PaperlayerView exact revision on arXiv

Source-linked AI summary

From Parameter Rebasin to Functional Transfer Across Heterogeneous Transformers

Filippo Rinaldi, Aniello Panariello, Giacomo Salici, Simone Magistri, Angelo Porrello, Andrew D. Bagdanov, Simone Calderara

arXiv:2610.32581v1cs.LG

TL;DR

Fine-tuned task updates are difficult and costly to reuse across Transformer architectures with different depths and widths. ARIADNE aligns source activation changes to a target base and fits a target-native update in closed form. It improves transfer across vision, language, and multi-task settings while reducing update-construction cost.

  • Problem

    Task-specific updates are tied to the parameterization of their source architecture, limiting reuse across different Transformer backbones and making retraining for each variant costly.

    Source: §1 · p. 1

  • Method

    ARIADNE aligns fine-tuning-induced activation displacements at corresponding blocks with local Procrustes maps, then uses closed-form ridge regression to synthesize target-native updates.

    Source: §1 · p. 2, §3.1 · p. 3, §3.2 · p. 5

  • Results

    ARIADNE improves vision, language, and multi-task transfer while substantially reducing update-construction time and memory relative to previous cross-architecture rebasin approaches.

    Source: §1 · p. 2, §5 · p. 10

  • Takeaways & Limitations

    The results support local functional change as a practical object for cross-architecture transfer.

    Source: §5 · p. 10

    Evaluation covers related vision and language backbones within two model families, not arbitrary architectures or modalities; the method also requires representative paired calibration inputs and block correspondences.

    Source: §5 · p. 10

Abstract

from arXiv · show

Adapting pretrained Transformers to downstream tasks produces task-specific updates that are expensive to reproduce for every model variant. Reusing these updates across different depths and widths is challenging because their parameters have incompatible shapes and do not share a canonical coordinate system. We introduce Ariadne, a training-free method that transfers task-specific behavior across heterogeneous Transformers without resizing the source model or transporting its parameters. Rather than treating the source task vector, i.e. the difference in parameter space between the fine-tuned and the base models, as the object to move, Ariadne characterizes fine-tuning by the activation changes it induces at matched blocks. First, it aligns these effects to a target base through local Procrustes maps. Then, it directly synthesizes a native target update by closed-form ridge regression. Across Transformer expansion, reduction and language-model transfer, Ariadne improves target-model performance while substantially reducing the time and memory required to construct an update relative to previous cross-architecture rebasin approaches. These results show that local functional effects provide an efficient alternative to parameter transport.

1 INTRODUCTION

Fine-tuned task behavior is tied to the source architecture, making reuse across different Transformer depths and widths difficult and retraining costly. ARIADNE transfers fine-tuning-induced activation effects into native target updates, avoiding parameter transport.

1 INTRODUCTION

  • Different Transformer parameter spaces prevent direct reuse of source task vectors, tying valuable fine-tuned model families to their original architectures.Retraining for a new architecture may be unavailable or prohibitively expensive and repeats substantial compute and energy use.Source: §1 · p. 1
  • Cross-architecture rebasin requires accurate parameter or global-functional correspondence and often heuristic depth resampling that may not preserve informative task vectors.More elaborate resampling can mitigate the issue but requires gradient access.Source: §1 · pp. 1–2
  • ARIADNE transfers source fine-tuning effects through local Procrustes alignment and closed-form ridge regression, producing target-native updates without transporting task vectors.The method aligns source and target base residual-stream states for matched blocks, then fits an update to a target component.Source: §1 · p. 2
  • ARIADNE formulates transfer across Transformer depths and widths as reproducing fine-tuning-induced functional effects rather than rebasining parameters.Its target update is synthesized directly from a source base and fine-tuned counterpart.Source: §1 · p. 2
  • The paper evaluates direct functional transfer across Vision Transformer expansion and reduction and across language-model variants.The stated evaluation scope includes multiple model sizes and architecture variants.Source: §1 · p. 2

2 RELATED WORK

Prior work reuses models through parameter alignment, representation matching, distillation, or architectural transformation. ARIADNE instead fits a native target update to the activation difference induced by source fine-tuning.

2 RELATED WORK

  • THESEUS and BiCo transport task vectors after architectural alignment, whereas ARIADNE fits a native target update to source fine-tuning’s aligned activation difference.THESEUS aligns input and output activations; BiCo uses output-side gradients instead of output activations.Source: §2 · p. 2
  • Activation-matching approaches include model stitching, relative representations, latent-space translation, representation metrics, RegMean, and PLeaS.RegMean minimizes output discrepancies when fitting merged linear layers, while PLeaS uses partial permutation matching and layer-wise least squares to reconstruct features.Source: §2 · p. 2
  • Distillation transfers teacher predictions without requiring parameter correspondence, while Transformer transformations and growth or compression methods reuse models through function preservation, continued training, or layer removal.TinyBERT also applies supervision to Transformer representations; Boomerang Distillation reinserts teacher blocks to obtain intermediate student sizes.Source: §2 · pp. 2–3
  • ARIADNE’s Figure 1 depicts source and target base activations aligned by component-wise Procrustes maps, followed by ridge regression that converts the desired shift into target-native updates.The mapped source fine-tuning effect becomes the desired target activation shift.Source: Figure 1 · p. 3

3 METHOD

Ariadne transfers fine-tuning effects across Transformer blocks instead of transporting source parameters, then synthesizes an update directly in the target model. It uses local alignment and regression to produce a sparse update in the target’s native parameterization.

3 METHOD

  • Ariadne compiles matched-block activation effects into a sparse task vector in the target’s native parameters, without resizing the source or requiring functional equivalence between base models.The update can be added to the target checkpoint; it is not obtained by transporting the source task vector.Source: §3 · p. 3, §3.2 · p. 5

3.1 BLOCK-LEVEL FUNCTIONAL CORRESPONDENCE

  • A user-specified, possibly partial block map pairs target blocks with source blocks; relative-depth pairing can reuse source blocks for expansion or omit them for reduction.The map may also be based on structural ancestry or domain knowledge.Source: §3.1 · p. 3
  • For each mapped pair, Ariadne aligns centered source and target base activations with a rectangular Procrustes map, then expresses the source fine-tuning displacement in target coordinates.A small calibration set supplies the activation banks; source activations may be interpolated when token grids differ.Source: §3.1 · p. 4
  • The desired target displacement is invariant to orthogonal changes in source hidden-state coordinates, unlike a parameter update that changes under such reparameterization.Ariadne transfers this local activation effect rather than recovering parameter-coordinate transformations across architectures.Source: §3.1 · p. 4
  • The gradient-alignment variant uses task-loss gradients instead of base activations and requires backward passes, so the paper treats it as an alternative to the default.Its aligned displacement remains unchanged under the stated orthogonal source-coordinate transformation.Source: §3.1 · p. 5

3.2 DIRECT COMPONENT REGRESSION

  • Ariadne turns each desired residual-stream displacement into an additive update to a selected target-block component using closed-form ridge regression.The component is chosen to absorb the effect encoded in the displacement; regressions use target-base inputs and are solved independently before assembly.Source: §3.2 · p. 5
  • The ridge scale is set from calibration statistics, and the resulting updates are compiled only for selected block components into the target task vector.For components with bias, the bias update is adjusted using the means of the displacement and component-input banks; bias-free components use zero bias.Source: §3.2 · p. 5

4 EXPERIMENTS

Across vision and language models, ARIADNE transfers task-specific effects across heterogeneous architectures, improving target performance while reducing update-construction cost. Calibration budget and alignment choice affect transfer quality, and multi-task gains depend partly on the source-side merge.

4 EXPERIMENTS

  • The experiments cover ViT-B/16 and ViT-L/14 transfers in both directions, plus Qwen2.5 instruction-following, mathematical-reasoning, and refusal-removal transfers at 0.5B, 1.5B, and 3B scales.The protocol compares THESEUS and BiCo in both directions and uses a shared calibration budget; task-vector and merge multipliers are selected on validation data.Source: §4 · p. 6

4.1 SINGLE-TASK VISION TRANSFER

  • Activation-based ARIADNE needs neither labels nor gradients, while gradient alignment can transfer better when labeled calibration data and backward-pass costs are acceptable.Increasing calibration from one to ten batches substantially improves activation-based transfer; larger budgets bring more modest gains.Source: §4 · p. 6, §4.1 · p. 7
  • With B = 1, ARIADNE surpasses the best B = 10 baseline in both vision extension and reduction; at larger budgets it improves over the target base by 10.68 and 12.00 points, respectively.At the larger budget, fitting takes 53 s for extension and 21 s for reduction, versus roughly 32 min and 14 min for parameter-transport baselines.Source: §4.1 · p. 6
  • Across vision tasks, transferred final-block CLS changes are generally most similar to native fine-tuning effects from the same task, indicating task-specific rather than generic changes.The comparison uses CKA between native target changes and transported source changes.Source: §4.1 · p. 7, Figure 2 · p. 7
  • Ablations indicate that Procrustes orthogonality, transferring the fine-tuning-induced displacement, regularization, and computationally ordered block matching all contribute to the default system’s performance.Unregularized fitting can be unstable with limited calibration and correlated features, while reversed or shifted block pairings reduce accuracy.Source: §4.1 · pp. 7–8

4.2 LANGUAGE-MODEL TRANSFER

  • For language-model extension, ARIADNE beats both baselines across tasks, improves the IFEval average by +18.15 points over the target base, and is the only method to improve GSM8K (+1.15).On HarmBench, it raises the non-refusal rate from 27.92% to 67.92%; reduction results are less pronounced, though ARIADNE performs best across settings.Source: §4.2 · pp. 8–9, Table 3 · p. 8

4.3 MULTI-TASK TRANSFER

  • ARIADNE alone improves the target average for both source merge rules and both transfer directions, but merging first is not uniformly as accurate as fitting vectors separately.TSV raises ARIADNE’s average from 75.68% to 79.11% in extension and from 68.21% to 71.62% in reduction; merging first uses one eighth as many fits, and some datasets remain at or below the target base.Source: §4.3 · p. 9, Table 4 · p. 9

5 CONCLUSIONS AND LIMITATIONS

ARIADNE transfers fine-tuning effects across Transformers of different depths and widths without parameter transport, improving transfer at lower construction cost. Evaluation is limited to related vision and language backbones in two model families and requires paired calibration inputs and block correspondences.

5 CONCLUSIONS AND LIMITATIONS

  • ARIADNE transfers fine-tuning effects across Transformers with different depths and widths by aligning activation changes and fitting target-native updates in closed form.The method avoids transporting parameters.Source: §5 · p. 10
  • Evaluation covers related vision and language backbones within two model families, so transfer across arbitrary architectures or modalities is not established.The method also requires representative paired calibration inputs and block correspondences.Source: §5 · p. 10
  • The authors report using generative AI to assist manuscript wording and code, with all AI-assisted text reviewed and code inspected and validated.They state that AI was not used to produce experimental data or make final scientific decisions.Source: AI USE STATEMENT · p. 10

A.1 CALIBRATION BUDGET AND STABILITY

More calibration batches improve average accuracy and stability, especially at small budgets, but the benefits vary by task and do not make every source effect useful to the target.

A.1 CALIBRATION BUDGET AND STABILITY

  • Moving from one to five calibration batches raises mean accuracy from 79.92% to 83.54% in extension and from 74.48% to 77.97% in reduction.At 50 batches, the means reach 85.68% in extension and 80.20% in reduction, with smaller subsequent gains.Source: §A.1 · p. 13
  • In reduction, average standard deviation falls from 0.98 with one batch to 0.24 with 50 batches, with EuroSAT and GTSRB fits becoming more consistent.Source: §A.1 · p. 13
  • Calibration gains are task-dependent: EuroSAT, SVHN, GTSRB, and MNIST improve, while Cars and SUN397 remain near their target-base values.More calibration reduces estimation error but cannot make every source effect useful to the target.Source: §A.1 · p. 13

A.2 TRANSPORT THEN MERGE

Transferring task vectors separately before merging improves target averages in most reported comparisons, but costs eight fits instead of one and does not improve every dataset.

A.2 TRANSPORT THEN MERGE

  • Separate transfer improves over the target base by 4.54–5.08 points in extension and 5.40–5.68 points in reduction for both merge rules.This strategy transfers each of eight task vectors independently before merging the target vectors.Source: §A.2 · p. 13
  • After separate transfer, TA and TSV differ by at most 0.53 points, less than when source vectors are merged before transfer.The authors suggest separate functional fits reduce source-side interference that makes the merge rule consequential.Source: §A.2 · p. 13
  • Separate transfer costs eight fits rather than one, and average gains coexist with small losses on some datasets.Gains on EuroSAT, SVHN, GTSRB, and MNIST coexist with small losses on some of DTD, Cars, RESISC45, and SUN397; the average is not Pareto improvement on every task.Source: §A.2 · p. 13

A.3 ACTIVATION AND GRADIENT ALIGNMENT

Gradient-derived alignment gains an advantage over activation-derived alignment as calibration budgets grow, while activation alignment remains label-free and competitive on some tasks.

A.3 ACTIVATION AND GRADIENT ALIGNMENT

  • At 20 calibration batches, gradient alignment is 2.41 points better in extension and 4.20 points better in reduction than activation alignment.With one calibration batch, the two alignment signals are essentially tied in both directions.Source: §A.3 · p. 13
  • Gradient alignment helps EuroSAT, GTSRB, RESISC45, and DTD, but activation alignment remains competitive on SVHN and MNIST at several budgets.Gradient alignment is useful when labels and backward passes are available; activation alignment allows label-free calibration.Source: §A.3 · p. 13

A.4 RESOURCE SCALING

Ariadne’s runtime advantage is largest with little calibration data and remains substantial as calibration grows, while host-memory growth is modest. Its speedup is concentrated in transformation rather than data collection or update application.

A.4 RESOURCE SCALING

  • At one batch, ARIADNE is about 70–77× faster for extension and 90× faster for reduction; at 20 batches, its advantage remains 32–35×.ARIADNE runtime rises from 13 to 234 seconds in extension and from 5 to 95 seconds in reduction between one and 50 batches.Source: §A.4 · p. 13
  • Host-memory growth is only about 20–23% across the calibration range, and ARIADNE uses substantially less host memory than either baseline at every jointly measured budget.The 50-batch baseline entries are unavailable, so no relative comparison is made at that budget.Source: §A.4 · p. 14
  • At B = 10, transformation takes 45 of ARIADNE’s 53 extension seconds versus roughly 1,850–1,880 baseline seconds; its speedup is concentrated in transformation.In reduction, ARIADNE’s 21 seconds are split almost evenly between collection and transformation, while baseline transformations take about 13 minutes; applying the fitted update is negligible for ARIADNE in both directions.Source: §A.4 · p. 14

A.5 TARGET FORMULATION AND REGULARIZATION

The target-formulation ablation favors transferring the source-induced activation change over reproducing its absolute fine-tuned endpoint. The tested ridge choices perform similarly over a broad range, while excessive shrinkage is costly.

Target formulation

  • Matching the source-induced change outperforms the absolute source fine-tuned endpoint by 8.14 points in extension and 3.66 points in reduction.The endpoint remains better than the target base on average, particularly in reduction; broad losses on EuroSAT, SVHN, GTSRB, and RESISC45 drive the larger extension penalty.Source: §A.5 · pp. 14–15, Table 11 · p. 17

Alignment

  • The alignment ablation compares activation-derived Qact with gradient-derived Qgrad across calibration budgets, but the supplied passage gives no outcome of that comparison.Source: Table 8 · p. 15

Regularization

  • Fixed trace-scaled ridge coefficients from 0.001 to 0.1 yield similar averages, while ρ = 10 loses 3.04 points in extension and 3.82 points in reduction versus the best fixed choice.The empirical-Bayes ridge is within 0.12 points of the best fixed coefficient in extension and 0.51 in reduction without validation-time tuning.Source: §A.5 · p. 15, Table 12 · p. 18

Related papers

Loading 2610.32581v1…