PaperlayerView exact revision on arXiv

Source-linked AI summary

Platonic Task Arithmetic

Junghwan Park, Woojin Cho

arXiv:2610.00929v2cs.LGcs.CVstat.ML

TL;DR

Weight-space task arithmetic cannot combine updates across models with incompatible parameter coordinates. This paper introduces architecture-independent Universal Task Descriptors and transfers them through a closed-form operator or a trained low-rank adapter. Cross-model transfer retains 74 to 80 percent of the gain from a target’s own descriptors, despite a substantial model-specific residual.

  • Problem

    Weight-space task arithmetic requires task updates to share an architecture and initialization, limiting composition across differently built models.

    Source: §1 · p. 1, §3.1 · p. 3

  • Method

    Universal Task Descriptors record fine-tuning’s affinity changes in architecture-independent matrices, which the paper transfers by matching a target edit to the descriptors.

    Source: §1 · p. 2

  • Results

    Across six model families and eight tasks, cross-model transfer retains 80% and 74% of the gain achieved by target-specific descriptors.

    Source: §1 · p. 2

  • Takeaways & Limitations

    Both the closed-form operator and adapter routes transfer tasks across models, including to audio–text models.

    Source: §1 · p. 2

    Both realizations require semantic prompts and unlabeled probes for each task, so they are label-free but not data-free.

    Source: §4.5 · p. 7, §6 · p. 10

Abstract

from arXiv · show

Distinct pre-trained models specialized for the same task converge to closely similar behavior, yet the parameter updates that produce it share no common coordinate system. Weight-space task arithmetic is therefore confined to a single model, and transporting an update between models requires a structural correspondence. Drawing on Plato's allegory of the cave, we hypothesize that these model-specific updates are shadows cast by one shared, model-agnostic object, the platonic task vector. To make it operational across models of different architectures, we introduce Universal Task Descriptors, matrices whose shape is independent of architecture and embedding dimension, which record a task's functional effect and admit addition and negation as ordinary matrix operations, and we transfer a descriptor into a target in two ways. A single least-squares solve returns a linear operator folded into the target's last layer, and a bank of such operators, one per source and task, realizes any composition as a signed sum of its entries. Alternatively, a low-rank adapter of the target's encoder is trained on the same objective at the price of one optimization per edit. Despite a model-specific residual comparable in norm to the shared component, transfer from another model retains 74 to 80 percent of the gain the target's own descriptors attain. Experiments across six model families, eight tasks and audio-text models confirm both realizations.

1 Introduction

The paper argues that cross-model task arithmetic is blocked by incompatible parameter coordinates, not by a lack of shared task knowledge. It introduces architecture-independent descriptors and two ways to transfer and compose them in a target model.

Motivation

  • Weight-space task arithmetic requires task vectors to share an architecture and initialization, while structural transport methods apply only to structurally corresponding models.The paper frames this restriction as increasingly consequential as vision–language models diversify across architectures, training data, and objectives.Source: §1 · p. 1
  • The proposed platonic task vector treats model-specific task updates as shadows of a shared, model-agnostic object.The paper asks whether the convergence proposed for model representations also governs what fine-tuning changes.Source: §1 · pp. 1–2

Approach

  • Universal Task Descriptors record fine-tuning’s functional effect in architecture- and embedding-dimension-independent matrices that can be added, negated, and combined.The closed-form transfer is a least-squares solve folded into the target’s last layer; alternatively, a low-rank adapter is trained on the same objective.Source: §1 · p. 2

Results

  • Across six model families and eight tasks, cross-model transfer retains 80% and 74% of the target’s own-descriptor gain, while negation removes 35.2 points for 4.5 on the same control.The closed-form route also transfers two tasks from other models with gains of 18.1 and 11.6 points for about two points of CIFAR-100.Source: §1 · p. 2

Evaluation

  • The evaluation covers addition and negation across eight tasks, six source families, and six targets, and also tests a dataset that enters no fit and audio–text models.The comparison includes weight-space composition, weight transport, gradient ascent, distillation, and supervised tuning.Source: §1 · p. 2

2 Background and Motivation

Task arithmetic composes, negates, and analogizes fine-tuning updates, but its parameter-space operations require updates from the same base model. The paper motivates a model-agnostic alternative by measuring how task-specific fine-tuning changes image–text affinity across models.

Task arithmetic

  • A weight-space task vector is defined relative to a base model’s parameters, confining addition, negation, and analogy to updates from the same initialization.Task arithmetic composes by adding updates, negates by subtracting an update with strength λ, and forms analogies from differences between task vectors.Source: §2 · p. 3

Cross-model consistency

  • Same-task descriptors across models have mean cosine 0.473 ± 0.066, versus 0.227±0.169 for different tasks within one model.The cross-task pattern is structured: EuroSAT and RESISC45 agree at 0.573, while Cars and SUN397 agree at −0.149.Source: §2 · p. 3

3 Universal Task Descriptors and the Platonic Hypothesis

The paper posits a shared task-displacement object whose model-specific updates are shadows, then measures task effects with Universal Task Descriptors. Its cross-model consistency hypothesis is supported by greater same-task agreement across models and by transfer results, but retains a substantial model-specific residual.

3.1 The Platonic Task Vector

  • A platonic task vector is a model-independent task-displacement map whose projection into each model’s coordinates is that model’s task update.Platonic task arithmetic adds or subtracts these shared task vectors regardless of which models learned them.Source: §3.1 · p. 3

3.2 Universal Task Descriptors

  • A Universal Task Descriptor measures fine-tuning-induced changes in image–prompt affinities using task probes and prompts derived from class names.Its N × M shape is independent of embedding dimension and architecture, and differencing removes each model’s pretrained affinities.Source: §3.2 · p. 4

3.3 The Cross-Model Hypothesis

  • Same-task descriptors across models agree at mean cosine 0.473, versus 0.227 for different tasks within a model; cross-model transfer retains 80% and 74% of target gains.The model-specific residual is comparable in norm to the shared component, so consistency is clear but imperfect.Source: §3.3 · p. 4

4 Cross-Model Task Arithmetic and Transfer to a Target

The paper represents task operations as signed combinations of descriptors and transfers them by matching those descriptors on a target’s probes. A closed-form linear edit supports reusable operator banks, while a low-rank adapter offers a richer jointly fitted edit at a per-edit optimization cost.

4.1 Addition and Negation in UTD Space

  • Negating a descriptor removes the fine-tuning-induced change rather than an absolute output configuration, leaving pretrained behavior untouched by construction.The transfer uses only the removed task’s descriptor, without control data or a preservation term.Source: §4.1 · p. 5

4.2 Transfer as Descriptor Matching

  • Descriptor matching seeks a target edit whose induced descriptor matches each prescribed task descriptor on that task’s probes and prompts.The objective weighs probe images equally and does not compare matrices of different shapes.Source: §4.2 · p. 5

4.3 The Descriptor Factorizes, and Transfer Is Linear

  • The descriptor observes an embedding map only through its projection onto span(G), making descriptor reproduction weaker than reproducing a fine-tuned encoder.The linear edit is modeled as input-independent so it can be folded into a target weight layer.Source: §4.3 · p. 5

4.4 Composing by Adding Operators

  • The closed-form solution map is linear in the descriptor, enabling task operators to compose as signed sums; numerically, the identity holds to relative difference below 5 × 10^-14 in double precision.The result applies at full rank, not to the nonlinear rank-capped variant.Source: §4.4 · p. 6
  • On a shared probe set, signed combinations of task operators compose exactly, but operators estimated on different probe sets can interfere without a smallness guarantee.The margin condition certifies 47.8% of evaluation images on 2,240 cells and predicts observed loss at rank correlation 0.66.Source: §4.4 · p. 6

4.5 Realization by a Low-Rank Adapter

  • A low-rank adapter fits all terms jointly across the target’s vision encoder, reaching every target and allowing richer edits, but requires one optimization per edit.Each edit takes about ten minutes on one GPU for a pair of tasks, and no reusable bank is stored.Source: §4.5 · p. 6

5 Experiments

Experiments evaluate descriptor-based task addition and negation across six model families and eight tasks, with comparisons spanning held-out datasets, alternative transfer routes, and audio–text models. Cross-model operators compose tasks and transfer substantial gains, while the experiments also expose interference and preservation trade-offs.

5.1 Experimental Setup

  • Six Large-scale model families and eight tasks were evaluated; rank-4 sources used unlabeled training probes, with no evaluation images included as probes.CIFAR-100 and ImageNet served as controls; adapter and closed-form strengths and fitting conditions were fixed across the experiments.Source: §5.1 · p. 7

5.2 Cross-Model Task Addition

  • 18.11 and 11.63 points over zero-shot are the two-task gains from cross-model composition, with paired costs of 1.47 ± 2.78 and 1.25 ± 3.14 points versus single-source transfers.The closed form acts at α = 0.5; Table 1 reports addition in percent against each block’s zero-shot row.Source: §5.2 · p. 7, Table 1 · p. 8

5.3 Cross-Model Task Negation

  • The negation sweep plots accuracy against strength λ, with λ = 0 marking the unedited target and selectivity defined by the removed task falling faster than controls.Figure 2 covers 160 cross-model cells from four operator-absorbing targets, a narrower population than Table 2.Source: §5.3 · p. 8, Figure 2 · p. 8

5.4 Reaching a Dataset That Enters No Fit

  • With both terms from one other family, the closed form gains 5.67 points over zero-shot; the second term adds 0.57 ± 1.04 points and leads on 13 of 18 cells.Table 3 evaluates compositions on datasets that enter no fit.Source: §5.4 · p. 9, Table 3 · p. 9
  • Across held-out dataset compositions, the adapter stays within 0.6 points of the closed form across families and reaches 76.47 with two distinct families.Within the target’s own family, the adapter leads the closed form by 2.1 points; CIFAR-100 stays within three points.Source: §5.4 · p. 9, Table 3 · p. 9

5.5 Comparison With Alternative Routes

  • On 168 architecture cells, the closed form lifts tasks by 20.9 and 15.7 points and the adapter by 22.0 and 20.0, exceeding GradFix and Theseus; negation comparisons show different control costs.At λ = 1, closed-form negation removes 36.3 points for 2.7 of CIFAR-100, while weight-space negation removes 30.1 for 13.4; architecture-cell GradFix spends 16.3 CIFAR-100 points versus 4.8 and 0.6 for the two descriptor routes.Source: §5.5 · pp. 9–10

5.6 Generality to Audio–Text Models

  • On four CLAP checkpoints with different widths and depths, the closed form lifts ESC-50 by 1.66 points and GTZAN by 2.33 while keeping UrbanSound8K within half a point.At λ = 1, it removes 14.4 points for 2.0 of UrbanSound8K; the adapter removes 11.1 for 1.0.Source: §5.6 · p. 10

6 Conclusion and Limitations

This section concludes the study of architecture-independent functional descriptors transferred through target-specific edits and examines limitations from cross-model residuals and task interference. It distinguishes the approach from prior methods, diagnoses residuals and composition, and discusses rank constraints, descriptor equivalence classes, and a ridge-regularized pairwise regression solve.

6 Conclusion and Limitations

  • The framework replaces coordinate-based task vectors with architecture-independent descriptors, transferred by a closed-form last-layer operator or a more accurate adapter.The adapter keeps ImageNet closer to zero-shot but requires one optimization per edit. The residual remains comparable in norm to the shared component; semantic prompts and unlabeled task probes mean the methods are label-free, not data-free, and exclude single-modality models.Source: §6 · p. 10

B Related Work

  • Unlike task-arithmetic methods tied to shared weight spaces [Ilharco et al., 2022], this work transfers functional changes; unlike output-matching distillation [Hinton et al., 2015], its source-relative descriptors compose by addition and negation.The least-squares edit fits an operator on the target’s embedding to reproduce a descriptor, rather than aligning model parameters or activations. The paper frames its question as whether fine-tuning changes, like pre-trained representations, share structure across architectures.Source: B Related Work · pp. 15–16

C.1 Derivation

  • The closed-form transfer is a pairwise kernel regression with two factor-specific ridge filters, and cached target-task decompositions build a 48-operator bank in 17 seconds.A solve takes well under a second; the regression is evaluated on new images and, in some settings, new prompts as well.Source: §C.1 · pp. 16–17

C.3 Composition Across Grids

  • Composition has no assumption-free guarantee: independently solved operators can leak across task grids, and the resulting departure from each term’s task behavior is not guaranteed to be small.The conclusion also notes that transfer gains at most two points on Cars and SUN397, and that reducing transfer costs can move undesirable capabilities too; negation is the corresponding removal tool.Source: §6 · p. 10, §C.3 · pp. 18, 21
  • Across 4,480 records, no prediction covered by the composition guarantee changed; the bound was 22.7% against an observed loss of 1.45 points.The guarantee covered 47.8% of evaluation images, and the observed loss was below the bound in every record.Source: Table 7 · p. 20, §C.3 · p. 20

C.4 Where the Cross-Model Residual Reaches the Edit

  • The solve amplifies descriptor residuals by a median factor of 1.5, and operator agreement with the target’s own fine-tune is lower than descriptor agreement.Mean cosine agreement is 0.39 for operators versus 0.59 for descriptors; shift-field agreement is 0.43. The residual’s effect depends on the target’s factors, which can suppress or amplify it.Source: §C.4 · p. 21
  • −0.83 for operators versus −0.48 for descriptors: across eight tasks, operator agreement better tracks the fraction of attainable gain transport forfeits.Cars and EuroSAT have similar descriptor agreement (0.81 and 0.79) but forfeit 61% and 11% of attainable gain, respectively; after the solve, operator agreement is 0.07 for Cars and 0.71 for EuroSAT.Source: Table 9 · p. 22, §C.4 · p. 22

C.5 Is a Low-Rank Prior Helpful?

  • At rank 16, transferred-task accuracy is 75.91% versus 74.15% at that cap, while SVHN reaches 47.64% at its selected cap, supporting no rank cap.The cap is selected by held-out descriptor reconstruction without labels or evaluation data; rank truncation is nonlinear and would forfeit the solve’s linearity for scaling and composition.Source: §C.5 · p. 23

C.6 What the Descriptor Does Not Determine

  • A descriptor determines an equivalence class of shift fields, and tested Procrustes transport and prompt-Gram whitening both weaken transfer or discriminability.The descriptor constrains shifts only within span(G); ridge selects a minimum-norm representative rather than a uniquely determined field.Source: §C.6 · p. 23

C.2 Proof of Proposition 1

The appendix checks the same-grid linearity behind composing solved operators, derives the shared-descriptor hypothesis under PRH assumptions, and reports the residual diagnostic and experimental setup. It also clarifies that independently solved operators may interfere across grids and describes the operator-bank composition procedure.

C.2 Proof of Proposition 1

  • Below 5 × 10^-14 in double precision and 4 × 10^-7 at stored single precision, the same-grid operator sum matches the composed descriptor; cross-grid bank sums may interfere.The numerical check compares the operator for U1 − 1.5 U2 + U3 with the same combination of the three solved operators; the cross-grid caveat follows because each entry is constrained on its own task’s grid.Source: §C.2 · pp. 17–18, Table 6 · p. 17

D The Hypothesis under the PRH Factorization

  • Under the PRH factorization, the shared-task hypothesis follows to first order only after positing one displacement across models; its residual depends on displacement size and model-specific isometry defects.The factorization does not establish that training produces the same displacement in every model; that premise concerns training dynamics.Source: D The Hypothesis under the PRH… · p. 24

D.1 Measuring the Residual

  • 0.99 is the mean plug-in dispersion across evaluated model-task pairs; pairwise cosine 0.473 corresponds to residuals near the shared component’s norm only under an orthogonality-and-equal-norm idealization.The diagnostic has family means from 0.77 to 1.27 and supplies no upper bound on the residual ratio.Source: §D.1 · pp. 24–25

E.1 Model Checkpoints

  • Sources use CLIP-style dual encoders and fine-tune only vision self-attention query and value projections, while text encoders remain frozen; SigLIP targets receive edits through the head’s last affine.The edit through the last affine is needed for the SigLIP families’ residual-sum attention head.Source: §E.1 · p. 25, Table 11 · p. 25, §E.2 · p. 25

E.2 Sources, Probes and Hyperparameters

  • Task grids use unlabeled training-split probes and class-name prompts, while evaluation uses disjoint held-out test data; random image sampling represents classes only in expectation.Grid sizes range from 300 images and 10 prompts for a ten-class task to 11,910 images and 397 prompts for SUN397.Source: §E.2 · pp. 25–26

E.3 Algorithms

  • The operator bank composes edits as signed sums on one target embedding space without another solve; when entries share a grid, their sum is also the operator for the composed descriptor.The atomic-solve procedure reuses task-level decompositions across source descriptors, while absorption costs one d × d matrix product.Source: §E.3 · p. 26

F The Adapter Realization

The adapter fits target edits to prescribed descriptor shifts, while the section evaluates its fit and cost alongside closed-form transfer across strength, targets, tasks, and controls.

F The Adapter Realization

  • The adapter matches descriptor shifts using a low-rank target-encoder edit, with one optimization per edit.The text encoder and other weights are frozen; the adapter is attached to query and value projections in every vision-encoder attention layer.Source: F The Adapter Realization · p. 27
  • Median adapter residuals are 0.041 for 438 addition cells and 0.077 for 134 negation cells, while median edits take about ten and four minutes, respectively.The closed form solves a target’s bank of 48 operators in seconds; each subsequent cell is a matrix sum.Source: F The Adapter Realization · p. 27

G.1 Strength Sweep

  • At α = 0.5, acquisition peaks across the two composed tasks, while control damage continues to grow.Task A gains only 0.3 points more at α = 1, while task B loses 1.2 points.Source: §G.1 · p. 28, Table 12 · p. 28

G.2 Per-Target Breakdown

  • The four targets that absorb the operator gain composed tasks by 13.04 to 18.17 points, while EVA-CLIP pays roughly five times what OpenCLIP pays on ImageNet.Targets receiving an update to a pooling head gain a little less, but their control changes remain within the range of the other targets.Source: Table 13 · p. 28

G.3 Per-Task Breakdown

  • Six of eight tasks retain 62–87% of their Oracle gain, and every one of 180 target-source-task combinations gains on average for those six tasks.All 38 combinations with nonpositive mean gain occur on Cars or SUN397.Source: §G.3 · p. 28, Table 14 · p. 29

G.4 Per-Cell View

  • On OpenCLIP-L, composed cells move up and right together from zero-shot, showing gains on both tasks that a single-source edit cannot produce.Figure 3 plots each cell by accuracy on the two composed tasks, normalized by their fine-tuning ceilings.Source: §G.4 · p. 29, Figure 3 · p. 29

G.5 The Cost of Acquiring a Task

  • Two-operator composition costs 1.70 points on controls, within the −7.14-point average control cost of single-task fine-tuning, but the operator costs 3.92 ImageNet points versus −0.29 for fine-tuning.The −1.70-point composition cost is reported on the same four targets at α = 0.5.Source: §G.5 · pp. 29–30, Table 15 · p. 30

G.6 Multi-Task Scaling

The multi-task sweep adds stored operators in varying task subsets on four targets. Acquisition generally persists as tasks accumulate, but control costs and gains vary by target.

G.6 Multi-Task Scaling

  • Each sweep prefix sums the same stored task operators, so its band reflects which J tasks are included, not their arrival order.The figure uses three task orderings and one source family per task on each of four targets.Source: §G.6 · p. 31, Figure 4 · p. 31
  • At α = 0.5 on CLIP-L, mean composed-task gain stays between +9.78 and +19.21 points across J = 1 to 8, while CIFAR-100 falls from −1.27 to −15.30.ImageNet moves from −2.41 points at J = 1 to −16.23 at J = 8.Source: §G.6 · p. 31, Figure 4 · p. 31
  • At J = 8, α = 0.0625 restores CLIP-L controls to +0.50 CIFAR-100 and −0.32 ImageNet points, while composed-task gain falls to +4.08.This uses α = 0.5/J; at fixed α, the summed operator’s Frobenius norm grows with J.Source: §G.6 · p. 31
  • At J = 8 and α = 0.5, acquisition remains positive on OpenCLIP-L, MetaCLIP-L, and CLIP-L, but EVA-CLIP-L’s mean gain falls to −4.41 points.On EVA-CLIP-L, ImageNet gives up 31.30 points; the passage characterizes accumulation as stopping pay off well before eight tasks on that target.Source: §G.6 · p. 31

H Full Negation Results

Negation removes a targeted task while affecting control benchmarks; the amount of erasure, retention, and cost depends on strength and target.

H Full Negation Results

  • Across 240 cross-model cells, λ = 0.5 removes 23.04 points of the targeted task for 1.00 CIFAR-100 and 1.73 ImageNet points, versus 35.22, 4.55, and 6.66 at λ = 1.The table covers 160 cross-model and 32 same-model cells over the same eight tasks and four targets.Source: H Full Negation Results · p. 32, Table 16 · p. 32
  • From λ = 2 to λ = 4, erasure rises from 15.4% to 7.3% remaining, while CIFAR-100 falls from 58.9 to 34.2 and ImageNet from 47.0 to 21.6.The passage identifies the last points of erasure as especially costly and recommends the smallest λ that reaches the desired erasure.Source: H Full Negation Results · p. 32
  • At λ = 1, OpenCLIP-L gives up 1.9 CIFAR-100 points for 27.6 points of erasure, while the other three targets give up 4.7 to 8.1 points for 31.3 to 38.2 points erased.OpenCLIP-L retains the most at every swept strength and for every source family.Source: H Full Negation Results · p. 32

I Sensitivity to the Source Fine-Tuning Seed

The seed-sensitivity experiment repeats source fine-tuning and transfer with other settings held fixed. Reported effects vary little across seeds, consistent with agreement in descriptors despite weak weight-update agreement.

I Sensitivity to the Source Fine-Tuning Seed

  • Across three seeds, composing two tasks yields EuroSAT 85.79 ± 0.23 and DTD 65.02 ± 0.13 percent, while negation leaves the removed task at 12.83 ± 0.08 at λ = 1.Controls move by less than a tenth of a point across seeds.Source: I Sensitivity to the Source… · p. 33, Table 17 · p. 33
  • Across cells, seed-related variation is at most 0.7 points, and the study’s own source rows remain within 0.6 points of the seeded means on every axis.The experiment varies only the low-rank update produced by source fine-tuning; probes, prompts, strengths, evaluation splits, and controls remain fixed.Source: I Sensitivity to the Source… · pp. 32–33
  • Across 18 seed pairs, task descriptors agree at cosine 0.93–0.97, while merged weight updates agree at only 0.14–0.24.The passage attributes low sensitivity in part to the solve damping directions that the target’s probes and prompts constrain least.Source: I Sensitivity to the Source… · p. 33

J Reaching a Dataset That Enters No Fit, in Full

Composing descriptors from other model families improves performance on a dataset that supplies neither source terms nor fit data. The closed form tracks the within-model reference, while the adapter performs better on the pooled assignments and stays close to control accuracy.

J Reaching a Dataset That Enters No Fit, in Full

  • Compositions lead their first transferred term alone on RESISC45 (+2.88 vs +2.34), UCMerced (+9.80 vs +9.04), and AID (+4.35 vs +3.93).The evaluated remote-sensing dataset enters neither fit through its images nor through its class names.Source: J Reaching a Dataset That Enters No… · p. 34
  • CIFAR-100 moves by at most three points across the rows, from 73.48 zero-shot to between 70.49 and 74.45.On every row, the composition’s control movement is smaller than its gain on the evaluated dataset.Source: J Reaching a Dataset That Enters No… · p. 34
  • 4.76–7.65 points: every cross-model composition beats zero-shot on the dataset that enters neither source term nor fit.The within-model oracle reaches 90.21, so related-task operators partially substitute for, but do not replace, the target’s own descriptor.Source: J Reaching a Dataset That Enters No… · p. 34, Table 18 · p. 34
  • 76.63 with both terms from one other family and 76.64 with one term from each of the other families: the adapter exceeds the closed form’s 76.08 and 76.47.The adapter reaches 80.16 within the model, against 78.06 for the closed form.Source: J Reaching a Dataset That Enters No… · p. 34
  • At full strength, the adapter keeps CIFAR-100 within 0.3 points of its 73.50 unedited accuracy on every row; at strength 0.5, cross-family adapter results fall below the closed form.At strength 0.5, the adapter reaches 75.47 with both terms from one other family and 74.26 with the first term alone, versus 76.08 and 75.51 for the closed form.Source: J Reaching a Dataset That Enters No… · pp. 34–35

K The Analogy Operation

The section tests task analogy and compares routes by their structural correspondence and target-side signal requirements. In the cross-family addition setting, the descriptor routes outperform transport baselines; in the tested analogy products, the subtraction offers little benefit and stronger subtraction harms results.

K The Analogy Operation

  • The analogy experiment evaluates each held-out task across six model families and compares the assembled expression with its two positive terms alone and an oracle.The two products cross domain and subtask axes; the second uses a semantic split and a narrower domain gap intended to favor the operation.Source: K The Analogy Operation · p. 35, Table 19 · p. 36

L.2 Protocols

  • The protocols fix route strengths in advance; the analogy comparison uses identical cells and scores a seeded 1,200-image held-out-task split plus 500 CIFAR-100 images.The analogy products are fine-tunes of one base model, so weight-space arithmetic is defined without cross-model transport.Source: K The Analogy Operation · p. 35, §L.2 · p. 38

K The Analogy Operation

  • At λ = 1, analogy gains 0.91 and 0.73 points in weight space and 0.53 and 0.39 in the descriptor route across the two products, but margins over the better single term are about a tenth of a point or less.Across 24 cells per product, those margins are smaller than the 0.87-point standard error of a single entry.Source: K The Analogy Operation · p. 36, Table 19 · p. 36
  • At α = 1, analogy falls below zero-shot on both products and both routes by 0.19–1.05 points; setting λ = 0 is better on 14, 15, 18, and 16 of 24 cells across products and routes.The subtracted term is at best unnecessary and more often harmful in these tests.Source: K The Analogy Operation · p. 36

L.1 What Each Route Requires

  • Table 20 compares routes by structural correspondence, target-side signal, supported operations, and whether cross-family use is defined; N/A means undefined, not a failed run.The descriptor routes require no structural correspondence and differ in whether they solve an operator for the last layer or fit an adapter of the encoder.Source: §L.1 · p. 37, Table 20 · p. 37

L.3 Addition Against the Transport Methods

  • 13.7 and 16.9 points: GradFix improves the two composed tasks, while the closed form leads it by 7.2 points on the first and trails by 1.1 on the second.The adapter leads GradFix by 8.2 and 3.2 points and stays within 1.1 and 0.3 points of zero-shot on CIFAR-100 and ImageNet; these are 168 cells with three distinct ViT-L/14 width-1024 families.Source: §L.3 · p. 38

L.4 Negation Against Weight-Space Negation and Gradient Ascent

Negation routes trade off erasure of the targeted task against preservation of controls, with results depending on each route’s requirements. Descriptor transfer supports negation across model families, while distilling a negated teacher’s outputs also degrades CIFAR-100 substantially.

L.4 Negation Against Weight-Space Negation and Gradient Ascent

  • The table reports changes from each route’s own zero-shot control draw, and each baseline appears only on the cells where it is defined.Weight-space negation requires the source and target to coincide, transport methods require corresponding parameters, and gradient ascent requires labeled examples.Source: §L.4 · p. 39, Table 21 · p. 40
  • At λ = 0.5, same-model descriptor negation removes 24.8 points versus 22.0 for weight-space negation, while giving up 0.4 CIFAR-100 points versus 4.2.Its ImageNet cost is 1.1 points versus 0.6 for weight-space negation.Source: §L.4 · p. 39
  • 35.3 points: GradFix removes more of the targeted task than the cross-model descriptor at λ = 1, but gives up 16.3 points of CIFAR-100 versus 4.8.At λ = 0.5, the cross-model descriptor removes 23.5 points for 1.0 of CIFAR-100; the adapter removes 28.5 for 0.6 and 0.7 of ImageNet.Source: §L.4 · p. 39, Table 21 · p. 40
  • Gradient ascent takes the removed task from a 56.4% mean to 7.7%, while CIFAR-100 falls to 26.6% and ImageNet to 55.7%; the larger budget reaches 4.1% on the removed task.The larger budget leaves CIFAR-100 at 3.6% and ImageNet at 28.1%.Source: §L.4 · p. 39

L.5 Addition and Negation Against Distillation and Supervised Tuning

  • Posterior distillation reaches 96.36 and 77.62 on the composed tasks, versus 92.45 and 64.41 for the adapter, but gives up 6.75 CIFAR-100 points versus 0.47.The routes are compared on six EuroSAT and DTD addition cells and twelve negation cells; trained rows share the adapter recipe and differ in loss.Source: §L.5 · p. 41, Table 23 · p. 42
  • Negated-teacher distillation takes CIFAR-100 from 73.30 to 7.40 and 14.29, while descriptor negation at λ = 1 removes 41.0 task points for 2.3 CIFAR-100 points with the closed form.The adapter removes 33.4 task points for 1.0 CIFAR-100 point.Source: §L.5 · p. 41

M Targets That Pool Through a Residual Head

For SigLIP and SigLIP2 residual-head targets, the closed-form update can be solved directly for the head while preserving linear composition. The head route works closely to the unrestricted solve, while the adapter performs better on composed tasks in the reported comparisons.

M Targets That Pool Through a Residual Head

  • The head update is obtained directly because changing W2 and b2 shifts the embedding by ∆W2u(x) + ∆b2 when upstream parameters are fixed.The solve uses normalized rows [u(x_i); 1]/∥z(x_i)∥, including the bias term.Source: M Targets That Pool Through a Residual… · p. 42
  • 16.43 and 7.07 points on SigLIP-L, and 16.55 and 9.30 on SigLIP2-L: head edits lift the two composed tasks, with controls costing at most 2.5 points.At λ = 1, negation removes 39.5 and 40.1 points for 2.5 and 4.5 CIFAR-100 points, respectively.Source: M Targets That Pool Through a Residual… · p. 43, Table 24 · p. 43
  • For SigLIP-family targets, the head edit trails the same solve applied outside the model by 0.5–1.2 points on composed tasks and gives up 0.3–0.6 more control points.The head edit touches only the head’s existing weights, adding no layer or parameters.Source: M Targets That Pool Through a Residual… · pp. 42–43
  • The adapter leads the head edit on composed tasks by 1.42 points on SigLIP-L and 1.88 on SigLIP2-L, while its CIFAR-100 comparison is 0.19 points ahead on SigLIP-L and 3.10 behind on SigLIP2-L.The head edit reproduces cached-quantity accuracy to a maximum absolute embedding difference of 1.2 × 10^-7, with identical top-1 predictions; solution linearity differs by 6.9 × 10^-8 on a mixed-sign combination.Source: M Targets That Pool Through a Residual… · pp. 42–43

N Targets at Another Scale

Universal Task Descriptors transfer across embedding scales without retraining the target or requiring a source at its scale. Cross-scale additions and negations retain substantial effects, with larger target dimensions generally paying a smaller control-task cost.

N Targets at Another Scale

  • Descriptors are N × M regardless of embedding dimension, and transferring them requires only the target’s probe-image and prompt forward pass.The 48 Large descriptors are reused verbatim for targets at other scales, with each operator solved on its task’s probes and prompts.Source: N Targets at Another Scale · p. 43
  • Negation removes 20.80–30.19 points across scales at λ = 0.5, compared with 22.13 points for Large targets.The table has no oracle or same-model row because those would require a source fine-tune on the target itself.Source: N Targets at Another Scale · p. 44, Table 25 · p. 44
  • At α = 0.5, 1024-dimensional targets lose 1.27 and 1.10 CIFAR-100 points, versus 1.70 at Large and 4.61 and 6.99 at 512 dimensions.ImageNet losses follow the same ordering: 0.94 and 2.00 at 1024 dimensions, 3.92 at Large, and 9.38 and 13.80 at 512.Source: N Targets at Another Scale · p. 44
  • The scale comparison cannot isolate dimension from model because the two Base targets are also the weakest zero-shot models.The results establish that another-scale targets need nothing retrained, not that dimension alone explains the performance spread.Source: N Targets at Another Scale · p. 44
  • Addition gains 11.38–16.83 points across cross-scale targets, versus 16.13 for Large targets; Table 25 reports these results at α = 0.5.The largest gain, +16.83, is on a target with 50.28 percent composed-task zero-shot accuracy; the target starting at 70.31 gains +11.38.Source: N Targets at Another Scale · p. 43, Table 25 · p. 44

O Ablations and Negative Results

The ablations show that task accuracy is not reliably predicted by shift-field reconstruction, and that probe quantity and the descriptor’s landmark frame matter. Several plausible alternatives underperform the closed-form construction, while the approximate fitting objective acts as a regularizer.

O Ablations and Negative Results

  • 76.01 versus 65.46 transferred-task accuracy favors the closed form over Procrustes, even though Procrustes better reproduces the target’s shift field.The corresponding shift-field cosines are 0.199 for the closed form and 0.358 for Procrustes; CIFAR-100 is 76.23 versus 74.75.Source: O Ablations and Negative Results · p. 44
  • Prompt-Gram whitening leaves fitted accuracy within 0.02 points but reduces same-task cross-model agreement from 0.47 to 0.10.The different-task control falls from 0.23 to 0.15, shrinking the discriminative gap from 0.25 to −0.05; shared prompt-frame structure explains why whitening harms descriptor comparison.Source: O Ablations and Negative Results · p. 45
  • Text-space alignment reaches 68.17 transferred-task accuracy versus 76.01 for the closed form, despite the non-orthogonal map reconstructing held-out words better.The 12,000-word vocabulary is task-agnostic; its non-orthogonal alignment has held-out reconstruction error 0.477 versus 0.576 for the orthogonal map.Source: O Ablations and Negative Results · p. 45
  • The unconstrained operator reaches 76.01 transferred-task accuracy and 76.23 CIFAR-100, exceeding every tested subspace-constrained alternative.At q = 512, the constrained result is 72.87 and 72.60; leading feature directions carry 59–68 percent of feature energy, so the tested subspace does not separate behavior to change from behavior to preserve.Source: O Ablations and Negative Results · p. 45
  • The exact first-order objective lowers residual error but can sacrifice control accuracy, whereas the approximate objective is retained as a regularizer.The corrected solve reaches 80.05 transferred-task accuracy for 67.08 CIFAR-100 at α = 1; at a comparable control score, it trails the closed form, and smaller ridge can sharply reduce control accuracy.Source: O Ablations and Negative Results · pp. 45–46
  • 75.69 to 69.99 task accuracy follows reducing each task’s probe pool from 2,274 to 284 images, while CIFAR-100 changes by only 0.2 points.Successive pool doublings add 2.5, 1.9, and 1.3 task points, but held-out reconstruction barely changes from 1.196 to 1.164 and cannot choose probe count.Source: O Ablations and Negative Results · p. 46

P Audio–Text Instantiation

This section tests task composition and negation in an audio–text instantiation across four CLAP checkpoints, evaluating both the closed-form operator and a descriptor-trained adapter. It also states that LLMs were not an important, original, or non-standard component of the research methodology.

P Audio–Text Instantiation

  • Four CLAP checkpoints serve as both sources and targets, with tower widths of 768 or 1024 and depths of 12 or 18 blocks; weight-space task vectors are undefined across the two classes.The operator folds into the projection head’s final linear layer and bias; ESC-50 and GTZAN are composed, while UrbanSound8K is reserved for retention measurement.Source: P Audio–Text Instantiation · p. 46
  • Across six cross-model addition cells, ESC-50 rises in every cell and GTZAN rises in five, while the never-composed control moves by at most 0.8 points.Table 27 reports per-cell ESC-50, GTZAN, and UrbanSound8K accuracy at α = 0.5.Source: Table 27 · p. 47
  • At λ = 1, the operator removes 14.37 points of the targeted task for 2.03 points of UrbanSound8K, versus 15.08 and 1.85 for the same-model reference.At λ = 0.5, the operator removes 7.03 points of the targeted task for 0.78 points of UrbanSound8K, versus 8.66 and 0.60 for the same-model reference.Source: P Audio–Text Instantiation · p. 47
  • The adapter’s addition lifts ESC-50 by 2.49 points and GTZAN by 0.83, while UrbanSound8K rises by 0.84.In negation, it removes 11.12 targeted-task points for 0.97 UrbanSound8K points on cross-model cells, and 13.34 for 1.75 on same-model cells, leaving the retained task within 0.3 points.Source: P Audio–Text Instantiation · p. 47

NeurIPS Paper Checklist

  • The paper reports that LLMs were not used as a core component of its methodology.Accordingly, it marks the checklist question about LLM use as [N/A].Source: NeurIPS Paper Checklist · p. 54

Related papers

Loading 2610.00929v2…