PaperlayerView exact revision on arXiv

Source-linked AI summary

No Task Vector Is an Island: A Comprehensive Study on the Composability of Task Vectors from On-Policy Distillation

Jingang Zhou, Feiyu Han, Han Zhu, Yuyi Zhou, Ruiyang Zhang, Jian Xu, Sirui Gao, Qingpei Guo, Xu-Yao Zhang

arXiv:2609.39405v1cs.LG

TL;DR

The composability of task vectors from on-policy distillation remains underexplored, despite OPD updates differing from those of their RL teachers. This study compares within-task OPD–RL merges and cross-task OPD and RL compositions across five domains and two architectures, finding evidence that OPD vectors can complement teacher updates and compose effectively across tasks.

  • Problem

    How OPD task vectors compose with RL teacher updates and across tasks remains largely unexplored, although the training procedures produce different updates.

    Source: §1 · pp. 1–2

  • Method

    The study evaluates within-task OPD–RL merges and cross-task compositions across five domains and two model architectures, and analyzes update geometry.

    Source: §1 · p. 3

  • Results

    OPD compositions score higher than corresponding RL compositions in seven of eight backbone–merging-rule comparisons, while within-task OPD–RL merges can outperform both constituent models.

    Source: §1 · p. 2, §6 · p. 9

  • Takeaways & Limitations

    Task-vector composability is distinct from standalone model performance, and weaker OPD students can contribute useful updates to stronger RL teachers.

    Source: §1 · p. 2, §6 · p. 9

    The evaluation uses general-purpose merging rules; OPD-aware merging methods and evaluation across additional tasks and model families remain future work.

    Source: §6 · p. 9

Abstract

from arXiv · show

Task vectors provide a simple mechanism for composing learned capabilities through model merging. However, the composability of task vectors produced by on-policy distillation (OPD) remains largely unexplored. OPD trains a student using teacher feedback on student-generated trajectories, yielding parameter updates that differ from those produced by the teacher model, usually by reinforcement learning (RL). We therefore ask whether OPD task vectors can complement their RL teacher updates and compose effectively across tasks. Across five domains and two model architectures, we find evidence for both forms of composability. Within a task, merging OPD and RL task vectors can outperform both constituent models, even when the OPD student is weaker than its RL teacher. Across tasks, OPD task-vector compositions achieve higher average scores than corresponding RL compositions in seven of eight backbone-merging-rule comparisons. Parameter-space analyses reveal substantial non-collinearity between OPD and RL updates. Experiment in CODE domain on SMOLLM3-3B shows that the combined direction outperforms either constituent direction at the tested global update norm, supporting directional complementarity in this configuration. Across tasks, OPD updates also show lower overlap among the top-10% feed-forward channels ranked by update energy. Together, these results show that weaker standalone performance does not imply weaker task-vector composability. OPD task vectors can complement stronger RL teacher updates and combine effectively across tasks, highlighting composability as a distinct property for understanding and evaluating post-training updates.

1 INTRODUCTION

The introduction asks whether OPD task vectors complement their RL teachers within tasks and compose effectively across tasks. Across five domains and two architectures, it reports evidence for both, with OPD compositions generally retaining or improving task performance despite weaker standalone OPD students.

Research question

  • The study tests whether OPD updates can complement their RL teacher within a task and reduce interference when independently trained task vectors are combined across tasks.OPD trains a student on student-generated trajectories using token-level feedback from a stronger teacher.Source: §1 · p. 2

Within-task composition

  • On QWEN3-4B CODE, a within-task OPD–RL TIES merge scores 41.24%, versus 37.51% for the RL teacher and 37.18% for the OPD student.The merge improves on the teacher by 3.73 pp.Source: §1 · p. 2

Cross-task composition

  • OPD compositions achieve higher overall scores than corresponding RL compositions in seven of eight backbone–merging-rule comparisons.The exception is SMOLLM3 Raw Sum; OPD also shows a more favorable change from its own specialist average across all eight comparisons.Source: §1 · p. 2

Update geometry

  • Within-task OPD updates have substantial components orthogonal to RL updates, and a norm-matched CODE experiment supports directional complementarity at the tested norm.The authors caution that non-collinearity alone does not establish functional benefit.Source: §1 · p. 2

Study scope

  • The study covers five domains and two architectures, and characterizes update geometry alongside within-task and cross-task composition.The domains are Math, Code, Instruction Following, Science, and Agent; the architecture/domain coverage varies by model family.Source: §1 · pp. 2–3

2 PRELIMINARIES

The preliminaries distinguish RLVR from policy-gradient OPD and define task vectors as parameter displacements from a shared initialization. They frame composability across within-task, cross-task, and joint combinations evaluated with multiple merging operators.

2.1 POST-TRAINING

  • RLVR uses response-level answer checks or executable tests, whereas OPD uses a fixed teacher’s token-level feedback on student-generated trajectories.OPD uses sampled-token feedback or supervision over multiple top-k candidates, following Li et al. (2026).Source: §2.1 · p. 3

2.2 COMPOSABILITY OF TASK VECTORS

  • A task vector is the parameter displacement from the shared initialization for a model trained on task t.The paper measures an OPD task vector from the student’s initialization, not from its teacher.Source: §2.2 · p. 3
  • Composition uses a general merging operator; Task Arithmetic uses a scaled sum, while TIES-Merging and TSV-M transform vectors before aggregation.The general formulation also includes configurations of the merging operator.Source: §2.2 · p. 3
  • The paper defines composability as retaining or improving task-specific capabilities when multiple task vectors are combined.It distinguishes within-task, cross-task, and joint compositions across tasks and post-training methods.Source: §2.2 · p. 3

3 COMPOSABILITY OF OPD TASK VECTORS

The section evaluates OPD task-vector composability within tasks, across tasks, and in joint OPD–RL multi-task merges, using two model families across five domains. It reports benefits in all three settings, with outcomes varying by task and merging rule.

3.1 EXPERIMENTAL SETUP

  • SmolLM3-3B covers Math, Code, and IF, while Qwen3-4B adds Science and Agent; checkpoints use matched evaluation settings and scores average three repetitions.The study compares TA, TIES, and TSV-M, including Raw Sum, across the task vectors from each model family and anchor.Source: §3.1 · p. 4

3.2 WITHIN-TASK COMPLEMENTARITY OF OPD TASK VECTORS

  • Seven of 12 Qwen task–merging configurations exceed both endpoints; Code TIES and Raw Sum beat RL by 3.73 and 3.50 pp.Raw Sum also gains 1.30 pp on IF; on Math, TA gains 0.37 pp and TIES 0.21 pp. Complementarity depends on task and merging rule and is not consistently recovered.Source: Table 1 · p. 4, §3.2 · p. 4

3.3 CROSS-TASK COMPOSABILITY OF OPD TASK VECTORS

  • OPD cross-task compositions had higher averages in seven of eight backbone–merging-rule comparisons; relative to their specialists, they gained more or lost less in all eight.The exception for final averages is Raw Sum on SmolLM3-3B, where RL reaches 32.45% and OPD 32.06%.Source: Table 2 · p. 5, §3.3 · p. 5

3.4 OPD TASK VECTORS IN JOINT MULTI-TASK COMPOSITION

  • 33.28% on SmolLM3-3B and 56.67% on Qwen3-4B were joint-composition averages, above both single-source averages on each backbone.The method averages OPD and RL updates within each task, then sums the combined updates across tasks.Source: Table 3 · p. 6, §3.4 · p. 6
  • Joint composition also exceeds both single-source compositions in each SmolLM3-3B domain, while Qwen3-4B shows task-dependent trade-offs.On Qwen3-4B, joint composition improves Math, Science, and Agent relative to RL-only, while Code and IF decline.Source: §3.4 · p. 6
  • Joint composition has the highest displayed SmolLM3-3B average among Table 4 baselines, but published baselines use separate runs and Math avg@64 rather than avg@8.Open-MOPD has the highest IF score (49.58%) and an average of 31.24%.Source: §3.4 · p. 6, Table 4

4 UNDERSTANDING THE TASK VECTORS FROM OPD

The section analyzes OPD update structure, its within-task relationship to RL updates, and how update structure relates to cross-task composition. Norm-matched Code controls support directional complementarity, while cross-task analyses show lower OPD hotspot overlap and task-dependent direction-retention changes.

4.1 PARAMETER STRUCTURE OF OPD TASK VECTORS

  • For a common SmolLM3-3B base, OPD update norms are 59.7% of RL’s for Math and 68.7% for Code; OPD also changes fewer saved BF16 coordinates.Spectral concentration F(k) aggregates rank-k retained energy over 252 attention and MLP projection matrices, using approximate randomized SVD and normalizing by total update energy.Source: §4.1 · pp. 6–7
  • For Code, the final 12 layers hold 37.7% versus 32.1% of each vector’s squared L2 energy; for Math, the shares are 44.7% versus 32.3%, despite lower absolute late-layer energy.The larger relative shares coexist with OPD’s lower update norms.Source: §4.1 · pp. 6–7, Figure 2 · p. 7

4.2 WITHIN-TASK PARAMETER COMPLEMENTARITY BETWEEN OPD AND RL

  • Orthogonal energy shares of 93.0% for Math and 95.2% for Code indicate substantial non-collinearity, but do not establish a functional benefit.RL–OPD alignment also varies across layers and projections.Source: §4.2 · p. 7, Figure 3 · p. 7
  • At the tested global update norm, Raw Sum exceeds norm-matched RL-only and OPD-only controls by 1.77 pp and 2.07 pp, respectively.The comparison uses SMOLLM3-3B Code controls.Source: §4.2 · p. 7, Table 5 · p. 8

4.3 HOW UPDATE STRUCTURE SHAPES MULTI-TASK COMPOSITION

  • OPD’s top-10% MLP-channel Jaccard overlap is lower across all ten task pairs, averaging 0.11064 versus RL’s 0.15368, a 28% reduction.The high-energy regions of OPD updates are less shared across tasks.Source: §4.3 · p. 8, Figure 4 · p. 8
  • At λ = 1, OPD direction retention improves by 0.016, 0.021, and 0.071 for Code, Science, and Agent, but falls by 0.011 for Math and 0.028 for IF.Lower relative incoming energy coincides with improved retention for Code, Science, and Agent, while Math and IF show the opposite pattern.Source: §4.3 · pp. 8–9, Figure 4 · p. 8

5 RELATED WORK

The related work situates this study at the intersection of on-policy distillation, task-vector merging, and analyses of post-training update geometry.

On-Policy Distillation and Capability Integration

  • Prior OPD work studies teacher–student objectives and multi-teacher integration during training; this paper instead studies composition of task-specific OPD updates in parameter space.The cited work includes GKD, MiniLLM, MOPD, and Open-MOPD.Source: §5 · p. 9

Task Vectors and Model Merging

  • Model Soups averages fine-tuned models, while Task Arithmetic, TIES-Merging, and DARE compose or edit task vectors through parameter-space operations.These merging methods motivate examining OPD-derived task-vector composition.Source: §5 · p. 9

Geometry and Composability of Post-Training Updates

  • Prior geometry studies examine post-training update structure and merging behavior; this work connects OPD update geometry to within-task RL composition and cross-task OPD composition.The cited context includes SFT–RLVR differences, update sparsity, alignment, and OPD checkpoint structure.Source: §5 · p. 9

6 DISCUSSION AND CONCLUSION

The paper studies whether OPD task vectors compose with RL updates within tasks and across tasks, and reports evidence for both settings. Its experiments use general-purpose merging rules, with further OPD-aware methods and broader evaluations left for future work.

6 DISCUSSION AND CONCLUSION

  • Across five domains and two architectures, within-task OPD–RL merges can beat both constituent models, while OPD cross-task compositions outperform RL compositions in seven of eight comparisons.SmolLM3-3B Code controls support directional complementarity at the tested global update norm.Source: §6 · p. 9
  • The evaluation uses general-purpose merging rules; future work could develop OPD-aware methods and evaluate additional tasks and model families.These are the stated scope boundary and directions for further study.Source: §6 · p. 9

REPRODUCIBILITY STATEMENT

The reproducibility statement describes the paper’s experiment documentation and score aggregation.

REPRODUCIBILITY STATEMENT

  • Appendix B documents data, OPD training configurations, merging parameters, and score aggregation; reported performance scores are arithmetic means over three repetitions.The statement says all experimental results are supported by statistical analyses.Source: REPRODUCIBILITY STATEMENT · p. 10

A POST-TRAINING OBJECTIVES

The section distinguishes verifier-based reinforcement learning from teacher-feedback on-policy distillation, then describes the models, training prompts, and domain-specific datasets used to study them. Each OPD student is trained from a shared task-vector anchor using its corresponding RL expert as teacher.

A POST-TRAINING OBJECTIVES

  • RLVR derives learning signals from verifier-defined outcomes, whereas OPD uses token-level teacher feedback on student-generated trajectories.The study considers sampled-token and student-top-k OPD feedback; both methods use closely related policy-gradient structures.Source: A POST-TRAINING OBJECTIVES · p. 15

B DETAILED EXPERIMENTAL SETUP

  • The experiments use SmolLM3-3B in three domains and Qwen3-4B-Instruct-2507 in five, training one OPD student per domain from the shared anchor with its RL expert as teacher.SmolLM3 uses student-top-k feedback and Qwen uses sampled-token feedback.Source: B DETAILED EXPERIMENTAL SETUP · p. 16

B.1 TRAINING DATASETS

  • Within each family and domain, RL and OPD use the same released training prompts, but RL receives verifier rewards and OPD receives teacher feedback on student-generated responses.The reported counts refer to processed, prepared, and filtered release splits, not the full upstream collections.Source: §B.1 · p. 16
  • SmolLM3 training uses 17,917 processed Math examples, 23,667 Code prompts after excluding LiveCodeBench-source examples, and 45,347 processed IF examples.Math examples have integer-answer verification; Code uses executable tests, while IF examples have programmatically checkable constraints.Source: §B.1 · pp. 16–17
  • Qwen uses 38,131 Math prompts, 50,000 sampled Science prompts, and processed Code, IF, and Agent splits of 19,169, 16,575, and 10,229 examples, respectively.The Math and Science splits apply difficulty filtering; the other domains cover executable coding tests, verifiable instruction constraints, and workplace-assistant tool tasks.Source: §B.1 · p. 17

B.2 EVALUATION DATASETS AND SCORING

Evaluation uses domain-specific benchmark subsets and metrics, with repeated scores averaged in stages to form aggregate results. The reported averages are not significance tests, and the two model families’ overall averages cover different task sets.

B.2 EVALUATION DATASETS AND SCORING

  • Math scores average AIME 2024 and 2025 avg@8, while published Open-MOPD baselines retain their original Math avg@64 protocol.Each AIME edition contains 30 problems, and avg@k averages correctness over k independently generated responses per problem.Source: §B.2 · p. 18
  • Code scores equally average LiveCodeBench v5 and v6 avg@10 results, while IF scores equally average strict prompt-level accuracy on IFEval and IFBench.The Code increments contain 167 and 175 problems; the IF evaluations use 541 and 300 prompts, respectively.Source: §B.2 · p. 18
  • Qwen’s Science score uses the 198-question GPQA-Diamond subset, and its Agent score is the percentage of successful instances on a 200-instance BFCL v3/test subset.The five Qwen domains contain 1,641 question-level evaluation instances in total.Source: §B.2 · p. 18
  • Overall scores average benchmarks within each domain, then give each evaluated domain equal weight; positive mean differences are not interpreted as statistical significance.SmolLM3 covers three domains and Qwen five, so their aggregate averages are not directly comparable in absolute value.Source: §B.2 · pp. 18–19

B.3 TRAINING CONFIGURATIONS

The study applies full-parameter OPD training and compares several task-vector merging rules. Parameter diagnostics show that OPD and RL updates differ in their layer energy, BF16-visible coordinates, and spectral concentration.

B.3 TRAINING CONFIGURATIONS

  • OPD training uses four rollout responses per prompt for three epochs, with shared optimization and rollout settings from the corresponding GRPO experts.For Qwen, a 0.001 coefficient applies to a separate reference-policy KL loss.Source: §B.3 · p. 19, Table 8 · p. 19

B.4 MODEL MERGING

  • The merging comparisons cover Task Arithmetic, TIES-Merging, and TSV-M, with Raw Sum reported as Task Arithmetic at λ = 1 where applicable.Reported configurations are selected from evaluated candidates; multi-task rows use one materialized checkpoint across the included domains.Source: §B.4 · pp. 19–20

C.1 PARAMETER-STRUCTURE MEASUREMENTS

  • At rank 256, OPD captures 33.4% of Code update energy and 33.7% of Math energy, versus 45.9% and 39.2% for RL, respectively.OPD has lower spectral concentration at all six measured ranks from 8 through 256.Source: Figure 5 · p. 20, §C.1 · p. 20
  • The final 12 layers contain 32.1% →37.7% of Code update energy and 32.3% →44.7% of Math energy for RL versus OPD, despite OPD having smaller update norms.OPD norms are 68.7% of RL for Code and 59.7% for Math; BF16-visible nonzero-coordinate fractions are 1.31% versus 27.98% for Code and 1.27% versus 21.04% for Math, OPD versus RL.Source: Figure 5 · p. 20, §C.1 · p. 20

C.2 WITHIN-TASK DIAGNOSTICS AND CONTROLS

The within-task diagnostics are based on SmolLM3-3B checkpoint pairs, and the Code controls test updates at a shared norm. The analysis cautions that orthogonality alone does not determine functional effect and that removing the orthogonal component also changes update magnitude.

C.2 WITHIN-TASK DIAGNOSTICS AND CONTROLS

  • The orthogonal residual’s functional effect depends on its interaction with the task loss, so orthogonality to the RL direction alone does not establish benefit.The local loss approximation is an interpretation, not a measured curvature result.Source: §C.2 · p. 21
  • The norm-matched Code controls support a directional contribution, but removing the orthogonal component also changes magnitude and therefore cannot isolate direction.The within-task diagnostics use the same SmolLM3-3B RL–OPD checkpoint pairs; Table 5 reports mean score differences relative to the corresponding Raw Sum model.Source: §C.2 · p. 21
  • For Qwen, within-task Raw Sum changes relative to RL specialists are −0.36 pp on Math and +3.50 pp on Code, showing that these comparisons vary by task.These within-task values are distinct from the cross-task Raw Sum compositions and from the SmolLM3-3B controls.Source: §C.2 · p. 21

C.3 SCALE AND LOCALIZATION WITHIN TASK HOTSPOTS

The hotspot analysis separates incoming update energy into scale and localization, with task-specific ratios changing through different factor patterns. The surrounding context positions the study as parameter-space reuse of OPD updates.

C.3 SCALE AND LOCALIZATION WITHIN TASK HOTSPOTS

  • Science’s ratio decreases from 3.613 to 3.262 as localization factor Lt falls despite a slight increase in relative scale At; Agent’s ratio falls from 4.637 to 3.103 as At drops despite higher Lt.The two examples show different factor patterns behind lower ratios.Source: §C.3 · p. 21
  • Code shows reductions in both At and Lt, while for Math and IF the increase in At outweighs lower Lt.Source: §C.3 · p. 21
  • Jaccard measures shared membership among selected channels, whereas Lt measures energy localization; Eq. 5’s direction retention captures alignment through the actual vector sum.Source: §C.3 · p. 21

D ADDITIONAL CONTEXT ON OPD AND CAPABILITY INTEGRATION

  • Prior work studies OPD for teacher–student transfer and multi-teacher integration, while this study examines task-specific OPD updates for later parameter-space composition.The cited approaches include GKD, MiniLLM, MOPD, and Open-MOPD [Gao et al., 2026].Source: D ADDITIONAL CONTEXT ON OPD AND… · p. 21

Related papers

Loading 2609.39405v1…