PaperlayerView exact revision on arXiv

Source-linked AI summary

Activation-Conditioned Self-Distillation

Zhexi Lu, Subhajit Chaudhury, Tejaswini Pedapati, Keerthiram Murugesan, Lei Yu

arXiv:2609.38342v1cs.LG

TL;DR

Outcome-only rewards provide limited intermediate-step feedback, and privileged reference conditioning does not ensure effective supervision throughout long responses. ACSD conditions a frozen copy of the base model on activation contrasts from its own trajectories, then distills its predictions on student-generated prefixes. It achieves the highest mean accuracy across four mathematical benchmarks on each of five evaluated models, with 71.9 versus 69.0 for OPSD on DeepSeek-R1-0528-Qwen3-8B.

  • Problem

    Outcome rewards lack fine-grained feedback on intermediate reasoning, and reference conditioning alone does not ensure effective token-level supervision throughout long responses.

    Source: §1 · pp. 1–2

  • Method

    ACSD steers a frozen base-model copy using activation contrasts from self-generated trajectories, then trains the student to match its next-token distributions on student-generated prefixes.

    Source: §1 · p. 2

  • Results

    ACSD achieves the highest mean accuracy across four mathematical benchmarks on each of five models; on DeepSeek-R1-0528-Qwen3-8B, mean accuracy is 71.9 versus 69.0 for OPSD.

    Source: §1 · p. 2, §4.2 · p. 7

  • Takeaways & Limitations

    Contrasts among correct trajectories also support distillation, extracted directions transfer across mathematical training datasets and difficulty levels, and ACSD maintains more stable late-position updates than OPSD on fixed student trajectories.

    Source: §1 · pp. 2–3

    The construction fixes one direction, layer, and strength per training setting.

    Source: §6 · p. 9

Abstract

from arXiv · show

On-policy self-distillation uses a model as its own teacher to provide dense supervision for reasoning, often through reference-solution conditioning. Providing privileged information does not by itself ensure effective token-level supervision throughout long responses. We introduce Activation-Conditioned Self-Distillation (ACSD), which extracts a steering vector by contrasting activations of self-generated trajectories that reach verified correct answers within a generation budget with those of all remaining trajectories. A frozen copy of the base model applies this vector at each prediction position, and the student learns from its next-token distributions on student-generated prefixes. Outcome verification is used for direction construction and calibration; distillation requires neither problem-specific reference text nor teacher parameter updates. The distilled student is used alone at inference. On each of five models, ACSD achieves the highest mean accuracy over four mathematical benchmarks among the evaluated methods. On DeepSeek-R1-0528-Qwen3-8B, mean mathematical accuracy reaches 71.9\% and LiveCodeBench v6 pass@12 reaches 70.9\%, compared with 69.0\% and 66.3\% for the reference-conditioned OPSD baseline. Contrasts among correct trajectories also support distillation, and extracted directions can be reused across mathematical training datasets. On fixed student trajectories, ACSD maintains more stable late-position logit-update magnitudes than OPSD.

1 INTRODUCTION

ACSD addresses limits in outcome-only and reference-conditioned supervision by conditioning a frozen self-teacher on activation contrasts among the model’s own trajectories. The paper reports leading mathematical benchmark accuracy across five models, along with more stable late-position updates and reusable directions.

1 INTRODUCTION

  • Outcome-only rewards lack fine-grained feedback on intermediate reasoning, while sampling multiple long responses increases training cost; reference-conditioned supervision may also weaken at late positions.The OPSD fixed-trajectory analysis finds decreasing update magnitudes as the student prefix grows.Source: §1 · pp. 1–2
  • Prior self-distillation methods condition teachers on reference answers, expert demonstrations, or external feedback, while RLCSD contrasts predictions conditioned on correct and incorrect solutions.The cited methods include OPSD and SDFT [Zhao et al., 2026; Shenfeld et al., 2026], SDPO [H¨ubotter et al., 2026], SD-Zero [He et al., 2026], and RLCSD [Pan et al., 2026].Source: §1 · p. 1
  • ACSD extracts a steering vector from the model’s own trajectory contrasts and applies it to a frozen base-model copy for dense supervision on student-generated responses, without reference solutions or additional teacher training.The student matches the teacher’s next-token distributions; the distilled student is used alone at inference.Source: §1 · p. 2
  • On fixed student trajectories, ACSD maintains more stable late-position update magnitudes than reference-conditioned OPSD.The paper also examines the basis and reusability of the steering vector.Source: §1 · pp. 2–3
  • Contrasts between shorter and longer correct trajectories support distillation, and extracted directions transfer across mathematical training datasets and difficulty levels.The contrasts capture differences among successful trajectories that correctness labels alone do not capture.Source: §1 · pp. 2–3
  • 4.3% average accuracy across four mathematical reasoning benchmarks and 5.2% LiveCodeBench v6 pass@12 improvements over the base model accompany ACSD’s highest mean accuracy on each of five models.On DeepSeek-R1-0528-Qwen3-8B, ACSD also achieves the highest LiveCodeBench v6 pass@k at all evaluated k values.Source: §1 · pp. 2–3

2 PRELIMINARIES

On-policy distillation supervises a student along its own sampled responses, and activation steering modifies hidden states while keeping model parameters fixed. A fixed steering direction can influence predictions throughout a response, with its effect depending on the hidden state and subsequent computation.

2.1 ON-POLICY DISTILLATION

  • On-policy distillation matches teacher next-token distributions at each position in student-sampled responses, then samples subsequent responses from the updated student.In self-distillation, the teacher is derived from the model being trained and may receive different conditioning, such as a reference answer unavailable to the student.Source: §2.1 · p. 3

2.2 ACTIVATION STEERING

  • Activation steering adds a scaled steering vector to a residual-stream state while model parameters remain fixed; applying it at successive prediction positions can influence a full response.The vector’s effect depends on the hidden state and subsequent network computation.Source: §2.2 · p. 3

3 ACTIVATION-CONDITIONED SELF-DISTILLATION

ACSD turns verified self-generated trajectory contrasts into token-level supervision by steering a frozen teacher on student prefixes. The direction is constructed and calibrated before on-policy distillation, after which only the student is used at inference.

3.1 MOTIVATION: FROM TRAJECTORY CONTRASTS TO A SELF-TEACHER

  • The contrast labels trajectories reaching verified correct answers within budget B as positive and places all others, including longer correct attempts, in the contrast group.The motivation is that successful trajectories for the same problem can differ substantially in length.Source: §3.1 · p. 4, §3.2 · p. 4
  • 26.6% to 66.0%: the selected intervention increased verified correct continuations on held-out calibration prefixes under matched sampling settings and budgets.The authors note that calibration recovery alone does not establish that the teacher provides better distillation targets.Source: §3.1 · p. 4

3.2 SELF-TEACHER CONSTRUCTION

  • The steering vector is the normalized difference between group means, using normalized activations from each trajectory’s first min(|r_i|, B) generated tokens.Each token activation contributes to the mean, so longer trajectories can contribute more vectors than shorter ones.Source: §3.2 · p. 5
  • At every response-prediction position, a frozen base-model copy adds the fixed direction to its residual state before producing a context-dependent next-token distribution.Steering begins at the final prompt state and continues as the student prefix grows.Source: §3.2 · p. 5

3.3 CONFIGURATION SELECTION

  • The intervention layer and strength are selected on a calibration pool by comparing verified continuation recovery with and without steering on fixed prefixes.After selection, the teacher parameters, direction, layer, and strength remain fixed throughout distillation; construction uses no reference solution or teacher parameter updates.Source: §3.3 · p. 5

3.4 ON-POLICY STUDENT TRAINING

  • The student matches the steered teacher’s next-token distributions on its own generated prefixes using a clipped, position-averaged objective, and is used alone at inference.Training uses one student response per problem; gradients flow only through the student distribution.Source: §3.4 · pp. 5–6

4 EXPERIMENTS

ACSD is evaluated on mathematical reasoning and code generation, with ablations testing trajectory contrasts, direction choices, token-level updates, and extraction data. It achieves the highest mean mathematical accuracy on all five models and improves code-generation scores, while analyses examine how its supervision behaves and transfers.

4.1 EXPERIMENTAL SETUP

  • Experiments use five models for mathematics, with the main comparisons training on OpenThoughts-Math-30K and evaluation across four benchmarks using 12 responses per problem.Code generation, ablations, and further analyses use DeepSeek-R1-0528-Qwen3-8B; mathematical comparisons use a shared training framework and LoRA for 200 updates, while ACSD and OPSD share a forward-KL objective with pointwise clipping.Source: §4.1 · pp. 6–7

4.2 MAIN RESULTS

  • 71.9 Avg@12 on DeepSeek-R1-0528-Qwen3-8B and 68.0 on Qwen3-8B are ACSD’s highest reported mathematical scores, and it leads the average on all five models.On DeepSeek-R1-0528-Qwen3-8B, the comparison values are 70.8 for GRPO and 69.0 for OPSD and RLCSD; on Qwen3-8B, the base model scores 63.3 and GRPO 66.4.Source: §4.2 · p. 7
  • ACSD reaches 51.0 pass@1 and 70.9 pass@12 on LiveCodeBench v6, the highest scores across the four reported code metrics.It improves both single-sample accuracy and multi-sample problem-solving probability; ACSD and OPSD use 59.93 and 61.82 H100 GPU-hours for 200 updates, respectively.Source: §4.2 · p. 7

4.3 ABLATION STUDIES

  • Contrasting correct trajectories completed within versus beyond the generation budget improves the student throughout training, while contrasting length-matched correct and incorrect trajectories helps initially but later falls below the base model.Full ACSD reaches a higher peak, suggesting that retaining incorrect and unfinished trajectories adds complementary information.Source: §4.3 · p. 7
  • Random, coordinate-shuffled, and reversed directions lead to declining accuracy, whereas the extracted ACSD direction consistently improves the student.Shuffling preserves coordinate values and norm, while reversal preserves the axis but changes its sign.Source: §4.3 · p. 7, Figure 3 · p. 8

4.4 TOKEN-LEVEL SUPERVISION FROM THE TEACHER

  • ACSD’s logit-update magnitude rises early and then remains relatively stable, while OPSD’s declines from about 97% of ACSD’s magnitude in the first 2k tokens to 66% beyond 16k.ACSD also improves over the base model at every tested training window from 1k to 16k tokens.Source: §4.4 · pp. 7–8, Figure 4 · p. 8
  • ACSD’s signed updates vary by reasoning role: it shifts uncertainty and final-answer markers in distinct directions, and conclusion-marker updates are larger in final-answer spans.For final answers, ACSD’s signs for contrast and conclusion markers are opposite to OPSD’s.Source: §4.4 · p. 8

4.5 THE EFFECT OF DIRECTION EXTRACTION DATA

  • Directions extracted from easy DeepMath problems reach 70.6 Avg@12 on hard problems, while a direction from 64 trajectories on 16 problems reaches 70.8 versus 67.6 for the base model.The transfer comparisons hold the model, layer, strength, and optimizer fixed; the small-set result comes from nested discovery subsets with four trajectories per problem.Source: §4.5 · pp. 8–9, Table 2 · p. 9

5 RELATED WORK

Prior work includes on-policy distillation and self-distillation approaches that use different teacher contexts, objectives, or rollout signals. ACSD instead derives its direction from verifier-labeled reasoning rollouts and applies token-level supervision across student-generated rollouts.

5 RELATED WORK

  • Self-distillation methods differ in how they condition teachers or use rollout feedback: OPSD uses reference solutions, while RLCSD uses correctness contrasts to modulate GRPO advantages.Other approaches include reverse KL, skew KL with response reuse, reviser distillation, and teacher-logit adjustments from successful and failed contexts.Source: §5 · p. 9
  • Activation-steering work studies behavior-related directions, while ACSD constructs its direction from verifier-labeled reasoning rollouts and supervises complete student-generated rollouts.Prior examples include steering for truthfulness, reasoning behaviors, and refusal distillation.Source: §5 · p. 9

6 DISCUSSION

ACSD uses trajectory-level verification to support token-level supervision through activation differences. The discussion highlights the scope of the fixed intervention and identifies extensions for future work.

6 DISCUSSION

  • ACSD’s correct short–long ablation supports useful supervision from successful trajectories sharing a correctness label, while recovery evaluation improves continuation success under a fixed budget.The discussion states that distillation retains the gains in the student’s parameters; the construction fixes one direction, layer, and strength per training setting.Source: §6 · p. 9

A.1 EXPERIMENTAL SETUP

The appendix specifies ACSD’s direction construction, training, and evaluation configurations for mathematical reasoning and code generation.

A.1 EXPERIMENTAL SETUP

  • Directions contrast verified correct trajectories within budget B against all remaining trajectories, using normalized pooled activations through the first min(|ri|, B) tokens.The positive group contains trajectories with verified final answers and at most B generated tokens; the contrast group includes longer correct, incorrect, and unfinished trajectories.Source: §A.1 · p. 12
  • ACSD applies a fixed direction to a frozen base-model teacher at each response-token prediction position, then updates the student on its own generated responses.The teacher parameters, direction, layer, and strength remain fixed; subsequent rollouts use the updated student.Source: §A.1 · pp. 12–13
  • Mathematical evaluation samples 12 responses per problem across AIME 2024, AIME 2025, HMMT February 2025, and AIME 2026.The appendix specifies 30 problems per benchmark and shared decoding settings for methods using the same model.Source: §A.1 · p. 13, Table 4 · p. 13
  • Code experiments train on 5,000 Codeforces problems, while LiveCodeBench v6 evaluation uses 175 problems with 12 samples per problem.ACSD trains on problem statements alone; SFT and OPSD use reference implementations, and evaluation uses the official checker.Source: §A.1 · p. 13

A.2 RECOVERY EVALUATION AND CONFIGURATION SELECTION

The recovery scan selects an intervention setting on held-out calibration continuations, and the selected setting performs best among the tested distillation configurations.

A.2 RECOVERY EVALUATION AND CONFIGURATION SELECTION

  • Recovery is measured by continuing fixed prefixes with and without steering under matched sampling settings and continuation budgets.The calibration pool contains 256 held-out problems, and each layer–strength setting uses 512 continuations.Source: §A.2 · pp. 13–14
  • 66.0% recovery with layer 18 and α = 0.3, up from 26.6% without steering.This setting was selected after scanning candidate layers and strengths on DeepSeek-R1-0528-Qwen3-8B.Source: Figure 5 · p. 14, §A.2 · p. 14
  • 71.9 Avg@12 for the recovery-selected setting, versus 68.7–69.9 across the five alternative intervention settings.The configurations share the model, training data, optimizer, and evaluation protocol.Source: Figure 6 · p. 14, §A.2 · p. 14

A.3 COMPLETE DIRECTION CONTROLS

The appendix compares alternative trajectory contrasts and direction perturbations under shared activation construction or intervention settings.

A.3 COMPLETE DIRECTION CONTROLS

  • The short–long contrast uses 914 correct trajectories finishing within budget and 913 correct trajectories finishing beyond it.The length-matched correctness contrast pairs 221 incorrect trajectories with 221 correct trajectories selected by nearest-neighbor matching on response length; their mean lengths differ by 78 tokens.Source: §A.3 · pp. 14–15
  • The length-matched correctness direction contrasts incorrect and correct trajectories, with mean lengths differing by 78 tokens.Both groups use the same activation normalization and difference-in-means construction.Source: §A.3 · p. 15
  • Perturbation controls are an isotropic random unit vector, a coordinate-shuffled ACSD direction, and the reversed direction −d.These controls share ACSD’s intervention layer, strength, training data, and optimizer.Source: §A.3 · p. 15

A.4 TOKEN-LEVEL UPDATE ANALYSIS

The token-level analysis compares teacher updates on identical held-out student trajectories, measuring update magnitudes by response position and examining marker groups across reasoning roles.

A.4 TOKEN-LEVEL UPDATE ANALYSIS

  • The analysis uses 512 fixed student trajectories, split into 256 discovery and 256 held-out trajectories, with both teachers processing the same prefixes.Marker groups and reasoning-role labels are determined separately; role analysis uses full held-out trajectories to retain less frequent roles.Source: §A.4 · p. 15
  • Update magnitude is summarized across five absolute response-position bins: 0–2k, 2–4k, 4–8k, 8–16k, and beyond 16k tokens.The measured quantity is the negative gradient of the clipped distillation loss with respect to student logits.Source: §A.4 · p. 15
  • 97% of ACSD’s update magnitude in the first 2k tokens, 85% in the next 2k, and 66% beyond 16k; OPSD’s update-vector cosine similarity falls from 0.298 to 0.108.Both teachers compute updates on the same observed text.Source: §A.4 · p. 15
  • Figure 7 maps marker groups by rows and reasoning roles by columns; expanded marker and leave-one-marker-out analyses preserve the reported main sign patterns.The aggregate EOS update is near zero and slightly negative.Source: Figure 7 · p. 16, §A.4 · p. 16

A.5 TRAINING-WINDOW EXPERIMENTS

Across tested training windows, ACSD improves over the base model, while OPSD does so only early at 4k tokens. The section also examines direction transfer, discovery-set size, and inference-time steering.

A.5 TRAINING-WINDOW EXPERIMENTS

  • ACSD improves over the base model at every tested window; OPSD at 4k reaches 67.4 at update 100, below the base model’s 67.6.Figure 8 shows the first 100 updates; OPSD’s other tested windows remain below the base model.Source: §A.5 · p. 16, Figure 8 · p. 17

A.6 DIRECTION TRANSFER AND DISCOVERY-SET SIZE

  • At layer 18, easy- and hard-problem directions have cosine similarities of 0.949 and 0.711, respectively, with the OpenThoughts-Math-30K direction.Both directions are distilled on the same hard problems under matched model and intervention settings; generation budgets correspond to each direction’s setting.Source: §A.6 · p. 17
  • Discovery sets of 16–512 problems are compared using direction cosine similarity to the 512-problem direction and distillation results.Each problem contributes four trajectories; layer 18, α = 0.3, and the main mathematical evaluation settings are held fixed.Source: §A.6 · p. 17

A.7 INFERENCE-TIME STEERING

  • Inference-time steering applies the teacher-construction direction, layer, and strength throughout autoregressive generation, whenever a response token is predicted.Table 7 reports average Avg@12 across four mathematical reasoning benchmarks and defines ∆ as Steered minus Base; the supplied passages do not give its values.Source: §A.7 · p. 17, Table 7 · p. 17
  • The evaluations separately examine full generation from the first token, fixed-prefix recovery under a shared continuation budget, and the distilled student’s performance.ACSD teacher targets are computed on prefixes generated by the student, while final evaluation uses the distilled student.Source: §A.7 · p. 18
Loading 2609.38342v1…