PaperlayerView exact revision on arXiv

Source-linked AI summary

Where Do Distilled Students Store Thinking?

Taeyeong Kim, Jaemin Jung, Ahhyun Kim, Unggi Lee

arXiv:2611.04141v1cs.CLcs.LG

TL;DR

Small students struggle with long LRM solutions, and output changes alone cannot show what their weights retain. The paper compares LoRA adapters across interpolated teachers and identifies a shared subspace tied to thinking-like behavior, whose removal largely turns that behavior off.

  • Problem

    Teacher thinking and solution length fall together, so student outputs alone cannot show which behavior the student learned or what its weights retain.

    Source: §1 · p. 1

  • Method

    The authors compare LoRA adapters across three teacher pairs and three students as λ varies, identify a pre-switch shared subspace, and test it by editing the adapters.

    Source: §1 · p. 1, §3.2 · p. 3, §3.4 · p. 4

  • Results

    Across nine cells, removing the shared region largely turns thinking-like behavior off; grafting it at twice its weight turns early markers on, though students rarely finish.

    Source: §7 · p. 8

  • Takeaways & Limitations

    The findings provide a weight-level account of what distillation from an interpolated teacher transfers.

    Source: §1 · p. 1

    Specificity beyond dominant directions holds only in pair A and C/Qwen, separation from length only in pair A, and model family and scale are confounded across pairs.

    Source: §7 · p. 8, Limitations · p. 9

Abstract

from arXiv · show

Small students degrade on large reasoning models' (LRMs') long solutions, so recent work shortens them by interpolating an LRM with a short-answer model at ratio $\lambda$. We ask what student weights retain when this teacher stops thinking, comparing LoRA adapters of three students from three teacher pairs at each $\lambda$. In each pair, the adapters' update subspace shifts most where the teacher stops thinking and solutions shorten most, locating the change without attributing it. We identify a shared adapter subspace tied to thinking-like behavior (reflective markers, monologue format and long solutions). Shared by adapters before that point, it keeps about half (post hoc) while one pair's teacher halves its solutions but keeps its think tag, and largely empties after thinking stops. Removing it largely turns this behavior off for every pair, student and benchmark. Only in that pair and one other student does it leave clearly fewer markers than removing the update's top components; elsewhere, same-energy random directions often nearly match it. Grafted into a short-answer student at twice its weight, it turns early markers (a post hoc count) on, like the whole update, but the student rarely finishes. Our work identifies a weight subspace tied to distilled thinking-like behavior.

1 Introduction

Small students struggle with long LRM solutions, and teacher output changes alone cannot reveal what students retain in their weights. This section introduces a LoRA-based comparison of students across interpolated teachers and identifies a shared subspace tied to thinking-like behavior.

1 Introduction

  • The shared adapter region is tied to reflective markers, monologue format, and long solutions, spanning 5–8 of 16 update directions at the chosen eigenvalue cut.Its unweighted build holds 47–72% of update energy; in pair A, it keeps about half while the teacher's solutions halve but its think tag remains.Source: §1 · p. 1
  • Removing the region largely turns thinking off in all nine cells, while grafting it at twice its weight turns early markers on but rarely yields finished solutions.The effect exceeds removal of other directions; only in pair A and C/Qwen does it exceed removal of the update's top components.Source: Figure 1 · p. 2, §1 · p. 2
  • In each teacher pair, adapter update subspaces shift most where the teacher stops thinking and solutions shorten, while the pre-switch shared region largely empties later.This locates the change without attributing it.Source: Figure 1 · p. 2, §1 · p. 2
  • Removal of the shared region also turns thinking off on GSM8K and GPQA-Diamond, but no edit beats untrained mean MATH-500 accuracy.The reported removal effect also holds with a region found from two λ = 0 seeds alone.Source: §1 · p. 2

2 Related Work

Prior work studies both distillation of long reasoning into small students and edits to model internals. These lines of work leave open what student weights retain as an interpolated teacher changes its reasoning style.

2.1 Chain-of-Thought Distillation to Small Students

  • Small students can degrade on long reasoning data, prompting curricula and interpolated teachers; these approaches tune student training inputs but leave open what weights retain.Interpolated-teacher students often do best at intermediate λ, while long solutions give larger gradients and, from 3B students on, more concentrated update directions.Source: §2.1 · p. 2

2.2 Reading and Editing Model Internals

  • Internal studies edit directions and SAE features to affect reasoning behavior, but often examine one model or pair; this work varies the teacher continuously and edits a shared subspace.Prior work also reports that SAE reasoning features can respond to cue tokens injected into non-reasoning text.Source: §2.2 · pp. 2–3

3 Method

The method distills students across interpolated teachers, compares LoRA update subspaces across teacher and data variations, and extracts a shared pre-switch region for intervention. Three teacher pairs and three students provide the experimental grid.

3.1 Distillation from an Interpolated Teacher

  • Students learn by sequence-level distillation from solutions sampled by a teacher interpolating an LRM (λ=0) and a same-base instruct model (λ=1).Source: §3.1 · p. 3

3.2 Subspace Analysis

  • Adapter update subspaces are compared by overlap, with teacher differences normalized against data-only differences and tested for coincident drops across λ.Source: §3.2 · p. 3
  • The switch is the first λ interval where teacher think rate drops by more than half; averaging pre-switch adapter projectors defines V from eigenvectors above 0.5.The eigenvalue represents the mean squared share of a direction within an adapter update subspace.Source: §3.2 · p. 3

3.3 Subspace Interventions

  • Interventions remove V, graft its λ=0 component into the λ=1 adapter at β=1 or 2, or amplify it, with energy-matched controls.Source: §3.3 · p. 4

3.4 Experimental Setup

  • The study varies only the teacher across six λ values, three teacher pairs, and three students; family and scale are confounded, so scale comparisons are avoided.Source: §3.4 · p. 4
  • Evaluation tracks pass@1, reflective markers, tokens, generation-limit hits, and SAE features on MATH-500, GSM8K, and GPQA-Diamond.Adapters are evaluated on the first 200 MATH-500 problems; markers serve as a surface proxy for long-solution reasoning.Source: §3.4 · p. 4

4 Where the Adapter Switches

Student outputs track teachers’ shifts in thinking, while adapter directions shift most where each teacher stops thinking and solutions shorten. A region shared before the switch largely empties afterward, though the evidence does not isolate which aspect of thinking it carries.

4.1 Student Outputs

  • Thinking markers and solution length fall with the teacher, while accuracy rises mostly through recovered truncation but falls over completed responses in eight of nine cells.Students score below their untrained models at 41 of 54 cell–λ points, and no edit raises mean accuracy above that level.Source: §4.1 · pp. 4–5

4.2 Adapter Directions

  • Adapter subspaces shift most near each pair’s teacher switch; pooled think-rate changes give p = 0.0035, while training-length drops give p = 0.0002.No individual pair reaches p < 0.05 on think rate, so the test locates the change rather than attributing it.Source: §4.2 · p. 5

4.3 Directions Shared Before the Switch

  • Before the switch, adapters share a region of 5–8 weighted directions out of 16; in pair A it keeps about half as solutions halve, then largely empties after thinking stops.In pair A, the teacher keeps its think tag while its markers fall threefold; in most cells, edits switch format and hesitation together, preventing attribution to one feature.Source: §4.3 · pp. 5–6
  • Pre-switch adapters cover 41–57% of the shared region, versus 5–19% for post-switch adapters and 0.5–1.8% for random adapters.Pair A’s λ = 0.4 adapters retain 0.09–0.12 of the λ = 0 region; these coverage comparisons use a post hoc reference.Source: §4.3 · p. 5

5 What the Shared Region Carries

Removing the shared region largely suppresses reflective behavior, while grafting it into a short-answer adapter restores early markers but rarely yields completed responses. Amplification and student-feature analyses further qualify how the region relates to thinking-like behavior.

5.1 Removing

  • Removal reduces whole-response markers to 0.2–10% of their original level and early markers from 10–16 to 0.07–1.3 per 1,000 words.Other-direction removal or scaling down the whole update leaves more markers; specificity beyond dominant directions appears only in pair A and C/Qwen.Source: §5.1 · p. 6, Figure 3 · p. 7

5.2 Grafting

  • At twice its weight, the region turns early reflective markers on in all nine cells but yields completion rates of only 0–23%.The early-marker metric was added after grafted responses looped; most region grafts loop, at 69–96%.Source: §5.2 · pp. 7–8, Figure 3 · p. 7

5.3 Amplifying

  • At fixed norm, the region’s weight rises only 1.2–1.6×; whole-response markers fall 41–74% at γ = 4 in five of nine cells.Early markers rise faster than the region’s weight in C/Qwen and C/Llama, about proportionally in A/Qwen and C/OLMo, and more slowly elsewhere.Source: Figure 3 · p. 7, §5.3 · p. 8

5.4 Student Features

  • Llama SAE features track markers at r = 0.88 in pair A and 0.92 in pair C, but r = −0.08 in pair B.The features may reflect style because each edited model reads its own responses, mixing text and edit.Source: §5.4 · p. 8

6 How General the Switch Is

Removing the region largely suppresses thinking-like markers beyond MATH-500, while a region built from two λ = 0 seeds also reduces markers when removed. The two-seed overlap is not independent evidence because λ = 0 adapters also help build the shared region.

6.1 Other Benchmarks

  • Removal leaves at most 10.4% of whole-response markers on GSM8K and GPQA-Diamond, far fewer than removing other directions.GSM8K accuracy rises partly through resolved truncation; all GPQA-Diamond conditions are below chance, so only markers and length are compared there.Source: §6.1 · p. 8
  • Table 3 reports λ = 0 adapter tokens and accuracy on GSM8K and GPQA-Diamond, plus marker, token, and accuracy measures for MATH-500 interventions.Untrained students ran on MATH-500 only.Source: Table 3 · p. 8

6.2 A Region from Two λ = 0 Seeds

  • A region built from two λ = 0 adapters differing only in seed spans more than half the shared region, and its removal leaves far fewer markers than removing other directions.Because λ = 0 adapters also help build the shared region, this overlap is not independent evidence, though the edited adapter has no shared initialization with those seeds.Source: §6.2 · p. 8

7 Conclusion

The paper identifies a shared adapter subspace associated with thinking-like behavior in small LoRA students distilled from interpolated teachers. Its specificity over dominant directions and separation from solution length are limited to particular settings.

7 Conclusion

  • The shared adapter subspace is tied to thinking-like behavior, shifts where the teacher stops thinking, and largely empties afterward.Removing it largely turns the behavior off in all nine cells and on other benchmarks; grafting it at twice its weight turns early markers on, though students rarely finish.Source: §7 · p. 8
  • The region is more specific than dominant update directions only in pair A and C/Qwen, and separates from solution length only in pair A.Separating thinking from solution length and scaling up remain future work.Source: §7 · p. 8
  • The authors’ conclusion is limited to a weight subspace tied to distilled thinking-like behavior.The paper does not claim a broader account of thinking independent of solution length.Source: §7 · p. 8

Limitations

The study’s conclusions are bounded by its small-model, LoRA-based off-policy MATH distillation setting and limited evidence outside mathematics. Measurement choices, grafting conditions, and the filtered training data further constrain interpretation.

Limitations

  • The study examines off-policy MATH distillation into students of at most 1B parameters, using LoRA as an instrument rather than the standard distillation method.Because full fine-tuning updates differ structurally, the identified region may not arise under full fine-tuning.Source: Limitations · p. 9
  • Only GPQA-Diamond provides evidence outside math, and model family and scale vary together across teacher pairs, precluding claims about scale.Students score below their untrained models at 41 of 54 cell–λ points; no λ beats the untrained model in five cells.Source: Limitations · p. 9
  • In pairs B and C, removing the region returns students to untrained behavior, so the region’s thinking-specific role cannot be separated from its carrying most teacher-added content.Same-energy random directions also leave at most 14% of markers in pair B and C/OLMo, while C/Qwen and C/Llama retain 28–31%.Source: Limitations · p. 9
  • The problem filter favors easier items, reducing the level-5 share from 30% to 16–17%, and only three seeds per cell support intervals with little seed variation.Thinking is measured by markers and, for Llama only, SAE features that may reflect style; grafting has only two usable doses and no cross-seed graft.Source: Limitations · p. 9
  • At fixed norm, the region’s weight rises only 1.2–1.6×, so above that range it may act as a dial in C/Qwen and C/Llama.OLMo grafts are confounded by a 3,840-token cap and truncated training solutions.Source: Limitations · p. 9

Use of AI Assistants

The paper reports using AI assistants across coding, literature search, analysis, and writing. The authors state that they verified generated content and take responsibility for the paper.

Use of AI Assistants

  • AI assistants were used to write and run code, search the literature, suggest analyses, and draft and revise text.Claude via Claude Code and Gemini for early drafts of Figure 1 are named.Source: Use of AI Assistants · p. 9
  • Gemini was used for early drafts of Figure 1.Source: Use of AI Assistants · p. 9
  • The authors say they verified all generated content and take full responsibility for the paper.Source: Use of AI Assistants · p. 9

A Experimental Details

The experiments train LoRA adapters on filtered MATH solutions while varying only the interpolated teacher, then measure reflective behavior and compare adapter subspaces. Region removal and grafting are evaluated against energy-matched controls.

Data and training

  • From 4,000 MATH training problems without MATH-500 overlap, the study keeps those solved completely and correctly within 8,192 tokens at all six λ values, holding out 300 for validation.With OLMo-2’s 4,096-token context, λ = 0 training solutions are truncated in 9%, 29%, and 6% of cases for pairs A, B, and C, respectively. Pair A is also evaluated at λ = 0.1 and 0.3.Source: A Experimental Details · p. 13

Measuring thinking

  • Thinking is measured by reflective-marker counts per 1,000 words, with early markers counted in the first 300 words; think tags and template tokens are stripped.An auxiliary Llama SAE read-out leaves 0.56–0.59 of variance unexplained and raises next-token loss by 0.01–0.04 when spliced in.Source: A Experimental Details · p. 13

Tests and regions

  • The shared region averages pre-switch adapter projectors and retains directions with eigenvalues above 0.5; a two-adapter direction’s eigenvalue is (1 + cos θ)/2.At this cut the two-seed region keeps directions within 60°; its size is 5.4–7.7 directions weighted or 6.4–8.0 unweighted.Source: A Experimental Details · p. 13

Edits

  • Region removal is compared with controls that remove matched update energy from other directions, the strongest singular components, or the whole update.Grafts add matched energy from outside the region or the whole λ = 0 update; removal also uses a same-length cut control.Source: A Experimental Details · p. 13

B Measuring the Switch

The analysis compares normalized adapter-subspace retention with teacher changes in thinking and solution length. Across pairs, the largest drops generally occur at teacher switch intervals, though length and switch status are not fully separable.

Normalizing R

  • Dividing overlap by the expected data-half overlap places the largest retention drop at the teacher switch in all nine cells, while R partly follows solution length.A regression switch coefficient is 0.46, but its cell-bootstrap interval spans zero and is unstable because length and interval class are confounded in pairs B and C.Source: B Measuring the Switch · pp. 13–14

Loss per unit of length

  • Pair A’s region loses 2.46–2.62 times more per unit of log-length drop at its switch than over its held interval; pairs B and C also average lower losses over held intervals.Pair B’s 0.6-to-0.8 interval loses more than its switch, where the teacher still produces 5.0 to 4.2 markers per 1,000 words.Source: B Measuring the Switch · p. 14

Floors and regression

  • The switch-versus-held loss ratio depends on the half-data reference and floor, and becomes unstable when solutions barely shorten.Pair A’s switch loses 1.93–3.07 times as much per unit length under the two-seed floor analysis; pair C’s first step does not separate under either floor.Source: B Measuring the Switch · p. 14

Pre-specified and post hoc analyses

  • Several analyses were post hoc, including symmetric R, bootstrap intervals, the two-seed region, overlap normalization, and per-marker normalization.The overlap ratio, permutation test, eigenvalue cut, switch definition, edits and controls other than random and post-switch removal were fixed before edit results were seen.Source: B Measuring the Switch · p. 14

C Results per Cell

Removing the shared region sharply reduces reflective markers and truncation, while student accuracy varies across λ and the region’s grafts often fail to finish. The comparisons also show that marker changes do not consistently distinguish the region from other update directions.

Student outputs

  • The λ = 1 student falls below untrained accuracy in six of nine cells, while accuracy peaks at λ = 0.8 in six cells and at λ = 1 in the other three.The λ = 1 accuracy drop persists on completed responses, by 1.2–8.5 percentage points.Source: C Results per Cell · p. 14, Table 10 · p. 16
  • The two-seed region retains 43–67% of a held-out λ = 0 adapter’s energy and 1–10% of a λ = 1 adapter’s, versus 0.2–2.3% by chance.Table 4 describes region dimensions, update-energy shares relative to random subspaces, and λ = 0 energy by layer third.Source: C Results per Cell · p. 14, Table 4 · p. 15

Controls and grafts

  • At β = 2, region grafts recover early verification markers in several cells but rarely finish; the region turns on long, open-ended responses without restoring closure.The graft finishes 22% and 19% of first responses in A/Qwen and A/Llama, respectively.Source: C Results per Cell · p. 14, Table 17 · p. 19

Logit lens

  • The logit-lens cue-word percentile is 64.8 for the region and 59.8 for random directions, but its 64.7 post-switch score does not separate the two reliably.The region beats the post-switch region in only five cells.Source: C Results per Cell · pp. 14–15

Seeds and intervals

  • Removal beats the cut in accuracy for 25 of 27 seeds and top removal for 19 of 27, but paired bootstrap intervals vary with whether cells or teachers are treated as units.The intervals reflect little seed variation because only three seeds were used per cell.Source: C Results per Cell · p. 15, Table 15 · p. 19

Results per Cell

  • At fixed norm, raising the region changes early markers by 0.96–1.58× at γ ≤4, while nine-cell accuracy falls from 13.9% to 10.1–11.5%.Moving the whole update into the region changes early markers by 0.87–1.79× and lowers completion from 37% to 28–31%.Source: Table 5 · p. 15
  • Removing the region leaves 0.2–9% of GSM8K markers and 0.1–10% of GPQA-Diamond markers, while GSM8K truncation falls from 11–85% to 0.4–1.9%.Figure 4 reports per-cell seed means for the λ = 0 adapter after removals and the λ = 1 adapter; nine-cell means appear in Table 3.Source: Figure 4 · p. 16
  • The removal controls include matched-energy alternatives, while the MATH-500 student table reports accuracy across λ and distinguishes changes on completed responses.Table 9 classifies intervals by teacher behavior and reports region coverage; Table 13 includes student structure means and an auxiliary SAE read-out.Source: Table 7 · p. 15, Table 9 · p. 16, Table 11 · p. 17

Related papers

Loading 2611.04141v1…