PaperlayerView exact revision on arXiv

Source-linked AI summary

Post-Training Leaves Behavioral Shadows on Unrelated Decisions

Ziyang Zhang, Yubin Jing, Yuanhao Zeng, Yuyao Li, Haofan Wang, Yichen Gong

arXiv:2609.29233v1cs.CLcs.AIcs.LG

TL;DR

Post-training may change a model’s ordinary, task-unrelated word choices, but whether those decisions reveal information about improved capabilities has been unclear. ATD uses public-ancestor near-ties to collect single-word teacher responses and train a same-ancestor student. The paper reports transfer across tasks and models, including a 5.34 pp HumanEval+ gain over an exact-matched control.

  • Problem

    It is unclear whether task-unrelated decisions reveal information about capabilities improved by a private post-training update.

    Source: §1 · p. 1

  • Method

    ATD selects unrelated near-tie prompts using the public ancestor and trains a same-ancestor student on one-word teacher responses, without target-task examples or teacher logits.

    Source: §1 · p. 1, §4 · p. 4

  • Results

    5.34 pp on HumanEval+ over an exact-matched control, with transfer also reported across scientific knowledge, commonsense reasoning, reading comprehension, and additional model generations, sizes, and families.

    Source: §1 · p. 2, §7 · p. 9

  • Takeaways & Limitations

    The behavioral shadow retains source- and strength-related information, but its usefulness for capability transfer varies across tasks and models.

    Source: §7 · p. 9

    The setup assumes a known public ancestor shared by teacher and student, and restricts teacher access to one next token per query with queries fixed in advance.

    Source: §3 · p. 3

Abstract

from arXiv · show

We find that language models can transfer capabilities through task-unrelated text. Post-training typically improves language models using task-specific data. Prior work on subliminal learning shows that information about these updates can pass through unrelated generations, but has largely focused on traits or preferences using extensive teacher outputs. We introduce Active Taskless Distillation (ATD), which achieves capability transfer using only a single word from the teacher per prompt. ATD probes the behavioral shadow of post-training by selecting prompts where the teacher and student's shared public ancestor is nearly indifferent between two ordinary words. A student initialized from this ancestor learns solely from the resulting prompt-word pairs, without target-task examples, teacher logits, or teacher parameters. In the primary coding experiment with Qwen2.5-1.5B, 5,664nses yield a 5.34 pp gain on HumanEval+ over an exact nuisance-matched control thadisrupts prompt-resperiments showtransfer in scientific knowledge, commonsense reasoning, and reading comprehensins across additional model generations, sizes, and families. Functional analyses show that the learned sid composable, andthat its strength tracks the teacher's update strength.

1 Introduction

The paper asks whether post-training changes on task-unrelated inputs reveal information about a model’s improved capabilities. It introduces ATD to collect such decisions and reports that students trained on them can improve on held-out target tasks.

1 Introduction

  • Prior subliminal-learning work showed that behavioral traits can pass through semantically unrelated training data, motivating tests of whether individual decisions reveal private updates and transfer capabilities.The paper frames these as observability and capability-transfer questions.Source: §1 · p. 1
  • ATD uses the public ancestor to find task-unrelated prompts where two ordinary words are nearly tied, then trains a same-ancestor student on the teacher’s single-word responses.The student receives no target-task examples, teacher logits, or updated parameters.Source: §1 · p. 1, Figure 1 · p. 2
  • 5.34 pp on HumanEval+ over an exact-matched control: Qwen2.5-1.5B also transfers across six multiple-choice benchmarks and additional model generations, sizes, and families.Code- and science-trained teachers produce their largest benefits on corresponding tasks; gains vary across tasks and models.Source: §1 · p. 2
  • The findings indicate that unrelated decisions can expose a private update and that many such observations can support target-task improvement in a student.The paper describes this information as capability-relevant, not as target-task training data.Source: §1 · p. 2

2 Related work

The paper situates ATD among work on subliminal learning, distillation, and informative query selection. Its distinction is to study capability transfer from actively selected, single-token observations of a private post-training update.

Subliminal learning and its mechanisms

  • Subliminal-learning studies transmit behavioral traits through semantically unrelated data, with follow-up work examining informative tokens, representations, paraphrases, preference labels, and independent examples [2] [3] [4] [5] [6] [7].ATD instead focuses on capability transfer and what selected single-token observations reveal about a private update.Source: §2 · p. 2

Knowledge distillation and on-policy distillation

  • Classical and language-model distillation use teacher distributions, generated responses, rationales, or solutions [8] [9] [10] [11] [12], while on-policy methods query student trajectories [13].ATD studies learning without target-task responses or teacher probabilities.Source: §2 · p. 2

Active learning, model extraction, and fingerprinting

  • Active learning, extraction, fingerprinting, and data-free distillation use informative queries or transfer inputs [17] [20] [22]; ATD tests whether decisions on unrelated inputs improve absent target tasks.The query set uses uncertainty from the public ancestor to observe a private update.Source: §2 · p. 3

Task vectors

  • Task arithmetic represents post-training as a parameter difference from an ancestor [25], motivating behavioral study of the changes that update induces.Source: §2 · p. 3

3 Problem setup

The setup assumes a public ancestor from which both a private post-trained teacher and a student originate. ATD observes the teacher through a restricted black-box interface and uses task-unrelated decisions to test for capability-relevant information.

3 Problem setup

  • The teacher is a private post-training of public model M0, and the student is initialized from the same ancestor.Source: §3 · p. 3
  • The teacher returns one greedy next token per query; the student gets no target-task data, and the query set must be fixed before teacher responses are observed.Probabilities from M0 may be used to construct queries.Source: §3 · p. 3
  • A carrier is a prompt–response pair whose visible text is unrelated to the target capability; improvement over controls that break the pairing is evidence of transfer.Source: §3 · p. 3
  • Near ties let small update-induced preference changes reverse a word choice; across prompts, choices constrain the update along carrier-gradient directions, while orthogonal components leave those logit differences unchanged.The paper presents this as a local approximation and tests empirically whether students learn capability-relevant information.Source: §3 · p. 3

4 Active Taskless Distillation

ATD selects task-unrelated prompts using the public ancestor’s near-ties, queries the teacher once per prompt, and trains a student on retained prompt–word pairs. The student’s loss is based on the teacher’s response token, with the prompt providing context.

4 Active Taskless Distillation

  • ATD selects task-unrelated prompts with two ordinary single-token candidates, using probabilities from the public ancestor to identify near-ties.Source: §4 · pp. 3–4
  • The normalized probability of one candidate must be within 0.02 of 0.5, and the public model’s full-vocabulary top token must be one of the pair.Source: §4 · p. 4
  • Prompts and candidate pairs are fixed before querying; each prompt is queried once, and responses outside the candidate pair are discarded without resampling.Each retained binary choice is a one-bit observation of the behavioral shadow.Source: §4 · p. 4
  • The same-ancestor student minimizes full-vocabulary cross-entropy on retained prompt–word pairs, where only the response token contributes to the loss.The prompt supplies context.Source: §4 · p. 4

5 Experiments

The experiments test whether a student initialized from a public ancestor can recover a private post-training capability using only task-unrelated carrier responses. They measure coding transfer against matched controls, then examine robustness, breadth, the structure of the recovered signal, and conditions under which transfer does not occur.

5.1 Identification of a specific capability

  • 51.22% on HumanEval+ matches the teacher’s aggregate coding score and exceeds the exact nuisance-matched permutation by +5.34 pp, the shuffle by +5.03 pp, and the teacher-free arm by +4.57 pp.The exact nuisance-matched control fixes the completion multiset and per-bin difficulty, so the comparison isolates prompt–token pairing.Source: Table 1 · p. 5, §5.1 · p. 5

5.2 Robustness

  • +4.80 pp on HumanEval+ across five acquisitions and three training seeds; all 15 signal–control pairs and all five acquisition means are positive.The between-acquisition standard deviation is 0.83 pp, and students outperform exact-matched controls across three matched adaptation regimes.Source: §5.2 · p. 5, Figure 2 · p. 6

5.3 Generalization

  • All seven task-specific students outperform teacher-label-shuffle controls, with gains from 0.81 pp to 5.03 pp across coding and multiple-choice tasks.The multiple-choice endpoints cover scientific knowledge, commonsense reasoning, and reading comprehension.Source: Table 2 · p. 6, §5.3 · p. 6
  • Mean HumanEval+ pass@1 gains over shuffle controls are positive for Qwen3-1.7B, Qwen3-4B, and Llama-3.2-1B.The paired 95% confidence intervals include zero for the three additional models.Source: §5.3 · p. 6, Figure 3 · p. 7

5.4 Analysis of the transferred signal

  • Across three sources and endpoints, each shadow’s largest positive effect is on its matched target, while off-diagonal effects are near zero or negative.A 50/50 mixture of near-orthogonal math and code shadows recovers both directions.Source: Figure 4 · p. 7, §5.4 · p. 7
  • On 76 seed–task cells where the teacher fixed a HumanEval+ task, ATD students solved 64.47% (49/76), versus 21.05% (16/76) for control; their task-level changes aligned more with the teacher.The reported cosine alignment is .701 for signal and .398 for control; recovered direction also tracks teacher update strength.Source: §5.4 · p. 7
  • Active near-boundary acquisition exceeds passive acquisition by +4.27 pp at matched query budget and +5.03 pp at matched training rows on HumanEval+.The comparison attributes the difference to concentrating queries near the ancestor’s decision boundary, rather than query count or data volume.Source: §5.4 · pp. 6–7, Table 3 · p. 8

5.5 Large teacher gains do not guarantee transfer

  • Neither a memorized-solution teacher nor a substitution-cipher teacher with a large target-task advantage produces transfer in its ATD student.The authors interpret these cases as evidence that a large teacher gain alone is insufficient; the discussion also notes reliance on actively constructed carriers and a known public ancestor, with transfer not reliable in every setting.Source: §5.5 · p. 8

6 Instrument extensions

The section explores two refinements of ATD on the same near-boundary carriers: a reference-relative objective and wider multi-token observations. RPM can strengthen CE in some settings but does not consistently outperform it; the wider readout is strongest on arithmetic and smaller but consistently positive elsewhere.

6 Instrument extensions

  • The section presents RPM and multi-token answers as exploratory refinements of the same near-boundary ATD instrument, not separate methods.RPM targets the residual preference introduced by the update by subtracting the frozen public margin between the candidate words.Source: §6 · p. 8, §6.1 · p. 8

6.1 Reference-relative pair-margin (RPM) objective

  • On HumanEval+, RPM’s shuffle contrast rises from +5.03 to +6.30 pp, while RPM does not dominate CE across settings.On PIQA both objectives remain positive; on ScienceQA their student accuracies are essentially equal, and RPM’s larger shuffle-relative gap reflects a lower shuffle baseline.Source: §6.1 · p. 8, Table 4 · p. 9

6.2 Multi-token observations

  • At K=20, the wider readout gives +6.37 pp on GSM8K, while HumanEval+ and ScienceQA effects are near +1 pp and positive across all six seeds.The intervals for the two non-arithmetic endpoints reach close to zero at this scale; the extension uses the same carriers and exact nuisance-matched control.Source: §6.2 · p. 9, Table 5 · p. 9

7 Conclusion

Post-training changes decisions even on inputs unrelated to the training task, and those behavioral shadows can carry information useful for a same-ancestor student. ATD extracts that information through single-word responses to near-boundary prompts; the primary coding experiment reports a 5.34 pp HumanEval+ gain over an exact nuisance-matched control.

7 Conclusion

  • 5.34 pp on HumanEval+: a same-ancestor student trained on 5,664 single-word responses outperforms an exact nuisance-matched control.The effect persists across independently constructed acquisitions.Source: §7 · p. 9
  • The behavioral shadow also retains information about the source and strength of the teacher’s update, and observations from two teachers can jointly influence the student.Its transfer usefulness varies across tasks and models; alignment with the teacher does not always yield downstream gains.Source: §7 · p. 9

A Worked ATD examples

The examples show how ATD turns neutral, target-unrelated prompts into single-word training pairs and how specific coding rules can appear in signal-trained solutions. The examples also differ in how clearly they separate signal seeds from matched controls.

A Worked ATD examples

  • A retained carrier uses a neutral short-story prompt and asks the model to choose one of two ordinary words; the public ancestor’s near-tie makes the pair eligible.For the jacket/tie example, the public model assigns P(tie)=0.5044 after normalization over the pair; the teacher returns jacket, and ATD stores only the prompt–word pair.Source: A Worked ATD examples · p. 12
  • The HumanEval/158 find_max example separates signal from controls: every signal seed and the private teacher count distinct characters, while the base and all controls rank by word length.The correct distinct-character rule does not appear in the carrier data, which a frozen audit finds free of code, digits, or task terms.Source: A Worked ATD examples · p. 12
  • The HumanEval/18 example is a weaker separator: all four signal seeds use overlapping substring counts, but one matched control does too.The public base and three other matched controls advance by the full substring length and undercount.Source: A Worked ATD examples · p. 12

B Carrier dataset and training setup

The coding acquisition uses public-ancestor near-tie prompts and retains one teacher token per prompt as target-unrelated training data. Controls reuse the carriers and training procedure while changing label assignments; the exact control additionally matches token frequencies and public difficulty while disrupting prompt–token correspondence.

B Carrier dataset and training setup

  • 5,664 of 17,858 coding near-tie queries were retained as carriers, each contributing one teacher token and no audited code, math, benchmark, or task terms.The 128-word vocabulary and near-tie selection use only the public ancestor; the retained rows are the entire training signal.Source: B Carrier dataset and training setup · p. 12
  • The code teacher is DPO-updated on filtered preference pairs, while six multiple-choice teachers use task-specific SFT LoRA; the student never trains on the teacher’s task corpus.The code pairs overlapping HumanEval+ and MBPP+ were removed before teacher training.Source: B Carrier dataset and training setup · p. 13
  • Students use rank-16 all-linear LoRA on a fixed 157-step schedule, and target-task endpoints are evaluated only after training.The exact control reuses the signal arm’s prompts, row count, data order, optimizer, and update count, differing only in label assignment.Source: B Carrier dataset and training setup · p. 13

C Control-arm construction

  • Global shuffle permutes teacher-chosen sides across carriers, while public-label and random-marginal controls use ancestor choices and independently sampled sides, respectively.The exact control complements shuffle by holding the chosen-word multiset fixed.Source: C Control-arm construction · pp. 13–14
  • The exact nuisance-matched control preserves the teacher’s token multiset and per-margin-bin flip counts while forcing only 50% teacher agreement, breaking prompt–token correspondence.The 50% agreement target is an imposed design choice; satisfying the side constraints can make the integer program infeasible.Source: C Control-arm construction · p. 14, Figure 5 · p. 14

D Identification analyses

Identification analyses find that teacher-direction alignment tracks prompt–token correspondence and is concentrated in high-divergence carrier states. A mixed student also recovers both near-orthogonal math and code directions, while composition-ratio endpoint evidence is less conclusive.

D Identification analyses

  • With the completion-token multiset fixed, teacher-direction alignment rises monotonically from .399 to .460 as correct prompt–token pairings increase from 0% to 100%.Spearman ρ=1 in all three seeds; the mean 100%−0% contrast is +.0613.Source: D Identification analyses · p. 15, Table 7 · p. 15
  • Destroying the top 20% most teacher–ancestor-divergent states costs .022 alignment, 1.56× the random-destruction loss and 2.86× the bottom-20% loss.The ordering holds in all three seeds, and top-state destruction still leaves cosine .437 above the full-shuffle floor of .395.Source: D Identification analyses · p. 15, Table 8 · p. 15

E Two-source composition

  • A 50/50 math–code mixture recovers both near-orthogonal teacher directions, with alignment advantages of +.046 for math and +.062 for code in 3/3 seeds.The between-source cosine is −.0002, consistent with recovery of both directions rather than a single winner.Source: E Two-source composition · p. 15, Table 9 · p. 15
  • As the carrier mixture shifts from all-code to all-math, the mathematics coefficient rises and code coefficient falls monotonically, but the GSM8K endpoint difference is not significant.The control-corrected 100%-versus-0% endpoint difference is +0.28 pp; the dose-response evidence is functional rather than an endpoint claim.Source: E Two-source composition · p. 15

F Functional-mechanism diagnostics

Functional diagnostics measure whether students’ representational changes align with teacher updates and how that alignment varies with training. The recovered functional direction tracks update strength, while downstream endpoint scores are noisier and do not establish the same fine-grained ordering.

F Functional-mechanism diagnostics

  • The diagnostic uses held-out target-unrelated prompts to form model fingerprints and compares student changes with teacher directions using distinct cosine and projection metrics.Teacher logits enter only this held-out diagnostic; the student’s training labels remain single hard tokens.Source: F Functional-mechanism diagnostics · pp. 15–16
  • In the carrier-mixture sweep, recovered mathematics coefficients rise and code coefficients fall, while the executable GSM8K endpoint’s +0.28 pp 100%-versus-0% difference is small and not significant.The endpoint is noisier than the functional coefficients and does not recover their fine strength ordering.Source: Figure 6 · p. 16
  • The teacher-aligned functional contrast is positive by step 4 and clearly formed by step 16, while HumanEval+ first separates significantly at step 80 and reaches +5.34 pp at step 157.The authors describe the timing as suggestive of an early teacher-aligned change, not evidence that the direction is acquired before capability.Source: F Functional-mechanism diagnostics · p. 16, Table 10 · p. 16

G Graded strength recovery

  • Recovered projections reproduce teacher update-strength ordering on code and SciQ trajectories, with Spearman ρ=1 in 3/3 seeds each, including a non-monotone code trajectory.The executable endpoint is too noisy to rank checkpoints; graded recovery is reported as a property of the functional direction.Source: G Graded strength recovery · p. 17, Table 11 · p. 17

H Boundaries of transfer

The tested transfer boundaries include failures with incompatible ancestors and near-zero recovery on a coding benchmark where the teacher’s own gain is weak. The authors treat these as empirical scope limits, not universal impossibility results.

H Boundaries of transfer

  • Neither tested incompatible-ancestor pair transferred despite large teacher gaps, but the failures do not establish a general identifiability condition.The pairs also differ in tokenizer, family, and scale.Source: H Boundaries of transfer · p. 17, Table 12 · p. 17
  • On MBPP+, where the teacher gap is +2.38 pp, ATD effects are +0.33 pp and +0.53 pp, both including zero.The authors contrast this with the HumanEval+ teacher gap of +7.93 pp and characterize coding transfer as conditional rather than unconditional.Source: H Boundaries of transfer · p. 17, Table 13 · p. 17
Loading 2609.29233v1…