PaperlayerView exact revision on arXiv

Source-linked AI summary

Conflict-Aware Logit Adapters for Utility-Preserving Anti-Distillation

Quoc Phong Dao, Binh Anh Nguyen, Linh Ngo Van, Thanh-Toan Do, Mehrtash Harandi, Trung Le

arXiv:2610.31046v1cs.AI

TL;DR

Reasoning traces can improve model capability, but they also provide high-information supervision for student imitation, and existing defenses do not fit deployed black-box teachers that must preserve useful outputs. CALA trains a residual adapter on a frozen teacher to balance anti-distillation with utility. Across reasoning benchmarks, it reduces student imitation while preserving teacher utility, with better surface fluency than ADS on GSM8K.

  • Problem

    Reasoning traces can improve multi-step reasoning while providing high-information supervision for student imitation, and existing defenses do not fit black-box teachers that must preserve useful outputs.

    Source: §1 · p. 1, §3 · p. 3

  • Method

    CALA trains a lightweight residual adapter on a frozen teacher, using proxy-guided anti-distillation signals and conflict-aware updates to balance imitation resistance with utility.

    Source: §1 · p. 2

  • Results

    Across reasoning benchmarks, CALA reduces student imitation while preserving teacher utility; at approximately 79–80% mean teacher accuracy on GSM8K, mean student accuracy is 50.8% under ADS versus 47.7% under CALA.

    Source: §5.2 · p. 7, §6 · p. 10

  • Takeaways & Limitations

    CALA avoids proxy-student computation during decoding and has better surface fluency than ADS on GSM8K.

    Source: §6 · p. 10

    Evaluated robustness covers only the tested settings; arbitrary or adaptive students and paraphrasing- or verifier-based attacks remain unevaluated.

    Source: §5.2 · p. 10

Abstract

from arXiv · show

While reasoning traces improve model capability, they also provide supervision signals for downstream distillation. Initial efforts to mitigate this issue rely on decoding-time interventions, but they incur high inference overhead and can harm the teacher model's natural fluency. We propose Conflict-Aware Logit Adapters (CALA), a lightweight training-time defense. CALA attaches a small residual adapter to a frozen teacher model and trains only the adapter. A proxy student identifies positions where imitation is most sensitive, and the adapter perturbs the output distribution at these positions while remaining within high-probability regions. A conflict-aware optimization strategy balances anti-distillation and utility objectives, while regularization preserves the teacher's performance and fluency. At inference time, CALA incurs modest overhead and requires no auxiliary model. Experiments on reasoning benchmarks show that CALA substantially reduces successful student imitation while maintaining near-original teacher performance and generation quality. These results establish CALA as a practical and efficient alternative to decoding-time approaches.

1 Introduction

Reasoning traces can improve multi-step reasoning while also supplying high-information supervision for student imitation. CALA shifts trace protection from decoding to training, aiming to limit imitation while preserving useful teacher outputs.

1 Introduction

  • Reasoning traces support multi-step capability but also expose solution patterns that can help smaller students imitate a teacher.This creates a security tension between useful reasoning outputs and model replication.Source: §1 · p. 1
  • Existing output-privacy and data-poisoning defenses do not fit a deployed black-box teacher that must keep serving useful outputs.The setting gives the defender control over output distributions, not the learner’s training set or the teacher’s internals.Source: §1 · pp. 1–2
  • CALA trains a lightweight residual logit adapter on a frozen teacher, using proxy-student signals to perturb sensitive positions within high-probability supports.Utility regularizers and conflict-aware updates keep the output distribution close to the original teacher and reduce surface artifacts.Source: §1 · p. 2
  • The proposed evaluation jointly measures teacher utility, student imitation, and surface fluency across reasoning benchmarks and operating points.The approach shifts protection from online decoding to model training and avoids proxy-model computation at inference.Source: §1 · p. 2

2 Related Work

Related work establishes that reasoning traces serve both as capability aids and as supervision, while security work studies output leakage and defenses against imitation. CALA addresses the narrower setting in which the defender controls released teacher traces but not the attacker’s training pipeline.

2 Related Work

  • Reasoning methods show that intermediate traces can improve difficult multi-step tasks, while teacher-generated rationales and synthetic data can train smaller models.This dual use motivates preserving useful traces while reducing their value as imitation supervision.Source: §2 · p. 2
  • Output-leakage research studies extraction from black-box access, whereas data-poisoning work assumes influence over upstream training data.Here, the defender controls emitted traces rather than the attacker’s dataset construction.Source: §2 · p. 2
  • ADS targets the utility-distillability tension at decoding time, while DOGe [23] is a training-time alternative that fine-tunes the teacher’s output layer.Related approaches also include watermarking and trace-rewriting, but CALA aims to reduce traces’ usefulness as distillation supervision rather than attribute or detect text.Source: §2 · pp. 2–3

3 Motivation and Problem Formulation

The paper formalizes black-box trace distillation as a student learning from teacher-generated input-trace pairs, with the defender controlling only emitted traces. It motivates a defense that limits imitation without sacrificing trace fluency and usability.

3 Motivation and Problem Formulation

  • An attacker can collect full input-trace pairs from a deployed teacher and train a smaller student with next-token prediction, while seeing no teacher internals.The defender controls only the distribution of traces emitted by the teacher.Source: §3 · p. 3
  • ADS adds proxy-model inference cost and can preserve a correct final answer while degrading reasoning into incoherent, mixed-language, or malformed traces.Figure 1 contrasts ADS artifacts with the cleaner trace produced by CALA; accuracy alone therefore misses fluency and usability.Source: §3 · p. 3, Figure 1 · p. 3
  • The design goal is to reduce traces’ value as distillation data while preserving fluent reasoning for legitimate users.CALA moves intervention to training time and lets a single strength parameter control its effect without a decoding-time proxy model.Source: §3 · p. 3

4 Method

CALA trains a residual adapter on a frozen teacher, combining proxy-guided anti-distillation with teacher-derived utility anchors and conflict-aware updates.

4.1 Training-time defended teacher

  • The adapter alone changes teacher logits, while top-k final-layer and intermediate-layer anchors constrain updates to preserve high-probability continuations and fluency.The final-layer anchor stabilizes likely continuations; the intermediate projection favors smoother continuations.Source: §4.1 · pp. 4–5

4.2 Proxy-guided anti-distillation

  • A non-committed virtual ascent step probes a nearby proxy state to identify trace positions where imitation is locally harder.The virtual proxy update is used as a stop-gradient probe, not committed to the proxy model.Source: §4.2 · p. 5

4 Method

  • CALA encodes an ADS-like anti-distillation direction during training using a non-committed proxy ascent probe and entropy-weighted, top-k targets.The proxy highlights imitation-sensitive positions while the utility anchors shape the adapted distribution.Source: §4.2 · pp. 5–6

4.3 Conflict-aware optimization

  • An alignment-based adaptive coefficient increases anti-distillation pressure when utility gradients are orthogonal or opposed, but reduces it when they already align.This is intended to prevent utility regularizers from washing out the protective update.Source: §4.3 · p. 6

4.4 Generation-time strength control

  • At generation, scalar δ interpolates original and adapted logits: 0 restores the original teacher, 1 applies full defense, and intermediate values tune the trade-off without retraining.Generation then uses standard temperature and top-k sampling.Source: §4.4 · p. 6

5 Experiments

The experiments compare CALA with decoding-time antidistillation and temperature sampling across three benchmarks, measuring teacher utility, student distillability, fluency, and inference cost. CALA improves utility-distillability trade-offs in evaluated settings while retaining fluency and staying close to the base teacher’s inference cost.

5.1 Experimental Setup

  • The evaluation uses DeepSeek-R1-Distill-Qwen-7B as teacher, Qwen2.5-1.5B as training proxy, and Llama-3.2-3B as the downstream student across GSM8K, MATH, and MMLU.The adapter is trained on 6,800 MetaMathQA samples; each method is evaluated at four operating points, with traces distilled using a consistent procedure.Source: §5.1 · p. 7

5.2 Results & Discussion

  • At approximately 79–80% mean teacher accuracy on GSM8K, student accuracy is 50.8% under ADS and 47.7% under CALA; CALA also has strong high-utility trade-offs on MATH, while ADS leads at MMLU’s most aggressive setting.Temperature sampling follows a sub-optimal trade-off curve on GSM8K; ADS’s extreme MMLU reduction accompanies severe surface-level degeneration.Source: §5.2 · p. 7, Figure 3 · p. 8
  • CALA maintains higher fluency than ADS on GSM8K and MMLU, while its inference cost stays close to the base teacher’s and ADS is substantially slower with nearly twice the peak memory.Fluency is judged for surface quality while ignoring mathematical correctness and the final answer.Source: §5.2 · p. 8, Figure 4 · p. 9
  • On GSM8K, the full objective gives the best overall trade-off; removing the base anchor sharply reduces teacher utility, while removing virtual ascent or conflict-aware updates raises student accuracy from 34.2% to 41.0% and 42.0%.Jointly removing Lmid and the top-k restriction reduces fluency under stronger defense settings.Source: §5.2 · p. 9, Figure 5 · p. 10
  • Protection transfers from the Qwen training proxy to Gemma-2-2B-IT and remains competitive for Qwen2.5-3B in the high-utility regime; a 100-trace audit found no statistically significant validity difference from ADS.The validity audit reports 36/50 strictly valid CALA traces versus 31/50 for ADS (exact McNemar p = 0.332), without claiming superiority in reasoning correctness.Source: §5.2 · pp. 8–9
  • Under adaptive filtering that discards the 20% highest-perplexity traces, filtering partially recovers student performance; CALA is stronger near 90% and 84% teacher accuracy, while ADS is stronger at the two lower-utility settings.Across three seeds, CALA has lower mean student accuracy than ADS at the first three operating points, with much larger variability at the most aggressive setting; robustness is limited to evaluated settings, not arbitrary students or attacks.Source: §5.2 · pp. 9–10, Table 2 · p. 10

6 Conclusion

CALA trains a lightweight residual logit adapter on a frozen teacher to reduce black-box trace distillation while preserving teacher utility. Its scope is the studied setting; aggressive settings may reduce accuracy or fluency.

6 Conclusion

  • CALA reduces downstream student imitation while preserving teacher utility in high-utility regimes, with better surface fluency than ADS on GSM8K.It amortizes proxy-guided anti-distillation pressure into the model and avoids proxy-student computation during decoding.Source: §6 · p. 10
  • CALA is limited to the studied black-box trace-distillation setting, not all model extraction or adaptive distillation.Its effectiveness may depend on the training-time proxy; tokenizer-mismatched proxies and several attack types remain open, and aggressive settings can reduce teacher accuracy or fluency.Source: §6 · p. 10

A Analysis of the Conflict-Aware Update

The conflict-aware update combines utility-preserving and anti-distillation gradients. It stays closest to the utility gradient while enforcing a minimum first-order decrease in the anti-distillation objective.

A Analysis of the Conflict-Aware Update

  • The update is the closed-form solution of a constrained projection problem, up to a numerical stabilizer.The direction is d⋆ = gu + λ⋆ga; λ⋆ is zero when the constraint is inactive and otherwise adjusts the direction to meet it.Source: A Analysis of the Conflict-Aware Update · p. 14
  • The update chooses the direction closest to the utility gradient while requiring a minimum first-order decrease in the anti-distillation objective.The two objectives can induce conflicting gradients.Source: A Analysis of the Conflict-Aware Update · p. 14
  • When the utility gradient already has sufficient anti-distillation alignment, the update follows it; otherwise, it adds the smallest correction needed to meet the margin.The correction prevents the utility objective from washing out the anti-distillation signal.Source: A Analysis of the Conflict-Aware Update · p. 15

B.1 Models and Training Setup

The experiments freeze the teacher backbone and language-modeling head, training only a residual logit adapter. A proxy constructs the training signal, while separate students are used for downstream trace-distillation evaluation.

B.1 Models and Training Setup

  • DeepSeek-R1-Distill-Qwen-7B is the defended teacher; Qwen2.5-1.5B is the training-time proxy, and Llama-3.2-3B models are downstream students.The proxy’s token embeddings are resized to match the teacher vocabulary; ADS uses a Qwen2.5-3B proxy during decoding.Source: §B.1 · p. 15, Table 3 · p. 15
  • The teacher backbone and language-modeling head are frozen, and only the residual logit adapter is trained.The adapter is trained on 6,800 MetaMathQA samples with teacher-generated responses.Source: §B.1 · p. 15
  • The proxy takes a virtual gradient-ascent step to construct the anti-distillation target, then the virtual update is discarded.The training signal uses masked next-token cross-entropy gradients for each accumulation window.Source: §B.1 · p. 15

B.2 Adapter Training Hyperparameters

Adapter training computes utility and anti-distillation losses on restricted top-k token supports. The supplied hyperparameter table identifies the configuration but gives no additional values in these passages.

B.2 Adapter Training Hyperparameters

  • k = 50 restricts both anti-distillation and utility-preservation KL and JS terms to plausible high-probability tokens.Table 4 summarizes the full adapter training configuration.Source: §B.2 · p. 15, Table 4 · p. 15

B.3 Evaluation Details

The evaluation generates teacher traces at multiple operating points and uses them to train downstream students under a shared distillation protocol. Fixed prompts specify math and multiple-choice reasoning formats, with answer forcing for consistent final answers.

B.3 Evaluation Details

  • Four operating points vary CALA’s adapter strength and temperature, ADS’s temperature and antidistillation weight, or only temperature for the baseline.CALA traces use top-k sampling with k = 50 unless noted, and generation is capped at 1024 tokens.Source: §B.3 · p. 16

Math trace-generation prompt

MMLU trace-generation prompt

Answer-forcing string

  • Students are trained on problem-plus-trace completions with prompt tokens masked, then evaluated by extracted answers against task references.GSM8K and MATH use numeric or symbolic verification; MMLU uses the selected option against its ground-truth label.Source: Answer-forcing string · p. 16, Table 6 · p. 17

C Additional Experiments

Additional experiments examine matched-utility protection, robustness, ablations, manual trace quality, transfer, control strength, and a DOGe comparison. Results are mixed across settings, and some studies are reported as supplementary rather than primary evidence.

Additional experiments

Multi-seed robustness

  • At GSM8K operating points P1–P3, CALA’s mean student accuracy is lower than ADS’s by 4.46, 4.10, and 3.13 points, respectively.P4 has much larger variance and is not used as primary evidence; three seeds do not establish formal significance.Source: C Additional Experiments · p. 18, Table 9 · p. 18

Additional ablation result

  • At comparable 65–67% teacher accuracy, full CALA achieves 34.2% student accuracy, versus 41.0% without virtual ascent and 42.0% without the conflict-aware update.The ablation table includes joint objective/support removals and individual removals of Lbase, virtual ascent, and the conflict-aware update.Source: C Additional Experiments · p. 18, Table 10 · p. 19

Manual reasoning-quality audit

  • The manual audit found no statistically significant difference in strict trace validity between CALA and ADS (McNemar p = 0.332), and did not observe a higher invalid-rationale rate for CALA.The audit covered 100 traces from 50 GSM8K problems at approximately 79% matched teacher accuracy; it does not establish mechanistic faithfulness.Source: C Additional Experiments · p. 19, Table 11 · p. 19

Cross-task distribution shift

  • At matched 37.9% teacher accuracy, CALA reduces student accuracy from 31.2% under ADS to 27.2%; at 36.9%, the methods are effectively tied at 23.3% and 23.1%.The authors limit the transfer claim to the evaluated shift and do not claim uniform dominance over ADS.Source: C Additional Experiments · p. 20, Figure 7 · p. 20

Transfer across downstream students

  • CALA yields lower Gemma-2-2B-IT student accuracy at all four tested points, while the Qwen2.5-3B comparison is operating-point dependent.For Qwen, the methods differ by only 0.9 and 1.2 points at the first two high-utility points, with one favoring each method; ADS is stronger at the two more aggressive points.Source: C Additional Experiments · p. 20, Table 13 · p. 21

Inference-strength control

  • With τ = 0.5 fixed, increasing δ from 0.25 to 0.80 and 0.85 changes teacher/student accuracy from 89.7/55.5 to 79.2/47.3 and 65.1/34.2, respectively.The authors describe protection as strengthening monotonically while teacher utility falls nonlinearly.Source: C Additional Experiments · p. 21

DOGe comparison

  • DOGe’s usable checkpoints reach 82.93–86.87% teacher accuracy and 55.03–57.03% student accuracy, a weaker trade-off than CALA in this setup.DOGe results are checkpoint-sensitive and lack matched operating points; changing its defense strength requires retraining, unlike CALA’s δ control.Source: C Additional Experiments · p. 21

D Limitations & Broader Impacts

The reported defense is scoped to standard next-token distillation from reasoning traces, with untested attack variants and proxy constraints. The paper also details its surface-fluency judge and limits the judge’s role to diagnosis.

D Limitations & Broader Impacts

  • The evaluation does not cover paraphrasing, verifier-selected traces, larger training budgets, or arbitrary auxiliary data, and tokenizer-mismatched proxies would require probability mapping or a sequence-level surrogate.The evaluated setting is an attacker training a smaller student from teacher-generated traces with standard next-token prediction.Source: D Limitations & Broader Impacts · p. 21
  • Results are most consistent on structured reasoning tasks such as GSM8K and MATH, while MMLU and some aggressive settings are more mixed; stronger defense can reduce teacher accuracy or fluency.The operating point should be chosen with the intended use case in mind.Source: D Limitations & Broader Impacts · p. 22

E Fluency Judge Details

  • The fluency judge scores 100 sampled traces per operating point on a 1–100 scale, assessing surface quality while ignoring correctness and the final answer.The protocol removes model-specific control tokens and delimiters but preserves mathematical notation and raw LaTeX commands.Source: E Fluency Judge Details · p. 22

Fluency judge prompt

REQUIRED EVALUATION STEPS

F Example Traces

This appendix presents qualitative GSM8K traces illustrating surface degeneration under stronger ADS settings, alongside checklist disclosures on reproducibility, impact, statistical-significance reporting, and LLM use.

F Example Traces

  • The selected qualitative GSM8K examples illustrate a surface-degeneration failure mode observed under stronger ADS settings, while the displayed Bill solution computes his destination floor step by step as 4 × 3 + 6 = 18.Source: F Example Traces · pp. 23–24
  • The book-thickness example shows a trace with junk tokens, repeated reasoning, and long junk-token runs.The arithmetic remains visible, but the output is contaminated by repeated “pistons” and “pistotika” strings.Source: F Example Traces · p. 26
  • A separate example records an output that truncates abruptly after a stray “absurd:” token.The following passage gives a final answer of 69 gumballs.Source: F Example Traces · p. 27
  • The example labels compare Antidistillation Sampling and Conflict-Aware Logit Adapters at teacher accuracies of 65.1% and 64.8%, respectively.Source: F Example Traces · p. 31
  • Example traces show Antidistillation Sampling misreading Charmaine’s future age, repeating “clap” tokens, truncating output, or producing an empty reasoning block before answering 20.Source: F Example Traces · p. 31

NeurIPS Paper Checklist

  • The checklist says reproducibility details and open code with publicly available datasets are provided, but statistical analyses do not cover every experiment supporting the main claims.The reported analyses cover the GSM8K robustness study and manual audit.Source: NeurIPS Paper Checklist · pp. 33–34

10. Broader impacts

  • The checklist says the paper discusses both positive and negative societal impacts in its conclusion and Appendix D.Source: §10 · p. 35

11. Safeguards

  • The checklist marks safeguards as not applicable because the paper releases no data or models considered high risk for misuse.Source: §11 · p. 36

12. Licenses for existing assets

  • The checklist says the paper credits its experimental datasets and models and identifies their licenses, including MIT, Apache 2.0, and the Llama 3.2 Community License.Source: §12 · p. 36

13. New assets

  • The checklist marks new assets as not applicable because no new dataset, benchmark, or model checkpoint is a primary contribution.Source: §13 · p. 37

14. Crowdsourcing and research with human subjects

  • The checklist says the work involves neither crowdsourcing nor research with human subjects, so the human-subjects and IRB items are not applicable.Source: §14 · p. 37

16. Declaration of LLM usage

  • GPT-4o mini judges reasoning-trace fluency, while LLMs also serve as the defended teacher, proxy models, and downstream students.The passage identifies these uses as part of the paper’s core experiments and methods.Source: §16 · p. 38

Related papers

Loading 2610.31046v1…