PaperlayerView exact revision on arXiv

Source-linked AI summary

Compress-Distill: Reasoning Trace Compression for Efficient Knowledge Distillation

Maxime Griot, Paul Steven Scotti, Tanishq Mathew Abraham

arXiv:2606.05988v1cs.LGcs.CL

TL;DR

Long reasoning traces raise distillation costs and encourage verbose student outputs. This paper compresses teacher traces before distillation and finds substantial efficiency gains, but raw traces retain the highest accuracy.

  • Problem

    Long chain-of-thought traces increase supervised fine-tuning costs and teach students to generate verbose, expensive outputs.

  • Method

    The pipeline generates correct teacher traces, rewrites them with instruction-tuned compressors, and fine-tunes students on raw, compressed, or answer-only targets.

  • Results

    Compressed traces reduce training tokens to 12–30% of raw, cut wall-clock training by 2.0–7.6×, and shorten inference reasoning by 3–19×, while raw traces remain most accurate.

  • Takeaways & Limitations

    Reasoning-trace compression is a practical accuracy–efficiency trade-off rather than a free improvement, retaining up to 96% of raw-trace accuracy.

  • Takeaways & Limitations

    Compressed students do not Pareto-dominate raw students: they use fewer tokens but have lower accuracy.

Abstract

from arXiv · show

Reasoning models produce long chain-of-thought traces that are costly to distill and encourage verbose student outputs. We study post-hoc compression of such traces before knowledge distillation. Two teachers, Qwen3.5-397B-A17B and gpt-oss-120B, generate about 283k correct traces each; two instruction-tuned models then compress them to 8.6-21.0% of their original character length. Across a 48-run main grid plus seven Qwen-teacher truncation ablations, compressed traces reduce training tokens to 12-30% of raw, speed up training by 2.0-7.6x, and shorten inference outputs by 3-19x with smaller reductions under the shorter gpt-oss teacher. However, raw traces retain the highest downstream accuracy at every scale and for both teachers. A length-matched raw-trace truncation ablation shows that compression is not merely benefiting from a smaller token budget: model-compressed traces usually beat or match naive truncation, especially for smaller students, while maintaining shorter inference outputs. Overall, reasoning-trace compression offers an accuracy-efficiency trade-off rather than a free improvement: students retain up to 96% of raw-trace accuracy while gaining up to 18x higher per-token efficiency, and at the 0.8B scale under LoRA compressed traces narrow the raw-vs-compressed gap but do not exceed raw.

1 Introduction

The paper studies whether teacher reasoning traces can be compressed before distillation to reduce training and inference costs. Across a broad empirical evaluation, compression improves efficiency but remains less accurate than raw traces, making it an accuracy–efficiency trade-off rather than a free improvement.

  • Motivation: Long chain-of-thought traces improve step-by-step reasoning but increase fine-tuning costs and encourage verbose, expensive student outputs.
  • Method: The pipeline generates correct teacher traces, rewrites them with instruction-tuned compressors, and fine-tunes students on raw, compressed, or answer-only targets.
  • Results: Raw traces outperform compressed traces at every evaluated scale and under both teachers, while answer-only training is cheapest but performs worst and is unstable under full fine-tuning.
  • Analyses and limitations: A length-matched truncation ablation tests whether compressed traces help beyond a reduced training-token budget, alongside analyses of termination failures and unstable answer-only training.The paper characterizes compression across teachers, domains, compressor models, training cost, accuracy, inference length, truncation, and per-token efficiency.

2 Related Work

Prior work establishes chain-of-thought prompting and teacher-trace distillation as ways to improve and transfer reasoning, while this paper inserts a separate trace-compression step before distillation. Related efficiency work reduces reasoning costs through trace optimization or inference-time control; this study fixes greedy decoding and an 8,192-token cap to isolate training-time compression.

  • Chain-of-thought reasoning: Chain-of-thought prompting improves arithmetic, commonsense, and symbolic reasoning, including in zero-shot settings.Reasoning-tuned models such as DeepSeek-R1 and QwQ are explicitly trained to emit long deliberation traces.
  • Knowledge distillation for reasoning: Teacher-trace distillation transfers reasoning ability to smaller models, typically using the teacher’s full verbose trace as the target.This paper instead rewrites the teacher’s trace with a separate compressor before distillation.
  • Reasoning trace optimisation: Reasoning-trace optimization reduces long-trace costs through token pruning, shorter generated CoTs, token-budget estimation, RL, pruning, or self-training.The cited approaches include Xia et al. (2025), Kang et al. (2025), Han et al. (2025), Aggarwal and Welleck (2025), Luo et al. (2025), Munkhbat et al. (2025), and Zhang et al. (2026).
  • Inference-time efficiency for reasoning: Inference-time efficiency work controls decoding budgets through overthinking analyses, adaptive slow/fast reasoning, and sketch-like rationales.This study keeps greedy decoding and an 8,192-token cap fixed to isolate the training-time intervention.

3 Method

COMPRESS-DISTILL is a three-stage, independently resumable pipeline that samples and verifies correct teacher reasoning traces, compresses them, and fine-tunes students on alternative trace formats. Its evaluation grid compares raw, compressed, and answer-only supervision across teachers, students, and training methods, with a length-matched raw-trace ablation for the Qwen teacher.

  • Pipeline: Three independently resumable stages write JSONL outputs that feed the next stage.This defines the overall COMPRESS-DISTILL pipeline structure.
  • Stage 1: Trace collection: Teacher responses are answer-extracted, verified against canonical answers, and resampled up to K times; only correct traces are retained.The sampled response contains a reasoning trace and final answer, with correctness filtering before downstream processing.
  • Stage 2: Compression: A low-temperature compressor generates shorter traces that preserve essential reasoning, key insights, and final answers, while recording per-example character compression ratios.Two compressors run independently to produce parallel compressed datasets.
  • Stage 3: Distillation: Students are fine-tuned with assistant-only next-token prediction on raw traces, compressed traces, or answer-only outputs using sample packing.The answer-only condition replaces the entire reasoning block with an empty string.
  • Evaluation design: 48 main-grid runs span two teachers, four students, four data sources, and LoRA or full/FSDP one-epoch fine-tuning, alongside a length-matched raw-trace truncation ablation.The main grid contains 27 Qwen-teacher runs and 21 gpt-oss-teacher runs; the ablation matches each raw Qwen trace to the corresponding Ministral-compressed token length while keeping the final answer unchanged.

4 Experimental Setup

The experiments use in-distribution datasets spanning four domains and two MMLU subject groups, with additional OOD-reasoning evaluation. Training uses standardized Axolotl configurations, while evaluation reports group accuracy and generation-efficiency metrics.

  • Datasets: In-distribution experiments cover ten datasets across four domains plus two MMLU subject groups, with SVAMP and GPQA Diamond used for trace generation only.Downstream evaluation also includes OOD-reasoning datasets.
  • Training: Training uses Axolotl with BF16, FlashAttention 2, CutCrossEntropy, 16,384-token sequences with sample packing, and a 5% validation split.LoRA and full-fine-tuning configurations use separate learning rates and optimization settings.
  • Evaluation: Each student is evaluated with the same chat template used during training.
  • Evaluation: Generation is greedy with an 8,192-token cap, and results report per-group accuracy, overall accuracy, inference output length, and per-token efficiency.Reported accuracy groups include Math, Science, Medicine, Common-sense, OOD-Reason., and OOD-Knownledge.

5 Results

Compression removes substantial redundancy from reasoning traces and improves training and inference efficiency, especially for the longer Qwen teacher. However, raw traces retain the highest downstream accuracy across scales, making compression an accuracy–efficiency trade-off rather than a strict improvement.

  • Trace generation: 283,335 Qwen traces are substantially more verbose than gpt-oss-120B traces, with GPQA Diamond averaging 18,734 versus 13,407 characters and MultiArith 1,750 versus 716.The teachers have similar accuracy, but gpt-oss answers 281,911 questions correctly and produces traces roughly half as long character-for-character.
  • Compression behavior: ρ=0.086 is the most aggressive aggregate compression ratio, achieved by Ministral-14B under Qwen, versus ρ=0.142 for Llama-70B.Under gpt-oss, mean ratios are ρ=0.147 for Ministral-14B and ρ=0.210 for Llama-70B; GPQA Diamond compresses hardest at ρ=0.054 under Qwen Ministral-14B.
  • Efficiency: 5.8–7.6× speedups at 12–16% of raw tokens occur for the 0.8B student under Qwen, while gpt-oss yields only 2.0–4.1× at 22–30%.The 8B/9B Qwen students gain 3.2–4.9×, and gpt-oss-20B gains 5.1–6.3×; wall-clock benefits lag token reductions.
  • Downstream accuracy: Raw traces achieve the highest overall accuracy in every evaluated teacher–student–method configuration, including the Qwen truncation ablation.Representative full-finetuning scores are 0.866 / 0.834 / 0.817 for Qwen3.5-9B and 0.844 / 0.776 / 0.767 for gpt-oss-20B, ordered raw / Llama-70B / Ministral-14B.
  • Length-matched ablation: At matched training-token budgets, model-compressed traces usually beat or match naive truncation, especially for Qwen3.5-0.8B LoRA and Llama-3.1-8B methods.Truncation is competitive with aggressive Ministral compression for Qwen3.5-9B and gpt-oss-20B, but trails or ties the less aggressive Llama-70B compressor.
  • Accuracy–efficiency trade-off: 3–5 accuracy points are exchanged for about 5–10× per-token efficiency at 9B, so compression is preferable when inference cost or latency dominates rather than peak accuracy.Compressed students do not Pareto-dominate raw: raw has strictly higher accuracy, while compressed traces use strictly fewer tokens.

6 Analysis

Compression’s accuracy cost varies with compressor, student scale, training method, output structure, and domain: raw traces remain strongest overall, while compressed traces consistently transfer brevity and can outperform when raw outputs truncate. The analysis identifies where compression is most cost-effective and where its accuracy disadvantage is largest.

  • Compressor comparison: Llama-70B-compressed traces usually outperform Ministral-14B-compressed traces, while Ministral compresses more aggressively under both teachers.Mean ρ is 0.086 vs. 0.142 under Qwen and 0.147 vs. 0.210 under gpt-oss for Ministral-14B versus Llama-70B.
  • Accuracy gap: The compressed-vs-raw accuracy gap appears at every scale, but is narrowest for 0.8B LoRA and varies with student scale and training method.For LoRA, the reported 0.8B values are 0.528 / 0.506 / 0.500 for raw / Llama-70B / Ministral-14B.
  • Training method: Compressed traces are most cost-effective for LoRA, whereas full fine-tuning is weakest at the smallest scale and answer-only training generally collapses.The answer-only baseline remains competitive only at Qwen3.5-9B LoRA, at 0.781 vs. 0.862 raw.
  • Inference brevity: Compression transfers shortened training traces to inference: students trained on compressed traces produce similarly short median reasoning lengths across all students and both teachers.Under Qwen, medians are 1,004–1,085 chars for Llama-70B-compressed and 685–745 chars for Ministral-14B-compressed; under gpt-oss, they are 831–1,020 and 612–717 chars.
  • Inference brevity: Length matching alone does not reproduce compression’s brevity, because students often decode longer from truncated traces; learned output policy depends on shortened-trace structure, not only token count.Qwen3.5-9B Full decodes 316 vs. 148 median reasoning tokens, and gpt-oss-20B Full decodes 249 vs. 156.
  • Domain structure: Medicine has the largest raw advantage, Science the smallest, and Math can favor compression when raw traces truncate heavily.Under Qwen Full across three larger students, Medicine’s raw advantage is 4.7–8.5 points versus 1.4–4.0 points for Science; at Llama-3.1-8B, Math compression beats raw under Full and LoRA.

7 Conclusion

Post-hoc compression makes reasoning-trace distillation substantially more efficient, reducing training and inference costs, but raw teacher traces retain the highest accuracy across scales and teachers. The raw-versus-compressed accuracy gap is smallest for 0.8B LoRA.

  • Conclusion: 12–30% of raw training tokens, 2.0–7.6× faster wall-clock training, and roughly 3–19× shorter median inference reasoning summarize compression’s efficiency gains.These results come from a 48-run main grid plus seven Qwen-teacher truncation ablations.
  • Conclusion: Raw teacher traces remain best in accuracy at every scale and for both teachers.Compression therefore presents an efficiency gain with an accuracy cost rather than a universal improvement.
  • Conclusion: The raw-versus-compressed accuracy gap is smallest at 0.8B under LoRA.

Limitations · A Per-Dataset Trace Generation · B Per-Dataset Compression

The section identifies prompt and decoding-cap limitations, then organizes per-dataset trace-generation and compression statistics for both teachers. Compression ratios are milder for the shorter-trace gpt-oss teacher, while truncation can underestimate compression’s length advantage.

  • Limitations: A single generic compression prompt leaves open whether domain-specific instructions would change per-domain accuracy gaps.Mathematics-specific preservation instructions are given as an example but are not explored.
  • Limitations: An 8,192-token decoding cap causes raw-trace truncation across teachers and student scales, ranging from 0.6–1.8% to 2.5–8.6% of records.The reported reasoning-length advantage of compression is therefore a slight underestimate at larger scales.
  • Limitations: The Qwen-teacher 0.8B Llama-70B Full compressed run truncates 23.1% of records despite short median outputs, suppressing per-token efficiency.This additional truncation is specific to that run.
  • A Per-Dataset Trace Generation: Table 4 reports per-dataset trace-generation statistics for both teachers under rejection sampling.Its total counts correct traces retained, while average reasoning and total measures report mean reasoning length and completion tokens per correct trace.
  • B Per-Dataset Compression: Table 5 reports compression statistics by teacher, compressor, and dataset.The table defines ρ as mean compressed length divided by mean original length.

C Training-Time Efficiency · D Downstream Evaluation Results

Compression substantially improves training-time efficiency across student scales, while raw traces remain the strongest downstream-accuracy reference. The evaluation covers 48 main-grid runs and seven Qwen-teacher truncation ablations, with several comparisons excluded by design.

  • C Training-Time Efficiency: Table 6 reports wall-clock training time and source-dataset trainable tokens for each student, method, teacher, and data source.The table compares both runtime and the number of trainable source-dataset tokens.
  • C Training-Time Efficiency: 3.5–7.6×, 2.0–4.9×, and 3.2–6.3× are the compression speedups for 0.8B, 8B/9B, and 20B students, respectively.These are wall-clock training reductions across both teachers, measured against raw traces.
  • C Training-Time Efficiency: 15.8–28.0× is the answer-only baseline’s training-cost advantage, but it is uniformly worse on downstream accuracy.The answer-only baseline is cheaper than compression but performs worse downstream.
  • D Downstream Evaluation Results: 48 main-grid runs plus seven Qwen-teacher truncation ablations comprise the downstream-accuracy evaluation.Table 7 reports per-group and overall accuracy across these records.
  • D Downstream Evaluation Results: “Overall” averages every evaluated record, while bold and underline mark within-(student, method) winners and second-best results per column.Table 7 defines these formatting conventions for interpreting downstream accuracy.
  • D Downstream Evaluation Results: Entries render as “—” for designated unavailable settings, including gpt-oss-20B LoRA under either teacher and truncation under the gpt-oss teacher.Other exclusions include answer-only for gpt-oss-20B or under the gpt-oss teacher.

E Length-Matched Truncation Ablation … G.1 Method

The truncation ablation matches raw-trace lengths to Ministral-14B-compressed traces, while inference efficiency is measured across the main grid and ablations. Faithfulness is evaluated by a judge that scores compressed traces against originals for fidelity, coverage, clarity, verdict, answer alignment, and failure modes.

  • E Length-Matched Truncation Ablation: Trunc. cuts each raw trace to the same per-example token length as its Ministral-14B-compressed counterpart, while L70 and M14 are model-compressed sources.This defines the Qwen-teacher length-matched truncation ablation reported in Table 8.
  • F Inference-Time Efficiency: Inference-time efficiency statistics cover the 48 main-grid runs plus seven Qwen-teacher truncation ablations.Table 9 reports truncation rate and accuracy normalized by mean completion tokens.
  • G Faithfulness of Compressed Traces: The compressor is treated as a black box that maps (q_i, t_i, a_i) to a shorter trace assumed to preserve answer-bearing reasoning.The canonical answer is verified against a_i* before compression and re-appended unchanged afterward.
  • G Faithfulness of Compressed Traces: The raw-versus-compressed downstream-accuracy gap provides an indirect upper bound on the cost of silent reasoning corruptions.Potential corruptions include merged derivation steps, hallucinated constants, and omitted case-split branches.
  • G.1 Method: A third LLM judge compares each compressed trace with its original using structured verdicts and scores for faithfulness, coverage, and clarity.Faithfulness requires claims and conclusions to remain supported; coverage measures preservation of essential steps, and clarity measures coherence as deliberation.
  • G.1 Method: The judge additionally records a faithful, partially_faithful, or unfaithful verdict and whether the compressed trace independently supports the canonical answer.Answer alignment is Boolean and is assessed from the compressed trace alone.
  • G.1 Method: Failure-mode labels include contradiction, unsupported_claim, omitted_key_step, answer_mismatch, overcompression, incoherent, and empty-trace conditions.The judge also supplies a short rationale for its structured assessment.
  • G.1 Method: Server-side JSON-schema validation makes structural parsing essentially free, with truncation at max_completion_tokens identified as the realistic parse failure.The judge prompt includes the question, canonical answer, and both traces, middle-truncated to 40k and 12k characters when necessary.

G.2 Results … H.5 Energy and Carbon Footprint

Compressed traces were judged overwhelmingly faithful, while the full pipeline required substantial upstream and student compute. Across training, trace generation, and compression, estimated emissions totaled 44.8 kgCO2eq.

  • G.2 Results: 98.1% of traces compressed by Llama-70B were faithful, versus 98.0% for Ministral-14B, with mean faithfulness of 4.98/5 and 4.97/5, respectively.Positive answer alignment was 97.4% for Llama-70B and 97.2% for Ministral-14B across the 283,260-trace corpus.
  • G.2 Results: ∼2% of verdicts were non-faithful, dominated by omitted_key_step, answer_mismatch, and unsupported_claim, while overcompression, contradiction, incoherent, and empty outputs were rare.The non-faithful rate was small relative to the raw-vs-compressed accuracy gap of ≤8.5 points at ≥8B, but remained non-zero.
  • H.1 Hardware: 48 main-grid student runs and seven Qwen-teacher truncation ablations used NVIDIA H100 SXM GPUs, with 4 or 8 GPUs per job depending on model and method.Runs used NVLink/NVSwitch within nodes and InfiniBand between nodes; upstream generation and compression used 8 H100s on one node.
  • H.2 Trace Generation and Compression Stack: Both upstream stages used vLLM with continuous batching and FP8 teacher weights, serving Qwen3.5-397B-A17B-FP8 and two compressors on 8×H100.The Llama-70B compressor used tensor parallelism 4 and data parallelism 2; Ministral-14B used tensor parallelism 1 and data parallelism 8.
  • H.3 Student Training Stack: Student training used Axolotl with BF16, sequence length 16,384, sample packing, FlashAttention 2, and optimizer-specific LoRA or FSDP configurations.The answer-only ablation stripped each correct trace’s entire <think>...</think> block while retaining the matching training setup.
  • H.4 Compute per Run: ≈280.9 H100-hours were consumed across the 55 student runs, including 264.9 H100-hours for the main grid and 16.0 H100-hours for truncation ablations.Raw-trace and full-FSDP runs dominated per-student compute, while answer-only, truncated, and compressed runs together used less than a third of each student total.
  • H.4 Compute per Run: ≈424 H100-hours were spent upstream: ≈348 H100-hours for teacher generation and ≈75.7 H100-hours for compression, roughly 1.5× student-training compute.Compression took 1 h 09 m with Ministral-3-14B and 8 h 19 m with Llama-3.3-70B.
  • H.5 Energy and Carbon Footprint: 44.8 kgCO2eq were estimated for student training plus upstream generation and compression, based on 704.9 GPU-hours, 805 W effective draw, and 79 gCO2eq/kWh.The effective draw included a 700 W H100 SXM TDP and datacentre PUE of 1.15.
Loading 2606.05988v1…