PaperlayerView exact revision on arXiv

Source-linked AI summary

CRISP: Compressed Reasoning via Iterative Self-Policy Distillation

Hejian Sang, Yuanda Xu, Zhengze Zhou, Ran He, Zhipeng Wang, Jiachen Sun

arXiv:2603.05433v7cs.LG

TL;DR

Reasoning models often spend unnecessary tokens on easy problems, increasing cost and sometimes compounding errors. CRISP distills a model’s own concise, instruction-conditioned behavior into its ordinary policy using on-policy reverse-KL training, shortening reasoning while preserving or improving accuracy across models and benchmarks.

  • Problem

    Reasoning models generate excessive deliberation that raises inference cost and can introduce errors, especially when easy inputs require only short answers.

  • Method

    CRISP conditions the same model on a conciseness instruction and minimizes per-token reverse KL between that teacher and the student on student-generated rollouts.

  • Results

    Across three model families, CRISP substantially shortens reasoning while preserving accuracy on strong Qwen3 models and improving accuracy when the base model has room to improve.

  • Takeaways & Limitations

    The results indicate that concise reasoning behavior can become the default without reducing general capability or requiring ground-truth answers, token budgets, or difficulty estimators.

  • Takeaways & Limitations

    CRISP’s teacher signal depends on the base model’s ability to follow conciseness instructions, and empirical evaluation currently focuses on mathematical reasoning.

Abstract

from arXiv · show

Reasoning models often generate far more tokens than a task requires, which raises inference cost and can compound errors. We introduce CRISP (Compressed Reasoning via Iterative Self-Policy Distillation), an on-policy self-distillation method that teaches a model to reason more concisely by distilling its own concise behavior back into itself. The method uses a single idea: condition the same model on a "be concise" instruction to obtain teacher logits, then minimize the per-token reverse KL divergence between the student and this teacher on the student's own rollouts. It requires no ground-truth answers, no token budgets, and no difficulty estimators. The reverse-KL objective is naturally difficulty-adaptive: it compresses easy problems aggressively while preserving the reasoning steps that hard problems require. On Qwen3-14B, CRISP cuts reasoning length by up to 56% on MATH-500 and 38% on the harder AIME 2024, while improving MATH-500 accuracy by up to 3.3 points over the base model and holding AIME 2024 accuracy within about one point. This behavior generalizes across model sizes and families: Qwen3-8B shows the same compression with accuracy preserved, and DeepSeek-R1-Distill-Llama-8B improves accuracy on all five benchmarks while shortening its responses. General capabilities are preserved across all three models. Code is available at https://github.com/HJSang/OPSD_Reasoning_Compression.

1 Introduction

CRISP addresses excessive reasoning by distilling a model’s concise, instruction-conditioned behavior into its ordinary policy without ground-truth answers. It combines on-policy self-distillation with difficulty-adaptive compression and preserves or improves benchmark performance.

  • Motivation: Reasoning models can waste tokens on easy problems, increasing inference cost and latency while sometimes introducing errors.Long deliberation remains useful on hard problems, but excess reasoning can degrade otherwise correct solutions.
  • Method: CRISP instructs the same model to be concise, then distills that behavior back into the model using per-token reverse KL on student rollouts.The teacher and student share parameters but differ in conditioning context.
  • Method comparison: CRISP uniquely combines on-policy training, no ground-truth answers, difficulty-adaptive compression, and entropy preservation.These are the four desiderata used in the method comparison.
  • Results: 32% to 57%: CRISP reduces Qwen3-8B and Qwen3-14B MATH-500 reasoning length while preserving accuracy.The reported range spans the two Qwen3 model sizes.
  • Results: Up to 3.3 points: CRISP improves Qwen3-14B MATH-500 accuracy, while AIME accuracy remains within about one point of the base model.Compression is larger on easier MATH-500 than on harder AIME benchmarks, and the method also improves accuracy on all five DeepSeek-R1-Distill-Llama-8B benchmarks while shortening responses.

2 Related Work

Prior reasoning-compression methods use reinforcement learning, supervised traces, prompting, or self-distillation, but each has important constraints. CRISP is positioned as an on-policy alternative that uses concise behavior rather than privileged answers or externally curated traces.

  • Reinforcement learning: Reinforcement-learning methods penalize length or impose budgets, often requiring ground-truth answers, difficulty estimates, or reduced exploration.Representative methods include L1, DiPO, DIET, Leash, DLER, and ThinkPrune.
  • Supervised fine-tuning: Supervised fine-tuning methods curate or distill short reasoning traces, but training on other reasoning distributions can cause shift and forgetting.Examples include SEER, TokenSkip, DAP/LiteCoT, and S3-CoT.
  • Training-free compression: Training-free methods alter prompts or decoding and are easy to deploy, but their compression is limited and can vanish when the prompt changes.Examples include Chain of Draft, TrimR, NoWait, and FlowSteer.
  • On-policy self-distillation: Earlier on-policy self-distillation methods use ground-truth answers, rich feedback, or other privileged contexts, whereas CRISP uses a conciseness instruction.CRISP contributes the application of instruction-defined concise behavior to on-policy self-distillation.

3 Method

CRISP treats the same model’s instruction-conditioned policy as a concise teacher and trains the unconditional student to match it on its own rollouts. Periodic teacher refreshes progressively strengthen compression while reverse KL and on-policy training stabilize the update.

  • Problem formulation: CRISP seeks shorter reasoning traces while maintaining accuracy, representing outputs as a reasoning trace followed by an answer.The model generates y = (r, a), with reasoning typically inside think delimiters.
  • Problem formulation: The teacher is the same model conditioned on a conciseness instruction, while the student uses the original input without that instruction.Both policies share parameters but receive different contexts.
  • Training objective: CRISP minimizes per-token reverse KL between student and stop-gradient teacher distributions on student-generated rollouts.The teacher weights are periodically synchronized with the student, and no gradients flow through the teacher.
  • Training objective: Reverse KL removes tokens avoided by the concise teacher while retaining reasoning steps the teacher still uses.Forward KL is unstable in this stale-teacher setting: on Qwen3-8B it collapses within about 100 steps, with near-zero accuracy and response length diverging to the token budget.
  • Teacher parameterization: Periodic teacher refreshes create progressively stronger compression targets beyond what a single frozen teacher can provide.The refreshed teacher is a more concise version of the student’s earlier behavior.
  • Difficulty adaptation: Compression adapts to difficulty because easy problems yield a stronger concise-teacher signal, whereas hard problems still require extensive reasoning.This behavior is formalized theoretically and verified empirically.
  • Implementation: CRISP uses standard supervised-training infrastructure with student and teacher forward passes, periodic weight copies, and no reward models or value functions.Per-token supervision also permits truncated rollouts; across 1K, 4K, 8K, and 30K limits, training remains robust with comparable compression and accuracy.

4 Theoretical Analysis

The analysis connects CRISP’s per-token loss to sequence-level KL and interprets it as implicit reward shaping. It explains accuracy preservation, difficulty-adaptive compression, bounded forgetting, and potential accuracy gains from reducing compounding reasoning errors.

  • KL interpretation: The per-token CRISP loss equals sequence-level KL by the autoregressive chain rule, enabling standard information-theoretic analysis.The identification supports later uses of Pinsker’s inequality and data processing.
  • Implicit reward: CRISP implicitly maximizes a reward combining task performance with a conciseness preference.The reward is the teacher-versus-student log-probability difference, with an entropy-like student term.
  • Accuracy preservation: The accuracy bound separates teacher quality from the distillation gap, so accuracy is preserved when teacher quality offsets the KL gap.The analysis states that improvement occurs when the teacher’s gain exceeds the distillation gap, matching the benchmark pattern.
  • Difficulty-adaptive compression: The compression signal is non-increasing with difficulty: easy problems receive stronger pressure, while hard problems receive weaker pressure because more steps are essential.This follows from decomposing tokens into essential and compressible contributions.
  • Difficulty-adaptive compression: Up to 56% on MATH-500 versus 32–38% on AIME: Qwen3-14B shows stronger compression on the easier benchmark without explicit difficulty estimation.The empirical pattern matches the proposition’s predicted allocation of compression pressure.
  • Bounded forgetting: For hard problems where the conciseness instruction has little effect, the conciseness gap is approximately zero, so forgetting is minimal where it matters most.The bounded-forgetting analysis contrasts this with off-policy SFT, whose distribution mismatch can be arbitrarily large.
  • Compounding error: Compression can improve accuracy under an independent per-token error model because removing tokens reduces opportunities for compounding errors.The predicted accuracy benefit grows exponentially with the number of removed tokens; correlated reasoning errors may amplify it in practice.

5 Experiments

CRISP is evaluated across in-domain math, out-of-domain capability, model-family, instruction, training, and sampling settings. It substantially shortens reasoning while generally preserving accuracy, with compression adapting to difficulty and excessive compression eventually harming performance.

  • Main results: CRISP reduces reasoning length by 32% to 57% on Qwen3-8B and Qwen3-14B while preserving accuracy, with Qwen3-14B gaining up to 3.3 points on MATH-500.DeepSeek-R1-Distill-Llama-8B improves accuracy on all five benchmarks while shortening responses.
  • Difficulty adaptation: 32% to 57% compression on MATH-500 exceeds the 5% to 38% reduction on harder AIME benchmarks, without explicit difficulty estimation.The compression signal adapts automatically through the KL objective.
  • General capabilities: Out-of-domain accuracy on GPQA-Diamond and MMLU stays within about one point of each model’s base accuracy, with DeepSeek improving slightly.The models are trained only on math prompts, while these benchmarks test general capabilities.
  • Qualitative behavior: CRISP examples reach the same correct answers as the base model with much shorter responses; v1 compresses more than v2, which retains additional reasoning on difficult cases.Both variants also produce the requested Answer: line in the cited format-change example.
  • Compression limits: About 2× compression is the sweet spot: further training yields modest extra reduction while accuracy drops first on AIME 2024 and later on MATH-500.At the end of the epoch, MATH-500 accuracy declines to ∼93% and AIME 2024 accuracy to ∼57%.
  • Instruction and training ablations: The difficulty-aware v2 instruction preserves accuracy best, while v1 achieves more compression; more prescriptive instructions trade additional reduction for accuracy loss.With M=50, v2 retains base-level accuracy at 31.6% / 17.1% compression, whereas v1 reaches 57.4% / 36.5%.

6 Limitations and Future Work

The study identifies instruction-following, evaluation scope, and teacher-quality characterization as boundaries for CRISP and directions for future work.

  • Instruction-following as an enabler: CRISP depends on the base model’s ability to follow conciseness instructions, while the minimum capability needed for effective self-distillation remains uncharacterized.Models with weaker instruction-following would provide a weaker teacher signal.
  • Progressive compression dynamics: On harder benchmarks such as AIME, the compression signal is weaker because the teacher itself requires more extensive reasoning.The paper interprets this behavior as difficulty-adaptive compression rather than a limitation.
  • Scope of evaluation: The current evaluation focuses on mathematical reasoning, so empirical validation on broader reasoning tasks remains a next step.The paper identifies code generation and scientific question answering as possible extensions.
  • Teacher quality characterization: A finer-grained characterization is still needed of when and why conciseness instructions improve or degrade accuracy across model families.The experiments show accuracy improvements from the conciseness-conditioned teacher, while the paper calls for deeper analysis of teacher quality.

7 Conclusion

CRISP uses on-policy self-distillation to make a model’s concise reasoning behavior persistent, without ground-truth answers, token budgets, or difficulty estimators. Across model families, it substantially shortens reasoning while preserving accuracy and improving it when the base model has room to improve.

  • Method: CRISP distills a conciseness-conditioned teacher into the same model by minimizing per-token reverse KL on the student’s own rollouts.The objective is also equivalent to sequence-level KL minimization for autoregressive models.
  • Method: CRISP requires no ground-truth answers, token budgets, difficulty estimators, reward engineering, or externally curated training traces.The teacher is the model itself conditioned on a conciseness instruction.
  • Conclusion: Across three model families, CRISP substantially shortens reasoning traces while preserving accuracy and improves accuracy when the base model has room to improve.The conclusions also report preserved general capability and no reduction in entropy.
  • Accuracy and forgetting: The accuracy-preservation guarantee depends on both the concise teacher’s accuracy gap and the student–teacher divergence, with the bound improving when the teacher is more accurate than the base model.In experiments, the concise teacher’s accuracy gap is consistently negative, so the bound can explain accuracy gains.
  • Difficulty-adaptive compression: CRISP compresses easier problems more aggressively while preserving more reasoning on harder problems, under a structural assumption that hard problems contain a larger fraction of essential tokens.The formal proposition relies on assumptions about essential-token fractions and category-level KL values; the latter is explicitly described as a modeling simplification.
  • Accuracy and forgetting: On-policy self-distillation controls divergence from the original model, and forgetting is predicted to be minimal on hard problems where the conciseness instruction changes the output distribution little.The bounded-forgetting result is stated for converged training and extends across periodic teacher-update windows.

A.5 Compression Reduces Compounding Error

The paper models reasoning errors as independently occurring at each token and shows that shortening a trace can improve accuracy by reducing opportunities for error accumulation. The predicted gain grows with the number of removed tokens and is illustrated quantitatively on MATH-500.

  • Probabilistic model: A trace of length L has correct-answer probability (1 − perr)^L when each token independently introduces an error with probability perr.The model assumes an incorrect intermediate step corrupts the final answer.
  • Compression effect: If compression reduces the trace from L to αL without increasing per-token error, the accuracy ratio is (1 − perr)^−(1−α)L.The number of removed tokens is (1 − α)L.
  • Compression effect: The accuracy improvement grows exponentially with the number of removed tokens, (1 − α)L.The proof uses −ln(1 − perr) ≥ perr and e^u ≥ 1 + u.
  • Quantitative interpretation: On MATH-500 with Qwen3-14B, removing approximately 2,330 tokens yields a lower-bound accuracy ratio of 1.23 and an exact exponential estimate of about 1.26 when perr = 10^-4.The calculation uses a base length of approximately 4,139 tokens and α ≈ 0.44.
  • Assumption: The independence assumption is conservative because positively correlated reasoning errors would make the benefit of removing error-prone tokens exceed the independent-error prediction.This is presented as a lower-bound interpretation of compression’s improvement.

B Survey of Reasoning Compression Methods

The survey organizes reasoning-compression methods by their reliance on correct answers, difficulty adaptation, and training or inference strategy. CRISP is motivated by gaps in comparability and by the need to compress while preserving accuracy.

  • Method survey: Table 4 surveys 13 reasoning-compression methods across length penalties, difficulty dependence, correct-answer requirements, and hard budgets.LP denotes Length Penalty in reward, DD Difficulty-Dependent, CA Correct Answer required, and HB Hard Budget.
  • Comparability: Reported accuracy and token-reduction results are not directly comparable because methods use different base models, benchmark splits, and evaluation protocols.NoWait is the only cited baseline evaluated on a CRISP base model, but its reported hard-set accuracy loss contrasts with CRISP’s accuracy preservation.
  • Main results: The concise-prompt baselines shorten responses, while CRISP extends compression further and better preserves accuracy.The comparison separates inference-only prompting from trained compression.
  • Main results: Table 5 compares base, concise-prompt, and CRISP rows using accuracy, average reasoning length, and token reduction relative to the base model.CRISP uses periodic teacher updates with M=50, while concise prompting applies the instruction only at inference.
  • Evaluation: Qwen3-14B accuracy is reported under answer-only, boxed-only, and dual scoring because correct answers can appear in either output format.The dual scorer counts correctness under either format and matches the main results table.
  • Evaluation: CRISP raises Qwen3-14B MATH-500 answer-only accuracy to 92.8% and dual accuracy to 95.2%, while boxed-only accuracy falls to 10.0%.The change reflects consolidation onto the Answer: format rather than a loss of dual-scored correctness.

E Training and Implementation Details

CRISP trains on student-generated rollouts while periodically synchronizing a concise teacher and optimizing reverse KL between their token distributions. The implementation evaluates this procedure across held-out math benchmarks and compares teacher-update and divergence choices.

  • Technical setup: Experiments run on one node with eight NVIDIA H200 GPUs using verl, FSDP with CPU offloading, SGLang rollouts, and degree-4 Ulysses sequence parallelism.These components provide actor–rollout–reference co-location and distributed training.
  • Training data: Training uses approximately 13,600 DAPO-Math-17k problems, with approximately 3,400 held out for validation and teacher prompts prepended with a conciseness instruction.The split is random with a fixed seed.
  • Training procedure: Each step samples a student response and applies one gradient update minimizing reverse KL on the student’s generated tokens, without filtering for correctness.Teacher and student forward passes are computed for each micro-batch.
  • Evaluation: Evaluation uses MATH-500, AIME 2024, and AIME 2025 with eight responses per problem and reports mean accuracy and average response token count.Sampling uses temperature 0.6, top-p 0.95, and top-k 20.
  • Teacher parameterization: The default periodic teacher updates use M=50 to balance progressive compression with training stability.Periodic refreshes allow the teacher to internalize earlier compression without continuous co-adaptation.
  • Teacher parameterization: The frozen teacher provides a stable target but becomes a weaker compression oracle as the student improves.This limits the maximum achievable compression.
  • Teacher parameterization: EMA teachers provide smooth progressive compression, while concurrent stop-gradient teachers offer aggressive compression but risk collapse.With α=0.995, EMA can exceed frozen-teacher compression but requires monitoring.
  • Teacher parameterization: M=1 causes training collapse: accuracy falls sharply and response length grows rather than shrinking.The moving teacher creates a positive feedback loop toward degenerate short-sequence behavior, while the reported ablation observes length growth.

G.3 Results

Reverse KL and JSD provide stable compression near base accuracy, whereas forward KL collapses into maximum-length generation and severe accuracy loss. The gradient analysis links this divergence choice to the observed training behavior.

  • Ablation results: Reverse KL and JSD maintain near-base accuracy with strong compression, while forward KL collapses to single-digit accuracy and maximum-length responses.JSD compresses slightly less than reverse KL, which is therefore used as the default.
  • Ablation results: By step 100, forward KL accuracy falls to single digits on MATH-500 and near zero on AIME 2024, while reverse KL and JSD remain near base accuracy.The collapse occurs after forward KL tracks the stable methods briefly.
  • Ablation results: Forward KL drives response length toward the 30K-token budget on both benchmarks, whereas reverse KL and JSD shorten responses smoothly.The length increase and accuracy collapse occur together under forward KL.
  • Mechanism: Reverse-KL updates are weighted by the student’s current distribution, limiting abrupt shifts, while forward-KL updates use the teacher distribution independently of student coverage.JSD inherits enough student weighting for stability but compresses less than pure reverse KL.

H Entropy Preservation During Training

CRISP shortens reasoning without collapsing policy entropy. Across Qwen3-8B and Qwen3-14B, entropy remains near its base-model level during training.

  • Motivation: Length-penalized reinforcement learning can cause entropy collapse, but CRISP’s compression does not induce that failure mode.The concern is that shorter outputs might accompany sharper next-token distributions and reduced diversity.
  • Measurement: Entropy is logged as mean full-vocabulary per-token Shannon entropy over the student’s on-policy response tokens.The quantity is measured for monitoring and does not enter the loss.
  • Results: Figure 11 shows entropy fluctuating narrowly around each model’s starting value without a downward trend toward zero.Thus the compression gains do not arise from a collapsed, overconfident policy.
  • Results: Qwen3-8B entropy ranges approximately from 0.27 to 0.34, while Qwen3-14B ranges approximately from 0.28 to 0.34 during training.The reported step-0 to step-99 values are 0.33 to 0.29 for Qwen3-8B and 0.33 to 0.32 for Qwen3-14B.

I Deep Planning Agentic Task

CRISP is evaluated on DeepPlanning agentic tasks requiring tool calls, constraint satisfaction, and long-horizon planning. Training substantially reduces response length while largely preserving planning quality.

  • Benchmark: DeepPlanning evaluates travel and shopping agents on tool use, constraint satisfaction, and multi-step planning.Travel planning covers itinerary construction with hard and commonsense constraints; shopping planning measures correctly carted products.
  • Setup: Qwen3-14B checkpoints are evaluated during CRISP training with a function-calling agent that iteratively executes tool calls.The setup evaluates 100 travel samples every 10 steps, with up to 400 LLM calls per case and repeated base-model runs for variance estimation.
  • Results: 51% fewer shopping response tokens and 42% fewer travel response tokens are achieved during training.Shopping tokens decrease from 12.5k to 6.2k by step 120, while travel tokens decrease from 5.3k to 3.1k.
  • Results: Planning quality remains largely preserved during substantial compression.Shopping match rate remains within baseline variance through step 70, at 25.8% versus 24.2% for the base model; the supplied passage truncates the travel-score result.
  • Output format: CRISP also consolidates model outputs onto the requested “Answer:” format, avoiding strict-grader failures caused by boxed-only answers.The base model may produce a correct boxed result without the requested answer line, whereas CRISP emits the requested format and shorter responses.
Loading 2603.05433v7…