PaperlayerView exact revision on arXiv

Source-linked AI summary

Improving the Chain-of-Thought Compressibility of Pruned Reasoning Models via Long--Short Contrastive Pruning

Tao Jing, Chen Kang, Feifei Zhai, Pengyuan Liu, Dong Yu

arXiv:2610.34336v1cs.CLcs.AI

TL;DR

Pruning and CoT compression address model-size and generated-token costs, but pruning can add redundant or incomplete reasoning and make models less compressible. The paper combines CSD pruning with preference pairs covering problems a pruned model always or never solves. At 20% sparsity, this combination uses 8–22% fewer thinking tokens than RAC-calibrated pruning followed by the same compression, with accuracy within 0.5 points.

  • Problem

    Pruning criteria do not distinguish necessary from redundant reasoning, while preference-based compression lacks training signal on problems a pruned model always or never solves.

    Source: §1 · p. 2

  • Method

    CSD contrasts activations on long and short correct solutions of the same problem to preferentially prune neurons enriched in redundant reasoning, and preference pairs cover problems the pruned model always or never solves.

    Source: §1 · p. 2

  • Results

    At 20% sparsity on four math benchmarks, the combination uses 8–22% fewer thinking tokens than RAC-calibrated pruning followed by the same compression, with accuracy within 0.5 points.

    Source: §1 · p. 2

  • Takeaways & Limitations

    The paper concludes that how a reasoning model is pruned matters more than how it is recovered, and pruning-side control of reasoning length works only while most capability is retained.

    Source: §6 · p. 8

    The study covers one model and one training run per configuration; pruned models decode at higher cost than dense models compressed by the same recipe, so the combination pays off when memory is the binding constraint.

    Source: §6 · p. 8

Abstract

from arXiv · show

Large reasoning models are costly to serve because they are large and because they generate long chains of thought (CoT). Weight pruning and CoT compression address these two costs separately, but they interfere in that pruning adds redundant reasoning on problems the model still solves and incomplete reasoning on those it no longer solves, and on harder problems makes the model less compressible by subsequent length-preference optimization. We trace this to a blind spot on each side: pruning criteria do not distinguish necessary from redundant reasoning, and preference-based CoT compression receives no training signal on problems a pruned model always or never solves. We propose Contrastive Set-Difference (CSD), which identifies neurons enriched in redundant reasoning by contrasting their activations on long and short correct solutions of the same problem and prunes them preferentially, and we extend preference pairs to the problems a pruned model always or never solves. Pruning 20\% of DeepSeek-R1-Distill-Qwen-7B, the combination uses 8--22\% fewer thinking tokens on four math benchmarks than pruning with reasoning-aware calibration (RAC) followed by the same compression, with accuracy within 0.5 points.

1 Introduction

Pruning and CoT compression target different inference costs, but pruning can lengthen reasoning and make it less compressible. The paper introduces CSD and pruning-aware preference pairs to address these problems, reporting fewer thinking tokens than RAC-calibrated pruning followed by the same compression.

1 Introduction

  • Pruning reduces per-token cost by removing parameters, while CoT compression reduces the number of generated tokens.Combining them could multiply savings for deployments needing both a smaller model and shorter outputs.Source: §1 · p. 1
  • Pruning can add redundant reasoning on problems the model still solves, incomplete reasoning on problems it no longer solves, and reduce compressibility on harder problems.The paper defines compressibility as the relative reduction in thinking length under length-preference optimization.Source: §1 · pp. 1–2
  • Pruning criteria score units on the model’s reasoning traces without separating necessary from redundant reasoning, while preference-based compression lacks contrasting samples when a pruned model always or never solves a problem.The paper identifies these as blind spots in the two approaches.Source: §1 · pp. 1–2
  • CSD contrasts activations on long and short correct solutions of the same problem to identify neurons enriched in redundant reasoning, while pruning-aware preference pairs cover problems the pruned model always or never solves.The approach leaves the compression objective unchanged and treats redundancy as excess over the shortest correct solution.Source: §1 · p. 2
  • At 20% sparsity, CSD with the proposed preference pairs uses 8–22% fewer thinking tokens than RAC-calibrated pruning followed by the same compression, with accuracy within 0.5 points.The comparison covers four math benchmarks.Source: §1 · p. 2
  • CSD’s removed neurons form a distributed population, and before compression its token savings come from shorter completed reasoning on problems the model still solves.These neurons are not redundant as parameters: the base criterion is less likely to prune them than an average neuron.Source: §1 · p. 2

2 Related Work

Prior work reduces reasoning-model size through pruning and shortens chains of thought through several training and inference-time approaches. These methods do not distinguish reasoning needed for a correct solution from reasoning that can be omitted, and preference-based compression may lack contrasting samples from a pruned model.

Reasoning-aware pruning

  • Structured pruning can remove neurons, attention heads, or hidden dimensions, but reasoning-model pruning methods retain units needed to reproduce reasoning without separating necessary from redundant reasoning.Calibration choice strongly affects outcomes, especially at high sparsity, and reasoning models are particularly sensitive to pruning.Source: §2 · p. 2

CoT compression

  • CoT compression shortens reasoning through fine-tuning, length-reward reinforcement learning, preference optimization, activation steering, or neuron-based early exit.Preference-based methods contrast samples from the same model on the same problem.Source: §2 · p. 2
  • Existing compression methods take the model as given, and preference-based methods need contrasting samples that a pruned model often cannot provide.The paper identifies this as a gap in applying preference-based compression after pruning.Source: §2 · p. 2

3 Method

The method treats reasoning redundancy as both removable neurons and neurons enriched in reasoning that a shorter correct solution omits. It combines contrastive pruning with preference pairs designed for the pruned model’s successes and failures.

3.1 Revisiting Redundancy in LRM Pruning

  • The base score estimates the output change from removing a neuron using its activation on the model’s own reasoning traces; layers prune the lowest-scoring neurons sequentially.Removed neurons are physically deleted, yielding a smaller dense model.Source: §3.1 · p. 3

3.2 CSD: Contrastive Set-Difference

  • CSD compares long and short correct solutions of the same problem, using position-matched activations to identify neurons enriched in reasoning beyond the shared core.Pairing solutions within a problem controls for difficulty; the short solution is the shortest correct sample and the long solution is drawn from other correct samples.Source: §3.2 · pp. 3–4
  • CSD removes neurons enriched in redundant reasoning alongside low-base-score neurons, while preserving the architecture and per-layer pruning budget.If no neuron passes the enrichment test, CSD reduces exactly to RAC-calibrated structured pruning.Source: §3.2 · pp. 3–4

3.3 Pruning-Aware Length Preference Optimization

  • Preference training uses self, dense-anchored, and length-anchor pairs, with shortest correct solutions chosen to target redundant or incomplete reasoning.Dense-anchored pairs cover problems the pruned model never solves; length-anchor pairs cover problems without self pairs where it has at least two correct solutions.Source: Table 1 · p. 5, §3.3 · p. 5

4 Experiments

The experiments compare 20%-pruned models using CSD or RAC-S, then evaluate length compression and benchmark performance. CSD paired with PLPO shortens reasoning more than RAC-S with PLPO, while retaining similar accuracy.

4.1 Setup

  • The setup prunes 20% of decoder linear parameters from MLP neurons, comparing RAC-S and CSD under shared per-layer budgets and calibration data.CSD uses long and short streams from correct dense-model solutions for 211 problems, while both criteria calculate the base score on the long stream.Source: §4.1 · p. 5
  • PLPO trains each pruned model on 3,983 problems, and evaluation covers MATH-500, GSM8K, AIME24, and AIME25 with accuracy and thinking-token measures.Training uses one epoch; benchmark sampling uses four samples per problem on MATH-500 and GSM8K and 16 on each AIME set.Source: §4.1 · pp. 5–6

4.2 Pruning Alone

  • RAC-S pruning lengthens reasoning and lowers accuracy versus the dense model, with the largest verbosity increase on GSM8K, where thinking tokens rise from 386 to 799.RAC-S also lengthens mean thinking by 15.5% on MATH-500 and 24.8% on AIME, where truncation rises by 13.1 points; accuracy falls by 3.1, 1.9, and 14.0 points on MATH-500, GSM8K, and AIME.Source: §4.2 · p. 6
  • Before compression, CSD reduces thinking by 4.4% in geometric mean on MATH-500 and 8.0% on GSM8K versus RAC-S, with no shortening on AIME.Accuracy differs by −0.4 points on MATH-500, is unchanged on GSM8K, and differs by +0.3 points on AIME, where truncation rises from 21.4% to 24.0%.Source: §4.2 · p. 6

4.3 Compressing Pruned Models

  • CSD's PLPO compressibility is higher than RAC-S's on every benchmark, but pruning makes compression less effective than for the dense model on MATH-500 and AIME.On MATH-500, geometric-mean reduction is 48.9% for CSD and 42.9% for RAC-S; PLPO reduces RAC-S by 30.3% on MATH-500 and 18.3% on AIME, versus 44.7% and 35.4% for the dense model.Source: §4.3 · p. 6
  • CSD+PLPO thinks 11.1%, 21.7%, and 9.6% less than RAC-S+PLPO on MATH-500, GSM8K, and AIME, with accuracy differences of −0.5, −0.4, and +0.1 points.Table 3 reports the comparison relative to RAC-S before and after PLPO; Table 2 defines Acc as pass@1 and Think as mean thinking tokens.Source: §4.3 · p. 6, Table 3 · p. 7, Table 2 · p. 7
  • CSD+PLPO requires about 0.59× the dense model's decoding compute on MATH-500; pruned models use 17% fewer parameters than the dense model.Per-token linear-layer FLOPs and weights are 0.815× the dense model, while total per-token cost rises from 0.82× at 1k context to 0.87× at 16k.Source: §4.3 · pp. 6–7

5 Analysis

The analyses find that CSD changes a small subset of RAC-S's neuron choices, primarily shortens reasoning on problems the pruned model still solves, and indicate that pruning-side improvements matter more than recovery training.

5.1 What Does CSD Remove?

  • CSD changes 6,456 of RAC-S's 121,464 pruned-neuron selections, displacing units that rank near the middle of the base criterion rather than just below its threshold.The displaced neurons' median base-score percentile is 46, and their enrichment is distributed across depth and across tokens in long solutions.Source: §5.1 · pp. 7–8
  • For most audited neurons, enrichment reflects stronger activation across long-solution tokens rather than specific words or rare tokens.For 53%, no single token contributes more than a quarter of enrichment; stronger activation on the same tokens accounts for a median 80% of enrichment.Source: §5.1 · p. 8

5.2 Where Do the Saved Tokens Come From?

  • RAC-S lengthens reasoning in all three MATH-500 difficulty strata, while accuracy loss and increased truncation concentrate in the longest 20%.Length increases are 19.8%, 13.4%, and 16.0%; the longest stratum loses 10.5 accuracy points and gains 11.0 truncation points.Source: §5.2 · p. 8
  • CSD significantly shortens the shorter half of MATH-500 problems versus RAC-S, with no significant accuracy difference in any stratum.The shorter-half reduction is 5.2%; changes in the middle and longest strata are smaller and not significant.Source: §5.2 · p. 8

5.3 Pruning-Side or Training-Side Optimization

  • Recovery of pruned models closes only part of the accuracy gap, while RAC-S retains much of the dense model's accuracy before recovery.The comparison reports RAC-S within 3.1 points on MATH-500 and 1.9 points on GSM8K, and 35.5% AIME accuracy versus 49.5% for the dense model; the models and protocols differ.Source: §5.3 · p. 8

6 Conclusion

The paper concludes that CSD-based pruning combined with preference pairs reduces thinking tokens relative to RAC-calibrated pruning and the same compression. Its gains are most relevant when memory, rather than decoding compute, is the binding constraint.

6 Conclusion

  • At 20% sparsity, the combination uses 8–22% fewer thinking tokens than RAC-calibrated pruning followed by the same compression, with accuracy within 0.5 points.The authors attribute most of the reduction to greater compressibility of the CSD-pruned model.Source: §6 · p. 8
  • CSD prunes neurons enriched in redundant reasoning by contrasting long and short correct solutions of the same problem, alongside preference pairs for problems the pruned model always or never solves.Source: §6 · p. 8
  • The study covers one model and one training run per configuration.Source: §6 · p. 8
  • Pruned models decode at higher cost than the dense model compressed by the same recipe, in exchange for 17% fewer parameters; the combination pays off when memory rather than decoding compute is the binding constraint.Source: §6 · p. 8

Related papers

Loading 2610.34336v1…