PaperlayerView exact revision on arXiv

Source-linked AI summary

Self-Training Elicits Concise Reasoning in Large Language Models

Tergel Munkhbat, Namgyu Ho, Seo Hyun Kim, Yongjin Yang, Yujin Kim, Se-Young Yun

arXiv:2502.20122v3cs.CLcs.AIcs.LG

TL;DR

Current LLM reasoning traces can contain redundant tokens, motivating methods that reduce output length without sacrificing correctness. The paper self-trains models on concise paths obtained through best-of-N sampling and few-shot conditioning, achieving 30% shorter outputs on average while maintaining accuracy. It also examines how concise reasoning varies across models and tasks, with scope limited to the evaluated settings and prompting choices.

  • Problem

    Current LLMs often generate more reasoning tokens than necessary, incurring extraneous inference costs.

  • Method

    The method fine-tunes models on self-generated concise reasoning paths obtained through best-of-N sampling and few-shot conditioning.

  • Results

    30% average output-length reduction is achieved across GSM8K and MATH while maintaining average accuracy across five model families.

  • Takeaways & Limitations

    Curated self-generated data can unlock latent concise reasoning and support more efficient inference without additional inference-time overhead.

  • Takeaways & Limitations

    The study focuses on the evaluated task-specific settings and does not explore advanced few-shot prompting methods such as exemplar selection or many-shot prompting.

Abstract

from arXiv · show

Chain-of-thought (CoT) reasoning has enabled large language models (LLMs) to utilize additional computation through intermediate tokens to solve complex tasks. However, we posit that typical reasoning traces contain many redundant tokens, incurring extraneous inference costs. Upon examination of the output distribution of current LLMs, we find evidence on their latent ability to reason more concisely, relative to their default behavior. To elicit this capability, we propose simple fine-tuning methods which leverage self-generated concise reasoning paths obtained by best-of-N sampling and few-shot conditioning, in task-specific settings. Our combined method achieves a 30% reduction in output tokens on average, across five model families on GSM8K and MATH, while maintaining average accuracy. By exploiting the fundamental stochasticity and in-context learning capabilities of LLMs, our self-training approach robustly elicits concise reasoning on a wide range of models, including those with extensive post-training. Code is available at https://github.com/TergelMunkhbat/concise-reasoning

1 Introduction

CoT improves complex-task performance by allocating computation through intermediate tokens, but current LLMs often use redundant reasoning that increases inference cost. The paper hypothesizes that shorter correct paths already exist in model output distributions and proposes fine-tuning to elicit them.

  • Intermediate CoT tokens provide additional computation but increase inference cost and latency roughly with output-token count.
  • Current LLMs often generate redundant reasoning containing verbose explanations and repetitive phrasing that do not contribute to solutions.
  • Pretraining and emergent CoT reasoning were not explicitly optimized for efficient use of intermediate tokens, leaving open the possibility of more efficient reasoning.
  • Shorter correct reasoning paths within model output distributions indicate a latent capacity for concise reasoning.
  • The proposed approach uses best-of-N sampling, few-shot conditioning, and standard fine-tuning to distill concise reasoning into the model without inference-time overhead.
  • 30% average output-length reduction is achieved across GSM8K and MATH while preserving overall accuracy across diverse model families.

2 Preliminary Investigation

The preliminary investigation defines concise reasoning relative to a model’s default correct-path length and tests whether models can produce shorter correct solutions. Models frequently do so, whereas zero-shot prompting reduces length inconsistently and often harms accuracy.

  • Concise reasoning means correctly solving a problem with fewer output tokens than the model’s default output.
  • The default output length is estimated as the average length of correct paths across multiple stochastic generations rather than greedy decoding.
  • The study evaluates instruction-tuned and math-specialized models on GSM8K and MATH to assess robustness and performance preservation.
  • Frequent normalized path lengths below 100% show that models can produce correct solutions shorter than their default outputs.
  • 32.2% shorter reasoning from Fixed Budget prompting coincides with a 10.1% relative accuracy decrease on average.
  • Zero-shot prompting is inconsistent across model families and can fail to elicit concise reasoning from math-specialized models.

3 Method

The method self-trains models on concise correct paths generated through best-of-N sampling and few-shot conditioning. Few-shot conditioning improves sample efficiency, while combining it with best-of-N sampling yields stronger reduction without test-time sampling overhead.

  • Self-training fine-tunes models on their own output distribution to elicit concise reasoning while preserving reasoning capability and avoiding external data dependencies.
  • Naive best-of-N sampling selects the shortest correct path among N generated paths for each training question.
  • Naive best-of-N becomes sample-inefficient because further length reduction requires exponentially increasing generation costs.
  • Few-shot conditioning uses human, GPT-4o, or self-generated exemplars to bootstrap concise training generations.
  • 8-shot conditioning significantly reduces reasoning length, with FS-Human outperforming naive best-of-N sampling at N = 256.
  • Combining few-shot conditioning with best-of-N sampling produces largely independent and additive length reductions.
  • Self-training internalizes length reduction so concise reasoning does not require repeated test-time sampling or long prompts.

4 Experimental setup

The experiments evaluate concise-reasoning methods on mathematical reasoning tasks using five main post-trained models, with accuracy and output length as primary metrics. Baselines include prompting and supervised fine-tuning variants, while self-training methods are compared under matched generation budgets.

  • Models: Five main models, including moderately sized general and math-specialized post-trained models, are used for the key experiments.The models are Llama-3.2-3B, Gemma-2-2B, Qwen2.5-3B, Qwen2.5-Math-1.5B, and DeepSeekMath-7B.
  • Tasks and datasets: GSM8K and MATH are used as challenging mathematical reasoning datasets where only the final answer is relevant.The task selection also targets settings where chain-of-thought improves performance and models achieve moderate performance, enabling assessment of accuracy preservation.
  • Evaluation metrics: Accuracy and length are the two primary evaluation metrics, with length defined as the average number of output tokens across all reasoning paths.The evaluation also reports relative accuracy and relative length against a strong zero-shot baseline, using greedy decoding for reproducibility.
  • Evaluation metrics: Output-token length includes incorrect reasoning paths because generated tokens incur inference costs regardless of answer correctness.The study focuses on output tokens because input lengths are similar across methods and output tokens more strongly affect wall-clock latency.
  • Baseline methods: Baselines include zero-shot prompting, ground-truth-answer fine-tuning, human- and GPT-4o-supervised concise-reasoning fine-tuning, and Rational Metareasoning.Rational Metareasoning is reproduced as an existing fine-tuning method for concise reasoning.
  • Self-training budget allocation: Self-training methods match generation budgets by controlling the number of paths generated per question, while fine-tuning costs are treated as relatively negligible.Naive BoN uses 16 paths, standalone few-shot conditioning uses one, and FS-BoN uses 16, with additional default-distribution paths.

5 Results

Self-training methods reduce reasoning length while generally preserving accuracy across tasks, models, difficulty levels, and scales. Few-shot conditioning, especially combined with best-of-N sampling and augmentation, provides the strongest reductions, while shorter training rationales transfer to shorter outputs.

  • Main results: Naive BoN fine-tuning reduces output length by 12% on average with N = 16, but greater compression becomes progressively less sample-efficient.The method remains effective on models where zero-shot prompting failed to reduce length.
  • Main results: FS-Human reaches 67.96% relative length on GSM8K versus 87.17% for naive BoN.Few-shot conditioning therefore produces substantially shorter self-training targets than naive BoN.
  • Main results: FS-BoN delivers the largest length reduction among the self-training methods while maintaining relative accuracy on average.For math-specialized models, FS-GPT4o-BoN gives the greatest reduction except methods using external data, which substantially sacrifice accuracy.
  • Main results: Augmentation improves accuracy for few-shot conditioning and, under matched budgets, achieves the greatest length reduction with minimal accuracy degradation.Its reduction rates remain superior to naive BoN and Rational Metareasoning.
  • Generalization: 10–12% length reductions with minimal accuracy drops are observed in cross-task evaluation on out-of-domain mathematical problems.The method also achieves 16.51% average accuracy improvement and 26.82% average length reduction across business, chemistry, and physics domains.
  • Adaptive length reduction: 20%–40% reductions occur at easier MATH difficulty levels, while harder questions receive less compression.The method adapts output length to question complexity rather than applying uniform shortening.

6 Discussion

Current training practices and reasoning-oriented models prioritize reasoning performance or additional tokens rather than conciseness. The paper argues that lightweight self-generated fine-tuning can elicit concise reasoning already latent in models.

  • Default reasoning behavior: Current LLM training data and recent thinking-model reinforcement do not promote concise reasoning and may encourage redundant token use.The discussion links this behavior to training practices emphasizing reasoning performance rather than token-budget efficiency.
  • Implications: The paper argues that concise reasoning supervision or rewards could improve model efficiency, especially for thinking models with lengthy internal reasoning.This is presented as a potential benefit for future training pipelines.
  • Lightweight fine-tuning: Standard fine-tuning on self-generated samples is presented as sufficient to substantially reduce reasoning length because models already possess relatively concise correct reasoning paths.The claim echoes the Superficial Alignment Hypothesis.

7 Conclusion

The paper addresses redundant chain-of-thought reasoning by hypothesizing latent concise reasoning capacity and eliciting it through self-generated fine-tuning data. The FS-BoN method reduces reasoning length while maintaining accuracy.

  • The paper hypothesizes that language models possess latent concise reasoning capacity, evidenced by shorter correct reasoning paths.
  • The proposed fine-tuning methods use self-generated data from best-of-N sampling and few-shot conditioning to elicit concise reasoning.
  • 30% reduction in reasoning length is achieved by FS-BoN while maintaining accuracy.

Limitations and Future Work

The paper identifies boundaries around its task-specific fine-tuning study and points to broader training, prompting, scaling, and task-generalization studies as future work. It also situates concise verbal reasoning as complementary to reducing reasoning stages.

  • Advanced Training Schemes: The study focuses on standard fine-tuning with self-generated samples, while advanced reinforcement-learning-based training schemes remain unexplored.Preliminary expert iteration and BoN experiments showed promising accuracy and length-reduction results.
  • Few-Shot Prompting Exploration: Advanced few-shot prompting methods, including exemplar selection and many-shot prompting, were not explored and could further enhance fine-tuning.Direct few-shot prompting achieved comparable length reduction but incurred a slight accuracy loss, whereas fine-tuning incorporated BoN sampling and augmentation.
  • Extended Scaling Studies: The scaling study was limited to Llama 3.x models with 1B, 3B, and 8B parameters, leaving larger-model effectiveness for future study.
  • Concise Reasoning in General LLMs: Generalizing the approach beyond task-specific tuning remains an important direction, including possible multi-task training for efficient reasoning.
  • Related Approaches: The method focuses on reducing verbal redundancy within sentences, complementing prior work that reduces the number of reasoning stages.

B Methodological Details

The methodology generates concise few-shot exemplars from the target model through best-of-N sampling and a two-phase sampling-and-selection process. Candidate paths are sorted, filtered for correctness, and validated before use.

  • Few-shot exemplars are obtained from the target model itself through a two-phase process consisting of sampling and selection.
  • Sampling Phase: 128 training questions each receive 128 diverse reasoning paths generated with sampling temperature T = 0.7.
  • Selection Phase: Candidate paths are sorted by token count in ascending order before further selection.
  • Selection Phase: Only correct samples are retained by comparing oracle labels with model answers using parsing code.
  • Selection Phase: GPT-4o sequentially validates candidates until eight valid and concise examples are obtained, while alternative solutions from selected questions are excluded.
  • Dataset-Specific Selection: For MATH, exemplar selection is distributed across algebra and other categories to avoid choosing only very easy questions.

C.1 Datasets

The experiments use English mathematical-reasoning datasets with original train/test splits, model-specific answer parsing, and output-token-based evaluation. Fine-tuning and generation use standard tooling with separate decoding and sampling settings.

  • Datasets: GSM8K contains grade-school multi-step word problems, while MATH covers advanced mathematical categories and difficulty levels.Both datasets use English, straightforward language, and standard mathematical notation.
  • Datasets: The original train/test splits are used for fine-tuning and evaluation, with MATH results reported on MATH-500.
  • Evaluation: Accuracy is evaluated with Python-based parsing that standardizes numerical formats and handles mathematical notation.
  • Evaluation: Model-specific parsing strategies are used for MATH because different models require different answer-extraction approaches.
  • Length Metric: Length is defined as the average number of output tokens, excluding input tokens because output decoding contributes more to wall-clock latency.
  • Implementation: Fine-tuning uses HuggingFace Trainer, generation uses vLLM, evaluation uses greedy decoding, and analysis uses temperature sampling with T=0.7.

D.1 Full Analysis on Reasoning Length Distribution (Section 2.1)

The analysis finds substantial variation in reasoning length across model families, while normalized distributions show that models can often solve problems with fewer tokens than their typical outputs. The proposed methods reduce reasoning length while largely preserving accuracy and also yield practical time and memory savings.

  • Model variation: DeepSeekMath-7B averages 179 output tokens on GSM8K, compared with 298 tokens for Qwen2.5-3B.The same pattern persists on MATH, where solutions are generally longer because the problems are more complex.
  • Measurement validation: Character-level distributions preserve relative model patterns, indicating that output-length differences are not artifacts of tokenization schemes.Raw character counts also measure solution verbosity without tokenization effects.
  • Normalized reasoning length: All models show probability mass for correct solutions below their normalized typical length, although flexibility in concise reasoning varies across model families.Normalized length divides each solution’s token count by the mean length of correct solutions for that question.
  • Selection strategy: Question-wise shortest-sample selection reaches 99.03% average relative accuracy and 70.54% average relative length, outperforming selection across all questions on accuracy.Selection across all questions reduces relative length to 64.33% but lowers average relative accuracy to 96.03%.
  • Practical efficiency: 15.38% to 52.94% wall-clock savings and 2.50% to 6.25% peak-memory reductions are reported after fine-tuning with FS-GPT4o-BoN.Wall-clock measurements use vLLM, while memory measurements use HuggingFace with batch size 128 on a single H100 GPU.

E Sample Study

The sample study compares concise-reasoning examples and generated rationales across models, prompts, and datasets. Fine-tuning with FS-GPT4o-BoN consistently shortens generated reasoning while maintaining accuracy, with augmentation increasing correct training samples.

  • Generated rationale examples: Fine-tuned Llama-3.2-3B and DeepSeek-7B examples show consistently shorter rationales through reduced verbosity on GSM8K and MATH500.The examples are presented for both general and math-fine-tuned models.
  • Prompt examples: Eight-shot prompts use randomly shuffled chat-formatted examples, with representative human, GPT-4o-generated, and self-generated concise-reasoning samples.The examples cover GSM8K and MATH and are used in FS-Human, FS-GPT4o, and FS-Self.
  • Training-data augmentation: Augmentation increases the number of correct fine-tuning samples while keeping average solution length relatively stable.It adds previously unsolvable long solutions and replaces solutions with shorter zero-shot BoN solutions.
  • Evaluation tables: Tables 13 and 14 report relative accuracy and length across five main models, alongside absolute accuracy and token length for each model on GSM8K and MATH.The tables compare main methods against default prompting.
Loading 2502.20122v3…