Source-linked AI summary
O1-Pruner: Length-Harmonizing Fine-Tuning for O1-Like Reasoning Pruning
Haotian Luo, Li Shen, Haiying He, Yibo Wang, Shiwei Liu, Wei Li, Naiqiang Tan, Xiaochun Cao, Dacheng Tao
TL;DR
Long-thought reasoning improves accuracy but incurs substantial inference overhead because reasoning length is not always well matched to problem difficulty. O1-Pruner uses length-aware RL-style fine-tuning to shorten reasoning under an accuracy constraint, and experiments report reduced solution length with modestly improved accuracy.
Problem
Long-thought models can generate redundant reasoning and do not reliably adjust reasoning length to problem difficulty, increasing inference overhead.
Method
O1-Pruner formulates length reduction relative to a reference model while preserving accuracy, using length-harmonizing fine-tuning across simpler and more complex problems.
Results
O1-Pruner significantly reduces generated solution length while achieving a modest improvement in accuracy across the reported experiments.
Takeaways & Limitations
The results support reducing redundant long-thought reasoning while maintaining high accuracy for more efficient inference.
Takeaways & Limitations
The method assumes that long-thought reasoning paths contain redundancies and lack proper coordination.
Abstract
from arXiv · showhide
Recently, long-thought reasoning LLMs, such as OpenAI's O1, adopt extended reasoning processes similar to how humans ponder over complex problems. This reasoning paradigm significantly enhances the model's problem-solving abilities and has achieved promising results. However, long-thought reasoning process leads to a substantial increase in inference time. A pressing challenge is reducing the inference overhead of long-thought LLMs while ensuring accuracy. In this paper, we experimentally demonstrate that long-thought reasoning models struggle to effectively allocate token budgets based on problem difficulty and reasoning redundancies. To address this, we propose Length-Harmonizing Fine-Tuning (O1-Pruner), aiming at minimizing reasoning overhead while maintaining accuracy. This effective fine-tuning method first estimates the LLM's baseline performance through pre-sampling and then uses RL-style fine-tuning to encourage the model to generate shorter reasoning processes under accuracy constraints. This allows the model to achieve efficient reasoning with lower redundancy while maintaining accuracy. Experiments on various mathematical reasoning benchmarks show that O1-Pruner not only significantly reduces inference overhead but also achieves higher accuracy, providing a novel and promising solution to this challenge. Our code is coming soon at https://github.com/StarDewXXX/O1-Pruner
1. Introduction
Long-thought reasoning improves problem-solving but increases inference overhead, while models exhibit redundant reasoning and length disharmony. O1-Pruner addresses this by optimizing reasoning length under an accuracy constraint.
- Long-thought reasoning enhances capabilities and accuracy but produces longer outputs and greater computational overhead.
- The method is evaluated against SFT and DPO on open-source long-thought reasoning models, with additional analyses of hyperparameters and dataset difficulty.
- The paper identifies length disharmony as redundant inference overhead caused by poorly coordinated reasoning lengths.
- O1-Pruner formulates inference efficiency as an optimization problem that maintains accuracy while reducing reasoning length.
- The paper presents O1-Pruner as a fine-tuning approach for improving reasoning efficiency while preserving accuracy.
2. Related Work
Prior work studies inference-time scaling, alignment, and chain-of-thought compression, while O1-Pruner analyzes long-thought reasoning through length distribution and RL-based optimization.
- Inference-time Scaling: Inference-time scaling improves outputs by allocating additional computation, including longer generated reasoning sequences.
- LLM Alignment: LLM alignment methods seek precise, logically consistent, safe, and preference-aligned responses; this paper extends that setting toward shorter outputs and accurate reasoning.
- CoT Compression: Chain-of-thought methods improve reasoning but add computational overhead, motivating approaches that allocate token budgets according to problem complexity.
- CoT Compression: Concurrent work addresses O1 overthinking with preference optimization, whereas O1-Pruner studies length distribution and uses an RL-based optimization method.
3. Revisiting the “Length Disharmony” in Long Thought Reasoning
The paper finds that accuracy does not increase consistently with reasoning length: shorter solutions can remain highly accurate, and average accuracy is higher for shorter responses. This motivates O1-Pruner to reduce redundant inference overhead.
- Length Disharmony: Length disharmony describes cases where shorter responses are sufficiently accurate, making longer reasoning a superfluous computational expense.
- Experimental Setup: The study samples 512 solutions for each of 64 MATH problems using Marco-o1 and QwQ-32B, then groups solutions into length intervals.
- Instance-Level Relationship: At the instance level, the highest accuracy occurs in short, intermediate, or long length intervals depending on the problem.
- Distribution-Level Relationship: At the distribution level, shorter response lengths are associated with higher average accuracy rates.
- Distribution-Level Relationship: Shorter responses may indicate more efficient identification of an optimal solution path with fewer reflection and backtracking steps.
- Implication: These findings motivate O1-Pruner, which seeks to maintain high accuracy while reducing redundant reasoning during inference.
4. Methodology
O1-Pruner formulates length-harmonizing fine-tuning as constrained optimization that reduces redundant reasoning while preserving accuracy. It uses reference-model samples and RL-style, off-policy optimization to train shorter, difficulty-sensitive reasoning paths.
- Training and Inference: The training procedure samples from the reference model, evaluates optimized-model outputs against those samples, and applies RL-style fine-tuning.The optimized model is then used for inference with reported improvements in speed and accuracy.
- Length-Harmonizing Objective: O1-Pruner targets redundant and poorly coordinated reasoning by shortening solution paths while maintaining the reference model’s accuracy.For simpler problems, it encourages shorter answers; for harder problems, it preserves longer correct reasoning paths.
- Reference Sampling: The method estimates reference behavior by sampling K solutions per problem and computing mean length and accuracy values.These sampled expectations provide the baseline used to evaluate candidate solutions.
- Optimization: Because the reward and accuracy functions are nondifferentiable, the method uses policy gradients and a PPO-style loss for off-policy training.The off-policy strategy samples from πref rather than πθ, simplifying data preparation and training.
- Reward Design: Length-Harmonizing Reward combines a shorter-output reward with an accuracy reward relative to the reference model.Outputs matching the expected reference length receive zero length reward, while longer outputs receive negative length reward.
5. Experiments
Experiments evaluate O1-Pruner on mathematical reasoning tasks using two long-thought models, multiple baselines, and metrics that jointly reflect accuracy and solution length. Across datasets and inference-time analyses, O1-Pruner reports the strongest accuracy-efficiency balance, while outcomes vary with penalty settings and training-data difficulty.
- Experiment Setup: The experiments evaluate Marco-o1-7B and QwQ-32B-Preview on MATH, GSM8k, and GaoKao using accuracy, solution length, and AES.Training uses 5,000 MATH problems with 16 sampled solutions for Marco-o1-7B and 12 for QwQ-32B-Preview.
- Main Results: O1-Pruner achieves the best trade-off between reasoning accuracy and length across the three mathematical datasets and both evaluated models.The comparison includes Fast-Solving Prompt, SFT, and DPO baselines.
- Main Results: 76.8% average accuracy accompanies a 40.5% solution-length reduction for Marco-o1-7B, while QwQ-32B-Preview reaches 89.3% accuracy with a 34.7% reduction.These figures are reported relative to the baseline in the respective experiments.
- Main Results: Fast-Solving Prompt reduces solution length but compromises accuracy, whereas SFT and DPO provide weaker accuracy-length balances than O1-Pruner.Fast-Solving Prompt has lower AES values, SFT makes marginal length gains, and DPO remains below O1-Pruner.
- Inference Time-Cost Analysis: O1-Pruner records the shortest inference times, taking slightly over 1 minute for Marco-o1-7B and 4 minutes for QwQ-32B-Preview.The time-cost analysis uses one A800 GPU for Marco-o1 and four A800 GPUs for QwQ-32B-Preview.
- Ablation Studies: Increasing λ raises both accuracy and inference length, with λ = 2 reported as a favorable accuracy-efficiency trade-off for Marco-o1-7B.Training-data difficulty also affects behavior: harder datasets produce longer solutions and higher accuracy, while easier datasets shorten solutions without improving accuracy.
6. Conclusion
O1-Pruner addresses length disharmony in long-thought reasoning by shortening generated solutions while preserving and modestly improving accuracy. Further analyses examine hyperparameters and dataset difficulty.
- O1-Pruner significantly reduces the length of model-generated solutions while modestly improving accuracy.The method is evaluated through extensive experiments and analyses of hyperparameters and datasets with varying difficulty.
- The method targets redundant computational overhead caused by length disharmony during inference.
- Additional experiments analyze how key hyperparameters and dataset difficulty affect O1-Pruner.