PaperlayerView exact revision on arXiv

Source-linked AI summary

Unlocking Efficient Long-to-Short LLM Reasoning with Model Merging

Han Wu, Yuxuan Yao, Shuqi Liu, Zehua Liu, Xiaojin Fu, Xiongwei Han, Xing Li, Hui-Ling Zhen, Tao Zhong, Mingxuan Yuan

arXiv:2503.20641v2cs.CL

TL;DR

Overthinking in deliberate LLM reasoning increases response length without proportional quality gains, motivating long-to-short methods. This paper systematically studies model merging, finding up to 55% shorter responses without compromising performance, while identifying scale- and calibration-related boundaries.

  • Problem

    Iterative reasoning can generate redundant steps without proportional output-quality improvements, while existing long-to-short methods include computationally expensive or unstable approaches.

  • Method

    The paper empirically evaluates task-vector, SVD-based, and activation-informed merging of quick-thinking and reasoning models for long-to-short tasks.

  • Results

    Model merging reduces response lengths by up to 55% without compromising performance, with task-vector and activation-informed methods showing especially promising results.

  • Takeaways & Limitations

    Model merging offers a cost-efficient approach to mitigating overthinking while preserving reasoning quality.

  • Takeaways & Limitations

    Effectiveness is constrained by model-scale and candidate-model performance gaps, while activation-based methods depend substantially on calibration-data choices.

Abstract

from arXiv · show

The transition from System 1 to System 2 reasoning in large language models (LLMs) has marked significant advancements in handling complex tasks through deliberate, iterative thinking. However, this progress often comes at the cost of efficiency, as models tend to overthink, generating redundant reasoning steps without proportional improvements in output quality. Long-to-Short (L2S) reasoning has emerged as a promising solution to this challenge, aiming to balance reasoning depth with practical efficiency. While existing approaches, such as supervised fine-tuning (SFT), reinforcement learning (RL), and prompt engineering, have shown potential, they are either computationally expensive or unstable. Model merging, on the other hand, offers a cost-effective and robust alternative by integrating the quick-thinking capabilities of System 1 models with the methodical reasoning of System 2 models. In this work, we present a comprehensive empirical study on model merging for L2S reasoning, exploring diverse methodologies, including task-vector-based, SVD-based, and activation-informed merging. Our experiments reveal that model merging can reduce average response length by up to 55% while preserving or even improving baseline performance. We also identify a strong correlation between model scale and merging efficacy with extensive evaluations on 1.5B/7B/14B/32B models. Furthermore, we investigate the merged model's ability to self-critique and self-correct, as well as its adaptive response length based on task complexity. Our findings highlight model merging as a highly efficient and effective paradigm for L2S reasoning, offering a practical solution to the overthinking problem while maintaining the robustness of System 2 reasoning. This work can be found on Github https://github.com/hahahawu/Long-to-Short-via-Model-Merging.

1 INTRODUCTION

LLM reasoning has shifted toward deliberate System 2 processes, but iterative overthinking limits practical efficiency. This work studies model merging as a long-to-short alternative that combines quick and methodical reasoning capabilities.

  • System 1 models produce rapid, intuitive responses, whereas System 2 models use deliberate reasoning for complex tasks.
  • Iterative reasoning can add redundant steps without proportional gains in output quality, creating the overthinking problem.
  • The study systematically evaluates task-vector, SVD-based, and activation-informed model merging for long-to-short reasoning.
  • Up to 53% average response-length reduction is achieved while preserving baseline performance, with some configurations improving performance.
  • Model scale correlates with merging efficacy, with smaller models struggling to obtain long-CoT reasoning ability through parameter fusion.

2 RELATED WORK

Long-to-short reasoning seeks to reduce Chain-of-Thought inference costs while retaining accuracy, using compression, training, inference-control, and prompt-based approaches. Model merging provides a complementary parameter-level route, while this study excludes architecture-changing MoE methods.

  • Chain-of-Thought compression reduces inference costs by selectively omitting tokens or controlling the reasoning process.Related approaches include TokenSkip and CoT-Valve.
  • Reinforcement learning and target-budget methods allocate computation or calibrate rewards to control reasoning length.
  • Prompt-based long-to-short methods encourage concise responses but can be unstable because performance is sensitive to prompt variations.
  • Model merging integrates task-specialized models into one model without additional training and can mitigate catastrophic forgetting.
  • MoE-based merging is excluded because it typically changes model architecture and model size.

3 BACKGROUND

The paper frames long-to-short reasoning and reviews parameter-merging strategies that combine capabilities from fine-tuned models. These include averaging, task-vector arithmetic, conflict resolution, lightweight pruning, activation protection, low-rank estimation, and routing.

  • Model merging combines K fine-tuned models derived from a pretrained model into one model intended to inherit their capabilities.
  • MODEL MERGING: Average merging forms a model by arithmetically averaging model weights, with effectiveness depending on base-model quality and diversity.
  • MODEL MERGING: Task Arithmetic computes each task vector as δk = θk − θ0, then adds weighted task vectors to the base model.
  • MODEL MERGING: TIES-Merging prunes insignificant deviations, resolves parameter conflicts, and aggregates significant parameters with weights.
  • MODEL MERGING: DARE drops redundant parameters, adjusts retained directions to resolve conflicts, and weights key parameters during integration.
  • MODEL MERGING: AIM uses activation information to identify and protect critical pretrained-model weights during merging.
  • MODEL MERGING: LoRE-Merging approximates the base model and task vectors with low-rank optimization to reduce task-vector interference.
  • Long-CoT models generate both an extensive thinking process and a final answer, whereas short-CoT models directly output the answer.

4 EXPLORING MODEL MERGING METHODS FOR LONG-TO-SHORT REASONING

The study evaluates task-vector, SVD-based, and activation-based merging for long-to-short reasoning on Qwen models. Task-vector and activation-based methods substantially compress responses while preserving or improving reasoning performance, whereas SVD-based methods are less effective and calibration choices constrain activation-based methods.

  • Task-vector based merging: Task-vector methods are evaluated through Average Merging, Task Arithmetic, TIES-Merging, and DARE.Average Merging is treated as a special case of task-arithmetic merging.
  • Task-vector based merging: 34.6% response-length reduction accompanies a 15.8% accuracy improvement over the quick-thinking baseline for average merging.TA and TIES-Merging achieve 48-53% length reduction with accuracy within 1.0% of the reasoning model or a +0.3% average-score gain.
  • Task-vector based merging: 48-53% length reduction with accuracy parity or marginal gains makes TA and TIES-Merging the strongest task-vector approaches.DARE consistently underperforms the original merging methods.
  • SVD-based merging: SVD-based methods outperform average merging but remain weaker than advanced task-vector methods and depend on task-vector singular-value distributions.LoRE-Merging and Twin-Merging are examined as low-rank approaches to task-vector interference.
  • Activation-based merging: Activation-based methods surpass the DeepSeek-R1-7B baseline while reducing response length, with AIM-TIES reaching a 55.3% compression ratio.AIM-TIES improves performance by 0.2 points, while Sens-Merging achieves reasoning performance comparable to DPO with approximately 50% shorter responses.
  • Activation-based merging: Activation-based performance depends on calibration data, while Sens-Merging requires backward-pass gradient computation and AIM uses only forward-pass activations.Alternative calibration datasets produced inferior performance in the experiments.

5 ANALYSIS ON MODELS WITH DIFFERENT SCALES

The study examines model merging across 1.5B, 14B, and 32B models. Merging remains useful on simple tasks at smaller scale, but smaller models struggle with complex long-CoT reasoning, while larger models make substantial length reduction difficult without performance loss.

  • Smaller models: Task-vector methods remain effective across scales, while Sens-Merging consistently achieves the best overall performance in the scale evaluations.The evaluation covers GSM8K, MATH500, and AIME24.
  • Smaller models: On 1.5B models, merging performance lags behind 7B models, especially on complex tasks such as AIME24.Reflective responses are negatively correlated with final performance, suggesting that merged models can produce false reflections.
  • Smaller models: 1.5B-scale merging remains effective on simple tasks, but smaller models struggle to acquire long CoT reasoning through merging.This pattern is reported for TA, TIES-Merging, and Sens-Merging.
  • Large-scale models: On 14B and 32B models, reasoning performance is largely preserved, but response-length reductions are less significant than at smaller scales.Average merging and Sens-Merging can even increase response length.
  • Large-scale models: 58.6% length reduction from TA at coefficient 0.3 caused performance changes of +1.2 on GSM8K, -7.2 on MATH500, and -30.0 on AIME24.The result illustrates the difficulty of reducing length substantially while maintaining reasoning performance on larger models.
  • Large-scale models: Large-scale merging makes simultaneous performance preservation and substantial response-length reduction challenging, especially when merging models have large performance gaps.Average merging of QwQ-32B with Qwen-32B or R1-32B tends to produce excessively lengthy responses, with the worst performance when merging two System 2 models.

6 FURTHER ANALYSIS

Further analysis examines whether merged models retain reflection, adapt response length to difficulty, and expose practical limitations. Response length and reflective-response frequency increase with question difficulty, while merging is sensitive to hyperparameters, calibration data, and model-pair performance gaps.

  • Response Length and Difficulty: Response length positively correlates with question difficulty across System 1, System 2, and merged models.The same pattern appears across datasets and difficulty levels within MATH500.
  • Self-Critique and Self-Correction: Merged models retain self-critique and self-correction abilities, but reflection ratios do not predict reasoning accuracy.TIES-Merging and AIM-TIES achieve strong performance despite relatively low reflection ratios on GSM8K and MATH500.
  • Self-Critique and Self-Correction: Reflective responses become more common as question difficulty increases, and reflection ratio is positively associated with response length.The reflective-response measure identifies responses containing predefined keywords such as “re-examine” or “double-check.”
  • Failure Experience and Future Directions: Task-vector merging hyperparameter α is typically set above 0.5, but changes within 0.1 can shift overall performance by approximately 0.5-1.0 points.Increasing α is positively correlated with both reasoning accuracy and response length, creating an experimental burden when tuning settings.
  • Failure Experience and Future Directions: Activation-based merging depends substantially on calibration-data choice and sample size, with 10 versus 20 samples producing an approximately 1.0-point average-score gap.AIM-TIES prefers short-answer activations, whereas Sense-Merging performs better with both short and long answers plus the question.
  • Failure Experience and Future Directions: Existing merging methods struggle when 14B and 32B quick- and slow-thinking models have substantial performance disparities.The paper identifies surpassing the superior model with shorter responses as a future research direction.

7 CONCLUSION

The study presents model merging as a cost-efficient approach to Long-to-Short reasoning by combining System 1 efficiency with System 2 rigor. It reports response-length reductions of up to 55% without compromising performance and identifies future work in theory and broader applicability.

  • Conclusion: Model merging reduces response lengths by up to 55% without compromising performance in Long-to-Short reasoning.The approach integrates System 1 efficiency with System 2 rigor.
  • Conclusion: Task-vector merging balances simplicity and effectiveness, while activation-informed methods show promising improvement potential.The conclusion positions model merging as a cost-efficient approach to mitigating overthinking while preserving reasoning quality.

A.1 EXPERIMENTS CONFIGURATION

The experiments describe training, evaluation, and merging configurations used for the study. DPO training uses s1K with LoRA, while Table 7 defines the principal hyperparameters for the merging methods.

  • DPO Training: DPO training uses the full s1K dataset with LoRA rank 16, maximum length 4096, learning rate 5e-5, and 3 epochs.The best saved checkpoint is reported.
  • Configurations: Merged models are evaluated in BF16 except Sens-Merging models, which use FP32 because sensitivity factors are computed on CPUs.Evaluations use the standard Qwen evaluation toolkit.
  • Configurations: Table 7 defines α for TA, p for DARE, k for TIES-Merging, ω for AIM, and T for Sens-Merging.These symbols denote the methods’ coefficient, drop rate, trim ratio, balance factor, and temperature, respectively.
Loading 2503.20641v2…