PaperlayerView exact revision on arXiv

Source-linked AI summary

GRIP: Granular Reward-Guided Parameter Interpolation for Efficient Reasoning

Lam So, Canhui Wu, Han Lin

arXiv:2608.25583v1cs.CL

TL;DR

Verbose reasoning improves problem solving but raises inference cost and latency, while concise instruction models may lack comparable reasoning ability. GRIP freezes identical-architecture reasoning and instruction models, optimizes module-wise interpolation ratios with correctness-and-conciseness rewards, and achieves a stronger accuracy-efficiency trade-off than fixed-ratio and search-based merging.

  • Problem

    Verbose reasoning increases inference cost and latency, creating a need to preserve reasoning accuracy while improving response efficiency.

  • Method

    GRIP freezes identical-architecture reasoning and instruction models and uses reward-guided RL to optimize module-wise interpolation ratios for correct, concise responses.

  • Results

    GRIP reduces generation length while preserving or improving average accuracy and yields a stronger accuracy-efficiency trade-off than fixed-ratio merging and search-based baselines.

  • Takeaways & Limitations

    Learned per-layer, per-module fusion coefficients support efficient reasoning and cannot be reduced to a single global ratio.

  • Takeaways & Limitations

    GRIP is evaluated only at 4B scale on dense Transformers and assumes reasoning and instruction models have identical architectures, leaving larger-scale, MoE, and cross-family transfer open.

Abstract

from arXiv · show

Reasoning-oriented large language models often achieve strong problem-solving performance by generating long chains of thought, but this behavior substantially increases inference cost and latency. In contrast, instruction-tuned models tend to answer more concisely, yet often lack comparable reasoning ability. This accuracy-efficiency mismatch motivates a lightweight approach that combines the strengths of both models without full model retraining. In this paper, we propose GRIP (Granular Reward-guided Interpolation of Parameters), a reward-guided parameter interpolation framework for efficient reasoning. Given a reasoning model and an instruction model with identical architectures, GRIP assigns learnable interpolation ratios to individual modules and optimizes only these ratios while keeping both source models frozen. The interpolation ratios are trained with a reward signal that favors responses that are both correct and concise. Experiments show that GRIP achieves a better accuracy-efficiency trade-off than fixed or search-based merging baselines and further reveals module-wise fusion patterns associated with efficient reasoning.

1 Introduction

GRIP addresses the efficiency costs of verbose reasoning by combining reasoning and concise instruction behavior through reward-guided, module-wise parameter interpolation. It freezes both source models, optimizes only interpolation ratios, and improves the accuracy-efficiency trade-off over merging baselines.

  • Motivation: Overthinking increases token consumption and latency, while redundant reasoning can introduce self-contradictions or compounding errors.The paper frames efficient reasoning as preserving accuracy without unnecessarily long explanations.
  • Motivation: Prompt-based methods depend on instruction or token-budget design, whereas training-based methods usually require costly model-level optimization.These limitations motivate a lighter alternative that adjusts the accuracy-efficiency trade-off without updating the full model.
  • Motivation: Existing merging methods use fixed global coefficients or black-box search, limiting task-adaptive control over module-specific reasoning and concise instruction behavior.GRIP targets this limitation with granular module-wise fusion.
  • GRIP: GRIP combines identical-architecture reasoning and instruction models by freezing both sources and optimizing learnable module-wise interpolation ratios with RL-based rewards.The reward jointly favors answer correctness and response conciseness.
  • GRIP: GRIP achieves a stronger accuracy-efficiency trade-off than existing merging baselines and reveals module-wise patterns associated with reasoning behavior.The method is presented as a lightweight approach rather than full model retraining.

2 Related Work

Related work studies model merging as a way to combine independently trained models and efficient reasoning methods that reduce verbosity while preserving reasoning quality. These approaches include parameter averaging, prompting, token constraints, and supervised fine-tuning.

  • Model Merging: Model merging integrates independently trained models into a unified parameter space to combine capabilities across settings such as continual and multi-task learning.Architectural alignment enables direct parameter combination, including elementwise weight averaging.
  • Efficient Reasoning: Prompt-based efficient-reasoning methods encourage concise outputs through explicit instructions or strict token constraints.Examples include CCoT, CoD, and Tokenbudget.
  • Efficient Reasoning: Supervised fine-tuning methods train models on concise reasoning traces to improve efficiency while maintaining reasoning quality and accuracy.The passage lists methods including C3oT, CoT-Valve, TokenSkip, and LS-Mixture.
  • Efficient Reasoning: Some approaches combine reasoning and instruction-following models to obtain efficiency gains without additional training.This line of work connects model merging with concise reasoning objectives.

3 Method

GRIP builds a fused model from same-architecture reasoning and instruction models by learning module-wise interpolation ratios while freezing both source models. It optimizes these ratios with reinforcement learning that rewards correctness and shorter responses.

  • 3.1 Problem Setup: GRIP constructs a fused model by interpolating parameters from same-architecture reasoning and instruction models module by module.The interpolated modules can include attention, FFN, embeddings, and the language-modeling head; tied weights share a coefficient.
  • 3.2 Module-wise Sigmoid-controlled Fusion: Each module receives a sigmoid-controlled coefficient α_k = σ(ρ_k) between 0 and 1, combining reasoning-model and instruction-model parameters.Only the unconstrained logits ρ_k are optimized, while both source models remain frozen.
  • 3.2 Module-wise Sigmoid-controlled Fusion: One trainable scalar per module preserves per-token reinforcement-learning credit while keeping optimization lightweight.The gradient is projected onto each module’s parameter difference through the chain rule.
  • 3.3 Reward-Guided Interpolation Optimization: GRIP samples response groups from the current fused policy and computes group-relative advantages before updating the fusion logits by gradient ascent.The procedure uses GRPO-style updates with clip-higher and KL-free modifications, while the original model parameters remain fixed.
  • 3.3 Reward-Guided Interpolation Optimization: The reward favors responses that are correct and concise by penalizing excessive generated length among sampled responses.Correctness is determined from the extracted answer, and length regularization is controlled by λ.

4 Experiments

Experiments show that GRIP improves the accuracy-efficiency trade-off across five reasoning benchmarks, while module-wise analyses reveal distinct fusion behavior across attention and FFN layers. Reward-guided updates also move more smoothly than black-box search.

  • 4.2 Main Results: GRIP reduces average generation length by 27.0% relative to Qwen3-Thinking while increasing average accuracy from 76.0 to 76.5.The result indicates shorter outputs without sacrificing task-critical reasoning behavior.
  • 4.2 Main Results: GRIP matches SLERP’s average accuracy while using 14.5% fewer tokens.Adaptive module-wise fusion improves the allocation of inference computation relative to fixed-ratio merging.
  • 4.3 Validating Module-wise Interpolation: FFN fusion drives most accuracy and length changes, whereas attention fusion is largely inert; global coefficients conflate these roles.Sweeping FFN changes macro pass@1 from 0.612 to 0.782 and length by 178%, while attention pass@1 remains in [0.690, 0.722] with only a 26% length increase.
  • 4.3 Validating Module-wise Interpolation: The best FFN sweep reaches 0.782 at α=0.9, exceeding the best global setting’s 0.760, while also avoiding the global sweep’s unnecessary length cost.Separating attention and FFN motivates the per-layer extension used by GRIP.
  • 4.4 Tracking Module-wise Fusion Coefficients: Inter-layer fusion-ratio standard deviations rise from near 0 to approximately 0.30 within about 300 steps, showing strong per-layer differentiation.Attention and FFN differentiate across different depth regions and trajectories rather than merely shifting toward one endpoint.
  • 4.5 Reward-Guided vs. Black-Box Search: GRIP’s median per-coordinate RMS shift is 0.018 versus 0.083 for CMA-ES, and its path-to-net-displacement ratio is 4.9× versus 21.8×.GRIP also receives gradient feedback for each module-level coefficient, whereas CMA-ES assigns one scalar score across all 74 coordinates.

5 Conclusion

GRIP is a lightweight reward-guided parameter interpolation framework for efficient reasoning. Across five benchmarks, it reduces generation length while preserving or improving accuracy and reveals module-specific fusion patterns.

  • GRIP freezes identical-architecture reasoning and instruction models, then learns module-wise fusion coefficients using rewards for correct, concise responses.
  • Across five reasoning benchmarks, GRIP reduces generation length while preserving or improving average accuracy relative to the original reasoning model.
  • GRIP achieves a stronger accuracy-efficiency trade-off than fixed-ratio merging and search-based baselines.
  • Per-layer, per-module coefficients cannot be reduced to a single global ratio, with distinct patterns emerging across attention and FFN modules.
  • Reward-guided updates produce a smoother optimization trajectory than black-box search over the same interpolation space.

6 Limitations

GRIP's evaluation is limited to 4B dense Transformer models and identical-architecture model pairs. Its transferability to larger models, Mixture-of-Experts architectures, and cross-family fusion remains unvalidated or unsupported.

  • GRIP is evaluated only at the 4B scale, leaving transfer to substantially larger reasoning models such as 30B+ models open.
  • The experiments use dense Transformer backbones and do not validate GRIP on Mixture-of-Experts architectures.
  • GRIP assumes identical depth, width, and head count, restricting the method to within-family reasoning and instruction-model pairs.
  • Cross-family fusion is not directly supported or evaluated under the current identical-architecture assumption.

A.1 Experimental environment

The experimental environment reports approximately 42 hours of total training time on Intel Xeon Platinum 8558 CPUs.

  • Approximately 42 hours of total training time were reported on Intel Xeon Platinum 8558 CPUs.

A.2 Qwen3-4B training curves

GRIP training on the Qwen3-4B pair gradually shortens responses while maintaining a stable reward signal. The training curves track mean response length and raw reward through step 750.

  • GRIP gradually shortens responses while maintaining a stable reward signal during training.
  • Figure 7 tracks mean response length during Qwen3-4B GRIP training through step 750.
  • Figure 8 tracks raw reward during Qwen3-4B GRIP training through step 750.

A.3 Ablation: layer-wise versus module-wise interpolation

The ablation finds that module-wise interpolation is more effective than layer-wise interpolation because attention and FFN modules play different roles. It improves accuracy while maintaining comparable generation length.

  • Module-wise interpolation assigns separate coefficients to attention and FFN in every layer, plus the same two special coefficients.This design has 72 + 2 trainable parameters.
  • Tying attention and FFN within each layer forces one coefficient to control modules that affect accuracy and response length differently.The paper links this distinction to the structural asymmetry observed in Figure 2.
  • Module-wise interpolation improves average accuracy by 3.0 points while using a comparable number of generated tokens.The comparison uses the same five evaluation benchmarks; the layer-wise result is evaluated at step 230.
  • The layer-wise model is slightly shorter on average but loses substantial accuracy because it cannot independently preserve reasoning-sensitive FFN components and shift attention modules differently.The additional module-wise parameters encode this structural asymmetry rather than merely adding capacity.

A.4 Artifact licenses and terms

The work uses publicly available models, datasets, frameworks, and evaluation artifacts under their released licenses and terms. Usage is limited to the described training and evaluation setting without redistributing modified datasets or checkpoints.

  • LightEval, LiveCodeBench, GPQA, GSM8K, and MATH-500 are released under the MIT License.
  • SLIME, Qwen3-4B, and AIME25 are released under the Apache-2.0 License.
  • The artifacts are used only for the paper’s stated training and evaluation setting, without redistributing modified datasets or model checkpoints.
  • The authors will release GRIP source code to support reproducibility.The implementation uses the listed software environment and LightEval commit.
  • The study uses publicly available datasets, models, and automated evaluation pipelines, with no human subjects, crowdworkers, or annotators.
Loading 2608.25583v1…