Source-linked AI summary
RelayGen: Intra-Generation Model Switching for Efficient Reasoning
Jiwon Song, Yoongon Kim, Jae-Joon Kim
TL;DR
Large reasoning models incur substantial deployment cost because long reasoning trajectories scale inference, while difficulty varies within individual outputs. RelayGen addresses this gap with training-free, segment-level switching based on empirical uncertainty cues, preserving large-model use for difficult reasoning and offloading lower-difficulty segments. Across benchmarks it preserves most large-model accuracy while reducing latency, reaching up to 2.2× end-to-end speedup with speculative decoding.
Problem
Inference-time scaling makes long-reasoning models costly, while existing efficiency methods either overlook intra-generation difficulty variation or require supervised token-level routing.
Method
RelayGen uses offline uncertainty analysis and model-specific transition cues to switch at segment boundaries, retaining the large model for difficult reasoning and handing lower-difficulty continuation to a smaller model.
Results
Across multiple reasoning benchmarks, RelayGen preserves most large-model accuracy while substantially reducing latency and achieves up to 2.2× end-to-end speedup with speculative decoding.
Takeaways & Limitations
Coarse-grained, training-free runtime switching can allocate model capacity within a single long generation and compose effectively with speculative decoding.
Takeaways & Limitations
RelayGen targets explicitly externalized long-form reasoning with varying difficulty, and its offloading assumes the smaller model retains sufficient capacity for lower-difficulty segments.
Abstract
from arXiv · showhide
Large reasoning models (LRMs) achieve strong performance on complex reasoning tasks by generating long, multi-step reasoning trajectories, but inference-time scaling incurs substantial deployment cost. A key challenge is that generation difficulty varies within a single output, whereas existing efficiency-oriented approaches either ignore this intra-generation variation or rely on supervised token-level routing with high system complexity. We present \textbf{RelayGen}, a training-free, segment-level runtime model switching framework that exploits difficulty variation in long-form reasoning. Through offline analysis of generation uncertainty using token probability margins, we show that coarse-grained segment-level control is sufficient to capture difficulty transitions within a reasoning trajectory. RelayGen identifies model-specific switch cues that signal transitions to lower-difficulty segments and dynamically delegates their continuation to a smaller model, while preserving high-difficulty reasoning on the large model. Across multiple reasoning benchmarks, RelayGen substantially reduces inference latency while preserving most of the accuracy of large models. When combined with speculative decoding, RelayGen achieves up to 2.2$\times$ end-to-end speedup with less than 2\% accuracy degradation, without requiring additional training or learned routing components.
1 Introduction
Long-form reasoning outputs contain heterogeneous difficulty, creating an opportunity to allocate large-model capacity selectively. RelayGen uses training-free segment-level switching guided by empirical difficulty analysis and reports latency gains with limited accuracy degradation.
- Inference-time scaling improves reasoning accuracy but increases deployment cost, making generation efficiency central to practical LRM deployment.
- Long reasoning trajectories interleave high-difficulty reasoning with lower-difficulty continuation or consolidation, so difficulty is heterogeneous within one generation.
- RelayGen allocates large-model capacity to difficult segments and hands subsequent lower-difficulty segments to a smaller model after detecting a transition.
- RelayGen uses a training-free, segment-level routing mechanism grounded in empirical difficulty analysis rather than supervised routing.
- 2.2× latency reduction with less than 2% accuracy degradation is reported on AIME 2025 when RelayGen combines with speculative decoding.
2 Related Works
Prior efficiency methods route at the input, token, or segment level, with trade-offs among granularity, supervision, heuristic design, and deployability. RelayGen belongs to the coarser switching family while emphasizing empirically validated difficulty transitions and compatibility with speculative decoding.
- Input-level routing: Input-level routing assigns one model to an entire generation and therefore cannot account for difficulty variation within a long reasoning output.
- Token-level routing: Token-level routing enables per-token difficulty-aware decisions but typically requires trained routers and additional supervision.
- Segment- or step-level switching: Segment- or step-level methods intervene over multi-token units, but some rely on manually specified, model-agnostic heuristic cues rather than model-specific difficulty analysis.
- Training-based switching: Training-based switching learns a model-internal handoff policy, creating a distinct accuracy–efficiency trade-off from training-free runtime allocation.
- Speculative decoding: Speculative decoding is orthogonal to routing and naturally compatible with coarse-grained switching, whereas token-level routing can disrupt draft–verify generation.
3 Difficulty Variation in LRM Generation
Long reasoning outputs contain distinct reasoning and answer stages, with substantial difficulty variation during reasoning and comparatively stable answer generation after reasoning.
- LRM outputs comprise a reasoning stage followed by an answer stage that consolidates conclusions for the user.
- Probability margin measures generation uncertainty as the difference between the top-1 and top-2 token probabilities.
- Probability margins fluctuate substantially across long reasoning trajectories, indicating non-uniform generation difficulty.
- Discourse-level cues are lexical markers associated with shifts in reasoning flow, and their post-sentence margins measure subsequent within-sentence uncertainty.
- Different cues produce different post-sentence margins: Thus often precedes higher-margin conclusions, whereas so can be associated with lower margins.
- 99.86% matching across 728 samples shows that small-model answer generation closely reproduces large-model answers when conditioned on preceding reasoning.
4 RelayGen
RelayGen performs training-free, segment-level runtime switching, retaining the large model for difficult reasoning and delegating selected easier continuations and answer generation to a smaller model.
- 4.1 Overview of RelayGen: RelayGen dynamically allocates model capacity within one generation, matching model size to local generation difficulty.
- 4 RelayGen: During reasoning, the large model generates until a selected switch cue, after which continuation segments are delegated to the small model.
- 4 RelayGen: RelayGen requires neither additional training nor auxiliary learned routers or control modules.
- 4 RelayGen: Switch cues are selected offline when post-sentence margin exceeds the global average by at least one standard error; inconsistent cues are excluded.
- 4 RelayGen: The small model stops at sentence boundaries during reasoning, allowing repeated handoffs back to the large model before final answer completion.
- 4 RelayGen: Coarse-grained switching remains compatible with speculative decoding, which can operate whenever the large model is active.
5 Experiments
RelayGen is evaluated across model pairs and reasoning benchmarks against single-model and mid-generation switching baselines. It preserves most large-model accuracy while reducing latency, including up to 2.20× speedup when combined with speculative decoding.
- Experimental Setup: RelayGen is evaluated with Qwen3-32B/1.7B and R1-Distill-Qwen-32B/1.5B across AIME 2025, MATH500, and GPQA-Diamond.The evaluation reports pass@1 using four outputs per problem and compares single-model generation with Speculative Thinking and R2R.
- Accuracy Evaluation: RelayGen substantially improves accuracy over the small-only baseline while preserving most of the large model’s performance across both model pairs.It consistently outperforms Speculative Thinking, which can delegate difficult reasoning segments to the small model and incur significant accuracy degradation.
- Accuracy Evaluation: RelayGen achieves comparable overall accuracy to R2R, outperforming it across all three benchmarks for Qwen3 and on MATH500 and GPQA-Diamond for R1-Distill.R2R performs slightly better on AIME 2025 for the R1-Distill pair.
- Inference Latency and Composability: 2.21× speedup is achieved by Speculative Thinking, but its low large-model utilization comes with substantial accuracy degradation.The comparison reports speedup relative to large-model-only inference and tracks the fraction of tokens generated by the large model.
- Inference Latency and Composability: RelayGen attains speedup comparable to R2R while retaining substantially higher large-model utilization by avoiding per-token routing overhead.Its segment-level switching preserves the large model for high-difficulty reasoning segments and yields a more favorable accuracy–latency trade-off.
- Inference Latency and Composability: 2.20× final speedup is achieved when RelayGen is combined with Eagle-3, while remaining training-free and adding 1.22× speedup over Eagle-3 alone.The combined approach is compatible with speculative decoding and avoids aggressive early switching.
- Ablations and Robustness: Selected switch cues outperform indiscriminate activation of all discourse-level cue candidates on AIME 2025 and GPQA-Diamond.The ablation evaluates Qwen3-32B/1.7B and finds that using all candidate cues consistently reduces pass@1.
- Ablations and Robustness: RelayGen remains effective with heterogeneous model pairs when the small model has sufficient capacity, while performance degradation primarily reflects limited small-model capacity.The heterogeneous-pair evaluation compares Qwen3-32B/R1-Distill-Qwen-1.5B with R1-Distill-Qwen-32B/Qwen3-1.7B.
6 Conclusion
RelayGen addresses heterogeneous difficulty within long reasoning trajectories by allocating model capacity at the segment level during inference. The framework is training-free and avoids learned routers or additional training signals.
- Generation difficulty within long reasoning trajectories is heterogeneous, so not every segment requires the full capacity of a large model.
- RelayGen dynamically allocates model capacity within a single generation through training-free, segment-level runtime model switching.
- The framework is based on offline empirical analysis of token-level uncertainty and does not rely on learned routers or additional training.
Limitations
RelayGen is best suited to long-form reasoning settings where difficulty varies across extended trajectories. Its effectiveness depends on suitable output structure and a small model with sufficient capacity for lower-difficulty segments, while multilingual applicability remains untested.
- RelayGen targets long-form reasoning tasks with extended trajectories and varying generation difficulty.Its benefits may be limited for tasks without sustained reasoning or clear long-sequence structure.
- Effective offloading assumes a moderate capability gap between the large and small models.If the small model cannot handle relatively low-difficulty segments, offloading can degrade overall performance.
- Experiments focus on English reasoning data, leaving systematic evaluation of multilingual extensions for future work.
7 Ethical Considerations
RelayGen changes runtime allocation between existing models without introducing new capabilities, data sources, or learning signals. Its biases and risks therefore come from the underlying models rather than from additional mechanisms introduced by RelayGen.
- RelayGen is a training-free inference-time framework that controls runtime allocation between the underlying models.
- The framework introduces no new model capabilities, data sources, or learning signals beyond those of the underlying models.
- Any biases or risks are inherited from the base models, and RelayGen does not raise additional ethical concerns compared with standard deployment.
A Discourse-level Cues
RelayGen defines discourse-level cues as lexical signals of structural transitions in reasoning, then categorizes them by function without assuming that every cue is an effective switch signal.
- Discourse-level cues are lexical tokens that signal progression, reconsideration, or consolidation without adding problem-specific content.
- The cue pool extends reflection keywords with tokens such as so and therefore that signal result consolidation or summary transitions.
- The cue pool is fixed before model-pair calibration, while its categorization remains descriptive rather than predictive of switching effectiveness.
- Progression and Continuation Cues: Progression cues indicate forward movement or continuation, including now, then, next, and again.
- Reconsideration and Branching Cues: Reconsideration cues signal pauses, corrections, or alternative reasoning paths, including wait, however, alternatively, and instead.
- Inference and Transition Cues: Inference cues mark transitions from intermediate reasoning to inferred statements or implications, including thus, hence, and therefore.
B Switch Cue Selection Details
RelayGen selects switch cues through training-free, model-pair-specific profiling of post-cue probability margins, using offline calibration to identify easier continuation segments without inference-time overhead.
- Profiling Method: RelayGen profiles each cue by measuring average post-sentence probability margins and comparing them with the global calibration-trace average.
- Switch-cue effects vary across models, so RelayGen selects cues independently for each large–small model pair.
- Calibration Setup: The training-free calibration uses 40 AMC 2023 problems, with four large-model traces per problem, yielding 160 calibration samples reused under the small model.
- Calibration Overhead: RelayGen incurs about 100 minutes of one-time offline calibration, with no inference-time overhead, and this cost can decrease with smaller calibration sets.
C Granularity Mismatch between Token-Level Routing and Speculative Decoding
Token-level routing fragments large-model computation into very short segments, undermining speculative decoding, whereas RelayGen’s coarser segment-level switching supports longer contiguous spans and avoids per-token routing overhead.
- Speculative decoding requires long contiguous draft spans so the larger model can verify many tokens in bulk.
- Token-level routing invokes the large model at sparse positions, often for one token, before immediately returning control to the small model.
- Extremely short large-model segments collapse speculative-decoding verification spans to one or two tokens, eliminating most bulk-verification amortization.
- Token-level routing speedups are often far from proportional to small-model token usage because per-step routing decisions add non-negligible overhead.
- RelayGen switches at segment-level granularity, enabling longer contiguous spans that compose naturally with speculative decoding.