PaperlayerView exact revision on arXiv

Source-linked AI summary

ThinkRouter: Efficient Reasoning via Routing Thinking between Latent and Discrete Spaces

Xin Xu, Tong Yu, Xiang Chen, Haoliang Wang, Julian McAuley, Saayan Mitra

arXiv:2602.11683v1cs.AIcs.CLcs.LG

TL;DR

Latent reasoning can improve efficiency, but its effectiveness varies and low-confidence alternatives may introduce noise that yields unreliable high-confidence trajectories. ThinkRouter routes low-confidence steps through discrete tokens and other steps through latent representations, achieving broad accuracy gains with shorter generation and calibrated errors.

  • Problem

    Latent reasoning seeks efficient reasoning through continuous representations, but its effectiveness varies across settings and low-confidence alternatives may introduce propagating noise.

  • Method

    ThinkRouter is an inference-time mechanism that routes each thinking step to discrete tokens when confidence is low and to probability-weighted latent embeddings otherwise.

  • Results

    ThinkRouter robustly improves accuracy and reduces generation length across diverse reasoning models and STEM and coding benchmarks, with up to 19.70-point average Pass@1 improvement over CoT sampling.

  • Takeaways & Limitations

    ThinkRouter can calibrate errors from CoT and Soft Thinking while accelerating end-of-thinking token generation by globally lowering model confidence.

Abstract

from arXiv · show

Recent work explores latent reasoning to improve reasoning efficiency by replacing explicit reasoning trajectories with continuous representations in a latent space, yet its effectiveness varies across settings. Analysis of model confidence dynamics under latent reasoning reveals that thinking trajectories ending in incorrect answers contain fewer low-confidence steps than those ending in correct answers. Meanwhile, we suggest that soft embeddings aggregated by multiple low-confidence thinking alternatives may introduce and propagate noise, leading to high confidence in unreliable reasoning trajectories. Motivated by these observations, ThinkRouter, an inference-time confidence-aware routing mechanism is proposed to avoid high confidence and noise for efficient reasoning. ThinkRouter routes thinking to the discrete token space when model confidence is low, and to the latent space otherwise. Extensive experiments on STEM reasoning and coding benchmarks across diverse large reasoning models demonstrate that ThinkRouter outperforms explicit CoT, random routing, and latent reasoning baselines in terms of accuracy, achieving an average improvement of 19.70 points in Pass@1, while reducing generation length by up to 15.55%. Further comprehensive analysis reveals that ThinkRouter can calibrate errors arising from explicit CoT and latent reasoning, and accelerates end-of-thinking token generation by globally lowering model confidence.

1. Introduction

ThinkRouter addresses variable and sometimes degraded latent-reasoning performance by routing thinking between discrete tokens and latent representations according to model confidence. Experiments report higher accuracy, shorter generation, and confidence-based error calibration across models and tasks.

  • Motivation: Latent reasoning reduces tokens but requires tuning and varies across settings, sometimes reducing performance.Training-free Soft Thinking instead uses token-probability-weighted soft embeddings.
  • Motivation: Incorrect latent-reasoning trajectories contain fewer low-confidence steps than correct trajectories.The analysis uses maximum next-token probability as a proxy for LRM confidence.
  • Motivation: Low-confidence alternatives may aggregate into noisy soft embeddings that propagate across latent reasoning steps and produce unreliable high-confidence solutions.This motivates preventing both excessive confidence and representational noise.
  • ThinkRouter: ThinkRouter routes low-confidence thinking to the discrete token space and otherwise uses probability-weighted latent soft embeddings.The routing decision is made at each thinking time step using LRM confidence.
  • Results: 19.70 points is the reported maximum average Pass@1 improvement over CoT sampling, while generation length is reduced comparably with baselines.The evaluation spans LRMs from 1.5B to 32B, Qwen3 and gpt-oss architectures, and STEM reasoning and coding datasets.
  • Results: ThinkRouter calibrates errors from explicit CoT and Soft Thinking while accelerating end-of-thinking generation by globally lowering model confidence.The paper presents these effects as analyses of ThinkRouter’s effectiveness.

2. Related Work

Prior work compresses reasoning into latent representations or learns policies for switching among thinking modes. The paper positions its confidence-aware hybrid approach against these training-dependent latent and hybrid reasoning methods.

  • Latent Reasoning: Latent-reasoning methods move reasoning from discrete CoT tokens to latent thoughts using hidden states, mappings, distillation, reinforcement learning, or iterative refinement.These methods improve reasoning efficiency but often rely on costly training or distillation.
  • Latent Reasoning: Soft Thinking provides a training-free latent-reasoning alternative based on probability-weighted soft embeddings.The supplied related-work passage introduces Soft Thinking after describing training-dependent methods.
  • Hybrid Reasoning: Hybrid reasoning work includes RL-learned gating and learned policies that switch between latent, discrete, long-thinking, and non-thinking modes.Examples include HRPO, Thinkless, AdaptThink, LHRM, Qiao et al., and MixReasoning.
  • Confidence Dynamics: Under Soft Thinking on GPQA Diamond, incorrect predictions are associated with fewer low-confidence thinking time steps than correct predictions.The figure-related passage describes the ratio of time steps with pmax below the routing threshold.

3. Preliminary

The paper contrasts discrete and latent reasoning spaces, then analyzes confidence patterns in latent-only reasoning. Incorrect answers are associated with fewer low-confidence steps, motivating concern about noisy aggregation of alternative reasoning paths.

  • Reasoning spaces: Discrete reasoning samples one token per step, while latent reasoning uses probability-weighted soft embeddings to integrate multiple potential reasoning paths.Latent reasoning preserves information about possible thoughts rather than committing immediately to one explicit path.
  • Answer generation: After thinking, the model generates the final answer through standard discrete-space decoding.The reasoning trajectory ends when a special end-of-thinking token is generated.
  • Confidence analysis: Latent-only confidence analysis evaluates two reasoning models across STEM and coding tasks, comparing confidence patterns for correct and incorrect answers.The analysis uses maximum next-token probability as the confidence proxy and measures low-confidence steps across thresholds.
  • Confidence analysis: Incorrect answers usually contain fewer low-confidence reasoning steps than correct answers, especially for thresholds τ ∈ [0.4, 0.9].This indicates an association between unsuccessful reasoning and relatively high confidence.
  • Confidence analysis: When maximum next-token probability is low, soft embeddings aggregate multiple alternatives that may represent distinct or incompatible reasoning directions.The paper hypothesizes that this aggregation can produce diffuse representations and propagate noise across latent reasoning steps.

4. THINKROUTER

THINKROUTER selects between discrete and latent reasoning at each thinking step using maximum next-token probability. It uses discrete tokens for low-confidence states and latent soft embeddings for higher-confidence states.

  • Routing mechanism: THINKROUTER routes thinking between discrete and latent spaces according to the maximum next-token probability.The mechanism is inference-time and determines the reasoning space at every thinking step.
  • Low-confidence routing: When pmax_t < τ, THINKROUTER samples one discrete token to avoid aggregating incompatible alternatives and reduce overconfident unreliable reasoning.The threshold τ identifies states where all alternatives are considered low confidence.
  • High-confidence routing: When pmax_t ≥ τ, reasoning proceeds in the latent space using a soft embedding that represents multiple plausible reasoning paths.This preserves richer exploration in the latent concept space following Soft Thinking.

5. Experiments and Results

THINKROUTER is evaluated across STEM and coding benchmarks with diverse reasoning models, using confidence-aware routing between discrete and latent thinking spaces. It consistently improves accuracy over baselines, reduces generation length, and calibrates errors by stabilizing confidence dynamics.

  • Setups: At each thinking step, THINKROUTER uses maximum next-token probability to route low-confidence reasoning to discrete tokens and other reasoning to latent soft embeddings.The method samples a discrete token when pmax is below the routing threshold; otherwise it computes a probability-weighted soft embedding.
  • Setups: THINKROUTER is evaluated on five reasoning benchmarks spanning STEM reasoning and coding, using four large reasoning models.The benchmarks include AIME 2024, AIME 2025, GPQA Diamond, HumanEval, and MBPP; models include Qwen3 variants and gpt-oss-20b.
  • Main Results: +19.70 points is THINKROUTER’s largest reported average Pass@1 improvement over CoT (sampling), achieved with Qwen3-1.7B on STEM reasoning.Against Soft Thinking, average Pass@1 gains are 9.22, 6.18, 5.63, and 13.80 points for Qwen3-1.7B, 8B, 32B, and gpt-oss-20b, respectively.
  • Main Results: -4.56%, -6.86%, and -4.35% are the reported STEM generation-length reductions relative to Soft Thinking for Qwen3-1.7B, 8B, and 32B.On coding benchmarks, THINKROUTER has comparable reduction to Soft Thinking and always produces shorter outputs than CoT.
  • Error Calibration: Up to 77.3% of baseline errors are corrected by THINKROUTER, with precision reaching 90.6% across models and benchmarks.The analysis uses Fix Rate to measure baseline-error coverage and precision to assess the reliability of corrections.
  • Analysis: THINKROUTER increases low-confidence steps, stabilizes confidence trajectories, reduces Cold Stop termination, and promotes shorter reasoning paths.These changes are associated with mitigating premature high-confidence reasoning and reducing error-prone trajectories.

6. Conclusion

ThinkRouter provides efficient reasoning by routing between discrete and latent spaces using confidence, improving accuracy, reducing generation length, and calibrating reasoning errors.

  • ThinkRouter routes thinking between discrete token and latent spaces based on large reasoning model confidence.
  • ThinkRouter robustly improves reasoning accuracy and reduces generation length across diverse models and benchmarks.
  • ThinkRouter calibrates errors from chain-of-thought and Soft Thinking while accelerating end-of-thinking token generation.
  • ThinkRouter globally lowers model confidence during reasoning.

Impact Statement

The paper reports no societal consequences that require specific highlighting.

  • The authors identify no potential societal consequences requiring specific emphasis.

A. Preliminary Experiments

The preliminary experiments define the confidence analysis, datasets, decoding baselines, and implementation choices used to evaluate reasoning behavior.

  • Confidence analysis: Figure 6 reports the ratio of low-confidence thinking steps for correct and incorrect HumanEval trajectories under latent-only Soft Thinking.
  • Confidence analysis: A low-confidence time step occurs when the maximum next-token probability is lower than τ.
  • Confidence analysis: Each dataset sample is run three times, with next-token distributions recorded at every thinking time step for analysis.
  • Datasets: The evaluation spans three STEM reasoning benchmarks and two coding benchmarks across different domains and scales.
  • Datasets: AIME 2024 and AIME 2025 require exact integer answers from 000 to 999 without partial credit.
  • Datasets: HumanEval evaluates function-level Python code generation using exact-match execution correctness, while MBPP contains short Python tasks with tests.
  • Baselines: CoT sampling is a discrete-space chain-of-thought baseline using top-k, top-p, and min-p strategies.
  • Baselines: CoT greedy decoding is a discrete-space chain-of-thought baseline using greedy decoding.

B.4. THINKROUTER Implementation with Soft Thinking

ThinkRouter's implementation is evaluated across routing thresholds and random seeds, with performance driven primarily by the threshold τ.

  • Performance varies primarily with the routing threshold τ, while random seeds introduce stochastic fluctuations.
  • The overall performance trend with respect to τ remains consistent across different random seeds.
  • Table 6 lists the τ values used in the reported experiments.

B.5. Main Results on Whole Datasets

On the original benchmark samples, THINKROUTER preserves its reported advantage across STEM and coding tasks, combining strong Pass@1 accuracy with short generation lengths.

  • THINKROUTER outperforms all baselines on Pass@1 for STEM reasoning benchmarks and has the shortest generation length in most cases.These results are reported on the original all-sample benchmark data.
  • THINKROUTER achieves the highest Pass@1 accuracy in most coding-benchmark cases, including cases where Soft Thinking performs worse than sampled CoT.It also remains competitive with Soft Thinking in reducing generation length.

C. Further Analysis

Further analysis examines confidence dynamics, stopping behavior, output lengths, and routing trajectories across models and STEM or coding benchmarks. The analyses report that THINKROUTER promotes end-of-thinking generation, while incorrect predictions generally produce longer outputs.

  • Confidence dynamics: The analysis records next-token distributions from three runs per sample to construct confidence-related figures.Samples with different generation lengths are normalized to relative positions and discretized into 100 bins.
  • Confidence dynamics: Low-confidence time-step ratios are compared across correct and incorrect trajectories for Qwen3-8B and gpt-oss-20b on GPQA Diamond and HumanEval.The comparison uses relative generation positions to align trajectories of varying lengths.
  • Thinking termination: THINKROUTER helps trigger end-of-thinking token generation in most cases compared with Soft Thinking.The stop-mode comparison also includes Cold Stop as a reference condition.
  • Generation length: Incorrect predictions generally have longer outputs than correct predictions, based on generation-length distributions.The distributions are shown separately for Qwen3-8B and gpt-oss-20b.
  • Routing trajectories: Figures 15–18 illustrate top-3 next-token probabilities over 100 thinking steps for THINKROUTER across Qwen3-8B and gpt-oss-20b tasks.Red boxes identify steps routed to the discrete token space; otherwise, thinking is routed to the latent space.
  • Confidence and termination: Low-confidence time-step ratios and maximum next-token probabilities are visualized across generation steps, including the final steps before termination.The figures cover GPQA Diamond and HumanEval for both evaluated models.
Loading 2602.11683v1…