PaperlayerView exact revision on arXiv

Source-linked AI summary

CounterRoute: Self-Routed Reasoning via Hierarchical Counterfactual Credit Assignment

Ruochen Jiao, Besnik Fetahu, Zhenyu Shi, Priyanka Nigam

arXiv:2609.29109v1cs.AI

TL;DR

Reasoning models must decide when extended deliberation is worthwhile, but routing targets can shift with policy improvements and route and response learning can be entangled. CounterRoute jointly learns routing and mode-conditioned responses with paired current-policy rollouts and a paired-to-self-routed curriculum. Across benchmarks, it improves the accuracy–efficiency trade-off, including lower token use than always-thinking checkpoints while improving accuracy.

  • Problem

    Adaptive reasoning must choose between thinking and direct answering, but existing methods leave current-policy counterfactual routing credit, distinct route and response credit, and continued evidence from both modes unresolved.

    Source: §1 · p. 2

  • Method

    CounterRoute jointly trains routing and both mode-conditioned response policies with paired current-policy counterfactual rollouts, disjoint routing and response credit, and a paired-to-self-routed curriculum.

    Source: §1 · p. 2, §3.2 · p. 4

  • Results

    CounterRoute achieves higher macro-average accuracy with fewer mean generated tokens than Always Think, same-data Think-Only GRPO, and HGPO for both models, reducing tokens versus always-thinking checkpoints by 51% for Qwen3-8B and 41% for Qwen3-14B.

    Source: §5.1 · p. 7

  • Takeaways & Limitations

    A single dual-mode model can learn both how to answer and when reasoning is worth its cost, with routing that broadly tracks the empirical value of thinking on held-out domains.

    Source: §8 · p. 9

    The study covers binary thinking-versus-direct routing and single-turn tasks; paired interventions also add rollout cost, reaching about 1.3× at steady state.

    Source: §7 · p. 9

Abstract

from arXiv · show

Reasoning-capable language models often produce long chains of thought when direct answers suffice, wasting inference compute. Many dual-mode models leave this choice to users. Automating it is challenging because routing targets evolve with the policy, initial mode preferences destabilize exploration, and sequence-level objectives entangle routing with response learning. We introduce CounterRoute, an online reinforcement-learning framework that jointly learns routing and modeconditioned responses in one shared policy directly from a native dual-mode checkpoint, without method-specific SFT warm-up. Paired current-policy counterfactual rollouts assign cross-mode credit only to the routing token, while within-mode GRPO trains response tokens. A paired-to-self-routed curriculum stabilizes early training with forced rollouts from both modes, then increases self-routed updates to improve autonomous routing. Across nine benchmarks, CounterRoute better balances accuracy and efficiency than heuristic and learned adaptive-routing methods. Relative to always-thinking checkpoints, it improves macro-average accuracy while reducing mean generated tokens by 51% for Qwen3-8B and 41% for Qwen3-14B. On instruction-following and commonsense benchmarks where direct answering is strong, think rates fall as low as 1% while response quality improves. Despite training only on math and instruction following, its routing behavior and response quality generalize to held-out coding, science, knowledge, and commonsense benchmarks.

1 INTRODUCTION

Reasoning can help on difficult prompts, but direct answers suffice for many others, leaving users or applications to choose a mode. CounterRoute jointly learns routing and mode-conditioned responses online, and improves accuracy–efficiency trade-offs across held-out benchmarks.

1 INTRODUCTION

  • Explicit chains of thought can benefit difficult math, science, and coding prompts, while many knowledge, commonsense, and instruction-following requests can be answered directly.Current interfaces leave users or applications to infer the appropriate mode, risking unnecessary computation or insufficient reasoning.Source: §1 · p. 2
  • Existing adaptive-reasoning methods leave unresolved moving counterfactual routing targets, entangled route and response credit, and self-reinforcing exploration that can suppress one mode.Some prior methods use additional method-specific SFT or hybrid fine-tuning before RL, making routing behavior depend on the warm-up distribution.Source: §1 · p. 2
  • CounterRoute learns routing and both mode-conditioned response policies through online RL from a native dual-mode checkpoint, without method-specific SFT warm-up, routing annotations, an external router, or separately served models.Paired current-policy rollouts compare thinking and direct modes; routing credit goes to the initial routing token, while within-mode GRPO trains response tokens.Source: §1 · pp. 2–3
  • 79.1% macro-average accuracy with Qwen3-8B and 82.4% with Qwen3-14B outperform same-data Think-Only GRPO; versus always-thinking checkpoints, generated tokens fall 51% and 41%, respectively, with accuracy improved for both.Training uses only math and synthetic instruction-following prompts, while evaluation spans nine held-out benchmarks.Source: §1 · p. 2
  • Training on math and synthetic instruction following yields routing that broadly tracks the empirical value of reasoning on held-out coding, science, knowledge, and commonsense benchmarks.On benchmarks where direct answering is strong, think rates fall as low as 1% while direct-answer capability improves.Source: §1 · pp. 2–3

2 PROBLEM FORMULATION

The formulation treats adaptive reasoning as a binary choice between thinking and direct answering, followed by mode-conditioned response generation. It defines an efficiency-adjusted route and separates routing credit from response-quality credit.

2 PROBLEM FORMULATION

  • For each prompt, the policy samples a mode—thinking (TH) or no-thinking (NT)—then generates a response conditioned on that mode.The formulation focuses on this binary setting to isolate routing–response credit assignment from multi-level reasoning budgets.Source: §2 · p. 3
  • The utility-optimal route maximizes mode-conditioned value after subtracting margin b for thinking; ties favor NT.The margin is a reward-space prior favoring the cheaper mode when estimated quality is similar, not a direct measure of token or latency cost.Source: §2 · p. 3
  • The return decomposition separates between-mode routing credit from within-mode response-quality credit, with the mode-mixture expectation serving as a conceptual baseline rather than a learned critic.It motivates distinct credit for routing and response decisions without prescribing that the terms be broadcast across tokens.Source: §2 · p. 3

3 METHOD

CounterRoute jointly trains routing and mode-conditioned responses in one shared policy, assigning cross-mode utility credit to the routing token and within-mode GRPO credit to response tokens. A paired-to-self-routed curriculum preserves forced coverage while shifting training toward autonomous routing.

3.1 METHOD OVERVIEW

  • The overview combines paired and self-routed updates in one shared policy, with paired comparisons refreshing routing targets as response policies improve.The rollout cost is 2.0× a same-data GRPO baseline initially, declining to 1.3× at steady state.Source: §3.1 · p. 3, Figure 2 · p. 4, §3.3 · p. 5

3.2 PAIRED COUNTERFACTUAL UPDATES

  • The per-prompt utility gap estimates the current benefit of thinking, and a scheduled reward-space margin initially favors no-think when mode rewards are similar.The margin is larger early to counter the initial thinking preference and can decay to zero; it affects routing credit, not within-group response rewards.Source: §3.2 · p. 4
  • Paired counterfactual updates compare current-policy rollouts under both forced modes, assigning signed cross-mode credit to the routing token and within-mode GRPO credit to subsequent response tokens.Separate mode-specific rollout groups normalize response advantages within each route; the routing advantage is bounded using ±s tanh(∆i).Source: §3.2 · pp. 4–5

3.3 SELF-ROUTED UPDATES AND CURRICULUM

  • Self-routed updates sample complete responses under the policy’s own mode choice and apply standard GRPO to the full sequence, including the routing token.The curriculum starts with paired coverage, then reaches a steady state of 70% self-routed and 30% paired prompt slots.Source: §3.3 · p. 5

4 EXPERIMENTAL SETUP

The experiments initialize CounterRoute from native dual-mode Qwen3 checkpoints and train on math and synthetic instruction-following data. They evaluate self-routed generation across nine held-out benchmarks and compare against fixed-mode and adaptive-routing baselines.

Models

  • The study initializes directly from officially released dual-mode Qwen3-8B and Qwen3-14B checkpoints, without additional method-specific SFT warm-up.The routing action uses native <think> and </think> tokens.Source: §4 · p. 5

Training data

  • Training uses only math and synthetic instruction-following data, with equal sampling and matched-data fixed-mode GRPO baselines; coding, science, knowledge, and commonsense data are excluded.The runs start directly from the corresponding dual-mode Qwen3 checkpoints.Source: §4 · p. 6

Evaluation

  • Evaluation covers nine held-out benchmarks across reasoning, coding, knowledge, commonsense, and instruction following, using self-routed generation and reporting accuracy, think rate, and mean generated tokens.Aggregate lengths equally weight benchmark-level means; token savings are computed relative to Always Think.Source: §4 · p. 6

Baselines

  • Comparisons include same-data fixed-mode references, routing baselines, and method-faithful HGPO Stage-II reproductions; the router baselines require serving two models.Controlled comparisons match data, rollout budgets, and optimization settings where the methods permit.Source: §4 · pp. 5–6

5 MAIN RESULTS

CounterRoute improves the accuracy–efficiency trade-off across both Qwen3 model sizes, and its self-routing transfers to held-out domains while broadly tracking the empirical value of thinking. Ablations and prompt-level analyses also reveal where routing remains imperfect.

5.1 OVERALL RESULTS AND BASELINE COMPARISONS

  • 51% for Qwen3-8B and 41% for Qwen3-14B fewer mean generated tokens than the corresponding Always Think checkpoints, with higher macro-average accuracy.CounterRoute strictly Pareto-dominates Always Think, same-data Think-Only GRPO, and HGPO on macro-average accuracy and mean generated tokens.Source: Table 1 · p. 6, §5.1 · p. 7, Table 2 · p. 7
  • On IFBench, CounterRoute reaches 46.7% accuracy with Qwen3-8B and 56.9% with Qwen3-14B, while routing only 0.7% and 4.6% of prompts to thinking, respectively.Its Qwen3-8B forced no-think and forced-think accuracies are 46.1% and 46.2%, close to the 46.7% self-routed result.Source: §5.1 · p. 7
  • Under comparable inference compute, CounterRoute improves average accuracy over Random Same-Rate by 2.4 points for Qwen3-8B and 3.1 for Qwen3-14B.Its gains over Dataset Router are 1.2 points for Qwen3-8B and 2.5 points for Qwen3-14B.Source: §5.1 · p. 7
  • HGPO trails CounterRoute in macro-average accuracy: 77.9% versus 79.1% with Qwen3-8B and 80.0% versus 82.4% with Qwen3-14B.HGPO routes to thinking on 97–100% of benchmarks and uses 19% more mean tokens with Qwen3-8B and 17% more with Qwen3-14B.Source: §5.1 · p. 7
  • AdaptThink reaches 72.2% with Qwen3-8B and 74.9% with Qwen3-14B, respectively 6.9 and 7.5 points below CounterRoute.The authors attribute its lower performance in this setting to frozen reference supervision that does not track changing relative mode values; CounterRoute recomputes the gap from paired current-policy rollouts.Source: §5.1 · p. 8
  • Ablations associate trajectory-wide routing credit and shared GRPO grouping with degraded quality, removing either paired or self-routed groups with near-always-thinking, and scheduled b-decay with a better balance.Training curves show improving mean correctness rewards and a non-collapsed self-routed mode mixture.Source: §5.1 · p. 8

5.2 ROUTING BEHAVIOR BEYOND TRAINING DOMAINS

  • After training only on math and synthetic instruction following, Qwen3-8B self-routing transfers to held-out coding, science, knowledge, and commonsense, reaching 79.10% accuracy versus 78.84% for forced thinking.Median response length falls from 534 to 400 words.Source: §5.2 · p. 8
  • The selected mode is no worse than the forced alternative on 95.08% of examples, counting both-correct and both-wrong outcomes as ties.Residual misrouting includes Qwen3-8B thinking on 25% of HellaSwag despite a 2.5-point disadvantage; GSM8K receives 100% thinking despite only a 4.6-point gain.Source: §5.2 · pp. 8–9

6 RELATED WORK

Related work addresses adaptive mode choice, routing after warm starts, and hierarchical credit assignment. CounterRoute differs by learning from a native dual-mode checkpoint with paired current-policy comparisons and disjoint routing and response credit.

Adaptive mode selection

  • Adaptive-reasoning methods include AdaptThink’s reference-performance constraint, TON’s thought dropout with RL, and AutoThink’s trigger-based activation and multi-stage rewards.The cited work also covers chain-of-thought prompting, reasoning-oriented post-training, and native dual-mode models.Source: §6 · p. 9

Warm starts and routing generalization

  • Thinkless, TON, R-4B, and HGPO use method-specific supervised or hybrid warm starts; CounterRoute instead starts from a native dual-mode checkpoint and trains on a narrow math-and-instruction mixture.The comparison frames held-out coding, science, knowledge, and commonsense as transfer beyond CounterRoute’s training mixture.Source: §6 · p. 9

Credit assignment

  • Prior credit-assignment approaches include separating control-token and response objectives, restricting inter-group advantage, and applying hierarchical rewards; CounterRoute combines paired current-policy comparisons with disjoint token credit.Its evaluation also tests route–value alignment beyond the training domains.Source: §6 · p. 9

7 DISCUSSION

The evaluation is limited to binary thinking-versus-direct routing and single-turn tasks, while paired counterfactual rollouts add training cost.

7 DISCUSSION

  • The study covers binary thinking versus direct answering, not multiple reasoning levels or continuous compute allocation.Extending the method would require counterfactual groups over more than two actions.Source: §7 · p. 9
  • The experiments are single-turn; multi-turn routing would require choosing intervention turns and assigning delayed credit across later turns.The authors plan to extend the framework to multi-level and multi-turn routing.Source: §7 · p. 9
  • Paired interventions reach at most 2× the rollout groups during warm-up and about 1.3× at steady state, plus response-length differences.Source: §7 · p. 9

8 CONCLUSION

The conclusion presents CounterRoute as joint online learning of routing and response generation, with hierarchical credit and a curriculum toward autonomous self-routing. Its reported accuracy–efficiency balance and route–value alignment extend to held-out domains.

8 CONCLUSION

  • CounterRoute jointly learns routing and response generation through paired current-policy rollouts for routing credit and within-mode comparisons for response learning.A curriculum transfers control to autonomous self-routing without additional SFT warm-up or an external router.Source: §8 · p. 9
  • Across Qwen3-8B and Qwen3-14B, CounterRoute improves the accuracy–efficiency trade-off over fixed-mode, heuristic, and learned adaptive baselines.Its routing broadly tracks the empirical value of thinking on held-out domains.Source: §8 · p. 9
  • The conclusion’s practical claim is that a single dual-mode model can learn both how to answer and when reasoning is worth its cost.Source: §8 · p. 9

A ROUTING GRADIENT DERIVATION

The routing update uses the paired-mode reward gap to adjust the two routing-token log probabilities in opposite directions, with bounded update magnitude. The training step combines forced paired rollouts with self-routed rollouts and applies their distinct credit assignments in one policy update.

A ROUTING GRADIENT DERIVATION

  • The paired-mode gap sets opposite log-probability updates for thinking and no-thinking; its sign selects which routing token’s log-odds rise, while s tanh(∆i) bounds update magnitude.Source: A ROUTING GRADIENT DERIVATION · p. 11

B TRAINING ALGORITHM

  • The training step combines paired forced-mode samples with self-routed samples, assigns opposite position-0 routing advantages to forced modes, then performs one clipped actor update.Source: B TRAINING ALGORITHM · p. 12
  • For each paired prompt, the algorithm samples K responses under each forced route, computes the mode gap, and calculates response advantages separately within the two mode groups.Source: B TRAINING ALGORITHM · p. 12

C TRAINING CONFIGURATION

The section identifies shared training settings, reports math-training reward and routing dynamics, and compares forced-mode estimates with the policy’s self-routed behavior on benchmarks.

C TRAINING CONFIGURATION

D.1 TRAINING DYNAMICS

  • Figure 4 plots mean correctness reward during math training, not held-out evaluation accuracy, and the self-routed think fraction remains non-collapsed after early exploration.Source: Figure 4 · p. 13

D.2 SAME-CHECKPOINT ROUTING ANALYSIS

  • 79.1% macro accuracy at 400 median words and a 65.6% think rate under self-routing, versus 78.84% at 534 words for forced thinking and 67.77% at 22 words for forced no-thinking.Source: Table 4 · p. 13, §D.2 · p. 13

D.3 COMPONENT ABLATIONS

The ablations distinguish the roles of counterfactual pairing, autonomous routing exposure, token-level credit assignment, and mode-separated response optimization. The additional comparison and examples describe the method’s performance against a Thinkless-style adaptation and illustrate both concise direct answering and residual over-routing.

D.3 COMPONENT ABLATIONS

  • More than 25 points of accuracy are lost and mean output rises to 2645.6 words when route credit is broadcast across the trajectory; shared response normalization reduces accuracy by 5.1 points.Source: Table 5 · p. 14, §D.3 · p. 14
  • The scheduled-decay control reaches 78.58% accuracy, 877.1 mean words, and 68.77% thinking, improving accuracy and mean length relative to the fixed-margin peak; fixed-margin controls show that b selects the routing operating point.Source: Table 5 · p. 14, §D.3 · pp. 14–15
  • Removing counterfactual pairing or self-routed updates leaves accuracy near the control but drives thinking to approximately 99–100% of prompts and lengthens responses.Source: §D.3 · p. 14, Table 5 · p. 14

D.4 THINKLESS COMPARISON

  • Thinkless-style adaptation saves 10% of mean tokens with Qwen3-8B and 16% with Qwen3-14B while thinking on 100% and 97% of prompts, respectively; CounterRoute saves 51% and 41% while retaining comparable or higher accuracy.Source: Table 7 · p. 15, §D.4 · p. 15

D.5 QUALITATIVE EXAMPLES

  • On HellaSwag, CounterRoute answers correctly in four words where Always No-Think fails; on GSM8K, concise within-mode reasoning coexists with residual conservative routing despite a correct No-Think answer.Source: §D.5 · pp. 15–16
Loading 2609.29109v1…