PaperlayerView exact revision on arXiv

Source-linked AI summary

Knowing When Thinking Is Not Enough: Teaching Small Reasoning Models to Reason Beyond Their Parametric Knowledge

Chanuk Lee, Minki Kang, Sangwoo Park, Woongyeong Yeo, Jinheon Baek, Sung Ju Hwang

arXiv:2609.34327v1cs.AIcs.CL

TL;DR

Small reasoning models are cheap to serve, but additional thinking may not resolve failures caused by missing parametric knowledge. FlyBy trains them to reason first and selectively query stronger models; on hard problems it improves pass@8 over a larger baseline at lower serving cost.

  • Problem

    Small reasoning models are attractive because they are cheap to serve, but whether additional thinking is the right operation when failures reflect knowledge gaps remains an open question.

    Source: §1 · pp. 1–2

  • Method

    FlyBy trains small models to reason first, diagnose unresolved needs, and selectively query stronger models for external knowledge.

    Source: §1 · p. 2

  • Results

    46.0% pass@8 lets FlyBy-4B surpass Qwen3-14B at 2.7× lower serving cost on hard problems.

    Source: §7 · p. 10

  • Takeaways & Limitations

    Effective test-time scaling requires choosing whether to think longer or seek external information, rather than only how much to compute.

    Source: §7 · p. 10

    FlyBy relies on external models, raising practical concerns about availability, privacy, reliability, latency, and deployment constraints; incorrect or hallucinated outputs may propagate into reasoning.

    Source: §I · p. 23

Abstract

from arXiv · show

Scaling test-time computation is a powerful way to improve language-model reasoning, and is particularly appealing for small reasoning models (sRMs) that are cheap to serve. However, is additional thinking always the right operation? By intervening at intermediate reasoning states across two model families and multiple scales, we find that self-refinement largely consolidates probability mass onto solutions already reachable from the current state, rather than making new ones reachable. These interventions reveal two failure regimes: execution bottlenecks, where the correct path is reachable and reflection can recover it, and knowledge bottlenecks, where relevant external information makes it reachable. Motivated by this distinction, we introduce FlyBy, a selective querying framework, and train 4B and 8B variants to reason first, diagnose what remains unresolved, and, at a knowledge bottleneck, query stronger models whose parametric knowledge extends beyond its own. Supervised fine-tuning bootstraps a multi-depth query action, and cost-aware reinforcement learning calibrates whether to query, what to ask, and how much to spend. On 1,158 hard problems across six benchmarks, FlyBy-4B achieves 45.96% pass@8, surpassing Qwen3-14B (41.64%) at 2.7 times lower serving cost, while also exceeding Qwen3-8B in pass@1 (16.85% vs. 15.31%). Scaling to FlyBy-8B further improves pass@8 to 51.81%.

1. Introduction

Additional thinking does not reliably help small reasoning models when failures reflect missing knowledge rather than recoverable execution errors. FlyBy addresses this distinction by training models to reason first and selectively seek external help, improving benchmark performance at lower serving cost than a larger baseline.

1. Introduction

  • Small reasoning models are inexpensive to serve but lag larger models, and their failures can reflect gaps in parametric knowledge that reflection alone cannot fill.The benefit of additional computation also varies across problems and reasoning states.Source: §1 · pp. 1–2
  • Execution bottlenecks permit recovery through additional reasoning, whereas knowledge bottlenecks require relevant external information to make a correct path reachable.The paper frames the key decision as whether more thinking is the right operation for the current state.Source: §1 · p. 2
  • Existing tool-use systems generally optimize invocation without explicitly diagnosing whether a reasoning state is internally recoverable or needs new information.The paper proposes diagnosing the bottleneck before deciding what information to request and how much external computation to spend.Source: §1 · p. 2
  • Counterfactual interventions show that reflection mainly recovers already reachable solutions, while relevant external information makes new paths reachable.The interventions span eight reasoning models across two families and multiple scales; smaller models encounter knowledge bottlenecks more often and benefit less from assistance.Source: §1 · p. 2
  • 46.0% pass@8 lets FlyBy-4B surpass Qwen3-14B at 2.7× lower serving cost.The comparison is on 1,158 hard problems across six benchmarks; FlyBy-4B more than doubles its Qwen3-4B base model’s pass@8 from 21.2% to 46.0%.Source: §1 · p. 2
  • 51.8% pass@8 for FlyBy-8B comes without increased serving cost, and reinforcement learning supports repeated cheap queries to approach the strongest backend’s performance.The repeated-query analysis reports this performance at nearly the cost of the cheapest backend.Source: §1 · p. 2
  • FlyBy trains 4B and 8B models to reason first and selectively query stronger models, outperforming larger models at lower serving cost.The framework’s policy learns whether, what, and how much to query.Source: §1 · p. 2

2. Preliminaries

The preliminaries define how the paper probes intermediate reasoning states and measures the effects of reflection-related interventions. They use answer success and answer diversity to characterize what remains reachable from a state.

Reasoning states and state probing

  • A reasoning state is probed by appending a cue and sampling continuations; its value is the fraction that reach the correct answer.The setup begins with a problem, a model-generated trace, and the state before a trace step.Source: §2 · p. 3
  • Answer entropy measures how varied the resulting answers are, while the null cue leaves the state unmodified.The paper places states on a value–entropy plane, with (1, 0) representing a correct, settled answer.Source: §2 · p. 3

Epistemic verbalizations and paired interventions

  • Epistemic verbalizations are short hedging or checking expressions, and paired interventions compare their effects against a counterfactual from the same state.The paper distinguishes expressions emitted spontaneously from cues inserted at a chosen state; entropy changes are defined analogously.Source: §2 · p. 3

3. Understanding When Thinking Is Not Enough

The analysis distinguishes failures recoverable by further reasoning from those needing external information: self-refinement mainly helps with the former, while relevant information expands what is reachable. Smaller models face both more knowledge-like bottlenecks and greater difficulty using hints.

3. Understanding When Thinking Is Not Enough

  • The analysis samples eight Qwen3 and Gemma4 models across mathematical and scientific reasoning, retaining model-problem pairs with solve rates from 0.25 to 0.75.It uses 16 rollouts per model-problem pair and analyzes 272K counterfactual and information-conditioned continuations.Source: §3 · p. 3

3.1. Execution and knowledge bottlenecks

  • Execution bottlenecks have a practically reachable correct solution, whereas knowledge bottlenecks require information not reliably accessible through further internal reasoning.Operationally, states with an observed successful continuation are execution-like; states with none are knowledge-like, a bounded-reasoning distinction rather than absolute reachability.Source: §3.1 · p. 4

3.2. Self-refinement in small reasoning models

  • Self-refinement benefits uncertain, execution-like states, primarily helping realize solutions already internally reachable.EVs remain common in smaller models, but their causal effect on value increases substantially with model scale.Source: §3.2 · p. 4

3.3. Effect of external information

  • Relevant information benefits knowledge-like bottlenecks, and its utilization generally increases with model scale.Smaller models more often enter knowledge-like states and are less capable of incorporating a relevant hint; the apparent largest-model drop reflects changes in the residual problem set.Source: §3.3 · p. 5

4. Teaching Small Models When Thinking Is Not Enough

FlyBy trains a small reasoning model to decide when to query an external model, what to ask, and how much computation to use. Its learned policy extends coverage on problems the base model did not solve and is cost-efficient across most evaluated price configurations.

4.2. Two-stage optimization

  • FlyBy-4B uses supervised fine-tuning to bootstrap query actions, then reinforcement learning to calibrate when querying is worth its cost and how much computation to allocate.The SFT data includes query-augmented failed trajectories and supervision for query generation and post-observation integration; RL uses problems disjoint from SFT synthesis.Source: §4.1 · p. 6, §4.2 · p. 6

4.1. Training Setup

  • The multi-depth tool lets FlyBy choose query depth, receive an observation, and resume reasoning; deeper queries provide stronger models and longer responses at higher cost.External models see only the query, and queries with excessive n-gram overlap with the original problem are rejected.Source: §4.1 · p. 6, Table 1 · p. 6

4.2. Two-stage optimization

  • The modified reinforcement-learning objective penalizes cost only for successful trajectories, avoiding rewards for simply failing cheaply.It can favor external information when that reduces the overall cost of reaching a correct solution.Source: §4.2 · pp. 6–7
  • On problems with zero Qwen3-4B successes across 16 rollouts, FlyBy-4B reaches 28.7% pass@8; it also offers the best cost efficiency across most evaluated price configurations.Coverage gains over Query Opening are larger on harder problems.Source: Figure 5 · p. 7

5. Experiments

Experiments on 1,158 hard problems across six benchmarks compare FlyBy with larger models and other tool-use approaches. FlyBy improves hard-problem coverage through targeted querying, with cost-aware RL and larger cognitive cores further strengthening results.

5.1. Experiment setup

  • Evaluation covers 1,158 hard problems across six reasoning benchmarks, with hard problems defined by vanilla Qwen3-4B pass@1 ≤0.25.The evaluated approaches include larger models, retrieval, one-time query opening, tool-interface prompting, forking, and FlyBy variants.Source: §5.1 · pp. 7–8

5.2. Main Results

  • 45.96% pass@8: FlyBy-4B surpasses Qwen3-14B at 2.7× lower serving cost.Light SFT achieves 36.39% pass@8, while cost-aware RL improves it with only a marginal increase in serving cost.Source: §5.2 · p. 8
  • FlyBy-4B uses short external responses averaging 105 output tokens, while Search-R1 remains substantially below it despite 2,015-character contexts per call.ForkingRL provides little improvement over the base model, while Search-R1 improves coverage through retrieval.Source: §5.2 · p. 8
  • As problems become harder, Query Opening falls further behind adaptive querying, which first reasons about missing information before seeking targeted assistance.The reported explanation is that a broad upfront query becomes less likely to surface the particular knowledge needed as difficulty increases.Source: §5.2 · p. 8
  • On 50 SuperGPQA problems, querying raises state value by 5.30 pp on average, whereas further reasoning yields essentially no improvement.Figure 6 also shows RL sustaining query probability at later turns and adaptive depth allocation reaching near-deep-backend performance at near-shallow-backend cost.Source: §5.2 · pp. 8–9, Figure 6 · p. 8
  • FlyBy-4B remains optimal across a broad range of GPU and API price regimes for pass@8 per USD.The price sweep indicates that its economic advantage is not tied to a particular pricing assumption.Source: §5.2 · pp. 8–9
  • 51.81% pass@8: FlyBy-8B improves over FlyBy-4B by 5.85 percentage points and Qwen3-14B by 10.17 points while remaining cheaper to serve.The paper reports that larger models possess greater parametric knowledge and benefit more from utilizing given information.Source: §5.2 · p. 9

6. Related Work

Related work studies self-refinement and uncertainty signaling, capability gaps in small reasoning models, and adaptive tool use. This paper distinguishes which failures need external information and uses that distinction to motivate targeted assistance.

6. Related Work

7. Conclusion

The conclusion presents FlyBy as selective external-computation acquisition for small reasoning models, with reported gains over a larger baseline. The probing appendix describes how reasoning states and interventions are compared.

7. Conclusion

  • FlyBy-4B reaches 46.0% pass@8 on hard problems and outperforms Qwen3-14B at 2.7× lower serving cost.The conclusion frames effective test-time scaling as knowing when additional thinking is not enough.Source: §7 · p. 10
  • FlyBy relies on external models, bringing practical concerns about availability, privacy, and reliability.The authors discuss these limitations and future directions in §I.Source: §7 · p. 10

A. Probing Reasoning States

  • A fixed probe policy places states generated by different policies on a common comparison plane; unless stated otherwise, the producing policy is also the probe policy.The probe policy evaluates a reasoning state and need not be the policy that produced it.Source: A. Probing Reasoning States · p. 14
  • The probing procedure appends a cue to a reasoning state and samples independent continuations; the null cue leaves the state unmodified.The final answers from those continuations define the observed answer set and its size.Source: A. Probing Reasoning States · p. 14
  • The appendix estimates answer entropy and notes that its second term is the Miller-Madow bias correction; paired interventions compare a cue with the null intervention on the same state.It also measures reflection by comparing continuations with epistemic-verbalization items allowed or banned.Source: A. Probing Reasoning States · p. 14

B. Robustness to the Continuation Budget

The section tests whether zero observed success at N=8 means a solution is unreachable and reports that higher continuation budgets reveal rare successes, without closing the intervention gap. It also details how FlyBy’s oracle information, query tool, costs, and training data are constructed.

B. Robustness to the Continuation Budget

  • 28 of 76 states with no success in the first eight samples become positive by N=64, so zero observed success is a finite-sample criterion rather than literal unreachability.The analysis classifies states using up to 64 sampled continuations.Source: B. Robustness to the Continuation Budget · p. 15
  • Newly positive states retain a large intervention gap: rare successful continuations do not imply that self-refinement substantially amplifies their probability.External information can shift low-reachability states by supplying information that further internal reasoning alone has difficulty recovering.Source: B. Robustness to the Continuation Budget · p. 15, Figure 7 · p. 15

C. Generating oracle information

  • Oracle information is a minimal, problem-specific missing fact or hint intended to aid recovery from an incorrect trajectory without revealing the final answer.Hints primarily provide domain knowledge for knowledge-intensive problems and a reasoning strategy or derivation direction for mathematical or symbolic problems.Source: C. Generating oracle information · p. 15

D.1. Details on Tool

  • The tool offers three query depths with stronger models and longer responses at higher cost, while the external model receives only the query, not the original problem.The interface asks the model to query only for a specific external knowledge gap and use the shallowest sufficient depth.Source: §D.1 · p. 16

D.2. Details on Cost

  • Serving cost combines local GPU inference and external API usage into a common monetary measure.The local estimate uses measured prefill and decoding throughput; API cost sums input and output token charges across tool calls.Source: §D.2 · pp. 16–17

D.3. Details on SFT

  • SFT rescue trajectories are retained only when tool assistance robustly rescues a failed prefix, yielding 95 single-call rescues from 800 training problems.The criterion requires no successful plain continuations and at least three successful tool-assisted continuations among four samples.Source: §D.3 · p. 17
  • Full-trajectory supervision alone rarely induced queries, while query-only supervision often led to repeated queries after observations.The training data therefore also supervises integration of the returned observation into subsequent local reasoning.Source: §D.3 · p. 18

D.4. Details on RL; D.6. Training 8B Model

  • RL training mixes ambiguous-solve-probability problems with problems likely to benefit from external information at approximately a 6:4 ratio.For FlyBy-8B, the SFT set contained 87 retained trajectories, and no-tool-solvable examples were reduced to 50% of the RL mixture.Source: §D.4 · p. 18, §D.6 · p. 19

E.2. Details on baselines implementation

FlyBy-4B leads Qwen3-8B on hard-subset pass@1 while using less than half its serving cost; the section also specifies baseline implementations and evaluation sampling settings.

E.2. Details on baselines implementation

  • The baselines include standalone ForkingRL, retrieval-based Search-R1, and Query Opening, which obtains external information once before reasoning.ForkingRL uses high-entropy-token updates without tools; Search-R1 interleaves reasoning with retrieval, while Query Opening uses a depth-1 external model.Source: §E.2 · pp. 19–20

E.3. Hyperparameters during evaluation

  • Evaluation uses temperature 0.6, top-p 0.95, and top-k 20, with other hyperparameters kept as reported in Table 4.Source: §E.3 · p. 20

E.4. Additional Results

  • 16.85% pass@1 for FlyBy-4B exceeds Qwen3-8B’s 15.31% at less than half the serving cost, 0.93 vs. 1.92 m$ per rollout.FlyBy-8B reaches 22.19% pass@1, matching Qwen3-14B at 1.4× lower serving cost.Source: §E.4 · p. 20

F. Analysis on Reinforcement Learning

RL produces iterative querying beyond the single-query behavior shown in SFT, while the cost penalty calibrates query depth. Backend substitutions retain most performance, and excessive cost pressure can suppress some successful trajectories.

F. Analysis on Reinforcement Learning

  • RL increases generation turns for all tested cost penalties, learning repeated alternation between local reasoning and external information acquisition despite SFT containing only single-query rescues.Source: F. Analysis on Reinforcement Learning · pp. 20–21
  • The cost penalty changes query depth: smaller penalties permit more expensive backends, while larger penalties preserve a preference for shallow calls.The analysis distinguishes RL’s role in composing tool interactions from the penalty’s role in calibrating their computation.Source: F. Analysis on Reinforcement Learning · p. 21
  • At λ=0.2, a correct but relatively expensive trajectory can receive negative advantage, whereas this behavior is negligible at smaller penalties.The main experiments use λ=0.1 as a middle regime.Source: F. Analysis on Reinforcement Learning · p. 21
  • SFT teaches query issuance and observation integration, while RL organizes those actions over an evolving reasoning trajectory under a cost constraint.Source: F. Analysis on Reinforcement Learning · p. 21

G.1. Generalization across query backends.

  • 43.21% pass@8 with GPT-5.6 Luna and 43.96% with HY3 compare with 45.96% using the training-time DeepSeek backend; both variants reduce cost versus the baseline by 56.5% and 60.6%.The fixed policy and query interface are retained when the backend is substituted.Source: §G.1 · p. 21

G.2. Analysis on Leakage

The leakage analysis finds that tool observations alone add little accuracy when reasoning is disabled. The section also describes cost-perturbation comparisons and examples in which FlyBy obtains targeted information, then completes the reasoning itself.

G.2. Analysis on Leakage

  • Adding a tool observation raises problem-weighted accuracy with thinking disabled only from 10.20% to 12.16%, a gain of 1.96 percentage points.The results suggest that querying helps primarily when observations are incorporated into subsequent reasoning, rather than through direct answer leakage.Source: §G.2 · pp. 21–22, Table 8 · p. 21

G.3. Additional economic analysis

  • The economic analysis compares cost-performance Pareto frontiers when API prices rise 10× and GPU prices rise 3×.Source: Figure 9 · p. 22

H. Qualitative Examples

  • In the medical example, a query supplies the missing connection between paravalvular leak and mechanical hemolysis, raising continuation success from 0/8 to 7/8.FlyBy-4B then integrates that fact with its reasoning to infer TEE as the next step.Source: Figure 10 · p. 22, H. Qualitative Examples · pp. 22–23
  • In the physics example, the returned projected-area relation helps correct the model’s drag calculation, raising continuation success from 2/8 to 8/8.The model independently reaches a total torque of 7.5425 N m.Source: Figure 11 · p. 23, H. Qualitative Examples · p. 23
  • Across both examples, FlyBy identifies missing information locally, queries for it, and resumes the remaining reasoning rather than outsourcing the original problem.Source: H. Qualitative Examples · p. 23

I. Limitations and future directions

FlyBy's reliance on stronger external models creates deployment and reliability constraints. The authors identify privacy-aware selective querying as a future direction.

I. Limitations and future directions

  • External API access may be unavailable or undesirable because of privacy, latency, and deployment constraints.Source: §I · p. 23
  • External models may produce incorrect or hallucinated information that propagates into subsequent reasoning.Source: §I · p. 23
  • Future work could selectively reveal or fragment context to minimize sensitive-information exposure while acquiring external information.Source: §I · p. 23
Loading 2609.34327v1…