When Should Agents Think? Adaptive Reasoning via Cross-Turn Estimation

1 INTRODUCTION

• Earlier reasoning can support subsequent actions for varying ranges: some steps help for one turn, while others remain useful across multiple turns.The paper calls this cross-turn reasoning persistence and suggests plans, constraints, or intermediate judgments may remain useful beyond their original turns.Source: §1 · pp. 1–2
• Existing turn-level verification overlooks varying cross-turn persistence and requires repeated action generations, adding cost in data synthesis and on-policy reinforcement learning.The paper identifies this overhead as increasingly pronounced as trajectories grow longer and policies evolve.Source: §1 · p. 2
• RACE uses LoGiC to test candidate reasoning removals against likelihood decreases for current and subsequent reference actions, retaining reasoning when the decrease exceeds a predefined threshold.The retained steps form a reasoning cover that supports actions throughout the trajectory.Source: §1 · pp. 2–3
• RACE incorporates removal signals into supervised fine-tuning and agentic reinforcement learning, where task rewards optimize performance and closure supervision teaches when to stop reasoning.LoGiC transforms full-reasoning trajectories during fine-tuning and derives signals from successful on-policy trajectories during reinforcement learning.Source: §1 · pp. 2–3
• 32.3%, 80.2%, 46.4%, and 62.0% fewer reasoning tokens on ScienceWorld, WebShop, AppWorld, and DeepSearch, respectively, with comparable performance versus standard SFT followed by RL.The paper reports that RACE maintains or improves task performance while substantially reducing reasoning cost across four representative benchmarks.Source: §1 · p. 3

2.1 PROBLEM FORMULATION

• The problem formulation optimizes a policy to invoke reasoning selectively across decision turns while preserving task performance.At each turn, the agent may reason before acting or act directly without additional reasoning.Source: §2.1 · p. 3

2.2 MEASURING CROSS-TURN SUPPORT THROUGH GENERATION

• Earlier reasoning can support subsequent reference actions, but its effective range varies from one turn to multiple turns.Measuring recovery requires generating candidate actions for each truncated context, which is costly for large-scale synthesis and on-policy reinforcement learning.Source: §2.2 · p. 4

2.3 LIKELIHOOD-BASED ESTIMATION OF ACTION SUPPORT

• Likelihood decreases closely track reference-action recovery rates, offering a lightweight proxy for whether later actions remain recoverable using earlier reasoning.Unlike action generation, the proxy requires only batched teacher-forced scoring of reference actions.Source: §2.3 · p. 4

3.1 LIKELIHOOD-GUIDED PROGRESSIVE REASONING COVER DETECTION

• LoGiC initializes reference-action likelihoods on the full trajectory, tests removals sequentially from turn two, and accepts them only when current and future likelihood drops stay within the threshold.Accepted removals update the context for later evaluations; retained reasoning steps form the trajectory’s reasoning cover.Source: §3.1 · pp. 4–5

3.2 AGENT REASONING SYNTHESIS FOR SUPERVISED FINE-TUNING

• LoGiC converts removable reasoning in agent trajectories into adaptive SFT data, with supervision on model-generated reasoning and action tokens.This stage provides an initial adaptive reasoning policy for subsequent agentic reinforcement learning.Source: §3.2 · p. 6

3.3 REASONING-COVER-AWARE AGENTIC REINFORCEMENT LEARNING

• On successful on-policy trajectories, LoGiC masks verified-SKIP reasoning from GRPO while retaining action tokens and adds reasoning-closure supervision; failed trajectories use standard GRPO.Closure supervision is normalized by rollout turns, and the actor objective combines task optimization with closure supervision.Source: §3.3 · pp. 6–7

4.1 EXPERIMENTAL SETTINGS

• The evaluation spans scientific reasoning, web shopping, application use, and knowledge-intensive reasoning, reporting success rate and average reasoning tokens per trajectory.Baselines include ReAct-style agents and reasoning-efficiency methods, with the latter evaluated using Qwen3.5-9B as the backbone.Source: §4.1 · pp. 7–8

4.3 QUANTITATIVE ANALYSIS

• RACE-SFT stays within 1.88 points of standard SFT while reducing reasoning tokens by 4.3% to 20.0%; RACE-SFT-RL stays within 1.40 points of standard SFT+GRPO while cutting tokens across benchmarks.Removing LoGiC from RL raises token counts from 1,403 to 2,573 on ScienceWorld, 535 to 2,670 on WebShop, 1,900 to 8,574 on AppWorld, and 822 to 3,507 on DeepSearch; removing reasoning masking reduces success on all four.Source: Figure 4 · p. 8, §4.3 · p. 9

4.2 MAIN RESULTS

• RACE also achieves competitive success rates against substantially larger and proprietary LLM agents across the four benchmarks.Its success rates are 73.28 on ScienceWorld, 53.40 on WebShop, 65.13 on AppWorld, and 35.00 on DeepSearch.Source: §4.2 · p. 8
• Against Qwen3.5-9B, RACE-SFT-RL improves success while reducing reasoning tokens across all four benchmarks.Success gains are 54.76, 37.80, 18.98, and 5.75 points; reasoning-token reductions are 36.5%, 80.3%, 39.4%, and 35.8% on ScienceWorld, WebShop, AppWorld, and DeepSearch, respectively.Source: §4.2 · p. 8
• Compared with RACE-SFT, RACE-SFT-RL raises success and reduces reasoning tokens on all four benchmarks.Success increases by 30.23, 8.00, 2.05, and 6.75 points, while reasoning tokens fall by 28.6%, 56.6%, 42.9%, and 55.7% on ScienceWorld, WebShop, AppWorld, and DeepSearch, respectively.Source: §4.2 · p. 8

4.3 QUANTITATIVE ANALYSIS

• Across Qwen3.5-2B, 4B, and 9B, RACE generally gains task performance with scale, while RACE-SFT-RL generally improves performance and reduces tokens versus RACE-SFT.AppWorld RL gains vary by model size and may reflect limited training data and less stable RL at smaller scales; token reductions are especially large on WebShop.Source: Figure 5 · p. 9, §4.3 · p. 9
• RL increases the skip ratio on all four benchmarks, while reasoning tokens per thinking turn rise on ScienceWorld and AppWorld but fall on WebShop and DeepSearch.RACE skips reasoning at more turns rather than uniformly shortening it; retained reasoning varies with task characteristics.Source: §4.3 · p. 9

Reasoning-Efficient Methods

• Prior efficiency work improves planning, execution, or direct optimization, while adaptive-reasoning studies dynamically determine when or how much to reason (Xu et al., 2023; Hu et al., 2024; Chen et al., 2026b; 2025; Lewis-Lim et al., 2025; Zhang et al., 2025; Fang et al., 2025; Lou et al., 2025; Wu et al., 2025; Lu et al., 2025; Su et al., 2026; Liang et al., 2025; Yang et al., 2026a; Ning et al., 2026; Yang et al., 2026b).RACE extends adaptive reasoning to turn-wise agent interaction, using a lightweight proxy for fresh-reasoning need and applying its signals to SFT and agentic RL.Source: §5 · pp. 9–10

Agentic Reinforcement Learning

• Agentic RL research has emphasized policy optimization, rollout exploration, and credit assignment, with less attention to adaptive reasoning during interaction (Schulman et al., 2017).RACE complements this work by introducing adaptive reasoning into agentic reinforcement learning.Source: §5 · p. 10

6 CONCLUSION

• RACE uses reference-action likelihood decreases as a lightweight signal for estimating cross-turn reasoning support and identifying removable reasoning steps.LoGiC produces removal signals used in both supervised fine-tuning and agentic reinforcement learning.Source: §6 · p. 10
• Across four representative agent benchmarks, RACE substantially reduces reasoning cost while maintaining or improving task performance.Source: §6 · p. 10
• The paper reports AI assistance with editing, organization, and literature discovery, and says the authors reviewed and verified the assisted content.The authors state that they take responsibility for the final content.Source: AI USE STATEMENT · p. 10

A TASKS AND DATASETS

• The method is evaluated on scientific reasoning, web shopping, application use, and knowledge-intensive search tasks.Source: A TASKS AND DATASETS · p. 17

A.1 SCIENTIFIC REASONING TASK

• ScienceWorld covers 30 tasks across 10 science topics, from changes of state and circuits to plants, life stages, and Mendelian genetics.The topics also include temperature measurement, friction, object classification, chemical mixtures, pollinators, and life spans.Source: §A.1 · p. 17
• Each task requires a sequence of textual observations and actions to accomplish a specified scientific goal.Source: §A.1 · p. 17
• Training uses the official training split, and evaluation uses all 1,819 instances in the official test split.Source: §A.1 · p. 17

A.2 WEB SHOPPING TASK

• WebShop contains 1,181,436 real-world products collected from Amazon and 12,087 crowd-sourced natural-language instructions.Source: §A.2 · p. 18
• Each instruction specifies desired product requirements, such as attributes, options, and price, and agents must find and purchase a matching item.Source: §A.2 · p. 18
• The official splits contain 10,587 training, 1,000 development, and 500 test instances; experiments train on the official training split and evaluate on all test instances.Source: §A.2 · p. 18

A.3 APPLICATION USE TASK

• AppWorld tasks use simulated applications and APIs for realistic day-to-day activities, often requiring coordination across apps and reasoning over intermediate results.Completion is evaluated with state-based unit tests; the experiments use 147 available Train and Dev tasks and evaluate on 168 Test-N and 417 Test-C tasks.Source: §A.3 · p. 18

A.4 DEEP SEARCH TASK

• DeepSearch combines seven question-answering datasets, including general and multi-hop questions; training uses four datasets and evaluation samples 400 development examples across all seven.Source: §A.4 · p. 18

B.1 REACT-STYLE ITERATIVE REASONING AGENTS •

• The experiments describe DeepSeek-V4-Pro, GLM-5.3, GPT-5.5, Claude-Opus-4.6, Kimi-K3, and Qwen3.8-27B as agent models, with medium reasoning or thinking settings specified for GPT-5.5 and Claude-Opus-4.6.Source: §B.1 · pp. 18–19

B.2 REASONING-EFFICIENT AGENTS •

• The ReAct-style baselines include Sparse ReAct, which skips reasoning at selected turns, and ADaPT, which decomposes tasks when an executor cannot complete a subtask directly.Source: §B.2 · p. 19
• Think Tool prompts agents to choose an auxiliary deliberation tool, while adapted DART uses agreement among no-thinking drafts to decide whether to reason at each turn.DART was originally designed for query-level reasoning and is adapted here to multi-step agent scenarios.Source: §B.2 · p. 19
• AdaptThink chooses thinking or non-thinking at each decision turn and uses GRPO with a 0.05 bonus for non-thinking turns on successful trajectories.Source: §B.2 · p. 19

C.1 EVALUATION PROTOCOL

• Across four benchmarks, episodes are capped at 30 turns; evaluation reports task success and average reasoning tokens per trajectory, using benchmark-specific success criteria.ScienceWorld uses 1,819 instances, WebShop 500, AppWorld 585, and DeepSearch 400; DeepSearch uses Exact Match.Source: §C.1 · pp. 19–20

C.2 TRAINING

• SFT uses successful DeepSeek-V4-Pro-0813 trajectories labeled by LoGiC, while RL starts from benchmark-specific RACE-SFT checkpoints and applies LoGiC to successful on-policy trajectories.Both stages use a likelihood-decrease threshold of ϵ = 0.001; RL closure-loss weights are 0.02 for ScienceWorld, WebShop, and AppWorld and 0.01 for DeepSearch.Source: §C.2.1 · p. 20, §C.2.2 · p. 20

D PROMPT TEMPLATE

• Task-specific prompts follow each benchmark’s interaction protocol, including one action per turn for ScienceWorld and WebShop and API-based multi-call turns for AppWorld.The templates also instruct DeepSearch agents to search only as needed and return a minimal answer.Source: D PROMPT TEMPLATE · pp. 20–21

A. General instructions: / B. App-specific instructions:

• AppWorld instructions require autonomous, bounded task execution, prohibit guessed values and unrelated operations, and direct agents to retrieve personal information and current time through app APIs.Source: A. General instructions · p. 21, B. App-specific instructions · p. 21

C. Task-completion instructions:

• AppWorld task completion requires calling the supervisor completion API; if the task cannot be solved, the agent may report failure, and final answers should be minimal.Source: C. Task-completion instructions · p. 21

E THE ALGORITHM WORKFLOW OF THE LIKELIHOOD-GUIDED PROGRESSIVE REASONING COVER DETECTION

• LoGiC progressively tests reasoning-turn removals against action likelihoods, accepting removals that meet its threshold and outputting a compressed trajectory and reasoning cover.Source: E THE ALGORITHM WORKFLOW OF THE… · p. 22

F.1 CASE STUDY FOR SCIENTIFIC REASONING

• In the ScienceWorld case, an infeasible direct pickup is rejected, prompting a revised pouring strategy; after the first pour succeeds, the policy performs supported remaining actions without further reasoning.Source: §F.1 · p. 22

F.2 CASE STUDY FOR WEB SHOPPING / F.3 CASE STUDY FOR APPLICATION USE

• In WebShop, reasoning returns to verify an uncertain wrapping attribute; in AppWorld, it reappears at subgoal transitions, while routine browsing and operations proceed directly.Source: §F.2 · p. 23, §F.3 · p. 23

• The policy decomposes the question into identifying both film directors and their birth dates, then skips extra reasoning for the second director lookup.Source: §F.4 · p. 23
• The trajectory concludes that Rex Ingram was born earlier than William A. Wellman, using reasoning for evidence synthesis but not straightforward retrieval.Source: §F.4 · p. 23
• Table 7 marks reasoning, tool calls, tool results, and the final answer with distinct symbols across the interaction.Source: Table 7 · p. 28