PaperlayerView exact revision on arXiv

Source-linked AI summary

Learning Agentic Policy from Action Guidance

Yuxiang Ji, Zengbin Wang, Yong Wang, Shidong Yang, Ziyu Ma, Guanhua Chen, Zonghua Sun, Liaoni Wu, Xiangxiang Chu

arXiv:2605.12004v1cs.CL

TL;DR

Agentic RL can stall when the base policy cannot reach reward states. ActGuide-RL uses adaptive plan-style action guidance with mixed-policy training, consistently improving zero RL across search-agent benchmarks and matching SFT+RL without cold-start initialization.

  • Problem

    Agentic RL lacks effective learning signals when reward states lie beyond the base policy’s reachable region, causing training to stall on difficult tasks.

  • Method

    ActGuide-RL injects action data as adaptive plan-style fallback guidance and jointly optimizes guided and unguided rollouts to internalize exploration gains.

  • Results

    Across search-agent benchmarks, ActGuide-RL consistently improves zero RL, including +10.68 pp on GAIA with Qwen3-4B-Instruct, while matching SFT+RL without cold-start initialization.

  • Takeaways & Limitations

    ActGuide-RL reduces reliance on supervised cold-start data while providing consistent gains over vanilla RL in the evaluated search-agent setting.

  • Takeaways & Limitations

    The experiments focus on search agents, so effectiveness in CLI, GUI, API-based, and embodied environments remains to be explored.

Abstract

from arXiv · show

Agentic reinforcement learning (RL) for Large Language Models (LLMs) critically depends on the exploration capability of the base policy, as training signals emerge only within its in-capability region. For tasks where the base policy cannot reach reward states, additional training or external guidance is needed to recover effective learning signals. Rather than relying on costly iterative supervised fine tuning (SFT), we exploit the abundant action data generated in everyday human interactions. We propose \textsc{ActGuide-RL}, which injects action data as plan-style reference guidance, enabling the agentic policy to overcome reachability barriers to reward states. Guided and unguided rollouts are then jointly optimized via mixed-policy training, internalizing the exploration gains back into the unguided policy. Motivated by a theoretical and empirical analysis of the benefit-risk trade-off, we adopt a minimal intervention principle that invokes guidance only as an adaptive fallback, matching task difficulty while minimizing off-policy risk. On search-agent benchmarks, \textsc{ActGuide-RL} substantially improves over zero RL (+10.7 pp on GAIA and +19 pp on XBench with Qwen3-4B), and performs on par with the SFT+RL pipeline without any cold start. This suggests a new paradigm for agentic RL that reduces the reliance on heavy SFT data by using scalable action guidance instead.

1 Introduction

Agentic RL can stall when reward states lie beyond the base policy’s reachable capability, but abundant human-generated action data can provide plan-style guidance to overcome this barrier. ACTGUIDE-RL adaptively combines guided and unguided rollouts to improve exploration while reducing reliance on cold-start SFT.

  • Motivation: Online RL stalls when reward states fall outside the base policy’s reachable region, because group-based advantage estimates collapse to zero gradient.The introduction distinguishes reachable in-region tasks from unreachable out-region tasks.
  • Motivation: Abundant open-world and task-construction action data offers a direct training source for models learning to act, interact, and decide on long-horizon tasks.Examples include step-by-step GUI/CLI interactions and API-mediated actions.
  • Method: ACTGUIDE-RL injects action data as plan-style reference guidance, helping policies cross capability barriers and visit out-region states for effective learning.The method is motivated by an empirical analysis of agentic-policy capability barriers.
  • Results: ACTGUIDE-RL consistently improves zero RL across tested base models, with larger gains on harder benchmarks where unguided RL struggles to obtain effective training signals.The evaluation spans search-agent benchmarks, varying base models, task difficulties, and in-domain and out-of-domain settings.

2 Method

ACTGUIDE-RL addresses structural reachability barriers in agentic RL by using action trajectories as reference-plan guidance to cross regions the base policy cannot explore. It adaptively applies the minimal sufficient guidance and jointly optimizes guided and unguided rollouts with source-aware importance weighting.

  • Reachability barriers: A reachability barrier causes zero group-based advantage gradients beyond a critical interval, so increasing rollout count cannot recover learning signals.The failure is structural: the policy must first be steered across the interval to access out-region reward states.
  • Action guidance: Action guidance treats human action trajectories as reference plans, and guided rollouts reveal where the base policy diverges and barriers are repaired.Guidance-induced logit shifts spike during barrier intervals, while unguided Pass@K recovers after the guided trajectory crosses them.
  • Action guidance: Guidance is provided as an appended list of future reference actions rather than a forced prefix, with prefix length k controlling intervention strength.The ordered family gk enables searching for the minimal sufficient intervention needed to cross a task-specific barrier.
  • Adaptive intervention: As guidance level k increases, off-policy risk Rk keeps rising, while barrier-repair benefit exhibits threshold behavior; therefore, utility peaks near the minimal successful level.The adaptive policy selects the smallest guidance level whose estimated success exceeds threshold δ.
  • Mixed-policy optimization: Mixed-policy optimization trains guided and unguided rollouts together, using source-specific importance ratios to transfer guided credit back to the unguided target policy.The objective remains unchanged because minimal intervention limits distribution shift between guided rollouts and the base policy.

3 Experiment

Experiments on search-agent benchmarks show that ACTGUIDE-RL expands exploration beyond vanilla RL’s reachable region, mitigates capability regression, and discovers more effective learning signals. It also enables complex interaction skills without cold-start initialization while remaining robust to action noise and relying on mixed-policy optimization to transfer guided behaviors.

  • Benchmarks and setup: Experiments evaluate ACTGUIDE-RL on stateless search-agent benchmarks spanning GAIA, WebWalkerQA, XBench, and BrowseComp-ZH.The study uses web-search and web-visit tools, raw tool outputs, and a few-shot reference-based binary LLM judge.
  • Overall comparison: Vanilla RL can regress when training-data exploration difficulty mismatches the base model, whereas adaptive guidance alleviates these regressions through more effective state visitation.The reported examples include Qwen2.5-7B-Instruct on GAIA and Qwen3-8B on BC-ZH.
  • Overall comparison: Action guidance helps the policy access effective states beyond its current reachable region when vanilla RL fails on harder tasks.This effect is especially evident for Qwen3-4B-Instruct.
  • Training dynamics: Action data produces effective learning signals in a higher proportion of rollout groups, while unguided training wastes rollouts behind exploration barriers.The training-dynamics analysis attributes the difference to more effective state discovery under guidance.
  • Towards complex interaction: ACTGUIDE-RL enables Qwen3-4B-Instruct to gradually acquire complex interaction capability without cold-start initialization.This is reflected by steady increases in interaction turns and generated tokens during training.
  • Ablation and robustness: Removing mixed-policy optimization causes a substantial performance drop because guided behaviors no longer transfer into test-time unguided capability.The method is also reported to maintain stable performance under simulated task-irrelevant action noise.

4 Related Work

Related work positions agentic RL as a framework for training interactive, sequential decision-making agents, while emphasizing its dependence on base-model exploration and its connection to reinforcement learning from demonstrations.

  • Recent RL advances enable end-to-end training of agents that interact with environments, make sequential decisions, and optimize long-horizon objectives.
  • Agentic RL supports both foundation-model capability building and domain-specific agent post-training.
  • Effective agentic RL depends strongly on the base model exploring valid training signals, motivating existing reliance on cold-start methods.
  • RL from demonstrations typically uses expert trajectories containing full reasoning-and-action traces in agent settings.
  • Classical RLfD bootstraps exploration in sparse-reward settings by retaining demonstrations in replay buffers and combining RL updates with auxiliary imitation losses.

5 Conclusion · Appendix

ACTGUIDE-RL uses readily available action data as plan-style guidance to help agentic RL overcome exploration barriers beyond the base policy’s reachable region. Adaptive fallback guidance and joint optimization of guided and unguided rollouts internalize exploration gains while reducing off-policy risks, yielding consistent gains over vanilla RL on search-agent benchmarks.

  • 5 Conclusion: ACTGUIDE-RL leverages readily available action data as plan-style guidance for agentic RL.
  • 5 Conclusion: The framework helps agentic RL overcome exploration barriers beyond the base policy’s reachable region.
  • 5 Conclusion: Guidance is introduced only as an adaptive fallback rather than applied continuously.
  • 5 Conclusion: Guided and unguided rollouts are optimized jointly during training.
  • 5 Conclusion: Joint optimization internalizes exploration gains while reducing the off-policy risks of excessive intervention.
  • 5 Conclusion: The design choices yield consistent gains over vanilla RL across search-agent benchmarks.

A Datasets · A.1 Train

The training setup uses sampled ASearcher instances for search-agent RL, supplemented by expert-generated action guidance and a separate SFT dataset. Action data retains atomic tool operations, whereas SFT data preserves complete trajectories with reasoning, calls, and responses.

  • A Datasets: The datasets comprise search-agent RL training data and separately sampled SFT data from ASearcher.The RL and SFT subsets are disjoint, according to the training-data description.
  • A.1 Train: 2k ASearcher instances were sampled for RL training across all experimental settings.
  • A.1 Train: Tongyi-DeepResearch-30B-A3B generated the action data through rejection sampling as the expert model.
  • A.1 Train: The action-data toolset was restricted to web-search and web-visit.
  • A.1 Train: Correct expert trajectories supplied atomic per-step operations, including tool call names and corresponding arguments, as candidate complete action guidance.
  • A.1 Train: 4k complete search-agent trajectories were produced for SFT from a disjoint ASearcher subset using the same expert model and rejection sampling.
  • A.1 Train: Unlike action data, SFT data preserves full Chain-of-Thought reasoning, explicit tool calls, and corresponding tool responses.

A.2 Evaluation

The evaluation uses challenging deep-search benchmarks to measure complex reasoning, web browsing, traversal, and information gathering, alongside general-purpose tests of out-of-domain reasoning, factual robustness, and instruction following.

  • Search-agent benchmarks: The search agent is evaluated on standard and challenging deep-search benchmarks for complex reasoning and deep search.These benchmarks assess the system’s capabilities in the intended search-agent setting.
  • Search-agent benchmarks: GAIA uses 103 text-only real-world questions requiring deep reasoning and web browsing to test fundamental capabilities.GAIA is a general AI assistant benchmark.
  • Search-agent benchmarks: WebWalkerQA contains 680 QA tasks requiring systematic traversal of multiple dynamic web pages to discover multi-layered information through multi-hop reasoning.It evaluates complex web traversal and information gathering.
  • Out-of-domain benchmarks: Out-of-domain generalization is assessed with GPQA, TruthfulQA, and IFEval.These benchmarks extend evaluation beyond the search-agent setting.
  • Out-of-domain benchmarks: GPQA tests graduate-level scientific reasoning, TruthfulQA tests factual robustness against misconceptions, and IFEval tests instruction following under verifiable constraints.Together, they cover reasoning, truthfulness, and constrained instruction following.

B Experiment Details

The experiments implement action guidance as plan-style references during guided rollouts and compare alternative guidance and distillation strategies. OPSD improves performance but remains limited because unguided state visitation cannot reach useful states independently.

  • Implementation Details: Guided rollouts inject action data into the query as plan-style reference guidance, allowing the policy to follow partial trajectories while completing missing steps itself.The exact prompt format is provided in Template B.
  • Compute Resources: Training and rollout experiments use nodes with 8 NVIDIA H20 GPUs, while the LLM judge uses a separate node with 8 NVIDIA H20 GPUs.The judge assigns rewards and performs test-time evaluation.
  • Different Guidance Methods: The study compares unguided rollouts with assistant-prefix guidance and user-assistant messages that reconstruct action data as tool calls and responses.These formats test different ways of injecting action guidance into LLM-based agents.
  • Action Data for On-policy Self Distillation: OPSD samples trajectories from the unguided policy but uses action-conditioned guided logits as stop-gradient distillation targets, leaving inference unguided.The guided distribution serves as a token-level teacher on on-policy rollouts.
  • Action Data for On-policy Self Distillation: OPSD can improve performance, but its gains remain limited because visited states are still determined by the base unguided policy.It therefore does not fundamentally resolve ineffective state visitation when the agent cannot reach useful states independently.

C Theoretical Analysis · C.1 Covariance Form of the Token-Level Off-Policy Risk

The analysis formulates token-level off-policy risk through the variance of cumulative log-ratio shifts. It decomposes this risk into token-wise variance and cross-token covariance, showing why autoregressive dependencies make covariance important.

  • C.1 Covariance Form of the Token-Level Off-Policy Risk: The analysis defines a token-level importance ratio for each generated token under guidance level g_k.
  • C.1 Covariance Form of the Token-Level Off-Policy Risk: It then aggregates token-level deviations through the cumulative log-ratio shift.
  • C.1 Covariance Form of the Token-Level Off-Policy Risk: The off-policy risk is defined from the cumulative log-ratio shift.
  • C.1 Covariance Form of the Token-Level Off-Policy Risk: Variance expansion separates the risk into token-wise variance terms and cross-token covariance terms.The covariance summation covers all distinct token pairs in the rollout.
  • C.1 Covariance Form of the Token-Level Off-Policy Risk: Token-wise variance captures local distribution mismatch, while covariance captures dependence among mismatches across the autoregressive trajectory.Stronger guidance can increase both token-level deviation magnitude and cross-token correlation, raising internalization risk.
  • C.1 Covariance Form of the Token-Level Off-Policy Risk: If token-level log-ratio shifts were independent, covariance terms would vanish and risk would reduce to summed token-wise variances.Autoregressive generation intrinsically creates token dependencies, so covariance generally cannot be ignored.
  • C.1 Covariance Form of the Token-Level Off-Policy Risk: The variance of the cumulative log-ratio shift therefore serves as a compact measure of off-policy risk.

C.2 Risk-Constrained View of Minimal Intervention

The paper formalizes minimal intervention as a risk-constrained selection problem: choose the weakest guidance level that meets a target recovery requirement. Under monotone recovery and risk, this minimal sufficient level is risk-optimal, and recovery probabilities can be estimated empirically from repeated rollout groups.

  • Risk-constrained selection: Minimal intervention selects a guidance level by requiring sufficient group recovery probability while minimizing off-policy risk.The selection is defined for guidance levels g_k, rollout group size N, success threshold δ, and target recovery level ρ.
  • Risk-optimality: Under monotone recovery and risk, the least guidance level satisfying the recovery requirement is an optimal solution to the risk-constrained problem.The guidance family assumes non-decreasing recovery probabilities Q_k and risks R_k.
  • Risk-optimality: Because risk is non-decreasing across guidance levels, no feasible level has lower off-policy risk than the minimal sufficient level.Thus, stronger guidance is unnecessary when a weaker level already satisfies the recovery constraint.
  • Empirical identification: When recovery probabilities are unknown, repeated rollout groups estimate each level’s recovery probability using empirical means of independent group-recovery indicators.The empirical identification result assumes monotone recovery and risk, with a positive margin between levels.

D Training Cases of ACTGUIDE-RL

This section illustrates how ACTGUIDE-RL uses complete reference action trajectories as plan-style guidance while requiring the policy to reason and act with tools. Representative cases span factual research and multi-step mathematical and sports investigations, including partial trajectories that require additional tool calls.

  • Guidance mechanism: ACTGUIDE-RL uses full guidance gK, supplying the complete available reference action trajectory while requiring reasoning and tool-based task completion.The guidance is injected as plan-style reference information rather than replacing policy reasoning or execution.
  • Training cases: Case 1 asks which political party last controlled West Berkshire Council; the ground-truth answer is the Liberal Democrats.The question requires tracing a Berkshire village, the Kennet & Avon Canal, and a tied election outcome.
  • Training cases: Case 2 presents a chess-tournament problem asking how many players participated and includes a partial trajectory of searches and a Reddit visit.The hint searches for the number of squares reachable by a knight in two moves on an infinite board.
  • Training cases: Case 3 combines a baseball-history question with a sports-identification query whose ground-truth answer is 1972, Mark DiFelice.The supplied trajectory includes searches and visits concerning the 2009 World Baseball Classic, Mark DiFelice, and related players.

E Limitations

The study is evaluated primarily in a controlled search-agent setting, while broader agent-task validation remains open. Its lightweight plan-level use of action data and the systematic collection and processing of interaction records are also limitations and opportunities for further work.

  • Experimental scope: The main experiments use a relatively simple search-agent setup, so effectiveness beyond search agents remains to be established.This setting enables controlled study of reachability barriers and guidance-induced off-policy risk.
  • Guidance formulation: Action data are injected only as high-level plan-style guidance; finer-grained step-level guidance remains unexplored.The formulation avoids costly reasoning traces while keeping the method lightweight and broadly applicable.
  • Action-data curation: The work does not specify how action data should be systematically collected, cleaned, or filtered from existing interaction records.Potential sources include backend logs from different agent applications.
Loading 2605.12004v1…