PaperlayerView exact revision on arXiv

Source-linked AI summary

CAPO: Critic-Guided Action-Aligned Policy Optimization for Advancing LLM Agent Capabilities

Daoyu Wang, Qingchuan Li, Mingyue Cheng, Jie Ouyang, Shuo Yu, Chunli Liu, Shijin Wang, Qi Liu, Enhong Chen

arXiv:2604.18401v5cs.CL

TL;DR

Multi-turn RL can assign the same advantage to every token in a trajectory, limiting credit assignment across decisions. CAPO aligns credit assignment and policy updates with complete actions, and consistently outperforms representative RL methods across multi-hop QA, academic paper search, and text-world tasks.

  • Problem

    Many multi-turn RL methods share one advantage across all trajectory tokens, providing coarse credit assignment despite agents making distinct decisions across actions.

  • Method

    CAPO estimates action-boundary state values for complete-action advantages and uses a length-calibrated action-aware policy ratio while retaining token-level gradients.

  • Results

    CAPO achieves the best results across all metrics for both backbone models on multi-hop QA, academic paper search, and text-world action tasks.

  • Takeaways & Limitations

    The results support complete actions as optimization units for training capable LLM agents in multi-turn reinforcement learning.

  • Takeaways & Limitations

    Unlike critic-free group-based methods, CAPO requires critic training, trading additional critic modeling for finer-grained credit assignment and improved performance.

Abstract

from arXiv · show

Reinforcement learning (RL) has become a key technique for improving the agentic capabilities of large language models (LLMs). Although critic-free methods such as GRPO are increasingly popular, we argue that critic-based methods remain well suited to long-horizon agentic tasks because their critic models can assess each state and assign credit to different decisions. However, representative critic-based methods such as PPO still organize value estimation, credit assignment, and policy updates at token granularity, whereas the environment state changes only after the agent completes a full action. In this work, we propose CAPO, a critic-guided action-aligned policy optimization method for advancing LLM agent capabilities. CAPO introduces two action-aligned designs. First, it estimates the state value at the action boundary using a critic model and assigns the corresponding advantage to the complete action rather than individual tokens. Second, CAPO introduces a novel action-aware policy ratio that aggregates token ratios within each action with length calibration for consistent updates across actions of different lengths. These two designs effectively achieve fine-grained credit assignment and consistent policy updates while remaining compatible with token-level gradient computation. Experiments across multi-hop QA, academic paper search, and text-world action tasks show that CAPO consistently outperforms representative RL methods. Further analyses validate both designs and reveal how action-aligned optimization improves multi-turn RL training. We hope this work provides a practical path for training more capable LLM agents.

1 Introduction

CAPO addresses the mismatch between action-level environment transitions and token-level policy optimization in multi-turn LLM agents. It aligns credit assignment and policy updates with complete actions while retaining token-level gradient computation, and shows consistent improvements across diverse agent tasks.

  • Motivation: LLM agents interact with environments through complete actions, but representative critic-based methods such as PPO estimate values, assign credit, and update policies at token granularity.This creates a fundamental granularity mismatch between the agent’s decision unit and optimization unit.
  • Motivation: Critic-free methods such as GRPO are popular for simplicity and scalability, but extending them to multi-turn training creates limitations because all trajectory tokens shareThe supplied passage ends mid-sentence after describing this limitation.
  • Method: CAPO formulates each complete agent action as the decision unit and introduces action-level credit assignment alongside action-aware policy updates.A critic estimates state values at action boundaries and assigns the corresponding advantages to complete actions rather than individual tokens.
  • Method: CAPO aligns policy optimization with action granularity while remaining compatible with token-level gradient computation.The design follows the principle that actions determine environment transitions, whereas tokens are underlying generation units.
  • Evaluation: CAPO consistently outperforms representative RL algorithms across multi-hop QA, academic paper search, and text-world action tasks.Ablation and sensitivity analyses validate both action-level credit assignment and action-aware policy updates.

2 Related Work

Prior RL work for LLMs increasingly favors scalable critic-free methods, while multi-turn settings expose challenges in credit assignment and policy updates. These developments motivate aligning both optimization components with complete agent actions.

  • Overview: Related work examines how RL methods choose the granularity of credit signals and policy updates, motivating alignment with complete agent actions.The section focuses on RL for LLMs and its extension to multi-turn agent tasks.
  • Single-turn RL for LLMs: Critic-free methods such as GRPO are increasingly favored for their simple, scalable formulation and relative advantages from same-prompt response comparisons.They avoid training a separate critic and support efficient optimization for long-form reasoning, but typically provide coarse credit signals.
  • Multi-turn RL: Multi-turn RL optimizes long-horizon LLM–environment trajectories, motivating designs for sparse-reward credit assignment and stable policy updates.Methods including GiGPO, ST-PPO, and Turn-PPO explore finer-grained credit assignment in multi-turn settings.

3 Methodology

CAPO aligns critic-based policy optimization with the environment’s action granularity by assigning credit to complete actions and using a length-calibrated action-aware policy ratio. It retains token-level score gradients while applying shared action-level weighting and clipping.

  • Action-aligned decision granularity: CAPO treats each complete response as the decision unit because environment transitions occur only after action completion.The policy autoregressively generates reasoning and tool-call tokens, then receives the next observation after the complete action.
  • Action-aligned optimization objective: CAPO combines action-level advantages with action-aware ratios so tokens share one advantage, ratio, and clipping decision while retaining token-level gradient computation.The L_t^-1/2 calibration prevents longer actions from dominating solely through token count, and the ratio is a surrogate rather than an exact action likelihood ratio.
  • Action-level credit assignment: CAPO estimates the pre-action state value with a critic and assigns the resulting GAE advantage to every token in the complete action.The action-level advantage is computed along the trajectory timeline, including bootstrapped value differences when rewards are terminal-only.
  • Action-aware policy ratio: Token-wise weighting and clipping remain misaligned because all tokens share an action advantage while PPO- and GRPO-style objectives operate on individual tokens.Exact action likelihood ratios are sensitive to action length, whereas geometric-mean aggregation can reduce clipping for longer actions.
  • Action-aware policy ratio: α = 0.5 yields more comparable out-of-range and active clipping fractions across action lengths than α = 0 or α = 1.For α = 0, both fractions increase with length; for α = 1, both decrease, indicating length-dependent clipping bias.
  • Action-aware policy ratio: CAPO uses square-root scaling over centered token log-ratios to produce length-calibrated weighting and clipping across actions of different lengths.Under the i.i.d. variance approximation, normalization gives variance σ2 independent of action length, while centering removes the length-dependent mean shift.

4 Experiments

CAPO achieves the best results across the evaluated multi-hop QA, academic paper search, and text-world tasks on both backbone models, with especially strong gains on long-horizon agent tasks. Ablation, training-dynamics, policy-update, behavior, and cost analyses further examine why CAPO improves credit assignment, optimization stability, and action efficiency.

  • Overall Results: CAPO achieves the best results on all reported metrics for both backbone models, with consistent gains on in-domain and out-of-domain multi-hop QA.The improvements are more pronounced on agent tasks, where effective credit assignment over interaction steps is important.
  • Ablation Analysis: Credit assignment contributes more on interaction-intensive RealResearchQuery, ALFWorld, and WebShop, while policy updates matter relatively more on HotpotQA.On HotpotQA, removing both designs causes a larger drop than the sum of their individual drops, indicating that the designs reinforce each other.
  • Training Dynamics: CAPO attains the highest rewards for most training and maintains more controlled critic loss than PPO, whose ALFWorld critic loss becomes highly unstable.CAPO reaches reward above 0.5 by roughly 100 HotpotQA updates and is the only method whose ALFWorld reward stays near 0.9 in the final segment, while PPO remains below 0.6.
  • Policy Updates: Length-scaled policy-ratio aggregation improves both tasks, with square-root aggregation outperforming mean aggregation and product aggregation underperforming token-wise ratios.The action-aware ratio improves ALFWorld Seen from 90.71 to 92.86 and WebShop from 76.18 to 78.12 over square-root aggregation, while remaining length-stable.
  • Trajectory Efficiency: CAPO matches or exceeds PPO and GRPO across task horizons and produces smaller gaps from golden action sequences, indicating more efficient trajectories on longer tasks.Across every group from three to nine golden steps, CAPO has the lowest reported step gap.
  • Behavior and Cost: CAPO obtains the highest F1@all in academic paper search while using the fewest action tokens and the most citation/reference expansion calls among compared methods.CAPO reaches 0.318 F1@all versus the next-best value of 0.306; its measured WebShop iteration time is 189.54 seconds versus 195.37 seconds for PPO.

5 Conclusion

CAPO is a critic-guided, action-aligned policy optimization method that addresses the mismatch between agent decisions and token-level optimization. It combines action-boundary credit assignment with length-calibrated, action-aware policy updates while retaining token-level gradient computation.

  • Conclusion: CAPO advances LLM agent capabilities through critic-guided, action-aligned policy optimization.The method is designed for LLM agent capability advancement.
  • Conclusion: CAPO assigns credit at action boundaries using state value estimation rather than token-level decisions.This design addresses the granularity mismatch between agent decisions and token-level optimization.
  • Conclusion: CAPO performs action-aware policy updates with a length-calibrated policy ratio.The ratio supports consistent updates across actions while preserving token-level gradient computation.
  • Conclusion: Together, CAPO’s designs provide fine-grained credit assignment and consistent policy updates while retaining token-level gradient computation.The approach was evaluated across multi-hop QA, academic paper search, and text-world action tasks.

A Theoretical Analysis · A.1 Detailed Step-Level MDP Transitions · A.2 Action-Level Value and Advantage Estimation

CAPO models agent–environment interaction as a step-level MDP in which complete actions, rather than individual tokens, trigger transitions and receive rewards. It therefore estimates values and advantages at action boundaries, assigning each complete action’s advantage across its valid tokens during optimization.

  • A Theoretical Analysis: CAPO models agent–environment interaction as a step-level MDP MStep = (S, A, PStep, R, γ).
  • A.1 Detailed Step-Level MDP Transitions: The complete environment-facing action at step t is autoregressively factorized into tokens, which are decoded internally without independently changing the environment state.
  • A.1 Detailed Step-Level MDP Transitions: The environment returns reward r_t and observation o_t+1 only after executing the complete action, from which the next state is constructed.
  • A.1 Detailed Step-Level MDP Transitions: Thus, environment dynamics advance once per complete action rather than once per generated token.
  • A.2 Action-Level Value and Advantage Estimation: Because transitions occur after complete actions, CAPO defines action values over complete interaction steps and state values before an action is specified.
  • A.2 Action-Level Value and Advantage Estimation: The population action-level advantage compares the realized complete action with its pre-action expected return.
  • A.2 Action-Level Value and Advantage Estimation: CAPO places value estimation at the decision boundary, using the final state-token representation conditioned on the complete state prefix but not the action.
  • A.2 Action-Level Value and Advantage Estimation: The resulting action-level advantage is assigned to the complete action and shared by its valid action tokens during policy optimization.

A.3 Analysis of Action-Aligned Policy Optimization · B Detailed Experimental Settings

CAPO analyzes action-aware policy-ratio aggregation, batch-mean centering, and length-calibrated gradients as modifications to PPO’s update granularity. The analysis also states that the surrogate is not an unbiased importance-sampling correction and lacks a directly applicable TRPO-style guarantee.

  • A.3 Analysis of Action-Aligned Policy Optimization: Product aggregation is the exact complete-action likelihood ratio, but its log-ratio becomes increasingly sensitive to action length.Geometric-mean aggregation controls accumulation more strongly but may reduce clipping frequency for long actions, whereas square-root aggregation moderates length dependence while retaining within-action accumulation.
  • A.3 Analysis of Action-Aligned Policy Optimization: CAPO centers token log-ratios before square-root aggregation and restores their overall center to calibrate action-aware ratios.The batch mean is token-weighted, removes the shared policy shift before length scaling, and preserves the overall ratio center.
  • A.3 Analysis of Action-Aligned Policy Optimization: At the rollout policy, all token log-ratios and the batch mean are zero, so the action-aware ratio equals 1.The batch mean is an empirical calibration statistic that can vary with optimization-set composition rather than an additional learned parameter.
  • A.3 Analysis of Action-Aligned Policy Optimization: Stop-gradient lets the batch mean affect the forward ratio and clipping decision without sending gradients across steps through the shared statistic.This prevents one step from receiving gradient contributions from other steps through the batch mean.
  • A.3 Analysis of Action-Aligned Policy Optimization: CAPO combines token score gradients within each action under one action advantage, action ratio, and clipping decision, with L_t^-1/2 calibrating magnitude across lengths.This retains PPO’s clipped-surrogate structure while changing the update granularity from individual tokens to complete actions.
  • A.3 Analysis of Action-Aligned Policy Optimization: CAPO constructs one ratio and one clipping decision for each complete action, unlike PPO’s token-wise weighting and clipping; when L_t = 1, the ratio is standard.Near the rollout policy, the per-action gradient follows the summed action-score direction with length calibration.
  • A.3 Analysis of Action-Aligned Policy Optimization: CAPO’s length-calibrated surrogate is not an unbiased importance-sampling correction, so a TRPO-style guarantee does not directly apply.The analysis instead characterizes changes to PPO’s scale, gradient weighting, and clipping granularity for complete-action alignment.

B.1 Datasets Preparation · B.2 RL Environment Construction · B.3 Baselines

The paper evaluates agent learning across multi-hop QA, academic paper search, ALFWorld, and WebShop, using task-specific datasets and interactive environments. It compares CAPO with prompting, critic-based, and critic-free RL baselines operating at token, sequence, trajectory, step, or turn granularity.

  • B.1 Datasets Preparation: HotpotQA training uses 90,447 questions, with 7,405 development questions for in-domain evaluation and 12,576 2Wiki plus 2,417 MuSiQue questions for cross-dataset testing.Each question is paired with 10 Wikipedia context paragraphs.
  • B.1 Datasets Preparation: Academic paper search trains on 33,551 RealResearchQuery queries and tests on 50, scoring discovered papers with pasa-7b-selector against query-level relevance annotations.Reported metrics are post-threshold F1@all and Recall@all against annotated arXiv IDs.
  • B.1 Datasets Preparation: ALFWorld uses 3,553 training, 140 valid-seen, and 134 valid-unseen solvable tasks spanning six household task families.Episodes provide goal instructions and evaluate whether the agent completes the specified household task.
  • B.1 Datasets Preparation: WebShop uses approximately 1.18 million products and 12,087 shopping goals, split into 11,587 training and 500 development goals.Its reward combines task satisfaction and required product-attribute matching with an intermediate formatting component.
  • B.2 RL Environment Construction: The multi-hop QA environment retrieves from benchmark-specific indexes containing 509,308 HotpotQA, 5,902,082 2Wiki, and 139,416 MuSiQue passages, encoded with BAAI/bge-large-en-v1.5 and indexed by FAISS.The environment exposes retrieved passages and prior search information at each step.
  • B.2 RL Environment Construction: The paper-search environment maintains query-specific paper pools and exposes search(query) and expand(paper_id), using approximately 3 million arXiv papers and 30 million citation edges.The corpus comes from a January 2026 Semantic Scholar snapshot restricted to papers with abstracts.
  • B.2 RL Environment Construction: ALFWorld runs packaged TextWorld games as independent interaction instances, returning textual observations and admissible commands while requiring exactly one valid command per step.Supported commands include navigation and household operations.

B.4 Training Settings · C Prompt Templates · D Representative Trajectory

The appendix specifies CAPO’s training implementation and comparison settings, details rollout-time prompt construction for tool-calling tasks, and presents an action-level academic paper search trajectory. These materials connect the experimental infrastructure and prompting interface to complete environment-facing actions.

  • B.4 Training Settings: B.4 Training Settings uses Agent-R1 with veRL and vLLM on a server equipped with 8 NVIDIA H100 GPUs.These components define the training framework, reinforcement-learning backend, generation system, and hardware setup.
  • B.4 Training Settings: The default actor learning rate is 1 × 10−6, while critic-based methods use a critic learning rate of 1 × 10−5.The passage also specifies a training batch size of 128 and an actor micro-batch size of 4 per GPU.
  • B.4 Training Settings: Group-based baselines use 8 rollouts per prompt and batch size 16, producing an effective batch size of 128 for fair comparison.This setting aligns the effective batch size with the default training batch size.
  • B.4 Training Settings: Policy clipping uses ϵ = 0.001 for CAPO and ϵ = 0.0003 for GSPO, while remaining baselines follow their original papers’ configurations.The passage explicitly distinguishes clipping choices for CAPO, GSPO, and other baselines.
  • C Prompt Templates: C Prompt Templates populate each dataset row with task inputs, including HotpotQA questions, paper-search queries, ALFWorld goals, and WebShop instructions.The stored prompt field contains the task input used to instantiate the tool-calling interface.
  • C Prompt Templates: During rollout, prompts combine observations or retrieved evidence, action history, available tools or admissible actions, and the expected output format.The construction preserves {observation} and {history_actions} as rollout-time placeholders and represents each response as one environment-facing action.
  • D Representative Trajectory: D Representative Trajectory presents an abbreviated academic paper search rollout in which the agent expands a paper pool through search and citation/reference expansion.The example preserves the pattern of selecting complete actions at each interaction step while abbreviating long observations, paper lists, and repetitive parallel tool calls.

E Reproducibility Details

The experiments average RL results across three random seeds and specify deterministic ReAct evaluation, hyperparameter comparisons, and a detailed hardware and software environment. These details support reproducibility across the reported experiments.

  • Evaluation protocol: RL results are averaged over random seeds 41, 42, and 43, while ReAct uses one deterministic evaluation at temperature 0.0.ReAct is evaluated with deterministic decoding.
  • Hyperparameter comparisons: Hyperparameter comparisons cover γ ∈{0.95, 0.99, 1.00} and the policy-ratio variants reported in the policy ratio analyses.The comparisons include both discount-factor settings and policy-ratio variants.
  • Computing environment: Experiments run on eight NVIDIA H100 80GB HBM3 GPUs with dual Intel Xeon Platinum 8468 CPUs under Ubuntu 22.04.5 LTS.The computational setup is specified at the server, processor, accelerator, and operating-system levels.
  • Software environment: The software stack uses Python 3.10.20, CUDA 12.8, PyTorch 2.8.0+cu128, Transformers 4.57.6, and veRL 0.7.0.These versions identify the principal runtime, acceleration, modeling, and RL-training components.

F Limitations … Paper Search Selector Prompt

The paper identifies computational and scope limitations for CAPO, then specifies agent prompts for Wikipedia search, paper retrieval and expansion, and paper-selection decisions. These prompts require explicit analysis, structured tool calls, and query diversification where applicable.

  • F Limitations: CAPO has comparable per-iteration training time to PPO but still requires critic inference and updates, unlike critic-free methods such as GRPO.Extending action-aligned optimization to critic-free RL is identified as future work.
  • F Limitations: The study mainly evaluates single ReAct-style agents and does not explore more complex multi-agent or hierarchical systems.The passage raises whether action-aligned optimization extends to these settings.
  • HotpotQA Prompt: The HotpotQA prompt directs a research agent to analyze the user query, action history, retrieved passages, and tool or format issues before choosing actions.The analysis and decision logic must be enclosed in <analysis>...</analysis> tags.
  • HotpotQA Prompt: The HotpotQA search tool retrieves Wikipedia passages relevant to the user question.It accepts one natural-language or keyword query that should differ from prior history queries when possible.
  • HotpotQA Prompt: The HotpotQA search function requires a single string query and mandates that it differ from all history queries when seeking new evidence.The query is the function's required parameter.
  • Paper Search Prompt: The paper-search prompt asks the agent to analyze the user query, action history, and paper list before selecting the next actions.Its analysis and decision logic must be enclosed in <analysis>...</analysis> tags, and parallel tool calls are supported.
  • Paper Search Prompt: The paper-search tools support hybrid retrieval with a single query and expansion from an existing paper by merging its citations and references.Search queries must differ from all history queries, while expansion requires the identifier of a paper already in the current list.
  • Paper Search Selector Prompt: The paper-search selector asks an elite AI researcher to decide whether a searched paper fully satisfies the user query's detailed requirements and provide consistent reasoning.The input includes the paper title and abstract, and the output format is Decision: True/False followed by a reason.

Paper Search Trajectory · ALFWorld Prompt · WebShop Prompt

The paper-search trajectory progressively broadens and diversifies a crypto-based private-learning literature pool while avoiding redundant searches and expansions. The ALFWorld and WebShop prompts constrain agents to execute exactly one admissible environment action per turn through env_step, with task-specific action and purchase rules.

  • Paper Search Trajectory: The search begins with parallel coverage of homomorphic encryption, federated learning, secure computation, and differential privacy for a broad seed pool.The initial pool is empty, and no prior queries or expanded papers need to be avoided.
  • Paper Search Trajectory: Seed papers on encrypted inference, private training, federated learning, and privacy-preserving neural networks are then expanded without repeating the four initial queries.The expansion targets papers identified as directly matching the query, including homomorphic-encryption and encrypted-inference work.
  • Paper Search Trajectory: Subsequent expansion diversifies the graph toward FHE inference and training, transfer learning, transformers, activation approximations, secure pipelines, and systems.The trajectory explicitly prioritizes new papers while skipping papers expanded in previous steps.
  • Paper Search Trajectory: Later targeted citation and reference expansion adds CryptoDL, THE-X, MedBlindTuner, CURE, Pencil, SPEED, surveys, split learning, graph learning, recurrent models, and hybrid cryptographic protocols.The final step improves recall by covering remaining subtopics while avoiding repeated searches and already-expanded papers.
  • ALFWorld Prompt: The ALFWorld prompt requires exactly one command from the currently admissible commands each turn, issued through env_step after brief reasoning in <think>...</think> tags.The official observation is the source of truth, and the agent must not output explanations or a final natural-language answer.
  • ALFWorld Prompt: ALFWorld env_step accepts a single string command that must exactly match one currently admissible ALFWorld TextWorld command.Examples include ‘go to dresser 1‘, ‘open cabinet 3‘, ‘take mug 1 from cabinet 3‘, and ‘use desklamp 1‘.
  • WebShop Prompt: The WebShop prompt requires exactly one executable action per turn through env_step to find and buy the product that best satisfies the shopping instruction.Search queries use concise product keywords, and available actions must be followed exactly except for replacing <your query> in search[<your query>].
  • WebShop Prompt: WebShop permits product, option, information-page, navigation, or Buy Now actions only when listed, and purchasing is allowed only after the product and options satisfy the instruction.Example actions include search[wireless headphones] and click[Buy Now].
Loading 2604.18401v5…