PaperlayerView exact revision on arXiv

Source-linked AI summary

Agentic Critical Training

Weize Liu, Minghui Liu, Sy-Tuyen Ho, Souradip Chakraborty, Xiyao Wang, Furong Huang

arXiv:2603.08706v1cs.AIcs.CLcs.LG

TL;DR

LLM agent training often teaches what to do without explaining why actions are preferable, and reflection-based extensions still imitate pre-generated text. ACT instead uses reinforcement learning to train agents to choose the better action among alternatives, yielding consistent benchmark gains and broader generalization. Across three agent benchmarks, it improves over imitation learning, reinforcement learning, and knowledge-distilled reflection, while also improving general reasoning without reasoning-specific training data.

  • Problem

    Imitation learning teaches agents successful behavior but not the relative quality of alternative actions, while reflection-based approaches remain imitation of pre-generated reasoning text.

  • Method

    ACT contrasts expert and model-generated actions and uses reinforcement learning with rewards based only on correct selection, encouraging autonomous reasoning about action quality.

  • Results

    Across three agent benchmarks, ACT consistently improves performance, with average gains of 5.07 points over imitation learning, 4.62 over reinforcement learning, and 2.42 over Early Experience.

  • Takeaways & Limitations

    ACT supports out-of-distribution generalization on agentic benchmarks and improves MATH-500 and GPQA-Diamond without reasoning-specific training data.

  • Takeaways & Limitations

    ACT requires collecting model-generated alternative actions, which can be expensive, and its data-transferability evaluation assumes alternatives collected from a larger model can train a smaller model without re-collection or adaptation.

Abstract

from arXiv · show

Training large language models (LLMs) as autonomous agents often begins with imitation learning, but it only teaches agents what to do without understanding why: agents never contrast successful actions against suboptimal alternatives and thus lack awareness of action quality. Recent approaches attempt to address this by introducing self-reflection supervision derived from contrasts between expert and alternative actions. However, the training paradigm fundamentally remains imitation learning: the model imitates pre-constructed reflection text rather than learning to reason autonomously. We propose Agentic Critical Training (ACT), a reinforcement learning paradigm that trains agents to identify the better action among alternatives. By rewarding whether the model's judgment is correct, ACT drives the model to autonomously develop reasoning about action quality, producing genuine self-reflection rather than imitating it. Across three challenging agent benchmarks, ACT consistently improves agent performance when combined with different post-training methods. It achieves an average improvement of 5.07 points over imitation learning and 4.62 points over reinforcement learning. Compared to approaches that inject reflection capability through knowledge distillation, ACT also demonstrates clear advantages, yielding an average improvement of 2.42 points. Moreover, ACT enables strong out-of-distribution generalization on agentic benchmarks and improves performance on general reasoning benchmarks without any reasoning-specific training data, highlighting the value of our method. These results suggest that ACT is a promising path toward developing more reflective and capable LLM agents.

1. Introduction

Imitation-based agent training teaches successful behavior but not why alternatives are worse, while ACT trains agents via reinforcement learning to judge action quality directly. Across three agent benchmarks, ACT improves performance over imitation learning, reinforcement learning, and knowledge-distilled reflection, while also generalizing beyond the training distribution.

  • Motivation: Imitation learning teaches agents to replicate successful demonstrations but provides no understanding of which alternatives are suboptimal.Early Experience adds generated reflections, but the model still imitates pre-generated text rather than autonomously developing reasoning.
  • Agentic Critical Training: ACT pairs expert actions with model-generated alternatives and uses reinforcement learning to train agents to select the better action.Only the correctness of the selection is supervised, requiring the model to develop reasoning about action quality.
  • Results: 5.07 points is ACT's average gain over imitation learning across ALFWorld, WebShop, and ScienceWorld.ACT also outperforms reinforcement learning by an average of 4.62 points and Early Experience by 2.42 points.
  • Generalization: ACT exhibits strong out-of-distribution generalization on agentic benchmarks and improves general reasoning without reasoning-specific training data.The reported reasoning benchmarks are MATH-500 and GPQA-Diamond.

2. Agentic Critical Training

Agentic Critical Training transforms agent training from imitating expert actions into judging better actions among alternatives, then uses that capability for direct action generation. Its two-stage RL pipeline constructs contrastive examples, trains critical judgment with verifiable rewards, and applies the resulting foundation to action training.

  • Motivation: IL teaches agents to replicate expert behavior but provides no signal about why one action is preferable to alternatives.ACT changes the objective from imitating the expert action to identifying the better action.
  • Training Pipeline: The pipeline first constructs contrastive examples, then trains critical judgment with GRPO before performing RL action training on expert trajectories.The stages are data construction, Agentic Critical Training, and direct action generation with the ACT-enhanced model.
  • Data Construction: ACT pairs each expert action with model-generated alternatives and trains the agent via RL to identify the better action.The approach assumes expert actions are generally superior to alternatives sampled from the initial policy.
  • Agentic Critical Training: ACT uses randomized candidate ordering and verifiable rewards, so the expert action appears in either position and correctness determines the training signal.The prompt asks the model to reason about which candidate is better and output the selected action in designated tags.
  • RL Action Training: RL Action Training samples responses for each context and rewards those whose extracted action matches the expert action.The shared reward design also gives partial credit to admissible non-expert actions and penalizes responses without proper action tags.

3. Related Work

Related work spans inference-time reflection, imitation-based agent training, critique RL, and agentic RL. ACT differs by learning self-reflection as a policy competence through RL in multi-turn agent settings rather than reproducing reflection text or focusing on single-turn tasks.

  • LLM-based Agents: ReAct interleaves reasoning and acting, while Reflexion uses verbal self-reflection at inference time to improve performance.ACT instead trains self-reflection as a learned competence through RL.
  • Training LLM Agents: Early Experience enriches demonstrations with generated reflections but still trains the model to reproduce pre-generated reflection text through supervised fine-tuning.Its training objective therefore remains imitation learning.
  • Critique RL Training: Recent critique-RL methods train reward models or policies through critique, but they focus on single-turn chat or code-generation settings.ACT is distinguished by operating in multi-turn agentic settings.
  • Agentic RL: Agentic RL addresses multi-turn, long-horizon decisions in complex environments, while RL with verifiable rewards can incentivize reasoning without supervised chain-of-thought data.This places ACT within a broader shift toward reinforcement learning for agent training.

4. Experiments

Experiments evaluate ACT across embodied, web, and scientific agent benchmarks, comparing it with imitation learning, reinforcement learning, and reflection-based training. ACT improves downstream performance, generalizes to unseen configurations and reasoning benchmarks, and supports failure recovery and cross-size data transfer.

  • Experimental Setup: The evaluation covers ALFWorld, WebShop, and ScienceWorld, using success rates or next-action prediction accuracy, with ALFWorld including ID and OOD splits.Methods are trained on the same expert trajectories, enabling comparison of training paradigms.
  • Main Results: RL consistently achieves higher success rates than IL across all three benchmarks when trained on the same expert data.This comparison isolates the training paradigm under matched expert-data conditions.
  • Main Results: 5.07 percentage points is ACT’s average improvement over IL, while 4.62 percentage points is its average improvement over RL across all benchmarks.The gains come from adding ACT before IL or RL; RL with ACT achieves the highest overall performance.
  • Main Results: 2.42 percentage points is the average advantage of IL with ACT over Early Experience across all benchmarks.Both IL with ACT and RL with ACT consistently outperform the reflection-text imitation baseline.
  • Generalization: ACT’s gain over RL is larger on OOD than ID tasks, at 3.73pp versus 2.15pp, indicating generalization to unseen task configurations.Adding ACT improves both IL and RL on ALFWorld’s OOD split.
  • Cross-Size Data Transferability: ACT data collected from Qwen3-8B remains effective for Qwen3-4B, improving both ID and OOD tasks without re-collection or adaptation.The transferred data supports amortizing alternative-action collection costs across model sizes.
  • Generalization to General Reasoning Benchmarks: ACT achieves the highest scores on MATH-500 and GPQA-Diamond despite training exclusively on ALFWorld agentic data.On GPQA-Diamond, ACT improves over the CoT baseline by 1.85pp, reaching 53.37% versus 51.52%.
  • Generalization to General Reasoning Benchmarks: ACT exhibits self-verification by substituting answer options into the original equation, while the base model does not systematically verify all options.This behavior matches ACT’s objective of evaluating and comparing candidate actions.

5. Conclusion

ACT trains LLM agents to reason about action quality by contrasting expert and self-generated actions through reinforcement learning rather than imitating reflection text. Across three benchmarks, it improves both imitation- and reinforcement-learning approaches, supports out-of-distribution generalization, and improves general reasoning without reasoning-specific data.

  • ACT contrasts expert and self-generated actions via reinforcement learning to train agents to reason about action quality.
  • Unlike next-token imitation of pre-generated self-reflection, ACT produces autonomous critical reasoning through reinforcement learning.
  • Across three agent benchmarks, ACT consistently improves both imitation learning and reinforcement learning and achieves the highest performance across all benchmarks.
  • ACT enables strong out-of-distribution generalization on agentic benchmarks.
  • ACT improves GPQA-Diamond and MATH-500 performance without reasoning-specific training data, while other training methods degrade or fail to improve reasoning.

A. Experimental Details

ACT uses a two-stage GRPO pipeline with verifiable rewards for action correctness, admissibility, and response format. The procedure first constructs alternatives from the initial policy, then trains the model on group-relative rewards before direct action training.

  • Reward Function Design: The reward design extracts an action from tagged spans, applies semantic rewards to that action, and applies the format reward to the full response.
  • Reward Function Design: Invalid or missing action tags produce zero semantic reward and only the format penalty.
  • Reward Function Design: The composite reward includes exact expert-action accuracy, partial credit for valid admissible actions, and a format component.
  • Reward Function Design: For WebShop reinforcement-learning action training, admissible-action reward is disabled because open-ended search queries cannot be enumerated.
  • GRPO Algorithm: GRPO samples groups of responses, computes rewards and group-relative advantages, and updates the policy with policy optimization and KL regularization.
  • Training Algorithm: ACT data collection samples alternatives from the initial policy, followed by GRPO training with verifiable rewards based on action correctness.

A.2. Implementation Details

The implementation uses OpenRLHF and DeepSpeed ZeRO-3 for GRPO training on four NVIDIA GH200 GPUs, with experiment settings specified in Table 4.

  • GRPO training uses OpenRLHF with DeepSpeed ZeRO-3 for memory efficiency.
  • Training uses 4 NVIDIA GH200 GPUs, and Table 4 lists the experimental hyperparameters.

A.3. Data Statistics

The experiments use state-action pairs from expert trajectories across ALFWorld, WebShop, and ScienceWorld, with shared training data across compared methods. ScienceWorld uses a sampled subset because of its large action space and resource constraints, and the benchmarks have separate ACT and RL prompts.

  • Data Statistics: Each training sample is a state-action pair from a successful trajectory, and all compared methods use the same pairs for fair comparison.
  • Data Statistics: ScienceWorld training randomly samples 10,240 state-action pairs from expert trajectories because of its large action space and resource constraints.
  • Data Statistics: Table 5 reports dataset statistics for training and distinguishes in-distribution from out-of-distribution evaluation.
  • Data Statistics: Expert trajectories come from model runs for ALFWorld, official human demonstrations for WebShop, and official gold trajectories for ScienceWorld.
  • Prompts: ALFWorld, WebShop, and ScienceWorld each provide separate ACT prompts that select between two candidate actions and RL prompts that generate the next action.

B. Additional Case Study: Agentic Task Performance

An additional WebShop case study shows that imitation learning can fail through rigid execution without state awareness.

  • The WebShop case study illustrates a failure mode of imitation learning: rigid execution without state awareness.

B.1. Rigid Execution Without State Awareness

Imitation learning follows fixed action scripts without checking whether the current environment state still satisfies the task constraints. ACT instead evaluates candidate actions against the current state, enabling recovery when a product page is unsuitable.

  • IL models replicate expert trajectories as fixed action sequences and cannot detect or adjust when the environment state deviates from training.
  • ACT evaluates candidate actions against the current state, allowing an agent to recognize unmet product constraints and search again.
  • The WebShop request requires men’s shirts with specified washing, material, and attribute constraints priced below $50.
  • The IL model executes search →click →buy without checking the item’s price, purchasing at $55 despite the $50 budget.

C.1. IL Causes Reasoning Collapse

The case studies describe reasoning collapse in IL after agentic fine-tuning, including unfocused meandering and prolonged algebraic loops. ACT instead preserves deep reasoning by optimizing outcome correctness through reinforcement learning rather than imitating behavioral patterns.

  • C.1. IL Causes Reasoning Collapse: IL fine-tuning on short, action-heavy trajectories can overwrite the original model’s deep reasoning capabilities, a phenomenon termed reasoning collapse.
  • Unfocused meandering: ACT produces a focused 10,669-character derivation and the correct answer, whereas IL generates a 37,924-character trace that is 3.5× longer and incorrect.
  • Unfocused meandering: The high-energy physics case concerns the threshold energy for γ-ray annihilation with CMB photons into electron-positron pairs.
  • Algebraic death loops: IL enters repetitive algebraic loops exceeding 80,000 characters and produces an incorrect answer, while ACT recognizes the geometric structure and derives the result cleanly.
  • Algebraic death loops: The algebraic-loop example is a MATH-500 rectangle-probability problem involving the probability that a point is closer to the center than to any vertex.
  • C.1. IL Causes Reasoning Collapse: ACT avoids reasoning collapse by optimizing outcome correctness through reinforcement learning rather than imitating behavioral patterns.
Loading 2603.08706v1…