PaperlayerView exact revision on arXiv

Source-linked AI summary

Agentic Critical Training

Weize Liu, Minghui Liu, Sy-Tuyen Ho, Yongkyun Lee, Andrew Adams Schoen, Souradip Chakraborty, Xiyao Wang, Furong Huang

arXiv:2603.08706v2cs.AIcs.CLcs.LG

TL;DR

Imitation learning identifies expert actions but not which plausible alternatives to reject, a gap that matters because one incorrect action can derail an agent's trajectory. Agentic Critical Training (ACT) uses reinforcement learning to judge expert actions against sampled alternatives without reference rationales. Across two model families and three agent benchmarks, ACT improves downstream pipelines over no-ACT counterparts and outperforms supervised reflection baselines.

  • Problem

    Imitation learning identifies expert actions but not rejected alternatives, a gap that matters because one plausible but incorrect action can derail an agent's trajectory.

    Source: §1 · pp. 1–2

  • Method

    ACT uses reinforcement learning to choose between an expert action and a sampled alternative, rewarding the choice without requiring a reference rationale.

    Source: §1 · pp. 2–3

  • Results

    Across two model families and three agent benchmarks, ACT improves both downstream pipelines over no-ACT counterparts and outperforms supervised reflection baselines.

    Source: §1 · p. 3, §5 · p. 11

  • Takeaways & Limitations

    The standalone ACT-stage checkpoint also achieves the highest reported mean on MATH-500 and GPQA-Diamond, extending the reported results beyond agent benchmarks.

    Source: §1 · p. 3

    ACT assumes that alternatives sampled from the initial policy are, on average, inferior to expert actions.

    Source: §2.2 · p. 5

Abstract

from arXiv · show

Imitation learning (IL) teaches language-model agents to reproduce expert actions but not to distinguish them from plausible mistakes. Self-reflection methods expose models to alternatives yet use supervised fine-tuning (SFT) to imitate fixed rationales and actions. We introduce Agentic Critical Training (ACT), which uses reinforcement learning with verifiable rewards (RLVR) to train models to judge actions directly. At each expert-trajectory state, ACT pairs an expert action with an alternative sampled from the initial policy and randomizes their order. The model generates its own reasoning but is rewarded only for selecting the expert action. ACT reuses demonstrations, requires no reference rationales, and allows pair reuse across model sizes. ACT is a warm-up before IL, optionally followed by RL; inference requires no candidate comparison. Across Qwen3-8B and Olmo-3-7B-Instruct on ALFWorld-ID, WebShop, and ScienceWorld, ACT yields average gains of 5.85 points over IL and 4.12 points over IL$\to$RL without ACT, while also improving ALFWorld-OOD. With Olmo on ScienceWorld, the full pipeline gains 15.36 points over CoT prompting and 9.23 points over IL$\to$RL without ACT. Both ACT$\to$IL and the full pipeline outperform supervised reflection baselines. Controls with fixed pairs or matched training durations show that the gains stem from the ACT objective rather than additional data or training. Without reasoning-specific post-training, the standalone ACT checkpoint achieves the highest mean among evaluated models on MATH-500 and GPQA-Diamond, showing that action comparison complements generation.

1. Introduction

Agentic Critical Training (ACT) uses reinforcement learning to teach agents to judge expert actions against plausible alternatives, rather than merely imitate actions or fixed rationales. It serves as a warm-up before policy training, and the resulting policy acts directly at inference without candidate comparison.

Motivation

  • Unlike imitation learning, which does not teach rejection of competing actions, Early Experience uses SFT to reproduce a fixed rationale and expert action rather than train judgment directly.ACT targets that judgment itself.Source: §1 · p. 2

Method

  • ACT pairs each expert action with an initial-policy alternative, randomizes their order, and rewards selecting the expert action without evaluating reasoning or requiring a reference rationale.The model generates its own reasoning; alternatives matching the expert action are discarded, and the expert action is treated as preferred.Source: §1 · pp. 2–3

Generalization

  • Pairs collected from Qwen3-8B can train Qwen3-4B, while the standalone ACT-stage checkpoint achieves the highest reported mean on MATH-500 and GPQA-Diamond without reasoning-specific post-training data.The paper also reports improved agentic OOD performance and causal interventions connecting reasoning to action selection.Source: §1 · p. 3

Training and inference

  • ACT is a warm-up before IL, with optional RL continuing from the IL checkpoint, and inference uses the resulting policy without candidate pairs.The paper calls the sequence ACT→IL→RL the full ACT pipeline and IL→RL w/o ACT its corresponding baseline.Source: §1 · p. 3

Results

  • ACT warm-up improves IL and IL→RL across Qwen3-8B and Olmo-3-7B-Instruct on ALFWorld, WebShop, and ScienceWorld, including ALFWorld-OOD.ACT→IL and the full pipeline also outperform supervised reflection baselines; same-pair and matched-continuation controls attribute gains to the ACT objective rather than data or added training alone.Source: §1 · p. 3

2. Agentic Critical Training

Agentic Critical Training (ACT) shifts agent training from imitating expert actions to judging which of two actions is better. It uses that judgment as a training-time warm-up before direct action generation.

2.1 Problem formulation

  • IL teaches agents to reproduce expert actions but provides no signal about why an expert action is better than alternatives.ACT addresses this gap by training agents to develop a discriminative understanding of action quality.Source: §2.1 · p. 4

2.2 Data construction

  • ACT pairs each expert action with a sampled alternative, assuming the initial policy’s actions are generally inferior to expert actions.The authors report that most sampled alternatives differ from the expert action and are identifiable as suboptimal in context.Source: §2.2 · pp. 4–5

2.3 Training pipeline

  • ACT uses GRPO to train action selection with a verifiable reward, without reference rationales, and randomizes candidate order.The reward gives 1 for selecting the expert action, 0.1 for another admissible action, and −0.5 for missing the required action tags. ACT is followed by IL, with optional RL; at test time, the policy generates actions directly without candidate pairs.Source: §2.3 · pp. 5–6

3. Related work

Related work often trains agents by imitating demonstrations or fixed reflection targets, while ACT uses reinforcement learning to select expert actions without prescribing rationales. It complements agentic RL by teaching action discrimination before policy training.

LLM-based Agents

Training LLM Agents

  • ACT replaces imitation of fixed reflection targets with reward-based selection between expert and model-generated actions (Early Experience; SAND; Agent-R).Early Experience augments demonstrations with alternative-action outcomes; SAND critiques alternatives, and Agent-R builds revision trajectories, but these methods imitate constructed text targets.Source: §3 · p. 6

Critique RL Training

  • Unlike RL critics focused on standalone outputs, ACT distinguishes expert from suboptimal actions within a multi-turn decision process.Prior critique-RL work trains reward models (Chen et al., 2026, Zhang et al., 2026) or improves policies through critique generation (Ruan et al., 2026, Wang et al., 2025).Source: §3 · p. 6

Agentic RL

  • ACT adds a warm-up objective to agentic RL: discriminating expert from suboptimal actions before policy training.Agentic RL extends post-training to long-horizon interaction, and RL with verifiable rewards can elicit reasoning without supervised chains of thought (Guo et al., 2025).Source: §3 · p. 6

4. Experiments

The experiments compare prompting, ACT, IL, ACT→IL, IL→RL without ACT, and the full ACT→IL→RL pipeline using shared expert trajectories across ALFWorld, WebShop, and ScienceWorld.

4.2 Main results

  • ACT→IL and the full pipeline outperformed all three supervised reflection baselines in every evaluation setting across both model families.On ALFWorld-OOD, ACT→IL improved Qwen3 from 79.85 to 83.58 and Olmo from 56.97 to 67.41; the full pipeline improved over IL→RL w/o ACT from 90.80 to 91.54 on Qwen3 and from 77.11 to 83.83 on Olmo.Source: §4.2 · pp. 7–8
  • ACT→IL→RL improved over IL by 6.32 average points on Qwen3 and 5.38 on Olmo, and over IL→RL w/o ACT by 3.10 and 5.14, respectively; it led all four settings.The four settings comprise ALFWorld-ID, ALFWorld-OOD, WebShop, and ScienceWorld.Source: §4.2 · p. 7, Table 1 · p. 8, Table 2 · p. 8

4.3 Training-objective and continuation controls

  • Under matched training budgets, ACT→IL beat IL→IL on all four settings, and the full pipeline beat IL→RL→RL on all four.ACT→IL's gains over IL→IL were 5.00 points on ALFWorld-ID, 4.47 on ALFWorld-OOD, 0.60 on WebShop, and 10.84 on ScienceWorld; full ACT's gains over IL→RL→RL were 6.19, 1.74, 9.40, and 14.49 points, respectively. Additional IL barely changed ScienceWorld (21.94 to 22.01), while ACT→IL reached 32.85; additional RL reduced means in all four settings. In objective controls, full ACT led on ALFWorld-ID, WebShop, and ScienceWorld, while SFT-on-choice led on ALFWorld-OOD.Source: §4.3 · pp. 8–9, Table 3 · p. 9

4.4 Causal rationale intervention

  • On 400 held-out ALFWorld pairs, ACT retained 50.75% accuracy under an incorrect rationale versus 2.25% for the original model, and removing rationale lowered ACT accuracy from 86.50% to 80.75%.The model followed the injected action at a 28.25% rate versus 81.75% for the original model. Case studies show full ACT self-correcting after failed ALFWorld actions, while an IL WebShop trajectory bought without checking a violated price constraint.Source: §4.4 · pp. 9–10, Figure 3 · p. 10

4.5 Generalization to general reasoning benchmarks

  • After training exclusively on ALFWorld agent data, standalone ACT achieved the highest tested means on MATH-500 (87.73±0.19%) and GPQA-Diamond (53.37±0.63%).IL and supervised reflection did not consistently improve general reasoning; IL scored 44.61% versus 51.52% for CoT prompting on GPQA-Diamond, while SAND scored 81.00±0.86% on MATH-500 and Agent-R scored 86.53±0.81% on MATH-500 and 52.86±1.04% on GPQA-Diamond. IL→RL w/o ACT roughly preserved the original model's performance on both benchmarks.Source: Figure 4 · p. 10, §4.5 · pp. 10–11

5. Conclusion · A. Experimental details · A.1 Training formulation and algorithm

ACT trains agents to judge actions by contrasting expert and model-generated actions, using verifiable rewards without prescribing reference rationales. The conclusion reports gains across agent benchmarks and reasoning evaluations; the appendix details reward design and GRPO training.

5. Conclusion

  • ACT contrasts expert and model-generated actions, evaluates the selected action without a reference rationale, and serves as a warm-up before IL with optional downstream RL.The conclusion describes the ACT stage followed by IL, with RL optionally continuing from the resulting IL checkpoint.Source: §5 · p. 11
  • ACT improves both downstream pipelines, and the full pipeline achieves the highest mean in every evaluation setting while outperforming supervised reflection baselines.Same-action-pair and compute-matched controls attribute gains to the training objective rather than additional optimization alone.Source: §5 · p. 11
  • The standalone ACT checkpoint achieves the highest mean on MATH-500 and GPQA-Diamond, and improvements also extend to unseen agent tasks.The checkpoint received only agentic training.Source: §5 · p. 11

A.1 Training formulation and algorithm

  • ACT extracts the tagged action from each response and combines action-accuracy, admissibility, and format rewards; invalid action tags receive no semantic reward and incur a format penalty.Semantic rewards apply to the extracted action, whereas the format reward applies to the full response.Source: §A.1 · p. 14
  • The accuracy reward requires an exact match to the expert action, while admissibility gives partial credit to valid but suboptimal actions in environments with defined action spaces.The admissibility reward is 0.1 for a non-expert action in the admissible set and 0.0 otherwise.Source: §A.1 · p. 14
  • For WebShop, ACT disables admissibility reward because search queries are open-ended, using only accuracy and format rewards.The admissible-action list includes clickable elements and a search[<query>] template when a search bar is present.Source: §A.1 · p. 14
  • ACT and optional downstream RL use GRPO, which samples response groups and estimates advantages from group-level reward statistics; IL uses supervised fine-tuning.The KL penalty coefficient is set to 0 in all experiments.Source: §A.1 · pp. 14–15
  • ACT training has two phases: collect alternatives sampled from the initial policy to construct the dataset, then optimize the policy with GRPO and verifiable action-correctness rewards.Source: §A.1 · p. 15

A.2 Implementation details

The implementation specifies the GRPO training stack and hardware, with model-specific hyperparameters listed in Table 4.

A.2 Implementation details

A.3 Data statistics · A.4 Choice of 𝐾 · A.5 Expert trajectory collection

These appendix sections describe the expert data and ACT training setup, explain the choice of one alternative per state, and identify the sources of expert trajectories across benchmarks.

A.3 Data statistics

  • ACT and IL use the same expert state–action pairs for a fair comparison, while RL starts from the corresponding IL checkpoint; each stage uses three epochs.The appendix identifies Table 5 as summarizing expert-data statistics, but the supplied passage does not provide its cell values.Source: §A.3 · p. 15, Table 5 · p. 17
  • ACT randomizes candidate positions, samples G responses for each prompt, computes ACT rewards, and updates the policy with GRPO.Source: §A.3 · p. 16
  • ScienceWorld training uses 10,240 randomly sampled state–action pairs from the full expert-trajectory set.Source: §A.3 · p. 16

A.4 Choice of 𝐾

  • All experiments set K = 1 because the expert datasets provide enough training samples; smaller datasets can use larger K to add contrastive pairs per state.Source: §A.4 · p. 16

A.5 Expert trajectory collection

  • ALFWorld expert trajectories are collected by running the model released by Feng et al. on the ALFWorld training set.Source: §A.5 · p. 16
  • WebShop trajectories come from the benchmark’s official human demonstrations, while ScienceWorld trajectories come from its official gold trajectories.Source: §A.5 · pp. 16–17

A.6 Additional evaluation protocol … A.9 Cross-size transfer to Qwen3-4B

The appendices specify evaluation metrics and comparison protocols, define reflection and objective controls, test rationale interventions, and report that Qwen3-8B ACT pairs transfer to improve Qwen3-4B downstream pipelines in ID and OOD settings.

A.6 Additional evaluation protocol

  • ALFWorld evaluation uses 140 valid-seen and 134 valid-unseen episodes for ID and OOD success rates, while WebShop reports success over 500 episodes.ALFWorld episodes are limited to 50 interaction steps.Source: §A.6 · p. 17
  • ScienceWorld uses 149 official-test episodes and reports the mean highest score per episode on a 0–100 scale, plus full-task completion at score 100.Episodes run for at most 50 steps, and the primary score retains credit for completed subgoals.Source: §A.6 · p. 17
  • Prompting baselines share a step-by-step action-selection instruction, and evaluations use temperature 0.6, top-p 0.95, and three runs.CoT and no-CoT prompts differ only by the instruction; reported error-bar values are mean ± standard deviation across runs.Source: §A.6 · p. 17

A.7 Baseline and control implementations

  • Early Experience, SAND, and Agent-R use synthesized reflection or deliberation targets with SFT, while all three baselines optimize next-token cross-entropy rather than policy gradients.Early Experience follows Zhang et al. (2025), SAND is described by Xia et al. (2025), and Agent-R by Yuan et al. (2025).Source: §A.7 · pp. 17–18
  • Same-action-pair controls retain ACT’s expert–alternative pairs and the IL→RL pipeline while replacing the stage-1 objective with choice-only SFT, DPO, or reasoning-free ACT.DPO treats the expert action as chosen and the alternative as rejected; the no-reasoning variant keeps ACT’s RL objective and reward.Source: §A.7 · p. 18

A.8 Causal rationale intervention protocol

  • A causal rationale intervention compares the original Qwen3-8B and standalone ACT checkpoints on 400 held-out ALFWorld pairs under free, no-rationale, and incorrect-rationale conditions.Only rationale context changes; the protocol measures choice accuracy and how often the final choice follows an injected alternative (Han et al., 2026).Source: §A.8 · p. 18, Table 6 · p. 18

A.9 Cross-size transfer to Qwen3-4B

  • ACT pairs collected from Qwen3-8B improve both Qwen3-4B downstream pipelines in ALFWorld ID and OOD settings without re-collection or adaptation.The compared pipelines are IL versus ACT→IL and IL→RL w/o ACT versus ACT→IL→RL; the supplied passages do not give the table’s numerical values.Source: §A.9 · p. 18, Table 7 · p. 19

A.10 Full general-reasoning results · A.11 ACT prompt templates

The appendix reports full standalone-ACT results for general-reasoning benchmarks and documents task-specific ACT prompts that compare randomized expert and alternative actions.

A.10 Full general-reasoning results

  • Accuracy is reported as mean ± standard deviation over three runs.Source: §A.10 · p. 19
  • Table 8 reports full ACT results on MATH-500 and GPQA-Diamond, with ACT denoting the standalone checkpoint before downstream IL or RL.Source: Table 8 · p. 19

A.11 ACT prompt templates

  • ACT templates randomize the order of expert and alternative actions, which appear as action_1 and action_2.Source: §A.11 · p. 20
  • The ALFWorld prompt presents full context and two candidate actions, asking the model to select the better one with reasoning.Source: Figure 5 · p. 20, §A.11 · p. 20
  • The WebShop prompt presents full context and two candidate actions, asking the model to select the better one with reasoning.Source: Figure 6 · p. 21, §A.11 · p. 21
  • The ScienceWorld prompt presents full context and two candidate actions, asking the model to select the better one with reasoning.Source: Figure 7 · p. 22, §A.11 · p. 22

B. Additional case studies: agentic task performance … C.1 Self-verification behavior

The case studies contrast ACT’s failure recovery and self-verification with IL’s rigid or uncorrected behavior. They illustrate state-aware action evaluation and verification in the standalone ACT-stage checkpoint.

B. Additional case studies: agentic task performance

B.1 Failure recovery through self-critique

  • The full ACT pipeline diagnoses an ALFWorld failure as a wrong location and navigates correctly, while IL repeats the failed action for over 30 steps.Figure 8 contrasts the ACT model breaking the loop with the IL model repeating the same action until termination.Source: Figure 8 · p. 23, §B.1 · p. 23

B.2 Rigid execution without state awareness

  • On WebShop, IL buys a $55 item despite the user’s $50 price limit, receiving a score of 0 without checking the product constraints.ACT targets evaluating whether the current product page satisfies the task constraints and searching again when it does not.Source: §B.2 · p. 23, Figure 9 · p. 24

C. Additional case studies: general reasoning after agentic

  • The general-reasoning case studies examine self-verification and reasoning degradation after IL using Qwen3-8B traces from GPQA-Diamond.The supplied case example concerns the standalone ACT-stage checkpoint’s self-verification behavior.Source: C. Additional case studies: general… · p. 25, §C.1 · p. 25

C.1 Self-verification behavior

  • On a difficult GPQA-Diamond problem, the standalone ACT-stage checkpoint substitutes answer options into the energy-conservation equation to eliminate inconsistent choices.The corresponding original-model trace derives an answer but does not systematically verify the options.Source: Figure 10 · p. 25, §C.1 · p. 25

C.2 Reasoning collapse after IL

The paper describes reasoning collapse after IL as a degradation in general reasoning linked to imitating short, action-heavy trajectories. A GPQA-Diamond example contrasts a focused, correct ACT trace with a longer, unfocused IL trace, which the authors treat as qualitative evidence.

C.2 Reasoning collapse after IL

  • IL on short, action-heavy expert trajectories can disrupt the original model’s chain-of-thought behavior and degrade general reasoning, a pattern the paper calls reasoning collapse.These trajectories contain minimal extended reasoning.Source: §C.2 · p. 25
  • 3.6× longer, the IL trace wanders through contradictory estimates and guesses incorrectly, while the standalone ACT checkpoint gives a focused 10,669-character derivation and the correct answer.The example concerns a high-energy physics threshold problem, GPQA-Diamond #7.Source: §C.2 · pp. 25–26, Figure 11 · p. 26
  • The authors present the trace as qualitative evidence that ACT may preserve more original reasoning behavior than IL, not as a claim about every problem.They attribute the contrast to ACT optimizing correct action selection via RL rather than imitating short, action-centric targets.Source: §C.2 · p. 26

Related papers

Loading 2603.08706v2…