PaperlayerView exact revision on arXiv

Source-linked AI summary

Beyond Corrected Actions: Training LLM Agents on Textual Critiques

Thomas Zeng, Zelei Cheng, Mingyang Zhou, Sambit Sahu, William Campbell, Genta Indra Winata

arXiv:2610.19367v1cs.CLcs.AI

TL;DR

Multi-turn agents must handle diverse action errors, but action supervision alone does not explain why a student’s original action was wrong. CAFT trains on expert critiques and corrected actions from student-generated trajectories; across four models and two agentic domains, it improves over action-only supervision by 8% on average.

  • Problem

    Action-focused supervision shows what an expert would do but not why a student’s original action is incorrect, even though agent errors can arise from diverse policy, information, or tool-use failures.

    Source: §1 · p. 1

  • Method

    CAFT uses an expert to critique incorrect actions and provide corrected actions on student trajectories, then fine-tunes the student on both targets.

    Source: §1 · p. 2, §2 · p. 3

  • Results

    Across four Nemotron 3 and Gemma 4 students, CAFT improved over action-only supervision by 8% on average and provided a stronger RL initialization than the base model or expert-model distillation.

    Source: §1 · p. 2

  • Takeaways & Limitations

    Expert critiques are a natural-language feedback channel that can be incorporated directly into training for multi-turn LLM agents.

    Source: §5 · p. 9

    The precise conditions under which CAFT+BC mixtures succeed remain unresolved.

    Source: §5 · p. 9

Abstract

from arXiv · show

Large language models (LLMs) are increasingly used for multi-turn agentic tasks that require interactions with users and environments over extended sequences of actions. A common approach to improving LLM agents is to use more capable expert models to provide training supervision, either by behavior cloning the expert's trajectories or by using the expert to correct the actions in trajectories generated by the weaker student model. However, expert LLMs can provide richer supervision than actions alone: they can directly generate textual critique that explain why specific actions are incorrect. We study whether training on such feedback benefits LLM agents and propose Critique-Augmented Agentic Fine-Tuning (CAFT), a framework that elicits textual expert feedback on student-generated trajectories. For each incorrect action, the expert yields both a corrected action and a textual critique explaining the student's mistake--which we then use as the training target for fine-tuning. Across four student models from the Nemotron 3 and Gemma 4 families on the agentic domains $\tau^2$-Bench Airline+Retail and WebShop, adding critiques to action-level supervision improves performance by 8% on average over action-only distillation. Additionally, CAFT checkpoints can also provide a better warm start for reinforcement learning than expert-trajectory distillation. Our results demonstrate that textual critiques can strengthen supervision for multi-turn LLM agents.

1 INTRODUCTION

Multi-turn agents face diverse action errors, while action-focused supervision does not explain why a student’s original action was wrong. CAFT trains on expert critiques and corrected actions from student trajectories, improving over action-only supervision across the evaluated models and domains.

1 INTRODUCTION

  • Action-focused methods supervise what the expert would do, but not why the student’s action is wrong, despite errors involving policies, information needs, or tool-use constraints.The paper motivates critiques as additional learning signal beyond corrected actions.Source: §1 · pp. 1–2
  • CAFT collects student trajectories, has an expert identify incorrect actions and provide critiques plus corrections, then trains on both while retaining the erroneous action in context.The critique explains what was wrong with the student’s original action.Source: §1 · p. 2
  • 8% on average, CAFT improves over action-only supervision across four student models on WebShop and τ 2-Bench Airline and Retail, with gains up to 27% on WebShop for Nemotron 3 Nano 4B.The paper also reports a stronger reinforcement-learning initialization than the base model or standard expert-model distillation in a single-model case study.Source: §1 · p. 2
  • Targeted ablations show that textual critiques provide a stronger learning signal than corrected actions alone.CAFT consistently improves over corrected-action training across four student models and two agentic environments.Source: §1 · p. 2
  • The paper also examines privileged information for critique generation, compares CAFT with expert-trajectory distillation and repeated fine-tuning, and evaluates it as an RL initialization.Source: §1 · p. 2

2 TRAINING LLM AGENTS ON TEXTUAL CRITIQUES

CAFT builds fine-tuning supervision from student-generated trajectories, pairing expert critiques with corrected actions for incorrect turns. The caption reports that CAFT-Critique and CAFT outperform other methods across all student–environment pairs.

2 TRAINING LLM AGENTS ON TEXTUAL CRITIQUES

  • CAFT rolls out the student policy, has an expert assess each turn, and uses critiques and corrected actions from incorrect turns to fine-tune the student.In the experiments, the expert has access to privileged task information, including reference actions and evaluation criteria.Source: §2 · p. 3
  • The setting models agent actions and environment observations as an interaction history, with task success evaluated by a binary correctness grader.CAFT aims to improve expected task success using training data from the base policy’s trajectories.Source: §2 · p. 3
  • CAFT-Critique and CAFT consistently outperform all other methods across student–environment pairs in task success.Table 1 covers τ² AR and WebShop, comparing base policies, action distillation, and CAFT variants.Source: Table 1 · p. 4
  • For incorrect turns, training retains the erroneous action in context, appends an interjection, and targets the expert’s critique followed by its correction.Correct turns instead retain the base policy’s sampled action as behavior-cloning supervision.Source: §2 · p. 4
  • The resulting dataset is used to fine-tune the policy with a standard autoregressive language-modeling objective.Student-policy trajectories supply the turns used to construct the supervised dataset.Source: §2 · p. 4

3 EXPERIMENTS

Experiments compare CAFT with action-only supervision and expert-trajectory distillation, test repeated CAFT training, and evaluate CAFT as an RL initialization across τ 2 AR and WebShop. CAFT outperforms action distillation across the reported settings, while results for other comparisons vary by environment and student.

3.1 EXPERIMENTAL SETUP

  • Experiments use Airline and Retail in τ 2-Bench and WebShop, with four student policies and Gemma 4 31B as the critique-generating expert.Each student–environment pair is rolled out on training tasks with 16 rollouts per task, and the expert annotates every agent turn.Source: §3.1 · p. 5

3.2 DO CRITIQUES PROVIDE SUPERVISION BEYOND CORRECTED ACTIONS?

  • 8.4% average gain over Action Distillation across all eight settings, with CAFT strongest in six.The experiments also report that Action Distillation consistently improves over the base model.Source: §3.2 · p. 5

3.3 HOW MUCH DOES THE EXPERT NEED TO KNOW TO CRITIQUE EFFECTIVELY?

  • Removing privileged task information reduces success by 7.5% on average; detailed prompting matters less and varies by student.With privileged information available, a minimal instruction retains most performance; detailed prompting matters most for Gemma 4 E2B and has no impact on Nemotron 3 Nano 4B.Source: §3.3 · p. 6, Table 2 · p. 6

3.4 COMPARISON TO EXPERT-TRAJECTORY DISTILLATION

  • On WebShop, expert-trajectory distillation yields 15.0–19.1% success, below base models’ 24.3–43.1%, while distilled policies repeatedly search without purchasing.The Table 3 caption says CAFT+BC was not run on WebShop because expert distillation collapses there.Source: §3.4 · pp. 6–7, Table 3 · p. 7

3.5 DOES A SECOND ITERATION OF CAFT HELP?

  • A second CAFT iteration improves every student on WebShop by 5.3% on average, but gains on τ 2 AR are mixed and may saturate.The two smaller Gemma students improve by 1.6% and 3.1% on τ 2 AR; Nemotron 3 Nano 4B sees no further improvement.Source: Table 4 · p. 7, §3.5 · p. 7

3.4 COMPARISON TO EXPERT-TRAJECTORY DISTILLATION

  • On τ 2 AR, CAFT+BC beats both components for Gemma 4 E2B (53.7%) and E4B (60.1%), but trails CAFT for Nemotron 3 Nano 4B (56.0% vs. 60.3%).Expert distillation alone is slightly better than CAFT for the two smaller Gemma students, while CAFT is better for Nemotron 3 Nano 4B.Source: §3.4 · p. 7, Table 3 · p. 7

3.6 CAN WE USE CAFT AS A WARM START FOR RL?

  • For Nemotron 3 Nano 4B on τ 2 AR, CAFT initialization beats base and expert-distillation starts by around 5% on held-out tasks.The CAFT-initialized run matches the base-initialized run’s best held-out success after about 60 of 300 GRPO steps; expert distillation’s higher training reward does not transfer.Source: Figure 2 · p. 8, §3.6 · p. 8

4 RELATED WORK

Prior agent-supervision methods train on expert behavior or corrected actions at student-visited states, while CAFT retains the student’s mistake and adds critique supervision. Related work also uses critiques for reasoning, reflection, and recovery, but CAFT tests whether explaining mistakes improves multi-turn decisions beyond corrected actions alone.

Expert supervision for training LLM agents

  • CAFT keeps the student’s mistake in context and trains on the expert’s critique, unlike prior action-relabeling methods that replace the mistake with the teacher’s action.This addresses supervision on states induced by the student policy while adding feedback about why its action was wrong.Source: §4 · p. 8

Training from natural language critiques

  • Prior critique methods span single-turn reasoning, response refinement, and agent feedback, while CAFT asks whether explaining an agent’s own mistakes improves multi-turn decisions beyond corrected actions.The cited agent work includes reflection at expert-visited states and conditioning tool policies on critic verdicts.Source: §4 · p. 9

Reflection and error recovery in LLM agents

  • CAFT’s correction specifies what the agent should have done instead of the mistake, rather than how to proceed afterward as in some recovery-oriented approaches.The paper distinguishes its focus on the critique’s contribution to task performance from approaches that train recovery behavior.Source: §4 · p. 9

5 CONCLUSION

The paper concludes that expert critiques add learning signal beyond corrected actions for multi-turn agents, and identifies broader validation and explanation of the effect as open directions.

5 CONCLUSION

  • CAFT-Critique outperforms CAFT-Correction and action-only distillation across all tested settings, and the authors report that training agents to explain mistakes improves correct action performance.The paper describes critiques as a cheap, rich feedback form that can be incorporated directly into training.Source: §5 · p. 9
  • The authors leave scaling to larger students and broader domains, extending privileged self-critique, clarifying critique-plus-demonstration mixtures, and explaining why critique targets help for future work.They also identify direct user feedback and online training on arriving corrections as possible extensions.Source: §5 · p. 9

A ENVIRONMENTS, TASK SPLITS, AND EVALUATION PROTOCOL

The appendix specifies the task splits and evaluation procedures for τ²-Bench Airline and Retail and WebShop, then describes the expert annotation prompts used in each environment. The prompts provide task-relevant information and judge actions step by step, with a criticality filter for the final τ²-Bench annotations.

A ENVIRONMENTS, TASK SPLITS, AND EVALUATION PROTOCOL

  • τ²-Bench Airline and Retail use 60 held-out tasks, 16 rollouts per task, and mean task success; WebShop uses the first 100 goals for evaluation, the next 100 for training, and 16 rollouts per goal.τ²-Bench uses a 40-turn limit and a binary outcome grader; WebShop uses a 30-action limit and scores whether the purchased item satisfies all request constraints.Source: A ENVIRONMENTS, TASK SPLITS, AND… · p. 12

B EXPERT ANNOTATION AND INTERJECTION PROMPTS

  • Both environments use expert prompts that provide privileged information, judge one action step at a time, and request a textual critique plus a counterfactually corrected action for flagged steps.The prompts differ to match the environments’ action spaces and reward structures.Source: B EXPERT ANNOTATION AND INTERJECTION… · p. 13

B.1 τ 2 AR EXPERT ANNOTATION PROMPT

  • For τ²-Bench, the expert receives the task, hidden user scenario, gold actions, evaluation criteria, relevant database context, conversation, and action being judged; its MISSTEP analysis supplies the critique and the text after “Instead:” supplies the corrected action.Final CAFT annotations retain only correct and incorrect steps labeled critical; the prompt’s criticality label is intended to curb over-labeling.Source: §B.1 · pp. 13–14

B.2 WEBSHOP EXPERT ANNOTATION PROMPT

  • For WebShop, the expert judges search or click actions using the shopping instruction, privileged target-product requirements, current clickable buttons, and episode history.The student harness appends a CAFT system addendum during fine-tuning.Source: §B.2 · p. 14

B.3 PROMPT-ABLATION VARIANTS

The prompt ablation varies privileged task information and critique guidance independently while re-annotating the same student trajectories with the same expert. This design isolates the annotation prompt’s contribution.

B.3 PROMPT-ABLATION VARIANTS

  • The ablation independently varies whether the expert receives privileged information and detailed critique guidance, re-annotating identical student trajectories with the same model across conditions.Withholding privilege removes gold actions, reward criteria, and the hidden scenario; minimal guidance removes the domain note and replaces the correctness and criticality clauses with bare questions.Source: §B.3 · pp. 14, 17
  • The no-privilege variant replaces the expert prompt’s opening and removes references to gold actions and reward criteria from the correctness and criticality clauses.The hidden user scenario is also withheld in this condition.Source: §B.3 · p. 14, Figure 9 · p. 17
  • The minimal-guidance condition retains the output format and first-person directive while replacing the detailed domain note and correctness and criticality clauses.The detailed version uses Figure 3’s guidance; the minimal version uses the bare questions shown in Figure 10.Source: §B.3 · p. 17, Figure 10 · p. 17

B.4 INTERJECTION ρ AND ABLATION VARIANTS

The interjection ρ is a fixed user-role message inserted after an expert-flagged action, while the student’s system prompt discloses that a coach is reviewing its actions. These strings are used during training, not evaluation, and distinguish CAFT from its ablations.

B.4 INTERJECTION ρ AND ABLATION VARIANTS

  • CAFT and its ablations differ only in the student system addendum, interjection ρ, and corresponding target y_t.Figure 11 identifies the CAFT (full) system addendum and interjection ρ.Source: §B.4 · p. 17, Figure 11 · p. 17
  • The interjection ρ is inserted as a fixed user-role message after an expert-flagged action.The system prompt’s coach-review disclosure makes this training-time interjection in-distribution during supervised fine-tuning; neither interjection is used at evaluation.Source: §B.4 · p. 17

C DATA COLLECTION AND SUPERVISED FINE-TUNING DETAILS

The paper collects student trajectories with expert annotations and uses them for supervised fine-tuning, while its reinforcement-learning warm-start experiments use GRPO with binary episode-level rewards.

C.1 TRAJECTORY COLLECTION AND ANNOTATION

  • Each student and environment receives one base-policy rollout over the training tasks, with 16 rollouts per task and expert annotations for every agent turn.Figures 12 and 13 identify the CAFT-Correction and CAFT-Critique student system addenda and interjection ρ.Source: §C.1 · p. 17, Figure 12 · p. 18, Figure 13 · p. 18
  • CAFT, ablation, and action-distillation runs use batch size 64, learning rate 3×10^-6, and two epochs on τ² AR or one on WebShop.Expert distillation and CAFT+BC mixtures on τ² AR instead use learning rate 1×10^-5 for two epochs; iteration-2 CAFT keeps the first iteration’s hyperparameters. Compute-matched mixtures contain 1,664 trajectory-equivalents split between expert trajectories and CAFT examples.Source: §C.1 · p. 18

D REINFORCEMENT LEARNING DETAILS

  • GRPO warm-start runs use binary τ² outcome rewards at episode end, and only the initialization differs across the three runs.The policy self-serves through vLLM on the training GPUs, while the user simulator and outcome judge run on an external Gemma 4 31B endpoint; each run occupies one 8-GPU node.Source: D REINFORCEMENT LEARNING DETAILS · p. 18

E.1 HOW MUCH SHOULD CORRECT TURNS BE SUPERVISED?

The correct-turn loss weight affects τ² AR task success: λ = 0.1 is best for both tested students, while both upweighting and nearly removing correct-turn supervision reduce performance.

E.1 HOW MUCH SHOULD CORRECT TURNS BE SUPERVISED?

  • λ = 0.1 is best for both students; λ = 0.5 costs 7.2% for Gemma 4 E2B and 5.6% for Nemotron 3 Nano 4B, while λ = 0.01 costs 0.8% and 7.3%, respectively.The sweep uses τ² AR validation data; Table 6 reports task success as a function of correct-turn loss weight.Source: §E.1 · p. 18, Table 6 · p. 19
  • Masking correct-turn loss entirely costs Nemotron 3 Nano 4B 7.1%: success falls to 53.2% versus 60.3% for the matched λ = 0.1 run.The authors suggest λ = 0.5 lets imitation of the student’s executed actions dominate critique learning; without correct-turn loss, performance degrades on the ordinary correct turns that make up most of a trajectory.Source: §E.1 · p. 18, Table 6 · p. 19

E.2 MIXING RATIO FOR CAFT+BC

The section reports that the preferred expert-trajectory share in CAFT+BC varies by student, and that CAFT-trained policies need fewer expert interventions than their base policies. The intervention results differ by environment and by whether success is measured before or after expert corrections.

E.2 MIXING RATIO FOR CAFT+BC

  • An equal expert-trajectory and CAFT mix is best for both Gemma students, while Nemotron prefers the mixture with the most critique data.The ratio sweep matches total training compute; Nemotron is the only student for which CAFT alone already beats expert distillation.Source: §E.2 · p. 19, Table 7 · p. 19

E.3 DO CAFT-TRAINED AGENTS RECEIVE LESS INTERVENTION FROM THE EXPERT MODEL?

  • The coach-in-the-loop validation rollout replaces expert-flagged mistakes with corrected actions and tracks corrected task success and interventions per episode.The experiment tests whether CAFT-trained checkpoints require fewer interventions than their base policies.Source: §E.3 · pp. 19–20
  • CAFT checkpoints require substantially fewer interventions than base policies in both environments, but corrected success rises on τ² AR and drops on WebShop.CAFT+BC requires the fewest interventions overall on τ² AR, where its remaining errors are harder to recover from; on WebShop, the authors suggest errors not caught by the expert become more fatal.Source: §E.3 · p. 20, Table 8 · p. 20
Loading 2610.19367v1…