PaperlayerView exact revision on arXiv

Source-linked AI summary

Training Advisors for LLM Agents from Task Outcomes

Sergei Polezhaev, Barys Liskavets, Ori Press, Alexander Golubev

arXiv:2610.09858v1cs.CLcs.AIcs.LG

TL;DR

Multi-step agents can get stuck or lose sight of their goals, and natural-language feedback can help them reconsider their approach. Caddie trains a critic from the final outcomes of a frozen agent’s continuations; its trained critic improves success across base models and out-of-domain tasks.

  • Problem

    Multi-step agents may repeat unsuccessful actions or lose sight of their goals, motivating feedback that identifies problems and suggests useful next actions.

    Source: §1 · p. 1

  • Method

    Caddie trains a natural-language critic using task outcomes after a frozen base model continues acting with its feedback, without step-level annotations or reference critiques.

    Source: §1 · p. 2

  • Results

    The critic transfers to three base models not used in training and out-of-domain tasks; on attached MuSiQue, it raises Qwen3-4B’s success rate by 25.31 percentage points.

    Source: §7 · p. 9

  • Takeaways & Limitations

    The same outcome-based training procedure produces feedback useful across base models and task settings, including out-of-domain tasks without further training.

    Source: §7 · p. 9

    Training requires G×N base-model continuations per sampled state and restorable environment states, and the effect of critic scale remains unverified because only Qwen3-4B critics were trained.

    Source: §5 · p. 8

Abstract

from arXiv · show

Large language model agents tackle multi-step tasks by interleaving reasoning and tool calls with observations from the environment. Prior work has shown that natural-language feedback can help these agents revise their decisions during task execution. We introduce Caddie, a method for training critics to provide natural-language analysis and advice as agents work through a task. Unlike approaches that rely on step-level labels or reference critiques, Caddie learns from whether the agent ultimately succeeds after receiving the critic's feedback. We optimize the critic through reinforcement learning while keeping the base model frozen. Trained on multi-hop question answering with a single base model, our Qwen3-4B critic improves success rates across four base models of different scales and architectures, including three not used during critic training. On the MuSiQue benchmark, the trained critic improves Qwen3-4B's success rate by more than 25 percentage points, surpassing the performance of Kimi K3 without a critic. The same critic also yields gains on out-of-domain interactive benchmarks, including $\tau^3$ and DeepDive, with no additional training. Our results show that agents can decide when to seek help from a critic at inference time and that outcome-based critic training can produce guidance that transfers across base models and task domains.

1 Introduction

Multi-step agents can repeat unsuccessful actions or lose sight of their goal, while conventional step-level assessment requires costly supervision and can misclassify useful exploratory actions. Caddie trains a critic to advise agents using downstream task outcomes, without step-level labels or reference critiques.

Motivation

  • Agents may repeat unsuccessful actions or lose sight of their goals, and external feedback can identify problems and suggest useful next actions.Natural-language critique can reduce the need for costly search, branching, or replay while making proposed next steps easier to interpret.Source: §1 · p. 1

Limitations of step-level verification

  • Step-level verification requires costly labels and can misclassify interactive actions that gather information or explore the environment.Automatic labeling can be costly and noisy, while human annotation is expensive at scale.Source: §1 · p. 1

Prior approaches

  • Prior learned-critic approaches include synthesizing reference critiques or separately optimizing critics to judge and improve solutions.The paper motivates simpler training for agents that continue acting and observing.Source: §1 · p. 2

Caddie

  • Caddie trains generative advisors to guide ongoing agent interactions using downstream task outcomes, without step-level annotations or reference critiques.The agent continues acting after receiving the advice, and its final task outcome rewards the critic.Source: §1 · p. 2

Contributions

  • A Qwen3-4B critic trained with one base model improves three additional models and yields gains on out-of-domain benchmarks without further training.The paper also examines feedback across inference strategies and base models.Source: §1 · p. 2

2 Method

Caddie adds natural-language critique to a frozen agent’s ongoing interaction with an environment. The critic is trained from the final outcomes of the agent’s continuations, and the agent can request critique during inference.

2 Method

  • Caddie trains a critic to guide an agent during task execution, using final outcomes of the frozen base model’s continuations as reward.The method needs neither task-specific step labels nor reference critiques.Source: §2 · pp. 2–3

2.1 Critique intervention

  • At each step, the frozen base policy samples an action, the environment returns an observation, and the resulting interaction is appended to the state.The episode ends with a terminating action or an exhausted interaction budget and receives a final task reward.Source: §2.1 · p. 3
  • When invoked, the critic analyzes the current state and advises next steps; the base agent then acts conditioned on the augmented state.An intervention protocol determines whether the critic is invoked, and a trajectory may contain zero, one, or several critiques.Source: §2.1 · p. 3
  • The three intervention protocols are windowed, probabilistic, and self-call, with self-call letting the agent choose when to request critique.Windowed and probabilistic protocols may intervene too early, too late, or when help is unnecessary; their performance depends on the chosen schedule.Source: §2.1 · p. 3

2.2 Outcome-based critic training

  • For each sampled prefix, the critic generates critiques whose rewards are the average final task rewards of N frozen-agent continuations.This evaluates feedback through its effect on downstream task performance.Source: §2.2 · p. 4
  • DAPO updates only the critic using group-normalized advantages; zero-variance reward groups are discarded and replaced with additional samples.Task instructions, observations, and base-model continuation tokens are excluded from the loss.Source: §2.2 · p. 4
  • On self-contained tasks with small token budgets and no tool calls, an optimal critic may solve the task and instruct the base model to copy its solution.This formulation constrains the choice of tasks and environments.Source: §2.2 · p. 4

3 Experiments

Experiments test Caddie in its training settings and then across unseen base models and task families. The critic improves in-domain success and transfers without adaptation, though gains vary by model and retrieval setting.

3.1 Experimental setup

  • Experiments cover MuSiQue and ALFWorld in-domain, plus DeepDive and τ3 transfer, testing multiple base models with self-call and eight seeded rollouts per task.Critics are trained separately for the in-domain settings while the Qwen3-4B base model remains frozen.Source: §3.1 · pp. 4–5

3.2 In-domain performance

  • 25.31 percentage points with attached passages, 10.75 points with Wikipedia retrieval, and 25.93 points on ALFWorld are Caddie’s gains over no critic.It also beats both prompted critics in all three settings; on MuSiQue-attached, feedback raises a trained generator from 49.69% to 63.56% (+13.88 percentage points).Source: Table 1 · p. 5, §3.2 · p. 5

3.3 Transfer to unseen base models

  • 9.31–13.19 percentage points is the attached-passage gain range across three unseen base models; Wikipedia-retrieval gains range from 0.81 to 6.19 points.Caddie exceeds both prompted critics in all six conditions, while the magnitude of transfer depends on model and retrieval setting.Source: Table 2 · p. 6, §3.3 · p. 6

3.4 Transfer to unseen task families

  • The wiki-trained critic exceeds the no-critic mean in all six base model–task combinations, while the attached-trained critic does so in five.With Qwen3.8-27B, wiki-trained Caddie raises τ3 Retail from 50.31% to 58.12% and Airline from 74.38% to 78.12%; on DeepDive, attached training drops success from 48.83% to 44.53%, while wiki training reaches 49.80%.Source: §3.4 · p. 6, Table 3 · p. 7

4 Analysis

The analysis finds that self-requested feedback is associated with higher solve rates, and that useful advice varies with both task and generator. These comparisons are descriptive rather than causal.

Self-call feedback

  • Feedback-bearing rollouts have higher solve rates in all six settings than no-critic rollouts on the same tasks, but the comparison is conditional on the generator requesting advice.The generator most often calls Caddie once; the comparison does not isolate a causal effect.Source: Figure 2 · p. 7, §4 · p. 7

Advice adapts to the task

  • Advice changes with the environment: DeepDive suggestions focus on evidence search or answering, while τ3 advice more often recommends verifying state, asking the user, or acting.Figure 3 illustrates matched τ3 Retail trajectories following the same change in user intent.Source: §4 · pp. 7–8, Figure 3 · p. 8

Advice by generator scale

  • Targeted search advice helps Qwen3-4B most, whereas Qwen3.8-27B benefits most from advice to answer using evidence already gathered.Additional search can help the smaller generator but distract the larger one.Source: Figure 4 · p. 8, §4 · p. 8

Out-of-domain interpretation

  • The results suggest that selective calls and advice tailored to a domain and generator weakness are a possible mechanism for out-of-domain gains.The figure comparisons are not randomized interventions.Source: §4 · pp. 7–8

5 Discussion and Limitations

The discussion identifies boundaries in task structure, training cost, and what the experiments establish about generative feedback relative to scalar verification.

Task structure

  • Caddie is intended for multi-step, tool-using tasks that require further environment interaction; on self-contained tasks, it may learn complete solutions that limit transfer.Under the outcome-based objective, the critic can become a task-solving policy for the base model to copy.Source: §5 · p. 8

Practical constraints

  • Training requires G N_base-model continuations per sampled state and restoring the same environment state across branches, which can be costly for long trajectories or expensive tools.Only Qwen3-4B critics were trained, so the effects of critic scale on performance and transfer remain unverified.Source: §5 · p. 8

Scalar verification

  • The experiments do not establish that natural-language feedback is generally superior to scalar verification; testing that requires matched inference procedures and compute budgets.Scalar verification can rank or select candidate continuations, while generative critics provide text conditioning the base model’s next actions.Source: §5 · p. 8

6 Related Work

Caddie is positioned among learned critique, advice, revision, and agent-training methods. Its distinguishing focus is outcome-based advice during an ongoing trajectory while the acting model remains frozen.

Learning critics from task outcomes

  • Unlike RL4F and CTRL, which initialize critics with supervised critique examples, and Critique-RL, which first trains correctness judgments, Caddie trains from final outcomes of ongoing agent continuations.Caddie uses neither supervised critique initialization nor a separate correctness-judgment stage.Source: §6 · p. 9

Learned feedback for agents

  • Advisor Models also reward advice through a frozen student’s task outcome, but Caddie branches from an intermediate state to correct an episode in progress and lets the agent request advice.Advisor Models deliver advice before a response, after a completed attempt, or on a fixed schedule in coding-agent experiments; Caddie reports gains on unseen base models and task families without adaptation.Source: §6 · p. 9

Revision and reflection

  • Unlike methods that rewrite outputs or reflect on failed attempts for later retries, Caddie advises the agent’s next steps during the same attempt.The cited revision methods include Welleck et al. (2023), GLoRe, and Retroformer.Source: §6 · p. 9

Training agents with feedback

  • Unlike approaches that train the critic alongside the agent, learn both reflection and action, or alternate critic and generator training, Caddie updates only the critic.The acting model’s parameters remain unchanged.Source: §6 · p. 9

7 Conclusion and Future Work

Caddie trains natural-language critics from the final outcomes of frozen base-model continuations, without task-specific step labels or reference critiques. The trained critic improves MuSiQue success and transfers across base models and task settings, while single-pass tasks expose a task-structure limitation.

7 Conclusion and Future Work

  • Caddie learns natural-language analysis and advice from final outcomes of a frozen base model’s continuations, avoiding task-specific step labels or reference critiques.The critic is optimized using a GRPO-based approach.Source: §7 · p. 9
  • 25.31 percentage points: the trained Qwen3-4B critic improves Qwen3-4B’s success rate on MuSiQue in the attached setting, with gains also reported across other base models and out-of-domain tasks.The same critic transfers to Qwen3-30B-A3B, Qwen3.8-27B, and Kimi K3, and yields gains on τ3 and DeepDive without further training.Source: §7 · p. 9
  • In self-contained tasks solvable in one pass, the critic can learn to generate complete solutions for the base model to copy, which may be optimal under the outcome-based objective.Future work could explore fine-tuning the base policy to use a critic more effectively or jointly training the policy and critic.Source: §7 · p. 9

A Alternative Objectives

The appendix compares alternative reward and continuation choices, then specifies feedback formatting, critic-call behavior, and evaluation budgets. Several plotted comparisons lack uncertainty estimates, so the curves do not establish reliable rankings or convergence differences.

A Alternative Objectives

  • Using N=1 or N=5 continuations produces similar final full-validation results, while N=1 requires fewer base-model continuations and less compute.Figure 5 compares the variants on a 20-task MuSiQue validation subset.Source: A Alternative Objectives · p. 12
  • The Gumbel counterfactual reward pairs continuations with and without a critique from the same state using shared positional noise, but its plotted early rise does not establish faster convergence.Shared noise reduces reward-difference variance only when paired rewards have positive covariance, which the authors do not measure after trajectories diverge.Source: A Alternative Objectives · p. 12, Figure 5 · p. 13
  • No-critique baseline subtraction assigns positive or negative advantage according to whether a critique’s estimated reward exceeds the baseline; baseline estimation adds noise that decreases with M.The baseline’s contribution to the expected unclipped score-function gradient is zero, but this does not mean it leaves the clipped objective unchanged.Source: A Alternative Objectives · pp. 12–13
  • The plotted baseline-subtraction variant improves early but later falls below terminal-outcome curves; without uncertainty estimates, this does not establish a reliable ranking.The authors suggest that changing advantage signs as critiques outperform the baseline may affect optimization under clipping, and leave a gradual baseline shift for future work.Source: A Alternative Objectives · p. 13, Figure 5 · p. 13
  • Training with self-call yields similar in-domain performance but less consistent transfer; training only on states chosen by one base model may limit exposure to other models’ states.Source: A Alternative Objectives · p. 13

B Format

  • Critiques must contain nonempty analysis and next-step advice, without tools, a complete solution, or a full action sequence; malformed feedback receives −1 without a continuation.Source: B Format · pp. 13–14
  • In self-call evaluation, feedback arrives as a user-role observation without advancing the environment; trajectories allow at most three critic calls.After the third call, only an environment action is permitted.Source: B Format · p. 14
  • ALFWorld evaluation allows 50 environment actions, while training uses a 12-action budget shared by prefix and continuation; episodes also terminate on invalid calls, malformed output, or context overflow.Source: B Format · p. 14

C Tasks, Splits, and Evaluators

The evaluation uses multi-hop MuSiQue questions, ALFWorld interactive tasks, and transfer benchmarks in DeepDive and τ3. Each benchmark has specified splits, trial counts, and task-specific evaluation procedures.

C Tasks, Splits, and Evaluators

  • MuSiQue comparisons use 200 validation questions with eight trials each; the subset includes 104 two-hop, 63 three-hop, and 33 four-hop questions.The attached setting searches 20 supplied passages, while the wiki setting uses a frozen Wikipedia retrieval service.Source: C Tasks, Splits, and Evaluators · p. 14
  • MuSiQue attached-setting evaluation requires at least three searches and scores normalized exact matches against reference answers.Normalization case-folds text, replaces punctuation with spaces, removes English articles, and collapses whitespace.Source: C Tasks, Splits, and Evaluators · p. 14
  • ALFWorld evaluation covers all six task families on 134 unseen-validation games with eight trials per game, while critic training uses 1,557 eligible place, heat, and examine-under-light games.Source: C Tasks, Splits, and Evaluators · p. 14
  • Transfer evaluations use 64 DeepDive questions, 40 τ3 Retail tasks, and 20 τ3 Airline tasks, with eight trials per task.DeepDive answers are judged by Qwen3-32B, and τ3 uses predefined test splits inherited from τ2-bench.Source: C Tasks, Splits, and Evaluators · p. 14

D Models and Training Details

The critic is initialized from Qwen3-4B-Instruct-2507 and trained while the corresponding base model remains frozen. Training uses scheduled intervention windows and fixed final checkpoints selected after validation plateaus.

D Models and Training Details

  • The critic is initialized from Qwen3-4B-Instruct-2507 with all parameters trainable, while the Qwen3-4B-Instruct-2507 base model remains frozen.Source: D Models and Training Details · p. 14
  • MuSiQue training samples one intervention uniformly from steps 1–10; ALFWorld samples from steps 2–10 and uses a task-dependent seed.For ALFWorld, the prefix and continuation share a 12-action budget, and evaluation uses self-call with up to three interventions and a 50-action budget.Source: D Models and Training Details · p. 14
  • MuSiQue training stops at 2,000 steps and uses the final checkpoint; ALFWorld stops at 800 steps, also using one checkpoint across evaluations.Both stopping points followed an observed validation plateau.Source: D Models and Training Details · pp. 14–15

E Critic Feedback on a GRPO-Trained Generator

On attached retrieval, the critic trained with a GRPO-trained generator adds more success than the critic trained with the original generator; its wiki gains test transfer beyond its training retrieval setting. The paper presents critic training as complementary to generator optimization, not a replacement for task-specific optimization.

E Critic Feedback on a GRPO-Trained Generator

  • The critic is trained after freezing the GRPO-trained generator, using terminal success from one generator continuation per critique.The generator is first trained on MuSiQue; the retained generator checkpoint is step 1,120 and the critic run is evaluated at step 2,000.Source: E Critic Feedback on a GRPO-Trained… · p. 15
  • Evaluations used 200 questions, eight trials per question, self-call intervention, and at most three critiques per trajectory.The generator chooses when to call, and the critic uses visible interaction history to provide feedback.Source: E Critic Feedback on a GRPO-Trained… · p. 15, Table 5 · p. 15
  • 13.88 percentage points over no-critic versus 2.00 points for the original-generator critic on attached retrieval.Both results are for the GRPO-trained generator.Source: Table 5 · p. 15, E Critic Feedback on a GRPO-Trained… · p. 15
  • 3.69 points on wiki retrieval for the GRPO-generator critic versus 4.75 points for the original-generator critic; wiki tests transfer beyond critic training retrieval.The GRPO-generator critic was trained only on attached retrieval, and the largest gain was in that training setting.Source: E Critic Feedback on a GRPO-Trained… · pp. 15–16
  • The paper does not claim critics outperform task-specific policy optimization; critic training after generator plateau provided additional gains.Direct GRPO was slightly better than pairing the original instruction-tuned generator with a critic.Source: E Critic Feedback on a GRPO-Trained… · p. 16

F Results with Other Inference Protocols

This section examines Caddie under alternative inference protocols and across out-of-domain tasks, including how intervention timing, self-call, and critic requests shape guidance. Trajectory examples show advice preserving constraints and helping agents use decisive evidence, while alternatives can lead to costlier routes, transfers, or false leads.

F Results with Other Inference Protocols

  • Fixed-step intervention yields a higher mean than self-call in some MuSiQue settings, but these diagnostic runs allow at most one self-call per trajectory.The main evaluations instead use a three-call limit; fixed-step intervention is a special case of Pwin with K = {k}.Source: Table 6 · p. 16, F Results with Other Inference Protocols · p. 16
  • Moving the Qwen3-4B intervention from step 3 to step 4 raised mean success from 17.75% to 30.88%; the best reported step also varies by base model.These are descriptive comparisons, and the reported SEMs quantify uncertainty in each mean, not in the paired difference.Source: Table 7 · p. 16, F Results with Other Inference Protocols · p. 16

H.1 How often is Caddie called?

  • One critic request dominates every setting; on DeepDive, a second request does not coincide with lower accuracy for either generator.The smaller generator repeats requests more often on DeepDive, while the larger generator almost always asks once.Source: Figure 6 · p. 19, §H.1 · p. 19, Figure 7 · p. 20

H.2 What advice is given, and is it used?

  • Generators follow most advice exactly or partially, with DeepDive advice emphasizing search and answer completion and τ3 advice more often emphasizing verification, clarification, or environment action.Targeted search helps Q4, whereas Q38 benefits more from advice to answer using evidence already available.Source: Figure 8 · p. 20, §H.2 · p. 20

H.3 Complementary views

  • The task-family view of DeepDive indicates that useful interventions either direct the search or help the generator commit to an answer.The complementary analysis groups behavior by task family.Source: §H.3 · p. 20, Figure 10 · p. 21

I Performance across τ3 task families

  • The τ3 task-family analysis compares Caddie, prompted critics, and Agent-RRM on identical task subsets, but the supplied passages do not report its comparative outcomes.Tasks are clustered by the user's primary intent, with matched replicates aggregated within each family.Source: Figure 11 · p. 22

J.1 Preserving transactional state on τ3 Retail

  • Retail examples show that preserving transactional state and action order can determine success: returning to the original card or completing an exchange first succeeds, while contrary actions fail.The examples also show that ordering address changes before item changes can preserve both requested updates.Source: §J.1 · pp. 22–23

J.1 Preserving transactional state on 𝜏3 Retail

  • In the 𝜏3 Retail case, one trajectory transfers an actionable cancellation to a human, while another confirms and cancels the order and refunds the original Visa.Source: §J.1 · p. 23

J.2 Preserving itinerary constraints on 𝜏3 Airline

  • In the cited case, the successful trajectory books the cheapest itinerary for $207 and refunds $2,580 to the original gift card, while the other changes the reservation to a more expensive route.Source: §J.2 · p. 24
  • For a two-stage 𝜏3 Airline change, advice calls for validating and performing the upgrade before the nonstop-flight change; the alternative treats the intermediate charge as over budget and transfers the user.The advice also calls for confirming willingness and validating the total cost before finalizing.Source: §J.2 · p. 24

J.3 Committing to retrieved evidence on DeepDive

  • Caddie returns the correct exact titles for both DeepDive tasks and succeeds, while the alternative trajectories pursue false leads and return no valid titles.The cases concern an IEEE service-chain paper and a semiconductor heterostructure paper.Source: §J.3 · p. 24
  • Across the DeepDive cases, Caddie preserves decisive state for the next action, while comparator advice substitutes an unseen value, drops operation order, narrows candidates, or summarizes a stale request.These are the failure modes identified across the cases.Source: §J.3 · p. 24

K Additional Discussion and Limitations

Outcome-based critic training can favor complete solutions when the base model reliably follows them, although this behavior may not establish transferable guidance. The discussion also identifies sampling, environment restoration, critic scale, and training cost as practical considerations.

Additional Discussion and Limitations

  • In self-contained mathematical reasoning, the critic increasingly supplied solutions the base model could follow directly after a few hundred training steps.MuSiQue instead requires gathering missing evidence through retrieval calls.Source: K Additional Discussion and Limitations · p. 25
  • Complete solutions that succeed across continuations receive the largest positive advantages, while consistently failing solutions can receive negative advantages in mixed groups.Repeated updates can reinforce correct solution generation and suppress incorrect solutions; this incentive depends on higher mean rewards, not reduced variance alone.Source: K Additional Discussion and Limitations · p. 25
  • With binary rewards, a correct complete solution always copied by the base model reaches the maximum expected reward of one.In this limiting case, the critic is optimized as a solver, which can improve success without establishing transfer across models or tasks.Source: K Additional Discussion and Limitations · p. 25
  • Advisor Models also study self-contained mathematical reasoning, but their analysis does not establish whether the advisor increasingly supplies complete solutions.Their setup rewards correctness of a single revised response without further tool interaction.Source: K Additional Discussion and Limitations · p. 25
  • With independent continuations, increasing N reduces reward-estimation noise but does not necessarily accelerate convergence.N=5 produced validation curves similar to N=1, with smaller later fluctuations and similar final performance, at additional continuation cost.Source: K Additional Discussion and Limitations · p. 25
  • Branching from shared states requires environment restoration, which can be difficult or costly when tools affect external services.Stateful software-engineering tasks may require checkpointing both files and running processes.Source: K Additional Discussion and Limitations · p. 25
  • Only Qwen3-4B critics were trained, so the effects of critic capacity on training, performance, and transfer remain unestablished.Transfer was evaluated to larger base models, but critic scaling itself was not established.Source: K Additional Discussion and Limitations · p. 26
  • Reaching a performance plateau required approximately 2,000 training steps, with cost depending on the base model and environment.Expensive tool execution, long trajectories, and larger base models can substantially increase CPU and GPU requirements.Source: K Additional Discussion and Limitations · p. 26
Loading 2610.09858v1…