Source-linked AI summary
CAST: Critique-Aware Supervision for Training Reliable Long-Horizon Tool-Calling Agents
Amir Saeidi, Zehua Zhang, Rishitosh Singh, Naman Ahuja, Vivek Gupta, Ali Payani, Gaowen Liu, Jayanth Srinivasa, Chitta Baral
TL;DR
Long-horizon tool-calling agents can fail because individual actions become unreliable across evolving states and repeated trials, while existing critique methods lack systematic action-level supervision. CAST generates structured, step-local verification rationales, trains a critique model, and uses critique-enriched trajectories to optimize policies. Across dynamic tool-calling benchmarks, CAST improves reliability across domains and also reduces inference cost relative to resource-intensive agentic baselines.
Problem
Long-horizon agents can fail from inconsistent actions across evolving states and repeated trials, while existing methods lack systematic action-level verification supervision.
Method
CAST converts sparse trajectory outcomes into structured, step-local verification rationales, trains a critique model, and uses critique-enriched trajectories to optimize the policy.
Results
CAST improves reliability across four dynamic tool-calling domains while reducing inference cost compared with resource-intensive agentic baselines.
Takeaways & Limitations
Learning to critique intermediate actions is a practical path toward more consistent performance in complex long-horizon tool use.
Takeaways & Limitations
CAST uses supervised fine-tuning rather than on-policy interaction with critique feedback and focuses on current-action verification rather than downstream consequences.
Abstract
from arXiv · showhide
Large language model (LLM) agents are increasingly deployed in long-horizon, interactive, and stateful environments. In these settings, a single wrong action, such as refunding the wrong purchase, can cause irreversible task failure and must be intercepted before execution. Such failures may not appear in every single run, but can emerge across repeated trials, making reliability across steps and trials critical. However, ensuring agentic reliability is challenging: even frontier LLMs struggle to explain why an action may be wrong, especially in long, intertwined trajectories governed by domain-specific policies. Much recent work relies on prompt-based critique agents, while optimization-based methods lack a systematic way to produce rich verification rationales for training. We address this gap with CAST, a critique-aware training framework that converts sparse task outcomes into action-level supervision for critique learning and policy optimization. CAST analyzes agent trajectories to synthesize structured rationales explaining action validity under partial observability. The resulting critique model is used to construct critique-aware training data for optimizing the policy model. Fine-tuning Qwen3-family models on dynamic tool-calling benchmarks, CAST improves reliability across domains, outperforming GPT-OSS-120B by over 10% pass^4 on Retail tasks and yielding an additional 9% improvement on Telehealth in an out-of-domain setting. These results demonstrate that critique-aware training improves the robustness of LLM agents in realistic dynamic environments.
1 Introduction
CAST addresses unreliable long-horizon tool use by converting trajectory outcomes into structured action-level critiques for policy training. Across dynamic domains, it improves reliability while reducing inference cost relative to resource-intensive agentic baselines.
- Motivation: Repeated trials expose reliability failures that single-run task success can miss, especially when small mistakes accumulate across long interactions.Dynamic observations, tool outputs, and prior actions can make otherwise identical tasks fail inconsistently.
- Results: 22.6% on pass^1 and 12.94% on pass^4 are the reported in-domain Retail gains of fine-tuned Qwen3 models over base instruct models.Across out-of-domain settings, CAST reports average gains of 3.7% on pass^1 and 5.1% on pass^4.
- CAST: CAST converts trajectory-level outcomes into structured action-level verification signals for training tool-calling agents.The framework is designed to provide supervision for judging individual actions rather than only final task outcomes.
- CAST: Its pipeline generates step-level critique rationales, trains a critique model, and constructs critique-enriched data for policy optimization.The critique model is reused to annotate new interactions before policy training.
- Results: CAST improves agent reliability while reducing inference cost compared with more resource-intensive agentic baselines.The evaluation spans two models and four domains.
2 Related Work
Related work improves language-agent behavior through tool augmentation, textual feedback, actor-critic methods, and generative verification. However, existing approaches face limits in long-horizon settings, including sparse rewards, post-hoc critiques, and deployment costs.
- Tool-Augmented LLM Agents: Tool-augmented agents plan and act across multi-step environments, but long-horizon tool errors can compound into unrecoverable failures.Examples include hallucinated arguments, domain-rule violations, and incorrect tool calls.
- Language-Based Feedback and Self-Correction: Reflexion and Self-Refine use textual critiques for revision, but their feedback is post-hoc and does not formalize information available when actions are taken.Reflexion requires a terminal reward, while Self-Refine requires a finished output.
- Actor-Critic Paradigms with Textual Supervision: Natural-language actor-critic and process-supervised methods use textual feedback or turn-level credit assignment instead of conventional value estimates.Generative verifiers frame reward modeling as next-token prediction with rationalized judgments.
3 Method
CAST learns action-level verification under partial observability and uses critique feedback to train a tool-calling policy. Its three-stage process collects repeated trajectories, creates annotated critiques, and optimizes the policy on successful critique-enriched interactions.
- Problem Definition: A single erroneous action can trigger cascading or irreversible consequences, motivating verification before errors propagate.The verifier must judge actions using only information available at the current step.
- Problem Definition: The action verifier maps current interaction context and a candidate action to a validity label and structured rationale.Accepted actions execute; rejected actions return feedback for revision, without using future observations.
- Overview of CAST: CAST repeatedly samples successful and failed trajectories from the same tasks to capture diverse outcomes in dynamic environments.These trajectories form the buffer used for critique annotation.
- Overview of CAST: A teacher-driven verification procedure labels each action with a rationale and validity decision, using privileged annotation-time information unavailable to the deployed policy and student critic.The resulting critique dataset contains context, action, rationale, and label fields.
- Verification Supervision: CAST classifies action failures as hallucination, domain violation, or wrong tool usage, then explains the verification decision through a structured rationale.Specialized agents evaluate each failure type and an orchestrator aggregates their outputs.
- Critique Model Learning: The critique model jointly learns rationale generation and verification classification, separating critique learning from policy optimization.The objective combines rationale-generation and verification-classification losses.
- Critique-Aware Policy Optimization: CAST-Critic accepts or rejects proposed actions, supplies revision feedback, and produces critique-enriched trajectories for supervised policy optimization.The optimized policy can run with the critic for explicit guidance or alone as a standalone agent.
4 Experiments
CAST improves repeated-run reliability across dynamic tool-calling domains while using critique-aware verification to support practical inference. Its learned critics provide more calibrated and actionable feedback than frontier critics, and smaller policy models can outperform larger models on pass^4.
- Overall results: CAST improves reliability across in-domain and out-of-domain dynamic tool-calling evaluations, with gains strongest on repeated-run pass^4.The evaluation covers Retail, Airline, Telecom, and Telehealth, with Retail used for training and the other domains for transfer.
- In-domain reliability: 18.9% on pass^1 and 10.4% on pass^4: CAST-Policy-4B improves over the Retail base instruct model.CAST-Policy-4B also improves pass^4 over RFT-4B by 4.3%, showing gains beyond training only on successful trajectories.
- Model scaling: 3.4% and 1.7% on pass^4: CAST-Policy-4B and CAST-Policy-8B outperform the best Qwen3-32B setup despite lower pass^1 performance.The comparison indicates stronger repeated-execution reliability for the smaller CAST-Policy models.
- Critic calibration: 46.8% of correct actions are falsely flagged by GPT-4.1-Critic, versus 13.6% and 11.4% for CAST-Critic-4B and CAST-Critic-8B.CAST critics also correctly leave 62.2% and 64.7% of valid actions unflagged, compared with 24.0% for GPT-4.1-Critic.
- Agentic baselines: 20.9, 15.3, and 14.9 percentage points: CAST-Critic-8B exceeds PALADIN on pass^1, pass^3, and pass^4.Against EvoTool, it gains 2.6, 4.5, and 4.3 percentage points on the same metrics.
- Corrective feedback: 82–88% of cases: CAST-Critic feedback leads to successful correction across hallucinations, domain violations, and wrong tool calls.CAST achieves this while using smaller critique agents than the Qwen2.5-72B helpers in IRMA and FAMA, with practical token and latency overhead.
5 Conclusion
CAST is presented as a critique-aware framework for improving reliability in dynamic, long-horizon tool-calling interactions. It generates structured verification rationales, trains critique and policy agents, and improves task success and repeated-run reliability across evaluation settings.
- CAST generates structured verification rationales for agent actions and uses them to train a critique agent and critique-aware tool-calling policy.
- The framework targets failures where plausible intermediate actions later derail long-horizon tasks.
- Experiments across in-domain and out-of-domain settings show improvements in task success and repeated-run reliability.
Limitations
The paper identifies two limitations of CAST: supervised fine-tuning does not directly train policies through on-policy critique interaction, and current critiques do not explicitly model downstream consequences.
- CAST uses supervised fine-tuning rather than directly training the policy through on-policy interaction with critique feedback.Future work could use the learned critique agent as a feedback model for reinforcement learning.
- CAST verifies the current policy action without explicitly modeling its downstream consequences.Future extensions could add forward-looking rationales describing effects on later states or task outcomes.
Ethics Statement
The paper reports using AI assistants for language editing and describes the Qwen3 fine-tuning setup and its hyperparameters.
- AI assistants, specifically Grammarly and ChatGPT, were used to correct grammatical errors and rephrase sentences.
- Qwen-series models were fine-tuned using multi-turn agentic data generated on a supercomputing cluster equipped with NVIDIA H100 GPUs.
- The maximum sequence length for Qwen3 models was set to 32,768.
- Fine-tuning hyperparameters for Qwen3 models are provided in Table 4.
B Details about Dataset
CAST fine-tunes separate critique and policy models using multi-turn agentic trajectories collected from τ-Bench tasks, then trains the policy on successful critique-enriched trajectories.
- CAST fine-tunes two models: a critique model and a policy model.
- Initial trajectory data comes from Qwen3-32B acting in ReAct mode as teacher policy on the 500-task τ-Bench training split, with five trials per task.Qwen2.5-72B serves as the user simulator, and trajectories are annotated using the agentic verification framework.
- After critique-model fine-tuning, the 500 τ-Bench tasks are run again for five trials with teacher actions verified by the learned critique model.
- Only reward-1 trajectories are retained as critique-enriched successful data for training the critique-aware policy.
C Details about Benchmarks
The benchmarks evaluate conversational tool-using agents in realistic, stateful environments, with τ-Trait extending the setting to diverse user behavior and domain shift.
- τ-Bench evaluates tool-using conversational agents interacting with simulated users, domain policies, and backend APIs.
- τ-Bench: The Retail domain contains 115 tasks spanning cancellations, returns, exchanges, address changes, and information requests.
- τ-Trait: τ-Trait preserves multi-turn tool use while adding persona-aware simulation and expanding evaluation to Telecom and Telehealth.
- τ-Trait: τ-Trait is designed to test robustness, personalization, and fairness under realistic behavioral variation.
D Details Results
The detailed-results section includes benchmark results alongside the checker prompts and a complete Retail trajectory case study. The case study contrasts repeated failure by Qwen3-8B with consistent success by CAST-Policy-8B.
- Tables 4–10 report the study’s detailed experimental results.
- Tool Extractor: The tool extractor selects candidate tools by matching tool descriptions to the observation while respecting domain constraints.
- Rule Extractor: The rule extractor returns minimal rules directly relevant to the current observation, emphasizing prerequisites and action constraints.
- Domain Violation Checker: The Domain Violation Checker evaluates policy violations separately from tool-selection errors and does not reflag prior-turn violations.
- Wrong Tool Checker: The Wrong Tool Checker assumes policy compliance and evaluates only whether the selected tool matches the current turn.
- Trajectory Case Study: 0/4 trials failed for Qwen3-8B, whereas CAST-Policy-8B succeeded in all 4/4 trials on the reported Retail task.
F.1 Qwen3-8B Task Trajectory User Agent
The Qwen3-8B trajectory fails after an initial account lookup error and ultimately receives reward 0. Its execution proceeds through authentication and product inspection but exchanges the item before confirmation and uses the original payment method.
- The agent first fails to find the user by email, then successfully locates Sofia Li using her name and ZIP code.
- The retrieved order is delivered and contains a 3x digital camera, making it eligible for the requested exchange.
- Product inspection identifies an available 10x camera variant as the maximum-zoom option.
- The agent executes the exchange before obtaining user confirmation and initially charges the original credit card rather than the requested PayPal account.
- Qwen3-8B fails all four trials of the reported task, receiving 0/4 successes and reward 0.
F.2 CATC-8B Task Trajectory
The CATC-8B trajectory resolves the account and inventory checks, confirms the available 10x option with the user, and completes the exchange using PayPal. It succeeds in all four reported trials.
- The agent locates Sofia Li, retrieves the delivered camera order, and identifies the available 10x replacement option.
- When the user requests approximately 60x zoom, the agent reports that no available model exceeds the 10x maximum.
- The user then confirms the 10x exchange and requests PayPal for the additional payment.
- The completed exchange records the PayPal payment method and a $170.68 price difference.
F.3 CATC-8B+CAST-8B Task Trajectory
The trajectory shows an initially premature exchange action, followed by a confirmed 10x exchange, and a separate critique-verification failure caused by reversed item IDs.
- Initial exchange attempt: The assistant initially selected the 10x camera exchange without explaining that no higher-zoom option existed or obtaining explicit confirmation.The required response was to clarify that 10x was the only available choice and ask whether to proceed.
- Corrected exchange trajectory: The corrected response explained that 10x was the highest available zoom capacity and asked the user to confirm the exchange.The user then confirmed the 10x exchange and supplied PayPal as the payment method.
- Corrected exchange trajectory: After confirmation, the assistant proceeded with the exchange using the specified order, item, replacement item, and PayPal payment method.The order reached an exchange-requested state with replacement item 9228757377 and payment method paypal_8194385.
- Verification failure case: A Retail Task 23 verification failure approved a tool call with the current and target grill item IDs reversed, causing deployment and task failure.The pending order contained item 5946177616, while delivered item 7082455361 was the configuration the user wanted to match; the tool call reversed these roles.
- Verification failure case: The reversed-ID call produced an error because item 7082455361 was not present in the pending order.The critique agent incorrectly verified the proposed call despite the item mismatch.