Source-linked AI summary
Self-Challenging Language Model Agents
Yifei Zhou, Sergey Levine, Jason Weston, Xian Li, Sainbayar Sukhbaatar
TL;DR
Training multi-turn tool-use agents requires large pools of high-quality tasks, but manually creating and annotating them is costly and hard to scale. Self-Challenging has an agent generate Code-as-Task problems with executable verification, then trains an executor on the filtered tasks. Across four environments, it improves Llama-3.1-8B-Instruct substantially using self-generated data, including a 95.8% average relative success-rate improvement on existing test tasks.
Problem
Training generally capable tool-use agents requires many high-quality tasks, while human task creation and annotation are costly, labor-intensive, and not scalable.
Method
Self-Challenging has an agent explore tools as a challenger, generate Code-as-Task problems with executable verification and tests, filter them, and train an executor with reinforcement learning.
Results
95.8% average relative success-rate improvement was achieved on existing test tasks across four multi-turn tool-use environments.
Takeaways & Limitations
Self-generated, automatically verified tasks can support self-improvement of multi-turn tool-use agents without manually creating diverse task, tool, and evaluation data.
Takeaways & Limitations
False negatives from semantic ambiguity or missing information leave a gap between training on SCA tasks and oracle tasks, while improvements tend to remain environment-specific.
Abstract
from arXiv · showhide
Large language models are quickly becoming the foundation for intelligent agents that are capable of using tools. However, training such agents is challenging because it requires human creation and annotation of a diverse set of tasks, tools, and evaluation criteria. In this paper, we propose the Self-Challenging framework for training an agent on high-quality tasks that are generated by itself. The agent first plays the role of challenger and generates a task after interacting with the given tools. The tasks take the form of a novel general class of problems termed Code-as-Task, which are defined by an instruction, a verification function and solution and failure cases which serve as tests, allowing to filter only for high-quality tasks. The agent then takes an executor role and trains on those tasks with reinforcement learning using the evaluation feedback as a reward. Evaluation on two existing multi-turn tool-use agent benchmarks, M3ToolEval and TauBench, shows the Self-Challenging framework achieves over a two-fold improvement in Llama-3.1-8B-Instruct, despite using only self-generated training data.
1 Introduction
Training general multi-turn tool-use agents requires many high-quality tasks, but human task creation is costly and difficult to scale. Self-Challenging addresses this by having an agent generate and verify Code-as-Task problems before using them for training.
- Human creation and annotation of diverse, high-quality tasks remains costly, labor-intensive, and insufficiently scalable for reinforcement-learning agents.
- Self-Challenging assigns the agent challenger and executor roles, using tool interactions to generate tasks and then solve them in the same environment.
- Code-as-Task represents each task with an instruction, verification function, example solution, and failure cases, enabling executable quality checks.
- Automatic filtering removes erroneous or impractical tasks before training, mitigating noise that could contaminate or destabilize executor learning.
- Across four tool-use environments, SCA uses only synthetic tasks and improves Llama-3.1-8B-Instruct by 20.2% absolute average success rate in distillation, while self-improvement doubles success from 12.0% to 23.5%.
2 Related Work
Prior work explores self-play, autonomous evaluation, self-generated instructions, and code-based tool interfaces. The paper targets the unresolved challenge of combining task self-synthesis with quality preservation in open-ended, partially observable tool-use environments.
- Asymmetric Self-Play has succeeded in constrained domains but has not established how to preserve task quality in open-ended, partially observable tool-use environments.
- Autonomous evaluations and self-generated instructions support agent improvement without human supervision, but prior approaches do not fully address general task synthesis for tool-rich settings.
- Code has emerged as a unified interface for LLM agents to call APIs, navigate the web, plan travel, and control robots.
3 Problem Setup
In the standard setting, an executor policy interacts with a tool-containing environment through actions, observes responses, changes the environment state, and ultimately receives verification-based feedback.
- The executor policy outputs tool-calling code or user questions, then appends tool or user responses to its context for subsequent actions.
- Actions change the environment state through a transition function, while the initial observation may include API documentation and task instructions.
- After task completion, a verification method evaluates the resulting environment state and answer to provide task feedback.
4 Self-Challenging Agents
Self-Challenging creates synthetic tool-use tasks through a challenger role and trains an executor on the filtered tasks. Code-as-Task provides executable verification and failure tests, while the framework supports self-improvement and teacher-student distillation.
- 4.2 Code-as-Task: Code-as-Task tasks combine instructions, verification functions, example solutions, and failure cases, then retain tasks whose tests distinguish valid from invalid solutions.
- 4.1 Task Challenger: The challenger explores the environment with tool calls and outputs a task specification containing a verification function.
- 4.2 Code-as-Task: Generated tasks may not match real-world tasks exactly, but shared skills can allow training on synthetic tasks to improve performance on real tasks.
- 4.2 Code-as-Task: If challenger-generated tasks are infeasible or ambiguous, including them in executor training reduces signal-to-noise ratio and contaminates learning.
- 4.3 Task Executor: In self-improvement, the same LLM generates tasks, collects rewarded trajectories, and applies reinforcement learning; with binary rewards, this is effectively rejection fine-tuning on successful trajectories.
- 4.3 Task Executor: Synthetic tasks also support distillation: a stronger model produces trajectories for a weaker student, and failed trajectories may provide useful mistake-related learning signals.
5 Experiments
SCA is evaluated across four tool-use environments in distillation and self-improvement settings, using synthetic training tasks and real out-of-distribution test tasks. Results show strong gains from SCA, while analyses identify benefits and limitations of RL choice, task filtering, and data scaling.
- Distillation Results: +20% average success-rate improvement for SCA demonstrates out-of-distribution generalization from synthetic-task training in the distillation setting.The comparison uses real test tasks despite substantial differences between synthetic and real tasks; Table 1 reports results for four environments and 800 synthetic tasks.
- Self-improvement Results: +11.5% average Pass@1 over the base model and +10.6% over PAE show SCA improves self-improvement across fully and partially observable environments.SCA performs better in Calculation and the three partially observable environments, using synthetic tasks generated by the task challenger and Code-as-Task reward feedback.
- CaT Components: CaT filtering reduces invalid and wrongly labeled tasks by combining executable verification, passing example solutions, and failure-case checks.Passing-task rates decrease from 47.7% with verification only to 9.5% with solutions and 5.2% with full CaT; failure-case filtering removes false positives, while false negatives remain.
- Task Diversity and Scaling: CaT filtering makes task difficulty less diverse for Llama-3.1-8B but preserves the stronger Llama-3.1-70B distribution, while 800 tasks are needed for scaling trajectories to generalize.Train-set gains from more trajectories do not reliably transfer to test performance with 200 or 400 tasks; sufficient task diversity and coverage at 800 tasks supports steady test improvements.
6 Limitations
SCA still produces false negatives because ambiguity or missing information can make tasks difficult to distinguish correctly. Improvements also tend to remain environment-specific rather than transferring to general agentic capabilities.
- False negatives remain a non-trivial limitation because semantic ambiguity or missing information can cause incorrect task judgments.This creates a gap between RL training with SCA tasks and training with oracle tasks.
- SCA mainly improves environment-specific skills, leaving environment-general agentic capability as an open research problem.
7 Conclusion
The Self-Challenging Agent enables an LLM to generate tasks for itself and learn from them through reinforcement learning using Code-as-Task. Across four multi-turn tool-use environments, the method substantially improves success on existing test tasks without manually created training tasks.
- SCA combines self-generated task creation with Code-as-Task and reinforcement learning to support LLM self-improvement.
- 95.8% average relative success-rate improvement was achieved on existing test tasks across four multi-turn tool-use environments.
- The approach reduces reliance on manual creation of diverse tasks, tools, and evaluation criteria, while remaining a preliminary step toward an effective self-improvement loop.
A Additional Environment Details
The evaluation covers calculation, web browsing, retail, and flight-booking environments from M3ToolEval and Tau-Bench. Environment-specific details include observability differences, customer-service interactions, and automated validation of generated tasks.
- M3ToolEval includes a fully observable Calculation environment whose tool information is largely contained in the initial observation.
- Tau-Bench models multi-turn customer service in Retail and Airline environments, where agents query databases and modify records to fulfill user requests.
- The task challenger operates after resetting hidden environment states and uses an automatic checker to validate runnable code, successful solutions, and failing cases.
B Additional Discussions
Additional analyses examine task quality, cross-environment transfer, unsuccessful trajectories, annotation, and broader research context. They show a remaining oracle-task gap, weak aggregate transfer, and potential value in learning from unsuccessful teacher trajectories.
- Cross-environment transfer: Aggregate training across environments provides marginal or worse benefits than training separate models, because SCA mainly addresses environment-specific challenges.
- Task quality versus oracle tasks: SCA-generated tasks improve performance but retain a sub-optimality gap compared with oracle tasks under identical PPO training on 800-task sets.
- Distillation ablation: When the teacher substantially outperforms the student, distillation using successful and unsuccessful trajectories tends to outperform using successful trajectories alone.
- Human annotation: The annotation process classifies trajectories as true or false positives and negatives using task completion, rewards, feasibility, and verifier correctness.
- Broader impact: The broader motivation is to reduce costly human supervision while recognizing that current models still struggle with relatively simple tasks and that safer self-improvement requires further research.
F Compute Usage
Challenge generation and multi-turn rollouts are the main compute bottlenecks, while rejection fine-tuning and evaluation require much less time. Hyperparameter sensitivity also differs across training methods.
- Challenge generation and rollout generation are the main compute bottlenecks because they require multi-turn environment interactions.
- Rejection Fine-Tuning and evaluation take much less time than challenge and rollout generation.
- Rejection Fine-Tuning and DPO are relatively stable across hyperparameter choices, whereas PPO and GRPO require more careful tuning.
H Qualitative Examples for Synthetic Tasks
Qualitative examples compare failure modes across task-generation variants and show that Code-as-Task reduces several problems but still permits underspecified instructions.
- PAE-generated tasks are often ambiguous and non-verifiable.
- Verification-only generation commonly produces infeasible verification functions, while adding an example solution can still allow unsuccessful attempts to pass.
- Code-as-Task can still generate instructions lacking enough information to solve the task, requiring semantic analysis to eliminate.
I Example Interaction of Task Challenger
The task challenger explores tool environments to gather relevant information before generating tasks, with examples spanning retail, airline, calculation, browsing, and evaluation settings. The qualitative examples also expose recurring generation failures across variants.
- The Retail challenger trajectory illustrates exploration of relevant task information before generating a high-quality task.
- Retail challenger prompts redact the order number so the agent cannot know it before logging into the user account.
- Example challenger prompts cover Airline, Calculation, Web Browsing, Retail, autonomous evaluation, and user simulation environments.
- A high-quality TauBench Code-as-Task includes an instruction, verification function, solution, and failure cases, with filtering retaining tasks whose solutions pass and failures do not.
- The illustrated failure modes include ambiguous PAE tasks, infeasible verification functions, weak verification checks, and instructions missing user-account information.