Source-linked AI summary
Revisiting DAgger in the Era of LLM-Agents
Changhao Li, Rushi Qiang, Jiawei Huang, Chenxiao Gao, Chao Zhang, Niao He, Bo Dai
TL;DR
Long-horizon LM agents face state-distribution mismatch and sparse feedback during multi-turn interaction. This paper adapts DAgger with teacher-interleaved rollouts and finds consistent gains over prior post-training methods, including a 3.9-point improvement at 4B on SWE-Bench Verified.
Problem
Existing SFT and RLVR recipes trade off deployment-state coverage against dense supervision, limiting efficient post-training for long-horizon agents.
Method
DAgger collects trajectories from a decaying mixture of student and teacher turns, then trains the student on teacher labels from those visited states.
Results
+3.9 points on SWE-Bench Verified at 4B and +3.6 points at 8B over OPD, with consistent gains across both evaluated splits.
Takeaways & Limitations
DAgger-style state-distribution correction improves software-engineering agents at 4B and 8B scales, with smaller models approaching substantially larger SWE-agent systems.
Takeaways & Limitations
Experiments focus on software-engineering agents under the OpenHands scaffold, and transfer to other long-horizon domains remains unvalidated.
Abstract
from arXiv · showhide
Long-horizon LM agents learn from multi-turn interaction, where a single early mistake can alter the subsequent state distribution and derail the whole trajectory. Existing recipes fall short in complementary ways: supervised fine-tuning provides dense teacher supervision but suffers from covariate shift because it is trained on off-policy teacher trajectories; while reinforcement learning with verifiable rewards avoids this off-policy mismatch by learning from on-policy rollouts but with only sparse outcome feedback. We address this dilemma by revisiting Dataset Aggregation (DAgger) for multi-turn LM agents: the algorithm collects trajectories through a turn-level interpolation of student and teacher policies, and the student is then trained on these trajectories using supervised labels provided by the teacher. By directly interacting with environments, we expose the model to realistic states likely to be encountered during deployment, thereby effectively mitigating covariate shift. Besides, since the student is learned by mimicking the teacher's behavior, it receives rich feedback during learning. To demonstrate DAgger enjoys the benefits of both worlds, we tested the algorithm to train a software-engineering agent with 4B- and 8B-scale student models. On SWE-bench Verified, our DAgger-style training improves over the strongest post-training baseline by +3.9 points at 4B and +3.6 points at 8B. The resulting 4B agent reaches 27.3%, outperforming representative published 8B SWE-agent systems, while the 8B agent achieves 29.8%, surpassing SWE-Gym-32B and coming within 5 points of stronger 32B-scale agents. Together with consistent gains on the held-out SWE-Gym split, these results suggest the effectiveness of DAgger for modern long-horizon LM agents.
1 Introduction
Long-horizon LM agents face a post-training dilemma: SFT offers dense supervision but suffers from covariate shift, while RL uses on-policy rollouts but provides sparse feedback. The paper revisits DAgger, using teacher-interleaved trajectories to supervise students on states they encounter themselves.
- Limitations of Existing Recipes: SFT provides dense, token-level supervision but trains exclusively on expert-induced states, causing covariate shift in long-horizon agentic tasks.This limitation arises because student states encountered during deployment may not be covered by teacher trajectories.
- Limitations of Existing Recipes: RL learns from student rollouts and outcome-level rewards but suffers from sparse credit assignment and expensive group sampling.Advantage estimates can also collapse when sampled trajectories lack diversity in correctness.
- DAgger for LM Agents: DAgger reduces covariate shift by gradually supervising students on self-visited states collected through teacher-interleaved trajectories.Each trajectory is generated through a stochastic mixture of student and teacher policies.
- Application to Software Engineering: Teacher-interleaved DAgger is especially suited to software-engineering tasks, where minor early mistakes can derail multi-turn interactions and leave expert demonstrations without coverage.The method is presented as a practical recipe combining on-policy coverage with dense supervision for these long-horizon settings.
2 Preliminaries
Behavioral cloning trains on teacher-induced states, but deployment under the student policy can cause compounding errors and covariate shift. DAgger addresses this mismatch by aggregating teacher-labeled states visited under student-influenced trajectories, while AggreVaTe uses student-prefix, teacher-completion sampling.
- Behavior cloning and covariate shift: Behavioral cloning minimizes supervised loss on states drawn from the teacher-induced distribution.For discrete actions, the loss is typically cross-entropy.
- Behavior cloning and covariate shift: During deployment, the student follows its own state distribution, where compounding prediction errors can create covariate shift outside the training support.In the worst case, imitation error scales quadratically with the horizon T.
- Dataset Aggregation (DAgger) and AggreVaTe: DAgger generates trajectories with a mixture policy, queries the teacher for actions on encountered states, and aggregates these state-action pairs into its dataset.The mixture coefficient β_i is typically annealed toward zero across iterations.
- Dataset Aggregation (DAgger) and AggreVaTe: DAgger trains the next learner by supervised learning on the aggregated dataset of states it is actually likely to visit.This changes the state distribution used for training relative to behavioral cloning.
- Dataset Aggregation (DAgger) and AggreVaTe: AggreVaTe samples an initial trajectory prefix from the student and has the teacher complete the sequence from a specified intervention point.This student-prefix, teacher-completion protocol is adopted as one sampling option.
3 Methods
The method adapts DAgger to multi-turn LM agents by mixing student and teacher actions during rollouts, collecting teacher labels at visited states, and training the student with cross-entropy. It also frames DAgger and related post-training methods through shared choices of state distributions, label distributions, and sample weighting.
- Rollout Protocols: DAgger-style rollouts independently select the teacher at each turn with probability β_i, which decays toward 0 across iterations.This turn-level mixture increasingly exposes the student to its own deployment-time states.
- Rollout Protocols: AggreVaTe-style rollouts let the student control a sampled prefix of length κ before the teacher completes the trajectory.The prefix distribution ρ_i is scheduled so student prefixes grow over training.
- Teacher-Labeled Data: Every visited state receives an expert action label, regardless of whether the executed action came from the teacher or student.The training batch pairs each execution-trace state s_t with an expert label ã_t.
- Training Objective: The student is optimized with cross-entropy against the logged expert-provided labels collected after each rollout.Expert actions may consist of multiple tokens, and shared-prefix transitions can be packed with a loss mask for equivalent gradients.
- Unified Post-Training View: A unified objective compares post-training methods through their context distribution p_s, label distribution p_a, scoring function w(s, a), and optional regularizer Ω_i.DAgger uses rollout-induced student–teacher context distributions while retaining the expert as the label source to mitigate covariate shift and provide information-rich feedback.
4 Experiments
Experiments evaluate DAgger-style training across model scales, matched post-training baselines, training stability, covariate shift, and agent behavior on SWE-Gym and SWE-Bench Verified. DAgger-style training achieves strong task-resolution gains while improving early training and limiting trajectory-distribution shift.
- Experimental setup: The study trains Qwen3-4B-Instruct-2507 and Qwen3-8B students with a fixed Qwen3-Coder-30B-A3B-Instruct teacher on SWE-Gym, including a 100-instance held-out split.Trajectories are generated and evaluated with OpenHands.
- Main results: 17.0% on SWE-Gym Holdout and 27.3% on SWE-Bench Verified are achieved by the 4B DAgger-style model, improving over OPD by +1.0 and +3.9 points, respectively.These results use the matched OpenHands scaffold and SWE-Gym training data.
- Training stability: At 3K effective samples, DAgger-style training reaches 12% on SWE-Gym Holdout and 20% on SWE-Bench Verified-100, versus 9% and 13% for on-policy distillation.AggreVaTe-style shows a similar early advantage at 13% and 20%.
- Covariate shift: DAgger-style and AggreVaTe-style training remain nearly flat around 0.10 after the initial drop, yielding roughly a 20% reduction relative to SFT at the largest sample budget.SFT’s reverse KL later rebounds to about 0.126 after initially dropping from roughly 0.27 to 0.10.
- Agent behavior: SFT raises submission rates to 97.6% and on-policy distillation to 98.5%, but SFT has the highest syntax/runtime error rate among trained methods at 20.3%.DAgger-style no-submission failures are dominated by context overflow at 57.5%.
5 Related Works
Prior work establishes LLM agents as capable software-engineering systems operating through developer-like interactions, while post-training methods trade dense supervision against sparse but on-policy verifiable rewards.
- Coding Agent and SWE Tasks: LLMs have demonstrated capabilities in code generation, software testing, and automated debugging, and are increasingly embedded in agentic frameworks that use tools and modify code.These frameworks reason over goals, execute commands, and interact with realistic repositories.
- Coding Agent and SWE Tasks: SWE-Bench evaluates repository-level issue resolution using real GitHub issues and unit tests, highlighting the difficulty of long-horizon software maintenance.Related benchmarks, training environments, and scaffolds such as SWE-agent and OpenHands support developer-like interactions including file navigation, code editing, command-line execution, and testing.
- LLM Agent Post Training: Verifiable-reward post-training is prominent in SWE because unit tests provide automatic outcome signals, but sparse rewards limit credit assignment over long trajectories.This motivates denser supervision such as rubric-based feedback and on-policy distillation, which improves over offline SFT by training on student-generated data.
6 Conclusion
The paper revisits DAgger for multi-turn LM agents to address train-test state-distribution mismatch from compounding local errors, combining on-policy coverage, teacher-guided recovery, and dense supervision. Experiments show consistent gains over SFT, GRPO, and on-policy distillation at 4B and 8B scales.
- 6 Conclusion: DAgger targets train-test state-distribution mismatch caused by compounding local errors during long-horizon tool interactions.The method is designed for multi-turn LM agents.
- 6 Conclusion: Teacher-interleaved mixture rollouts combine on-policy state coverage with teacher-guided recovery and dense supervised feedback.These components are presented as the method’s central mechanism.
- 6 Conclusion: Experiments with software-engineering agents show consistent gains over SFT, GRPO, and on-policy distillation at both 4B and 8B scales.The reported comparison covers both student-model scales.
A Derivation of the Unified Post-Training View
The section presents a unified post-training formulation in which algorithms repeatedly construct weighted supervised datasets over agent contexts and actions, differing in state distributions, label sources, and sample weights. Under this view, DAgger combines increasingly student-induced states with teacher action labels to reduce train-test mismatch while retaining supervised learning.
- Unified formulation: Post-training methods share a weighted supervised-update structure over agent contexts and turn-level actions, differing mainly in context distributions, action labels, and sample weights.Contexts include interaction prefixes with messages, tool calls, and environment observations; stop-gradient treats data and weights as fixed during policy updates.
- SFT and RL: SFT uses expert-induced contexts and expert labels with unit weights, whereas RL uses student-induced contexts and actions weighted by advantage estimates.For SFT, ps = dπe and pa(· | s) = πe(· | s); for RL, ps = dπθi, pa = πθi, and w(s, a) = Ai(s, a).
- On-policy distillation: OPD trains on student-induced contexts and student-sampled actions using a teacher-based distillation signal, which can allocate supervision to low-quality or prematurely failed trajectories early in training.Its unified weight is log πe(a | s) − log πθi(a | s).
- DAgger-style rollout: DAgger executes each turn with the teacher at probability βi and the student at probability 1 − βi, then labels every visited state with the teacher action.As βi decays, contexts shift from mostly teacher-induced toward more student-induced states while retaining uniform teacher supervision.
- AggreVaTe-style rollout: AggreVaTe uses a trajectory-level mixture by letting the student execute a sampled prefix before the teacher completes the trajectory, preserving teacher labels on all visited states.Compared with turn-level DAgger mixing, it exposes the teacher to longer contiguous student-induced prefixes and provides a simpler trajectory-level intervention.
B Limitations · C Potential Social Impact · D Ethical Statement
The paper limits its evidence to software-engineering agents under the OpenHands scaffold and notes dependence on stronger teacher policies. It describes potential benefits of more reliable agents alongside risks from misuse, insecure outputs, and deployment without oversight.
- B Limitations: The experiments focus on software-engineering agents under the OpenHands scaffold, so transfer to web navigation, data analysis, and scientific computing remains unvalidated.The paper identifies these domains as directions for further validation.
- B Limitations: DAgger-style training relies on a stronger teacher policy to provide action labels on visited states, making effectiveness dependent on teacher quality and availability.The passage also begins to identify additional dependencies, but the supplied text is truncated.
- C Potential Social Impact: More capable software-engineering agents could help developers localize bugs, repair code, maintain open-source projects, and reduce routine debugging and maintenance costs.The paper frames these as potential positive societal impacts.
- C Potential Social Impact: By mitigating covariate shift and improving agent stability, DAgger-style training may make tool-using systems less brittle during deployment-time interactions.This is presented as a potential benefit rather than a guaranteed outcome.
- C Potential Social Impact: More capable code-editing agents could be misused to automate harmful software modifications, discover exploitable vulnerabilities, or scale malicious development workflows.The passage identifies these as potential negative societal impacts.
- C Potential Social Impact: Without sufficient review and testing, agents may introduce subtle bugs, insecure code, or incorrect patches even in benign settings.The paper emphasizes that stronger smaller-model agents can also create deployment risks.
- D Ethical Statement: The study evaluates long-horizon LLM agents in controlled software-engineering benchmarks using publicly available datasets, models, and evaluation protocols.The proposed method is intended for research on post-training algorithms.
- D Ethical Statement: The method is not designed for autonomous deployment in high-risk or security-critical software systems, where incorrect, insecure, or harmful code requires oversight.The paper also warns that stronger coding agents could be misused to scale malicious software development.
E Dataset and Task Details
The study uses SWE-Gym as the primary training corpus for agent post-training methods and SWE-Bench Verified as the final out-of-domain evaluation benchmark. SWE-Gym tasks require repository inspection, code modification, and patch submission, while SWE-Bench Verified evaluates whether patches satisfy issue descriptions and hidden regression tests.
- SWE-Gym: SWE-Gym provides real GitHub issue tasks requiring agents to inspect repositories, localize bugs or missing functionality, edit source files, and submit patches.It is used as the primary training corpus for SFT, OPD, DAgger-style, and AggreVaTe-style methods.
- SWE-Bench Verified: SWE-Bench Verified is the final out-of-domain benchmark, measuring task-resolution rate under the OpenHands scaffold.Its instances require generated patches to satisfy issue descriptions and pass hidden regression tests.
F Experimental Details · F.1 Overall Experimental Configuration
The experiments use a shared rollout-update setup across DAgger-style and AggreVaTe-style methods, with DAgger progressively increasing student-induced states. Training and evaluation use fixed decoding, context, interaction, and task-resolution conventions, while AggreVaTe shifts toward longer student prefixes and teacher completion.
- F.1 Overall Experimental Configuration: Each iteration collects 512 fresh mixed-policy task instances, updates the student on teacher-labeled data, and trains for 5 iterations at learning rate 3 × 10−6.This optimization and rollout-update configuration is shared by DAgger-style and AggreVaTe-style methods.
- F.1 Overall Experimental Configuration: DAgger starts with β1 = 1.0 and decays the teacher-mixture coefficient by 0.2 per iteration to a floor of 0.6.The schedule keeps early trajectories strongly teacher-guided while increasing student-induced states over training.
- F.1 Overall Experimental Configuration: AggreVaTe bases its student-prefix support on the observation that student rollouts typically terminate within about 40 turns.The supplied passage introduces the prefix-length distribution but does not include its displayed formula.
- F.1 Overall Experimental Configuration: AggreVaTe shifts probability mass toward longer student prefixes over training, after which the teacher completes the remaining trajectory for recovery from student-induced states.This provides trajectory-level recovery after the sampled student prefix.
- F.1 Overall Experimental Configuration: Rollouts use temperature 0.7 and top-p = 0.9, whereas evaluation uses greedy decoding across all methods, including baselines.These settings apply to rollout collection and evaluation respectively.
- F.1 Overall Experimental Configuration: Training and evaluation use a maximum context length of 64K tokens and allow at most 100 environment interactions per trajectory.Unless otherwise specified, reported results use task-resolution rate as the evaluation metric.
F.2 Training Implementation Details · G Prompt Templates
The paper specifies shared compute and filtering procedures for post-training, then fixes the OpenHands scaffold, tools, system instructions, and task prompt used for SWE-agent training and evaluation.
- F.2 Training Implementation Details: All 4B and 8B experiments perform rollout collection and model training on 4 A100 GPUs.
- F.2 Training Implementation Details: SFT retains 684 successful teacher trajectories whose final patches pass executable tests, then trains the student for 3 epochs with batch size 16.
- F.2 Training Implementation Details: DAgger-style, AggreVaTe-style, and OPD share optimization settings, while collecting 512 trajectories per online iteration and filtering invalid final patch submissions.
- G.1 OpenHands System Prompt: The OpenHands system prompt defines the agent role, command and code-editing behavior, file-system constraints, version-control rules, and problem-solving workflow.It emphasizes thorough repository exploration, focused changes, testing when feasible, and verification.
- G.1 OpenHands System Prompt: The prescribed workflow covers exploration, analysis, testing, implementation, and verification, with troubleshooting guidance for repeated failures and major execution issues.The prompt requires considering multiple approaches, testing real code paths, making minimal changes, and consulting the user when setup or plans require it.
- G.2 OpenHands Tool Specifications: The OpenHands action space contains execute_bash, str_replace_editor, and finish tools for running commands, viewing or editing files, and reporting completion.The specifications include command timeouts, exact-match replacement requirements, file-path parameters, and a completion message.
- G.3 Initial User Message: Each SWE task’s initial user message places the repository at /testbed, supplies the issue description, prohibits test changes, and requests minimal edits to non-test files.