PaperlayerView exact revision on arXiv

Source-linked AI summary

ReSAIL: Mitigating Collapse in Iterative Agent Self-Distillation

Shengjie Jin, Hengbo Xu, Zelong Sun, YuJie Guo, Zhiwu Lu

arXiv:2609.39306v1cs.LGcs.AI

TL;DR

Iterative self-distillation can collapse in deployment performance, while PI-conditioned task success also declines. ReSAIL prioritizes PI-sensitive interaction steps and retains PI-conditioned behavior for future supervision. It yields sustained gains across cycles, including an average 22.5-percentage-point final-cycle success-rate gain over SDPO and OEL baselines.

  • Problem

    Existing iterative self-distillation methods can lose deployment performance across cycles, and PI-conditioned task success also declines.

    Source: §1 · pp. 1–2

  • Method

    ReSAIL augments iterative PI-based self-distillation by selecting PI-sensitive steps, balancing their losses across trajectories, and retaining PI-conditioned behavior.

    Source: §1 · p. 2

  • Results

    Across ALFWorld and TextCraft, ReSAIL gains persist over three cycles and model scales, with an average 22.5-percentage-point final-cycle success-rate gain over SDPO and OEL baselines.

    Source: §1 · p. 2

  • Takeaways & Limitations

    The findings provide evidence that a more robust learning mechanism can mitigate performance collapse in iterative agent self-distillation over deployment trajectories.

    Source: §5 · p. 9

Abstract

from arXiv · show

Iterative self-distillation enables LLM agents to learn from successive deployments, offering a path toward recursive self-improvement (RSI). Yet our experiments with existing methods reveal a collapse in deployment performance across cycles, while task performance with privileged information (PI) also declines. We address this collapse by prioritizing informative interaction steps for distillation and preserving PI-conditioned behavior as the student becomes the next teacher. We introduce Retentive and Selective Augmentation for Iterative Self-Distillation (ReSAIL), a plug-in augmentation for iterative PI-based self-distillation. ReSAIL selects interaction steps where PI most strongly changes the teacher's predictions and balances the resulting distillation losses across trajectories. It also regularizes the student's PI-conditioned output distributions toward those of the frozen teacher at selected and unselected steps to preserve PI-conditioned behavior for supervision in the next cycle. On ALFWorld and TextCraft, ReSAIL sustains substantial gains across model scales over three cycles, with an average absolute gain of 22.5% in final-cycle success rates when added to self-distillation baselines. Sensitivity-guided selection of offline data also improves action prediction accuracy for multimodal GUI agents on AITZ. These findings provide the first evidence that a more robust learning mechanism can effectively mitigate performance collapse in iterative agent self-distillation over deployment trajectories.

1 INTRODUCTION

Iterative self-distillation can collapse across deployment cycles, while PI-conditioned success also declines. ReSAIL targets both patterns by prioritizing PI-sensitive steps and retaining PI-conditioned behavior for future supervision.

1 INTRODUCTION

  • The paper frames iterative experience-based learning as a feedback loop: agents use interaction trajectories to refine decisions, which shapes the experience they later generate.Repeated self-distillation is presented as one path toward recursive self-improvement, but continued improvement is not guaranteed.Source: §1 · p. 1
  • Experiments with SDPO and OEL show that deployment performance can collapse over successive cycles.Figure 1(a) compares both methods with and without ReSAIL on ALFWorld OOD with Qwen3-4B.Source: §1 · p. 1, Figure 1 · p. 2
  • With tasks and PI held fixed, SDPO and OEL also show declining mean PI-conditioned task success across cycles.This motivates preserving PI-conditioned competence as the updated student becomes the next cycle’s teacher.Source: §1 · p. 2
  • ReSAIL is a plug-in augmentation that selects steps where PI most changes teacher predictions, balances selected losses across trajectories, and regularizes the student’s PI-view predictions toward the frozen teacher.Privileged retention applies at both selected and unselected steps to preserve behavior used for supervision in the next cycle.Source: §1 · p. 2
  • ReSAIL improves final-cycle success rates by an average of 22.5 percentage points over SDPO and OEL baselines across ALFWorld and TextCraft, with gains across model scales over three cycles.Sensitivity-guided selection also improves action prediction accuracy for multimodal GUI agents on AITZ.Source: §1 · p. 2

2 RELATED WORK

Prior work reuses agent experience through context or parameter updates, including self-distillation with privileged information and iterative deployment cycles. ReSAIL addresses iterative self-distillation using already collected trajectories.

2 RELATED WORK

  • Agent learning from experience includes context-based reflection and retrieval, as well as parameter updates through supervised fine-tuning or reinforcement learning.The cited approaches reuse past interactions without parameter updates or train on interaction trajectories and environmental rewards.Source: §2 · pp. 2–3
  • ReSAIL combines PI-sensitive selection, trajectory loss balancing, and privileged retention, contrasting with base self-distillation’s uniform averaging over interaction steps.The overview depicts retention on both selected and unselected steps.Source: Figure 2 · p. 3
  • Self-distillation with privileged information transfers behavior conditioned on training-only auxiliary context into model parameters, without requiring that context at deployment.Examples include reference solutions, demonstrations, hindsight feedback, and knowledge extracted from interaction trajectories.Source: §2 · p. 3
  • Unlike an approach that requires additional environment interaction for successful PI-conditioned teacher trajectories, ReSAIL strengthens self-distillation on already collected deployment trajectories.The cited alternative uses off-policy context-distillation with a forward-KL objective on successful trajectories.Source: §2 · p. 3

3 METHODOLOGY

ReSAIL augments offline PI-based self-distillation with selective, trajectory-balanced training and privileged retention. Its setup uses a fixed trajectory dataset, and its joint objective combines both mechanisms.

3 METHODOLOGY

  • The methodology introduces ReSAIL as an offline self-distillation augmentation, then presents its setup, selective distillation, privileged retention, and joint optimization.Figure 2 summarizes the method’s contrast with uniform all-step base distillation.Source: §3 · p. 3

3.1 SETUP AND BASE SELF-DISTILLATION

  • In each cycle, training uses a fixed trajectory dataset without further environment interaction; the ordinary-view student matches the privileged-view teacher using losses averaged over training steps.The updated student deploys without PI and initializes both models in the next cycle; OEL samples training responses per update, whereas SDPO-offline reuses cycle-start responses.Source: §3.1 · p. 4

3.2 TRAJECTORY-BALANCED SELECTIVE DISTILLATION (TBSD)

  • Trajectory-Balanced Selective Distillation scores steps by the frozen teacher’s PI sensitivity, retains the highest-scoring steps, and averages selected losses within each trajectory before normalizing by batch size.This avoids assigning greater total weight to trajectories merely because they contain more selected steps.Source: §3.2 · pp. 4–5

3.3 PRIVILEGED RETENTION (PR)

  • Privileged Retention matches the student’s privileged-view output distributions to the current frozen teacher’s across every step in U, including steps not selected for distillation.It targets PI-conditioned behavior used when the updated student becomes the next cycle’s teacher.Source: §3.3 · p. 5

3.4 JOINT OBJECTIVE AND OPTIMIZATION

  • ReSAIL combines selective distillation and privileged retention in a joint objective, recomputing teacher sensitivity scores and the selected set at each update while holding them fixed during backpropagation.The retention strength is controlled by λ.Source: §3.4 · p. 5

4 EXPERIMENTS

Experiments evaluate ReSAIL over three deployment cycles on ALFWorld and TextCraft, and test sensitivity-guided selection as an offline filter on AITZ. ReSAIL improves final-cycle success, while ablations show distinct roles for selection and privileged retention.

4.2 MAIN RESULTS

  • 7.0–26.1 percentage points for SDPO and 20.3–37.0 points for OEL: ReSAIL improves final-cycle success over each parent method across ALFWorld ID/OOD and TextCraft at both model sizes.On ALFWorld OOD with Qwen3-8B, SDPO+ReSAIL and OEL+ReSAIL reach 75.3% and 78.4% success, respectively.Source: Table 1 · p. 6, §4.2 · p. 6
  • 18.3 percentage points for SDPO and 26.6 points for OEL: the equally averaged advantage over parent methods grows from Cycle 1 to Cycle 3 across six model–benchmark settings.Both augmented methods outperform their parent methods at every reported cycle and maintain non-decreasing mean success across the three cycles.Source: §4.2 · p. 6

4.3 ABLATION STUDIES

  • The ablation shows complementary timing: SGS leads first-cycle performance, TLB improves third-cycle success, and adding PR yields the highest Cycle 2 and 3 success.The full method exceeds TBSD by 17.7 and 21.6 points at Cycle 3, and PR alone by 9.9 and 13.3 points.Source: §4.3 · p. 7, Table 2 · p. 7
  • 12.5 and 11.5 percentage points: top PI-sensitivity selection beats random selection on ID and OOD, while bottom selection performs worse than random on both splits.The comparison fixes selection ratio at ρ = 0.05, with TLB and PR fixed, indicating that the selection criterion matters beyond subset size.Source: §4.3 · p. 7, Table 3 · p. 7
  • 74.2% on ID and 67.2% on OOD: privileged-view retention achieves the highest third-cycle success among the tested retention variants.Both ordinary-view and privileged-view retention substantially improve over no retention.Source: §4.3 · p. 7, Table 3 · p. 7
  • 70.3%: with privileged retention, PI-conditioned success rises from 58.9% at Cycle 1, whereas without retention it falls from 61.5% to 44.8%.At Cycle 3, privileged retention's advantage over ordinary-view retention is 8.1 points with PI and 1.3 points without PI.Source: §4.3 · p. 8, Figure 3 · p. 8
  • 5.4 and 7.0 percentage points: with λ = 0.5, selecting the top 5% of steps improves first-cycle ID and OOD success over using all steps.Across three cycles, success rises as λ increases from 0 to 0.5, while λ = 1 yields lower performance.Source: §4.3 · p. 8, Figure 4 · p. 8

4.4 EFFICIENCY AND GENERALIZATION

  • Approximately 74%: SGS reduces student optimization cost and cuts total training-loop cost from 4.33 to 4.08 GPU-hours while improving first-cycle success.On AITZ, OEL+SGS finishes at 65.95% action accuracy versus 64.89% for OEL, using the same update budget and a fixed top-80% subset.Source: Figure 5 · p. 9, §4.4 · p. 9

5 CONCLUSION

ReSAIL combines PI-sensitive selection, trajectory-balanced distillation, and privileged retention to improve learning within each cycle and preserve PI-conditioned behavior across cycles. Experiments report sustained gains across three deployment cycles and improved AITZ action prediction with sensitivity-guided offline filtering.

5 CONCLUSION

  • ReSAIL improves SDPO and OEL across model scales and sustains gains over three deployment cycles.Ablations and evaluations with fixed privileged information support complementary roles for selection and retention.Source: §5 · p. 9
  • ReSAIL combines PI-sensitive selection and trajectory-balanced distillation with privileged retention across cycles.The method is presented as a plug-in augmentation for iterative experience-based self-distillation.Source: §5 · p. 9
  • Sensitivity-guided offline filtering improves action prediction for multimodal agents on AITZ.Source: §5 · p. 9
  • The findings highlight preserving PI-conditioned competence that the student will use as teacher in subsequent self-distillation cycles.Source: §5 · p. 9

A.1 BENCHMARKS AND DATA SPLITS

The experiments use Qwen3-4B and Qwen3-8B for text tasks and Qwen3-VL-4B-Instruct for AITZ. Table 4 summarizes test sets held fixed across methods and cycles.

A.1 BENCHMARKS AND DATA SPLITS

  • Text experiments use Qwen3-4B and Qwen3-8B with thinking disabled, while AITZ uses Qwen3-VL-4B-Instruct.Source: §A.1 · p. 14
  • Table 4 summarizes the test sets, which are held fixed across methods and cycles.The caption identifies the table's scope as evaluation scope per decoding seed.Source: §A.1 · p. 14, Table 4 · p. 14

A.2 TRAINING DATA AND CYCLES

Text experiments run three cycles, carrying forward final model parameters while resetting optimizer and random states. AITZ uses a shared set of externally collected trajectories, with one run filtering steps by a fixed global sensitivity ranking.

A.2 TRAINING DATA AND CYCLES

  • Text agents run three cycles, carrying forward final model parameters and resetting optimizer and random states each cycle.Source: §A.2 · p. 14
  • Each cycle samples task IDs without replacement and collects one trajectory per task; later cycles use each method's preceding checkpoint to collect a fresh corpus.Methods of the same model size share the Cycle 1 corpus and summaries, while later corpora replace the previous ones.Source: §A.2 · p. 14
  • AITZ training uses 1,728 externally collected complete CoAT trajectories, with a fixed seed sampling and ordering 1,600 trajectories shared across runs.The filtered run keeps a fixed global top 80% of steps before training; only retained steps contribute to its loss.Source: §A.2 · p. 14

A.3 EVALUATION AND REPORTING

Text-task results evaluate final checkpoints after each cycle and report means and standard deviations across three decoding seeds. AITZ monitoring uses a fixed test subset and matched generation settings across methods and checkpoints.

A.3 EVALUATION AND REPORTING

  • Text tasks evaluate each cycle's final checkpoint without intermediate validation or checkpoint selection, reporting means and standard deviations across three decoding seeds.Success rate uses a 30-step interaction limit; invalid actions consume a step and the agent continues from environment feedback.Source: §A.3 · p. 14
  • AITZ monitoring evaluates a fixed random subset of 101 test episodes every ten updates using one fixed decoding seed.Monitoring does not guide hyperparameter changes, early stopping, or checkpoint selection.Source: §A.3 · p. 14
  • AITZ accuracy averages exact-match action scores over non-stop evaluation steps, with invalid predictions scoring zero.Matching follows the official screen-wise rules for the category and associated arguments rather than literal equality of raw arguments.Source: §A.3 · p. 14
  • Generation uses temperature 0.4, top-p 1.0, no top-k truncation, and a 1,024-token cap, with settings and evaluation seeds matched across methods and checkpoints.Source: §A.3 · p. 14

B IMPLEMENTATION AND BASELINE DETAILS

This appendix documents how experience is constructed, how ReSAIL is integrated into training, and how baseline methods are adapted. It also identifies the scope of the detailed experimental results.

B IMPLEMENTATION AND BASELINE DETAILS

  • The training stack uses slime, Megatron-LM, and SGLang, with BF16 model parameters and FP32 optimizer states and main gradients.Megatron data parallelism is degree 8; tensor, pipeline, and context parallelism are degree 1.Source: B IMPLEMENTATION AND BASELINE DETAILS · p. 15

B.1 INPUTS AND EXPERIENCE CONSTRUCTION

  • PI adds a trajectory-derived summary to the ordinary task-and-history input, while deployment evaluation uses ordinary input.Text-task summaries use task snapshots, actions, resulting observations, and outcomes; AITZ extraction also uses diagnostic attempts.Source: §B.1 · p. 15

B.2 SELECTION, RETENTION, AND OPTIMIZATION

  • Each update selects high-sensitivity steps, computes selective distillation and retention losses, and optimizes the student with their weighted sum.Distillation covers selected steps, retention covers all steps in U, and the losses average within trajectories and then across the sampled batch; the AdamW learning rate is 10−6.Source: §B.2 · pp. 15–16, Table 5 · p. 15

B.3 BASELINE ADAPTATIONS

  • Baseline adaptations include ordinary-history cross-entropy for RFT, logged-action matching for GRPO, and cached teacher-response fitting for EPD.Offline SDPO uses cycle-start ordinary responses as cached KL prefixes, with token-level truncated importance sampling shared by baseline and augmented SDPO.Source: §B.3 · pp. 16–17

C DETAILED EXPERIMENTAL RESULTS

  • The detailed-results appendix supplements the main results and primary ablations with full hyperparameter results, AITZ training dynamics, and training-loop cost breakdowns.The reported cost analysis measures the training loop, not the full deployment cycle.Source: C DETAILED EXPERIMENTAL RESULTS · p. 18

C.1 HYPERPARAMETER ABLATIONS

The appendix reports parameter-sweep settings, AITZ training dynamics, and training-loop costs, then examines privileged retention and its evaluation scope. Results indicate that both the strength and conditioning view of retention matter.

C.1 HYPERPARAMETER ABLATIONS

  • The selection-ratio sweep keeps trajectory balancing and retention enabled, so Top-100% is not the unaugmented OEL baseline.The retention-weight sweep fixes ρ = 0.05 across three cycles and removes only retention at λ = 0.Source: §C.1 · p. 18

C.2 AITZ TRAINING DYNAMICS

  • At update 100 on the AITZ monitoring subset, OEL reaches 64.89% accuracy and OEL + SGS (Top-80% steps) reaches 65.95%.The subset contains 843 non-stop steps from 101 test episodes and is not used for model selection.Source: §C.2 · p. 19

C.3 COMPUTATIONAL COST

  • The ALFWorld cost breakdown covers approximate Cycle 1 training-loop costs on eight H800 GPUs, with auxiliary JSD diagnostics excluded.Required SGS scoring and privileged-view passes for PR are included; “Other” is the residual cost, and the scope excludes the full deployment cycle.Source: §C.3 · p. 19, Table 8 · p. 19

D ANALYSIS AND DISCUSSION

  • The appendix's empirical analyses use ALFWorld with Qwen3-4B and treat trajectory-derived summaries as privileged information.PR targets the student's ability to use PI because the student becomes the next cycle's teacher and supplies privileged-view supervision.Source: D ANALYSIS AND DISCUSSION · p. 20, §D.1 · p. 20

D.1 PRIVILEGED RETENTION VERSUS REFERENCE KL REGULARIZATION

  • PR constrains predictions given both interaction history and PI; matching ordinary-view predictions alone does not directly constrain the PI-conditioned predictions.The ordinary-view control and PR match KL direction, steps, averaging, and coefficient; only the conditioning views differ.Source: §D.1 · p. 20
  • For OEL+TBSD at Cycle 3 on OOD tasks, ordinary-view retention improves both input-view means over no retention but decreases success when Experience is supplied; under PR, Experience raises mean success by 3.1 percentage points.Source: §D.1 · p. 20
  • Under privileged retention, Cycle 3 mean success rises from 63.0% to 74.2% on ID and from 49.7% to 67.2% on OOD as λ increases from 0.01 to 0.5.Increasing λ to 1 lowers both means.Source: §D.1 · p. 20

D.2 DUAL-VIEW EVALUATION DETAILS

The dual-view evaluation compares checkpoints with and without fixed Experience under matched task conditions. Table 9 reports mean success rates and sample standard deviations for ALFWorld OOD tasks.

D.2 DUAL-VIEW EVALUATION DETAILS

  • Ten checkpoints are evaluated with and without Experience on the same 128 OOD tasks, using matched initial states and three decoding seeds.The set includes the shared initial model and final checkpoints from Cycles 1–3 for OEL+TBSD with no retention, ordinary-view retention, or PR.Source: §D.2 · p. 20
  • The 128 task-specific Experience summaries are collected once by the frozen initial model and held fixed across configurations, checkpoints, and seeds.Each evaluation episode restarts from its task-specific initial state; only the privileged view receives the summary at each interaction step.Source: §D.2 · p. 20
  • Table 9 reports mean success rates and sample standard deviations over three decoding seeds on 128 ALFWorld OOD tasks.Trained configurations all use OEL+TBSD and differ only in retention; the shared initial checkpoint is listed once.Source: Table 9 · p. 21

D.3 PI-SENSITIVITY MEASUREMENT AND DYNAMICS

This section describes how PI sensitivity is measured and changes across training cycles, along with the prompts and experience-extraction procedures used for text and GUI agents. It also outlines conservative guidelines for turning GUI trajectories into student guidance, using demonstrated correct actions as the authority and focusing on failed steps.

D.3 PI-SENSITIVITY MEASUREMENT AND DYNAMICS

  • The fixed probe uses 16 logged trajectories and 315 interaction steps, measuring response-token-averaged JSD on a consistent Top-20-plus-tail vocabulary support.Evaluations occur at initialization and after each update; the probe is diagnostic, not task-held-out, and all curves use one training seed.Source: §D.3 · p. 21
  • PI sensitivity generally decreases across training, with less pronounced local reversals under ReSAIL than under OEL and SDPO.Adding PR to OEL+TBSD does not uniformly raise endpoint JSD across cycles; these distributional curves do not measure functional competence.Source: §D.3 · p. 22, Figure 6 · p. 22

E.1 TEXT AGENT PROMPTS

  • Text-agent calls serialize prior observations and actions, retain episode history up to 30 steps, omit earlier reasoning, and reinsert TextCraft crafting commands when available.ALFWorld prompts extract the task description from the initial observation and remove help from admissible actions.Source: §E.1 · p. 23

E.2 TEXT EXPERIENCE EXTRACTION

  • Experience summaries use outcome-specific prompts, require a matching heading and four nonempty fields, and are cached for the cycle.Action and observation evidence is capped at 240 and 480 characters, respectively, and the task snapshot is limited to four nonempty lines of at most 240 characters each.Source: §E.2 · p. 27

E.3 GUI AGENT PROMPT

  • GUI calls include the system message, mobile-use tool definition, and a multimodal user message; the two views differ only in the inserted guideline.Both views include prior ground-truth actions and the current screenshot, but exclude earlier screenshots and model predictions.Source: §E.3 · p. 29

E.4 GUI EXPERIENCE EXTRACTION

  • The initial GUI student attempts each logged step under ordinary input, one accepted remark is cached per trajectory, and OEL variants sample fresh ordinary-view responses at each update.Correctness is evaluated using AITZ EM.Source: §E.4 · p. 31
  • Only correct-step actions and explicitly labeled CORRECT actions are ground truth; wrong attempts may appear only as warnings, never as recommended operations.The instructions prohibit using a wrong attempt as positive evidence.Source: §E.4 · p. 31
  • For each trajectory step, identify correct actions or show the failed-step screenshot and its correct alternative, with most detail on failures.The requested format distinguishes correctly handled steps from failed ones and directs attention to the difference between the attempt and the correct action.Source: §E.4 · p. 31
  • Summarize only the demonstrated route, preserve required typed text and action distinctions, and include visible cues and one useful fallback without inventing steps.The instructions also prohibit coordinates and tool calls.Source: §E.4 · p. 31

Related papers

Loading 2609.39306v1…