More Methods, Better Agents? A Reality Check on Self-Distillation in Agentic Reinforcement Learning

• On-policy self-distillation supplies denser supervision for student-generated trajectories through a teacher branch conditioned on training-only privileged information.Recent work integrates teacher guidance through auxiliary losses, advantage modulation, or reward shaping.Source: §1 · p. 1
• Published evaluations offer an incomplete basis for judging progress because direct comparisons are scarce, reused baselines may differ in conditions, and stability and cost are less systematically examined.Source: §1 · p. 1
• The authors argue that independent evaluation should take priority over proposing another method to determine what is worth building on.Source: §1 · p. 2
• The benchmark compares ten representative methods with GRPO across four benchmarks and three model sizes, using 132 training runs totaling 12,124.4 recorded H800 GPU-hours.It includes repeated evaluation over eight rollout seeds for pretrained backbones and trained methods.Source: §1 · p. 2
• No method consistently outperforms GRPO across settings; strong final performance can mask unstable training, and added training cost does not consistently improve performance.Source: §1 · p. 2

• Agent research includes inference-time reasoning or verbal feedback, training from annotations or interaction trajectories, and reward-based optimization.This benchmark focuses on capabilities acquired through parameter updates rather than changes to inference procedures.Source: §2 · p. 2
• Distillation transfers teacher predictions, sequences, or behavior; privileged information and reasoning rationales enrich supervision, while learner-visited states can reduce distribution mismatch.MiniLLM and DistiLLM study divergence objectives and sampling strategies, while iterative self-training uses selected self-generated traces.Source: §2 · pp. 2–3
• Table 1 summarizes prior-work coverage of key evaluation dimensions.Source: Table 1 · p. 3
• Agentic self-distillation methods use auxiliary objectives, advantage modulation, reward shaping, or environment-verified teacher continuations.Prior analyses identify supervision inconsistency and privileged-information bias as potential failure modes.Source: §2 · p. 3

• Table 1 summarizes whether source papers address repeated evaluation, training-failure analysis, and measured computational cost.Gaps in coverage limit assessment of reliability and practical value beyond reported performance gains.Source: §3 · p. 3
• Repeated evaluation is rarely reported, making it difficult to distinguish consistent improvements from favorable sampling outcomes.Reporting variability alongside mean performance helps assess claimed advantages.Source: §3 · p. 3
• Training-failure analyses are inconsistent across studies, so final scores may conceal regressions and failure risks.Source: §3 · p. 3
• Measured computational cost remains sparsely reported, limiting comparisons of resources required for competing methods’ gains and their practical value.Source: §3 · p. 3
• None of the surveyed source papers covers all three evaluation dimensions; the benchmark compares consistency of gains, training stability, and computational cost rather than final scores alone.Source: §3 · p. 3

• The benchmark combines shared model–task settings and evaluation procedures with method-specific objectives and hyperparameters informed by source papers and released implementations.Source: §4 · p. 4
• Four benchmarks—ALFWorld, ScienceWorld, WebShop, and Search-QA—are paired with Qwen3.5-2B, 4B, and 9B to create twelve model–task settings.Search-QA uses a fixed seed-0 subset of 3,125 questions, and its Overall metric weights questions equally.Source: §4 · p. 4
• The ten alternatives use auxiliary objectives, privileged signals for credit or reward shaping, or verified teacher continuations; all methods are evaluated without privileged teacher context.Skill-SD and RLSD use skill-bank adaptations, and GRPO+OPSD follows SDAR’s constant-gate implementation.Source: §4 · p. 4
• Each method is trained for 100 global steps in every setting, with validation every five steps; final evaluations use eight rollout seeds and report mean and sample standard deviation.Training progress and stability are assessed using validation-curve AUC, peak-to-final declines, and low-performing outcomes.Source: §4 · p. 4

5 Comparative Performance under Common Evaluation

• 54.26 (+4.58 points) for PCSD and 51.34 (+1.66 points) for AgentOPSD exceed GRPO’s twelve-setting mean of 49.68; the other eight alternatives fall below it.All ten alternatives improve on GRPO somewhere, but isolated gains do not establish broad progress beyond the outcome-only baseline.Source: §5 · p. 4
• Setting-specific leaders vary by benchmark and model size: even PCSD, the aggregate leader, leads no individual setting and beats GRPO in only six.GRPO+OPSD leads WebShop at all three scales; ScienceWorld’s leaders are OVCSD, OCSD, and GRPO at 2B, 4B, and 9B.Source: §5 · pp. 4–5, Table 2 · p. 5
• ICSD wins 7 of 12 settings but averages 44.75 versus GRPO’s 49.68, including 0.00% success on 4B WebShop.GRPO+OPSD wins six settings but averages 43.05, with 1.52% success on 4B ALFWorld.Source: §5 · p. 5

6 Training Progress and Stability

• Validation AUC is normalized by the training-step horizon and averaged equally across twelve settings, while Figure 2 covers all 132 runs and 2,772 validation sweeps.The section also notes that intermediate declines and peak-to-final gaps reveal aspects of stability that aggregate AUC and final scores can each miss.Source: §6 · pp. 5–6, Figure 2 · p. 7
• On 2B ScienceWorld, most methods share an early low-score plateau with overlapping validation curves and little further improvement.Appendix F.3 examines this early stagnation.Source: §6 · p. 6
• PCSD has the highest mean final performance (54.26), but its validation AUC (38.45) is below OCSD’s (39.88); AgentOPSD ranks second in final performance (51.34) but ninth by AUC (34.41).GRPO’s AUC is 36.89, above AgentOPSD’s; across twelve settings, PCSD ranks first by final performance while OCSD ranks first by AUC.Source: §6 · p. 6
• On 4B ALFWorld, AgentOPSD falls to at most 4.3% for six validations before recovering to 79.3%, while GRPO+OPSD ends at 2.1%.An endpoint-only assessment would miss AgentOPSD’s prolonged near-zero phase.Source: §6 · pp. 6–7
• Nine runs finish at least 20 points below their validation peak; OCSD on 4B ALFWorld and ICSD on 9B ScienceWorld remain well below their peaks over their final three validations.OCSD falls from 85.0% to 51.4%; ICSD falls from a task-score peak of 55.23 to 18.77.Source: §6 · p. 7

7 The Cost of Performance Gains

• In 112 of 120 comparisons, methods require more net training time than GRPO; 58 also score lower and seven tie, so added compute often brings no performance gain.Figure 3 plots mean final performance over eight rollout seeds against training time across twelve model–task settings.Source: §7 · p. 8, Figure 3 · p. 8
• Within GRPO’s training-time budget, AgentOPSD’s ALFWorld success falls from 68.10% to 37.63%, while PCSD’s WebShop score falls from 73.17% to 63.83%.Both methods’ budget-matched scores fall below GRPO’s respective 58.10% and 68.20%.Source: §7 · p. 8

8 Discussion

• A shared basis for comparison is needed to clarify which ideas merit further development, and independent evaluation can reveal methods’ strengths, limitations, risks, and costs.The authors prioritize evaluation alongside method development rather than proposing another training method.Source: §8 · p. 9

9 Conclusion

• Across four benchmarks and three model sizes, no method is uniformly superior; strong final results can conceal unstable training, and extra computation does not consistently improve performance.The authors frame these findings as reasons to assess gain breadth, learning stability, and improvement cost together.Source: §9 · p. 9

Limitations

• Limited resources and researcher time prevented an exhaustive evaluation, so the benchmark covers ten representative methods rather than every relevant method.Extending the benchmark would help establish how broadly the observed patterns hold.Source: Limitations · p. 9

Ethical Considerations

• The study is confined to simulated environments and retrieval-based question answering, without human-participant studies or real-user deployment.Its empirical scope therefore does not include human studies or deployment on behalf of users.Source: Ethical Considerations · p. 9

B Method Descriptions and Adaptations

• The eleven methods span auxiliary distillation objectives, advantage reweighting, reward shaping, and environment-verified teacher continuations.Teachers share the student’s model backbone, while privileged information is available only during training.Source: B Method Descriptions and Adaptations · p. 13
• Skill-SD and RLSD use skill-bank adaptations, while GRPO+OPSD adds a constant-gate auxiliary objective independent of teacher scores.For Skill-SD and RLSD, the pre-update actor scores student responses under skill-conditioned prompts.Source: B Method Descriptions and Adaptations · pp. 13–14
• AgentOPSD initializes its belief from clipped mean trajectory-deduplicated returns, while ADRS computes L2 teacher utility over task-grouped interaction rows.These implementations follow the released return-based statistics.Source: B Method Descriptions and Adaptations · p. 14
• ICSD uses an extra gradient-free student-scoring pass for minibatch calibration, whereas PCSD uses a frozen skill-conditioned teacher initialized from the pretrained backbone.ICSD’s scoring computation is included in reported training cost; PCSD reuses the fixed reference model’s weights for its teacher branch.Source: B Method Descriptions and Adaptations · p. 14

C.1 Training Configuration

• The study trains eleven methods for 100 steps on four benchmarks and three model sizes, using common optimization defaults alongside benchmark- and method-specific configurations.Method-specific algorithm hyperparameters remain fixed across model sizes within each benchmark; task rewards and configuration sources vary by benchmark.Source: §C.1 · p. 14

C.2 Teacher Models and Contexts

• Most methods score student responses with the pre-update actor conditioned on skills, while PCSD uses a fixed reference teacher and OCSD compares replay contexts with or without future observations.Skill banks combine general guidance with task-specific instructions; OVCSD uses skill-conditioned inputs for continuations.Source: §C.2 · pp. 15–16

C.3 Numerical Precision and Stability

• Training uses FP32 actor weights with BF16 mixed-precision computation, while final-model evaluation uses BF16 actor and inference-engine weights.The authors specify numerical thresholds for masked means, standardization, belief probabilities, advantage selection, and other method statistics.Source: §C.3 · p. 16

C.4 Validation and Repeated Evaluation

• Repeated final evaluation uses the same 0.4 sampling protocol and eight rollout seeds across four benchmarks, with fixed task sets shared across methods and model sizes.Overall performance averages per-instance metrics over the full evaluation set, giving equal weight to instances rather than batches or categories.Source: §C.4 · p. 17

C.5 Experimental Records

• The experimental records link 132 training runs and 2,772 validation sweeps to logs, from which learning curves, computational costs, and repeated-evaluation summaries are derived.Analysis scripts generate tables and figures from these records and verify metric aggregation against source data.Source: §C.5 · p. 17

D Training Objectives and Algorithms

• The eleven evaluated algorithms comprise GRPO and ten alternatives that change auxiliary objectives, advantage weights, collected trajectories, or reward signals.Shared operations are defined once, with method-specific changes specified separately.Source: D Training Objectives and Algorithms · p. 18

D.1 Shared Training Procedure

• The shared loop samples task groups, collects trajectories, group-normalizes returns into GRPO advantages, and applies PPO-epoch AdamW updates; GRPO sets auxiliary loss to zero.Teacher scores are computed on sampled responses and held fixed through the actor update.Source: §D.1 · pp. 19–20

D.2 Auxiliary Distillation Objectives

• Auxiliary objectives range from constant or confidence-gated skill distillation (GRPO+OPSD, SDAR) and importance-weighted K3 supervision (Skill-SD) to calibrated token weights (ICSD) and window-smoothed teacher-support weights (PCSD).PCSD uses a fixed-reference skill-conditioned teacher; ICSD calibrates trust and influence signals and keeps its resulting weights fixed during each minibatch.Source: §D.2 · pp. 20–22

D.3 Advantage Reweighting

• RLSD schedules teacher-reweighted advantages toward zero, OCSD selects high-NLL turns for full-versus-ablated future-evidence replay, and AgentOPSD uses teacher-informed signed turn credit.AgentOPSD updates group success beliefs from teacher–old-policy log-ratios, then standardizes turn-level belief increments to reweight advantages.Source: §D.3 · pp. 22–23

D.4 Verified Teacher Continuations

• OVCSD verifies skill-conditioned teacher continuations for groups without successful trajectories, pairs teacher and failed-student turns at their first action divergence, and trains on comparative advantages and suffix distillation.Its masks preserve ordinary updates for non-intervened groups and restrict policy-gradient updates for intervened failed groups to accepted divergent turns.Source: §D.4 · pp. 23–25

D.5 Teacher-Based Reward Shaping

• ADRS shapes rewards with normalized teacher-alignment signals and group-level teacher-value advantages, then applies an advantage correction with task-specific normalization rules.Teacher-score normalization differs by task: it uses batch-wide sample SD for ALFWorld, ScienceWorld, and Search-QA, but per-row population SD for WebShop.Source: §D.5 · p. 25
• ADRS adds a gated teacher-reward term before recomputing GRPO advantages, then applies a clipped within-group correction when the shaped-return standard deviation meets a threshold.Reward, TVA, and advantage calculations are detached; the correction is clipped to [−δad, δad].Source: §D.5 · p. 25
• Table S5 aggregates twelve model–benchmark settings using benchmark-specific metrics and reports mean and worst paired differences from GRPO plus win/loss/tie counts.The caption specifies success rate for ALFWorld and WebShop, exact-match accuracy for Search-QA, and task score for ScienceWorld; it supplies no values.Source: Table S5 · p. 26
• Figure S1 compares training-curve AUC rankings and eight-seed final performance across twelve settings, with OCSD and PCSD identified as the AUC and final-performance leaders in its validation trajectories.The figure caption does not provide the rankings or numerical performance values.Source: Figure S1 · p. 26

E.1 All Twelve Settings

• Only PCSD and AgentOPSD improve the twelve-setting mean over GRPO, by 4.58 and 1.66 points; ICSD wins seven settings but averages 4.93 points below GRPO.OCSD improves on GRPO in 2B and 4B ScienceWorld but trails by 14.69 points at 9B, leaving its mean 0.18 points below GRPO.Source: §E.1 · p. 26

E.2 Low-Performance Outcomes

• Fifteen trained models have repeated final-evaluation means below 5; low endpoints include lost pretrained capability and limited progress from weak starting points.One ALFWorld and five WebShop models underperform their pretrained backbones; nine 2B ScienceWorld models score below 2 despite improving from initially negative scores.Source: §E.2 · p. 27

F Training Dynamics and Stability

• Across 132 runs, aggregate comparisons equally weight model–benchmark settings and connect performance summaries to instability, late regression, and low-score plateaus.Metrics remain benchmark-specific rather than forming a pooled task-success probability.Source: F Training Dynamics and Stability · pp. 27–28

F.1 Aggregate Progress and Method Rankings

• OCSD leads training-curve AUC, but PCSD leads final performance; stage leaders shift from SDAR at steps 20 and 40 to OCSD at 60 and PCSD at 80 and 100.The endpoint advantage need not represent performance sustained throughout optimization.Source: §F.1 · p. 28, Table S7 · p. 28

F.2 Intermediate Instability and Late Regression

• On 4B ALFWorld, AgentOPSD recovers after six validations at or below 4.3%, while GRPO+OPSD loses early gains and OCSD declines from 85.0% to 51.4%.A near-zero endpoint screen would miss AgentOPSD’s temporary collapse and OCSD’s late regression.Source: Figure S2 · p. 27, §F.2 · p. 28
• On 2B Search-QA, OVCSD’s greedy accuracy drops 36.67 points despite sampled final accuracy of 33.21 ± 0.33%, versus GRPO’s 32.38 ± 0.28%.ICSD’s greedy accuracy also regresses, peaking at 39.52% before ending at 31.49%, despite leading the sampled comparison at 39.39 ± 0.19%.Source: §F.2 · pp. 28–29

F.3 Early Stagnation on ScienceWorld

• At 2B ScienceWorld, eight methods including GRPO plateau at 0.83, while OCSD and OVCSD reach 13.78 and 13.85; the leader changes to OCSD at 4B and GRPO at 9B.The plateau tests whether methods move beyond limited initial task competence, not only improve agents already learning effectively.Source: §F.3 · pp. 29–30

F.4 Per-Run Training Profiles

• Per-run profiles distinguish peak, final, Tail-3, peak-to-final Drop, and normalized training-curve AUC because these summaries capture different aspects of trajectory behavior.A low Tail-3 indicates persistently weak late performance, while a small Drop does not rule out earlier instability.Source: §F.4 · p. 31

F.5 Task Failure and Numerical Diagnostics

• The below-5 validation threshold identifies 16 runs spanning regression, stagnation, and decoding-dependent outcomes, rather than identifying uniform training collapse.The ALFWorld and WebShop cases also regress below pretrained final-evaluation baselines, while most low ScienceWorld scores reflect stagnation.Source: §F.5 · p. 32, Table S12 · p. 33
• All five low-performing WebShop runs use zero student rollout success and 14.99–15.00 of 15 actions in the final 20 steps; near-limit interaction is not universal across low scores.Low-performing ScienceWorld runs average 12.88–20.00 of 20 turns, while Search-QA OVCSD averages 2.48 of four turns.Source: §F.5 · pp. 32–33
• Only 3 of 16 low-performing runs have nonfinite actor gradients, while 18 of 21 runs with nonfinite gradients finish at or above 5; gradient checks and behavior metrics are complementary.All nine runs with peak-to-final declines of at least 20 points have finite recorded gradients throughout.Source: §F.5 · p. 33

G.1 Performance and Time Budgets

• Only PCSD and AgentOPSD improve mean final performance, with training-time ratios of 1.17 and 1.19; OVCSD costs 1.37× without improving aggregate performance.Nine alternatives take more mean paired training time than GRPO; OCSD is approximately tied at 0.999×.Source: §G.1 · pp. 33–34
• Within matched GRPO time budgets, AgentOPSD loses its ALFWorld advantage and PCSD loses its WebShop advantage, while OCSD retains a smaller ScienceWorld advantage.For 2B OVCSD on Search-QA, the within-budget step-55 score is 36.93% before its late greedy-decoding collapse.Source: §G.1 · p. 35

G.2 Total Compute and Stage-Level Costs

• The 132 runs used 12,124.4 H800 GPU-hours: 10,580.7 for training, 1,515.3 for in-loop validation, and 28.4 for checkpoint saving.The total excludes initialization, initial validation, and subsequent repeated evaluation.Source: §G.2 · pp. 35–36, Table S16 · p. 36
• OVCSD rollout generation averages 218.0 seconds per step versus GRPO’s 124.9, while OCSD advantage computation averages 10.6 seconds versus 0.2.Teacher-related work can occur outside separately timed teacher forward passes; Figure S3 decomposes costs by training stage.Source: §G.2 · p. 37

H.1 Search Strategy and Selection Criteria

• The survey screens 635 records to 47 papers from a 90-day first-submission window, covering distillation for agent action-policy training and checking evaluated methods within that period.The search retrieved arXiv metadata on 23 September 2026 and included full-text inspection where needed.Source: §H.1 · p. 37

H.2 Publication Statistics

• The 47-paper inventory comprises 32 papers on privileged-context policy supervision, 10 on other on-policy distillation, and 5 on trajectory-based policy distillation.The inventory counts each unique arXiv paper once by first-submission date.Source: §H.2 · p. 37