Progress-Aware Distillation for Mitigating Stagnation in Small Language Model Agents

1 Introduction

• Large-backbone agents achieve high task success, but their token consumption and inference latency pose barriers to practical deployment.Agent distillation transfers capabilities from large models to SLMs through supervised fine-tuning on teacher trajectories.Source: §1 · pp. 1–2
• SFT-distilled agents can repeat failed strategies in unproductive loops, a problem that worsens with longer trajectories and smaller models.The analysis reports stagnation across model families and task domains.Source: §1 · p. 2
• ProD detects and penalizes stagnation in student-generated trajectories, encoding the student–teacher stagnation gap as an adaptive margin.The margin strengthens correction for stagnant behavior while limiting intervention when student progress aligns with the teacher.Source: §1 · p. 2
• Across in-domain and out-of-domain benchmarks, ProD mitigates stagnation while improving success rates and task-completion efficiency for distilled SLM agents.Experiments cover eight SLMs ranging from 0.6B to 8B parameters.Source: §1 · p. 2

2.1 Preliminaries: Agent Distillation via SFT

• SFT distillation trains a student on successful teacher trajectories of thoughts, actions, and observations, calculating loss on teacher-generated tokens while masking observations.The teacher trajectories are collected from teacher-model interactions with a user query and agent instruction.Source: §2.1 · pp. 2–3

2.2 Progress Stagnation in SFT-Distilled Agents

• Turn-level stagnation scores measure whether a thought–action pair revisits earlier turns, using lexical overlap, semantic similarity, or an LLM judge.Lexical and semantic scores compare each turn against all previous turns; analysis focuses on the teacher–student gap because tasks can have different benign overlap.Source: §2.2 · pp. 3–4
• Distilled SLMs stagnate more than the teacher, with the disparity growing as model scale decreases or trajectory length increases.The analysis reports lexical, semantic, and judgment scores; lexical stagnation approaches 0.9, and the three metrics correlate at r > 0.92. The trend also appears across model families, under full-parameter fine-tuning, and on distillation training data.Source: §2.2 · p. 4, Figure 2 · p. 4, Table 2 · p. 4

• Starting from an SFT student, ProD generates student trajectories for dataset queries and compares their progress with teacher trajectories.Source: §3 · p. 5
• The adaptive margin encodes the student–teacher stagnation gap, increasing corrective updates for pronounced stagnation while limiting intervention as student progress approaches the teacher’s.Source: §3 · p. 5
• ProD combines stagnation severity with answer correctness, assigning the largest margins to student trajectories that are both highly stagnant and incorrect.Source: §3 · p. 5
• ProD retains the SFT loss to preserve teacher-trajectory likelihood and can iteratively recompute penalties from fresh student generations.Source: §3 · p. 5

4.1 Experimental Settings

• Eight student models from 0.6B to 8B parameters are evaluated on MATH and HotpotQA in-domain, with additional out-of-domain benchmarks and stagnation measures.Experiments use CodeAct, LoRA, greedy decoding, and a 10-turn inference horizon; Figure 3 compares turn-level lexical stagnation and average semantic and judgment stagnation.Source: §4.1 · pp. 6–7, Figure 3 · p. 7

4.2 Main Results

• 4.7% in-domain and 4.9% out-of-domain accuracy gains over SFT accompany reduced stagnation and interaction length across benchmarks.ProD also improves average accuracy over base models by 29.2% in-domain and 18.5% out-of-domain.Source: §4.2 · p. 7, Table 4 · p. 7

4.3 Comparison with Repetition-Control Methods

• ProD outperforms decoding-time repetition penalties and shortened teacher demonstrations; the latter reduce average trajectory steps from 4.98 to 2.65 but yield minimal gains over SFT.The comparison indicates that these approaches do not constrain the student's own behavior during optimization.Source: §4.3 · pp. 7–8, Table 5 · p. 8

4.4 Effects of Iterative ProD Optimization

• Two ProD iterations further reduce stagnation and improve final-answer accuracy, bringing student progress dynamics closer to the teacher when one round leaves a gap.Each iteration uses the previous student as the new base, regenerates trajectories, and recomputes stagnation gaps and adaptive margins.Source: §4.4 · p. 8, Figure 4 · p. 8

4.5 Ablation Studies

• Removing both SFT loss and adaptive margin produces unstable results and underperforms SFT on Qwen3-4B and Gemma3-4B-it; fixed-margin ProD beats SFT but trails full ProD.The results support complementary roles for teacher-trajectory anchoring and adaptive correction.Source: §4.5 · pp. 8–9, Figure 5 · p. 9
• The best average accuracy is consistently achieved at α = 0.5, which balances stagnation gap and answer correctness; using the stagnation gap alone still improves over SFT.The study compares α = 1, which uses only the stagnation gap, with α = 0, which uses only answer correctness.Source: §4.5 · pp. 8–9, Table 6 · p. 9

Chain-of-Thought Distillation

• Prior chain-of-thought distillation methods shorten or structure teacher reasoning, while Pipis et al. [2025] study looping in single-turn distillation without a dedicated mitigation objective.Examples include concise rationales [Feng et al., 2024], reasoning segmentation [Chen et al., 2025], redundant-step pruning [Wu et al., 2025, Jiang et al., 2025], and counterfactual training [Wang et al., 2023].Source: §5 · p. 9

Agent Distillation

• Agent distillation trains students on interactive teacher trajectories, but the SFT paradigm does not adequately address stagnation during student interactions.The cited paradigm masks environmental observations in the training loss to encourage coherent student thoughts and actions.Source: §5 · p. 9

6 Conclusion

• ProD measures student–teacher behavioral discrepancies and adaptively penalizes stagnation; experiments report higher task success and execution efficiency, while future work targets finer progress metrics and stagnant-action localization.The conclusion states that stagnation worsens with longer interaction horizons and smaller model scales.Source: §6 · p. 9

A LLM-as-Judge Setup for Progress Stagnation

• GPT-4.1 trajectories are scored by an LLM judge from 0.0 to 1.0 starting at turn two, with stagnation defined by repeated failed strategies lacking meaningful progress.The appendix identifies GPT-5.4 as the evaluator and describes score levels from new directions to near-identical repeated actions.Source: A LLM-as-Judge Setup for Progress… · p. 13

B Additional Experimental Details

• Agents use the CodeAct framework with smolagents, and experiments run on one NVIDIA A100 GPU and one NVIDIA H200 GPU.The teacher is gpt-4.1-2025-04-14.Source: B Additional Experimental Details · pp. 13–14

B.1 Hyperparameters

• SFT checkpoints are selected by in-domain accuracy after evaluating up to five epochs; a learning rate of 2 × 10−4 performs best across models and tasks.Three epochs suffice for most models, while Llama3.1-8B and Llama3.2-3B peak after two epochs on MATH and degrade with further training.Source: §B.1 · p. 14
• ProD uses β = 2.5, λ = 1, and default α = 0.5, with its learning rate and initial margin c0 selected using in-domain results.Out-of-domain evaluations use the selected models without additional training.Source: §B.1 · p. 14

B.2 Evaluation Prompts

• For factual and mathematical reasoning, an LLM judge evaluates answer correctness to handle equivalent answers and surface-form variation that strict string matching may miss.The factual prompt accepts meaning-preserving phrasing and aliases but treats verbose answers missing key information as incorrect; the prompts return a JSON explanation and binary score.Source: §B.2 · p. 15

C Evaluation of Progress Stagnation under Full-Parameter Fine-Tuning

• FFT-distilled agents still show pronounced progress stagnation, and Qwen agents sometimes stagnate more than their LoRA-trained counterparts.The authors suggest unrestricted parameter updates may amplify imitation bias and exacerbate unproductive interaction loops.Source: C Evaluation of Progress Stagnation… · p. 15, Figure 6 · p. 16

D Evaluation of Progress Stagnation on the Distillation Training Data

• Students show stagnation on 1,000 sampled training queries at levels similar to those on held-out evaluation sets, while teacher trajectories are compared alongside them.This indicates SFT alone does not give students the teacher’s ability to avoid stagnation, even on distillation data.Source: D Evaluation of Progress Stagnation on… · p. 16, Figure 7 · p. 16

E Gradient Analysis of the ProD Objective

• More-stagnant student trajectories with incorrect answers receive larger margins and gradient weights, while comparable-progress trajectories with correct answers receive smaller weights.The margin captures the student–teacher stagnation gap and directs updates toward unproductive student behavior.Source: E Gradient Analysis of the ProD… · pp. 16–17

F ProD with Alternative Progress Metrics

• On HotpotQA, ProD-Semantic achieves performance comparable to ProD.The authors state this confirms that overall distillation performance is robust to the choice of stagnation metric.Source: F ProD with Alternative Progress Metrics · p. 17, Table 9 · p. 17
• ProD trained with the lexical stagnation score substantially outperforms standard SFT distillation across models and tasks.Source: F ProD with Alternative Progress Metrics · p. 17
• The authors report strong agreement among lexical, semantic, and LLM-as-judge stagnation scores.Source: F ProD with Alternative Progress Metrics · p. 17

G Additional Results on Gemma 3 Models

• ProD similarly reduces stagnation in Gemma3 student agents, further validating the method's generality across model families.Source: G Additional Results on Gemma 3 Models · p. 17, Figure 8 · p. 17
• The paper also reports that ProD alleviates stagnation for Qwen3 and Llama3 agents across lexical, semantic, and LLM-as-judge metrics.Source: G Additional Results on Gemma 3 Models · p. 17
• Figure 8 shows turn-level lexical stagnation on MATH and HotpotQA, plus average semantic and judgment stagnation across turns and datasets.Source: Figure 8 · p. 17

H Training Overhead

• ProD atop SFT incurs additional cost mainly from student rollout generation, lexical stagnation-score computation and ProD training; one rollout per training question is preprocessed once, while inference token usage drops 20.7%.The main experiments use the lexical stagnation-score variant, and all experiments run on a single H200 GPU; Table 10 reports component wall-clock times for two representative models.Source: H Training Overhead · p. 17, Table 10 · p. 18

I Prompt for Concise-Trajectory SFT

• For teacher trajectories with at least four interaction steps, the concise-trajectory SFT procedure prompts the teacher to solve the question again using its successful trajectory as context.The prompt encourages avoiding redundant searches, combining information, and skipping unnecessary steps while requiring at least one tool-based search.Source: I Prompt for Concise-Trajectory SFT · p. 18

J Comparison with Preference-Based Objectives

• Both on-policy SimPO and DPO underperform ProD, with DPO also below SFT; ProD’s adaptive margin accounts for teacher–student stagnation gaps and trajectory severity.The authors attribute the difference to fixed-margin or margin-free objectives’ inability to account for trajectory-specific stagnation, risking overcorrection or under-optimization.Source: J Comparison with Preference-Based… · p. 18, Table 11 · p. 19

K Qualitative Study

• In two HotpotQA cases, SFT agents persist with near-duplicate failed searches, while ProD agents use intermediate evidence or change strategy to escape stagnation and solve the questions.Figure 9 shows a two-hop search guided by intermediate observations; in Figure 10, the ProD agent changes strategy after two unsuccessful trials.Source: K Qualitative Study · p. 19, Figure 9 · p. 19, Figure 10 · p. 20

L Discussion

• The trajectory-level stagnation scores identify teacher–student behavioral gaps but do not distinguish causes such as misread observations, failure to update plans, over-trusting failed tool calls, or insufficient task knowledge.The authors propose finer-grained progress metrics and targeted corrective signals as future work.Source: L Discussion · p. 21
• The implementation uses CodeAct sandboxing with restricted tool access as a basic safeguard, while acknowledging that sandboxing is incomplete and promising usage guidelines and safeguards at release.The discussion notes that smaller, locally deployable agents may interact with external tools, execute code, or retrieve web information.Source: L Discussion · p. 21

NeurIPS Paper Checklist

• The authors report no experimental error bars, citing the high computational and evaluation costs involved.Source: NeurIPS Paper Checklist · p. 25
• The checklist marks societal-impact discussion and responsible-release safeguards as addressed in Appendix L.The checklist answers yes to both questions but does not specify the discussion or safeguards in these passages.Source: NeurIPS Paper Checklist · p. 26
• The paper reports that creators and original owners of used assets are credited, and that it releases no new assets.The checklist marks the existing-assets credit and licensing question yes, and the new-assets documentation question N/A because no new assets are released.Source: NeurIPS Paper Checklist · p. 27
• The paper states that it involves no crowdsourcing experiments or research with human subjects.Source: NeurIPS Paper Checklist · p. 28
• LLMs are used as teacher agents to collect training data and as judges to evaluate task-execution outcomes, with details in Appendix B.Source: NeurIPS Paper Checklist · p. 28