PaperlayerView exact revision on arXiv

Source-linked AI summary

EDGE: Experience-Distillation for Guided Exploration in Agentic Reinforcement Learning

Can Xie, Yuyi Zhou, Wen Yang, Ziyi zhang, Siyao Song, Yingzhuo Deng, Shuo Ren, Jiajun Zhang

arXiv:2608.21946v2cs.CLcs.AIcs.LG

TL;DR

Agentic RL can discard reusable exploration patterns, while inference-time experience methods may ignore policy-dependent utility and require persistent retrieval. EDGE uses retrieved experiences as temporary training scaffolds, admits only positive marginal gains, and distills their behavior into the policy. Across embodied, web, and search-based QA tasks, it improves over RL and experience-augmented baselines while retaining performance without deployment-time scaffolds.

  • Problem

    Agentic RL discards reusable trajectory patterns, while inference-time experience augmentation can apply policy-mismatched guidance and create persistent retrieval dependence.

  • Method

    EDGE partitions rollout groups to estimate experience marginal gains without extra sampling, distills positive-gain behavior via reverse-KL, and co-evolves its experience bank with policy changes.

  • Results

    EDGE improves across embodied, web, and search-based QA tasks, retains 96.0% of scaffolded performance without inference-time experiences, and improves over Search-R1 by 5.9 points on average.

  • Takeaways & Limitations

    Retrieved experience is used as temporary training-time scaffolding whose benefits are progressively internalized into a retrieval-free parametric policy.

  • Takeaways & Limitations

    Evaluation covers two interactive environments and seven search-based QA benchmarks, with limited model scales; broader validation remains future work.

Abstract

from arXiv · show

Reinforcement learning with outcome-based objectives such as GRPO enables LLM-based agents to solve complex, long-horizon tasks, yet the reusable exploration patterns embedded in interaction trajectories are largely discarded after a single policy update. Existing experience-augmented approaches retrieve historical guidance at inference time, but they apply experiences without accounting for the policy's evolving capability and create persistent dependencies on external retrieval. We propose EDGE (Experience-Distillation for Guided Exploration), a framework that treats retrieved experiences as temporary training-time scaffolds and progressively internalizes their benefits into the parametric policy. Concretely, EDGE partitions each rollout group into experience-conditioned and experience-free trajectories to estimate and admit only positive marginal gains without extra sampling, then distills the induced behavior into the base policy via a reverse-KL objective on its own empirical support. A co-evolutionary experience bank further synthesizes guidance from emerging failure modes and prunes obsolete entries as the policy evolves. Across embodied, web, and search-based QA tasks, EDGE improves over strong RL baselines by up to 12.5 points and remains effective without inference-time scaffolds or a proprietary reflector. The code is available at https://github.com/xvolcano02/EDGE.

1 Introduction

EDGE reframes experience reuse as a dynamic transition from temporary training-time scaffolds to parametric policy behavior. It estimates experience utility under the current policy, distills only beneficial guidance, and evolves the experience bank as capabilities change.

  • Motivation: Agentic RL often discards reusable decomposition, recovery, and dead-end-avoidance patterns after a single policy update.This forces rediscovery in sparse-reward, long-horizon environments.
  • Motivation: Inference-time experience augmentation can apply guidance unconditionally and create persistent dependence on external retrieval as the policy evolves.Experience may become redundant or harmful after the behavior is internalized.
  • EDGE framework: The co-evolutionary experience bank synthesizes guidance from failures and prunes obsolete entries as the policy changes.The resulting deployed policy does not require external retrieval.
  • Results: 96.0% of scaffolded performance is retained without external experiences at inference, versus 82.9% for SkillRL and 92.3% for EMPO2.Across ALFWorld, WebShop, and seven search-based QA benchmarks, EDGE also improves over prior methods and exceeds Search-R1 by 5.9 points on average.
  • EDGE framework: EDGE partitions rollout groups into experience-conditioned and experience-free trajectories to estimate marginal experience value without additional environment sampling.Only positive-gain instances are retained for subsequent training.
  • EDGE framework: Gain-gated privileged distillation transfers scaffold-induced behavior into the standard policy through reverse-KL on the student’s empirical support, activating only for positive marginal gain.This is intended to prevent negative transfer.

2 Preliminaries

The paper models LLM agents as sparse-reward, partially observable sequential decision-makers and builds on GRPO’s group-relative policy updates. It motivates guided exploration because unsuccessful rollout groups can make GRPO’s advantage estimates uninformative.

  • Agent formulation: An LLM agent is formalized as a POMDP whose observations are natural-language strings and whose actions are autoregressively generated token sequences.Episodes terminate on task completion or a maximum step limit and produce a binary outcome reward.
  • Agent formulation: A trajectory contains the task instruction, interaction history, observations, actions, and terminal binary reward.The interaction history records prior observation-action pairs.
  • GRPO: GRPO samples G parallel trajectories per task and computes group-normalized advantages.It contrasts outcomes within each group without a learned value function.
  • GRPO: GRPO updates the policy with a clipped surrogate objective and KL regularization using an importance-sampling ratio and reference policy.The coefficient β controls the KL penalty strength.
  • Limitation: In sparse-reward, partially observable environments, unguided exploration can produce groups with few or no successes, making advantage estimates uninformative.This motivates the guided-exploration mechanism introduced next.

3 Method: EDGE

EDGE uses retrieved experience as temporary training-time scaffolding: it validates experience utility, distills beneficial behavior into the policy, and evolves the experience bank as training progresses. The resulting policy can reproduce privileged reasoning without external scaffolds at deployment.

  • EDGE converts retrieved experience from persistent inference-time memory into dynamically validated training-time scaffolding that is discarded at deployment.The framework selectively guides exploration, distills beneficial behavior into policy parameters, and avoids deployment-time retrieval dependence.
  • 3.1 Experience-Guided Exploration Scaffolding: Experience-guided exploration retrieves relevant bank entries and partitions each GRPO rollout group into equal teacher and student subsets without extra rollouts.Teacher trajectories receive privileged context x ⊕ e, while student trajectories use the task instruction x alone.
  • 3.1 Experience-Guided Exploration Scaffolding: A positive marginal gain Δe admits the experience for distillation and pooled advantage estimation, whereas Δe ≤ 0 excludes teacher trajectories and falls back to standard GRPO.The gain compares outcomes under shared policy conditions that differ only in retrieved experience visibility.
  • 3.2 Gain-Gated Privileged Distillation: EDGE uses asymmetric self-distillation: the same policy supplies a privileged teacher and standard-context student, with gradients stopped on the teacher side.The asymmetry is informational rather than architectural, and distillation is activated only when the scaffold yields a verified advantage.
  • 3.2 Gain-Gated Privileged Distillation: Reverse-KL distillation on the student’s own empirical support concentrates the policy on effective scaffold-induced reasoning modes while avoiding out-of-distribution transitions.The student and teacher are compared on identical student-generated action sequences conditioned on different contexts.
  • 3.3 Experience Bank Evolution and Management: The co-evolutionary experience bank expands from contrasting failed and successful trajectories and prunes entries whose EMA utility falls below threshold η.New experiences are deduplicated before insertion, while EMA tracking smooths stochastic utility fluctuations and retires absorbed or conflicting scaffolds.

4 Experiments

EDGE is evaluated across interactive decision-making and search-based QA settings, where it improves performance, retains scaffolded gains after retrieval removal, and benefits from gain gating, distillation, and pruning.

  • Experimental setup: EDGE is evaluated on ALFWorld, WebShop, and seven search-based QA benchmarks using Qwen2.5 and Qwen3 backbones.The setup compares EDGE with closed-source agents, prompting methods, vanilla RL, and experience-augmented training baselines.
  • Main results: 90.4% ALFWorld success and 82.6% WebShop success at 7B improve over GRPO by 8.3 and 12.5 points, respectively.EDGE also exceeds EMPO2 by 1.3 and 5.5 points, with consistent gains at 1.5B.
  • Main results: EDGE’s advantage concentrates on exploration-heavy subtasks, while its WebShop success-rate gain over GRPO exceeds its score gain.The reported WebShop improvements are +12.5 in success rate and +9.3 in score.
  • Generalization: Self-reflection retains 97.3% of GPT-4o-reflected performance and surpasses both baselines on every search-based QA benchmark.With GPT-4o, EDGE leads on five of seven benchmarks, while the results support transfer to a newer backbone without requiring a proprietary reflector.
  • Inference without external scaffolds: 96.0% of scaffolded performance remains after external experiences are removed, compared with 92.3% for EMPO2 and 82.9% for SkillRL.This result is presented as evidence that scaffold-induced behavior is transferred into the parametric policy.
  • Ablation studies: Removing the gain gate reduces overall success to 72.3%, while removing distillation and pruning yields 83.6% and 86.7%, respectively.The ablations associate gain gating with avoiding harmful unfiltered injection, distillation with durable behavioral transfer, and pruning with experience-bank maintenance.
  • Training dynamics: After approximately step 100, GRPO plateaus and regresses while EDGE continues toward 90% validation success and reduces environment steps and trajectory length more rapidly.The analysis attributes EDGE’s behavior to structured exploration guidance combined with gain gating.
  • Experience gain tracking: With a static bank, mean marginal gain becomes negative after roughly step 100, whereas pruning sustains more stable positive gain and shrinks the bank after step 100.Unchecked bank growth beyond 650 entries introduces retrieval noise, while bank shrinkage is described as a signature of internalization.

5 Related Work

Related work spans RL objectives, inference-time experience augmentation, and self-distillation; EDGE instead uses GRPO-style group sampling to validate and internalize currently useful experience.

  • Reinforcement Learning for LLM Agents: Agentic RL has progressed from PPO-based methods to critic-free objectives such as GRPO and RLOO, including turn-level and stepwise credit signals.
  • Reinforcement Learning for LLM Agents: Reusable exploration patterns in RL trajectories are often consumed once and discarded, motivating EDGE’s reuse of group sampling for experience evaluation and transfer.
  • Experience-Augmented LLM Agents: Experience-augmented agents use reflections, persistent retrieval, skill libraries, and selective replay of past reasoning traces.
  • Experience-Augmented LLM Agents: EDGE uses paired-rollout gains to validate and distill online experience while co-evolving its bank through utility-based pruning.
  • Privileged Information and Self-Distillation: Prior privileged-information and self-distillation approaches include LUPI, asymmetric actor-critic methods, context distillation, and on-policy distillation.The cited discussion notes that many approaches are off-policy and can suffer distribution mismatch with the student’s visitation.

6 Conclusion

EDGE treats retrieved experience as temporary training-time scaffolding, validates it under the current policy, and internalizes beneficial behavior for deployment without retrieval.

  • Conclusion: EDGE uses retrieved experience as a temporary training-time scaffold rather than a persistent inference-time dependency.
  • Conclusion: The framework admits only positive-gain scaffolds, distills their behavioral effect into the experience-free policy, and co-evolves the experience bank.
  • Conclusion: Across ALFWorld, WebShop, and seven search-based QA benchmarks, EDGE improves over RL and experience-augmented baselines, with especially large gains on exploration-intensive subtasks.
  • Conclusion: The results suggest that external experience is most useful when dynamically validated, selectively applied, and ultimately internalized.

Limitations

EDGE avoids extra environment rollouts but adds training-time overhead and depends on reward quality. Its empirical evaluation covers two interactive environments and seven search-based QA benchmarks with Qwen2.5 models at 1.5B and 7B.

  • EDGE avoids extra environment rollouts but requires overhead to maintain the experience bank, compare teacher–student behavior, and invoke the reflector LLM.
  • Gain-gating depends on reward quality, so noisy or misspecified rewards reduce the reliability of marginal-gain estimates.
  • The evaluation covers two interactive environments and seven search-based QA benchmarks using Qwen2.5 models at 1.5B and 7B.

A Theoretical Analysis of EDGE

The analysis follows an experience through utility estimation, smoothing, gating, advantage calibration, and on-support reverse-KL distillation. It formalizes when privileged rollouts influence learning and how EDGE limits harmful transfer.

  • Marginal-Gain Estimation and Gating: EDGE estimates experience value as the return gap between privileged and standard contexts using paired teacher and student rollout subsets.
  • Marginal-Gain Estimation and Gating: The empirical marginal gain is unbiased under conditional iid assumptions, with variance determined by the two success probabilities and subset size K.
  • Marginal-Gain Estimation and Gating: EDGE converts the estimated gain into a binary gate that masks teacher rollouts when the experience is not admitted.
  • Utility Smoothing: An exponential moving average reduces utility-estimate variance while trading slower adaptation for greater smoothing.
  • Pooled Advantage Calibration: When active, pooled calibration shifts student advantages downward and teacher advantages upward because privileged rollouts outperform standard-context rollouts.
  • On-Support Reverse-KL Distillation: On-support reverse-KL evaluates teacher behavior only at student-generated prefixes, avoiding teacher-only states and concentrating probability on teacher-preferred actions.
  • On-Support Reverse-KL Distillation: The actor loss combines the RL and distillation pathways when the gate activates, while inactive experiences revert to student-only GRPO.

B Implementation Details

EDGE is implemented as an experience-augmented training system with diverse baselines, an evolving experience bank, and a joint RL–distillation objective.

  • Baselines: The evaluated experience-augmented baselines include SkillRL, EMPO2, EvolveR, and GRPO+Mem0.
  • Baselines: The implementation compares EDGE with proprietary agents, prompting frameworks, GRPO, self-distillation, and experience-augmented training baselines.
  • Training and Retrieval: EDGE is implemented on verl-agent and trained with the joint optimization objective in Eq. 19.
  • Training and Retrieval: Reverse-KL vocabulary loss is approximated using the top-k student tokens, while the experience bank starts empty with zero initial utility scores.
  • Experience-Bank Updates: The bank retrieves a top-m candidate pool and uses its highest-scoring experience as the current rollout scaffold.
  • Experience-Bank Updates: After each step, low-success categories trigger reflection-based expansion with up to three synthesized experiences, while the bank also prunes entries.
  • Hardware: Training and profiling experiments use 8× A100 80GB GPUs.

B.3 Computational Cost

EDGE’s computational cost is measured per training step and includes retrieval, privileged-context computation, and reflection, while its distillation coefficient creates an RL–distillation trade-off.

  • Per-Step Cost: The overhead mainly comes from retrieval, privileged-context forward computation, and reflection, with reflector latency reported separately.
  • Per-Step Cost: 417.72 seconds per step represents a 20.3% overhead over GRPO, while EDGE is 7.8% faster than SkillRL and 6.0% faster than EMPO2.
  • Per-Step Cost: EDGE uses the same rollout budget as the comparison methods and requires no additional environment interactions.
  • Reading the Displays: Table 4 is labeled as an RL hyperparameter table, whereas Figure 5 reports sensitivity to the distillation weight λ.
  • Distillation Sensitivity: At λ = 1, distillation dominates and keeps performance near its initial ∼15%, preventing exploration beyond scaffold-prescribed behavior.
  • Distillation Sensitivity: At λ = 0.01, RL-driven improvement returns, but scaffold-induced patterns are internalized more slowly and convergence remains roughly 5 points below the best setting.

C.2 Case Study: Experience Bank Evolution Dynamics

The experience bank evolves with the policy’s changing failure distribution: broad early scaffolds give way to specialized entries whose utility can recover when new failure modes emerge.

  • Case Study Overview: Training logs trace four cases linking rollout failures to generated experiences, later retrieval, and behavioral changes across bank expansion and refinement.Cases 1–2 represent expansion, while Cases 3–4 represent late-stage refinement and non-stationary utility.
  • Early-Stage Expansion: Early structural failures produce generic, sustained-use scaffolds, driving initial bank growth as the policy addresses destination-first wandering and hallucinated object presence.These failures affect many task variants and correspond to the expansion phase.
  • Late-Stage Refinement: By step 165, coordinating two targets remains fragile, motivating a search-then-collect scaffold for the difficult Pick2 subtask.The scaffold targets object counting, source search, and final placement simultaneously, with final utility U=0.375.
  • Non-Stationary Utility: Entry step_914’s utility recovers from −0.085 at step 165 to 0.452 by step 200 after no-op navigation loops become the dominant Pick2 failure mode.Its utility first hovers near the pruning boundary, showing that entry usefulness changes as the policy’s failure distribution shifts.
  • Bank Evolution: The aggregate bank trajectory reflects initial accumulation followed by contraction toward a compact set of specialized, currently relevant entries.The contraction occurs as the policy absorbs broad patterns and obsolete entries lose relevance.
  • Training Loop and Prompts: Algorithm 1 alternates experience-guided scaffolding, gain-gated privileged distillation, and experience-bank evolution during training.The supplied prompts implement rollout generation and state-aware experience updates for ALFWorld and WebShop.

F Dataset License

This section documents the environments, algorithm specification, and rollout and update prompts used for ALFWorld and WebShop experiments, alongside the stated LLM-use disclosure.

  • Dataset License: Experiments use publicly available ALFWorld and WebShop environments under their licenses and usage terms for academic research.Training, evaluation, and experience-bank construction use trajectories generated from the environments’ task instances and interfaces.
  • LLM Usage Statement: The authors state that an LLM assisted only with editorial phrasing, grammar, clarity, and readability, not research ideas, experiments, or analyses.They also state that scientific contributions and findings are entirely the authors’ work.
  • Algorithm Specification: Algorithm 1 initializes a policy, experience bank, rollout settings, distillation weight, EMA momentum, and pruning and success-rate thresholds.The loop processes tasks in batches and alternates among three EDGE stages.
  • Algorithm Specification: EDGE retrieves a top experience, samples conditioned and experience-free trajectories, computes marginal gain, and applies gain-gated privileged distillation.The algorithm also computes advantages over the rollout group or experience-free trajectories before distillation.
  • Algorithm Specification: Experience-bank evolution updates EMA utility, prunes entries below η, and identifies task categories whose success rate is below ξ.The trained policy is output for deployment without the experience bank E.
  • Prompt Templates: ALFWorld and WebShop rollout prompts contrast retrieved-experience context with experience-free execution based on task information, history, observations, and admissible actions.Separate update prompts compare failed and successful trajectories to generate or revise state-aware experiences tied to environment cues and actions.
Loading 2608.21946v2…