PaperlayerView exact revision on arXiv

Source-linked AI summary

Experiential Reinforcement Learning

Taiwei Shi, Sihao Chen, Bowen Jiang, Linxin Song, Longqi Yang, Jieyu Zhao

arXiv:2602.13949v1cs.LGcs.AI

TL;DR

Language models struggle to learn corrective behavior from sparse, delayed environmental feedback. ERL adds an experience–reflection–consolidation loop that guides revised attempts and internalizes successful corrections, yielding stronger learning efficiency and final performance across control and agentic reasoning tasks.

  • Problem

    Sparse and delayed rewards require language models to infer how failures should translate into corrective behavior, making learning unstable and sample-inefficient.

  • Method

    ERL uses feedback to generate reflection-guided second attempts, then selectively internalizes successful behavior into the base policy through self-distillation.

  • Results

    ERL consistently outperforms RLVR across six evaluated settings, with gains up to +81% in Sokoban, +27% in FrozenLake, and +11% in HotpotQA.

  • Takeaways & Limitations

    Embedding structured experiential revision into training provides a mechanism for translating feedback into durable behavioral improvement without reflection at inference.

  • Takeaways & Limitations

    In Olmo3-7B-Instruct Sokoban, the no-memory variant slightly outperforms full ERL, suggesting persistent memory can propagate inaccurate reflections in some complex or stochastic settings.

Abstract

from arXiv · show

Reinforcement learning has become the central approach for language models (LMs) to learn from environmental reward or feedback. In practice, the environmental feedback is usually sparse and delayed. Learning from such signals is challenging, as LMs must implicitly infer how observed failures should translate into behavioral changes for future iterations. We introduce Experiential Reinforcement Learning (ERL), a training paradigm that embeds an explicit experience-reflection-consolidation loop into the reinforcement learning process. Given a task, the model generates an initial attempt, receives environmental feedback, and produces a reflection that guides a refined second attempt, whose success is reinforced and internalized into the base policy. This process converts feedback into structured behavioral revision, improving exploration and stabilizing optimization while preserving gains at deployment without additional inference cost. Across sparse-reward control environments and agentic reasoning benchmarks, ERL consistently improves learning efficiency and final performance over strong reinforcement learning baselines, achieving gains of up to +81% in complex multi-step environments and up to +11% in tool-using reasoning tasks. These results suggest that integrating explicit self-reflection into policy training provides a practical mechanism for transforming feedback into durable behavioral improvement.

1 Introduction

ERL addresses the difficulty of learning corrective behavior from sparse, delayed rewards by embedding experience, reflection, and consolidation into reinforcement learning. Across control and agentic reasoning settings, it improves training efficiency and final performance while internalizing reflection-driven gains.

  • Motivation: Sparse and delayed outcome rewards make it difficult for language models to infer corrective behavior, especially in multi-step agentic tasks.These settings can amplify small errors and obscure credit assignment.
  • Approach: ERL embeds an explicit experience–reflection–consolidation loop inside reinforcement learning.The model generates an initial attempt, uses environment feedback to produce structured reflection, and guides a refined second attempt.
  • Approach: Successful corrections are internalized into the base policy, allowing improved behavior to persist without reflection during deployment.This mechanism consolidates reflection-driven improvements into the deployable policy.
  • Results: ERL consistently outperforms RLVR across six sparse-reward control and agentic reasoning settings spanning two model scales.The evaluation covers both control environments and reasoning benchmarks.
  • Results: +81% in Sokoban, +27% in FrozenLake, and +11% in HotpotQA are reported gains for ERL over RLVR.These results are presented across the evaluated control and reasoning tasks.

2 Experiential Reinforcement Learning (ERL)

Experiential Reinforcement Learning (ERL) embeds reflection and consolidation within reinforcement learning, turning feedback into structured behavioral corrections. Its loop combines selective retries, reflection memory, reward-driven updates, and distillation so successful improvements persist in the base policy.

  • Training loop: The training loop samples an initial response, obtains feedback and reward, generates a reflection, and uses it to produce a second response.The second attempt receives new feedback and reward, while the reflection is assigned the second attempt’s reward.
  • Selective revision: ERL selectively triggers reflection and retry when the first attempt underperforms a reward threshold, concentrating computation on trajectories likely to benefit from revision.Successful corrective patterns can be stored in cross-episode memory for reuse.
  • Optimization: Reinforcement learning updates the first attempt, reflection, and second attempt using their associated rewards and advantage estimates.The objective preserves reward-driven policy optimization while operating over a richer trajectory structure.
  • Internalization: Selective distillation trains the policy to reproduce successful second attempts from the original input alone, removing reflection context at deployment.This internalization step is intended to preserve corrections when feedback and reflection are unavailable.
  • Comparison to standard RLVR: Compared with RLVR’s scalar endpoint feedback, ERL uses explicit behavioral correction to reduce reliance on undirected exploration while remaining compatible with standard reinforcement learning.The method combines reflection-guided retries, memory, reinforcement learning, and distillation into a self-improvement loop.
  • Core idea: ERL adds an explicit experience–reflection–consolidation stage to reinforcement learning, converting feedback into a reflection that guides a refined attempt.The refined trajectory is reinforced and later internalized through selective distillation.

3 Experiment

The experiments compare ERL with standard RLVR on sparse-reward control and agentic reasoning tasks across two language-model scales. The evaluation uses Frozen Lake, Sokoban, and HotpotQA under interaction-focused task settings and a shared policy-gradient optimizer.

  • Evaluation tasks: ERL is evaluated against standard RLVR on three agentic reasoning tasks: Frozen Lake, Sokoban, and HotpotQA.The tasks span sparse-reward control environments and tool-assisted multi-hop question answering.
  • Sparse-reward control: Frozen Lake and Sokoban provide reward only at episode completion, with +1 for success and 0 otherwise.The model receives no explicit game rules or environment dynamics and must infer task structure through interaction.
  • Agentic reasoning: HotpotQA is evaluated as an agentic multi-hop retrieval task, with correctness measured by token-level F1 against ground-truth answers.The model performs iterative tool-assisted retrieval before producing a final answer.
  • Models and optimization: Olmo-3-7B-Instruct and Qwen3-4B-Instruct-2507 are trained with both RLVR and ERL using GRPO as the underlying policy-gradient optimizer.Training also uses clipping, KL regularization, and importance sampling for stability.

4 Result and Discussion

ERL improves final reward and training efficiency across sparse-reward control and agentic reasoning tasks, with benefits linked to structured reflection and cross-episode memory. Ablations indicate that reflection is especially important, while memory generally supports cumulative refinement but can sometimes propagate inaccurate reflections.

  • Performance Across Tasks: ERL improves final evaluation performance over RLVR across all tasks and both model backbones.The evaluation spans FrozenLake, Sokoban, and HotpotQA.
  • Performance Across Tasks: Qwen3-4B-Instruct improves from 0.06 to 0.87 on Sokoban, while Olmo3-7B-Instruct improves from 0.04 to 0.20.Sokoban produces the largest reported effect and requires long-horizon planning and recovery from compounding errors.
  • Learning Efficiency: ERL reaches higher validation reward earlier and maintains a persistent margin over RLVR across tasks and models.The acceleration is especially pronounced in FrozenLake and Sokoban.
  • Mechanistic Role of Reflection: Post-reflection trajectories achieve higher training reward than both RLVR and pre-reflection trajectories.The within-episode comparison indicates that reflection guides improved second attempts after feedback.
  • Ablation Study: Memory and Reflection Mechanisms: Full ERL generally performs best, followed by no-memory, while no-reflection shows the largest degradation in most settings.Removing memory slows convergence, whereas removing reflection substantially reduces learning speed and final reward.
  • Ablation Study: Memory and Reflection Mechanisms: In Olmo3-7B-Instruct Sokoban, no-memory slightly outperforms full ERL.The authors suggest persistent memory can propagate early inaccurate reflections in complex or stochastic settings.

5 Related Work

Related work frames reinforcement learning as a route beyond static imitation and emphasizes learning from agent-generated experience. This motivates mechanisms that convert interaction failures into usable learning signals.

  • Reinforcement Learning for LLMs: Reinforcement learning research for LLMs has expanded from preference alignment toward mathematical reasoning and verifiable rewards.The passage situates ERL within broader efforts to improve language-model capabilities through reinforcement learning.
  • Learning from Experience: Learning-from-experience work argues that continual agent-generated data and long-horizon decision-making may extend beyond imitation of human corpora.This perspective motivates algorithmic mechanisms for converting failures into usable learning signals.

6 Conclusion

The conclusion presents ERL as a training loop that converts environmental feedback into structured behavioral correction through reflection and consolidation. Across control and reasoning tasks, it improves learning efficiency and final policies while preserving deployment without reflection at inference time.

  • Conclusion: ERL incorporates an explicit experience–reflection–consolidation stage into reinforcement learning to convert feedback into structured behavioral correction.The framework pairs reflection-guided revision with selective internalization.
  • Conclusion: ERL improves learning efficiency, stabilizes optimization, and produces stronger final policies relative to standard reinforcement learning baselines.The reported scope covers sparse-reward control and agentic reasoning tasks.
  • Conclusion: The resulting deployable policy operates without reflection at inference time.Corrective strategies are learned during training and consolidated into the base policy.

A Full Algorithm and Gated Reflection

ERL uses an explicit experience–reflection–consolidation loop, with gated reflection on unsuccessful first attempts and mechanisms for memory and policy internalization.

  • Gated Reflection: Gated reflection triggers a refined second attempt only when the first-attempt reward is below threshold τ.This preserves successful trajectories as purely on-policy while applying reflection to failed or suboptimal trajectories.
  • Gated Reflection: Reflection on successful attempts caused unstable training, reward hacking, and weaker generalization in early experiments.The authors attribute this to instance-specific shortcuts and domination of the optimization signal by off-policy second attempts.
  • Memory Extensions: ERL stores successful reflections in plain-text system-prompt memory and could retrieve relevant past reflections before later reflection steps.The memory mechanism is presented as extensible toward structured agentic memory.
  • On-Policy Distillation: An on-policy reverse-KL objective can train the deployment policy to match a contextual policy that has access to reflection and memory.This provides an alternative to supervised internalization while remaining on-policy with respect to the deployment policy.

B.1 Frozen Lake

Frozen Lake evaluates navigation on procedurally generated grids with abstract observations, sparse terminal rewards, and textual feedback that the agent must interpret through interaction.

  • Environment: Frozen Lake requires moving from a start location to a goal location on an n × n grid.Grid sizes are sampled uniformly from [2, 9], with randomly selected distinct start and goal positions and at least one valid path.
  • Training: Algorithm 2 samples a first answer, performs an RL update, and conditionally generates a reflection-conditioned second attempt when the first reward is below τ.Successful second-attempt reflections are stored when their reward exceeds the threshold.
  • Observations: The environment uses abstract symbols for the agent, goal, hole, and safe frozen tile to reduce reliance on pretrained symbolic priors.The encoding is A = agent, B = goal, C = hole, and D = safe frozen tile.
  • Rewards and Termination: The reward is 1.0 for reaching the goal and 0.0 otherwise, with termination on success, falling into a hole, or exhausting 8 actions.Invalid actions or boundary moves produce no state change, while the episode continues unless the step budget is exhausted.
  • Data: Training uses 10,000 procedurally sampled instances, while evaluation uses 100 disjoint instances generated by the same process.The setup separates training and evaluation instances while retaining the same generation procedure.

B.2 Sokoban

Sokoban evaluates short-horizon box-pushing with abstract grid observations, deterministic movement constraints, sparse success rewards, and procedurally separated train and test layouts.

  • Environment: Sokoban requires placing all boxes on designated goal tiles in procedurally generated n × n grids.The experiments use single-box, single-goal layouts with n sampled uniformly from [6, 8].
  • Environment: Layouts are accepted only when their shortest valid solution is at most 8 moves, ensuring solvability and short-horizon episodes.Train and test splits are disjoint at the layout level.
  • Observations: Abstract symbols encode the agent, box, goal, wall, and floor in the full textual grid observation.The encoding includes special symbols for the agent or box occupying a box or goal location.
  • Dynamics: Deterministic actions can push one adjacent box only when the cell behind it is free; boxes cannot be pulled or moved through obstacles.Invalid moves produce no state change.
  • Rewards and Termination: The reward is 1.0 if and only if all boxes are on goals and 0.0 otherwise, with termination on success or after the step budget.The generated dataset uses an 8-step budget per instance.
  • Data: Training uses 10,000 procedurally sampled instances and evaluation uses 100 disjoint instances built with the same generation process.The evaluation set is separated from training by instance layout.

B.3 HotpotQA

HotpotQA tests tool-augmented multi-hop question answering through retrieval and answer synthesis, with answers extracted from a prescribed format and scored using graded reward.

  • Task: HotpotQA requires answering compositional questions by retrieving and synthesizing evidence across multiple documents.Each instance contains a natural-language question and a reference answer.
  • Interaction: The agent alternates among reasoning, retrieval, and answer generation rather than observing an explicit environment state.The interaction is tool-augmented and supports external retrieval before producing a final textual answer.
  • Evaluation: Final answers must be formatted inside \boxed{} so they can be reliably extracted for scoring.The extracted answer is normalized by lowercasing and whitespace canonicalization before evaluation.
  • Retrieval: The retrieval interface exposes local search(query, top k) over a dense-retrieval server backed by an indexed Wikipedia corpus.The implementation uses FAISS, E5-base-v2 embeddings, and permits up to 5 interaction turns per episode.
  • Evaluation: The reward is 1.0 for exact matches, proportional to F1 when F1 is at least 0.3, and 0 otherwise.Correctness is measured using token-level F1 against the ground-truth answer.

C Training Configuration Details

The training configuration specifies the infrastructure, optimization settings, sampling procedures, evaluation schedule, and task prompts used for ERL experiments.

  • Infrastructure: Models are trained with the rLLM stack, GRPO, vLLM, FlashAttention, and eight H100 GPUs on a single node.
  • Optimization: Optimization uses a 1e-6 learning rate, 64-example minibatches, dynamic batching, token-length limits, and parameter or optimizer offloading.
  • Sampling: Training uses batch size 64, 8,196-token prompt and response limits, asynchronous rollouts, and temperature-0.7 sampling with specified top-p and top-k validation settings.
  • Regularization: KL regularization uses coefficient 0.001, with a 0.28 actor clipping upper bound and disabled entropy, rejection sampling, and stepwise advantage estimation.
  • Experiment schedule: RLVR generates 10 samples per prompt, whereas ERL generates four samples per attempt to match compute budgets; evaluation occurs every five iterations with manual early stopping.
  • Task materials: Appendix A documents ERL implementation details, while supplementary tables provide prompts and example instances for FrozenLake, Sokoban, and HotpotQA, including a no-reflection ablation prompt.
Loading 2602.13949v1…