PaperlayerView exact revision on arXiv

Source-linked AI summary

RetroAgent: From Solving to Evolving via Retrospective Dual Intrinsic Feedback

Xiaoying Zhang, Zichen Liu, Yipeng Zhang, Xia Hu, Wenqi Shao

arXiv:2603.08561v6cs.AI

TL;DR

Success-driven RL for LLM agents can encourage premature convergence and makes useful experience difficult to retrieve and reuse. RetroAgent addresses this gap with hindsight dual intrinsic feedback and SimUtil-UCB retrieval, achieving state-of-the-art performance across four agentic benchmarks. The paper also reports a limitation in jointly optimizing reflection and decision-making objectives.

  • Problem

    Success-driven RL can prematurely favor suboptimal policies while useful experience remains implicit in model parameters, limiting retrieval and reuse.

  • Method

    RetroAgent combines hindsight intrinsic numerical feedback for incremental subtask progress with intrinsic language feedback stored and retrieved through SimUtil-UCB.

  • Results

    RetroAgent achieves state-of-the-art performance across four agentic benchmarks and consistently outperforms strong baselines.

  • Takeaways & Limitations

    Combining progress-aware exploration with explicit experience reuse supports more effective policy evolution than success-driven RL alone.

  • Takeaways & Limitations

    The in-context variant slightly outperforms the RL-trained reflection variant on ALFWorld and Sokoban, likely because jointly optimized objectives interfere.

Abstract

from arXiv · show

Standard reinforcement learning (RL) for large language model (LLM) agents primarily optimizes extrinsic task rewards, often favoring isolated task completion over continual adaptation. This paradigm can cause premature convergence to suboptimal policies and leaves useful experience only implicitly encoded in model parameters, limiting its retrieval and reuse for future decisions. We introduce RetroAgent, an online RL framework that trains agents to master interactive environments not merely by solving tasks, but by evolving across episodes. Inspired by human retrospective self-improvement, RetroAgent augments extrinsic rewards with hindsight-generated dual intrinsic feedback: (1) Intrinsic Numerical Feedback, which rewards beneficial exploration by measuring incremental subtask progress relative to prior attempts; and (2) Intrinsic Language Feedback which distills successes and failures into reusable textual lessons for explicit experience reuse. To leverage these lessons effectively, we propose Similarity & Utility-Aware Upper Confidence Bound (SimUtil-UCB), a retrieval strategy that balances semantic relevance, historical utility, and exploration. Across four challenging agentic benchmarks, RetroAgent achieves new state-of-the-art performance, outperforming GRPO by +18.3% on ALFWorld, +15.4% on WebShop, +27.1% on Sokoban, and +8.9% on MineSweeper, while demonstrating strong test-time adaptation and out-of-distribution generalization.

1 Introduction

RetroAgent addresses limitations of success-driven RL by coupling exploration with explicit experience reuse, enabling agents to evolve across episodes under sparse supervision. It combines retrospective dual intrinsic feedback with retrieval of reusable lessons.

  • Existing RL frameworks mainly optimize extrinsic task-success rewards, favoring one-off solving over continuous adaptation.
  • Success-driven optimization can cause premature convergence to suboptimal policies and leave useful experience implicit in model parameters.These limitations make prior interactions difficult to retrieve, inspect, and reuse.
  • RetroAgent uses hindsight self-reflection after each episode to generate intrinsic feedback for task solving and continuous adaptation.The framework is inspired by retrospective reflection and combines extrinsic rewards with intrinsic signals.
  • Intrinsic Numerical Feedback rewards incremental subtask progress relative to prior attempts, encouraging exploration that may precede task success.Capability evolution is approximated through progress such as locating a target item even when the final purchase fails.
  • Intrinsic Language Feedback distills successful and failed trajectories into reusable lessons, while SimUtil-UCB balances similarity, utility, and exploration during retrieval.The retrieval strategy combines semantic relevance with historical utility and under-used lesson exploration.
  • RetroAgent is evaluated with Qwen-2.5-7B-Instruct and Llama-3.1-8B-Instruct across ALFWorld, WebShop, Sokoban, and MineSweeper.The framework is implemented with GRPO for the decision policy and REINFORCE for the self-reflection policy.

2 Related Work

Related work spans LLM agents, reinforcement learning for interactive tasks, and retrospective learning from experience. RetroAgent connects exploration-oriented feedback with explicit reflection memory.

  • LLMs as Decision-Making Agents: LLM-agent research uses prompting, in-context examples, memory retrieval, and external tools to tackle complex decision-making tasks.
  • Reinforcement Learning for LLM Agents: RL methods for LLM agents address multi-turn environments, hierarchical value estimation, long-horizon optimization, and credit assignment.Examples include ArCHer, LOOP, and group-based methods building on GRPO.
  • Learning from Experience through Retrospection: Retrospective learning methods generate verbal critiques or lessons and use them for iterative self-improvement through in-context learning.
  • Learning from Experience through Retrospection: RetroAgent combines incremental-progress exploration rewards with distilled trajectory lessons retrieved from a reflection memory.Its framework is presented as an approach to leveraging both intrinsic feedback forms across episodes.

3 RetroAgent

RetroAgent is an online RL framework that combines hindsight self-reflection with dual intrinsic feedback to improve exploration, experience reuse, and policy evolution across episodes.

  • 3 RetroAgent: RetroAgent uses hindsight self-reflection to generate intrinsic feedback alongside extrinsic task-success rewards.After each episode, the agent analyzes its trajectory and produces feedback for subsequent learning.
  • 3 RetroAgent: Sparse terminal rewards are uniformly redistributed across steps, while intrinsic rewards are likewise assigned uniformly to form the composite objective.The standard objective maximizes expected discounted extrinsic return; RetroAgent augments it with intrinsic feedback.
  • 3.3 Intrinsic Language Feedback via Reflection Memory: SimUtil-UCB retrieves memories by balancing semantic relevance, historical utility, and exploration.Its stated behavior favors higher utility among similarly relevant entries and higher relevance among similarly useful entries.
  • 3.3 Intrinsic Language Feedback via Reflection Memory: Intrinsic language feedback stores retrospective lessons in a memory buffer for retrieval during future decision-making.Each memory entry records the task instruction, lesson, source trajectory, utility, retrieval count, and originating outcome.
  • 3.2 Intrinsic Numerical Feedback for Directed Exploration: Intrinsic numerical feedback rewards trajectories whose reflected progress exceeds the best previously observed group-level performance.This directed signal distinguishes failures with meaningful progress from uninformative failures.
  • 3.3 Intrinsic Language Feedback via Reflection Memory: Reflection memory can constrain plausible task-solving hypotheses and identify progress, failure modes, or corrective actions beyond scalar rewards alone.The formal results characterize these effects up to an approximation error.

4 Experiments

RetroAgent is evaluated across four agentic benchmarks using multiple model backbones, baselines, and analyses of performance, adaptation, memory retrieval, and self-reflection quality. The experiments show improved benchmark performance, rapid test-time adaptation, robustness to harder tasks, and benefits from pairwise induction and selective memory augmentation.

  • Experimental Setup: RetroAgent is evaluated on ALFWorld, WebShop, Sokoban, and MineSweeper using Qwen-2.5-7B-Instruct and Llama-3.1-8B-Instruct.The evaluation reports Success Rate across tasks and additionally Task Score for WebShop.
  • Main Results: +14.4, +12.0, +21.4, and +8.6 percentage points over GRPO are reported on ALFWorld, WebShop, Sokoban, and MineSweeper, respectively.RetroAgent achieves state-of-the-art performance on all four benchmarks; on WebShop it also exceeds GiGPO and SkillRL by +6.1–6.2%.
  • Main Results: RetroAgent outperforms memory-augmented and uncertainty-reward baselines, supporting complementary roles for numerical exploration feedback and language-based experience reuse.The comparisons include MemRL, EvolveR, SimpleMem+GRPO, SkillRL, and GRPO w/ EMPG.
  • Self-Reflection Analysis: 95.6%, 82.3%, and 38.3% success are reached on ALFWorld, WebShop, and Sokoban when the self-reflection policy is jointly trained with the decision policy.The result indicates that reflective feedback generation can itself be optimized to support policy evolution.
  • Adaptation and Analysis: Discovery within three attempts rises from 82.3% to 99.0% on WebShop and from 92.9% to 100.0% on ALFWorld, while RetroAgent remains robust as MineSweeper difficulty increases.Removing test-time retrieval causes only minor early-attempt drops and leaves Discovery@3 unchanged; pairwise induction improves downstream success to 72.9% versus 70.3%, while half-group augmentation reaches 75.3% versus 72.9% for full-group augmentation.

4.5 Effect of Intrinsic Numerical Feedback

Intrinsic numerical feedback improves policy optimization by rewarding incremental progress, while language feedback and SimUtil-UCB enable explicit reuse of experience. Jointly trained reflection preserves feedback quality during policy improvement, although contrastive reflection can improve accuracy without improving task performance.

  • Discounted returns improve GRPO by +8.7 percentage points in task score and +7.8 in success rate on WebShop.
  • SimUtil-UCB reaches an 86.4% task score and a 78.6% success rate by balancing semantic relevance, historical utility, and exploration.The UCB bonus reduces over-reliance on similar or high-utility memories and promotes under-explored lessons.
  • SimUtil-UCB distributes memory access more evenly, with most instances retrieved about 5 times, unlike similarity retrieval where many exceed 15 accesses.
  • Dual intrinsic feedback yields approximately +3% success-rate gains over either capability-evolution rewards or SimUtil-UCB retrieval alone.The in-context variant slightly underperforms GRPO with capability-evolution rewards only, suggesting interference between simultaneous exploration signals.
  • RL-trained self-reflection maintains accuracy during training, whereas in-context reflection declines as the decision policy improves.Pairwise induction gives higher reflection accuracy but does not improve task performance over single induction.

4.8 Effect of Intrinsic Feedback on Exploration

RetroAgent’s intrinsic feedback improves training efficiency, retrieval behavior, and performance across model architectures and scales. Utility-aware retrieval and self-reflection optimization provide measurable gains, while joint objectives can interfere on some tasks.

  • 46% and 32% reductions in time to GRPO’s peak performance occur for the in-context and RL-trained variants, respectively.RetroAgent has higher total training cost but reaches the GRPO peak at steps 65 and 73.
  • Utility-prioritized retrieval at α = 0.3 produces higher WebShop task scores and success rates than relevance-oriented retrieval.
  • Increasing λ_reflect from 0 to 1 raises success rate from 75.8% to 82.3% and task score from 86.0% to 88.9%.
  • RetroAgent consistently achieves state-of-the-art performance across four tasks with Llama-3.1-8B-Instruct.All improvements reported in Table 9 are statistically significant at p < 0.01.
  • The RL-trained self-reflection variant slightly underperforms the in-context variant on ALFWorld and Sokoban, likely because joint objectives interfere.The authors leave improved multi-objective balancing for future work.
  • Scaling Qwen2.5-Instruct from 7B to 14B improves task score by +0.9%–+3.8% and success rate by +1.3%–+1.6%.The paper reports limited benefits from scaling alone when pretraining distributions and reasoning capabilities are similar.
  • The step-65 failed WebShop trajectory becomes a successful step-150 trajectory after relevant lessons are retrieved during training.

5 Conclusion

RetroAgent bridges one-off task solving and continuous adaptation through hindsight self-reflection and dual intrinsic feedback. Across four agentic tasks, it achieves state-of-the-art performance while showing test-time adaptation and out-of-distribution generalization.

  • RetroAgent combines incremental-progress rewards, reusable language lessons, and SimUtil-UCB retrieval in an online RL framework.
  • Across four diverse agentic tasks, RetroAgent consistently achieves state-of-the-art performance with strong test-time adaptation and out-of-distribution generalization.

Ethics Statement

The study follows privacy, fairness, and responsible-development guidelines in fully simulated benchmark environments. It uses no human subjects, personally identifiable information, or private data.

  • All experiments use publicly available ALFWorld, WebShop, Sokoban, and MineSweeper benchmarks in sandboxed environments.
  • The study does not involve human subjects, personally identifiable information, or private data.

A Theoretical Analysis – Joint benefit of dual intrinsic feedback

The analysis shows how memory retrieval and intrinsic numerical feedback jointly improve training: retrieval changes which trajectories are sampled, while intrinsic feedback changes their optimization weight. Shared parameters can transfer memory-guided behaviors to the base prompt when score gradients align.

  • Intrinsic feedback: The capability-evolution reward selectively upweights trajectories whose reflected potential satisfies ϕ_k(τ) ≥ δ and gives no intrinsic reward below the current baseline.The analysis therefore emphasizes trajectories that clear the current capability threshold.
  • Intrinsic feedback: Even when Rext(τ) = 0, a trajectory can receive positive gradient weight if its augmented advantage is positive.This separates failures containing meaningful reflected progress from uninformative failures.
  • Joint benefit: The mixture objective combines base-prompt and memory-augmented objectives with mixture coefficient ρ, and RetroAgent uses ρ = 1/2.This formalizes joint training across the two rollout prompt distributions.
  • Joint benefit: If memory augmentation raises the probability of sampling a δ-valuable trajectory by ΔM, the intrinsic-reward lower bound improves by at least ρδΔM.For ρ = 1/2, the guaranteed gain is at least δΔM/2.
  • Internalization: Shared policy parameters allow a high-advantage trajectory sampled under one prompt to increase the likelihood of related behavior under the other prompt when score gradients are positively aligned.The transfer follows from a positive first-order update for sufficiently small learning rates.
  • Internalization: A memory-augmented policy-gradient update increases the expected base-prompt log-likelihood of valuable trajectories to first order when memory rollouts are used.The result holds up to O(η^2) terms and relies on sufficiently small η.
  • Joint mechanism: Memory retrieval shifts sampling toward trajectories with higher reflected potential, while intrinsic feedback assigns larger returns to trajectories exceeding the capability baseline.Together, these mechanisms reinforce partial progress even when extrinsic task rewards are absent.
  • Conclusion: The joint analysis attributes improvement in exploration, credit assignment, and online adaptation to retrieval, selective intrinsic weighting, and cross-prompt transfer through shared parameters.The conclusion is framed for sparse extrinsic rewards.

B Implementation Details

The implementation uses a specified sentence-transformer model and runs all experiments on four NVIDIA H200 GPUs.

  • Implementation Details: Experiments use sentence-transformers/all-MiniLM-L6-v2 and 4 NVIDIA H200 GPUs.These are the reported embedding-model and hardware settings.

C Superiority of Pairwise Induction over Single Induction

The pairwise induction analysis evaluates whether reflection scores discriminate trajectory quality on WebShop. Pairwise induction better recognizes intermediate progress than single induction, which often assigns partial successes scores near zero.

  • Evaluation setup: The analysis samples four sequences per prompt every 10 training steps and randomly selects 800 sequences for evaluation.GPT-4o evaluates score alignment with oracle outcomes and the quality of summarized lessons.
  • Metrics: The reflection analysis measures both alignment between induced subtask scores and oracle outcomes and the quality of summarized lessons.GPT-4o serves as the external evaluator for both metrics.
  • Evaluation setup: Figure 13 compares single- and pairwise-trajectory induction using subtask completion scores on WebShop.The comparison concerns Qwen-2.5-7B-Instruct within the training-free self-reflection mechanism.
  • Findings: Pairwise induction assigns distinctly higher scores to Early Progress trajectories and separates them clearly from Total Failure cases.Both methods assign minimal scores to Total Failure cases, but pairwise induction discriminates intermediate outcomes more effectively.
  • Findings: Single induction struggles to distinguish Early Progress from Total Failure, producing a distribution heavily skewed toward zero for partial successes.This limits its discrimination of trajectories that contain intermediate progress.
  • Configuration: Table 10 reports the default hyperparameters and training configurations used across environments.The table is presented as the configuration reference for RetroAgent experiments.

E Task Prompts

The task prompts require structured reflection over trajectory outcomes, subtask completion, causal actions, and reusable lessons. Pairwise prompts additionally compare the current attempt with a reference trajectory, while task-specific criteria define progress across environments.

  • E.1 Prompt Templates for In-Context Self-Reflection: WebShop prompts request task success, subtask statuses, action lessons, and navigation lessons in JSON format.The subtasks include search, filtering, item selection, and purchase completion.
  • E.1 Prompt Templates for In-Context Self-Reflection: Across prompts, subtask statuses must reflect trajectory events and lessons must explain the factors leading to the outcome.The templates also request specific trajectory references and JSON-only output.
  • E.1 Prompt Templates for In-Context Self-Reflection: The prompts permit null lessons only when truly inapplicable and consistently require structured JSON evaluation.These constraints apply across the task-specific reflection templates.
  • E.1 Prompt Templates for In-Context Self-Reflection: WebShop pairwise induction compares the current trajectory with a reference trajectory to identify successful and failed subtasks and valuable lessons.The prompt asks for concrete actions or decisions leading to the outcome.
  • E.1 Prompt Templates for In-Context Self-Reflection: ALFRED prompts evaluate pickup, navigation, appliance use, and object placement, while recording action and spatial lessons.The reflection should identify which steps succeeded or failed and what caused the outcome.
  • E.1 Prompt Templates for In-Context Self-Reflection: ALFRED reflection prompts ask successful attempts to identify effective sequences and failed attempts to identify missed or failed steps.Lessons should reference concrete objects, locations, and appliances.
  • E.1 Prompt Templates for In-Context Self-Reflection: Minesweeper prompts require analysis of attempted subtasks, outcome-causing decisions, mistakes or successes, and a concise plan for the next trial.They use board notation for cell states and one-based row and column coordinates.
  • E.1 Prompt Templates for In-Context Self-Reflection: Minesweeper subtask criteria mark valid moves, exploration progress, logical attempts, error recovery, cascade usage, and systematic approach as completed or incomplete.Examples include revealing more than 10% of the board and correcting errors within three attempts.
Loading 2603.08561v6…