PaperlayerView exact revision on arXiv

Source-linked AI summary

RetroAgent: From Solving to Evolving via Retrospective Dual Intrinsic Feedback

Xiaoying Zhang, Zichen Liu, Yipeng Zhang, Xia Hu, Wenqi Shao

arXiv:2603.08561v6cs.AI

TL;DR

Standard LLM-agent reinforcement learning often focuses on task-success rewards, while useful experience can be difficult to retrieve and reuse. RetroAgent adds retrospective numerical and language feedback with lesson retrieval, achieving state-of-the-art results across four benchmarks and outperforming GRPO on each.

  • Problem

    Existing LLM-agent RL primarily optimizes extrinsic task-success rewards, while prior interactions are difficult to retrieve and reuse for later policy improvement.

    Source: §1 · pp. 1–2

  • Method

    RetroAgent uses hindsight-generated numerical feedback on incremental subtask progress and language lessons stored in a reflection memory retrieved with SimUtil-UCB.

    Source: §1 · p. 2, §3.3 · p. 7, §5 · p. 20

  • Results

    RetroAgent achieves state-of-the-art performance on all four benchmarks, outperforming GRPO by +14.4, +12.0, +21.4, and +8.6 percentage points on ALFWorld, WebShop, Sokoban, and MineSweeper, respectively.

    Source: §4.2 · p. 11

  • Takeaways & Limitations

    Results indicate that numerical feedback for exploration and language feedback for experience reuse play complementary roles in these benchmarks.

    Source: §4.2 · p. 11

    Jointly trained reflection slightly underperforms the in-context variant on ALFWorld and Sokoban, and improved multi-objective balancing is left to future work.

    Source: §4.12 · p. 20

Abstract

from arXiv · show

Standard reinforcement learning (RL) for large language model (LLM) agents primarily optimizes extrinsic task rewards, often favoring isolated task completion over continual adaptation. This paradigm can cause premature convergence to suboptimal policies and leaves useful experience only implicitly encoded in model parameters, limiting its retrieval and reuse for future decisions. We introduce RetroAgent, an online RL framework that trains agents to master interactive environments not merely by solving tasks, but by evolving across episodes. Inspired by human retrospective self-improvement, RetroAgent augments extrinsic rewards with hindsight-generated dual intrinsic feedback: (1) Intrinsic Numerical Feedback, which rewards beneficial exploration by measuring incremental subtask progress relative to prior attempts; and (2) Intrinsic Language Feedback which distills successes and failures into reusable textual lessons for explicit experience reuse. To leverage these lessons effectively, we propose Similarity & Utility-Aware Upper Confidence Bound (SimUtil-UCB), a retrieval strategy that balances semantic relevance, historical utility, and exploration. Across four challenging agentic benchmarks, RetroAgent achieves new state-of-the-art performance, outperforming GRPO by +18.3% on ALFWorld, +15.4% on WebShop, +27.1% on Sokoban, and +8.9% on MineSweeper, while demonstrating strong test-time adaptation and out-of-distribution generalization.

1 Introduction

RetroAgent couples retrospective intrinsic feedback with explicit experience reuse to support policy evolution across episodes. Across four agentic benchmarks, it reports new state-of-the-art results and strong test-time adaptation and out-of-distribution generalization.

1 Introduction

  • Prior RL often prioritizes extrinsic task-success rewards, which can favor one-off solving, premature convergence, and experience stored only implicitly in model parameters.These limitations can slow learning and weaken generalization, especially when relevant experience is sparse or context-dependent.Source: §1 · pp. 1–2
  • Approximately +10% on WebShop and +16% on Sokoban in success rates accompany RetroAgent’s new state-of-the-art results across four benchmarks.The framework also exhibits strong test-time adaptation and out-of-distribution generalization.Source: Figure 1 · p. 2, §1 · p. 3
  • Exploration-focused methods often lack explicit reusable experience, while memory-augmented methods commonly retrieve by fixed similarity and remain decoupled from policy optimization.Noisy or unavailable memory can make agents brittle, while over-reliance on retrieved experience may suppress exploration.Source: §1 · p. 2
  • The central aim is to couple exploration with explicit experience reuse so lessons guide policy evolution and become internalized rather than persistently supplied by external memory.The framework asks whether agents can use intrinsic signals from their own trajectories under sparse external supervision.Source: §1 · p. 2
  • RetroAgent reflects on each episode and combines numerical feedback on incremental subtask progress with language lessons distilled from successes and failures.SimUtil-UCB combines semantic relevance and historical utility, using UCB to balance high-utility lessons against under-used ones.Source: §1 · pp. 2–3
  • Across evaluations with Qwen-2.5-7B-Instruct and Llama-3.1-8B-Instruct on four agentic benchmarks, RetroAgent improves SOTA success rates by approximately +10% on WebShop and +16% on Sokoban.In the implementation, GRPO optimizes the decision policy and REINFORCE optimizes the self-reflection policy.Source: §1 · p. 3

2 Related Work

Prior work has moved from prompting frozen LLM agents to training agents through supervised learning or RL, while retrospective methods use feedback and memory for self-improvement. RetroAgent differs by jointly using hindsight-generated numerical rewards for exploration and language feedback for experience reuse.

LLMs as Decision-Making Agents

  • Prompted frozen LLM agents use examples, structured prompts, memory, and tools, while supervised fine-tuning and RL train agents from environmental interactions.The passage names ReAct and Reflexion among prompting approaches and notes that prompting remains bounded by the underlying model’s capabilities.Source: §2 · p. 3

Reinforcement Learning for LLM Agents

  • RL agent training spans hierarchical value functions, long-horizon advantage estimation, group-based credit assignment, reward shaping, and cross-episode meta-RL.The passage notes that these methods primarily optimize extrinsic environmental feedback; RetroAgent instead uses hindsight reflection to produce dual intrinsic feedback for continuous adaptation.Source: §2 · p. 3

Learning from Experience through Retrospection

  • Retrospective approaches generate language critiques, internalize reflections through meta-RL or policy optimization, or retrieve stored trajectories, lessons, and skills for similar tasks.RetroAgent combines intrinsic numerical rewards that guide exploration with intrinsic language feedback that supports experience reuse, and jointly uses both signals for policy optimization.Source: §2 · pp. 3–4

4 Experiments … 4.4 Analysis of In-Context Self-Reflection

Across four agentic benchmarks, RetroAgent’s retrospective intrinsic feedback improves performance over GRPO, while further analyses show test-time adaptation and benefits from pairwise reflection and selective memory augmentation.

4.1 Experimental Setup

  • Experiments span four agentic tasks, compare prompting, RL, memory-augmented, and meta-RL baselines, and evaluate GRPO-based RetroAgent on Qwen-2.5-7B-Instruct and Llama-3.1-8B-Instruct.Success Rate is reported across tasks, with WebShop Task Score also reported; training lessons are retrieved at test time.Source: §4.1 · pp. 10–11

4.2 Main Results

  • RetroAgent outperforms GRPO by 14.4, 12.0, 21.4, and 8.6 percentage points on ALFWorld, WebShop, Sokoban, and MineSweeper, respectively.It also surpasses GiGPO and SkillRL by 6.1–6.2% on WebShop.Source: §4.2 · p. 11, Table 1 · p. 12
  • Numerical exploration feedback and reusable language lessons are complementary: RetroAgent beats memory-augmented methods and GRPO w/ EMPG, which provide only one of these capabilities.On WebShop, RetroAgent achieves 78.9–82.3% success versus 17.6% for EvolveR, which reuses raw trajectories.Source: §4.2 · p. 11
  • Jointly training the self-reflection policy raises success to 95.6% on ALFWorld, 82.3% on WebShop, and 38.3% on Sokoban.This indicates that reflective feedback generation can itself be optimized to support policy evolution.Source: §4.2 · p. 12

4.3 Test-Time Adaptation and Generalisation

  • Within three attempts, RetroAgent reaches 99.0% Discovery@k on WebShop and 100.0% on OOD ALFWorld, consistently outperforming LaMer.Removing test-time retrieval causes only minor Discovery@1 and Discovery@2 drops for in-context self-reflection (78.9% to 76.8%) and leaves Discovery@3 unchanged, suggesting feedback is largely internalized in policy parameters.Source: §4.3 · p. 12, Figure 3 · p. 13, Table 2 · p. 13
  • RetroAgent consistently outperforms all baselines on harder MineSweeper settings, adapting rapidly and degrading gracefully as difficulty increases.The settings increase mine count from 3 to 4 or vary it from 3 to 5.Source: §4.3 · pp. 12–13, Figure 4 · p. 14

4.4 Analysis of In-Context Self-Reflection

  • Pairwise-trajectory induction improves reflection quality and downstream GRPO success over single-trajectory induction (72.9% vs. 70.3%).Its numerical feedback aligns more closely with oracle subtask scores, and its language lessons have lower hallucination rates and higher estimated utility.Source: Table 3 · p. 13, §4.4 · p. 14, Figure 5 · p. 14
  • Half-group memory augmentation yields higher success than full-group augmentation (75.3% vs. 72.9%), as unguided trajectories preserve exploration and reduce premature convergence.Applying memory-guided generation to every sampled trajectory reduces diversity and can promote suboptimal strategies.Source: §4.4 · pp. 14–15, Table 4 · p. 15

4.5 Effect of Intrinsic Numerical Feedback · 4.6 Effect of Intrinsic Language Feedback · 4.7 Effect of Combining Dual Intrinsic Feedback

The ablations show that numerical shaping and language-memory retrieval each improve WebShop performance under the right design, while combining both feedback channels yields further gains. Exploration-aware retrieval and RL-trained reflection help preserve broader memory use and reflection quality during training.

4.6 Effect of Intrinsic Language Feedback

  • 86.4% task score and 78.6% success rate are reached by SimUtil-UCB, whereas similarity-only and similarity–utility retrieval degrade performance when added to discounted returns.This contrasts with standard GRPO, where similarity-based retrieval improves performance.Source: Table 6 · p. 16, §4.6 · p. 16

4.5 Effect of Intrinsic Numerical Feedback

  • +8.7 percentage points in task score and +7.8 in success rate come from discounted returns, while capability-evolution rewards raise performance to 88.2% task score and 79.7% success rate.Capability-evolution rewards show consistent gains from step 25 onward.Source: Figure 6 · p. 15, §4.5 · p. 16
  • Capability-evolution rewards outperform progress-guided rewards, indicating that self-reflection-based potential scores provide richer shaping than binary extrinsic outcomes alone.Progress-guided rewards substitute the environment’s binary success score for the self-assessed progress potential.Source: §4.5 · pp. 15–16

4.6 Effect of Intrinsic Language Feedback

  • SimUtil-UCB’s UCB exploration bonus balances relevance, utility, and exploration to reduce over-reliance on similar or historically useful memories and encourage under-explored lessons.The experiments use half-group memory augmentation.Source: §4.6 · p. 16
  • Most memories are retrieved about 5 times with SimUtil-UCB, while similarity-based retrieval concentrates usage on a small subset, many accessed over 15 times.The UCB bonus diversifies memory use and contributes to stronger final performance.Source: §4.6 · p. 16, Figure 7 · p. 16

4.7 Effect of Combining Dual Intrinsic Feedback

  • ≈+3% success rate from dual intrinsic feedback exceeds either capability-evolution rewards or SimUtil-UCB alone, while the in-context variant slightly underperforms numerical feedback alone.The authors suggest simultaneous exploration signals may interfere during action selection.Source: Table 7 · p. 17, §4.7 · p. 17
  • Pairwise reflection with a reference trajectory achieves the highest reflection accuracy but does not improve task performance, suggesting contrastive comparison does not ensure robust standalone evaluation.The authors infer that contrastive comparison helps the reflector infer outcomes from relative trajectory differences.Source: Table 7 · p. 17, §4.7 · p. 18
  • RL-trained self-reflection maintains accuracy as the policy improves, unlike the in-context variant’s steady decline; after a slight dip before step 75, its accuracy recovers and rises.The initial gap is attributed to single induction being less informative than pairwise induction.Source: §4.7 · p. 17, Figure 8 · p. 17

4.8 Effect of Intrinsic Feedback on Exploration … 4.14 Qualitative Analysis

These evaluations examine how intrinsic feedback affects exploration, training efficiency, memory retrieval, self-reflection, generalization, and policy evolution. RetroAgent reaches GRPO’s peak performance earlier, benefits from utility-aware retrieval and a higher self-reflection weight, and generalizes across model architectures and sizes.

4.8 Effect of Intrinsic Feedback on Exploration

  • WebShop exploration tests compare capability-evolution rewards, SimUtil-UCB memory retrieval, and RetroAgent self-reflection using Vendi Score diversity over successful and failed trajectories.The proposed feedback mechanisms aim to encourage promising action sequences and discourage repeated failures; the supplied passages describe the evaluation but provide no diversity results.Source: §4.8 · p. 18, Table 8 · p. 18

4.9 Training Efficiency

  • 46% and 32% less time is needed for the in-context and RL-trained variants, respectively, to reach GRPO’s peak performance.The variants match that peak at steps 65 and 73, respectively; RetroAgent’s total training cost is higher, and RL-trained reflection converges slightly more slowly.Source: §4.9 · p. 19, Figure 10 · p. 19

4.10 Sensitivity: Relevance–Utility Tradeoff Coefficient

  • Higher WebShop task scores and success rates occur when retrieval prioritizes historical utility at α = 0.3 rather than semantic relevance at α = 0.7.This comparison varies the relevance–utility coefficient for RetroAgent with in-context self-reflection.Source: to the Relevance–Utility Tradeoff… · p. 19

4.11 Sensitivity to the Self-Reflection Objective Weight

  • 82.3% success rate and 88.9% task score at λreflect=1 improve on 75.8% and 86.0%, respectively, at λreflect=0.Increasing the RL-trained self-reflection objective weight consistently improves WebShop performance.Source: §4.11 · p. 19

4.12 Generalization Across Model Architectures

  • RetroAgent achieves state-of-the-art performance across four tasks with Llama-3.1-8B-Instruct, with all improvements statistically significant at p < 0.01.The RL-trained variant slightly underperforms the in-context variant on ALFWorld and Sokoban, possibly because joint optimization interferes with the decision-making objective.Source: Table 9 · p. 19, §4.12 · pp. 19–20

4.13 Scaling Across Model Sizes

  • +0.9%–+3.8% task score and +1.3%–+1.6% success rate accompany scaling Qwen2.5-Instruct from 7B to 14B on WebShop.RetroAgent outperforms competitive baselines at both scales, while the passage attributes the modest gains to shared pretraining distributions and similar underlying reasoning capabilities.Source: Figure 11 · p. 20, §4.13 · p. 20

4.14 Qualitative Analysis

  • At step 150, RetroAgent retrieves relevant lessons and selects the required WebShop item with higher confidence, unlike its failed step-65 trajectory.The earlier trajectory chose the wrong item and missed the pink variant, while assigning low token-level confidence to the correct category, “youth.”Source: §4.14 · p. 20

Ethics Statement

The study uses public interactive benchmarks in simulated, sandboxed environments, without human subjects or private data. Its feedback mechanisms remain within those simulations, and training uses open-source base models.

Ethics Statement

  • Experiments use publicly available ALFWorld, WebShop, Sokoban, and MineSweeper benchmarks in fully simulated, sandboxed environments.The study involves no human subjects, personally identifiable information, or private data.Source: Ethics Statement · p. 22
  • Capability-evolution rewards and memory-based language feedback operate only in simulations and do not interact with real users or real-world systems.These measures are intended to minimize potential negative impacts on users and society.Source: Ethics Statement · p. 22
  • Training uses open-source base models to support transparency and reproducibility.Source: Ethics Statement · p. 22

A Theoretical Analysis – Joint benefit of dual intrinsic feedback

The analysis attributes RetroAgent’s adaptation under sparse extrinsic rewards to memory retrieval shifting sampling toward higher-reflected-potential trajectories, intrinsic feedback increasing their return and advantage weights, and shared parameters transferring aligned behaviors across prompts.

Intrinsic feedback as selective gradient weighting

  • Trajectories above the current capability threshold receive at least δ intrinsic return weight, letting valuable extrinsic failures gain positive gradient weight when augmented advantage is positive.Trajectories below the current baseline receive no intrinsic reward.Source: A Theoretical Analysis – Joint benefit… · pp. 32–33

Joint effect

  • The joint lower bound improves by at least ρδ∆M over base-only sampling (δ∆M/2 when ρ=1/2), assuming retrieval raises δ-valuable trajectory sampling by ∆M>0.The theorem’s gain combines the retrieval-induced increase in valuable-trajectory probability with the intrinsic reward assigned to those trajectories.Source: A Theoretical Analysis – Joint benefit… · p. 33

Internalization through shared parameters

  • With shared policy parameters, a valuable trajectory sampled under one prompt increases its likelihood under the other to first order when score gradients align positively and η is sufficiently small.The trajectory may be extrinsically failed yet still be reinforced if its augmented advantage is positive.Source: A Theoretical Analysis – Joint benefit… · p. 34
  • A policy-gradient update on memory-augmented valuable trajectories raises their expected log-likelihood under the base prompt to first order, up to O(η^2) terms.This expected transfer holds for sufficiently small η.Source: A Theoretical Analysis – Joint benefit… · p. 34

B Implementation Details

The implementation uses a specified sentence-transformer model and four NVIDIA H200 GPUs, with detailed hyperparameters and training configurations reported in Table 10.

B Implementation Details

  • Experiments use sentence-transformers/all-MiniLM-L6-v2 and run on 4 NVIDIA H200 GPUs; Table 10 provides detailed hyperparameter settings and training configurations.Source: B Implementation Details · p. 35

C Superiority of Pairwise Induction over Single Induction

In WebShop, pairwise-trajectory induction more clearly recognizes partial progress than single-trajectory induction, although both assign minimal scores to total failures.

C Superiority of Pairwise Induction over Single Induction

  • The analysis sampled four reflection sequences per task prompt every 10 training steps and used GPT-4o to evaluate score alignment and lesson quality across 800 sequences.The 800 sequences were randomly selected from the sampled pool.Source: C Superiority of Pairwise Induction… · p. 35
  • Pairwise induction better separates early progress from total failure than single induction, while both assign minimal scores to total failures.The comparison used Qwen-2.5-7B-Instruct and measured subtask completion scores.Source: C Superiority of Pairwise Induction… · p. 35, Figure 13 · p. 35

Related papers

Loading 2603.08561v6…