Source-linked AI summary
Experiential Reflective Learning for Self-Improving LLM Agents
Marc-Antoine Allard, Arnaud Teinturier, Victor Xing, Gautier Viaud
TL;DR
LLM agents struggle to adapt to unfamiliar environments and reuse prior interactions. ERL reflects on single-attempt trajectories to create transferable heuristics, retrieves relevant guidance at test time, and achieves a 56.1% Gaia2 success rate, improving 7.8% over ReAct while outperforming prior experiential methods.
Problem
General-purpose LLM agents struggle with unfamiliar tools and domain conventions, while fine-tuning is resource-intensive, unavailable for closed-source models, and unsuitable for continuous learning.
Method
ERL reflects on task trajectories and outcomes to generate structured heuristics, then uses LLM-based retrieval to inject relevant heuristics into the context of new tasks.
Results
56.1% overall success rate on Gaia2, a +7.8% improvement over ReAct, with ERL outperforming ExpeL and AutoGuide.
Takeaways & Limitations
Heuristics transfer better than raw trajectories, selective retrieval outperforms random and embedding-based selection, and failure-versus-success benefits depend on task type.
Abstract
from arXiv · showhide
Recent advances in large language models (LLMs) have enabled the development of autonomous agents capable of complex reasoning and multi-step problem solving. However, these agents struggle to adapt to specialized environments and do not leverage past interactions, approaching each new task from scratch regardless of their accumulated experience. We introduce Experiential Reflective Learning (ERL), a simple self-improvement framework that enables rapid environment adaptation through experiential learning. ERL reflects on task trajectories and outcomes to generate heuristics, capturing actionable lessons that transfer across tasks. At test time, relevant heuristics are retrieved based on the current task and injected into the agent's context to guide execution. On the Gaia2 benchmark, ERL improves success rate by 7.8% over a ReAct baseline, with large gains in task completion reliability, and outperforms prior experiential learning methods. Through systematic ablations, we find that selective retrieval is essential and that heuristics provide more transferable abstractions than few-shot trajectory prompting. These results demonstrate that reflecting on single-attempt experiences to extract transferable heuristics enables effective agent self-improvement.
1 INTRODUCTION
ERL addresses the difficulty of adapting LLM agents to specialized environments by reflecting on experience to produce reusable heuristics and retrieving relevant guidance for new tasks. On Gaia2, it improves success over ReAct while emphasizing transferable, task-specific lessons.
- Motivation: General-purpose LLM agents often struggle with unfamiliar tools and domain conventions, while fine-tuning is costly and does not support continuous learning.These limitations motivate parameter-free experiential memory approaches.
- Approach: ERL reflects on past trajectories and outcomes to build a pool of reusable heuristics, then retrieves relevant heuristics for each new task.The retrieved guidance is injected into the agent’s context.
- Approach: ERL extracts heuristics from single-attempt trajectories without curated training sets or repeated execution.The heuristics retain granular trajectory details that cross-task aggregation can lose.
- Results: 56.1% overall success rate, a +7.8% improvement over ReAct, was achieved by ERL on Gaia2’s Search and Execution splits.ERL also outperformed ExpeL and AutoGuide.
2 METHODS : EXPERIENTIAL REFLECTIVE LEARNING
ERL has two linked components: generating structured heuristics from completed task experiences and retrieving a selective, task-specific subset to guide execution on new tasks.
- Framework: ERL combines heuristic generation from task experience with retrieval-augmented execution for new tasks.These are the framework’s two main components.
- Heuristic generation: After each task, the agent reflects on its description, trajectory, and binary outcome to produce an analysis and a guideline with triggers and recommended actions.The guideline can encode concrete tool-use procedures.
- Retrieval-augmented execution: For a new task, an LLM scores heuristics using task similarity, experience diversity, and guideline informativeness before injecting the top-k items into the system prompt.Selective retrieval avoids overwhelming the agent with the full heuristic pool.
3 EXPERIMENTS
Experiments on Gaia2 evaluate ERL against ReAct and alternative retrieval or experience-selection strategies across held-out universes. Results show that selective, relevant heuristics outperform raw trajectories, random selection, and broad heuristic inclusion.
- Experimental setup: Gaia2 evaluates long-horizon agents in a simulated mobile environment with 12 applications and 101 tools, focusing on Search and Execution tasks.Heuristics are accumulated on eight universes and tested on two held-out universes.
- Experimental setup: ERL augments Gaia2’s default ReAct scaffold by injecting retrieved heuristics into the system prompt without modifying the core ReAct loop.The default configuration retrieves k = 20 heuristics with an LLM.
- Retrieval quality: 56.1% overall success rate from LLM-based retrieval exceeds embedding retrieval at 53.3% and the best random configuration at 53.8%.Random-selection performance is non-monotonic and degrades with excessive inclusion.
- Heuristic source: Failure heuristics achieve +14.3% over baseline on Search, while success heuristics achieve +9.0% on Execution.Failure-only retrieval reaches 58.9% overall but sacrifices Execution performance, making mixed retrieval a practical compromise for unknown task distributions.
4 CONCLUSION
The paper concludes that ERL improves Gaia2 task-completion reliability by distilling trajectories into reusable heuristics and retrieving relevant guidance. Its analyses identify better transfer than raw trajectories, superior LLM-based retrieval, and task-dependent value from failure versus success experiences.
- Conclusion: ERL distills trajectories into reusable heuristics and retrieves relevant guidance at test time to improve task-completion reliability on Gaia2.The framework outperforms prior experiential learning methods.
- Conclusion: Heuristics transfer better than raw trajectories, while LLM-based retrieval outperforms random and embedding-based selection.These findings summarize the paper’s main comparative analyses.
- Conclusion: The impact of learning from failures versus successes varies with task type.The conclusion identifies this outcome as a central analysis finding.
A τ 2-BENCH EVALUATION
ERL improves overall performance on τ 2-bench, especially in Airline and Retail, while Telecom exposes limitations in user-coordination settings.
- ERL improves overall success rate over the baseline, with gains on Airline and Retail but a slight drop on Telecom.Overall success rate is 0.380 versus 0.367 for the baseline.
- Telecom reverses the pattern: success rate drops and pass^3 falls to zero, although pass@3 improves from 0.575 to 0.625.
- The Telecom divergence may reflect combinatorial task composition and dual-control user interaction, which make retrieved heuristics less consistently applicable.Users have their own tools, randomly assigned personas, and variable device states.
- τ 2-bench results corroborate Gaia2 findings in single-control domains but highlight limits in capturing user-coordination strategies.
B GAIA2 EVALUATION SETUP
The Gaia2 evaluation uses ARE's ReAct scaffold and randomly selected test universes, with implementations of experiential-learning baselines for comparison.
- ARE provides the infrastructure and default ReAct agentic scaffold used as the baseline for Gaia2 evaluation.Each scenario includes a task and ground-truth trajectory for automatic completion verification.
- Test universes 22 and 25 were randomly selected, and GPT-5-mini was chosen to balance reasoning performance, latency, and computational cost.
- The ExpeL implementation collects success and failure experiences with Reflexion and up to three retries, then retrieves three task-relevant few-shot examples.
- AutoGuide generates context-aware guidelines from contrastive successful and failed trajectories and feeds the top three relevant guidelines into agent context after each ReAct turn.
C.1 TOKEN-MATCHED COMPARISON WITH A FEW-SHOT BASELINE
When feedback is matched by experience count and token context, heuristic feedback produces higher success rates than raw few-shot trajectories across the evaluated settings.
- +5.5% Execution gain occurs with heuristics at approximately 20 scenarios compared with raw trajectories.
- +23.8% Search gain occurs with heuristics at 40 scenarios, where raw trajectories approach long-context degradation.
- For any number of experiences, heuristics achieve higher success rates than raw trajectories.The comparison uses the same number of experiences provided in context.
- Distilling experiences into heuristics provides a more transferable learning signal while fitting substantially more experiences within a fixed context budget.
C.2 HEURISTIC RETRIEVAL
ERL evaluates retrieval count and strategy, finding that selective retrieval matters and that LLM-based ranking performs best at k = 20, while imperfect outcome feedback reduces performance.
- Embedding retrieval selects heuristics by cosine similarity between test and training tasks, with best performance at k = 20 and a 53.3% success rate.
- LLM-based ranking considers task similarity, error patterns, and heuristic lessons, achieving optimal performance at k = 20 with a 56.1% success rate.It outperforms embedding-based retrieval for most k values, particularly on Search.
- Without environment validation rewards, the agent identifies success or failure correctly only 70% of the time and ERL success rate drops to 51.2% (-4.8%).This remains above the 48.3% baseline.
- Accurate outcome feedback remains important for optimal performance, although ERL retains value with imperfect reward signals.
D TOKEN COUNT AND COST ANALYSIS
The cost analysis compares baseline ReAct with ERL and shows that ERL substantially increases input-token usage while prompt caching limits the scenario-rollout cost increase.
- Table 4 compares token usage and API costs for baseline ReAct and ERL evaluations.
- Input tokens for scenario rollout increase by 85% under ERL, mainly because roughly 20k retrieved-heuristic tokens are appended to a roughly 12k-token system prompt at every turn.The number of turns remains roughly constant: 16.6 versus 17.6.
task.
ERL’s heuristic workflow reflects on completed trajectories, retrieves relevant guidance for new tasks, and can accumulate guidance iteratively. The examples illustrate how structured heuristics connect task outcomes to reusable execution advice.
- Iterative ERL: ERL’s iterative variant retrieves guidance from the current pool before execution, then adds newly generated heuristics after each task.Tasks are processed in batches, creating a cumulative learning effect across batches.
- Iterative ERL: Iterative ERL achieves higher source-task performance but 5.4% lower test performance than standard ERL.The reported gap may reflect narrower failure coverage and reduced transferability from guided trajectories.
- Heuristic examples: Heuristics combine an analysis of why an agent succeeded or failed with a guideline containing generalizable takeaways.Guidelines can reference specific tools and clarify idiosyncratic behaviors or errors.
- Heuristic examples: A heuristic from a failed rescheduling trajectory guides a structurally similar event-replacement task by creating the replacement before deleting the original.The agent repeatedly applies this guideline while completing the new task.
- Heuristic generation: Heuristic generation uses the task, agent trajectory, and binary reward to produce constructive lessons for avoiding repeated mistakes and preserving effective reasoning patterns.
- Heuristic retrieval: At test time, an LLM decomposes the task, scores stored heuristics, and selects the top k using task similarity, experience diversity, and guideline informativeness.The selected heuristics are injected into the agent’s system prompt.