Source-linked AI summary
FLEX: Continuous Agent Evolution via Forward Learning from Experience
Zhicheng Cai, Xinyuan Guo, Yu Pei, Jiangtao Feng, Jinsong Su, Jiangjie Chen, Ya-Qin Zhang, Wei-Ying Ma, Mingxuan Wang, Hao Zhou
TL;DR
LLM agents remain static after training, motivating a learning paradigm that can use experience continually without changing model parameters. FLEX constructs and retrieves a structured experience library through forward exploration and distillation, improving performance across mathematics, chemical retrosynthesis, and protein fitness prediction while supporting scalable and inheritable evolution.
Problem
LLM agents remain static after training, while existing gradient-based and self-evolving approaches face computational, forgetting, task-generalization, scalability, or model-transfer limitations.
Method
FLEX uses forward exploration, experience distillation, library updating, and retrieval-guided reasoning without gradient back-propagation or parameter updates.
Results
FLEX significantly enhances baseline performance across mathematical reasoning, chemical retrosynthesis, and protein fitness prediction, including improvements from 40% to 63% on AIME25.
Takeaways & Limitations
FLEX supports continuous agent evolution through accumulated experience, with experience scaling and inheritance enabling knowledge reuse across agents.
Takeaways & Limitations
Gradient-based learning remains computationally costly, vulnerable to catastrophic forgetting, and unable to continually evolve models through environmental interaction.
Abstract
from arXiv · showhide
Autonomous agents driven by Large Language Models (LLMs) have revolutionized reasoning and problem-solving but remain static after training, unable to grow with experience as intelligent beings do during deployment. We introduce Forward Learning with EXperience (FLEX), a gradient-free learning paradigm that enables LLM agents to continuously evolve through accumulated experience. Specifically, FLEX cultivates scalable and inheritable evolution by constructing a structured experience library through continual reflection on successes and failures during interaction with the environment. FLEX delivers substantial improvements on mathematical reasoning, chemical retrosynthesis, and protein fitness prediction (up to 23% on AIME25, 10% on USPTO50k, and 14% on ProteinGym). We further identify a clear scaling law of experiential growth and the phenomenon of experience inheritance across agents, marking a step toward scalable and inheritable continuous agent evolution. Project Page: https://flex-gensi-thuair.github.io.
1 Introduction
FLEX addresses the inability of static or parameter-focused agents to learn continually from interaction by building and using an evolvable experience library. Across mathematics, chemical retrosynthesis, and protein fitness prediction, it reports consistent performance improvements while supporting scalable and inheritable evolution.
- Motivation: Existing gradient-based approaches are poorly suited to continuous evolution because of computational cost, catastrophic forgetting, and infeasible parameter optimization for closed-source models.Related self-evolving approaches also remain task-specific, non-scalable, and model-specific.
- FLEX: FLEX shifts learning from modifying model parameters to constructing and leveraging an evolvable experience library.The paradigm uses forward exploration, library updating, and experience-guided reasoning rather than gradient-based tuning.
- FLEX: FLEX continuously aggregates cross-task experiences in a persistent library whose expansion enables performance to scale with accumulated knowledge.The library stores distilled strategies in semantic form for future use.
- Experiments: From 40% to 63% on AIME25, 20% to 30% on USPTO50k, and 46% to 60% on ProteinGym, FLEX improves performance across three challenging scientific tasks.These results span Olympiad-level mathematics, chemical retrosynthesis, and protein fitness prediction.
- FLEX: Experience inheritance transfers distilled experience between agents in a plug-and-play manner, enabling knowledge assimilation without redundant learning.The semantic representation is described as transferable and interpretable.
2 Forward Learning with Experience
FLEX replaces parameter fine-tuning with forward learning over an evolvable experience library, which is constructed from trajectories, refined through updates, and retrieved to guide future reasoning. Its Meta-MDP formulation coordinates sample-level exploration with library-level evolution, while the information-theoretic view links useful experiences to reduced predictive uncertainty.
- Meta-MDP Formulation: The resulting closed-loop process continuously refines accumulated experiential knowledge while preserving gradient-free forward learning across tasks.Base-level and meta-level updates jointly support ongoing experience accumulation and library evolution.
- Optimization Objective: FLEX constructs an experience library that maximizes expected task correctness and uses retrieved experiences to condition the frozen LLM's predictions.The library is optimized as the central learning object rather than the model parameters.
- Optimization Process: FLEX learns through forward exploration, probabilistic library updates, and retrieval-guided reasoning instead of backward gradient-based parameter updates.An actor generates trajectories, an updater distills them into the library, and relevant experiences guide subsequent tasks.
- Information-Theoretic Insights: Retrieved experiences provide information about target outputs beyond the query, so minimizing conditional entropy is equivalent to maximizing their conditional mutual information.The reformulation interprets effective experiences as reducing predictive uncertainty and increasing information gain.
- Meta-MDP Formulation: The Meta-MDP separates base-level intra-sample exploration and distillation from meta-level integration that evolves the global experience library.This hierarchy coordinates fine-grained exploration with long-term library optimization.
3 Concrete Instantiation of FLEX
FLEX instantiates forward learning through actor–critic exploration and a hierarchically organized experience library that is continually updated and retrieved. The resulting system refines, consolidates, and reuses semantic experiences for ongoing agent evolution.
- 3 Concrete Instantiation of FLEX: FLEX combines actor–critic forward exploration with experience-library evolution to realize continual semantic evolution.The actor–critic loop explores and refines experiences, while the meta-level process consolidates and reuses their semantics.
- 3 Concrete Instantiation of FLEX: Parallel trajectory sampling and sequential refinement provide both broad coverage of reasoning paths and progressive improvement.Rejection sampling retains diverse, high-quality outputs, while critic feedback guides iterative refinement.
- 3 Concrete Instantiation of FLEX: Incorrect trajectories are converted into abstract, task-agnostic improvement suggestions that are fed back into the actor as new experiences.Validated experiences are distilled after refinement and passed to the meta-level process for long-term accumulation.
- 3 Concrete Instantiation of FLEX: The hierarchical textual library supports modular indexing and retrieval across abstraction levels, enabling top-down guidance and bottom-up consolidation.Its explicit storage also externalizes reasoning behaviors and supports interpretability and transparency.
- 3 Concrete Instantiation of FLEX: The updater removes duplicates, selectively merges similar experiences, and inserts novel experiences into appropriate hierarchical partitions.During inference, contextualized retrieval conditions relevance on the query, reasoning state, consistency, and goal alignment.
- 3 Concrete Instantiation of FLEX: Together, update and retrieval form a self-organizing system that progressively refines the library’s structure and semantic granularity.The refinement loop and meta-level updater dynamically organize distilled experiences into the evolving library.
4 Experiments
FLEX improves performance across mathematics, chemistry, and biology benchmarks, while experience accumulation follows measurable scaling behavior and transfers across models. Case studies attribute these gains to reusable procedural strategies and guardrails distilled from prior trajectories.
- 4 Experiments: FLEX achieves significant and consistent improvements across all evaluated scientific domains and benchmarks.The experiments cover mathematical reasoning, chemical retrosynthesis, and protein fitness prediction.
- Math: 23.3 percentage points lifts Claude-Sonnet-4 on AIME25 from 40.0% to 63.3% after learning from 49 examples.The reported increase is a 58.7% relative improvement.
- Chemistry: 10.0 percentage points lifts Claude-Sonnet-4.5 on USPTO50k from 20.0% to 30.0% after learning from 50 examples.GPT-5 and Gemini-2.5-Pro also rise from 9.0% to 16.0% and 18.0%, respectively.
- Biology: ProteinGym shows an average improvement of roughly 0.10 in Spearman’s ρ through reusable rules and heuristics extracted from a small training set.The forward-learned model transfers these insights to a larger and more diverse test set.
- The Scaling Law of Experience: Training accuracy rises from 81.2% to 94.2% as the experience library grows from 1,001 to 1,904 items, while test accuracy increases from 81.3% to 83.3%.Test performance surpasses the 80.8% baseline, and increasing experience reduces variance.
- The Scaling Law of Experience: Experience accumulation changes from rapid expansion of +576 items between epochs 1 and 2 to selective refinement of +64 items between epochs 4 and 5.The paper characterizes this as a logistic-like, phase-changing scaling law.
- Inheritance of the Experience Library: Cross-model transfer distills expertise from stronger to weaker agents and generalizes strategies from weaker to stronger agents.Claude-Sonnet-4.5’s library boosts Gemini-2.5-Pro by 11 absolute points on USPTO50k, while Claude-Sonnet-4’s experience gives DeepSeek-V3.1-Terminus a 6.7-point AIME25 gain.
- Case Study: FLEX case studies show that retrieved experiences supply procedural scaffolds and logical guardrails that convert heuristic exploration into solve-and-verify reasoning.In the mathematics example, these safeguards lead to the correct final area of 1043.
5 Discussion
The discussion frames FLEX as a step toward agents that learn continuously from experience, share knowledge across agents, and make learning more transparent.
- Conclusion: The discussion presents these contributions as one step in an ongoing effort to build dynamic, collaborative, and transparent AI systems.It explicitly avoids framing the work as a final conclusion.
- Qualitative Case Studies: FLEX’s qualitative case studies show retrieved golden rules and warnings helping agents overcome reasoning failures in mathematics, chemistry, and biology.Baseline LLM and ReAct responses fail in the cases, whereas FLEX reaches the correct solution.
- Toward Continuous Learning: FLEX’s experience-driven learning is presented as a promising direction for continuous, more auditable agent evolution.The discussion contrasts this with static AI systems and traditional numerical optimization.
- Heading to a Collective Wisdom: Decoupling learned experience from model parameters supports inheritance of knowledge across distinct agents.This addresses the lack of a robust medium for transferring wisdom between agents.
- Learning in a transparent way: Human-readable experience records can make an agent’s learning process more transparent and support meaningful human-in-the-loop interaction.The discussion characterizes this as movement toward a more white-box model of AI development.
6 Related Work
Related work spans gradient-based optimization, non-parametric adaptation, self-evolving agents, and experience-driven evolution, but leaves continuous gradient-free learning across challenging scientific tasks insufficiently established.
- Learning Paradigms: Gradient-based learning improves models through optimization but remains computationally costly, vulnerable to catastrophic forgetting, and static after training.These limitations prevent continual evolution through environmental interaction.
- Non-parametric and Self-evolving Agents: Non-parametric methods adapt prompts or contexts without gradient updates, while self-evolving agents extend capabilities through tools and architecture changes.The related work surveys prompt engineering, in-context learning, tool evolution, and architecture evolution.
- Experience-driven Evolution: Experience-driven methods accumulate and reuse interaction trajectories, but differ in generalization, interpretability, or explicit learning processes.The discussion names AgentKB, Memento, ReasoningBank, and TF-GRPO as examples.
- Research Gap: Existing approaches do not establish a principled continual gradient-free learning paradigm for LLM agents.The section also notes that most evaluations remain limited to simple reasoning benchmarks.
7 Conclusion
FLEX is proposed as a paradigm for continuous agent evolution through experience rather than parameter updates. It builds an evolving experience library that enhances reasoning across mathematical, chemical, and biological tasks.
- Contribution: FLEX enables LLM agents to continuously evolve by learning from accumulated experience.The conclusion states this as the paper’s central proposal.
- Method: FLEX constructs an evolving experience library through exploration and semantic distillation of environmental interactions, without gradient back-propagation or parameter updates.The library is used to augment reasoning and expert-knowledge use.
- Results: Experiments report improved baseline performance across mathematical reasoning, chemical retrosynthesis, and biological protein fitness prediction.The conclusion characterizes these as diverse challenging scientific tasks.
A.1 Biology
The biology study applies FLEX to protein fitness prediction by transforming agentic reasoning into proxy-metric construction under scarce-data conditions. It evaluates the method against protein models and native LLM baselines, with ablations isolating three core components.
- Agentic Task Transformation: Agents are tasked with reasoning, ranking, knowledge synthesis, and few-shot learning rather than directly inferring mutational effects from protein sequences.This reframes prediction around constructing enhanced fitness metrics despite limited biochemical specialization.
- Benchmark and Evaluation: FLEX evaluates protein fitness using Spearman correlation between ground-truth DMS scores and agent-generated proxy metrics on ProteinGym.The benchmark uses a small experience set of 100 sequences on average 1.47% of available data.
- Results: Figure 7 reports ProteinGym results from both protein language models and large language models.
- Baselines: The study compares FLEX with zero-shot protein models spanning single-sequence, homolog-based, and structural representations, plus direct LLM baselines.The listed models include VespaG, PoET, ProSST, and VenusREM.
- Ablation Study: Ablations identify Experience Exploration, Experience Evolution, and Regression Tools as contributors to protein fitness performance.Experience Evolution updates lessons with new cross-validation facts, while exploration supplies success and failure cases.