Source-linked AI summary
Experience Funnel: A State-Policy Alternating Loop for Self-Evolving Agents
Wenbo Gao, Zhaomou Song, Zhiyuan Ji, Renxi Liu, Xing Li, Xianzhi Yu, Xiaoguang Li, James Chung-wai Cheung, Weizhe Lin, Yaoyuan Wang
TL;DR
Experience Funnel addresses how agents can transform abundant interaction experience into reusable competence while preserving rapid, editable adaptation. It alternates validated textual-state evolution with transition-aware consolidation of behavior that remains useful across state revisions. Across diverse benchmarks, it improves capability over state-only and internalization-only alternatives while progressively converting explicit experience into autonomous policy competence.
Problem
Agents must convert interaction experience into reusable competence while avoiding the context costs of accumulating textual guidance and the risk of indiscriminate parameter updates.
Method
Experience Funnel alternates validated textual-state updates with selective transition-aware distillation of state-enabled behavior into a state-free policy.
Results
Across question answering, embodied interaction, and web navigation, alternating state adaptation and policy consolidation consistently improves performance over state-only evolution and policy-internalization approaches.
Takeaways & Limitations
Useful explicit experience is progressively converted into autonomous policy competence while complementary guidance remains available in the evolving textual state.
Takeaways & Limitations
The framework assumes repeated related tasks with sufficiently informative environment-level feedback, limiting direct applicability to one-shot or highly subjective, sparse-feedback domains.
Abstract
from arXiv · showhide
Autonomous agents powered by large language models (LLMs) continuously accumulate experience through interaction, creating an opportunity to improve future behavior through self-evolution. A fundamental challenge is how to transform abundant, task-specific interaction experience into reusable model competence without sacrificing the ability to adapt rapidly to newly observed evidence. Explicit textual states, such as skills and agent harnesses, provide fast, human-readable and editable adaptation, but incur persistent dependence on external context; parametric policies provide compact and reusable competence, but are substantially slower to update. We present \textit{Experience Funnel}, a self-evolving framework that couples fast state adaptation with slow policy consolidation in an alternating loop. Interaction trajectories are first distilled into an explicit textual state, where newly acquired experience can be rapidly incorporated and validated. The framework then selectively identifies state-enabled behavior that remains useful across state revisions and consolidates it into the policy through transition-aware distillation. The updated state--policy pair subsequently generates new rollouts, providing fresh evidence for the next round of state adaptation and policy consolidation. Experiments across diverse agent benchmarks show that \textit{Experience Funnel} consistently improves agent capability over state-only evolution and policy-internalization approaches, while progressively converting useful explicit experience into autonomous policy competence.
1 Introduction
Experience Funnel addresses the challenge of converting interaction experience into reusable policy competence while retaining rapid, editable adaptation. It alternates validated textual-state evolution with selective policy consolidation, and experiments report consistent gains over state-only and policy-internalization approaches.
- Textual states adapt quickly and remain editable, while parametric policies offer reusable competence but update more slowly.
- Indefinite textual accumulation creates retrieval, context, redundancy, and management costs, whereas indiscriminate parameter updates may consolidate noisy or sample-specific behavior.
- Experience Funnel first aggregates trajectories into validated task-specific textual states containing recurring procedures, failure modes, and corrective strategies.
- Transition-Aware Skill Distillation selects state-enabled behavior that remains useful across state revisions and transfers its attributable effects into a state-free policy.
- Across question answering, embodied interaction, and web navigation, alternating state adaptation and policy consolidation improves performance while progressively converting explicit experience into autonomous competence.
2 Experience Funnel for Self-Evolving Agents
The framework alternates textual-state evolution with selective policy consolidation: validated experience is organized into state, useful cross-state behavior is identified, and the updated pair generates evidence for another round.
- Experience Funnel jointly evolves complementary state and policy representations, organizing newly acquired experience in task-specific state and consolidating reusable experience into policy.
- State Evolution: Candidate state updates summarize recurring procedures, failure modes, and corrective strategies across trajectories, then use held-out validation to filter unreliable edits.
- State–Policy Alternation: The evolution loop sends the updated state–policy pair back to deployment, where subsequent trajectories support retaining, refining, or retiring textual guidance.
- Transition-Aware Skill Distillation: Transition-Aware Rollout Selection compares state-free, previous-state, and updated-state execution to identify newly useful and persistently useful behavior for consolidation.
- Transition-Aware Skill Distillation: Token-level attribution compares identical prefixes with and without accepted state, isolating local behavioral responses that guide focused distillation into a state-free policy.
- Policy Consolidation: The policy objective combines selective distillation with state-free reward optimization, and candidate updates are committed only after satisfying the validation criterion.
3.1 Experimental Setup
The experiments evaluate state–policy consolidation on SearchQA with a Qwen3.5-4B deployment policy and fixed Qwen3.5-27B teacher, using separated data roles and five evolution rounds. Additional ALFWorld and WebShop results provide descriptive cross-environment evidence under differing OPD budgets.
- SearchQA uses Qwen3.5-4B as the evolving deployment policy and Qwen3.5-27B as the fixed OPD teacher, with hard answer accuracy as the primary metric.
- Training, validation, and test splits are separated across experience collection and updates, state-revision acceptance, and final reporting, respectively.
- Five state-evolution rounds are run; rounds 1 and 4 pass validation and trigger policy consolidation, while rounds 2, 3, and 5 are rejected.
- ALFWorld and WebShop results are treated as descriptive summaries because environments use different OPD budgets rather than identical controlled training budgets.
3.2 Overall Performance
Across three agent environments, the State–Policy Consolidation Loop achieves the strongest overall performance while progressively converting explicit experience into autonomous policy competence. Evolution trajectories show that coordinated state and policy updates outperform using either representation alone.
- The method achieves the best performance across SearchQA, ALFWorld, and WebShop, with the highest average score of 57.6%.Scores are 62.4% on SearchQA, 67.9% on ALFWorld, and 42.4% on WebShop.
- Compared with SkillOpt, OPID, and SkillRL, the method improves average performance by 3.7, 2.2, and 1.4 points, respectively.SkillOpt averages 53.9%, while the method averages 57.6%; gains over SkillRL occur on all three environments.
- Across accepted consolidation rounds, state-free policy accuracy rises from 58.1% to 61.3%, indicating progressive conversion of experience into autonomous competence.
- Combining the final consolidated policy with its evolved state reaches 62.4%, whereas the evolved state with the initial policy reaches 60.2%.The comparison indicates that explicit experience becomes more valuable when paired with the policy with which it evolves.
- The coordinated evolution of state and policy yields stronger performance than state-centric, policy-centric, and existing joint-evolution approaches.
3.3 Ablation Study
Both state evolution and policy consolidation independently improve SearchQA performance, while their combination provides the strongest result. The ablation supports their complementary roles in retaining and internalizing useful experience.
- Both adaptation mechanisms improve agent performance independently on SearchQA.
- Policy-Only Consolidation reaches 62.8% ACC, exceeding State-Only Evolution at 61.1% ACC.
- Combining state evolution and policy consolidation achieves 63.6%, outperforming State-Only and Policy-Only by 2.5 and 0.8 points, respectively.
- Consolidation absorbs reusable experience into the policy, while the evolving state retains additional guidance beyond parametric adaptation alone.
3.4 Analysis of State–Policy Consolidation
The analysis shows that policy consolidation benefits from state–policy co-evolution and selective experience selection. Consolidation transfers reusable experience into state-free policy competence while retaining complementary guidance in the explicit state.
- Dynamic State–Policy Coupling: The advantage of co-evolved state–policy pairs grows during evolution, reaching 62.4% ACC versus 60.2% when the same state uses the initial policy.At the intermediate stage, the corresponding policy improves ACC from 61.1% to 61.9%.
- Cross-Version Experience Selection: After the second consolidation round, combining newly useful and persistently useful experiences reaches 63.0% ACC, outperforming unfiltered training at 58.9%.The combined strategy improves over unfiltered consolidation by 4.1 points and over the stronger single-transition strategy by 3.1 points.
- Teacher Capacity and State-Conditioned Transfer: The teacher-capacity study finds that the 27B teacher is most effective when conditioned on the updated state, reaching 0.6121 ACC versus 0.5629 without conditioning.The unconditioned result is below the 0.5814 base policy, indicating that teacher scale alone does not explain the strongest result.
- Residual State after Policy Consolidation: Policy consolidation raises state-free performance from 58.1% to 61.3%, while the full evolved state further increases accuracy to 62.4%.The residualized configuration preserves the consolidated policy’s 61.3% performance without the complete evolved state.
4 Related Work
Related work develops both persistent textual guidance and parametric policy internalization, increasingly treating them as coupled components. Experience Funnel extends this direction by tracking how state utility changes across successive state versions and using those transitions for consolidation.
- Explicit Experience Evolution: Earlier self-evolving agents externalize feedback and successful behaviors as textual reflections, episodic experience, or reusable skills.These approaches guide later interactions without modifying model parameters.
- State–Policy Coupling: Recent methods co-evolve textual guidance with policy behavior by updating, evaluating, refining, and pruning skills under evolving policies.SkillRL and ReSkill exemplify increasingly explicit coupling between state evolution and policy optimization.
- Skill Distillation and Experience Internalization: Experience-internalization methods distill externally represented patterns or context-conditioned behavior into autonomous parametric policies.Examples include Self-Consolidation, OPCD, Skill-SD, and related skill-conditioned approaches.
- Cross-Version Experience Management: Experience Funnel differs by tracking state utility across state-free, previous-state, and updated-state executions rather than only measuring a current skill’s effect.These cross-version transitions identify newly useful, persistently useful, regressive, and inactive effects for trajectory and token-level consolidation.
5 Conclusion
The paper introduces a State–Policy Consolidation Loop that evolves textual experience and parametric policy competence together. Across several agent domains, alternating state evolution and policy consolidation improves autonomous competence while reducing dependence on accumulated textual guidance.
- Framework: The State–Policy Consolidation Loop maintains evolving textual state and selectively consolidates useful state-enabled behavior into the policy.Transition-Aware Skill Distillation selects targets using behavioral utility across state-free, previous-state, and updated-state execution.
- Empirical Conclusion: Across embodied interaction, web navigation, and execution-verified coding, alternating state evolution and policy consolidation improves autonomous policy competence over state-only and internalization-only alternatives.The process also reduces persistent dependence on accumulated textual guidance.
Limitations
The framework is best suited to repeated, interactive, and verifiable tasks with informative environment feedback, and its iterative evolution process incurs additional computation. These constraints motivate alternative utility signals and more efficient asynchronous consolidation for broader deployment.
- Scope Boundary: The framework assumes repeated related tasks with sufficiently informative environment-level feedback, limiting direct applicability to one-shot tasks and subjective domains with sparse feedback.Such settings may require learned verifiers or preference-based feedback.
- Compute Cost: Iterative evolution adds computation for multi-condition rollout evaluation, state revision, and policy consolidation, creating a compute–adaptation tradeoff.Periodic or asynchronous updates can reduce serving-latency impact but introduce delays between experience acquisition and model updates.