PaperlayerView exact revision on arXiv

Source-linked AI summary

MetaExp: Learning to Self-Improve through Experience via Verifiable Cross-Task Environment Scaling

Renhao Li, Xinjie Shen, Yang Su, Jianhong Tu, Derek F. Wong, Dayiheng Liu, Min Yang

arXiv:2610.21492v1cs.AI

TL;DR

Language-model agents often lose useful procedures between episodes, and it remains unclear whether experience learning itself can transfer beyond task-specific training. MetaExp trains this capability with verifiable task families and evaluates frozen-parameter evolution on held-out benchmarks. It reports gains across three benchmarks, supporting experience learning as a transferable capability for continued non-parametric improvement.

  • Problem

    Agents often lose useful procedures after an episode, and it remains unclear whether experience learning can be trained to transfer beyond task-specific distributions.

    Source: §1 · p. 1

  • Method

    MetaExp trains experience acquisition, updating, and reuse with verifiable task families that preserve reusable factors while varying task realizations.

    Source: §1 · p. 2, §3.1 · p. 3

  • Results

    MetaExp gains +8.5 points on SpreadsheetBench, peaks at +2.3 points on APEX-Agents, and gains +14.8 points on AutomationBench during cold-start evolution.

    Source: §1 · p. 2

  • Takeaways & Limitations

    The results support experience learning as a transferable capability for continued non-parametric improvement on unseen task distributions.

    Source: §6 · p. 9

Abstract

from arXiv · show

Language-model agents can accumulate interaction feedback, but reliably turning that feedback into reusable experience remains challenging. Existing methods either optimize external experience scaffolds or train experience-related behavior on specific task distributions, leaving open whether experience learning itself can become a transferable model capability. Training such a capability calls for supervision beyond isolated tasks, which mainly support in-episode problem solving rather than revealing what knowledge remains useful across episodes. In this work, we introduce MetaExp, a scalable framework that trains this capability through verifiable task families with controlled cross-task structure. Each family preserves a reusable factor while varying task-specific realizations, enabling transfer without direct solution reuse. We propose two complementary designs: Shared-Tool families keep black-box tools fixed across tasks to elicit operational experience, while Shared-Workflow families preserve latent procedures across changing realizations to elicit procedural experience. We scale environments under both designs, collect and filter sequential interaction trajectories, and fine-tune Qwen3.5-122B-A10B on 4,956 trajectories from 1,000 task families. With parameters frozen during downstream evolution, MetaExp improves by +8.5 points on SpreadsheetBench, peaks at +2.3 on APEX-Agents, and improves by +14.8 on AutomationBench. The gains persist under domain-excluded training, supporting experience learning as a transferable capability for continued non-parametric improvement after deployment.

1 Introduction

The paper asks whether a model can learn to acquire, update, and reuse experience across unseen task distributions, rather than merely retaining experience or solving isolated tasks. MetaExp trains this capability with structured task families and reports gains during frozen-parameter evolution on held-out benchmarks.

1 Introduction

  • Experience retention alone is insufficient: agents must diagnose outcomes, abstract useful knowledge, update experience, and apply it in later tasks.Most agents remain episodic, so procedures discovered during interaction are often lost when an episode ends.Source: §1 · p. 1
  • Prior work either improves external experience systems or trains experience-related behavior on task-specific distributions, leaving transferability difficult to assess.System-level approaches depend on the frozen model’s ability to diagnose and abstract; model-level approaches typically optimize and evaluate on related distributions.Source: §1 · p. 1
  • Verifiable task families preserve reusable factors while varying task realizations, providing cross-episode supervision without permitting direct reuse of earlier solutions.Shared-Tool families encourage operational experience from recurring tools; Shared-Workflow families encourage procedural experience across changing domains, tools, and data.Source: §1 · p. 2
  • MetaExp gains +8.5 points on SpreadsheetBench, peaks at +2.3 points on APEX-Agents, and gains +14.8 points on AutomationBench during cold-start evolution.The gains persist when semantically related training domains are excluded; MetaExp also makes better use of externally provided skills.Source: §1 · p. 2
  • MetaExp frames experience learning as a transferable capability: after one general training stage, an agent builds and reuses experience on unseen distributions without further parameter updates.The evaluation begins with an empty experience store and uses sequential interaction and feedback.Source: §1 · p. 2

2 Related Work

The related work distinguishes system-level experience scaffolds from model-level training, while noting that transferability is difficult to assess when training and evaluation use related task distributions. MetaExp instead scales task families with controlled cross-task structure and tests cold-start evolution on heterogeneous held-out benchmarks.

Experience-driven agent self-evolution

  • System-level methods externalize experience while keeping model parameters fixed, whereas model-level methods train experience acquisition, maintenance, or skill-conditioned execution.Prior approaches include verbal memories, code, and executable skills; the cited work includes Shinn et al. (2023), Zhao et al. (2024), and Wang et al. (2024; 2025b;a).Source: §2 · p. 2
  • Existing approaches often train and evaluate on the same or closely related distributions, while MetaExp evaluates cold-start evolution across heterogeneous held-out task distributions.The related work also describes CoD (Chen et al., 2026) as evaluating cross-domain generalization primarily in controlled game-like and simulated science and engineering environments.Source: §2 · pp. 2–3

Environment scaling and evaluation

  • MetaExp scales verifiable task families with controlled cross-task structure, supervising operational and procedural experience without enabling direct solution reuse.Its evaluation focuses on cold-start skill evolution across heterogeneous, work-oriented agent benchmarks.Source: §2 · p. 3

3 Problem Setup

The setup asks whether one general training stage can improve experience learning on unseen task distributions while keeping model parameters fixed during test-time evolution. It defines a sequential interaction-and-update process and a cold-start protocol that measures performance as experience accumulates.

3.1 Experience Learning Setting

  • After each task, the model may update a persistent, file-based bank of structured agent skills using the trajectory and task-level feedback.The update behavior is Uθ; experience learning includes both deciding what to update and using the resulting skills on later tasks.Source: §3.1 · p. 3

3 Problem Setup

  • Meta experience learning asks whether a single general training stage transfers the ability to acquire, update, and reuse experience to unseen task distributions.At evaluation, there are no target-benchmark parameter updates or pre-accumulated target-domain experience; adaptation occurs through the evolving experience store.Source: §3.1 · p. 3, §3 · p. 3

3.2 Cold-Start Sequential Evolution

  • SeqEvo starts with an empty experience bank and carries it forward across adaptation tasks from a held-out distribution, without future-task or target-set feedback.The adaptation tasks are processed one at a time, and k counts test-time adaptation tasks rather than gradient updates.Source: §3.2 · pp. 3–4
  • Each accumulated bank is evaluated unchanged on a fixed target set disjoint from adaptation tasks, separating zero-experience performance from the change as experience accumulates.S(k; θ) denotes performance with the accumulated bank, and Δ(k; θ) = S(k; θ) − S(0; θ) measures its change from zero-experience performance.Source: §3.2 · p. 4

4 MetaExp: Verifiable Cross-Task Environment Scaling

MetaExp trains experience-learning behavior with verifiable task families that preserve reusable structure while varying task-specific realizations. Sequential trajectories from these families supervise experience acquisition, updating, and reuse.

4.1 Cross-Task Family Design

  • Each family preserves a reusable factor while varying task-specific factors, enabling potential transfer without direct reuse of earlier solutions.The reusable factor is z; task-specific variation is ξ_i.Source: §4.1 · p. 4
  • Shared-Tool families preserve tool semantics and interfaces, while Shared-Workflow families preserve latent procedures across changing domains and tools.The two designs elicit complementary operational and procedural experience.Source: §4.1 · p. 4

4.2 Verifiable Family Scaling

  • Family scaling instantiates varied tasks and retains packages only when reference solutions execute successfully and pass executable verification.Synthesis uses a propose–solve–confirm loop; reference solutions, verifiers, and latent workflows are not exposed to the solver.Source: §4.2 · p. 4

4.3 Sequential Experience Trajectory Collection

  • Sequential collection orders family members by difficulty and interleaves some other-family tasks to encourage selective reuse of accumulated skills.The teacher solves with its current bank, receives executable feedback, then updates that bank.Source: §4.3 · p. 5

4.4 Meta-Experience Training

  • Retained trajectories train execution-stage experience-conditioned solving and reflection-stage feedback-driven updating using an assistant-token objective.System, user, and environment-observation tokens are masked from the loss.Source: §4.4 · p. 5
  • Frozen-parameter evaluation starts each held-out task distribution with an empty bank, where adaptation uses only newly accumulated experience.The skill banks used during trajectory collection are discarded after training.Source: §4.4 · p. 5

5 Experiments

Experiments evaluate cold-start experience evolution across heterogeneous benchmarks, domain-excluded training, family-design ablations, and skill use. Results examine both accumulated experience and the use of externally provided skills.

5.1 Experimental Setup

  • The study fine-tunes Qwen3.5-122B-A10B on 4,956 filtered trajectories and evaluates it under three-run SeqEvo with disjoint adaptation and target sets against a matched backbone.The training corpus draws on 500 Shared-Tool and 500 Shared-Workflow families, each with five task instances.Source: §5.1 · pp. 5–6

5.2 Main Results

  • MetaExp success rises from 65.0 to 73.5 (+8.5) on SpreadsheetBench, peaks at 14.5 from 12.2 on APEX-Agents, and rises 14.5 to 29.3 (+14.8) on AutomationBench.SpreadsheetBench gains emerge mostly within the first 10 adaptation tasks; AutomationBench gains are monotonic.Source: Table 1 · p. 6, §5.2 · pp. 6–7, Figure 2 · p. 7
  • On APEX-Agents, law gains +2.8 and management consulting +5.0 points, while investment banking falls 2.5 points as Qwen3.5 improves 3.7.The authors attribute part of this reversal to near-binary exact-value evaluation and different cold-start headroom, while noting that case-specific skills can overgeneralize.Source: §5.2 · p. 7

5.3 Domain-Excluded OOD Evaluation

  • After excluding three related training domains, Strict-OOD reaches ∆20 = +9.8 and ∆40 = +9.3 on SpreadsheetBench, comparable to full MetaExp.The retained corpus contains 3,634 trajectories, and the matched untrained backbone degrades under the same SeqEvo setting.Source: Table 2 · p. 8, §5.3 · p. 8

5.2 Main Results

  • With curated skills held fixed, MetaExp improves over its matched backbone from 33.3 to 41.4 in average score and from 47.1% to 62.1% in average pass rate.This evaluation tests using provided experience, complementing the benchmarks that measure experience accumulation.Source: Table 3 · p. 8, §5.2 · p. 8

5.5 Behavioral Analysis

  • MetaExp invokes existing skills more often despite a smaller bank, and consolidates feedback into fewer, longer skills with more failure- and verification-oriented content.The consolidated skills contain less instance-specific structure.Source: §5.5 · p. 8

5.4 Ablation Study

  • Combining designs gives the broadest gains: AutomationBench +14.8 versus Shared-Tool +7.6 and Shared-Workflow +7.1, while reducing execution by 7.0 turns.Shared-Tool has the larger SpreadsheetBench gain (+9.7 vs. +6.1); Shared-Workflow reduces execution by 2.6 turns on APEX-Agents and 2.4 on AutomationBench.Source: §5.4 · p. 8, Table 4 · p. 9

6 Conclusion

MetaExp studies whether experience learning can be trained as a transferable model capability. Its verifiable cross-task families train models to acquire, update, and reuse external experience while parameters remain fixed during downstream evolution.

6 Conclusion

  • The results suggest experience learning can transfer beyond the task distributions used for training.Source: §6 · p. 9
  • MetaExp uses verifiable cross-task families to train models to acquire, update, and reuse external experience.The model parameters remain fixed during downstream evolution.Source: §6 · p. 9
  • Across heterogeneous held-out benchmarks, MetaExp benefits more from accumulated experience than its matched backbone.The effect persists under domain-excluded training and extends to externally provided skills.Source: §6 · p. 9

A Further Analysis

The further analysis examines how MetaExp-Qwen3.5’s advantage over its matched backbone appears in experience use and updating. It focuses on SpreadsheetBench.

A Further Analysis

A.1 Experience-Conditioned Execution

On SpreadsheetBench, MetaExp-Qwen3.5 consults accumulated skills more often and earlier than the baseline. Skill-invoking tasks also show a positive success-rate difference against matched empty-bank runs for MetaExp, unlike the baseline.

A.1 Experience-Conditioned Execution

  • 79.6% of MetaExp-Qwen3.5 tasks invoke a skill, versus 14.1% for Baseline-Qwen3.5.A skill is the first tool invoked on 67.1% of MetaExp tasks and 4.8% of baseline tasks; MetaExp maintains fewer skills throughout adaptation.Source: §A.1 · p. 13
  • The skill-invocation counts include generated skills available in the current bank, excluding built-in harness skills and invalid names.Source: §A.1 · p. 13
  • +9.5 points is the average success-rate change for MetaExp skill-invoking tasks versus matched empty-bank runs, while the baseline invoked subset is 4.2 points lower.Source: §A.1 · p. 13

A.2 Feedback-Driven Experience Updating

The analysis finds that MetaExp’s feedback-driven skill banks differ in organization and content from the baseline’s. The authors caution that structural statistics do not directly measure skill quality.

A.2 Feedback-Driven Experience Updating

  • After 40 adaptation tasks, MetaExp stores fewer, longer skills than the baseline despite similar total text volume.Its banks contain more failure- and verification-related language, no detected opaque identifiers, and a larger share of invocations concentrated on a few skills.Source: §A.2 · p. 13
  • The bank statistics characterize structure, not skill quality directly.Qualitative examples show MetaExp more often records applicability conditions, causes, corrective actions, and verification, whereas the baseline favors direct procedures or local recipes.Source: §A.2 · p. 13
  • MetaExp makes accumulated skills a more active part of later task execution and consolidates interaction feedback for future reuse.Source: §A.2 · p. 13

B Seed Diversity for Environment Scaling

MetaExp draws on corpus passages, task examples, and tool schemas to provide diverse starting points for constructing controlled task families. The seed sources cover varied domains, intents, and tool capabilities, while their statistics describe the scaling substrate rather than the final training distribution.

Corpus seeds

  • Both construction modes cover all 28 corpus-seed domains, with independent sampling yielding a slightly more even task-level distribution for Shared-Workflow.Shared-Tool reuses each of 500 sampled passages across its five family members, whereas Shared-Workflow independently samples passages for all 2,500 task instances.Source: B Seed Diversity for Environment Scaling · pp. 13–14, Figure 3 · p. 15

Task-example seeds

  • The 500 task-example seeds emphasize agent configuration, software and system operations, automation, and finance, while retaining a long tail of other intents.Each example receives one primary intent through reproducible researcher annotation rather than a source-provided label.Source: B Seed Diversity for Environment Scaling · p. 14

Tool-example seeds

  • The 500 tool-seed records contain 2,338 examples spanning 16 top-level tool names, grouped into six capability categories for analysis.Each record has 1–10 examples, averaging 4.676; tool-example share and seed-record coverage measure category prevalence in different ways.Source: B Seed Diversity for Environment Scaling · p. 14

Seed diversity

  • Together, the three sources contribute semantic contexts, procedural patterns, and executable affordances without tying construction to a narrow set of manually specified environments.These statistics characterize the breadth of the scaling seeds, not the final training distribution.Source: B Seed Diversity for Environment Scaling · pp. 13, 15

C Discussion

The discussion motivates supervised fine-tuning as a practical starting point for experience learning and describes a more direct, but costlier, way to filter updates by downstream utility. The prompt templates use shared execution and reflection stages, with feedback guiding skill-bank changes.

C Discussion

  • Reinforcement learning faces delayed cross-episode credit assignment because an experience update’s utility appears only when reused on later tasks, so this work uses rejection sampling and supervised fine-tuning as a practical first step.Executable verification filters reliable teacher trajectories at substantially lower cost, providing an initial capability for acquiring and reusing experience.Source: C Discussion · pp. 15–16
  • Held-out members of a task family can evaluate old and updated skill banks on the same tasks, making downstream experience utility a stronger trajectory-rejection criterion.This targets transfer to related tasks not used to construct the bank, unlike a verifier that checks only the current task.Source: C Discussion · pp. 16–17
  • Downstream-utility filtering requires extra evaluation rollouts for each candidate update, and the current training strategy does not explicitly optimize each update’s downstream utility.The discussion leaves utility-based filtering and reinforcement learning with cross-episode reward signals as future directions.Source: C Discussion · pp. 15, 17

D Prompt Templates for Experience Learning

  • Fixed harness-level execution and reflection templates are shared across models, while runtime context such as the system message and available-skill inventory is supplied separately.The execution template initializes the task, specifies the sandbox, and instructs the agent to consult relevant accumulated skills when available.Source: D Prompt Templates for Experience… · p. 17
  • The execution prompt directs the agent to read the task, inspect available data and tools, use relevant skills, write outputs to the designated directory, verify them, and finish the task.The template specifies separate source, input, and output locations in the sandbox.Source: D Prompt Templates for Experience… · p. 17
  • After evaluation, the reflection prompt supplies the preceding trajectory and outcome so the agent can create, revise, or retain skills for future similar tasks.It asks the agent to analyze reward and failure criteria, check existing skills, and keep the skill bank within its capacity limit.Source: D Prompt Templates for Experience… · pp. 17–18

Related papers

Loading 2610.21492v1…