PaperlayerView exact revision on arXiv

Source-linked AI summary

SEA-Eval: A Benchmark for Evaluating Self-Evolving Agents Beyond Episodic Assessment

Sihang Jiang, Lipeng Ma, Zhonghua Hong, Keyi Wang, Zhiyu Lu, Tengfei Wang, Shisong Chen, Jinghao Zhang, Tianjun Pan, Weijia Li, Jiaqing Liang, Yanghua Xiao

arXiv:2604.08988v3cs.AI

TL;DR

Existing episodic evaluations do not capture how agents accumulate and reuse experience across task boundaries. The paper formalizes self-evolving agents, proposes SEA-Eval with sequential task streams, and finds that comparable success rates can mask divergent token-consumption trajectories.

  • Problem

    Existing benchmarks reset memory at task boundaries or emphasize static metrics, leaving cross-task evolution and efficiency convergence unmeasured.

    Source: §1 · p. 1, §2.3 · p. 3

  • Method

    The paper formalizes SEAs and introduces SEA-Eval, which assesses execution reliability and long-term evolution using sequential task streams and SR and token-consumption metrics.

    Source: §1 · p. 2, §6 · p. 15

  • Results

    With comparable success rates, OpenClaw consumes up to 31.2× as many tokens as GenericAgent on individual tasks, and their token-consumption trajectories diverge across repeated execution.

    Source: §1 · p. 2

  • Takeaways & Limitations

    Comparable success rates alone can create a capability illusion; sequential token-consumption convergence is the key signal for distinguishing genuine evolution from pseudo-evolution.

    Source: §1 · p. 2

    SEA-Eval does not assess autonomous bootstrapping: current deployments rely on human-provided task specifications to start the first execution cycle.

    Source: §7.1.1 · p. 16

Abstract

from arXiv · show

Current LLM-based agents demonstrate strong performance in episodic task execution but remain constrained by static toolsets and episodic amnesia, failing to accumulate experience across task boundaries. This paper formalizes the Self-Evolving Agent (SEA) from the perspective of digital embodiment and continuous cross-task evolution, introduces the Evolutionary Flywheel as its minimal sufficient architecture, and presents SEA-Eval -- the first benchmark designed specifically for evaluating SEAs. Grounded in Flywheel theory, SEA-Eval establishes SR and T as primary metrics and, through sequential task stream design, is designed to quantify evolutionary gain, evolutionary stability, and implicit alignment convergence. Empirical evaluation reveals that, under comparable success rates, token consumption differs by up to 31.2 times between frameworks on individual tasks, with divergent evolutionary trajectories emerging under sequential analysis -- demonstrating that success rate alone creates a capability illusion and that the sequential convergence of $T$ is the key criterion for distinguishing genuine evolution from pseudo-evolution.

1 Introduction

The paper frames self-evolving agents as systems that carry experience across tasks, then introduces SEA-Eval to assess both task execution and long-term evolution. Its evaluations show that comparable success rates can mask sharply different token costs and evolutionary trajectories.

Motivation

  • Existing episodic benchmarks reset memory at task boundaries and emphasize static success rates, leaving cross-task knowledge accumulation and execution-efficiency convergence unobserved.Consequently, frameworks with comparable success rates can differ in evolutionary quality without existing benchmarks distinguishing them.Source: §1 · pp. 1–2

SEA-Eval

  • SEA-Eval evaluates immediate execution reliability and long-term evolutionary performance, using SR and sequential token-consumption convergence as its primary evaluative primitives.It uses correlated, orthogonal, and implicit-intent task streams under two noise conditions.Source: §1 · p. 2

Empirical findings

  • With comparable success rates (85%–96.7%), OpenClaw consumes up to 31.2× as many tokens as GenericAgent on individual tasks; GenericAgent’s consumption falls roughly 68% across repetitions.OpenClaw’s consumption barely decreases, fails to form a stable convergence trajectory, and can rise when unrelated tasks are interleaved.Source: §1 · p. 2

Contributions

  • The paper formalizes SEA around digital embodiment and continuous cross-task evolution, with the Evolutionary Flywheel as its minimal sufficient architecture.It uses this formalization to distinguish genuine evolution from pseudo-evolution and characterize distillation and retrieval failures.Source: §1 · p. 2

Empirical findings

  • The paper reports a significant evolutionary bottleneck: comparable static success rates coexist with large differences in token consumption and divergent sequential trajectories.The reported maximum individual-task difference is 31.2×.Source: §1 · p. 2

2 Related Work

The related work contrasts episodic agents and isolated agent capabilities with the paper’s architecture-level, closed-loop view of evolution. It also identifies why existing benchmarks miss evolving behavior and how SEA-Eval addresses that gap.

2.1 LLM Agents and Episodic Amnesia

  • Current LLM agents can reason within tasks but reset their cognitive state at task boundaries, whereas SEA defines evolution through persistent external memory rather than parameter updates (Luo et al., 2025; Du, 2026; Kirkpatrick et al., 2017).The SEA framing also targets open-ended digital environments rather than closed task distributions.Source: §2.1 · p. 3

2.2 Technical Foundations of Self-Evolving Agents

  • Prior work advances tool creation, embodied interaction, memory, planning, and reflection separately, while this paper formalizes their interdependence as a closed-loop system.It presents that loop as the first formal characterization of the structure and its minimal sufficient conditions.Source: §2.2 · p. 3

2.3 Limitations of Existing Evaluation Frameworks

  • Mainstream benchmarks use isolated episodes, while newer benchmarks treat efficiency as static; neither captures how cross-task evolutionary dynamics unfold.Token consumption is presented as a dynamic efficiency proxy, unlike step counts or hardware-dependent latency.Source: §2.3 · p. 3
  • SEA-Eval replaces isolated episodes with persistent-memory task streams and tracks token-consumption convergence across sequential tasks.Failure to converge after repeated structurally similar tasks is identified as a signal of absent evolutionary capability (Gao et al., 2026; Cai et al., 2026; Zheng et al., 2025).Source: §2.3 · pp. 3–4

3 The SEA Paradigm

The SEA paradigm combines open-ended digital interaction with persistent cross-task evolution. Its proposed hubs close an execution-and-memory loop whose accumulated experience can improve later execution, while failures in distillation or retrieval can prevent that improvement.

3.1 Foundational Characteristics

  • SEA requires open-ended digital interaction and persistent memory across task boundaries, with prior experience reconfiguring later execution.Its formal definition includes an extensible action space, stratified memory, experience distillation, and selective cross-task transfer.Source: §3.1 · p. 4

3.2 Necessary Consequences of the SEA Characteristics

  • Open-ended environments require autonomous tool synthesis when existing tools cannot meet interaction demands, expanding the action space over deployment time.The paper treats this as a structural consequence of open-ended interaction, not an optional enhancement.Source: §3.2 · p. 5
  • With a consistent user, accumulated corrections and interaction friction are distilled into declarative memory, progressively aligning execution with latent preferences.The paper describes this alignment as an emergent consequence of sustained multi-turn evolution.Source: §3.2 · p. 5

3.3 The Evolutionary Flywheel

  • The Evolutionary Flywheel is a minimal sufficient dual-hub architecture coupling environmental interaction with persistent memory updates.Removing either hub or their interaction would violate at least one foundational SEA characteristic.Source: §3.3 · p. 5

3.3.1 Execution Hub: The Digital Limb

  • The Execution Hub gathers multimodal environmental signals, acts within the environment, and can synthesize persistent tools when existing ones are insufficient.Its inputs include screen visuals, DOM structures, system logs, and terminal outputs.Source: §3.3.1 · p. 5

3.3.2 Cognition Hub: The Cognitive Core

  • The Cognition Hub guides execution using task state, retrieved memory, and an updated plan, then distills reusable strategies, failure attributions, and user preferences into selective memory updates.Selective integration is intended to retain generalizable knowledge while suppressing instance-specific noise.Source: §3.3.2 · pp. 5–6

3.3.3 The Evolutionary Flywheel and Its Failure Modes

  • The flywheel turns interaction trajectories into memory updates and retrieves prior strategies for later tasks; distillation or retrieval failure can leave execution costs from decreasing.Both failure modes share a behavioral signature: execution cost fails to decrease across sequential variants and scales linearly with task frequency.Source: §3.3.3 · p. 6

4 SEA-Eval: Evaluation Framework and Benchmark Design

SEA-Eval evaluates both immediate execution reliability and long-term evolution, using sequential task streams to assess whether experience improves efficiency and remains stable across task boundaries. Its design combines primary and diagnostic metrics with varied task sequences and retrieval-noise conditions.

4.1 Evaluation Principles

  • SEA-Eval tracks execution cost across sequential task variants because genuine evolution should converge while pseudo-evolution does not.The framework’s evaluation principles call for tracking task success and execution overhead across temporally connected sequences rather than isolated episodes.Source: §4.1 · p. 6

4.1.1 Intra-task Execution Reliability

  • Intra-task reliability measures adaptation to complex, changing environments and recovery from failures without relying on prior accumulated experience.Agents should maintain coherent progress through long decision chains and use self-reflection and self-correction to avoid deadlocks or cascading failures.Source: §4.1.1 · pp. 6–7

4.1.2 Long-term Evolutionary Performance

  • Long-term evolutionary performance covers improvement across deployments, stability across task boundaries, and declining user intervention as preferences are internalized.The framework expects execution overhead to decrease while acquired capabilities remain usable when new tasks expand memory.Source: §4.1.2 · p. 7

4.2.1 Primary Metrics

  • Success Rate (SR) measures goal achievement, while Token Consumption (T) tracks execution overhead and whether experience-driven efficiency converges across tasks.SR is the minimum reliability guarantee; a failure of T to converge indicates pseudo-evolution even when SR is maintained.Source: §4.2 · p. 7, §4.2.1 · p. 8

4.2.2 Intra-task Auxiliary Metrics

  • Intra-task auxiliary metrics record decision steps, autonomous error recoveries, and user interventions to diagnose planning efficiency and execution robustness.Long-term metrics add adjacent-step trajectory measures: Keff tracks token-consumption change, KSR tracks success-rate change, and Kalign tracks intervention-rate change.Source: §4.2.2 · p. 8

4.2.3 Long-term Auxiliary Metrics

  • Long-term auxiliary metrics use sequential changes to diagnose efficiency, success-rate growth, and alignment; Keff converging from below zero indicates experience consolidation.Jointly analyzing Keff and KSR helps identify stagnation; Kalign tracks declining user interventions, with its negative-side convergence indicating internalization of user preferences.Source: §4.2.3 · p. 8

4.3.1 Atomic Task Construction

  • Atomic tasks span web and local-system operations at three difficulty levels, with variable-slot variants designed to preserve task logic while limiting answer memorization.Normal tasks provide the primary evolutionary discriminability; hard tasks stress-test reliability, while easy tasks establish a stable baseline.Source: §4.3.1 · p. 9

4.3.2 Sequential Task Construction

  • The benchmark combines repeated, similar, and interference-separated task sequences with noisy-memory testing; its 32 atomic tasks yield 92 sequences.Correlated sequences test learning and generalization, while orthogonal sequences test retention after unrelated tasks; noisy conditions preload 20 unrelated skills.Source: §4.3.2 · p. 10, §4.3.3 · p. 10, §4.3.4 · p. 10

5 Experiments

Across five agent frameworks, the experiments compare execution cost and success with learning, transfer, interference stability, and multidimensional capability. Results distinguish substantial learning gains from flat or unstable trajectories, while showing tradeoffs between learning depth and stability.

5.1 Experimental Setup; 5.2 Overall Performance Overview

  • In the controlled sandbox evaluation of five agents, OpenClaw used 1905K tokens at R1—14.8× Codex’s 129K—and 1119K at R3—17.6× GA’s 64K.Foundation models followed each agent’s native architecture and task requirements; usage and memory handling followed task requirements.Source: §5.2 · p. 11, §5.1 · p. 11

5.2 Overall Performance Overview

  • Learning trajectories differed: GA’s R1→R2 reduction accounted for 88% of its total optimization, Claude Code decreased 63% then 6%, and Codex decreased only 10% from R1 to R3.Low-token agents maintained 85%–95% success rates; Hermes was a counterexample, with high token consumption even on tasks not timed out.Source: §5.2 · p. 12

5.3 Evolutionary Capability Analysis

  • On identical tasks, GA’s R3 vs. R1 median change was −78%; Codex’s was −28%, with limited headroom given its 129K R1 consumption.The experiments characterize GA as iterative optimization, Claude Code and Codex as first-shot convergence, and OpenClaw as near-zero-amplitude pseudo-evolution; caching makes Claude Code and Codex consumption partly opaque.Source: §5.3 · p. 12
  • On similar tasks, GA’s A3 vs. R1 median change was −55%, but transfer magnitude varied less across agents than transfer stability, which was volatile for several frameworks.OpenClaw’s apparently flat transfer reflects persistently high consumption rather than robust transfer.Source: §5.3 · p. 12
  • After unrelated tasks, GA retained median improvements of −35.2% on identical and −29.2% on similar tasks; OpenClaw shifted from −13.1% under Orth-Same to +22.2% under Orth-Similar.GA nevertheless showed some increased-token cases attributed to memory-retrieval instability; OpenClaw’s shortcut memorization was fragile under variants or disrupted context.Source: §5.3 · pp. 12–13

5.4 Comprehensive Capability Profile

  • GA ranks first across both aggregation methods, followed by Claude Code, Codex, OpenClaw, and Hermes.The comprehensive scores are based on eight evaluation dimensions.Source: §5.4 · p. 14, Table 4 · p. 14
  • For GA’s Keff(Ret), median aggregation yields 0.643, while mean aggregation drops to 0 because a few extreme degradation tasks pull the average into negative territory, clipped to 0.Source: §5.4 · p. 14
  • The capability profiles contrast GA’s 68.0% identical-task efficiency gain and strongest learning with memory-related instability, against Claude Code’s learning–stability balance and Codex’s efficiency and consistency with little learning.GA’s orthogonal-interference Keff(Stab-Id) was 0.648, the lowest; Claude Code alone continued optimizing on similar tasks after interference, at −38.9%.Source: §5.4 · pp. 14–15

6 Conclusion

The paper formalizes Self-Evolving Agents and presents SEA-Eval to measure both immediate capability and cross-task evolution. Its results distinguish genuine evolution from pseudo-evolution, while the evaluation principles provide a foundation for future work.

6 Conclusion

  • At identical success rates, frameworks show up to approximately 30× divergence in token consumption, and only architectures with effective experience distillation demonstrate monotonic convergence.The evaluation principles provide a foundation for addressing remaining challenges in future research.Source: §6 · p. 15
  • SEA-Eval evaluates intra-task reliability and long-term evolutionary performance using Success Rate and Token Consumption metrics and sequential task streams.The framework formalizes SEAs around digital embodiment and continuous cross-task evolution, with the Evolutionary Flywheel as their structural necessity.Source: §6 · p. 15
  • Challenges remain in fully autonomous bootstrapping, subjective preference alignment, and systemic governance.The evaluation principles established in this work provide a foundation for addressing them in future research.Source: §6 · pp. 15–16

7 Discussion and Future Work

SEA-Eval addresses the evaluation of single-agent evolutionary dynamics, while the discussion identifies capability, evaluation, and deployment questions beyond its scope. These include autonomous bootstrapping, efficiency, coordination, alignment, governance, and physical embodiment.

7 Discussion and Future Work

  • SEA-Eval evaluates single-agent evolutionary dynamics, while the broader capability, evaluation, and deployment frontiers remain beyond its current scope.The discussion organizes these open directions around what SEAs can do, what remains difficult to measure, and safe, broad real-world deployment.Source: §7 · p. 16

7.1.1 Toward Fully Autonomous Bootstrapping

  • Current SEA deployments depend on human-provided task specifications to start the Evolutionary Flywheel; autonomous initiation remains unassessed.The paper identifies self-generated bottleneck detection, synthetic experiences, and iteration without external prompts as a cold-start problem and prerequisite for generalization in unknown domains.Source: §7.1.1 · p. 16

7.1.2 Balancing Reasoning Depth and Execution Efficiency

  • Large-model inference latency limits SEA applicability in real-time, high-frequency, and latency-sensitive workflows.Heterogeneous orchestration is proposed as one direction, but model responsibility boundaries and latency-aware evaluation remain open problems.Source: §7.1.2 · pp. 16–17

7.1.3 Toward Collective Evolution: Multi-agent Co-evolution

  • The formalization covers a single agent with private memory, whose bounded capacity limits evolutionary gains.Multi-agent co-evolution raises open questions about selectively sharing distilled knowledge, coupled flywheels, specialization, and system-level evaluation metrics.Source: §7.1.3 · p. 17

7.2.1 Subjective Preference and Deep Alignment

  • Kalign measures reduced user interventions as a behavioral proxy, but does not capture subjective judgments such as aesthetic or stylistic quality.The discussion calls for richer preference elicitation to assess convergence with latent user intent.Source: §7.2.1 · p. 17

7.3.1 Safety and Systemic Governance

  • Broader permissions increase evolutionary potential while scaling the risks of autonomous self-modification, and agent-level safeguards may fail through distillation.The paper points to permission management and sandboxed execution as system-level controls, while principled permission calibration remains unresolved.Source: §7.3.1 · p. 17

7.3.2 From Digital Embodiment to Embodied AI

  • The modality-independent SEA loop may transfer from digital to physical embodiment through modality adaptation rather than architectural reinvention.Digital environments are presented as a high-fidelity, low-cost testbed for evolutionary mechanisms needed by physical robots.Source: §7.3.2 · p. 17

8 Ethics Statement

The benchmark uses expert-constructed tasks based on public web resources and local system scenarios, with evaluation conducted in sandboxed environments. The paper also describes limited AI assistance in writing and diagram presentation.

8 Ethics Statement

  • SEA-Eval tasks use publicly available web resources or local scenarios; web tasks target public sites, local operations run in isolated sandboxes, and experiments use containerized sandboxes.The paper states that no personal or user-sensitive data are involved and that agent operations do not affect host systems or external environments.Source: §8 · pp. 17–18

AI-Assisted Writing and Figure Generation

  • Claude Sonnet 4.6 assisted with language polishing and Gemini 3 with framework diagrams; the authors report completing the research design, implementation, analysis, and core arguments independently.The stated AI use was limited to English expression and diagram presentation rather than substantive research contributions.Source: AI-Assisted Writing and Figure… · p. 18

A Normalization Methodology

The appendix normalizes raw metrics to [0, 1] for radar-chart comparison using metric-specific strategies. Stability metrics use exponential decay to reflect the differing practical significance of small and large fluctuations.

A Normalization Methodology

  • Raw metrics are normalized to [0, 1] for radar-chart comparison using metric-specific strategies.The strategies are shown in Table 5.Source: A Normalization Methodology · p. 20
  • Stability metrics use exponential decay so fluctuations below 20% incur less penalty while fluctuations above 50% are substantially penalized.The function exp(−|x|) has a gentle gradient near zero and rapid decay farther away.Source: A Normalization Methodology · p. 20

B Functional Boundary Between Φ and Ψ

Φ distills a single-task trajectory into candidate memory updates, while Ψ selectively integrates updates into persistent memory. Their sequential coupling means evolutionary quality depends on both generalizable abstraction and selective commitment.

B Functional Boundary Between Φ and Ψ

  • Φ extracts a critical decision subsequence from one task trajectory and abstracts it into candidate updates for strategy, episodic, and declarative memory.Its output does not directly modify persistent memory.Source: B Functional Boundary Between Φ and Ψ · p. 20
  • Ψ selects which candidate updates enter persistent memory, suppressing nongeneralizable or redundant entries to preserve stability and limit retrieval overhead.The goal is memory that grows in quality rather than merely in cardinality.Source: B Functional Boundary Between Φ and Ψ · p. 20
  • Φ runs before Ψ, so future memory quality depends jointly on abstraction quality and integration selectivity.An overly conservative Ψ can suppress useful updates, while an overly permissive Ψ can admit instance-specific noise.Source: B Functional Boundary Between Φ and Ψ · p. 20
  • Episodic agents have null Φ and Ψ operations; the SEA definition requires both functions to be non-trivial.This boundary applies regardless of within-task reasoning sophistication.Source: B Functional Boundary Between Φ and Ψ · pp. 20–21
  • Table 5 is identified as covering normalization strategies for each metric dimension.Source: Table 5 · p. 21

C Diagnostic Analysis of Evolutionary Flywheel Failure Modes

Distillation and retrieval failures share a primary behavioral signature: execution cost does not converge as tasks accumulate. Auxiliary execution traces can help distinguish their causes.

C Diagnostic Analysis of Evolutionary Flywheel Failure Modes

C.1 Formal Characterization of Failure Modes

The appendix distinguishes distillation failure, where updates do not yield reusable strategies, from retrieval failure, where relevant strategies exist but are not surfaced. It outlines separate diagnostics using memory updates and retrieval traces.

C.1 Formal Characterization of Failure Modes

  • Distillation failure occurs when Φ misses the outcome-critical subsequence or produces an update too poorly abstracted to transfer to similar tasks.The update may encode instance-specific details even when Ψ commits it.Source: §C.1 · p. 21
  • In distillation failure, memory entries may accumulate without reducing execution overhead on subsequent tasks.Source: §C.1 · p. 21
  • Distillation failure can stem from noisy or contradictory trajectory actions obscuring causal structure, or insufficient abstraction capacity to form reusable strategies.Source: §C.1 · p. 21
  • Retrieval failure occurs when relevant, generalizable strategies exist in Mp but similarity ranking leaves them below the retrieval threshold.Environmental drift, such as interface changes or domain shifts, can make surface-level similarity unreliable.Source: §C.1 · p. 21

C.2 Diagnostic Distinction via Auxiliary Execution Traces

  • With detailed logs, memory-update monitoring can diagnose distillation failure when Mp grows but overhead does not fall and strategy reuse remains near zero.The reuse rate is the proportion of task steps invoking retrieved strategies rather than zero-shot reasoning.Source: §C.2 · pp. 21–22
  • Retrieval traces support diagnosis when relevant entries receive low similarity scores and are not surfaced to the Execution Hub.Large rank changes under surface-level reformulations that preserve task structure are another diagnostic signal.Source: §C.2 · pp. 21–22

C.3 Implications for SEA-Eval

Token consumption is the first-order signal for pseudo-evolution, while execution steps and self-correction frequency offer partial diagnostic information. Full diagnosis depends on internal logs that may be unavailable in black-box evaluations.

C.3 Implications for SEA-Eval

  • Token consumption trajectory is the first-order signal for detecting pseudo-evolution in SEA-Eval.Source: §C.3 · p. 22
  • High Nfix with stable S is more consistent with distillation failure, while low Nfix with high T is more consistent with retrieval failure.These auxiliary metrics provide partial diagnostic information.Source: §C.3 · p. 22
  • Full diagnosis requires internal memory-update logs and retrieval traces, which may be inaccessible in black-box evaluations.SEA-Eval treats diagnosis as best-effort auxiliary analysis and encourages standardized logging interfaces in future extensions.Source: §C.3 · p. 22
Loading 2604.08988v3…