PaperlayerView exact revision on arXiv

Source-linked AI summary

PAST-Bench: Benchmarking the Foundations of Recursive Self-Improvement in Personal Agents

Shuhan Xue, Zixin Ding, Yichen Shen, Yinjie Wang, Zhenfei Yin, Yingcheng Wu, Yuxin Chen, Mengdi Wang, Ling Yang

arXiv:2608.04003v1cs.CL

TL;DR

Personal agents need to show that retained experience improves future behavior, but existing evaluations cannot reliably distinguish persistence gains from other sources. PAST-Bench tests this with trajectory-level persistence controls and mechanism evidence, finding capability-specific improvement and clearer Update gains from Hermes+ without a stable overall advantage.

  • Problem

    Existing one-shot evaluations cannot distinguish later-task gains caused by retained experience from gains caused by models, runtimes, prompts, shortcuts, or scoring noise.

  • Method

    PAST-Bench evaluates ordered fresh-session task families with persistence-on/off controls and trace evidence across four capabilities.

  • Results

    Across seven models and four frameworks, self-evolution is capability-specific; Hermes+ raises reported Overall Δ from +0.13 to +0.15 and Mech from 0.64 to 0.73.

  • Takeaways & Limitations

    Similar persistence gains can follow different mechanisms, so persistent-agent evaluation should report both later-task performance and evidence of the intended persistence pathway.

  • Takeaways & Limitations

    Hermes+’s +0.02 Overall improvement is smaller than run-to-run variation, so it is not interpreted as a stable overall gain.

Abstract

from arXiv · show

Recursive self-improvement requires agents to turn accumulated experience into better future behavior. Personal AI agents offer a concrete setting for studying this capability because they retain preferences, task histories, tool routines, and learned skills across sessions. Yet whether retained experience actually improves them over time has not been systematically tested. We introduce PAST-Bench, a benchmark designed to isolate this question. Each agent runs through ordered sequences of fresh-session tasks under matched conditions that turn retained experience on and off. It spans 26 scenarios and 204 episodes across memory, procedural reuse, information gathering, and update. We report both later-task gains and whether those gains follow the intended save, retrieve, and update pathway. Across seven base models and four agent frameworks, improvement is real but uneven across capabilities. Agents with the same headline gain can differ markedly in whether that gain is supported by evidence of the intended pathway. Guided by these findings, we develop Hermes+, which extends Hermes with five targeted interventions across stages of the agent loop. Hermes+ raises the average gain from retained experience and provides clearer pathway evidence, with its strongest improvement on tasks requiring outdated state to be replaced, although the effect remains capability- and model-dependent. Together, PAST-Bench and Hermes+ provide an evaluation and diagnostic foundation for studying how persistent agents can progress from retaining experience to systematically improving through it. Code: https://github.com/Gen-Verse/PAST-Bench

1. Introduction

PAST-Bench studies whether personal agents can turn retained experience into improved future behavior through online self-evolution. It isolates performance gains from retained experience and diagnoses whether agents actually save, retrieve, apply, and revise useful state.

  • Motivation: Personal agents provide a testbed for online self-evolution: reusing prior experience to change future behavior without model retraining, prompt optimization, or long-context adaptation.The capability forms a concrete substrate for stronger recursive self-improvement.
  • Motivation: Accumulated experience does not guarantee improvement because agents may store incorrect evidence, retrieve irrelevant memories, reuse brittle procedures, or apply stale state.Later-session gains may also reflect the base model, runtime, prompt, retrieval shortcuts, task difficulty, or scoring noise.
  • PAST-Bench: PAST-Bench uses trajectory-level evaluation with matched persistence-on/off controls, holding model, task family, and interface fixed while traces diagnose retained-experience pathways.Saved artifacts and execution traces reveal whether the agent wrote, retrieved, applied, or revised the intended state.
  • PAST-Bench: PAST-Bench covers 26 scenarios and 204 episodes across four capabilities, combining performance attribution with per-stage diagnosis of where retained experience helps.The benchmark evaluates agent trajectories rather than one-shot per-task scores.
  • Findings: Across seven models and four agent frameworks, Hermes and nanobot show the same Δ = +0.13 but different mechanism evidence: 0.64 versus 0.57.This discrepancy demonstrates why one-shot task scoring cannot fully diagnose self-evolution.
  • Hermes+: Hermes+ raises the reported means from Δ: +0.13 →+0.15 and Mech: 0.64 →0.73 using five runtime mechanisms, with Update reaching Δ = +0.24.Update exceeds any single mechanism, including closeout alone at +0.16 and retrieval gate alone at +0.06; the Overall Δ difference is smaller than run-to-run variation.

2. Related Work

Prior agent evaluations typically assess isolated tasks, trajectories, or individual persistence mechanisms, leaving unclear whether retained experience improves later executable tasks. PAST-Bench instead evaluates ordered episode sequences within task families and separates outcome gains from evidence that the claimed persistence mechanism caused them.

  • Evaluation scope: PAST-Bench grades episode sequences within task families, testing whether state created in earlier episodes is reused in later ones rather than evaluating a single task instance.This distinguishes it from three broad groups of prior agent evaluation organized by evaluation unit.
  • Evaluation scope: Prior interactive benchmarks score complete agent stacks on isolated task instances, conflating model capability, prompting, tool policy, and retained experience into one number.Trajectory-level benchmarks additionally score the action sequence within a task, but still differ from PAST-Bench’s episode-sequence focus.
  • Persistence mechanisms: Prior memory, skill, and architectural studies isolate individual persistence substrates but do not test whether retained experience improves later executable tasks under family-level controls.The cited lines of work cover long-horizon conversational memory, curated or self-generated skills, and architectural choices.
  • Mechanism attribution: PAST-Bench separates later-task score gains from trace-level evidence that the persistence mechanism caused them, addressing contamination concerns in self-evolution evaluation.The concern is that benchmark familiarity or artifacts can produce real score gains without transferable problem solving or causal use of persistence.
  • Benchmark comparison: PAST-Bench supports all four methodology axes jointly: cross-session retained experience, fixed-framework model comparison, fixed-model framework comparison, and trajectory-level diagnostics beyond one-shot task success.Existing benchmarks cover strict subsets, whereas the joint design is presented as necessary for retained-experience attribution.

3. PAST-Bench

PAST-Bench isolates persistent self-evolution from in-context propagation by testing whether agents retain and actively reuse state across ordered fresh-session tasks. It evaluates four capabilities with matched persistence ablations, control episodes, and pathway-oriented mechanism evidence.

  • Benchmark design: PAST-Bench separates persistent learning from long-context visibility and uninterrupted sequential state propagation by using ordered task families of fresh-session episodes.This design addresses two regimes that otherwise conflate persistent learning with in-context propagation.
  • Capability coverage: The benchmark covers Memory, Procedural Reuse, Information Gathering, and Update, requiring retention across sessions and active reuse later.Families share a latent rule, reusable artifact, correction, or pre-established reference whose retrieval must be tested.
  • Episode roles: Each family sequences cold, learn, evaluation, and control episodes to measure first-contact behavior, deposit target state, test later reuse, and rule out shortcut explanations.Evaluation episodes remove the trigger wording, while Update families also include an Update episode that deposits the correction.
  • Persistence and ablation: Matched ablations compare evaluation with and without access to family-produced state while holding prompts, graders, and tool stacks constant.The w/o-evolve condition denies access to family-produced state; the w/-evolve condition permits it.
  • Metrics and evidence: The primary metric is the self-evolution gap Δ_f between within-family mean scores with and without evolution access, accepted only after clearing family control bounds.Capability-level Δ is the macro-average across families, and the benchmark also reports writes, reads, searches, diffs, and updates as mechanism evidence.

4. Experiments

PAST-Bench compares persistence-on and persistence-off agents using task gains and mechanism evidence, revealing model- and framework-dependent self-evolution. Hermes+ improves the overall gap and especially Update, but component interactions, transfer, and run-to-run variation limit uniform conclusions.

  • 4.1 Controlled setting: PAST-Bench pairs persistence-on and persistence-off conditions, reporting later-task performance gaps alongside mechanism evidence for the intended persistence pathway.Task score and Δ measure behavior, while saved artifacts and runtime telemetry support attribution.
  • 4.2 Main results: +0.13 to +0.24: every base model gains from persistence, but models concentrate gains on different capabilities, including Memory, Update, or Information Gathering.GPT-5.4 distributes movement across Memory (38%) and Update (35%), while GLM-5.1 places 46% on Update and Kimi K2.6 places 49% on Memory.
  • 4.2 Main results: Frameworks also differ sharply: Hermes is the only baseline improving all four capabilities, whereas Agent-Zero regresses on three and Hermes+ lifts Memory, Information Gathering, Update, and Overall.Hermes ties nanobot at +0.13 Overall Δ while showing stronger mechanism evidence, 0.64 versus 0.57.
  • 4.3 Hermes+: Hermes+ raises Overall Δ from +0.13 to +0.15, ties Base Hermes at 0.66 persistence-on score, and achieves Update score 0.74 with gap +0.24.Its Procedural result declines slightly, and isolated mechanisms support their diagnosed targets: E3 yields Procedural Δ +0.10, E4 Info Δ +0.17, and E5 Update Δ +0.16.
  • 4.4 Diagnostics and transfer: Mechanism interactions and transfer are non-uniform: removing E2 raises Procedural Δ from +0.085 to +0.108, while removing E3 or E5 lowers it to +0.062 or +0.042.Across five base-model configurations, Hermes+ matches or improves Hermes on three, but DeepSeek-V4-Pro and Claude Opus 4.6 regress slightly.
  • 4.4 Diagnostics and transfer: 0.13 ± 0.04 versus 0.15 ± 0.06: Hermes+ has a higher three-run Overall gap than Hermes, but the +0.02 difference is smaller than run-to-run variation.The clearer mean shift is on Update, from +0.12 to +0.24, alongside increased σΔ from 0.01 to 0.09.

5. Conclusion

PAST-Bench attributes later-task performance to base models, runtimes, and retained experience by pairing persistence-on/off evaluations with mechanism evidence. Across seven models and four frameworks, self-evolution is capability-specific; Hermes+ improves MiniMax-M2.7’s reported mean Overall Δ and Mech, especially on Update.

  • Benchmark and findings: PAST-Bench pairs persistence-on/off evaluations within task families with mechanism evidence, separating base-model, runtime, and retained-experience contributions to later-task performance.The benchmark reports both task scores and evidence about the mechanisms underlying persistence effects.
  • Benchmark and findings: Across seven models and four frameworks, self-evolution is capability-specific, and similar persistence gaps can conceal different persistence paths.The experiments show that comparable performance gaps do not necessarily reflect the same underlying pathway.
  • Hermes+ results: +0.15 is Hermes+’s reported mean Overall Δ under MiniMax-M2.7, rising from +0.13.This is the reported mean Overall Δ comparison for Hermes+ under MiniMax-M2.7.
  • Hermes+ results: 0.73 is Hermes+’s reported mean Mech under MiniMax-M2.7, rising from 0.64.Hermes+ provides clearer mechanism evidence according to the reported mean Mech comparison.
  • Hermes+ results: Hermes+ shows its clearest gain on Update under MiniMax-M2.7.The passage identifies Update as the capability with the clearest Hermes+ improvement.

6. Future Work · A. PAST-Bench Benchmark Details · A.1. Family Taxonomy

PAST-Bench establishes a foundation for attributing cross-session improvement to retained experience, while identifying broader validity, capability coverage, mechanism attribution, and adaptive persistence as priorities for future work. Its task taxonomy distributes instances across main and sub-families, with each sample assigned to exactly one of each.

  • 6. Future Work: PAST-Bench should broaden ecological validity and temporal scope through human-authored and interaction-derived scenarios, longer sequences, and cross-family experience transfer.Current task families are synthetically constructed and evaluated in isolation.
  • 6. Future Work: Future benchmarks should expand beyond memory, procedural reuse, information gathering, and update to stronger forms of recursive improvement.Proposed directions include acquiring new tool-use strategies, revising long-horizon plans, coordinating experience across agents, and improving artifact-selection mechanisms.
  • 6. Future Work: Mechanism attribution should move beyond consistency with an expected persistence pathway toward causal tests using trace evidence and counterfactual artifact interventions.Deleting, replacing, or corrupting candidate artifacts could reveal resulting behavioral changes; larger-scale human annotations and uncertainty estimates could strengthen construct validity.
  • 6. Future Work: Hermes+’s capability-specific and model-dependent behavior suggests persistence mechanisms should not be treated as uniformly composable.Future agents could dynamically route experience across memory, skills, and session history while detecting conflicts, redundancy, and stale state.
  • 6. Future Work: Adaptive persistence should decide when structured memory rendering and procedural skill routing interact, rather than relying on fixed composition.This direction follows the observed interaction between these two persistence substrates.
  • A.1. Family Taxonomy: Table 7 reports the full PAST-Bench task distribution across main capability families and their sub-families.Each sample belongs to exactly one main family and exactly one sub-family, with sub-families covering distinct focus areas.

A.2. Task Construction and Quality Control · A.3. Trace-Backed Mechanism Case Studies · B. Metric Definitions and Aggregation

PAST-Bench uses fully synthetic, author-checked tasks to test cross-episode dependencies across four capabilities, with representative traces exposing whether agents recover the intended dependency. It reports task and mechanism-evidence scores computed from raw traces and aggregated across benchmark levels.

  • A.2. Task Construction and Quality Control: 26 task families and 204 episodes are fully synthetic, covering user-specific facts, procedural reuse, noisy-history evidence retrieval, and outdated-state replacement.Rules specify weak triggers, ambiguity, distractors, conflicting records, and transfer across fresh sessions; no task contains real-user data.
  • A.2. Task Construction and Quality Control: Tasks, prompts, graders, and expected artifacts were generated with Codex–GPT-5.4 and Claude Code–Claude Opus 4.6, without copying or adapting another benchmark.Existing benchmarks informed only the high-level scenario taxonomy.
  • A.2. Task Construction and Quality Control: Three authors checked every family for capability alignment, prompt consistency, intentional ambiguity, cross-episode consistency, nonrevealing controls, and grader-artifact agreement.Each family was checked by at least one author, including partial-credit and stale-answer rules.
  • A.3. Trace-Backed Mechanism Case Studies: Each family tests a distinct cross-episode dependency established in a learn episode and recovered in a later evaluation episode.The families are Constraint Retention (B1), Fact Correction (B2), SOP Bootstrap (B3), Exception List Recall (B4), and Failure-to-Rule (B5).
  • A.3. Trace-Backed Mechanism Case Studies: Figures 4–8 contrast representative failing and successful traces, reporting the evaluation user message and final answer while abridging intermediate reasoning and tool outputs.Captions identify the failure mode each family is designed to surface.
  • A.3. Trace-Backed Mechanism Case Studies: Figure 4 shows planning turning a retained constraint into an active filter before an irreversible share call.The representative task asks the agent to extract action items and share the source note with the appropriate recipients.
  • B. Metric Definitions and Aggregation: The evaluation framework computes task score and mechanism-evidence score (Mech) from raw traces and aggregates them to the family, capability, and benchmark levels.These are the two reported scores specified in the metric definitions section.
  • B. Metric Definitions and Aggregation: Figure 5 attributes corrected-value retrieval to typed binding metadata and a render filter; without the binding, the agent lacks an anchor and asks the user.The case concerns surfacing the current Phoenix freeze date when the current task does not restate it.

B.1. Task Score … C.1. Personal-Agent Frameworks

PAST-Bench scores task outcomes, persistence mechanisms, and grader validity across aggregated capabilities, then tests mechanism sensitivity and framework implementations. The methodology distinguishes performance gains from evidence of the intended persistence pathway while controlling framework comparisons where possible.

  • B.1. Task Score: Each episode receives completion, robustness, and safety scores from a task-specific grader evaluating the agent trace, tool calls, and audit data.Completion measures output quality, robustness measures recovery from tool-call errors, and any safety violation zeros the entire score.
  • B.2. Aggregation: Family scores average evaluation episodes, capability scores macro-average families, and overall scores average the four capability scores under matched persistence ablation.The w/o condition strips persistence while keeping prompt, grader, tools, and seed matched; it is not a cold-start score.
  • B.3. Mechanism-Evidence Score (Mech): Mech measures whether the agent completed the intended write→retrieve→correct-application persistence cycle rather than merely improving task scores.Mech = 1 indicates the full expected cycle, whereas Mech = 0 indicates the pathway was entirely absent.
  • B.3.1. Per-Episode Computation: Each episode’s mechanism scorer compares the trace with a family contract specifying artifact type, keyword patterns, minimum write/read counts, and retrieval signals.Artifact quality averages keyword hit rate and count-delta score when both are specified.
  • B.3.2. Family-Level Mechanism Score: Family-level Mech aggregates write precision, recall accuracy, update correctness, retention horizon, and pollution rate across learning and evaluation episodes.Retention horizon compares distant with near evaluations, while pollution penalizes irrelevant or out-of-scope written entries.
  • B.4. Human Validation of the LLM Judge: Human scorers agree closely, but judge–human agreement is imperfect, so MiniMax-M2.7 at temperature 0 serves as a scalable, human-validated grader.The study does not vary the judge model or prompt.
  • B.5. Sensitivity of Mechanism Evidence: 0.970–0.997 Spearman correlations and 98.8%–100% threshold agreement show Mech rankings are generally stable under tested weight and retrieval changes.Removing keyword-based artifact-content credit produces the largest change, and Mech is a pathway signal rather than causal proof.
  • C.1. Personal-Agent Frameworks: Hermes is selected as the augmentation substrate because its loop exposes memory, user-model, skill, and session-search surfaces, while other frameworks are adapter-standardized baselines.Agent-Zero receives a 1200s budget because its recursive multi-agent design often exceeds the default 300s budget; framework comparisons characterize design points rather than isolate one component.

C.2. Evaluation on General-Purpose Agents … D.3. Agent-Level Attribution Frontier

PAST-Bench extends beyond personal-agent frameworks: both tested general-purpose agents show positive persistence gains under the matched protocol. Additional analyses clarify interface requirements, mechanism-specific effects, procedural routing, and agent-level attribution.

  • C.2. Evaluation on General-Purpose Agents: The evaluation tests whether PAST-Bench applies to general-purpose agents supporting broad open-ended tasks rather than only personal assistance.
  • C.2. Evaluation on General-Purpose Agents: Codex CLI and Claude Code with MiniMax-M2.7 use identical tasks, graders, and matched persistence-on/off conditions, with results averaged over three independent runs.Table 9 reports persistence-on scores and matched gaps Δ = S_on − S_off.
  • C.2. Evaluation on General-Purpose Agents: Both general-purpose agents achieve positive matched gaps across all four capabilities, demonstrating retained-state measurement outside personal-agent frameworks.This result does not establish that either system is a personal agent or generalize to every general-purpose agent.
  • C.3. Support for Different Persistence Interfaces: The matched protocol requires only a switch controlling retained-state access, allowing black-box agents to report Task Score and Δ.Mech additionally requires observable persistence events; adapters can map exposed memory, skill, or history events to benchmark event types.
  • D. Additional Experimental Results: Figure 9 compares per-capability persistence gaps for each single-mechanism addition and the full Hermes+ system.
  • D.2. Procedural Routing Trace: Removing E2 creates a reusable DB-migration skill and raises the focused Procedural gap from +0.085 to +0.108.Under full Hermes+, learning sessions call skills_list without skill_manage, leaving no DB-migration skill for evaluation reuse; without E2, later sessions read and open it with skill_view.
  • D.1. Mechanism Ablation Heatmap: Three capability-specific mechanisms produce the largest target-capability single-mechanism gains, while the full system shows its clearest gain on Update.E3 targets Procedural, E4 targets Info, E5 targets Update, E2 raises Memory persistence-on score, and E1 provides a cross-cutting plan-time check.
  • D.3. Agent-Level Attribution Frontier: Figure 10 compares fixed-model agents along the two attribution axes reported by PAST-Bench.

D.4. Per-Family Paired Results · D.5. Score Variance Across Runs

D.4 reports full per-family paired results for Hermes and Hermes+ with MiniMax-M2.7, while D.5 summarizes cross-run score variance by capability for the two configurations with three independent runs.

  • D.4. Per-Family Paired Results: Table 10 provides the full 26-family breakdown for Hermes and Hermes+ with MiniMax-M2.7.The table reports values averaged over three runs.
  • D.4. Per-Family Paired Results: The D.4 score is measured with persistence on, and Δ is the matched persistence-on/off gap.These definitions establish how the paired results should be interpreted.
  • D.5. Score Variance Across Runs: Table 11 covers Hermes/MiniMax-M2.7 and Hermes+/MiniMax-M2.7, the two configurations with three independent runs.The table reports mean and standard deviation for both the w/ evolve score and Δ.
  • D.5. Score Variance Across Runs: D.5 aggregates the mean and standard deviation of the w/ evolve score and Δ by capability.The table is explicitly organized as per-capability score variance for Hermes and Hermes+ under MiniMax-M2.7.
  • D.5. Score Variance Across Runs: In D.5, μ denotes mean w/ evolve score, σ denotes its across-run standard deviation, μΔ denotes mean Δ, and σΔ denotes Δ’s standard deviation.These symbols define the reported variance statistics.
  • D.5. Score Variance Across Runs: Capability rows rank higher μ and μΔ as better, lower σ and σΔ as better, while the Overall row is excluded from ranking.Bold marks the best capability row and underline marks the second-best within each framework sub-column.

D.6. Computational Cost

Hermes+ substantially increases token usage over Base Hermes while adding only a modest wall-clock-time increase per episode. The added cost comes from several agent-loop interventions, and Kimi K2.6 has the highest wall-clock time because of API latency.

  • Hermes vs. Hermes+ cost: 2.5× more tokens per episode: Hermes+ uses 31,859 versus Base Hermes at 12,615.The increase comes from planning prompts, structured memory rendering, skill-list queries, gating retries, and closeout review.
  • Measurement: Table 12 reports mean computational cost per episode across all episodes using input-plus-output tokens and wall-clock time.The reported configurations are evaluated through observed wall-clock time and token usage.
  • Cross-model cost: 257.9 s wall-time per episode: Kimi K2.6 has the highest observed wall-clock time because of higher API latency.Wall-clock time includes model inference, tool execution, and overhead.

E. Reproducibility Details … F. Extended Related Work

The appendix specifies matched persistence conditions, inference and execution policies, and failure handling for reproducibility. It also situates PAST-Bench among trajectory-level evaluation and memory, procedural, and architectural research.

  • E.1. Context and Persistence Handling: Each episode starts in a fresh session, with persistence-on exposing earlier state and persistence-off removing access to it.Earlier dialogue is never appended; retained state is accessed through the agent’s native memory, skill, or history interface.
  • E.1. Context and Persistence Handling: Within matched pairs, model, agent, prompt, tools, context, limits, and compaction remain fixed; only retained-state access changes.Δ controls for context policy within a model–agent pair, while absolute cross-system scores still reflect native context-management differences.
  • E.2. Model Inference Settings: Model-side settings distinguish code-sent values, provider-documented defaults marked with asterisks, and unset fields shown as dashes.These conventions apply to the reported model inference settings.
  • E.2. Model Inference Settings: The open-ended LLM judge uses MiniMax-M2.7 at temperature 0 with a maximum output of 8,192 tokens.These settings define the judge configuration used in the experiments.
  • E.3. Agent Limits and Retry Policies: The fixed-model comparison holds MiniMax-M2.7 constant while preserving each agent’s native loop and applying the same settings to both persistence conditions.Task files impose an outer limit of 25 turns, with stopping and retry rules reported separately.
  • E.3. Agent Limits and Retry Policies: Timed-out or crashed episodes score zero, the benchmark proceeds to the next episode, and failed episodes are not rerun.This rule applies to episode-level failures during evaluation.
  • F. Extended Related Work: PAST-Bench extends interactive and trajectory-level evaluation by using within-task action evidence to diagnose whether retained state is reused.Related methods include AgentBoard, TRAJECT-Bench, and ATBench, which evaluate progress, tool-call sequences, or multi-turn safety traces.
  • F. Extended Related Work: Related work also studies information extraction, multi-session and temporal reasoning, knowledge updates, abstention, reusable procedures, and skill lifecycle management.Examples include LongMemEval, LoCoMo, SkillsBench, and research on skill optimization, curation, and lifecycle management.
Loading 2608.04003v1…