PaperlayerView exact revision on arXiv

Source-linked AI summary

Your Agents Are Aging Too: Agent Lifespan Engineering for Deployed Systems

Jianing Zhu, Yeonju Ro, John Robertson, Kevin Wang, Junbo Li, Haris Vikalo, Aditya Akella, Zhangyang Wang

arXiv:2605.26302v1cs.AIcs.CLcs.MA

TL;DR

Long-lived agents are evaluated too often at day one, leaving their reliability over deployment insufficiently characterized. AgingBench provides longitudinal, mechanism-level diagnosis of agent aging, finding that degradation is multidimensional and can occur at different memory-pipeline stages.

  • Problem

    Day-one benchmarks do not establish whether long-lived agents remain dependable as their full harness state changes across sessions.

  • Method

    AgingBench evaluates four aging mechanisms longitudinally and uses paired counterfactual probes to diagnose failures across memory writing, retrieval, utilization, and lifecycle handling.

  • Results

    Agent aging is multidimensional: behavioral tests can remain clean while factual precision decays, derived-state tracking can collapse within one model, and maintenance can trigger abrupt regressions.

  • Takeaways & Limitations

    Reliable deployment requires lifespan evaluation, mechanism-level diagnosis, and stage-targeted repair rather than relying only on day-one model capability.

  • Takeaways & Limitations

    The benchmark’s mechanisms are characterized within controlled session horizons, leaving validation at real-user timescales in production deployments as an open frontier.

Abstract

from arXiv · show

Long-lived AI agents are increasingly deployed as persistent operational systems, yet they are still evaluated like freshly initialized models. Day-one benchmarks miss a basic systems question: how long does an agent remain reliable after deployment? Even when model weights are frozen, an agent's effective state keeps changing as it compresses interaction history, retrieves from a growing memory store, revises facts after updates, and undergoes routine maintenance. Reliability therefore becomes a lifespan property of the full agent harness, not only a snapshot property of the base model. We introduce AgingBench, a longitudinal reliability benchmark for agent lifespan engineering: measuring not only whether deployed agents degrade, but what form the degradation takes and where repair should target. AgingBench organizes agent aging into four mechanisms: compression aging, interference aging, revision aging, and maintenance aging. To diagnose these failures, AgingBench uses temporal dependency graphs and paired counterfactual probes that produce diagnostic profiles for the write, retrieval, and utilization stages of the memory pipeline. Across 7 scenarios, 14 models, multiple memory policies, and both runner-controlled and autonomous agents, over ~400 runs spanning 8 - 200 sessions show that agent aging is not one-dimensional: behavioral tests can remain clean while factual precision decays; derived-state tracking can collapse sharply within a single model; and the same wrong answer can require different repairs depending on what the diagnostic profile points to. These results suggest that reliable agent deployment requires lifespan evaluation, mechanism-level diagnosis, and stage-targeted repair, not only stronger day-one models.

1. Introduction

Long-lived agents can become less reliable over time even with frozen model weights, because their full harness accumulates changing memory state, interaction history, revisions, and maintenance effects. AgingBench frames this as agent aging and evaluates its mechanisms, lifespan curves, and repair targets through temporal dependencies and counterfactual memory-pipeline diagnostics.

  • Motivation: Deployed agents are time-evolving harnesses whose reliability cannot be captured by day-one benchmark scores or frozen model weights alone.Their behavior changes through memory writing, storage, retrieval, utilization, tools, prompts, workspaces, and maintenance procedures.
  • Problem: Agent aging names time-dependent reliability degradation caused by changing memory state, accumulated interaction history, and lifecycle events.The resulting failures include omissions, confusion between similar entities, stale facts, and regressions after routine memory operations.
  • Framework: AgingBench organizes degradation into four mechanisms: compression, interference, revision, and maintenance aging.These mechanisms respectively capture write-time loss of future-relevant details, crowding by similar memories, failure to update changed or derived state, and regressions triggered by lifecycle events.
  • Evaluation: AgingBench measures operational-lifetime aging with temporal dependency DAGs, mechanism-specific metrics, and curves rather than a single snapshot score.The DAG represents superseding facts, long-separated dependencies, confusable entities, and controlled lifecycle events.
  • Diagnostics: Paired counterfactual probes diagnose whether failures arise from writing, retrieval, utilization, or lifecycle shocks.The harness replaces retrieval with an oracle over agent-written memory, then replaces writing and retrieval with gold context to produce repair-oriented profiles.
  • Findings: Across 7 scenarios and 14 models, agent aging was multi-dimensional: behavioral compliance could remain clean while factual precision decayed, derived-state tracking could collapse, and maintenance could cause abrupt regressions.Strong models could preserve information yet fail to reuse it, and identical aggregate failure rates could conceal different repair needs.

2. Related Work

Existing work studies multi-session memory and long-horizon agent capabilities, but snapshot benchmarks miss deployment-time degradation and evolving memory. AgingBench provides a lifespan-engineering evaluation foundation with aging curves, temporal dependency graphs, lifecycle event injection, and component-aware diagnostics.

  • Multi-session memory and long-horizon capabilities: Existing work increasingly examines multi-session memory and long-horizon capabilities, while AgingBench focuses on evaluating agent lifespan engineering.AgingBench instruments this evaluation with aging curves, a temporal dependency DAG, lifecycle event injection, and component-aware diagnostic profiles.
  • Degradations in deployed agents: Deployed agents can degrade through context compression, cross-entry retrieval interference, and other pressures that snapshot benchmarks do not capture.Examples include losing low-frequency project details after compression and retrieving the wrong client’s budget from similar memory entries.
  • Lifecycle events and attribution for system harness: Existing benchmarks rarely control lifecycle events or measure their effects on reliability, and failure attribution remains largely unaddressed.Operational events such as memory compaction or flushing are routine in deployed agents, but existing evaluations generally assume static environments in which memory does not evolve.

3. Agent Aging Taxonomy

Agent aging comprises four mechanisms: compression, interference, revision, and maintenance. Compression and interference accumulate as state grows, whereas revision and maintenance are triggered by discrete changes; their prominence depends on deployment regime and determines how failures should be interpreted.

  • Aging mechanisms: Four aging mechanisms organize long-lived agent degradation: compression, interference, revision, and maintenance.Compression and interference are accumulation-driven; revision and maintenance are event-driven.
  • Compression aging: Compression aging discards low-frequency details at write time because future queries are unknown, with higher compression preserving mainly high-level summaries.Dollar amounts, proper nouns, and constraint values are discarded before broader summaries.
  • Interference aging: Interference aging crowds out target facts during retrieval as stored state grows, even without information loss or factual revision.Freezing all facts does not prevent interference, making it distinct from revision aging.
  • Revision aging: Revision aging fails to propagate changed facts, while one missed update to dynamic latent state can contaminate every subsequent derived answer with compounding errors.These errors can remain invisible to standard keyword recall.
  • Maintenance aging: Maintenance aging arises when recompaction, prompt updates, or log cleanup silently alter behavior, producing a performance cliff or regression.Unlike the other mechanisms, maintenance aging is driven by actions taken on the agent.
  • Deployment scenarios: All four mechanisms can co-occur, but their relative prominence varies by deployment regime, so the same wrong answer may require different interpretations and repairs.Different deployments naturally encounter different subsets of the mechanisms.

4. AgingBench : A Benchmark for Agent Lifespan Engineering

AgingBench makes agent aging measurable through scalable, cross-session task generation, temporal dependency graphs, and a session-based evaluation loop. It produces mechanism-linked aging curves and shows downward trajectories across scenarios, with rate and shape varying by mechanism.

  • Benchmark structure: AgingBench generates scalable multi-session task streams with temporal dependency DAGs encoding fact version chains, latent-state accumulators, dependency depth, and confusable entities.Programmatic generators produce the task stream, fact registry, and DAG for arbitrary session counts, while configurable pressures control dependency density, update rate, chain depth, and interference.
  • Evaluation procedure: The evaluation runs agents over N sessions, reading compressed memory, answering tasks and held-out probes, scoring each session, then compressing interaction history into the next state.Runner-injected maintenance events can disrupt memory or policy parameters, enabling controlled measurement of maintenance effects.
  • Mechanism-linked scoring: AgingBench computes aging curves using half-life, decay slope, and hazard proxy, while tying metrics to DAG structures and their corresponding compression, interference, revision, or maintenance mechanisms.Revision metrics include current-version accuracy and derived-value accumulation, rather than relying on a single aggregate recall score.
  • Aging curve preview: Every scenario shows an overall downward trajectory across the evaluation horizon, although the rate and shape of decline vary by aging mechanism.The benchmark preview reports these trajectories under two contrasting memory policies; detailed mechanism-level findings are deferred to Section 6.2.

5. Component-Level Attribution

Component-level attribution decomposes agent aging across write, store, retrieval, and utilization stages, then uses oracle counterfactual probes to identify which stage most reduces observed error. Diagnostic profiles show that similar aggregate failure rates can reflect different bottlenecks and therefore require different repairs.

  • Pipeline decomposition: The memory pipeline comprises Write/Compression Policy (W), Memory Store (S), Read/Retrieval Algorithms (R), and Utilization Logic (U).W persists session history, S holds persistent data, R extracts task-relevant context, and U plans retrieval and synthesizes responses.
  • Counterfactual attribution: Three oracle counterfactual probes form an ablation ladder whose accuracy gaps identify the first non-oracle component consistent with failure.The P1/P2/P3 ladder yields a stage-level diagnostic profile across Write, Retrieval, and Utilization.
  • Maintenance attribution: Maintenance shocks are separated temporally from gradual write errors because both can appear as missing facts in the memory store.Maintenance effects are measured immediately across lifecycle event time t using ΔS = WriteError_t+ − WriteError_t−.
  • Diagnostic profiles: Across three models, total error rates cluster between ∼0.60 and 0.82, while component compositions remain heterogeneous across scenarios and models.S1 is Utilization-dominated, S2 Write-dominated, and S5 shifts from near-pure Write failure to substantial Read/Interference depending on model.
  • Repair targeting: Attribution distinguishes repair paths: S1 needs utilization-stage operators, S2 value-preserving compaction, and strong-model S5 a planning-loop fix forcing re-reads.Aggregate error alone would prescribe the same memory-increasing fix, whereas decomposition identifies stage-specific interventions.

6. Results

AgingBench shows that deployment-time memory aging is multi-dimensional: models can trade off across mechanisms, while behavioral compliance remains intact as factual accuracy deteriorates. Mechanism-level evaluation reveals distinct failures in revision, self-managed memory retrieval, and model-specific maintenance responses, enabling different repair paths.

  • Finding I: No single model dominates across aging mechanisms; methods leading under one mechanism are often average or worst under another.Rank reversals recur across Table 3 rather than appearing as isolated comparisons.
  • Finding I: Routine maintenance events produce abrupt, model-specific post-event regressions.The Δshock contrasts in Table 3 and Figure 7d distinguish post-event responses across maintenance variants.
  • Finding II: Behavioral compliance can remain near-perfect while factual precision deteriorates after compression, producing confident incorrect answers that behavioral metrics miss.On S2, explicit constraint violations remain near zero while constraint precision drops; mechanism-level fact-recall probes are required to detect the drift.
  • Finding III: Revision aging does not improve consistently with model size or memory-policy changes, indicating a representational state-update problem rather than capacity alone.The S2 accumulator-error results show no consistent larger-model advantage, and policy changes do not reliably reduce error across Tier 1.
  • Finding IV: Across all Tier 2 configurations, workspace fidelity exceeds downstream recall, revealing a persistent write–read gap in self-managed memory.The gap is smaller for Claude Code and larger for OpenHands, while correct responses involve more retrieval activity than incorrect ones.
  • Finding V: Within Claude Code, Opus-4.7 has the lowest pytest and ws_fid despite competitive interference resistance and revision accuracy, isolating degradation at write-time outputs.Multi-mechanism columns separate the model’s retrieval-stage strengths from its weaker write-time behavior.

7. Conclusion … B Metric Definitions and Scoring

Long-lived agents can degrade quietly after deployment even with frozen model weights, making reliability a lifespan property of the full agent harness. AgingBench supports agent lifespan engineering by organizing degradation into four mechanisms and measuring them systematically.

  • 7. Conclusion: Frozen model weights do not prevent reliability degradation as memory state drifts and accumulates error across sessions.The conclusion frames reliability as a property of the deployed harness rather than the model alone.
  • 7. Conclusion: AgingBench organizes agent aging into compression, interference, revision, and maintenance mechanisms.These four mechanisms provide the paper’s structure for analyzing degradation.
  • 7. Conclusion: AgingBench enables agent lifespan engineering through systematically generated scenarios that measure degradation over deployment.The supplied conclusion passage identifies systematic scenario generation as part of the benchmark’s approach.
  • Appendix: The Appendix includes a subsection on aging curve statistics.This is identified as Appendix B.1 in the supplied section listing.
  • B Metric Definitions and Scoring: Metric Definitions and Scoring covers aging curve statistics, DAG-derived metrics, and headline metric definitions and selection.The supplied contents list these as subsections B.1, B.2, and B.3.

C Scenario Details and Task Illustrations … Appendix

The appendix organizes supplementary material on long-lived agent aging across scenario details, diagnosis, experiments, implementation, case studies, evaluation, broader discussion, and supporting appendices. It also includes a conceptual illustration of post-deployment agent aging.

  • C Scenario Details and Task Illustrations: Scenario details cover scenario curation, session anatomy, and per-scenario task examples and summaries.These materials appear in Appendix C.
  • D Component-Aware Diagnosis: Conditions and Agent Architectures: Component-aware diagnosis covers diagnostic-probe design, a typed-state revision-aging intervention, runtime control, production adaptation, and an Opus-4.7 re-read ablation.These extensions appear in Appendix D.
  • E Additional Experimental Results: Additional experimental results include a findings-matrix reference, detailed setup, supplementary evidence, multi-seed validation, and PressureConfig as a controlled evaluation tool.These materials appear in Appendix E.
  • F Implementation Details: Implementation details describe experiment execution, generator and pressure configuration, memory policies, compaction prompts, and cost and runtime footprint.These materials appear in Appendix F.
  • G Case Studies: Case studies address memory degradation under compression and the tracing of a compounding error.These case studies appear in Appendix G.
  • Appendix: The appendix supplements the paper’s exploration of long-lived AI agent aging and extends related work and metric formalization.Appendix A extends related-work discussion, while Appendix B provides further metric details.
  • Appendix: The table of contents identifies later sections on an evaluation card and broader discussion as part of the consecutive supplementary material.Sections H Evaluation Card and I Broader Discussion are listed in the merged section range, though their detailed passages are not supplied here.
  • Appendix: The appendix also contains a conceptual illustration of long-lived agent aging after deployment.This illustration is presented as Figure 8.

A. Extended Related Work · B. Metric Definitions and Scoring · B.1. Aging Curve Statistics

AgingBench extends prior work by evaluating longitudinal reliability across multi-session dependencies, lifecycle events, measurable aging, and component-aware diagnosis, rather than snapshot capability alone. It formalizes aging-curve statistics that quantify degradation speed, shape, and threshold crossing.

  • A. Extended Related Work: AgingBench frames longitudinal evaluation around multi-session evaluation, cross-session dependencies, lifecycle event control, measurable aging, and component-aware diagnosis.These five dimensions characterize how deployed agents change across their operational lifetime.
  • A. Extended Related Work: Existing memory benchmarks primarily measure capability at one evaluation point, whereas AgingBench studies how memory-supported behavior changes across the agent’s operational lifetime.This contrasts snapshot memory capability with longitudinal reliability.
  • A. Extended Related Work: Multi-session memory compaction shifts the bottleneck from single-window attention to the write→store→read pipeline operating under a fixed budget.Only the compacted memory artifact persists into later sessions, making the four aging mechanisms distinguishable as failure modes.
  • A. Extended Related Work: Prior benchmarks generally lack the joint combination of controlled longitudinal pressure across the five aging-evaluation dimensions.Related work addresses capabilities individually, but the surveyed combination is not present to the authors’ knowledge.
  • B. Metric Definitions and Scoring: AgingBench formalizes headline and DAG-derived metrics by defining their formulas and linking them to scoring functions.The metric framework is designed to support adding new metrics through the same definition-and-scoring process.
  • B.1. Aging Curve Statistics: From each aging curve m(t) = {s_0, s_1, ..., s_N}, the framework computes summary statistics over a session horizon of length N.These statistics summarize degradation trajectories rather than isolated evaluation points.
  • B.1. Aging Curve Statistics: Half-life is the first session crossing 0.5 · m(0), decay slope is the OLS per-session loss rate, and hazard proxy estimates per-session failure probability below a scenario-dependent threshold.Half-life uses linear interpolation and returns ∞ if the threshold is never crossed; negative decay slope indicates degradation.
  • B.1. Aging Curve Statistics: At t=0, m(t) is the mean over the run window when attribution conditions are summarized as a single bar per cell.This statistic provides the baseline summary used for aggregated attribution displays.

B.2. DAG-Derived Metrics … D.4. Production-Level Agent Adaptation

AgingBench defines mechanism-specific DAG metrics, generator-based longitudinal scenarios, and paired probes that locate failures in memory writing, retrieval, or utilization. It then evaluates targeted repairs, including typed-state overlays and lightweight runtime control, while extending diagnosis to production-agent architectures.

  • B.2. DAG-Derived Metrics: DAG-derived metrics map dependency edges, version chains, interference pairs, and accumulators to compression, revision, interference, and derived-state measurements.Per-hop analysis identifies the first failing dependency hop, while shock_delta compares seed-matched shock and control runs.
  • B.3. Headline Metric Definitions and Selection: Headline metrics are selected for meaningful variation across the session horizon while primarily isolating one aging mechanism.The selection avoids alternatives that saturate, such as CVR near zero for safety-tuned models and LA at 1.0 for some models.
  • C.3. Per-Scenario Examples and Summary: The seven scenarios cover research, lifestyle, enterprise knowledge, software engineering, naturalistic multi-domain, self-management, and self-planning deployments with distinct memory and mechanism pressures.Examples include specific-value recall, evolving constraints, retractions, workspace confusion, lifecycle shocks, API interference, and schema or storage migrations.
  • C.1. Scenario Curation and Generator Rationale: Scenario generators produce fresh, seed-reproducible tasks at arbitrary session counts to apply controlled longitudinal pressure across four aging mechanisms.The benchmark intentionally favors mechanism-faithful control over the full distribution of naturalistic user behavior, and its scenario set is designed to be extendable.
  • C.2. Session Anatomy: System Prompts and Input Construction: Each session combines a role-defining system prompt, compressed prior memory, and current tasks, while self-managed agents must proactively read workspace files.Tier 1 injects compressed memory; Tier 2 instead relies on the agent’s own workspace, making lossy compression and file selection observable.
  • S2 Accumulator Probe (Revision Aging — Latent State): Derived-state probes expose compounding revision failures: the correct remaining dining budget is $154, whereas missing one expense yields $222.The gold value is computed as $309 − $87 − $68, and the accumulator value never appears directly in session text.
  • D.1. Diagnostic Probes: Design Choices and Alternatives: The paired P1–P3 diagnostic set attributes memory loss to utilization, writing, or retrieval by replacing the deployed pipeline with progressively stronger oracles.The differences 1 − Acc_P3, Acc_P3 − Acc_P2, and Acc_P2 − Acc_P1 provide the three attributions.
  • D.2. Typed-State Overlay: A Targeted Intervention for Revision Aging: Targeted interventions reduce revision aging: a typed-state overlay lowers accumulator error on both backends, while a threshold controller captures roughly 91% of the always-on ceiling at 86% of its wall time.Retroactive recompaction backfires, with the aggressive forward-only trigger capturing −43% versus −24% for matched retroactive recompaction; diagnosis also guides adaptation to production-level agent architectures.

D.5. Opus-4.7 Re-read Ablation … E.4. Multi-Seed Validation

The re-read ablation separates Opus-4.7’s retrieval-related and task-phase failures, while supplementary experiments characterize AgingBench across models, agents, memory policies, temporal horizons, and multiple seeds. These results show aging trends, capability-dependent compaction benefits, temporal-distance effects, persistent horizon gaps, and run-to-run variability in probe outcomes.

  • D.5. Opus-4.7 Re-read Ablation: Forced re-reading raises retrieval metrics but leaves Opus-4.7’s late-session pytest collapse largely intact.Recall rises from 0.68 to 0.91, ws_fid from 0.75 to 0.83, and accum_err falls from 2.25 to 0.00; pytest improves only from 0.65 to 0.70.
  • E.1. Findings Matrix Reference: The supplementary findings matrix links five mechanism-level findings to supporting evidence and practical implications, including preliminary typed-state and runtime-controller interventions.The interventions target revision aging and maintenance-related behavior, respectively.
  • E.2. Detailed Experimental Setup: The evaluation spans 14 models, three agent frameworks, runner-controlled and autonomous tiers, multiple memory policies, and replicated 8–12- or 10-session runs.The models range from 7B to 120B open-source systems plus multiple closed-source API versions; 10-session runs total approximately 40K tokens and 200 calls.
  • E.3. Supplementary Evidence: Under lossy compression, Tier-1 aging curves trend downward across most tested configurations, consistent with compression aging’s structural-limit interpretation.The benchmark releases per-run aging-curve data with the code.
  • E.3. Supplementary Evidence: Careful compaction widens across-model performance gaps rather than lifting every configuration to a shared ceiling, with smaller models sometimes scoring lower than under lossy compaction.On S3 fidelity, a smaller model scores lower under careful than under lossy, indicating a capability threshold for exploiting preserved content.
  • E.3. Supplementary Evidence: Lag-recall decreases monotonically as the session gap grows, even when raw memory content remains preserved.On S6, same-session recall at lag = 1 exceeds recall at lag 8–10 across Gemma-4, GPT-4o, and Llama-3.1.
  • E.3. Supplementary Evidence: On GPT-4o S1, careful compaction preserves more content at short horizons, but both policies converge toward a shared floor at long horizons while the cumulative gap persists.The result is consistent with a write-before-query barrier limiting performance within the tested window.
  • E.4. Multi-Seed Validation: Three-seed validation finds workspace fidelity highly stable, with std ≤0.02, while per-probe outcome metrics contain most run-to-run variance.The validation fixes model and memory policy while varying each seed’s generated task stream; reported std excludes API-side decoding stochasticity.

E.5. PressureConfig as a Controlled Evaluation Tool … Lossy Compaction Prompt

PressureConfig provides controllable difficulty axes that isolate aging mechanisms, while the benchmark’s implementation exposes reproducible runs, generators, memory policies, and compaction prompts. Controlled sweeps show targeted degradation with largely stable bystander metrics, and compaction prompts define contrasting preservation endpoints.

  • E.5. PressureConfig as a Controlled Evaluation Tool: PressureConfig factorizes deployment difficulty into continuous axes for information density, fact revision, cross-domain interference, and dependency reach.It enables memory-relevant difficulty changes without retraining, redesigning tasks, or changing the agent.
  • E.5. PressureConfig as a Controlled Evaluation Tool: n_confusable_pairs from 0 to 12 drives interference_resistance from 1.0 to 0 while fidelity 𝑚𝐹 stays flat.The sweep provides the canonical controlled-evaluation signature: the targeted metric changes monotonically while untargeted metrics do not.
  • E.5. PressureConfig as a Controlled Evaluation Tool: Chain recall declines 0.84 →0.33 as source-session span widens from 1 to 8, and higher dependency density produces steeper span-decline.Denser dependency graphs surface more long-span probes, where recall fails.
  • E.5. PressureConfig as a Controlled Evaluation Tool: DAG pressure dials produce controllable, reproducible aging in targeted mechanisms while bystander metrics hold in place.The experiments support using each axis as an independent variable for aging studies, with dose-response generalizing across scenarios.
  • F. Implementation Details: The implementation documents benchmark subsystems for reproducibility, with metric definitions in Appendix B and emphasis here on code architecture and configuration.This frames the implementation material as supporting reproducible benchmark use.
  • F.1. Running an Experiment: Each run accepts a scenario, SUT configuration, session count, and diagnostic-probe selector, then emits per-session metrics, graph diagnostics, traces, and an aging-curve plot.Tier-1 runners seed random and torch.manual_seed; Tier-2 runners use random.Random(seed) because the external agent CLI is the source of randomness.
  • F.2. Generator and Pressure Configuration: Scenario generators maintain FactGraphs covering facts, version chains, dependency edges, interference pairs, and accumulators, with pressure parameters controlling DAG topology.Available knobs include tokens_per_session, dependency_density, update_rate, max_chain_depth, n_confusable_pairs, confusable_start_session, and warmup_sessions.
  • F.3. Memory Policies and Compaction Prompts: The main experiments use no_memory, append_only, summarize_store, growing_history, lossy_episodic, and workspace memory policies, with lossy policies generating summaries through LLM-governed compaction prompts.The compaction prompt is the single parameter varied across the two compaction endpoints.

F.4. Cost and Runtime Footprint … H. Evaluation Card

The evaluation card combines a low-cost pre-deployment subset with case studies showing how compression and revision aging arise, propagate, and require mechanism-specific diagnosis. AgingBench reports mechanism-level aging curves and counterfactual stage profiles while explicitly limiting causal claims.

  • F.4. Cost and Runtime Footprint: Under $5 and roughly half an hour per run, the reduced S1, S2, and S7 subset preserves four-mechanism diagnostic interpretability for pre-deployment checks.The subset uses 10 sessions × 3 seeds with one Haiku-class model.
  • G. Case Studies: The case studies demonstrate the four-mechanism taxonomy, temporal dependency DAG, and counterfactual attribution through compression-aging and revision-aging walkthroughs.The walkthroughs cover an S2 user profile and an S2 DeepSeek-R1-7B run.
  • G.1. Memory Degradation Under Compression: Lossy compaction replaces specific profile values with generic summaries, preserving 0/10 constraint values and producing the empirical ∼4.5× half-life gap.Careful compaction preserves names, numbers, and dollar amounts; both settings use Haiku 4.5 and differ only in θ.
  • G.2. Tracing a Compounding Error: In the S2 DeepSeek-R1-7B setup, each session adds a spending delta, and budget-balance probes test whether compaction preserves the running accumulator.The verified run uses lossy compression across 10 sessions with medium pressure.
  • Compaction discards transaction deltas (qualitative): Lossy compaction erases transaction deltas at write time, causing subsequent budget-balance probes to carry the omission forward across sessions.Specific “spent $X at <merchant>” content becomes a generic phrase such as “user has dining preferences.”
  • Per-session accumulator error (one verified run, oracle-store condition): 232 →38 →95 accumulator_error values show a non-monotonic trajectory, while the run-mean error remains ∼125 and compounding_detected fires under its sliding-window heuristic.Keyword recall remains around ∼0.7 even though the derived running budget is consistently wrong from session 3 onward.
  • H. Evaluation Card: AgingBench produces reproducible mechanism-level aging curves and paired-control diagnostic profiles, supporting pre-deployment checks and stage-targeted intervention without claiming unique causal identification.Violations of probe monotonicity are reported as diagnostic anomalies, and single-blob summaries jointly report W and R errors.

I. Broader Discussion

The discussion frames agent aging as a lifespan and runtime-control problem requiring longitudinal evaluation, mechanism-level diagnosis, and targeted state representations. It also identifies production deployment validation as an open frontier.

  • Implications for long-lived agent system design: Compaction policy is a capability multiplier, so compaction investment should be paired with realistic assessment of the target model.Careful prompts help models that exploit preserved content but may provide little benefit to weaker models or underperform terse lossy prompts.
  • Connection to task horizons: AgingBench extends task-horizon evaluation by measuring reliability across session-gap and overall deployment-horizon axes.Both structural axes erode reliability and are represented by aging curves.
  • Aging as a runtime control problem: AgingBench frames memory policies as runtime control policies governing writing, compression, retrieval, recompaction, and flushing decisions.Per-mechanism aging curves support closed-loop evaluation of these policies.
  • Aging as a runtime control problem: 91% of the always-on intervention ceiling was captured by an aggressive forward-only controller at 86% of its wall time.The preliminary S2 controller used per-session diagnostic signals to trigger one-shot corrective actions.
  • Typed state for revision-heavy variables: Revision-heavy variables may require typed state because derived values fail under regimes where added context or model scaling does not close the gap.Running totals, accumulated counters, and versioned constraints suggest text memory may be the wrong abstraction for delta-updated state.
  • Limitations and the open frontier: AgingBench provides a mechanism-level vocabulary and seeded multi-seed protocol covering compression, interference, revision, and maintenance aging.The protocol targets all four mechanisms within controlled session horizons and diagnoses aging at specific memory-pipeline stages rather than aggregate model quality.
Loading 2605.26302v1…