PaperlayerView exact revision on arXiv

Source-linked AI summary

Measure Before You Manage: Evaluating Agent Working Memory in Coding Agents

Le Chen, Zishen Wan, Baixi Sun, Xiaolong Ma, Chih-Hsuan Yang, Feng Yan, Sheng Di, Franck Cappello, Rajeev Thakur

arXiv:2608.31057v1cs.AI

TL;DR

Coding-agent working memory contains heterogeneous object types whose management and evaluation are not well captured by uniform rules or nominal token budgets. The paper characterizes typed objects across archived trajectories and studies object-aware compression and retrieval-based management. It finds distinct retention and compression behavior, limited transfer of calibration gains, and mismatches between nominal budgets, delivered context, and management work.

  • Problem

    Whether semantic heterogeneity should shape coding-agent working-memory management and how such management should be evaluated remains unclear.

  • Method

    The paper characterizes typed-object working memory across 55 archived trajectories and examines object-aware compression and retrieval-based management strategies.

  • Results

    Semantically different objects show distinct retention and compression behavior, while calibration gains may not transfer to held-out tasks and equal nominal budgets need not yield equal delivered context or management work.

  • Takeaways & Limitations

    Evaluating working-memory management requires separating stored state, delivered context, management work, and task or process outcome rather than relying on nominal token budgets alone.

  • Takeaways & Limitations

    The evidence is small, repository-clustered, conditional on completion, and lacks valid repair-success evaluation in the later study; the served model revision is also unpinned.

Abstract

from arXiv · show

Agent working memory is heterogeneous. Objects such as instructions, artifacts, tool outputs, and agent-generated state play different semantic roles and exhibit different size, retention, and representation profiles. Recent work has begun to explore memory-management mechanisms that account for such heterogeneity. This work focuses on semantic heterogeneity and studies how it should shape the management and evaluation of working memory in coding agents. Across 55 archived coding-agent trajectories, we find that semantically different working-memory objects exhibit distinct retention and compression behavior. This heterogeneity motivates semantically informed memory management. We study two semantically informed strategies: an object-aware compression policy and a retrieval-based policy. Their evaluation shows that calibration gains may not transfer to held-out tasks, and that equal token budgets do not imply equal delivered context or management cost. A real-system replay further exposes serving limits that nominal budgets alone do not capture. Together, these results show why semantic structure matters for agent working memory and why evaluating memory-management strategies requires more than a nominal token budget. We organize these lessons into four levels: stored state, delivered context, management work, and task or process outcome.

1 Introduction

Working memory in coding agents is semantically heterogeneous, but existing management mechanisms may treat its diverse objects uniformly. The paper characterizes this heterogeneity and evaluates semantically informed management strategies to clarify what reliable evaluation requires.

  • Instructions, source artifacts, tool outputs, and agent-generated state differ in semantic role, size, persistence, representation, and lifecycle.
  • It examines an object-aware compression policy and a retrieval-based policy as case studies of semantically informed management.
  • The study characterizes typed-object working memory across 55 archived coding-agent trajectories.
  • The evaluation asks whether apparent management gains transfer to held-out tasks and whether equal nominal budgets produce equal delivered context and management work.
  • The paper focuses narrowly on task-local working memory in coding agents rather than claiming novelty for memory hierarchies, retrieval, summarization, or agent-memory profiling.

2 Working-Memory Model and Study Design

The study represents coding-agent working memory as typed, ordered objects managed through a host message plan and evaluates process behavior under explicit accounting choices. Its measures distinguish retained state from provider cost, cache footprint, and causal utility.

  • The host maintains an ordered message plan over instructions, artifacts, tool outputs, and agent state, with records tracking size, creation step, representation, and optional dependencies.
  • Compression can use raw, compressed, summary, or pointer representations, with pointers recoverable through recall_object rather than requiring irreversible deletion.
  • The policy studies impose a 25-step ceiling and use an unpinned served model revision, while historical requests, retries, and supervision are incompletely reconstructable.
  • The later retrieval population reuses development tasks, and its 24 task–arm cells lack valid formal repair evaluation.
  • Repeated tool invocations with matching names and sorted-key JSON arguments form a process-regularity metric rather than a measure of repair success or total unnecessary work.
  • Content volume counts each retained record once, whereas retention-weighted cost weights records by residency; neither is a provider bill, KV-cache footprint, or causal-utility measure.

3 Semantic Heterogeneity in Agent Working Memory

Across 55 coding trajectories, object types contribute differently to working-memory volume, residency, and representation. These descriptive measurements show why pooled averages and stored-token counts alone can mischaracterize task-local memory behavior.

  • 1,350 in-context objects comprise 585 tool outputs, 165 artifacts, 110 instructions, and 490 agent-state objects across the 55 trajectories.
  • 55.5% of pooled content volume comes from tool outputs, versus 28.3% from artifacts; retention-weighted cost shares are 40.2% and 38.9%, respectively.
  • Artifacts have median recorded size 624 proxy tokens versus 73 for tool outputs and mean residency 10.71 versus 8.61 steps.
  • 50.8% is the task-equally weighted mean tool-output volume share, compared with 55.5% pooled, with a task-level P10–P90 range of 23.0–78.1%.
  • The structural probe reports mean compressed/raw ratios of 0.150 for artifacts and 0.673 for tool outputs, but these are properties of implemented rules and sampled objects.
  • No stored-state quantity identifies which object is safe to discard.

4 Managing Heterogeneous Working Memory

The paper studies object-aware compression and adapted retrieval as distinct management case studies rather than universally superior policies. Their comparisons show that calibration contrasts may not transfer, while deployment-style evaluations confound priority, representation, and restoration mechanisms.

  • 4.1 Object-Aware Compression: The object-aware policy combines type, age, access, staleness, supersession, and rendered-size signals across raw, compressed, summary, and pointer forms.
  • 4.1 Object-Aware Compression: Automatic prompt inclusion updates the inherited access clock, so LRU can collapse toward creation order and OA’s age/reuse inputs are not clean demand signals.
  • 4.1 Object-Aware Compression: −1.633 repeated calls is OA’s calibration contrast with FIFO, versus −0.500 held out; no held-out contrast survives Holm correction.
  • 4.1 Object-Aware Compression: The held-out result does not establish equivalence, prove calibration overfitting, or show that OA matches a baseline.
  • 4.2 Retrieval-Based Memory Management: The adapted retrieval policy uses recency, relevance, and importance components with coding-specific inputs, rather than replicating Generative Agents’ reflection, planning, or social environment.
  • 4.2 Retrieval-Based Memory Management: The six-arm retrieval comparison varies priority score, representation, and restoration mechanism together under a common cap, making it a deployment-style comparison rather than a score ablation.
  • 4.2 Retrieval-Based Memory Management: −0.375 is GA’s mean repeated-call difference from FIFO across eight complete blocks, with 95% CI [−1.250, 0.375]; no baseline contrast survives Holm correction.

5 Lessons for Evaluating Agent Working Memory

Evaluating working-memory policies requires measuring more than nominal token budgets: delivered context, management work, and signal validity can change what comparisons mean.

  • Effective budgets: Budget labels can hide effective-budget differences because floors, caps, and tokenization units vary across sweeps and studies.The earlier wide sweep’s 6,000-token floor collapses nominal 25% and 50% settings into identical absolute budgets, while 18 of 24 retrieval caps exceed the nominal 15% anchor.
  • Delivered context: None of ten constrained-arm pairs matched on delivered managed-state tokens across all eight complete tasks; FIFO–GA matched on six, with an 18.7% maximum gap.A shared cap therefore does not ensure that policies deliver equivalent context.
  • Serving boundary: A real-system replay found the unconstrained arm exceeded a frozen 32,768-token serving limit on six of 25 steps, whereas constrained arms stayed at or below 16,643 tokens.This exposes a hard delivered-context feasibility boundary that nominal budgets do not capture.
  • Management work: Management work differs by policy: GA added 285 importance calls and OA added 169 summary calls, while CPU embedding work was uncounted.GA–FIFO wall-time differences were positive on all eight tasks, averaging +67.45 seconds, but changed tool paths prevent isolating rating latency.
  • Signal validity: Lifecycle signals can misrepresent semantic change: unchanged source produced a new artifact version and five tool-output invalidations in one saved SymPy trajectory.Prompt inclusion also refreshed the inherited access clock, and no corrected-OA trajectories were available.

6 A Framework for Evaluating Agent Working Memory

The paper synthesizes a four-level framework for evaluating agent working memory, while bounding its claims by evidence that is small, conditional, and partly instrument-limited.

  • Four-level framework: Table 4 reports working-memory evaluation at four levels: stored state, delivered context, management work, and task or process outcome.The framework was synthesized post-hoc from the case studies rather than specified in advance.
  • Scope and limitations: The evidence base is small, repository-clustered, conditional on completion, and limited to eight complete six-arm retrieval blocks with one run per task–arm.The later study lacks valid repair-success evaluation.
  • Scope and limitations: The served model revision is unpinned, request and intervention history is incompletely reconstructable, and no corrected-OA trajectories exist.These boundaries constrain interpretation of serving conditions and the OA lifecycle mechanisms.
  • Evidence status: Saved records support expanded analyses, but recalculations, descriptive summaries, and confidence intervals are post-hoc audits rather than new confirmatory experiments.Historical protocol declarations and later amendments are distinguished.

A.1 Populations and evaluation units

The study distinguishes populations and evaluation units across 55 archived trajectories, with policy analyses using selected and completion-conditional subsets rather than interchangeable samples.

  • Evaluation units: Table A.1 distinguishes tasks, trajectories, objects, and task–policy cells as different population units.These units should not be treated as interchangeable when interpreting counts or comparisons.
  • Policy population: The eight held-out policy tasks come from seaborn, pylint, pytest, requests, and Sphinx.The held-out split is a subset of the archived SWE trajectories rather than the full characterization corpus.
  • Evaluation protocol: The earlier policy population was selected using successful full-context runs under a local Docker-free evaluator checking FAIL_TO_PASS and only the first 20 PASS_TO_PASS tests.These local verdicts are not official SWE-bench scores, and the follow-up estimand is conditional on all six arms completing.
  • Object representation: The ordered message plan references typed objects with creation step, size, representation, and optional path, version, or dependency metadata.Pointers retain object IDs recoverable through recall_object, so eviction need not mean permanent deletion.
  • Run conditions: Policy trajectories used a 25-step ceiling with the recorded claude-opus-4.8 alias, unpinned served revision, and no supplied gold patch.Auxiliary requests omitted temperature, and outside human supervision was not fully reconstructable.

B.1 Size, retention, and task heterogeneity

Working-memory composition varies across object types and tasks: size, retention, and volume shares depend on what is measured and how tasks are weighted. Temporal summaries also change their contributing population, limiting longitudinal interpretation.

  • Task heterogeneity: The 18-trajectory Terminal-Bench probe has more tool-output volume and less artifact volume than the SWE sample.It is a boundary observation and does not establish policy transfer or consistent semantic-staleness behavior across workloads.
  • Temporal composition: Step-level summaries use only trajectories with an LLM step at each checkpoint, so later rows represent fewer, longer-running tasks rather than a balanced longitudinal effect.A rising agent-state share among survivors does not demonstrate rising utility.

B.3 Structural compression probe

The structural probe finds strongly type-dependent compression behavior, supporting type-specific representations while limiting claims about intrinsic compressibility or semantic preservation. The object-aware policy is heuristic, and implementation details constrain interpretation of its comparisons.

  • Probe scope: 468 raw/compressed/pointer rows cover 156 sampled objects, but the sampler is not a random task sample and lacks task IDs.The probe does not enforce the characterization table’s in-context filter, so rows cannot be treated as mapped context objects or independent task-level replicates.
  • Type-dependent compression: Structural rules retain code signatures/imports for artifacts, selected diagnostic lines for tool outputs, and head portions for instructions and agent state.The literal-fact metric checks whether extracted paths, error names, symbols, test names, and test-count phrases remain verbatim.
  • Type-dependent compression: 0.150 artifact ratio versus 0.673 for tool outputs supports type-specific representations, but these ratios are properties of implemented rules and sampled objects.They do not establish intrinsic or optimal compressibility, semantic preservation, or unchanged task performance.
  • Object-aware policy: The object-aware policy is a hand-specified heuristic using type/subtype, age, access count, stale or supersession status, and rendered size.It demotes candidates through raw, compressed, summary, and pointer forms for at most four rounds, while protecting instructions and newly created objects.
  • Object-aware policy: OA’s inherited access counters are updated on automatic prompt inclusion, so access-age and reuse signals may reflect recent inclusion rather than demand.The archive reports implemented historical policies, not intended or corrected variants, and does not claim exhaustive tuning.

C.3 Per-task results and secondary measurements

Per-task results show that development improvements do not establish held-out gains, while secondary measurements and retrieval comparisons are shaped by aggregation, implementation, and completion constraints. The evidence therefore supports cautious interpretation of policy differences.

  • Secondary measurements: OA’s mean recorded input is slightly above FIFO’s under the same within-task aggregation, so fewer repeated calls do not imply fewer recorded input tokens.Table C.4 reports recorded main-agent prompt tokens per trajectory, not an all-service token bill.
  • Budget sensitivity: The effective-fraction sensitivity uses a ≤30% threshold and eight tasks, changing both the conditioning rule and sample relative to the nominal ≤15% primary analysis.It cannot replace the nominal primary analysis or identify budget mismatch as the cause of failed held-out transfer.
  • Retrieval comparison: The retrieval score combines normalized recency, relevance from embeddings, and importance, with elapsed agent steps replacing the source setting’s time scale.Normalization is over the current candidate set, and the retrieval-owned clock updates for admitted objects rather than automatic prompt inclusion.
  • Comparison design: UC and OA retrieval comparisons combine different selectors, candidate universes, packers, ladders, and restoration opportunities, so they compare system mechanisms rather than retrieval scores alone.Full context is an uncapped reference rather than a sixth capped arm.

D.3 All attempted task blocks and stopping

Across 24 attempted blocks, completion was limited by floor-rule or OA-transition stopping, while nominal budget labels did not always represent distinct pressure levels. The analysis therefore distinguishes recorded execution outcomes from budget coordinates and accounting units.

  • All attempted task blocks and stopping: 77 trajectories started, 61 completed, and 16 were interrupted across 24 attempted blocks.Eight blocks completed, 13 stopped by the greedy floor rule, and three stopped after OA exhausted its allowed transitions.
  • All attempted task blocks and stopping: The floor rule is a sequential, state-dependent raw/pointer construction rather than a proven global minimum over representation assignments.Consequently, floor-rule stopping does not establish intrinsic task infeasibility.
  • Nominal and absolute budgets: The wide sweep’s 6,000-token floor made all 25%/50% pairs identical absolute-budget conditions, not distinct pressure levels.The tight and subsequent 15% sweeps used a 1,200-token floor, so nominal fractions could map to different absolute conditions.
  • Nominal and absolute budgets: The archived narrative and saved cells report different medians, and the historical discrepancy remains unresolved.Recalculations use saved cell values with within-sweep references; later held-out data are not merged into the table.
  • Later budget freeze: In the later freeze procedure, 18 of 24 tasks were raised above the nominal anchor, with B_t/P_t ranging from 0.14994 to 0.76711.The slight undershoot of 0.15 reflects integer rounding, and five constrained arms share B_t within each task.
  • Accounting units: Managed-state cost tokenizes joined rendered message-plan objects once, while absolute budgets from the earlier and later hosts are not directly comparable token units.The API’s recorded prompt usage is a separate outcome, and managed-state cost excludes full provider-request protocol overhead.

E.2 Delivered-state matching and measured resources

Delivered-state matching is a distinct diagnostic from nominal budget matching, and the resource measurements show that management strategies can differ in whole-trajectory work and auxiliary-call counts. Reported dollar totals remain unresolved estimates rather than billed outcomes.

  • Delivered-state matching: No capped-arm pair passes the median managed-state delivery diagnostic on all eight tasks.The diagnostic requires every task’s largest pairwise median managed-state gap to be at most 10%; it is not an equality test on full API input or all-service compute.
  • Measured resources: GA’s paired whole-trajectory wall-time difference from FIFO is positive on all eight tasks, averaging +67.45 seconds.The median paired difference is +43.47 seconds, while the difference between the two arm medians is 40.89 seconds; these are distinct summaries.
  • Measured resources: Whole-trajectory wall-time differences include changed execution paths, tool work, and auxiliary work, so they do not isolate importance-rating latency or prove a serving-layer slowdown.CPU embedding work is not an LLM call.
  • Measured resources: OA made 169 summary calls and GA made 285 importance calls, preventing interpretation of same-cap outcomes as equal-compute comparisons.These auxiliary calls are separately counted from main agent LLM steps.
  • Cost accounting: The full-run estimate is $213.52 in run_result.json versus $235.99 in cost.json, and neither total is treated as actual spend.The ledger uses conservative preset prices rather than provider billing, and the discrepancy is unreconciled.

F.1 Lifecycle signals are implementation observations

Lifecycle analyses identify implementation-specific observations rather than direct measures of semantic change or utility. Additional evaluation and replay limitations constrain policy comparisons, serving conclusions, and reproducibility claims.

  • Lifecycle signals: A read of unchanged source content created a new disk artifact and advanced its version in sympy__sympy-19007.Full-source sidecar hashes establish equal content across each edit/read pair.
  • Lifecycle signals: Five dependency-change invalidations at step 14 show that a recorded version event is not proof of semantic change.The read path returns both a view and full artifact content, and the loop creates a versioned disk artifact before adding the context view.
  • Lifecycle signals: Automatic prompt inclusion updates the inherited access clock, so LRU may observe recent inclusion rather than recent demand.The later LRU-D selector uses a separate demand observer, while inherited OA retains the old counters.
  • Interpretation limits: Content-reference detection measures matching signatures rather than causal utility, lacks human ground truth, and varies in coverage by type.The paper therefore does not use these observations to claim that inexpensive objects have greater future utility.
  • Evaluation limits: The later 48 complete task–arm cells lack valid formal repair evaluation, and serving replays are excluded from performance claims.No cache benefit, serving speedup, or corrected implementation result is claimed.
  • Serving replay: The exploratory serving replay used a small calibration-task set, a different served model, and no sustained physical-memory-pressure regime.Its measurements are not comparable to the main policy contrasts and support no physical-memory-efficiency claim.
  • Reproducibility: Incomplete historical parameter-selection, serving, decoding, and intervention provenance prevents an unqualified claim that all experimental settings are specified.The archive supports saved-result checks but promises neither a complete anonymous raw-data release nor byte-identical trajectory regeneration.
Loading 2608.31057v1…