PaperlayerView exact revision on arXiv

Source-linked AI summary

Bookkeeping, Composition, or Unreachable Gold? Reading MemoryAgentBench's Conflict-Resolution Scores Against a Frozen Last-Write Resolver

Egor Pakhomov, Erik Nijkamp

arXiv:2610.09193v1cs.AIcs.CL

TL;DR

MemoryAgentBench’s Conflict Resolution split is treated as a measure of selective forgetting, but whether that interpretation is warranted depends on the items and scoring rule. This paper executes the benchmark’s newest-statement rule as a frozen resolver and audits its failures alongside outcome-only runs of three systems. On multi-hop questions, accuracy is much lower on items whose gold the last-write graph cannot reach than on resolver-solved items.

  • Problem

    It is unclear whether Conflict Resolution scores measure selective forgetting or reflect the item set and scoring rule rather than agent behavior.

    Source: §1 · p. 1

  • Method

    The paper implements the benchmark’s newest-statement rule as a frozen resolver and labels its failures by last-write reachability, then compares item-level outcomes from three systems.

    Source: §1 · p. 2

  • Results

    For multi-hop items, accuracy on resolver-solved versus ambiguous items is 74.4% vs. 8.3% for Gemini 2.5 Pro.

    Source: §4 · p. 4

  • Takeaways & Limitations

    The per-item reachability split, rather than the aggregate score alone, is the unit at which scores on this split can be interpreted.

    Source: §1 · p. 2, §3.3 · p. 3, §4 · p. 4

    The analysis covers four fact lists and no memory system with a learned or structured store was evaluated at the item level.

    Source: §4 · p. 4

Abstract

from arXiv · show

MemoryAgentBench's Conflict Resolution split is read as measuring "selective forgetting". We execute the benchmark's own rule - the newest statement about a fact wins - as a zero-learning resolver frozen on one of the four fact lists. Under the official metric the rule answers 80.25% of the questions (74.5% on the three held-out lists). Of the rest, 67 items have a released gold that the last-write graph cannot reach but overwritten statements would ("The capital of India is New Delhi." superseded by "The capital of India is Grosseto."; gold New Delhi); such items are a third of the multi-hop questions at 262K. Two long-context models and our pre-registered approximate re-implementation of the benchmark's BM25 agent, one retained run per item and outcomes only, score 84.7%, 82.6% and 41.6% on the items the rule solves against 10.4%, 11.9% and 6.0% on those 67. The failures are a reachability split plus a small parser-scope residual; the per-item split, not the aggregate, is the unit at which a score here can be read.

1 Introduction

The paper asks whether Conflict Resolution scores measure selective forgetting or reflect properties of the items and scoring rule. It executes the benchmark’s newest-statement rule as a frozen resolver and audits the items and outcomes.

1 Introduction

  • The paper treats score interpretation as a construct-validity question about the item set and scoring rule, not just agent behavior.It applies trivial-baseline and item-level outcome auditing to an agent-memory competency.Source: §1 · p. 1
  • The benchmark’s prompt makes conflict resolution executable as keeping the newest statement per fact and answering by lookup or composition.Where that rule matches the released gold, a zero-learning resolver can answer correctly.Source: §1 · pp. 1–2
  • The contributions are a frozen resolver with a predeclared held-out split, a documented labeling of its failures, and outcome-only runs of three systems.The authors state that they make no claims about mechanisms or memory systems in general.Source: §1 · p. 2

2 Setup

The setup applies a fixed parser and graph resolver to the released Conflict Resolution questions. The parser is developed on one fact list, frozen, and evaluated unchanged on the other three.

Corpus

  • The corpus contains 800 gold-answerable questions across four fact lists, each paired with single-hop and multi-hop question sets.The lists range from 455 to 18,332 statements, corresponding to benchmark context lengths of 6K–262K.Source: §2 · p. 2

Parser and resolver

  • A fixed set of 35 patterns maps statements to 34 relations; the resolver keeps the greatest-position tuple per subject–relation key and searches paths of up to six edges.Questions are grounded literally, and the resolver abstains when it cannot produce an answer; execution and scoring use no LLM API calls.Source: §2 · p. 2

Freeze protocol

  • The parser was developed on the 6K list, frozen, and then run unchanged on the other three lists, parsing 98.6% of their distinct statements.The 351 unparsed statements take three forms outside the pattern list, and no independent semantic audit of parses was conducted.Source: §2 · p. 2

Scoring

  • Headline scores use the repository’s substring_exact_match metric, count ABSTAIN as wrong, and report exact answer-level intervals conditional on the four lists.The four-list bootstrap is described as a width indicator; clustered uncertainty is the more appropriate scale.Source: §2 · p. 2

3 Results

The frozen resolver answers most questions, while its failures separate into misses where the last-write graph can reach gold and cases where it cannot. Item-level outcomes show large accuracy differences between these classes across the evaluated systems.

3.1 What the frozen symbolic pipeline recovers

  • 80.25% of all questions are answered correctly by the frozen resolver, including 74.50% on the three held-out lists.It answers 642/800 correctly and abstains on 32; single-hop accuracy is 88–100% across list lengths, while multi-hop accuracy is lower on larger lists.Source: §3.1 · p. 2

3.2 Position relative to published scores

  • At 262K, the resolver scores 88 / 36 against at most 60 / 7 for main-table systems in the benchmark’s arXiv v1.This tier-matched comparison is not a leaderboard: whole-list reading differs from chunked ingestion, prompting, retrieval, and formatting.Source: §3.2 · pp. 2–3

3.3 The operational split: what the failures demand

  • 78 failures (49.4%) have a gold reachable from a question mention by a last-write path of at most six edges, but the resolver does not return it.The reachability flag does not establish that the path answers the question; the resolver’s misses are mostly expected to involve grounding or path selection.Source: §3.3 · p. 3
  • 67 failures have gold present in the context but unreachable in the frozen last-write graph; re-admitting overwritten tuples makes every gold reachable within six edges.In 50 cases the gold-valued tuple was overwritten; in 17, only an intermediate tuple was.Source: §3.3 · p. 3
  • The remaining 13 failures require an edge found only in an unparsed statement, while 3 of the 67 unreachable-gold cases also contain gold in an unparsed statement.The 67-item label is operational relative to the frozen parser’s graph, not a claim that the released gold is wrong.Source: §3.3 · p. 4

3.4 Three systems, item by item

  • In one retained run per item, Gemini 2.5 Pro scores 84.7% on resolver-solved items versus 10.4% on ambiguous items; the approximate BM25 re-implementation scores 6.0% on the 67 ambiguous items.The long-context models and BM25 agent were evaluated with outcomes only, so these results do not establish mechanisms; class membership is confounded with hop type and length.Source: §3.4 · p. 4

4 Discussion

On this split, single-hop questions are mostly solvable by one-edge lookup, while multi-hop accuracy reflects both graph composition and a substantial fraction of golds unreachable under last-write. The authors limit interpretation to four fact lists and one frozen parser and resolver.

4 Discussion

  • Multi-hop accuracy falls on larger lists, where about a sixth of items—and a third at 262K—have golds the rule cannot reach.On those unreachable-gold items, accuracy is 8.3%, 11.7% and 0% for the compared systems; no multi-hop miss was attributed to semantics beyond last write under the documented rule.Source: §4 · p. 4
  • The analysis is conditional on four fact lists and one split; no memory system with a learned or structured store was analyzed at item level.The parser is a 35-pattern lower bound on a symbolic procedure, and the held-out lists share templates with the 6K development list.Source: §4 · p. 4

A Taxonomy rule, label production, and reachability check

The failure audit applies a six-category rule to resolver failures, combines scripted reachability labels with 13 agent designations, and checks whether overwritten statements recover the released golds. Re-adjudication and row-level review qualify how those labels should be read.

A Taxonomy rule, label production, and reachability check

  • The six-category rule separates parse failures, unreachable golds, reachable misses, beyond-last-write semantics, normalization artifacts and unclassifiable cases.Parse-scope failures are prioritized for required query edges found only in unparsed statements; reachable misses have a gold in the frozen graph but a different resolver output.Source: A Taxonomy rule, label production, and… · p. 8
  • 158/158 failures received labels through a script over reachability plus 13 agent designations, rather than the pre-specified operating-agent classification.The authors disclose this execution deviation; all three override lists were empty and no rows were labeled unclassifiable.Source: A Taxonomy rule, label production, and… · p. 9
  • With the reachability flag visible, blind re-adjudication agreed with frozen labels on 147/158 (93.0%; κ = 0.875); without the explicit flag, blind relabeling agreed on 141/158 (89.2%; κ = 0.82).Source: A Taxonomy rule, label production, and… · p. 9
  • All 158 failures have a gold string in context, so the rule’s ordering leaves the genuine-beyond-last-write category empty.Under the documented ordering, such items are assigned to parse failure, ambiguous gold or multi-hop composition failure before that category is considered.Source: A Taxonomy rule, label production, and… · p. 9
  • 67/67 ambiguous-gold rows reach a gold under breadth-first search with overwritten tuples re-admitted, versus 0/67 using only last-write tuples.A superseded gold-valued edge occurs on some qualifying path in 50 rows; for the remaining 17, only intermediate tuples on such paths were overwritten.Source: A Taxonomy rule, label production, and… · p. 10
  • 67 of 67 ambiguous-gold rows were confirmed by both reviewers as having released golds that require overwritten statements to be reached.The row-by-row review checked the raw statements, rather than parser tuples, and therefore does not replace the missing parse audit.Source: A Taxonomy rule, label production, and… · p. 10
  • Re-admitting overwritten statements expands the median reachable graph from 8 to 47 nodes, or 87 at 262K.The workbench records paths, parsed and unparsed gold occurrences, and the resolver’s answer and trace reason for each failure.Source: A Taxonomy rule, label production, and… · pp. 8, 10

B Examples per category

The examples distinguish failures where the last-write graph contains a route to the gold from cases where the released gold is unreachable under that graph or a needed statement is unparsed. The section also situates the ambiguous-gold examples against the benchmark’s construction description.

B Examples per category

  • A last-write-reachable miss can stop short of the gold: the Michael Mantler question’s gold, Canada, lies one edge beyond the resolver’s path.Other examples return Regina Ip instead of the chairperson’s occupation and Ireland instead of Burkina Faso’s public representative.Source: B Examples per category · p. 10
  • For the Rogério Ceni question, the released gold is New Delhi although a later statement overwrites it with Grosseto as India’s capital.The section gives two further ambiguous-gold examples, with London/Oderzo and Amposta/Grosseto as gold/output pairs.Source: B Examples per category · p. 10
  • The construction statement says newer numbered edits contain the answer, but neither benchmark version names the MQuAKE subset used.The paper notes documented MQuAKE conflict issues and treats the 67 items as operationally ambiguous relative to the frozen last-write graph, not as proof that the released gold is wrong.Source: B Examples per category · pp. 10–11
  • Parser-scope examples include a head-coach statement and a titled-office statement that the frozen parser leaves unparsed.The section’s examples include ABSTAIN outputs as well as a Grosseto answer for the Cleveland Indians question.Source: B Examples per category · p. 11

C Parser and resolver

The resolver extracts numbered statements with frozen full-line patterns, keeps the latest tuple for each subject–relation key, and searches question-grounded paths for an answer. Its parser also leaves several statement forms outside its pattern list.

C Parser and resolver

  • The parser segments numbered context lines, excluding the unnumbered heading, and accepts a statement only when exactly one frozen pattern matches.The 35 patterns map to 34 relation keys; more syntactically specific patterns precede generic ones.Source: C Parser and resolver · p. 11
  • For each subject–relation key, the graph retains the tuple with the greatest numbered position and discards earlier tuples.Subjects and objects are used as written after trimming surrounding whitespace, without entity linking.Source: C Parser and resolver · p. 11
  • Question entities are grounded by normalized literal-substring matches against graph nodes, with longer overlapping mentions taking precedence.The normalization includes Unicode NFKC, case-folding, quote unification and whitespace collapsing.Source: C Parser and resolver · p. 11
  • Paths of 1–6 edges are deterministically scored using mention length, lexical relation cues, terminal preference, edge and uncued-relation penalties, and a question-surface penalty.The highest-scoring path supplies the answer; fixed cue and preference tables were unchanged after the freeze.Source: C Parser and resolver · p. 11
  • On the three held-out lists, 351 of 25,222 distinct statements are unparsed, including head-coach, original-language and titled-office forms.The development list is parsed in full; the held-out lists each occur in two parquet rows.Source: C Parser and resolver · pp. 11–12

D Sensitivities

The overwrite sensitivity analysis shows that most items solved by the frozen resolver depend on choosing the newest write. The reported bootstrap interval is only a width indicator, not a precision claim.

D Sensitivities

  • Table 4 reports sensitivity of headline accuracy under normalized official scoring, while strict scoring gives identical counts in every row.The supplied caption does not specify the sensitivity dimensions or individual table values.Source: Table 4 · p. 12
  • 526 of the 642 resolver-solved items are lost under first-write resolution, showing that the split exercises overwriting on 82% of items the rule solves.All 266 multi-hop items solved by the last-write resolver are lost under first-write; 116 single-hop items are solved under both policies.Source: D Sensitivities · p. 13
  • With only four clusters, the bootstrap distribution has 256 equiprobable outcomes, so the interval is reported only as a width indicator.The reported four-list interval is [67.25, 92.75]; it supports no precision claim.Source: D Sensitivities · p. 13

E Accuracy grid with intervals; residual splits

The section presents per-list accuracy intervals and residual-failure counts split by hop type, output type, and question set.

E Accuracy grid with intervals; residual splits

  • Table 6 repeats Table 1’s accuracy grid with exact intervals, while Tables 5 and 7 report failure splits by hop/output type and question set.The supplied text identifies the tables’ scopes but does not provide their individual cell values.Source: E Accuracy grid with intervals;… · p. 13
  • Table 5 cross-tabulates the 158 residual failures by hop type and output type, with Gemini 2.5 Pro correctness identified.The caption describes counts from the row-level audit file.Source: Table 5 · p. 13
  • Table 6 gives per-list accuracy percentages with exact Clopper–Pearson 95% intervals and reports n = 100 per hop cell.Both-hops cells have n = 200; pooled and held-out columns with intervals appear in Table 1.Source: Table 6 · p. 13
  • Table 7 splits residual labels across eight question sets, comprising four fact lists crossed with multi-hop and single-hop questions.Each set contains 100 questions; counts come from the row-level audit file.Source: Table 7 · p. 13

F Published comparison: all cited rows

The published comparisons place the frozen resolver beside benchmark results across tiers, while the paper cautions that these comparisons are not a leaderboard. The paper also qualifies performance-gap interpretations using its item-level findings.

F Published comparison: all cited rows

  • The cited Hu et al. [1] values are transcribed from v1 and use the benchmark’s accuracy / substring exact match mapping, not recomputed scores.Table 10 organizes v1 reasoning-model rows above per-tier context-length rows.Source: F Published comparison: all cited rows · p. 14, Table 10 · p. 15
  • Table 8 places cited published Conflict Resolution scores beside the resolver by tier; Table 9 covers all 17 v1 main-table systems at 262K.Non-long-context rows in Table 9 use a GPT-4o-mini backbone.Source: Table 8 · p. 14, Table 9 · p. 14
  • The camera-ready renames Selective Forgetting and revises some reported rows, including BM25 single-hop at 262K from 56 to 48.It adds several systems, while the Claude-3.7-Sonnet multi-hop 262K value is corrected from 0 to 2.Source: F Published comparison: all cited rows · p. 14
  • Pooled resolver accuracy is 94.0 single-hop / 66.5 multi-hop, or 92.0 / 57.0 on the three held-out lists.At 6K, o4-mini reaches 100 / 80, within 15 pp of the resolver’s 95 multi-hop score on the development list.Source: F Published comparison: all cited rows · p. 14
  • The tier-matched comparisons are a reading aid rather than a leaderboard because whole-list reading and chunked ingestion, prompting, retrieval, and formatting differ.The paper further notes that this split’s gaps can reflect unreachable-gold items and long-context model failures on resolver-solvable items.Source: F Published comparison: all cited rows · p. 15

G Freeze and reproducibility record

The resolver and analyses were frozen and evaluated through scripted procedures, with development confined to one fact list. The paper discloses LLM assistance in selected labeling, model experiments, and manuscript work, alongside a parse-audit limitation.

G Freeze and reproducibility record

  • The released parquet contains eight rows, 800 questions, and four distinct fact-list contexts; development rows 0 and 4 were declared before other row content was inspected.Those rows comprise the 6K fact list’s multi-hop and single-hop sets.Source: G Freeze and reproducibility record · p. 15
  • The parser/resolver source was hashed and frozen before all-row evaluation, with no edits afterward; the development diagnostic was 195/200 correct and parsed 910/910 statements.The all-row evaluation refuses to run unless the source hash matches the freeze record.Source: G Freeze and reproducibility record · p. 15
  • The parse audit recorded a 30-statement sample as faithful without per-row checks, and its later same-pattern reapplication cannot independently verify semantic correctness.The authors state that no independent semantic audit of parses exists.Source: G Freeze and reproducibility record · p. 15
  • Resolver execution and official substring scoring were scripted locally without LLM API calls; scoring lowercases, strips punctuation, removes articles, collapses whitespace, and tests gold substrings.The official scoring implementation was reimplemented verbatim, with strict equality yielding identical counts in every group.Source: G Freeze and reproducibility record · p. 15
  • The frozen-source record corrects its pattern count from 32 to 35, mapping to 34 relation keys, and the authors make scripts and audit artifacts available on request.Available materials include frozen source, per-question results, parsed and unparsed statements, the workbench, and the classification script.Source: G Freeze and reproducibility record · p. 16

H LLM-use disclosure

  • LLMs assisted with parser-scope labeling and blind relabelings, while Gemini 2.5 Pro, Gemini 2.5 Flash, and a GPT-4o-mini-backed retrieval agent were experimental subjects.Parsing, resolution, and scoring used scripted pipelines without LLM calls; the authors retain responsibility for the work.Source: H LLM-use disclosure · p. 16

I Long-context and retrieval-agent experiments: protocol and tables

The long-context models and approximate BM25 agent were evaluated with one retained output per item, then compared by resolver item class. Their accuracy differs sharply between items the resolver solves and ambiguous-gold items, with the latter especially difficult at 262K multi-hop.

I Long-context and retrieval-agent experiments: protocol and tables

  • The cost rule selected Gemini 2.5 Pro and Flash for identical long-context protocols, with one run per item and retries for empty or transport-error outputs.Temperature was the provider default; the retrieval agent used BM25 over 512-token chunks, top 10, and GPT-4o-mini, as an approximate re-implementation.Source: I Long-context and retrieval-agent… · pp. 16–18
  • Pro scores 84.7% vs. 10.4%, and Flash 82.6% vs. 11.9%, on resolver-correct vs. ambiguous-gold items.The comparisons are class-specific, not overall model accuracies.Source: I Long-context and retrieval-agent… · p. 17, Table 13 · p. 18
  • On the 67 ambiguous-gold items, each long-context model contains neither the resolver prediction nor released gold in 43 outputs.On 78 last-write-reachable-miss items, neither model ever outputs the resolver’s wrong prediction.Source: I Long-context and retrieval-agent… · p. 18
  • The retrieval agent scores 41.6% on resolver-correct items and 6.0% on ambiguous-gold items (4/67).Overall, it scores 285/800 = 35.6%; on multi-hop 262K items, it gets 4/36 resolver-correct and 0/34 ambiguous-gold items right.Source: Table 14 · p. 19, I Long-context and retrieval-agent… · p. 19
  • At multi-hop 262K, Pro scores 4/36 vs. 1/34, Flash 13/36 vs. 4/34, and retrieval 4/36 vs. 0/34 on resolver-correct vs. ambiguous-gold items.For each system, the first count is on resolver-correct items and the second on ambiguous-gold items.Source: I Long-context and retrieval-agent… · p. 19
Loading 2610.09193v1…