PaperlayerView exact revision on arXiv

Source-linked AI summary

Reliable Post-Retrieval Assembly for Agent Memory: Separating Evidence Extraction from Policy Execution

Vikas Reddy, Sumanth Reddy Challaram

arXiv:2606.01435v2cs.AIcs.CLcs.IR

TL;DR

Memory agents can retrieve relevant evidence yet face a conflict-resolution question when answer generation must also filter evidence and apply a policy. This paper separates candidate extraction from policy execution; on FactConsolidation, structured assembly improves single-hop accuracy over direct answering, with gains that are task-scoped.

  • Problem

    Memory systems face an unresolved post-retrieval question: how to assemble retrieved evidence into a policy-compliant answer when facts conflict.

    Source: §1 · pp. 1–2

  • Method

    The pipeline retrieves evidence, extracts matching candidates into a structured representation, and applies an explicit answer policy in a separate stage.

    Source: §3 · p. 4

  • Results

    10.8 pp average and 21 pp at 262K: structured assembly improves single-hop accuracy over direct answering with identical retrieved evidence.

    Source: §1 · p. 2

  • Takeaways & Limitations

    Post-retrieval assembly is a distinct system boundary that should be designed and evaluated explicitly.

    Source: §6 · p. 8

    The comparison is task-level and does not rank overall memory-architecture quality.

    Source: §5.2 · p. 6

Abstract

from arXiv · show

LLM-based memory systems can retrieve relevant evidence yet still fail when answer generation entangles semantic filtering, conflict resolution, prior suppression, and output generation in one step. We study this failure as a problem of post-retrieval assembly. In the MemoryAgentBench (MAB) release used here, FactConsolidation explicitly states that newer facts have larger serial numbers, yet the best reported retrieval/memory result is 54% single-hop and all 22 reported systems score at most 7% multi-hop. We evaluate a structured assembly interface in which an LLM first extracts semantically matching evidence into a candidate representation and a separate stage executes the required answer policy. At 262K, this pipeline reaches 82%/93% single-hop and 27%/41% multi-hop with gpt-4o-mini/gpt-4o, exceeding every result reported in the MAB v3 FactConsolidation comparison. This is a task-level result, not a claim that the evaluated memory architectures are broadly inferior. A controlled whole-pipeline comparison, with identical backbone, retrieved top-10 evidence, chunking, and n=100 per cell, improves single-hop accuracy by 10.8 percentage points (pp) on average and 21 pp at 262K. A targeted comparison using the same extraction setup shows that changing only the final policy executor contributes 2.0 pp on average and 0 pp at 262K. Most of the gain therefore comes from separating evidence identification from final policy execution rather than from the freshness operator itself. A LongMemEval check finds no significant overall advantage (26/45 versus 29/45; paired exact McNemar p=0.45), bounding the result to current-value questions with explicit version metadata. The evidence identifies post-retrieval assembly as a distinct reliability boundary between retrieval and answer generation.

1 Introduction

FactConsolidation makes current-value conflict resolution an answer-time assembly problem: newer facts have larger serial numbers, but strong retrieval alone does not ensure the right value is used. The paper separates evidence identification from policy execution and reports gains on this task, while bounding the comparison to the benchmark rather than memory architectures generally.

1 Introduction

  • FactConsolidation explicitly says newer facts have larger serial numbers, yet retrieval and memory systems still struggle to resolve its conflicts.In the MAB v3 comparison, HippoRAG-v2 reaches 54% single-hop and all 22 reported systems score at most 7% multi-hop.Source: §1 · p. 1
  • The structured pipeline extracts matching candidates first, then applies the newest-version policy in a separate stage.The paper frames this as separating evidence identification from explicit policy execution when policy and metadata are known.Source: §1 · pp. 2–3
  • 10.8 pp average and 21 pp at 262K: structured assembly improves single-hop accuracy over direct answering despite identical retrieved evidence.The controlled comparison fixes the backbone, top-10 evidence, chunking, and evaluation items; the direct-answer baseline falls from 75% at 64K to 61% at 262K, while structured assembly reaches 82%.Source: §1 · p. 2, Table 1 · p. 2
  • 2.0 pp average and 0 pp at 262K: changing only the policy executor contributes little compared with the whole-pipeline gain.The targeted comparison re-ran candidate extraction, so it approximates rather than perfectly isolates executor choice.Source: §1 · p. 2, Table 1 · p. 2
  • At 262K, structured assembly reaches 82%/93% single-hop and 27%/41% multi-hop with gpt-4o-mini/gpt-4o, exceeding every reported MAB v3 FactConsolidation result.The benchmark's best reported retrieval/memory result was 54% single-hop, while all 22 systems scored at most 7% multi-hop.Source: §1 · pp. 1–2
  • The reported comparison is task-scoped and does not rank memory architectures across their broader capabilities.The evaluation is limited to current-value conflict resolution over data with explicit version markers, not partial orders, historical queries, or general temporal reasoning.Source: §1 · p. 3

2 Related work

Related work spans memory benchmarks and architectures, decomposition methods, structured reasoning interfaces, and temporal handling. The paper positions its contribution narrowly: an assembly interface for explicit version metadata and a fixed answer policy.

2 Related work

3 Structured post-retrieval assembly

Structured post-retrieval assembly separates candidate identification from policy execution: an LLM converts retrieved evidence into candidates, and an assembly operator applies the explicit policy.

3 Structured post-retrieval assembly

  • The single-hop pipeline retrieves top-10 BM25 evidence, extracts matching candidates, then returns the entity with the largest serial or “no answer”.Candidates match the query’s subject and predicate.Source: §3 · p. 4
  • The extractor preserves version metadata and returns every semantically matching candidate without choosing a winner, separating evidence identification from policy selection.This is the key interface between the two stages.Source: §3 · p. 4
  • For “X of Y of Z” questions, CAR resolves each atomic hop before substituting its answer into the next query.Each hop applies the single-hop conflict-resolution procedure.Source: §3 · p. 4
  • A prompt constraint allows at most one relationship word per hop, preventing four-hop chains from becoming unresolved composite hops.The passage identifies this as a hard constraint for gpt-4o-mini.Source: §3 · p. 5

4 Experimental setup

The experiments use MemoryAgentBench FactConsolidation v3 across four context lengths, with fixed retrieval settings and published comparisons based on the standard 262K setting.

4 Experimental setup

  • The evaluation uses MAB FactConsolidation v3 single-hop and multi-hop tasks at 6K, 32K, 64K, and 262K, with n = 100 per cell.Facts are counterfactual rewrites assigned higher serials; 6K is diagnostic, while 262K is used for comparisons with MAB’s main systems table.Source: §4 · p. 5
  • Experiments use gpt-4o-mini, gpt-4o, and diagnostic o4-mini; retrieval uses BM25Okapi with top-k = 10.Structured pipelines use temperature 0.0; the direct baseline uses 0.7, so temperature is part of the whole-pipeline comparison.Source: §4 · p. 5
  • Accuracy is measured with substring exact match, and published comparisons use MAB v3 Table 3 values at 262K without rerunning those systems.MAB’s released default backbone is used where a comparison entry does not name a model.Source: §4 · p. 5

5 Results

Structured assembly performs strongly on FactConsolidation, including against reported systems, while controlled comparisons locate most of its gain in structuring post-retrieval decisions. Its results vary with context length and query type, and the LongMemEval check does not demonstrate an overall advantage.

5.1 Master results

  • At 262K, structured gpt-4o-mini and gpt-4o reach 82% and 93% on single-hop questions, while gpt-4o CAR reaches 41% on multi-hop questions.The primary runs report 78.0% pooled for structured gpt-4o-mini and 94.8% pooled for gpt-4o.Source: §5.1 · p. 5

5.2 Comparison with published systems at 262K

  • At 262K, structured pipelines exceed every reported MAB v3 FactConsolidation system, with margins of 28 and 33 pp for FC-SH and 20 and 34 pp for FC-MH with gpt-4o-mini and gpt-4o, respectively.This is a task-level comparison, not a ranking of overall architecture quality.Source: §5.2 · pp. 5–6, Table 4 · p. 6

5.3 Where the gain comes from

  • Deterministic policy execution reaches 78.0% versus 76.0% for LLM execution, but both reach 82% at 262K, localizing most improvement upstream of final selection.The whole-pipeline gain is statistically reliable (χ2 = 14.6, p < 0.001), but its multiple changed components prevent attributing it to one component alone.Source: §5.3 · p. 6
  • Fact-level and chunk-4096 segmentation yield similar pooled means of 78.0% and 80.8%, with opposite context-length curves.Chunk-4096 wins at short lengths, while fact-level wins at 262K.Source: §5.3 · p. 6

5.4 Robustness to context length

  • The direct-answer baseline drops 14 points from 64K to 262K, whereas structured assembly does not show the same collapse.With identical top-10 retrieval, direct-answer accuracy on questions solved by structured assembly falls from 78% at 64K to 66% at 262K.Source: §5.4 · p. 6

5.5 Complementarity and a retrieval lower bound

  • Across 400 gpt-4o-mini FC-SH item-length observations, at least one pipeline solves 88.5%, a lower bound for the fixed top-10 retrieval setup.Structured assembly trades some recall, as strict extraction rejects valid candidates, for a larger precision gain from constrained selection.Source: §5.5 · pp. 6–7

5.6 Multi-hop results and backbone scaling

  • CAR with gpt-4o reaches 51.5% pooled and 41% at 262K, while decomposition hurts o4-mini at 6K but helps at 32K.Most failures begin with an incorrect first-hop extraction and then cascade.Source: §5.6 · p. 7

5.7 Cross-benchmark generalization on LongMemEval

  • On LongMemEval’s 45-question knowledge-update subset, structured assembly with the newest-timestamp policy does not outperform the baseline overall.The authors narrow the supported scope to current-value questions; broader conversational QA requires question-type-aware composition.Source: §5.7 · p. 7

6 Discussion

The discussion attributes the main gain to separating semantic evidence identification from explicit policy execution, while limiting the demonstrated scope to current-value conflicts with explicit version metadata.

6 Discussion

  • The ablations support architectural separation, not a special inability to compare serial numbers: the LLM usually applies the ordering rule once candidates are clean.The direct baseline combines semantic filtering, prior suppression, policy execution, and answer generation; the whole-pipeline comparison does not isolate which extraction component contributes how much.Source: §6 · p. 7
  • Explicit policy execution makes decisions consistent for identical candidates, auditable, independently testable, and replaceable for historical, thresholded, or aggregation queries.The small policy-executor accuracy difference does not capture these guarantees.Source: §6 · p. 7
  • The proposed division of labor uses an LLM for semantic evidence identification and a separate interface for answer-time policy execution.Memory design should preserve version metadata at conflict granularity and expose the evidence used for decisions.Source: §6 · p. 8

7 Conclusion

The conclusion frames memory conflict resolution as post-retrieval assembly: an LLM extracts structured candidates, then a separate stage executes an explicit answer policy. Results are strongest on MAB v3 FactConsolidation, with the demonstrated benefit bounded to current-value conflicts with explicit version metadata.

7 Conclusion

  • At 262K, the recipe reaches 82%/93% single-hop and 27%/41% multi-hop with gpt-4o-mini/gpt-4o, exceeding every reported MAB v3 FactConsolidation result.A controlled whole-pipeline comparison yields a 10.8 pp average single-hop gain and a 21 pp gain at 262K despite identical retrieved evidence.Source: §7 · p. 8
  • The targeted policy-executor comparison yields only 2.0 pp on average and 0 pp at 262K; the main lesson is to separate evidence identification from policy execution.The conclusion attributes much of the compound-decision burden reduction to this separation, rather than to the freshness operator itself.Source: §7 · p. 8
  • The LongMemEval check shows no overall advantage, bounding the demonstrated benefit to current-value conflicts with explicit version metadata.Source: §7 · p. 8

A Hyperparameters

Table 6 is identified as the experimental-settings table; the supplied passage reports the study's interval, sample-size, evaluation-count, cost, and runtime details.

A Hyperparameters

  • Wilson 95% intervals use z = 1.96, with n = 100 per cell and n = 400 pooled.Source: A Hyperparameters · p. 10
  • The experiments comprise 4,400 LLM-driven evaluations; total API cost was approximately $3, and four parallel processes completed runs in approximately 15 minutes.The cost was dominated by gpt-4o cells.Source: A Hyperparameters · p. 10

B Policy-executor comparison detail

The policy-executor detail compares LLM and deterministic selection on contested observations; the full comparison table covers FactConsolidation accuracy at 262K.

B Policy-executor comparison detail

  • The approximate policy-executor effect is +5.9 pp on contested observations but does not grow with context length; at 262K, both executors select the same serials and have identical accuracy.FactConsolidation generally caps the semantic candidate set at an original and one counterfactual value, so it does not test selection over many candidates.Source: B Policy-executor comparison detail · p. 10

C Full MAB v3 comparison at 262K

  • Table 8 compares 262K FactConsolidation accuracy for the task-specialized pipeline, given the freshness policy a priori, against published general-purpose systems without that task-specific assembly layer.Source: Table 8 · p. 11

D Qualitative failure detail

A qualitative spot-check grouped incorrect answers into approximate failure categories, while LongMemEval disagreements reflected output-format, historical-query, and interval-aggregation mismatches.

D Qualitative failure detail

  • The gpt-4o-mini FC-SH spot-check classified approximate errors as wrong-subject confusion (25%), predicate semantic gap (25%), empty extraction (10%), and other categories.Counterfactual absence from the top ten and real-world prior or plurality effects each accounted for 10%; ambiguous serial ties accounted for 5%, and other or unclassified errors for 15%.Source: D Qualitative failure detail · p. 11
  • The authors describe the spot-check as a qualitative pilot, and note that deterministic ordering removes one failure mode but does not solve retrieval or semantic matching.The categories were approximate and based on approximately 30 incorrect answers across four runs, rather than systematic labeling.Source: D Qualitative failure detail · p. 11
  • Five versus two discordant wins favored the direct baseline over the structured pipeline on LongMemEval, with paired exact McNemar p = 0.45.The five direct-baseline-only cases comprised three Yes/No prefix mismatches, one historical query, and one earlier-interval aggregation task.Source: D Qualitative failure detail · p. 11

E Reproducibility and use of language models

The implementation uses public MAB v3 FactConsolidation and LongMemEval datasets and provides code, prompts, traces, and a runner. Language models supported experiments and editorial work, but did not generate labels or measurements.

E Reproducibility and use of language models

  • Code, prompts, per-question traces, and a one-command runner accompany the public MAB v3 FactConsolidation and LongMemEval datasets.The release includes retrieved and selected serials, extracted candidates, predictions, and SubEM outcomes for every reported cell.Source: E Reproducibility and use of language… · p. 11
  • Language models served as experimental backbones and assisted with editorial and LaTeX work, but generated neither ground-truth labels nor reported measurements.The authors verified numerical results, statistical tests, citations, claims, and final text.Source: E Reproducibility and use of language… · p. 11

Related papers

Loading 2606.01435v2…