PaperlayerView exact revision on arXiv

Source-linked AI summary

Same Ranking, Different Winner: How Scoring Targets Shape LLM Memory Benchmarks

Sugam Panthi, Rabab Abdelfattah

arXiv:2605.24060v2cs.IR

TL;DR

Conversational-memory benchmarks must decide which stored forms derived from a source turn can receive retrieval credit. TIAP rescores fixed ranked outputs under Raw, Source, and Canonical targets; changing the target changes nDCG across shared queries and yields different target orderings in transfer runs.

  • Problem

    Conversational-memory benchmarks may index raw dialogue turns alongside derived memories but leave underspecified which stored forms from a source turn should receive retrieval credit.

    Source: §1 · p. 1

  • Method

    TIAP rescores saved ranked outputs under Raw, Source, and Canonical targets without rerunning retrieval.

    Source: §1 · p. 2, §3.1 · p. 3

  • Results

    83.4% to 94.0% of shared queries change Raw-versus-Canonical nDCG when only the scoring target switches, and Mem0 and MemoryOS transfer runs show different target orderings.

    Source: §1 · p. 2

  • Takeaways & Limitations

    Benchmarks should define and report the credited target set because target choice can alter scores, system rankings, and memory-store recommendations.

    Source: §1 · p. 2

    TIAP uses binary target membership and does not grade partial evidence or compose support across retrieved facts, limiting its fit for queries requiring multiple evidence units.

    Source: §7 · p. 10

Abstract

from arXiv · show

Conversational-memory systems increasingly transform dialogue history into facts, summaries, timelines, and other source-linked descendants, so a single source turn can coexist with several derived memories in the same retrieval index. This raises an underspecified evaluation question: which stored form should receive retrieval credit? We show that this scoring-target choice is often left implicit and can materially change benchmark conclusions. We present TIAP, a fixed-output audit that rescores saved ranked outputs under three targets -- Raw, Source, and Canonical -- without rerunning retrieval. On LoCoMo and LongMemEval-S, switching only the credited target changes nDCG on 83.4--94.0 percent of shared queries, flips target orderings on Mem0 and MemoryOS transfer runs, and reverses parser-density recommendations. A 1,902-case semantic audit further shows that relaxed source-linked credit is fully justified only 29.2 percent of the time, despite high rubric reliability in a validation subset. These results reveal target noninvariance: conclusions about memory architectures can silently flip with a single benchmark-design choice. Conversational-memory papers should therefore define and report the scoring target explicitly.

1 Introduction

Memory systems retain transformed descendants alongside source turns, leaving benchmark designers to decide which stored forms count as correct retrieval. This choice can alter benchmark conclusions even when ranked outputs are fixed.

1 Introduction

  • Only 29.2% of source-linked descendants receiving relaxed credit fully support the query in the 1,902-case audit.Source: §1 · p. 2
  • 83.4% to 94.0% of shared queries change Raw-versus-Canonical nDCG when only the target switches, while Mem0 and MemoryOS transfer runs show different target orderings.Source: §1 · p. 2
  • TIAP audits saved outputs under different admissible targets, and MTEL-Mem packages target mappings, source fixtures, and rescoring utilities.Source: §1 · p. 2
  • Because no target is universally correct, conversational-memory benchmarks should define and report which stored forms receive credit.Source: §1 · p. 2

2 Related Work

Prior work evaluates retrieval and answers, studies memory architecture, or analyzes sensitivity to evaluation choices. This paper focuses on which system-generated descendants of source evidence are eligible for retrieval credit.

2 Related Work

3 Method

TIAP defines Raw, Source, and Canonical target sets, then compares them on the same saved ranked outputs. It measures target sensitivity and audits whether expanded credit has semantic support.

3.1 Stage I: Constructing Scoring Targets

  • Raw credits only the exact source turn, Source credits all descendants linked to that evidence, and Canonical credits serving-form transformed descendants.Canonical may include multiple transformed descendants per source; Raw and Canonical are usually disjoint, while both are subsets of Source.Source: §3.1 · p. 3

3.2 Stage II: Fixed Output Rescoring

  • TIAP rescores fixed ranked traces under each target and compares targets on shared queries when target coverage differs.Shared-subset comparisons include only queries with at least one eligible stored ID under every compared target, separating target effects from changes in query population.Source: §3.2 · pp. 3–4

3.3 Stage III: Target Sensitivity and Semantic Audit

  • TIAP records score shifts, hit changes, winner flips, and design-recommendation changes, with a winner flip defined by a target-dependent sign change in the system gap.A changed preferred store or configuration on fixed ranked outputs makes the recommendation depend on the scoring target rather than retrieval behavior.Source: §3.3 · p. 4
  • The semantic audit checks contested Source or Canonical credits for full, partial, or absent support because shared lineage alone does not ensure that a memory supports the answer.A transformed memory can omit the answer, remove a relevant condition, change temporal scope, or preserve only weakly related context.Source: §3.3 · pp. 4–5

4 Experimental Setup

The experiments map benchmark evidence to three targets and rescore saved traces across conversational-memory datasets and retrievers. A validation subset checks rubric consistency before the full contested-credit audit.

4.1 Datasets

  • LoCoMo and LongMemEval-S provide source-turn or session evidence anchors, while BEAM-1M serves only as a supplementary provider robustness check.The benchmarks hold out queries, reference answers, and evidence labels from the parser that produces transformed memories.Source: §4.1 · p. 5

4.2 Target Mappings

  • Lineage metadata deterministically maps each source fixture to its raw ID and transformed descendants; Canonical includes only non-raw descendants and may have lower coverage than Raw.Across 13,620 query-system pairs, Source and Raw contain the same IDs in 44.3%, Source adds one ID in 32.3%, and adds two or more in 23.4%.Source: §4.2 · p. 5

4.3 Retrieval and Memory Systems

  • At k = 20 versus k = 60, Source–Raw relative shifts grow from +57% to +84% on LoCoMo and from +123% to +215% on LongMemEval-S; Canonical exceeds Raw on every shared subset.Source exceeds Canonical on all four LoCoMo shared subsets.Source: §4.3 · p. 6, Table 2 · p. 6

4.4 Semantic Audit Methodology

  • A stratified 115-case validation with a 36-case human overlap achieved five-rater Fleiss’ κ = 0.83 before five frontier models labeled all 1,902 contested cases.The validation tested rubric reproducibility; each model judged cases independently at temperature 0.Source: §4.4 · p. 6, Table 3 · p. 6

5 Results

Across the results, changing the credited memory target changes scores and engineering recommendations on fixed ranked outputs, with effects across query types and transferred architectures. Relaxed credit is often only partly or not semantically supported, so target selection should be explicit and deployment-aligned.

5.1 Scoring Target Alters Retrieval Scores

  • 83.4% to 94.0% of shared queries show changed nDCG in Raw-versus-Canonical rescoring across LoCoMo and LongMemEval-S, shifting scores and the relative evaluation landscape.The ranked outputs remain fixed; the target change alone alters which stored forms receive credit.Source: §5.1 · p. 7, Table 4 · p. 7
  • Canonical-covered queries are easier under Raw by a mean run-level nDCG gap of +0.086, so Raw-versus-Canonical comparisons use shared covered subsets.Answer F1 changes less, with a mean gap of +0.008.Source: §5.1 · p. 7

5.2 Score Shifts Are Not Explained by Target-Set Size

  • Target-set size does not explain most score shifts: it explains at most 1.7% of Canonical-minus-Raw shift variance, and one extra eligible ID yields a +0.153 mean shift.For Source versus Raw, target-set size increase correlates with the nDCG shift (r = 0.38), explaining about 14% of its variance; shifts saturate after two to three extra IDs.Source: §5.2 · p. 7

5.3 Scoring Target Changes Store Design Recommendations

  • On a fixed matched subset, Raw prefers parser density D1 over D5, Source prefers D5, and Canonical returns to D1; Source favors denser settings in all six pairwise comparisons.The sweep uses parser_max_facts values of 1, 5, and 8 (D1, D5, and D8) on n = 453; Raw prefers the sparsest setting in four of six comparisons, while Canonical is split.Source: §5.3 · pp. 7–8, Table 5 · p. 8

5.4 Target Sensitivity Is Broad Across Query Types

  • Raw-versus-Canonical rescoring changes nDCG for 83.6% to 93.2% of queries within major LoCoMo categories and 77.1% to 100.0% within LongMemEval-S categories.For LongMemEval-S, Source-versus-Canonical instability is concentrated in multi-session and knowledge-update questions and is near zero for single-session assistant questions.Source: §5.4 · p. 8

5.5 Relaxed Credit Is Only Partially Reliable

  • Only 29.2% of 1,902 contested credits fully support the query; 39.6% are partial and 31.2% do not support it.Across 1,876 cases with valid labels from all five judges, mean pairwise agreement is 90.9% for binary labels and 85.9% for three-class labels.Source: §5.5 · p. 8

5.6 Target Sensitivity Transfers Across Memory Architectures

  • Mem0 and MemoryOS remain target-sensitive across Recall@60, MRR, and nDCG@60, but target orderings vary by architecture, dataset, and metric.On BEAM’s full 625-query run, nDCG@60 orders targets as Raw 0.055, Source 0.097, and Canonical 0.197.Source: §5.6 · pp. 8–9, Table 7 · p. 9

5.7 Choosing a Scoring Target

  • The appropriate target depends on what the deployed system serves: Canonical matches transformed-memory deployments, while Raw is primary for original-turn use or exact provenance.The authors position Source as a provenance-relaxed stress test, not a default headline target; they recommend shared subsets when coverage differs and explicit reporting of winner flips.Source: §5.7 · p. 9, Table 8 · p. 9

6 Conclusion

Changing only the credited target can alter scores, system rankings, and store-design recommendations on fixed ranked outputs. The semantic audit also finds that relaxed source-linked credit is only partially reliable.

  • Changing the credited target alters scores on 83% to 94% of shared queries, changes store-design recommendations, and transfers across memory architectures.These changes occur under fixed ranked outputs.Source: §6 · p. 9
  • The semantic audit finds that source-linked descendants do not always preserve the evidence needed to answer a query.Relaxed credit is only partially reliable.Source: §6 · p. 9
  • Benchmark reports should specify the credited target set, use shared-subset rescoring when coverage differs, and flag cases where target choice changes the winner.TIAP makes target sensitivity visible and reproducible.Source: §6 · p. 9

7 Limitations

The study focuses on transformed conversational memory, with its native headline matrix concentrated in one parser-derived store and only a small sample of other architectures. Its audit also has boundaries in validation, query coverage, and how it represents evidential sufficiency.

  • The study concerns transformed conversational memory, not IR evaluation generally, and its native headline matrix centers on one parser-derived store.Mem0 and MemoryOS broaden the evidence but remain a small architectural sample.Source: §7 · p. 9
  • Prevalence estimates from the 1,902-case semantic audit should be interpreted in light of a 115-case human validation subset.That subset had one primary annotator and a 36-case adjudicated overlap.Source: §7 · pp. 9–10
  • Canonical coverage is nonrandom, and queries with canonical coverage have higher Raw nDCG than uncovered queries.The main Raw–Canonical claims therefore use matched canonical-covered subsets.Source: §7 · p. 10
  • TIAP uses binary target membership and does not grade partial sufficiency or compose support across multiple retrieved facts.This especially constrains queries whose answers are distributed across evidence units, including multi-hop, temporal, and aggregation questions.Source: §7 · p. 10
  • The Raw / Source / Canonical basis assumes stable source anchors, recoverable lineage, enumerable descendant targets, and exported fixed ranked traces.Graph-native memories may need node-, edge-, or path-level target definitions beyond this minimal basis.Source: §7 · p. 10
  • TIAP and MTEL-Mem make target sensitivity auditable rather than prescribing a universal target set or pass/fail threshold.Extending the audit to graph-native memories and other retrieval settings remains future work.Source: §7 · p. 10

Supplementary Material

Supplementary analyses detail query-level instability, test aggregation and agreement filtering as mitigations, report BEAM stress tests, and illustrate qualitative disagreements.

A Query-Level Instability Details

  • 83.4%–94.0% of Raw-versus-Canonical comparisons change nDCG, while 31.3%–66.2% of Raw-versus-Source and 44.2%–55.2% of Source-versus-Canonical comparisons also change.Figure 3 shows shifts of at least 0.1 nDCG for 65% of LoCoMo and 71% of LongMemEval-S query–provider pairs, with mean |∆| of 0.25 and 0.33, respectively.Source: A Query-Level Instability Details · p. 13, Figure 3 · p. 13

B Aggregation and Mitigation Checks

  • No tested aggregation creates a novel provider ranking, and agreement filtering leaves nDCG divergent and LongMemEval-S rankings target-dependent.Agreement filtering retains 72% to 85% of queries, but mean per-query nDCG spreads remain between 0.16 and 0.30.Source: B Aggregation and Mitigation Checks · p. 13, Table 9 · p. 13

C Supplementary BEAM Runs

  • BEAM preserves the Raw < Source < Canonical ordering on the full run and the canonical-covered shared subset.The full 625-query BEAM run reports nDCG@60 of 0.055 under Raw, 0.097 under Source, and 0.197 under Canonical.Source: C Supplementary BEAM Runs · pp. 13–14, Table 11 · p. 14

D Qualitative Disagreement Cases

E Auxiliary Answer-Level Alignment

The auxiliary analysis compares target hits with answer-level alignment and reports semantic-audit rubric agreement. Together, these checks do not identify one universally preferred target set.

E Auxiliary Answer-Level Alignment

  • Raw has the strongest average answer-alignment signal across eight native LoCoMo runs, while Source and Canonical align better on LongMemEval-S across four native runs.The difference is consistent with LoCoMo's lexical scoring and LongMemEval-S's more paraphrase-tolerant judge-based scoring; answer-level alignment does not identify one universally preferred target set.Source: E Auxiliary Answer-Level Alignment · p. 14, Table 13 · p. 14

F Semantic Audit Agreement

  • All four frontier LLMs achieved substantial to almost perfect consistency with human validation labels on the 115-case subset.The validation tested rubric reproducibility rather than model accuracy.Source: F Semantic Audit Agreement · p. 14

G Full-Audit Multi-Model Agreement

The 1,902-case audit found high agreement among five LLM judges, and its majority-vote labels largely matched the original Sonnet 4 labels.

G Full-Audit Multi-Model Agreement

  • 1,876 of 1,902 cases received valid labels from all five models; 26 returned empty responses from GPT-5.5 or DeepSeek V4 Pro after retries.Pairwise agreement was reported on the 1,876 common-valid cases.Source: G Full-Audit Multi-Model Agreement · p. 14
  • Mean agreement was 90.9% across binary labels and 85.9% across three-class labels.Binary agreement collapses supports and partial into relevant.Source: G Full-Audit Multi-Model Agreement · p. 14
  • The five-model majority vote matched the original Sonnet 4 labels on 90.1% of cases; 29.2% fully supported relaxed source-linked credit.Majority-vote labels were 29.2% supports, 39.6% partial, and 31.2% does not support.Source: G Full-Audit Multi-Model Agreement · p. 14
  • The dataset-level fully supportive gap—approximately 33% on LoCoMo versus 21% on LongMemEval-S—was reproduced by all five judges.The reported distribution is based on majority-vote labels.Source: G Full-Audit Multi-Model Agreement · p. 14

H Reporting-Practice Audit

An audit of publicly stated practices in 11 conversational-memory papers found incomplete specification of which transformed memories receive retrieval credit. Benchmarks and systems provide some provenance information, but no paper fully maps target credit through transformations to source evidence.

H Reporting-Practice Audit

  • The audit checked publicly stated or released practices in 11 cited benchmark and memory-system papers, without inferring that authors lacked internal conventions.It examined transformed representations, retrieval-unit definitions, target mappings, and descendant credit.Source: H Reporting-Practice Audit · pp. 14–15
  • No paper fully specifies a target mapping from scored descendants to source evidence, and none addresses credit assignment when one source turn yields multiple stored descendants.The reporting-practice table covers 11 conversational-memory papers and marks descendant-credit specification as absent across them.Source: H Reporting-Practice Audit · p. 15, Table 15 · p. 15
  • LoCoMo labels QA gold answers with turn IDs, LongMemEval-S labels evidence sessions and positions, and BEAM includes a sometimes-empty source_chat_ids field.Ong et al. (2025) note missing gold mappings in existing long-term conversation datasets; Rasmussen et al. (2025) report episodic links that their experiments do not examine.Source: H Reporting-Practice Audit · p. 15
  • The audit recommends stating the scored target set and releasing target mappings alongside retrieval traces.LoCoMo alone explicitly labels and separately evaluates distinct unit types.Source: H Reporting-Practice Audit · p. 15

I Cardinality Analysis

The cardinality analysis finds that nDCG shifts saturate after a few extra eligible IDs, while shallower retrieval depth amplifies target differences. The parser derives memories only from conversation turns and uses lineage metadata for target mappings.

I Cardinality Analysis

  • The Source-minus-Raw size difference is an upper bound on Canonical expansion, making the 1.7% variance figure an upper bound on cardinality contribution.The analysis rescores the same saved ranked traces and includes 13,620 query-system pairs in the full-set Source-minus-Raw comparison.Source: I Cardinality Analysis · p. 15, Table 17 · p. 16
  • At one extra eligible ID, the nDCG shift was significant in both settings (t = 46.8 full set; t = 27.7 covered Canonical minus Raw; both p < 10^-10), then saturated after two to three extra IDs.The full-set bins contained n = 6,032, 4,399, 1,688, 807, and 694 pairs; covered bins contained n = 4,241, 1,672, 806, and 694.Source: I Cardinality Analysis · p. 15

J Retrieval-Depth Sensitivity

  • As retrieval depth k decreases, the relative Source-versus-Raw shift grows because higher-ranked positions carry more nDCG weight at shallow depth; k = 60 is most conservative.Table 19 compares mean Raw and Source nDCG at k = 20, 40, and 60.Source: J Retrieval-Depth Sensitivity · pp. 15–16, Table 19 · p. 16

K Parser Operational Details

  • The parser uses conversation turns only, caps extracted facts at 1, 5, or 8 per turn, and derives target mappings from lineage metadata rather than text similarity.Queries, reference answers, and evidence labels are held out; not every source turn yields a canonical serving memory. Native runs use extractive answer mode with seed 1337 and locally served Ollama embeddings.Source: K Parser Operational Details · pp. 16–17
Loading 2605.24060v2…