PaperlayerView exact revision on arXiv

Source-linked AI summary

Before Answering: Evidence Sufficiency under Size-Matched Memory Construction

Joyanta Jyoti Mondal, Md. Shifatul Ahsan Apurba, Mridul Banik, Md Masud Al Mahmud, Ibne Farabi Shihab

arXiv:2609.32269v1cs.CLcs.LG

TL;DR

Agents using compressed or retrieved memory need to assess whether query-required evidence remains, but deletion-based benchmark construction can expose memory size as a shortcut. The paper introduces size-matched construction and evaluates MEMSAFE, a cross-encoder with set-level aggregation. MEMSAFE leads on the reported benchmark metrics, though one dataset is saturated, and its sufficiency score also improves answer gating in the reported setting.

  • Problem

    Deletion-based insufficient-evidence examples can leak labels through memory size, complicating evaluation of whether estimators detect missing evidence.

    Source: §1 · pp. 1–2

  • Method

    The paper matches memory size across integrity states and evaluates MEMSAFE, which cross-encodes query–unit pairs and aggregates them with a set transformer.

    Source: §1 · p. 2, §3.3 · p. 4

  • Results

    MEMSAFE achieves the best unsafe AUROC and integrity macro F1 on all three datasets, while 2WikiMultiHopQA is treated as saturated.

    Source: §5.2 · p. 5, §5.4 · p. 7

  • Takeaways & Limitations

    The paper concludes that sufficiency estimators should be evaluated against size-matched constructions and reports that MEMSAFE is a better answer gate than reader confidence and, on average, the integrity label.

    Source: §7 · p. 9

    The evaluated memories come from QA benchmarks and clinical records rather than deployed agents, and the main models rely on gold supporting-fact labels.

    Source: §6 · p. 9

Abstract

from arXiv · show

Agents that answer questions from compressed or retrieved memory must recognize when the evidence a query needs is no longer in memory. Benchmarks for this task usually create insufficient-evidence examples by deleting supporting passages. We show that this construction leaks the label through memory size: on MuSiQue, a classifier that only counts paragraphs reaches an area under the ROC curve (AUROC) of $0.979$ for detecting unsafe memory, higher than the lexical estimator we initially evaluated. We propose a size-matched construction that provably removes this shortcut, and use it to study MemSafe, an estimator that cross-encodes the query with each memory unit and aggregates the units with a set transformer. Across three multi-hop question answering datasets and five seeds, MemSafe reaches $0.968$ and $0.983$ AUROC on MuSiQue and HotpotQA, $0.26$ to $0.39$ above a lexical baseline, while the third dataset, 2WikiMultiHopQA, is saturated. A frozen pretrained cross-encoder with a logistic head already closes $41\%$ of the MuSiQue gap between the lexical baseline and MemSafe. At the same time, MemSafe degrades more than a weak baseline on the unanswerable questions released with MuSiQue, reaches only $0.639$ AUROC on SQuAD~2.0, and needs several thousand clinical training examples before it outperforms a feature-based estimator. Used as a gate for a 7B reader, it reduces the error rate on answered questions from $0.850$ to $0.631$ at $5\%$ coverage, outperforming both reader confidence and, on average, the ground-truth integrity label, although a 7B LLM judge is the better gate at $10\%$ coverage. These results indicate that the way insufficient evidence is constructed matters as much as the estimator that detects it.

1 INTRODUCTION

Agents can lose query-relevant evidence when they compress or retrieve memory, so they need to assess sufficiency before answering. This work examines that assessment after correcting a memory-size shortcut in deletion-based benchmarks.

1 INTRODUCTION

  • 0.979 unsafe AUROC from paragraph counts exceeded the lexical estimator’s 0.968 on MuSiQue when supporting paragraphs were removed without replacement.With memory size held fixed, the same size-only classifier performs at chance.Source: §1 · p. 2
  • MEMSAFE cross-encodes each query with memory units and aggregates them with a set transformer to estimate memory sufficiency.The paper compares it with lexical and matched comparators, long-context encoders, an LLM judge, and surface-level controls.Source: §1 · p. 2
  • Size-matched construction removes the memory-size shortcut and gives size-only estimators chance-level AUROC when each base query contributes equal numbers of integrity states.Otherwise, the construction provides a bound on deviation from chance.Source: §1 · p. 2
  • The study separates the contributions of pretrained cross-encoding, fine-tuning, set-level interaction, and gold versus silver supervision.Source: §1 · p. 2
  • Stress tests examine a second construction, SQuAD 2.0, and clinical records at several scales, including settings where the estimator underperforms.Source: §1 · p. 2
  • A continuous sufficiency score is evaluated as an answer gate and is reported to outperform the ground-truth integrity label empirically and formally.Source: §1 · p. 2

2 RELATED WORK

Prior work studies context sufficiency, evidence verification, construction artifacts, retrieval control, and selective prediction. This paper uses an answer-free sufficiency estimator and controls how insufficient examples are constructed.

Context sufficiency and evidence verification

  • The paper follows Joren et al. (2025)’s answer-free sufficiency setting, replacing an LLM autorater with a supervised estimator that also identifies evidence-bearing units.It also differs from passage-set verification by controlling how insufficient examples are constructed.Source: §2 · p. 3

Construction artifacts

  • Deletion-based NEI benchmarks can contain exploitable artifacts; the paper removes memory size as one such artifact and tests transfer across constructions.Source: §2 · p. 3

Multi-hop evidence and retrieval control

  • Unlike adaptive retrieval controllers that decide whether more information is needed, this estimator assesses whether the current memory contains required evidence and predicts what is missing.The authors describe it as complementary because retrieval controllers can use that information.Source: §2 · p. 3

Selective prediction

  • Selective prediction commonly thresholds model-confidence scores, whereas this work thresholds an evidence-derived score on validation data and evaluates answer gating at fixed coverage.The threshold is chosen without a distribution-free risk guarantee.Source: §2 · p. 3

3 METHOD

The method defines memory integrity relative to query-required evidence, constructs equal-sized memories across integrity states, and estimates sufficiency with a two-stage model.

3.1 PROBLEM FORMULATION

  • A memory is unsafe when its integrity state is not COMPLETE; the model predicts integrity, unsafe probability, unit-level evidence presence, and the number of missing items.The four states are COMPLETE, MISSING, RELATION-LOST, and STALE; QA evidence annotations may not enumerate every sufficient route to an answer.Source: §3.1 · p. 3
  • A gate answers when the unsafe score is at most a threshold, but evidence risk and reader risk measure different outcomes, and controlling one does not bound the other.Source: §3.1 · pp. 3–4

3.2 SIZE-MATCHED MEMORY CONSTRUCTION

  • Each integrity state receives a memory of the same size, using retained evidence units and hash-ordered distractors; unmatched queries are discarded.Deletion-only construction makes complete memories larger than missing memories, exposing integrity through size.Source: §3.2 · p. 4
  • Under size matching, any estimator that uses only memory size has chance-level AUROC when each base query contributes equal numbers of unsafe and complete examples.Otherwise, the deviation from chance is bounded by the total-variation distance between unsafe and complete base-query distributions.Source: §3.2 · p. 4
  • The guarantee depends on example counts: a query with many unsafe variants can dominate the unsafe distribution even if exceptional queries are rare.Source: §3.2 · p. 4

3.3 TWO-STAGE ESTIMATOR

  • Stage A cross-encodes query–unit pairs and adds a top-unit-conditioned encoding to capture second-hop evidence that may be lexically unrelated to the query.The pretrained MiniLM cross-encoder truncates inputs to 256 tokens.Source: §3.3 · p. 4
  • The encoder is fine-tuned to identify evidence-bearing units, then a set transformer lets unit representations interact and predicts unsafe probability, integrity state, missing count, and unit-level evidence.Comparators use the same embeddings with mean pooling or aggregated relevance scores; another baseline encodes the concatenated memory.Source: §3.3 · p. 4

4 EXPERIMENTAL SETUP

Experiments use three multi-hop QA datasets and two memory-construction families. The study fixes question-level splits, uses five pre-specified seeds, and registers hypotheses before final runs.

4 EXPERIMENTAL SETUP

  • MuSiQue, HotpotQA, and 2WikiMultiHopQA provide 21,902, 97,406, and 180,030 base questions, respectively, yielding 65,297, 292,218, and 540,090 memory variants.Family A is size-matched; Family B uses MuSiQue unanswerables with an answer paragraph removed at equal context length.Source: §4 · p. 5
  • Splits are assigned by salted hash before corruption; five pre-specified seeds are used, but two reuse encoders, so they are not five independent full-pipeline repetitions.Unsafe AUROC is primary; bootstrap intervals omit encoder-training variation, and five seeds cannot yield a two-sided Wilcoxon p-value below 0.05.Source: §4 · p. 5
  • Pre-registered hypotheses test MEMSAFE against mean pooling, LEXSET, and TF-IDF; transfer across constructions; bridge conditioning; and performance on MIMIC-IV.Source: §4 · p. 5

5 RESULTS

Size matching removes the paragraph-count shortcut, enabling a comparison of sufficiency estimators on matched memories. MEMSAFE leads across the three datasets, but its gains and utility vary with task difficulty, construction, coverage, and training scale.

5.1 SIZE MATCHING REMOVES THE SHORTCUT

  • 0.979 unsafe AUROC from paragraph counting under deletion falls to 0.5007 on MuSiQue after size matching; counts score exactly 0.5000 on HotpotQA and 2WikiMultiHopQA.Text length remains a surface cue, reaching 0.525 to 0.633 AUROC, so the construction is count-matched rather than artifact-free.Source: §5.1 · p. 5

5.2 MAIN RESULTS

  • MEMSAFE outperforms every baseline on all three datasets and five seeds, with paired AUROC advantages over LEXSET of +0.3862 on MuSiQue, +0.3222 on HotpotQA, and +0.2648 on 2WikiMultiHopQA.All three reported intervals exclude zero; 2WikiMultiHopQA is nevertheless saturated, with encoder-based models above 0.99 AUROC.Source: §5.2 · pp. 5–6

5.3 SOURCES OF THE IMPROVEMENT

  • Fine-tuned cross-encoding raises AUROC from 0.582, 0.661, and 0.735 to 0.922, 0.961, and 0.995; set interactions add smaller gains that shrink on easier tasks.A frozen cross-encoder already closes 41% of the MuSiQue gap between LEXSET and the full model; fine-tuning and set modeling show diminishing returns when combined.Source: §5.3 · p. 7
  • Silver unit labels reduce AUROC from 0.9674 to 0.9444 but lower per-unit AUPRC from 0.9123 to 0.4252; gold supporting facts still define variants and integrity labels.The detection change is modest compared with the loss in evidence localization.Source: §5.3 · p. 7

5.4 TRANSFER ACROSS CONSTRUCTIONS AND SATURATION

  • Under family-A-to-family-B transfer, MEMSAFE reaches 0.8948 AUROC versus 0.6496 for flat TF-IDF but loses more AUROC, 0.0734 versus 0.0550; 2WikiMultiHopQA's near-perfect scores likely reflect templated entity cues.The authors treat 2WikiMultiHopQA as saturated and rely on MuSiQue to compare strong models.Source: §5.4 · p. 7

5.5 SELECTIVE PREDICTION AT A FIXED RISK BUDGET

  • At a shared 5% validation evidence-risk target, MEMSAFE accepts 0.2479 of MuSiQue memories versus 0.1334 for mean pooling, though their realized test risks differ.On 2WikiMultiHopQA, both methods are near the prevalence-limited coverage ceiling, so that dataset does not separate them at this operating point.Source: §5.5 · p. 8

5.6 GATING A READER

  • At 5% coverage, MEMSAFE gating lowers 7B-reader error from 0.8503 to 0.6306 ± 0.0206, outperforming reader confidence and, on average, the integrity-label gate.The LLM judge is the better gate at 10% coverage; about two thirds of answered questions remain wrong at 5% coverage.Source: §5.6 · p. 8

5.7 CLINICAL EVALUATION

  • On clinical OMR, LEXSET leads at 2,788 training variants, while MEMSAFE leads from 6,844 onward; LEXSET remains the better ranker for small clinical cohorts.At the largest reported scale, MEMSAFE reaches 0.9911 AUROC versus 0.9648 for LEXSET; realized patient counts also grow along the learning curve.Source: §5.7 · pp. 8–9, Figure 3 · p. 9

6 LIMITATIONS

The evaluation is limited by constructed memories, annotation dependence, and narrow clinical and reader-study settings; retrieval and abstention policies remain future work.

6 LIMITATIONS

  • Evaluated memories come from QA benchmarks and clinical records, not from deployed agents.The main models use gold supporting-fact labels, and gold annotations also define memory variants and integrity labels.Source: §6 · p. 9
  • Learning from memories without annotated evidence remains open because silver targets preserve detection but not localization.Gold annotations still define the memory variants and integrity labels.Source: §6 · p. 9
  • Re-derived splits prevent direct comparison with published development-set results, while the clinical curve uses one event source and three seeds per point.The reader study covers one dataset with greedy decoding.Source: §6 · p. 9
  • The LLM judge uses one pinned model with fixed prompts, and the answer-gating evaluation leaves retrieval and abstention policies for future work.Inference rests on paired bootstrap intervals.Source: §6 · p. 9

7 DISCUSSION AND CONCLUSION

The discussion argues that memory-size shortcuts must be removed before assessing sufficiency estimators. It concludes that MemSafe can gate answers effectively, but robustness and clinical data scale constrain its scope.

7 DISCUSSION AND CONCLUSION

  • Deletion-based benchmarks let a trivial size control outperform the estimator initially evaluated, motivating size-matched controls and evaluation on held-out construction families.The authors also recommend testing at the data scale intended for deployment.Source: §7 · p. 9
  • Most of MemSafe’s signal comes from a pretrained cross-encoder, while fine-tuning and set-level interaction add overlapping gains.This qualifies the emphasis on set-level verification in Qiu et al. (2026b) for the answer-free setting.Source: §7 · p. 9
  • As an answer gate, MemSafe outperforms reader confidence and, on average, the ground-truth integrity label, but the strongest in-construction model is not the most robust.A small clinical study would favor the lexical estimator.Source: §7 · p. 9
  • Future work should localize missing evidence without gold annotations and combine answer-free sufficiency estimates with reader-aware judges.Source: §7 · p. 9

AI USE STATEMENT

The paper reports AI assistance and author responsibility, states the scope of its clinical evidence, and points readers to its notation, appendix materials, and related work.

AI USE STATEMENT

  • Generative AI tools assisted with language editing, literature discovery, mathematical exposition, and source-code preparation; the authors verify the proofs, citations, experimental records, and final claims.Source: AI USE STATEMENT · p. 9
  • The clinical component uses de-identified retrospective data and supports no claim about clinical deployment, improved patient outcomes, or clinician-level performance.Corrupted-memory states are constructed artifacts rather than observed clinical failures.Source: ETHICS STATEMENT · p. 10
  • The appendix provides proofs, dataset construction and exclusion counts, training details, pre-registered criteria, per-seed metrics, checksums, and run-verification code; MIMIC-IV requires credentialed access.Source: REPRODUCIBILITY STATEMENT · p. 10

A OVERVIEW AND NOTATION

B EXTENDED RELATED WORK

  • Related work covers retrieval gates based on uncertainty, hidden-state probes, or non-generator features, alongside selective prediction and abstention methods.It also discusses retrieval QA, conflicting evidence, and detection of unanswerable questions.Source: B EXTENDED RELATED WORK · p. 14

C PROOFS

The proofs establish when size-only scores are uninformative under matched construction and why coarse integrity gates can be suboptimal. They also bound the residual size-only deviation and illustrate its scale in MuSiQue.

C PROOFS

  • Equal positive counts of complete and unsafe examples for every base query make both distributions uniform and yield AUROC = 1/2 exactly.Within a base query, a size-only estimator assigns the same score to every state.Source: C PROOFS · pp. 14–15
  • The deviation bound must depend on the unsafe and complete example distributions, not merely on the fraction of exceptional queries.A single query with many unsafe variants can dominate the unsafe-example distribution.Source: C PROOFS · p. 15
  • For MuSiQue, the construction-based AUROC deviation bound is 0.0046, while the observed test deviation is 0.0007.Only comparisons between the two query groups can deviate from one half.Source: C PROOFS · p. 15
  • At any fixed coverage, the minimum selective risk comes from accepting examples with the lowest probability of reader error.A binary integrity gate is strictly worse at some coverage unless reader error is conditionally independent of the query and memory given that gate.Source: C PROOFS · p. 15

D.1 DATASETS AND EXCLUSIONS

The paper evaluates three multi-hop QA datasets under reproducible preprocessing and preregistered criteria. MemSafe outperforms tested baselines on the main datasets, while transfer tests expose limits on SQuAD 2.0 and in reader gating.

D.1 DATASETS AND EXCLUSIONS

  • Processed datasets use or exclude every raw record for recorded reasons: MuSiQue excludes 22,355 unanswerable records and 453 low-distractor questions; HotpotQA excludes 446 low-distractor questions; 2Wiki excludes 12,576 test questions without supporting facts.HotpotQA uses 97,852 distractor-setting training and development records in total; MuSiQue reuses its unanswerable records as family B.Source: §D.1 · pp. 15–16, Table 4 · p. 16

D.2 TRAINING AND EVALUATION

  • The Stage B configuration is selected by MuSiQue validation AUROC from a prespecified eight-point grid, with macro-F1 as the tie-break.All eight configurations fall between 0.963 and 0.968 validation AUROC; reported bootstrap intervals, rather than corrected Wilcoxon p-values, are treated as inferential.Source: §D.2 · p. 16

D.3 PRE-REGISTERED CRITERIA AND OUTCOMES

  • H1 and H2 hold, H3 is refuted, and H4 holds on MuSiQue but is untested on 2WikiMultiHopQA.H5 is partially refuted: LEXSET ranks better at the smallest clinical scale, while MemSafe ranks better at the two larger scales.Source: §D.3 · pp. 16–17
  • The preregistered reader-gating criteria (i) and (ii) were not met as stated; the study instead reports fixed-coverage results after observing the high base error rate.Criterion (iii), lower answer-correctness AURC than reader confidence, holds.Source: §D.3 · p. 17

E.1 STRONGER BASELINES AND EXTERNAL TRANSFER

  • 0.7832 zero-shot and 0.7825 four-shot unsafe AUROC: the tested 7B LLM judge trails MemSafe’s 0.9719 on the same subset.The comparison concerns the tested prompts and checkpoint, not LLM judges in general.Source: §E.1 · p. 17
  • 0.6393 AUROC: frozen MemSafe retains positive discrimination on SQuAD 2.0, but MuSiQue-selected thresholds fail to meet the target evidence risks.At 5% and 10% risk budgets, realized risks are 0.2759 and 0.2900.Source: §E.1 · p. 17, Figure 5 · p. 18
  • 0.9019 MuSiQue and 0.9559 HotpotQA unsafe AUROC: the 8k joint encoder remains below MemSafe and matched mean pooling on both datasets.Its mean 5% risk coverage is 0.0028 on MuSiQue and 0.1206 on HotpotQA, versus MemSafe’s 0.2479 and 0.2922.Source: §E.1 · p. 17, Figure 4 · p. 18

E.2 COMPONENT ABLATIONS

Ablations show that the pretrained cross-encoder contributes substantially, while fine-tuning and set-level modeling have overlapping gains. Silver unit supervision preserves detection better than localization, and memorization audits find differing effects on lexical baselines and MemSafe.

E.2 COMPONENT ABLATIONS

  • 0.7407 unsafe AUROC: a frozen pretrained cross-encoder with a logistic head closes 41% of the gap between LEXSET and the full model.The full model reaches 0.9674, versus 0.5820 for LEXSET; the ratio describes the attribution ladder and does not allocate causal credit among components.Source: §E.2 · p. 18, Figure 6 · p. 19
  • Fine-tuning and the set model are not additive: each component’s marginal AUROC gain is three to four times larger when the other is absent.The comparison is descriptive, not a matched factorial interaction estimate; bridge-conditioned encoding is the second-largest single-component ablation at −0.0251 on MuSiQue.Source: §E.2 · pp. 18–19, Figure 7 · p. 19
  • Removing per-unit evidence and missing-count heads barely changes detection, although the evidence head provides per-unit localization with 0.9123 AUPRC.The changes are +0.0002 and −0.0014 across the two detection metrics.Source: §E.2 · p. 19

E.3 SILVER SUPERVISION

  • Silver-supervised MemSafe reaches 0.9444 ± 0.0021 test unsafe AUROC, three points below gold supervision but 0.36 above LEXSET.Gold unit targets remain important for per-unit localization: evidence AUPRC falls from 0.9123 to 0.4252 with silver targets, whose training-label precision and recall are 0.653 and 0.365.Source: §E.3 · pp. 19–20

E.4 CONSTRUCTION TRANSFER AND MEMORIZATION

  • When gold evidence is absent from training, MemSafe changes by +0.019, −0.003, and −0.000 across the three datasets, while lexical baselines lose accuracy.Shared paragraphs occur across splits, but restricting evaluation changes the question population and leaves pretraining overlap untested.Source: §E.4 · p. 20, Figure 8 · p. 20

E.5 SELECTIVE PREDICTION AND ERRORS

Selective-prediction results depend on evidence-risk prevalence and the task’s error structure. MemSafe can improve reader gating at some coverage levels, but clinical performance depends on training scale, and the reported diagnostics include localization and calibration measures.

E.5 SELECTIVE PREDICTION AND ERRORS

  • Neither estimator reaches useful 5% risk coverage on MIMIC-IV outpatient measurement records: LEXSET reaches 3.1% and MemSafe 0.2%.Figure 9 plots test risk-coverage curves, with the dotted line marking the 5% evidence-aware risk budget.Source: §E.5 · p. 20, Figure 9 · p. 21
  • 0.3509 coverage and 0.3005 AURC: MemSafe reaches the prevalence limit on 2WikiMultiHopQA, so its small difference from mean pooling does not reflect ranking quality.At 5% evidence risk, the prevalence bound applies because two-thirds of variants are unsafe.Source: §E.5 · p. 20, Table 5 · p. 21
  • 0.1985, 0.1396, and 0.0103 test error: MemSafe’s errors on MuSiQue, HotpotQA, and 2WikiMultiHopQA concentrate between relation-lost and missing states.Errors peak when exactly one required evidence item remains; unsafe memories predicted complete make up 6.6%, 6.0%, and 0.4% of unsafe cases, respectively.Source: §E.5 · pp. 20–21

E.6 READER GATING

  • At 5% coverage, the 7B-reader error is 0.6306 with MemSafe gating versus 0.8503 when answering everything; the LLM judge is better at 10% coverage.The oracle integrity label is not an oracle for answer correctness; Figure 10 also reports that MemSafe gating beats both answering everything and the oracle label across tested corruption prevalences.Source: §E.6 · pp. 21–22, Table 6 · p. 22, Figure 10 · p. 22

E.7 CLINICAL RESULTS

  • At the largest clinical scale, MemSafe reaches 0.9908 ± 0.0002 unsafe AUROC on laboratory events and 0.9960 ± 0.0003 on scale-matched outpatient records.LEXSET is preferable for small cohorts; the learning-curve crossover narrows to between 2,788 and 6,844 training variants, with MemSafe winning all three seeds at the latter scale.Source: §E.7 · p. 23, Table 7 · p. 23, Table 8 · p. 23

E.8 ADDITIONAL DIAGNOSTICS

  • Against mean pooling, MemSafe improves integrity macro F1 by +0.0295, +0.0176, and +0.0014 on MuSiQue, HotpotQA, and 2WikiMultiHopQA.Its per-unit evidence AUPRC and missing-count MAE also outperform mean pooling on all three datasets; MemSafe’s ECE is 0.0128, 0.0065, and 0.0013.Source: §E.8 · p. 23

Related papers

Loading 2609.32269v1…