PaperlayerView exact revision on arXiv

Source-linked AI summary

Evaluating Compressed Memory: Dependence, Utility, and Content Benefit

Minsu Oh, Seunghui Jwa, Chanjun Park, Yeo-Chan Yoon

arXiv:2610.19373v1cs.CLcs.AI

TL;DR

The paper asks whether compressed memory improves prediction because it stores useful document information, rather than merely because a model relies on a learned input. It combines zeroing with separately trained alternatives and content controls; the comparisons show that reliance or gains over no document do not alone establish benefit from the question’s document.

  • Problem

    A score drop after zeroing or a gain over no document does not establish that stored document information improves prediction beyond a trained alternative or a learned input without document facts.

    Source: §1 · p. 1

  • Method

    The paper pairs post-training memory zeroing with separately trained no-memory and shared constant-input controls to distinguish reliance, utility, and document-content benefit.

    Source: §1 · p. 1, §7 · p. 8

  • Results

    Across nine systems, some models rely on memory yet perform worse than trained alternatives, while QA comparisons show that some gains also occur without the question’s document.

    Source: §1 · pp. 1–2, §8 · p. 8

  • Takeaways & Limitations

    Evaluations should specify the comparison model, report absolute performance, and state generation and scoring conditions before attributing gains to document content.

    Source: §7 · p. 8, §8 · p. 8

    QASPER uses a 128-question development panel and ten training orders, providing limited optimization coverage.

    Source: Limitations · p. 9

Abstract

from arXiv · show

Does a language model answer better because its compressed memory contains useful document information? Replacing memory vectors with zeros can lower a trained model's score without showing that it outperforms a model trained without memory. We combine zeroing with separately trained models that cannot access stored text or receive the same constant input for every document. Across nine systems, some models rely on memory yet perform worse than the trained alternative. For example, in experiments that predict states in generated stories, zeroing memory worsens a model's prediction loss, yet a model trained without memory achieves lower loss than the original model with intact memory. On TriviaQA, giving xRAG memory from another question's passage raises F1, which measures word overlap with accepted answers, from 19.38 without a document to 24.78. Its effect on how often outputs contain an accepted answer is statistically uncertain. The question's retrieved passage scores higher still (29.88 F1). Thus a gain over no document alone does not establish that the gain requires the question's passage. Evaluations must therefore specify the comparison model, absolute performance, and generation and scoring conditions.

1 Introduction

Memory evaluations can confuse reliance on a trained input with a performance benefit over a trained alternative, or with a benefit from document content. Across the reported comparisons, zeroing damage and gains over no document do not by themselves establish that the stored content helps.

1 Introduction

  • The evaluation distinguishes reliance by zeroing after training, utility by comparison with a trained model without memory, and content benefit by comparing document vectors with shared constant input.The constant-input module cannot convey the current document’s facts, so this comparison tests whether gains require document content.Source: §1 · p. 1
  • Across nine systems, four generated-story predictors are harmed by zeroing memory yet perform worse than trained alternatives; these low-performing cases do not establish the disagreement for models that solve the task well.A separate controlled study also finds a positive memory advantage when both models worsen, because the no-memory model worsens more.Source: §1 · pp. 1–2
  • A constant-input module’s zeroed output lowers F1 by 5.73 points despite conveying no current-document facts, while document-vector training separately outperforms constant-input training.F1 measures word overlap with an accepted answer, accounting for missing and extra words.Source: §1 · pp. 1–2
  • Passages from other questions improve some xRAG scores over no document on three datasets, while the question’s retrieved passage performs better.Generation settings and scoring rules also change which conclusions the comparisons support.Source: §1 · p. 2
  • The paper’s central inference rule is to match evidence to the claim and report the comparison model and absolute performance alongside relative gains.The authors argue that these details clarify what a memory gain does and does not demonstrate.Source: §1 · p. 2

2 Related Work

The paper brings together controls for interpreting performance gains and trained comparisons used in compressed-memory research. It applies these distinctions to ask whether memory helps beyond no memory and whether gains require document content.

2.1 Interpreting Performance Gains

  • Zeroing after training tests reliance, while a separately trained no-memory model tests whether memory training improves prediction; constant-input comparison asks whether document input helps beyond a learned input without document-specific information.This approach connects compressed-memory evaluation to comparisons of altered evaluation contexts and trained controls for learned representations.Source: §2.1 · pp. 2–3
  • Existing controls include retraining after feature removal, learned prompts, constant replacements, and activation patching, whose conclusions can depend on the input alteration and scoring metric.The paper applies the same distinction between zeroing and replacement with another document’s vectors to memory evaluation.Source: §2.1 · p. 3

2.2 Evaluating Compressed Memory

  • Prior compressed-memory evaluations use trained alternatives, including fine-tuned baselines for Auto-Compressors and full- or hidden-instruction comparisons for Gist.The study tests whether zeroing damage agrees with the advantage over a trained model without memory across several memory designs.Source: §2.2 · p. 3
  • The QA studies distinguish gains from additional training from gains requiring document content, including comparisons of document-vector and constant-input training.The decoder study keeps document memory fixed and uses equal training budgets for correct-document, other-document, or no-document training; both document-trained models receive the same correct-document memory at evaluation.Source: §2.2 · p. 3
  • Other-question passages, as well as constant-input training, show why a gain over no document or a positive normalized score alone cannot establish that document-specific information helps.The paper also compares retrieved, other-question, and no-document inputs under different generation settings and scoring rules.Source: §2.2 · pp. 3–4

3 Comparisons for Memory Evaluation

The paper separates reliance, utility, and content benefit through distinct comparisons. It also specifies how model interfaces, interventions, absolute scores, and uncertainty affect what those comparisons establish.

3.1 Three Questions, Three Comparisons

  • Zeroing memory after training tests dependence, whereas a model trained without the tested memory tests utility; a constant-input model tests whether document-vector training adds benefit beyond a learned input without current-document facts.For higher-is-better scores, D = Sintact − Szero, U = Sintact − Sr, and C = Sintact − Sconstant; positive values favor intact memory over the named reference.Source: §3.1 · p. 4
  • For NLL, lower is better; for F1 or accuracy, higher is better, and the score differences use those respective directions.NLL measures how little probability a model assigns to the correct next token and is reported in nats.Source: §3.1 · p. 4

3.2 What Is Held Fixed

  • PlanGen compares memory vectors with a parameter-matched local adapter that sees only recent text; xRAG projector studies preserve architecture while varying access to document content, visible text, or constant input.The xRAG encoder and LLM are frozen, but the projector’s information access and inference cost differ across conditions.Source: §3.2 · p. 4
  • With a shared zero-input predictor, the performance lost by zeroing document memory includes the benefit of a learned constant input, so that loss is not another test of document use.In PlanGen, separately trained parameters remain after zeroing, so predictors can differ; D − U compares the trained local model with the zeroed model without isolating causal mechanisms.Source: §3.2 · p. 4

3.3 Content Checks and Uncertainty

  • Memory swaps replace one document’s vectors with another’s, while evidence masking hides answer-supporting text and sham masking hides equally long text at a similar position outside designated evidence.The study reports absolute scores beside named-comparison advantages and uses paired bootstrap intervals; intervals containing zero do not establish an absent effect.Source: §3.3 · pp. 4–5

4 Dependence and Utility

Across nine systems, zeroing damage does not consistently identify a trained performance advantage; synthetic boundary cases also show that relative memory gains can coexist with poor prediction and uncertain benefit over constant input.

4 Dependence and Utility

  • Four systems harmed by zeroing on generated-story prediction perform worse than their trained baselines, and these comparisons concern specific procedures rather than architecture rankings.The low-performing cases do not establish the same disagreement in models that solve the task well.Source: §4 · p. 5, Table 1 · p. 6
  • H2O and xRAG are more sensitive to hiding answer-supporting than matched non-evidence text, yet both have higher NLL than their trained baselines.H2O: +0.405 [0.353, 0.460] nats and 2.068 versus 1.849; xRAG: +1.268 [1.216, 1.321] nats and 3.211 versus 2.279.Source: §4 · p. 5

4.1 Synthetic Boundary Cases

  • Across six PlanGen base models, all pooled-memory configurations have D > 0 and U < 0, but accuracy is 14.78% versus 18.55% for training-frequency prediction.For Llama-3.2-3B, D = 0.4643 and U = −0.1417 nats; predictions remain unreliable.Source: §4.1 · p. 5
  • Under target-only training, pooled memory beats the adapter by +0.7808 [0.3611, 1.2187] nats, while its advantage over constant input is unresolved at −0.0354 [−0.1418, 0.0590].Constant input also beats the adapter by +0.8162 [0.3619, 1.2961] nats; all three models have higher held-out answer-token NLL under target-only than all-token training.Source: §4.1 · p. 5, Table 2 · p. 6
  • All target-only losses exceed the 1.8871 NLL uniform-candidate predictor, and 43/60 models reach the 1,024-update cap without convergence.The positive relative utility therefore does not establish reliable state prediction or superiority to constant input.Source: §4.1 · p. 6

5 What a QA Score Gain Establishes

Trained controls and alternate-passage tests distinguish document-content benefits from gains that also occur without the question’s retrieved passage. The conclusions vary with dataset, training history, and scoring rule.

5.1 Trained Controls on QASPER

  • On QASPER, document-trained projectors score 19.98 F1 versus 12.54 for constant-input training, while zeroing either projector yields 6.82.The document-versus-constant advantage is +7.43 [6.15, 8.75] points; zeroing damage includes a 5.73-point drop also seen without current-document facts.Source: §5.1 · p. 6, Table 3 · p. 6
  • On CoQA, correct-document training gives 66.56 F1 versus 34.22 for other-document training and 18.86 without documents, supporting content benefit within that dataset.Upstream xRAG training includes CoQA, so this does not establish unseen-task transfer.Source: §5.1 · pp. 6–7

5.2 Gains That Do Not Require the Retrieved Passage

  • On TriviaQA, F1 is 19.38 without a document, 24.78 with donor passages, and 29.88 with the retrieved passage.The donor gain over no document is 5.40 points, while the retrieved-passage gain over donors is a further 5.10 points.Source: §5.2 · p. 7, Table 4 · p. 7
  • On TriviaQA, donor passages improve EM by +3.80 [3.22, 4.42] points, but their Match difference is uncertain at −0.57 [−1.50, 0.37].The retrieved passage beats donors by +9.28 [8.26, 10.31] Match points; higher overlap or exact match need not mean more outputs contain an accepted answer.Source: §5.2 · p. 7
  • Donor F1 gains are positive on TriviaQA, NQ-open, and WebQuestions, while retrieved passages beat donors on Match throughout; donor Match and EM results vary across datasets.The NQ-open and WebQuestions comparisons use a separate twelve-comparison adjustment.Source: §5.2 · p. 7, Table 5 · p. 8
  • With QASPER questions, memory, prompts, and generation held fixed, document training widens the correct-versus-other-paper F1 gap from 3.31 to 14.14 points.The increase is 10.83 [9.07, 12.59]; training improves answers with correct memory while reducing performance with other-paper memory.Source: §5.2 · p. 7

5 What a QA Score Gain Establishes

  • QASPER uses annotated evidence, whereas the released-model study uses retrieved passages, and their answer-generation procedures differ; the studies do not isolate each difference’s contribution.These differences, together with scoring, can affect measured gains.Source: §5.2 · p. 7

6 Supporting Lexical Checks

Lexical probes provide a positive document-content result on repeated words but a harder substitution test does not establish reliable factual recovery.

6 Supporting Lexical Checks

  • On PG-19 repeated-word targets, document input yields 4.5088 NLL versus 4.8145 for constant input and 4.8254 for suffix input.Its accuracy advantage over constant input is +2.13 [1.15, 3.09] points, with a crossed simultaneous lower bound of 0.85 points.Source: §6 · p. 8
  • Zeroing the constant input raises NLL by +1.2360 nats, although its advantage over suffix input is uncertain at +0.0109 [−0.0042, 0.0259].This separates reliance on a learned input from advantage over another trained input.Source: §6 · p. 8
  • For NASA word substitutions, document-input accuracy falls from 47.56% to 0.23%, and the probes do not establish factual recovery.The original-input NLL contrast over constant input is +0.2098 nats, but its interval [−0.0058, 0.4424] includes zero.Source: §6 · p. 8

7 Discussion

The discussion argues that memory claims require comparisons matched to the claim, because observed score changes alone do not identify benefits from current-document content.

7 Discussion

  • Relative gains can grow while both models worsen, zeroing a learned constant input can hurt, and other questions’ passages can raise QA scores.Together, these cases show why a score change alone cannot identify the benefit of the current document’s content.Source: §7 · p. 8
  • Zeroing tests reliance after training, a separately trained no-memory model tests performance benefit, and replacement controls test whether gains require document-dependent input or the question’s passage.The positive QA results support document benefit, but not every observed gain demonstrates it.Source: §7 · p. 8

8 Conclusion

Memory can be relied on without outperforming a trained alternative, and gains over no document may also occur with another passage. Evaluations should specify the comparison model, absolute performance, and generation and scoring conditions.

8 Conclusion

  • Memory can be relied on even when a model trained without it performs better; relative gains can also grow while both models’ prediction losses worsen.QA comparisons show document content helps, but some gains over zeroed memory or no document also occur without the question’s document.Source: §8 · p. 8
  • Within QASPER, decoder training increases sensitivity to correct versus other-document memory, while released-model QA conclusions about other-passage gains change with generation and scoring choices.Source: §8 · p. 8
  • Before attributing gains to document content, evaluations should name the comparison model, report absolute performance, and specify how answers are generated and scored.Source: §8 · p. 8

Limitations

The reported benefits are conditional on the comparison model, task, and training procedure, and the experiments do not establish architecture-wide optima, convergence, or matched total costs.

Limitations

  • The study does not identify an architecture’s best performance, establish convergence, or match total pretraining and inference costs.Optimization differences may matter, but the results do not isolate them as a causal explanation.Source: Limitations · p. 9
  • Generated stories permit copying repeated values, and predicting one distinguishing token with preceding gold text is easier to constrain than generating an entire state value.Restricted attention deliberately hides earlier text and does not represent every deployment setting.Source: Limitations · p. 9
  • PG-19 repeated-word prediction tests lexical prediction rather than annotated facts, while QASPER’s annotated evidence results do not evaluate retrieval or show passages are uniquely necessary.Source: Limitations · p. 9
  • The studies use different tokenizers, memory modules, and multiple-comparison adjustments; raw NLL is not ranked across tokenizers.Local holdouts do not exclude foundation-model pretraining exposure, and internal validation does not remove it.Source: Limitations · p. 9
  • Intervals over items with trained models fixed omit training uncertainty; crossed intervals cover order variation at shared initialization, not arbitrary initializations or full configuration search.Finite bootstrap samples limit tail precision, and an interval containing zero is not proof of no effect.Source: Limitations · p. 9
  • The NASA task’s automatic word substitutions lack human semantic validation, and original versus substituted examples differ in difficulty.The document-versus-constant difference describes trained models, not a unique causal fraction of utility; released-model TriviaQA also uses one frozen checkpoint and fixed donor assignments, with some descriptions post hoc.Source: Limitations · p. 9

Ethics Statement

The paper uses synthetic templates and public texts and datasets, without recruiting participants or commissioning new human annotation. AI tools assisted research and analysis, and the authors report checking outputs and retaining responsibility.

Ethics Statement

  • The work uses synthetic templates and public texts and datasets, including TriviaQA, PG-19, QASPER, and NASA articles.No participants were recruited or new human annotation commissioned; upstream terms apply, and public access does not imply unrestricted redistribution.Source: Ethics Statement · p. 9
  • The numerical supplement preserves comparison families and settings, including results that qualify the claims, and distinguishes author-generated evidence from third-party checkpoints and corpora.Source: Ethics Statement · p. 9

AI Assistance

  • Generative AI assisted literature search, study design, implementation, experiments, analysis, writing, and figure preparation; the authors report checking key inputs, outputs, and rendered PDFs.The authors remain responsible for the research and presentation.Source: AI Assistance · pp. 9–10
  • The literature audit’s agreement statistics describe AI-assisted coding passes, not agreement between independent human annotators.Source: AI Assistance · p. 10

A Controlled Probe: Implementation and Data

The controlled probe tests how compressed memory and trained alternatives perform on story-state prediction, with interventions that distinguish dependence from utility. A separate objective and constant-input study tests whether document content accounts for observed advantages.

A Controlled Probe: Implementation and Data

  • The probe summarizes completed 16-token blocks into delayed memory slots, with a two-block delay preventing tokens from reading summaries of their own unfinished block.Local attention is limited to recent tokens, and the memory bank resets within each evaluation window.Source: A Controlled Probe: Implementation and… · p. 13
  • Pooled reading averages available projected slots and adds their gated contribution to the local hidden state; indexed reading selects the current slot.Scored targets exclude recurrences within 384 tokens, beyond both local and indexed reach.Source: A Controlled Probe: Implementation and… · p. 13
  • Four conditions share backbone, tokenizer, data, objective, and LoRA configuration, while the local adapter cannot access memory and matches the memory module’s parameter count within 0.06%.Stories provide preceding gold text, and evaluation scores the first token distinguishing the true value from alternatives on the same targets within each backbone.Source: A Controlled Probe: Implementation and… · p. 13, Table 6 · p. 13
  • All six pooled-memory configurations have U < 0 and D > 0, indicating worse utility than the parameter-matched adapter despite damage from zeroing intact memory.Candidate accuracy does not establish state recovery; the table reports state-token NLL and candidate accuracy with marginal document-bootstrap intervals.Source: Table 7 · p. 14
  • The study’s candidate-token accuracy does not establish reliable story-state recovery; its uniform-candidate benchmarks already know each item’s candidate set.NLL uses the full vocabulary, unlike candidate-token accuracy.Source: A Controlled Probe: Implementation and… · p. 14
  • Zeroing only document-summary slots nearly reproduces the damage from zeroing all slots, while other listed interventions change NLL by amounts within ±0.01 nat.The interventions characterize inference inputs that matter to the trained model, not how training produced that dependence.Source: A Controlled Probe: Implementation and… · p. 14

B Earlier Objective and Constant-Input Controls

  • The constant-input path receives one shared vector across documents but remains trainable and retains local hidden states and sequence length, changing more than document information alone.The earlier-objective study pairs seeds and input order, while its separate ten-seed study uses equal development searches; its sign reversal should not be transferred to that later recipe.Source: B Earlier Objective and Constant-Input… · pp. 14–15
  • Under target-only training, the document-versus-constant-input NLL interval includes zero, while under all-token training the difference lies within ±0.01 nat; accuracy contrasts also include zero.These contrasts do not establish that content contributes negligibly or assign a causal contribution to either changed model.Source: B Earlier Objective and Constant-Input… · p. 15

C Gold-Answer QA: Data, Fitting and Contrasts

These studies compare document inputs with trained controls and interventions, separating memory reliance from benefits attributable to document content. Results vary by task and scoring rule: document-trained systems can improve on trained controls, but not every apparent gain is document-specific or robust across metrics.

C Gold-Answer QA: Data, Fitting and Contrasts

  • QASPER supplies annotated evidence without retrieval errors, and the trained-input comparison varies evidence, question, or a constant while keeping question text and decoder input interface fixed.The constant projector can produce only one learned 4,096-dimensional vector, giving it less input-dependent capacity than the other projectors.Source: C Gold-Answer QA: Data, Fitting and… · pp. 15–16
  • +1.14 [0.40, 1.91] EM points: document input exceeds question input, while zeroing and swapping lower document-input EM by +4.29 [3.20, 5.43] and +2.85 [1.89, 3.86] points.The document-versus-constant-input EM difference is +1.16 [0.46, 1.91] points; the reported sensitivity bounds adjust across a separate four-effect family.Source: C Gold-Answer QA: Data, Fitting and… · p. 17

D LLM Adaptation at Fixed Storage

  • The fixed-storage decoder comparison preserves the stored vector bytes but trains decoder LoRA parameters; its interpretation includes a change in input distribution for the mean-trained decoder.The study states that decoder adaptation adds trainable parameters and training cost while preserving storage.Source: D LLM Adaptation at Fixed Storage · p. 17
  • +4.975 [3.933, 6.023] F1 points: with the same correct document vector, the document-trained decoder outperforms the mean-trained decoder.The crossed interval is [3.719, 6.230]; the EM difference is +0.848 [0.148, 1.563] points, but its crossed interval includes zero.Source: D LLM Adaptation at Fixed Storage · pp. 17–18, Table 11 · p. 18

E Released-Model QA Audit

  • On the released xRAG audit, question-vector input gains on Match over closed-book input, but the primary gain does not extend to strict EM.In an exploratory family, donor input exceeds closed-book EM by 10.958 [10.250, 11.666] points, and document input exceeds donor EM by 7.348 [6.635, 8.063].Source: E Released-Model QA Audit · pp. 18–19, Table 13 · p. 19
  • Batch size changes the released-model audit: batch four changes 9,905 answers and yields Match 56.96%, strict EM 1.29%, and F1 19.39, versus 51.59%, 30.43%, and 43.54 at batch one.The comparison does not isolate a single cause; the audit also cautions that its records and scores need not match the original paper’s main table.Source: E Released-Model QA Audit · pp. 18–20

F Lexical Content: Methods and Complete Primary Effects

  • The lexical probe uses paired repeated-word substitutions in NASA articles, but its hash-based replacement does not match word frequency, part of speech, or semantic plausibility.The authors note that recent publication alone does not establish absence from model pretraining.Source: F Lexical Content: Methods and… · p. 20
  • After replacing remote text and the target while keeping the visible suffix fixed, several accuracy contrasts include zero; full-context accuracy falls from 66.36% to 17.05%.This pairwise preference criterion differs from selecting the correct token from the full vocabulary.Source: F Lexical Content: Methods and… · p. 20

G Other Comparisons and Boundaries

Additional comparisons test normalized scores, memory controls, inference boundaries, and review findings. They show that positive gains over closed-book input can occur without document access, while the reviewed literature and within-paper tests support narrower conclusions than broad claims about controls or full information recovery.

G Other Comparisons and Boundaries

  • On MuSiQue, zeroing query memory improves NLL, with D = −0.0230 [−0.0388, −0.0079]; all four NLL utility intervals in the further study include zero.Synthetic pooled-memory zeroing damage is 0.0278 [0.0111, 0.0446].Source: G Other Comparisons and Boundaries · p. 21
  • Four synthetic configurations suffer zeroing damage yet underperform their trained local comparators, showing that memory sensitivity can coexist with negative utility.The systems differ in adapted components and input access, so their results do not isolate architectural causes.Source: G Other Comparisons and Boundaries · p. 22, Table 19 · p. 22
  • 25.87% for the trained constant and 58.24% for document input: both normalized F1 gains are fractions of the full-text-to-closed-book gap, not measures of recovered information.Across 722/9,790 seed–question pairs, document input beats the frozen closed-book model but loses to the trained constant.Source: G Other Comparisons and Boundaries · p. 23

H Inference, Scope and Reproduction

  • The intervals do not cover arbitrary initialization choices, configuration searches, or selection among analyses; repeated analysis of the same public test is not independent confirmation.Crossed intervals capture training-order variation at the shared initialization and recipe.Source: H Inference, Scope and Reproduction · pp. 23–24

I Review of Controls in Selected Memory Papers

  • Among 19 learned-compression papers, no clear trained independent-input control was located; among 21 KV papers, StreamingLLM provides a component-specific precedent.The review does not support claims that comparable controls are absent throughout the field or that any paper’s performance claims are invalid.Source: I Review of Controls in Selected… · pp. 24–25

J Within-Paper Evidence and Matched Decoder Controls

  • 19.9785 versus 4.7420 mean F1: intact versus same-paper-swapped evidence leaves a 15.2365-point gap, although unmatched answer-bearing text may remain in donor paragraphs.Same-paper donors match topic and encoder length, but do not isolate answer-bearing semantics from every formatting or content difference.Source: J Within-Paper Evidence and Matched… · p. 25
  • +7.4844 F1: correct-document training exceeds wrong-document training; correct-document training also exceeds closed-book training by +7.9442 F1.Both differences remain positive under the prespecified six-primary Bonferroni intervals, without establishing complete recovery of source text.Source: J Within-Paper Evidence and Matched… · pp. 25–26
  • Compressed-document F1 is 41.3318 points below full-text F1, so positive gains over other controls do not establish full recovery of available text’s predictive benefit.These are exploratory secondary comparisons, and their nominal intervals do not adjust for selection or multiplicity.Source: J Within-Paper Evidence and Matched… · p. 26

K Evaluation with Fixed Software Versions and Batch Size Four

The section compares fixed-runtime QA conditions and additional training and evaluation checks. Its results show that conclusions depend on the comparison model and evaluation setup.

K Evaluation with Fixed Software Versions and Batch Size Four

  • +3.5181 Match points for encoded questions and +8.7156 for encoded documents over closed-book input; donor EM/F1 improve, but donor Match includes zero.The 40.4% ratio is descriptive, not the fraction of document benefit caused by question information.Source: K Evaluation with Fixed Software… · p. 27

L Objective Comparison with Equal Development Searches

  • Target-only training raises adapter NLL by 0.9539 and pooled-memory NLL by 0.1730; pooled memory's relative edge reflects a worse comparator, not improved own NLL.The target-only adapter NLL is 2.5805, worse than uniform prediction over permitted candidates.Source: L Objective Comparison with Equal… · p. 28, Figure 2 · p. 28

M CoQA Evaluation without Retraining QASPER Models

  • The CoQA evaluation reuses fixed QASPER-trained models on 405 first-turn questions with short annotated evidence, not full-document or multi-turn QA.Upstream CoQA instruction training prevents interpreting this as transfer to an unseen task or freedom from pretraining contamination.Source: M CoQA Evaluation without Retraining… · p. 29

N Planned Follow-up Comparisons

  • Six follow-up contrasts were specified before their new outputs, with separate interval families and resampling schemes for fitted models or frozen-model passage clusters.The bounds do not cover analysis selection or arbitrary initialization; other evaluation families remain separate.Source: N Planned Follow-up Comparisons · pp. 29–30

O Training Configurations for the Nine-System Comparison

  • Nine-system local comparators differ in backbone, input, adapted components, and training recipe, so raw NLL cannot rank architectures across conditions.The four synthetic configurations that underperform their local comparators also have low absolute task performance and differing decoder adaptations.Source: O Training Configurations for the… · p. 30

P Expanded Learning-Rate Search for QA Projectors

  • 8 × 10^-4 for document input and 3.2 × 10^-3 for question and constant input are the best tested expanded-grid rates; larger rates worsen development NLL.These checks do not retrain the test models or establish equal optimization quality or a global optimum.Source: P Expanded Learning-Rate Search for QA… · p. 31

Q Fixed-Software Generation on NQ-open and WebQuestions

  • Donor input improves Match over closed-book input after adjustment on WebQuestions, but not on NQ-open; document input improves Match over donors on both datasets.Donor F1 gains are positive while strict EM gains are small or unresolved; output-length observations do not establish length as the cause.Source: Q Fixed-Software Generation on NQ-open… · p. 32

S Reconciling the QA Comparisons

The QA comparisons differ in what they change: training input, decoder training history, or inference-time document vectors. Their results are compatible, but cross-study differences do not isolate individual causes.

S Reconciling the QA Comparisons

  • The synthesis is post hoc, and cross-study differences cannot isolate dataset, retrieval quality, answer format, or runtime; CoQA exposure and short evidence preclude a matched-counterexample claim.QASPER uses annotated scientific-paper evidence, unlike retrieved-passage benchmarks with different answer distributions.Source: S Reconciling the QA Comparisons · pp. 33, 35
  • The projector comparison changes training input, whereas the decoder comparison changes training history; both document-trained decoders receive correct memory at evaluation.The 21.21 versus 13.73 F1 contrast compares decoder training histories, not an inference-time document intervention.Source: S Reconciling the QA Comparisons · pp. 33–34
  • Correct-memory F1 rises by 7.08 points while other-paper F1 falls by 3.74; decoder training changes sensitivity to memory without isolating semantic use.The correct-minus-other difference increases by 10.83 points on identical inputs.Source: S Reconciling the QA Comparisons · p. 34, Table 42 · p. 34
  • +4.2944 F1 points separate other-paper memory from no document in the descriptive QASPER synthesis, while correct memory scores higher still.The contrast uses separate saved runs, and the no-document prompt omits the background instruction, so it does not control document semantics alone.Source: S Reconciling the QA Comparisons · p. 34
  • On pinned-runtime TriviaQA, F1 is 19.38 without a document, 24.78 with donor memory, and 29.88 with the retrieved passage; donor Match remains uncertain.The retrieved passage exceeds donors on both F1 and Match, while donor F1 gains do not measure a fraction of information preserved.Source: S Reconciling the QA Comparisons · p. 35

Related papers

Loading 2610.19373v1…