MemoReason: Evaluating the Effect of Parametric Memory on Contextual Reasoning in LLMs

• The study distinguishes familiarity-driven memorization bias from the Strong Parametric Shortcut Hypothesis, in which models recall answers instead of completing necessary reasoning steps.It asks whether performance degrades when new information resembles training data textually but has a different meaning.Source: §1 · pp. 1–2
• Prior CofCa comparisons used distinct evidence-question pairs for factual and fictitious examples, so systematic error could explain the reported performance gap (Wu et al., 2024a).Source: §1 · p. 2
• MemoReason pairs factual reasoning tasks with structurally identical fictitious counterparts, replacing familiar entities under rule constraints to enable controlled comparisons.The benchmark is human-curated and designed to test memorization bias and the Strong Parametric Shortcut Hypothesis.Source: §1 · p. 2
• LLMs show significant performance drops on reasoning questions in fictitious versus factual settings, but direct factual recall is not the dominant memorization bias.Source: §1 · p. 2

2 BENCHMARK CONSTRUCTION

• The benchmark separates familiarity from reasoning by preserving text structure while abstracting content and generating fictitious variants of factual tasks.The construction is intended to isolate parametric memory without changing the underlying reasoning.Source: §2 · p. 2

2.1 FACTUAL DATASET

• Fictitious counterparts replace real entities and values while retaining the original task pattern, as illustrated by the Apollo-to-Stellara example.The example changes names, places, dates, and counts while retaining corresponding questions and reasoning operations.Source: §2 · p. 2, §2.1 · p. 3
• The factual dataset pairs Wikipedia article introduction excerpts with questions and answers based on each excerpt.It is represented as D = {(di, qi, ai)}N.Source: §2.1 · p. 2

Rules

• MemoReason contains 100 entity-rich Wikipedia template documents with 12 question-answer tasks each, yielding 1,200 factual tasks paired with fictitious counterparts across nine themes.The excerpts cover famous topics that the authors expect many LLMs encountered during training. Questions span extractive, arithmetic, temporal, and non-arithmetic inference tasks.Source: Rules · pp. 3–4

2.2 TEMPLATE ANNOTATION

• Templates annotate entities with type and attribute using a 14-type taxonomy, then apply manually encoded rules to preserve the document’s logic and core semantics during replacement.Rules constrain relationships such as sums, preserve numerical and temporal ordering, and restrict replacement values to plausible intervals.Source: §2.2 · pp. 4–5
• AI-drafted annotations were reviewed by 15 human annotators, with nearly 50% of question-answer pairs requiring human intervention across an estimated 200 hours of annotation.The annotators independently checked and modified drafts, then met to reach agreement.Source: §2.2 · p. 5

2.3 FICTITIOUS DATASET

• Each template yields 10 fictitious variants, generated by sampling coherent, type-compatible named entities and rule-constrained numerical and temporal values.Named-entity candidates are checked for non-existence; numerical and temporal values are sampled with a Mixed Integer Linear Programming solver, with dates and months manually verified.Source: §2.3 · p. 5
• Answer classes distinguish variant answers that change with substituted entities, invariant answers that remain unchanged on fully fictitious documents, and refusal questions testing whether models stay grounded.The dataset contains 12,000 fictitious question-answer pairs across the 10 variants per document-template task.Source: §2.3 · p. 5

• Performance variation is measured by comparing factual-example performance with the average across K corresponding fictitious variants, with confidence intervals bootstrapped over complete documents.Source: §3 · p. 6
• Predictions are scored first by Exact Match, then incorrect predictions are checked by LLM Judge Match, calibrated on 200 examples with 100% human-machine agreement.The calibration supports Judge Match’s reliability in capturing semantically equivalent answers missed by Exact Match and identifying incorrect predictions.Source: §3 · p. 6
• Evaluation spans nine recent high-performing reasoning models across multiple providers, combining open-weight models and a frontier API model to compare biases across families and scales.Source: §3 · p. 6

4.1 THE IMPACT OF ENTITY REPLACEMENT ON LLMS’ REASONING PERFORMANCE

• All 9 models show statistically significant reasoning drops on fictitious variant-answer tasks; inference-class questions drive the decline.Inference drops are in the 10% range; temporal drops are significant for 5 models (4.4%–15.6%), and arithmetic drops for 6 (4.7%–7.2%). Invariant-answer changes range from a 6.0-point decrease to a 1.5-point increase, with significant decreases for 3 of 9 models; refusals show a nonsignificant inverse-bias hint, extractive variant-answer performance falls for all 9 models (significantly for 6), and paired responses favor the factual setting.Source: §4.1 · pp. 6–7, Table 2 · p. 7

4.2 VARYING THE PROPORTION OF REPLACED ENTITIES

• Accuracy declines as entity replacement rises toward 50%, then partially recovers toward full replacement, as partial replacement adds conflict between parametric knowledge and contextual assertions.The experiment compares the factual baseline (0% replacement) with full fictitious replacement (100%) and intermediate replacement proportions.Source: Figure 2 · p. 8, §4.2 · p. 8

4.3 CHAIN-OF-THOUGHT EFFECT

• 15.00 and 12.62 percentage-point CoT gains on factual and fictitious reasoning, respectively, came with a 2.38% larger factual–fictitious performance drop.For GPT-OSS models, factual/fictitious accuracy rises 3.50/2.25 points from low to medium effort, then falls 1.58/1.59 points at high effort; the gap remains significant at each setting and is not removed by additional effort.Source: §4.3 · p. 8, Table 3 · p. 8

4.4 STRONG PARAMETRIC SHORTCUT HYPOTHESIS

• The Shortcut Rate is only 1.0%–3.2% for reasoning failures on fictitious variants and 0.0%–1.0% for extractive failures, so direct factual-answer copying is rare.The rate measures incorrect predictions matching the corresponding factual answer among failed fictitious examples; the results do not identify the mechanisms behind errors.Source: Table 4 · p. 9, §4.4 · p. 9

4.5 EVALUATION CONTRACT

• Claude Sonnet 4.6 falls from 95.67% factual accuracy to 92.30% fictitious accuracy, a statistically significant 3.37-point gap despite its high factual score.The evaluation pairs fictitious accuracy with the factual–fictitious gap, error ratio, and Shortcut Rate; for example, Llama-3.1-8B-Instruct has a 1.37-point gap but ranks seventh in fictitious accuracy.Source: Table 5 · p. 9, §4.5 · pp. 9–10

5 CONCLUSION

• Accuracy drops by up to 15.7% in the fictitious setting, and failures rarely reproduce factual answers directly.The reported drops are statistically significant; the underlying mechanisms remain an open question.Source: §5 · p. 10
• MemoReason pairs factual tasks with structurally identical fictitious counterparts, preserving templates, answer expressions, replacement constraints, and specified reasoning operations.This design isolates familiar versus unfamiliar entity anchors rather than testing robustness to arbitrary problem variation.Source: §5 · p. 10, §A.1 · p. 14
• Regenerable templates enable paired comparisons and targeted entity, numerical, or temporal interventions; fresh fictitious instances help mitigate leakage.The annotation interface and generation and evaluation pipeline make experiments inspectable, reproducible, and extensible.Source: §5 · p. 10

AI USE STATEMENT

• Generative AI assisted benchmark pre-annotation, fictitious entity generation, and some software development, with generated outputs reviewed and corrected by human annotators.The paper points to Section 2.2 for details on human review.Source: AI USE STATEMENT · p. 10

REPRODUCIBILITY STATEMENT

• The released pipeline preserves raw outputs, scores, confidence intervals, manifests, and dataset-binding checksums, with statistical details specified in Section 3.The statement also identifies where source selection, template annotation, fictitious-data generation, intervention settings, model and inference details, prompts, and calibration are documented.Source: REPRODUCIBILITY STATEMENT · p. 10

A.1 EVALUATING REASONING VIA INTERVENTIONS AND ROBUSTNESS

• Prior work probes reasoning brittleness under input changes, while MemoReason focuses on dependence of contextual reasoning on parametric familiarity.Studies perturb numerical values or constraints (Stolfo et al., 2023; Hong et al., 2025; Mirzadeh et al., 2025), test scaled numerical entities (Shrestha et al., 2025), or link arithmetic performance to training-data term frequencies (Razeghi et al., 2022).Source: §A.1 · p. 14

A.2 FICTITIOUS AND COUNTERFACTUAL BENCHMARKS

• Shared annotated templates hold document structure and question logic fixed, enabling the performance gap to be attributed to entity familiarity and supporting Shortcut Rate analysis.Unlike designs with different evidence-question pairs, MemoReason provides paired factual answers for testing direct recall on fictitious-setting failures.Source: §A.2 · pp. 14–15

A.3 KNOWLEDGE-SOURCE SELECTION AND CONFLICT MECHANISMS

• Mechanistic studies of knowledge-source selection complement MemoReason’s behavioral evaluation, whose paired templates test familiarity and knowledge conflict without attributing differences to a specific internal mechanism.Source: §A.3 · p. 15

B.1 ENTITY TYPES

• MemoReason uses 14 entity types and a medium-grained taxonomy to replace familiar anchors while preserving entity roles, document structure, answer expression, and reasoning path.Entity attributes keep coupled mentions consistent; the taxonomy balances semantically safe replacements against brittle, overly fine-grained categories.Source: §B.1 · p. 15

B.2 RULES

• Replacement rules preserve document-specific arithmetic, compatibility, plurality, offset, and value constraints, while the generation pipeline handles generic ordering constraints.Figure 3 depicts the taxonomy, rule layer, and question-answer contract as benchmark construction layers.Source: §B.2 · p. 15, Figure 3 · p. 16

B.3 ELIMINATING UNIQUE FACTUAL REFERENCES

• Uniquely identifying descriptions are kept literal when structurally necessary or softened to prevent factual cues from turning fictitious examples into contradiction tests.Broadening a description such as “the greatest sprinter” avoids forcing a model to reconcile a fictitious name with world knowledge.Source: §B.3 · p. 16

C JUDGE MATCH CALIBRATION

• Judge Match achieved 100% positive and negative recall on 200 annotated examples, with a 98.2% two-sided 95% lower bound for each recall.Corrected performance estimation accounts for judge bias; with both recalls equal to 1, the corrected estimate reduces to the raw success rate.Source: C JUDGE MATCH CALIBRATION · pp. 16–17

D ANNOTATION INTERFACE

• A unified interface supports document, rule, and question-answer annotation, with independent reviews, adjudication, and audit checks across the quality-control process.An audit found 15 problematic questions, approximately 1.4% of the then-current set; all were corrected and independently reviewed. Annotators check entity consistency, replacement constraints, answerability, refusal evidence, and answer expressions.Source: D ANNOTATION INTERFACE · p. 17, §D.1 · p. 17, §D.2 · p. 17

E.1 DATASET THEMES

• The factual dataset spans nine thematic categories, using the same taxonomy, replacement rules, question types, and answer types to diversify contexts for testing document-grounded reasoning and parametric shortcuts.Source: §E.1 · p. 17

E.2 RELEASED DATASETS

• The release includes ten dataset settings, with factual data as the reference and interventions paired through shared templates for factual–fictitious comparisons.Partial-replacement datasets also use the same templates as the factual and fully fictitious settings.Source: §E.2 · p. 18, Table 6 · p. 18
• Public-facing dataset aliases use the fictitious* prefix, while frozen experimental manifests retain their original keys for hash compatibility.Source: §E.2 · p. 18

F EXPERIMENTAL SETUP DETAILS

• Each question is evaluated independently from its document, and the model must return exactly one line in the form ANSWER: <answer>.The system prompt is shown in Appendix K.1.Source: F EXPERIMENTAL SETUP DETAILS · p. 18
• The evaluation does not request chain-of-thought; any hidden or tagged reasoning is stored separately, and only the final answer channel or ANSWER: line is parsed.Source: F EXPERIMENTAL SETUP DETAILS · p. 18

F.1 MODEL INFERENCE SETTINGS

• Main-experiment decoding is deterministic and greedy, with GPT-OSS API and local runs both using low-reasoning settings and no requested reasoning traces.Source: §F.1 · p. 18

F.2 HARDWARE

• Local open-weight model generation, including ablations, used approximately 200–250 GPU hours on NVIDIA H100 and A100 GPUs with 80 GB of VRAM.The estimate excludes provider-side compute for API-hosted models.Source: §F.2 · p. 19

F.3 MODEL LICENSES

• Open-weight models are used under provider licenses: Apache 2.0 for GPT-OSS (OpenAI et al., 2025), Qwen3.5, and OLMo-family models (Olmo et al., 2026), with Gemma and Llama under their respective license terms.API-only Claude systems are governed by Anthropic’s service terms (Anthropic, 2026b;a), and this work redistributes no API-model weights.Source: §F.3 · p. 19

G.1 ROBUSTNESS AND MECHANISM CHECKS

• The robustness analyses ask whether the gap depends on familiar names versus numerical or temporal information, answer changes after replacement, factual-answer retrieval, stochastic decoding, or training-data entity frequency.Confidence intervals resample complete source documents to preserve dependence among their questions and variants.Source: §G.1 · p. 19

G.1.1 WHICH SUBSTITUTIONS AFFECT PERFORMANCE WHEN THE ANSWER STAYS UNCHANGED?

• Names-only replacement significantly reduced OLMo-3-7B-Instruct accuracy by 3.78 percentage points, while values-only replacement reduced Qwen3.5-35B-A3B accuracy by 4.45 points.Neither isolated intervention produced a statistically detectable decrease for OLMo-3-7B-Think; the remaining confidence intervals include zero, which does not demonstrate an absence of effect. The comparison covered 400 invariant-answer questions across four question types.Source: §G.1.1 · pp. 19–20, Table 7 · p. 19

G.1.2 HOW DOES REPLACEMENT CHANGE INDIVIDUAL ANSWERS?

• The forward flip rate exceeded the mirror rate for all 9 models, with a positive difference whose 95% CI excluded zero for 7; two models’ differences were inconclusive.Rates are conditional on successes in their respective settings, so their difference is not the percentage of all questions lost; replacement could also turn failures into successes, and the flips do not identify a mechanism.Source: §G.1.2 · p. 20, Table 8 · p. 20

G.1.3 DOES RETAINING FAMILIAR NAMES INCREASE THE SHORTCUT RATE?

• With factual names retained and values replaced, reasoning Shortcut Rates ranged from 1.95% to 3.22% across 4 models, with a 5.00% maximum for Qwen3.5-35B-A3B on inference.The question-averaged rate is the fraction of failed variants reproducing the original factual answer, averaged across questions; questions with no failed variants contribute zero. Familiarity may influence intermediate reasoning without the final response reproducing that answer.Source: §G.1.3 · pp. 20–21, Table 9 · p. 21

G.1.4 IS THE FACTUAL ADVANTAGE SPECIFIC TO DETERMINISTIC DECODING?

• The factual–fictitious gap remained positive and significant in all 6 model–temperature combinations; changes from temperature 0 ranged from 0.03 to 0.29 percentage points, with all four CIs including zero.The test used OLMo-3-7B-Instruct and Qwen3.5-27B at temperatures 0, 0.5, and 1.0; nonzero temperatures averaged three generation seeds.Source: §G.1.4 · p. 21, Table 10 · p. 21

G.1.5 DO MORE FREQUENT NAMES PREDICT A LARGER FACTUAL ADVANTAGE?

• For OLMo-3-7B-Think, the association between named-entity frequency and the factual–fictitious accuracy gap was weakly negative (Pearson r = −0.103; 95% CI [−0.252, 0.028]), with no clear relationship.The analysis used mean entity occurrence counts as a proxy for training exposure across 100 documents; it does not rule out an influence of training exposure.Source: §G.1.5 · pp. 21–22, Figure 4 · p. 22

H ANNOTATION INTERFACE

• The authors developed a web annotation interface customized for MemoReason and made its code publicly available.Figure 5 shows the interface for the Toyota template, including entity spans, replacement rules, and question-answer fields.Source: H ANNOTATION INTERFACE · p. 23, Figure 5 · p. 23

H.1 PARTIAL REPLACEMENTS

• Claude Sonnet 4.6 accuracy falls from 95.67% in the factual setting to 91.88% at 80% replacement, then recovers by 0.42 points at the fully fictitious endpoint.Source: §H.1 · p. 24
• Partial replacements use deterministic accepted-answer rules alongside EM/JM evaluation, with reference answers derived from the evaluated documents.Source: §H.1 · p. 24
• Six of seven sweeps recover 0.42–0.87 points toward the fully fictitious endpoint, while OLMo-3-7B-Instruct changes by 0.07 points and plateaus.The sweeps range from a factual baseline with 0% replacement to a fully fictitious counterpart with 100%; Figure 6 shows 95% confidence intervals. These descriptive point-estimate recoveries alone establish neither statistical significance nor a causal mechanism.Source: Figure 6 · p. 24, §H.1 · p. 24

I.1 QUALITATIVE THINKING-CHECKPOINT CONTRAST OVER FICTITIOUS CONTEXTS

• The qualitative analysis presents chain-of-thought as helping fictitious-context reasoning.Source: §I.1 · p. 25
• The model first answered 9 years incorrectly, then used the document’s dates to calculate 11 years and answer correctly.It identified 1938 as the liberation year and 1949 as the council’s formation year, then subtracted the dates.Source: §I.1 · p. 25
• The initial response gave “ANSWER: 9” and was marked incorrect.Source: §I.1 · p. 25
• The chain-of-thought located the two event years and calculated their difference as 11 years.It reasoned from the 1938 liberation and 1949 formation dates before giving its final answer.Source: §I.1 · p. 25

J RESPONSIBLE RELEASE

• The release provides benchmark examples and evaluation code, not a trained model.Source: J RESPONSIBLE RELEASE · p. 25
• The release anonymizes publication identifiers, removes annotator names, and includes a canary field in dataset rows.Source: J RESPONSIBLE RELEASE · p. 25
• Explicit dataset splits distinguish factual, fictitious, partial-replacement, and temporal-perturbation settings.Source: J RESPONSIBLE RELEASE · p. 26
• The dataset is intended for diagnostic evaluation of contextual faithfulness; users should not present fictitious documents as real-world claims.Source: J RESPONSIBLE RELEASE · p. 26

K.1 EVALUATION SYSTEM PROMPT

• The evaluation prompt requires one answer line, forbids explanations, and specifies “Cannot be determined” when the document does not determine the answer.Otherwise, it requests only the shortest final answer.Source: §K.1 · p. 26

K.2 TAXONOMY SPECIFICATION PROMPT

• The taxonomy assigns consistent IDs and attributes to entity types, including people, places, events, and products.Product entries include a name attribute; the taxonomy also specifies person attributes such as full name.Source: §K.2 · pp. 26–28

K.3 ENTITY ANNOTATION PROMPT

• An entity-annotation prompt supplies AI-agent pre-annotations to assist human annotators.Source: Figure 7 · p. 29

K.4 RULE GENERATION PROMPT

• Fictionalization rules constrain replacements so documents retain semantic and linguistic coherence.The prompt asks for a minimal set of explicit constraints protecting facts stated or directly implied by the document.Source: §K.4 · pp. 29–30
• Rules should capture arithmetic relations, comparisons or bounds, and document-specific dependencies that could become incoherent after replacement.They must refer to annotated entities and attributes rather than fixing their original factual values.Source: §K.4 · pp. 30–31
• The prompt excludes rules that only preserve numeric, age, or temporal ordering, while requiring explicit exact offsets when the document implies them.It also calls for explicit century constraints when changing a century and an associated exact year or date independently would create a contradiction.Source: §K.4 · p. 30
• Rules must use entity references rather than factual values, allowing constants only for structural relations such as consecutive years.For example, a sum should be tied to its annotated result entity, not to that result’s original value.Source: §K.4 · pp. 30–31
• Rule generation targets explicit document constraints, including temporal or numeric intervals, identity-linked values, and number plurality.Source: §K.4 · p. 31
• The rule prompt rejects generic ordering, name-content, speculative, redundant, or automatically enforced constraints, returning no rules when only such candidates remain.Source: §K.4 · p. 31

K.5 FICTITIOUS ENTITY POOL GENERATION PROMPT

• The entity-pool prompt generates document-specific candidate pools of fictional named entities; later code samples the final fictional document from them.Source: §K.5 · p. 32
• Variant indices are coordinated across linked references so each index yields a coherent document version that satisfies relevant rules.Source: §K.5 · pp. 32–34
• Numbers, dates, temporals, pronouns, and related fields stay outside the entity pools and are generated later by seeded Python code.Source: §K.5 · pp. 32–33
• Pool entries must remain distinct across entity references and include only attributes required for each specific reference.Source: §K.5 · p. 34
• Output must be YAML only, limited to requested buckets and listed reference IDs, with no null values and counts matching valid variants provided.The prompt also requires relevant keys only and says organization variants should contain only name entries.Source: §K.5 · pp. 34, 36
• The prompt requests 15 fictional variants for each of two person entries and for countries, demonyms, and places.Person entries use full-name and first-name fields; place entries specify fictional countries and demonyms.Source: §K.5 · p. 35
• It requests 15 fictional names for events, military organizations, enterprise organizations, and NGOs.Event entries may also include an event type when needed.Source: §K.5 · p. 35
• Government, educational, and media organizations each receive 15 fictional name variants.Organization variants contain only a name field.Source: §K.5 · pp. 35–36
• The prompt requests 15 fictional names for awards and legal entities, with legal reference codes included if needed.Legal entries specify a name and an optional fictional legal code.Source: §K.5 · pp. 35–36
• It also requests 15 fictional product names.Source: §K.5 · p. 36