Empty Shelves or Lost Keys? Recall Is the Bottleneck for Parametric Factuality

• Knowledge profiling shifts analysis from questions to facts, distinguishing whether each fact is encoded and whether it is directly recalled, unrecalled, or accessible only through thinking.Encoding and recall are operationalized behaviorally across varied contexts, phrasings, and relational orderings.Source: §1 · p. 2
• WikiProfile contains 2,150 natural-document facts with 10 questions each and evaluates encoding, recall, and recognition using an automated web-search-grounded construction pipeline.The benchmark supports realistic evaluation of indirectly expressed facts and complex, context-dependent relations across more than 4 million graded answers from 13 LLMs.Source: §1 · p. 2
• 95–98% of facts are encoded by frontier models, yet they still cannot recall 25–33% without thinking, making recall the primary factuality bottleneck.Scaling primarily improves encoding and does not necessarily improve recall.Source: §1 · p. 2
• Long-tail facts show only a small encoding disadvantage but often over 25% lower recall, while reverse questions impair generation despite no recognition disadvantage.These findings reframe long-tail errors and the reversal curse (Berglund et al., 2024; He et al., 2025; Kandpal et al., 2023; Mallen et al., 2023) as recall failures rather than missing knowledge.Source: §1 · p. 2
• Thinking recovers 40–65% of encoded-but-not-directly-known facts, with the largest gains on long-tail facts and reverse questions.These results motivate post-training and inference-time methods that improve utilization of existing knowledge.Source: §1 · p. 2

2. Knowledge Profiling

• Factual knowledge is evaluated without external tools through two behavioral notions: encoding and knowledge, enabling model-agnostic diagnosis of whether errors reflect storage or recall.The framework applies to both closed- and open-weight LLMs.Source: §2 · p. 3

2.1. Operationalizing Encoding and Knowledge

• Encoding requires reproducing a fact in a strongly primed, pre-training-like context, whereas knowledge requires correct answers across four varied direct and reverse questions, with and without thinking.Encoding excludes thinking to avoid conflating storage with inference; knowledge is evaluated both with and without thinking.Source: §2.1 · pp. 3–4

2.2. Knowledge Profiles

• The resulting five profiles distinguish encoding failure, recall failure, direct recall, recall with thinking, and inference without encoding, while excluding an exceptionally rare unencoded-but-directly-known case.Encoding failures suggest capacity or data-coverage limits; recall failures indicate inaccessible stored facts, whereas inference without encoding is less reliable and may promote hallucinations.Source: §2.2 · p. 4

3. The WikiProfile Benchmark

• WikiProfile evaluates encoding, knowledge, and fact verification using 2,150 facts, each paired with ten questions spanning proposition-completion, contextual, direct, reverse, and multiple-choice tasks.Wikipedia provides the factual source and ground truth; every question has a single unambiguous gold answer for automatic validation.Source: §3 · p. 4

knowledge profiling.

• The benchmark pipeline extracts Wikipedia facts, generates and refines direct and reverse questions, creates contextual, natural-phrasing, and multiple-choice variants, and filters questions using Google Search.Facts are discarded whenever an associated question is rejected, and the pipeline is fully automated using prompted LLMs.Source: §3 · p. 4, Figure 3 · p. 5, knowledge profiling · p. 5
• Fact extraction samples 10,000 Wikipedia pages and selects non-trivial, non-guessable, time-independent completions with single unambiguous answers, balancing categories and entity types before downsampling.The resulting candidates have a median context length of 100 words and a mean of 116 words (±56 std).Source: knowledge profiling · p. 5
• Strict automated filtering and manual validation reduce the benchmark to a final 2,150 facts, with fewer than 2% of facts discarded during manual review.The automated filtering stage initially leaves approximately 2,200 facts after rejecting any fact with an ambiguous or otherwise invalid question.Source: knowledge profiling · p. 5

LLMs

• 13 LLMs from five families are evaluated with and without thinking, using default thinking settings or zero effort/budget, and chain-of-thought prompting where applicable.The models include Gemini-3, Gemini-2.5, GPT-5.2, GPT-5, GPT-5-mini, GPT-4.1, GPT-4.1-mini, and Gemma3 variants.Source: §4 · p. 6

Response Generation

• Approximately 4.5 million responses are generated using eight samples per example, temperature 1, and prompts for completion, closed-book QA, and multiple-choice QA.Multiple-choice options are shuffled so each answer choice is correct exactly twice.Source: §4 · p. 6

Response Evaluation

• Question grades use only CORRECT and INCORRECT responses, excluding PARTIALLY and OTHER because their weighting is subjective and they occur in fewer than 5% of responses.Facts with only PARTIALLY or OTHER responses are excluded from aggregate statistics, and alternative handling strategies produce negligible differences.Source: §4 · p. 6
• Two Gemini-2.5-Pro autoraters with thinking assign CORRECT, INCORRECT, PARTIALLY, or OTHER labels, achieving 98.2% agreement with GPT-5-based grading.The remaining disagreements primarily involve PARTIALLY or OTHER labels excluded from analysis, indicating results are not sensitive to the underlying grader model.Source: §4 · p. 6

5.1. The Recall Bottleneck

• Frontier models encode 95–98% of facts, but fail to directly recall 26–34% and fail to recall 11–12% even with thinking, making recall the dominant bottleneck.Recall failures account for more than 70% of GPT-5.2’s errors and a larger share in stronger models.Source: §5.1 · pp. 6–7, §5 · p. 6
• Scaling primarily reduces encoding failures, while larger models increasingly retain errors caused by inaccessible encoded knowledge rather than absent knowledge.In Gemma3, increasing capacity from 1B to 27B parameters reduces encoding failures from 85% to 23%, while recall failures reach 33% with thinking and 40% without.Source: §5.1 · p. 7

5.2. Why Recall Fails

• Recall is harder for rare and reverse-query facts, indicating that utilization depends on whether questions match training-time popularity, contexts, phrasings, and directions.The reversal effect is a recall failure rather than absent bidirectional knowledge: models can recognize reverse answers in verification even when generation fails to produce them.Source: §5.2 · p. 7, Figure 5 · p. 7, Figure 6 · p. 7
• Reverse questions reduce recall for nearly all models in generation, but verification makes them no harder than direct questions: reverse performance is higher for 9 models and similar for 4.For GPT-5, recall falls from 82.9% on direct questions to 74% on reverse questions; phrasing itself shows no significant effect across 104 corrected hypothesis tests.Source: §5.2 · pp. 7–8, Figure 6 · p. 7

5.3. Thinking as a Recovery Mechanism

• Thinking preferentially recovers weakly recalled facts, improving rare-fact recall by 20.1 points for Gemini-3-Pro and reverse-question recall by 19 points for GPT-5.It narrows the popularity gap from Δ = 21.4 to 12.5 and the directionality gap from Δ=9 to 2; thinking recovers 40–65% of encoded versus 5–20% of non-encoded facts, supporting recall facilitation over response diversity or complex inference.Source: §5.3 · pp. 8–9, Figure 8 · p. 8

5.4. Connections to Human Cognition

• The encoding–recall gap parallels human tip-of-the-tongue and feeling-of-knowing phenomena, while the authors limit the claim to functional similarity rather than human-like cognition.Both comparisons describe stored or recognizable information that cannot be immediately generated but can be accessed through additional effort or presented alternatives.Source: §5.4 · p. 9

Encoding and Memorization

• The behavioral encoding criterion connects to memorization research, repurposing encoding-via-memorization for knowledge profiling rather than privacy or data-leakage analysis.It aligns specifically with the probabilistic framework of Hayes et al. (2025).Source: §6 · p. 9

Latent Knowledge

• Latent knowledge may reflect context-dependent recall: facts surface under training-matched encoding queries but become harder to access under divergent knowledge queries.This behavioral account complements internal-representation studies of latent or hidden knowledge, including Burns et al. (2023) and Kadavath et al. (2022).Source: §6 · p. 9

The Reversal Curse

• Comparing verification with generation shows that reverse answers can be recognized without being produced, indicating recall asymmetry rather than failure to learn bidirectional associations.Thinking can mitigate this reversal curse, complementing prior explanations based on autoregressive objectives, training dynamics, and data asymmetries.Source: §6 · p. 9

Thinking and Factuality

• Thinking functions as a recall mechanism, with its strongest effects on rare facts and reverse questions amid mixed prior evidence on factuality and consistency.This perspective complements Gekhman et al. (2026), which identifies a computational buffer and factual priming as mechanisms expanding parametric knowledge recall.Source: §6 · pp. 9–10

• Frontier LLMs encode over 95% of benchmark facts, making recall rather than encoding the primary factuality bottleneck, especially for rare facts and reverse questions.Recall failures appear sensitive to surface-level query characteristics because stored facts remain tied to their acquisition contexts.Source: §7 · p. 10
• Distinguishing encoding from recall failures could extend beyond short-form factuality to long-form generation, multi-step reasoning, and tool-using systems.Parametric knowledge remains important for fluency, speed, and integrating information across contexts, even with retrieval-augmented generation (RAG).Source: §7 · p. 10
• Pre-training data augmentation, post-training alignment, and inference-time thinking are proposed routes for improving access to already encoded knowledge.The discussion cites self-generated question-answer pairs Lin et al. (2025), alignment studies Gekhman et al. (2024); Lin et al. (2024a); Zhou et al. (2023), and thinking as recall interventions.Source: §7 · p. 10
• WikiProfile’s conclusions may not generalize beyond Wikipedia, and its knowledge definition covers only directionality and phrasing for single-hop facts.The study also does not exhaustively characterize all query forms, while full frontier-model profiling costs ±$500, reducible through fewer question types or samples.Source: §7 · p. 10

Impact Statement

• The findings could inform deployment decisions by identifying failures involving rare facts and reverse questions, as well as cases where inference-time computation may help.The authors report no negative societal consequences specific to this work.Source: Impact Statement · p. 10

A. WikiProfile – Full Details

• WikiProfile defines ten tasks for each fact extracted from Wikipedia, spanning three formats and including two encoding measures.The encoding tasks are proposition completion and a contextual question; the passage does not provide the remaining task definitions or results.Source: A. WikiProfile – Full Details · p. 18
• WikiProfile uses four knowledge tasks covering direct and reverse questions with two phrasings each, plus multiple-choice variants for every task.Source: A. WikiProfile – Full Details · p. 18
• An automated pipeline powered by Gemini-2.5-Pro with thinking enabled constructs WikiProfile for application to other, including domain-specific, corpora.Prompt development used iterative error analysis and refinement on approximately 100 documents and facts, which were excluded from the final dataset.Source: A. WikiProfile – Full Details · p. 18

A.1. Tasks and Properties

• WikiProfile varies format, context, verbatim overlap, and relational direction to measure encoding, knowledge access, and verification across ten task types.Encoding is measured with completion and contextual tasks; knowledge is measured with direct, natural, reverse, and multiple-choice questions.Source: §A.1 · pp. 18–19

A.2. Fact Extraction Process

• Facts are extracted from 10,000 Wikipedia summaries and selected through entity recognition, proposition suitability checks, verification, and category-balanced downsampling to 5,000 entities.The extraction avoids simple schema-based triplets and targets non-trivial facts that are likely represented in pretraining data.Source: §A.2 · pp. 19–20

A.3. Question Generation Process

• Questions undergo generation, refinement, and Google Search–grounded filtering to enforce specificity, minimalism, unambiguous answers, and resistance to plausible alternatives.Reverse questions receive additional validation because reversing the relation can create multiple valid answers or trivial formulations.Source: §A.3 · pp. 20–21
• After filtering and length constraints, 2,200 facts retain all five question types, while reverse-question construction removes 31.5% of facts before an additional 13.1% filtering loss.The final set excludes questions over 40 words and left contexts below 30 words.Source: §A.3 · p. 21

A.4. Manual Quality Validation

• Manual validation flagged 237 facts, removed 43 low-quality cases, and found only 3 errors among a 50-fact subset, supporting high benchmark quality.Validation prioritized facts on which four frontier models unanimously failed at least one question.Source: §A.4 · p. 21

B. Thinking and Recall

• The analysis evaluates three mechanisms for thinking’s factual gains, each defined by distinct predictions about availability, robustness, and whether the target fact is encoded.The mechanisms are response diversity, inference from related facts, and recall facilitation of encoded but directly inaccessible facts.Source: B. Thinking and Recall · p. 22, §B.1 · p. 22

B.1. Three Possible Mechanisms

• Response diversity increases availability through broader sampling but cannot improve robustness, because variance does not create reliable convergence on the correct answer.The mechanism may yield some correct samples even when direct generation repeatedly produces the same incorrect answer.Source: §B.1 · p. 22
• Inference chains related encoded premises to answer a target question, so it can produce correct answers even when the specific target fact is not encoded.Recall facilitation instead uses additional computation to activate an encoded but inaccessible target fact and is predicted to improve both availability and robustness.Source: §B.1 · p. 22

B.2. Measuring Availability and Robustness

• correct≥p measures the proportion of questions whose estimated probability of correctness across samples is at least p.It captures availability and robustness along a continuous spectrum.Source: §B.2 · p. 22
• For each question, n independent samples produce k correct responses and an empirical correctness rate p̂ = k/n, which parameterizes a Binomial(n, p̂) estimate.The method estimates the probability that at least m = ⌈pn⌉ of n repeated answers are correct.Source: §B.2 · p. 22
• Aggregated across questions, correct≥p counts those whose estimated probability of at least ⌈pn⌉ correct answers exceeds 0.5.Formally, the criterion is P̂(K ≥ ⌈pn⌉) > 0.5.Source: §B.2 · p. 22
• Low p values test whether the model can produce any correct answer, whereas high p values test whether it is correct almost all of the time.For example, p = 0.125 corresponds to at least 1 correct answer out of 8 and is analogous to pass@k, while p = 0.875 demands near-consistent correctness.Source: §B.2 · pp. 22–23

Ruling Out Response Diversity

• For thinking-optimized LLMs, +Thinking improves robustness across all thresholds, ruling out response diversity as the sole explanation.At p>0.5, the +Thinking curve remains above the Base curve; increased output variance alone cannot produce greater consistency in correctness.Source: §B.3 · p. 23
• At low thresholds (p<0.5), thinking improves correct≥p for all 12 LLMs, consistent with response diversity and other mechanisms.These low thresholds correspond to Availability.Source: §B.3 · p. 23
• For Gemma3 models, CoT-elicited thinking degrades robustness as the curves cross at intermediate p.This reverses the pattern observed in thinking-optimized LLMs.Source: §B.3 · p. 23
• GPT-4.1 shows modest robustness gains with CoT prompting, suggesting that CoT mainly increases response diversity and that recall or inference benefits depend on capacity or training.GPT-4.1 therefore occupies a middle ground between the thinking-optimized and Gemma3 patterns.Source: §B.3 · p. 23

• Thinking recovers 40–65% of encoded but not directly known facts, versus less than 20% of non-encoded facts, favoring recall facilitation.This encoding dependence is consistent with recall facilitation, although it does not by itself rule out inference.Source: §B.4 · p. 23
• Thinking improves robustness, ruling out response diversity as the primary explanation for its factuality gains.Source: §B.4 · p. 23
• With n=8 samples, availability gains in the extreme low-probability tail may be underestimated, while robustness at high p remains well-estimated.Correct ≥p is insensitive to changes when p≲0.1 because such cases frequently produce zero correct samples.Source: §B.4 · p. 23
• Thinking gains concentrate mostly on encoded facts, and single-hop questions make complex deductive inference unnecessary.Thus, recall of an encoded fact is a more parsimonious explanation than inference for the observed gains.Source: §B.4 · p. 23

B.5. The Role of Thinking-Specific Training

• Native thinking consistently improves availability and robustness, whereas CoT prompting has mixed effects across model families.CoT improves availability but reduces robustness for Gemma3, while producing modest gains in both for GPT-4.1.Source: §B.5 · p. 23
• Recall facilitation may depend on thinking-specific training rather than inference-time token generation alone, but model-family differences prevent a conclusive attribution.The experimental design cannot fully disentangle training from differences in model size, data, and post-training procedures.Source: §B.5 · p. 23

C. Additional Results

• The additional-results section documents computation procedures and reports all evaluated LLMs to support the main-text trends.It provides supplementary analyses of profile distributions, popularity gaps, and phrasing robustness.Source: C. Additional Results · p. 23

C.1. Profiles

• The profile analysis estimates distributions after excluding non-gradable questions and defines Knows as Direct Recall, Recall with Thinking, and Inference without Thinking.Knows is treated as knowledge potential because it can slightly overestimate actual performance with thinking enabled.Source: §C.1 · pp. 23–24, Figure 13 · p. 24
• Figure 12 characterizes LLMs by correct≥p across samples, with low p reflecting availability and high p reflecting robustness.The figure reports this quantity for 12 LLMs and shows distinct model-specific patterns.Source: Figure 12 · p. 24

C.2. Fact Popularity

• For 10 of 13 LLMs, the recall gap between rare and popular facts is substantially larger than the encoding gap.Popularity tiers use Wikipedia page visits as an imperfect proxy, and recall is measured conditional on facts being encoded.Source: §C.2 · p. 24

C.3. Phrasing

• None of 104 phrasing tests were significant after Benjamini–Yekutieli FDR correction at q=0.05, indicating phrasing did not confound knowledge assessment.The tests covered 13 LLMs with and without thinking and four direct/reverse question-pair variants.Source: §C.3 · p. 25

C.4. Reverse Questions

• For all LLMs, the direct–reverse gap is much smaller for verification than generation, and reverse verification is easier for 9 of 13 models.Recall is evaluated conditional on facts being encoded, requiring both corresponding questions to exceed the grading threshold.Source: §C.4 · p. 25
• The reverse-generation gap persists without encoding conditioning and after controlling for answer type, whereas verification shows a much smaller or absent gap and often favors reverse questions.Thinking also mitigates the reverse-versus-direct gap, supporting the robustness of the observed pattern.Source: §C.4 · p. 25
• The reverse-question and thinking analyses condition on encoded facts to isolate recall failures, then test robustness by removing that conditioning and controlling for answer type.The same qualitative trends persist across these robustness analyses.Source: §C.4 · p. 25
• Reverse-question errors consistently exceed direct-question errors, with smaller models most often failing both directions.As model scale increases, failures shift toward reverse-only errors, which become the dominant failure mode.Source: §C.4 · p. 25

C.5. Thinking

• Thinking mitigates the directional recall gap for 9 of 13 LLMs and the popularity gap for 8 of 13, with substantial mitigation among frontier proprietary models.The analysis separates baseline recall from the additional share of encoded facts recovered only with thinking.Source: §C.5 · p. 25

D. Methodological and Design Choices

• The methodological section describes and justifies evaluation and analysis details that influence the study’s design and interpretation.This subsection frames the methodological choices rather than reporting an additional empirical result.Source: D. Methodological and Design Choices · p. 25

D.1. Handling Partially and Other Grades

• Question grades use accuracy over CORRECT and INCORRECT responses only, excluding PARTIALLY and OTHER because assigning them weights is subjective.For example, 3 CORRECT and 1 INCORRECT response yields a grade of 0.75 when six PARTIALLY or OTHER responses are excluded.Source: §D.1 · p. 25
• Excluding facts with any non-gradable profiling question would remove around 10% of frontier-model facts and over 20% of Gemma3 facts.This motivates the less aggressive pair-level exclusion rule.Source: §D.1 · p. 25
• The adopted fact-level strategy excludes a fact only when an entire encoding, direct, or reverse task pair is non-gradable, reducing exclusions to 1–4% with nearly unchanged profile distributions.This pair-level approach is preferred over excluding any fact with one non-gradable question because it substantially retains facts without materially changing the profiles.Source: §D.1 · p. 25
• Reverse-question errors exceed direct-question errors both across all facts without thinking and among encoded facts.The decomposition distinguishes failures on only direct questions, only reverse questions, or both.Source: Figure 14 · p. 26
• Assigning PARTIALLY responses a 0.51 weight lowers exclusions below 1% and closely matches pair-level results, but introduces subjective judgment, so it is not adopted.The paper therefore uses the second, pair-level strategy throughout.Source: §D.1 · pp. 25–26
• Facts that are not encoded but are known without thinking are excluded from profiling because they occur in less than 0.5% of cases and do not change results.The paper attributes these rare cases to sampling noise or grader error.Source: §D.1 · p. 26

D.2. Threshold Selection

• Across threshold choices, encoding remains nearly saturated for frontier LLMs, recall failures grow under stricter thresholds, and over 10% of facts are known only with thinking.At τ=0.99, frontier-model encoding remains nearly saturated, while thinking acts as a recovery mechanism under every tested threshold.Source: §D.2 · p. 26
• The Gemini-2.5-Pro and GPT-5 graders achieve 98.2% overall agreement, with most disagreements involving OTHER or PARTIALLY predictions.This provides a cross-grader consistency check for the response labels used in profiling.Source: Figure 15 · p. 26

D.3. Grader Evaluation

• Across 4,160 uniformly sampled responses from six tasks and all evaluated LLMs, Gemini-2.5-Pro and GPT-5 graders agreed on 98.2%, supporting insensitivity to the grader’s underlying LLM.The remaining disagreements primarily involved OTHER or PARTIALLY labels, which were excluded from the analysis.Source: §D.3 · pp. 26–27
• Gemini-2.5-Pro with thinking served as the primary grader, using a SimpleQA-adapted prompt refined through manual error analysis.The comparison grader used GPT-5 with thinking at medium effort, and both graders were evaluated on 4,160 responses across six tasks and all models.Source: §D.3 · p. 26
• Most grader disagreements occurred when Gemini-2.5-Pro assigned OTHER or PARTIALLY while GPT-5 assigned a definitive label.Manual examination found Gemini-2.5-Pro generally more accurate, though it occasionally drew on external knowledge.Source: §D.3 · pp. 26–27

D.4. Multiple Response Generation

• n=8 responses per question reduce sampling noise, replace unavailable or ill-defined probability estimates, and distinguish availability from robustness.Availability means at least one correct response, whereas robustness requires all responses to be correct; this distinction supports analysis of inference-time thinking.Source: §D.4 · p. 27
• The multiple-response analysis evaluates encoding and knowledge across completion, contextual, direct, and reverse tasks rather than relying on a single response per question.This design targets fact-level profiles and assesses how the fraction of facts with g(q) > 0.5 changes with the number of sampled responses.Source: §D.4 · p. 27
• At n=8, the 90% bootstrap confidence-interval width falls below 1% for every task across evaluated models, making estimated fact fractions stable within ±0.5%.The analysis used 1,000 bootstrap resamples across completion, contextual, direct, and reverse tasks, motivating n=8 throughout the paper.Source: §D.4 · p. 27, Figure 19 · p. 31

• Gemini’s grading treats permissible paraphrases, rounding, omitted modifiers, and broader descriptions as correct or partially correct when they remain factually compatible.Examples include “approximately 160 million” for 159.2 million, “Kasur” for Kasur District, and formal versus abbreviated exposition names.Source: Box D.1: Analysis of Disagreements… · pp. 32–33
• Answers can be judged other rather than incorrect when they provide a broader category, generic description, or related role without contradicting the requested fact.The examples include “Taskmaster” for Antonia Dreykov, “Black Panther Party” for Fredrick Hampton, and “Perimeter Wall” for Elephant Wall.Source: Box D.1: Analysis of Disagreements… · p. 32
• Incorrect judgments are reserved for responses that contradict the question or locate the answer in the wrong place, including rejecting the premise date or naming the wrong temple.The fossil example explicitly claims the question’s date is wrong, while the Vairocana Buddha response places the statue in a different temple.Source: Box D.1: Analysis of Disagreements… · p. 32
• A response is other when it is vague or irrelevant to the requested answer, such as stating that people built a town or describing US 1’s purpose instead of naming the highway.These answers do not contradict the gold answer but fail to supply the requested person or highway name.Source: Box D.1: Analysis of Disagreements… · pp. 32–33
• Refusals and nonsense fillers remain other or incorrect rather than acceptable approximations, as illustrated by “I cannot answer” and “Sure.”The grader labels the refusal incorrect because it contradicts the question’s premise and treats “Sure” as refusal or nonsense filler.Source: Box D.1: Analysis of Disagreements… · p. 33

Box E.1: Prompt for Classifying Pages into Categories

• The classification prompt assigns each document to one category from a predefined taxonomy spanning people, history, geography, arts, philosophy and religion, everyday life, society, science, technology, and other.The input consists of a page title and summary, and the taxonomy includes descriptions for each category.Source: Box E.1: Prompt for Classifying Pages… · pp. 33–34
• The classifier must output only one category name, without explanation, punctuation, or other text.This constrains the classification response to a standardized single-label format.Source: Box E.1: Prompt for Classifying Pages… · p. 34

Box E.2: Prompt for Performing Named Entity Recognition

• Named-entity recognition selects up to eight concise named entities from the paragraph’s second part and replaces them with typed placeholders while preserving the original text.Entities longer than five words or descriptive multi-concept phrases are excluded, and the entity name must match the source exactly.Source: Box E.2: Prompt for Performing Named… · p. 34
• The NER prompt defines a broad entity-type inventory, including people, organizations, locations, events, works of art, products, organisms, dates, numeric values, titles, languages, groups, laws, ideas, classifications, identifiers, and other.The typed replacement format is illustrated with labels such as PERSON, DATE, TITLE, EVENT, and NUMERIC_VALUE.Source: Box E.2: Prompt for Performing Named… · pp. 34–35

Box E.3: Prompt for Selecting Object Entities

• OBJECT selection targets entities whose values require recalling factual knowledge while the left-context uniquely constrains the expected entity type and completion.Selections must avoid trivial guesses, ambiguous contexts, sentence-initial entities, and information appearing after the target.Source: Box E.3: Prompt for Selecting Object… · pp. 35–36
• Selected OBJECT entities are returned in a list sorted from most to least suitable, with each entity reproduced exactly in the annotated-text format.The prompt uses factual examples to distinguish excellent candidates from guesses, ambiguous completions, and targets whose disambiguating information occurs later.Source: Box E.3: Prompt for Selecting Object… · pp. 36–37

Box E.4: Prompt for Verifying Object Entity Suitability – with Gold Target

• The suitability verifier accepts an OBJECT only when the left-context supports a coherent, specific completion with no plausible alternatives or contradictions.The verifier requires all listed conditions, including a clearly constrained entity type and a unique factual completion.Source: Box E.4: Prompt for Verifying Object… · p. 38
• The gold-target suitability prompt accepts an OBJECT only when the left-context identifies a unique semantic entity, not merely a matching surface form.Capitalization, punctuation, grammatical variants, omitted modifiers, and common aliases preserve uniqueness; multiple distinct entities make the completion unsuitable.Source: Box E.4: Prompt for Verifying Object… · p. 38

Box E.5: Prompt for Verifying Object Entity Suitability – without Gold Target

• The no-gold-target prompt requires the context to constrain the expected entity type and a unique completion whose alternatives would clearly contradict the conveyed facts.Examples treat vague contexts, multiple plausible entities, and underspecified comparison classes as unsuitable, while biographical, award, aircraft, and chart contexts can provide unique completions.Source: Box E.5: Prompt for Verifying Object… · p. 40
• Semantic uniqueness in the no-gold-target prompt tolerates alternate forms such as capitalization, punctuation, grammatical variation, omitted modifiers, and aliases, but not multiple distinct entities.This criterion is illustrated by rejecting contexts with several plausible foods, stabilizers, or comparison classes and accepting contexts that identify a specific city, award, aircraft, or chart.Source: Box E.5: Prompt for Verifying Object… · p. 40

Box E.6: Prompt for Verifying Completion Difficulty

• The completion-difficulty prompt requires challenging, static factual recall that cannot be trivially guessed, logically inferred, or directly disclosed by the context.It excludes common-sense answers, answers already contained in the prompt, and time-sensitive values unless the context fixes them to a specific year.Source: Box E.6: Prompt for Verifying… · p. 42
• Time sensitivity is a key exclusion criterion: an unspecified-year median household income is unsuitable, whereas a value tied to a specific year is treated as static.The examples classify $67,521 as time-sensitive without a specified year and 50 billion Indian Rupee as stable when restricted to 2018.Source: Box E.6: Prompt for Verifying… · pp. 42–43
• The difficulty prompt treats recalling Vincent Price’s spoken-word performance and identifying the first modern Olympics as suitable nontrivial factual completions.The latter remains suitable despite contextual hints because the specific fact cannot be determined from the stadium’s location alone.Source: Box E.6: Prompt for Verifying… · p. 43

Box E.7: Prompt for Generating the Direct Task

• The direct-task prompt begins from an incomplete left-context and an OBJECT entity to construct a factual completion task.The supplied passage introduces the prompt inputs but does not specify further criteria in this section part.Source: Box E.7: Prompt for Generating the… · p. 43
• Direct-task prompts generate only the question, using context-derived wording and essential details to make the OBJECT the sole unambiguous answer.Questions must be self-contained, precise about the required answer type, minimal, and free of outside references or synonyms unless needed for clarity.Source: Box E.7: Prompt for Generating the… · p. 43
• Examples operationalize the prompts by retaining identifiers needed for disambiguation while excluding unnecessary dates, genres, biographical facts, and other contextual hints.Examples include adding an artist to distinguish an album title, retaining an incident to identify a condition, and removing a defending-champion clue.Source: Box E.7: Prompt for Generating the… · p. 44, Box E.9: Prompt for Refining Minimalism · p. 47

Box E.8: Prompt for Refining Specification

• Specification-refinement prompts verify whether a question uniquely targets the gold answer and specifies its entity type, granularity, and required format.Questions that fail can receive minimal clarifying changes, such as changing “Who” to “Which company,” adding units, or narrowing a qualifier without leaking non-trivial information.Source: Box E.8: Prompt for Refining… · p. 45
• Unfixable questions are rejected when correction would leak part of the answer or require a major rewrite rather than a minimal clarification.The required output is only the original question, a revised question, or “REJECT”.Source: Box E.8: Prompt for Refining… · p. 46

Box E.9: Prompt for Refining Minimalism

• Minimalism prompts remove non-essential details and answer-hinting clues while retaining the information needed for unique entity identification and a single gold answer.Minimalism is defined by disambiguation rather than brevity, so even long questions remain acceptable when every component is essential.Source: Box E.9: Prompt for Refining Minimalism · p. 47
• Minimalist revisions may only delete words or phrases, preserve the remaining wording, and retain grammaticality, logical sense, and unique answer targeting.The process removes redundant specifiers, superfluous descriptors, and leaked clues when another identifier already disambiguates the question.Source: Box E.9: Prompt for Refining Minimalism · p. 47
• Minimalism refinement removes unnecessary hints while preserving essential context, official names, abbreviations, and information needed to disambiguate the question.The prompt returns either the original question or only a revised question string, never explanatory text.Source: Box E.9: Prompt for Refining Minimalism · p. 48

Box E.10: Prompt for Filtering with Google Search Grounding

• The filtering prompt labels a question suitable only when the expert answer confirms an unambiguous, specific question with one direct and unique answer.It rejects clarification requests, multiple valid answers, and answers that differ across interpretations, outputting only SUITABLE or NOT.Source: Box E.10: Prompt for Filtering with… · p. 49

Box E.11: Prompt for Generating the Reverse Task

• Reverse-task generation swaps the direct question’s subject and object, producing a self-contained question whose answer is the original subject.The reverse must be unambiguous, specific, minimal, verbatim, challenging, and uniquely answerable; unsuitable items are rejected.Source: Box E.11: Prompt for Generating the… · p. 50
• Reverse questions may add context only when necessary for uniqueness, without shifting focus or creating a shortcut to the answer.For example, edition and director details disambiguate a film premiere, whereas unrelated adaptation details are disallowed.Source: Box E.11: Prompt for Generating the… · pp. 50–51
• Reverse generation rejects cases where no added detail can preserve ownership as the question’s focus while ensuring a single answer.The radio-station example is rejected because the owner controls many stations and available details would make the question a slogan or frequency question.Source: Box E.11: Prompt for Generating the… · pp. 50, 52

Box E.12: Prompt for Verifying a Valid Reverse and Direct Pair

• Direct/reverse pairs are suitable only when the object appears in the reverse question, both answers require specific factual knowledge, and dynamic facts include a timeframe.Pairs are rejected when answers are guessable, the direct answer is absent from the reverse question, or changing information lacks a specified timeframe.Source: Box E.12: Prompt for Verifying a Valid… · p. 52
• Verification excludes trivial pairs when an answer is explicitly contained in the question, such as a book title revealing the requested language.The required relationship is a proposition between subject and object, with the direct question targeting the object and the reverse targeting the subject.Source: Box E.12: Prompt for Verifying a Valid… · p. 52
• Reverse-question validation accepts pairs only when the direct answer is included to resolve ambiguity, remains time-stable, and requires nontrivial knowledge rather than repeating question terms.Examples reject time-sensitive answers without a year, structurally invalid reversals, and trivial term repetition.Source: Box E.12: Prompt for Verifying a Valid… · p. 53

Box E.13: Prompt for Generating the Natural Questions

• Natural-question generation preserves a question’s exact answer while making it self-contained, unambiguous, specific, minimal, and conversational.Clarifying phrases may specify whether an answer should be a date or year, while minor wording can be removed sparingly when it does not affect disambiguation.Source: Box E.13: Prompt for Generating the… · p. 54, Box E.14: Prompt for Generating the… · p. 55

Box E.14: Prompt for Generating the Contextual Questions

• Contextual-question generation prepends the relevant context to an unambiguous question, excluding its incomplete final sentence and allowing minor edits for fluency and redundancy.The resulting question tests comprehension within the supplied context while retaining the original answer target.Source: Box E.14: Prompt for Generating the… · p. 55

Box E.15: Prompt for Generating Options for Multiple-Choice Questions

• Multiple-choice construction keeps the gold answer first and adds three plausible but definitively incorrect distractors matched to its type, category, and granularity.Distractors may exploit contextual hints, similar entities, thematic relations, or popular associations.Source: Box E.15: Prompt for Generating… · p. 57

Box E.16: Prompt for Generating the Reverse Task

• The reverse-task prompt maps each answer to one entity type from a defined taxonomy spanning people, organizations, locations, events, works, quantities, conditions, laws, ideas, and other categories.The output must contain only one taxonomy label, such as GROUP or DATE.Source: Box E.16: Prompt for Generating the… · pp. 57–58

Box E.17: Task Instructions

• Task instructions standardize evaluation outputs by requiring either answer-only completions, answer-labeled step-by-step responses, or only the letter of the correct option.These formats distinguish open-answer, chain-of-thought-style, and multiple-choice tasks.Source: Box E.17: Task Instructions · p. 59

Box E.18: Autorater Prompt for Text Completion Tasks

• The text-completion autorater prompt supplies an incomplete text, a gold completion, and the expected entity continuation for judging responses.The visible instructions frame the gold entity as the expected completion of the left context.Source: Box E.18: Autorater Prompt for Text… · p. 59
• Autoraters assign one of four labels—CORRECT, PARTIALLY, INCORRECT, or OTHER—when evaluating predicted completions against the gold completion and left context.The closed-book QA prompt uses the same four-way label set while comparing predicted answers with the gold target and question.Source: Box E.18: Autorater Prompt for Text… · p. 59, Box E.19: Autorater Prompt for… · p. 62
• CORRECT requires the prediction to preserve the gold information without contradiction, while allowing semantic placement, minor omissions, variants, formatting differences, and harmless extra details.Hedging or guessing remains acceptable when the gold content is included and neither the prediction nor its extra details contradict the context.Source: Box E.18: Autorater Prompt for Text… · p. 60
• INCORRECT applies when a predicted completion contradicts the gold completion or context, including when the contradiction is hedged with uncertainty.The closed-book QA criteria apply the same rule to predicted answers that contradict the question or gold target.Source: Box E.18: Autorater Prompt for Text… · p. 60, Box E.19: Autorater Prompt for… · p. 63
• PARTIALLY covers predictions that identify the core meaning but are more general, omit descriptive detail, or express a closely related outcome rather than the exact gold completion.Examples include “photographer” for “documentary photographer,” “dopaminergic medication” for “dopamine agonist medications,” and “end-stage kidney disease” for “chronic kidney disease.”Source: Box E.18: Autorater Prompt for Text… · p. 61
• OTHER is used when the prediction omits the gold information without contradicting it, including refusals, requests for context, or unrelated continuations.This category separates non-answering or irrelevant responses from factual contradictions.Source: Box E.18: Autorater Prompt for Text… · p. 61
• The rubric does not penalize information inferable from context, clear name typos, or minor lexical and grammatical differences when the intended answer remains identifiable.For example, “Outstanding Paper” can stand for “Outstanding Paper Award,” and several misspellings of a person’s name remain acceptable.Source: Box E.18: Autorater Prompt for Text… · p. 62

Box E.19: Autorater Prompt for Closed-book QA Tasks

• The closed-book QA prompt transfers the same semantic grading principles from text completion, comparing each answer with the question and gold target before emitting the required label format.It provides parallel examples for correct, incorrect, and partially correct answers, with labels output as CORRECT, INCORRECT, PARTIALLY, or OTHER.Source: Box E.19: Autorater Prompt for… · pp. 62–63
• The autorater labels answers as CORRECT, INCORRECT, PARTIALLY, or OTHER based on their relation to the gold target and question.The required output uses one tagged grade for each answer.Source: Box E.19: Autorater Prompt for… · pp. 64–65
• A predicted answer can be PARTIALLY correct when it gives a more general or less specific target, preserves the core meaning, or omits a descriptive modifier.Examples include “dopamine agonist medications” versus “dopaminergic medication” and “photographer” versus “documentary photographer.”Source: Box E.19: Autorater Prompt for… · p. 64
• Closely related answers are also accepted as PARTIALLY correct when they represent a common outcome or progression, even without strict identity to the gold target.The prompt gives end-stage kidney disease as a related severe form of chronic kidney disease.Source: Box E.19: Autorater Prompt for… · p. 64
• Answers are OTHER when they omit the gold target’s important information without contradicting either the target or the question.The prompt illustrates this category with vague, evasive, or merely related completions that do not supply the requested fact.Source: Box E.19: Autorater Prompt for… · p. 64
• Numeric answers must match the gold target to its last significant figure, while approximate or lower-bound responses are OTHER when they neither confirm nor contradict it.For a gold value of 120k, 120k, 124k, and 115k are CORRECT, whereas 100k and 113k are INCORRECT.Source: Box E.19: Autorater Prompt for… · pp. 64–65
• The grader should not penalize information clearly inferred from the question or minor typographical variations when the intended person’s name remains clear.For example, “Outstanding Paper” can match “Outstanding Paper Award,” and several misspellings of “Hyung Won Chung” are accepted.Source: Box E.19: Autorater Prompt for… · p. 65
• The prompt requires a strict tagged response format, grading every supplied answer with one of the four labels and forbidding apologies or self-correction.Each grade must appear as <answer_i>...</answer_i> around the corresponding answer’s label.Source: Box E.19: Autorater Prompt for… · p. 65