Source-linked AI summary
LUMOS: Tracing Parametric Knowledge from Training Data to Behavioral Outputs in LLMs
Seoyeon Ye, Gayoung Kim, Jiyoung Hong, Sookyung Kim, Hyunsoo Cho
TL;DR
Output-centric evaluation cannot verify what a model saw during training, leaving memorization and generalization difficult to distinguish. LUMOS traces knowledge from verified exposure to behavior using OLMo 2 and its transparent corpus. It finds a gap between internal encoding and behavioral expression, alongside self-verification that falters on unseen content.
Problem
Output-centric analyses do not verify training exposure, limiting evidence about whether correct answers reflect memorization or generalization.
Source: §1 · pp. 1–2Method
LUMOS traces knowledge from verified training exposure through internal representations and probabilistic preferences to behavior, using OLMo 2 and its transparent corpus.
Source: §1 · p. 2Results
Rare facts can be internally encoded yet fail to appear in outputs; self-verification is reliable on trained content but approaches random-baseline performance on unseen content, and chain-of-thought inflates confidence without improving calibration.
Source: §1 · p. 2Takeaways & Limitations
The authors conclude that grounding knowledge assessment in training exposure makes diagnoses empirically verifiable and should be a standard component of LLM knowledge evaluation.
Source: §1 · p. 2, §6 · p. 9The study relies on accessible pretraining corpora, string matching and frequency statistics that may miss implicit or paraphrased exposure, and a specific model suite whose generalizability remains to be validated.
Source: A Limitations · p. 13
Abstract
from arXiv · showhide
Current analyses of LLMs' parametric knowledge are largely output-centric, drawing conclusions about what a model knows without verifying what it was actually trained on. This leaves fundamental questions, such as whether a correct response reflects genuine generalization or rote memorization, grounded in speculation rather than evidence. To resolve these ambiguities, we introduce LUMOS, a diagnostic framework that traces knowledge along the causal chain from training-data exposure to behavioral output, leveraging OLMo 2 with its fully transparent training corpus. By grounding analysis in verified exposure, we reveal that models internally encode rare facts with high separability (84%) yet fail to express them behaviorally (54%), though this retrieval gap narrows with scale. Furthermore, when models are asked to self-reflect on their own answers, they perform reliably on trained content (83%) but drop to random-baseline levels (49%) on unseen content. This collapse persists even under chain-of-thought prompting, which inflates confidence signals rather than improving calibration. Collectively, these findings demonstrate that incorporating the training-data axis into LLM evaluation transforms speculative diagnoses into verifiable claims, and we advocate that this axis should be a standard component of knowledge assessment in LLMs.
1 Introduction
Output-centric evaluation leaves unclear whether correct answers reflect memorization or generalization, and prior probing omits the training data that shaped model representations. LUMOS grounds evaluation in verified exposure and finds that internal knowledge, behavioral expression, and self-assessment can diverge.
1 Introduction
- Output-centric evaluation cannot establish whether a fluent answer reflects robust internal knowledge or a stochastic coincidence, nor reliably map outputs back to training origins.Behavioral, internal, and mechanistic probes examine outputs, representations, and parameters, but omit the training-data exposure that shaped them.Source: §1 · pp. 1–2
- LUMOS traces knowledge from training exposure through internal representations and probabilistic preferences to behavioral output, using OLMo 2 and its transparent corpus.Its evaluation pairs seen and unseen benchmarks across factual and reasoning domains, varying frequency and prompting while combining behavioral and internal probes.Source: §1 · p. 2
- Models can encode rare facts without expressing them, self-verify trained content more reliably than unseen content, and show confidence inflation rather than improved calibration under chain-of-thought prompting.The introduction also reports that the retrieval gap narrows with scale, while the exposure boundary remains consequential across factual and reasoning evaluations.Source: §1 · p. 2
- The authors present exposure-grounded evaluation as a way to make previously conjectured knowledge phenomena empirically verifiable and reveal failure modes missed by output-only assessment.Source: §1 · p. 2
2 Related Work
Prior work probes observable behavior and internal model signals, but these approaches rarely verify what the model encountered during training.
2 Related Work
- Behavioral probing measures observable outputs such as benchmark accuracy, confidence estimates, and uncertainty or selective-answering behavior.Methods also aggregate behavior across paraphrased prompts, controlled knowledge conditions, or many-shot sampling.Source: §2 · p. 2
- Internal probing examines logits or hidden representations, with studies reporting that correct-answer probabilities can track factual accuracy and that latent probes reveal an expression gap.Semantic entropy across answer clusters is used to separate uncertainty from linguistic variability.Source: §2 · p. 2
- Despite their different signals, these research lines rarely ground conclusions in what the model actually saw during training.Source: §2 · p. 2
3 LUMOS Taxonomy
LUMOS traces parametric knowledge through exposure, internal states, and output, then combines exposure and behavioral accuracy with measures of epistemic depth. This taxonomy distinguishes kinds of apparent success and failure by where knowledge is grounded or breaks down.
3 LUMOS Taxonomy
- LUMOS formalizes a knowledge trace from training corpus D through hidden representations h and probabilistic preferences p to behavioral output y, aligning knowledge assessment with exposure, expression, and calibrated uncertainty.Source: §3 · pp. 2–3
3.1 Dual-Axis: Exposure vs. Performance
- The exposure and accuracy axes yield four states: robust memorization and knowledge attrition for exposed facts, and systematic generalization and epistemic void for unexposed facts.The taxonomy distinguishes correct from incorrect output within both seen and unseen conditions.Source: §3.1 · p. 3, Figure 1 · p. 3
3.2 Depth Axis: Quantifying Epistemic Grounding
- Epistemic Depth measures stability along the internal-state-to-output chain; Decisional Consistency captures whether stochastic outputs converge, while Probabilistic Confidence probes the grounding of internal states.High decisional consistency can indicate stable output regardless of correctness; confidence can be operationalized with predictive uncertainty or latent probes.Source: §3.2 · pp. 3–4
3.3 Multidimensional Knowledge Diagnostics
- Combining exposure, accuracy, and depth distinguishes robust encoding from brittle pattern matching, latent knowledge from weight-level erasure, and grounded from spurious generalization.For unexposed incorrect answers, low confidence indicates calibrated uncertainty while high confidence flags overconfident hallucination; the analysis focuses on attrition and epistemic-void cases.Source: §3.3 · p. 4
4 LUMOS Dataset
The dataset pairs seen and unseen factual and reasoning questions, grounding exposure judgments in the OLMo 2 corpus. Its construction and evaluation distinguish frequency, domain, prompting, and behavioral versus internal signals.
4 LUMOS Dataset
- LUMOS uses OLMo 2-Instruct because its accessible pretraining corpus enables precise exposure verification alongside access to model internals.Source: §4 · p. 4
4.1 Evaluation Domains: Factual vs. Reasoning
- The evaluation separates factual memory from mathematical and code reasoning, and compares OLMo 2-Instruct with OLMo 2-SFT for reasoning under different post-training paradigms.Seen and unseen QA pairs are drawn from or excluded from the pretraining corpus, respectively.Source: §4.1 · pp. 4–5, Table 1 · p. 5
4.2 Factual Domain Construction
- Factual seen data comes from Web and Academic sources, with entities stratified by exposure frequency; unseen queries use temporally novel sources after the OLMo 2 release cutoff.Unseen queries are reformulated as open-ended questions and retained only if the model fails to answer correctly in all 11 sampling trials.Source: §4.2 · p. 5
4.3 Reasoning Domain Construction
- Mathematical reasoning uses GSM8K training data and a filtered MetaMathQA subset as seen benchmarks, and GSM8K test, GSM1K, and GSM-Symbolic as unseen benchmarks.The unseen sets retain structural alignment with training templates while changing numerical values and entities; code reasoning is treated as a complementary domain.Source: §4.3 · pp. 5–6
4.4 Evaluation Metrics
- Evaluation combines output-based accuracy, Pass@k, decisional consistency, and self-verification with probabilistic confidence measures from answer likelihoods, hidden-state probes, semantic uncertainty, and activation-based prediction.Source: §4.4 · p. 6
5 Results
Factual results show persistent Seen–Unseen reliability gaps, alongside retrieval failures for encoded facts and self-verification failures on unseen content. Reasoning generalization depends on training method and model scale.
5.1 Results on Factual Domain
- At 32B, Seen Academic Self-Verify Accuracy reaches 83.76%, while Unseen accuracy remains near the 50% baseline and F1 falls to 41–45%.At 7B, Unseen Current Events accuracy of 66.30% coexists with precision and recall both below 16%.Source: Table 3 · p. 7, §5.1 · p. 8
- At 7B, Tail entities reach 53.64% Accuracy but show lower DC than Head entities (74.01% vs. 92.71%); probe ranks indicate encoded knowledge is not reliably expressed.The Tail probe rank is 9 versus Accuracy rank 23 at 7B; the expression gap narrows with scale and reverses for Seen Academic at 32B.Source: Table 4 · p. 7, §5.1 · p. 7
- Under CoT, 32B Unseen Self-Verify remains near baseline (48.98% on RealTimeQA; 49.87% on Current Events), while Seen performance remains stable (83.56%).At 7B, Unseen RealTimeQA Norm. P rises from 28.18% to 52.83% under CoT while Accuracy decreases from 33.88% to 31.74%.Source: Table 5 · p. 8, §5.1 · p. 8
- A 30–40 percentage point gap separates Seen and Unseen Accuracy, with Unseen performance below 44% even at 32B; few-shot gains are negligible on Unseen RealTimeQA.At 7B, few-shot examples raise Seen Web Accuracy from 70.09% to 76.04%, but Unseen RealTimeQA rises only from 33.88% to 35.70%; internal metrics also retain exposure gaps.Source: §5.1 · pp. 8–9
5.2 Generalization in Reasoning
- Reasoning generalization differs by training method: SFT overcomes its Seen-task advantage only as capacity grows, while RLVR shows Unseen GSM8K-Test outperforming Seen MetaMath from 7B.At 32B, RLVR’s GSM8K-Test and GSM-Symbolic Pass@1 scores are 87.64% and 83.37%, while SFT is more sensitive to numerical perturbation (75.55% vs. 67.14%).Source: Table 7 · p. 9, §5.2 · p. 9
6 Conclusion
LUMOS grounds knowledge evaluation in verified training exposure and traces knowledge from pretraining data to behavioral output. The authors conclude that exposure is the dominant determinant of factual reliability and advocate making the training-data axis standard in LLM knowledge assessment.
6 Conclusion
- Training exposure is the dominant determinant of factual reliability; scaling, prompting, and chain-of-thought reasoning cannot overcome this structural constraint.The authors advocate including the training-data axis as a standard component of LLM knowledge assessment.Source: §6 · p. 9
A Limitations
The study’s findings are based on models with accessible pretraining corpora, an exposure measure based on explicit string matching and frequency statistics, and a specific model suite. The authors identify these choices as boundaries on applicability and generalizability.
A Limitations
- LUMOS’s direct applicability to closed-source or proprietary LLMs is limited by the reliance on fully accessible pretraining corpora.The authors also note that explicit string matching and frequency statistics may miss implicit, paraphrased, or indirectly encoded knowledge, and that validation on other open-source model suites remains future work.Source: A Limitations · p. 13
B Experimental details
Experiments used 11 generated responses per question, while Figure 2 presents entity frequency distributions under head–tail stratification.
B Experimental details
- Each question received 11 outputs: one greedy-decoded response and ten samples generated at temperature 1.0.These outputs were used uniformly across behavioral and internal metrics on a system with four NVIDIA RTX PRO 6000 GPUs and approximately 98 GB total VRAM.Source: B Experimental details · p. 13
- Figure 2 presents entity frequency distributions under head–tail stratification.Source: Figure 2 · p. 13
C Evaluation Metrics
The evaluation combines output behavior, internal signals, self-verification, and sampling-based reasoning metrics. It also includes a head-versus-tail comparison under chain-of-thought prompting.
C.1 Accuracy
- Accuracy averages the fraction of correct responses across 11 generations per question, rather than using majority vote.The dataset-level measure averages per-question accuracy.Source: §C.1 · p. 14
C.2 Decisional Consistency (DC)
- Decisional Consistency measures the normalized frequency of the most common answer across 11 generations, regardless of correctness.Dataset-level consistency averages this per-question measure.Source: §C.2 · p. 14
C.3 Normalized Probability
- Normalized Probability uses length-normalized conditional likelihood to measure how strongly the model favors a candidate answer.For reasoning tasks, probabilities are computed over the concatenated question, reasoning, and answer sequence.Source: §C.3 · pp. 14–15
C.4 Linear Probing
- Linear Probing trains layer-specific classifiers on hidden states to predict answer correctness from paired correct and incorrect examples.The best layer is selected by development-set AUROC, and scores are averaged over correct test answers.Source: §C.4 · p. 15
C.5 Meaning-Aware Response Scoring (MARS)
- MARS weights token probabilities by sequence length and semantic importance to score generated responses.Its token-importance coefficients are estimated by masking phrases and measuring changes in a correctness prediction.Source: §C.5 · pp. 15–16
C.6 Statement Accuracy Prediction based on Language Model Activations (SAPLMA)
- SAPLMA uses hidden activations and a classifier to predict statement truthfulness, adapted here to rank multiple-choice answers.The adapted method scores each question-choice input without additional prompting and selects the highest-scoring choice.Source: §C.6 · p. 16
C.7 Self-verify
- Self-verify prompts the model to judge a proposed answer’s correctness and compares that binary judgment with ground truth.Evaluation includes accuracy, precision, recall, and F1, and uses the same data splits as output-based metrics.Source: §C.7 · pp. 16–17
C.8 Pass@k
- Pass@k measures the chance of finding at least one correct answer among k sampled responses, averaged across questions.It does not account for diversity among correct solutions or the probability mass distribution across responses.Source: §C.8 · p. 17
D.2.1 Dataset
HumanEval (164 problems) and MBPP (500 problems) are used as out-of-distribution code-generation tests because neither dataset was included in the OLMo 2 pretraining corpus.
D.2.1 Dataset
- HumanEval (164 problems) and MBPP (500 problems) are treated as out-of-distribution test sets because neither appeared in OLMo 2 pretraining.Both benchmarks come from the unseen split.Source: §D.2.1 · p. 18
D.2.2 Generalization in Code Dataset
On code benchmarks, RLVR offers limited advantages over SFT, unlike the gains observed in math-based reasoning. The findings emphasize that RLVR’s efficacy depends on alignment between its reward signal and the task.
D.2.2 Generalization in Code Dataset
- RLVR gains are most pronounced when its reward aligns with well-structured reasoning; they diminish for execution-dependent code tasks.The passage attributes the difference to code correctness being less directly captured by the immediate reward signal.Source: §D.2.2 · p. 18
- RLVR provides limited advantages over SFT on code benchmarks, with SFT’s lead clearer at higher k, especially on MBPP.HumanEval Pass@1 is comparable, while SFT surpasses RLVR across most MBPP settings.Source: §D.2.2 · p. 18, Table 9 · p. 19
E.1 Unseen Filtering Prompt
- Unseen-filtering prompts request concise answers as a single word, number, or short phrase rather than a complete sentence.Source: §E.1 · p. 19
E.2 Factual Domain Prompt
- Factual-domain prompts require a single-letter multiple-choice answer, with chain-of-thought variants asking for stepwise reasoning before the final letter.Source: §E.2 · pp. 19–20
E.3 Reasoning Domain Prompt
- Reasoning-domain prompts ask models to solve math word problems step by step and provide a numerical final answer.Source: §E.3 · p. 21