MAPLE: A Multi-Axis Parametric Knowledge Extraction Lens for LLM Evaluation

1 Introduction

• The introduction frames LLM outputs as probabilistic inferences from parametric representations, not direct readouts of learned knowledge.Prior work cited in the passage finds that pretraining-data presence and a correct answer alone are insufficient criteria for judging what a model knows.Source: §1 · p. 1
• Aggregate accuracy can conceal different prediction changes: methods with the same score may differ in recovered failures and disrupted successes.The introduction motivates consistent comparisons across KEE techniques beyond aggregate accuracy.Source: §1 · pp. 1–2
• MAPLE combines a standardized factual-recall benchmark with verified pretraining-corpus presence for OLMo2 and diagnostics spanning surface, probability, and beyond-sampling levels.Queried facts are retained when their sentences appear verbatim in OLMo2’s pretraining corpus; this establishes an opportunity to learn, not successful encoding or extractability.Source: §1 · p. 2
• Prompting methods recover the most failures, including answers missed by fixed-budget sampling, but also disrupt correct responses; ITI stays near baseline, while decoding and CAA alter ranks without consistently improving them.The contrast between ITI and CAA shows that methods acting at the same inference stage can have different effects.Source: §1 · p. 2
• Its metrics separate recovery from disruption, track gold-token rank shifts on unrecovered failures, and measure recovery beyond fixed-budget sampling.The framework compares each technique with the model’s own baseline.Source: §1 · p. 2

2 Related Work

• Decoding methods recalibrate output distributions using contrastive signals from smaller models, early layers, masked attention heads, or activation entropy (Contrastive Decoding; Li et al., 2023b; DoLa; Chuang et al., 2024; DeCoRe; Gema et al., 2025; Activation Decoding; Chen et al., 2024).These are examples of the diverse methodological approaches covered in the related-work section.Source: §2 · p. 2
• Steering methods shift internal activations using vectors from linear probes (ITI; Li et al., 2023a), contrastive examples (CAA; Rimsky et al., 2024), or input-adaptive semantics (SADI; Wang et al., 2025).Source: §2 · p. 2
• Prompting methods first elicit implicit knowledge as text, then condition generation on that context through demonstrations (RECITE; Sun et al., 2023) or zero-shot instructions (GenRead; Yu et al., 2023).Source: §2 · pp. 2–3
• Existing evaluation benchmarks were designed for other purposes: some target reasoning or misconceptions, while FACTOR and FEVER test selection among explicit candidates rather than answer generation from model knowledge.The section also notes that TriviaQA, Natural Questions, and PopQA use different source or entity distributions (Joshi et al., 2017; Mallen et al., 2023; Lin et al., 2022; Yang et al., 2018; Thorne et al., 2018; Muhlgay et al., 2024).Source: §2 · p. 3

3.1 Benchmark Construction

• The benchmark uses pre-cutoff Wikipedia triplets, filters for valid single-answer facts, and verifies corpus presence to reduce exposure-related ambiguity.Triplets came from a March 2022 Wikipedia dump, were organized across ten domains, and were checked against OLMo2’s public pretraining corpus.Source: §3.1 · pp. 3–4
• 1,303 of 3,254 candidate triplets passed all three filtering steps and formed the final benchmark.Corpus-presence filtering required at least one supporting sentence to appear in OLMo2’s pretraining corpus; the filter may exclude facts present in paraphrased or unrecovered sentences.Source: §3.1 · p. 4

3.2 Evaluation Metrics

• Gain and Loss separately measure the proportions of baseline failures recovered and baseline successes disrupted by a KEE technique.Their denominators are the baseline failures and baseline successes, respectively, rather than the full dataset.Source: §3.2 · p. 4
• Probability Rank Shift tracks whether the gold token’s rank improves, stays tied, or worsens on failures that remain incorrect under both settings.The method compares the first gold token’s rank under the baseline and KEE technique, excluding instances where it ranks first under either setting.Source: §3.2 · pp. 4–5
• Shallow and deep gain rates compare KEE recovery with temperature sampling for failures that sampling can or cannot reach within K samples.At K = 20, the method ranking’s Kendall correlation with rankings at other K values is at least 0.89 for both SGR and DGR.Source: §3.2 · p. 5, Table 1 · p. 5

4.1 Settings

• The evaluation covers OLMo2-7B/32B-Instruct, Qwen2.5-7B/32B-Instruct, and Llama3-8B/70B-Instruct, with two representative methods from each KEE family.DeCoRe and AD represent decoding, ITI and CAA steering, and RECITE and GenRead prompting; temperature sampling at τ=0.7 and τ=1.2 serves as a comparison.Source: §4.1 · p. 5
• Greedy decoding makes top-1 prediction changes attributable to KEE interventions, while temperature-sampling answers use majority votes across 10 samples.Correctness accepts gold answers and aliases after normalization.Source: §4.1 · p. 6

4.2 Main Results

• Gain and Loss separate recovered failures from disrupted successes, PRS tracks gold-token rank changes on unrecovered failures, and SGR/DGR test recovery beyond temperature sampling.The metrics are based on answer flips or ranks, enabling comparisons across decoding, steering, and prompting techniques.Source: §4.2 · p. 6
• Prompting recovers the most failures beyond sampling but also incurs the largest Loss; ITI stays near baseline, while decoding and CAA reshape distributions without consistently moving toward gold answers.Decoding and CAA recovery largely remains within the reach of temperature sampling.Source: §4.2 · p. 6

5.1 Surface Level

• RECITE and GenRead achieve the highest Gain across all models, while GenRead also has the highest Loss in every model.RECITE maintains comparably high Gain with lower Loss than GenRead in every model.Source: §5.1 · p. 6, Table 2 · p. 7
• On Llama3, decoding techniques achieve 1.6–4.9× the temperature-sampling Gain, but on OLMo2 and Qwen2.5-32B they achieve only 0.3–1.5×.Their Loss is comparable to temperature sampling and markedly below prompting methods, except Qwen2.5-7B, where DeCoRe and AD reach 9.96% and 11.21% Loss.Source: §5.1 · pp. 6–7
• CAA’s Gain and Loss stay comparable to temperature sampling on OLMo2 and Qwen2.5, but reach 1.8–5.1× and 1.3–4.7× higher levels on Llama3, respectively.ITI stays close to baseline across models, with Gain between 1.21% and 4.91% and consistently low Loss.Source: §5.1 · p. 7

5.2 Probability Level

• Only 29.7%, 34.7%, and 33.3% of unrecovered failures remain Tied for DeCoRe, AD, and CAA, respectively; on average, PRSWorsened exceeds PRSImproved.On Llama3-70B, AD has PRSWorsened of 46.7% against PRSImproved of 13.1%.Source: §5.2 · p. 7, Figure 2 · p. 7
• ITI has the largest PRSTied share at 62.5%, ranging from 53.0% to 72.2% across models, while PRSImproved and PRSWorsened are nearly balanced at 18.2% and 19.3%.This minimal reshaping is consistent with small Gain and low Loss at the surface level.Source: §5.2 · pp. 7–8

5.3 Beyond-Sampling Level

• Decoding and CAA gains are largely concentrated in shallow failures; ITI stays at or below 2.06% DGR in every model at both temperatures.On OLMo2-7B, AD has SGR of 5.96% and DGR of 0.00%, versus τ1.2 at 16.06% and 0.42%; Llama3-8B CAA reaches DGR of 4.10%, versus τ1.2 at 0.24%.Source: §5.3 · p. 8, Table 3 · p. 8
• RECITE and GenRead decisively outperform τ0.7 and τ1.2 on SGR and DGR; on OLMo2-32B, their DGR reaches 15.76% at τ=0.7 and 11.43% at τ=1.2, versus 0.00% for both sampling baselines.Source: §5.3 · p. 8, Table 3 · p. 8

• MAPLE’s benchmark contains facts verified to be present in OLMo2’s pretraining corpus and pairs them with diagnostics across three levels of knowledge extraction.Source: §6 · p. 8
• Across three model families and six techniques, prompting recovers the most failures beyond sampling but incurs the largest Loss, while ITI stays near baseline and decoding and CAA reshape distributions without moving toward gold answers.Source: §6 · p. 8
• MAPLE offers a fine-grained way to examine KEE trade-offs and a multi-axis standard to guide development of more effective and stable methods.Source: §6 · p. 8

Limitations

• Corpus presence is verified against OLMo2’s pretraining corpus, but this does not establish parameter encoding; presence for Qwen2.5 and Llama3 is likely, not verified.Facts present in a corpus may still be weakly encoded or not encoded at all.Source: Limitations · p. 9
• The benchmark uses a March 2022 Wikipedia dump, covers models whose pretraining post-dates that cutoff, and evaluates six models and two representative techniques per KEE category.The authors say these constraints limit transferability and leave broader model and technique coverage for future work.Source: Limitations · p. 9
• The benchmark contains single-answer factual triplets from Wikipedia, excluding ambiguous multi-answer and non-English knowledge.The authors identify extending the benchmark to these settings as future work.Source: Limitations · p. 9

A.1 Domain Classification Prompt

• The benchmark organizes source entities into ten categories and classifies triplets into ten domains using a shared LLM-based relevance-classifier prompt template.The template varies the domain name and in-domain entity scope.Source: A Benchmark Construction Details · p. 11, §A.1 · p. 11

A.2 Answer Cardinality Filtering Prompt

• An LLM-based cardinality classifier labels candidates SINGLE, MULTIPLE, or UNSURE, and the benchmark retains only SINGLE triplets.The system prompt and few-shot demonstrations are shown in Figure 4 and Table 6, respectively.Source: §A.2 · p. 11

A.3 Benchmark Statistics

• Benchmark statistics are presented in Figure 5, but the supplied passage does not report their values.Source: §A.3 · p. 11

B Experiments Details

• Unless otherwise noted, experiment hyperparameters follow each method’s original-paper recommendations, and Figure 4 presents the answer-cardinality classifier’s system prompt.Source: B Experiments Details · p. 11, Figure 4 · p. 11

B.1 Decoding Methods

• DeCoRe uses 10 retrieval heads; Activation Decoding sets its information layer at 81.25% of model depth and uses α = 0.5.The Activation Decoding layer position follows the original 26th-layer setting for a 32-layer model.Source: §B.1 · p. 11

B.2 Steering Methods

• ITI uses 48 heads selected with TruthfulQA training and validation splits, then applies an intervention strength of α=15 at inference.Source: §B.2 · pp. 11–12
• Table 4 lists benchmark categories with representative Wikidata instance-of (P31) types.Source: Table 4 · p. 12
• CAA selects an intervention layer by ground-truth first-token probability change, using two folds so each fact’s steering vector and layer are derived without that fact.Each fold uses 80% of its pool to construct the vector and 20% to select the layer; the steering multiplier is 1.0.Source: §B.2 · p. 12

B.3 Prompting Methods

• RECITE runs its recitation and answer-generation stages 20 times and selects by majority vote; GenRead also uses separate background-generation and answer-generation stages.Source: §B.3 · p. 12

C Benchmark Prompt Details

• The baseline uses a generation prompt with five few-shot demonstrations for each short-answer question; Table 5 records each domain name and its substituted scope string.Source: C Benchmark Prompt Details · p. 12, Table 5 · p. 13

D Additional Results

• Full raw-count results for Table 2 are in Table 11; raw-count results for Tables 3 and 12 are in Tables 13 and 14, respectively; Figure 2’s per-model and raw-count breakdowns are in Figure 12 and Table 15.Source: D Additional Results · p. 13

E Use of AI assistants

• AI assistants were used intermittently for template code and sentence rephrasing, but not for complete paper sections or code scripts; authors inspected and verified generated content.Source: E Use of AI assistants · p. 13
• The generation materials include a baseline prompt, five few-shot demonstrations, and staged prompts for RECITE and GenRead.The RECITE Stage 1 system message provides five demonstrations.Source: Table 10 · p. 15, Figure 6 · p. 15, Figure 7 · p. 15
• At K = 20, every model reaches about 90% or more shallow-failure coverage at both sampling temperatures, motivating this balance between coverage and cost.Coverage is measured relative to shallow failures identified with 30 samples; the figure shows faster gains at small K followed by saturation.Source: Figure 11 · p. 17

OLMo2-32B

• Table 15 reports per-model PRS categories for OLMo2-32B unrecovered failures, excluding cases where either method ranks the gold token first.The remaining failures are categorized as improved, tied, or worsened; RECITE uses the most frequent category across its 20 recitation paths.Source: Table 15 · p. 18, Figure 12 · p. 19