PaperlayerView exact revision on arXiv

Source-linked AI summary

NanoKnow: How to Know What Your Language Model Knows

Lingwei Gu, Nour Jedidi, Jimmy Lin

arXiv:2602.20122v2cs.CLcs.AIcs.IRcs.LG

TL;DR

It remains unclear how LLMs acquire knowledge because pre-training data is often inaccessible. NanoKnow projects Natural Questions and SQuAD onto transparent FineWeb-Edu data to separate supported from unsupported answers and evaluates eight nanochat checkpoints. The results show frequency-dependent recall, partial mitigation from external evidence, complementary parametric and external knowledge, and harm from distractors.

  • Problem

    LLM knowledge can come from entangled parametric and external sources, but inaccessible pre-training data limits systematic analysis of how that knowledge was acquired.

  • Method

    NanoKnow projects Natural Questions and SQuAD onto FineWeb-Edu, using retrieval, answer-string matching, and LLM verification to label supported and unsupported questions.

  • Results

    Across eight nanochat checkpoints, closed-book accuracy rises with answer frequency, external evidence mitigates this dependence, supported answers remain more accurate, and distractors reduce accuracy.

  • Takeaways & Limitations

    NanoKnow enables controlled studies showing that parametric knowledge and external evidence are complementary, while retrieval precision matters because distractors degrade performance.

  • Takeaways & Limitations

    NanoKnow was built on FineWeb-Edu, although its construction methodology can be extended to other open corpora.

Abstract

from arXiv · show

How do large language models (LLMs) know what they know? Answering this question has been difficult because pre-training data is often a "black box" - unknown or inaccessible. The recent release of nanochat - a family of small LLMs with fully open pre-training data - addresses this as it provides a transparent view into where a model's parametric knowledge comes from. Towards the goal of understanding how knowledge is encoded by LLMs, we release NanoKnow, a benchmark dataset that partitions questions from Natural Questions and SQuAD into splits based on whether their answers are present in nanochat's pre-training corpus. Using these splits, we can now properly disentangle the sources of knowledge that LLMs rely on when producing an output. To demonstrate NanoKnow's utility, we conduct experiments using eight nanochat checkpoints. Our findings show: (1) closed-book accuracy is strongly influenced by answer frequency in the pre-training data, (2) providing external evidence can mitigate this frequency dependence, (3) even with external evidence, models are more accurate when answers were seen during pre-training, demonstrating that parametric and external knowledge are complementary, and (4) non-relevant information is harmful, with accuracy decreasing based on both the position and the number of non-relevant contexts. We release all NanoKnow artifacts at https://github.com/castorini/NanoKnow.

1 Introduction

NanoKnow makes it possible to disentangle how LLM knowledge is acquired by partitioning questions according to whether their answers appeared in transparent pre-training data. Experiments show that answer frequency, external evidence, and distractors shape nanochat’s accuracy.

  • Motivation: NanoKnow addresses the difficulty of tracing LLM knowledge by using transparent pre-training data to distinguish seen answers from unseen answers.The resource supports controlled evaluation of parametric and external knowledge sources.
  • Dataset: NanoKnow partitions Natural Questions and SQuAD into supported and unsupported splits based on whether answers exist in the FineWeb-Edu pre-training corpus.Its pipeline retrieves candidate documents, matches answer strings, and uses LLM verification to remove coincidental matches.
  • Findings: Closed-book accuracy increases with answer frequency in the pre-training corpus, while external evidence mitigates this dependence.The experiments use multiple nanochat checkpoints and replicate related findings across the literature.
  • Findings: Even with the oracle answer document, nanochat is more accurate on supported than unsupported questions, showing that parametric and external knowledge complement each other.External evidence does not fully eliminate the advantage associated with having seen the answer during pre-training.
  • Findings: Non-relevant documents reduce accuracy, with the decline depending on both distractor position and distractor count.The answer document’s position relative to distractors matters, and more distractors further harm performance.

2 NanoKnow

NanoKnow projects NQ and SQuAD questions onto FineWeb-Edu, separating questions with relevant answer evidence from those without it. Its pipeline combines retrieval, answer-string matching, and LLM verification, producing reproducible relevance judgments and released artifacts.

  • Dataset and splits: NanoKnow partitions NQ and SQuAD into supported and unsupported questions based on whether answers appear in relevant FineWeb-Edu contexts.Supported questions pass both string matching and LLM verification; unsupported questions lack a match or have only unrelated matches.
  • Projection Pipeline: The projection pipeline retrieves the top 100 BM25 candidates, checks for the gold answer string, and uses LLM verification to remove coincidental matches.Verification examines a 512-word context window around each answer-string match.
  • Projection Results: 66.2% of NQ questions and 70.9% of SQuAD questions were confirmed supported after String Match followed by LLM Verify.String matching alone identified 73.9% of NQ and 78.9% of SQuAD questions, with verification filtering coincidental matches.
  • Projection Results: Closed-book exact-match accuracy on unsupported questions was 0.8% for NQ and 1.5% for SQuAD using nanochat d32.This validation supports the accuracy of the unsupported labels.
  • Released Artifacts: NanoKnow releases qrels, unsupported questions, a 326GB Lucene index, evaluation code, and artifact access through its project repository.Qrels link supported questions to answer documents, while unsupported questions are provided with their question text.

3 Experimental Setup

The experiments use multiple nanochat checkpoints across three model sizes and compare closed-book prompting with supplied FineWeb-Edu or original SQuAD contexts. Accuracy is measured using exact match and an LLM-based judge.

  • Research Questions: The study examines answer-frequency effects, external evidence, supported-versus-unsupported questions, and distractor effects on QA accuracy.These questions define the paper’s experimental investigation of how pre-training data and context affect model responses.
  • Models: Experiments use nanochat models d20, d32, and d34, ranging from approximately 561M to 2.2B parameters, with multiple checkpoints per scale.Multiple checkpoints are used to assess robustness to training variation.
  • Prompting Setups: The prompting setups are Closed-Book, w/ FineWeb Context, and, for SQuAD, w/ Original Context.FineWeb Context supplies the oracle answer passage, while Original Context supplies the original SQuAD answer context.
  • Evaluation: Accuracy is evaluated with exact match and an LLM-Judge that classifies outputs against predefined correct answers.Exact match uses standard benchmark scripts, while Qwen3-14B performs the judge classification.

4 Results

NanoKnow experiments show that answer frequency, model scale, evidence quality, and distractor placement all shape nanochat’s question-answering accuracy. External context reduces frequency dependence but does not eliminate the advantage of answers present during pre-training.

  • 4.1 Impact of Answer Frequency in Pre-training: Accuracy more than doubled from rare to high answer-frequency questions in closed-book QA, except for the smallest d20 checkpoint.The frequency effect appeared on both NQ and SQuAD; d20 did not show the same pattern.
  • 4.1 Impact of Answer Frequency in Pre-training: External evidence reduced, but did not remove, the relationship between answer frequency and accuracy.Open-book accuracy still generally increased with answer frequency, but the improvement rate was much lower, especially on SQuAD.
  • 4.2 Closed-Book QA vs. Open-Book QA: Larger nanochat checkpoints improved closed-book LLM-Judge accuracy by 26.9 points on NQ and 21.7 points on SQuAD between the best d20 and d34 checkpoints.These changes corresponded to relative improvements of 19.2% and 19.7%, respectively.
  • 4.2 Closed-Book QA vs. Open-Book QA: Providing the oracle FineWeb-Edu answer passage produced large accuracy gains, with average improvements of 2.4×, 2.1×, and 1.9× on NQ across d20, d32, and d34.For SQuAD, the corresponding average improvements were 4.4×, 3.2×, and 2.6×; smaller models benefited more.
  • 4.2 Closed-Book QA vs. Open-Book QA: Open-book QA remained more accurate for supported than unsupported SQuAD questions across all model scales, even with original context.This indicates that external evidence did not fully replace knowledge associated with pre-training exposure.
  • 4.3 Influence of Distractors: A non-relevant context reduced LLM-Judge accuracy by 3.2 points on NQ and 1.5 points on SQuAD relative to closed-book prompting.Adding distractors also consistently reduced accuracy relative to answer-only prompting, with the strongest performance when the answer context was near the question without intervening distractors.

5 Related Work

Related work studies how LLM outputs relate to pre-training data and how parametric and external knowledge interact. NanoKnow addresses the difficulty of disentangling these sources when pre-training data is unknown.

  • Tracing an LLM’s capabilities to its pre-training data: Training-data attribution methods seek pre-training examples that explain a model’s output, commonly using gradients or representations.Other approaches map task-specific data to pre-training data in advance to study how corpus properties relate to downstream capabilities.
  • Tracing an LLM’s capabilities to its pre-training data: Prior work measures how question-and-answer frequency in pre-training data relates to memorization, generalization, and output distributions.These methods connect task-specific entities or outputs to their observed pre-training frequency.
  • Interplay of parametric versus external knowledge: Retrieval-augmented generation supplies external knowledge at inference time to address sparse or obsolete parametric knowledge.Prior studies examine the interaction between knowledge stored in model parameters and retrieved evidence.
  • Interplay of parametric versus external knowledge: Because many studies use LLMs with unknown pre-training data, they cannot confidently disentangle parametric knowledge from external knowledge.NanoKnow provides a benchmark designed to support that controlled analysis.

6 Conclusion

NanoKnow makes it possible to separate knowledge supported by pre-training from unsupported knowledge and examine how frequency, external evidence, and distractors shape model accuracy. The findings show complementary parametric and external knowledge, while extending the methodology beyond FineWeb-Edu remains future work.

  • NanoKnow identifies whether answers to Natural Questions and SQuAD appear in nanochat’s FineWeb-Edu pre-training data, enabling controlled knowledge evaluation.
  • Answer frequency strongly shapes recall: frequently seen answers are recalled more reliably, while rare answers benefit from external evidence.
  • Retrieval-augmented generation improves accuracy where parametric knowledge is weakest, but models remain more accurate when answers were seen during pre-training.
  • Non-relevant contexts reduce accuracy, especially when the answer is buried in the middle, highlighting retrieval precision’s importance in practical RAG systems.
  • NanoKnow was built on FineWeb-Edu, while its methodology can extend to any open corpus and support future studies of training-data effects.
Loading 2602.20122v2…