"According to ...": Prompting Language Models Improves Quoting from Pre-Training Data

1 Introduction

• The study motivates according-to prompting as a way to encourage quotation from curated pre-training sources, with empirical evidence in current open- and closed-source LLMs.Source: §1 · p. 1
• QUIP-Score measures how much generated text consists of n-gram spans exactly contained in the corpus, using DATA PORTRAIT membership testing.The introduction presents it as an efficient metric for measuring groundedness against pre-training data.Source: §1 · pp. 1–2
• According-to prompts increase corpus overlap by 5–105% while maintaining or improving downstream performance across human-authored prompts and the tested corpora.Experiments include Wikipedia, PubMed, and the U.S. legal tax code, with open- and closed-source LLMs.Source: §1 · p. 2
• Prompts that discourage grounding or target other corpora can reduce Wikipedia overlap and lower performance on tasks relying on Wikipedia content.Source: §1 · p. 2

Memorization in LLMs

• Research on memorization finds that models retain training data, a concern for sensitive or low-quality material but potentially beneficial for trusted, curated corpora.Prior work also measures memorization using n-gram overlap against Google Search results.Source: §2 · p. 2
• DATA PORTRAIT membership testing, including Bloom-filter storage of n-grams, supports fast, lightweight, scalable quotation measurement against large corpora.Source: §2 · p. 2

Hallucination and grounding

• Prior work addresses hallucination through retrieval, decoding, tuning, and training-data analysis; this paper focuses on quotation as a subset of grounding and is complementary to those approaches.The paper's premise is that quoted content is grounded and not hallucinated.Source: §2 · p. 2

Attribution

• Prior attribution work connects to macro-level source attribution here, but this paper does not provide fine-grained attribution to originating documents.Source: §2 · pp. 2–3

LLM Steerability via prompting

• Prior work reports greater steerability in larger and instruction-tuned models; this paper applies that capability to eliciting more quotations from a target corpus.Source: §2 · p. 3

Improving LLMs through prompting

• Prompting research often targets benchmark performance, whereas this work generates answers before explanations and focuses on quoting rather than end-task performance.Source: §2 · p. 3

3 Methodology

• The narrow definition counts only exact quotations, excluding semantic paraphrases; orthographic variation can also cause QUIP-Score to miss quotations.The authors describe QUIP-Score as a lower bound on actual quoting performance.Source: §3 · p. 3, §3.1 · p. 4

3.1 QUIP-Score: Measuring Grounding to Pre-Training Data

• Bloom-filter DATA PORTRAITs enable memory-efficient membership queries and constant-time corpus lookups, whereas scaling conventional n-gram counts to Wikipedia was estimated to require ∼1.5 TB of RAM.This approach supports one-time indexing of a large corpus.Source: §3.1 · p. 3
• QUIP-Score is character n-gram precision: the share of generated text n-grams found in the corpus, macro-averaged across generations.The indicator function returns 1 when an n-gram is in the corpus and 0 otherwise.Source: §3.1 · p. 3
• QUIP scores are corpus-specific and therefore are not comparable across datasets.The authors consider this acceptable for experiments comparing generations against one corpus.Source: §3.1 · p. 3

3.2 Validity of QUIP-Score

• QUIP-Score averages 99.9% on sampled Wikipedia documents and 17.0% ± 0.8% on sampled Pile documents.The authors use these results to show that the metric recognizes full quotations and that random text has approximately 17% QUIP-Score.Source: §3.2 · p. 4
• As QUIP-Score increases, sampled generations show more quoting and fewer hallucinations.The authors report that this analysis empirically demonstrates the metric's desired properties.Source: Table 1 · p. 4, §3.2 · p. 4

4 Grounding via according-to Prompting

• According-to prompting appends a corpus-grounding instruction to a task prompt and compares generations against a null prompt.The experiments measure the change in QUIP-Score.Source: §4 · p. 4
• Anti-grounding prompts either direct models to another source or tell them not to use the target corpus.The experiments test whether models can ground, or avoid grounding, to a given corpus when asked.Source: §4 · p. 4
• The experiments use no retrieval models because they aim to measure what the LLM recalls on its own.Source: §4 · p. 4
• Multiple prompt phrasings test whether gains are consistent rather than artifacts of wording, since prompting is described as brittle.The authors motivate this test by noting that changing phrasing can affect results.Source: §4 · p. 5

4.1 Datasets

• Long-form QA provides enough n-grams to measure grounding; short-form QA prompts request an explanation whose n-grams can also be measured.The experiments check consistency and whether grounding affects task performance.Source: §4.1 · p. 5
• The study targets grounding rather than state-of-the-art task performance, using Wikipedia QA datasets spanning short- and long-form, single- and multi-hop questions.According-to prompting often achieves competitive or improved performance compared with other prompting baselines.Source: §4.1 · p. 5

4.2 Models and Prompting

• Experiments span open- and closed-source models, multiple instruction-tuning variants, and sizes from 220 million to 175B parameters, using prompts without fine-tuning.Source: §4.2 · p. 5
• For models below 15B parameters, short-form QA uses two-step prompting: first the answer, then the explanation with the grounding prompt if applicable.The two-step format addresses smaller models' difficulty following one prompt to provide both outputs in a parseable format.Source: §4.2 · p. 5

5 Results

• On ChatGPT, according-to prompts generally matched or improved QUIP-Score over the null prompt, while anti-grounding prompts scored the same or worse; end-task performance was often similar or better.Reported end-task improvements reached 6% on NQ and 2.5% on HotpotQA; ELI5 ROUGE-L measures similarity to Reddit rather than Wikipedia.Source: §5 · p. 5, Table 2 · p. 6

5.1 Results from Other Models

• Across evaluated models, grounding prompts improved both end-task performance and QUIP-Score by 5–105%; Table 3 reports 5–15% gains in nearly every dataset and metric.Table 3 omits smaller-model EM/F1 scores when grounding and null prompts produced the same answer, and includes GPT-4 only on ELI5.Source: §5.1 · p. 5, Table 3 · p. 6

5.2 Impact of Model Size

• Grounding prompts outperform null prompts on QUIP-Score for larger models, while smaller FLAN-T5 models tie and smaller OpenAI models perform worse.The reported model-size comparison uses FLAN-T5 and OpenAI models.Source: §5.2 · p. 6, Figure 3 · p. 7

5.3 Impact of Entity Popularity

• QUIP-Score rises with entity co-occurrence frequency for both prompt types, with a stronger positive relationship under grounding prompts.The authors report better recall of Wikipedia information when question-answer entities co-occur frequently.Source: §5.3 · p. 6, Figure 2 · p. 7

5.4 Impact of Instruction Tuning

• Instruction tuning increases the grounding-prompt advantage: FLAN-T5’s QUIP-Score is roughly twice as high as its null-prompt score, unlike T5-v1.1-Adapt’s similar scores.Source: §5.4 · p. 7, Figure 4 · p. 7

5.5 Qualitative Examples

• According-to prompts produce larger Wikipedia text chunks, but grounded generations can still answer incorrectly, as both TriviaQA models quoted the wrong Smokey the Bear passage.Source: §5.5 · p. 7

• Across PubMed and tax-code prompts, Table 4 reports more quoting with maintained task performance; on SARA, QUIP-Scores more than triple with a slight performance increase.Source: Table 4 · p. 8, §6 · p. 8
• The PubMed tests cover PubMedQA, MedQA, and MedicationQA; PubMedQA is evaluated closed-book without providing its abstract.Source: §6 · p. 8
• The legal-domain evaluation uses SARA, a natural-language-inference dataset of tax cases.Source: §6 · p. 8
• In the medical domain, grounding to PubMed slightly improves performance and raises QUIP-Scores on all datasets.Source: §6 · p. 9

• Prompting may support provenance grounding across a wide range of task spaces, beyond the tasks considered here.Source: §7 · p. 9
• According-to prompting is orthogonal to retrieval augmentation, and the authors encourage testing the two together.Source: §7 · p. 9

• Across varied domains and corpora, according-to prompts raise QUIP-Score, while anti-grounding prompts reduce it.Source: §8 · p. 9
• QUIP-Score measures the percentage of model generations that appear as exact quotes in the pre-training corpus.Source: §8 · p. 9
• QUIP-Score increases with the popularity of the question’s entity and with model size.Source: §8 · p. 9

9 Limitations

• QUIP-Score is a lower bound on quoting-based grounding, misses non-exact grounded statements, and is DATA PORTRAIT-specific; the metric is compared across prompts using the same portrait.Source: §9 · p. 9
• For private models such as ChatGPT, limited pre-training information leaves possible differences between their training data and the Wikipedia version used for analysis.Source: §9 · p. 9

A Dataset Details

• Dataset choices include grounded KILT ELI5 and full-development NQ and HotpotQA, plus filtered 7k-instance TriviaQA.Source: A Dataset Details · pp. 12–13

B.1 Sources for SOTA Performance in Table 3

• The Table 3 zero-shot baselines come from LLaMA 33B and 65B, PaLM 540B, and BART; retrieval-augmented baselines come from Izacard et al. (2022) or Su et al. (2022), depending on the dataset.Izacard et al. is used for NQ, TriviaQA, and HotpotQA; Su et al. is used for ELI5.Source: §B.1 · p. 13

B.2 Additional Models for According-To vs Null

• Additional model results that did not fit in Table 3 are shown separately.Source: §B.2 · p. 13
• Short-form QA prompts request an answer followed by a semicolon and explanation, with a two-step answer-then-explanation alternative for models that do not follow the combined format.Source: §B.2 · p. 13
• For T5v1.1-Adapt and GPT-J-Instruct on ELI5, adding “Answer:” to both null and grounding prompts avoids very short or nonsensical outputs and yields fluent text.Source: §B.2 · p. 13

C Conventional N-Gram Metrics

• 163 MiB for an optimal Bloom filter versus 11,107 MiB for a typical CPython set or dictionary on a sample of about 100M n-grams; naive storage extrapolates to around 1.5TB for Wikipedia.The Bloom filter uses around 14 bits per element for the chosen parameters; Table 6 compares data-structure sizes.Source: C Conventional N-Gram Metrics · p. 13, Table 6 · p. 14
• The memory measurements cover only one n-gram width; comparing metrics that store multiple widths could further increase memory use.Source: C Conventional N-Gram Metrics · pp. 13–14
• GPT-J and Koala have low EM scores because they fail to produce short answers zero-shot; GPT-4 was run only on ELI5 due to cost.Source: Table 5 · p. 14

D Prompts for Non-Wikipedia Datasets

• For non-Wikipedia tasks, prompts retain the same style but adapt to task format; SARA needs balancing wording, and MedQA’s grounding prompt specifies multiple-choice answers.PubMed uses “According to PubMed,” while MedicationQA appends a grounding prompt without a prompt before the question.Source: D Prompts for Non-Wikipedia Datasets · p. 14

E Computational Resources

• Experiments used OpenAI model APIs or one A100 GPU for local models, and each dataset experiment took approximately less than an hour.Source: E Computational Resources · p. 14