PaperlayerView exact revision on arXiv

Source-linked AI summary

Is Memorization Context-Sensitive? Prefix-Based Extraction Beyond Isolated Prefixes

Ali Satvaty, Narjes Sharafi, Jirui Qi, Suzan Verberne, Fatih Turkmen

arXiv:2610.12085v1cs.AI

TL;DR

The paper asks whether retrieved context mitigates memorization or changes which samples are extractable, and measures target-suffix probability for paired prefix–suffix samples across empty and retrieved contexts. It finds a context-robust core alongside a context-sensitive boundary, so contextual conditioning reshapes exposure rather than simply erasing it.

  • Problem

    It is unclear how contextual conditioning changes prefix-based extraction for individual samples, despite its relevance to prompts containing retrieved documents or instructions.

    Source: §1 · p. 1

  • Method

    The study compares probabilistic target-suffix scores for paired prefix–suffix samples under empty, relevant, and irrelevant contexts across three open-weight instruction-tuned models.

    Source: §1 · pp. 1–2

  • Results

    Many extractable samples remain exposed across contexts, while context mainly shifts marginal samples near the threshold, suppressing some and enabling others.

    Source: §1 · p. 2, §7 · p. 9

  • Takeaways & Limitations

    Context can reduce aggregate extraction without eliminating robust exposures, and prefix-only evaluation misses samples made extractable by context.

    Source: §1 · p. 2, §7 · p. 9

    The experiments cover three open-weight model families and randomly sampled Pile-CC data, not the largest open-weight models, closed APIs, or other domains.

    Source: Limitations · p. 9

Abstract

from arXiv · show

Large language models (LLMs) can expose memorized training sequences under prefix-based extraction: given a prefix from a training example, the model may assign high probability to the original continuation. In deployed systems, however, prefixes are rarely evaluated in isolation. They often appear together with instructions, retrieved documents, or other task-specific context, as in retrieval-augmented generation (RAG). This motivates examining whether contextual conditioning mitigates memorization or merely changes the set of memorized samples that become extractable. We investigate this issue through paired item-level measurements of probabilistic suffix extraction. For each prefix-suffix pair, we score the target suffix under an empty prompt and under retrieved contexts of varying relevance, across three open-weight instruction-tuned models. We find that context does not simply erase memorization. Instead, extractable memorization consists of a context-robust core and a context-sensitive boundary. Many samples that are extractable without context remain extractable under the retrieved context, especially as the prefix length increases. At the same time, context mainly affects marginal samples near the extraction threshold: it suppresses some exposures, but also enables new ones that are missed by prefix-only evaluation. These findings qualify the view that RAG reduces memorization risk. Context can lower aggregate extraction by suppressing boundary cases, yet robustly extractable samples persist, and context-enabled extractability remains security-relevant.

1 Introduction

The paper asks whether retrieved context mitigates prefix-based memorization or changes which samples are extractable. Paired item-level comparisons show that contextual conditioning reshapes exposure: a robust core persists while context-sensitive boundary cases can be suppressed or newly exposed.

1 Introduction

  • The paper examines prefix–suffix extraction, in which a matching training prefix can elicit high probability for its original continuation.This behavior can enable verbatim recovery of memorized text.Source: §1 · p. 1
  • The study compares each prefix–suffix sample under an empty prompt and relevant or irrelevant preceding context.Paired item-level measurements track both aggregate extraction and which samples become extractable.Source: §1 · pp. 1–2
  • Contextual conditioning leaves a context-robust core while shifting a boundary of samples whose exposure changes when context is added.Context suppresses some marginal samples and makes others extractable; many core samples persist, especially with longer prefixes.Source: §1 · p. 2
  • Context can lower aggregate extraction by suppressing boundary cases, but robust exposures persist and context-enabled samples are missed by prefix-only evaluation.The findings qualify the view that retrieval reduces memorization risk.Source: §1 · p. 2

2 Background and Related Work

Prior work studies memorization through prefix-based extraction, while this paper focuses on whether the same target suffix remains probable when surrounding context changes. It uses probabilistic, item-level comparisons to track stable, suppressed, and newly exposed samples.

2.1 Memorization in LLMs

  • Extraction attacks can recover verbatim training examples by prompting a model and searching generated outputs for memorized text (Kim et al., 2023; Carlini et al., 2021).Source: §2.1 · p. 2
  • Verbatim suffix extraction means the model generates the target continuation exactly after receiving its training prefix.The paper focuses on whether the model assigns high probability to that corresponding suffix.Source: §2.1 · p. 2

2.2 Measuring Memorization

  • Greedy-decoding evaluations use a binary exact-match criterion, which can miss suffixes that receive high probability without appearing in a greedy continuation.Source: §2.2 · p. 2
  • Probabilistic extraction measures the target suffix’s probability rather than relying only on an exact decoded match.The continuous per-sample score supports paired comparisons across contexts.Source: §2.2 · p. 2
  • Unlike prior RAG privacy work on entity-level disclosure, this study tracks whether each verbatim prefix–suffix pair remains exposed, is suppressed, or becomes newly exposed.The item-level view shows how aggregate leakage can decrease while a robust subset remains extractable.Source: §2.2 · p. 3

3 Problem Formulation

The formulation tests whether extraction belongs to a prefix–suffix pair alone or depends on surrounding text. It holds the target pair fixed while varying context and distinguishes samples that remain exposed from those whose exposure changes.

3 Problem Formulation

  • The central question is whether memorization depends only on a training prefix–suffix pair or also on retrieved documents preceding the prefix.Source: §3 · p. 3
  • In the standard setting, the model’s likelihood of the target suffix is evaluated conditioned on its prefix.The prefix and suffix come from the same source sequence.Source: §3 · p. 3
  • Contextualized extraction evaluates the same target suffix with both context and prefix, varying the preceding context while keeping the pair fixed.Contexts may be empty, semantically related, or intentionally irrelevant.Source: §3 · p. 3
  • The three conditions are empty context, the nearest retrieved document, and the farthest retrieved document as a low-relevance context.Source: §3 · p. 3
  • The formulation distinguishes a context-robust core exposed under all conditions from a context-sensitive boundary whose exposure depends on context.The boundary samples are concentrated near the extraction threshold.Source: §3 · p. 3

4 Experiments

Experiments compare probabilistic suffix scores for fixed prefix–suffix samples with and without retrieved context. The setup uses three open-weight instruction-tuned models, randomly sampled Pile-CC prefixes, and relevant or irrelevant retrieved documents.

4 Experiments

  • Each sample is scored with an empty context and with semantically related or irrelevant context, changing only the preceding document.Paired item-level comparisons support aggregate and sample-level analysis.Source: §4 · p. 3
  • The evaluation uses Llama-3-8B-Instruct, Falcon-7B-Instruct, and OLMo-2-1124-7B-Instruct.The models are open-weight and instruction-tuned.Source: §4 · p. 4
  • The sample comprises 50,000 randomly selected Pile-CC prefixes at lengths 32, 64, or 128 tokens, each with a 32-token suffix.Prefixes were not filtered or pre-selected for memorization.Source: §4 · p. 4
  • E5 retrieves the most- and least-relevant documents from a 67,655-document collection of StackExchange, PubMed, and Wikipedia text.Source: §4 · p. 4
  • The measurement uses target-suffix likelihood as a graded signal rather than only exact greedy continuation.It compares suffix log-probabilities across contexts and analyzes exposed-set membership at a threshold.Source: §4 · p. 4

5 Results

Retrieved context reshapes rather than erases prefix-based memorization: longer prefixes make suffix likelihoods and exposed sets more stable, while context shifts mainly boundary cases. The results also separate generic context effects from the additional semantic effect of relevant retrieval.

5 Results

  • Retrieved context reshapes rather than erases prefix-based memorization, with a persistent core and a context-sensitive exposure boundary.Source: §5 · p. 4

5.1 Observation 1: Longer prefixes make memorization increasingly context-independent

5.2 Observation 2: Strongly exposed samples persist across contexts, while weaker exposures are context-sensitive

  • The dominant all-context exposure core grows with prefix length, while context-induced flips cluster near the exposure threshold and become rarer as prefixes lengthen.Source: §5.2 · p. 6

5.3 Observation 3: Context mostly suppresses, but context-enabled extraction remains security-relevant

  • Context generally suppresses more samples than it enables, yet robust exposures persist and context-enabled exposures remain nonzero.Source: §5.3 · p. 6, Figure 3 · p. 7

5.4 Observation 4: Relevant context adds a semantic effect, but generic context also matters

  • The semantic effect is generally negative, especially for shorter prefixes, while the generic effect of adding irrelevant context varies by model.Source: Table 1 · p. 7, §5.4 · p. 7

5.5 Ablation analysis

  • Under instruction framing, relevant–irrelevant exposure overlap rises with prefix length, while chat templates lower measured memorization under a changed response-formatting regime.Source: Figure 4 · p. 8, §5.5 · p. 8

6 Discussion

The discussion separates probabilistic memorization from observed leakage and argues that retrieval does not eliminate extractability. Its controlled retrieval-and-prompting operation is also used in real-world RAG systems.

6 Discussion

  • The study’s controlled operation—adding a retrieved document to an instruction-tuned model’s prompt—is also used by real-world RAG systems.The authors state that production components operate around this core operation rather than replacing it.Source: §6 · p. 8
  • Probabilistic memorization measures a model’s tendency to assign probability to a memorized continuation, while observed leakage depends on the attack regime.The authors use this measure to compare conditions without conflating them with leakage rates under particular attacks.Source: §6 · p. 8
  • Items whose memorization persists in both plain-LLM and RAG conditions remain equally leakable operationally, although observed leakage rates depend on the attack model.Source: §6 · p. 8

7 Conclusion

Across three instruction-tuned models, contextual conditioning leaves many extractable samples exposed, particularly with longer prefixes. Contextual changes cluster near the exposure threshold, so retrieval can suppress some boundary cases while enabling others.

7 Conclusion

  • Across three instruction-tuned models, many samples remain exposed under related and irrelevant contexts, especially as prefix length increases.Source: §7 · p. 9
  • Context-induced changes concentrate near the exposure threshold, while samples farther from it are largely stable.Source: §7 · p. 9
  • Context can reduce aggregate extraction by suppressing boundary cases, but robust exposures persist and some samples become extractable only after context is added.The authors recommend extending the analyses to additional datasets and experimental settings.Source: §7 · p. 9

Limitations

The experiments cover three approximately 10B-scale open-weight model families and a random Pile-CC subset, with suffix length fixed at 32 tokens. The authors identify model, domain, retriever, and suffix-length coverage as boundaries on the study.

Limitations

  • The study tests Llama-3, OLMO-2, and Falcon at approximately 10B parameters on a randomly sampled CommonCrawl subset of Pile-CC.It does not evaluate the largest open-weight models or closed APIs; domains such as code, biomedical text, or creative writing may have different absolute memorization rates.Source: Limitations · p. 9
  • Scores use temperature T = 1 and top-p = 1, and retrieval results use E5 with nearest and farthest retrieval.Preliminary tests suggest other lexical or dense retrievers have limited effect, but a broader retriever sweep remains future work.Source: Limitations · p. 9
  • The evaluation uses prefix lengths of 32, 64, and 128 tokens with suffix length fixed at 32.Longer suffixes may produce different quantitative patterns, so a systematic suffix-length sweep is left to future work.Source: Limitations · p. 9

Ethical Considerations

The paper limits its ethical analysis to public Pile-CC prefixes and reports aggregate results. Its threshold-sensitivity analysis finds a stable exposure-set pattern, while flips cluster among samples near the exposure boundary.

Ethical Considerations

  • The study uses public Pile-CC prefixes, not private or proprietary training data, closed APIs, or deployed systems.It reports aggregate metrics, includes only a limited set of illustrative public-text examples, and does not release all per-example extraction outputs.Source: Ethical Considerations · p. 9

A Threshold Choice and Sensitivity Analysis

B Flip Analysis

  • Samples close to the exposure threshold have the highest context-induced flip rates, while samples far from it almost never flip.This indicates that context primarily affects marginal rather than strongly exposed or strongly non-exposed samples.Source: B Flip Analysis · pp. 11–12

C Exposure Set Analysis

The appendix examines how contextual conditions, instruction framing, and chat-template formatting affect exposed-set overlap and measured extraction. Longer prefixes make exposure more stable across contexts, while context changes concentrate near the exposure boundary.

C Exposure Set Analysis

  • The analysis uses pairwise Jaccard overlap to compare exposed sets under empty, relevant, and irrelevant contexts at τ = −16.Table 2 reports this comparison.Source: C Exposure Set Analysis · p. 12
  • Across models, longer prefixes substantially increase overlap between exposed sets, making extraction more consistent across contextual conditions.The exposure-boundary analysis also finds that context-induced flips are most common near the threshold and approach zero far from it.Source: C Exposure Set Analysis · p. 13, Figure 6 · p. 13

D Instruction-Framed Contextualization

  • As prefix length increases from 32 to 128, relevant–irrelevant overlap rises from 0.69 to 0.93 for Llama, 0.69 to 0.92 for OLMo, and 0.74 to 0.96 for Falcon.The shared exposed category dominates and grows substantially with prefix length, while relevant-only and irrelevant-only categories remain comparatively small.Source: D Instruction-Framed Contextualization · p. 13

E Effect of Chat Templates

  • Chat-template formatting substantially lowers measured memorization across models and prefix lengths, so the core analysis excludes it to isolate preceding-context effects.The template inserts an assistant-role boundary, changing the scored continuation from the suffix after the prefix to the suffix after an assistant tag.Source: E Effect of Chat Templates · p. 14

G.1 Semantic Relevance Effect

  • Relevant context is often suppressive, especially for shorter prefixes, while the effect of irrelevant context varies by model and can be neutral or increase extraction.The suppressive semantic effect weakens as prefix length increases.Source: §G.1 · p. 14

G.2 Relevant versus Irrelevant Context

  • At the item level, points near the relevant–irrelevant context diagonal indicate similar extraction likelihood; deviations indicate sensitivity to retrieval choice.Under instruction-framed prompting, relevant and irrelevant exposed sets still share a growing robust core as prefixes lengthen.Source: §G.2 · p. 14, Figure 7 · p. 15

H Examples of Recalled and Forgotten Samples in Presence of Context

The appendix illustrates both forgotten and recalled samples around the extraction threshold, alongside comparisons of context effects and scoring formats. Semantic relevance is often suppressive, while chat-template formatting changes the measured continuation.

H Examples of Recalled and Forgotten Samples in Presence of Context

  • Chat-template scoring inserts an assistant-role boundary before the suffix and substantially lowers measured memorization across models and prefix lengths.This scores a different continuation than direct prefix–suffix scoring.Source: Figure 9 · p. 17, Figure 10 · p. 17
  • Tables 5 and 6 show forgotten and recalled examples, each selected to straddle the −16 threshold with a log-probability gap of at least 5.Each table gives one example per model: forgotten items cease to be extractable with retrieved context, while recalled items become extractable.Source: H Examples of Recalled and Forgotten… · p. 15, Table 5 · p. 17, Table 6 · p. 22
  • Relevant context is often suppressive, especially for shorter prefixes, while the generic effect of irrelevant context varies by model.With grounding instructions, the semantic effect remains generally negative but is less pronounced than without them.Source: Table 4 · p. 16, Figure 11 · p. 18
  • In the item-level plot, proximity to the diagonal marks similar suffix log-probabilities under relevant and irrelevant context; deviations mark context-sensitive samples.The axes are log-probability under relevant context (x) and irrelevant context (y).Source: Figure 12 · p. 19
Loading 2610.12085v1…