PaperlayerView exact revision on arXiv

Source-linked AI summary

LLMs Can Leak Training Data But Do They Want To? A Propensity-Aware Evaluation of Memorization in LLMs

Gianluca Barmina, Peter Schneider-Kamp, Lukas Galke Poech

arXiv:2606.06286v1cs.CLcs.AI

TL;DR

Existing memorization evaluations emphasize whether models can be forced to reproduce training data, leaving ordinary-use leakage propensity less directly assessed. The paper introduces PropMe and SimpleTrace to compare adversarial capability with non-adversarial propensity, finding substantially stronger memorization under prefix attacks but low propensity overall.

  • Problem

    Existing memorization evaluations primarily test adversarial extractability, leaving models’ propensity to reproduce training data under ordinary use less directly assessed.

  • Method

    PropMe compares generic or dataset-specific prompts with prefix attacks, while SimpleTrace deterministically attributes generations to large-scale training corpora using verbatim and near-verbatim metrics.

  • Results

    Prefix attacks consistently elicit stronger memorization than generic or specific prompts, while propensity remains low and DFM Decoder shows weaker Common Pile memorization than Comma.

  • Takeaways & Limitations

    Memorization audits should report both capability and propensity because worst-case extractability and ordinary leakage risk capture different aspects of model behavior.

  • Takeaways & Limitations

    The evaluation depends on access to full training corpora and covers only a narrow set of model families and languages.

Abstract

from arXiv · show

Large language models can reproduce training data, but existing memorization evaluations mostly measure whether models can be forced to do so, rather than whether they do so under ordinary use. We introduce PropMe, a propensity-aware framework for memorization evaluation that contrasts prefix-based capability attacks with non-adversarial evaluations. We propose a metric transformation that, applied to existing functions, allows to create propensity metrics. We further introduce SimpleTrace, a lightweight tracing pipeline built on infini-gram that deterministically attributes model generations to large-scale training corpora and computes verbatim, near-verbatim, and propensity-transformed memorization metrics. Evaluating two fully-open models: Comma and DFM Decoder on two datasets: Common Pile and Dynaword in two languages, we find a consistent gap between capability and propensity: prefix attacks elicit substantially stronger memorization signals than generic or dataset-specific prompts, while propensity scores remain low overall. Thus, the models can reveal training data when directly elicited, but rarely do so in more common non-adversarial settings. We also find that DFM Decoder, which is continually pre-trained from Comma, exhibits reduced memorization and memorization propensity for Common Pile, confirming that memorization capability can decrease when later training emphasizes partially different data. Our results suggest, and we encourage, that memorization audits should report both worst-case extractability and ordinary leakage propensity in order to have a more comprehensive view of this phenomenon.

1 Introduction

The paper frames memorization auditing as a distinction between capability under adversarial attacks and propensity under ordinary use, motivated by leakage and regulatory risks. It introduces PROPME and SIMPLETRACE to evaluate, transform, and attribute memorization across models, datasets, and languages.

  • Motivation: Existing memorization research emphasizes adversarial attacks, including membership inference, prefix attacks, and resource-referencing prompts.Prior work has documented regeneration of copyrighted books and sensitive personal identifiers, motivating closer study of memorization.
  • Motivation: Assessing ordinary-use propensity can provide evidence of foreseeable training-data leakage risks relevant to GDPR and EU AI Act obligations.The cited obligations include confidentiality, accountability, data protection by design, risk management, robustness, cybersecurity, and systemic-risk evaluation.
  • PROPME: PROPME evaluates memorization across three levels, spanning generic propensity-focused prompts to prefix attacks focused on capability.The framework is designed to compare willingness to leak data under real-world prompting with leakage elicited by adversarial attacks.
  • SIMPLETRACE: SIMPLETRACE is an open-source, lightweight tool that traces model outputs to source documents in large-scale training data.The tool is built on infini-gram and supports attribution of potentially memorized sequences to training-set documents.
  • PROPME: A novel transformation converts standard memorization metrics into propensity metrics conditioned on adversarial and non-adversarial settings.This enables measurement of a model’s propensity to leak training data rather than only its memorization capability.
  • Experimental scope: The evaluation covers two models, Comma and DFM Decoder, across public permissibly licensed datasets in English and Danish.DFM Decoder is continually trained from the monolingual English Comma model on a lower-resource language, enabling study of continual pre-training effects.

2 Related Work

Prior memorization research spans different model types and measurement methods, while recent work distinguishes what models can exhibit from what they tend to exhibit. This work applies that capability–propensity distinction to memorization and builds on text-comparison metrics and infini-gram-based tracing.

  • Memorization: Memorization research covers closed or commercial models and open models, using model-internal measurements or comparisons with external texts.
  • Memorization: Existing work broadly addresses detection of whether sequences were seen during training and extraction of training sequences through model outputs.
  • Propensity vs Capability in Large Language Models: Propensity-aware evaluations distinguish behaviours models can exhibit when elicited from behaviours they tend to exhibit under realistic deployment conditions.
  • Memorization Metrics Based on Text/Token Comparison: Verbatim memorization length declares a sequence memorized when at least 32 tokens are reproduced exactly from a prefix of at most 32 tokens.
  • Tracing Training Set Data: Infini-gram scales n-gram counting and probability estimation to trillions of tokens, enabling OLMoTrace to trace generations to large training corpora.SIMPLETRACE adapts this infrastructure for offline, systematic large-scale analysis and adds indexing, unigram precomputation, parallel batch processing, and metric aggregation.

3 Proposed Method: Propensity-Aware Memorization Evaluation

PropMe evaluates memorization by contrasting ordinary prompting with targeted prefix attacks, then transforms existing metrics to quantify propensity separately from capability. SimpleTrace deterministically traces generations to large corpora through infini-gram-based span extraction, retrieval, merging, and aggregation.

  • Propensity-aware evaluation: PropMe contrasts realistic, low-overlap prompts with prefix attacks that directly condition models on training-data prefixes.The first setting measures reproduction under ordinary use, while the second maximally elicits memorization through targeted intervention.
  • Propensity-aware evaluation: The propensity-aware transformation combines propensity and capability evaluations to distinguish spontaneous behavior from behavior revealed only under adversarial elicitation.High capability with low propensity produces a low propensity score, whereas low capability with high propensity produces a high score.
  • Propensity-aware evaluation: Existing memorization metrics become propensity memorization metrics through the proposed transformation, which measures tendency under standard settings rather than manifestation degree.The paper explicitly distinguishes propensity degree from behavior degree and applies the transformation to memorization metrics in Section 4.
  • SimpleTrace pipeline: SimpleTrace uses infini-gram suffix-array queries to extract maximal verbatim spans, filter rare spans, retrieve matching documents, merge regions, and aggregate memorization statistics.The pipeline is augmented with multi-worker parallelization and metrics aggregation, producing over 30 summary fields.
  • SimpleTrace validation: SimpleTrace achieves perfect retrieval and exact-match results on Dynaword and near-perfect source-document retrieval with exact span recovery on Common Pile queries.It also recovers every Common Pile full document perfectly; 100 queries take approximately 1 minute for Common Pile and approximately 10 seconds for Dynaword using 4 CPU cores.

4 Experimental Setup

The experiments index two openly licensed datasets and compare generic, specific, and prefix prompts under deterministic evaluation. Memorization is measured with four SimpleTrace metrics, including propensity-aware variants designed to distinguish ordinary prompting from adversarial extraction.

  • Datasets: Common Pile uses the 521 GB Comma v0.1 corpus with 463.6B tokens, while Danish Dynaword contains 5.66M samples and 6.83B Llama 3 tokens.Common Pile was indexed across three balanced shards; Dynaword comprises 10.5 GB of data, and both datasets contain exclusively open, permissibly licensed data.
  • Prompt settings: Each dataset has 100 Generic, Specific, and Prefix prompts, with the first two targeting memorization propensity and Prefix targeting memorization capability through 50-token training-example prefixes.Generic and Specific prompts are plausible and naturally phrased with low expected training overlap; Prefix prompts use random training examples of at least 100 tokens.
  • Prompt validation: SimpleTrace validation finds that Generic and Specific prompts have substantially lower training-data overlap than Prefix prompts, while Specific overlap exceeds Generic overlap.These non-adversarial sets are therefore used to measure reproduction propensity under realistic, non-targeted prompting.
  • Metrics: SimpleTrace computes average longest span length, generations full matches ratio, average nv-recall, and average near-verbatim recall to quantify memorization.Average longest span length measures the mean longest verbatim span; generations full matches ratio measures the fraction of generations fully matched by a retrieved training document.
  • Metrics: The propensity-aware transformation is applied to generations full matches ratio and average nv-recall, producing variants that characterize memorization under ordinary and adversarial prompting.Generations full matches ratio is also abbreviated FMR or full matches ratio, while average longest span is ALS and average nv-recall is NVR or nv-recall.

5 Results

Prefix attacks consistently elicit much stronger memorization than generic or specific prompts, while propensity scores remain low overall. Memorization profiles differ across corpora, and DFM Decoder’s Common Pile memorization is lower than Comma’s but remains stable across its training stages.

  • Prompt settings: For Comma on Common Pile, ALS is 50.35 tokens under prefix attacks versus 27.95 generic and 29.47 specific, while specific and prefix FMR both equal 0.02.Non-adversarial memorization is therefore non-negligible despite weaker average spans than prefix attacks.
  • Prompt settings: Prefix attacks dominate memorization: DFM Decoder on Dynaword reaches NVR 0.036 versus 0.001 for both generic and specific prompts, a 36× difference.FMR is 0.07 under prefix prompting, versus 0.00 for generic and 0.01 for specific prompts.
  • Corpus profiles: Common Pile produces longer DFM Decoder verbatim spans than Dynaword, while Dynaword shows stronger prefix full-generation memorization with FMR 0.07 versus 0 for Common Pile.Common Pile ALS ranges from 23.57–40.83 tokens, compared with 15.68–24.75 for Dynaword.
  • Model comparison: On Common Pile, Comma exceeds DFM Decoder in ALS under generic prompts, 27.95 versus 23.57 tokens, and prefix attacks, 50.35 versus 40.83 tokens.Comma alone has non-zero FMR, equaling 0.02 under specific prompts and prefix attacks; DFM Decoder remains at 0.
  • Propensity: Propensity scores remain below neutral 0.5: DFM Decoder on Dynaword has PMNVR 0.026 generic and 0.018 specific, with PMFMR at most 0.125.For Common Pile, DFM Decoder’s specific PMNVR is 0.281, indicating targeted non-adversarial prompts recover some prefix-elicited signal but remain below neutral.
  • Training stages: Across DFM Decoder’s training stages, memorization profiles remain essentially unchanged for both corpora, with Dynaword ALS fixed at 15.68, 17.37, and 24.75 tokens.Propensity scores are similarly flat, with generic PMnv-recall ranging from 0.023 to 0.027 across stages.

6 Discussion

The discussion distinguishes memorization capability from propensity: prefix attacks elicit substantially more memorization than ordinary prompts, while propensity remains low overall. It also finds that later training on partially different data can reduce accessible memorization.

  • Capability versus propensity: Prefix attacks consistently elicit stronger near-verbatim recall, more full-generation matches, and longer verbatim spans than generic or specific prompts.The models can reproduce training data when directly conditioned on it, but this behavior is much less likely under ordinary prompting.
  • Capability versus propensity: Propensity is overall low across datasets and models, so models rarely reveal memorized data in common non-adversarial settings.This suggests memorization capability alone overstates practical leakage risk.
  • Capability versus propensity: Low propensity does not imply absent memorization: specific prompts can still recover memorized content, so propensity evaluation should complement adversarial extraction tests.The discussion presents propensity evaluation as complementary rather than a replacement for capability-oriented audits.
  • Effects of continued training: DFM Decoder shows weaker memorization of Common Pile than Comma, indicating that training on partially different data can reduce accessible memorized content.Memorization remains comparatively stable across later DFM training stages.

7 Conclusion

The paper introduces PropMe and SimpleTrace to evaluate memorization propensity and attribute model generations across multiple matching levels. Experiments show that adversarial prompts elicit much stronger memorization than ordinary prompts, while later training on partially different data can reduce access to earlier memorized data.

  • Contributions: PropMe measures memorization propensity by comparing ordinary prompting with adversarial capability settings.SimpleTrace complements the framework by attributing generations to large-scale training data.
  • Contributions: SimpleTrace enables memorization analysis using verbatim, near-verbatim, and full-generation matches against large-scale training data.
  • Findings: Memorization is much stronger under prefix-based capability evaluations than under non-adversarial propensity evaluations, although models rarely reveal training data in common prompting conditions.Training on a partially different corpus can also reduce accessible memorization of earlier data.

8 Limitations

The study’s full-corpus comparisons provide high measurement accuracy but limit applicability when training data is unavailable. Its experiments also cover a narrow set of models and languages, motivating broader evaluation across architectures and multilingual settings.

  • Data access: Full training-corpus access improves measurement accuracy but limits applicability to models whose training data is not publicly available.The framework can still be combined with logit-, weight-, or probability-based memorization methods when corpus access is unavailable.
  • Method scope: The propensity transformation and PropMe framework are architecture-agnostic and compatible with alternative memorization methods.These include logit-, weight-, and probability-based methods for settings without training-data access.
  • Evaluation scope: The experiments cover one model family, four checkpoints derived from two base models, and two languages.Three checkpoints are continual pre-trainings of the fourth; broader architectures and additional languages could clarify architectural and multilingual effects on memorization propensity.

9 Ethical Considerations

The study uses research models trained exclusively on open, permissibly licensed data and releases SIMPLETRACE openly to support transparent, reproducible research. Its findings also motivate evaluating both adversarial memorization capability and ordinary prompting propensity.

  • All experiments use models trained exclusively on open, permissibly licensed data and intended for research use.
  • Adversarial elicitation can surface memorized content even when propensity under ordinary prompting is low.
  • The findings underscore the importance of evaluating memorization capability alongside propensity.
  • SIMPLETRACE is released as open-source to support transparent and reproducible research.

A Validation of SIMPLETRACE … C.1 Memorization of Dynaword Across Training Stages

SIMPLETRACE passes controlled and corpus-level validation, achieving near-perfect or perfect recovery on Common Pile and Dynaword. Across DFM Decoder’s Dynaword training stages, memorization metrics and propensity scores remain essentially stable, with low propensity under ordinary prompts.

  • A Validation of SIMPLETRACE: Controlled unit tests verify exact-span recovery, cross-document attribution, negative cases, full-document matching, summary statistics, and exported metadata.Together, these tests validate the tracing pipeline and its aggregate outputs.
  • A Validation of SIMPLETRACE: Validation uses 100 queries per corpus from 25 sampled documents, combining full-document queries with start-, middle-, and end-anchored 128-token partial queries.A query passes when SIMPLETRACE retrieves the source document or finds an exact span covering the query text.
  • A Validation of SIMPLETRACE: Common Pile reaches 0.99 source-document retrieval, 0.99 exact-text match, and a 1.00 overall pass rate across all 100 queries.Full-document, middle, and end partial queries recover perfectly; start partial queries return 0.96 for both source-document retrieval and exact-text match.
  • A Validation of SIMPLETRACE: Common Pile’s sole source-document miss still has a partial-span exact-query match rate of 1.00, indicating ambiguous attribution rather than failed text tracing.The missed originating document ID occurred despite an exact span match being found elsewhere in the corpus.
  • A Validation of SIMPLETRACE: Dynaword achieves 1.00 source-document retrieval and 1.00 exact-text match across all 100 queries, including full-document and every partial-query setting.No missing-document cases were logged in the sampled Dynaword validation set.
  • B Prompt Validation: Overlapping metrics increase across prompt settings, distinguishing propensity-oriented evaluations from capability-oriented prefix attacks.The full prompt-validation results are reported using overlapping metrics in Figure 2.
  • C Memorization Across Training Stages: Memorization profiles are essentially stable across DFM Decoder’s three Dynaword training stages, with nearly identical metrics within generic, specific, and prefix prompts.ALS is unchanged across stages at 15.68, 17.37, and 24.75 tokens for generic, specific, and prefix prompts, respectively.
  • C.1 Memorization of Dynaword Across Training Stages: Dynaword propensity remains low across stages: generic NVR propensity ranges from 0.023 to 0.027, while specific FMR propensity stabilises at 0.125 from Stage 2 onward.Both values are below the neutral score of 0.5; prefix attacks also produce broader matched-span distributions than non-adversarial prompts.

C.2 Exp. 4: Memorization of Common Pile Across Training Stages … E Generating Prompt Settings

Across training stages, Common Pile memorization and propensity remain stable, while prefix attacks produce deeper and broader memorized spans than generic or specific prompts. The prompt-generation procedures define generic prompts from domains and specific prompts from dataset context without extracting prompts directly.

  • C.2 Exp. 4: Memorization of Common Pile Across Training Stages: Common Pile memorization remains stable across Stage 1, Stage 2, and the Final DFM Decoder model.ALS is identical within each prompt setting, while NVR remains nearly flat and FMR is 0.00 across stages and settings.
  • C.2 Exp. 4: Memorization of Common Pile Across Training Stages: Common Pile span-length distributions are unchanged across training stages, with prefix attacks retaining a broader tail toward longer spans.Generic and specific prompts concentrate mainly in the (7-10) and (11-20) token buckets, whereas prefix attacks add mass in the (21-50) and (51-100) buckets.
  • C.2 Exp. 4: Memorization of Common Pile Across Training Stages: Generic NVR propensity remains around 0.013 and specific NVR propensity around 0.28 across stages, while full-match propensity stays at 0.00.Both propensity values remain below the neutral score of 0.5, indicating low propensity to reproduce Common Pile content in non-adversarial conditions.
  • D.1 Dynaword Span Lengths in DFM Decoder: Under generic and specific prompts, Dynaword matched spans concentrate in the short (11–20) token bucket, whereas prefix attacks increase mass in longer buckets.Prefix attacks also introduce a small non-zero mass in the longest bucket, confirming increased depth of memorized reproduction.
  • D.2 Common Pile Span Lengths in Comma Model: Approximately 23% of Common Pile spans under prefix attacks fall in the (21–50) range, versus 16% for generic and 12% for specific prompts.The prefix setting also has presence in the (51–100) and (151–∞) buckets, which are largely absent from non-adversarial settings.
  • D.3 Common Pile vs. Dynaword Span Lengths (DFM Decoder): Common Pile is shifted toward longer spans than Dynaword across settings, while both corpora show broader distributions under prefix attacks.Dynaword places more mass in shorter buckets, particularly below 10 tokens, reflecting its shorter average document length.
  • D.4 Common Pile Span Lengths: Comma vs. DFM Decoder: Comma shifts more strongly toward longer Common Pile spans than DFM Decoder under prefix attacks.This supports the interpretation that continual pre-training on Dynaword data partially attenuates memorization depth, but not its rate, in DFM Decoder.
  • E Generating Prompt Settings: Generic prompts request sentence starts across specified domains, while specific prompts use dataset and domain context without extracting prompts directly from the dataset.Both settings instruct generation of 10 prompts per domain with varied lengths.

F Additional Memorization Metrics … spans_length_counts_distribution

The paper defines complementary memorization and tracing metrics, spanning extractability, near-verbatim overlap, full-generation matches, threshold exceedance, and matched-span length distributions. SimpleTrace reports these measures at both document and corpus levels.

  • F Additional Memorization Metrics: k-Eidetic memorization flags strings occurring in at most k training examples, while near-duplication, ROUGE-L, and Token Accuracy capture additional overlap signals.Near-duplication uses weighted Jaccard similarity JW(a, b) ≥0.6 between token-frequency vectors.
  • H SIMPLETRACE Metrics: SimpleTrace produces per-document retrieval metrics for traced spans and corpus-level summary metrics aggregated across generations.Identifiers such as document ID lists and output paths are metadata rather than metrics.
  • H SIMPLETRACE Metrics: Near-verbatim recall measures the fraction of generation words reappearing in retrieved documents as sufficiently long, aligned contiguous blocks.Related counts include matched generation words, reference words, candidate words, and missing words.
  • generations_with_n_token_span_ratio: The n-token span metric is the fraction of generations containing at least one traced span of length at least N.Span-length summaries also include total generations, generations with spans, total spans, average longest-span length, and minimum and maximum span lengths.
  • generations_full_normalized_matches_ratio: Full-match metrics count retrieved documents containing the complete generation verbatim, whereas normalized full-match metrics allow light normalization.The framework also reports distinct-document counts and ratios for exact and normalized full matches.
  • unique_full_normalized_matches_ratio: Partial-match metrics count retrieved documents with only partial span overlap, alongside mean, maximum, and non-zero near-verbatim recall statistics.The user-defined nv_recall_threshold flags especially strong near-verbatim matches.
  • generations_above_nv_recall_threshold: Threshold metrics count generations containing retrieved documents whose nv_recall exceeds a user-defined threshold and report the corresponding fraction and document count.The ratio is computed over generations, while docs_above_nv_recall_threshold counts distinct retrieved documents.
  • spans_length_counts_distribution: Span-length distributions provide histograms of retrieved documents grouped by matched-span token length and normalized versions reported as proportions.Figures cover Common Pile and Dynaword in DFM Decoder, plus Common Pile in Comma and DFM Decoder, across generic, specific, and prefix prompts.
Loading 2606.06286v1…