PaperlayerView exact revision on arXiv

Source-linked AI summary

Extracting alignment data in open models

Federico Barbero, Xiangming Gu, Christopher A. Choquette-Choo, Chawin Sitawarin, Matthew Jagielski, Itay Yona, Petar Veličković, Ilia Shumailov, Jamie Hayes

arXiv:2510.18554v2cs.AI

TL;DR

String matching is poorly aligned with extracting semantic patterns, leaving the scale of memorisation difficult to assess. The paper prompts open-weight models with post-training chat-template tokens and uses embeddings to identify training matches; it reports substantial undercounting by string metrics, recovered model performance from extracted data, and regurgitation of RL samples.

  • Problem

    String-matching measures are poorly suited to extraction focused on semantic patterns and templates, rather than verbatim or near-verbatim text.

    Source: §1 · p. 2, §3.2 · p. 5

  • Method

    The authors prompt open-weight models with post-training chat-template tokens, then use neural embeddings to find semantically similar training samples.

    Source: §3.3 · p. 6, §6 · p. 12

  • Results

    At least an order of magnitude: string-matching methods undercount extraction relative to embedding-based detection; extracted data also recovers parts of original performance and includes RL samples.

    Source: §6 · p. 12

  • Takeaways & Limitations

    Model distillation can function as indirect training-data extraction, putting advantages gained from alignment data at risk of leakage.

    Source: §6 · p. 12

    The attack applies to open models; using it on closed models would require spoofing the behavior of chat-template tokens.

    Source: §6 · p. 12

Abstract

from arXiv · show

In this work, we show that it is possible to extract significant amounts of alignment training data from a post-trained model -- useful to steer the model to improve certain capabilities such as long-context reasoning, safety, instruction following, and maths. While the majority of related work on memorisation has focused on measuring success of training data extraction through string matching, we argue that embedding models are better suited for our specific goals. Distances measured through a high quality embedding model can identify semantic similarities between strings that a different metric such as edit distance will struggle to capture. In fact, in our investigation, approximate string matching would have severely undercounted (by a conservative estimate of $10\times$) the amount of data that can be extracted due to trivial artifacts that deflate the metric. Interestingly, we find that models readily regurgitate training data that was used in post-training phases such as SFT or RL. We show that this data can be then used to train a base model, recovering a meaningful amount of the original performance. We believe our work exposes a possibly overlooked risk towards extracting alignment data. Finally, our work opens up an interesting discussion on the downstream effects of distillation practices: since models seem to be regurgitating aspects of their training set, distillation can therefore be thought of as indirectly training on the model's original dataset.

1. Introduction

The introduction frames alignment data as a valuable asset whose semantic patterns may be extracted from models, especially when string matching is used to measure memorisation. It presents a chat-template prompting attack, reports close matches to alignment data, and notes that the attack depends on access to the template structure.

1. Introduction

  • The paper targets semantic patterns and templates of proprietary data, which string metrics may miss despite their utility for model performance and distillation.The authors argue that existing extraction methods and string-matching metrics are not well aligned to this task.Source: §1 · p. 2
  • Because chat templates are introduced in post-training, the authors hypothesize that prompting with the template induces alignment-data generation, then match outputs to post-training data using embeddings.They generate synthetic samples and find each sample’s closest post-training example by embedding similarity.Source: §1 · p. 2
  • Generated outputs can be much closer to alignment samples than expected; Figure 1’s maths question and answer has embedding similarity 0.97 versus string-match similarity 0.68.The example is described as near identical to an alignment-dataset sample.Source: §1 · p. 2, Figure 1 · p. 3
  • The attack relies on user control over chat-template structure, so it does not immediately apply to closed models.The authors note this control is not normally available for closed models.Source: §1 · p. 2
  • Approximate or verbatim string matching likely undercounts memorisation by at least 10×; embeddings reveal more, and extracted datasets recover some original performance.The authors list these as contributions of the work.Source: §1 · p. 3

2. Nomenclature

The nomenclature broadens alignment data and memorisation beyond narrow post-training and verbatim-match meanings, while specifying related terms used in the paper.

2. Nomenclature

  • Alignment data means targeted collections used to steer behavior or improve capabilities, including SFT, RL, mathematics, reasoning, and long-context understanding.The term is used more broadly than safety and instruction-following data alone.Source: §2 · p. 3
  • Training data covers all model-creation phases and, in this paper, semantically equivalent representations of that data.Source: §2 · p. 3
  • Post-training means stages after initial large-scale pretraining; the paper treats post-training data as a subset of alignment data.The authors note that alignment data is increasingly included earlier in training.Source: §2 · p. 3
  • Memorisation includes regurgitating training data and reproducing its patterns, templates, or semantic structure beyond verbatim matches.The paper uses embeddings to detect approximate memorisation when semantics matter.Source: §2 · p. 4
  • A chat template formats messages with special tokens and is typically introduced during post-training.Source: §2 · p. 4
  • An embedding score measures semantic similarity using the dot product of neural text embeddings, capturing equivalence despite superficial differences.The authors propose it as an alternative to string-matching metrics such as Levenshtein distance.Source: §2 · p. 4

3. Extracting alignment data with neural embeddings

The section uses post-training chat-template prompts to generate alignment-like samples, then compares them with available training data using neural embeddings. It argues that semantic matching better captures close or equivalent samples than string matching, while noting constraints on the data and threshold choices.

3. Extracting alignment data with neural embeddings

  • The pipeline confirms that many generated outputs are verbatim or very close to true training samples under an appropriate similarity measure.It embeds the post-training set, generates samples from repeated prompts, and retrieves each generation’s closest match.Source: §3 · p. 4
  • The pipeline embeds the post-training set, repeatedly prompts the model, and retrieves the closest training-sample match for each generated output.Once the dataset is embedded, retrieval can be implemented as a single matrix multiplication.Source: §3 · p. 4
  • The study requires access to models that disclose their post-training mixtures, so it focuses on OLMo 2 for SFT and Open-Reasoner-Zero for RL.Source: §3 · p. 4

3.1. Chat templates to steer generations

  • Chat-template prompting produces outputs resembling post-training samples, as illustrated by generations from Qwen2.5 7B Instruct.The authors investigate whether this resemblance reflects actual training data in later sections.Source: §3.1 · p. 5, Figure 2 · p. 5

3.2. Measuring aproximate semantic memorisation

  • In the OLMo 2 example, string matching penalizes changed multiple-choice options despite preserved semantics, while the embedding score better aligns with human judgment.The figure reports an embedding score of 0.986 and normalized Levenshtein similarity of 0.699.Source: §3.2 · p. 5, Figure 3 · p. 6

3.3. Measuring memorisation with neural embeddings

  • The method compares plain-text samples using normalized neural embeddings and their dot product, retrieving the highest-scoring training sample.Special tokens are removed before embedding; the experiments use gemini-embedding-001.Source: §3.3 · p. 6
  • The manually selected 0.95 embedding threshold affects measured memorisation, although threshold choice also matters for string-matching measures.The authors describe 0.95 as a conservative choice based on qualitatively similar samples.Source: §3.3 · p. 6

4. Large scale extraction of SFT data

The study tests extraction from OLMo 2’s SFT data using chat-template prompts and compares string matching with neural embeddings. Embedding-based extraction also yields synthetic data that achieves comparable benchmark performance after SFT, except on IFE.

4. Large scale extraction of SFT data

  • OLMo 2’s post-training includes 939k SFT samples, and extraction uses the chat-template starting tokens to prompt generations for matching.The SFT set is one stage of a three-stage post-training mixture; the other stages are DPO and RLVR.Source: §4 · p. 7

4.1. String matching poorly captures memorisation

  • String-matching scores suggest negligible memorisation, yet miss examples that appear clearly memorised because trivial differences deflate similarity.Figure 4 presents histograms of the highest Levenshtein and Indel similarity scores for each generation.Source: §4.1 · p. 7, Figure 4 · p. 7

4.2. Large scale extractions with embeddings

  • Neural embeddings reveal more memorisation than string matching, whose Levenshtein scores are poorly aligned with semantic similarity and biased against longer generations.Figure 5 shows embedding-score distributions and compares embedding scores with Levenshtein distance, with points coloured by string length.Source: §4.2 · p. 8, Figure 5 · p. 8
  • Some post-training samples are more likely to be memorised when similar samples occur in pre- or mid-training, or when the sample appears repeatedly.Figure 6 reports each true sample’s best embedding score across one million generations and cumulative coverage.Source: §4.2 · p. 8, Figure 6 · p. 8
  • Using the full chat template rather than only the beginning-of-sequence token produces generations that are, on average, semantically closer to the post-training set.Table 1 compares mean best embedding scores for the two prefixes.Source: §4.2 · p. 9, Table 1 · p. 9

4.3. Direct distillation on extracted data

  • SFT on extracted synthetic data achieves benchmark performance comparable to the reproduction, except on IFE.The synthetic dataset contains about 930k samples, but 850M tokens versus about 1.3B in the original training set.Source: §4.3 · p. 9, Table 2 · p. 9

5. Large scale extraction of RL data

The paper extracts RL training data by prompting an RL-trained model and matching generations to training samples. Extracted data supports comparable benchmark performance, while RL training also increases the likelihood of many training prompts.

5. Large scale extraction of RL data

  • From 100k generations, the model sometimes regurgitates RL samples verbatim, including a thinking trace absent from the question-and-answer training data.The authors note that this is surprising given the PPO objective’s apparent misalignment with memorising training samples.Source: §5 · p. 10, Figure 7 · p. 10
  • Likelihood-based measurement captures only pointwise memorisation and may miss recall of a question style when the exact training example is absent.The authors describe this as a limitation of measuring training-sample likelihoods.Source: §5 · p. 10
  • RL training increases the likelihood of many training prompts, with some samples rising from 10^-11 to 10^-5.Figure 8 counts training prompts above likelihood thresholds before and after RL; the authors say the mechanism behind the increase is unclear.Source: §5 · p. 11, Figure 8 · p. 11

RL on extracted dataset

  • For the extracted-data experiment, the authors generate 100k samples, filter invalid or incorrect examples, then randomly select 57k for training.The original RL dataset contains questions and answers, while reasoning traces arise during training rollouts.Source: §5 · pp. 10–11
  • A Qwen2.5 7B model trained on 57k extracted RL samples achieves benchmark performance comparable to the baseline across four maths benchmarks.Figure 9 reports AMC, MATH500, Minerva Math, and OlympiadBench results.Source: §5 · p. 11, Figure 9 · p. 11
  • The authors leave improving the extraction pipeline to obtain higher performance and a higher-quality dataset as future work.They frame the work’s focus as training-sample extraction.Source: §5 · p. 12

6. Conclusion

Open-weight models can be prompted with post-training chat templates to regurgitate alignment-like data. Embedding-based evaluation reveals more semantic memorisation than string matching, and extracted data can recover performance when used to train a base model.

6. Conclusion

  • Chat templates and their special tokens, introduced during post-training, serve as prefixes that induce open-weight models to regurgitate alignment-like data.The attack relies on access to tokenization and chat-template structure, so it applies to open models; exploiting closed models would require spoofing template tokens.Source: §6 · p. 12
  • High-quality embeddings reveal semantic memorisation that string matching undercounts by at least an order of magnitude.The paper argues semantic similarity better captures the utility of extracted data than approximate or verbatim string matches.Source: §6 · p. 12
  • Extracted post-training data can meaningfully recover parts of the original model’s performance when used to train a new base model in SFT and RL settings.The authors frame distillation as a possible indirect form of training-data extraction, putting advantages from alignment data at risk of leakage.Source: §6 · p. 12
  • Models readily regurgitate samples from RL training, although increased likelihood of RL prompts after post-training suggests a more complex relationship between alignment and memorisation.The authors identify that relationship as a topic for future investigation.Source: §6 · p. 12

A. Extended background

The background situates the paper at the intersection of model alignment, training-data memorisation, and model distillation, where it argues a risk emerges.

A. Extended background

  • Model alignment uses curated data and specialised training techniques to transform a base model into a useful assistant.Source: A. Extended background · p. 17
  • Training-data memorisation is the observed capacity of models to regurgitate training data.Source: A. Extended background · p. 17
  • Model distillation transfers a strong model’s capabilities to another, and the paper argues the intersection of these three areas carries risk.Source: A. Extended background · p. 17

A.1. Model alignment

This background describes post-training approaches and the role of specialised data, then explains how chat templates and their special tokens can provide an attack vector for eliciting alignment data.

A.1. Model alignment

  • The study focuses mainly on SFT and PPO post-training, which are common techniques with strong open models and publicly available post-training data.The OLMo 2 family used SFT with 939,344 question-answer pairs, providing a resource for the study.Source: §A.1 · pp. 17–18
  • SFT trains on question-answer pairs by increasing the likelihood of answers conditioned on questions, typically masking question-token loss contributions.Source: §A.1 · p. 17
  • PPO is presented as a policy-optimization method, with its objective using a token advantage estimate computed through GAE with a learned value model and reward function.Source: §A.1 · p. 18
  • GRPO removes PPO’s value function and estimates advantages across groups of responses, whose rewards normally reflect answer correctness and response format.Source: §A.1 · p. 18
  • Several methods aim to improve GRPO: Dr. GRPO removes response-length and question-difficulty biases, while DAPO and GSPO are other examples.The passage says Dr. GRPO improves token efficiency with comparable reasoning performance.Source: §A.1 · p. 18
  • Chat templates mark user, assistant, and system messages with special tokens introduced during post-training, making them an attack vector for eliciting alignment data.This work generalises an effect noted by Xu et al. (2024) to study memorisation, addressing their conjecture that models memorize such data.Source: §A.1 · p. 18

A.2. Memorisation

The background reviews verbatim and non-verbatim memorisation, then frames distillation as a route by which a teacher’s training data may pass to a student.

A.2. Memorisation

  • Memorisation research has often focused on public-facing risks such as leaking private information or copyrighted material.Source: §A.2 · pp. 18–19
  • Discoverable extraction counts a sample as extractable when a prefix prompts greedy generation of it, while probabilistic extraction considers extraction likelihood across multiple queries.Source: §A.2 · p. 19
  • Prior work describes non-verbatim forms including mosaic memory from overlapping fragments and distributional memorisation linked to related input-output frequency.Mosaic memory challenges an exact-repetition account; Wang et al. (2024) report distributional memorisation in knowledge-intensive tasks, while reasoning tasks rely more on generalisation.Source: §A.2 · p. 19
  • The authors establish memorisation rates under an embedding definition that they say are higher than expected from prior work, including findings that reasoning relies more on generalisation [Wang et al. (2024)].Source: §A.2 · p. 19
  • Post-training may increase memorisation because it comes later, but masked prompt tokens and prior findings that RL memorizes less than instruction tuning point the other way.Jagielski et al. (2022) found later training data more likely to be memorised; Pappu et al. (2024) found RL less likely to memorize than standard instruction tuning at the same compute budget.Source: §A.2 · p. 19

A.3. Model Distillation

  • Distillation trains a student on a stronger teacher’s outputs, raising the question of how much of the teacher’s original training data is passed down unintentionally.The concern is especially pronounced in hard distillation that directly uses teacher labels as examples [Xu et al., 2024]; proprietary strategic data is costly, and leakage could erode its owner’s competitive advantage.Source: §A.3 · p. 20

B. Accidental gradient alignment

The section proposes that training on an answer conditioned on a question can also increase the question’s likelihood when their gradients align. It also notes that evaluating a single sequence may miss broader template-level similarity and gives generation-prefix examples.

B. Accidental gradient alignment

  • The authors identify verbatim training-data memorization despite masked gradients and ask whether updating on an answer conditioned on a question can increase the question’s likelihood.They offer mathematical speculation about how this may happen.Source: B. Accidental gradient alignment · p. 20
  • The answer-update changes the question’s log-probability by an amount proportional to the inner product of the answer and question gradients.The question’s likelihood increases when that inner product is positive; the authors describe this as correlated gradients.Source: B. Accidental gradient alignment · pp. 20–21
  • Single-sequence likelihood may be uninformative when a math sample preserves a template but changes numerical values, while integrating over reasonable sequences is quickly intractable.The caveat is that a single sequence may not capture the broader template-level pattern.Source: B. Accidental gradient alignment · p. 21

C. Prefixes used for generation

  • The generation prefixes are taken from the model developers’ released templates and specify how to format reasoning and answers.One prefix places reasoning in <think> tags and the answer in <answer> tags; another requires <answer> tags and a \boxed{} final answer.Source: C. Prefixes used for generation · p. 21

D. Failure cases of string matching

The examples show that string matching can assign low similarity to generations that are semantically memorized under the paper’s embedding threshold. It can penalize added reasoning, template differences, and small changes in notation or numbers.

D. Failure cases of string matching

  • All examples in this section exceed the 0.95 embedding threshold, yet are presented as failure cases for string matching, which uses a 0.90 threshold.The authors characterize the examples as semantically memorized and correctly identified by neural embeddings.Source: D. Failure cases of string matching · p. 21
  • The examples include simple-interest arithmetic, a petrol-or-water choice, and a height-growth question.The passages provide these as example question text.Source: D. Failure cases of string matching · pp. 22–24
  • Adding options and reasoning can lower the string-matching score even when a question is regurgitated almost verbatim.Figure 10 compares the generation with its best match selected by embeddings and by string matching.Source: Figure 10 · p. 22
  • Embedding similarity preserves the petrol-versus-water semantics, while string matching favors a more template-like but poor match.Figure 11 contrasts the semantically relevant match with one sharing a more similar question template.Source: Figure 11 · p. 23
  • When both methods find the same best match, string matching still penalizes small differences in wording, notation, numbers, or numerical values.These comparison cases are shown in Figures 12–14.Source: Figure 12 · p. 24, Figure 13 · p. 25, Figure 14 · p. 26

E. Examples of data extracted using our attack

The examples compare generations with their closest training samples using gemini-embedding-001. One sample is above the 0.95 semantic memorization threshold, while five are below it.

E. Examples of data extracted using our attack

Related papers

Loading 2510.18554v2…