PaperlayerView exact revision on arXiv

Source-linked AI summary

Learning to Detect Language Model Training Data via Active Reconstruction

Junjie Oscar Yin, John X. Morris, Vitaly Shmatikov, Sewon Min, Hannaneh Hajishirzi

arXiv:2602.19020v1cs.LGcs.AIcs.CL

TL;DR

Existing MIAs passively query fixed model weights, potentially missing latent membership signals. This paper introduces ADRA, which uses on-policy RL, reconstruction metrics, and contrastive rewards to actively reconstruct candidate data; ADRA and ADRA+ consistently outperform prior MIAs across training stages.

  • Problem

    Prior MIAs passively query fixed weights, while the paper investigates whether latent membership signals in model parameters can be actively elicited.

  • Method

    ADRA fine-tunes a target-initialized policy with on-policy RL and reconstruction rewards, using contrastive matching with distractors; ADRA+ adds a loss-derived adaptive membership prior.

  • Results

    ADRA and ADRA+ consistently outperform prior MIAs across pre-training, post-training, and distillation settings, with a 10.7% average improvement over the previous runner-up.

  • Takeaways & Limitations

    The results suggest model weights encode more training-data information than fixed outputs reveal, and RL can surface this latent information for stronger reconstruction and membership inference.

  • Takeaways & Limitations

    In prefix-matched distillation, members and non-members share the same prefix, making ADRA+ not directly applicable; only ADRA is used with N = 16.

Abstract

from arXiv · show

Detecting LLM training data is generally framed as a membership inference attack (MIA) problem. However, conventional MIAs operate passively on fixed model weights, using log-likelihoods or text generations. In this work, we introduce \textbf{Active Data Reconstruction Attack} (ADRA), a family of MIA that actively induces a model to reconstruct a given text through training. We hypothesize that training data are \textit{more reconstructible} than non-members, and the difference in their reconstructibility can be exploited for membership inference. Motivated by findings that reinforcement learning (RL) sharpens behaviors already encoded in weights, we leverage on-policy RL to actively elicit data reconstruction by finetuning a policy initialized from the target model. To effectively use RL for MIA, we design reconstruction metrics and contrastive rewards. The resulting algorithms, \textsc{ADRA} and its adaptive variant \textsc{ADRA+}, improve both reconstruction and detection given a pool of candidate data. Experiments show that our methods consistently outperform existing MIAs in detecting pre-training, post-training, and distillation data, with an average improvement of 10.7\% over the previous runner-up. In particular, \MethodPlus~improves over Min-K\%++ by 18.8\% on BookMIA for pre-training detection and by 7.6\% on AIME for post-training detection.

1. Introduction

ADRA reframes membership inference as active reconstruction: RL updates a target model to expose latent membership signals, exploiting differences in member and non-member reconstructibility. Across pre-training, post-training, and distillation settings, ADRA variants outperform prior MIAs.

  • Prior MIAs passively query fixed weights using loss, log-probabilities, or generations, potentially missing latent membership signals in model parameters.
  • ADRA uses RL with reconstruction rewards to elicit latent membership signals from a policy initialized from the target model.The approach is motivated by findings that RL sharpens behaviors already encoded in weights while transferring minimal new information.
  • ADRA generates continuations from candidate prefixes and uses contrastive rewards against true suffixes and distractors; ADRA+ adaptively uses a loss-derived membership prior.
  • 10.7% average improvement over the previous runner-up is reported across pre-training, post-training, and distillation detection settings.The experiments cover 6 open-weight LLMs, 6 newly constructed datasets, and 2 established benchmarks.
  • RL-based reconstruction suggests model weights encode more training-data information than fixed outputs reveal, enabling stronger reconstruction and membership inference.

2. Background

Membership inference determines whether candidate data appeared in an LLM’s training set, traditionally using loss or token log-probabilities as membership scores. Reconstruction-based methods instead sample completions and compare them with candidate suffixes.

  • Membership inference classifies whether a candidate datapoint was present in a model’s training data using a thresholded membership score.
  • Most prior language-model MIAs use candidate-sequence loss or token-level log-probabilities because seen sequences are expected to receive lower loss.
  • R-Loss calibrates target-model loss with a reference model, while Zlib Entropy normalizes loss by the candidate’s zlib compression size.
  • Min-K% averages the lowest-likelihood tokens, whereas Min-K%++ z-scores next-token log-probabilities using vocabulary statistics before averaging the smallest values.
  • Reconstruction-based MIA samples completions from a prefix and scores their n-gram overlap with the ground-truth suffix.

3. Active Data Reconstruction Attack

ADRA actively fine-tunes a target-initialized policy with reconstruction rewards, then scores candidates by regenerated similarity. Contrastive matching with distractors yields ADRA, while a loss-derived prior enables adaptive ADRA+.

  • ADRA asks whether active weight updates can elicit latent membership signals that passive log-probability and generation queries fail to capture.
  • The RL pipeline initializes θ0 from the target model, samples completions, computes rewards, updates the policy, and evaluates a final membership score.
  • The policy is trained on lexical similarity rewards measuring how much of the reference suffix a completion reconstructs, then evaluated by aggregating similarities over resampled completions.GRPO uses a clipped/KL-regularized surrogate during on-policy updates.
  • 3.2. Data Reconstruction Metrics: Token-set similarity measures unique-token recall, longest common subsequence rewards correct order, and n-gram coverage measures reference n-grams appearing in the completion.
  • 3.3. Contrastive Rewards: Contrastive rewards compare each rollout with the true suffix and distractors, limiting supervision to relative evidence already encoded in model weights.
  • 3.3. Contrastive Rewards: ADRA selects the best match in the reference pool, while ADRA+ adjusts true-suffix inclusion using a loss-derived membership prior.

4. Experimental Setup

The evaluation covers membership inference across pre-training, post-training, and distillation, using eight datasets and both original and paraphrased settings. ADRA methods are compared with passive loss- and reconstruction-based baselines using AUROC and reconstruction similarity metrics.

  • The study evaluates MIA across pre-training, post-training, and distillation stages.
  • The authors develop six new LLM-MIA datasets to reflect later knowledge cutoffs and larger post-training corpora.
  • The benchmark includes eight datasets and evaluates both original and Gemini-paraphrased candidate data.
  • Baselines include five loss-based MIAs and N-Sampling, which scores sampled continuations with lexical and embedding similarity metrics.
  • Performance is reported with AUROC, while reconstruction quality uses lexical and embedding similarity metrics.

5. Results

ADRA and ADRA+ improve reconstruction and membership detection across pre-training, post-training, and distillation settings. The methods achieve broad gains over passive baselines, including strong distillation performance and higher post-training reconstructibility.

  • Pre-training: 15.1%, 10.0%, and 21.3% are ADRA+’s average gains over the best loss-based MIAs on BookMIA, WikiMIA2024 Hard, and Dolma3 arXiv.
  • Pre-training: ADRA and ADRA+ consistently improve reconstruction, especially lexical Jaccard and embedding cosine, although RL creates metric trade-offs on WikiMIA2024 Hard.
  • Post-training: 11.0% and 13.2% are ADRA+’s AUROC gains over N-Sampling on Olympia Math and AIME, respectively.
  • Post-training: Post-training data are more extractable than pre-training data, with stronger reconstruction gains after RL finetuning.
  • Distillation: Distillation examples share prefixes and differ only in generated suffixes, making ADRA+ inapplicable and requiring ADRA with N = 16 generations.
  • Distillation: 97.0% and 86.9% are ADRA’s membership-inference results under DeepSeek-R1 and Gemini distillation, respectively.

6. Ablations & Analysis

Ablations indicate that RL optimization and contrastive objectives drive the MIA gains, while lexical rewards outperform model-based alternatives. Performance remains stronger than existing MIAs when scaling to Tulu2-13B.

  • RL vs. SFT: RL-based optimization and contrastive objectives are critical to gains in MIA performance.
  • Model Size: ADRA and ADRA+ continue outperforming existing MIAs when scaling from Tulu2-7B to Tulu2-13B on AIME.
  • Reward Type: Lexical rewards outperform embedding and LLM-as-judge rewards under noncontrastive RL.
  • Reward Type: Model-based rewards improve over baseline but underperform lexical rewards and are more prone to reward hacking.

7. Related Work

Prior MIAs mostly passively query fixed model weights through losses, log-probabilities, or generations. ADRA instead actively trains a policy to reconstruct candidate data, targeting latent membership information in model parameters.

  • Membership Inference Attacks: Membership inference determines whether candidate data appeared in a model’s training set, commonly using loss or log-probability signals.
  • Membership Inference Attacks: Generation-based methods sample continuations under fixed prompts, so they remain passive despite using model outputs.
  • Active Reconstruction: ADRA actively incentivizes reconstruction by maximizing reconstruction-aligned rewards during policy training.
  • Training Data Detection in LLMs: Training-data detection is difficult because corpora are enormous, frontier datasets are often undisclosed, and distribution shifts confound MIA efficacy.
  • Training Data Extraction and Memorization: Language models can memorize and regurgitate harmful or copyrighted training information, while alignment can make extraction more difficult.

8. Conclusion

ADRA uses reinforcement learning for active data detection and reconstruction, improving over passive MIAs across all stages of model training. The results suggest model weights encode more training-data information than previous methods reveal.

  • ADRA leverages reinforcement learning for active data detection and reconstruction.
  • ADRA consistently improves over passive MIAs across pre-training, post-training, and distillation settings.
  • The results suggest model weights encode more about training data than previous methods reveal.

Impact Statement

The paper evaluates active reconstruction across diverse training stages and datasets using controlled candidate construction, paraphrasing, contrastive distractors, and lexical reconstruction rewards. Its experiments use dataset-specific prefixes, sampling protocols, and training configurations.

  • The study investigates privacy risks from detecting and reconstructing training data across pretraining and post-training model lifecycles.
  • Paraphrased evaluation rewrites examples while preserving semantic content, using paraphrased inputs for training experiments and evaluation examples for existing models.
  • The pre-training dataset samples 3,000 Dolma3 arXiv documents for members and 1,000 post-2025 papers verified absent from training data for non-members.
  • Post-training mixtures use Aya and WildChat members from the Tulu3 SFT Mix and excluded original-dataset examples as non-members, capped at 1,024 tokens per turn.
  • ADRA trains reconstruction rewards using TRIO, which averages token-set similarity, longest common subsequence, and N-gram coverage, or N-GRAM, which uses N-gram coverage alone.
  • Each generation is scored against eight candidate suffixes consisting of one ground-truth suffix and seven independently sampled distractors.

D.3. ADRA Evaluation

RL-trained policies are evaluated with matched rollout budgets and lower-temperature sampling, and outperform N-Sampling on paraphrased pre-training reconstruction across the reported datasets and metrics.

  • After RL training, evaluation generates N rollouts per input with temperature 0.7, top-p 0.95, and top-k 50.
  • ADRA and ADRA+ consistently outperform N-Sampling on average lexical Jaccard and embedding cosine for paraphrased pre-training data.
  • On BookMIA, ADRA+ achieves the strongest average performance with 13.5 Jaccard, 44 LCS, and 1.0 coverage.
  • On WikiMIA2024 Hard, ADRA achieves the best average Jaccard at 13.4 and embedding cosine at 84.1, while N-Sampling leads LCS and coverage.

F. Additional Results: Post-training Reconstruction

For paraphrased post-training data, ADRA and ADRA+ improve reconstruction over N-Sampling across controlled-contamination datasets and metrics. Qualitative examples span pre-training, post-training, and distillation, while model-based rewards remain vulnerable to reward hacking.

  • Post-training reconstruction: ADRA and ADRA+ consistently outperform N-Sampling across all metrics on Olympia Math and AIME.
  • Post-training reconstruction: On AIME, ADRA improves average Jaccard by 6.7, LCS by 15, coverage by 7.9, and embedding cosine by 5.7 over N-Sampling.
  • Qualitative results: Qualitative reconstruction examples cover pre-training in Olmo3 Mix arXiv, post-training in AIME, and distillation in S1.1 DeepSeek-R1.
  • Ablation limitation: Model-based rewards are prone to reward hacking, and their effectiveness depends on prompt design and model choice.

I. Limitations & Discussions

The paper identifies compute cost, training instability, and unresolved memorization–generalization disentanglement as limitations of ADRA and its analysis. It also notes that improved RL algorithms may further improve the approach.

  • ADRA is substantially more compute-intensive than current MIAs because it requires on-policy RL training, potentially limiting large-scale use.
  • MIA performance can peak during early RL training, while continued training may degrade performance on some seeds and datasets.
  • The analysis cannot precisely disentangle latent memorization elicited by RL from generalization, and this distinction varies by dataset.
  • The paper uses vanilla GRPO despite reported collapse and instability, leaving exploration of improved RL algorithms to future work.
  • The authors conclude that active reconstruction reveals more training-data information in model weights than passive methods can surface.
Loading 2602.19020v1…