PaperlayerView exact revision on arXiv

Source-linked AI summary

OLMo-Detect: A Multi-Stage, Confounder-Controlled Benchmark for Membership Inference on Large Language Models

Tao Shi, Chaoyi Xiang, Qiongkai Xu, Jey Han Lau

arXiv:2610.02986v1cs.CL

TL;DR

LLM membership inference asks whether a text was in a model’s training data, but existing benchmarks have limited stage coverage, member–non-member alignment, and corpus-overlap filtering. OLMo-Detect addresses these gaps with a multi-stage, controlled benchmark and evaluates attacks across OLMo 2. Detection remains limited, varies with data type, and is not robust to distribution shifts.

  • Problem

    Existing LLM membership-inference benchmarks have limited training-stage coverage, insufficient member–non-member distributional alignment, and inadequate filtering of non-members against training data.

    Source: §1 · p. 1

  • Method

    OLMo-Detect covers pre-training, mid-training, and post-training, aligns members and non-members on three axes, and filters non-members against training data with infini-gram.

    Source: §1 · p. 2, §4.4 · p. 4, §4.5 · p. 5

  • Results

    The best MIA reaches AUC 0.68; detectability is higher for curated math data, unsupervised attacks are not robust to shifts, and findings generalize to OLMo 3 and non-OLMo models.

    Source: §1 · p. 2, §6 · p. 10

  • Takeaways & Limitations

    OLMo-Detect provides a rigorous testbed for future MIAs, and the reported main patterns are not artifacts of OLMo 2.

    Source: §6 · p. 10

    The benchmark does not perform additional PII screening itself, though peS2o and SFT inherit filtering from their sources.

    Source: ETHICS STATEMENT · p. 10

Abstract

from arXiv · show

Membership inference on large language models (LLMs) aims to determine whether a given text sample was included in an LLM's training data, without access to its training corpus. Despite recent progress, existing benchmarks suffer from three limitations: limited coverage of training stages, insufficient distributional alignment between members and non-members, and lack of rigorous filtering of non-members against the training corpus. To address these limitations, we propose OLMo-Detect, a multi-stage, confounder-controlled benchmark built upon the fully open OLMo 2 pipeline. OLMo-Detect spans pre-training, mid-training, and post-training, explicitly aligns members and non-members on three key axes, and rigorously filters non-members via infini-gram. To assess robustness to distribution shifts, we further introduce OLMo-Detect (Shifted), a variant where members are misaligned with non-members. We evaluate 15 unsupervised and 3 supervised membership inference attacks (MIAs) across the OLMo 2 family, finding that: (i) overall performance is limited: the best unsupervised and supervised MIAs both reach an AUC of only 0.68, and supervised MIAs degrade under cross-domain evaluation; (ii) MIA performance peaks at mid-training and is lower at pre-training and post-training, a pattern driven by data type rather than a stage effect: curated math data is far more detectable than other types; (iii) overall scores improve from 1B to 13B but plateau at 32B; and (iv) no unsupervised MIA is robust to distribution shifts, with AUCs shifting by up to 0.42. Finally, we find that our findings on OLMo 2 generalize to OLMo 3 and non-OLMo models.

1 INTRODUCTION

Existing LLM membership-inference benchmarks lack three-stage coverage, explicit member–non-member alignment, and rigorous corpus-overlap filtering; OLMo-Detect addresses those gaps, while evaluation finds detection remains limited and data-dependent.

1 INTRODUCTION

  • Existing benchmarks lack broad training-stage coverage, explicit alignment on attack-relevant axes, and rigorous filtering of non-members against training data.Table 1 compares benchmarks on stage coverage, explicit alignment, and rigorous filtering; semantic alignment alone may leave lexical shifts.Source: §1 · pp. 1–2, Table 1 · p. 2
  • OLMo-Detect spans pre-training, mid-training, and post-training, aligns members and non-members on three axes, and filters non-members using infini-gram.The benchmark aligns text quality, temporal range, and lexical similarity, and uses strict 13-gram filtering with a 20% threshold.Source: §1 · p. 2
  • Best unsupervised and supervised MIAs each reach average AUC 0.68; detectability tracks data type, improves from 1B to 13B then plateaus at 32B, and shifted AUCs move by up to 0.42.Curated math data is considerably more detectable than other data types.Source: §1 · p. 2
  • No unsupervised MIA is robust to distribution shifts, while findings on OLMo 2 generalize to OLMo 3 and non-OLMo models.The benchmark evaluates a shifted variant with misaligned members and non-members.Source: §1 · p. 2

2 RELATED WORK

Related work groups membership-inference benchmarks by how they construct members and non-members, and attacks by whether and how they use supervision.

2 RELATED WORK

  • Benchmarks are grouped as temporal-cutoff, partition-based, checkpoint-based, or model-output-based according to how members and non-members are constructed.The cited passage points to Appendix A for detailed descriptions.Source: §2 · pp. 2–3
  • Unsupervised MIAs fall into likelihood-based, perturbation-based, prompt-based, and output-distribution-based families.The passage lists prior work associated with each family.Source: §2 · p. 3
  • Supervised MIAs include a supervised CAMIA variant and tuning-based methods.The passage points to Appendix B for details.Source: §2 · p. 3

3 TASK DEFINITION

The task is to infer whether a candidate text belongs to an LLM’s training corpus from a score, under an assumed grey-box access setting.

3 TASK DEFINITION

  • Membership inference scores a candidate text and thresholds the score to predict whether it belongs to the model’s training corpus.The prediction is 1 for member and 0 for non-member.Source: §3 · p. 3
  • The task assumes grey-box access: output statistics such as loss, logits, and token probabilities are available, but internal parameters are not.The passage notes that some methods additionally assume the ability to fine-tune the model.Source: §3 · p. 3
  • Some membership-inference methods assume the ability to fine-tune the model.Further description is deferred to Appendix B.Source: §3 · p. 3

4 BENCHMARK CONSTRUCTION

OLMo-Detect uses OLMo 2’s open training pipeline to build a benchmark spanning three training stages, with members and non-members aligned and non-members filtered against training data. Verification reports low residual n-gram overlap and near-chance blind-classifier AUCs; a shifted variant relaxes the alignment constraints.

4.1 MOTIVATION FOR USING OLMO 2

  • OLMo 2 was chosen because it provided access to all three training stages, and the paper reports that its findings generalize to OLMo 3.When the work began, OLMo 2 was the available model with complete access across those stages.Source: §4.1 · p. 3

4.2 TRAINING DATA OF OLMO 2

  • OLMo 2’s benchmark training data spans web, code, academic, math, and encyclopedic material in pre-training; instruction, Q&A, and curated math in mid-training; and SFT, DPO, and RLVR in post-training.The data summary distinguishes reused pre-training domains from new mid-training domains.Source: §4.2 · p. 3

4.3 BENCHMARK OVERVIEW

  • OLMo-Detect comprises nine domains across three stages; four domains are excluded because matched sources, filtered non-members, or instance-length-matched members were unavailable.The exclusions are pre-training Wikipedia/Wikibooks, arXiv, and Algebraic Stack, plus mid-training FLAN.Source: §4.3 · p. 3

4.4 CONSTRUCTION OF NON-MEMBERS

  • Non-members come from source-matched pools, use random or boundary sampling depending on quality thresholding, and pass a strict infini-gram 13-gram filter at a 20% threshold.The retained candidates are allocated a roughly uniform budget of 217K tokens per domain.Source: §4.4 · pp. 3–5

4.5 CONSTRUCTION OF MEMBERS

  • Members are selected with simulated annealing to align lexical similarity and, where available, temporal range, while matching instance counts and keeping token counts within a 250-token margin.The loss includes Jensen–Shannon divergence and timestamp alignment; δ is set to 250.Source: §4.5 · p. 5

4.6 BENCHMARK VERIFICATION

  • Across domains, non-members have 5.31% average and 9.31% maximum 13-gram overlap with OLMo 2’s training corpus.Table 3 reports per-domain overlap averaged across model sizes.Source: §4.6 · p. 5, Table 3 · p. 5
  • Every domain’s blind bag-of-words AUC is within 0.02 of 0.50, supporting lexical alignment between members and non-members.The blind classifier predicts membership from text alone, without access to the LLM.Source: §4.6 · pp. 5–6, Table 3 · p. 5

4.7 OLMO-DETECT (SHIFTED)

  • OLMo-Detect (Shifted) keeps non-members unchanged but samples members using only matched instance and token counts, omitting alignment on quality, time, and lexical similarity.Mid-training Math Mix and post-training RLVR are excluded because sampled members do not yield substantially different JSD from their non-members.Source: §4.7 · p. 6

5 EXPERIMENTS

The experiments compare supervised and unsupervised MIAs across OLMo 2 models and training stages, then test distribution-shift robustness and generalization to other model families. Detection is limited, varies strongly with data type, and changes under distribution shifts.

5.1 EXPERIMENTAL SETTINGS

  • The evaluation covers four instruction-tuned OLMo 2 sizes, 15 unsupervised and three supervised MIAs, and reports macro-averaged AUC and TPR@5%FPR.Macro-averaging aggregates scores from subsets to domains, stages, and final scores.Source: §5.1 · p. 6

5.2 OVERALL PERFORMANCE

  • 0.68 overall AUC is reached by both Zlib and Sup-CAMIA; supervised methods perform worse or no better under LODO.Sup-CAMIA falls to 0.61 AUC under LODO, while Zlib reaches 0.68 overall AUC.Source: Table 5 · p. 7, §5.2 · p. 7
  • Likelihood-based MIAs generally lead, but ReCaLL reverses on three rigid-format math sets (AUC 0.10, 0.09, 0.45); MinK++ trails MinK, 0.56 vs. 0.66.The rigid formats turn ReCaLL’s non-member prefix into an in-context demonstration rather than disrupting member predictions.Source: §5.2 · p. 7

5.3 PERFORMANCE ACROSS STAGES AND MODEL SIZES

  • Curated math subsets are more detectable (p = 0.008 for all MIAs; p = 0.004 for effective MIAs), and excluding them narrows stage differences, indicating a composition effect.CoT-GSM8K reaches 0.90 AUC among effective MIAs.Source: Table 6 · p. 8, §5.3 · p. 8
  • Average unsupervised AUC rises from 0.56 at 1B to 0.60 at 13B and only 0.61 at 32B; effective methods gain just 0.01 from 13B to 32B.Source: §5.3 · p. 8

5.4 ROBUSTNESS TO DISTRIBUTION SHIFTS

  • Every unsupervised MIA changes significantly on 3–6 of seven shifted domains, with largest per-method changes of 0.09–0.42 AUC; likelihood-based methods are most affected.Eight of ten likelihood-based MIAs change by 0.20–0.42 AUC.Source: Table 7 · p. 9, §5.4 · p. 9

5.5 GENERALIZATION BEYOND OLMO 2

  • On OLMo 3, GSM8K AUC remains 0.95–1.00; on three non-OLMo models, every method is within 0.04 AUC of size-matched OLMo 2 and domains remain weakly detectable (0.51–0.64).Source: Table 8 · p. 9, §5.5 · pp. 9–10

6 CONCLUSION

OLMO-DETECT spans all three OLMo 2 training stages, aligns members and non-members, and filters non-members against the training corpus. Its evaluations find limited MIA performance and greater detectability for curated math data.

  • The benchmark spans pre-training, mid-training, and post-training, aligns members and non-members on three axes, and filters non-members against the training corpus.Source: §6 · p. 10
  • The best MIA reaches only 0.68 AUC, while curated math data is substantially more detectable than other data types.Source: §6 · p. 10
  • The main patterns generalize to OLMo 3 and non-OLMo models, and the authors propose OLMO-DETECT as a testbed for future MIAs.Source: §6 · p. 10
  • The benchmark uses OLMo 2 training data or data from the same source distributions and performs no additional PII screening itself.peS2o and SFT inherit filtering from their sources.Source: ETHICS STATEMENT · p. 10

A DETAILED DESCRIPTIONS OF EXISTING BENCHMARKS FOR MEMBERSHIP INFERENCE ON LLMS

This survey details how existing membership benchmarks construct member and non-member examples, then reviews gray-box attack methods and OLMo 2 training data. It highlights distributional and lexical confounds in benchmark design and describes the signals exploited by different attacks.

A DETAILED DESCRIPTIONS OF EXISTING BENCHMARKS FOR MEMBERSHIP INFERENCE ON LLMS

  • Temporal-cutoff benchmarks label pre-cutoff texts as members and post-cutoff texts as non-members, but natural temporal drift can make apparent detection reflect distribution shifts instead (Duan et al., 2024).WikiMIA, StackMIA, PatentMIA, and WikiMIA-24 use different domains or cutoff windows.Source: A DETAILED DESCRIPTIONS OF EXISTING… · p. 20
  • MIMIR’s permissive 13-gram filter leaves residual overlap above 30% across measured domains, whereas OLMo-Detect’s ≤20% threshold reduces average residual overlap to 5.31%; RL-MIA lacks explicit alignment controls.MIMIR discards candidates only when more than 80% of their 13-grams occur in training data.Source: A DETAILED DESCRIPTIONS OF EXISTING… · p. 20
  • OLMoMIA aligns checkpoint-based splits semantically, but semantic similarity need not imply lexical similarity; DETCON’s different model states may also expose LoRA-related shortcuts and limit transfer to real-world training.OLMoMIA uses examples before step 100k as members and examples from steps 400k–500k as non-members; DETCON simulates SFT membership with LoRA.Source: A DETAILED DESCRIPTIONS OF EXISTING… · pp. 20–21

B DETAILED DESCRIPTIONS OF EXISTING MIAS

  • Gray-box unsupervised attacks include likelihood, prompt, perturbation, and output-distribution methods; likelihood approaches calibrate token probabilities against text properties, variants, other models, or context.Examples include Min-K%, Neighborhood Attack, prompt-based completion or cloze probes, and output-concentration measures.Source: B DETAILED DESCRIPTIONS OF EXISTING MIAS · pp. 21–22
  • Prompt-based attacks trade broader free-form probing, which requires soft matching, for slot-filling exact-match checks; output-distribution methods detect abnormal concentration in generated responses.DCQ instead asks the model to choose the original among semantically equivalent paraphrases.Source: B DETAILED DESCRIPTIONS OF EXISTING MIAS · p. 22
  • Supervised methods either combine likelihood-derived signals with a classifier, tune a soft prompt or model weights, or use score changes after non-member fine-tuning as the detection signal.FSD requires only non-members for tuning, unlike MIA-Tuner, which uses both members and non-members.Source: B DETAILED DESCRIPTIONS OF EXISTING MIAS · pp. 22–23

C DETAILED DESCRIPTIONS OF OLMO 2 TRAINING DATA

  • The appendix’s OLMo 2 training-data breakdown reproduces token counts, document counts, and filtering details from the original OLMo 2 paper.The breakdown covers the three training stages.Source: C DETAILED DESCRIPTIONS OF OLMO 2… · p. 23

C.1 PRE-TRAINING: OLMO 2 MIX 1124

  • OLMo 2 Mix 1124 combines seven domains from DCLM, Dolma 1.7, and StarCoder; the 1B and 7B models use 4 trillion tokens, 13B uses 5 trillion, and 32B uses 6 trillion.StarCoder repositories require at least two GitHub stars and are filtered for repetitive word distributions.Source: §C.1 · p. 23

C.2 MID-TRAINING: DOLMINO MIX 1124

This section describes OLMo 2’s mid- and post-training data mixtures and the construction of OLMO-DETECT member and non-member samples, including source matching, quality-based sampling, overlap filtering, and lexical statistics. It also explains alignment constraints and simulated-annealing member sampling, and examines how randomly sampled members shift across domains in OLMO-DETECT (Shifted).

C.2 MID-TRAINING: DOLMINO MIX 1124

  • OLMo 2’s Dolmino Mix 1124 combines a five-domain High-Quality Subset with a Dolmino Math Mix containing synthetic math, GSM8K, and filtered external data.The Math Mix includes TuluMath, DolminoSynthMath, TinyGSM-MIND, MathCoder2-Synthetic, GSM8K, and OpenWebMath-filtered Metamath and CodeSearchNet; CodeSearchNet was not used in final training.Source: §C.2 · p. 23, Table 10 · p. 24

C.3 POST-TRAINING: TÜLU 3 RECIPE

  • Post-training follows an adapted Tülu 3 recipe with sequential SFT, DPO, and RLVR phases.DPO uses judged candidate responses to form chosen/rejected pairs; all four model sizes receive one RLVR pass, with additional passes for 1B and 13B.Source: §C.3 · pp. 25, 27

D DETAILED PER-DOMAIN STATISTICS OF OLMO-DETECT

  • A lexical JSD of 0.30 for StarCoder Java reflects its smaller token budget, not residual misalignment; a blind bag-of-words classifier reaches AUC 0.51.Java has 22.1K tokens versus 184K–217K elsewhere; splitting its non-members yields JSD 0.40, and restricting Python to Java’s budget raises its JSD from 0.16 to 0.36.Source: D DETAILED PER-DOMAIN STATISTICS OF… · p. 28

E.1 SOURCE-MATCHED CANDIDATE POOLS

  • For RLVR, the benchmark selects held-out CoT-GSM8K and CoT-MATH candidates over IFEval because lexical alignment takes priority over within-domain diversity.The two selected subsets share a similar chain-of-thought construction.Source: §E.1 · pp. 29, 32

E.2 DOMAIN-DEPENDENT SAMPLING STRATEGIES

  • Quality-thresholded non-members are sampled near selection boundaries, while StarCoder candidates also retain the training data’s GitHub-star constraint.Stack Exchange candidates come from the rejected remainder of the same dated dump and are processed into matched Q&A format.Source: §E.1 · p. 29, §E.2 · p. 33
  • StarCoder sampling could not meet the proportional Java allocation: only 56 Java non-members (22K tokens) were available, so Python filled the domain’s ∼217K-token budget.Java candidates were selected to match word-frequency and GitHub-star constraints; the available Java members and non-members were retained while Python supplied the remainder.Source: §E.2 · p. 33

E.3 INFINI-GRAM-BASED 13-GRAM FILTERING

  • No arXiv or Algebraic Stack candidates survived 13-gram filtering, so both domains were excluded.The candidates came from held-out files; filtering used infini-gram indexes built over each OLMo 2 model’s full training corpus, including repetition.Source: §E.3 · p. 33

F.1.2 QUALITY-THRESHOLDED DOMAINS

  • In domains where OLMo 2 training data passed a quality threshold and was higher quality than corresponding non-members, instances closest to the threshold form the candidate pool to align text quality.For other specified domains, the corresponding OLMo 2 training data serves as the candidate pool.Source: §F.1.2 · p. 34, §F.1.1 · p. 34

F.2 ALIGNMENT AXES

  • The alignment design stops at three axes because additional constraints shrink feasible candidate pools and subset sizes, undermining statistical reliability.The authors prioritize subset size and statistical reliability over adding further constraints, which would also make the simulated-annealing search substantially harder.Source: §F.2 · p. 35

F.3 CONSTRAINED SAMPLING VIA SIMULATED ANNEALING

  • Constrained simulated annealing selects domain-specific member subsets matching non-member counts and token totals while minimizing lexical and, when available, temporal distributional loss.The token-count margin is δ = 250; the loss combines Jensen–Shannon divergence with a temporal term when metadata is available, and swap acceptance uses a Metropolis criterion during alignment optimization.Source: §F.3 · p. 35

G DETAILED PER-DOMAIN STATISTICS OF OLMO-DETECT (SHIFTED)

  • Every shifted domain differs from non-members on at least one axis, with lexical shifts concentrated in domains whose source corpora vary lexically.DCLM-Baseline, peS2o, and Stack Exchange shift mainly in quality or time while lexical values change by at most 0.03; StarCoder, OpenWebMath, SFT, and DPO lexical values rise two- to fourfold, from 0.09–0.15 to 0.32–0.36. The shifts are not maximized, emulating unaligned benchmark construction rather than a worst case.Source: G DETAILED PER-DOMAIN STATISTICS OF… · p. 36

H DETAILED DESCRIPTIONS OF EXPERIMENTAL SETTINGS

The appendix specifies a shared evaluation protocol, reports uncertainty for aggregate AUCs, and documents adaptations and exclusions for the compared attacks.

H DETAILED DESCRIPTIONS OF EXPERIMENTAL SETTINGS

  • Every method uses batch size 1; generative methods use 4,096-token contexts, other methods use 1,024, and hyperparameters are tuned on development data.Experiments use A100 GPUs and a fixed seed of 42.Source: H DETAILED DESCRIPTIONS OF… · p. 36
  • A single seed suffices for deterministic methods, but five stochastic methods use fixed random draws or data order without multi-seed reporting.The stochastic methods are Neigh., CDD, PAC, FSD, and MIA-Tuner.Source: H DETAILED DESCRIPTIONS OF… · pp. 36–37
  • Aggregate AUC uncertainty is estimated with 2,000 bootstrap resamples and 95% intervals, but these intervals omit seed-to-seed variance for five stochastic methods.The intervals apply to the overall AUC aggregation, not an individual evaluation.Source: H DETAILED DESCRIPTIONS OF… · p. 37
  • DCQ and Guided receive continuous scores for AUC by replacing categorical decisions with scores derived from their original signals.DCQ uses next-token log-probabilities; Guided uses ROUGE-L F1, with variants selected on development data.Source: H DETAILED DESCRIPTIONS OF… · pp. 37–38
  • The comparison tables distinguish unsupervised from supervised methods and list tuned hyperparameter search spaces, while four surveyed MIAs are excluded for setting incompatibility.Excluded methods require an additional model, need specialized text, or lack an open-source implementation.Source: Table 26 · p. 38, Table 27 · p. 38, H DETAILED DESCRIPTIONS OF… · p. 38

I PER-DOMAIN AUC RESULTS FOR UNSUPERVISED MIAS ON OLMO-DETECT

The appendix points readers to detailed unsupervised-MIA AUC and TPR@5% FPR results, complementing stage-level aggregates.

I PER-DOMAIN AUC RESULTS FOR UNSUPERVISED MIAS ON OLMO-DETECT

  • Per-domain unsupervised AUCs are reported in Tables 37–49, complementing stage-level results; bootstrap intervals for overall AUC are in Tables 28 and 29.The intervals apply to the overall AUC aggregation; SelfCrit and FSD are omitted for their stated coverage/reporting constraints.Source: I PER-DOMAIN AUC RESULTS FOR… · p. 38, Table 28 · p. 39, Table 29 · p. 39
  • The reported bootstrap intervals use 2,000 resamples and recompute the full subset-to-domain-to-stage aggregation, rather than intervals for individual evaluations.Source: Table 28 · p. 39

J TPR@5% FPR RESULTS FOR UNSUPERVISED MIAS ON OLMO-DETECT

K PER-DOMAIN AUC RESULTS FOR SUPERVISED MIAS ON OLMO-DETECT

Across domains, supervised methods seldom outperform Zlib, and fine-tuning often fails to improve its base detector; the appendix also details shifted benchmark coverage.

K PER-DOMAIN AUC RESULTS FOR SUPERVISED MIAS ON OLMO-DETECT

  • Per-domain tables report Sup-CAMIA and MIA-Tuner AUCs under In-domain and LODO settings, plus FSD’s gain over its base detector.LODO trains on the other eight domains’ development sets.Source: K PER-DOMAIN AUC RESULTS FOR… · p. 39
  • The better supervised MIA exceeds Zlib on five of nine domains, ties on OpenWebMath, and falls below it on DCLM-Baseline, SFT, and RLVR.The largest stated FSD losses reach −0.10 or worse on DCLM-Baseline.Source: K PER-DOMAIN AUC RESULTS FOR… · p. 39
  • FSD improves its base detector in 13 of 45 domain-by-detector combinations In-domain and 11 of 45 under LODO.Every base detector degrades on DCLM-Baseline under both settings.Source: K PER-DOMAIN AUC RESULTS FOR… · p. 39
  • The shifted benchmark covers seven of nine domains; its members are sampled without alignment on text quality, temporal range, or lexical similarity, while non-members remain unchanged.GSM8K and RLVR are excluded because neither has a shiftable axis.Source: Table 30 · p. 40

L TPR@5%FPR RESULTS FOR SUPERVISED MIAS ON OLMO-DETECT

At the 5% FPR operating point, supervised MIAs do not outperform Zlib overall, and their performance weakens under LODO evaluation. Their advantage over Zlib is domain-dependent, while FSD rarely improves its base MIA.

L TPR@5%FPR RESULTS FOR SUPERVISED MIAS ON OLMO-DETECT

  • The appendix reports supervised MIAs by TPR@5% FPR, with overall results aggregated by training stage and per-domain results in separate tables.In-domain training uses the evaluated domain’s development set; LODO uses the other eight domains’ development sets.Source: L TPR@5%FPR RESULTS FOR SUPERVISED… · p. 40
  • Sup-CAMIA In-domain reaches 0.26 TPR@5% FPR versus Zlib’s 0.28, while Sup-CAMIA under LODO reaches 0.14 and MIA-Tuner reaches 0.09 In-domain and 0.05 under LODO.The MIA-Tuner LODO result is indistinguishable from the 0.05 of a random detector.Source: L TPR@5%FPR RESULTS FOR SUPERVISED… · p. 40
  • On four of nine domains the better supervised method exceeds Zlib, but on five it falls below, with the largest shortfall on StarCoder (−0.10) and largest gains on Math Mix and DPO (+0.08 each).Source: L TPR@5%FPR RESULTS FOR SUPERVISED… · p. 40
  • FSD improves its base MIA in 12 of 45 domain-by-base-MIA combinations under In-domain and 11 under LODO.Source: L TPR@5%FPR RESULTS FOR SUPERVISED… · p. 40

M EFFECT OF THE LABELED-DATA BUDGET ON SUPERVISED MIAS

Increasing Sup-CAMIA’s labeled-data budget raises macro-averaged AUC modestly, with gains concentrated in the smallest pre-training domains. The larger budget does not change the conclusion that matched-distribution supervision adds little beyond unsupervised signals.

M EFFECT OF THE LABELED-DATA BUDGET ON SUPERVISED MIAS

N DOMAIN-UNKNOWN EVALUATION

When domain identity is unknown, pooled evaluation lowers absolute MIA performance, especially for Sup-CAMIA, but method rankings closely track the domain-known setting. CAMIA is the strongest unsupervised method in the pooled setting.

N DOMAIN-UNKNOWN EVALUATION

  • The domain-unknown setting pools members and non-members from all nine domains and computes one AUC rather than averaging per-domain AUCs.For accuracy, it uses one threshold tuned on pooled data instead of a threshold tuned within each domain.Source: N DOMAIN-UNKNOWN EVALUATION · p. 62
  • Pooling lowers AUC for 12 of the 15 unsupervised MIAs, and the unsupervised average drops by 0.02 AUC.CAMIA is unaffected at 0.62 in both settings and is strongest once the domain is unknown.Source: N DOMAIN-UNKNOWN EVALUATION · p. 62
  • Sup-CAMIA falls from 0.66 to 0.60 AUC when pooled, while MIA-Tuner changes from 0.56 to 0.55.Sup-CAMIA’s loss is attributed to losing the ability to calibrate to a single domain.Source: N DOMAIN-UNKNOWN EVALUATION · p. 62
  • The ranking remains highly consistent across settings, with Spearman ρ = 0.95 across the 17 MIAs.Source: N DOMAIN-UNKNOWN EVALUATION · p. 62

O PER-DOMAIN MATCHED-VS-SHIFTED RESULTS

Distribution shifts produce method- and domain-dependent AUC changes, with each method’s largest change occurring in a pre-training domain. For SmolLM2 evaluation, the authors filter DCLM non-members against FineWeb-Edu and pair the comparison with OLMo 2-1B on the verified set.

O PER-DOMAIN MATCHED-VS-SHIFTED RESULTS

  • OpenWebMath inflates AUC for PPL, MinK, CAMIA, and EMMIA but degrades it for Zlib, Lower, and PAC, showing that shift direction depends on the method–domain pair.Source: O PER-DOMAIN MATCHED-VS-SHIFTED RESULTS · p. 62
  • Every MIA’s largest shift-related AUC change occurs in a pre-training domain: OpenWebMath for 7 of 14 MIAs, StarCoder for 4, and peS2o for 3.DCLM never carries a method’s largest change despite being shifted on the same axes.Source: O PER-DOMAIN MATCHED-VS-SHIFTED RESULTS · p. 62

P NON-MEMBER VERIFICATION FOR SMOLLM2

  • DCLM non-members absent from DCLM are not necessarily absent from FineWeb-Edu, because OLMo 2 trained on both and FineWeb-Edu deduplicates within rather than across dumps.Source: P NON-MEMBER VERIFICATION FOR SMOLLM2 · p. 63
  • Applying the benchmark’s 13-gram filter at the 0.20 threshold to FineWeb-Edu excluded eight of 134 test non-members and one of 16 development non-members; the remaining 126 test non-members had overlaps no greater than 0.128.The excluded indices are released with the benchmark.Source: P NON-MEMBER VERIFICATION FOR SMOLLM2 · p. 63
  • SmolLM2 is evaluated on the 126 verified non-members, and OLMo 2-1B is re-evaluated on the same set for a paired comparison.SmolLM2 members are not filtered because their appearance in FineWeb-Edu is expected and they remain members of its training data.Source: P NON-MEMBER VERIFICATION FOR SMOLLM2 · p. 63
Loading 2610.02986v1…