Source-linked AI summary
Detectable Only Where It Is Confounded: What Verified Duplication Counts Say About Membership Evidence in Language Models
Arman Nik Khah
TL;DR
Membership inference has often relied on guessed member and non-member sets, making exposure difficult to separate from sentence differences. This paper uses exact counts from two published corpora and same-sentence comparisons, finding only a faint exposure trace where the corpora disagree and stronger signal mainly where high exposure coincides with fame.
Problem
Existing membership tests often guess which sentences were in training, allowing source, date, style, and length differences to confound the inference.
Method
The paper combines exact sentence-copy counts from public OLMo-mix and Pile indexes with same-sentence comparisons across OLMo-2 and Pythia models.
Results
At ordinary duplication levels, exposure leaves only a faint trace, with rank correlation near −0.08 where the two corpora disagree; stronger signal appears above roughly 1,000 copies where sentences are famous.
Takeaways & Limitations
Exposure signal is difficult to distinguish from fame when it becomes strong, so membership evidence must be interpreted alongside verified duplication counts and confound-controlled comparisons.
Takeaways & Limitations
The study is limited to models up to 13B, English literary prose from six books, exact-match floors, and a twelve-sentence famous-line control.
Abstract
from arXiv · showhide
When a language model finds a sentence unusually cheap to predict, it is tempting to conclude that the sentence was in its training data. Almost every published test of that inference has had to guess which sentences were in the training data, the members, and which were not. This paper removes the guessing. Two model families, OLMo-2 and Pythia, publish their pretraining corpora, and a public index over those corpora returns the exact number of times any sentence appeared in each. Those counts make three questions answerable directly. The answers form a pincer, closing from two sides. At the duplication levels ordinary text actually has, five models from 1B to 13B parameters carry at most a faint trace of their own exposure. We measure that trace with a design that reads the same sentence through two models, which cancels fluency and quality by construction, and it comes to a rank correlation near -0.08, where -1 would be a perfect relation and 0 none. Where the trace does become strong, above roughly a thousand copies, the two corpora agree on which sentences those are, because they are the famous ones, so exposure can no longer be told apart from fame. Two further measurements show how apparent membership signal gets manufactured. A common way to build a non-member is to change one word of a member. The model does prefer the original, but the gap is the same whether the original appeared once or a hundred times, so what the model is rewarding is the author's word choice, not memory. Above a thousand copies the gap grows with model size on the twelve sentences we can test there, at the same boundary where the pincer closes. And swapping the controls for sentences that differ from the members in register moves a detector from 0.83 to 0.94 AUC, on a scale where 0.5 is a coin flip and 1.0 is perfect separation. We release the sentence banks, counts, and code.
1. The question, and why it has been hard to answer
Published membership tests often confound exposure with differences in date, source, style, or length. This paper uses exact corpus counts and same-sentence comparisons to measure exposure directly while holding sentence properties fixed.
- The question, and why it has been hard to answer: Membership tests can score well by detecting date, source, style, or length differences rather than training exposure.Prior evaluations guessed membership, allowing distribution shifts to masquerade as memorization.
- The question, and why it has been hard to answer: Membership is a dose rather than a binary label, but existing tests lacked exact exposure counts and controls that held sentence properties fixed.Web text can occur once, dozens of times, or thousands of times, while conventional comparisons use different sentences.
- The question, and why it has been hard to answer: Two open model families and infini-gram provide exact corpus counts, enabling exposure variation across corpora while keeping each sentence’s fluency, quality, and length fixed.The design reads the same sentence through models from OLMo-2 and Pythia and compares their loss differences.
- The question, and why it has been hard to answer: The paper’s pincer result is that exposure is faint where the corpora disagree and detectable only where they agree on highly exposed, famous sentences.The paper identifies this as the joint conclusion of its first two findings.
2. Counting exposure
The study counts exact sentence copies in the published OLMo-mix and Pile corpora using infini-gram. These counts are exact-match floors, so formatting changes and an uncovered OLMo annealing stage leave exposure uncounted.
- Counting exposure: Infini-gram returns the number of positions containing an exact token sequence in the OLMo-mix and Pile corpora.The public suffix-array indexes answer queries in about thirty milliseconds without a key.
- Counting exposure: Exact-match counts are exposure floors because punctuation, quotation, line-break, and other tokenization changes create different sequences.OLMo counts also omit exposure from a separate annealing mix not covered by the index.
- Counting exposure: The pipeline checks a known-present sentence before accepting a batch, preventing index failures or malformed queries from being recorded as zero counts.This presence check is distinct from the paper’s positive-control bank.
- Counting exposure: Sentences of 10 to 16 words balance exact-match survival against specificity: most spans remain stable to forty words, while some collapse after formatting differences begin.A 140-word paragraph typically returns zero under exact matching.
- Counting exposure: Across 747 sentences, OLMo-mix and Pile counts have Spearman correlation 0.15, with a roughly twenty-seven-fold change in their count ratio across the distribution.This cross-corpus disagreement makes differential exposure measurable on the same sentences.
3. The sentence bank
The sentence bank combines mid-book literary sentences with a small famous-line positive control. It constructs non-members through same-source one-word edits to reduce distribution shifts while exposing a remaining word-choice confound.
- The sentence bank: The bank samples 10- to 16-word sentences from the middles of six public-domain novels and adds twelve famous lines from broader sources.The famous bank spans multiple authors, centuries, genres, and public texts.
- The sentence bank: Skipping book openings avoids a nearly saturated low-loss baseline caused by famous lines, which reached 0.90 AUC in an initial attempt.Opening lines are both highly quoted and unusually cheap for models to predict.
- The sentence bank: Each non-member is made by replacing one content word with a same-length, same-register near-synonym in the member sentence.This keeps source and broad style aligned, unlike controls drawn from different sources or dates.
- The sentence bank: The authors explicitly retain this control’s remaining flaw so Section 6 can measure whether the apparent membership gap reflects word choice.The design therefore treats the edit construction as a testable confound rather than a clean ground truth.
- The sentence bank: The famous bank has median exposure 1,192 copies, whereas mid-book banks have medians in the tens; the differential design scores the same 747 sentences across model pairs.Passes 3 and 4 are pooled for the differential analysis.
4. Where loss-based membership turns on
Loss-based membership detection is essentially flat across 1–999 exact copies but becomes measurable for the famous high-copy control. The paper therefore treats the low-thousands range as the exposure boundary visible to this instrument.
- Where loss-based membership turns on: The loss test is defined as area under the ROC curve using model mean per-token loss, with 0.5 as chance and 1.0 as perfect separation.Members are treated as positives.
- Where loss-based membership turns on: 0.60 AUC on the full mid-book bank shows weak loss-based separation, with copy bands from 1 to 999 remaining within overlapping intervals.The 1–9, 10–99, and 100–999 bands score 0.66, 0.59, and 0.57 AUC respectively.
- Where loss-based membership turns on: 0.83 AUC on the famous-line control demonstrates detectable exposure at a median of roughly 1,200 copies.The 95 percent interval is 0.65–0.98 across twelve members and ten edits, so this is a positive control rather than a precise estimate.
5. The differential design
The differential design compares matched OLMo-2 and Pythia losses and corpus counts within books, isolating exposure from sentence quality and book-level taste. Exposure correlations are small within books, while large between-book effects are plausibly taste rather than memory.
- 5.1 What it cancels, and what it does not: The differential test subtracts each sentence’s OLMo and Pythia losses and compares that difference with the log corpus-count difference.Using the same sentence on both sides cancels fluency, quality, length, and topic.
- 5.1 What it cancels, and what it does not: Within-book comparisons remove book-level taste, whereas pooled correlations can mistake family-specific preferences for exposure.The analysis centres loss and count differences on each book and reports between-book correlations separately.
- 5.2 Results: −0.084 at 7B and −0.065 at 13B: within-book exposure correlations are small, accounting for under one percent of loss-difference variance.The 7B result is statistically significant, while the 13B result is marginal.
- 5.2 Results: −0.94 at 7B and −0.49 at 13B: large between-book correlations are not reliable evidence of memorization across six novels.The size comparison moves opposite to the expected memorization trend, though the evidence is compatible with no size trend.
- 5.2 Results: Figure 4’s pooled correlations split by model family, but book-centred correlations for four models remain between −0.016 and −0.056 and are nonsignificant.The pooled pattern is identified as the taste confound that book centring removes.
6. Edit controls manufacture membership
Edit controls make original sentences cheaper than one-word edits, but the gap is flat across ordinary copy counts, indicating word-choice fit rather than membership memory. Above roughly a thousand copies, the gap grows with model size and begins to reflect memory.
- 6. Edit controls manufacture membership: 0.4 nats per token: originals beat one-word edits at every model size, but the gap is unchanged from 1 to 999 copies.The original is about 1.5 times as probable as the edit on average, and the result holds in 92–100 percent of pairs.
- 6. Edit controls manufacture membership: The unchanged ordinary-count gap measures fit: an author’s chosen word suits the sentence better than a near-synonym.The paper illustrates this with “Quite a week” versus “rather a week.”
- 6. Edit controls manufacture membership: Above about a thousand copies, the edit gap grows with model size, reaching 0.932 nats at 13B versus 0.568 at 100 to 999 copies.The same famous-bank pairs score 0.550 at 1B and 0.787 at 7B.
- 6. Edit controls manufacture membership: A paired original-versus-edit test can exceed 0.9 AUC while detecting fluency rather than membership, even when originals appeared only once.The perturbation itself is detectable, so minimally edited member controls inherit the problem.
7. Register does the rest
Changing the controls changes apparent membership performance: for the same twelve famous members, composed-prose controls yield higher AUC than same-source edits, plausibly because of register differences.
- 7. Register does the rest: 0.83 AUC rises to 0.94 when same famous members are compared with composed-prose rather than same-source controls.The intervals are 0.65–0.98 and 0.85–1.00, respectively.
- 7. Register does the rest: The authors identify register as the most plausible source of the 0.11 AUC increase, while noting that the intervals overlap.The famous members remain unchanged across the two comparisons.
- 7. Register does the rest: Figure 6 compares loss-test AUC across mid-book members, famous members, and famous members against composed-prose controls.Bars show 95 percent bootstrap intervals.
8. The pincer
The pincer is weak exposure sensitivity below roughly a thousand copies and strong but confounded sensitivity above it. Larger models provide a preliminary crack in this boundary, but the evidence remains limited.
- 8. The pincer: At high duplication, famous sentences prevent public corpora from varying exposure while holding sentence identity fixed.The authors report that famous lines are commoner in both corpora and have a narrower cross-corpus count ratio than mid-book sentences.
- 8. The pincer: Below the boundary, exposure can be varied across corpora, but the trace reaches only −0.08 in rank correlation at the largest tested sizes.This result concerns the same-sentence design and corresponds to under one percent of the variance.
- 8. The pincer: Exposure is either weakly detected below roughly a thousand copies or confounded with fame above that boundary.The paper calls this the pincer: ordinary duplication levels produce faint traces, while highly duplicated sentences are famous across corpora.
- 8. The pincer: Three proposed experiments could break the pincer: larger models, documented corpora that disagree on famous text, or controlled sequence injection during pretraining.The paper notes that injection would avoid the natural-experiment constraint but require a pretraining run.
- 8. The pincer: At 13B, paired gaps across twelve famous sentences correlate with copy count at 0.206, but the result is based on one bank and 8-bit scores.The two smaller models show correlations of −0.042 and 0.036.
9. Related work
Prior membership-inference work often evaluates classifiers on guessed member and non-member sets, while this paper adds measured copy counts and fixed-sentence exposure variation using public corpora.
- 9. Related work: Prior evaluations found membership tests near chance and sensitive to how non-members are constructed.The cited literature identifies post-hoc set construction and distribution shifts as central weaknesses.
- 9. Related work: This paper adds exact copy counts as a measured exposure dose and varies that dose while keeping the sentence fixed.These additions address limitations described in prior membership-inference evaluations.
- 9. Related work: Unlike extraction, which asks whether a model reproduces a sequence, the paper tests whether sequence loss carries a membership trace.The copy counts come from a different index but measure the same corpus quantity used in prior duplication analyses.
- 9. Related work: Reference-model attacks use a similar-data model as a fluency baseline, whereas this paper uses the second model as a separately measured exposure condition.That design is possible because both pretraining corpora are public.
10. Limitations, and what the release enables
The study is bounded by model size, quantization, narrow English literary data, exact-match counting, absent injection experiments, and a small famous-line bank. Its release provides reusable sentence banks, counts, code, and figure scripts.
- 10. Limitations, and what the release enables: The measurements stop at 13B, use 8-bit precision for 13B runs, and cover 10–16-word English literary sentences from six books.The study also lacks an injected-sequence condition.
- 10. Limitations, and what the release enables: Exact-match counts are floors, OLMo counts omit an annealing stage, and the famous-line bank contains only twelve sentences.These constraints make intervals based on the famous bank correspondingly wide.
- 10. Limitations, and what the release enables: Quantization noise at 13B could flatten a size trend, while larger models are the principal experiment suggested for breaking the pincer.The paper treats precision as the limitation most relevant to interpreting the size trend.
- 10. Limitations, and what the release enables: The release includes sentence banks with both copy counts, scoring and analysis code, and figure scripts under an MIT licence.The paper states that new sentences can be indexed with one query and a 7B bank can be scored in seconds of GPU time.
Appendix A. A method that did not work
Gradient-ascent resistance did not provide membership information beyond the sentence’s starting loss. The authors therefore discontinued this signal and focused on the two-corpus design.
- AUC was 0.54 on the famous bank after regressing out baseline loss, providing little separation at the tested 44-step budget.The budget was the largest tested before unrelated control text began to degrade.
- On a 100-sentence subset of pass 3, the baseline-adjusted resistance measure reached only 0.40 AUC at 46 steps.
- Because resistance carried no information beyond starting loss, the paper does not report the method further.The authors instead built sentence banks and a two-corpus design to test membership signals more directly.