Source-linked AI summary
Surprise Masking: Private Data Filtering with Model Likelihoods
Riya Tyagi, Sahra Ghalebikesabi, Vinnie Monaco, Avi Schwarzschild, Om Thakkar
TL;DR
Useful language-model training data can contain private or unwanted information, while discarding risky examples sacrifices useful content. Surprise Masking uses a public reference model to identify surprising spans and suppress learning from them. It improves privacy–utility tradeoffs in training, though specialized detectors generally perform better on broader credential and PII sanitization benchmarks.
Problem
Useful training examples can contain private information, leaving practitioners to choose between discarding useful samples and using safeguards with costs or limitations.
Source: §1 · pp. 1–2Method
Surprise Masking uses a publicly available reference model to identify unlikely spans and mask them from training loss while retaining the rest of each example.
Source: §1 · p. 2Results
SM extends the privacy–utility frontier in pretraining and fine-tuning, while specialized detectors are generally stronger on broader credential and PII sanitization benchmarks.
Source: §1 · p. 2Takeaways & Limitations
SM is an inexpensive, adaptable complement to scrubbers, rewriters, and other tools in a privacy suite, particularly where private and useful content are separable, as often occurs with code.
Source: §1 · p. 2, §7 · pp. 10–11SM provides no formal privacy guarantee, because removing a span's direct training signal does not prevent learning through context or repeated exposure.
Source: §7 · p. 10
Abstract
from arXiv · showhide
Data useful for language model (LM) training can contain private or unwanted information that is at risk of being learned or memorized. We propose Surprise Masking (SM), a simple form of perplexity filtering that uses a publicly available reference model to identify unwanted spans and suppress learning from them while preserving the rest of each example. On synthetic code benchmarks, we show SM improves the privacy-utility tradeoff over standard masking methods and PII filtering. In fine-tuning experiments, SM reduces approximate extraction of private values from revealed prefixes from 66.3% to 2.3% and the reproduction of stylistic coding quirks from 23.6% to 1.3%, both while preserving performance on the evaluated coding tasks. We also evaluate SM for sanitizing datasets by directly removing unwanted content. Here, SM outperforms DePA, a perplexity-based defense for detecting poisoned dead code, while specialized detectors perform better on broader credential and PII benchmarks. Across eleven settings, we find that SM's downstream success is predictable from the separation of unwanted and useful content in the distribution, and it is particularly useful with code data. Ultimately, SM is an inexpensive, adaptable method to preserve valuable examples while reducing the learning of unwanted information.
1 INTRODUCTION
Useful training examples can also contain private or unwanted content, forcing a tradeoff between discarding valuable data and training with safeguards that have costs or limitations. Surprise Masking uses a public reference model to identify surprising spans for masking; results indicate it can improve privacy–utility tradeoffs in training and is most effective in code.
1 INTRODUCTION
- The motivating tradeoff is that practitioners can discard risky samples and lose useful information, or train with safeguards that have costs and limitations.The passage also notes that training-time interventions lack explicit or theoretical guarantees and therefore need empirical testing.Source: §1 · pp. 1–2
- SM masks spans unlikely under a publicly available reference model to suppress learning of private and other unwanted content.The paper notes that private content may be unfamiliar to the reference model, while sensitive content can be publicly known and therefore not private.Source: §1 · p. 2
- Training-time masking keeps the rest of each example, while sanitization directly removes unwanted content and measures how much other content is preserved.Sanitization may be useful when sensitive information must be removed before dataset release.Source: §1 · p. 2
- During pretraining and fine-tuning, SM extends the privacy–utility frontier beyond existing baselines; for sanitization, it helps on some narrow tasks but specialized detectors are generally stronger on broader credential and PII benchmarks.The paper identifies code as a particularly promising setting and presents SM as an inexpensive, adaptable complement to other privacy tools.Source: §1 · p. 2
2 METHOD
Surprise Masking scores windows with a reference model, selects those meeting a surprise threshold, and converts selected windows into token-level training masks. Its training and sanitization uses differ in how they treat collateral masking and in the resulting utility tradeoff.
2 METHOD
- SM takes an example, reference model, window function, and threshold function, then returns window-level masking decisions that post-processing converts to token-level masks.The reference model should be publicly available and not trained on the private content targeted for exclusion.Source: §2 · pp. 2–3
- Windows may be fixed-width or correspond to complete code lines, document fields, or chat responses; the reference model's mean negative log likelihood scores each window, with higher values indicating greater surprise.Some positions may fall outside all windows and receive no window-based mask.Source: §2 · pp. 2–3
- Global SM applies one threshold to every window, Local SM ranks windows against nearby windows, and Prompted SM conditions the reference model on a description of non-target content.The variants provide different ways to select windows for masking.Source: §2 · p. 3
- The simplest token mask covers every token in at least one selected window; consensus masking requires appearances in at least k selected windows, and duplicate propagation masks repeated selected spans.A masked token is excluded from the loss, but the paper's training-time setup retains it in context for predicting later tokens.Source: §2 · p. 3, §2.1 · p. 3
2.1 USE CASES
- For training interventions, masked tokens remain in context and privacy is measured by unwanted learning; for sanitization, masked tokens are redacted and every removed non-target token counts against utility.The sanitization evaluation covers credentials, personal information, and poisoned code; it measures target recall (TPR) and non-target preservation (1−FPR).Source: §2.1 · p. 3
- SM works best as a training intervention and on narrow sanitization tasks such as detecting poisoned code; frequent false positives limit its usefulness on broader sanitization tasks.On broader tasks, SM can achieve high recall, but collateral masking reduces usefulness.Source: §2.1 · p. 3
3 SURPRISE MASKING SUPPRESSES UNWANTED CODE ATTRIBUTES WHILE PRESERVING UTILITY
Surprise Masking (SM) improves privacy–utility tradeoffs on synthetic code tasks by masking selected content while retaining the surrounding examples. Its coverage of target and non-target tokens helps explain downstream outcomes in both Code PII and Code Quirks.
3 SURPRISE MASKING SUPPRESSES UNWANTED CODE ATTRIBUTES WHILE PRESERVING UTILITY
- Across pretraining and fine-tuning, SM extends the Code PII privacy–utility frontier; extraction falls from 66% to 2% in fine-tuning and from 97% to 2% in pretraining.Random masking stays between 66% and 80% in fine-tuning and above 90% in pretraining; Goldfish reduces extraction at substantial utility cost, while OPF does little to reduce privacy risk.Source: Figure 2 · p. 4, §3 · pp. 4–5
- Code PII embeds repeated synthetic private values in code, while Code Quirks assigns each synthetic user a harmless stylistic habit that PII scrubbers may miss.The datasets use ExStruct tasks and code solutions modified with controlled target spans; comparisons include unmasked, oracle, matched-budget random, Goldfish, and OPF training.Source: §3 · pp. 4–5
- SM cuts Code Quirks display from 24% to 1.3% in fine-tuning (oracle: 1%; matched-budget random: 8.5%) and from 32% to 4–8% in pretraining.At comparable pretraining utility, random masking remains at 14–15% quirk display.Source: Figure 3 · p. 5, §3 · p. 5
3.1 MASK COVERAGE HELPS EXPLAIN PRIVACY AND UTILITY
- On labeled, in-distribution validation data, target-coverage estimates correlate above 0.92 and collateral-masking estimates above 0.95 with training-token masking fractions.Figure 4 relates target-token coverage to extraction or quirk display, and collateral masking to fine-tuning pass@5 and pretraining perplexity.Source: §3.1 · p. 6, Figure 4 · p. 6
- Masking more target tokens generally improves privacy, while masking more non-target tokens generally reduces utility; at matched budgets, SM masks more target and fewer non-target tokens than random masking.Methods with equal target coverage can still differ in extraction risk if one leaves every occurrence of a private value exposed.Source: §3.1 · p. 5, Figure 4 · p. 6
4 SURPRISE MASKING PROVIDES SANITIZATION SIGNAL BUT USUALLY
Surprise Masking’s sanitization performance varies by dataset: it performs strongly on reconstructed dead-code poisoning but trails specialized detectors on broader PII benchmarks. Its results also expose a gap between credential ranking and performance at a single threshold.
Credentials
- On CredData, SM reaches 0.900 AUROC but only 0.471 ± 0.012 character F1; category-specific thresholds raise F1 to 0.504, still trailing OPF and CredSweeper.Both F1 values exceed those of Gitleaks and detect-secrets; the AUROC–F1 gap reflects the difficulty of choosing one threshold that avoids masking too much code.Source: §4 · p. 6, Figure 5 · p. 7
- CredData threshold transfer varies across subsets: out-of-distribution F1 is lower for Credential 1 and backend languages, but higher for Credential 2, Text/Config, and frontend languages.Source: §4 · p. 7
Broad PII
- On PIIBench, Global SM reaches 0.280 character F1 and prompted SM 0.286, below native OPF at 0.325, OPF token log-odds at 0.513, and DeBERTa at 0.785.Thresholds were selected on 10k validation records; results are reported on an approximately 100k-record test set.Source: §4 · p. 7, Figure 5 · p. 7
Dead code poisoning
- 0.639 mean line F1 for Global SM exceeds DePA’s published mean of 0.240 on the reconstructed dead-code benchmark.The comparison uses HumanEval and a reconstructed version of DePA’s data-generation procedure.Source: §4 · p. 7, Figure 5 · p. 7
SM and OPF combinations
- In appendix evaluations, combinations of SM and OPF produce limited gains compared with whichever of the two individual methods is stronger, as reported in Figure 7.Source: §4 · p. 7
5 VALIDATION DATA HELP ASSESS WHERE SURPRISE IS INFORMATIVE
Across eleven settings, surprise separates target from non-target content to varying degrees, and validation ranking usually resembles application ranking. Validation is a useful initial check, but thresholds and downstream outcomes still require separate evaluation.
Evaluation scope
- The comparisons cover training and sanitization benchmarks plus anonymization, insecure code, medical and financial advice, and citation authenticity.The evaluated units and scoring rules differ across settings.Source: §5 · p. 8, Figure 6 · p. 8
Application score separation
- Application AUROC ranges from 0.50 for fabricated citations to 1.00 for financial advice, with stronger separation for dead-code poisoning (0.99) and weaker separation for RedacBench (0.53).These comparisons use different scoring rules and evaluation units across settings.Source: §5 · p. 8, Figure 6 · p. 8
Validation and application data
- In nine of eleven settings, validation and application AUROCs differ by at most 0.03; Code Quirks falls from 0.85 to 0.78 and RedacBench from 0.74 to 0.53.Validation data approximate application data, so these results do not establish robustness under broader distribution shifts.Source: Figure 6 · p. 8, §5 · p. 9
Practical use
- Use representative labeled validation data to check target coverage at an acceptable collateral-masking rate, then verify the chosen threshold on separate representative data using the complete masking procedure.For training-time interventions, separately evaluate extraction or unwanted behavior and task utility; score separation and mask coverage do not determine these outcomes.Source: §5 · p. 9
6 RELATED WORK
Related work places Surprise Masking at the intersection of likelihood-based filtering and selective learning. The paper distinguishes its span-localization approach from methods that select whole documents, suppress known targets, or transform text more broadly.
Likelihood-based anomaly detection
- ONION and DePA use perplexity-based anomaly detection, while SM scores observed tokens directly rather than constructing a deletion variant for each candidate.The DePA comparison uses reconstructed poisoned data, not identical released artifacts.Source: §6 · p. 9
Selective loss masking
- Selective loss methods suppress examples or token contributions for efficiency, task performance, or forgetting; SM tests likelihood-based selection against privacy and utility outcomes.STM is an operational precedent: both it and SM fix likelihood-based selections before fine-tuning and keep masked tokens available as context.Source: §6 · p. 9
Likelihoods for data selection
- Perplexity-based data curation is corpus-dependent, and harmful-content filtering can score specified continuations at the document level rather than localizing observed spans.SM uses application-specific validation to choose an operating point and retains surrounding examples.Source: §6 · pp. 9–10
Extraction and memorization
- Goldfish masks loss positions consistently from context hashes without identifying sensitive spans, whereas SM evaluates reference scores for targeting unwanted content; neither establishes a formal privacy guarantee from loss masking alone.SM’s masked tokens remain visible as context, so its training-time risk reductions are empirical.Source: §6 · p. 10
Privacy filtering and broader privacy goals
- OPF is a supervised token classifier for a predefined privacy taxonomy, while anonymization and semantic-privacy benchmarks address broader editing or outcome goals than span masking.Source: §6 · p. 10
7 CONCLUSION
Surprise Masking’s suitability depends on separating private or unwanted content from useful training signal, a distinction the authors find easier in code. The paper’s datasets span training-time masking, sanitization, anonymization, and several additional content domains.
7 CONCLUSION
- SM is limited when useful but unfamiliar content is masked alongside private information, and its strongest results occur in code where the two are often easier to distinguish.The authors report utility evaluation on one coding library, LoRA fine-tuning, and GPT-2-Small pretraining; SM also provides no formal privacy guarantee against learning from context or repeated exposure.Source: §7 · p. 10
- Practitioners should tune the complete method on deployment-like validation data and use another intervention if private and non-private content do not separate acceptably.The authors recommend one threshold, overlapping-window agreement, and repeated-span propagation for code; neighboring-window ranking may suit broader or more heterogeneous data.Source: §7 · pp. 10–11
A DATASETS
- The dataset suite covers training interventions, sanitization, anonymization, and additional settings including insecure code, advice, and fabricated citations.Training interventions use Code PII and Code Quirks; sanitization uses CredData, PIIBench, and dead-code reconstruction, while anonymization uses RAT-Bench and RedacBench.Source: A DATASETS · p. 15
A.1 CODE PII
- Code PII embeds 100 synthetic private values across 625 executable 24-file repository bundles, with every generated test suite passing.The values include usernames, workspace paths, account identifiers, and recovery credentials; each appears 100 times.Source: §A.1 · pp. 15–16
A.2 CODE QUIRKS
- Code Quirks contains 4,000 correct programs from ten synthetic users, each applying one behavior-preserving stylistic transformation consistently.The labeled targets are disconnected edits such as punctuation, rewritten identifiers, or added statements.Source: §A.2 · pp. 16–17
A.3 CREDDATA
- CredData contains 1,995 credential contexts from 259 repositories, with repository-disjoint validation–application folds and character-level target labels.Its examples include true credentials, false credential-like strings, and ambiguous cases across six credential families and 17 language or document groups.Source: §A.3 · p. 17
A.4 PIIBENCH
- PIIBench annotates 82 personal-information types in text, with character-level overall evaluation and family-specific results on test documents containing each family.The official test set contains 100,002 records; family-specific false-positive estimates apply only to documents containing the evaluated family.Source: §A.4 · p. 17
A.5 DEAD-CODE POISONING
- The dead-code study calibrates on MBPP and applies its line-level mean-NLL threshold unchanged to four HumanEval attack variants, but does not establish detection of subtle insecure completions.The authors reconstruct DePA’s protocol because the original poisoned artifacts and random seeds are unavailable.Source: §A.5 · p. 17
A.7 ADDITIONAL DATASETS
Additional ROC diagnostics test whether surprise distinguishes risky or fabricated content in several settings, with limits on how broadly the held-out checks generalize.
A.7 ADDITIONAL DATASETS
- The insecure-code, medical-advice, and financial-advice diagnostics each use 500 paired completions, while the fabricated-citation diagnostic uses matched genuine and plausible fabricated citations.For citations, validation has 101 examples per class and the official held-out split has 93 per class; both classes share formatting and recognizable publication details.Source: §A.7 · p. 18
- Advice evaluation uses internally held-out groups rather than independent unseen benchmarks, and financial-control rewrites may differ in wording, length, or generating model.The authors inspected full collections and earlier exploratory splits before fixing the grouped splits; the advice records retain their original redistribution and training-use restrictions.Source: §A.7 · p. 18
B EVALUATION METRICS
The evaluation metrics quantify extraction and task utility for training interventions, masking quality for sanitization, and identity or disclosure outcomes for anonymization.
B.1 CODE PII EXTRACTION
- Approximate extraction measures whether any of 128 continuations from each revealed canary prefix matches the hidden suffix under edit distance.The reported rate is the fraction of distinct private values recovered at least once, also assessed across smaller continuation counts.Source: §B.1 · pp. 18–19
B.2 CODE PII UTILITY
- Code PII fine-tuning utility is pass@5 on 300 development prompts, while pretraining utility is mean negative log likelihood on 30 development tasks.The pretraining figures and tables use different utility directions: token log likelihood is better when higher, whereas perplexity and NLL are better when lower.Source: §B.2 · p. 19
B.3 CODE QUIRKS REPRODUCTION AND UTILITY
- Code Quirks reproduction averages the ten quirk rates equally, counting syntactic-pattern matches regardless of whether generated code passes functional tests.Fine-tuning utility is pass@5 on 60 tasks; pretraining utility is NLL on 30 tasks, with separate functional checks.Source: §B.3 · p. 19
B.4 SANITIZATION
- Sanitization reports TPR for target content masked, FPR for other content masked, and F1; preservation of other content is 1 − FPR.Character counts are pooled for CredData and PIIBench F1, and AUROC measures target-versus-nontarget ranking, with ties counting one-half.Source: §B.4 · p. 19
B.5 ANONYMIZATION AND RE-IDENTIFICATION
- RAT-Bench security is the fraction of transcripts where the attacker fails to identify the person, while transcript BLEU measures wording similarity rather than factual retention.RedacBench separately tracks whether sensitive propositions cease to be disclosed and nonsensitive propositions remain available.Source: §B.5 · p. 20
C TRAINING DETAILS
The training and scoring procedures fix masks from validation-selected rules, then apply them consistently across seeds. Surprise Masking includes global and local window selection, overlap consensus, prompted scoring variants, and alternative training interventions.
C TRAINING DETAILS
- Validation-selected scoring rules and thresholds are fixed before training, and all seeds use the same masks.Training excludes selected tokens from loss while retaining them in context; selection intervals are projected onto downstream tokenization as needed.Source: C TRAINING DETAILS · p. 20
C.1 CODE PII FINE-TUNING AND PRETRAINING
- Code PII pretraining and fine-tuning use GPT-OSS-20B reference scores; Code Quirks uses the same LoRA configuration with a separate training mixture.The GPT-2-small pretraining conditions include 248,321,000 OpenWebText tokens and 1,679,000 Code PII tokens; Code Quirks processes 3.75 billion tokens.Source: §C.1 · p. 20, §C.2 · p. 21
C.3 BASELINES AND CONTROL CONDITIONS
- Baselines include exact-span oracles, token-budget-matched random masks, Goldfish loss, and OpenAI Privacy Filter variants.Native OPF uses decoded positive labels; thresholded and soft OPF add validation-selected surprise thresholds.Source: §C.3 · p. 21
D.1 GLOBAL WINDOW SCORING
- Global scoring selects windows whose mean reference-model negative log likelihood meets or exceeds a threshold.Invalid windows with missing or nonfinite scores are not selected.Source: §D.1 · p. 22
D.2 LOCAL RANKING
- Local ranking compares each Code Quirks window’s mean log probability with up to 32 neighboring windows and selects ranks below rmax.At sequence boundaries, only available neighbors are used, and ties count against selection; this neighborhood does not limit reference-model attention.Source: §D.2 · p. 22
D.3 CONSENSUS AND SEQUENCE BOUNDARIES
- Overlapping selected windows vote on token positions, and tokens meeting consensus level k are selected, with duplicate propagation as an additional route.Tokens covered by fewer than k valid windows can only be selected through duplicate propagation.Source: §D.3 · p. 22, §D.5 · p. 23
D.4 PROMPTED AND CONTRASTIVE SCORES
- Prompted and contrastive variants score content under compliant or violating instructions, while other settings use prompted likelihoods or line- and sentence-level scores.The methods vary by benchmark: PIIBench and RAT-Bench compare prompted likelihoods, medical and financial advice use prompted single-likelihood scores, and insecure code combines prompt-change and line-NLL features.Source: §D.4 · p. 22, §D.5 · p. 22
E.1 MASKING ACTIONS
- Filtering selected-token records leaves 293 examples and, under the fixed schedule, yields 76.79 effective passes versus two for the other conditions.The compared masking actions use the same SM95 selection, while the filtering condition retains the fixed 1,125-step schedule and batch size 20.Source: §E.1 · p. 23, Table 6 · p. 23
E.3 EXTRACTION VERSUS SAMPLE COUNT
The Code Quirks results are tabulated for fine-tuning and pretraining, while combining SM with OPF does not improve overall performance despite a PIIBench F1 increase under residual prompting.
E.3 EXTRACTION VERSUS SAMPLE COUNT
- The Code PII extraction table reports approximate extraction across continuation counts; its budget conditions cover two SM95 seeds and yield 2.5% at 128 samples, rather than the three-seed mean of 2.3%.The table uses temperature 1 and top p = 1; SM conditions mask loss.Source: Table 8 · p. 24
E.4 CODE QUIRKS RESULTS
- Tables 9 and 10 report Code Quirks display, utility, and seed counts for fine-tuning and pretraining; baseline conditions use a single seed.Table 9 covers fine-tuning on 60 development tasks, with Display@5 averaged equally across ten quirks; Table 10 reports pretraining NLL on 30 development tasks and quirk display from unexecuted samples.Source: §E.4 · p. 24, Table 9 · p. 24, Table 10 · p. 25
F.1 COMBINING SURPRISE MASKING WITH PII SCRUBBERS YIELDS LIMITED GAINS
- Adding prompted residual SM to OPF token log-odds raises PIIBench F1 from 0.513 to 0.517, while the jointly selected union does not improve overall performance.The methods’ union closely tracks OPF on CredData and SM on Code PII and Code Quirks, and falls below OPF on PIIBench.Source: §F.1 · p. 25
F.2 CREDDATA AND PIIBENCH RESULTS
Gold-category thresholds improve aggregate CredData character F1, while PIIBench joint threshold selection has lower F1 on the official test than on validation. The dead-code section describes a reconstructed DePA calibration setup.
F.2 CREDDATA AND PIIBENCH RESULTS
- 0.504±0.013 aggregate CredData F1 with gold-category thresholds exceeds 0.471±0.012, but requires a category oracle.The standard deviations are over three folds; each method otherwise uses one validation threshold per fold across categories.Source: §F.2 · p. 25, Table 11 · p. 26
- Joint selection among 576 PIIBench threshold pairs gives F1 0.5026 on 4,937 validation records and 0.3549 on the official test.Table 14 covers combinations on the corrected 100,002-record test set.Source: §F.2 · p. 25, Table 14 · p. 27
F.3 DEAD CODE SANITIZATION BY ATTACK
- The reconstructed DePA study calibrates a 3.583587-nat-per-token threshold on five MBPP folds and applies it to every HumanEval attack.Published DePA values are from the original report; the poisoned artifacts were unavailable for evaluation.Source: §F.3 · p. 28, Table 15 · p. 28
F.4 RAT-BENCH AND REDACBENCH RESULTS
The anonymization results include an OPF routing cost above full-document rewriting at a 20% RedacBench routing rate. The accompanying analysis bounds generation probability under a specific global masking rule and explains how decoding choices affect span probability.
F.4 RAT-BENCH AND REDACBENCH RESULTS
- At 20% RedacBench routing, OPF costs $41.97 per million source tokens, above $37.53 for a full document rewrite.OPF makes 1,402 calls across 513 documents, using 789,669 input and 56,832 output tokens; repeated policy and context raise the routing cost.Source: §F.4 · p. 30
G GENERATION-PROBABILITY BOUND FOR GLOBAL SURPRISE MASKING
- The generation-probability bound applies only to global length-n window masking; exact duplicate propagation preserves it by only adding masked positions.The anonymization figures compare security, utility, and cost, with different security definitions for RAT-Bench and RedacBench; direct masking and routing use different intervention budgets.Source: Table 16 · p. 28, Table 17 · p. 28, Figure 9 · p. 29
G.1 PROBABILITY BOUND
- The probability bound assumes every fully contained length-n window is scored, finite token log probabilities, and masking all tokens covered by a window at or above threshold.Repeated independent samples from the same reference-model conditional distribution have a geometric waiting time, which yields the expected-sample bound.Source: §G.1 · p. 30
G.2 TEMPERATURE AND DECODING
- Temperature can raise or lower a target span’s probability, while top-p truncation can make it zero by excluding a required token.As temperature approaches zero, the span probability approaches one if each target token is the unique maximum, and zero if any target token is not a maximum; ties share the limiting mass.Source: §G.2 · p. 31