PaperlayerView exact revision on arXiv

Source-linked AI summary

TRAP: Understanding and Mitigating Privacy Memorization in Language Models

Muhammed Ustaomeroglu, Ziyue Xu, Hanshen Xiao, Peter Cnudde, Guannan Qu, Holger R. Roth

arXiv:2609.32293v1cs.CLcs.AIcs.LG

TL;DR

Fine-tuning on sensitive records can leave language models able to reproduce personal information, raising questions about when memorization occurs and how to mitigate it without identifying sensitive spans in advance. The paper measures tokenwise memorization with TRA, a target-versus-complementary-reference comparison, and studies its drivers before introducing TRAP, a penalty on excess TRA. Across the evaluated testbeds, TRAP removes most memorization near undefended utility, while compared generic regularizers barely change memorization and differential privacy often sacrifices much of fine-tuning’s utility.

  • Problem

    Fine-tuning on sensitive records can leave models able to reproduce particular records, motivating ways to understand and reduce memorization without advance identification of sensitive spans.

    Source: §1 · p. 1, §4 · p. 7

  • Method

    The paper compares a target model’s token likelihood with a model trained on the complementary data half to define TRA, then uses a one-sided tokenwise penalty on TRA in TRAP.

    Source: §1 · p. 2

  • Results

    Across testbeds, TRAP removes most memorization near undefended utility, while generic regularizers barely move and differential privacy gives up most of fine-tuning’s utility.

    Source: §1 · p. 2, §4 · p. 9

  • Takeaways & Limitations

    The reported comparison favors TRAP for reducing memorization while retaining utility across the evaluated settings.

    Source: §1 · p. 2, §4 · p. 9

    The comparison excludes compute costs, treating total compute as comparable across methods.

    Source: §4 · p. 7

Abstract

from arXiv · show

Fine-tuning a language model on sensitive records can leave it able to reproduce them. We ask when this memorization arises and how to prevent it without knowing in advance which spans are sensitive. Our starting point is that most memorization scores and attacks share one statistical core: whether the model assigns a token more probability than some reference would. Taking as the reference a model trained on the complementary half of the same corpus gives the Target Reference Advantage (TRA), a per-token signal that separates what a model fit to a particular record from what it learned across records, and is cheap and differentiable. We then study what drives memorization during fine-tuning: it keeps growing well past the validation minimum, is larger on small datasets and at higher learning rates, and higher when the underlying task is harder. Early stopping removes much of it, but because it is chosen by aggregate validation loss it helps least for rare, hard-to-predict spans embedded in otherwise learnable text, which is exactly what sensitive information tends to be. We therefore introduce TRAP, a one-sided penalty on tokenwise TRA that acts only where the target model pulls ahead of its reference. On student essays with annotated personal information and clinical cases with patient identifiers, TRAP brings memorization near the level of an untrained model at little utility cost, where generic regularizers barely move and differential privacy gives up most of what fine-tuning bought.

1 INTRODUCTION

Fine-tuning on sensitive records can expose their specific contents, leaving organizations reluctant to use valuable data. The paper introduces TRA as a reference-based memorization signal, studies when memorization arises, and presents TRAP as a mitigation that requires no sensitive-token annotations.

1 INTRODUCTION

  • Fine-tuning can make models reproduce particular sensitive records, and sharing model parameters rather than data does not remove that risk.This can leave organizations avoiding the use of valuable sensitive data.Source: §1 · p. 1
  • TRA compares a target model’s token likelihood with a reference, and using a model trained on the other data half yields a cheap, differentiable signal whose expectation is the generalization gap.The reference comparison aims to separate record-specific fit from what is supported across records.Source: §1 · pp. 1–2
  • The study maps sensitive-span memorization across student essays, synthetic applicant dossiers, and clinical case reports, examining training duration, data size, learning rate, task hardness, data mixtures, and model size.TRAP applies a one-sided penalty to tokenwise TRA without requiring annotation of sensitive tokens.Source: §1 · p. 2
  • Across three testbeds and models up to 8B parameters, TRAP lowers worst-case leakage at little utility cost and outperforms a broad set of existing defenses.The paper reports that it moves the memorization–utility frontier further than the compared baselines.Source: §1 · p. 2

2 TARGET REFERENCE ADVANTAGE AND TRAP

TRA measures a target model’s tokenwise likelihood advantage over a reference trained on complementary data, distinguishing target-record-specific fit from patterns learned across records. TRAP uses this signal as a one-sided tokenwise training penalty, without requiring sensitive-token labels.

2.1 TARGET REFERENCE ADVANTAGE AS A MEMORIZATION SIGNAL

  • TRA compares the target model’s average token log-likelihood with a reference conditional distribution, with the choice of reference determining which predictive advantage counts as memorization.A uniform reference counts learned structure beyond random prediction, while a language-model reference discounts ordinary linguistic structure.Source: §2.1 · p. 2
  • Training target and reference models independently on complementary corpus halves makes positive TRA identify tokens the target predicts better, separating target-record-specific fit from patterns supported by both halves.Restricting evaluation to sensitive positions measures this advantage on the spans of interest.Source: §2.1 · pp. 2–3
  • When evaluated over all tokens trained by fine-tuning, TRA’s expected value on a random training example equals the expected generalization gap.The passage characterizes span memorization measured by TRA as a local form of that aggregate quantity.Source: §2.1 · p. 3

2.2 TARGET REFERENCE ADVANTAGE PENALTY

  • TRA requires one target and one reference rather than many subset-trained models, and its tokenwise form is differentiable, allowing it to serve as a training signal.The reference model is trained on complementary data; the described frozen-reference variant adds one forward pass per step, at most about a third more compute.Source: §2.2 · pp. 3–4
  • TRAP adds a per-token hinge penalty only when TRA exceeds a threshold, applying it to every trained token without needing labels for sensitive tokens.Below threshold, tokens contribute their ordinary cross-entropy gradient; above it, the gradient is scaled by 1 − w, removed at w = 1, or reversed for w > 1.Source: §2.2 · p. 3
  • If every token in a span of length L stays below threshold ε, the target assigns the span at most e^(Lε) times the reference probability, but TRAP training no longer preserves TRA’s expected-generalization-gap identity.The identity fails because the target is trained with the penalty while the reference is not.Source: §2.2 · p. 3, §2.1 · p. 3

3 WHEN DO LMS MEMORIZE SENSITIVE INFORMATION?

Fine-tuning memorizes information specific to individual records, especially when training continues beyond the validation minimum or the task is difficult. Validation-based stopping helps, but can miss rare, hard-to-predict spans within otherwise learnable text.

3 WHEN DO LMS MEMORIZE SENSITIVE INFORMATION?

  • The study measures sensitive spans across essays, synthetic applicant dossiers, and clinical cases, comparing target and complementary-half models alongside other span scores.Essays use annotated personal information; APEX and MedCase are evaluated closed-book, with their sensitive fields scored as completions.Source: §3 · p. 4

3.1 WHAT MOVES MEMORIZATION

  • After the best-validation checkpoint, memorization keeps rising until roughly ten times that epoch before plateauing, so early stopping removes much but not all of it.The large majority of configuration pairs also show more memorization when trained longer.Source: §3.1 · p. 5, Figure 1 · p. 6
  • For APEX-L, unlearnable rules are memorized more than learnable ones at best validation, while harder unlearnable rules show more memorization when training continues.At best validation, the unlearnable levels are not ordered because validation loss bottoms out before most record-specific binding forms.Source: §3.1 · p. 5, Figure 1 · p. 6
  • The rarer the impossible-to-predict minority, the more it is memorized; at the rarest share, it is memorized more in a mixture than when trained alone.The majority keeps validation loss improving, leading to additional passes over minority examples that can only be memorized.Source: §3.1 · pp. 5–6, Figure 1 · p. 6
  • Higher learning rates increase memorization across testbeds, and smaller training sets increase it across testbeds and regimes.Memorization at a higher rate remains greater when runs are compared at the same training loss.Source: §3.1 · pp. 6–7
  • Model-size effects depend on the setting: overtrained larger models memorize more, but best-validation results show no consistent size pattern.For essays, the size ordering depends on learning rate; MedCase shows more memorization for larger models in both regimes.Source: §3.1 · p. 7

4 MITIGATING PRIVACY MEMORIZATION

TRAP improves the memorization–utility trade-off across the evaluated panels, while most generic regularizers remain near the undefended model. Its reference-based penalty targets example-specific fit without sensitive-span annotations.

4 MITIGATING PRIVACY MEMORIZATION

  • Evaluation uses each model’s best-validation epoch and compares held-out cross-entropy with the worst nAUC across four scores, sweeping method-specific hyperparameters without sensitive-token annotations.DP results are measured scores, not certified guarantees; the displayed panels cover essays and MedCase at Pythia-1B and the rare-minority mixture at Pythia-160M.Source: §4 · pp. 7–9, Figure 2 · p. 8
  • Across essays, MedCase, and the rare-minority mixture, TRAP improves the memorization–utility frontier; TRAP + LoRA is best on essays, while TRAP leads on MedCase and the mixture.Figure 2 compares held-out cross-entropy with worst-case nAUC; lower is better on both.Source: Figure 2 · p. 8, §4 · p. 9
  • On the rare-minority mixture, the undefended model nearly perfectly memorizes twenty secrets despite healthy aggregate loss; TRAP brings worst-case memorization near the floor for a few thousandths of a nat.TRAP + LoRA reaches that level below the undefended cross-entropy, without labels identifying the minority.Source: §4 · p. 9
  • Generic objectives and regularizers mostly sit near the undefended model, while DP and, on MedCase, LoRA reach the floor only after giving up much of fine-tuning’s utility.LoRA competes on essays; on MedCase and the mixture, TRAP leads.Source: §4 · p. 9
  • TRAP penalizes target-over-reference advantage, intended to isolate example-specific fit; other defenses act on signals such as gradient size, loss, confidence, random masking, or model capacity.The adapter limits how much can be stored, while TRAP targets what is stored.Source: §4 · p. 9

5 CONCLUSION

The conclusion presents TRA as a reference-based measure and TRAP as a tokenwise defense, reporting improved trade-offs alongside empirical limitations. The related-work discussion situates these contributions among prior measurement, memorization-driver, and mitigation research.

5 CONCLUSION

  • Memorization grows with training past best validation, smaller datasets, higher learning rates, harder tasks, and rare subgroups; no consistent model-size effect appears at best validation.Early stopping removes much, but not all, memorization; lower learning rates memorize less at matched task performance, for reasons the authors could not explain.Source: §5 · p. 9
  • TRAP yields a better memorization–utility trade-off than differential privacy, adapters, and a dozen regularizers on every testbed run, but remains an empirical defense rather than a formal guarantee.The attacker is assumed not to adapt; calibration against an attacker’s own shadow models is untested.Source: §5 · p. 9
  • The reported model learns from only half the data; the authors note that mutually defended models could use both halves, while leaving no data unused.The conclusion also identifies the unexplained lower-learning-rate association as an open question.Source: §5 · p. 9

A BACKGROUND AND RELATED WORK

  • Prior work studies membership inference, exposure, extraction, and counterfactual memorization; whether membership scores work on large models remains contested.Earlier findings also connect memorization to model size, duplication, and context length (Carlini et al., 2023).Source: A BACKGROUND AND RELATED WORK · p. 13
  • Differential privacy offers a formal guarantee at a utility cost, while TRAP instead penalizes target-over-reference likelihood advantage token by token without sensitive-token annotations.Related defenses include overfitting-oriented tools, token dropping, parameter-efficient fine-tuning, federated training, and reference-divergence approaches.Source: A BACKGROUND AND RELATED WORK · pp. 13–14

B A COMMON LIKELIHOOD-AND-REFERENCE VIEW OF MEMORIZATION

Memorization scores and attacks compare the model’s probability for evaluated tokens with a reference, but differ in how they choose that reference and make decisions. TRA uses a model trained on the complementary half of the corpus, providing a task-matched comparison for sensitive spans.

B A COMMON LIKELIHOOD-AND-REFERENCE VIEW OF MEMORIZATION

  • Most memorization measures compare the probability of the evaluated tokens with a reference, while attacks apply a decision rule to that comparison.The reference varies across methods, including fixed thresholds, model predictions, decoys, perturbed texts, compressors, language models, and models trained without the text.Source: B A COMMON LIKELIHOOD-AND-REFERENCE… · p. 14, §B.8 · pp. 17–18

B.1 LOSS, PERPLEXITY, AND THE PROBABILITY SCORE

  • LOSS uses mean token log-likelihood and a fixed threshold, so text predictable to everyone can resemble text memorized through training.LOSS, perplexity, and exact-span probability rank spans identically and give the same AUC.Source: §B.1 · p. 14

B.2 MIN-K%, MIN-K%++, AND LOSS++

  • Min-K% focuses on the least likely tokens, while Min-K%++ standardizes token log-probabilities against the model’s own predictions.For short spans, the paper uses K = 100%, making Min-K% equivalent to LOSS and Min-K%++ the mean standardized score called loss++.Source: §B.2 · p. 15

B.3 EXPOSURE

  • Exposure ranks the true span among decoy values, and its values are comparable only across spans using the same kind and number of decoys.The testbed uses up to 63 same-type decoys, limiting exposure to at most 6 bits.Source: §B.3 · pp. 15–16

B.4 NEIGHBORHOOD REFERENCES

  • Neighborhood, compression, transformation, and language-model references estimate how probable a text should be, but a generic language model does not discount task-specific learning.A target can therefore appear to memorize a span it has never seen when compared with a generic reference.Source: §B.4 · p. 16, §B.5 · p. 16, §B.6 · p. 16

B.7 COUNTERFACTUAL MEMORIZATION AND TRA

  • TRA compares a target trained on a record with a same-recipe reference trained on the complementary half, matching the task while isolating the record’s contribution.In expectation, one target–reference pair corresponds to the counterfactual effect of including the record; estimating that effect through separate expectations requires many trained models.Source: §B.7 · pp. 16–17

B.8 DEFINITIONS AND ATTACKS ARE THE SAME THING USED DIFFERENTLY

C.1 Essays: STUDENT ESSAYS WITH ANNOTATED PERSONAL INFORMATION

  • The essay testbed uses token-labeled personal-information spans, document-level splits, and a validation set for checkpoint selection and early stopping.The training halves contain 3,268 and 3,267 documents; the test set contains 200 documents and 48 labeled spans.Source: §C.1 · p. 18, C TESTBED CONSTRUCTION · p. 18

C.2 APEX: APPLICANT DOSSIERS WITH A TARGET OF CONTROLLED DIFFICULTY

The appendix describes synthetic applicant dossiers and related medical testbeds designed to separate copying, rule learning, and memorization, then specifies mixture, provenance, and sweep setups. Its experiments use synthetic personal fields alongside essays and clinical cases, with scoring focused on sensitive spans or task completions.

C.2 APEX: APPLICANT DOSSIERS WITH A TARGET OF CONTROLLED DIFFICULTY

  • APEX uses noisy synthetic applicant dossiers with details scattered among distractor applicants; copyable fields appear verbatim, while derived and opaque fields vary in difficulty.Changing target class or rule level leaves the dossier unchanged, isolating the target-field difficulty.Source: §C.2 · pp. 18–20

C.3 THE APEX GENERATION PROMPT

  • The APEX generation process fills placeholders with synthetic values under uniqueness constraints and checks that each value appears exactly once in the generated dossier.Dossiers are generated without real personal information.Source: §C.2 · p. 19, §C.3 · p. 21

C.4 MedCase: AN IDENTIFIER BOUND TO A DIAGNOSIS

  • MedCase pairs clinician-authored diagnostic cases with unique synthetic identifiers, then evaluates diagnosis reproduction with the case removed and only the identifier supplied.A correct closed-book diagnosis therefore requires memorizing the identifier-to-diagnosis association.Source: §C.4 · p. 22

C.5 MEDEX AND THE CROSS-CORPUS MIXTURES

  • MEDEX combines copyable patient fields with opaque secrets, while cross-corpus mixtures place MedCase examples among APEX or MEDEX examples and compare them with matched controls.The mixture variants also include a flipped control and, in one case, a reasoning-plus-diagnosis target.Source: §C.5 · p. 22

C.6 PROVENANCE AND LICENSING

  • APEX and MEDEX use synthetic personal values, MedCase uses synthetic identifiers attached to published case text, and essays retain surrogate personal information in their labelled text.The external-source licenses must be confirmed before releasing derived data.Source: §C.6 · p. 22

D DETAILED SWEEPS FOR UNDERSTANDING MEMORIZATION

  • The detailed sweeps use plain cross-entropy fine-tuning and score sensitive spans; the smallest configurations have wide chance bands and should not be ranked.Overtraining budgets differ by testbed, and no series pools runs from local machines and cluster runs.Source: D DETAILED SWEEPS FOR UNDERSTANDING… · p. 23

D.1 TRAINING DURATION

Memorization generally increases when training continues beyond the validation minimum, though the magnitude and timing vary across testbeds and metrics.

D.1 TRAINING DURATION

  • 91–94% of APEX-full grid configurations worsened on secret-focused metrics with continued training, versus 66–69% on metrics averaged over all example tokens.The essays’ probability-score AUC rose from 0.94 to 1.00 while held-out cross-entropy rose from 3.5 to 11.1 nats.Source: §D.1 · p. 23, Table 3 · p. 24
  • Validation minima occur early in many runs, and memorization is already rising then; on two essay runs, near-tied validation losses selected checkpoints with leakage differing by about a factor of two.The 1B essays model memorized more and faster than 70M under every score, while APEX-L rules L2–L5 would have kept rising after their early minima.Source: §D.1 · p. 23
  • Late in MedCase training, loss++ becomes erratic as its per-position scale collapses, while probability, exposure, and TRA continue separating members from held-out spans.This behavior occurs once training loss is near zero.Source: §D.1 · p. 24

D.2 TRAINING-SET SIZE

Memorization generally declines as training-set size increases, although extra training passes do not explain the pattern across most sources.

D.2 TRAINING-SET SIZE

  • The APEX-full best-validation size marginal has Spearman ρ from −0.75 to −0.96 across metrics, with p<0.05 for all but MinK%++ (p=0.07).In overtrained runs, the probability ratio fell from about 5500× at 20 applicants to 2.7× at 2198.Source: §D.2 · p. 24
  • The size effect persists after controlling for training passes on the APEX-full hardest rule, where pass counts show no significant size trend (Spearman ρ=−0.34, exact p=0.46).The target-secret log-probability AUC fell from 0.92 at 50 examples to 0.66 at 2198; the partial Spearman association is descriptive only.Source: §D.2 · p. 24
  • MedCase is the exception to the pass-count pattern: its smallest sets stop several epochs later than larger sets, so extra passes may explain part of its size effect.The passage reports this pattern across several model scales and learning rates.Source: §D.2 · p. 24

D.3 TASK HARDNESS

APEX-L varies the learnability of a hidden rule while keeping dossiers fixed; memorization is strongest for rules that cannot be learned from unseen applicants and continues after early validation minima.

D.3 TASK HARDNESS

  • L1 was learned across the three scales, while L2–L5 were not; the learned rule’s validation minimum came at epochs 15–53, whereas the unlearned rules bottomed out within one to ten epochs and their scores then rose.Closed-book exact-identifier reproduction appeared only for unlearned rules and only when training continued.Source: §D.3 · pp. 24, 26
  • 15–20% of training applicants’ exact identifiers were reproduced at L4 after continued training for Pythia-160M and 1B, while the matched reference and unseen applicants remained at zero.Across 70 run-checkpoints, no run had both accuracy and TRA nAUC above one half.Source: §D.3 · pp. 24, 26
  • After 425 epochs on 200 applicants, the secret probability ratio rose from under 10× at the learnable rule to several thousand times at the hardest rule, with run-level Spearman ρ=0.9–1.0 across three seeds.The probability score reached AUC 0.99 and a true-positive rate of 0.93 at 1% false positives, while copied-field AUC stayed flat.Source: §D.3 · p. 26
  • In the mixture ladder, the validation minimum arrived at epoch 4–18 for every rule; after ten epochs the target reproduced 30–80% of minority secrets while the reference reproduced none.The probability ratio on the secret reached up to 70×.Source: §D.3 · p. 27

D.4 MIXTURES

The mixture experiments show that memorization of rare secrets can rise when a learnable majority keeps training after the minority itself would stop. Cross-corpus results qualify this pattern: it depends on task, model scale, and training dynamics.

D.4 MIXTURES

  • At 1% share, the minority’s probability ratio was 66× higher at 70M and 15× higher at 160M inside the mixture than in the matched-count pure control.Exact extraction was seven and two times more frequent, respectively; the seed spread at 1% was wide.Source: §D.4 · p. 28
  • At 1% share, MedCase secrets in APEX or MEDEX mixtures reached a 4417× probability ratio versus 5.1× for the same 20 cases trained alone.Swapping the roles so the majority was the unlearnable task stopped training at epoch 3 to 4 and left minority ratios at 0.92 to 1.18×.Source: §D.4 · p. 28
  • The cross-corpus mixture effect did not appear at Pythia-1B or Llama-3.1-8B, or when the minority target was a long reasoning narrative rather than a short label.In one mixture, a low-rate run trained for 129 epochs and stayed at chance, while the undefended run reached 4417× in 24 epochs.Source: §D.4 · p. 28

D.5 LEARNING RATE

Learning-rate sweeps generally show more memorization at higher rates, but the pattern depends on the score and can coincide with worse held-out fit. Sequential training and minority mixtures further illustrate how per-example update size and validation stopping shape memorization.

D.5 LEARNING RATE

  • 14 of 15 learning-rate series rose in mean nAUC across the four scores at the best-validation checkpoint.The sweeps covered essays, APEX and MedCase across rates, model scales and training-set sizes.Source: §D.5 · p. 28
  • At matched training loss, the smallest rate leaked less on MedCase for every seed, and remained lowest across the overlapping rate range on essays.After all MedCase runs drove training loss to zero, the smallest and largest rates still differed by 0.3 to 0.4 under the probability score.Source: §D.5 · p. 31
  • The rate effect depends on the score: probability rose from near chance to about 0.5 on essays and from about 0.3 to 0.8 on MedCase, while TRA moved less.TRA compares target confidence with a reference trained at the same rate, so rate-wide confidence increases cancel and the remaining signal is specific to the target’s records.Source: §D.5 · p. 31
  • Held-out cross-entropy worsened with learning rate throughout, gently up to 2 · 10−4 and sharply at 5 · 10−4, where MedCase reached several nats and APEX lost an order of magnitude.The rate changes therefore coincided with both memorization-score changes and held-out-fit changes.Source: §D.5 · p. 32
  • In cross-corpus mixtures, lower-rate or adapter runs trained for many more epochs yet stayed near chance, while the memorizing runs used full steps.DP runs had no validation-selected epoch and were not shown in the comparison.Source: Figure 19 · p. 34
  • On MedCase, one-example-at-a-time training memorized as much in a single pass as seventy batch-recipe epochs at the same rate, while ending with worse validation loss.The example-sized step gives one case the update that a batch step spreads over sixteen MedCase cases; on essays, one such step imprints little.Source: §D.5 · p. 34

D.6 MODEL SIZE

Model-size effects vary with dataset and learning rate: size-related increases were strongest for small MedCase sets, while the essays’ ordering depended on rate and APEX-L showed little size effect. The appendix also explains how to read the full mitigation frontiers and their controls.

D.6 MODEL SIZE

  • With 20 MedCase cases, best-val nAUC rose from about 0.3 at 70M to about 0.8 from 410M on; the overtrained arm rose from 0.65 to about 0.8.Both arms rose more gently with 200 cases, while with 2000 cases they stayed low up to 410M and best-val rose only at 1B and 1.4B; each 20-case cell had one seed.Source: §D.6 · p. 35
  • On essays, the 1B model memorized most at each shared rate, while APEX-L sizes stayed near 0.25 at best-val and near 0.55 at the last scored epoch.In the scale-adapted essay series, 1B appeared to memorize less than 70M, but that comparison changed rates as well as sizes; APEX-L’s small size effect reflects learnable levels being learned and unlearnable levels memorized at every size.Source: §D.6 · pp. 35–36

E FULL MITIGATION FRONTIERS

  • The full mitigation frontiers show each method’s lower-left boundary, with union-trained twins as more-data controls and the untrained model as an empirical floor.The hollow markers and dashed lines denote union-trained runs; methods using sensitive-token annotations are omitted.Source: E FULL MITIGATION FRONTIERS · p. 40

E.1 EVERY METHOD ON ITS OWN

The method-by-method frontiers compare privacy and held-out utility across testbeds, with results depending on setting and defense. The appendix clarifies frontier construction, score-specific behavior, directional over-correction, and the distinct risks of iterated references.

E.1 EVERY METHOD ON ITS OWN

  • In the rare-minority mixture, TRAP through an adapter at weight 1 or 2 sat at nAUC floor levels of 0.11 to 0.16 at 160M with cross-entropy at or below the undefended model’s.On this testbed, DP settings bringing leakage to 0.14 to 0.18 cost four to eighteen times the undefended cross-entropy, versus about 1.7 times for TRAP at weight 4.Source: §E.1 · pp. 40–41

E.2 THE SETTINGS BEHIND THE FRONTIERS

  • Frontier tables list each method’s settings and the associated held-out cross-entropy and worst-case nAUC, retaining only settings not beaten on both axes by another setting of that method.Runs use shared testbed recipes and best-validation stopping except for the fixed forty-epoch overtrained campaign; objectives and regularizers either kept undefended memorization or gave up utility before it moved.Source: §E.2 · p. 41

E.3 ONE SCORE AT A TIME

  • TRAP’s reduction holds under all four scores, although which score is strongest varies by testbed; TRA requires access to the reference model.On essays, exposure and TRA expose the undefended model most, while on MedCase all four scores are high.Source: §E.3 · p. 41

E.4 DIRECTIONAL AUC VERSIONS OF THE FRONTIERS

  • Directional AUC points below 0.5 indicate over-correction, where held-out spans rank above training spans; nAUC folds these cases upward.The strongest TRAP settings do this on essays at 1B, MedCase and the mixture, as do DP models at the floor; defended models should be compared with the untrained empirical floor, not zero.Source: §E.4 · pp. 41, 53

F REFERENCE SCHEDULES

  • Iterated reference rounds update the frozen reference from the previous round’s target, but recurring secrets can then become high-probability in the reference and stop the penalty firing.Unique secrets in these testbeds avoid that failure mode; for recurring secrets the paper identifies the frozen reference as the schedule to use.Source: F REFERENCE SCHEDULES · p. 59

F.1 OTHER REFERENCES, AND WHY THEY ARE NOT USED

The reference schedule determines whether TRAP reduces memorization while preserving utility: a frozen reference trained on the same task but not the target examples performs best. The appendix also states three properties of TRA and TRAP.

F.1 OTHER REFERENCES, AND WHY THEY ARE NOT USED

  • The compared schedules include lockstep, mutual, alternating, and pretrained-base references, alongside the frozen reference.The asymmetric alternating and the two base references were evaluated only at 70M.Source: §F.1 · p. 59, Figure 44 · p. 60
  • The frozen reference sets the frontier at both model scales, while other schedules either lose utility or leave more memorization.Mutual training reaches worst-case nAUC 0.46 at 70M versus 0.71 for the undefended model, but does not improve on the undefended model at 1B.Source: §F.1 · p. 59, Figure 44 · p. 60

G PROPERTIES OF TRA AND TRAP

  • The appendix states that expected training-example TRA equals the expected generalization gap, the penalty gates each token’s gradient, and tokenwise control bounds continuation probabilities relative to the reference.Source: G PROPERTIES OF TRA AND TRAP · p. 60

G.1 EXPECTED TRA EQUALS THE EXPECTED GENERALIZATION GAP

Under matched training procedures and a balanced random split, expected full-sequence TRA on target training examples equals the expected generalization gap. TRAP attenuates only above-threshold token contributions and yields a reference-relative bound on continuations.

G.1 EXPECTED TRA EQUALS THE EXPECTED GENERALIZATION GAP

  • Expected full-sequence TRA on target training examples equals test loss minus training loss when both split learners use the same randomized procedure.The target and reference are balanced halves of IID samples; during TRAP training, the reference is not penalized, so TRA is a training signal rather than an estimate of the gap.Source: §G.1 · pp. 60–61

G.2 THE PENALTY GATES EACH TOKEN’S GRADIENT

  • Above the threshold, TRAP attenuates a token’s gradient, while below it tokens retain their ordinary gradient.The raw contribution is removed at w = 1 and reversed when w > 1; minibatch aggregation, clipping, momentum, and Adam alter the final parameter update.Source: §G.2 · p. 61

G.3 A REFERENCE-RELATIVE BOUND ON CONTINUATIONS

  • If every token in a length-L span has TRA at most 𝜖, the target assigns that continuation at most e^(L𝜖) times the reference probability.The bound is reference-relative, not a differential-privacy guarantee; excess probability already assigned by the reference remains outside TRAP’s control.Source: §G.3 · p. 61

Related papers

Loading 2609.32293v1…