Provably Confidential Language Modelling

• An imperfect screening policy can leave sensitive text exposed to efficient black-box extraction, even when training satisfies selective differential privacy (Shi et al., 2021).Source: §1 · p. 2
• The paper introduces a confidentiality definition that quantifies the risk of leaking sensitive text, addressing a problem distinct from differential privacy.Source: §1 · pp. 1–2
• CRT combines deduplication and redaction to protect confidential text despite imperfect labeling, addressing the difficulty of manually screening huge datasets with imperfect automated tools.Source: §1 · p. 2
• CRT combined with differentially private stochastic gradient descent provides strong confidentiality guarantees by theoretical proof.Source: §1 · p. 2
• CRT-trained models achieve the same or better perplexity than existing solutions on MultiWOZ 2.2 and CustomerSim against the attacks of Carlini et al. (2019, 2021).The introduction also reports only mild utility loss for provably confidential transformer models.Source: §1 · p. 2

• Language models estimate sequence probabilities and are trained by minimizing negative log-likelihood, with transformers replacing RNNs in recent state-of-the-art models.Source: §2 · p. 2
• Differential privacy can substantially reduce language-model utility, motivating methods that protect sensitive text without requiring models to avoid all memorization.Source: §2 · p. 2
• Deduplication reduces how often language models emit memorized text but cannot prevent all unintended memorization, motivating its combination with redaction.Source: §2 · pp. 2–3
• Figure 1 illustrates confidential spans, redaction under a labeling policy, and false positives or false negatives when that policy is imperfect.Source: Figure 1 · p. 3
• Selective DP-SGD depends on a policy identifying sensitive tokens; with an approximate policy, it may provide no meaningful confidentiality guarantee and remain vulnerable to extraction.Source: §2 · p. 3

3.1 Formally defining confidentiality

• A secret is a contiguous token subsequence treated as sensitive, and confidentiality compares the trained model on data containing that secret with data where it is masked.Source: §3.1 · p. 3
• Masking all sensitive text in preprocessing ensures (0, 0)-confidentiality, although the paper distinguishes this content-focused protection from non-trivial differential privacy guarantees.Source: §3.1 · p. 4
• Group confidentiality protects a list of sensitive texts together across up to as many data points as secrets; uniform-confidentiality applies this to all secrets in the dataset.Source: §3.1 · p. 4
• Bayesian confidentiality evaluates protection when a secret is drawn from a distribution, allowing guarantees against adversaries with less than complete information about it.Source: §3.1 · p. 4

3.2 Confidentially redacted training

• CRT screens the corpus into public sentences without confidential information and private sentences containing it, using ordinary optimization on public data and DP-SGD on private data.Source: §3.2 · pp. 4–5
• Deduplication masks repeated sentences after their first occurrence, while redaction replaces spans labeled confidential by a sequence-labeling policy with a special mask token.Source: §3.2 · p. 4

8 end

• The training pipeline combines redaction and deduplication with a normal optimizer and DP-SGD, and protects every data point containing a <MASK>.Automated implementations can use a Bloom filter for deduplication and NER or PII detectors for redaction; the redaction policy may produce false positives and false negatives.Source: §8 · p. 5

3.3 Theoretical analysis

• After redaction, a token’s confidentiality parameter is ϵ(x) when π does not flag it and zero when π flags it.Redaction also yields a group confidentiality bound over the tokens not flagged by π.Source: §3.3 · p. 5
• With false-negative rate γ, redaction scales the Bayesian bound to (log(1 + γ(e^ϵ −1)), γδ) when the underlying algorithm is confidential under the conditioned secret distribution.The proof intuition is that redaction produces identical datasets with probability 1−γ; false positives do not change that comparison.Source: §3.3 · pp. 5–6
• Deduplication strengthens group confidentiality by replacing the group’s full set with its unique secrets, especially when a secret appears repeatedly.The resulting parameters are ϵ(Unique(S)) and δ(Unique(S)).Source: §3.3 · p. 6
• Under the stated screening assumptions, CRT provides token-level, group, and Bayesian confidentiality guarantees, with high recall strengthening the guarantees.The theorem gives a Bayesian bound depending on the recall of both screening policies.Source: §3.3 · p. 6

4 Experiments

• CRT is evaluated with LSTM and GPT-2 on MultiWOZ 2.2 and the simulated CustomerSim dialogue dataset.MultiWOZ contains real-world annotated dialogues; CustomerSim contains synthetic user information and uses regex detection for secrets.Source: §4 · p. 6
• The evaluation compares CRT with vanilla training and DP-SGD for GPT-2, and additionally with Selective DP-SGD for LSTM.The LSTM implementation follows Shi et al. (2021); the GPT-2 setup uses ghost clipping to alleviate DP-SGD’s computational challenge.Source: §4 · pp. 6–7
• With a perfect screen, redaction achieves ϵ = 0 and the perplexity of vanilla training without added noise.Figure 2 compares test-set perplexity with Bayesian confidentiality on MultiWOZ 2.2 and CustomerSim; other models use γ = 0.1 under an approximate policy.Source: Figure 2 · p. 7
• Models were trained five times, with parameters tuned using validation-set performance.For Bayesian confidentiality, the experiments use a uniform distribution over secret sequences.Source: §4 · pp. 6–7

5.1 Evaluation procedure

• Evaluation measures language-model utility with predictive perplexity and confidentiality with the theoretical (ϵ, δ) budget and attack tests.Canary insertion uses 10 randomly generated canaries, each inserted 20 times; membership inference ranks samples by perplexity, and the group attack classifies groups.Source: §5.1 · p. 7

5.2 Overall performance

• CRT-GPT’s performance is close to Non-private-GPT in PPL while preserving strong confidentiality, and its Bayesian ε is much lower than DPSGD-GPT’s.DPSGD-GPT and CRT-GPT generally perform better on the test sets than S-DP-LSTM, CRT-LSTM, and DPSGD-LSTM.Source: §5.2 · pp. 7–8
• CustomerSim models achieve overall better performance than MultiWOZ models, which the authors attribute to CustomerSim’s template-based simulated dialogs.The comparison covers model performance across the two datasets.Source: §5.2 · p. 8

5.3 Attack results

• CRT-LSTM and CRT-GPT have canary exposures close to 0, while exposure for several baselines rises sharply as the screening false-negative rate γ increases.Non-private models with masking protect well only at γ = 0; S-DP-LSTM has high exposure above 0 because false-negative samples are not protected.Source: Figure 3 · p. 8, §5.3 · p. 8
• CRT-LSTM and CRT-GPT membership-inference accuracy aligns with the 0.5 random-guess baseline, while Non-private and S-DP models become more vulnerable as γ increases.Non-private and S-DP models show worse protection under group inference than individual inference at certain γ values.Source: Figure 4 · p. 8, §5.3 · p. 8, Figure 5 · p. 8

5.4 CRT amplifies Bayesian Confidentiality guarantees

• With (ϵ′ = 1.0, γ = 0.1), CRT reduces Bayesian confidentiality ϵ to 0.12.The analysis fixes DP-SGD’s ϵ′ and varies screening policy; high recall (small γ) yields greater improvement.Source: Figure 6 · p. 8, §5.4 · p. 9

6 Conclusion

• CRT is proposed to protect secret texts in language models, with a confidentiality definition and theoretical and empirical evaluation across multiple datasets and models.The definition quantifies the risk of leaking sensitive content.Source: §6 · p. 9

7 Broader Impact

• CRT aims to support language-model fine-tuning on individual application data without leaking or memorizing confidential information.The paper contrasts this with using a general pretrained model without fine-tuning or manually filtering sensitive information before fine-tuning.Source: §7 · p. 9
• The experiments use simulated confidential content to mimic a real corpus and do not use real users’ confidential information.Source: §7 · p. 9

A.1 Illustration of our proposed algorithm

• Figure 7 illustrates preprocessing on two data points; deduplication masks are shown in red, and both operations replace token sequences with <MASK>.Source: Figure 7 · p. 11

A.2 Proofs of technical results

• Proposition 5 derives group confidentiality by iterating the single-secret guarantee after redaction, while Proposition 7 reduces neighboring deduplicated datasets to their unique-secret difference.Source: §A.2 · pp. 11–12
• Proposition 6 proves Bayesian confidentiality by bounding hockey-stick divergence for mixtures of distributions, using joint convexity and the stated confidentiality assumption.Source: §A.2 · pp. 11–12
• When conservative screening catches every secret missed by π, Theorem 8 bounds each secret’s ε(x) by ε and composes confidentiality parameters across its guarantee statements.Source: §A.2 · p. 12

A.3 More details on experiments

• CRT-LSTM uses lr=0.05, bs=10, and 100 epochs; CRT-GPT uses lr=5e-4, bs=256, and 10 epochs, with one epoch taking 2 hours and 30 minutes, respectively.The LSTM is one layer with 200-dimensional embeddings and hidden state; distill-gpt27 has 6 layers, dimension 768, and 12 heads.Source: §A.3 · p. 12

A.4 Redaction policy details

• The screen policy uses threshold 0.3, while conservative πc selects tagged predictions and plain text scoring below 0.9; manual labeling found 90% and 100% recall, respectively.The conservative policy has lots of false positives, while π has few.Source: §A.4 · p. 12

A.5 Membership inference attack details

• Membership attacks use 2,000 balanced secret-versus-random examples, while group attacks classify 20-sentence groups using summed perplexity.The individual-example construction gives random guessing 50% accuracy; group sets contain 1,000 sensitive and 1,000 insensitive groups.Source: §A.5 · p. 13

A.6 “The devil is in the details” – how things could go wrong with seemingly inocuous changes to the algorithm.

• Define confidentiality against a perfectly redacted dataset: unlike selective differential privacy, this yields amplification from redaction when a secret is detected by π.The passage says selective-DP framing does not provide this amplification effect.Source: §A.6 · p. 13
• Tokenize and split into whole natural sentences before redaction or deduplication; otherwise preprocessing can misalign neighbors or split sensitive text across data points.Source: §A.6 · p. 13