PaperlayerView exact revision on arXiv

Source-linked AI summary

The Privacy-Hallucination Tradeoff in Differentially Private Language Models

Krithika Ramesh, Krishna Pillutla, Danish Pruthi, Anjalie Field

arXiv:2609.00492v1cs.AIcs.CL

TL;DR

The paper addresses whether differentially private training compromises factual accuracy, an important concern when models handle sensitive information in high-stakes domains. It empirically studies DP pre-training and fine-tuning, examines predictive-distribution mechanisms and fact frequency, and finds increased hallucinations alongside impaired factual-association acquisition. The results motivate privacy-preserving methods that better balance privacy guarantees with factual accuracy.

  • Problem

    Prior work studied privacy and factual accuracy independently, leaving their interaction insufficiently investigated despite the need for both in high-stakes applications.

  • Method

    The paper compares DP and non-DP pre-training and fine-tuning across privacy budgets, analyzes predictive distributions, and controls fact frequency to study factual acquisition.

  • Results

    DP training increases hallucinations, flattens output distributions toward incorrect alternatives, and hinders acquisition of factual associations, especially under stricter privacy budgets.

  • Takeaways & Limitations

    Existing DP training methods can compromise factual reliability, motivating refined privacy-preserving approaches that balance privacy guarantees with factual accuracy in sensitive applications.

  • Takeaways & Limitations

    The experiments are limited to open-source models with known pre-training data cutoffs and do not examine other DP approaches such as DP retrieval-augmented generation.

Abstract

from arXiv · show

Both privacy and factual accuracy are paramount in high-stakes domains like healthcare. Concerningly, we uncover and investigate a privacy-hallucination tradeoff in differentially private (DP) language models. First, we empirically show that models pre-trained or fine-tuned with DP tend to produce more hallucinations than non-DP counterparts, with increased severity as the privacy budget grows stricter. Second, we investigate model properties driving this tradeoff, demonstrating that DP mechanisms flatten output distributions, potentially redistributing probability mass toward factually incorrect alternatives. Third, through experiments where we control fact frequency in training data, we characterize how information frequency can reduce hallucination risks in DP models. Overall, our findings underscore the need for more nuanced privacy-preserving interventions that offer rigorous privacy guarantees without compromising factual accuracy.

1 Introduction

The paper investigates whether differentially private training increases factual hallucinations, motivated by the need to preserve both privacy and accuracy in high-stakes settings. It finds increased hallucinations, flatter output distributions, and frequency-dependent factual acquisition in DP models.

  • Differential privacy is motivated by the need to protect sensitive training information in high-stakes domains while preserving factual accuracy.
  • The paper asks whether DP training increases hallucinations and examines the model properties and conditions associated with that effect.
  • DP training increases hallucinations in both fine-tuning and pre-training, with a more pronounced degradation during pre-training.
  • DP noise flattens predictive distributions by dispersing next-token probability mass across more candidate tokens, increasing the risk of factually incorrect content.
  • A fact must recur several times before a DP-trained model acquires it, and stricter privacy budgets can prevent acquisition even at substantially higher tested frequencies.
  • The findings call for caution when applying DP in high-stakes settings and further investigation of when it can be used without increasing hallucinations.

2 Experimental Design

The study evaluates DP effects using controlled Wikipedia fine-tuning and pre-training comparisons, open-ended article generation, automated FactScore, and human annotation. Its design separates training overlap from evaluation data and measures factuality rather than relying on utility metrics alone.

  • The experiments vary privacy budgets while controlling fine-tuning and pre-training overlap and evaluating factual accuracy in open-ended model outputs.
  • DP fine-tuning is studied for controllable text generation using LLMs, with LoRA applied to both DP and non-DP models.
  • GPT-J 6B is used because The Pile is open with a known cutoff, enabling fine-tuning data selection without pre-training overlap.
  • Private pre-training is evaluated with the 1B-parameter DP-pre-trained VaultGemma model at ε = 2 against non-private and additional baselines.
  • Wikipedia datasets contain post-2020 science and AI articles selected to separate pre-training knowledge from fine-tuning-acquired facts.
  • Models receive article titles and generate full Wikipedia articles, whose factual accuracy is measured so that higher accuracy indicates a lower hallucination rate.
  • FactScore extracts atomic claims and verifies them against the original Wikipedia article using an external knowledge source and an LLM verifier.
  • Human evaluation by five computer science graduate students assesses claim veracity and source-text support on stratified Wikipedia AI generations.

3 RQ1: What impact does DP training have on hallucinations in model outputs?

DP training is associated with more hallucinations in both fine-tuning and pre-training, with stronger degradation under stricter privacy and especially pronounced effects during pre-training. Fine-tuning can leave previously acquired knowledge intact, but DP models produce less factually consistent outputs on newly evaluated information.

  • 3.1 DP fine-tuning increases hallucinations: DP fine-tuning consistently lowers FactScore relative to non-DP training, following ε = ∞ ≻ ε = 16 ≻ ε = 8 across both datasets.Differences are significant at both response and topic levels except for ε = 16 on Wikipedia AI.
  • 3.1 DP fine-tuning increases hallucinations: 37.7% versus 32.9% correct claims shows successful non-DP fine-tuning on Wikipedia AI, while ε = 8 reaches only 31.6% mean FactScore.At ε = 16, the mean FactScore rises to 33.8% but remains below the non-DP model’s 37.7%.
  • 3.1 DP fine-tuning increases hallucinations: DP fine-tuning does not significantly disrupt knowledge acquired during non-private pre-training.The result suggests that the principal degradation concerns information learned under the DP fine-tuning setting rather than previously encoded knowledge.
  • 3.1 DP fine-tuning increases hallucinations: Human annotators found more unsupported facts and lower veracity for DP outputs among higher-quality generations.For generations with FactScore ≤0.5, both model types received equally low veracity ratings, while the DP model still exhibited more unsupported facts.
  • 3.2 DP pre-training and hallucinations: VaultGemma, pre-trained with DP at ε = 2, consistently produces more inaccurate facts than non-private Gemma baselines.On Wikipedia pre-training data, Gemma3 averages 26.6 FactScore versus 22.0 for VaultGemma, with statistically significant differences across Gemma comparisons.
  • 3.2 DP pre-training and hallucinations: VaultGemma can be as hallucinated as a model never directly exposed to the targeted information, despite having higher average FactScores than GPT-2 XL on some settings.On Wikipedia Science, VaultGemma does not outperform GPT-2 XL, even though the articles were reportedly created after GPT-2 XL’s cutoff.

4 RQ2: What impact does DP training have on model properties related to hallucination?

DP training alters model output distributions in ways associated with hallucination: probability mass becomes more dispersed across candidate tokens, and DP-induced miscalibration can shift mass toward incorrect alternatives. These effects become more pronounced under stricter privacy budgets.

  • In controlled bigram experiments, hallucination rate is measured as the proportion of generated tuples not belonging to the designated true-fact set.The setup uses synthetic Person-Food-Place facts and reports hallucination rates over 3000 generated tuples.
  • DP models have larger mean nucleus sizes, indicating probability mass is dispersed across more next-token candidates.Nucleus size is measured using top-p = 0.9.
  • DP models have higher effective vocabulary sizes, reflecting greater uncertainty in next-token distributions.Effective vocabulary size is defined as exp(H), the exponentiated entropy of the next-token distribution.
  • DP-induced miscalibration can increase hallucinations by shifting probability mass toward factually incorrect alternatives.The paper relates this mechanism to theory connecting hallucination rates with calibration and training-data fact frequency.
  • DP mechanisms differ from temperature scaling because they need not flatten distributions evenly or preserve token rankings.On unseen data, token rankings shift alongside increased hallucination rates, whereas rankings are largely preserved on pre-training data.

5 RQ3: Under what conditions could DP be usable without increasing hallucinations?

The paper tests whether factual frequency can enable correct association learning under DP. It finds that stricter privacy budgets require substantially more repetitions, and within the tested range can prevent acquisition entirely.

  • Stricter privacy budgets require increasingly frequent repetitions for factual associations to be learned.The controlled experiment varies fact frequency across f ∈ {1, 3, 7, 15, 30, 60} repetitions per epoch.
  • At ε = 1311, the pre-trained model requires 30 repetitions per epoch, while at ε = 30 it requires 60.Without DP, the pre-trained model recalls facts appearing once per epoch.
  • For ε ≤ 13, no fact is learned even at 60 repetitions per epoch in the pre-trained setup.This indicates that, within the tested frequency range, stricter DP can prevent learning rather than merely slow it.
  • At ε = 1311, the randomly initialized model requires 30 repetitions per epoch, compared with 7 for the pre-trained model.Beyond this setting, no fact is learned by the randomly initialized model at any strict ε within the tested frequency range.
  • The authors conclude that factual associations may not be acquired during DP fine-tuning over sensitive data at meaningful privacy guarantees.They characterize the frequency thresholds as considerably worse for DP pre-training because of its compute and memory costs.

6 Discussion and Conclusions

The paper identifies a privacy–hallucination tradeoff in differentially private language models. It concludes that DP can hinder factual acquisition and shift outputs toward incorrect content, motivating approaches that balance privacy with factual reliability.

  • DP significantly hinders acquisition of new factual associations during pre-training and fine-tuning, leading to hallucinations.
  • The proposed mechanism is that DP shifts model outputs toward factually incorrect content.
  • The findings show that moderate noise can preclude acquisition of a factual association.
  • The paper calls for refined privacy-preserving approaches that balance privacy guarantees with factual accuracy in sensitive applications.

Limitations

The paper’s scope and evaluation choices constrain how broadly its privacy–hallucination findings should be applied. It also leaves mitigation strategies and other DP approaches for future work.

  • The experiments are limited to open-source models with known pre-training data cutoffs.This choice ensures sufficient unseen evaluation and training data for the experimental setup to satisfy DP constraints.
  • The study does not investigate post-training methods intended to reduce hallucinations through uncertainty expression or refusal.The authors note that such approaches might reduce the tradeoff but do not expect them to eliminate hallucination risk.
  • The tradeoff is not examined in other DP approaches, including DP retrieval-augmented generation.The authors identify maintaining privacy guarantees under multiple queries as a challenge requiring further research.
  • The evaluation targets acquisition and reproduction of facts from previously unseen data, distinguishing failure to learn from hallucination.A model that fails to acquire a fact could instead produce no content or generic content.

C.2 Experimental Setup: Methods and Hyperparameters

The experiments use DP-SGD with LoRA-based fine-tuning, calibrated privacy accounting, and distinct privacy units for pre-training and fine-tuning. The setup also compares private and non-private models while acknowledging that data-matched DP pre-training is computationally infeasible at the required scale.

  • DP fine-tuning: DP-SGD clips per-sample gradients and adds calibrated Gaussian noise to provide sequence-level (ϵ, δ)-differential privacy.The noise scale is selected to meet the desired privacy guarantee.
  • Privacy accounting: The study uses tighter PLD privacy-loss estimates to determine the noise multiplier required by each fine-tuning privacy budget.This improves the accuracy of privacy accounting relative to alternate techniques.
  • Privacy units: Fine-tuning treats each document as the privacy unit, whereas VaultGemma pre-training uses fixed 1024-token sequences that can span document operations.The document-level unit reflects settings where each document corresponds to a distinct individual.
  • Baselines and scope: VaultGemma serves as a comparable-scale reference rather than a controlled data-matched ablation because DP pre-training from scratch is infeasible at the required scale.The authors leave controlled retraining experiments for future work.
  • Parameter-efficient adaptation: LoRA adapts frozen pre-trained weights through low-rank matrices, avoiding the memory overhead that makes full DP fine-tuning computationally infeasible at this scale.The study applies LoRA in all experiments and cites prior evidence that it can be competitive with full fine-tuning.

C.3 Dataset : Examples of Input

The dataset examples span AI, science, and other Wikipedia-derived topics, with figures documenting training loss under two fine-tuning data settings. The analysis also clusters recurring claims across generated documents to identify potential recurrent hallucinations.

  • Dataset examples: The examples include Wikipedia AI and Science articles, alongside topics such as cancer biology, medicine, and physics.The supplied examples list article titles from multiple domains.
  • Training examples: Figures 8 and 9 report training loss for models fine-tuned on large Wikipedia data and on private Wikipedia articles only.The two settings distinguish broader Wikipedia fine-tuning from fine-tuning restricted to private articles.
  • Claim analysis: The clustering procedure groups semantically similar and lexically consistent claims, then retains clusters recurring across at least two documents for the same topic.These recurring clusters are used to identify potential recurrent hallucinations rather than one-off statements.

D.1.1 Recurring Hallucination Analysis

DP models produce more recurring unsupported claims than non-DP models, indicating that increased hallucinations reflect structured shifts rather than random errors. These findings are not explained by generation length or lower perplexity, and DP fine-tuning does not consistently disrupt pre-training facts.

  • DP models produce fewer recurring supported claims and more recurring unsupported claims than non-DP models across both datasets.The stricter privacy settings show lower ratios of recurring supported to unsupported claims.
  • The recurring-claim analysis indexes claims by source document, clusters them using sentence embeddings and Jaccard distance, and retains clusters supported by at least two documents.
  • DP models can repeat the same hallucination: for “AlphaEvolve,” both DP models incorrectly describe it as a video game with fabricated development and gameplay details.
  • DP-generated text is slightly shorter, but stable claim density and length-normalized FactScore indicate that FactScore differences are not attributable to generation length.
  • Lower perplexity does not predict factual accuracy: DP fine-tuned models often have lower perplexity, while non-DP models achieve higher FactScores across topics.
  • DP fine-tuning causes only marginal FactScore differences on Wikipedia pre-training data, suggesting limited disruption to facts already learned during pre-training.

D.6 Distributional flattening for pre-training data

DP models flatten next-token distributions on pre-training data, increasing uncertainty and spreading probability mass across more tokens. However, token rankings remain relatively preserved on seen data, so the flattening does not affect hallucinations in the same way as on unseen data.

  • DP models have larger mean nucleus sizes at top-p = 0.9, indicating probability mass dispersed across more candidate tokens and greater next-token uncertainty.
  • DP models show higher effective vocabulary sizes, meaning probability mass is spread across more tokens.
  • On seen data, nucleus-token overlap remains higher than on unseen data, with ε = 16 reaching 0.707±0.071 versus 0.623 ± 0.051, and ε = 8 reaching 0.704±0.069 versus 0.619 ± 0.051.
  • For pre-training data, increased entropy reflects a more diffused but structurally intact next-token distribution, whereas on unseen data redistributed mass falls on incorrect alternatives.

D.7 Is the model reverting to its pre-training priors?

The evidence argues against DP fine-tuned models simply reverting to their pre-training priors. DP models differ more from the base model on unseen data, while their factual performance remains similar on data likely seen during pre-training.

  • DP models have higher KL divergence from the base model on unseen fine-tuning data, contradicting the expectation that they revert to pre-training priors.
  • Loss interpolation shows DP models remain closer to the base model than the non-private model but occupy regions associated with lower loss.
  • On Wikipedia pre-training data, DP and non-private GPT-J models achieve similar FactScores, suggesting DP fine-tuning does not disrupt pre-training knowledge.
  • The KL-divergence analysis compares token-position prediction shifts on Wikipedia Science and AI data against Wikipedia pre-training data.
  • Human factuality annotation labels claims as correct, incorrect, or unclear, and separately assesses whether each claim is supported by the source text.

D.9 Expanded FactScore Evaluations

Expanded evaluations examine robustness across claim-evaluation models, temperatures, datasets, and pre-training conditions. The supplied passages describe the evaluation coverage and stability analyses but do not report their numerical outcomes.

  • Additional FactScore evaluations vary claim decomposition and evaluation models for Wikipedia Science articles.
  • FactScore distributions are examined across temperature settings for GPT-J 6B and pre-trained models on Wikipedia Science, AI, and pre-training data.
  • Additional evaluations measure factuality on unseen articles after fine-tuning exclusively on Wikipedia training data or exclusively on unseen Wikipedia articles.
  • Per-topic FactScore standard deviations for Wiki AI and Wiki Science assess factuality stability across prompt variations.
  • Pre-trained-model results report average, median, quartile, per-topic maximum and minimum FactScores, plus counts of factual and non-factual responses.

E.1 ANADP: Adaptive Noise Allocation for Differential Privacy

ANADP adaptively allocates less noise to parameters deemed more important, aiming to preserve factual associations while maintaining the total noise budget. However, its privacy analysis is incomplete, and experiments find little mitigation of the privacy-hallucination tradeoff.

  • Experimental results: ANADP does not meaningfully mitigate the privacy-hallucination tradeoff compared with standard DP-SGD.On Wikipedia AI, it performs at par or worse at both privacy budgets; on Wikipedia Science, it marginally outperforms standard DP-SGD at ε = 8.
  • ANADP mechanism: ANADP assigns smaller noise to parameters with higher importance scores, computed from estimated parameter sensitivity and uncertainty.The algorithm’s parameter importance combines a loss-based sensitivity estimate with uncertainty measuring how that sensitivity changes from its moving average.
  • Privacy analysis: ANADP’s privacy guarantee is unverified because its analysis mishandles reduced coordinate noise and data-dependent covariance.The privacy analysis must account for sensitivity in the Mahalanobis norm, while ANADP computes importance from the same private gradients later receiving noise.
  • Experimental results: ANADP reduces recurring incorrect claims but does not reduce hallucinations overall.The paper reports that anisotropic Gaussian noise in this form reduces recurring incorrect claims while failing to reduce hallucinations.
  • Controlled factual recall: Higher DP noise requires greater fact frequency for successful recall, while pre-trained models learn facts at lower frequencies than randomly initialized models.This controlled benchmark varies noise multipliers σ and fact frequencies f to measure factual recall and its frequency threshold.
Loading 2609.00492v1…