PaperlayerView exact revision on arXiv

Source-linked AI summary

Adaptive Diffusion Freezing: Privacy-preserving Diffusion Models Against Membership Inference Attacks

Jialu Guo, Xiao Han, Junjie Wu

arXiv:2609.10608v1cs.CRcs.LG

TL;DR

Diffusion models raise privacy concerns because membership inference attacks can exploit training-set memorization, while existing defenses struggle to balance privacy, utility, and efficiency. ADF addresses this gap with risk-aware, cross-timestep adaptive freezing that controls subset participation across timesteps. The paper reports effective protection and a favorable privacy–utility–efficiency trade-off, while noting that masks must be prepared before training exposure occurs.

  • Problem

    Existing diffusion-model privacy defenses do not adequately balance membership-inference protection, generation utility, and computational efficiency.

  • Method

    ADF uses a timestep-wise mask matrix and risk-aware freezing policy to restrict higher-risk subset–timestep contributions during diffusion training.

  • Results

    ADF effectively mitigates membership leakage while preserving generative performance and computational efficiency, achieving a favorable privacy–utility–efficiency trade-off across datasets and attack settings.

  • Takeaways & Limitations

    ADF provides a diffusion-specific defense that regulates over-memorization through fine-grained, timestep-aware exposure control.

  • Takeaways & Limitations

    Jointly optimizing the mask with the diffusion model is unstable and cannot undo exposure that occurred before mask entries were updated.

Abstract

from arXiv · show

Diffusion models have achieved remarkable success in generative tasks across various areas, however their training process raises significant privacy concerns, particularly under membership inference attacks (MIAs). Prior studies on privacy-preserving of diffusion models fail to balance privacy, utility, and efficiency. To address this gap, we propose a novel framework of privacy-preserving diffusion models, Adaptive Diffusion Freezing (ADF), which can defend against MIAs with better trade-off. By leveraging cross-timestep adaptive freezing training, ADF explicitly control the participation of different data subsets across diffusion timesteps via a mask matrix, which reduces the over-memorization and leads to more uniform model behaviors between member and nonmember samples. To construct a freezing mask matrix that effectively reduce membership leakage without unnecessarily harming generation quality, we introduce a pretraining-based risk-aware freezing policy to estimate MIA risk based on memorization tendency, and suppress the contribution of the subset-timestep pairs with higher risk. Evaluations on multiple datasets demonstrate that ADF provides effective defense performance as well as state-of-the-art privacy-utility-efficiency trade-off performance compared to various baselines.

1 Introduction

Diffusion models face timestep-specific membership leakage that existing defenses do not adequately address. ADF controls subset participation across timesteps with risk-aware freezing to improve the privacy–utility–efficiency trade-off.

  • Existing defenses often overlook timestep-specific leakage, while removing high-risk timesteps can disrupt diffusion’s temporal dependencies and generation quality.
  • The framework targets efficient subset–timestep control compatible with standard DDPM training rather than costly per-sample estimation or complex reweighting.
  • ADF uses a timestep-wise mask matrix to freeze selected training subsets at selected diffusion timesteps, reducing over-memorization.
  • A pretraining-based risk-aware policy identifies higher-risk subset–timestep pairs from memorization tendency and suppresses their contribution.
  • ADF achieves effective membership-leakage mitigation while preserving generative performance and computational efficiency.
  • Experiments train diffusion models on four datasets and compare ADF with five baselines under varied membership inference attacks.

2 Related Work

Prior diffusion-model MIA defenses include augmentation, regularization, differential privacy, distillation, and adversarial training, but they retain important privacy, utility, scalability, or efficiency limitations. The paper motivates a diffusion-specific defense that addresses timestep-wise leakage.

  • Diffusion-model MIAs exploit signals from generated outputs, intermediate denoising states, losses, or gradients under black-box, gray-box, and white-box access.
  • Regularization and augmentation only partially limit membership leakage and may hinder convergence or degrade generative performance.
  • Differential privacy can defend against MIAs but substantially degrades generation quality and increases computational cost in diffusion models.
  • Knowledge distillation can improve the privacy–utility balance but is computationally expensive and less scalable.
  • Adversarial-training defenses can introduce considerable complexity and may protect only fine-tuning data.
  • Existing defenses still offer limited effectiveness, privacy–utility trade-offs, and high computational overhead, motivating a diffusion-tailored mechanism.

3 Preliminary and Problem Formulation

The paper introduces diffusion processes, membership inference attacks, and a formal defense objective. It seeks a timestep-wise mask that lowers membership distinguishability while preserving denoising-based generation utility.

  • 3.1 Diffusion Model: Diffusion models transform data through forward noising and reverse denoising, with DDPM using a Markov chain over T timesteps.
  • 3.1 Diffusion Model: The forward process progressively corrupts x_0 with Gaussian noise, while reparameterization allows intermediate x_t samples to be obtained directly.
  • 3.1 Diffusion Model: The reverse network learns to estimate added noise at each timestep by minimizing the discrepancy between predicted and ground-truth noise.
  • 3.2 Membership Inference Attack: An MIA determines whether a query sample belongs to the training set using attack features derived from model outputs, likelihood, denoising error, loss, or gradients.
  • 3.3 Problem Formulation: The defense addresses over-memorization because it creates distinguishable member and nonmember behavior during denoising.
  • 3.3 Problem Formulation: ADF regulates training-data exposure with a timestep-wise mask matrix M, whose entries determine whether each subset participates at each timestep.
  • 3.3 Problem Formulation: The formal objective minimizes membership-inference risk while keeping utility bounded through the defended denoising objective and mask design.

4 Methodology

ADF regulates diffusion-model memorization by controlling which training subsets contribute at each timestep, using adaptive freezing and a pretraining-based risk-aware policy. Its mask-based design suppresses high-risk subset-timestep signals while preserving broader denoising supervision.

  • Cross-timestep Adaptive Freezing Training: Cross-timestep adaptive freezing partitions training data into controllable subsets and restricts their exposure across the denoising trajectory.The subsets are training-exposure units rather than separate sub-models, enabling fine-grained regulation of training information.
  • Cross-timestep Adaptive Freezing Training: ADF uses a binary mask matrix to control each subset's participation in optimization at each diffusion timestep.A mask value of 1 keeps a subset active, whereas 0 freezes its loss contribution and gradient propagation at that timestep.
  • Cross-timestep Adaptive Freezing Training: ADF removes optimization signals for selected subset-timestep pairs without changing the network architecture, reducing consistent overfitting to specific data groups.Different subsets can remain active at other timesteps, so freezing is localized rather than global.
  • Multi-network Parameterization Scheme: The framework extends to separate denoising networks for timestep groups because noise levels, denoising subtasks, and privacy sensitivity can vary across the diffusion trajectory.This multi-network parameterization is intended to increase ADF's flexibility across timestep groups.
  • Risk-aware Freezing Policy: ADF constructs its mask before main training by using early standard-training fitting dynamics to estimate subset-timestep memorization risk.The risk-aware policy assigns higher freezing priority to higher-risk pairs and maps normalized risk to freezing probabilities rather than using a hard threshold.
  • Risk-aware Freezing Policy: The mask retains (1 − s)K active subsets at each timestep, where s controls the fraction of frozen subsets.This sparsity constraint limits freezing while maintaining timestep-wise denoising supervision.

5 Experiments

ADF is evaluated across four image datasets using multiple baselines, threat models, privacy metrics, and controlled diffusion-model settings.

  • Datasets: Four datasets cover natural, facial, and remote-sensing imagery, providing varied evaluation domains for the defense.The datasets are CIFAR-10, STL10_U, CelebA, and NWPU-RESISC45.
  • Baselines: The comparison includes an undefended DDPM and four representative defense strategies.The experiments use standard diffusion without defense as W/O and compare it with Cutout and three other defenses described in the paper.
  • Threat models: Evaluation considers gray-box SecMI and white-box GSA membership-inference attacks with different adversary capabilities.SecMI accesses denoising-related intermediate information, while the evaluation also includes the stronger white-box setting.
  • Metrics: Privacy is measured with ASR, AUC, TPR@1%FPR, and TPR@0.1%FPR, where lower values indicate stronger protection.FID is additionally adapted to measure generation fidelity, alongside efficiency-related evaluation described in the experiments.
  • Implementation: All variants share a DDPM U-Net backbone, 1000 diffusion steps, and the same architecture, optimization, noise schedule, and sampling protocol.Unless otherwise specified, ADF uses K=10 subsets, G=1 timestep group, sparsity s=0.3, and five pretraining epochs for risk estimation.

5.2 Experimental Results

ADF consistently improves the privacy–utility trade-off across diffusion architectures and attack settings by targeting risky subset–timestep contributions. Its benefits depend on calibrated sparsity, timestep-aware modeling, and risk-aware freezing, which reduce membership leakage while preserving generation quality.

  • Privacy-utility trade-off performance: ADF consistently achieves a superior privacy–utility trade-off across datasets and attack settings, lowering membership attacks while preserving competitive generation quality.On LDM, ADF significantly lowers attack metrics while maintaining competitive generation quality and outperforming other defenses in trade-off performance.
  • Generalizability across architectures: ADF slightly improves FID from 31.223 to 26.479 on Stable Diffusion 3 while consistently reducing all three membership-inference attacks.This supports applicability to MMDiT-style architectures without sacrificing generation quality.
  • Sensitivity analysis: Increasing mask sparsity lowers ASR or AUC, but extreme sparsity, especially s = 0.9, substantially increases FID and should prioritize privacy over generation quality.FID remains relatively stable as sparsity increases from 0.1 to 0.5, indicating a practical operating range.
  • Sensitivity analysis: Finer timestep grouping improves privacy with mild utility cost: FID increases only from 17.396 to 17.989 as G grows from 1 to 5.Grouping enables different freezing decisions across denoising stages and captures heterogeneous timestep-wise leakage.
  • Sensitivity analysis: Risk-aware freezing outperforms exposure-matched random timestep freezing, reducing ASR/AUC while maintaining nearly identical generation quality.Removing pretraining-based risk estimation produces consistently higher attack performance and worse FID.
  • Robustness analysis: Across partition seeds, ADF remains stable: average FID is 17.169 ± 0.165 versus 18.227 for DDPM, and worst privacy results remain below the undefended baseline.The reported ASR/AUC standard deviations across attacks range from 0.002/0.002 to 0.007/0.010.

6 Conclusion

The paper proposes ADF to mitigate membership leakage in diffusion models while maintaining privacy, utility, and computational efficiency across architectures and datasets.

  • 6 Conclusion: ADF provides effective privacy protection while maintaining high computational efficiency and utility across various datasets and attack settings.The framework is evaluated on DDPM, latent diffusion models, and Stable Diffusion 3.

B Open Science

The released artifacts support evaluation of ADF, including defense training, membership inference attacks, and generative utility measurement, while public datasets are downloaded through provided scripts.

  • B Open Science: The artifact package includes ADF implementation, training scripts, membership inference attack scripts, and FID computation scripts.Supported attacks include SecMIStat, SecMInn, and GSA.
  • B Open Science: Code and step-by-step instructions are available for environment setup, dataset preparation, model training, utility evaluation, and membership inference attacks.The repository provides a README and is hosted at the stated GitHub address.
  • B Open Science: Experiments use public image datasets including CIFAR-10, CelebA, STL10_U, and NWPU-RESISC45, with scripts provided for downloading and preprocessing.The original datasets are not redistributed because of file size and copyright.
  • B Open Science: Pre-trained model checkpoints are omitted, but training and evaluation scripts, command-line instructions, and execution guidelines are provided for reproduction.The omission is attributed to file size limitations.

C Ethical Considerations

The study frames membership inference attacks as evaluation tools for defensive research and conducts experiments on public benchmark datasets without releasing sensitive training data or identifying individuals.

  • C Ethical Considerations: The reported evaluation includes privacy-utility trade-offs across membership inference settings and sensitivity across partition seeds.The cited tables define the evaluated metrics and sensitivity analysis scope.
  • C Ethical Considerations: Computational efficiency is included as a comparison dimension among defense methods.The evaluation scope includes a dedicated computational-efficiency comparison.
  • C Ethical Considerations: The study uses existing membership inference attacks only to quantify membership leakage and evaluate the proposed defense, not to facilitate attacks on real-world systems.The authors state that no private or sensitive training data are released.
  • C Ethical Considerations: All experiments use publicly available benchmark datasets for controlled empirical evaluation and do not identify real individuals or infer sensitive attributes from generated samples.The stated datasets are widely used in generative modeling research.
Loading 2609.10608v1…