PaperlayerView exact revision on arXiv

Source-linked AI summary

Difficulty Calibration Across Time for Membership Inference in Diffusion Models

Michael Joseph, Yedid Hoshen

arXiv:2610.40616v1cs.CVcs.CRcs.LG

TL;DR

Absolute reconstruction loss can confuse image difficulty with training-set membership in diffusion models. The paper introduces reference-checkpoint and within-trajectory calibration methods; experiments report strong gains, including a doubled attack performance on CIFAR-10. Their use is bounded by access to denoising outputs for candidate images.

  • Problem

    Absolute denoising loss varies with image difficulty, so an easy non-member can score lower than a difficult member and confound membership inference.

    Source: §1 · p. 2, §3.2 · p. 3

  • Method

    RMC compares fine-tuned and pretrained-checkpoint losses, while TSC compares target-timestep loss with neighboring points on the same model’s denoising trajectory.

    Source: §1 · p. 2, §3.3.2 · p. 6

  • Results

    62.3% TPR@1%FPR on CIFAR-10: TSC doubles existing attacks, and it consistently outperforms prior baselines across DDPM and Stable Diffusion settings.

    Source: §1 · p. 2

  • Takeaways & Limitations

    RMC provides near-perfect uncaptioned auditing of fine-tuned data, while TSC frequently doubles the TPR@1%FPR of the strongest trajectory-based baselines in unconstrained settings.

    Source: §5 · p. 9

    Both methods assume access to denoising outputs for candidate images at selected noise levels, limiting direct application through APIs that expose only final generated images.

    Source: §5 · p. 9

Abstract

from arXiv · show

Modern diffusion models are trained on massive datasets that often include copyrighted or private images. Membership inference attacks (MIAs) address this by testing whether a given image was in a model's training set. The standard approach scores an image by its reconstruction loss and flags low-loss images as members. This often fails because some images are simply easier to reconstruct than others, so a simple unseen image can score a lower loss than a complex training image. Difficulty calibration separates memorization from how difficult an image is to begin with. We show how to calibrate along two axes of time: training time and diffusion time. First, across training time, we introduce Reference-Model Calibration (RMC), which compares a fine-tuned model's loss against an earlier checkpoint. This cleanly isolates newly memorized data and gives near-perfect auditing, even without text captions. Since earlier checkpoints are not always available, we next present Temporal Self-Calibration (TSC), which calibrates across diffusion time using only a single model. Our key observation is that an image's difficulty persists across noise levels, whereas memorization is localized in time. By comparing the loss at a target timestep against neighboring timesteps from the same model, TSC attenuates image difficulty using the model's own trajectory. Empirically, our methods beat the state of the art by a wide margin, and unlike most existing attacks, generalize across different model parameterizations such as $x$-, $v$-, and $\epsilon$-prediction models. Code is available at https://github.com/MichaelJoseph777/TSC.

1 INTRODUCTION

Diffusion-model membership inference is confounded by variation in image difficulty, making absolute reconstruction loss unreliable. The paper calibrates difficulty across training and diffusion time and reports strong auditing results across settings.

1 INTRODUCTION

  • MIAs test whether a sample belonged to a training set, but diffusion’s iterative denoising makes earlier attack approaches less effective.Prior MIAs were studied for GANs and VAEs, motivating attacks tailored to diffusion models.Source: §1 · p. 1
  • Images can have low reconstruction loss because they are intrinsically easy, while difficult training images can have higher loss, confounding membership scores.The paper describes this difficulty factor as making absolute loss an unreliable membership metric.Source: §1 · pp. 1–2
  • TSC compares losses at a target diffusion timestep with neighboring timesteps, using the local trajectory to suppress image-difficulty variance.The approach relies on image difficulty persisting across nearby noise levels while membership is localized in time.Source: §1 · p. 2
  • RMC compares a fine-tuned model’s loss with its base checkpoint to isolate newly memorized data from general reconstruction difficulty.The introduction notes that RMC is irrelevant for pretraining corpora and depends on access to an earlier checkpoint.Source: §1 · p. 2
  • 62.3% TPR@1%FPR on CIFAR-10: TSC doubles the performance of existing attacks.The paper also reports that TSC consistently outperforms prior baselines across DDPM and Stable Diffusion settings.Source: §1 · p. 2
  • The paper presents TSC as a general method and RMC as a calibration approach for auditing fine-tuned models.It reports evaluating across architectures, datasets, and training targets, including robustness to different parameterizations.Source: §1 · p. 2

2 RELATED WORK

Related work ranges from general membership-inference threat models to diffusion-specific attacks that use losses or denoising trajectories. These methods differ in access assumptions, query strategies, and calibration requirements.

2 RELATED WORK

  • MIAs test whether a record was in training; white-box, black-box, and gray-box attacks differ by access to parameters, gradients, or observable outputs.Diffusion attacks querying intermediate denoising states generally assume local model access without full white-box analysis.Source: §2 · p. 2
  • Threshold-based MIAs classify a sample as a member when its score falls below τ, with thresholds calibrated to a target false-positive rate (Yeom et al., 2018; Carlini et al., 2022).More sophisticated attacks may use shadow models to calibrate score distributions (Shokri et al., 2017; Carlini et al., 2022).Source: §2 · p. 3
  • Diffusion attacks include Naive loss [Matsumoto et al., 2023], SecMI, PIA [Kong et al., 2024], and conditional-loss CLiD [Zhai et al., 2024].LiRA achieved high precision but requires numerous shadow models, which the passage describes as computationally prohibitive (Carlini et al., 2022).Source: §2 · p. 3

3 METHODOLOGY

The section motivates calibrating membership scores against image-specific difficulty and develops references across training time and diffusion time. RMC uses an earlier checkpoint, while TSC uses the image’s own loss trajectory.

3.1 PRELIMINARIES

  • Diffusion and flow-matching trajectories are aligned by log-SNR, while denoising networks may predict x, v, or ϵ targets.The section evaluates denoising-loss trajectories across log-SNR for a fixed sample.Source: §3.1 · p. 3

3.2 MOTIVATION

  • Image difficulty confounds absolute loss: an easy non-member can score lower than a difficult member, so candidates need image-specific references.The section illustrates the overlap in CIFAR-10 DDPM loss distributions even at the optimal log-SNR.Source: §3.2 · p. 3, Figure 1 · p. 4
  • With 20 queries, uncaptioned RMC achieves near-perfect auditing across datasets; a two-query version outperforms 15-query fully captioned CLiD.Table 1 evaluates RMC on realistic Stable Diffusion fine-tunes without captions at λ = 1 log-SNR, using one or 10 paired noise draws; parentheses give queries per candidate.Source: §3.2 · p. 4, Table 1 · p. 5

3.3 DIFFICULTY CALIBRATION ACROSS TIME

  • Assuming difficulty is a persistent multiplicative factor, subtracting stabilized log-losses acts as a log-ratio that cancels the shared factor.The section applies this relative calibration across training time and diffusion time.Source: §3.3 · pp. 4–5

3.3.1 REFERENCE-MODEL CALIBRATION

  • RMC subtracts an image’s pretrained-checkpoint loss from its fine-tuned loss, using the earlier model as a baseline for difficulty before membership exposure.Members have more negative residuals; the MS-COCO example reports highly correlated checkpoint losses and members shifting below the ASR threshold after subtraction.Source: Figure 2 · p. 4, §3.2 · p. 4, §3.3.1 · p. 5

3.3.2 TEMPORAL SELF-CALIBRATION

  • TSC compares loss at a target log-SNR with a symmetric two-point chord, using a wide span to anchor baseline trajectory rather than local curvature.The target is placed at an intermediate memorization peak; its downward deviation yields a more negative residual for members than non-members.Source: §3.3.2 · p. 6

4 EXPERIMENTS

Across DDPM benchmarks, TSC outperforms prior attacks, including in the low-false-positive regime. Evaluations also test uncaptioned Stable Diffusion fine-tunes, different prediction and loss parameterizations, and choices in TSC’s temporal calibration.

4.1 EXPERIMENT SETUP

  • The experiments cover DDPM and DDPM++ image-synthesis models and Stable Diffusion v1.4 fine-tuned on Pokémon, MS-COCO, and Flickr30k.Performance is measured with ASR, AUC, and TPR@1%FPR.Source: §4.1 · p. 6

4.2 TEMPORAL SELF-CALIBRATION EVALUATIONS AND ANALYSIS

  • 62.31% TPR@1%FPR on CIFAR-10, versus 30.90% for PIAN, and TSC achieves state-of-the-art performance across the DDPM datasets.The evaluation compares TSC with Naive, SecMI, PIA, and PIAN; results are reported across four DDPM datasets.Source: §4.1 · p. 6, §4.2 · p. 7, Table 2 · p. 7
  • TSC outperforms all baselines on MS-COCO and Flickr30k across every metric, while PIA prevails on the visually uniform, limited-scale Pokémon fine-tune.The Stable Diffusion evaluation omits ground-truth captions to reflect auditors’ typically unknown training prompts.Source: §4.2 · p. 7, Table 3 · p. 7

4.3 IMPACT OF PREDICTION PARAMETERIZATION AND TRAINING TARGET

  • Across nine CIFAR-100 models, TSC has the highest TPR@1%FPR in every configuration, beating the strongest baseline by 23.84 percentage points on average.Peak leakage shifts toward higher log-SNR as the training target changes from x to v to ϵ, while changing the native prediction head has little effect on peak location.Source: §4.3 · p. 8, Table 4 · p. 8

4.4 ABLATIONS

  • Log-loss TSC reaches nearly 26% TPR@1%FPR, while raw-loss TSC falls below Naive; log-loss makes the multiplicative difficulty confound additive and easier to cancel.As query budgets rise from q = 3 to q = 45, log-loss TSC improves its temporal-contrast estimate, while Naive remains near 4%.Source: §4.4 · p. 8, Figure 5 · p. 8
  • Endpoints achieve the highest mean AUC and TPR@1%FPR at both tested query budgets, and repeated draws at fixed endpoints outperform adding temporal levels at q = 45.The filter comparison covers four DDPM datasets; the full Gaussian uses 45 levels and is unavailable at q = 15.Source: §4.4 · p. 9, Table 5 · p. 9
  • At q = 15, the symmetric stencil has the highest mean TPR@1%FPR, although an equal-weight (1.5, 3.5) window yields slightly higher overall AUC.Symmetry strictly dominates in the low-FPR regime, and interpolation weighting does not consistently improve asymmetric windows.Source: §4.4 · p. 9

5 CONCLUSION

Difficulty calibration across training time and diffusion time mitigates the image-difficulty confound in diffusion MIAs. RMC enables near-perfect uncaptioned auditing of fine-tuned data, while TSC often doubles low-FPR detection against trajectory-based baselines; the methods also have access and scope limits.

5 CONCLUSION

  • RMC enables near-perfect uncaptioned auditing of fine-tuned data, and TSC frequently doubles TPR@1%FPR of the strongest trajectory-based baselines in fully unconstrained settings.The conclusion reports substantial gains in strict low-FPR membership detection across diverse datasets, architectures, and training objectives.Source: §5 · p. 9
  • TSC requires denoising outputs for candidate images at selected noise levels, limiting direct application through APIs that expose only final generated images.The methods use forward predictions and do not require gradients.Source: §5 · p. 9
  • RMC requires the checkpoint used for fine-tuning, restricting applicability when that checkpoint is unavailable or the model’s training lineage is unknown.The authors suggest investigating related public checkpoints or ensembles of reference models.Source: §5 · p. 9
  • RMC measures loss changes between checkpoints but does not separately identify membership in the reference model’s pretraining corpus.The authors propose studying pretraining–fine-tuning data overlap and calibration across successive checkpoints.Source: §5 · p. 9

ETHICS STATEMENT

  • Membership inference has both privacy risks and legitimate auditing uses, so the authors advocate authorized auditing and conclusions that account for false positives and false negatives.A negative result does not prove exclusion from training, and an audit does not identify the complete training dataset.Source: ETHICS STATEMENT · p. 10

A PREDICTION AND LOSS PARAMETERIZATIONS

Diffusion and flow-matching models can predict clean data, noise, or velocity, and evaluate their training objective in any of these spaces. Deterministic links among these quantities permit conversion before loss computation.

A PREDICTION AND LOSS PARAMETERIZATIONS

  • Models can predict clean data x, noise ϵ, or velocity v, and evaluate their training objective in any of the three spaces.The forward process deterministically links these quantities.Source: A PREDICTION AND LOSS PARAMETERIZATIONS · p. 13
  • A prediction in one space can be algebraically converted to another before computing the loss.Table 7 gives conversion equations for all nine prediction-space and squared-error-space combinations under the linear interpolation schedule.Source: A PREDICTION AND LOSS PARAMETERIZATIONS · p. 13
  • Table 7’s columns specify direct network prediction space, while its rows specify the space in which squared error is measured.The table covers the nine combinations under the linear flow schedule.Source: Table 7 · p. 13, A PREDICTION AND LOSS PARAMETERIZATIONS · p. 13

B EXPERIMENTAL DETAILS

The experimental details specify common evaluation protocols and model-training settings, then describe temporal-filter selection and analyses of reference symmetry. The symmetry discussion explains what the contrast removes and the limits of that interpretation.

B EXPERIMENTAL DETAILS

  • All membership-inference experiments distinguish a designated member split from a disjoint hold-out split, using final EMA checkpoints unless stated otherwise.Images are normalized to [−1, 1], and models are trained with Adam.Source: B EXPERIMENTAL DETAILS · p. 13
  • The unconditional DDPM experiments use ϵ-prediction with T = 1000 diffusion steps and a linear noise schedule, alongside a specified UNet, optimizer, and training configuration.The UNet uses base width 128, channel multipliers (1, 2, 2, 2), two residual blocks per resolution, and attention at 16 × 16; runs use random horizontal flips.Source: B EXPERIMENTAL DETAILS · p. 13, Table 8 · p. 13
  • Stable Diffusion v1.4 is fine-tuned at 512 × 512 by updating the UNet while keeping the VAE and text encoder fixed.The settings use mixed precision, gradient checkpointing, EMA, batch size 1 with gradient accumulation 4, learning rate 10−5, and random cropping unless stated otherwise.Source: B EXPERIMENTAL DETAILS · p. 13, Table 9 · p. 14, Table 10 · p. 14
  • Pokémon, MS-COCO, and Flickr30k experiments use image-caption pairs with staged caption metadata and no caption dropout unless stated otherwise.Pokémon captions come from its BLIP-caption dataset; the other datasets use annotations from constructed train/hold-out splits.Source: B EXPERIMENTAL DETAILS · p. 13
  • The DDPM++ CIFAR-100 experiment varies nine linear flow-matching prediction/loss pairs while fixing architecture, data split, optimizer, and training budget.It uses EDM base code and 2000 epochs, with no data augmentation, to isolate prediction-head and training-loss-target effects.Source: B EXPERIMENTAL DETAILS · p. 14

B.1 TEMPORAL-FILTER SELECTION

  • Filter selection searches 45 log-SNR levels, with the center fixed at λ⋆ = 0.75, and tests even, Gaussian, windowed Gaussian, and reverse-window supports.The endpoint stencil is fixed at ∆ = 2.5, k = 1; filter families vary their support, weights, or Gaussian bandwidth.Source: §B.1 · p. 14
  • Restricted filter supports include the center and use m = 2k + 1 levels, with query budgets of q = 15 or q = 45 depending on filter family.Configurations are selected by mean AUC across four first-half shadow models, then reused on matching held-out checkpoints; lower temporal residuals are member-like.Source: §B.1 · p. 14

C TEMPORAL REFERENCE SYMMETRY

  • The symmetric temporal score removes sample-dependent offsets and odd trajectory variation, but symmetry is not a guarantee that all image difficulty is eliminated.Sample-dependent even variation remains, odd membership variation would also be removed, and nominally equal log-SNR spacing can map to slightly unequal discrete-schedule distances.Source: C TEMPORAL REFERENCE SYMMETRY · pp. 15–16

C.1 SYMMETRY ABLATION PROTOCOL

The symmetry ablation compares one- and two-reference temporal contrasts at fixed query budgets across four held-out DDPM checkpoints. Symmetry has the highest mean TPR@1%FPR at both budgets, but does not lead every checkpoint or metric.

C.1 SYMMETRY ABLATION PROTOCOL

  • The protocol compares single references with symmetric (2.5, 2.5) and unequal (1.5, 3.5) or (3.5, 1.5) two-reference placements, all spanning five nominal log-SNR units.The query center is λ⋆ = 0.75; placements and score direction are not selected using held-out membership labels.Source: §C.1 · p. 16
  • At q ∈ {15, 45}, three-point contrasts use q/3 draws per level, while one-reference contrasts allocate the same total query budget between center and reference.Odd-component diagnostics instead use q/3 draws at all three levels so the equation identities compare the same loss estimates.Source: §C.1 · p. 16
  • The symmetric triple’s requested log-SNR levels map to approximately (−1.752006, 0.755986, 3.264399), with a midpoint offset of approximately 2.11 × 10−4.The deployed baseline retains equal weights, while unequal interpolation weights use realized distances; AUC and TPR@1%FPR are averaged across checkpoints rather than pooled.Source: §C.1 · p. 16
  • Symmetry has the highest mean TPR@1%FPR across four held-out checkpoints at both budgets, but does not dominate every checkpoint or metric.The equal-weight (1.5, 3.5) layout has higher AUC on all four checkpoints; interpolation-weight effects depend on orientation and metric.Source: §C.1 · p. 16, Table 11 · p. 17
  • With equal per-point precision, the left-only contrast retains a larger mean gap than the symmetric contrast but has substantially greater spread.The symmetric contrast’s smaller endpoint-difference separation relative to its spread is consistent with difficulty-related variation being suppressed, but does not establish that the endpoint difference is pure nuisance.Source: §C.1 · p. 16, Table 12 · p. 17

D PREDICTION-LOSS ANALYSIS

This analysis separates explicit training-objective weights from model-dependent errors and empirical membership leakage. It finds that leakage peaks shift toward higher log-SNR from x-loss to v-loss to ϵ-loss, while clarifying that weight centers do not determine those peaks.

D.1 CONVERSIONS AND OBJECTIVE WEIGHTING

  • Common-coordinate factorization does not imply identical optimization or learned models across native prediction heads and loss targets.The analysis concerns interior timesteps, before numerical endpoint clamping.Source: §D.1 · p. 18
  • For one fixed model and interior timestep, changing evaluation loss space preserves image rankings, ROC curves, and AUC; separately trained configurations need not share those errors.The invariance follows because squared errors are scaled by a positive factor shared across images, including after averaging the same noise draws.Source: §D.1 · p. 18

D.2 TEMPORAL-WEIGHT BARYCENTERS AND THEIR ORDERING

  • The centers of explicit objective weights move from noisy toward clean timesteps as the training-loss target changes from x to v to ϵ.The ordering applies to the explicit weights, not to learned errors or membership peaks; a single-timestep sampling distribution gives equality throughout.Source: §D.2 · pp. 18–19

D.3 EMPIRICAL LOCALIZATION OF MEMBERSHIP LEAKAGE

  • The peak-localization experiment evaluates nine non-augmented CIFAR-100 DDPM++ models at epoch 2000, averaging 50 shared noise draws per image.Figure 6 reports single-timestep absolute-loss AUC, with lower loss indicating membership at every level.Source: §D.3 · p. 19, Figure 6 · p. 19
  • Membership peaks occur at λ = −0.75, 0.75, and 2.00 for x-, v-, and ϵ-loss, respectively, shifting toward higher log-SNR across the losses.The observed peaks are interior and differ from the explicit-weight barycenters.Source: Figure 6 · p. 19, §D.3 · p. 20

E ADDITIONAL EXPERIMENTS

Additional DDPM experiments test TSC at shorter training horizons and without horizontal-flip augmentation. TSC remains strongest across the reported datasets and metrics, though shorter training weakens leakage, especially on STL-10-U and Tiny ImageNet.

E ADDITIONAL EXPERIMENTS

  • Unless otherwise stated, results use a held-out second evaluation split, with operating points selected on first-half shadow checkpoints.TSC uses an endpoint stencil with Δ = 2.5 and five noise draws per trajectory point (q = 15), without frequency filtering.Source: E ADDITIONAL EXPERIMENTS · p. 20

E.1 EARLIER DDPM CHECKPOINTS

  • At 2K checkpoints, shorter training weakens membership leakage for every attack, especially on STL-10-U and Tiny ImageNet, while TSC remains strongest across metrics and datasets.Tables 13 and 14 report the CIFAR datasets and STL-10-U/Tiny ImageNet, respectively.Source: §E.1 · p. 20, Table 13 · p. 20, Table 14 · p. 20

E.2 DDPMS WITHOUT HORIZONTAL FLIPS

  • Without horizontal flips, TSC’s advantage persists at both 2K and 4K epochs and is especially pronounced at low FPR after longer training.The 4K checkpoints reproduce the no-flip training regime commonly used in prior diffusion-MIA evaluations.Source: §E.2 · p. 20, Table 15 · p. 21, Table 16 · p. 21

E.3 CONDITIONING AND DIFFUSION SCHEDULE

These experiments vary class conditioning and the diffusion noise schedule. TSC remains the strongest attack in both settings, while class-condition access strengthens every attack and absolute leakage differs substantially by schedule.

E.3 CONDITIONING AND DIFFUSION SCHEDULE

  • Access to the ground-truth class condition strengthens every attack, while TSC remains best with either the condition dropped or supplied.Both evaluations use the same class-conditional ϵ-prediction, ϵ-loss checkpoint.Source: §E.3 · p. 21, Table 17 · p. 21
  • Although absolute leakage differs substantially between cosine and linear noise schedules, TSC provides the strongest separation under both.The comparison uses class-conditional CIFAR-100 DDPM checkpoints at 2K epochs.Source: §E.3 · p. 21, Table 18 · p. 22

E.4 FULL PREDICTION–LOSS RESULTS AND OPERATING-POINT SELECTION

Across three realistic Stable Diffusion fine-tunes, reference calibration improves both reported metrics over uncalibrated loss around the operating region. The section also specifies shared operating-point and query protocols for full prediction–loss results.

E.4 FULL PREDICTION–LOSS RESULTS AND OPERATING-POINT SELECTION

  • The full DDPM++ results cover x-, v-, and ϵ-loss, each with and without augmentation, reporting AUC, ASR, and TPR@1%FPR.Non-augmented models were trained for 2000 epochs and augmented counterparts for 4000 epochs.Source: §E.4 · p. 22, Table 19 · p. 22, Table 20 · p. 22
  • Each loss uses one shared λ∗ across prediction parameterizations and augmentation variants; TSC uses 15 queries, versus 20 for Naive and 12 for SecMI.TSC uses five draws at each of three levels, with Δ = 2.5 and no frequency filtering; SecMI follows its original fixed protocol.Source: §E.4 · p. 22
  • The ϵ-loss tables provide the augmented and non-augmented DDPM++ configurations alongside the x- and v-loss results.Source: Table 23 · p. 23, Table 24 · p. 23

E.5 REFERENCE CALIBRATION ACROSS NOISE LEVELS

  • Around the Table 1 operating region, RMC substantially improves AUC and TPR@1%FPR over uncalibrated loss on all three datasets.The curves also show that membership signal can still vary along the diffusion trajectory.Source: §E.5 · p. 23, Figure 7 · p. 24

Related papers

Loading 2610.40616v1…