PaperlayerView exact revision on arXiv

Source-linked AI summary

Revisiting Certified Defense with Differential Privacy on Vision Transformers

Jun Yan, Weiquan Huang, Qixian Zhang, Yan Bai, Shutai Zhang

arXiv:2609.31310v1cs.CRstat.AP

TL;DR

The paper asks how PixelDP can provide useful certified robustness when ported to Vision Transformers. It analyzes sensitivity constraints, repairs a Laplace configuration, and derives a corresponding certificate; the inherited grouped bound collapses while the spectral repair restores trainability.

  • Problem

    How can PixelDP be ported to self-attention architectures while preserving a valid and useful robustness certificate?

    Source: §1 · p. 2

  • Method

    The paper ports PixelDP to Vision Transformers, compares sensitivity constraints, and derives a dimension-free (ε, δ) Laplace guarantee under ℓ2 sensitivity with a closed-form calibration.

    Source: §1 · p. 2, §5 · p. 6

  • Results

    The inherited grouped ℓ1 bound collapses for Laplace models, while a spectral constraint restores trainability across CIFAR-10, CIFAR-100, and SVHN and the derived guarantee closes the certification gap.

    Source: §1 · p. 2, §5 · p. 6

  • Takeaways & Limitations

    The closed-form calibration allows existing repaired checkpoints to be certified without retraining.

    Source: §1 · p. 2

    The ViT models are small, trained from scratch at low resolution, and the Laplace certificate costs about 1.47× the Gaussian noise at matched (ε0, δ).

    Source: §5 · p. 7

Abstract

from arXiv · show

Certified defenses that incorporate differential privacy have proven effective on Convolutional Neural Networks (CNNs), furnishing rigorous robustness guarantees against norm-bounded adversaries. However, the certified robustness behavior of Pixel Differential Privacy (PixelDP) remains largely unexplored with the self-attention architecture now dominating the deep-learning landscape. Given that the Transformer has a profound impact on our daily applications from the digital world to the physical world, it is crucial to study certified robustness through differential-privacy-style stability. To fill this research gap, we revisit this construction in Vision Transformers and identify a failure mode that is largely hidden in the convolutional setting. When noise is injected after the patch embedding, the Laplace mechanism with the inherited grouped $\ell_1$ sensitivity bound collapses to chance-level accuracy across noise scales, whereas the Gaussian mechanism remains trainable. This contrast isolates the source of failure: not the injected noise itself, but the geometry of the sensitivity constraint. We show that the attenuation induced by the inherited $\Delta_{1,1}$ projection increases with layer width and kernel size according to a random-matrix scale $C/(\sqrt{M}+\sqrt{N})$. Replacing the $\ell_1$-type constraint with a spectral-norm constraint eliminates the collapse across datasets and architectures, but creates a fundamental obstacle: the repaired models no longer satisfy the sensitivity condition required by the standard Laplace certificate. We resolve this mismatch by deriving a dimension-free $(\varepsilon,\ \delta)$-privacy guarantee for the Laplace mechanism under $\ell_2$ sensitivity through concentration of the privacy loss.

1. Introduction

PixelDP provides certified robustness through sensitivity-constrained noise, but its behavior in self-attention architectures remains insufficiently explored. This paper ports PixelDP to Vision Transformers, diagnoses the inherited Laplace sensitivity constraint, and develops a certification route for a spectral repair.

Motivation

  • PixelDP’s certified robustness in self-attention architectures remains insufficiently explored, motivating the question of how to port it while preserving a valid, useful certificate.PixelDP constrains a pre-noise mapping’s sensitivity and injects calibrated noise to stabilize randomized predictions.Source: §1 · pp. 1–2

Noise placement

  • Noise immediately after the patch embedding gives a controllable sensitivity point; noise after attention is absorbed by training and certifies nothing.The patch embedding is a linear layer whose sensitivity can be explicitly controlled.Source: §1 · p. 2

Sensitivity diagnosis

  • The inherited grouped ℓ1 bound makes the Laplace route collapse to chance accuracy, while Gaussian noise remains trainable; the bound is more conservative than the stride-aware operator norm.The passage attributes the failure to the sensitivity constraint’s geometry, not noise magnitude or the Laplace distribution itself.Source: §1 · p. 2
  • Attenuation scales as C/(√M+√N) under the Gaussian initialization model, and the measured norm ratio predicts a severe local signal-to-noise-ratio drop.The resulting optimization difficulty accounts for failure across the reported training recipes.Source: §1 · p. 2

Repair and certification

  • Spectral constraint restores Laplace trainability across CIFAR-10, CIFAR-100, and SVHN, while the inherited grouped-bound failure persists on ImageNet.The repair creates a certification gap because the standard Laplace guarantee requires ℓ1 sensitivity; the paper derives an (ε, δ) guarantee under ℓ2 sensitivity.Source: §1 · p. 2
  • The derived Laplace guarantee applies at every attack radius and re-certifies trained checkpoints without retraining, but calibrated Laplace noise has about 1.47× Gaussian standard deviation at matched (ε, δ).The analysis uses the shifted first absolute moment of a Laplace variable and Hoeffding’s inequality on independent, bounded privacy-loss coordinates.Source: §1 · p. 2

Adversarial training

  • Matched-budget adversarial training improves certified accuracy over the nonzero operating region on all three datasets; on SVHN, noisy pretraining followed by adversarial fine-tuning restores training.The protocol is designed to isolate the effect of the attack from the training recipe.Source: §1 · p. 2

Scope of evaluation

  • The contributions include an ImageNet study across Inception and ViT-Base backbones and a matched-budget ablation reporting certified gains at every radius on CIFAR-10, CIFAR-100, and SVHN.The study also identifies the patch embedding as a directly enforceable pixel-space sensitivity bound.Source: §1 · p. 2

2. Related Work

Related work establishes PixelDP and randomized smoothing as noise-based certified defenses, while prior ViT robustness results do not directly provide PixelDP’s needed pixel-space sensitivity control. This paper studies the inherited mechanism–constraint pairings rather than proposing a new smoothing distribution.

Certified defenses from noise

  • PixelDP constrains sensitivity before calibrated noise, while randomized smoothing injects Gaussian noise at the input and provides a tighter ℓ2 certificate.The paper’s stated aim is diagnostic rather than to outperform specialized input-space smoothing on pure ℓ2 certification.Source: §2 · pp. 2–3

Positioning

  • The paper studies how PixelDP’s inherited mechanism–constraint pairings interact with ViT patch embeddings, rather than proposing a new smoothing distribution.It connects the spectral repair to certification under ℓ2 sensitivity.Source: §2 · p. 3

Robustness of Vision Transformers

  • Prior attention bounds address token-space smoothness but do not directly control the full pixel-to-token prefix needed for PixelDP’s enforceable pixel-space sensitivity bound.The paper therefore places noise immediately after the patch embedding, before the first attention block.Source: §2 · p. 3

Privacy and robustness together

  • Unlike work on private training data, PixelDP uses privacy as an inference-time robustness analysis tool; the two goals are described as orthogonal and compatible.The paper adopts adversarial-training guidance from prior ViT recipes for its ablation.Source: §2 · p. 3

3. Porting PixelDP to Vision Transformers

The ViT port places noise after the patch embedding, where sensitivity is explicitly controllable, but the inherited grouped ℓ1 constraint severely attenuates the signal. A spectral constraint restores trainability; a dimension-free Laplace privacy guarantee then closes the resulting certification gap.

3.1. Where the Noise Can Go

  • The patch embedding provides an explicitly controllable sensitivity bound, while noise after an attention block is absorbed during training and has no certified pixel-space sensitivity bound.The patch embedding is linear, and enforcing Δ2,2 = 1 lets its token-space guarantee transfer to pixel space with constant one.Source: §3.1 · pp. 3–4

3.2. Sensitivity-Induced Signal Collapse

  • The inherited grouped ℓ1 projection reduces the weight-dependent signal-to-noise measure by ρ_grp, while the spectral constraint avoids this attenuation and improves Laplace trainability across datasets.At the ViT patch embedding, ρ ≈ 123 leaves about 4% of the injected noise as signal at L = 0.03; the spectral projection leaves signal at 2.4 times the noise. The repair creates a certification gap because it does not provide the ℓ1 bound used by the existing Laplace guarantee; naive conversion incurs √d ≈ 110.9 here.Source: §3.2 · p. 4

3.3. A Dimension-Free Certificate for Laplace under ℓ2 Sensitivity

  • Under spectral ℓ2 constraints, coordinatewise Laplace privacy-loss fluctuation is controlled by ∥v∥2, canceling dimension dependence; it applies at every shift, enabling certification with only a certification pass and no group privacy.Laplace noise was 3.73 L versus 2.54 L for Gaussian, a factor of 1.47. At ε0 = 1, eliminating d ≈ 110.9 corresponded to a 42× noise reduction at their layer.Source: §3.3 · p. 5

3.4. A Gaussian-DP Calibration Variant

  • The Gaussian-DP variant fixes a target µ0 at construction, converts the radius-dependent µ(L′) to (ε, δ), and is an alternative calibration rather than group-privacy scaling.For a Gaussian mechanism with sensitivity S and scale σ, µ = S/σ; the variant sets σ = L/µ0.Source: §3.4 · p. 5

3.5. Certification Meets Adversarial Training

  • Adversarial training improves certified accuracy over matched controls at positive radii, but on SVHN joint training collapses to the class prior; noisy pretraining followed by adversarial fine-tuning avoids that failure.The matched-budget protocol holds training hyperparameters fixed except for the attack to isolate its effect.Source: §3.5 · p. 5

4. Experiments

Experiments evaluate the port on CIFAR-10, CIFAR-100, SVHN, and ImageNet under specified training and certification protocols. The grouped-bound Laplace failure and spectral repair recur across datasets, while the ImageNet tests and re-certification results show their scale and certification trade-offs.

4.1. Setup

  • The evaluation covers CIFAR-10, CIFAR-100, SVHN, and ImageNet; certified tables use 300 draws and Clopper–Pearson bounds with a union bound at η = 0.95.ViT-Tiny models use patch size 4 at resolution 32; ImageNet uses a frozen autoencoder front end with Inception-v3 and ViT-Base backbones.Source: §4.1 · pp. 5–6

4.2. Main Result

  • The faithful port’s grouped-bound Laplace arm falls to 28.9% on CIFAR-10 and 7.5% on CIFAR-100, and reaches the majority-class prior on SVHN at every noise level; spectral-bound Laplace recovers near Gaussian accuracy.On CIFAR-10 and CIFAR-100, the repaired arm exceeds Gaussian at small L. On ResNet-50 with C = 576, the grouped-bound Laplace arm trains normally on all three datasets.Source: §4.2 · p. 6

4.3. ImageNet at Scale

  • On ImageNet, Inception’s 77.8% baseline matches the original paper’s 77.5%, and ViT-Base dominates Inception at every operating point despite nearly identical noise-related accuracy drops.For Gaussian at L = 0.1, drops are −4.1 versus −4.2 points; at L = 1, they are −27.0 versus −29.3 points. The grouped-bound Laplace collapse also reappears with the C = 3200 front end.Source: §4.3 · p. 6

4.4. Certified Accuracy of the Repaired Laplace Mechanism

  • Re-certified Laplace radii are necessarily smaller than Gaussian radii at matched (ε0, δ), reflecting both 1.47× larger calibrated noise and the superlinear growth of ε(L′).The reported radii are presented as the cost of certifying the heavier-tailed mechanism, not as a new state of the art.Source: §4.4 · p. 6

5. Conclusion

The conclusion frames the ViT PixelDP port as revealing an inherited grouped-bound failure and closing the repaired Laplace certification gap, while limiting empirical claims to large failure-and-repair effects.

5. Conclusion

  • The ViT port exposes inherited grouped ℓ1 Laplace collapse; stride-aware control recovers much, and a norm-ratio law explains the grouped-bound attenuation.The Laplace certificate under ℓ2 sensitivity closes the repair's certification gap.Source: §5 · p. 6
  • ImageNet experiments with a fixed autoencoder front end and two downstream backbones empirically illustrate the post-processing property.The accompanying table compares accuracy under standard DP and f-DP accounting for Gaussian and grouped-bound Laplace configurations.Source: §5 · p. 7, Table 2 · p. 7
  • Matched-budget ablations show adversarial training improves certified accuracy over the nonzero operating region.The table reports certified accuracy for repaired Laplace models at L = 0.1 using Monte Carlo certification with n = 300 draws.Source: §5 · p. 7, Table 3 · p. 7
  • Small, low-resolution scratch-trained ViTs and single-seed runs constrain claims: absolute accuracy trails modern recipes, and small configuration differences remain unquantified.The study examines ℓ1 and ℓ2 threats; the interaction with ℓ∞ training is untouched.Source: §5 · pp. 7–8

A.1. Differential Privacy over a Norm Metric

The appendix states norm-metric privacy properties, describes PixelDP's sensitivity-and-noise pipeline and certification procedure, and relates the construction to randomized smoothing.

A.1. Differential Privacy over a Norm Metric

  • Norm-metric differential privacy bounds how randomized outputs change for nearby inputs; post-processing preserves the guarantee, and bounded outputs have stable expectations.PixelDP uses an ℓp image metric rather than database record replacement.Source: §A.1 · p. 9

A.2. The PixelDP Construction

  • PixelDP splits the network around a noise layer and calibrates Laplace or Gaussian noise to sensitivity; its inherited grouped bound is conservative for non-overlapping patches.The framework's guarantee passes through subsequent computation provided no skip connection bypasses the noise.Source: §A.2 · p. 10

A.3. Certified Prediction

  • Certified prediction estimates class expectations with confidence bounds and returns the largest perturbation radius satisfying the generalized robustness condition.Certified accuracy counts test points that are correct and certified at or above a specified radius.Source: §A.3 · p. 10

A.4. Relation to Randomized Smoothing

  • Randomized smoothing is the input-noise special case of PixelDP, while placing noise after a trained layer creates sensitivity-driven interactions with that layer.Source: §A.4 · p. 10

B.1. Proof of Proposition 3.1 (norm-ratio law)

The appendix bounds the grouped-to-spectral norm ratio and supplies shifted-moment and tail arguments used in the Laplace privacy derivation.

B.1. Proof of Proposition 3.1 (norm-ratio law)

  • Under Gaussian initialization, the common weight scale cancels from the norm ratio, whose leading order increases with filter count and kernel size.The proof bounds grouped ℓ1 mass and the spectral norm before taking their ratio.Source: §B.1 · p. 10

B.2. Proof of Lemma 3.1 (shifted absolute moment)

  • For Laplace noise, the shifted absolute moment has a closed form, yielding an upper bound of v^2/(2b) on its increase over the unshifted moment.Source: §B.2 · p. 11

B.3. Proof of Lemma 3.2 (tail implies DP)

  • On the event Sε, the density ratio is at most e^ε; the complementary event contributes the tail-probability terms in the approximate-privacy bound.Source: §B.3 · p. 11

B.4. Proof of Theorem 3.1

The proof derives a dimension-free approximate-DP bound for Laplace noise under ℓ2 sensitivity, then calibrates the mechanisms and examines how placement and sensitivity constraints affect results.

B.4. Proof of Theorem 3.1

  • Independent, bounded privacy-loss coordinates permit Hoeffding concentration, yielding a dimension-free (ε, δ)-DP bound for every ordered neighboring pair.The derivation uses the shifted first absolute moment of Laplace noise and requires no group privacy.Source: §B.4 · p. 11

B.5. Proof of Corollary 3.2 (calibration)

  • Under Δ2,2 ≤ 1, sensitivity is S = L at the construction bound, and calibration solves for Laplace scale b from the target privacy level.At a candidate radius L′, the privacy parameter is recalculated using sensitivity S = L′.Source: §B.5 · p. 11

C. Gaussian-DP Calibration Formulas

  • Gaussian calibration fixes σ = L/µ0 at construction and scales the privacy parameter to µ(L′) = µ0L′/L at radius L′, without group privacy.The corresponding ε is obtained by inverting the Gaussian-DP profile at the chosen δ.Source: C. Gaussian-DP Calibration Formulas · p. 11

D.1. The constraint-decoupling Study

  • With noise after an attention block, Gaussian and Laplace accuracy stays near the non-private baseline across the construction grid, but the models lack a valid PixelDP certificate.The placement has no enforceable pixel-to-representation sensitivity bound under the protocol.Source: C. Gaussian-DP Calibration Formulas · p. 11, Figure 3 · p. 12
  • At L = 0.03, grouped Δ1,1 Laplace accuracy is 28.87% on CIFAR-10 and 7.51% on CIFAR-100, while SVHN reaches the majority-class prior; spectral Δ2,2 substantially improves low-noise performance.The spectral repair does not remove all high-noise failures: SVHN still reaches the class prior at L = 1.Source: §D.1 · pp. 11–12

D.2. Sensitivity to Monte Carlo sample size and δ

The sensitivity study finds that more Monte Carlo samples improve certified accuracy, while changing δ has different effects at zero and positive radii.

D.2. Sensitivity to Monte Carlo sample size and δ

  • Increasing n from 300 to 1,000 changes expected-prediction accuracy by 0.06 points but improves certified accuracy by up to 4.65 points; reducing δ from 0.05 to 0.01 slightly improves T = 0 certification and reduces certification at positive radii.The study uses the same repaired CIFAR-10 checkpoint and stored noise draws; the positive-radius effect follows from the increase in ε(T).Source: §D.2 · p. 12

D.3. Norm-ratio evolution during training

Constraint-only controls and training diagnostics examine the inherited grouped bound’s effect, while a separate comparison sets the scope of the repaired PixelDP model against input smoothing.

D.3. Norm-ratio evolution during training

  • At L = 0.03, CIFAR-10 accuracy rises from 28.87% with grouped Δ1,1 control to 70.83% with exact induced ℓ1→ℓ1 control and 85.50% with spectral Δ2,2 control.Only the patch-embedding sensitivity constraint changes in this comparison.Source: Table 5 · p. 12, §D.3 · p. 12
  • The measured patch-embedding ratio ρ falls from 136.99 at initialization to 53.83 by epoch 20 and remains near 55, versus the initialization prediction 117.93.The prediction is initialization-scale under i.i.d. Gaussian weights; the ratio remains large after optimization moves the layer outside that regime.Source: §D.3 · p. 12
  • The spectral constraint substantially mitigates low-noise Laplace degradation across three datasets, but failures remain at large noise levels.The diagnostics cover ViT-Tiny, three datasets, and two noise-calibration variants.Source: Figure 4 · p. 13

D.4. Comparison with input randomized smoothing

  • On CIFAR-10, repaired Laplace and input Gaussian smoothing achieve 81.14% and 81.57% prediction accuracy, respectively, while CA@0.05 is 51.65% versus 75.75%.The comparison uses ViT-Tiny and matched training budgets; PixelDP injects noise after the patch embedding, while input smoothing uses image-input noise.Source: §D.4 · p. 13

D.5. Adversarial Training Lifts the Certificate

Under matched-budget controls, adversarial training improves certified accuracy across the reported positive operating region, with a distinct two-phase training result on SVHN.

D.5. Adversarial Training Lifts the Certificate

  • The strong recipe uses 2.7× the training budget, so only the matched control reveals that the apparent deficit reflects budget rather than the attack.Near the certification boundary, the matched adversarial-training model approaches the strong recipe; CIFAR-100 is within 0.6 points at T = 0.15.Source: §D.5 · pp. 13–14, Table 7 · p. 15, Table 8 · p. 15
  • CIFAR-10 certified accuracy gains 8.30 and 10.70 points at T = 0.05 and 0.10; gains span 2.68–4.22 points on CIFAR-100 and 2.81–6.09 on SVHN through T = 0.15.These comparisons are against matched controls; both CIFAR-10 models have zero certified accuracy at T = 0.20 and 0.30.Source: §D.5 · p. 13, Figure 6 · p. 14, Table 7 · p. 15
  • On SVHN, joint training from scratch stays at the 19.6% class prior for 300 epochs, while noisy pretraining followed by adversarial fine-tuning restores the reported pattern.A constant classifier can certify every majority-class prediction at every radius, so certified accuracy should be read with conventional accuracy.Source: §D.5 · p. 14
  • The AT-only baseline is empirically stronger but lacks a PixelDP certificate because it has no randomized, sensitivity-controlled mechanism; adaptive checks show EOT stronger than PGD and BPDA weaker for the randomized model.For the deterministic model, BPDA coincides with PGD-20.Source: §D.5 · pp. 14–15, Table 9 · p. 15

D.6. Takeaways from the Experiment

The experiments identify patch-embedding placement as binding and summarize the sensitivity-bound failure, certificate repair, and matched-budget adversarial-training result. The reproduction section specifies training, certification, and evaluation protocols.

D.6. Takeaways from the Experiment

  • Only the patch embedding supports an enforceable ViT sensitivity bound; attention-side noise is absorbed into a certificate-free model.Source: §D.6 · p. 15
  • The inherited grouped constraint causes severe attenuation, with its initialization scale following a closed-form law from C = 144 to C = 3200.Source: §D.6 · p. 15
  • A dimension-free ℓ2 Laplace certificate re-certifies repaired checkpoints without retraining, at a 1.47× noise cost against Gaussian.Source: §D.6 · p. 15
  • Swapping frozen Inception for frozen ViT-Base shifts ImageNet baselines, not degradation curves; matched-budget adversarial training improves certified accuracy over the nonzero operating region.Source: §D.6 · p. 15

E. Reproduction

  • Reproduction uses ViT-Tiny with patch 4 at resolution 32, AdamW, cosine scheduling and warmup, with CIFAR trained for 400 epochs and SVHN for 300.Source: E. Reproduction · p. 15
  • Laplace certification uses calibration b=L/u and ε(L′); a load-time guard rejects checkpoints whose Δ2,2 is not 1.Source: E. Reproduction · p. 15
  • For SVHN, adversarial fine-tuning and a continued noise-only control fork from shared 300-epoch weights and train 150 epochs with reset schedules.Source: E. Reproduction · p. 15
  • AutoAttack judges averaged predictions over 8 draws, while reported accuracy averages 20 draws; single-draw judging underestimates the deployed averaged prediction.Source: E. Reproduction · p. 15

Related papers

Loading 2609.31310v1…