Source-linked AI summary
Camera-Noise Residuals for Face-Swap Detection: Redundant, Not Complementary, and Why
Danil Davydov, Bader Rasheed, Dmitriy Vatolin
TL;DR
The paper asks whether Noiseprint++ residuals complement RGB for face-swap detection and tests this on FaceForensics++ with model ablations and a bottleneck diagnostic. Fusion does not improve over RGB; the residual signal is statistical, is removed by input InstanceNorm, and remains redundant with RGB when preserved.
Problem
Whether a Noiseprint++ residual is complementary to an RGB backbone for face-swap detection, rather than redundant or lost through architectural choices, had not been settled by a controlled ablation.
Source: §I · p. 1Method
The authors compare RGB-only, residual-only, and late-fusion models on FF++, then use a seven-level diagnostic and fixed-fusion variants to probe signal loss and redundancy.
Source: §I · p. 1, §VI · p. 4Results
RGB-only and Late-Fusion are statistically indistinguishable on FF++ (Wilcoxon p = 0.977); six scalar noise statistics reach AUC 0.747 before InstanceNorm pooling and 0.554 after it.
Source: A. Ablation and cross-dataset transfer · p. 2, B. The bottleneck is per-sample… · p. 2Takeaways & Limitations
For face-swap detection in this setting, preserving the residual's statistical signal does not make it complementary to RGB, so noise-residual fusion should not be assumed to help.
Source: §V · p. 4The conclusion is limited to the FF++ face-swap distribution and the Noiseprint++/TruFor template; it does not claim sensor-noise residuals are useless for forgery detection generally.
Source: §V · p. 4
Abstract
from arXiv · showhide
Fusing a learned camera-noise fingerprint with an RGB appearance backbone is an appealing route to generator-independent deepfake detection, because the noise residual is grounded in image-formation physics rather than in the texture statistics of a particular generator. We test, on FaceForensics++, whether a Noiseprint++ residual channel carries information \emph{complementary} to an RGB Xception backbone for face-swap detection. A three-model ablation (RGB-only, residual-only, late-fusion) shows that fusion does not improve over RGB alone and that the residual branch alone is near chance. A seven-level bottleneck diagnostic localizes the cause: the noise maps do carry a discriminative signal, but it is statistical---carried by the per-sample first and second moments (mean, variance, energy) of the residual---and the per-sample \texttt{InstanceNorm} layer placed at the noise-branch input, following the TruFor template, standardizes exactly those moments away (five-fold cross-validated AUC drops from $0.747$ to $0.554$). A context-crop control rules out cropping geometry, and two fixed-fusion variants that remove the bottleneck recover the statistical signal yet still fail to beat RGB on every dataset. We conclude that, on this manipulation distribution, the noise residual is redundant with RGB rather than complementary, and we give concrete guidance for practitioners adopting noise-residual fusion for face-swap detection.
I. INTRODUCTION
The paper asks whether a Noiseprint++ camera-noise residual supplies complementary information to RGB for face-swap detection, rather than redundant information or a signal lost to architectural choices. On FaceForensics++, the ablations find no fusion improvement over RGB and trace the residual signal to statistics removed by input InstanceNorm.
I. INTRODUCTION
- The study uses sensor-noise residuals as a candidate image-formation invariant because compositing disrupts camera-model fingerprints.Noiseprint extracts a camera-model fingerprint disentangled from scene content; TruFor uses Noiseprint++ in a learned forgery detector.Source: §I · p. 1
- The controlled question is whether Noiseprint++ residuals are complementary to RGB—fusion exceeding the better channel—or redundant, with no such improvement.The paper evaluates this question on FaceForensics++ [1].Source: §I · p. 1
- RGB-only and Late-Fusion are statistically indistinguishable on FF++ (Wilcoxon p = 0.977), while Residual-Only is near chance.The seven-level diagnostic finds AUC 0.747 from six scalar noise statistics, but 0.554 after InstanceNorm pooling.Source: §I · p. 1
- A context-crop control and two fixed-fusion variants rule out cropping geometry and the normalization bottleneck as explanations for fusion failure, supporting redundancy with RGB.The bottleneck diagnostic localizes the signal loss to input InstanceNorm: six scalar statistics reach AUC 0.747 before pooling and 0.554 after it.Source: §I · p. 1
II. RELATED WORK
Prior work addresses deepfake generalization with self-blended training examples and image-forgery detection with camera-model fingerprints. This study tests the latter architectural approach specifically for face-swap detection.
II. RELATED WORK
- The Self-Blended Images framework, building on Face X-Ray, generates training fakes by self-blending a source image and is used here as an off-the-shelf reference baseline.The passage attributes markedly better cross-dataset transfer to SBI and says it removes dependence on labelled fakes.Source: §II · pp. 1–2
- Noiseprint learns camera-model fingerprints, while TruFor combines Noiseprint++ with an RGB stream through a learned fusion head and places InstanceNorm at the noise-branch input.The paper adopts this architecture as a template for a face-swap-specific ablation.Source: §II · p. 2
III. METHOD
Experiments use video-level partitions of face crops from FF++ and its DeepFakeDetection extension, with Noiseprint++ maps extracted on full frames before cropping. The protocol uses class-weighted cross-entropy for class imbalance.
III. METHOD
- The data comprise 19,571 training, 3,573 validation, and 4,675 test face crops, partitioned by video so source-video frames do not cross splits.Faces are detected with MTCNN, and the boxes are expanded by 1.3× and square-padded.Source: A. Data and noise extraction · p. 2
- Noiseprint++ maps are precomputed on full frames at native resolution before face cropping to preserve camera-model fingerprint context.The passage says extraction on small crops degrades receptive-field context.Source: A. Data and noise extraction · p. 2
- Class imbalance is handled with class-weighted cross-entropy, and the full protocol and training configuration are tabulated in Appendix Table IV.Source: A. Data and noise extraction · p. 2
B. Three ablation models
The section defines RGB-only, residual-only, and late-fusion models, then probes noise maps at seven levels to test whether scalar statistics or spatial patterns carry signal and where normalization affects it.
B. Three ablation models
- RGB-only uses an ImageNet-pretrained Xception on face crops, while Residual-only uses a ResNet on Noiseprint++ maps with input InstanceNorm.Late-Fusion concatenates parallel RGB and noise branches into a joint head to test whether noise adds complementary information.Source: B. Three ablation models • · p. 2
C. Seven-level bottleneck diagnostic
- The seven-level diagnostic tests scalar statistics, raw spatial patterns, and whether InstanceNorm pooling or pre-convolution standardization changes the readout.L1–L3 test whether scalar statistics suffice; L4 tests standardize-then-pool, L5–L6 test raw spatial patterns, and L7 tests a learned convolution after standardization.Source: C. Seven-level bottleneck diagnostic · p. 2
B. Three ablation models
- Table I reports RGB-Only approximately equal to Late-Fusion (Wilcoxon p = 0.977), with Residual-Only near chance across in-distribution and cross-dataset AUC results.The caption specifies video-level AUC, except frame-level AUC for DFF; SBI is an off-the-shelf reference.Source: Table I · p. 2
D. Context-crop control
The section tests whether wider crops capture blending-boundary cues and describes two fixed-fusion variants that bypass the diagnosed normalization bottleneck.
D. Context-crop control
- The crop control compares 1.3× and 2.7× face-box crops on FF++ and Celeb-DF using noise probes and a classifier-free corpus SNR.The wider crop is intended to capture the blending boundary where PRNU discontinuities may be visible.Source: D. Context-crop control · p. 2
E. Fixed-fusion variants
- StatNoise-Fusion bypasses spatial processing and combines thirteen scalar noise statistics with Xception features through an MLP.This variant directly removes the diagnosed bottleneck by fusing scalar noise statistics.Source: E. Fixed-fusion variants · p. 2
- ResAware-Fusion replaces noise-branch InstanceNorm with BatchNorm and processes the noise crop at native resolution.The design aims to preserve per-sample statistics through spatial processing.Source: E. Fixed-fusion variants · p. 2
A. Ablation and cross-dataset transfer
On FF++, architectural fusion does not improve over RGB alone, while a bottleneck diagnostic finds that InstanceNorm removes the residual’s statistical signal.
A. Ablation and cross-dataset transfer
- RGB-only and Late-Fusion are statistically indistinguishable on FF++ (Wilcoxon paired test, p = 0.977), while Residual-Only stays near chance across datasets.The comparison covers in-distribution and three further corpora; RGB-only and Late-Fusion are indistinguishable at both frame and video level.Source: A. Ablation and cross-dataset transfer · p. 2
B. The bottleneck is per-sample InstanceNorm
- AUC is 0.747 for the six-statistic MLP (L3) and 0.554 after InstanceNorm pooling (L4).Table II reports five-fold cross-validated AUC on 3,000 FF++ test samples.Source: B. The bottleneck is per-sample… · p. 2, Table II · p. 3
- The L3→L4 drop localizes the loss to standardize-then-pool, while L7 recovers part of the statistical ceiling.Fig. 1 shows seven-level AUC with five-fold standard-deviation bars.Source: Figure 1 · p. 3, B. The bottleneck is per-sample… · p. 3
- InstanceNorm subtracts each sample’s channel mean and divides by its standard deviation, removing per-sample first- and second-moment information.Mean, variance/energy, and scale-dependent descriptors such as max |x| are among the discriminative statistics; standardized higher-order moments can survive in principle.Source: B. The bottleneck is per-sample… · p. 3
C. Cropping geometry is not the cause
The wider context crop weakens the measured noise signal rather than improving it, and removing InstanceNorm still does not let fixed-fusion variants beat RGB-only.
C. Cropping geometry is not the cause
- Removing the InstanceNorm bottleneck recovers statistical signal but still underperforms RGB-only on both datasets.Table III summarizes fixed-fusion variants as mean ± standard deviation over five seeds.Source: Table III · p. 3
- FF++: corpus SNR is 0.188 (tight) versus 0.080 (context), and best-classifier AUC is 0.670 versus 0.652.On Celeb-DF, SNR is 0.158 versus 0.053 and AUC is 0.588 versus 0.580.Source: C. Cropping geometry is not the cause · p. 3
- Expanding the crop dilutes face-region signal with background pixels identical between real and fake; the expected PRNU boundary discontinuity is not stronger than the face-region signal.The tight crop outperforms the context crop at every probe level on both datasets.Source: C. Cropping geometry is not the cause · p. 3
D. Fixing the bottleneck does not recover complementarity
Fixed-fusion variants recover a noise contribution but do not outperform RGB-only, and perturbation tests likewise find no robustness gain from fusion.
D. Fixing the bottleneck does not recover complementarity
- Both fixed-fusion variants recover a non-trivial noise contribution yet underperform RGB-only and original Late-Fusion on every dataset.The stronger but redundant noise signal pulls fixed fusion down, unlike the original Late-Fusion model, which learned to ignore the near-chance noise gradient.Source: D. Fixing the bottleneck does not… · p. 3
E. Robustness corroborates redundancy
- Across JPEG, blur, resize, and gamma perturbations, mean ΔAUC is −0.003 (max |Δ| = 0.035) and is never significantly positive.The only point whose confidence interval excludes zero is Gaussian blur σ = 2: −0.017.Source: E. Robustness corroborates redundancy · p. 3
- The SBI reference has positive paired differences at every operating point (means +0.04 to +0.17), while the noise-fusion head learned to ignore the near-chance noise gradient.This contrast accompanies the noise-fusion difference remaining statistically indistinguishable from zero.Source: E. Robustness corroborates redundancy · pp. 3–4
V. DISCUSSION
On FaceForensics++, the residual carries a statistical signal that input InstanceNorm erases, but preserving it still does not make fusion outperform RGB. The conclusion is specific to the tested face-swap distribution and Noiseprint++/TruFor template.
V. DISCUSSION
- The residual’s face-swap signal lies in per-sample first and second moments, which input InstanceNorm standardizes away; tight crops are uniformly better.The bottleneck is exposed by a classifier-free probe, while loss curves do not reveal the collapse.Source: §V · p. 4
- Do not assume a noise-residual side channel is complementary for face-swap detection; input normalization can erase its signal, and preserving it may add nothing over RGB.The SBI reference transfers across datasets better when the physical prior enters through training data rather than an architectural side channel.Source: §V · p. 4
- The conclusion is limited to the FF++ face-swap distribution and the Noiseprint++/TruFor template, not sensor-noise residuals in forgery detection generally.The authors state that these residuals are redundant, not complementary, for face-swap detection in this setting.Source: §V · p. 4
VI. CONCLUSION
- On FF++, probe AUC drops from 0.747 to 0.554 after input InstanceNorm, and fixed-fusion variants still do not exceed RGB once the residual signal is preserved.The original late-fusion model also tracks RGB across perturbations; the context-crop control rules out cropping geometry as the cause.Source: §VI · p. 4, §V · p. 4
APPENDIX A DATA PROTOCOL AND TRAINING DETAILS
Table IV consolidates the data protocol and training configuration used for all three ablation models.
APPENDIX A DATA PROTOCOL AND TRAINING DETAILS
- Table IV consolidates the dataset partitioning, preprocessing, and training configuration used for the three ablation models to support reproducibility.Source: Appendix A · p. 4