ADD: Frequency Attention and Multi-View based Knowledge Distillation to Detect Low-Quality Compressed Deepfake Images

Introduction

• Detection performance can drop by up to 18% on compressed low-resolution images, where compression removes subtle discriminative artifacts and correlations.These low-quality images are common on social media and mobile platforms with bandwidth or storage constraints.Source: Introduction · pp. 1–2
• ADD uses knowledge distillation from a teacher pretrained on high-quality images to help a student learn distinctive features for detecting low-quality deepfakes.The approach assumes high-quality images are readily available and uses distillation to address the underexplored real-world low-quality detection setting.Source: Introduction · p. 1
• Frequency attention distillation guides the student to recover removed high-frequency components, which contain fine-grained deepfake artifacts lost during compression.The introduction argues that ordinary networks may overfit to high-frequency image-specific characteristics, while early stopping can omit deepfake artifacts under heavy compression.Source: Introduction · pp. 1–2
• Multi-view attention distillation slices teacher and student tensors into multiple views and uses sliced Wasserstein distance to transfer tensor distributions while preserving pixel correlations.It addresses degraded local correlations and contrasts with element-wise discrepancy or single-dimension channel-wise attention.Source: Introduction · p. 2
• The authors report that ADD outperforms well-known baselines, including attention-based distillation methods, on different low-quality compressed deepfake datasets.The stated contributions also include combining multi-view attention distillation with contrastive distillation.Source: Introduction · p. 2

Deepfake detection

• Most prior deepfake detectors target high-quality synthetic images, which are easier to classify using large training sets (Rossler et al. 2019; Li et al. 2020, 2019; Jeon et al. 2020).Li et al. exposed blending boundaries and reported effectiveness on unseen manipulation techniques; Jeon et al. introduced self-training with L2-starting point regularization for newly generated images.Source: Related Work · p. 2

Knowledge distillation

• Earlier distillation work largely targeted conventional datasets, while Zhu et al. used FitNets for low-resolution detection but matched only the teacher’s penultimate-layer distribution (Yim et al. 2017; Tian, Krishnan, and Isola 2019; Huang and Wang 2017; Passalis and Tefas 2018; Zhu et al. 2019; Romero et al. 2014).The paper notes that the penultimate layer does not possess rich lower-layer features.Source: Related Work · pp. 2–3
• Activation-based attention transfer improves on FitNets with spatial attention maps, and ADD extends it with random-dimension attention features using the Radon transform (Zagoruyko and Komodakis 2016; Helgason 2010).The authors say this multi-view approach has greater generalization ability and brings student backbone features closer to the teacher’s.Source: Related Work · p. 3
• CRD draws positive pairs closer and pushes negative pairs apart, but its large embedding-memory buffer is restrictive; this work instead samples positive and negative images within each mini-batch (Oord, Li, and Vinyals 2018; Tian, Krishnan, and Isola 2019; Bromley et al. 1994).The paper applies contrastive loss to the embedded features, similar to a Siamese network.Source: Related Work · p. 3

Frequency domain learning

• Frequency-spectrum discrepancies and GAN up-sampling checkerboard artifacts have served as forgery clues, but prior detectors degrade on compressed images; F 3Net decomposes frequencies but doubles backbone parameters (Jiang et al. 2020; Khayatkhoei and Elgammal 2020; Dzanic, Shah, and Witherden 2019; Frank et al. 2020; Zhang, Karaman, and Chang 2019; Odena, Dumoulin, and Olah 2016).F 3Net combines frequency components with local frequency statistics in a two-stream network.Source: Related Work · p. 3

Wasserstein distance

• Wasserstein distance has been applied to generative models, autoencoder latent distributions, and domain adaptation; this work uses it to guide the student toward the teacher’s tensor distribution despite compressed-input degradation (Villani 2008; Arjovsky, Chintala, and Bottou 2017; Deshpande, Zhang, and Schwing 2018; Kolouri et al. 2018; Xu et al. 2020; Lee et al. 2019).The paper uses Wasserstein’s geometrically meaningful distance property to provide guidance for matching distributions.Source: Related Work · pp. 3–4

Frequency Attention Distillation

• The frequency distiller compares teacher and student feature coefficients after channel-wise Fourier transforms, weighting discrepancies to focus on frequencies removed from the student, especially high frequencies.The exponential attention weighting increases as the student’s removed frequency components increase.Source: Frequency Attention Distillation · p. 4

Multi-view Attention Distillation

• Random-direction sliced Wasserstein matching compares binned projections of normalized teacher and student feature distributions, producing channel-, width-, or height-oriented attention views instead of fixed views.The method also applies CRD-inspired contrastive loss to encourage same-class representation similarity and discourage different-class similarity (Tian, Krishnan, and Isola 2019).Source: Multi-view Attention Distillation · p. 5, Figure 4 · p. 5

Overall Loss Function

• The overall KD objective combines frequency- and multi-view-attention losses with balancing hyperparameters; its parameter-free, architecture-independent attention losses can augment conventional detector losses.The paper states that ADD’s attention losses can be added directly to a detector’s conventional loss, such as cross-entropy.Source: Overall Loss Function · p. 5

Datasets

• Five deepfake datasets are split into 720 training, 140 validation, and 140 test videos each, with 64 face-cropped 128 × 128 frames sampled per video.Each dataset includes c23 and c40 H.264 compression versions.Source: Datasets · p. 6

Experiment Settings

• Experiments use a ResNet50 backbone with Adam, a 2 × 10^-4 learning rate, batches of 144, and validation-based early stopping.The learning rate follows a one-cycle schedule; the method’s listed hyperparameters are held fixed except β, which is tuned per dataset.Source: Experiment Settings · p. 6

Results

• ADD comparisons include Inception ResNet V1 pre-trained on VGGFace2, two-stream XceptionNet F3Net, ResNet50, and ResNet50-backbone FitNet, Attention Transfer, and Non-local distillation methods.Each distillation method is fine-tuned on the validation set to achieve its best performance.Source: Results · p. 6
• 1% to 6%: ADD improves detection accuracy over non-distillation baselines across all five datasets for both c23 and c40; it also outperforms other distillation methods on average and on c40.FitNet, attention, and non-local modules do not surpass ADD; the paper attributes their limitations to insufficient frequency information or lower attention dimensions.Source: Results · p. 6

Sensitivity of attention weights (α and β)

• Across five datasets, changing α and β still leaves ADD outperforming the baselines, indicating low sensitivity to both attention weights.Detailed results are in the supplementary materials.Source: Ablation Study and Discussions · p. 6

Effects of Attention Modules

• On c40 NeuralTextures, combined frequency and multi-view attention with contrastive loss reaches 68.53% accuracy, versus 68.14% with contrastive multi-view alone and 67.01% without it.Frequency attention improves accuracy by about 6.76%; the authors report that the modules make distinct, compatible contributions.Source: Ablation Study and Discussions · p. 6, Table 2 · p. 7

Experiment with other backbones

• ADD improves detection accuracy across all five datasets and compression qualities, by up to 7% with ResNet18, 5.8% with ResNet34, and 7.1% with EfficientNet-B0.The comparison uses ResNet18 and ResNet34 (He et al. 2016) and EfficientNet-B0 (Tan and Le 2019); EfficientNet-B0 uses γFR = 1e−3.Source: Table 3 · p. 7, Ablation Study and Discussions · p. 7

Grad-CAM

• Grad-CAM visualizations show ADD correcting facial-artifact attention and directing students toward facial areas, while the baseline attends to background regions and makes confident wrong predictions (Selvaraju et al. 2017).ADD also teaches the student to neglect background noise in complex scenes, matching the teacher’s facial focus.Source: Ablation Study and Discussions · p. 7

Conclusion

• ADD achieves competitive results in most cases when detecting extremely challenging, highly compressed low-quality deepfakes.The conclusion describes the method as highly effective.Source: Conclusion · p. 7
• Frequency-attention distillation helps the student retrieve and focus on high-frequency components from the teacher.This is one of ADD’s two distillation mechanisms.Source: Conclusion · p. 7
• Inspired by Sliced Wasserstein distance, multi-view attention pushes the student’s output-tensor distribution toward the teacher’s while maintaining correlated pixel features across views.It does so by slicing the teacher’s and student’s tensors into multiple views.Source: Conclusion · p. 7

A. Multi-View Attention Algorithm

• Algorithm 1 formulates multi-view attention distillation between corresponding student and teacher backbone features using Sliced Wasserstein distance (SWD).For simplicity, it describes how a single projection θ_i contributes to the total SWD.Source: A. Multi-View Attention Algorithm · p. 10
• In practice, deep-learning libraries can sample K uniform vectors θ_i in S^(d−1) simultaneously and vectorize projection or binning operations.The passage names TensorFlow and PyTorch as examples.Source: A. Multi-View Attention Algorithm · p. 10

Datasets

• NeuralTextures uses learned feature maps on 3D mesh proxies for video reenactment, with experimental modifications limited to mouth regions (Thies, Zollh¨ofer, and Nießner 2019).Other face regions remain unchanged.Source: B. Datasets · p. 10
• DeepFakes generates swapped faces by decoding a source face’s embedding with the decoder trained on the target face.The dataset description notes that DeepFakes now refers more broadly to AI-generated facial manipulation methods.Source: B. Datasets · p. 10
• Face2Face reenacts a source person’s expression on a target while preserving identity, lighting, and background (Thies et al. 2016).It transfers expression coefficients between faces after recovering the target identity from manually selected key-frames.Source: B. Datasets · p. 10
• FaceSwap fits a 3D model to 68 source-face landmarks, projects facial regions onto the target, and applies color correction (Accessed 2021-Jan-01c).Projection minimizes pair-wise landmark errors.Source: B. Datasets · p. 10
• FaceShifter uses an encoder-based feature extractor and Adaptive Attentional Denormalization to blend identity and features, then refines facial occlusions (Li et al. 2019).Its second stage is a Heuristic Error Acknowledging Refinement Network.Source: B. Datasets · p. 10

• FitNet (Romero et al. 2014) trains the student’s penultimate layer to predict the teacher’s penultimate-layer hints.Source: C. Distillation Baseline Methods · p. 10
• Attention Transfer (Zagoruyko and Komodakis 2016) transfers attention maps formed by summing spatial values across backbone feature channels.Source: C. Distillation Baseline Methods · p. 10
• Non-local (Wang et al. 2018) has the student mimic the teacher’s self-attention tensors generated from their backbone features.Source: C. Distillation Baseline Methods · p. 10

D. Evaluation Metrics

• Accuracy (ACC) measures the proportion of test samples whose true class is predicted with the highest probability.Source: D. Evaluation Metrics · p. 10
• Recall at k (R@k) measures the proportion of test samples with at least one same-class sample among their k nearest neighbors in feature space.The passage states that small R@k implies small intra-class variation, which usually leads to better accuracy.Source: D. Evaluation Metrics · p. 10
• The evaluation adopts R@1, using Euclidean distance between penultimate-layer features to find each test sample’s first nearest neighbor.Source: D. Evaluation Metrics · p. 10
• The sliced Wasserstein distance (SWD) procedure takes student features from low-quality compressed images and teacher features from raw images, along with projection count K and bin count g.Source: D. Evaluation Metrics · p. 11
• For each of K iterations, the SWD procedure samples a direction uniformly from the unit sphere in R3 and projects the normalized student and teacher features onto it.Source: D. Evaluation Metrics · p. 11
• The procedure applies a one-dimensional transportation cost with quadratic loss to the projected features and returns SWD.Source: D. Evaluation Metrics · p. 11

E. Hyperparameters’ Sensitivity

• ADD is less sensitive to α and β and almost always outperforms ResNet50 baselines across datasets; the study fixes α = 1 and fine-tunes β.The authors note that α and β can be further tuned to optimize performance.Source: E. Hyperparameters’ Sensitivity · p. 11

Gallery setup

• The gallery compares a raw-data ResNet50 teacher, a c40-trained ADD-ResNet50 student, and a c40-only ResNet50 baseline using Grad-CAM localization heat maps.Red indicates stronger activation; blue indicates weaker activation.Source: F. Gallery of Grad-CAM for the… · p. 11

Facial activation regions

• On deepfake examples from five datasets, the c40-only baseline’s facial activations differ from the teacher’s, while the ADD student’s activations closely follow the teacher’s and support correct predictions.In the figure, red arrows mark baseline activation regions and green arrows mark facial activation regions.Source: F. Gallery of Grad-CAM for the… · p. 11, Figure 6 · p. 13

Background confusion

• With complex backgrounds, the c40-only baseline activates background regions and makes high-confidence errors, while the ADD student correctly emphasizes facial areas like its teacher.The authors suggest exploring adversarial attacks on compressed images and KD attention as a possible defense against complex-background noise.Source: F. Gallery of Grad-CAM for the… · p. 11, Figure 7 · p. 14