Deep Convolutional Pooling Transformer for Deepfake Detection

I. INTRODUCTION

• Automatic detection is sought because Deepfake videos are increasingly realistic and numerous, making manual distinction of real and fake videos unrealistic.The introduction reports that video numbers almost doubled within nine months in 2019.Source: §I · p. 1
• Existing methods have difficulty learning global facial relationships, while randomly extracted low-resolution frames restrict available image features.Local artifacts have become harder to discover as synthetic work has improved.Source: §I · pp. 1–2
• The model combines CNN-based local feature extraction with a deep Transformer using convolutional pooling and re-attention to learn local and global features and relations.Convolutional pooling refines dominant features with dimension reduction, while re-attention maintains attention-map diversity in a deep model.Source: §I · p. 2
• The study extracts keyframes for training and testing because they retain complete, high-resolution image information unlike reconstructed normal frames.The authors describe keyframe importance as largely untapped in prior Deepfake detection work.Source: §I · p. 2
• The proposed approach outperforms several baselines in both within- and cross-dataset evaluations.It also visualizes richer learned facial features from keyframes than from normal frames.Source: §I · p. 2

A. Deepfake Generation

• Deepfake face swapping uses a shared encoder for identity-independent features and separate decoders to synthesize the desired identity while retaining the target’s expression.The target face is encoded into a context vector that the corresponding decoder uses to generate the swapped face.Source: A. Deepfake Generation · p. 2

B. Deepfake Detection

• CNN-based detectors focus mainly on local features, and even Xception is described as restricted in learning global features and generalizing.Methods discussed include optical-flow analysis, pre-trained CNNs, mouth-focused detection, and Xception.Source: B. Deepfake Detection · p. 2
• Recent detectors add convolutional attention or paired-stream analysis to CNN backbones, while DCL uses positive and negative paired data to target transferability.CViT and later models also apply Transformer ideas to Deepfake detection.Source: B. Deepfake Detection · pp. 2–3

C. Vision Transformers

• Vision Transformer work includes ViT, shifted-window Swin Transformer, Contextual Transformer, Wave-ViT, and Dual-ViT.The passage describes these approaches as addressing, respectively, downstream vision tasks, hierarchical processing, contextual information, information loss from down-sampling, and self-attention efficiency.Source: C. Vision Transformers · p. 3
• Transformer and self-attention had been barely discussed in Deepfake detection, motivating the authors to present a deep convolutional Transformer.Source: C. Vision Transformers · p. 3

A. Framework

• The model extracts local face features, projects them with depth-wise separable convolution, then analyzes global features with a pooling Transformer and re-attention.Unique learnable transformation matrices are applied to each attention map at each pooling Transformer layer; an MLP reshapes the learned features for detection.Source: A. Framework · p. 3
• The framework discussion introduces image keyframes from compressed video and presents convolutional pooling and re-attention within an enhanced deep Transformer.Source: A. Framework · p. 3
• In the illustrated compressed-video clip, B-frames depend on nearby I- or P-frames, P-frames on preceding I- or P-frames, and I-frames contain complete image information.Source: Figure 3 · p. 4

B. Keyframe Extraction

• Keyframes carry complete image information and have the largest sizes and resolutions in compressed video, while P- and B-frames encode only partial or interpolated information.The section argues that collecting as many keyframes as possible matters because Deepfake cues can be difficult to discover.Source: B. Keyframe Extraction · p. 4

C. Convolutional Local Feature Extraction

• A 3×224×224 face image is converted by the local feature extractor into a 512×7×7 feature map for Transformer-based global feature learning.The extractor uses 17 convolutional layers arranged in five groups, each followed by max pooling.Source: C. Convolutional Local Feature… · p. 4

1) Convolutional Pooling Transformer:

• Self-attention relates every pair of feature-patch positions, with QK entries representing patch relations and multiplication by V applying those relations to image features.The feature map is flattened into w^2 patches, and a learnable CLS token is appended for classification.Source: 1) Convolutional Pooling Transformer · p. 5

D. Deep Transformer with Convolutional Pooling and Re-attention

• Sinusoidal positional encoding maintains meaningful relations among feature patches after self-attention breaks their relative positional layout.The Transformer is organized into three phases separated by convolutional pooling layers that double channels and halve feature-map dimensions.Source: Figure 4 · p. 4, 1) Convolutional Pooling Transformer · p. 5

1) Convolutional Pooling Transformer:

• Depth-wise separable convolutions with batch normalization replace the fully connected Q, K, and V projections, enriching image feature learning while preserving feature dimensions.Depth-wise convolution modifies spatial dimensions while keeping channels unchanged; point-wise convolution changes channel count.Source: 1) Convolutional Pooling Transformer · pp. 5–6

2) Deep Re-attention:

• 24 Transformer blocks are arranged in three phases because deep ViT models can suffer attention collapse, with overly similar attention maps and declining performance.The authors state that multi-head self-attention efficacy tapers as the model deepens.Source: 2) Deep Re-attention · p. 6
• Re-attention multiplies the self-attention map along the head dimension by a learnable transformation matrix Θ, enabling cross-head communication.Each Transformer block uses a unique learnable Θ matrix.Source: 2) Deep Re-attention · p. 6
• The authors report that re-attention improves attention-map diversity, allowing richer features and relations to be learned and boosting Deepfake detection performance.Source: 2) Deep Re-attention · p. 6

• The final Transformer block’s learnable CLS token passes through a fully connected layer and softmax to predict the input face frame’s authenticity.Source: 3) Binary Classification · p. 6
• Cross-entropy loss tunes the model using the ground-truth value and softmax prediction for each class.Source: 3) Binary Classification · p. 6

• The experiments assess model performance on FF++ and several other Deepfake datasets.Source: §IV · p. 6
• Ablation studies examine keyframes, individual model components, and different model depths.Source: §IV · p. 6

A. Implementation Details

• Training and testing use aligned 224×224 face images, with a 24-block Transformer arranged in three phases and Adam optimization.The phases use 8, 8, and 8 blocks with 4, 8, and 16 attention heads, respectively; the learning rate and weight decay are both 1e−4.Source: A. Implementation Details · p. 6

B. Datasets

• FF++ contains 1,000 original videos and four 1,000-video manipulation subsets; experiments use its compressed HQ (c23) version and official 720:140:140 split.Source: B. Datasets · p. 6
• Keyframe faces are extracted and real keyframes augmented, but normal frames are added because keyframes alone are insufficient for training convergence.Source: B. Datasets · pp. 6–7
• The training set uses three times more normal-frame faces from each real video than from each fake video; FF++ testing mixes keyframe and normal faces.Source: B. Datasets · p. 7
• Cross-dataset evaluation uses DFDC, Celeb-DF, and DF-1.0; DFDC evaluation randomly samples 10% of its 128,154 videos.Source: B. Datasets · p. 7
• Celeb-DF evaluation used its official 518-video test set, imbalanced at 178 real and 340 fake; another evaluation tested 1,000 videos with mixed random distortion and perturbation levels, sampling equal numbers of real videos for balance.Source: B. Datasets · p. 7

• Within each same-video pair in FF++, keyframe and normal-frame feature heatmaps show a noticeable feature gap.Source: Figure 6 · p. 8
• In the comparative experiment on FF++, the proposed model outperformed all baseline models, although some comparative approaches achieved competitive performance.Source: C. Performance Evaluation · pp. 8–9
• The proposed model outperforms all state-of-the-art baselines in cross-dataset comparisons on DFDC, Celeb-DF, and DF-1.0.Table II reports frame-level ACC and AUC after training on FF++.Source: C. Performance Evaluation · p. 9
• For unseen test manipulation techniques, the method is the only compared approach exceeding 60% accuracy and 70% AUC on all three cross-dataset benchmarks.Source: C. Performance Evaluation · p. 9
• DF-1.0 performance is described as stable across random distortion and perturbation levels.Source: C. Performance Evaluation · p. 9

D. Keyframe Analyses

• 6.22%, 4.96%, 8.15%, and 10.19% performance boosts from keyframe attendance were reported on FF++, DFDC, Celeb-DF, and DF-1.0, respectively.The comparison holds the model architecture fixed and contrasts training with keyframes plus normal frames against normal frames alone.Source: D. Keyframe Analyses · p. 9
• Keyframe faces generally produced hotter Grad-CAM heatmaps with larger red areas than paired normal frames from the same videos.Hotter areas encode more features extracted by the model; the visualized pairs come from FF++ manipulation techniques.Source: D. Keyframe Analyses · p. 9
• Training with keyframes and normal frames achieved the best performance on keyframe, normal-frame, and combined-frame testing, supporting robust detection across unpredictable frame types.Sole-keyframe training performed unsatisfactorily because the training data were insufficient; augmentation improved performance, while sole-normal-frame training was relatively weak.Source: D. Keyframe Analyses · p. 10

E. Model Component Analysis

• Each added module—pooling, convolutional projection, and re-attention—consistently boosted detection performance in the cumulative ablation.The comparison begins with a vanilla Transformer while keeping local feature extraction fixed.Source: E. Model Component Analysis · pp. 10–11
• Around 3% AUC gains on FF++ were reported for pooling and convolutional projection, while both modules improved DF-1.0 AUC by around 4%.Pooling helped DFDC more than Celeb-DF, whereas convolutional projection was more effective on Celeb-DF than DFDC.Source: E. Model Component Analysis · p. 11
• Re-attention added less than 1% AUC on FF++ but was more instrumental in cross-dataset performance; all three components enhanced DF-1.0 performance, including with artificial noise.Source: E. Model Component Analysis · p. 11

F. Model Depth Analysis

• Depth 24 was optimal: it outperformed depth 14 by 5.8%, 7.51%, and 11.28% AUC on DFDC, Celeb-DF, and DF-1.0, while depth 32 reduced cross-dataset AUC versus depth 24 by 2.38%, 1.55%, and 5.18%, respectively.FF++ AUC increased only around 1% between adjacent depths, and the deeper model did not always perform better.Source: F. Model Depth Analysis · p. 11

V. CONCLUSION

• The approach achieved state-of-the-art performance in both within- and cross-dataset experiments, and feature visualizations showed richer information from keyframes than normal frames.The conclusion links exploring keyframes and decisive local and global features with further performance enhancement in Deepfake detection.Source: §V · p. 11
• The detector may be fooled if opponents raise the authenticity of Deepfakes generated upon high-quality keyframes.Source: §V · p. 11