Source-linked AI summary
Frequency-aware Discriminative Feature Learning Supervised by Single-Center Loss for Face Forgery Detection
Jiaming Li, Hongtao Xie, Jiahong Li, Zhongyuan Wang, Yongdong Zhang
TL;DR
Face forgery detection needs more discriminative embeddings and more adaptable frequency representations because softmax supervision and fixed handcrafted filters have stated limitations. The paper introduces SCL and AFFGM in an end-to-end framework, reporting state-of-the-art results on three FF++ versions while acknowledging limited generalization to unseen manipulations and ignored inter-frame information.
Problem
Softmax-supervised features lack explicit intra-class compactness and interclass separability, while fixed handcrafted filters inadequately capture diverse frequency forgery patterns.
Method
The framework combines single-center loss, which selectively compresses natural-face variation, with adaptive frequency feature generation for data-driven frequency clues.
Results
State-of-the-art results are achieved on three versions of the FF++ dataset.
Takeaways & Limitations
The framework supports more discriminative feature learning with less optimization difficulty and avoids overly relying on incomprehensive frequency-domain prior knowledge.
Takeaways & Limitations
The framework lacks generalization ability on unseen manipulation methods and ignores inter-frame information valuable for video forgery detection.
Abstract
from arXiv · showhide
Face forgery detection is raising ever-increasing interest in computer vision since facial manipulation technologies cause serious worries. Though recent works have reached sound achievements, there are still unignorable problems: a) learned features supervised by softmax loss are separable but not discriminative enough, since softmax loss does not explicitly encourage intra-class compactness and interclass separability; and b) fixed filter banks and hand-crafted features are insufficient to capture forgery patterns of frequency from diverse inputs. To compensate for such limitations, a novel frequency-aware discriminative feature learning framework is proposed in this paper. Specifically, we design a novel single-center loss (SCL) that only compresses intra-class variations of natural faces while boosting inter-class differences in the embedding space. In such a case, the network can learn more discriminative features with less optimization difficulty. Besides, an adaptive frequency feature generation module is developed to mine frequency clues in a completely data-driven fashion. With the above two modules, the whole framework can learn more discriminative features in an end-to-end manner. Extensive experiments demonstrate the effectiveness and superiority of our framework on three versions of the FF++ dataset.
1. Introduction
Face forgery detection is motivated by the societal risks of increasingly accessible manipulation and the need for efficient automatic analysis. The proposed framework addresses limited feature discrimination and inflexible frequency modeling with SCL and AFFGM.
- Increasingly accessible face manipulation technologies can generate high-quality forged faces and create severe societal trust issues.
- Efficient automatic detection is important because experts cannot review the countless videos uploaded to the Internet each day.
- Softmax-supervised features lack explicit intra-class compactness and inter-class separability, while regular metric learning compresses natural and manipulated faces indiscriminately.
- Fixed filter banks and hand-crafted features inadequately capture subtle frequency-domain forgery patterns across diverse backgrounds, demographics, and manipulation methods.
- SCL compresses natural-face variation while increasing manipulated-face distance from the natural-face center, without restricting manipulated-face compactness.
- AFFGM adaptively mines frequency clues in a data-driven fashion, preserving spatial-frequency block relationships and enabling flexible forgery-clue extraction.
- The framework achieves state-of-the-art results on three versions of the FF++ dataset in extensive experiments.
2. Related work
Prior face forgery detection methods evolved from hand-crafted features and network modifications toward deep models using additional information and prior knowledge. Related metric-learning methods motivate comparisons with the paper’s distinct SCL design.
- Face forgery detection: Early face forgery detection methods used hand-crafted features or modified existing neural-network structures.
- Face forgery detection: Later methods introduced information or prior knowledge, including manipulated-region location, skin-color rhythms, and blending-boundary cues.
- Metric learning: Center loss learns class centers and compresses same-class features, while triplet loss is identified as a closely related metric-learning method.
- Metric learning: Unlike a fixed-center two-branch approach, SCL uses an updatable center and constrains relative rather than absolute sample-to-center distances.
3. Proposed method
The framework combines RGB and adaptively mined frequency features with single-center and softmax supervision. Its preprocessing preserves spatial-frequency correspondence, while SCL clusters natural faces near a center and separates manipulated faces by relative distance.
- Overview: The framework extracts RGB- and frequency-domain features simultaneously, fusing them early with point-wise convolution to reduce parameters and computational expense.A classifier produces predictions after feature embedding, while joint SCL and softmax supervision shapes the embedding space.
- Adaptive frequency features generation module: The preprocessing converts RGB images to YCbCr and applies 2D DCT to each 8 × 8 image block before frequency-feature extraction.The pipeline is compatible with existing convolutional networks and uses JPEG-related transformations.
- Adaptive frequency features generation module: AFFGM adaptively mines frequency clues in a data-driven fashion instead of relying on fixed filter banks or hand-crafted features.This design targets subtle forgery patterns that vary with background, demographic factors, and manipulation methods.
- Single-center loss: Unlike conventional metric losses, SCL avoids indiscriminately compressing manipulated-face variations, whose distributions differ across manipulation methods.The paper attributes these differences to manipulation-specific artifacts such as GAN fingerprints, manipulated regions, and post-processing.
- Single-center loss: SCL reduces natural-face distances to one learnable center while requiring manipulated-face distances to exceed natural-face distances by a margin.Mnat and Mman denote the batch mean Euclidean distances of natural and manipulated representations to the center, respectively.
- Single-center loss: The total objective combines softmax loss with SCL, with λ controlling their trade-off; softmax provides global label guidance while SCL directly structures embeddings.The center is updated from mini-batches, motivating the use of softmax loss to guide its update.
4. Experiments
Experiments on FF++ evaluate SCL, AFFGM, fusion choices, feature distributions, and comparisons with prior methods. The framework and its components consistently improve detection, with strongest reported results from the complete FDFL model and gains across FF++ versions.
- Experiment setup: Experiments use FF++ with three compression versions, focusing mainly on the challenging c40 setting and reporting video-level AUC, pAUC0.1, and accuracy.The study samples frames from manipulated and original videos, detects faces with RetinaFace, and uses t-SNE for additional visualization.
- SCL parameter analysis: SCL remains effective across λ values from 0.001 to 1 when m is fixed at 0.1, indicating robustness to this hyper-parameter.The experiments analyze λ and m on FF++ c40; λ balances softmax and SCL losses, while m controls the relative distance to the center point.
- SCL parameter analysis: AUC 0.916 and pAUC0.1 0.790 are achieved at m = 0.3 and λ = 0.5, while combining SCL with softmax improves AUC by 4% ∼6% over softmax alone.Using only softmax gives the worst performance, with an AUC of 0.861.
- Loss comparison: SCL with softmax performs best among the evaluated losses, while triplet-plus-softmax and center-plus-softmax provide only subtle improvements over softmax alone.The comparison includes triplet loss with softmax and center loss with softmax on FF++ c40.
- Representation visualization: t-SNE shows SCL plus softmax compactly gathers natural-face representations while separating them from less compact manipulated-face representations.Softmax produces neighboring clusters, triplet loss has little distributional effect, and center loss can overfit by compressing manipulated-face features.
- Component and method comparisons: The complete FDFL reaches AUC 0.924 and pAUC0.1 0.810, while SCL and AFFGM separately improve the baseline and the full framework surpasses prior methods across FF++ versions.On c40, the reported improvements over comparison methods are 6.4% in AUC, 15.8% in pAUC0.1, and 2.86% in accuracy.
5. Limitations
The framework achieves strong results on FF++ but has limited generalization to unseen manipulation methods and does not use inter-frame information. The provided figure and table passages identify evaluation context but do not state additional limitations.
- Generalization: The framework lacks generalization ability on unseen manipulation methods.The authors report that supporting results appear in supplementary material and relate this limitation to manipulation-specific discriminative features.
- Evaluation context: Experiments evaluate the framework on FF++ videos under raw, light-compression, and heavy-compression settings.The table caption identifies c0, c23, and c40 as the three compression versions and distinguishes image-based from video-based methods.
- Temporal information: The framework ignores inter-frame information, although frame inconsistency may provide a useful cue for video face forgery detection.The authors note that current manipulation methods generally do not impose temporal constraints.
6. Conclusion
The conclusion presents FDFL as combining single-center metric learning with adaptive frequency-feature learning for face forgery detection. It reports state-of-the-art results on three FF++ versions while identifying future work on temporal information and generalization.
- Conclusion: Single-center loss compresses natural-face intra-class variations while boosting inter-class separability in the embedding space.The authors associate this design with learning more discriminative features with less optimization difficulty.
- Conclusion: The adaptive frequency feature generation module mines subtle frequency-domain artifacts in a data-driven fashion.This approach avoids overly depending on incomprehensive prior knowledge.
- Conclusion: The framework achieves state-of-the-art results on three versions of the FF++ dataset.The conclusion reports this outcome from extensive experiments.
- Future work: Future work will explore inter-frame information, semi-supervised and unsupervised learning, and applying SCL in other fields.The stated examples include improving generalization and face antispoofing.