PaperlayerView exact revision on arXiv

Source-linked AI summary

UIA-ViT: Unsupervised Inconsistency-Aware Method based on Vision Transformer for Face Forgery Detection

Wanyi Zhuang, Qi Chu, Zhentao Tan, Qiankun Liu, Haojie Yuan, Changtao Miao, Zixiang Luo, Nenghai Yu

arXiv:2210.12752v1cs.CV

TL;DR

UIA-ViT addresses the difficulty of learning intra-frame inconsistency without reliable pixel-level forged-region labels. It uses Vision Transformer attention with unsupervised patch consistency learning and progressive feature assembly; experiments demonstrate the method’s generalization ability and the effectiveness of unsupervised learning.

  • Problem

    Learning intra-frame inconsistency for forgery detection requires pixel-level forged-region annotations that are difficult to obtain reliably.

    Source: §1 · p. 2, §2.1 · p. 4

  • Method

    UIA-ViT uses Vision Transformer attention, MVG-based pseudo-location maps progressively optimized by UPCL, and PCWA to combine patch and classification embeddings.

    Source: §1 · pp. 2–3

  • Results

    Experiments demonstrate UIA-ViT’s superior generalization ability and the effectiveness of its unsupervised learning strategy.

    Source: §1 · p. 3

  • Takeaways & Limitations

    The paper concludes that UIA-ViT is effective for general face forgery detection.

    Source: §5 · p. 14

Abstract

from arXiv · show

Intra-frame inconsistency has been proved to be effective for the generalization of face forgery detection. However, learning to focus on these inconsistency requires extra pixel-level forged location annotations. Acquiring such annotations is non-trivial. Some existing methods generate large-scale synthesized data with location annotations, which is only composed of real images and cannot capture the properties of forgery regions. Others generate forgery location labels by subtracting paired real and fake images, yet such paired data is difficult to collected and the generated label is usually discontinuous. To overcome these limitations, we propose a novel Unsupervised Inconsistency-Aware method based on Vision Transformer, called UIA-ViT, which only makes use of video-level labels and can learn inconsistency-aware feature without pixel-level annotations. Due to the self-attention mechanism, the attention map among patch embeddings naturally represents the consistency relation, making the vision Transformer suitable for the consistency representation learning. Based on vision Transformer, we propose two key components: Unsupervised Patch Consistency Learning (UPCL) and Progressive Consistency Weighted Assemble (PCWA). UPCL is designed for learning the consistency-related representation with progressive optimized pseudo annotations. PCWA enhances the final classification embedding with previous patch embeddings optimized by UPCL to further improve the detection performance. Extensive experiments demonstrate the effectiveness of the proposed method.

1 Introduction

Face forgery detection must generalize beyond training datasets, but learning intra-frame inconsistency has relied on difficult-to-obtain pixel-level forged-region labels. UIA-ViT uses Vision Transformer attention, unsupervised patch consistency learning, and progressive feature assembly to learn from video-level labels.

1 Introduction

  • Face forgery detection matters because forged facial images can deceive people and be abused in ways that threaten security and privacy.The paper gives fake news and evidence as examples.Source: §1 · p. 1
  • Pixel-level supervision for intra-frame inconsistency is difficult to obtain, while existing synthesized-image and paired-image label approaches have important limitations.Synthesized data may not capture forgery-region properties, and paired real/fake images are often unavailable; resulting difference labels can be discontinuous and inaccurate.Source: §1 · p. 2
  • UIA-ViT uses Vision Transformer attention maps among patch embeddings as consistency representations and introduces UPCL and PCWA.The method uses multivariate Gaussian estimation to represent real/fake features and generate pseudo annotations.Source: §1 · p. 2
  • UPCL estimates forgery locations by comparing middle-layer patch embeddings with real/fake feature distributions, then progressively optimizes pseudo location maps.It constrains multi-layer attention maps to model consistency.Source: §1 · pp. 2–3
  • PCWA progressively weights final-layer patch embeddings using attention to the classification embedding, then concatenates them with that embedding for classification.The attention-producing layers are optimized by UPCL.Source: §1 · p. 3
  • The method combines global classification features with attention-weighted local patch features, and experiments demonstrate its generalization ability and the effectiveness of unsupervised learning.The approach is presented as improving detection generalization without pixel-level annotations or additional overhead.Source: §1 · p. 3

2 Related Work

Prior face-forgery research explores visual artifacts, frequency cues, and manipulation-related inconsistencies, but methods can struggle to generalize to unseen datasets. The paper motivates unsupervised inconsistency learning and situates its use of Vision Transformers among earlier transformer-based approaches.

2.1 Face Forgery Detection

  • Early detection work used visual cues such as blinking, head-pose inconsistency, and facial features, while frequency-based and attention methods also reported intra-dataset performance but struggled on unseen datasets.Examples include eye blinking, 3D head poses [32], hand-crafted facial features, frequency methods [16, 22, 11], and attention methods [3, 33, 19].Source: §2.1 · p. 3
  • Other work seeks universal forgery cues in warping, blending, temporal inconsistency, and intra-frame inconsistency.Pixel-level label generation can be time-consuming, miss forgery-region properties, or rely on unavailable pairs and yield discontinuous, inaccurate labels; the paper proposes unsupervised learning to address these issues.Source: §2.1 · pp. 3–4

2.2 Transformer

  • Prior face-forgery transformers model local features or temporal information, while this paper uses a Vision Transformer to model relationships among image regions for local consistency.The cited examples include Miao et al., Khan et al., and Zheng et al..Source: §2.2 · p. 4

3 Method

UIA-ViT estimates forgery locations from layer-K patch features and trains ViT attention maps as consistency representations, then uses consistency-weighted patch features alongside the classification embedding.

3.1 Unsupervised Patch Consistency Learning

  • Layer-K patch features are compared with real and forged multivariate Gaussian distributions to estimate pseudo forgery-location maps.The forged distribution is fitted using features from the center-square General Forgery Region; real samples use all-zero maps.Source: Unsupervised Approximate Forgery… · pp. 4–6
  • UPCL supervises ViT’s existing multi-layer patch Attention Map with consistency targets derived from the estimated location maps.Learnable consistency factors are initialized to limit instability when early MVG estimates are biased.Source: Patch Consistency Loss · p. 6

3.2 Progressive Consistency Weighted Assemble

  • PCWA progressively weights final-layer patch embeddings with classification-to-patch attention, then concatenates their weighted average with the classification embedding for prediction.The weighting transitions from uniform to attention-based consistency weighting during training.Source: §3.2 · p. 7

3.3 Loss Functions

  • The total objective combines classification and UPCL losses with terms that encourage c1 and c2 to increase and c3 to decrease during training.The hyperparameters balance classification and consistency losses and adjust the consistency factors.Source: §3.3 · p. 8

4 Experiment

Experiments assess cross-dataset and cross-manipulation generalization, then visualize predicted maps and examine component, layer, and GFR choices.

4.1 Experimental Setting

  • Experiments use four forgery datasets and a ViT-Base backbone, with center-face GFR patches and 110 frames per test video.The datasets are FaceForensics++, Celeb-DF, DFD, and DFDC preview.Source: Datasets · p. 8, Implement Details · p. 8

4.2 Quantitative Results

  • UIA-ViT achieves the best results on unseen datasets except DFDC-P and exceeds other methods by 3.0+% on DFD.The cross-dataset evaluation trains on FF++(HQ) and tests on DFD, Celeb-DF-v2, Celeb-DF-v1, and DFDC-P.Source: Table 1 · p. 9, Cross Dataset Experiment · p. 9
  • UIA-ViT reports 9.7+% and 6.7+% higher AUC than other methods, with especially strong results on Face2Face and FaceSwap.The cross-manipulation evaluation holds out one manipulation type while training on the other three.Source: Table 2 · p. 9, Cross Manipulation Experiment · p. 10

4.3 Visualization

  • Attention visualizations show similar patch embeddings within forged regions and original backgrounds, but lower similarity across those regions.Attention in later layers is closer to the consistency constraint; classification-to-patch attention also contains consistency cues.Source: Figure 2 · p. 10, §4.3 · pp. 11–12

4.4 Ablation Study

  • UPCL and PCWA together improve Celeb-DF-v1 performance by 11% over baseline, and the ablation describes the components as complementary.UPCL's MVG estimation amends the prior GFR, soft consistency factors stabilize early training, and PCWA outperforms CWA through progressive weighting.Source: Table 3 · p. 12, §4.4 · p. 13

4.3 Visualization

  • Predicted location maps concentrate on manipulated regions across training and unseen datasets, including central facial areas in face-swapping data.For NeuralTextures, manipulated regions are mainly on the lower face, especially the mouth and nose.Source: Figure 4 · p. 11, §4.3 · p. 12

4.5 Analysis

5 Conclusions

UIA-ViT combines UPCL and PCWA in a Vision Transformer-based face forgery detector. The authors report that it learns consistency-related representations and is effective for general face forgery detection.

5 Conclusions

  • UIA-ViT is a Vision Transformer-based face forgery detector equipped with UPCL and PCWA.Source: §5 · p. 14
  • UPCL learns consistency-related representations through unsupervised forgery location, while PCWA enhances features using those representations.Source: §5 · p. 14
  • Visualizations show the method’s capability to learn consistency-related representations, and experiments support its effectiveness for general face forgery detection.Source: §5 · p. 14
Loading 2210.12752v1…