PaperlayerView exact revision on arXiv

Source-linked AI summary

FairReL: Deepfake Detection using Fairness-Aware Representation Learning

Xiaoman Lu, Jiaqi Li, Shuntian Zheng, Huiping Chen, Yu Guan

arXiv:2608.28777v1cs.CV

TL;DR

Deepfake detectors can show uneven subgroup errors, while existing fairness methods offer limited control over which representation components are suppressed. FairReL targets spatial-frequency imbalance and fine-tuning-induced residual shifts with dedicated objectives, achieving lower cross-domain fairness disparities while preserving competitive detection performance.

  • Problem

    Existing fairness-aware detectors intervene at predictions or mixed representations, limiting control over subgroup-sensitive evidence and potentially leaving representation-level bias under distribution shift.

  • Method

    FairReL uses wavelet-based Group-Conditional Wavelet Decorrelation for spatial features and class-conditional Subspace-Localised Mean Alignment for residual representations in an SVD-decomposed backbone.

  • Results

    FairReL achieves the lowest FFPR and FMEO on three unseen datasets, lowers worst-subgroup FPR across four datasets, and preserves competitive AUC.

  • Takeaways & Limitations

    Targeting spatial frequencies and the trainable residual separately reduces cross-domain fairness disparities without regularising the whole representation.

  • Takeaways & Limitations

    The approach relies on preserving the principal/residual separation so that fine-tuning effects can be isolated.

Abstract

from arXiv · show

Although recent deepfake detectors achieve high overall accuracy, their errors remain unevenly distributed across demographic subgroups, with real faces from certain groups more often misclassified as fake. Existing fairness-aware detectors typically regularise the entire feature representation, without identifying or controlling the specific components that drive unfair predictions. Such coarse intervention can over-suppress useful forgery cues while leaving demographic structure in component-specific subspaces. To address this, we identify two subgroup-sensitive components: multi-scale spatial features, which encode local facial and forgery patterns, and fine-tuning-induced residual features, which adapt the backbone to the unfair training distribution. We propose FairReL, a fairness-aware representation-learning framework that targets both components with dedicated demographic supervision. FairReL uses an SVD-decomposed foundation-model backbone to isolate the fine-tuning-induced residual representation, and introduces two complementary losses. Group-Conditional Wavelet Decorrelation (GCWD) suppresses subgroup-imbalanced structure across spatial wavelet sub-bands, while Subspace-Localised Mean Alignment (SLMA) aligns subgroup means within each real/fake class in the residual representation. Experiments on FF++, Celeb-DF, DFD and DFDC show that, against the state-of-the-art fairness-aware detector, FairReL improves unseen-dataset AUC by 3.9% while reducing subgroup FPR disparity by 10.2%. Code is available at https://github.com/xiaoman89/FairReL .

1 Introduction

Deepfake detectors can achieve strong overall accuracy while producing substantially higher false positives for some demographic groups. FairReL addresses this gap by targeting spatial-frequency bias and fine-tuning-induced residual bias with dedicated representation-level supervision.

  • Motivation: Real Female-Black faces are misclassified as fake over three times as often as real Male-White faces by a vanilla Xception detector.The comparison uses a shared threshold and motivates reducing subgroup error gaps without sacrificing accuracy or unseen-distribution fairness.
  • Motivation: Existing fairness-aware detectors intervene at prediction or whole-representation levels, limiting control over which evidence is suppressed and leaving fairness gains vulnerable to distribution shift.Prediction-level methods can retain subgroup-sensitive shortcuts, while representation-level methods may regularise mixed demographic and forgery information.
  • Sources of bias: Spatial-scale mismatch separates relatively coarse demographic appearance patterns from local forgery cues such as blending boundaries, texture artefacts, and synthesis traces.This motivates decomposing spatial feature maps into frequency bands before global pooling.
  • Sources of bias: Uneven demographic coverage in fine-tuning data can introduce subgroup-specific correlations into task-adapted representations that detectors mistake for fakeness.The resulting adaptation-induced bias is a second target for FairReL.
  • FairReL: FairReL applies GCWD to wavelet sub-bands and SLMA to class-conditional subgroup means in a trainable residual representation.The framework decomposes adapted weights into frozen principal and trainable residual components, targeting spatial-frequency and adaptation-induced bias without regularising the whole embedding.
  • Evaluation: FairReL is evaluated on FF++, Celeb-DF, DFD, and DFDC to assess fairness–generalisation trade-offs across datasets and demographic distributions.The experiments include ablations and representation-level analyses of the two complementary objectives.

2 Related Work

Deepfake detection research has moved from artefact-based detectors toward methods designed to generalise across unseen manipulations and datasets. FairReL extends this setting by treating demographic fairness as a co-equal requirement and localising supervision to distinct representation components.

  • Generalisable detection: Early detectors rely on visual or temporal artefacts but often degrade on unseen datasets and forgery methods.Later work pursues common-forgery features, foundation-model detectors, larger benchmarks, and forgery-semantic decoupling.
  • Generalisable detection: FairReL treats transfer across forgery methods and demographic groups as joint requirements for deepfake detection.This positions fairness alongside, rather than after, generalisation.
  • Fairness in deepfake detection: Prior fairness methods supervise prediction outputs or pooled embeddings as undifferentiated representations, whereas FairReL targets spatial spectral structure and the trainable residual separately.The proposed decomposition preserves the transferable visual prior in the principal subspace while addressing adaptation-induced demographic shifts in the residual.
  • Fairness in deepfake detection: Adversarial removal, null-space projection, and moment matching commonly act on final embeddings, where demographic appearance and forgery evidence may already be mixed after pooling.This motivates fairness supervision before pooling and within a component-specific residual subspace.

3 FairReL

FairReL targets fairness at two representation stages: pre-pooling spatial-frequency features and fine-tuning-induced residual features. Its GCWD component measures subgroup imbalance in wavelet sub-bands and adaptively suppresses the most imbalanced components.

  • Subspace-Decomposed Backbone: FairReL decomposes the backbone into a frozen principal component and a trainable residual component, isolating task-adaptation changes for fairness control.The residual component has rank r, while the principal component retains the dominant pre-trained subspace; only the residual and classifier are trainable.
  • Training Pipeline: During training, the standard path produces spatial features for detection, while a separate residual-only path estimates adaptation-induced representations; inference retains only the standard path.The residual-only path is used for training-time fairness supervision rather than final prediction.
  • Group-Conditional Wavelet Decorrelation: GCWD operates before pooling because spatial features preserve scale information that can distinguish demographic appearance patterns from local forgery evidence.It applies a channel-wise Daubechies-4 DWT to decompose each feature map into frequency-localised sub-bands.
  • Group-Conditional Wavelet Decorrelation: GCWD computes each sub-band’s average energy within demographic subgroups and uses cross-subgroup variation as its fairness signal.A larger Dℓ indicates stronger subgroup imbalance, whereas a smaller Dℓ indicates more consistent use across subgroups.
  • Group-Conditional Wavelet Decorrelation: GCWD adaptively assigns larger weights to sub-bands with stronger subgroup discrepancy and weaker supervision to already consistent bands.A temperature-controlled softmax with stop-gradient produces the weights, preventing the model from changing importance scores instead of reducing discrepancies.

4 Experiments

FairReL is evaluated under cross-domain fairness protocols and consistently improves subgroup fairness while retaining competitive detection performance. Mechanistic analyses and ablations support targeting spatial-frequency structure and fine-tuning residuals rather than regularising the full representation.

  • Experimental Setup: FairReL is trained on FF++ and evaluated on source-domain and unseen Celeb-DF, DFD, and DFDC data using intersectional gender–race subgroups.The evaluation reports frame-level AUC alongside fairness metrics and uses six demographic subgroups.
  • Main Results: FairReL achieves the lowest FFPR and FMEO on all three unseen datasets while preserving competitive AUC.Against fairness-aware baselines, it further lowers cross-domain fairness disparities.
  • Main Results: FairReL achieves higher AUCavg and lower fairness disparities than GenD, despite slightly trailing GenD in overall AUC.Its subgroup performance is more even under domain shift, whereas GenD’s overall-AUC advantage is concentrated in White subgroups.
  • Main Results: At TPR=0.90, FairReL lowers worst-subgroup FPR and narrows the best–worst gap across all four datasets.The improvement is most visible for minority Female and Black subgroups.
  • Mechanistic Analysis: Frequency diagnostics show stronger subgroup discrepancy in coarse bands and stronger real–fake energy gaps in finer bands, motivating sub-band-specific GCWD regularisation.GCWD therefore applies stronger decorrelation where subgroup imbalance is larger rather than regularising the whole feature map.
  • Mechanistic Analysis: Residual-only analysis finds larger pooled subgroup mean shifts than principal-only analysis, especially for gender, while imbalanced fine-tuning spreads residual subgroup centroids.These findings support SLMA’s localisation to the residual representation.
  • Ablation Studies: The full-feature variant is consistently worse than FairReL in AUC and FFPR, supporting the necessity of SVD-based subspace decomposition.Applying fairness supervision to the trainable residual is more effective than applying it to the full pooled feature.
  • Ablation Studies: GCWD and SLMA each improve fairness individually, while their combination gives the best overall balance between AUC and FFPR.Applying SLMA to the residual outperforms full-feature alignment, and removing class conditioning sharply reduces AUC.

5 Conclusion

FairReL targets subgroup bias in spatial features and fine-tuning residuals, improving fairness and cross-dataset generalisation while preserving competitive detection performance.

  • 5 Conclusion: FairReL targets spectral imbalance in spatial features and adaptation-induced shifts in the trainable residual subspace.GCWD reduces subgroup-imbalanced wavelet statistics before pooling, while SLMA aligns class-conditional subgroup means in the residual representation.
  • 5 Conclusion: Experiments on FF++, Celeb-DF, DFD and DFDC show reduced cross-domain fairness disparities with competitive detection performance.These datasets cover standard face-manipulation benchmarks used to assess fairness generalisation.
  • 5 Conclusion: Ablations and representation analyses confirm that GCWD and SLMA are complementary objectives.

A Datasets and Demographic Statistics

The evaluation uses FF++ as the source dataset and Celeb-DF, DFD, and DFDC as unseen target domains, with fairness assessed across six gender–race intersectional groups.

  • Datasets: Four datasets test fairness generalisation across identities, recording conditions, and forgery distributions.FF++ is the source dataset; Celeb-DF, DFD, and DFDC are unseen target domains.
  • Datasets: FF++ contains controlled manipulations, Celeb-DF contains in-the-wild celebrity DeepFakes, DFD contains actor recordings, and DFDC provides diverse actors, scenes, and manipulation methods.
  • Demographic Statistics: The protocol retains six intersectional groups formed by gender categories male and female and race categories Asian, White, and Black.Samples annotated as Others are excluded from the reported demographic coverage.
  • Demographic Statistics: FF++ is split approximately 60/20/20 into training, validation, and test sets, while the other three datasets are used only for unseen target-domain testing.

B Evaluation Metrics

The evaluation reports subgroup fairness through Equal FPR and Max Equalized Odds, alongside group-averaged AUC and AUC gap for subgroup-level detection performance.

  • Fairness Metrics: The fairness metrics are reported in percent, are lower-is-better, and assign 0 to perfect subgroup parity.
  • Fairness Metrics: Equal FPR measures subgroup deviations from the overall false-positive rate, reflecting the deployment concern of flagging real faces as fake.The metric is motivated by the greater number of real faces in deployment and the stated harmful error.
  • Fairness Metrics: Max Equalized Odds is the larger of the worst-case FPR gap and worst-case TPR gap across subgroups.
  • Detection Metrics: AUCavg weights every subgroup equally, while the AUC gap measures the spread between the best- and worst-served subgroups.These metrics complement overall AUC, which is dominated by majority subgroups.

C Implementation Details

Training uses AdamW and a two-level Daubechies-4 wavelet decomposition, with balanced subgroup sampling and source-validation hyperparameter selection.

  • Implementation Details: Training uses AdamW with learning rate 1×10^-3, weight decay 4×10^-3, batch size 64, and 224×224 inputs on one H100 GPU.
  • Implementation Details: GCWD uses a 2-level Daubechies-4 decomposition producing seven wavelet sub-bands.The number of bands is L = 3J+1 = 7 for J=2.
  • Implementation Details: A group-balanced sampler draws approximately 10–12 samples per subgroup per batch, with exponential-moving-average fallback for sparse cells.The fallback is invoked on under 1% of batches.
  • Implementation Details: Hyperparameters are selected on the FF++ validation split using a combined source-domain AUC and FPR-disparity score at TPR=0.90.Target-domain test results are never used for selection; the chosen values are λorth=0.1, λG=λS=0.5, and r=1.

D Intra-Domain Comparison on FF++

On the FF++ test split, FairReL ranks best on both detection and fairness metrics under the fixed-threshold protocol, with stable results across random seeds.

  • FairReL attains the highest AUC and AUCavg and the lowest FFPR and FMEO on the FF++ test split.The comparison uses a threshold pinned on the FF++ validation split at TPR=0.90.
  • Results are averaged over three random seeds, with deviations no larger than 0.4 on AUC metrics and 0.6 on fairness metrics.The small deviations indicate stable rankings across seeds.

F Target-Domain Sensitivity at the Fixed Operating Point

At a threshold fixed from the source-domain operating point, FairReL’s lower target-domain FPR is accompanied by stronger sensitivity and balanced accuracy than the subspace-decomposed baseline.

  • FairReL’s lower FPR does not come from sacrificing target-domain detection performance at the shared threshold.The threshold is pinned on held-out FF++ at source-domain TPR=0.90 and reused across datasets.
  • FairReL’s reported means are based on three seeds, with small variation relative to the gaps to competing methods.
  • Target-domain TPR and balanced accuracy are consistently higher for FairReL than for the subspace-decomposed baseline on all three targets.

G Per-Subgroup AUC on Each Dataset

Across source and unseen datasets, FairReL produces the most uniform subgroup AUC while remaining competitive in overall detection, supported by scale-localised evidence for its spatial intervention.

  • G Per-Subgroup AUC on Each Dataset: Across all four datasets, FairReL achieves the highest AUCavg and smallest AUC gap while remaining detection-competitive in overall AUC.Celeb-DF and DFD omit Asian subgroup annotations, and Celeb-DF’s F-B subgroup has only 30 test samples.
  • G Per-Subgroup AUC on Each Dataset: The validated λorth=0.1 is the best cross-domain entry on every target, while both extremes are visibly suboptimal.The value is selected on the FF++ validation split with GCWD and SLMA active.
  • G Per-Subgroup AUC on Each Dataset: Demographic information is strongest in coarse low-frequency bands, whereas forgery-discriminative information is strongest in fine high-frequency bands.Linear-probe results support separating the spatial representation by scale before decorrelation.
  • G Per-Subgroup AUC on Each Dataset: The residual carries more subgroup-sensitive structure than the frozen principal, with ethnicity dominant in low-frequency discrepancy and gender in pooled mean shift.These measurements are reported on DFDC and normalised by the largest entry in each column.

I.3 Complementarity of GCWD and SLMA

GCWD and SLMA reduce different forms of recoverable demographic information: GCWD is stronger for ethnicity-linked spectral imbalance, while SLMA is stronger for gender-linked mean shifts.

  • GCWD reduces ethnicity-probe accuracy from 63.7 to 54.9, while SLMA reduces gender-probe accuracy from 78.4 to 65.8.The complementary effects mirror the distinct subgroup-sensitive signatures identified in the residual representation.
  • Ethnicity appears as a low-frequency spectral-energy discrepancy, whereas gender appears as a class-conditional first-moment shift of the pooled embedding.
  • FairReL uses demographic labels during training to compute fairness losses, while the deployed detector requires only the input image.The deployment stage does not require demographic classifiers, group-specific thresholds, or attribute-based routing.
Loading 2608.28777v1…