FairReL: Deepfake Detection using Fairness-Aware Representation Learning

Introduction

• A vanilla Xception detector misclassifies real Female-Black faces as fake over three times as often as real Male-White faces, illustrating subgroup-level error disparity.Figure 1 reports false-positive rates across six demographic subgroups at a shared threshold; Female-Black has the highest rate.Source: §1 · p. 2, Figure 1 · p. 2
• Prediction-level constraints can leave subgroup shortcuts encoded, while whole-feature regularisation can suppress useful forgery evidence along with harmful demographic cues.The introduction identifies brittle fairness under distribution shift and a potential fairness–detection-performance trade-off as limitations of these approaches.Source: §1 · p. 2
• FairReL addresses spatial-scale mismatch between coarse demographic appearance and local forgery cues, plus demographic correlations absorbed during fine-tuning on unevenly covered data.These are identified as spatial-frequency bias and adaptation-induced bias, respectively.Source: §1 · pp. 2–3
• FairReL improves average AUC on unseen datasets by 3.9 percentage points and reduces subgroup FPR disparity by 10.2% versus existing fairness-aware detectors.Experiments cover FF++, Celeb-DF, DFD, and DFDC.Source: §1 · p. 3
• GCWD penalises subgroup-dependent wavelet-band imbalance, while SLMA aligns subgroup means within each real/fake class in the trainable residual representation.FairReL decomposes adapted weights into a frozen principal component and trainable residual; demographic labels are needed only during training, with no inference-time overhead.Source: §1 · p. 3

2.1 Generalisable Deepfake Detection

• FairReL treats demographic fairness as co-equal with transfer across forgery methods, extending generalisable detection beyond artifact-based cues that often degrade on unseen datasets and forgery methods.Earlier detectors use visual or temporal artifacts [23] [24] [47], while newer approaches pursue common-forgery features, foundation models, larger benchmarks, and forgery semantics [32] [44] [46] [48].Source: §2.1 · pp. 3–4

2.2 Fairness in Deepfake Detection

• Prior fairness methods target prediction outputs or pooled embeddings, whereas FairReL intervenes on multi-scale spatial spectral structure and the trainable residual while preserving the principal-subspace visual prior.Ju et al. [19] use prediction-layer fairness losses, Lin et al. [26] find source-domain fairness may not transfer, and Cheng et al. [3] use same-label feature alignment.Source: §2.2 · p. 4

2.3 Fairness-Aware Representation Learning

• Unlike embedding-level fair representation methods [11] [28] [37] [38] [18] and Effort [46], FairReL aligns class-conditioned residual subgroup means and reduces subgroup imbalance across spatial wavelet bands.Effort separates a frozen principal subspace from a trainable residual using SVD but targets generalisation rather than fairness; deepfake fine-tuning can inherit demographic disparities [19] [42].Source: §2.3 · p. 4

3.1 Problem Definition

• FairReL applies GCWD to pre-pooling spatial features and SLMA to fine-tuning residual features, targeting subgroup imbalance without regularising the entire pooled representation.The standard path supplies the spatial feature map and prediction representation; only the pooled prediction feature reaches the classifier.Source: §3.1 · p. 5, Figure 2 · p. 5

3.2 Subspace-Decomposed Backbone

• Following [46], FairReL freezes the dominant SVD subspace and trains a rank-1 residual; an orthogonality penalty preserves the split during fine-tuning.For ViT-L/14 attention projections, the stated ranks are 1023 frozen and 1 trainable; a separate residual-only forward path estimates adaptation-induced representation changes during training.Source: §3.2 · p. 6

3.3 Group-Conditional Wavelet Decorrelation (GCWD)

• GCWD applies channel-wise Daubechies-4 wavelets to pre-pooling features, then uses stop-gradient adaptive weights to reduce cross-subgroup energy gaps most strongly in imbalanced bands.This frequency-localised supervision avoids forcing the entire feature map to be group-invariant, while applying weaker supervision to bands already consistent across subgroups.Source: §3.3 · pp. 6–8

3.4 Subspace-Localised Mean Alignment (SLMA)

• SLMA aligns subgroup means within each real/fake class in residual representations, limiting fine-tuning shifts without aligning real and fake samples or constraining the full pooled feature [14], [3] [11] [28].The method targets subgroup-dependent residual shifts that could become prediction shortcuts while leaving the frozen pre-trained visual prior unconstrained.Source: §3.4 · p. 8

3.5 Training Objective and Inference

• Training combines binary cross-entropy with orthogonality, GCWD, and SLMA losses, updates only residual weights and the classifier, and removes fairness components at inference for no overhead.Inference retains the standard backbone and classifier while discarding the residual-only path, wavelet transform, fairness losses, and demographic labels.Source: §3.5 · pp. 8–9

4.1 Experimental Setup

• The cross-dataset protocol trains on FF++ and tests on Celeb-DF, DFD, and DFDC; evaluation reports overall and group-averaged AUC plus FPR-based fairness metrics.Fairness is measured across six intersectional gender–race subgroups; FPR metrics use a threshold fixed on FF++ validation at source-domain TPR 0.90, and demographic labels are not required at inference.Source: §4.1 · p. 9

4.2 Main Results

• FairReL achieves the lowest FFPR and FMEO on Celeb-DF, DFD, and DFDC while preserving competitive AUC.The evaluation trains models on FF++ and tests them on these three unseen datasets.Source: Table 1 · p. 10, §4.2 · p. 10
• FairReL slightly trails GenD [49] in overall AUC but has higher AUCavg and lower disparities; on DFDC it has the highest AUCavg and smallest AUC gap.Figure 4 shows lower worst-subgroup FPR and narrower best–worst gaps across all four datasets, with the clearest gains on minority Female and Black subgroups.Source: §4.2 · pp. 10–11, Table 2 · p. 10, Figure 4 · p. 11

4.3 Mechanistic Analysis of FairReL

• FairReL mainly reduces low-frequency subgroup discrepancy while largely preserving higher-frequency bands with stronger real–fake separation.Subgroup variation peaks in coarse low-frequency bands, while the real–fake energy gap is stronger in finer bands; GCWD applies stronger decorrelation where imbalance is greater.Source: §4.3 · p. 11, Figure 5 · p. 12
• Pooled subgroup mean shifts are larger in the residual than principal representation, and imbalanced fine-tuning spreads residual subgroup centroids further apart.This motivates SLMA’s alignment of subgroup means within each real/fake class in the residual representation.Source: §4.3 · pp. 12–13, Figure 6 · p. 12

4.4 Ablation Studies

• Without SVD decomposition, fully fine-tuned CLIP with GCWD and full-feature SLMA is consistently worse than FairReL in AUC and FFPR.The joint ablation finds that GCWD and SLMA each improve fairness alone, their combination gives the best AUC–FFPR balance, and removing class conditioning sharply reduces AUC.Source: Table 3 · p. 13, §4.4 · pp. 13–14, Table 4 · p. 13
• External fairness objectives provide smaller fairness gains than FairReL and often reduce cross-domain AUC; full-feature mean alignment trails it on all cross-domain fairness metrics.The compared alternatives include prediction-level fairness loss, adversarial demographic classification, and full-feature mean alignment.Source: Table 5 · p. 13, §4.4 · p. 14

4.5 Visualisation Results

• Grad-CAM examples show FairReL’s activations are more spatially consistent across gender–race subgroups than Effort [46], focusing on central facial regions rather than hair or background.The visualisations cover six subgroups on FF++ and DFDC.Source: Figure 7 · p. 14, §4.5 · p. 14

5 Conclusion

• Across FF++, Celeb-DF, DFD and DFDC, FairReL reduces cross-domain fairness disparities while preserving competitive detection performance.The conclusion situates these results on standard face-manipulation benchmarks with available demographic annotations.Source: §5 · p. 14
• FairReL targets spectral imbalance in spatial features and adaptation-induced shifts in the trainable residual subspace.GCWD reduces subgroup-imbalanced wavelet statistics before pooling, while SLMA aligns class-conditional subgroup means in the residual representation of an SVD-decomposed backbone.Source: §5 · p. 14
• Ablations and representation analyses confirm that GCWD and SLMA provide complementary objectives.Source: §5 · p. 14

A Datasets and Demographic Statistics

• FF++ [39], Celeb-DF [25], DFD, and DFDC [9] provide distinct sources and forgery pipelines for testing fairness generalisation across domains.The datasets cover controlled manipulations, in-the-wild celebrity videos, paid-actor recordings, and diverse actors, scenes, and manipulation methods.Source: A Datasets and Demographic Statistics · p. 19
• The evaluation uses six intersectional groups formed from male and female gender categories and Asian, White, and Black race categories, excluding Others.Demographic annotations follow [26]; Table S2 reports sample counts for each retained group, with “–” indicating a subgroup absent from a dataset annotation.Source: A Datasets and Demographic Statistics · p. 19, Table S2 · p. 19
• FF++ is split approximately 60/20/20 for training, validation, and testing, while Celeb-DF, DFD, and DFDC are unseen target-domain test sets.Table S1 reports dataset splits and demographic coverage after excluding samples annotated as Others.Source: Table S1 · p. 19

Equal FPR (FFPR)

• Equal FPR (FFPR) is the primary fairness metric, summing subgroup deviations from overall FPR because real faces dominate deployment and false alarms harm them.Metrics are lower-is-better percentages, calculated across six intersectional gender–race groups; definitions follow Ju et al. [19] and Lin et al. [26] for comparability.Source: B Evaluation Metrics · p. 20

Max Equalized Odds (FMEO)

• Max Equalized Odds (FMEO) is the larger of the worst subgroup FPR gap and worst subgroup TPR gap, with TPR conditioned on fake samples.Both fairness metrics are reported in percent, with lower values indicating greater parity.Source: B Evaluation Metrics · p. 20

Detection metrics in the per-subgroup tables

• Per-subgroup analysis reports group-averaged AUC, which weights groups equally, and AUC gap, the spread between the best- and worst-served subgroups.These summaries accompany overall AUC, which is dominated by majority subgroups.Source: B Evaluation Metrics · p. 20

• Training uses AdamW with learning rate 1×10−3, weight decay 4×10−3, batch size 64, 224×224 inputs, and a single H100 GPU.Source: C Implementation Details · p. 20
• GCWD uses a 2-level (J=2) Daubechies-4 wavelet decomposition yielding 7 sub-bands, while a group-balanced sampler draws approximately 10–12 samples per subgroup per batch to define GCWD and SLMA subgroup moments.For (y,g) cells with fewer than two samples in a batch, GCWD and SLMA losses use an exponential-moving-average estimate.Source: C Implementation Details · p. 20
• Hyperparameters λorth=0.1, λG=λS=0.5, and r=1 are selected on FF++ validation using source-domain AUC and FPR disparity at TPR=0.90.Target-domain test results are never used for selection.Source: C Implementation Details · p. 21

D Intra-Domain Comparison on FF++

• The FF++ comparison pins the threshold on validation data at TPR=0.90.The protocol matches the fixed-threshold setup described in the main text.Source: D Intra-Domain Comparison on FF++ · p. 21, Table S3 · p. 21
• DAW-FDD, PG-FDD, and FairReL use demographic labels during training, whereas FairAdapter does not.Table S3 denotes this distinction as “Req. Demo.”Source: D Intra-Domain Comparison on FF++ · p. 21, Table S3 · p. 21
• On FF++, FairReL attains the highest AUC and AUCavg and the lowest FFPR and FMEO.The comparison uses the fixed-threshold protocol pinned on the FF++ validation split at TPR=0.90.Source: Table S3 · p. 21, D Intra-Domain Comparison on FF++ · p. 21
• The intra-domain comparison trains and tests methods on FF++ and evaluates them on the FF++ test split.Table S3 reports AUC and AUCavg for detection, and FFPR and FMEO for fairness, with lower fairness values better.Source: Table S3 · p. 21, D Intra-Domain Comparison on FF++ · p. 21

E Per-Run Standard Deviations

• FairReL’s reported results are mean ± standard deviation across three runs with different random seeds on each of four datasets.The runs use an identical schedule and data.Source: E Per-Run Standard Deviations · p. 21
• FairReL’s standard deviations are ≤0.4 on the AUC metrics and ≤0.6 on the fairness metrics.These deviations are reported as small.Source: E Per-Run Standard Deviations · p. 21
• The small deviations mean the rankings in the main tables are stable across seeds.This stability statement follows the reported per-run deviations for FairReL across the four datasets.Source: E Per-Run Standard Deviations · p. 21

F Target-Domain Sensitivity at the Fixed Operating Point

• All FPR metrics use one threshold pinned on a held-out FF++ split at source-domain TPR=0.90 and reused on every dataset.This fixed operating point prevents shrinking FPR-based gaps simply by raising the threshold.Source: F Target-Domain Sensitivity at the… · p. 21
• At the same threshold, FairReL has consistently higher target-domain TPR and balanced accuracy than the subspace-decomposed baseline on all three targets.Table S5 evaluates both metrics under the shared threshold; the accompanying text states that FairReL’s lower FPR does not come at a detection cost.Source: F Target-Domain Sensitivity at the… · pp. 21–22, Table S5 · p. 22

G Per-Subgroup AUC on Each Dataset

• Celeb-DF and DFD omit Asian subgroup entries, and Celeb-DF’s F-B AUC is high-variance because that subgroup has only 30 test samples.AUCavg and the AUC gap use available subgroups.Source: G Per-Subgroup AUC on Each Dataset · p. 22

H Sensitivity to the Orthogonality Weight λorth

• λorth was retuned on FF++ validation because loose settings let principal directions leak into the residual, while tight settings over-constrain it and dilute SLMA.The selected value differs from the fairness-unsupervised tuning in [46].Source: H Sensitivity to the Orthogonality… · p. 22

G Per-Subgroup AUC on Each Dataset

• Across all four datasets, FairReL attains the highest AUCavg and smallest AUC gap while remaining detection-competitive in overall AUC.This pattern improves the fairness–generalisation trade-off rather than trading accuracy for subgroup fairness.Source: G Per-Subgroup AUC on Each Dataset · p. 22, Table S6 · p. 23

H Sensitivity to the Orthogonality Weight λorth

• λorth=0.1 is the best cross-domain entry on every target in the sweep, while both tested extremes are visibly suboptimal.The value was selected on the FF++ validation split with GCWD and SLMA active.Source: H Sensitivity to the Orthogonality… · p. 22, Table S7 · p. 23

Extended Mechanistic Analysis

• The localisation analysis uses principal-only and residual-only intervention views to measure spatial sub-band discrepancy and class-conditional subgroup mean shift on DFDC.Because the backbone is nonlinear, these are diagnostic probes rather than an exact additive split; quantities are normalised by column for gender and ethnicity partitions.Source: Extended Mechanistic Analysis · p. 24, §I.2 · pp. 24–25

I.1 Per-Sub-Band Probing of Demographic and Forgery Information

• Demographic-probe accuracy is highest in coarse level-2 bands, especially LL2, whereas forgery-probe AUC strengthens toward fine bands and peaks in high-frequency level-1 bands.This separation supports decorrelating subgroup-imbalanced low-frequency bands, which carry little forgery-discriminative signal.Source: Table S8 · p. 24, §I.1 · p. 24

I.2 Localisation of Subgroup-Sensitive Structure

• The residual carries more subgroup-sensitive structure than the frozen principal in low-frequency discrepancy and pooled mean shift, while high-frequency structure stays low in both.Within the residual, ethnicity dominates low-frequency spatial discrepancy and gender dominates pooled mean shift.Source: Table S9 · p. 25, §I.2 · p. 25

I.3 Complementarity of GCWD and SLMA

• On DFDC splits, a one-layer linear probe predicts gender or ethnicity from the residual, with balanced-accuracy chance levels of 50.0% and 33.3%, respectively.The probe is trained for 20 AdamW epochs; lower probe accuracy indicates less recoverable demographic information.Source: §I.3 · p. 25, Table S10 · p. 25
• GCWD lowers ethnicity-probe accuracy from 63.7 to 54.9, while SLMA lowers gender-probe accuracy from 78.4 to 65.8, targeting different signatures.The passage links ethnicity to low-frequency spectral-energy discrepancy and gender to a class-conditional first-moment shift of the pooled embedding, describing the losses as complementary rather than redundant.Source: §I.3 · p. 25, Table S10 · p. 25

J Discussion

• Demographic labels raise ethical and privacy concerns because they involve sensitive attributes and their collection or prediction may create privacy risks or reinforce protected-attribute profiling.The passage frames these concerns in fairness-aware deepfake detection.Source: J Discussion · pp. 25–26
• Subgroup information remains necessary to diagnose and mitigate unfair detector behavior, because without it researchers cannot know which groups are underserved.Source: J Discussion · p. 26
• FairReL uses demographic labels only during training to compute group-level fairness losses; its deployed detector needs only the input image.Source: J Discussion · p. 26
• At test time, FairReL requires no demographic labels, demographic classifiers, group-specific thresholds, or attribute-based routing, using labels to remove subgroup-dependent shortcuts rather than condition individual predictions.The passage presents this as a less intrusive, more deployment-friendly approach under the ethical constraints of demographic annotation.Source: J Discussion · p. 26