PaperlayerView exact revision on arXiv

Source-linked AI summary

Does an Illumination Prior Help Face-Swap Detection? A Controlled Study of Temporal Self-Blended Images

Danil Davydov, Bader Rasheed, Dmitriy Vatolin

arXiv:2610.11706v1cs.LG

TL;DR

Face-swap detectors trained with self-blended images primarily learn blending artifacts, leaving open whether an illumination prior adds value. This study tests T-SBI with controlled training comparisons and finds no evidence of illumination-specific improvement, while observing threshold shifts and greater robustness to heavy compression.

  • Problem

    Self-blended training captures blending artifacts, and the study asks whether adding illumination inconsistencies improves face-swap detection.

    Source: §I · p. 1

  • Method

    T-SBI transfers illumination statistics between same-video frames, and controlled regime, ∆L, and attribute-binned comparisons test its effects.

    Source: §I · p. 1, B. Pair-level illumination gating · p. 2

  • Results

    Across the controlled comparisons, T-SBI shows no consistent performance direction, no measurable frame-level effect from training-pair ∆L, and no preferential harsh-illumination improvement.

    Source: §I · p. 1, A. Five-regime ablation: no consistent… · p. 3, §V · p. 5

  • Takeaways & Limitations

    T-SBI shifts prediction scores and optimal thresholds, so fixed-threshold comparisons can misstate its capability effects; the observed compression robustness is distinct from the targeted illumination mechanism.

    Source: §V · p. 5, §VI · p. 6

    The five-regime ablation is single-seed, and the high- and low-∆L pools differ in source-style composition, partially confounding ∆L with source style.

    Source: §V · p. 6

Abstract

from arXiv · show

Self-blended images are widely used to train face-swap detectors, but primarily capture blending artifacts. We investigate whether adding illumination inconsistencies improves detection. Temporal Self-Blended Images (T-SBI) transfer lighting statistics between frames of the same video, with the mismatch controlled by luminance difference ({\Delta}L). Using five training regimes and a three-seed comparison of high- and low-{\Delta}L training, we find no evidence of illumination-specific improvements. AUC differences remain within seed variability across four datasets, and an analysis of 506,328 attribute-binned samples shows no preferential reduction in errors under harsh lighting. Instead, T-SBI shifts prediction scores, changing optimal thresholds by approximately 0.34 on FaceForensics++ and 0.30 on Celeb-DF, making comparisons at a fixed threshold misleading. However, T-SBI improves robustness to heavy JPEG compression on DFDC (AUC 0.780 versus 0.696), potentially reflecting greater reliance on low-frequency cues. These findings highlight the importance of evaluating training methods against their intended targets and accounting for threshold effects.

I. INTRODUCTION

T-SBI adds an illumination mismatch to self-blended training pairs to test whether photometric inconsistency improves face-swap detection. The study measures whether the addition, the size of the illumination delta, or robustness to harsh lighting affects detection.

I. INTRODUCTION

  • T-SBI transfers illumination statistics between frames of the same video, adding a photometric inconsistency to the self-blending boundary.The proposed probe tests whether this targeted prior changes detection, whether ∆L is the active ingredient, and whether robustness improves on illumination-targeted samples.Source: §I · p. 1
  • The generator uses pair-level illumination gating in LAB space, with a seven-variant shortcut ablation checking for class-asymmetric pipeline artifacts.These controls support interpreting the comparison as a test of the intended illumination prior.Source: §I · p. 1
  • High-∆L and low-∆L training differ by less than the combined seed standard deviation on every corpus.The comparison isolates the training-pair illumination delta across three seeds.Source: §I · p. 1
  • The study finds no preferential reduction in harsh-illumination failures and no consistent performance direction for T-SBI; its observed effects are a threshold shift and compression robustness.The failure taxonomy spans 506,328 attribute-binned rows.Source: §I · pp. 1–2

II. RELATED WORK

The paper situates T-SBI among blending and photometric-forensics approaches, then describes a same-video training-pair generator and explicit illumination-delta gating. The realized accepted pairs often fall below the designed threshold, with some admitted through a fallback.

II. RELATED WORK

  • Face X-Ray and SBI use blending boundaries as a compositing cue, while T-SBI injects photometric constraints during training rather than applying lighting checks at inference.The paper also motivates threshold sweeps because threshold-0.5 metrics can conflate calibration with capability under cross-dataset imbalance.Source: §II · p. 2

A. Temporal self-blended image generation

  • T-SBI transfers a source frame’s illumination statistics to a self-blended target from the same video while preserving target geometry, identity, expression, and pose.Luminance-only histogram matching with percentile trimming and envelope-based fitting is used to suppress flat-face, blowout, and banding failures.Source: A. Temporal self-blended image… · p. 2

B. Pair-level illumination gating

  • Pair-level gating measures lightness differences within face boxes, using ∆Lmean for global offsets and ∆Lstd for changes in shading spread.A relax fallback accepts the most-different available pair when no pair clears the gate; the recorded deltas and relax flag enable quartile analysis.Source: B. Pair-level illumination gating · p. 2
  • Across 7,320 accepted pairs, median ∆Lmean is 5.62, below the design threshold, and 22.3% entered through the relax fallback.The sampler, rather than the source corpus, is identified as the binding constraint.Source: B. Pair-level illumination gating · p. 2

C. Five-regime ablation

The five regimes isolate supervised fakes, classic self-blended images, and temporal self-blended images. B mix versus C mix, and B pure versus C pure, provide matched contrasts for adding T-SBI.

C. Five-regime ablation

  • B mix versus C mix and B pure versus C pure differ only by adding T-SBI pairs, providing the clean contrasts for the temporal-illumination signal.Source: C. Five-regime ablation · p. 2
  • The five regimes factorize supervised fakes, classic SBI fakes, and T-SBI across real-plus-synthetic training recipes.Source: C. Five-regime ablation · p. 2
  • All regimes share an EfficientNet-B4 backbone, AdamW with cosine warmup, 380 × 380 inputs, twenty epochs, and class-balanced batches; JPEG quality is randomized over [75, 98].Randomization applies to real and synthetic samples in every regime, eliminating compression history as a class-asymmetric shortcut.Source: C. Five-regime ablation · p. 2

D. Shortcut ablation

The shortcut ablation tests whether pipeline steps create class-asymmetric signals by replacing active operations with identity operations. Photometric-transfer variants fall within the null band, while two variants are reported as caveats.

D. Shortcut ablation

  • N3–N6 photometric-transfer variants all fall within the |AUC −0.5| ≤ 0.05 null band.Source: D. Shortcut ablation · p. 2
  • N0 JPEG round-trip yields |AUC −0.5| = 0.428 but is present identically in B and C, so it cancels in their contrast.Source: D. Shortcut ablation · p. 2
  • N2 adjacent-frame pairing yields |AUC −0.5| = 0.431, lies outside the null band, and is unique to T-SBI.The paper reports N2 as a caveat.Source: D. Shortcut ablation · p. 2
  • The test trains a one-epoch classifier on real and null-output samples, measuring two-sided separability as |AUC −0.5|; values ≤0.05 indicate no class-asymmetric signal.The T-SBI pipeline has seven identity-step variants (N0–N6), with a parallel classic-SBI suite (P0–P5).Source: D. Shortcut ablation · p. 2

E. Decisive diagnostic: the ∆L-quartile experiment

The ∆L-quartile experiment isolates training-pair illumination delta, while the failure taxonomy measures errors across image-attribute bins.

E. Decisive diagnostic: the ∆L-quartile experiment

  • HIGHDL and LOWDL use otherwise byte-identical C mix recipes, differing only in T-SBI pairs from the upper versus lower ∆L quartile.Each set contains 1,830 T-SBI pairs; their mean ∆Lmean values are 12.43 and 1.57, respectively.Source: E. Decisive diagnostic: the… · p. 2
  • The five-regime table compares cross-dataset AUC and calibration; its C-vs-B contrast has no consistent sign and is single-seed.DFF is reported at frame level, while other datasets use video-level mean aggregation.Source: Table I · p. 3
  • Each variant is trained for twenty epochs across three seeds and evaluated on FF++, Celeb-DF, DFDC, and DFF.Real frames, supervised fakes, and classic SBI fakes are held identical.Source: E. Decisive diagnostic: the… · p. 3

F. Failure taxonomy

  • The failure taxonomy bins nine image attributes by within-dataset quantiles across 506,328 rows from nine models and four datasets.It tests failure status against attribute bin using Pearson’s χ2 and reports Cramér’s V as the effect size.Source: F. Failure taxonomy · p. 3

A. Five-regime ablation: no consistent T-SBI direction

The ablation finds no consistent T-SBI advantage across regimes or datasets; the isolated illumination-delta comparison likewise shows no frame-level difference beyond seed variability.

A. Five-regime ablation: no consistent T-SBI direction

  • C mix exceeds B mix by +0.014 AUC on Celeb-DF but trails by −0.003 on DFDC, while C pure trails B pure on both datasets.The C-vs-B ablation therefore licenses no universal claim about T-SBI.Source: A. Five-regime ablation: no consistent… · p. 3
  • Baseline A has frame ECE 0.382 versus 0.030 for C mix despite its Celeb-DF AUC lead of 0.044.The calibration comparison distinguishes the AUC lead from the substantially different calibration results.Source: A. Five-regime ablation: no consistent… · p. 3
  • All regimes sit near chance on DFF (0.54–0.57), and the paper scopes its claims to the face-swap family.The stated reason is that fully generated diffusion faces lack the blending boundaries and illumination differences supervised by SBI and T-SBI.Source: A. Five-regime ablation: no consistent… · p. 3

B. The illumination delta is not the active ingredient

  • Every HIGHDL-versus-LOWDL frame-level difference lies within combined seed standard deviation across four corpora, despite an order-of-magnitude ∆Lmean gap.Mean ∆Lmean is 12.43 for HIGHDL versus 1.57 for LOWDL; Table II reports the three-seed comparison.Source: B. The illumination delta is not the… · p. 3, Table II · p. 3
  • Video-level results reverse direction on DFDC: LOWDL wins all three aggregation strategies, but mean and vote differences are calibration-mediated rather than evidence of better features.Both variants are severely miscalibrated on DFDC, and the DFDC gap shrinks under max aggregation; Table III gives the video-level comparison.Source: B. The illumination delta is not the… · p. 3, Table III · p. 3
  • On frame-level AUC, HIGHDL is at or above LOWDL on every OOD corpus, but margins are small relative to seed variance and both variants discriminate weakly.The figure reports AUCs of 0.55–0.67; DFDC’s video-level mean comparison reverses, with frame-level ECE ≈0.42 for both.Source: Figure 1 · p. 4

C. A dataset-level illumination control

Celeb-DF’s within-video illumination variation does not explain the dataset asymmetry; threshold-sweep results instead show that T-SBI shifts the operating point and changes fixed-threshold comparisons.

C. A dataset-level illumination control

  • Celeb-DF has the largest median within-video ∆Lmean p90 (9.97) and smallest static-clip fraction (0.15), so it is not illumination-static relative to FF++ or DFDC.DFDC instead differs in shadow-structure variation and between-video luminance diversity, and the targeted within-video variable does not separate the corpora.Source: C. A dataset-level illumination control · p. 4

D. The mechanism is a threshold shift, not illumination robustness

  • At threshold 0.5, B mix and C mix failure rates rise from 12.8% to 15.9% on Celeb-DF, 35.1% to 42.3% on DFDC, and 28.0% to 34.3% on DFF.Yet bootstrapped Celeb-DF AUC CIs do not overlap: 0.736–0.742 for B mix versus 0.751–0.756 for C mix.Source: D. The mechanism is a threshold shift,… · p. 4
  • C mix lowers the argmax-MCC threshold by ≈0.30 on Celeb-DF (0.88 to 0.58) and ≈0.34 on FF++ (0.76 to 0.42).The table reports frame-level threshold-sweep optima, with smaller and inconsistent shifts on DFDC and DFF.Source: D. The mechanism is a threshold shift,… · p. 4, Table IV · p. 4
  • On Celeb-DF at threshold 0.5, B mix has 0.9% false positives and 80% false negatives, versus 7.9% and 58% for C mix.The authors describe the shift as a trade between real-precision and fake-recall, not a uniform capability change.Source: D. The mechanism is a threshold shift,… · p. 4

E. Robustness: one condition where T-SBI helps

T-SBI shows a robustness advantage under heavy compression and blur, although the reported perturbation runs share the headline ablation’s single-seed limitation.

E. Robustness: one condition where T-SBI helps

  • On DFDC at JPEG q40, C mix retains frame AUC 0.780 versus B mix’s 0.696, a +0.084 margin.This is the largest single-condition gap in the study and an order of magnitude larger than any clean-image difference.Source: E. Robustness: one condition where… · p. 4, Table V · p. 5
  • C mix also degrades more gracefully under heavy blur on all three face-swap corpora, with gains of +0.02–0.03 at σ = 3.The table covers frame-level AUC under JPEG compression and Gaussian blur.Source: E. Robustness: one condition where… · p. 5, Table V · p. 5
  • The authors suggest illumination-varied pairs shift reliance toward low-frequency global cues that survive quantisation, unlike high-frequency blending-boundary evidence.These runs share the headline ablation’s single-seed limitation.Source: E. Robustness: one condition where… · p. 5

F. The targeted attribute shows no effect

The attribute-binned analysis finds no preferential reduction in harsh-illumination failures. T-SBI’s observed changes instead concern score operating point and compression robustness, not its targeted illumination mechanism.

F. The targeted attribute shows no effect

  • C mix shows negligible illumination-harshness effects on every dataset, with Cramér’s V below 0.05 and no monotonic failure-rate ordering.The per-bin failure-rate spreads are only 1–5 percentage points.Source: Figure 3 · p. 5, F. The targeted attribute shows no… · p. 5
  • Of 216 model-dataset-axis triples, only 23 reach V ≥0.10, two exceed 0.20, and none exceed 0.30.The strongest illumination sensitivity belongs to the SBI reference on DFDC and peaks on the flat-illumination bin, opposite the T-SBI hypothesis.Source: F. The targeted attribute shows no… · p. 5

V. DISCUSSION

  • C-vs-B changes sign across corpora, while an eightfold training-pair ∆L difference stays within frame-level seed noise on all four.The taxonomy also finds no preferential harsh-illumination improvement, with T-SBI’s illumination effect sizes uniformly smaller than its counterpart’s.Source: §V · p. 5
  • T-SBI lowers the optimal threshold by ≈0.30 where its effect is largest and is associated with low-frequency evidence and greater compression robustness on DFDC.Only a threshold sweep distinguishes the apparent capability loss at 0.5 from the opposite reading at models’ own optima.Source: §V · pp. 5–6
  • A physically motivated prior should be evaluated against its target attribute, and the realized distribution of the gated quantity should be reported.The sampler admitted a median pair below its threshold despite much larger corpus deltas.Source: §V · p. 6
  • The five-regime ablation is single-seed, the ∆L pools partly confound source style, the illumination descriptor reduces spatial structure to two scalars, and the taxonomy uses threshold 0.5.The operating-point analysis mitigates but does not remove the threshold limitation.Source: §V · p. 6

VI. CONCLUSION

The paper uses T-SBI as a controlled probe of whether illumination priors improve face-swap detection. It finds no support for the targeted mechanism, while identifying threshold and compression effects and proposing reusable evaluation diagnostics.

VI. CONCLUSION

  • Across a five-regime ablation, three-seed illumination-delta experiment, dataset-level control, and 506,328-row taxonomy, the targeted illumination mechanism is unsupported.Training-pair illumination delta does not measurably change frame-level detection, and harsh-illumination bins show no preferential robustness.Source: §VI · p. 6
  • T-SBI shifts the optimal threshold downward by ≈0.30, so comparisons at 0.5 can reverse the apparent verdict relative to each model’s optimum.The conclusion also reports improved robustness to heavy compression, neither as the illumination mechanism the prior targeted.Source: §VI · p. 6
  • The paper reports T-SBI as a controlled probe rather than an improvement and offers realized-gate reporting, threshold sweeps, and attribute-binned failure taxonomies as reusable diagnostics.These diagnostics are proposed for evaluating physically motivated training priors.Source: §VI · p. 6

APPENDIX A TRAINING AND EVALUATION PROTOCOL

Table VI consolidates the configuration shared by the five training regimes and both ∆L-quartile variants for reproducibility.

APPENDIX A TRAINING AND EVALUATION PROTOCOL

  • Table VI consolidates the shared configuration for all five regimes.The configuration also covers both ∆L-quartile variants and is provided for reproducibility.Source: Appendix A · p. 6
  • Both ∆L-quartile variants use the configuration consolidated in Table VI.The stated purpose is reproducibility.Source: Appendix A · p. 6
  • Table VI presents the common configuration for the five regimes and two ∆L variants to support reproducibility.Source: Appendix A · p. 6

Related papers

Loading 2610.11706v1…