PaperlayerView exact revision on arXiv

Source-linked AI summary

PairCLIP-SWA: Similarity-Guided Balancing and Stochastic Weight Averaging for Generalized Deepfake Image Detection with OpenCLIP

Md. Nakib, Sudipto Sarkar, A. J. A. Jubayer Talukder, Mohammad Ariful Haque

arXiv:2610.37874v1cs.CVcs.AI

TL;DR

Checkpoint selection from a validation split is widely used, but its reliability in deepfake detection has rarely been quantified. PairCLIP-SWA adapts and averages a single OpenCLIP tower trained on a balanced subsample. Across splits, the averaging winner changes on selection splits, but averaging wins on the split used for neither training nor selection.

  • Problem

    Checkpoint selection is commonly treated as free, although finite validation sets can bias selection and deepfake checkpoint reliability has not been quantified.

    Source: A. Research Gaps · p. 1

  • Method

    PairCLIP-SWA adapts a single OpenCLIP ViT-L/14 tower through progressive unfreezing on a balanced subsample and averages its fine-tuning trajectory in weight space.

    Source: §VI · p. 13

  • Results

    Averaging wins by 5.91 points on the only split used for neither training nor selection, while the selection-split winner differs across DF-Wild-Cup and OpenForensics.

    Source: D. Results Across Splits · p. 7

  • Takeaways & Limitations

    The authors propose additional seeds and OpenForensics threshold sweeps so selection effects carry dispersion and threshold-free metrics.

    Source: §VI · p. 13

    Reported runs are single trajectories, so PairCLIP-SWA’s own seed-to-seed dispersion is unknown.

    Source: G. Limitations · p. 13

Abstract

from arXiv · show

Deepfake detectors are almost always deployed as the single checkpoint that scored best on a validation split, yet the reliability of that choice is rarely measured. We measure it with PairCLIP-SWA, a single-backbone detector that adapts an OpenCLIP ViT-L/14 visual tower by progressive block unfreezing on a class-balanced corpus built by CLIP-similarity-guided one-to-one subsampling, and averages the fine-tuning trajectory in weight space. Trained in-domain on the IEEE SP Cup 2025 (DF-Wild-Cup) benchmark under the same protocol as the strongest published three-backbone ensemble, the weight-averaged detector reaches 97.04% accuracy, 98.64% AU-ROC and a 4.07% equal error rate, against 94.46%, 97.60% and 5.00%. Its training log shows why averaging matters: over the averaged epochs the non-averaged checkpoint swings across 5.34 points of validation accuracy while its validation loss more than doubles, whereas the average is never worse and is up to 5.99 points better. On an OpenForensics-derived benchmark the same instability becomes a misranking: the checkpoint that validation accuracy selects (98.43%) scores 88.85% on a disjoint 10,905-image test partition, rejecting 21.1% of genuine photographs, while the terminal average, ranked last on validation, scores 94.76%. A study of twenty-two in-house configurations on the identical DF-Wild-Cup split, including preprocessing ablations, shows that CLIP pre-training rather than ensembling drives accuracy, that secondary face cropping costs about 34 points, and that frequency-domain inputs transfer poorly. We recommend per-class selection criteria measured off the early-stopping split.

I. INTRODUCTION

Deepfake detectors can generalize poorly across generators, compression, and capture conditions, while validation-selected checkpoints and preprocessing choices are not always measured or justified. The paper presents a single-backbone OpenCLIP detector and studies checkpoint selection, preprocessing, and representation geometry.

I. INTRODUCTION

  • Detectors can degrade sharply when generators, compression pipelines, or acquisition conditions change, making generalization a central challenge.These shifts can follow near-saturated performance within one manipulation family.Source: §I · p. 1

A. Research Gaps

  • Checkpoint-selection reliability is unquantified in deepfake studies, despite finite-validation optimism and the importance of selection protocols in domain generalization.The paper also identifies ImageNet-dominated initialization and rarely justified preprocessing as open questions.Source: A. Research Gaps · pp. 1–2
  • CLIP’s web-scale image–text pre-training has shown transfer across unseen generators, but frozen probes can underfit forensic evidence and unconstrained fine-tuning can distort pre-trained geometry.Whether a single adapted CLIP tower can replace a heterogeneous ensemble remains open.Source: A. Research Gaps · p. 1

B. Contributions

  • 97.04% accuracy, 98.64% AU-ROC, and 4.07% equal error rate exceed CAE-Net’s 94.46%, 97.60%, and 5.00%, respectively, under the same in-domain protocol.The paper also reports 5.34 points of validation-accuracy swing and a held-out OpenForensics misranking of the validation-selected checkpoint.Source: §I · p. 1, B. Contributions · p. 2
  • Twenty-two configurations and preprocessing ablations support CLIP pre-training over ensembling, while secondary face cropping costs about 34 points and frequency inputs transfer poorly.The study also reports that a weight-averaged single model outperforms a five-model prediction ensemble with one forward pass instead of five.Source: B. Contributions · p. 2

A. Spatial, Frequency and Hybrid Detection

  • Spatial detectors target blending boundaries, whereas frequency and wavelet methods target spectral signatures that can be attenuated by recompression and resizing.CAE-Net combines spatial, attention, and wavelet experts and trains sequentially on disjoint fake subsets.Source: A. Spatial, Frequency and Hybrid… · p. 2

B. CLIP for Forensics

Prior work motivates adapting web-scale CLIP features between frozen probing and full fine-tuning, while this paper studies how data construction and checkpoint selection shape detector behavior. Its preprocessing choices preserve full frames and use limited augmentation to avoid altering forensic traces.

B. CLIP for Forensics

  • Progressive unfreezing places adaptation between frozen CLIP probes and full fine-tuning, which can respectively underfit forensic evidence or distort pre-trained geometry.Prior work reports cross-generator transfer from frozen CLIP features; this paper focuses on behavior under checkpoint selection.Source: B. CLIP for Forensics · p. 2

C. Shortcuts and Data Construction

  • Similarity-guided fake selection is intended to reduce content shortcuts that count-based balancing leaves untouched, although self-blended fakes trade that route for bias toward one synthesis operator.The selection is described as a greedy approximation to an assignment problem.Source: C. Shortcuts and Data Construction · p. 2

D. Weight Averaging and Model Selection

  • SWA averages trajectory iterates, while validation selection can incur optimistic bias that grows with the number of candidates compared.The paper’s contribution is empirical measurement of forensic checkpoint trajectories and the cost of selecting among them.Source: D. Weight Averaging and Model Selection · p. 2

A. Problem Statement and Notation

  • The study treats both the classifier and the rule that maps its parameter trajectory to a deployed checkpoint as objects of analysis.Labels use real as 1 and fake as 0, while precision, recall, and F1 use fake as the positive class.Source: A. Problem Statement and Notation · p. 2

B. Preprocessing Pipeline and Design Rationale

C. Stage 1: CLIP-Similarity-Guided Subsampling

The subsampling stage pairs each real image with a similar fake in CLIP space to create a balanced corpus. It uses a greedy no-reuse assignment and substantially reduces the training images, while leaving much fake-pool diversity unused.

1) Motivation:

  • When real and fake content distributions differ, empirical risk minimization can exploit content rather than manipulation; the proposed pairing favors fakes most similar to reals.The motivation is to reduce content-only shortcuts when the majority class must be subsampled.Source: 1) Motivation · p. 3

2) Objective and greedy solution:

  • The method greedily processes real images in order, assigning each its most similar unused fake rather than solving the cubic-cost exact assignment.No-reuse prevents a few generic fakes from absorbing multiple real partners; block streaming limits peak similarity-matrix memory.Source: 2) Objective and greedy solution · p. 3

3) Data usage:

  • 42,690 pairs form an 85,380-image corpus, using 32.6% of available training images and retaining 19.5% of fakes.The remaining generator diversity is a cost: most of the fake pool never reaches the detector.Source: 3) Data usage · pp. 3–4

4) Scope of the claim:

The authors limit what can be attributed to similarity-guided selection because the reported corpus lacks a random-subsample control and was built from twice-rescaled inputs. Stage 2 maintains exact real–fake balance in every training batch.

4) Scope of the claim:

  • No identical-schedule random one-to-one control exists, and the reported similarities came from strongly attenuated, twice-rescaled inputs, so content alignment remains unestablished.The stage is best read as a deterministic similarity-guided subsample, not evidence that semantic matching improved detection.Source: 4) Scope of the claim · p. 4

D. Stage 2: Class-Exact Sampling

  • Each batch contains exactly 16 real and 16 fake images, and interleaving keeps both DataParallel replica halves balanced.The per-sample layer normalization means the shard split does not change the forward computation.Source: D. Stage 2: Class-Exact Sampling · p. 4

E. Stage 2: Backbone

The detector uses OpenCLIP ViT-L/14’s visual tower and projected class-token representation with an affine head. It progressively unfreezes backbone blocks and averages the fine-tuning trajectory to produce deployable checkpoints.

E. Stage 2: Backbone

  • The visual tower feeds a projected class-token representation to an affine classifier, retaining the CLIP projection because its space preserves learned contrastive geometry.The text tower is discarded; the input is split into 256 patches of 14 × 14 pixels and embedded to width 1024.Source: E. Stage 2: Backbone · pp. 4–5

F. Stage 2: Progressive Unfreezing and Optimization

  • At each epoch, the head and last k(e) blocks are trainable while the embeddings and projection stay frozen; k(e) grows from two blocks toward a cap of 14.Newly unfrozen tensors enter AdamW with zero moment estimates; balanced batches use unweighted two-way cross-entropy, and early stopping monitors non-averaged validation accuracy.Source: F. Stage 2: Progressive Unfreezing and… · p. 5

G. Stage 2: Stochastic Weight Averaging

  • From the SWA start epoch, the method maintains a running weight mean, anneals the learning rate, and checkpoints both averaged and non-averaged models each epoch.It yields best-regular, best-SWA, and final-SWA deployable models; the final average receives a normalization-statistics pass.Source: G. Stage 2: Stochastic Weight Averaging · p. 5

H. Stage 3: Representation Geometry

The representation-geometry analysis probes class separation in a two-dimensional feature projection and independently in the original 768-dimensional feature space.

H. Stage 3: Representation Geometry

  • For held-out features projected with t-SNE, the analysis reports the distance between fake and real class centroids and a scale-free ratio based on their radial dispersions.The centroid distance is Δ = ∥μf − μr∥2; the ratio is described as a two-cluster analogue of a Fisher ratio.Source: H. Stage 3: Representation Geometry · p. 5
  • A k-nearest-neighbour classifier fitted in the t-SNE embedding probes how much class information survives the projection.Source: H. Stage 3: Representation Geometry · p. 5
  • As a projection-independent check, DBSCAN partitions the 768-dimensional features, with silhouette coefficient and Davies–Bouldin index reported.Source: H. Stage 3: Representation Geometry · p. 5

IV. EXPERIMENTS

The experiments evaluate PairCLIP-SWA on DF-Wild-Cup and an OpenForensics-derived benchmark, with checkpoint selection and evaluation roles defined for each split. The reported training trajectory shows substantially greater validation stability for the weight average than for the non-averaged checkpoint.

A. Datasets and Splits

  • DF-Wild-Cup uses eight benchmarks and evaluates both PairCLIP-SWA and CAE-Net on the same 3,072-image official validation partition after training on the training partition.The separate OpenForensics-derived run is selected on validation and evaluated once on a disjoint 10,905-image test partition untouched by training or selection.Source: A. Datasets and Splits · p. 6

B. Implementation and Reproducibility

  • The reported DF-Wild-Cup run starts from OpenAI CLIP weights with a random head, trains for epochs 0–19, and averages seven iterates from epochs 13–19.SWA starts at epoch 12; the reported run uses two NVIDIA T4 GPUs.Source: B. Implementation and Reproducibility · p. 6
  • Stage 1 double-rescaled its inputs, so similarities for the reported subsample were computed on attenuated, near-black images rather than the intended input range.The channel dynamic range shrank by 255×; passing PIL images or setting do_rescale=False restores the intended path. Frozen-feature reproductions should also account for the QuickGELU versus standard GELU mismatch.Source: B. Implementation and Reproducibility · p. 6

C. Evaluation Protocol

  • DF-Wild-Cup accuracy is reported with 95% Wilson intervals, while the best-SWA checkpoint’s scores are swept across 1,000 thresholds to obtain AU-ROC, EER, and threshold-specific measures.OpenForensics confusion matrices are reported at τ = 0.5; in-loop trajectory values use float16 autocast, whereas saved checkpoints are re-evaluated in full precision.Source: C. Evaluation Protocol · p. 6, B. Implementation and Reproducibility · p. 6
  • The non-averaged checkpoint’s validation accuracy oscillates across 5.34 points and its validation loss rises 2.15×, while averaged validation accuracy stays within 1.86 points and is never lower.At epoch 17 the average leads by 5.99 points; its validation loss falls by 30% while training loss falls by 78%.Source: Figure 2 · p. 7

D. Results Across Splits

Across the selection splits, the winning checkpoint differs, but averaging wins by 5.91 points on the split used for neither training nor selection. On DF-Wild-Cup, PairCLIP-SWA beats CAE-Net on every reported metric, while its training log shows the non-averaged checkpoint is less stable than the average.

D. Results Across Splits

  • Averaging wins by 1.37 points on DF-Wild-Cup selection and loses by 0.76 on OpenForensics selection, but wins by 5.91 on the split used for neither training nor selection.The paper identifies the latter as the only split used for neither training nor selection.Source: D. Results Across Splits · p. 7

E. Training Dynamics: the Case for Averaging

  • Across eight epochs, non-averaged validation accuracy spans 5.34 points, from 95.70 to 90.36%, with a 2.33-point mean consecutive-epoch change.The average spans 1.86 points, with a 0.34-point mean consecutive-epoch change.Source: E. Training Dynamics: the Case for… · p. 7, Table VI · p. 8
  • Training loss falls 78% as non-averaged validation loss rises to 0.3603, 2.15 times its epoch-12 value; averaged validation loss falls 30% to 0.1178.The first large validation-loss increase coincides with the first unfreezing event in the averaged window, and the largest follows the second.Source: E. Training Dynamics: the Case for… · p. 7
  • From epoch 14 onward, the average beats the non-averaged iterate at every epoch by 1.37 to 5.99 points, averaging 3.44 points.The two models coincide at epochs 12–13 because the first SWA update copies the iterate.Source: E. Training Dynamics: the Case for… · p. 7, Table VI · p. 8

F. Comparison on DF-Wild-Cup

  • At the accuracy-optimal threshold, specificity is 99.03% and fake-class recall is 95.54%; 68 of 83 remaining errors are forgeries accepted as real.The threshold was selected on the evaluation split, so the reported 0.26-point gain from moving it to 0.680 is an upper bound.Source: F. Comparison on DF-Wild-Cup · p. 8
  • 97.04% vs. 94.46% accuracy, 98.64 vs. 97.60% AU-ROC, and 4.07 vs. 5.00% EER: PairCLIP-SWA beats CAE-Net on every reported metric.The comparison is on the DF-Wild-Cup validation set; PairCLIP-SWA is the best-SWA checkpoint.Source: Table VIII · p. 8, Table VII · p. 8, F. Comparison on DF-Wild-Cup · p. 8
  • The detector is 2.59× larger, uses 3.27× the MACs, and has 2.7× per-image latency versus CAE-Net, measured on different hardware.In exchange, it uses one backbone, one forward pass, one training stage, and a third of the training images.Source: Figure 3 · p. 7, F. Comparison on DF-Wild-Cup · pp. 8–9

G. OpenForensics: Convergence and Validation

On OpenForensics, validation favors the selected checkpoint, but a disjoint held-out test partition reverses that ranking. The terminal average scores higher on the test partition, where most selected-checkpoint errors are genuine photographs rejected as forgeries.

G. OpenForensics: Convergence and Validation

  • On validation, the selected checkpoint scores 98.43% and beats the terminal average, which scores 97.67%; validation alone recommends the selected checkpoint.The selected checkpoint has 99.03% fake recall and 97.83% specificity, while the terminal average trades recall for 99.60% specificity.Source: G. OpenForensics: Convergence and… · p. 9

H. OpenForensics: the Held-Out Test Partition

  • Validation and test partitions come from the same corpus, so the measured shift is the residual difference between disjoint samples of in-the-wild imagery.The paper describes this as the kind of shift that separates a benchmark split from deployment data.Source: H. OpenForensics: the Held-Out Test… · p. 9
  • 88.85% for the validation-selected checkpoint versus 94.76% for the terminal average on the held-out test; the selected checkpoint makes 2.1× as many errors.Its specificity falls from 97.83% to 78.88%, while fake-class recall changes from 99.03% to 98.67%; 94.0% of its test errors are genuine photographs rejected as forgeries.Source: H. OpenForensics: the Held-Out Test… · p. 9, Figure 5 · p. 10
  • In ten examples, the selected checkpoint rejects two severely degraded genuine images with high confidence, while the average classifies both correctly with lower confidence.The images are a heavily noised crowd scene and a portrait with an extreme magenta cast.Source: H. OpenForensics: the Held-Out Test… · p. 9, Figure 6 · p. 10

I. Representation Geometry

Embedding separation does not consistently track classifier quality. The ablations show substantial costs from secondary face cropping and frequency-domain inputs, while similarity-guided balancing is promising but not isolated from other changes.

I. Representation Geometry

  • Averaging lowers separation S from 2.033 to 1.917 yet raises fake recall 92.90→95.74% and MCC 0.922→0.944, showing separation and decision quality can diverge.On OpenForensics, checkpoints differ by at most 0.03 in S and not at all in probe accuracy (97.80%), yet differ by 5.91 points on held-out test data.Source: §I · pp. 10–11, Figure 7 · p. 11
  • Cluster indices are not reliable proxies for classifier quality; the probe uses one 1,000-image subsample, so small separation-score differences may reflect subsample variance.The probe agrees with the head to within 0.16 points on DF-Wild-Cup.Source: §I · p. 11

A. Preprocessing and Data-Preparation Ablations

  • 56.05% validation accuracy from secondary face cropping is about 34 points below the same model on full frames, despite fitting 84.34% of training data.The proposed explanation is loss of transferable evidence; context around the face may preserve blending-boundary cues.Source: A. Preprocessing and Data-Preparation… · pp. 11–12, Figure 8 · p. 11
  • FFT spectra score 49.61% on validation after fitting 83.72% of training data; wavelets reach 73.20%, still 7.55 points below spatial RGB with CLIP initialization.The cited ConvNeXt+wavelet result is 16 points above the wavelet-coefficient input, indicating that backbone choice also matters.Source: Figure 8 · p. 11, A. Preprocessing and Data-Preparation… · p. 12
  • 95.67% for similarity-guided balancing exceeds 93.36% for random one-to-one undersampling, but differing unfreezing depth and preprocessing prevent attributing the gap to content matching.The passage says the result is consistent with, but does not establish, a benefit from Stage 1.Source: A. Preprocessing and Data-Preparation… · p. 12, Table XII · p. 12
  • Augmentation raises a ViT-LoRA early validation peak from 86.75 to 91.44% but ends at 87.99%; triplet metric learning reaches 98.70% training accuracy yet trails cross-entropy by 6.19 validation points.Class-symmetric corruption is proposed as a remedy for a possible degradation shortcut, but was not tested.Source: A. Preprocessing and Data-Preparation… · p. 12

B. Backbone and Adaptation: a Design-Space Study

The design-space study favors OpenCLIP initialization with progressive unfreezing and weight averaging. Validation-selected checkpoints can misrank held-out performance, motivating selection criteria beyond validation accuracy.

B. Backbone and Adaptation: a Design-Space Study

  • OpenCLIP ViT-L/14 configurations span 93.36–97.04%, above the reported ImageNet CNN and non-CLIP transformer ranges, supporting the study's conclusion that pre-training dominates.The design-space runs vary in schedules, epochs, and sometimes training data, so the comparisons are not controlled ablations.Source: B. Backbone and Adaptation: a… · p. 12
  • Progressive unfreezing performs best among the tested adaptation strategies; full fine-tuning of a non-CLIP ViT falls from 89.58% after one epoch to 87.60% after two.The near-linear probe reaches 80.70%, while adapter-based LoRA falls between the extremes.Source: B. Backbone and Adaptation: a… · p. 12
  • 97.04% accuracy from one-pass ViT-L/14 with SWA exceeds 94.95% from an ensemble of five ViT-B/32 models, which requires five forward passes.The ensemble comprises models averaging 92.75% individually; feature fusion of three heterogeneous backbones scores 81.22%.Source: B. Backbone and Adaptation: a… · p. 12

C. What Weight Averaging Buys, and Why

  • The two validation splits rank averaging differently (+1.37 points on DF-Wild-Cup, −0.76 on OpenForensics), while on data used for neither training nor selection the ranking reverses by 5.91 points.The non-averaged DF-Wild-Cup iterate has 2.13-point validation-accuracy standard deviation over eight epochs; averaging integrates over that variability at zero inference cost.Source: C. What Weight Averaging Buys, and Why · pp. 12–13

D. Validation Accuracy Is a Poor Deployment Proxy

  • The validation-accuracy-selected OpenForensics checkpoint makes 2.1× as many held-out errors as the terminal average, with failure confined to one class.The passage recommends balanced accuracy or MCC on a partition disjoint from early stopping, plus per-class recall and weight averaging.Source: D. Validation Accuracy Is a Poor… · p. 13

E. Computational Complexity

The paper reports the detector's computational cost, identifies structural choices used instead of weight decay, and lists limitations and follow-up experiments. Its conclusion reports stronger DF-Wild-Cup metrics than the published ensemble, with evaluation-scope caveats.

E. Computational Complexity

  • 81.0 GMACs (162.0 GFLOPs) per image are reported for the 24-block tower; measured latency is 28.1 ms per image on 2×T4.Parameter memory is 1,160 MB in float32 versus 448 MB for CAE-Net; the quadratic attention term contributes 4.0%.Source: E. Computational Complexity · p. 13

F. On the Absence of Weight Decay

  • With λ = 0, regularization is structural: 54.42% of the tower never moves, batches are exactly balanced, and averaging smooths the trajectory.The non-averaged suffix still overfits, so the passage says a sweep over λ remains worthwhile.Source: F. On the Absence of Weight Decay · p. 13

G. Limitations

  • The reported runs use single trajectories, DF-Wild-Cup results are selection-split numbers, and most design-space comparisons confound schedules, epochs, data, or hardware.The limitations also include one-subsample geometry analysis, absent adversarial evaluation, threshold-free metrics only for the DF-Wild-Cup best-SWA checkpoint, and cost that rules out edge deployment without distillation.Source: G. Limitations · p. 13

VI. CONCLUSION AND FUTURE WORK

  • 97.04% accuracy, 98.64% AU-ROC, and 4.07% EER for PairCLIP-SWA compare with 94.46%, 97.60%, and 5.00% for CAE-Net under the same in-domain protocol.The conclusion also says PairCLIP-SWA uses a third of the training images.Source: §VI · p. 13
  • Planned follow-up includes additional seeds, corrected Stage 1 preprocessing with a matched random-subsample control, adversarial evaluation, symmetric corruption augmentation, and distillation.The authors also plan to repeat the OpenForensics threshold sweep and measure threshold-free metrics.Source: §VI · pp. 13–14
Loading 2610.37874v1…