Source-linked AI summary
Spectra: Multi-Scale Implicit Decomposition for Deepfake Detection
Dhananjaya Jayasundara, Kartik Narayan, Vishal M. Patel
TL;DR
Deepfake detectors can struggle to generalize across unseen generators, compression, and dataset shifts. Spectra combines semantic features with INR reconstruction cues, and experiments report strong cross-dataset deepfake performance and generalization to presentation attack detection.
Problem
Deepfake detectors may rely on synthesis-pipeline-specific cues and lose accuracy across unseen generators, compression, post-processing, or dataset shifts.
Source: §1 · p. 1Method
Spectra combines CLIP semantic features with multi-level, spatial and frequency INR reconstruction representations, adaptively fused and jointly modeled through bidirectional attention.
Source: §1 · p. 2, §2.3 · p. 4Results
Spectra achieves state-of-the-art cross-dataset deepfake detection and generalizes to presentation attack detection without task-specific redesign.
Source: §5 · p. 8Takeaways & Limitations
Abstract
from arXiv · showhide
Deepfakes increasingly undermine trust in digital media, yet existing detectors often struggle to generalize across unseen manipulation techniques, compression levels, and dataset shifts. Most approaches rely on RGB appearance, frequency-domain artifacts, or their combination, learning cues present in the final synthesized image. While effective in-distribution, such cues are frequently generator-specific and can diminish as synthesis models evolve. We argue that a complementary perspective can be obtained through implicit neural representations (INRs), which expose how an image is reconstructed rather than merely how it appears. By fitting an INR to an image and analyzing its internal activations, one can reveal reconstruction dynamics that are difficult to observe in the image domain alone and may capture more intrinsic differences between genuine and synthetic content. To this end, we propose Spectra, a deepfake detection framework that combines semantic visual representations with artifact-aware cues derived from INR reconstruction. Spectra extracts multi-level reconstruction heatmaps from image-specific INRs and jointly models them with semantic features through a bidirectional fusion mechanism. This enables direct interaction between image semantics and reconstruction artifacts, allowing the model to identify forgery cues that are less dependent on specific generation pipelines. Extensive experiments on multiple deepfake benchmarks demonstrate improved robustness and cross-dataset generalization, consistently outperforming detectors based solely on RGB appearance or semantic representations. Furthermore, we show that the proposed representation extends beyond deepfake detection to face presentation attack detection, highlighting its potential as a unified framework for facial attack detection.
1. Introduction
Spectra addresses deepfake detectors’ difficulty generalizing beyond known manipulation pipelines by combining semantic features with INR-derived reconstruction cues. The authors report stronger cross-dataset generalization than semantic-only baselines.
1. Introduction
- Detectors can deteriorate on unseen generators, compression levels, post-processing, and dataset shifts when they rely on pipeline-specific artifacts.The paper notes that visible artifacts may diminish as generative models improve.Source: §1 · p. 1
- Prior CLIP-based detectors benefit from transferable semantic representations but may overlook subtle generation traces needed for forensic analysis [7] [43].CLIP is pretrained for semantic understanding rather than forensic analysis [45].Source: §1 · p. 2
- INR reconstruction dynamics may expose intrinsic differences between authentic and manipulated content through multi-level, spatially and frequency-sensitive heatmaps.The paper proposes these cues as complementary to semantic visual features and less dependent on a particular synthesis pipeline.Source: §1 · p. 2
- Spectra combines frozen CLIP semantics with image-specific INR heatmaps and bidirectional attention to model interactions between image meaning and reconstruction artifacts.CLIP is used for semantic content, while INR cues represent how the image is reconstructed.Source: §1 · p. 2
- Across multiple deepfake benchmarks, Spectra outperforms strong semantic-only baselines and achieves superior cross-dataset generalization.The authors describe INR-derived representations as complementary forensic evidence largely absent from conventional image-domain features.Source: §1 · p. 2
2. Method
Spectra combines multiscale CLIP semantics with image-specific SIREN reconstruction maps, then models their interaction for facial attack detection. Its method covers dataset preparation, multi-level decomposition, adaptive spatial–frequency and semantic fusion, and joint classification and representation learning.
2. Method
- Spectra combines multiscale CLIP features with image-specific INR heatmaps, processes reconstruction cues through spatial and frequency branches, and fuses both streams bidirectionally for classification and contrastive learning.The framework also combines global CLIP and heatmap descriptors in a shared representation.Source: §2 · p. 2, Figure 2 · p. 3
2.1. Dataset Preparation
- Training uses FF++ c23 with 10 fake and 40 real frames sampled per video, and evaluation covers four out-of-distribution benchmarks: CDFv1, CDFv2, FaceShifter, and DFD.The preprocessing pipeline extracts frames, detects faces with RetinaFace, aligns them using five facial landmarks, and resizes the crops.Source: §2.1 · pp. 2–3
2.2. Multi-Level Implicit Decomposition
- SIREN hidden-layer activations are projected with PCA into heatmaps that reveal different genuine and deepfake representation pathways despite comparable reconstruction fidelity.For K=3, the decomposition contains three layer heatmaps and two inter-level difference maps; a hypernetwork predicts SIREN parameters to avoid per-image optimization.Source: §2.2 · pp. 3–4, Figure 3 · p. 4
2.3. Reconstruction-Aware Representation Learning
- Spatial encoding captures localized reconstruction structure while frequency encoding captures spectral irregularities, and a learned gate adaptively weights these cues for each image.The spatial and frequency features are combined using a channel-wise gate predicted from their global average pooled descriptors.Source: §2.3 · p. 4
2.4. Multi-Scale Semantic Representation Learning
- CLIP features from early, intermediate, and late transformer layers are projected into a shared space and combined with learnable normalized weights.For CLIP ViT-L/14, the selected layers are 11, 17, and 23; the weighted representation includes a global token and patch tokens.Source: §2.4 · p. 5
2.5. Bidirectional Artifact-Semantic Fusion
- Bidirectional fusion lets semantic features retrieve reconstruction evidence while reconstruction cues guide aggregation of semantic regions.The resulting representation combines global semantic and reconstruction descriptors with both interaction-aware descriptors.Source: §2.5 · pp. 5–6
2.6. Joint Representation Learning
- The fused representation is projected to a compact embedding used for both deepfake detection and presentation attack detection, with the training distribution changing between tasks.The representation combines image semantics, reconstruction-aware evidence, and their bidirectional interaction.Source: §2.5 · p. 6, §2.6 · p. 6
2.7. Training Objective
- Mixup regularizes the classification branch, while supervised contrastive learning pulls same-class embeddings together and pushes different classes apart.The full objective combines cross-entropy and supervised contrastive losses; contrastive learning is computed on the unmixed batch because mixup targets are fractional.Source: §2.7 · p. 6
3. Experiments
Spectra is evaluated on out-of-distribution deepfake benchmarks, feature-space class separation, and face presentation attack detection. It achieves the highest average AUC across four deepfake benchmarks and transfers to presentation attacks, while its reconstruction-aware representation improves class separation over an RGB-only ablation.
3. Experiments
- Training uses frames from four manipulation methods, while evaluation on datasets differing in technique, pipeline, and visual characteristics tests cross-dataset generalization.Performance is measured using AUC.Source: §3 · p. 6
- 0.856 average AUC across four benchmarks gives Spectra the highest average, exceeding TGFD (0.833) by 2.76%; it is second-best on DFD.Spectra is best or second-best on every benchmark and establishes a new state of the art on three; its AUC is 0.829 on CDFv1, 0.841 on CDFv2, and 0.858 on FSh.Source: §3 · p. 6, Table 1 · p. 7
3.1. How Well Does Spectra Separate Classes?
- Silhouette scores improve 2.4–3.2× and Davies–Bouldin scores 1.6–1.9× over the RGB ablation across four benchmarks, with more separable embeddings for Spectra.The t-SNE visualization compares Spectra with the RGB-only ablation on CDFv1; Spectra’s ROC curves are also shown for Celeb-DF v1 and v2.Source: §3.1 · p. 7, Figure 5 · p. 7, Figure 4 · p. 7
3.2. Can Spectra be Extended to Presentation Attack Detection?
- Spectra achieves the best average presentation-attack performance (97.17) across four leave-one-domain-out protocols and the best result on three, trailing on Idiap Replay-Attack.The INR decomposition shows the attacked sample developing a localized bright forehead region by Layer 2, while its Layer 3 structure is less uniform than the genuine sample.Source: §3.2 · pp. 7–8, Figure 6 · p. 8
- Presentation attacks perturb image statistics more globally and less locally than semantic deepfake manipulation, yet reconstruction-aware features generalize to this setting without task-specific architectural modification.The comparison describes distinct layer-wise INR responses for genuine and attacked samples.Source: §3.2 · pp. 7–8
4. Additional Experiments and Ablations
The supplementary material expands the INR architecture and design rationale, implementation details, ablations, and qualitative visualizations; Table 3 identifies the cross-dataset PAD evaluation scope.
4. Additional Experiments and Ablations
- Supplementary material details the INR architecture and design choices, including activation functions, network depth, hidden-layer dimensionality, and hypernetwork-based acceleration.It also provides implementation hyperparameters and descriptions of major architectural components.Source: §4 · p. 8
- Table 3 reports cross-dataset PAD performance as percent AUC across the M, C, I, and O evaluation protocols.Best and second-best results are marked in bold and underlined, respectively.Source: Table 3 · p. 8
- Ablations examine the CLIP backbone, reconstruction-aware heatmaps, multi-level decomposition, multi-scale semantic representations, and loss functions.Additional visualizations illustrate hierarchical INR structure for deepfake and face presentation attack detection.Source: §4 · p. 8
5. Conclusion
Spectra combines INR reconstruction dynamics with CLIP semantics for synthetic-image detection. The conclusion reports state-of-the-art cross-dataset deepfake performance, generalization to presentation attack detection without task-specific redesign, and improved embedding separability.
5. Conclusion
- Spectra combines INR dynamics with CLIP semantics to detect synthetic images, with reported cross-dataset deepfake performance and transfer to presentation attack detection without task-specific redesign.Its learned embeddings show improved class separability, which the authors interpret as evidence that reconstruction-aware features capture signatures transferable across manipulation and attack types.Source: §5 · p. 8