AIGC Video Detection based on the fusion of spatial-frequency-optical flow multimodal features

1 Introduction

• AIGC’s mass production can facilitate public deception, motivating detection to protect public trust and information integrity.The introduction situates AIGC forgery detection within the broader scope of video forgery.Source: §1 · pp. 1–2
• The framework’s central claim is that cross-modal inconsistency is a more generalizable and robust footprint than artifacts found in any single modality.The approach examines appearance and motion together rather than treating each perspective in isolation.Source: §1 · p. 2
• CrossAtt-VFD uses cross-attention as an inconsistency diagnostic engine that models the asymmetric relationship between spatial-frequency appearance and optical-flow motion features.The paper describes the module as more than a fusion tool.Source: §1 · p. 2
• 94.32% accuracy, 91.67% precision, and 96.25% recall are reported on a dataset spanning multiple state-of-the-art generative models, with enhanced generalization against unseen models.The paper presents these results as evidence of the framework’s performance and generalization capability.Source: §1 · p. 2

2.1 Common flaws in generative videos

• The paper treats predictable shortcomings in generated videos as common artifacts that can support a generalizable detector.It groups these flaws into physical-law violations, temporal-coherence defects, and abnormal frequency-domain characteristics.Source: §2.1 · p. 2, §2.1.3 · p. 3

2.1.1 Violation of physical laws

• Generated videos may show erratic motion or implausible occlusion relationships, including foreground objects incorrectly occluded by background elements.The passage links geometric inconsistencies to perspective or occlusion violations and notes that panoptic-aware depth estimation can identify such artifacts.Source: §2.1.1 · p. 2

2.1.2 Defects in temporal coherence

• Temporal-coherence defects include abrupt inter-frame mutations and longer-range scene discontinuities.Examples include a hand suddenly appearing on another person’s shoulder and a scene jumping from indoors to outdoors without a coherent transition.Source: §2.1.2 · pp. 2–3

2.1.3 Abnormal frequency-domain characteristics

• Generated content can lose high-frequency detail and show unnatural color banding, creating frequency-domain deficiencies that the paper treats as part of its AIGC fingerprint.The described effects include blurred edges, diminished texture, unnaturally smooth appearance, and banding where gradients should be smooth.Source: §2.1.3 · p. 3

3 Methodology

• The framework integrates spatial appearance, frequency-domain signatures, and optical-flow motion to detect AIGC videos.It is designed to cover artifacts ranging from static texture anomalies to dynamic physical implausibilities.Source: §3 · p. 3
• Combining complementary modalities is intended to support a more comprehensive and generalizable representation than any single perspective provides.The framework uses the joint information to identify a broader spectrum of artifacts.Source: §3 · p. 3
• The targeted artifacts span static texture anomalies and dynamic physical implausibilities.These examples connect appearance-based and motion-based evidence within the detection task.Source: §3 · p. 3

3.1 Multi-modal framework overview

• CrossAtt-VFD uses three synergistic modalities—spatiotemporal appearance, frequency-domain signatures, and optical flow—to mine inconsistencies between real and AIGC videos.The framework is organized as parallel spatial-frequency and optical-flow branches.Source: §3.1 · p. 3
• The spatial-frequency branch extracts frame features and models their temporal evolution, while the optical-flow branch encodes motion trajectories and their temporal dynamics.The overview figure presents the two branches in parallel.Source: §3.1 · p. 4, Figure 2 · p. 4
• Cross-attention lets spatial-frequency features query optical-flow features, adapting motion features to appearance before the fused representation produces a detection score.The optical-flow attention weights represent the importance of motion-feature dimensions relative to spatial-frequency features.Source: §3.1 · p. 4

3.2 Spatial feature extraction module

• ResNet-50 is selected over ResNet-18 because preliminary experiments found higher accuracy and F1-scores, with faster convergence and better generalization despite slightly higher memory use.The choice targets a lightweight detector deployable on standard office equipment.Source: §3.2 · p. 5
• A pre-trained ResNet extracts spatial features independently from each frame, which are then stacked along the temporal dimension for video-level representation.The framework uses the features to model temporal dependencies with a BiLSTM.Source: §3.2 · p. 5
• CBAM enhances salient visual patterns and suppresses noise before a BiLSTM models temporal dynamics and long-range dependencies in the refined feature sequence.The module addresses redundant noise and subtle generation artifacts in the extracted spatial features.Source: §3.2 · p. 5

3.3 Frequency-domain feature extraction module

• The frequency module applies FFT to video frames and uses convolutional feature learning to detect spectral anomalies, including high-frequency detail loss.Its input is a video frame sequence with dimensions [B, T, C, H, W].Source: §3.3 · p. 5
• Each frame is reshaped for independent analysis, transformed into a complex-valued spectrum, shifted to center low frequencies, and represented by a magnitude spectrum.The magnitude spectrum has shape [B × T, 1, 128, 128], with each element representing frequency-component energy intensity.Source: §3.3 · pp. 5–6
• Four spectral statistics are extracted: total energy, high-frequency energy ratio, frequency-domain entropy, and central low-frequency ratio.They characterize overall spectral energy, relative high-frequency content, spectral-energy randomness, and low-frequency concentration, respectively.Source: §3.3 · p. 6
• The four statistics are stacked into a [B × T, 4] tensor and projected to frequency-domain feature vectors of dimensions [B × T, 128].The high-frequency ratio uses a distance-thresholded mask, while the central low-frequency ratio uses a mask within 0.4dmax.Source: §3.3 · p. 6

3.4 Optical flow feature extraction module

• Classical optical-flow algorithms are used instead of learning-based methods because they are computationally efficient and require no annotated training data.The paper says this choice suits its hardware constraints and provides a robust estimate of motion inconsistencies.Source: §3.4 · p. 7
• The branch converts frames from RGB to grayscale and uses OpenCV’s DIS algorithm to compute dense optical flow between consecutive frames.In the visualizations, hue represents motion direction and intensity represents magnitude or speed.Source: §3.4 · p. 7
• Generated-video optical flow often shows fragmented motion vectors at object boundaries or abrupt intensity changes, unlike the smooth, continuous transitions described for real video.These motion artifacts are presented as complementary diagnostic clues alongside spatial-frequency features.Source: §3.4 · p. 7, Figure 3 · p. 7
• A CNN encodes normalized optical-flow fields, and a two-layer-default BiLSTM models dependencies across the motion sequence.The BiLSTM outputs temporally encoded features with dimensions [B, T −1, 2dh], where dh = 256.Source: §3.4 · p. 8

3.5 Cross-modal inconsistency mining via cross-attention

• Cross-attention aligns spatial-frequency and optical-flow features, selecting salient complementary components for video classification.Because the feature streams have temporal lengths T and T −1, linear projections first match their dimensions; the attention output is added to projected spatial-frequency features and normalized.Source: §3.5 · p. 8, §3.5.2 · p. 8

3.5.1 Intuition of asymmetric diagnosis

• Appearance context queries optical-flow motion evidence, encoding the premise that appearance governs plausible motion.Spatial-frequency features provide the query; optical-flow features provide the key and value.Source: §3.5.1 · p. 8

3.5.2 Diagnostic process

• Low cross-attention compatibility marks regions where observed motion is implausible given the appearance context.The weighted output forms appearance-rectified motion features for detecting generated content.Source: §3.5.2 · p. 8

3.5.3 Advantage over baselines

• Unlike concatenation or additive fusion, the method emphasizes modality disagreement rather than assuming equal reliability or direct alignment.It is designed to use the modalities’ points of disagreement.Source: §3.5.3 · p. 8

4.1 Experimental purpose; 4.3 Experimental setup; 4.4 Experiment design comparison

• The evaluation targets real-versus-AIGC detection using accuracy, precision, recall, and F1-score, with training and comparisons conducted under controlled conditions.The implementation uses spatial, frequency, and optical-flow encoders, with LSTMs for temporal modeling and attention for feature enhancement.Source: §4.1 · p. 9, §4.3 · p. 9, §4.4 · p. 10

4.2 Dataset construction

• The hybrid dataset combines proprietary and public videos, with 1,312 real and 1,312 fake training videos and 640 real and 703 fake test videos.Fake videos include GenVidBench samples and official Sora, Kling, and Hunyuan demonstrations; real videos include HAIC and Mixkit clips.Source: §4.2 · p. 9, Table 1 · p. 9

4.4.2 Comparative experiment; 4.4.3 Ablation experiment comparison

• The baseline comparison includes CNN-LSTM-Attention, EfficientFormer, and ViTranSP, while single-modality baselines establish reference performance for subsequent fusion.The modality ablation compares spatial-only, spatial plus frequency, spatial plus optical flow, and full tri-modal fusion under shared hyperparameters and classification heads.Source: §4.4.2 · p. 10

4.5.1 Results of a comparative experiment on different data loading methods

• Pre-screened keyframes outperformed complete-video input by 0.8% accuracy, 0.55% precision, 1.25% recall, and 0.89% F1-score, and cut epoch time by 112 s.Both methods converged without signs of overfitting; the authors attribute the keyframe advantage to filtering redundant or static frames.Source: Table 2 · p. 11, §4.5.1 · p. 11, Figure 4 · p. 11

4.5.2 Comparative experiment results

• 94.32% validation accuracy—4.55% above ViTranSP (89.77%)—and 91.67% precision were reported for CrossAtt-VFD against representative SOTA baselines.The comparison also reports greater convergence stability than CNN-LSTM-Attention and EfficientFormer.Source: Table 3 · p. 11, §4.5.2 · p. 12, Figure 5 · p. 12

4.5.3 Ablation experiment comparison results

• The full tri-modal model achieved 94.32% accuracy and 93.90% F1-score, while ablations associated frequency with recall gains and optical flow with precision gains.Spatial-only achieved 93.30% accuracy and 92.74% F1-score; spatial plus frequency reached 97.25% recall, while spatial plus optical flow reached 93.27% precision.Source: §4.5.3 · pp. 13–14

4.6 Model’s effect display in the test set

• On the held-out set, accuracy was 86% for unseen Jieyue videos, while latest-model results indicated a need for improved detection accuracy.Without fine-tuning on DVF, accuracy was 97.03% for “Stable video” and 90.67% for VideoCrafter1; stable video diffusion image-to-video accuracy was 36.61%, attributed to a distributional shift absent from training.Source: §4.6 · pp. 14–15, Figure 7 · p. 14, Figure 8 · p. 15

• Cross-attention correlates spatial-frequency features with optical-flow patterns to highlight discrepancies and violations of physical motion constraints.The authors report stronger diagnostic capabilities than simple feature-fusion techniques.Source: §5 · p. 15
• 94.32% accuracy on the benchmark dataset, with significant improvement over obsolete single-modality or simple-fusion baselines.The full model's evaluation also finds that each modality contributes and cross-attention enhances their performance.Source: §5 · p. 15
• Motion-entropy-based keyframe selection is used to improve computational efficiency.Source: §5 · p. 15
• Future work will add data from emerging generators and explore few-shot or semi-supervised active learning, alongside refining cross-attention for real-time authentication.The authors identify scarcity of samples for the latest generators as a motivation for adapting with limited annotated data.Source: §5 · p. 15