Diversifying Spatial-Temporal Perception for Video Domain Generalization

1 Introduction

• Source-trained video classifiers can rely on domain-specific cues, such as a backboard, that are absent in an unseen target domain, making recognition challenging without target-domain knowledge.The task trains on source videos only, with source and target distributions differing but sharing a label space.Source: §1 · pp. 1–2
• STDN explores spatial-temporal diversity to capture potential domain-invariant cues alongside domain-specific cues and alleviate source-domain overreliance.The approach aims to leverage diverse class-correlated cues from video contents when recognizing target-domain videos.Source: §1 · p. 2
• Spatial Grouping Module discovers groups of spatial cues within individual frames through a clustering-like process.This enriches diversity from the spatial modeling perspective.Source: §1 · p. 2
• Spatial-Temporal Relation Module models dependencies between video contents at multiple space-time scales.This enriches diversity from the spatial-temporal relation modeling perspective.Source: §1 · p. 2
• Experiments on three benchmarks of different types, including two newly designed benchmarks, demonstrate the method's effectiveness and versatility.The supplied introduction does not report benchmark scores.Source: §1 · p. 2

Video Classification

• Video classification architectures model spatial and temporal information in different ways, but typically assume test and training distributions are identical.The surveyed approaches include 3D CNNs, frame-level spatial modeling followed by temporal modeling, temporal shifts, and video Transformers.Source: §2 · pp. 2–3

Video Domain Generalization

• Prior video domain generalization methods assume cross-domain invariance, whereas this work explores diverse class-correlated spatial-temporal cues as an orthogonal approach.Examples include temporal-feature invariance and visual-audio consistency.Source: §2 · p. 3

General Domain Generalization

• General domain generalization includes feature alignment, adversarial learning, invariant risk minimization, meta-learning, augmentation, and ensembles; this work instead targets video-specific space-time diversity.The related work also distinguishes domain generalization from out-of-distribution identification, which it places outside the paper's scope.Source: §2 · p. 3

3.1 Problem Formulation

• Video domain generalization trains on labeled source videos to classify unseen target domains with different distributions but the same label space.Source: §3.1 · p. 3

3.2 Model Overview

• STDN extracts spatial cues from each sampled frame, models spatial-temporal dependencies at multiple scales, and aggregates diverse features for classification.The model overview describes spatial grouping and relation modeling as ways to enrich diversity in spatial and spatial-temporal modeling.Source: §3.2 · p. 4

3.3 Spatial Grouping Module

• The Spatial Grouping Module assigns spatial features to learnable anchors and aggregates each group into a distinct spatial cue, rather than one pooled frame feature.The grouping is described as a clustering-like process over spatial features within individual frames.Source: §3.3 · pp. 4–5
• Two entropy-based losses encourage confident per-feature assignments and spread features across different spatial groups.Entropy minimization sharpens each feature’s assignment, while maximizing the entropy of mean assignments encourages uniform use of the groups.Source: §3.3 · p. 5

3.4 Spatial-Temporal Relation Module

• The Spatial-Temporal Relation Module builds frame-level features from spatial cues and global features, modeling spatial dependencies at multiple space scales.It samples distinct spatial cues at each scale and concatenates the resulting relation features with a global feature.Source: §3.4 · p. 6
• Temporal dependency functions model ordered frame features at different time scales, producing N − 1 temporal relation features per video.For each scale, the model randomly samples frame features while preserving their relative order.Source: §3.4 · p. 6
• A relation discrimination loss trains different time scales to capture distinct class-correlated temporal cues and avoid feature collapse.The relation classifier distinguishes features across classes and also between scales within the same class.Source: §3.4 · p. 6

4.1 Benchmarks and Experimental Setups

• Experiments use source-domain training and validation splits, compare reproduced general and video domain-generalization methods, and report mean and standard deviation over three random trials.The benchmark study includes EPIC-Kitchens-DG and Jester-DG alongside UCF-HMDB.Source: §4.1 · p. 7
• The three benchmarks cover cross-domain video classification, cross-scene egocentric action recognition, and cross-domain hand-gesture recognition.EPIC-Kitchens-DG and Jester-DG are newly designed benchmarks; EPIC-Kitchens-DG has three domains and six subtasks, while Jester-DG has source and target domains.Source: §4.1 · p. 7
• The implementation uses an ImageNet-pretrained ResNet50 backbone, five sampled frames per video, and video-adapted MixStyle feature augmentation.The backbone receives 224 × 224 frames, and MixStyle calculates channel-wise mean and standard deviation across space and time.Source: §4.1 · pp. 7–8

4.2 Results

• STDN outperforms state-of-the-art methods on all three benchmarks, with 2.2% improvement on HMDB→UCF and 2.3% on EPIC-Kitchens-DG.Table 2 covers EPIC-Kitchens-DG and Jester-DG comparisons.Source: §4.2 · p. 8, Table 2 · p. 8
• 3.4% and 2.3% improvement over Backbone+SGM on UCF→HMDB and HMDB→UCF, respectively, follows adding full STRM; MixStyle further improves the full model.Without relation discrimination loss Lrel, temporal dependency modeling yields only very minor improvement.Source: §4.2 · pp. 8–9
• Higher normalized MSE indicates more diverse temporal relation features across time scales; Lrel increases diversity, and spatial grouping and spatial dependency modeling enlarge feature differences further.Without Lrel, temporal relation features at different time scales show very small differences, indicating feature collapse.Source: §4.2 · p. 9, Figure 4 · p. 9
• SGM produces spatial features with better cluster separation than the average-pooling baseline, as assessed qualitatively and by Davies-Bouldin Index.Figure 5 visualizes K-means clusters of spatial features.Source: §4.2 · p. 9, Figure 5 · p. 9
• Grad-CAM examples show STDN using basketball cues in the target domain when the baseline's domain-specific backboard cue is unavailable.The comparison concerns UCF-HMDB; STDN captures both domain-specific and domain-invariant class-correlated cues.Source: §4.2 · pp. 9–10, Figure 6 · p. 10

• STDN learns diverse spatial-temporal features that discover potential domain-invariant cues and alleviate heavy reliance on domain-specific cues.Source: §5 · p. 10
• Experiments on three benchmarks, including two newly designed benchmarks, demonstrate the approach’s effectiveness and versatility.Source: §5 · p. 10