Temporal Attentive Alignment for Large-Scale Video Domain Adaptation

Min-Hung Chen; Zsolt Kira; Ghassan AlRegib; Jaekwon Yoo; Ruxin Chen; Jian Zheng

Source-linked AI summary

Temporal Attentive Alignment for Large-Scale Video Domain Adaptation

Min-Hung Chen, Zsolt Kira, Ghassan AlRegib, Jaekwon Yoo, Ruxin Chen, Jian Zheng

arXiv:1907.12743v6Jul 30, 2019cs.CVcs.LGcs.MMeess.IV

TL;DR

Video domain adaptation is under-explored because prior evaluations largely use small, saturated datasets with limited domain discrepancy. The paper introduces two larger benchmarks, studies spatio-temporal feature alignment, and proposes TA3N, which attends to temporally discrepant dynamics; TA3N achieves state-of-the-art performance across four investigated datasets.

Problem
Video domain adaptation lacks large-scale, challenging benchmarks, while prior datasets have limited domain discrepancy and can become saturated.
Method
The paper creates two large-scale video domain adaptation datasets and develops TA2N and TA3N to align domains while learning temporal dynamics.
Results
TA3N achieves state-of-the-art performance on all four investigated cross-domain video datasets.
Takeaways & Limitations
Effective video domain alignment can be achieved by simultaneously aligning and learning temporal dynamics, without requiring sophisticated domain adaptation methods.
Takeaways & Limitations
The reported work uses a closed-set video adaptation setting; extending TA3N to open-set settings is identified as future work.

Abstract

from arXiv · show

Although various image-based domain adaptation (DA) techniques have been proposed in recent years, domain shift in videos is still not well-explored. Most previous works only evaluate performance on small-scale datasets which are saturated. Therefore, we first propose two large-scale video DA datasets with much larger domain discrepancy: UCF-HMDB_full and Kinetics-Gameplay. Second, we investigate different DA integration methods for videos, and show that simultaneously aligning and learning temporal dynamics achieves effective alignment even without sophisticated DA methods. Finally, we propose Temporal Attentive Adversarial Adaptation Network (TA3N), which explicitly attends to the temporal dynamics using domain discrepancy for more effective domain alignment, achieving state-of-the-art performance on four video DA datasets (e.g. 7.9% accuracy gain over "Source only" from 73.9% to 81.8% on "HMDB --> UCF", and 10.3% gain on "Kinetics --> Gameplay"). The code and data are released at http://github.com/cmhungsteve/TA3N.

1. Introduction

The paper addresses under-explored video domain adaptation by introducing larger, more discrepant benchmarks and aligning temporal dynamics alongside spatial features. It proposes TA2N and TA3N, with TA3N attending to temporally embedded representations that have greater domain discrepancy.

Motivation: Video domain adaptation lacks well-organized benchmarks, while existing datasets have few overlapping categories and limited domain discrepancy.Deep CNNs can achieve nearly perfect performance on some small datasets without domain adaptation.
Feature alignment: Video domain discrepancy occurs across both spatial and temporal directions, but most domain adaptation approaches do not explicitly address temporal shift.The paper therefore investigates which features to align and how to integrate domain adaptation with temporal modeling.
Feature alignment: Aligning temporal-dynamics features outperforms aligning only spatial features, and selecting what to align matters more than the domain adaptation approach.TA2N simultaneously aligns domains and learns temporal dynamics, outperforming approaches that directly apply sophisticated image-based methods to videos.
TA3N: TA3N attends to temporal representations with greater domain distribution discrepancy while simultaneously aligning domains and encoding temporal dynamics.Its attention mechanism focuses alignment on local temporal features contributing more to the overall domain shift.
Results: TA3N achieves state-of-the-art performance on all four investigated video domain adaptation datasets.The datasets include both small- and large-scale cross-domain video benchmarks.
Datasets: The paper introduces UCF-HMDBfull and Kinetics-Gameplay as larger-scale datasets for investigating video domain discrepancy.UCF-HMDBfull contains 12 overlapping categories and is around three times larger than UCF-Olympic and UCF-HMDBsmall; Kinetics-Gameplay contains 30 overlapping categories spanning virtual and real domains.

2. Related Works

Related work covers video representation learning, deep domain adaptation, and the limited literature on video domain adaptation. Existing approaches include distribution discrepancy, adversarial, normalization, ensemble, and attention-based methods, but video adaptation remains comparatively under-explored.

Video Classification: Video classification methods learn compact spatio-temporal representations using CNNs for spatial information and architectures for temporal dynamics.Some methods separate spatial and temporal learning, while others jointly encode spatio-temporal representations.
Domain Adaptation: Deep domain adaptation methods address domain shift by learning common feature spaces for labeled source and unlabeled target branches.The literature combines classification objectives with domain-alignment objectives.
Domain Adaptation: Domain adaptation includes discrepancy-based methods such as MMD and CORAL, adversarial methods, normalization-based methods, and ensemble-based methods.These approaches reduce or model differences between source and target feature distributions in different ways.
Domain Adaptation: Attention-based adaptation has been used to select transferable regions, and this paper extends the idea to important parts of temporal dynamics.The extension targets spatio-temporal domains rather than image regions alone.
Video Domain Adaptation: Video domain adaptation remains under-explored, with prior work concentrated on small-scale datasets containing few overlapping categories.Earlier methods reduce background effects, learn common feature spaces, or adapt pre-extracted C3D features on a Grassmann manifold.

3. Technical Approach

The approach progresses from a temporal-pooling adversarial baseline to explicit temporal-relation alignment and then domain-discrepancy-guided attention. TA3N combines these components to align temporal dynamics while learning video representations.

Baseline Model: The baseline extends image-based unsupervised domain adaptation to videos by mean-pooling frame features and adding adversarial domain discriminators.TemPooling produces video-level features from frame-level representations; gradient reversal connects domain classifiers to the main model.
Integration of Temporal Dynamics with DA: TemRelation explicitly encodes temporal dynamics by combining time-ordered frame representations across multiple temporal scales.An MLP fuses n-frame relations, and the resulting relation features are summed into the final video representation.
Integration of Temporal Dynamics with DA: TA2N integrates relation discriminators inside the temporal module so domain alignment occurs while temporal dynamics are learned.Each n-frame relation receives a corresponding relation discriminator because different relations represent different temporal characteristics.
Temporal Attentive Alignment for Videos: TA3N weights local temporal features using domain attention derived from relation-discriminator predictions and entropy.The attention emphasizes features with larger domain discrepancy, while a residual connection supports stable optimization.
Temporal Attentive Alignment for Videos: TA3N additionally uses attentive entropy regularization to minimize classifier entropy primarily for videos that are similar across domains.The overall objective combines class prediction, spatial, relation, temporal, and attentive entropy losses with tunable weights.

4. Datasets

The paper creates two cross-domain video datasets to address the scarcity of large-scale benchmarks: UCF-HMDBfull and Kinetics-Gameplay. They span real and virtual domains and provide larger, more challenging settings for evaluating video domain adaptation.

The authors create UCF-HMDBfull and Kinetics-Gameplay as two cross-domain datasets for evaluating video domain adaptation.These datasets address the limited availability of benchmark datasets for video DA.
UCF-HMDBfull: UCF-HMDBfull contains 12 overlapping categories and more than 3000 video clips, making it around three times larger than UCF-HMDBsmall and UCF-Olympic.
Kinetics-Gameplay: Kinetics-Gameplay combines real Kinetics-600 videos with virtual Gameplay videos collected from Detroit: Become Human and Fortnite.
Kinetics-Gameplay: Kinetics-Gameplay includes around 50K video clips across both domains.

5. Experiments

Experiments evaluate video DA on four datasets and show that temporal-dynamics-aware alignment consistently improves cross-domain classification. TA3N achieves state-of-the-art results through domain attention and spatio-temporal alignment.

Experimental Setup: The experiments evaluate DA approaches on UCF-Olympic, UCF-HMDBsmall, UCF-HMDBfull, and Kinetics-Gameplay.The self-collected datasets use labeled source training data, unlabeled target training data, and target-domain validation data.
UCF-Olympic and UCF-HMDBsmall: 6.5% and 9% absolute accuracy differences separate the proposed approach from previous methods on U →O and U →H, respectively.The reported accuracies are 98.15% versus 91.60% for U →O and 99.33% versus 90.25% for U →H.
UCF-HMDBfull: Temporal-relation features outperform temporal pooling for the same DA method, including gains of 3.61% versus 0.83% on U →H and 2.45% versus 0.17% on H →U with DANN.These results support applying DA to representations that encode temporal dynamics.
UCF-HMDBfull: TA2N reaches 77.22% with a 5.55% gain on U →H and 80.56% with a 6.66% gain on H →U by explicitly aligning temporal dynamics.TA2N outperforms methods extended from more sophisticated DA approaches that do not follow this integration strategy.
UCF-HMDBfull: TA3N reaches 78.33% with a 6.66% gain on U →H and 81.79% with a 7.88% gain on H →U, achieving state-of-the-art accuracy and gain.Its domain-attention mechanism attends to temporal dynamics using domain discrepancy.
Kinetics-Gameplay: 47.27% is the accuracy gap between “Source only” and “Target only” on Kinetics-Gameplay, where TA3N raises “Source only” accuracy from 17.22% to 27.50%.Kinetics-Gameplay is more challenging than UCF-HMDBfull because it spans real and virtual domains with more severe domain shifts.
Ablation Study and Analysis: Integrating relation discrimination improves performance by 0.92%, while combining all three discriminators improves TA2N by 4.20%.Temporal-relation discrimination simultaneously aligns and learns temporal dynamics across domains.
Ablation Study and Analysis: TA3N reduces MMD from 0.184 to 0.0842 and increases domain loss from 1.116 to 1.9286 relative to TemPooling.Lower MMD and higher domain loss indicate a smaller domain gap under the reported measures.

6. Conclusion and Future Work

The paper introduces large-scale real-and-virtual video DA datasets and TA3N, showing that jointly aligning and learning temporal dynamics is effective across the investigated datasets. Future work targets additional video tasks and open-set adaptation.

Conclusion: The paper presents UCF-HMDBfull and Kinetics-Gameplay to investigate video domain shift across real and virtual domains.These datasets address an under-explored video DA problem.
Conclusion: Simultaneously aligning and learning temporal dynamics achieves effective alignment without requiring sophisticated DA methods.TA3N extends this strategy by attending to, aligning, and learning temporal dynamics across domains.
Future Work: Future work proposes applying TA3N to video captioning, segmentation, and detection, and extending it to the open-set setting.The open-set setting has different categories between source and target domains and is described as more challenging and closer to real-world scenarios.

7. Supplementary

The supplementary material provides additional ablations, implementation details, and a complete introduction to the datasets.

Supplementary: The supplementary material includes more detailed ablation studies, implementation details, and a complete introduction of the datasets.

7.1. Visualization of distribution

The t-SNE visualizations compare source and target distributions and show how temporal-relation modeling improves clustering and cross-domain generalization relative to temporal pooling.

Distribution Visualization: Figures 6a and 6b show that TemPooling poorly aligns source and target distributions, even with image-based DA methods.Source data are shown in blue and target data in orange.
Distribution Visualization: The temporal-relation module groups source data into denser clusters but does not by itself fully generalize the distribution into the target domain.

7.2. Domain Attention Mechanism

Domain attention improves video domain adaptation by emphasizing domain-relevant features, benefiting both TemPooling and TemRelation architectures across adversarial discriminator choices.

Domain attention improves performance for both TemPooling and TemRelation architectures across all tested adversarial discriminators.The mechanism can benefit video DA even when the backbone does not explicitly encode temporal dynamics.
TemPooling applies domain attention to raw frame features.
Accuracy evaluations compare different positions for integrating the domain discriminator on U →H and H →U.

7.3. Implementation Details

The implementation uses frame-level CNN features sampled sparsely over time, while the baseline and proposed architectures integrate temporal and adversarial domain-alignment components.

TA3N architecture: TA3N generates n-frame relation features, weights them using domain predictions, and applies a residual attentive entropy loss.The method attends to relation features with domain attention and weights class entropy using domain entropy.
Baseline architecture: The baseline TemPooling architecture uses spatial and temporal adversarial discriminators positioned after the corresponding modules.
Implementation: The implementation uses an ImageNet-pretrained ResNet-101, samples five equally spaced frame features per video, and starts optimization with learning rate 0.03.
Compared methods: JAN adds JMMD to the final video representation and class prediction, while AdaBN uses separate source and target normalization statistics.
Compared methods: MCD adds a second classifier and iteratively optimizes generators and classifiers using classifier discrepancy.

7.4. Datasets

The paper evaluates four cross-domain video datasets, including two newly collected large-scale datasets designed to expose substantial domain discrepancy.

UCF-HMDBfull: UCF-HMDBfull contains 12 overlapping categories from UCF101 and HMDB51, with 1,438 UCF training videos and 840 HMDB training videos.The dataset also includes 571 UCF and 360 HMDB validation videos.
Kinetics-Gameplay: Kinetics-Gameplay combines 30 overlapping categories from Kinetics-600 and gameplay videos collected from Detroit: Become Human and Fortnite.Gameplay clips are split into training, validation, and testing sets at a 7:2:1 ratio.
Dataset examples: Figures show example category snapshots for UCF-HMDBfull and example screenshots contrasting Gameplay and Kinetics videos.

7.5. More Details

Additional analyses identify conditions affecting adaptation methods, including data imbalance, temporal consistency, and computational overhead.

JAN on Kinetics-Gameplay: JAN performs poorly on Kinetics-Gameplay because source-target imbalance makes MMD align only part of the source batch with the target batch.The resulting model remains overfitted to the source domain, whereas MMD works better with balanced data.
Evaluation protocol: The UCF-HMDBsmall comparison excludes evaluations using target-labeled fine-tuning because those settings violate unsupervised DA.
Temporal information: The paper uses RGB frames to attend to temporal dynamics, while treating optical flow as complementary rather than required.
Comparison with self-supervision: Cycle-consistency methods learn general representations from unlabeled videos, whereas this approach explicitly aligns source and target feature spaces.
Failure cases for TemRelation: TemRelation offers limited improvement over TemPooling for temporally consistent actions, including equal 97% ride-bike accuracy and lower ride-horse accuracy.
Testing cost: TA3N requires all domain discriminators during testing but adds only 10% more computation than TA2N.