Source-linked AI summary
VideoMoCo: Contrastive Video Representation Learning with Temporally Adversarial Examples
Tian Pan, Yibing Song, Tianyu Yang, Wenhao Jiang, Wei Liu
TL;DR
Unsupervised video representation learning needs task-agnostic features beyond pretext tasks designed around individual video properties. VideoMoCo extends MoCo with adversarial frame dropout and temporal decay for stale queue keys, and is reported as state-of-the-art on standard benchmarks.
Problem
Unsupervised video representation learning seeks task-agnostic features beyond pretext tasks designed around individual video properties.
Method
VideoMoCo extends MoCo with adversarial temporal frame dropout and temporal decay that reduces older keys’ contributions in the contrastive queue.
Results
VideoMoCo is reported as a state-of-the-art video representation learning method on standard benchmarks including UCF101 and HMDB51.
Takeaways & Limitations
VideoMoCo learns temporally robust video representations without empirically designing pretext tasks.
Takeaways & Limitations
Adversarial learning is difficult for the discriminator to converge when applied at the initial training stage.
Abstract
from arXiv · showhide
MoCo is effective for unsupervised image representation learning. In this paper, we propose VideoMoCo for unsupervised video representation learning. Given a video sequence as an input sample, we improve the temporal feature representations of MoCo from two perspectives. First, we introduce a generator to drop out several frames from this sample temporally. The discriminator is then learned to encode similar feature representations regardless of frame removals. By adaptively dropping out different frames during training iterations of adversarial learning, we augment this input sample to train a temporally robust encoder. Second, we use temporal decay to model key attenuation in the memory queue when computing the contrastive loss. As the momentum encoder updates after keys enqueue, the representation ability of these keys degrades when we use the current input sample for contrastive learning. This degradation is reflected via temporal decay to attend the input sample to recent keys in the queue. As a result, we adapt MoCo to learn video representations without empirically designing pretext tasks. By empowering the temporal robustness of the encoder and modeling the temporal decay of the keys, our VideoMoCo improves MoCo temporally based on contrastive learning. Experiments on benchmark datasets including UCF101 and HMDB51 show that VideoMoCo stands as a state-of-the-art video representation learning method.
1. Introduction
VideoMoCo adapts MoCo for unsupervised video representation learning by addressing temporal robustness within samples and stale keys across samples. It avoids empirically designed pretext tasks and achieves favorable benchmark performance.
- Approach: VideoMoCo combines temporal augmentation within each sample with temporally contrastive learning across different samples.This extends MoCo’s queue structure and moving-averaged encoder to video representation learning.
- Motivation: Contrastive video representation learning has received less attention than static-image learning, while existing video methods often rely on task-specific pretext tasks.The paper targets task-agnostic video representations using contrastive learning.
- Approach: VideoMoCo introduces temporally adversarial learning, using a generator to drop frames and train an encoder to produce robust representations.The remaining-frame sample is contrasted with keys in the memory queue.
- Approach: Temporal decay reduces contributions from older memory-queue keys whose representations become less up-to-date after momentum-encoder updates.The decay emphasizes recent keys during contrastive learning.
- Results: Experiments on action-recognition benchmarks show that VideoMoCo performs favorably against state-of-the-art video representation methods.The authors summarize the method as a state-of-the-art video representation learning approach.
2. Related Work
Related work spans self-supervised video representation learning, contrastive learning for images and videos, and generative adversarial learning. VideoMoCo differs by integrating adversarial temporal augmentation into contrastive video learning.
- Self-supervised video learning: Self-supervised video representation learning commonly exploits temporal coherence or constructs pretext tasks such as future prediction, pace prediction, and frame color estimation.These methods learn video features through temporal properties or manually specified tasks.
- Contrastive learning: Contrastive learning minimizes distances between positive pairs and maximizes distances between negative pairs during representation learning.MoCo improves this framework with a momentum encoder and dynamic dictionaries.
- Contrastive learning: SimCLR evaluates augmentation combinations, BYOL uses only positive samples, and mutual-information objectives provide another contrastive-learning perspective.These approaches extend the broader literature on contrastive representation learning.
- Generative adversarial learning: Generative adversarial networks use competing generator and discriminator subnetworks, with the generator synthesizing outputs and the discriminator distinguishing real from generated data.VideoMoCo instead uses adversarial learning to augment video training data and train a temporally robust encoder.
3. VideoMoCo
VideoMoCo extends MoCo for unsupervised video representation learning with temporally adversarial frame dropout and temporal decay for queued keys. These mechanisms target temporal robustness and key staleness, with visualizations showing more confident and motion-focused representations than MoCo.
- 3.1. MoCo Overview: VideoMoCo retains MoCo’s memory queue and moving-average encoder while adapting contrastive learning to fixed-length video clips.The encoder produces queries, keys remain in a queue, and the momentum encoder is updated after contrastive learning.
- 3.2. Temporally Adversarial Learning: Temporally adversarial learning uses a generator to remove frames and trains the encoder to produce similar features for original and altered clips.The generator adaptively drops frames, while the encoder acts as a discriminator against this temporal attack.
- 3.2. Temporally Adversarial Learning: The generator removes 25% of high-importance frames while preserving the query clip’s feature-map size.The remaining frames form xquery through temporal dropout, and the removed-frame content is retained as empty content within the original-sized representation.
- 3.2. Temporally Adversarial Learning: Adversarial training is delayed until the discriminator reaches a semi-stable state because applying it initially makes convergence difficult.Training first uses contrastive learning without the generator, then alternates adversarial updates for G and D.
- 3.3. Temporal Decay: Temporal decay downweights older queue keys because momentum-encoder updates make their representations less aligned with current inputs.For key ki, the decay ti decreases as its queue age i increases, reducing the contribution of historical keys.
- 3.4. Visualization: VideoMoCo’s visualizations show lower entropy under standard and partially occluded inputs, correct classification on an occluded BalanceBeam sequence, and stronger attention to motion regions.Compared with MoCo, VideoMoCo’s features degrade less under temporal occlusion and concentrate more on motion areas such as an eyebrow-pencil tip and hand region.
4. Experiments
Experiments evaluate VideoMoCo through implementation details, ablations, temporal-decay analysis, and comparisons on UCF101 and HMDB51. Temporally adversarial learning and temporal decay improve MoCo, while VideoMoCo performs favorably against existing self-supervised approaches.
- Experimental Setup: Experiments use K-400 for pretext training and UCF101 and HMDB51 for downstream action-recognition evaluation.The downstream setup uses training split 1 for finetuning and testing split 1 for evaluation on UCF101 and HMDB51.
- Ablation Studies: The ablation compares baseline MoCo, random dropout, adaptive adversarial dropout, and the combination with temporal decay.The study varies dropout amount k and evaluates temporal decay values t within the MoCo pretext-training process.
- Temporally Adversarial Training: 75.9%/46.2% versus 77.8%/49.1% shows adversarial learning improves recognition on both datasets, whereas random dropout degrades baseline performance.The highest performance occurs at k = 8; smaller or larger dropout amounts upset the adversarial-learning balance.
- Temporal Decay: t = 0.99999 achieves the best temporal-decay performance.This setting retains contributions from all keys while halving the contribution of the oldest key, with t^65536 ≈ 0.52.
- Comparison with State-of-the-Art Approaches: VideoMoCo outperforms DPC by 5.9% on UCF101 and 9.1% on HMDB51 under the same 3D-ResNet18 architecture and similar input size.Using R(2+1)D, VideoMoCo achieves the premier performance reported in the comparison.
5. Concluding Remarks
VideoMoCo improves MoCo for self-supervised video representation learning by adding temporally adversarial learning and temporal decay. Experiments on standard action-recognition datasets show improved MoCo performance and favorable comparison with existing approaches.
- Concluding Remarks: VideoMoCo uses a generator to augment videos by dropping frames and trains the encoder as a discriminator for temporally robust representations.The generator performs adaptive frame dropout during adversarial learning, while contrastive learning trains the discriminator.
- Concluding Remarks: Temporal decay reduces the contribution of older memory-queue keys because their representations attenuate after momentum-encoder updates.The method therefore emphasizes more recent keys during contrastive learning.
- Concluding Remarks: Experiments on UCF101 and HMDB50 show that VideoMoCo improves MoCo and performs favorably against state-of-the-art self-supervised video representation methods.The conclusion describes the learned feature backbone as effective for downstream recognition tasks.