Source-linked AI summary
Evolving Losses for Unsupervised Video Representation Learning
AJ Piergiovanni, Anelia Angelova, Michael S. Ryoo
TL;DR
Video representation learning needs transferable features without expensive video annotation. The paper combines multimodal self-supervision, distillation, evolved loss weighting, and unlabeled power-law matching to learn a single RGB representation. This representation outperforms prior unsupervised methods and several labeled-pretraining baselines, though large fully labeled video data remains strongest in one comparison.
Problem
Learning transferable video representations is difficult because videos are expensive to collect and annotate, while useful representations are needed for recognition and other perception tasks.
Method
The method combines multimodal, multitask self-supervision with distillation, evolutionary loss-function search, and power-law distribution matching as an unlabeled objective.
Results
The zero-label representation outperforms ImageNet and small or medium labeled-video pretraining, outperforms prior unsupervised methods, and is surpassed only by fully annotated large-scale Kinetics pretraining in the reported comparison.
Takeaways & Limitations
A single RGB network can retain multimodal information and provide powerful transferable video features without additional annotations.
Takeaways & Limitations
The evaluation uses randomly sampled YouTube videos and assumes no overlap with evaluation datasets; the method’s unsupervised Zipf matching also assumes equal cluster variance and prior.
Abstract
from arXiv · showhide
We present a new method to learn video representations from large-scale unlabeled video data. Ideally, this representation will be generic and transferable, directly usable for new tasks such as action recognition and zero or few-shot learning. We formulate unsupervised representation learning as a multi-modal, multi-task learning problem, where the representations are shared across different modalities via distillation. Further, we introduce the concept of loss function evolution by using an evolutionary search algorithm to automatically find optimal combination of loss functions capturing many (self-supervised) tasks and modalities. Thirdly, we propose an unsupervised representation evaluation metric using distribution matching to a large unlabeled dataset as a prior constraint, based on Zipf's law. This unsupervised constraint, which is not guided by any labeling, produces similar results to weakly-supervised, task-specific ones. The proposed unsupervised representation learning results in a single RGB network and outperforms previous methods. Notably, it is also more effective than several label-based methods (e.g., ImageNet), with the exception of large, fully labeled video datasets.
1. Introduction
The paper targets transferable video representations without costly annotation by combining self-supervised tasks across modalities and automatically evolving their losses using an unlabeled distribution-matching objective.
- Video representation learning supports recognition and other high-level perception tasks, but video annotation is costly because videos are high-dimensional and require ambiguous temporal labels.
- The framework combines self-supervised tasks with synchronized multimodal distillation, transferring information from modalities such as grayscale, optical flow, and audio into one RGB network.Tasks include frame ordering and video-audio alignment, and the losses require no human supervision.
- An evolutionary algorithm learns task and distillation-loss weights, while power-law distribution matching with KL constraints supplies an entirely unlabeled fitness signal.The matching objective replaces task-specific labels for evaluating candidate loss combinations.
- The resulting single RGB representation is intended to retain multimodal information while enabling faster representation computation and transfer to recognition tasks.
- Zero-label representations outperform ImageNet and small or medium labeled-video pretraining, rivaling fully labeled Kinetics except at its largest scale.After fine-tuning with Kinetics labels, the proposed representations also outperform Kinetics training.
2. Related Work
Prior work uses temporal or spatial video structure and multi-task self-supervision, but this paper extends unsupervised representation learning to multimodal inputs with automatic discovery of useful tasks.
- Existing video methods exploit temporal structure through tasks such as frame ordering, reversal, shuffling, and cross-frame color consistency.
- Multi-task self-supervised methods report promising results but generally assign equal task weights and do not combine modalities.
- The paper generalizes unsupervised representation learning to multimodal inputs and automatically discovers tasks that benefit recognition performance.
- Large activity-recognition datasets such as Kinetics and Moments in Time enable more accurate deep video CNNs, which the paper seeks to improve through unsupervised representation learning.
3. Method
The method combines self-supervised tasks across modalities, transfers their information into a single RGB network, and evolves the loss weights without labels. It evaluates candidate representations by matching cluster distributions to a Zipf prior.
- Loss evolution: Loss evolution searches task and distillation weights automatically instead of relying on manually constructed combinations.Each weight is constrained to [0, 1], and evolutionary individuals represent complete loss-weight configurations.
- Unsupervised multi-modal learning: The final objective combines per-modality task losses with distillation losses, producing a single RGB network that captures information from multiple tasks and modalities.The weighted sum trains the entire model, while distillation infuses audio, optical-flow, and temporal information into the RGB stream.
- Loss evolution: Figure 3 tracks how weights for modalities, self-supervised tasks, and layer-specific distillation change during evolutionary search.The symbols encode modalities, tasks, and the layer receiving each distillation loss.
- Distillation: Distillation matches activations between modality-specific networks and the main RGB network to transfer cross-modal features during joint training.The distillation loss is an L2 difference between corresponding activations, and the final distillation is performed into the RGB stream.
- Unsupervised Zipf distribution matching: The evolutionary fitness measure trains each candidate on unlabeled videos, clusters its representations, and compares the resulting cluster distribution with a Zipf prior.The method uses k-means-derived cluster probabilities and a KL-divergence objective to evaluate candidate losses.
4. Experiments
Experiments show that evolved, unsupervised representations trained on random unlabeled videos outperform prior unsupervised methods and several supervised baselines, while scaling benefits from more data and training.
- 4.1. Datasets: ELo reaches 67.4% and 93.8% on HMDB, outperforming ImageNet pretraining at 49.8% and 84.5% while approaching Kinetics pretraining at 74.3% and 95.1%.The comparison reports the two evaluation values in the supplied results table, while the accompanying discussion states that ELo outperforms ImageNet and is comparable to labeled Kinetics pretraining.
- 4.2. Comparison to previous methods: ELo outperforms prior state-of-the-art methods on HMDB51 and UCF101 despite using random, less task-aligned unlabeled video clips.The authors contrast their random clips with prior methods trained on activity-recognition datasets such as Kinetics.
- 4.2. Comparison to previous methods: Distillation is extremely important because it lets the single RGB stream receive information from high-performing flow, shuffling, and other modality-specific tasks.Without distillation, the RGB network can access other modalities only through a limited set of alignment tasks.
- 4.3. Improving supervised learning: 58.1% accuracy is obtained using only 25k labeled Kinetics samples, and performance matches the full-data baseline with about 120k samples, or roughly half the dataset.The full labeled set contains 225k samples; using all samples outperforms the baseline network, attributed to better initialization and modality distillation.
- 4.2. Comparison to previous methods: Adding more unlabeled data improves HMDB performance even when iterations stay fixed, while fixing epochs shows that combining more data with more training is best.With fixed iterations, additional data reduces epochs per sample but still improves performance; with fixed epochs, additional data increases iterations and improves it further.
- 4.2. Comparison to previous methods: Evolution assigns low weights to RGB frame shuffling and high importance to audio alignment, indicating that task selection and weighting matter for recognition.The evolved loss weights are examined across 250 evolution rounds, and the final learned loss is visualized in the accompanying figures.
- 4.2. Comparison to previous methods: CMA-ES converges fastest and reaches the best fitness, while Zipf matching strongly correlates with the alternative fitness measure.This supports Zipf matching as a suitable unsupervised representation-evaluation criterion.
5. Conclusion
The paper concludes that a unified multi-task, multi-modal unsupervised framework produces strong video representations, with evolutionary loss weighting guided by an unsupervised fitness measure.
- 5. Conclusion: The learned representations outperform prior self-supervised tasks and can match or improve networks trained with supervised data.The conclusion attributes this result to the unified framework and evolved weighting of self-supervised tasks and modalities.
A. Datasets
Evaluation spans Kinetics, HMDB, and UCF101, showing that the unsupervised representation generalizes across large and small recognition datasets and limited-label settings.
- A. Datasets: The representation works well on both large-scale and smaller video datasets, including when labeled data is limited.The evaluated datasets cover Kinetics, HMDB, and UCF101, with substantially different sizes and activity-class counts.
B. Visualization of loss evolution
The t-SNE visualization shows that ELo produces clearer video clusters than random and ImageNet representations, approaching supervised Kinetics representations.
- B. Visualization of loss evolution: ELo generates clearer video clusters than random weights and ImageNet weights, with clustering more comparable to supervised Kinetics features.
C. Supplemental Results
ELo learns filters that resemble those from labeled-data training while differing from them in some respects. Supplemental visualizations compare its learned representations with random, ImageNet, Kinetics, and prior self-supervised networks.
- ELo filters are quite similar to those learned with labeled data, although they also show some differences.
- t-SNE embeddings compare HMDB test-video representations from random, ImageNet-trained, Kinetics-trained, and ELo networks, with colors denoting activities.
- Filter visualizations compare random initialization, standard supervised learning, AVTS self-supervision, and ELo's evolved multi-modal, multi-task loss.