Source-linked AI summary
Learnable pooling with Context Gating for video classification
Antoine Miech, Ivan Laptev, Josef Sivic
TL;DR
Video classification commonly relies on frame-level features followed by temporal aggregation, but simple pooling can be inaccurate for long sequences and recurrent models can be cumbersome, data-intensive, and suboptimal for long videos. The paper evaluates learnable clustering-based pooling for audio and visual streams and introduces Context Gating to model activation interdependencies. On Youtube-8M V2, the approach improves pooling and gating results and achieves top performance in the Large-Scale Video Understanding challenge.
Problem
Temporal aggregation for video features remains limited by inaccurate simple pooling on long sequences and cumbersome, data-intensive recurrent models.
Method
The paper combines learnable clustering-based aggregation of audio and visual features with Context Gating, a nonlinear unit modeling interdependencies among activations.
Results
The method achieves top performance among more than 650 teams in the Youtube-8M Large-Scale Video Understanding challenge.
Takeaways & Limitations
Clustering-based pooling and Context Gating improve video classification, while the resulting models train faster and require less training data than common recurrent models.
Takeaways & Limitations
Recurrent-model comparisons are conducted in the Youtube-8M V2 setting, where experiments also report similar results for temporally ordered and randomly ordered frames.
Abstract
from arXiv · showhide
Current methods for video analysis often extract frame-level features using pre-trained convolutional neural networks (CNNs). Such features are then aggregated over time e.g., by simple temporal averaging or more sophisticated recurrent neural networks such as long short-term memory (LSTM) or gated recurrent units (GRU). In this work we revise existing video representations and study alternative methods for temporal aggregation. We first explore clustering-based aggregation layers and propose a two-stream architecture aggregating audio and visual features. We then introduce a learnable non-linear unit, named Context Gating, aiming to model interdependencies among network activations. Our experimental results show the advantage of both improvements for the task of video classification. In particular, we evaluate our method on the large-scale multi-modal Youtube-8M v2 dataset and outperform all other methods in the Youtube 8M Large-Scale Video Understanding challenge.
1 INTRODUCTION
The paper targets multi-label classification of variable-quality Internet videos with incomplete or noisy labels, revising temporal aggregation through clustering-based pooling, multimodal architecture, and Context Gating. On Youtube-8M V2, these choices improve performance and achieve top challenge results.
- Internet video classification is difficult because content and quality vary widely, while user-generated labels can be incomplete, ambiguous, or erroneous.
- Existing systems extract frame or clip features and aggregate them over time using averaging, VLAD, LSTM, or GRU.
- Average and maximum pooling may become inaccurate on long sequences, while recurrent aggregation is cumbersome to train, data-intensive, and potentially suboptimal for long videos.
- The paper integrates and enhances clustering-based aggregation techniques for video representation and classification.
- The contributions include a state-of-the-art audio-visual architecture, Context Gating for activation interdependencies, and evidence that clustering-based pooling benefits video classification over LSTM and GRU.
- On Youtube-8M V2, containing about 8M videos and 4716 unique tags, the method achieves top performance among more than 650 teams and trains faster with less data than common recurrent models.
2 RELATED WORK
Related work spans deep frame or clip representations, recurrent temporal modeling, orderless pooling, and gating mechanisms. The paper combines end-to-end learnable orderless aggregation of visual and audio inputs with gating, addressing a gap in applying gating beyond recurrent models.
- Video representations use CNN activations from individual frames or short clips, including separate appearance and motion streams.
- LSTM and GRU models aggregate frame-level features into a single representation while capturing temporal structure.
- The paper's architecture is illustrated as a Late Concat variant, with FC layers and a Mixture-of-Experts classifier.
- Orderless alternatives include average or maximum pooling, bag-of-visual-words, VLAD, and Fisher Vector encoding.
- End-to-end trainable orderless aggregation can learn encoding modules within neural networks, and this work extends it to visual and audio inputs and multiple aggregations.
- Gating provides multiplicative interaction between input features and gate values, but had not been combined with non-temporal FV, VLAD, or BoW aggregation in the described prior work.
3 NETWORK ARCHITECTURE
The architecture separately pools visual and audio features, then applies Context Gating to the pooled representation and classifier outputs. Context Gating learns feature interdependencies and reweights activations, while its gradient behavior is related to residual connections.
- Network architecture: The architecture extracts video and audio features, pools each stream into a compact representation, applies Context Gating, and predicts predefined label scores.The classification module uses a Mixture-of-Experts classifier followed by another Context Gating layer.
- Context Gating: Context Gating transforms X into Y by multiplying the input feature vector by sigmoid-generated, trainable gates.The gates lie between 0 and 1 and are applied element-wise to the input dimensions.
- Context Gating: Context Gating introduces non-linear interactions among activations and recalibrates their strengths through self-gating.The layer is motivated by both activation dependencies and selective reweighting.
- Relation to residual connections: Like residual connections, Context Gating has a gradient term involving ∇X, but that term is weighted by the learned gate values.Gradients propagate directly when gate activations are near 1 and are diminished when they are near 0.
- Context Gating: Applied to video features and output labels, Context Gating reweights representations and can model dependencies in label scores.The output-level use is intended to capture prior structure in label annotations.
- Context Gating: In a skiing example, Context Gating can down-weight Tree activations when Ski and Snow activations co-occur.This illustrates context-dependent suppression of less relevant visual cues.
4 LEARNABLE POOLING METHODS
The paper studies non-recurrent, clustering-based pooling methods that aggregate frame-level descriptors through learnable soft assignments. It extends NetVLAD toward BOW and Fisher Vector encodings and introduces NetRVLAD, which averages descriptors instead of residuals.
- Motivation: The pooling study focuses on non-recurrent aggregation because shuffled-frame experiments showed almost no performance change, suggesting reliance on static visual cues.The stated goal is to efficiently retain relevant visual cues over potentially long videos.
- NetVLAD: NetVLAD replaces hard cluster assignment with differentiable soft assignment, allowing cluster parameters to be tuned by backpropagation.The soft assignment measures how closely a descriptor matches each learnable cluster.
- NetVLAD: The NetVLAD descriptor aggregates weighted residuals between descriptors and learnable cluster anchor points.Each cluster receives a weighted sum of descriptor-to-cluster residuals.
- Beyond NetVLAD: The same soft-assignment mechanism supports differentiable imitations of bag-of-visual-words and Fisher Vector aggregation.These alternatives preserve the clustering-based aggregation idea while learning the assignment parameters.
- Beyond NetVLAD: NetFV extends the NetVLAD design to learn second-order feature statistics within clusters, alongside first-order statistics.Its learnable clusters have diagonal covariances, and the covariance parameters are constrained to remain positive.
- Beyond NetVLAD: NetRVLAD averages actual descriptors rather than residuals, reducing parameters and computation to about half of the original NetVLAD design.This is presented as a simplified residual-less VLAD variant.
5 EXPERIMENTS
Experiments on Youtube-8M compare learnable pooling, recurrent aggregation, Context Gating, audio-visual fusion, data-scale effects, and ensembles using GAP at top 20.
- Dataset and metric: The evaluation uses approximately 8 million multi-label videos with pre-extracted visual and audio features, reporting Global Average Precision at top 20.The original dataset split is 70% training, 20% validation, and 10% test; most results use a validation set of around 20K videos.
- Model evaluation: All pooled representations are fixed at 1024 dimensions, enabling comparison of clustering-based, recurrent, and average-pooling models.Clustering-based models include BoW, NetVLAD, NetRVLAD, and NetFV; recurrent models include LSTM and GRU.
- Model evaluation: Learned aggregation schemes significantly outperform average-pooling baselines, while shuffled NetVLAD and NetFV outperform GRU and LSTM.Context Gating consistently improves clustering-based pooling methods, but adding it to GRU and LSTM decreases performance.
- Context Gating ablation: Context Gating improves NetVLAD with 128 clusters by 0.8% GAP and also outperforms a less complex GLU comparison.The ablation adds Context Gating in both the feature-pooling and classification modules.
- Audio-video fusion: Late audio-visual fusion improves clustering-based pooling, whereas early fusion performs better for GRU and LSTM aggregations.Outer-product fusion performed worse than concatenation, while compact bilinear pooling produced underfitting models.
- Ensembling: A seven-model ensemble reaches 84.688 GAP on the private test set and achieves first place with substantially fewer models than other top-performing approaches.The final challenge ensemble contains 25 models, while the comparison covers 655 teams.
6 CONCLUSIONS
The paper concludes that learnable clustering-based pooling, Context Gating, and audio-visual aggregation improve large-scale video tagging, with ensembling enabling the challenge win.
- Conclusions: NetVLAD, NetFV, and BoW outperform LSTM and GRU for temporal aggregation of audio and visual features.The paper presents these as trainable variants of classical pooling methods for large-scale video tagging.
- Conclusions: Context Gating benefits the trainable versions of BoW, VLAD, and FV, while ensembling the individual models improves performance further.The resulting ensemble enabled the method to win the Youtube 8M Large-Scale Video Understanding challenge.