Source-linked AI summary
Delving Deeper into Convolutional Networks for Learning Video Representations
Nicolas Ballas, Li Yao, Chris Pal, Aaron Courville
TL;DR
Video representations must capture temporal variation at multiple spatial resolutions, but intermediate CNN maps create high-dimensional inputs. The paper uses convolutional GRUs over multi-level ImageNet CNN percepts, improving baseline performance on action recognition and captioning while reaching state-of-the-art-comparable YouTube2Text results without extra 3D CNN features.
Problem
Existing high-level recurrent convolutional architectures may miss fine motion because CNN top layers discard local spatial and short-term temporal variation.
Method
The paper feeds convolutional maps from multiple levels of an ImageNet-trained CNN into GRU-based recurrent convolutional networks with sparse local connectivity.
Results
3.4% and 10% gains over the baseline are reported for UCF101 action recognition and YouTube2Text video captioning, respectively.
Takeaways & Limitations
On YouTube2Text, the approach achieves results comparable to state of the art using a simpler text decoder and no additional 3D CNN features.
Abstract
from arXiv · showhide
We propose an approach to learn spatio-temporal features in videos from intermediate visual representations we call "percepts" using Gated-Recurrent-Unit Recurrent Networks (GRUs).Our method relies on percepts that are extracted from all level of a deep convolutional network trained on the large ImageNet dataset. While high-level percepts contain highly discriminative information, they tend to have a low-spatial resolution. Low-level percepts, on the other hand, preserve a higher spatial resolution from which we can model finer motion patterns. Using low-level percepts can leads to high-dimensionality video representations. To mitigate this effect and control the model number of parameters, we introduce a variant of the GRU model that leverages the convolution operations to enforce sparse connectivity of the model units and share parameters across the input spatial locations. We empirically validate our approach on both Human Action Recognition and Video Captioning tasks. In particular, we achieve results equivalent to state-of-art on the YouTube2Text dataset using a simpler text-decoder model and without extra 3D CNN features.
1 INTRODUCTION
Video understanding requires representations that preserve temporal information while capturing discriminative visual content. The paper uses convolutional maps from multiple CNN levels to model both coarse appearance and fine local motion.
- Video understanding is challenging, motivating general representations for action recognition, retrieval, and captioning.
- 2D CNNs perform strongly on images but discard temporal information, whereas RNNs can model temporal sequences.
- Existing high-level RCNs may miss fine motion because pooling removes local information and video motion is typically local.
- Higher CNN layers become temporally stable and discard short-window variation, while lower-level maps preserve finer spatial detail.
- The proposed GRU extension replaces fully connected recurrent transformations with convolutions, enforcing sparse local connectivity and reducing memory requirements.
- 3.4% and 10% gains over the baseline are reported for UCF101 action recognition and YouTube2Text video captioning, respectively.
2 GRU: GATED RECURRENT UNIT NETWORKS
GRUs are recurrent networks designed to model temporal dependencies using gated hidden-state updates. The paper presents their gates and connects them to a convolutional recurrent architecture for video representations.
- GRUs are RNN variants that model variable-length sequences through recurrent hidden states and require less memory than LSTMs.
- The update gate z_t controls how much the unit updates its activation, while the reset gate r_t controls access to the previous state.
- The candidate activation h̃_t is computed from the current input and gated previous hidden state before forming the new activation.
- The model uses convolutional maps from different CNN layers as inputs to convolutional GRU-RNNs at different time steps.
- Optional bottom-up connections between recurrent convolutional layers produce the Stacked-GRU-RCN architecture.
3 DELVING DEEPER INTO CONVOLUTIONAL NEURAL NETWORKS
The section develops recurrent convolutional architectures that model temporal patterns from convolutional maps at multiple spatial resolutions while controlling parameter growth through local, sparse connectivity.
- The approach applies recurrent networks to convolutional maps extracted from multiple CNN levels to capture temporal patterns at different spatial resolutions.
- The first architecture applies L RNNs independently to each convolutional map, whose outputs feed classification or text-decoder modules.
- Directly applying GRUs to intermediate maps creates many parameters, so GRU-RCN replaces fully connected transformations with convolutions.
- Convolutional recurrent units use local neighborhoods, sparse connectivity, and shared parameters while preserving spatial topology.The recurrent receptive field grows across earlier time steps, enabling spatio-temporal patterns with high spatial variation in time.
- GRU-RCN requires O(3TN1N2k1k2(OxOh + OhOh)) multiplications, compared with O(3TN1N2N1N2(OxOh + OhOh)) for a fully connected RNN.The comparison attributes the savings to sparse connectivity.
- Stacked GRU-RCN adds bottom-up connections by providing the previous RNN hidden representation as an extra input to convolutional GRU units.This adds flexibility for leveraging representations with different resolutions.
4 RELATED WORK
Prior video-representation work used 3D CNNs, two-stream CNNs, or RNNs over CNN features, whereas this approach uses recurrent convolutional units on percepts from multiple CNN layers.
- Earlier methods used 3D CNNs to learn video representations from large datasets such as Sport 1 Million.
- Two-stream models train separate RGB and optical-flow CNNs, allowing the RGB stream to use image-pretrained 2D CNNs.
- Other approaches placed RNNs on top of two-stream features or learned video representations unsupervised, typically using only high-level visual percepts.
- Unlike prior work on now-casting with LSTMs applied directly to pixels, this method uses recurrent convolutional units on pretrained CNN maps.
5 EXPERIMENTATION
The evaluation tests the proposed GRU-RCN and Stacked GRU-RCN architectures on human action recognition and video caption generation.
- The experiments evaluate GRU-RCN and Stacked GRU-RCN on human action recognition and video caption generation.
5.1 ACTION RECOGNITION
The action-recognition experiments evaluate GRU-RCN variants using RGB, flow, and fused inputs on UCF101. Bi-directional GRU-RCN performs best among the proposed variants, while fusion reaches performance close to a state-of-the-art reference.
- Dataset and inputs: The experiments use UCF101 split 1, with RGB and flow percepts extracted from VGG-16 convolutional maps and fc7.The CNNs are ImageNet-pretrained and fine-tuned on UCF-101.
- Model variants: GRU-RCN applies convolutional GRU-RNNs independently to convolutional maps, while stacked and bi-directional variants test bottom-up and reverse-temporal connections.The bi-directional model processes inputs in sequential and reverse temporal order before classification.
- RGB results: 80.7 accuracy is achieved by Bi-directional GRU-RCN on RGB inputs, a 3.4% gain over baselines focused on the VGG-16 top layer.The baselines are VGG-16 and VGG-16 RNN; VGG-16 RNN reaches 78.1 versus 78.0 for VGG-16.
- Flow results: 85.7 accuracy versus 85.4 is achieved by GRU-RCN on flow inputs, although the improvement is smaller because the flow baseline already captures motion information.The flow baseline uses 10 consecutive flow inputs to extract percepts.
- Fusion and comparisons: 90.8 accuracy is reached by combining RGB and flow Bi-directional GRU-RCN, a 1.9% gain over the fused VGG-16 baseline and near the 90.9 reference.The fused VGG-16 baseline achieves 89.1, while C3D reaches 85.2 using training over 1 million videos.
5.2 VIDEO CAPTIONING
The video-captioning experiments compare VGG-16 and Bi-directional GRU-RCN encoders within an encoder-decoder framework on YouTube2Text. The GRU-RCN encoder outperforms VGG-16 across all reported metrics, while combined representations reach other state-of-the-art results.
- Dataset: The YouTube2Text corpus contains 1,970 video clips with multiple descriptions spanning topics including sports, animals, music, and movies.The split uses 1,200 training clips, 100 validation clips, and the remaining clips for testing.
- Encoder: The encoder maps videos into abstract representations, using either VGG-16 fc7 activations or Bi-directional GRU-RCN representations from each segment.The GRU-RCN processes the segment’s first 10 frames, while VGG-16 uses the first frame’s fc7 activations.
- Decoder and training: An LSTM text generator with soft attention decodes captions from the video representations.Decoder parameters are estimated by maximizing training description log-likelihood, with hyperparameters selected by validation performance.
- Evaluation: The evaluation reports BLEU, METEOR, and CIDEr, selecting models by validation performance for each metric.Models are early-stopped using validation negative log-likelihood.
- Results: The combined GoogleNet and Bi-directional GRU-RCN approach is on par with a state-of-the-art method without requiring a C3D encoder.It also outperforms approaches using spatial-temporal attention or hierarchical RNN decoder schemes according to BLEU.
6 CONCLUSION
The paper learns video representations by modeling temporal variation across visual percepts at multiple CNN spatial resolutions. Its evaluations show gains over baselines on action recognition and video captioning, with competitive captioning results using simpler components.
- Conclusion: The method addresses learning discriminative and abstract video representations by modeling temporal variation across different spatial resolutions.High-level percepts are discriminative but low-resolution, whereas low-level percepts preserve spatial detail for finer motion patterns.
- Conclusion: The proposed recurrent convolutional architecture uses convolutional maps from all levels of an ImageNet-trained deep CNN.This design combines percepts from multiple spatial resolutions within the video representation.
- Conclusion: 3.4% and 10% gains over the baseline are reported for human action recognition and video captioning, respectively, using RGB inputs.The evaluations use UCF-101 and YouTube2Text.
- Conclusion: The approach achieves results comparable to state of the art on YouTube2Text with a simpler text decoder and without extra 3D CNN features.This conclusion concerns the video-captioning evaluation.