Source-linked AI summary
VideoLSTM Convolves, Attends and Flows for Action Recognition
Zhenyang Li, Efstratios Gavves, Mihir Jain, Cees G. M. Snoek
TL;DR
Video action models must represent spatial structure, motion, temporal memory, and action locality across varied visual content. VideoLSTM adapts soft-Attention LSTM with convolutional and motion-based attention modules, and uses its attention for class-label-only localization; experiments report consistent improvements from jointly modeling these properties. Attention may also select contextual background regions, so the method applies temporal smoothing to reduce box jitter.
Problem
Video action recognition requires jointly modeling appearance, motion, temporal memory, and spatio-temporal locality despite substantial variation across videos and actions.
Method
VideoLSTM extends soft-Attention LSTM with the Convolutional ALSTM and shallow convolutional motion-based attention, using attention maps for action localization.
Results
Experiments show consistent improvements when video properties are modeled jointly, and attention supports competitive action localization using only video-level action class labels.
Takeaways & Limitations
Adapting the recurrent architecture to video enables one model to integrate spatial correlations, motion-guided attention, and class-label-based localization.
Takeaways & Limitations
Attention sometimes selects discriminative background context rather than the action, requiring temporal smoothing to reduce sudden localization deviations.
Abstract
from arXiv · showhide
We present a new architecture for end-to-end sequence learning of actions in video, we call VideoLSTM. Rather than adapting the video to the peculiarities of established recurrent or convolutional architectures, we adapt the architecture to fit the requirements of the video medium. Starting from the soft-Attention LSTM, VideoLSTM makes three novel contributions. First, video has a spatial layout. To exploit the spatial correlation we hardwire convolutions in the soft-Attention LSTM architecture. Second, motion not only informs us about the action content, but also guides better the attention towards the relevant spatio-temporal locations. We introduce motion-based attention. And finally, we demonstrate how the attention from VideoLSTM can be used for action localization by relying on just the action class label. Experiments and comparisons on challenging datasets for action classification and localization support our claims.
1 Introduction
VideoLSTM adapts recurrent sequence modeling to video’s spatial, motion, and spatio-temporal locality properties. It jointly combines convolutions, motion-based attention, and attention-derived localization for action recognition.
- Video modeling must address appearance, motion, memory, and spatio-temporal locality despite substantial variation across videos and actions.The introduction contrasts actions driven by context, motion, or combinations of appearance, motion, and temporal transitions.
- Standard and attention LSTMs vectorize spatial inputs, discarding correlations between neighboring locations that convolutional networks model explicitly.The paper argues that convolutions should be hardwired into LSTM architectures to preserve spatial structure.
- VideoLSTM adapts the LSTM architecture to video rather than adapting video to an established architecture, jointly modeling video properties.Experiments reportedly show limited gains from modeling only subsets of properties, but consistent improvements from joint treatment.
- The architecture adds convolutions for spatial correlations and a shallow convolutional motion-based attention mechanism for relevant spatio-temporal locations.VideoLSTM starts from soft-Attention LSTM and introduces the Convolutional ALSTM and Motion-based Attention networks.
- VideoLSTM attention maps support competitive action localization using only video-level action class labels, without explicit localization supervision.The figure summarizes convolutional attention, motion-based attention, and attention-map-based localization as the proposed network’s components.
2 Related work
Related work spans convolutional, two-stream, recurrent, attention-based, and proposal-based approaches to video action recognition. VideoLSTM is positioned as an end-to-end model that learns salient action locations from class labels while avoiding explicit spatio-temporal annotations.
- ConvNet architectures: 3-D convolutional networks jointly model appearance and motion but increase learning complexity and parameter demands, requiring larger datasets.The cited work reports improved accuracy when such networks are trained on massive sports-video datasets.
- ConvNet architectures: Two-stream architectures separately learn 2-D filters for appearance and optical-flow variations, using consecutive frames to capture longer temporal patterns.A multi-task setting was proposed to address limited training data.
- LSTM architectures: LSTM approaches model short- and long-term sequential memories, increasingly using ConvNet activations from appearance and flow inputs.Prior work also explored fusing appearance- and flow-based LSTMs.
- ALSTM architectures: Attention-LSTMs extend temporal modeling by focusing on spatially local input regions through soft attention.This attention mechanism replaced fixed-length visual vectors in vision applications.
- Action localization: Unlike proposal methods that require spatio-temporal tubes and annotations, VideoLSTM learns an end-to-end representation from action class labels while predicting salient action locations.The localization approach is presented as distinct from separately encoding candidate action tubes.
3 VideoLSTM for action classification
VideoLSTM extends soft-Attention LSTM with convolutional state processing and motion-based attention, preserving spatial structure while using motion to guide attention in video.
- 3.1 Convolutional ALSTM: Using convolutional feature maps produces N × N × D representations, preserving regions rather than reducing each frame to a vector.
- 3.1 Convolutional ALSTM: Convolutional ALSTM replaces fully connected operations with convolutions, so gates, memory, and hidden states retain spatial dimensions.
- 3.1 Convolutional ALSTM: Its attention map is generated by a shallow convolutional network from the previous hidden state and current feature map, yielding a 2-d score map.
- 3.1 Convolutional ALSTM: Attention weights spatial feature-map locations without taking their expectation, suppressing activations at locations with lower saliency.
- 3.2 Motion-based Attention: Motion-based attention adds a bottom convolutional LSTM layer using optical-flow feature maps, conditioning attention on the current hidden state.
- 3.2 Motion-based Attention: The bottom layer helps generate motion-based attention but supplies no direct information to the top layer for final classification.
4 VideoLSTM for action localization
VideoLSTM converts framewise attention into saliency maps and uses thresholded connected regions to produce one temporally smoothed action proposal per video.
- VideoLSTM up-samples and Gaussian-smooths attention maps to create framewise saliency maps for action localization.
- Thresholding each saliency map and boxing its connected components produces candidate action regions for each frame.
- The method avoids bounding-box ground truth and generates only one detection by using a high threshold and selecting among multiple boxes when necessary.
- Temporal smoothing uses locally weighted linear regression to reduce sudden deviations and box jitter across frames.
- Framewise classification scores are averaged to obtain class confidence scores for the single proposal, enabling localization without bounding-box training labels.
5 Experiments
Experiments evaluate VideoLSTM components for action classification and localization across UCF101, HMDB51, and THUMOS13. Convolutions, motion-based attention, and joint modeling improve classification, while weakly supervised localization remains competitive with box-supervised methods.
- Datasets and setup: Experiments use UCF101 and HMDB51 for classification and THUMOS13 for action localization.The reported evaluation follows the first split for UCF101 and THUMOS13, while HMDB51 classification is averaged over its three splits.
- Action classification: ConvALSTM improves appearance classification by +2.2% on UCF101 and +1.1% on HMDB51 over the standard ConvNet baseline.Other LSTM-based architectures fail to improve over the standard ConvNet on appearance data, whereas ConvALSTM produces the largest gain.
- Action classification: Motion-based attention further improves both ALSTM and ConvALSTM when RGB appearance input is used.The mechanism explicitly models motion saliency with optical flow to help generate attention maps.
- Action classification: +3.8% over LSTM, +2.6% over ALSTM, and +1.7% over Convolutional LSTM on UCF101 are obtained by ConvALSTM with optical-flow input.With flow frames, all LSTM-based models improve over the ConvNet baseline, and the improvement is larger on HMDB51.
- Action localization: VideoLSTM obtains 89.2 mAP on UCF101 first-split localization versus 77.0 mAP for the compared approach.The passage attributes the difference to convolutions and motion-based attention.
- Action localization: VideoLSTM is competitive with box-supervised localization methods on THUMOS13 while using only video-level action labels and returning a single spatio-temporal proposal.It does better at two of four reported thresholds against Weinzaepfel et al., is competitive with APT, and outperforms Yu et al.
6 Conclusion
VideoLSTM adapts recurrent architecture to video by jointly modeling spatial correlations, motion-guided attention, and spatio-temporal action localization using video-level labels.
- VideoLSTM introduces convolutions to exploit spatial correlations in video frames and produce spatial frame encoding and attention.
- A shallow convolutional network uses motion information to generate motion-based attention maps over relevant spatio-temporal locations.
- VideoLSTM uses attention maps to localize actions spatio-temporally from only video-level action class labels.
- Experiments on three challenging datasets demonstrate the theoretical and practical merits of VideoLSTM.