Source-linked AI summary
Action Recognition using Visual Attention
Shikhar Sharma, Ryan Kiros, Ruslan Salakhutdinov
TL;DR
Action-recognition models often treat all video regions similarly, despite evidence that relevant information is spatially selective. This paper uses recurrent soft attention to pool video features dynamically and reports better performance than average- and max-pooled LSTM baselines across three datasets.
Problem
Traditional video-recognition methods largely lack attention mechanisms and treat different image or video regions similarly, limiting selective modeling of task-relevant information.
Method
A recurrent soft-attention model uses multilayer deep LSTMs to dynamically pool convolutional features while selecting informative video-frame locations.
Results
The attention model performs better than average- and max-pooled LSTMs across UCF-11, HMDB-51, and Hollywood2, while remaining competitive with RGB-only deep-learning models.
Takeaways & Limitations
Visualizing learned attention provides insight into which video elements the model uses and helps examine both successful and failed classifications.
Takeaways & Limitations
The attention mechanism can focus on background elements or field boundaries and consequently misclassify activities.
Abstract
from arXiv · showhide
We propose a soft attention based model for the task of action recognition in videos. We use multi-layered Recurrent Neural Networks (RNNs) with Long Short-Term Memory (LSTM) units which are deep both spatially and temporally. Our model learns to focus selectively on parts of the video frames and classifies videos after taking a few glimpses. The model essentially learns which parts in the frames are relevant for the task at hand and attaches higher importance to them. We evaluate the model on UCF-11 (YouTube Action), HMDB-51 and Hollywood2 datasets and analyze how the model focuses its attention depending on the scene and the action being performed.
1 INTRODUCTION
The paper proposes a soft attention-based recurrent model for action recognition that dynamically pools convolutional features and focuses on activity-relevant elements in video frames.
- Soft attention models are deterministic and trainable by backpropagation, unlike stochastic hard attention models that require sampling.Hard attention can also be trained using REINFORCE, variational lower bounds, or importance sampling, making learning computationally expensive.
- The paper proposes a soft attention-based recurrent model for action recognition.The model dynamically pools convolutional features and uses them for action recognition.
- The model dynamically pools convolutional features and performs better than average or max pooling used by many existing models.
- The model recognizes important elements in video frames according to the activities it detects.This reflects selective attention to relevant parts of the frames rather than treating all regions equally.
2 RELATED WORK
Prior video-action recognition methods extend image-based CNNs with temporal modeling through handcrafted descriptors, CNN-LSTM architectures, and temporal pooling. Attention models are introduced as a way to improve interpretability by indicating where models focus, while earlier action-recognition attention fixed focus to the frame center.
- Temporal video representations: Video classification adds a temporal dimension to image classification, motivating optical flow, HOG, and handcrafted descriptors that encode appearance and dynamics.Learning temporal dynamics is described as difficult.
- Recurrent temporal modeling: LSTMs have performed well across speech recognition, machine translation, image description, and video description.The cited applications include work by Graves et al. (2013), Sutskever et al. (2014), Xu et al. (2015), Vinyals et al. (2015), and others for video description.
- CNN-LSTM video models: Existing video approaches commonly combine CNNs with LSTMs, classify sequences directly, or temporally pool features before classification.LSTMs have also been used for unsupervised video representation learning with encoder-decoder models, alongside 3-D CNN features and LSTM decoders.
- Attention and interpretability: Attention models add interpretability to deep neural networks by capturing where the model focuses while performing a task.The passage frames attention as addressing the difficulty of interpreting internal representations learned by deep neural networks.
- Attention and interpretability: Karpathy et al. used a multi-resolution CNN for video action recognition and described a fovea-like mechanism whose attention remained fixed at the frame center.This contrasts with attention mechanisms that learn where to focus.
3 THE MODEL AND THE ATTENTION MECHANISM
The model extracts spatial feature slices from CNN feature cubes, uses soft attention to form inputs by weighting regions, and processes them with deep multi-layered LSTMs. It predicts locations and action classes while regularizing attention to cover frame regions over time.
- Feature extraction and recurrent architecture: CNN feature cubes provide K^2 D-dimensional feature slices, each corresponding to an overlapping region of the input frame.The experiments use a 7 × 7 × 1024 feature cube.
- Feature extraction and recurrent architecture: The recurrent network uses multi-layered deep LSTMs to predict the next location probabilities and class-label probabilities.The model uses LSTM cell and hidden states, with an additional hidden layer using tanh activations for predictions.
- Soft attention mechanism: At each time-step, a location softmax assigns probabilities to K × K regions, and soft attention computes the next input as their weighted expected feature value.The location probabilities represent the model’s estimated importance of corresponding input regions.
- Soft attention mechanism: Unlike hard attention, soft attention remains differentiable because it averages feature slices instead of sampling a single location.Hard attention samples a location from the softmax distribution and therefore requires sampling methods.
- Training objective: The training objective combines cross-entropy with a doubly stochastic attention penalty that encourages the model to inspect each frame region at some point in time.The regularization imposes Σ_t l_t,i ≈ 1 over locations.
4 EXPERIMENTS
Experiments evaluate the attention model across three action-recognition datasets and compare it with pooling and state-of-the-art baselines. The results show improved performance, adaptable attention behavior, and interpretable successes and failures.
- Datasets: The experiments use UCF-11, HMDB-51, and Hollywood2, with 975/625, 3570/1530 per split, and 823/884 training/testing videos, respectively.UCF-11 has 11 actions, HMDB-51 has 51 classes, and Hollywood2 has 12 classes; Hollywood2 videos may contain multiple actions.
- Experimental setup: The model uses 3-layer LSTMs selected by cross-validation, with hidden, cell, and hidden-layer dimensionality 512 for UCF-11 and Hollywood2 and 1024 for HMDB-51.Experiments with one to five LSTM layers produced no significant performance improvements.
- Comparisons: The attention model outperforms softmax regression and both average- and max-pooled LSTMs across the evaluated datasets.The softmax regression baseline receives the complete 7 × 7 × 1024 feature cube, whereas the attention model and other LSTM variants use feature slices or pooled representations.
- Attention analysis: The doubly stochastic penalty controls attention exploration: λ = 0 yields less variation, λ = 1 encourages different gaze locations, and λ = 10 makes attention resemble average pooling.Intermediate values produce dynamic weighted averaging of feature slices.
- Attention analysis: Attention focuses on task-relevant regions, such as parts of a cycle for cycling and dogs for walking with a dog.These examples show the model attending to distinct objects or regions while correctly recognizing the associated activity.
- Failure analysis: Attention can also expose errors: focusing on a court background caused diving instead of volleyball spiking, while soccer-juggling attention to field boundaries caused tennis-swinging.The authors also report that background context can support recognition when foreground content is difficult to discern.
5 CONCLUSION
The paper develops recurrent soft-attention models for action recognition and analyzes their focus on video-frame elements. The models outperform non-attention baselines but remain computationally expensive because they require all features for dynamic pooling.
- Contributions: The authors develop recurrent soft-attention models for action recognition and analyze where the models focus.The models use recurrent attention to examine relevant video-frame regions.
- Attention analysis: The model recognizes important video-frame elements according to the action being performed.Its focus varies with the action and scene content.
- Performance: The proposed model performs better than baselines without an attention mechanism.The conclusion specifically compares against baselines that do not use attention.
- Limitations: Soft-attention models remain computationally expensive because they require all features for dynamic pooling.Their computational cost persists despite using soft attention.
S. Hochreiter and J. Schmidhuber. Long short-term memory. Neural Computation, 9(8):1735–1780, 1997.
The section lists prior work spanning visual attention, action recognition, video classification, recurrent modeling, optimization, and neural-network regularization. It also cites applications of recurrent networks to video understanding and image captioning.
- Video understanding: The cited video-understanding work covers large-scale classification, long-range temporal modeling, unsupervised LSTM representations, video captioning, and dense action labeling.The references include methods designed to exploit temporal structure and recurrent representations for video tasks.
- Training and regularization: Additional references cover Adam optimization, dropout, recurrent-network regularization, sequence-to-sequence learning, and reinforcement-learning gradients.The section also cites foundational methods for training and regularizing neural networks.
- Visual attention: Related work includes visual-attention models for recurrent vision and image caption generation.The references include recurrent models of visual attention, spatial transformer networks, and visual-attention-based image captioning.
- Action recognition: Action-recognition research cited here uses convolutional networks, two-stream models, stacked Fisher vectors, slow feature analysis, and multi-skip feature stacking.These references address action recognition in videos through both deep-learning and feature-based approaches.
A ADDITIONAL EXAMPLES
The paper presents additional correctly and incorrectly classified examples from UCF-11, HMDB-51, and Hollywood2, illustrating where the model focuses over time. Attention visualizations show that white regions mark the model’s focus and brightness indicates its strength.
- Correctly Classified Examples: Additional correctly classified examples are presented from UCF-11, HMDB-51, and Hollywood2.These examples appear in Figure 10.
- Incorrectly Classified Examples: Incorrectly classified examples are also shown with attention visualizations over time.Figure 11 uses white regions to indicate where the model looks and brightness to indicate focus strength.
- Attention Visualization: The model’s attention in correctly classified videos focuses on relevant parts over time.White regions show where the model looks, while brightness indicates focus strength.