Source-linked AI summary
LSTA: Long Short-Term Attention for Egocentric Action Recognition
Swathikiran Sudhakaran, Sergio Escalera, Oswald Lanz
TL;DR
Egocentric activity recognition requires fine-grained action–object discrimination, while existing approaches can require costly supervision or neglect temporal consistency. The paper proposes LSTA, which combines recurrent spatial attention and output pooling in a two-stream architecture. LSTA achieves state-of-the-art accuracy on four standard benchmarks, with ablations showing gains from its components.
Problem
Fine-grained egocentric recognition is difficult because relevant action and object information must be identified without impractical frame-level annotations, while some attention methods process frames independently.
Method
LSTA extends LSTM with attention pooling and output pooling, tracking spatial attention across time and using a two-stream CNN-LSTA architecture with cross-modal fusion.
Results
LSTA achieves state-of-the-art accuracy on four standard benchmarks.
Takeaways & Limitations
Attention pooling, output pooling, and cross-modal two-stream processing provide a practical end-to-end approach for egocentric activity recognition.
Takeaways & Limitations
The attention-pooling design assumes that activity recognition involves a limited number of relevant pattern categories, despite variability within and across executions.
Abstract
from arXiv · showhide
Egocentric activity recognition is one of the most challenging tasks in video analysis. It requires a fine-grained discrimination of small objects and their manipulation. While some methods base on strong supervision and attention mechanisms, they are either annotation consuming or do not take spatio-temporal patterns into account. In this paper we propose LSTA as a mechanism to focus on features from spatial relevant parts while attention is being tracked smoothly across the video sequence. We demonstrate the effectiveness of LSTA on egocentric activity recognition with an end-to-end trainable two-stream architecture, achieving state of the art performance on four standard benchmarks.
1. Introduction
The paper targets fine-grained egocentric activity recognition, where action–object distinctions, ego-motion, and spatio-temporal variation make recognition difficult without costly annotations. It introduces LSTA, an LSTM-based recurrent unit with built-in spatial attention and revised output gating, deployed in a two-stream architecture.
- Challenge: Large-scale fine-grained annotation covering spatio-temporal variability is described as unfeasible in practice.
- Challenge: Fine-grained egocentric recognition depends jointly on the action and manipulated object, while strong ego-motion complicates motion encoding.Frame-level object or hand annotations could help but are costly and impractical at scale.
- Prior limitations: Existing methods may rely on separate pre-training with strong supervision and complex annotation operations.
- Approach: LSTA augments LSTM with built-in spatial attention and revised output gating to focus on relevant regions and expose a distilled memory view.
- Evaluation: The proposed system uses LSTA in a two-stream architecture with cross-modal fusion and reports state-of-the-art results on four public datasets.
2. Related Work
Prior egocentric methods often use additional annotations or framewise attention, limiting scalability or temporal consistency. LSTA instead generates attention sequentially by propagating an internal attention state and combines motion attention with cross-modal stream fusion.
- Annotation-based methods: Specialized hand-segmentation and object-localization methods require substantial annotated data and may encode appearance from single RGB images without temporal information.
- Annotation-based methods: State-of-the-art techniques commonly use hand segmentation, object bounding boxes, or gaze information, but annotating every video frame is impractical.
- Temporal attention: Most video attention techniques process frames independently, losing information from temporal consistency.
- LSTA: LSTA generates top-down spatial attention sequentially by propagating attention maps from past frames through an internal state.
- LSTA: The proposed approach applies attention to the motion stream and uses cross-modal fusion between appearance and motion streams.
3. Analysis of LSTM
The paper analyzes LSTM memory, attention, and output gating for video recognition. It motivates convolutional memory and built-in spatial attention because discriminative video information can be spatially localized, while output gating also influences memory tracking.
- LSTM: LSTM tracks a latent memory state across a sequence using forget and update gates.
- LSTM: The gating functions depend on the current input and a gated nonlinear view of the previous memory state.
- LSTM: LSTM computes input, forget, output, and innovation terms through gated parametric functions.
- LSTM: For sequence classification, the final memory state can serve as a fixed-length descriptor of the input sequence.
- LSTM: LSTM memory updates support erasing, resetting, preserving, or progressively memorizing state, helping maintain long-term dependencies.
- Design analysis: LSTA’s design uses recurrent attention and output pooling to track relevant features and provide a high-capacity output gate.
- Design analysis: The analysis identifies spatial attention and output gating as key LSTM extensions because localized features require filtering before gating and output gating affects memory tracking.
4. Long Short-Term Attention
LSTA extends LSTM with trainable spatial attention and output pooling, allowing recurrent memory to focus on discriminative regions while exposing a filtered memory representation.
- Attention pooling: LSTA extends LSTM with attention pooling that spatially filters input features before recurrent memory updates.The attention map is applied elementwise to the feature tensor, producing filtered features for the LSTM-style recurrence.
- Attention pooling: A learned selector chooses category-specific mappings from pooled feature representations to suppress activations unrelated to recognition.The pooling model assumes a limited number of relevant pattern categories, each with variable visual realizations.
- Attention pooling: The attention mapping is differentiable and can be trained from video-level annotations rather than direct selector supervision.This contrasts with class activation mapping approaches that train the selector directly using strong supervision.
- Output pooling: LSTA replaces standard output gating based on filtered inputs with gating based on a pooled view of the updated memory state.The filtered memory representation νc ⊙ct preserves recurrent output gating while exposing a distilled view of internal memory.
- Output pooling: The attention-filtered input also controls the output-pooling bias, coupling attention tracking with output gating and supporting active memory localization.A reduction and learned linear regression produce an instance-specific bias for the activation mapping.
5. Two Stream Architecture
The model uses a two-stream architecture to encode appearance and motion, then couples the streams through deep cross-modal bias control before late score fusion.
- Architecture: The two streams encode appearance from RGB frames and motion from optical-flow stacks.Both streams use convolutional processing before recurrent modeling.
- Architecture: The appearance stream uses a ResNet-34 feature extractor followed by LSTA processing of convolutional frame features.The output of ResNet-34 block conv5_3 is supplied to the LSTA module.
- Architecture: The motion stream uses a ResNet-34 trained on five-frame optical-flow stacks to encode action-related motion information.The motion network is first trained on action verbs before activity recognition training.
- Cross-modal fusion: Cross-modal fusion lets each stream influence the other by applying one modality’s features as bias to the other stream’s recurrent or convolutional processing.Flow features bias appearance-stream LSTA gates, while RGB features are summarized for motion-stream processing.
- Fusion: The final class scores are obtained by late average fusion of the two stream outputs.This combines the independently produced appearance and motion predictions after cross-modal interaction.
6. Experiments and Results
Experiments evaluate LSTA through ablations, stream variants, cross-modal fusion, comparisons, and four egocentric activity-recognition benchmarks. The results show gains from attention, output pooling, bias control, motion attention, and cross-modal fusion, with state-of-the-art comparisons reported across datasets.
- Datasets: Four benchmarks—GTEA 61, GTEA 71, EGTEA Gaze+, and EPIC-KITCHENS—are used to evaluate egocentric activity recognition.The datasets range from 61 and 71 activity classes to approximately 10K samples and more than 28K video samples.
- Ablation Study: 14% improvement over the vanilla ConvLSTM baseline results from adding attention pooling.Attention pooling identifies relevant regions and maintains their history across frames, enabling smoother attention tracking and better classification of activities involving multiple objects.
- Ablation Study: 16% gain results from combining attention and output pooling, while bias control adds an additional 6% recognition-accuracy improvement.The combined components increase capacity to classify both actions and objects.
- Ablation Study: 22% improvement over vanilla ConvLSTM is achieved by the complete LSTA model.The analysis attributes the improvement to attention pooling and output pooling that focus encoding on features relevant to the classification task.
- Stream and Fusion Analysis: 40.52% accuracy with motion attention exceeds the 36.21% baseline, while cross-modal fusion improves 1% over traditional late fusion.Motion attention focuses near the hands, whereas appearance attention focuses on objects; after cross-modal training, motion attention expands toward object regions.
- Comparative Analysis: LSTA gains 13% over eleGAtt by selecting attention maps from input-dependent weights rather than using one fixed weight matrix.The comparison also notes that ego-rnn generates per-frame attention without depending on previous-frame information, which can select different objects in adjacent frames.
7. Conclusion
LSTA extends LSTM with attention pooling and output pooling to track localized video representations. Combined with two-stream cross-modal fusion, it achieves state-of-the-art accuracy on four standard benchmarks.
- LSTA extends LSTM with attention pooling that spatially filters inputs and output pooling that exposes a distilled memory view.
- Both pooling mechanisms support smooth, focused tracking of latent video representations when discriminative features are spatially localized.
- A two-stream CNN-LSTA architecture uses novel cross-modal fusion for egocentric activity recognition.
- LSTA achieves state-of-the-art accuracy on four standard benchmarks.
8. Ablation Analysis
The ablation study shows that attention pooling and output pooling improve complementary recognition capabilities, while their combination with bias control produces the strongest gains.
- 13.79% gain in action recognition follows from replacing ConvLSTM output gating with output pooling.Output pooling localizes the active memory component and improves tracking of relevant spatiotemporal patterns.
- 20.69% gain in object recognition results from attention pooling that encodes features from spatially relevant areas.Attention pooling helps track active object regions, particularly for classes involving different actions or multiple objects.
- 13.72% action and 18.1% object recognition gains result when attention pooling and output pooling are combined.
- 22.41% gain in action recognition follows from adding bias control to output pooling in LSTA.The combined design further improves activity recognition capacity and active memory localization.
- Output pooling favors action recognition, attention pooling favors object recognition, and bias-controlled coupling improves both.The reported gains are +13.79% action versus +12,07% object for output pooling, +12,93% versus +16,38% for attention pooling, and +22.41% versus +21,55% after coupling.
9. Comparative Analysis
Compared with ego-rnn, LSTA is reported to identify activities involving multiple objects. The analysis attributes this capability to attention with memory for tracking previously attended regions.
- LSTA identifies activities involving multiple objects that are challenging for the ego-rnn comparison.Examples include variants involving mustard, hotdog, bread, honey, cup, and spoon.
- Attention with memory helps LSTA track previously attended regions during activity recognition.
10. Confusion Matrix
The confusion-matrix analysis aggregates available train/test splits into a single matrix for each dataset. The supplied passages do not state a specific confusion-matrix outcome.
- The supplied analysis identifies wrong regions in the frame but does not report a specific confusion-matrix comparison outcome.
- The analysis averages confusion matrices across available train/test splits to represent each dataset.
11. EPIC-KITCHENS
On EPIC-KITCHENS, the two-stream cross-modal fusion model improves activity recognition over the strongest TSN two-stream baseline in both reported settings. The section also uses confusion matrices and attention visualizations to examine class corrections and region tracking.
- EPIC-KITCHENS: 30.33% and 16.63% activity recognition performance are obtained on EPIC-KITCHENS S1 and S2, respectively, versus 20.54% and 10.89% for the TSN strongest baseline.The comparison evaluates verb, noun, and activity-class prediction settings described for the dataset.
- Ablation and comparison analyses: The pooling and fusion analyses plot corrected samples by true-label/predicted-label pairs and show corresponding differences between confusion matrices.The comparisons cover output pooling, attention pooling, their combination with bias control, LSTA against ego-rnn and eleGAtt-LSTM, and cross-modal fusion against late fusion.
- Confusion matrices: Confusion matrices are averaged across four train/test splits for GTEA 61 and GTEA 71, and across three splits for EGTEA Gaze+.
- Attention visualizations: Attention visualizations uniformly sample five frames from 25-frame inputs and compare ego-rnn, LSTA, and motion-stream attention maps across activity sequences.The displayed sequences include scoop sugar, take water, shake tea, take bread, and take spoon.