PaperlayerView exact revision on arXiv

Source-linked AI summary

Encouraging LSTMs to Anticipate Actions Very Early

Mohammad Sadegh Aliakbarian, Fatemeh Sadat Saleh, Mathieu Salzmann, Basura Fernando, Lars Petersson, Lars Andersson

arXiv:1703.07023v3cs.CV

TL;DR

Action anticipation asks how to identify an action from only partial video early enough for applications such as autonomous navigation. The paper combines context- and action-aware features in a multi-stage LSTM and introduces a loss favoring early correct predictions. Experiments show significant state-of-the-art gains across standard anticipation benchmarks.

  • Problem

    Action anticipation must predict the correct action from only a few initial frames, while existing losses remain limited for very early prediction such as 1% of a sequence.

  • Method

    The paper combines context-aware and action-aware features in a multi-stage LSTM and introduces a loss that encourages correct predictions as early as possible.

  • Results

    The approach significantly outperforms state-of-the-art action anticipation methods on UCF-101, UT-Interaction, and JHMDB21.

  • Takeaways & Limitations

    The method provides a unified action-anticipation approach that exploits both contextual and action-focused information while avoiding optical flow computation.

  • Takeaways & Limitations

    Existing loss functions remain limited for very early prediction, such as from 1% of the sequence.

Abstract

from arXiv · show

In contrast to the widely studied problem of recognizing an action given a complete sequence, action anticipation aims to identify the action from only partially available videos. As such, it is therefore key to the success of computer vision applications requiring to react as early as possible, such as autonomous navigation. In this paper, we propose a new action anticipation method that achieves high prediction accuracy even in the presence of a very small percentage of a video sequence. To this end, we develop a multi-stage LSTM architecture that leverages context-aware and action-aware features, and introduce a novel loss function that encourages the model to predict the correct class as early as possible. Our experiments on standard benchmark datasets evidence the benefits of our approach; We outperform the state-of-the-art action anticipation methods for early prediction by a relative increase in accuracy of 22.0% on JHMDB-21, 14.0% on UT-Interaction and 49.9% on UCF-101.

1. Introduction

Action anticipation requires predicting an action from only a few initial frames, motivating a method designed for highly accurate very-early prediction. The approach combines context- and action-aware features in a multi-stage LSTM with a loss that encourages correct predictions as early as possible.

  • Motivation: Action anticipation predicts the observed action as early as possible from only a few frames at the beginning of a video.This differs from recognition, which commonly uses complete sequences, and matters for applications that must react before an action is finalized.
  • Motivation: Existing losses encourage correct scores or ranks to increase over time, or increasingly penalize classification mistakes, but remain limited for prediction from 1% of a sequence.The limitation is reported specifically for very early prediction.
  • Method: The proposed loss models early ambiguity by avoiding excessive early false-positive penalties while penalizing false negatives from the beginning.It encourages high probability for the correct class as early as possible.
  • Method: The multi-stage LSTM jointly exploits global context-aware features and action-aware features focused on regions where the action occurs.The first stage extracts contextual information from the entire RGB image; the second merges it with localized, class-specific information.
  • Results: The approach significantly outperforms state-of-the-art action anticipation methods on UCF-101, UT-Interaction, and JHMDB21.The experiments also report that the new loss yields higher accuracy than existing losses for a given model.

2. Related Work

Prior action-anticipation work models temporal dynamics, ranking or score objectives, and future representations, while action modeling often emphasizes either context or localized action information. This paper combines context- and RGB-based action-aware representations in a multi-stage network that avoids expensive optical flow computation.

  • Action anticipation: Earlier anticipation methods model causal relationships, action dynamics, sub-actions, or partial histories to predict activities online.The cited approaches include integral and dynamic bag-of-words, spatial-temporal shape models, skeleton information, dynamic programming, and structural SVMs.
  • Action anticipation: CNN-LSTM approaches model spatial and temporal information, while ranking losses enforce increasing correct-class scores or margins over time.These methods extend anticipation objectives beyond conventional classification.
  • Action anticipation: Future-representation prediction targets visual features rather than future class labels, partly to support learning from unlabeled videos.The learned representation is not always related to the action itself.
  • Action modeling: Global representations provide context but may not focus on the action, whereas localized features can fail to exploit contextual information.Context may be unreliable because the same action can occur in different environments.
  • Action modeling: The proposed network uses RGB data to combine context-aware and action-aware information, explicitly leveraging appearance for localization without optical flow.This addresses the contextual and localization trade-off described for prior methods.

3. Our Approach

The approach combines a new early-anticipation loss with a multi-stage LSTM that progressively integrates context-aware and action-aware features for class prediction.

  • 3.1. A New Loss for Action Anticipation: The new loss encourages high probability for the correct class early while allowing early false positives caused by action ambiguity.False-negative penalties remain constant, whereas false-positive penalties increase linearly over time.
  • 3.2. Feature Extraction: The feature extraction network shares VGG-16 layers up to conv5-2 before branching into context-aware and action-aware sub-models.The context stream represents the global image, while the action stream uses a separate network to focus on action-relevant information.
  • 3.2.1. Context-Aware Feature Extraction: The context-aware sub-model extracts a deep representation of the whole scene for each activity and uses its fc7 output as context-aware features.Its final fully connected layer is adapted from ImageNet classification to the number of activities.
  • 3.2.2. Action-Aware Feature Extraction: The action-aware sub-model uses class activation maps with conv5-3 features to filter irrelevant regions and takes the resulting fc7 output as action-aware features.The class activation maps identify image regions contributing most to each class prediction.
  • 3.2.3. Sequence Learning for Action Anticipation: The multi-stage LSTM first processes context-aware features, then concatenates its hidden activations with action-aware features for refined class probabilities.The overall training objective combines losses from the context-aware and action-aware stages.
  • 3.2.3. Sequence Learning for Action Anticipation: During inference, the model averages predictions from all frames up to the current time, while the LSTM models long-range temporal information as more frames arrive.The network is trained with the proposed loss and produces increasingly accurate predictions as it processes the sequence.

4. Experiments

Experiments on three standard benchmarks show that the proposed approach anticipates actions accurately with very limited observations. Ablations further support the value of the new loss, temporal average pooling, and jointly learned context-aware and action-aware features.

  • Comparison to State-of-the-Art: On JHMDB-21, the method outperforms all baselines in both earliest and latest settings, including methods using optical flow or Fisher-vector features.Earliest prediction observes the first 20% of each sequence.
  • Comparison to State-of-the-Art: On UCF-101, the method outperforms context-only baselines when only the first 2 frames, approximately 1% of the sequence, are observed.The comparison uses baselines with alternative anticipation losses and highlights the joint use of context-aware and action-aware features.
  • Influence of the Loss Function: With only 2 frames observed, the proposed loss reaches accuracy comparable to other losses after 30–40 frames, corresponding to roughly 1 second earlier at 30 fps.The comparison is reported on UCF-101 as accuracy varies with the number of observed frames.
  • Influence of the Loss Function: Temporal average pooling increases accuracy, especially at very early stages, and performs best when combined with the proposed loss.The pooling strategy can generally be applied to action anticipation methods.
  • Influence of the Features: Jointly using context-aware and action-aware features consistently outperforms using either feature type individually on UCF-101.Single-feature models use one LSTM, whereas the joint model uses the multi-stage LSTM and the proposed anticipation loss.
  • Influence of the Features: Action-aware features outperform context-aware features on classes containing irrelevant motions or multiple actors, including Baseball pitch, Basketball, Cricket Shot, and Ice dancing.Reported accuracies are 66.1% vs. 58.2%, 83% vs. 76.4%, 65.1% vs. 58%, and 92.3% vs. 91.7%, respectively.

5. Conclusion

The paper concludes that its anticipation loss encourages earlier correct predictions and that its multi-stage LSTM benefits from combining context-aware and action-aware information. Across evaluated datasets, the approach significantly outperforms state-of-the-art anticipation methods.

  • 5. Conclusion: The proposed loss encourages correct predictions as early as possible and improves very early action anticipation over existing losses.The conclusion identifies early correct prediction as the loss’s purpose and reports benefits over existing alternatives.
  • 5. Conclusion: The multi-stage LSTM combines context-aware and action-aware information, whose joint use is supported by the experiments.The model and the feature combination are presented as complementary contributions.
  • 5. Conclusion: The approach significantly outperforms state-of-the-art action anticipation methods on all evaluated datasets.The conclusion reports this outcome across the datasets used in the paper.

Supplementary Material

The supplementary material extends evaluation beyond the main action-anticipation experiments to action recognition and additional optical-flow analyses.

  • Supplementary Material: The supplementary material compares the approach with state-of-the-art action recognition methods on UCF-101, JHMDB-21, and UT-Interaction.It also evaluates the effect of additional optical-flow features on both action recognition and anticipation.

Comparison to State-of-the-Art Action Recognition Methods

The paper compares its approach with state-of-the-art action recognition methods on three standard benchmarks using average accuracy over standard training and testing partitions.

  • Comparison to State-of-the-Art Action Recognition Methods: Action recognition comparisons cover UCF-101, JHMDB-21, and UT-Interaction using average accuracy over standard training and testing partitions.For UCF-101, the comparison reports baselines using only RGB images and activity labels.

Exploiting Optical Flow

Optical flow can be incorporated into the multi-stage LSTM through a two-stream design, improving UCF-101 anticipation accuracy when runtime is not a concern.

  • Architecture: The two-stream extension combines RGB and optical-flow features by merging flow features with the second stage of the multi-stage LSTM.Flow features are computed from stacks of 20 optical-flow frames.
  • Comparison: The two-stream approach achieved accuracy comparable to state-of-the-art methods using optical flow.
  • Optical-flow anticipation: 86.8% earliest and 91.8% latest prediction accuracy were achieved on UCF-101 with the flow-based approach.The method pads flow stacks with gray images when fewer than 10 frames are available.
  • Optical-flow anticipation: The optical-flow approach outperformed a two-stream action-recognition model adapted to anticipation, which achieved 83.2% earliest and 88.6% latest accuracy on UCF-101.

Effect of Different Feature Types

Experiments show that combining context-aware and action-aware features consistently improves recognition accuracy over using either feature type alone, while the proposed loss also improves results for a given model.

  • Feature combination: Combining context-aware and action-aware features consistently outperformed using either feature type individually across the evaluated settings.
  • Loss comparison: The proposed loss yielded higher accuracies than alternative losses for a given model.

Robustness to the Number of Hidden Units

The study evaluates 1024- and 2048-unit LSTMs because 512 units performed poorly on large UCF-101 data, while larger models were constrained by memory.

  • Hidden-unit count: 512 hidden units were insufficient to capture the complexity of large datasets such as UCF-101.
  • Hidden-unit count: The experiments compared 1024 and 2048 hidden units, while 4096 units were excluded because they would require too much memory.

Effect of the LSTM Architecture

The multi-stage LSTM benefits from processing context-aware features before action-aware features, outperforming simpler fusion strategies and a CNN-only variant.

  • Fusion strategies: Both multi-stage LSTMs outperformed single-stage concatenation and parallel-LSTM fusion, indicating the value of sequentially treating the two feature types.
  • Fusion order: Processing context-aware features first achieved higher accuracy than processing action-aware features first.The proposed order allows global information to be refined with localized action-aware features.
  • LSTM contribution: 80.5% earliest and 83.4% latest prediction accuracy were achieved by the complete multi-stage LSTM on UCF-101, compared with 69.53% and 73.80% for the CNN-only variant.The comparison evaluates the effect of explicitly training on multiple frames.
Loading 1703.07023v3…