PaperlayerView exact revision on arXiv

Source-linked AI summary

Rolling-Unrolling LSTMs for Action Anticipation from First-Person Video

Antonino Furnari, Giovanni Maria Farinella

arXiv:2005.02190v2cs.CV

TL;DR

Egocentric action anticipation asks systems to predict future actions from first-person video, including the objects the camera wearer will interact with. The paper proposes Rolling-Unrolling LSTM with sequence-completion pre-training and modality attention to combine RGB, optical flow, and object features. Across benchmark datasets, it reports state-of-the-art action anticipation performance, competitive ActivityNet results, and generalization to related recognition tasks.

  • Problem

    Egocentric action anticipation requires predicting future actions and interacted objects from incomplete first-person observations.

  • Method

    Rolling-Unrolling LSTM separates past encoding from future inference, while Sequence Completion Pre-training and Modality ATTention support disentanglement and multimodal fusion.

  • Results

    The approach achieves state-of-the-art action anticipation results across first-person and third-person scenarios and generalizes to early action recognition and action recognition.

  • Takeaways & Limitations

    The method provides a unified approach for multimodal action anticipation and related recognition tasks across three benchmark datasets.

  • Takeaways & Limitations

    For action recognition, the method does not generally outperform competing approaches, although it achieves competitive results in some cases.

Abstract

from arXiv · show

In this paper, we tackle the problem of egocentric action anticipation, i.e., predicting what actions the camera wearer will perform in the near future and which objects they will interact with. Specifically, we contribute Rolling-Unrolling LSTM, a learning architecture to anticipate actions from egocentric videos. The method is based on three components: 1) an architecture comprised of two LSTMs to model the sub-tasks of summarizing the past and inferring the future, 2) a Sequence Completion Pre-Training technique which encourages the LSTMs to focus on the different sub-tasks, and 3) a Modality ATTention (MATT) mechanism to efficiently fuse multi-modal predictions performed by processing RGB frames, optical flow fields and object-based features. The proposed approach is validated on EPIC-Kitchens, EGTEA Gaze+ and ActivityNet. The experiments show that the proposed architecture is state-of-the-art in the domain of egocentric videos, achieving top performances in the 2019 EPIC-Kitchens egocentric action anticipation challenge. The approach also achieves competitive performance on ActivityNet with respect to methods not based on unsupervised pre-training and generalizes to the tasks of early action recognition and action recognition. To encourage research on this challenging topic, we made our code, trained models, and pre-extracted features available at our web page: http://iplab.dmi.unict.it/rulstm.

1 INTRODUCTION

Egocentric action anticipation predicts future actions from incomplete first-person observations, a challenging task because past events must be related to uncertain future events. The paper separates this problem into encoding past observations and inferring future actions, introducing Rolling-Unrolling LSTM and complementary training and fusion techniques.

  • Problem: The task is challenging because models must relate past and future events despite incomplete observations and uncertain predictions.Multiple possible actions are therefore evaluated with Top-k measures.
  • Problem: Egocentric action anticipation predicts a future action from an observed video segment before the action begins.The task uses observation and anticipation times to define how much video is available and how far ahead the prediction must be made.
  • Approach: The proposed Rolling-Unrolling LSTM disentangles summarizing the past from inferring the future with two separate LSTMs.The Rolling LSTM continuously updates a summary of streaming observations, while the Unrolling LSTM generates future hypotheses when anticipation is required.
  • Approach: The work introduces Sequence Completion Pre-training and adaptive multimodal fusion with Modality ATTention.These techniques are presented as problem-specific additions to the Rolling-Unrolling architecture.
  • Contributions: The study systematically evaluates egocentric action anticipation and releases the approach’s code to support further research.The contributions include benchmarking existing approaches within the EPIC-Kitchens framework and reporting improvements over the state of the art.

2 RELATED WORK

Prior work spans action recognition, early recognition, and anticipation across third-person and first-person video. This paper focuses on predicting future actions from egocentric video, distinguishing anticipation from recognizing actions that are already partially observed.

  • Action recognition: Related research covers hand-designed and learned representations, multimodal cues, temporal modeling, and object-based features for video action recognition.Egocentric recognition studies additionally use cues such as gaze, hands, objects, head motion, and temporal pooling.
  • Early action recognition: Early action recognition identifies an ongoing action as early as possible from partial observations.Prior approaches include sparse coding, structured prediction, sequential detectors, LSTMs, and dedicated loss functions.
  • Action anticipation: Unlike early recognition, action anticipation predicts an action before it starts and before it can be partially observed.The paper nevertheless adapts ideas from early recognition, including LSTMs for streaming observations and dedicated loss functions.
  • Third-person anticipation: Third-person anticipation research has used future-path prediction, dual-agent interactions, hierarchical representations, autoregressive models, LSTMs, object affordances, and future-frame regression.These studies address varied anticipation settings, including driving maneuvers and responses between interacting agents.
  • First-person anticipation: Prior first-person anticipation studies address varied tasks and datasets, whereas this work targets the egocentric action anticipation challenge proposed for EPIC-Kitchens.The paper notes that few previous works had addressed this challenge under the same framework.

3 PROPOSED APPROACH

The proposed approach uses Rolling-Unrolling LSTMs to separate past encoding from future action inference, support predictions at multiple temporal scales, and fuse RGB, optical-flow, and object-based modalities adaptively.

  • Rolling-Unrolling LSTMs: The encoder-decoder formulation allows the model to anticipate actions at different temporal scales by varying the U-LSTM unrolling duration.The number of unrolling steps is determined by the temporal position relative to the action onset.
  • Rolling-Unrolling LSTMs: The architecture processes modality-specific video representations through recurrent branches that combine rolling encoding with unrolling-based future prediction.The model uses RGB frames, optical flow, and object-based features as separate modalities.
  • Rolling-Unrolling LSTMs: The R-LSTM recursively summarizes incoming representations, while the U-LSTM predicts future actions from the current representation and the R-LSTM state.The U-LSTM is iterated for a number of steps proportional to the current anticipation time.
  • Sequence Completion Pre-Training: Sequence Completion Pre-Training modifies network connections so the U-LSTM processes future representations, encouraging the R-LSTM to focus on summarizing the past.After pre-training, the network is fine-tuned for action anticipation using the standard recurrent connections.
  • Modality ATTention: Modality ATTention computes video-dependent fusion weights from modality-specific R-LSTM states and combines their action predictions with a weighted linear combination.This addresses cases where object detection is unreliable or optical flow is less informative because scene motion is limited.

4 EXPERIMENTAL SETTINGS

The experiments evaluate action anticipation across egocentric and third-person video datasets using accuracy, recall, timeliness, and observation-ratio measures, alongside comparisons with established methods and baselines.

  • Datasets: Experiments use EPIC-Kitchens, EGTEA Gaze+, and ActivityNet, covering egocentric and third-person videos.EPIC-Kitchens contains 39,596 action annotations across 125 verbs and 352 nouns.
  • Evaluation Measures: Top-5 accuracy and Mean Top-5 Recall measure class-agnostic and class-aware anticipation performance, respectively.A prediction is correct when the ground-truth action appears among the top-k predictions.
  • Evaluation Measures: The time to action measure captures the earliest anticipation time at which a correct top-k prediction is made.Mean time to action summarizes how early actions are anticipated across the test set; unsuccessful anticipations receive a score of zero.
  • Evaluation Measures: Minimum Observation Ratio measures the smallest fraction of an action observed before a correct Top-1 prediction.The observation ratio is the number of observed frames divided by the total number of frames.
  • Compared Methods: The study evaluates verb, noun, and action predictions and compares the proposed method with state-of-the-art approaches and sequence-to-sequence baselines.Verb and noun scores are obtained by marginalizing action scores for most methods.

5 RESULTS

RU consistently outperforms competing methods on egocentric anticipation benchmarks, with especially strong noun performance, while remaining competitive on ActivityNet. Ablations attribute gains to rolling-unrolling processing, multimodal attention, sequence-completion pre-training, and suitable temporal context.

  • EPIC-Kitchens: RU outperforms prior approaches by about 5% in Top-5 action anticipation accuracy across anticipation times on EPIC-Kitchens validation.It also achieves 51.79% Top-5 noun accuracy and 49.90% mean Top-5 noun recall.
  • EPIC-Kitchens: RU outperforms competitors on both seen and unseen EPIC-Kitchens test scenes, with action improvements reaching +5.52% in S1 and +1.81% in S2.Noun improvements reach +8.23% in S1 and +2.76% in S2.
  • EGTEA Gaze+: RU outperforms competitors on EGTEA Gaze+ at all anticipation times except τa = 2s, where it matches the LSTM baseline.The smaller margins are attributed to EGTEA Gaze+ having 106 actions versus 2,513 in EPIC-Kitchens.
  • ActivityNet: On ActivityNet, RU is generally second-best behind ED and substantially outperforms ED*, which does not use unsupervised pre-training.This contrasts with the egocentric results while preserving competitive performance without unsupervised pre-training.
  • Ablation study: MATT outperforms late fusion, while early fusion is consistently sub-optimal; all fusion schemes improve over single modalities.RGB+OBJ performs better than RGB+Flow, and combining all modalities gives the best performance.
  • Ablation study: The complete proposed system improves over a strong three-modality baseline by +1.48% to +4.06% across anticipation times.Sequence Completion Pre-Training provides small but consistent improvements, and Senc = 6 performs best across most anticipation times.

6 ADDITIONAL RESULTS ON EARLY ACTION RECOGNITION AND ACTION RECOGNITION

The paper extends Rolling-Unrolling LSTM beyond anticipation to early action recognition and action recognition, with results varying across datasets and tasks.

  • Early Action Recognition: The sequence-to-sequence design produces predictions at every timestep, enabling both early action recognition and final-timestep action recognition.The models process action videos sequentially and output predictions throughout the sequence.
  • Early Action Recognition: EPIC-Kitchens results show the proposed method consistently outperforms competitors across observation rates and recognition measures.The evaluation reports Top-1 accuracy and MOR for verbs, nouns, and actions; lower MOR indicates recognition from fewer frames.
  • Early Action Recognition: On EGTEA Gaze+, RU is weaker than the LSTM baseline through 50% observation but becomes comparable at later observation rates.Its MOR values indicate marginally less video content is needed on average.
  • Early Action Recognition: On ActivityNet, RU performs comparably on average while achieving a MOR 5.8% smaller than competing methods.This corresponds to recognizing actions after observing 5.8% less video content on average.
  • Action Recognition: For EPIC-Kitchens action recognition, RU remains competitive but trails the best methods by 4.44% in Top-1 and 3% in Top-5 accuracy.It nevertheless outperforms several baselines, including TSN, 2SCNN, TRN, and TSM.
  • Action Recognition: On EGTEA Gaze+, RU outperforms several recent approaches and remains close to the reported state of the art.Reported gains include +6.9% over Li et al. and +3.19% over Zhang et al.; gaps to two other methods are −0.56% and −1.66%.

7 CONCLUSION

The paper concludes that Rolling-Unrolling LSTMs separate past summarization from future prediction through two LSTMs and sequence-completion pre-training. Multimodal attention supports predictions from RGB, flow, and object features, while experiments show broad task generalization and released resources.

  • Conclusion: Rolling-Unrolling LSTMs use separate LSTMs to disentangle encoding the past from inferring the future.The architecture targets egocentric action anticipation.
  • Conclusion: Sequence-completion pre-training encourages the two LSTMs to specialize in their respective subtasks.This training technique is presented as part of the proposed architecture.
  • Conclusion: A modality attention network fuses predictions from RGB, optical flow, and object-based feature branches.The three modalities are processed by separate branches before fusion.
  • Conclusion: Experiments across three benchmark datasets report state-of-the-art action anticipation and generalization to early action recognition and action recognition.The conclusion covers both first-person and third-person anticipation scenarios.
  • Conclusion: The authors released source code, pretrained models, and extracted features through the project webpage.The release is intended to encourage research on egocentric action anticipation.

A IMPLEMENTATION AND TRAINING DETAILS OF THE PROPOSED METHOD

This section introduces the implementation and training details of the proposed method and directs readers to the online code for further implementation information.

  • Implementation and Training Details: The section documents implementation and training details for the components of the proposed method.It serves as the implementation-details section for the approach.
  • Implementation and Training Details: The online code provides an additional implementation reference for the proposed approach.The paper points readers to the project code for implementation details.
  • Implementation and Training Details: The implementation reference is provided through the Rolling-Unrolling LSTM project website.The cited webpage hosts the implementation materials referenced by the section.

A.1 Architectural Details

The architecture uses Inception-based spatial and motion representations, Faster R-CNN object features, and two 1024-unit LSTMs with strong input dropout.

  • Architectural Details: Batch Normalized Inception supplies the representation functions for the spatial and motion branches.These functions are denoted φ1 and φ2.
  • Architectural Details: The object branch uses a Faster R-CNN detector with a ResNet-101 backbone.The detector follows the implementation specified in the paper.
  • Architectural Details: Both the Rolling and Unrolling LSTMs contain one hidden layer with 1024 units.The architecture applies dropout with p = 0.8 to each LSTM input and the final classifier input.

A.2 Training Procedure

The training procedure pre-trains each RU-LSTM branch with Sequence Completion Pre-training, fine-tunes the branches for anticipation, and then jointly fine-tunes the assembled MATT network. Early stopping selects models using validation performance, with task-specific criteria for anticipation and early recognition.

  • A.2 Training Procedure: Each training stage uses SGD with cross entropy, learning rate 0.01, and momentum 0.9, averaging loss across samples and prediction timestamps.
  • A.2 Training Procedure: The three trained branches are assembled with MATT and jointly fine-tuned for 100 epochs using cross entropy.
  • A.2 Training Procedure: For early action recognition, early stopping selects the epoch with the best average Top-1 action accuracy across observation rates.
  • A.2 Training Procedure: For anticipation, early stopping selects iterations with the best Top-5 action accuracy at anticipation time τa = 1s on the validation set.

B IMPLEMENTATION AND TRAINING DETAILS OF THE COMPARED METHODS

Because the compared methods lacked official public implementations, the authors conducted experiments using their own implementations and report their implementation details.

  • B IMPLEMENTATION AND TRAINING DETAILS OF THE COMPARED METHODS: The compared methods were evaluated using author implementations because no official public implementations were available.

B.1 Deep Multimodal Regressor (DMR)

The compared methods were implemented with specified multimodal, temporal, loss, and fusion settings to support fair evaluation against the proposed architecture.

  • B.1 Deep Multimodal Regressor (DMR): The Deep Multimodal Regressor uses three multimodal branches with interleaved units and a BNInception backbone replacing the originally used AlexNet.
  • B.1 Deep Multimodal Regressor (DMR): DMR early stopping selects the MLP iteration achieving the highest Top-5 action accuracy on the validation set.
  • B.1 Deep Multimodal Regressor (DMR): The TSN-based comparison model predicts verbs and nouns separately, then computes action probabilities as p(a = (v, n)|x) = p(v|x) · p(n|x).
  • B.1 Deep Multimodal Regressor (DMR): Another TSN baseline uses the Verb-Noun Marginal Cross Entropy Loss and the authors’ official implementation.
  • B.1 Deep Multimodal Regressor (DMR): The implementation following uses RGB and Flow features but omits reinforcement learning because the comparison does not target earliest action-background discrimination.
  • B.1 Deep Multimodal Regressor (DMR): The architecture uses an optional LSTM, concatenation fusion, and two-stream RGB and Flow processing with equal-weight late fusion.
  • B.1 Deep Multimodal Regressor (DMR): The remaining baseline uses a five-layer network with kernel size 7, input dropout p = 0.8, and cross-entropy training.
  • B.1 Deep Multimodal Regressor (DMR): Three additional methods use a single LSTM with the proposed RGB and Flow features and equal-weight late fusion.

C ADDITIONAL QUALITATIVE EXAMPLES

Additional qualitative examples show that MATT can rely on contextual appearance or object detections, while detection failures and misleading detected objects produce anticipation errors.

  • C ADDITIONAL QUALITATIVE EXAMPLES: MATT assigns little weight to the object branch when RGB context alone reliably anticipates the next actions.
  • C ADDITIONAL QUALITATIVE EXAMPLES: Object detection is crucial for correctly anticipating “put down spoon” once the spoon is detected.
  • C ADDITIONAL QUALITATIVE EXAMPLES: Failure cases include missing-chair detection, which prevents correct anticipation of “adjust chair.”
  • C ADDITIONAL QUALITATIVE EXAMPLES: Detecting “pan” or “jar” can mislead the model toward “take curry” or jar-related actions instead of the correct action.
  • C ADDITIONAL QUALITATIVE EXAMPLES: The additional qualitative examples are supplemented by videos made available on the authors’ web page.
Loading 2005.02190v2…