PaperlayerView exact revision on arXiv

Source-linked AI summary

Rolling-Unrolling LSTMs for Action Anticipation from First-Person Video

Antonino Furnari, Giovanni Maria Farinella

arXiv:2005.02190v2cs.CV

TL;DR

Egocentric action anticipation asks systems to predict future actions from first-person video, including the objects the camera wearer will interact with. The paper proposes Rolling-Unrolling LSTM with sequence-completion pre-training and modality attention to combine RGB, optical flow, and object features. Across benchmark datasets, it reports state-of-the-art action anticipation performance, competitive ActivityNet results, and generalization to related recognition tasks.

  • Problem

    Egocentric action anticipation requires predicting future actions and interacted objects from incomplete first-person observations.

    Source: §1 · p. 1

  • Method

    Rolling-Unrolling LSTM separates past encoding from future inference, while Sequence Completion Pre-training and Modality ATTention support disentanglement and multimodal fusion.

    Source: §3.4 · p. 6, §7 · p. 14

  • Results

    The approach achieves state-of-the-art action anticipation results across first-person and third-person scenarios and generalizes to early action recognition and action recognition.

    Source: §7 · p. 14

  • Takeaways & Limitations

    The method provides a unified approach for multimodal action anticipation and related recognition tasks across three benchmark datasets.

    Source: §7 · p. 14

    For action recognition, the method does not generally outperform competing approaches, although it achieves competitive results in some cases.

    Source: §6.2 · p. 13

Abstract

from arXiv · show

In this paper, we tackle the problem of egocentric action anticipation, i.e., predicting what actions the camera wearer will perform in the near future and which objects they will interact with. Specifically, we contribute Rolling-Unrolling LSTM, a learning architecture to anticipate actions from egocentric videos. The method is based on three components: 1) an architecture comprised of two LSTMs to model the sub-tasks of summarizing the past and inferring the future, 2) a Sequence Completion Pre-Training technique which encourages the LSTMs to focus on the different sub-tasks, and 3) a Modality ATTention (MATT) mechanism to efficiently fuse multi-modal predictions performed by processing RGB frames, optical flow fields and object-based features. The proposed approach is validated on EPIC-Kitchens, EGTEA Gaze+ and ActivityNet. The experiments show that the proposed architecture is state-of-the-art in the domain of egocentric videos, achieving top performances in the 2019 EPIC-Kitchens egocentric action anticipation challenge. The approach also achieves competitive performance on ActivityNet with respect to methods not based on unsupervised pre-training and generalizes to the tasks of early action recognition and action recognition. To encourage research on this challenging topic, we made our code, trained models, and pre-extracted features available at our web page: http://iplab.dmi.unict.it/rulstm.

1 INTRODUCTION

Egocentric action anticipation predicts future actions from incomplete first-person observations, a challenging task because past events must be related to uncertain future events. The paper separates this problem into encoding past observations and inferring future actions, introducing Rolling-Unrolling LSTM and complementary training and fusion techniques.

Problem

  • The task is challenging because models must relate past and future events despite incomplete observations and uncertain predictions.Multiple possible actions are therefore evaluated with Top-k measures.Source: §1 · p. 1
  • Egocentric action anticipation predicts a future action from an observed video segment before the action begins.The task uses observation and anticipation times to define how much video is available and how far ahead the prediction must be made.Source: §1 · p. 1

Approach

  • The proposed Rolling-Unrolling LSTM disentangles summarizing the past from inferring the future with two separate LSTMs.The Rolling LSTM continuously updates a summary of streaming observations, while the Unrolling LSTM generates future hypotheses when anticipation is required.Source: §1 · pp. 1–2
  • The work introduces Sequence Completion Pre-training and adaptive multimodal fusion with Modality ATTention.These techniques are presented as problem-specific additions to the Rolling-Unrolling architecture.Source: §1 · p. 2

Contributions

  • The study systematically evaluates egocentric action anticipation and releases the approach’s code to support further research.The contributions include benchmarking existing approaches within the EPIC-Kitchens framework and reporting improvements over the state of the art.Source: §1 · p. 2

2 RELATED WORK

Prior work spans action recognition, early recognition, and anticipation across third-person and first-person video. This paper focuses on predicting future actions from egocentric video, distinguishing anticipation from recognizing actions that are already partially observed.

Action recognition

  • Related research covers hand-designed and learned representations, multimodal cues, temporal modeling, and object-based features for video action recognition.Egocentric recognition studies additionally use cues such as gaze, hands, objects, head motion, and temporal pooling.Source: §2.1 · pp. 2–3

Early action recognition

  • Early action recognition identifies an ongoing action as early as possible from partial observations.Prior approaches include sparse coding, structured prediction, sequential detectors, LSTMs, and dedicated loss functions.Source: §2.2 · p. 3

Action anticipation

  • Unlike early recognition, action anticipation predicts an action before it starts and before it can be partially observed.The paper nevertheless adapts ideas from early recognition, including LSTMs for streaming observations and dedicated loss functions.Source: §2.2 · p. 3

Third-person anticipation

  • Third-person anticipation research has used future-path prediction, dual-agent interactions, hierarchical representations, autoregressive models, LSTMs, object affordances, and future-frame regression.These studies address varied anticipation settings, including driving maneuvers and responses between interacting agents.Source: §2.3 · p. 3

First-person anticipation

  • Prior first-person anticipation studies address varied tasks and datasets, whereas this work targets the egocentric action anticipation challenge proposed for EPIC-Kitchens.The paper notes that few previous works had addressed this challenge under the same framework.Source: §2.4 · pp. 3–4

3 PROPOSED APPROACH

The proposed approach uses Rolling-Unrolling LSTMs to separate past encoding from future action inference, support predictions at multiple temporal scales, and fuse RGB, optical-flow, and object-based modalities adaptively.

Rolling-Unrolling LSTMs

  • The encoder-decoder formulation allows the model to anticipate actions at different temporal scales by varying the U-LSTM unrolling duration.The number of unrolling steps is determined by the temporal position relative to the action onset.Source: §3.2 · pp. 4–5, §3.1 · p. 4
  • The architecture processes modality-specific video representations through recurrent branches that combine rolling encoding with unrolling-based future prediction.The model uses RGB frames, optical flow, and object-based features as separate modalities.Source: §3.2 · pp. 4–5
  • The R-LSTM recursively summarizes incoming representations, while the U-LSTM predicts future actions from the current representation and the R-LSTM state.The U-LSTM is iterated for a number of steps proportional to the current anticipation time.Source: §3.2 · p. 5

Sequence Completion Pre-Training

  • Sequence Completion Pre-Training modifies network connections so the U-LSTM processes future representations, encouraging the R-LSTM to focus on summarizing the past.After pre-training, the network is fine-tuned for action anticipation using the standard recurrent connections.Source: §3.3 · p. 6, Figure 5 · p. 6

Modality ATTention

  • Modality ATTention computes video-dependent fusion weights from modality-specific R-LSTM states and combines their action predictions with a weighted linear combination.This addresses cases where object detection is unreliable or optical flow is less informative because scene motion is limited.Source: §3.4 · p. 6

4 EXPERIMENTAL SETTINGS

The experiments evaluate action anticipation across egocentric and third-person video datasets using accuracy, recall, timeliness, and observation-ratio measures, alongside comparisons with established methods and baselines.

Datasets

  • Experiments use EPIC-Kitchens, EGTEA Gaze+, and ActivityNet, covering egocentric and third-person videos.EPIC-Kitchens contains 39,596 action annotations across 125 verbs and 352 nouns.Source: §4.1 · p. 7

Evaluation Measures

  • Top-5 accuracy and Mean Top-5 Recall measure class-agnostic and class-aware anticipation performance, respectively.A prediction is correct when the ground-truth action appears among the top-k predictions.Source: §4.2 · p. 8
  • The time to action measure captures the earliest anticipation time at which a correct top-k prediction is made.Mean time to action summarizes how early actions are anticipated across the test set; unsuccessful anticipations receive a score of zero.Source: §4.2 · p. 8
  • Minimum Observation Ratio measures the smallest fraction of an action observed before a correct Top-1 prediction.The observation ratio is the number of observed frames divided by the total number of frames.Source: §4.2 · p. 8

Compared Methods

  • The study evaluates verb, noun, and action predictions and compares the proposed method with state-of-the-art approaches and sequence-to-sequence baselines.Verb and noun scores are obtained by marginalizing action scores for most methods.Source: §4.2 · p. 8, §4.3 · p. 8

5 RESULTS

RU consistently outperforms competing methods on egocentric anticipation benchmarks, with especially strong noun performance, while remaining competitive on ActivityNet. Ablations attribute gains to rolling-unrolling processing, multimodal attention, sequence-completion pre-training, and suitable temporal context.

EPIC-Kitchens

  • RU outperforms prior approaches by about 5% in Top-5 action anticipation accuracy across anticipation times on EPIC-Kitchens validation.It also achieves 51.79% Top-5 noun accuracy and 49.90% mean Top-5 noun recall.Source: §5.1 · p. 9
  • RU outperforms competitors on both seen and unseen EPIC-Kitchens test scenes, with action improvements reaching +5.52% in S1 and +1.81% in S2.Noun improvements reach +8.23% in S1 and +2.76% in S2.Source: §5.1 · p. 9

EGTEA Gaze+

  • RU outperforms competitors on EGTEA Gaze+ at all anticipation times except τa = 2s, where it matches the LSTM baseline.The smaller margins are attributed to EGTEA Gaze+ having 106 actions versus 2,513 in EPIC-Kitchens.Source: §5.2 · p. 9

ActivityNet

  • On ActivityNet, RU is generally second-best behind ED and substantially outperforms ED*, which does not use unsupervised pre-training.This contrasts with the egocentric results while preserving competitive performance without unsupervised pre-training.Source: §5.3 · p. 10

Ablation study

  • MATT outperforms late fusion, while early fusion is consistently sub-optimal; all fusion schemes improve over single modalities.RGB+OBJ performs better than RGB+Flow, and combining all modalities gives the best performance.Source: §5.4 · p. 11
  • The complete proposed system improves over a strong three-modality baseline by +1.48% to +4.06% across anticipation times.Sequence Completion Pre-Training provides small but consistent improvements, and Senc = 6 performs best across most anticipation times.Source: §5.4 · pp. 11–12

6 ADDITIONAL RESULTS ON EARLY ACTION RECOGNITION AND ACTION RECOGNITION

The paper extends Rolling-Unrolling LSTM beyond anticipation to early action recognition and action recognition, with results varying across datasets and tasks.

Early Action Recognition

  • The sequence-to-sequence design produces predictions at every timestep, enabling both early action recognition and final-timestep action recognition.The models process action videos sequentially and output predictions throughout the sequence.Source: §6 · p. 12
  • EPIC-Kitchens results show the proposed method consistently outperforms competitors across observation rates and recognition measures.The evaluation reports Top-1 accuracy and MOR for verbs, nouns, and actions; lower MOR indicates recognition from fewer frames.Source: §6.1 · p. 12
  • On EGTEA Gaze+, RU is weaker than the LSTM baseline through 50% observation but becomes comparable at later observation rates.Its MOR values indicate marginally less video content is needed on average.Source: §6.1 · p. 12
  • On ActivityNet, RU performs comparably on average while achieving a MOR 5.8% smaller than competing methods.This corresponds to recognizing actions after observing 5.8% less video content on average.Source: §6.1 · p. 12

Action Recognition

  • For EPIC-Kitchens action recognition, RU remains competitive but trails the best methods by 4.44% in Top-1 and 3% in Top-5 accuracy.It nevertheless outperforms several baselines, including TSN, 2SCNN, TRN, and TSM.Source: §6.2 · p. 13
  • On EGTEA Gaze+, RU outperforms several recent approaches and remains close to the reported state of the art.Reported gains include +6.9% over Li et al. and +3.19% over Zhang et al.; gaps to two other methods are −0.56% and −1.66%.Source: §6.2 · p. 13

7 CONCLUSION

The paper concludes that Rolling-Unrolling LSTMs separate past summarization from future prediction through two LSTMs and sequence-completion pre-training. Multimodal attention supports predictions from RGB, flow, and object features, while experiments show broad task generalization and released resources.

Conclusion

  • Rolling-Unrolling LSTMs use separate LSTMs to disentangle encoding the past from inferring the future.The architecture targets egocentric action anticipation.Source: §7 · p. 14
  • Sequence-completion pre-training encourages the two LSTMs to specialize in their respective subtasks.This training technique is presented as part of the proposed architecture.Source: §7 · p. 14
  • A modality attention network fuses predictions from RGB, optical flow, and object-based feature branches.The three modalities are processed by separate branches before fusion.Source: §7 · p. 14
  • Experiments across three benchmark datasets report state-of-the-art action anticipation and generalization to early action recognition and action recognition.The conclusion covers both first-person and third-person anticipation scenarios.Source: §7 · p. 14
  • The authors released source code, pretrained models, and extracted features through the project webpage.The release is intended to encourage research on egocentric action anticipation.Source: §7 · p. 14

A IMPLEMENTATION AND TRAINING DETAILS OF THE PROPOSED METHOD

This section introduces the implementation and training details of the proposed method and directs readers to the online code for further implementation information.

Implementation and Training Details

A.1 Architectural Details

The architecture uses Inception-based spatial and motion representations, Faster R-CNN object features, and two 1024-unit LSTMs with strong input dropout.

Architectural Details

  • Batch Normalized Inception supplies the representation functions for the spatial and motion branches.These functions are denoted φ1 and φ2.Source: §A.1 · p. 15
  • The object branch uses a Faster R-CNN detector with a ResNet-101 backbone.The detector follows the implementation specified in the paper.Source: §A.1 · p. 15
  • Both the Rolling and Unrolling LSTMs contain one hidden layer with 1024 units.The architecture applies dropout with p = 0.8 to each LSTM input and the final classifier input.Source: §A.1 · p. 15

A.2 Training Procedure

The training procedure pre-trains each RU-LSTM branch with Sequence Completion Pre-training, fine-tunes the branches for anticipation, and then jointly fine-tunes the assembled MATT network. Early stopping selects models using validation performance, with task-specific criteria for anticipation and early recognition.

A.2 Training Procedure

  • Each training stage uses SGD with cross entropy, learning rate 0.01, and momentum 0.9, averaging loss across samples and prediction timestamps.Source: §A.2 · p. 15
  • The three trained branches are assembled with MATT and jointly fine-tuned for 100 epochs using cross entropy.Source: §A.2 · p. 15
  • For early action recognition, early stopping selects the epoch with the best average Top-1 action accuracy across observation rates.Source: §A.2 · p. 15
  • For anticipation, early stopping selects iterations with the best Top-5 action accuracy at anticipation time τa = 1s on the validation set.Source: §A.2 · p. 15

B IMPLEMENTATION AND TRAINING DETAILS OF THE COMPARED METHODS

Because the compared methods lacked official public implementations, the authors conducted experiments using their own implementations and report their implementation details.

B IMPLEMENTATION AND TRAINING DETAILS OF THE COMPARED METHODS

B.1 Deep Multimodal Regressor (DMR)

The compared methods were implemented with specified multimodal, temporal, loss, and fusion settings to support fair evaluation against the proposed architecture.

B.1 Deep Multimodal Regressor (DMR)

  • The Deep Multimodal Regressor uses three multimodal branches with interleaved units and a BNInception backbone replacing the originally used AlexNet.Source: §B.1 · p. 15
  • DMR early stopping selects the MLP iteration achieving the highest Top-5 action accuracy on the validation set.Source: §B.1 · pp. 15–16
  • The TSN-based comparison model predicts verbs and nouns separately, then computes action probabilities as p(a = (v, n)|x) = p(v|x) · p(n|x).Source: §B.2 · p. 16
  • Another TSN baseline uses the Verb-Noun Marginal Cross Entropy Loss and the authors’ official implementation.Source: §B.3 · p. 16
  • The implementation following uses RGB and Flow features but omits reinforcement learning because the comparison does not target earliest action-background discrimination.Source: §B.4 · p. 16
  • The architecture uses an optional LSTM, concatenation fusion, and two-stream RGB and Flow processing with equal-weight late fusion.Source: §B.5 · p. 16
  • The remaining baseline uses a five-layer network with kernel size 7, input dropout p = 0.8, and cross-entropy training.Source: §B.6 · p. 16
  • Three additional methods use a single LSTM with the proposed RGB and Flow features and equal-weight late fusion.Source: §B.7 · p. 16

C ADDITIONAL QUALITATIVE EXAMPLES

Additional qualitative examples show that MATT can rely on contextual appearance or object detections, while detection failures and misleading detected objects produce anticipation errors.

C ADDITIONAL QUALITATIVE EXAMPLES

Related papers

Loading 2005.02190v2…