PaperlayerView exact revision on arXiv

Source-linked AI summary

What Would You Expect? Anticipating Egocentric Actions with Rolling-Unrolling LSTMs and Modality Attention

Antonino Furnari, Giovanni Maria Farinella

arXiv:1905.09035v2cs.CVcs.AI

TL;DR

Egocentric action anticipation asks which objects and actions a camera wearer will interact with next, but requires prediction from incomplete first-person observations. RU-LSTM separates past encoding from future prediction with two LSTMs, processes RGB, optical flow, and object features, and fuses them using adaptive modality attention. Across two datasets, it outperforms prior methods and generalizes to early action recognition and action recognition.

  • Problem

    Egocentric action anticipation requires recognizing future actions from complex, incomplete first-person video so systems can react before actions occur.

  • Method

    RU-LSTM uses separate Rolling and Unrolling LSTMs for past summarization and future prediction, with RGB, optical-flow, object features, and Modality ATTention.

  • Results

    RU-LSTM outperforms prior art on egocentric anticipation and generalizes across datasets and the tasks of anticipation, early recognition, and recognition.

  • Takeaways & Limitations

    The architecture anticipates actions at multiple temporal scales while leveraging complementary visual modalities.

Abstract

from arXiv · show

Egocentric action anticipation consists in understanding which objects the camera wearer will interact with in the near future and which actions they will perform. We tackle the problem proposing an architecture able to anticipate actions at multiple temporal scales using two LSTMs to 1) summarize the past, and 2) formulate predictions about the future. The input video is processed considering three complimentary modalities: appearance (RGB), motion (optical flow) and objects (object-based features). Modality-specific predictions are fused using a novel Modality ATTention (MATT) mechanism which learns to weigh modalities in an adaptive fashion. Extensive evaluations on two large-scale benchmark datasets show that our method outperforms prior art by up to +7% on the challenging EPIC-Kitchens dataset including more than 2500 actions, and generalizes to EGTEA Gaze+. Our approach is also shown to generalize to the tasks of early action recognition and action recognition. Our method is ranked first in the public leaderboard of the EPIC-Kitchens egocentric action anticipation challenge 2019. Please see our web pages for code and examples: http://iplab.dmi.unict.it/rulstm - https://github.com/fpv-iplab/rulstm.

1. Introduction

Egocentric action anticipation requires predicting future actions from incomplete, long, object-rich first-person video so intelligent systems can react before actions occur. The proposed approach separates past summarization from future prediction and combines RGB, optical-flow, and object modalities for predictions at multiple anticipation times.

  • Motivation: Egocentric action anticipation predicts actions before they occur from complex first-person videos containing many objects and user-performed actions.The task is relevant to intelligent systems that must react before an action is completed or starts.
  • Proposed approach: The method disentangles past summarization and future prediction with separate Rolling and Unrolling LSTMs.The Rolling LSTM encodes streaming observations, while the Unrolling LSTM uses its states to predict future actions.
  • Proposed approach: RU anticipates actions at multiple temporal scales rather than only at a fixed anticipation time.The paper gives 2s, 1.5s, 1s, and 0.5s as example anticipation times.
  • Proposed approach: Sequence Completion Pre-training encourages the network to separate the two anticipation subtasks.
  • Proposed approach: RU processes RGB frames, optical flow, and object-based features, then adaptively fuses modality-specific predictions with Modality ATTention.

2. Related Work

Prior work studied action recognition and anticipation using multimodal inputs, temporal models, encoder-decoder designs, and object features, but these approaches do not directly solve egocentric anticipation. The paper positions RU within a relatively underexplored egocentric challenge and distinguishes anticipation from early recognition.

  • Action Recognition: Action recognition research has used multiple modalities, Temporal Segment Networks, and explicit object-based features for video analysis.
  • Action Recognition: Methods designed for action recognition, including TSN and early or late multimodal fusion, are not directly applicable to egocentric action anticipation.
  • Early Action Recognition in Third Person Vision: Early action recognition identifies an ongoing action from partial observations, whereas anticipation predicts an action before it starts and cannot observe it partially.
  • Action Anticipation in Third Person Vision: Third-person anticipation research has used LSTMs, encoder-decoder frameworks, object-specific features, and future-representation regression.The paper reports that direct regression of future representations does not achieve satisfactory performance in its scenario.
  • Anticipation in First Person Vision: Prior egocentric anticipation studies used varied problems and evaluation frameworks, limiting direct comparison with the challenge targeted by this work.

3. Proposed Approach

The proposed RU architecture processes egocentric video through modality-specific branches, separating past encoding from future prediction and adaptively fusing multimodal outputs with MATT.

  • Processing Strategy: RU processes video online in encoding and anticipation stages, consuming short snippets at regular intervals.The encoding stage summarizes observed video, while anticipation operates at least α seconds before the action occurs.
  • Rolling-Unrolling LSTM: Each modality branch uses a Rolling LSTM to recursively encode the current feature and summarize past observations.The architecture supports multiple identical branches, each receiving modality-specific feature vectors.
  • Rolling-Unrolling LSTM: The Unrolling LSTM initializes from the Rolling LSTM state and iterates the current snippet representation to predict future actions at different anticipation times.The number of unrolling iterations is set by the time-steps required to reach the action onset, encouraging predictions to differ across anticipation times.
  • Sequence Completion Pre-training: Sequence Completion Pre-training specializes the two LSTMs by feeding future representations to the Unrolling LSTM during pre-training.This procedure is intended to let the Rolling LSTM focus on summarizing past representations without anticipating the future.
  • Modality ATTention: MATT computes adaptive modality-importance scores from branch states and uses normalized weights to linearly combine modality-specific predictions.The mechanism addresses the finding that a modality’s usefulness can vary across processed samples.
  • Branches and Representation Functions: The model uses RGB, optical-flow, and object-based branches to represent appearance, motion, and objects present in the scene.The object representation sums detection confidence scores by class and discards object position, while retaining object presence information.

4. Experimental Settings

The experiments use EPIC-Kitchens and EGTEA Gaze+, evaluate top-k anticipation performance and timeliness, and compare RU with established anticipation and recognition baselines.

  • Datasets: Experiments use EPIC-Kitchens with 2,513 unique actions and EGTEA Gaze+ as large-scale egocentric-video benchmarks.EPIC-Kitchens contains 39,596 action annotations, 125 verbs, and 352 nouns; EGTEA Gaze+ contains 10,325 annotations.
  • Evaluation Measures: Top-5 accuracy measures class-agnostic correctness, while Mean Top-5 Recall averages class-wise Top-5 recall.On EPIC-Kitchens, recall is averaged over many-shot verbs, nouns, and actions.
  • Evaluation Measures: EPIC-Kitchens official-test results use Top-1 accuracy, Top-5 accuracy, Precision, and Recall, with verb and noun scores generally obtained by marginalizing action scores.The compared methods include DMR, ATSN, MCE, ED, and adapted early-action-recognition baselines.
  • Evaluation Measures: Mean time to action measures the earliest anticipation time at which a correct top-k prediction is made, averaged across the test set.If no correct prediction occurs at any time-step, time to action is set to zero.

5. Results

RU consistently outperforms competing methods for egocentric anticipation and early recognition, while its rolling-unrolling design, adaptive modality fusion, and sequence-completion pre-training each contribute to performance.

  • Anticipation Results: RU improves EPIC-Kitchens Top-5 action accuracy over prior art by about 5% across anticipation times.The margin is reported as significant given the dataset’s 2,513 action classes.
  • Anticipation Results: RU anticipates verbs, nouns, and actions on average 1.62, 1.11, and 0.76 seconds in advance, respectively.Mean time to action is reported for the proposed method’s predictions.
  • Anticipation Results: RU outperforms competitors on both EPIC-Kitchens seen and unseen test sets, with action improvements reaching +5.52% on S1 and +5.39% on S2.Noun improvements reach +8.8% on S1 and +4.88% on S2.
  • Anticipation Results: RU outperforms all competitors at every anticipation time on EGTEA Gaze+, although improvement margins are smaller than on EPIC-Kitchens.The smaller margins are attributed to EGTEA Gaze+ having 106 actions versus 2,513 in EPIC-Kitchens.
  • Ablation Study: Rolling-unrolling systematically improves a three-branch late-fusion baseline across all anticipation times, especially for the object branch.The full component combination improves over the strong baseline by +1.53% to +4.08%.
  • Ablation Study: MATT always outperforms late fusion, while early fusion is consistently sub-optimal and all fusion schemes improve over individual branches.Attention weights suggest that MATT emphasizes objects when RGB and Flow are less informative, and emphasizes RGB and Flow when objects are unnecessary.
  • Ablation Study: Sequence-completion pre-training yields small but consistent improvements for most anticipation times.The ablation isolates pre-training as one contributor to the cumulative improvement of the complete RU system.
  • Early Action Recognition: For early action recognition, RU consistently outperforms competitors at all observation rates on EPIC-Kitchens and by small, consistent margins on EGTEA Gaze+.On EPIC-Kitchens, RU reaches 33.09% after observing 75% of an action, compared with 34.07% when observing the full action.

6. Conclusion

RU-LSTM processes RGB, optical flow, and object-based features through two LSTMs and modality attention for egocentric action anticipation. Experiments report superiority over prior methods and generalization across datasets and tasks.

  • RU-LSTM combines RGB, optical flow, and object-based features with two LSTMs and modality attention.The architecture anticipates actions from egocentric video.
  • Experiments on two datasets show superiority over prior art and generalization across anticipation, early recognition, and recognition.

A. Implementation Details and Training Procedure of the Proposed Method

The implementation uses a documented architecture and training setup, with BNInception-based spatial and flow representations and ReLU activations in the attention network.

  • Implementation and training details are provided for the proposed and compared methods, with an architecture diagram supplied for reference.
  • BNInception produces 1024-dimensional representations for the spatial and flow branches, while Faster R-CNN with a ResNet-101 backbone supports the object branch.Both recurrent modules use one layer with 1024 hidden units, and dropout is applied to the inputs.
  • ReLU activation functions are used within the attention network.

A.2. Training Procedure of RU-LSTM and MATT

Training uses separate branch-wise procedures with sequence-completion pre-training, followed by fine-tuning and assembly of the three-branch architecture. Early stopping selects validation-performing models, while the architecture includes three modality branches.

  • A.2. Training Procedure of RU-LSTM and MATT: End-to-end training was challenging because the indirect relationship between input video and future actions encouraged over-fitting.
  • A.2. Training Procedure of RU-LSTM and MATT: Each branch is first trained with Sequence Completion Pre-training, then fine-tuned for action anticipation before assembling the complete three-branch architecture.Appearance and motion branches use 100 pre-training epochs, while the object branch uses 200.
  • A.2. Training Procedure of RU-LSTM and MATT: The experiments use three modalities—RGB, Flow, and OBJ—although the illustrative architecture example contains M = 2 modalities.
  • A.2. Training Procedure of RU-LSTM and MATT: Early stopping selects intermediate and final models using validation accuracy at anticipation time τa = 1s, with the same strategy applied across methods for fair comparison.For early action recognition, the selected epoch maximizes average Top-1 action accuracy across observation rates.

A.2.1 Note on End-To-End Training

The authors fix feature extractors because end-to-end alternatives over-fit or fail to improve performance. The model produces time-specific anticipations over a configurable range, including settings covering up to two seconds ahead.

  • A.2.1 Note on End-To-End Training: Fixed feature extractors were chosen after end-to-end alternatives produced poor sequence-completion performance or no improvement.The authors attribute this difficulty to the indirect relationship between observed scenes and future actions.
  • A.2.1 Note on End-To-End Training: Anticipations can be inconsistent because predictions update as more video evidence arrives, with later predictions more likely to be correct.
  • A.2.1 Note on End-To-End Training: Each prediction is tied to a specific anticipation time, allowing fixed-time anticipation by processing the corresponding buffered time-steps and discarding others.For τa = 1s, the model processes the last 11 time-steps.
  • A.2.1 Note on End-To-End Training: With α = 0.25s and Sant = 8, the model anticipates actions up to 2s ahead while retaining predictions at τa = 1s.The encoder length was set to Senc = 6 after its effect on performance was found to be light.

B. Implementation Details of the Compared Methods

The compared methods were reimplemented with specified backbones, fusion strategies, losses, and training procedures to support fair evaluation against the proposed architecture.

  • Compared methods were evaluated using independent implementations because no official public implementations were available.The experiments report implementation details for each compared method.
  • The Deep Multimodal Regressor was adapted to three multimodal branches and a BNInception backbone for fair comparison.The original AlexNet backbone was replaced with ImageNet-pretrained BNInception.
  • Training procedures included cross-entropy, Verb-Noun Marginal Cross Entropy, ranking loss, SGD, and validation-based early stopping.The compared methods used method-specific losses and, where specified, early stopping on validation performance.
  • The TSN-based baseline predicts verbs and nouns separately, then computes action probabilities as their product.At test time, p(a = (v, n)|x) = p(v|x) · p(n|x).
  • Several baselines used RGB and Flow features from the proposed representation functions, with late fusion or architecture-specific fusion.The compared models included two-stream variants and concatenation-based fusion.

C. Additional Results

Additional experiments show that RU performs strongly across expanded anticipation and recognition evaluations, including precision, recall, and a public competition ranking.

  • RU outperforms all competitors on EPIC-Kitchens anticipation precision and recall except average verb precision.The largest gains occur for noun precision and recall, while noun predictions remain balanced on S1.
  • RU outperforms competitors on most EPIC-Kitchens recognition measures despite being designed for action anticipation.The expanded evaluation includes precision and recall alongside the challenge’s other measures.
  • RU outperforms many recent EGTEA Gaze+ recognition approaches by significant margins and is comparable with other state-of-the-art methods.This result also demonstrates generalization beyond the anticipation task.
  • RULSTM participated in the EPIC-Kitchens egocentric action anticipation competition using an ensemble with BNInception- and ResNet-50-based features.The cited passage describes the competition submission and its ensemble construction.

E. Additional Qualitative Examples

Qualitative examples illustrate adaptive use of appearance and object modalities, while failures show that object-detection errors can misdirect anticipated actions.

  • MATT reduces the object-branch weight when RGB context alone is sufficient for anticipation.The example attributes reliable anticipation to contextual appearance features.
  • Object detection is essential in one example for correctly anticipating “put down spoon” once the spoon is detected.The example contrasts modality contributions across successful predictions.
  • Object-detector failures lead to missed or incorrect predictions, including “adjust chair,” “close fridge,” and “pour coffee.”Undetected or newly appearing objects cause wrong object associations and misleading action hypotheses.
  • Supplementary material provides additional success and failure examples, including applications to early action recognition.
Loading 1905.09035v2…