PaperlayerView exact revision on arXiv

Source-linked AI summary

Joint Hand Motion and Interaction Hotspots Prediction from Egocentric Videos

Shaowei Liu, Subarna Tripathi, Somdeb Majumdar, Xiaolong Wang

arXiv:2204.01696v1cs.CVcs.LG

TL;DR

Forecasting future egocentric interactions requires representing both hand movement and object contact, which action labels or future pixels do not directly provide. The paper automatically generates trajectory and hotspot labels and trains an Object-Centric Transformer to predict them probabilistically; it reports that OCT outperforms state-of-the-art approaches on the evaluated prediction tasks.

  • Problem

    Action labels do not reveal future movement or interaction targets, while predicting future pixels is challenging because of high-dimensional outputs and uncertainty; hand trajectories alone also miss dense object interactions.

    Source: §1 · p. 1

  • Method

    The paper automatically collects trajectory and hotspot labels, then uses OCT to jointly predict future hand trajectories and object-contact hotspots with probabilistic modeling.

    Source: §1 · pp. 1–2

  • Results

    OCT significantly outperforms state-of-the-art approaches on hand trajectory and interaction-hotspot prediction across Epic-Kitchens-55, Epic-Kitchens-100, and EGTEA Gaze+.

    Source: §1 · p. 2, §6 · p. 8

  • Takeaways & Limitations

    The paper reports that trajectory estimation helps hotspot prediction and that the predictions can benefit future-action classification.

    Source: §1 · p. 2

Abstract

from arXiv · show

We propose to forecast future hand-object interactions given an egocentric video. Instead of predicting action labels or pixels, we directly predict the hand motion trajectory and the future contact points on the next active object (i.e., interaction hotspots). This relatively low-dimensional representation provides a concrete description of future interactions. To tackle this task, we first provide an automatic way to collect trajectory and hotspots labels on large-scale data. We then use this data to train an Object-Centric Transformer (OCT) model for prediction. Our model performs hand and object interaction reasoning via the self-attention mechanism in Transformers. OCT also provides a probabilistic framework to sample the future trajectory and hotspots to handle uncertainty in prediction. We perform experiments on the Epic-Kitchens-55, Epic-Kitchens-100, and EGTEA Gaze+ datasets, and show that OCT significantly outperforms state-of-the-art approaches by a large margin. Project page is available at https://stevenlsw.github.io/hoi-forecast .

1. Introduction

The paper predicts future hand motion and object-contact hotspots rather than action labels or pixels, addressing the need to represent both movement and object interaction. It introduces automatic label collection and an Object-Centric Transformer (OCT), and reports strong results on both prediction tasks.

1. Introduction

  • The paper motivates forecasting egocentric activity for applications such as timely augmented-reality guidance and smoother human-robot collaboration.The introduction illustrates this motivation with a person acting in a kitchen.Source: §1 · p. 1
  • The paper jointly predicts future hand trajectories and contact hotspots on the next-active object, representing future interactions beyond a single action label.Future hand locations and object-contact points are modeled probabilistically, with hotspots conditioned on the predicted hand trajectory.Source: §1 · p. 1
  • An automatic annotation pipeline uses hand and object detections and projects future locations into the last observation frame to generate large-scale trajectory and hotspot labels.The labels are generated for Epic-Kitchens-55, Epic-Kitchens-100, and EGTEA Gaze+ without human labor.Source: §1 · pp. 1–2
  • OCT uses Transformer self-attention to reason over hand, object, and environment features, with a C-VAE decoder for probabilistic joint prediction.The decoder uses cross-attention between encoder outputs and predicted future hand locations.Source: §1 · p. 2
  • OCT significantly outperforms state-of-the-art approaches on hand trajectory and interaction-hotspot prediction across Epic-Kitchens and EGTEA Gaze+ datasets.The experiments also find that trajectory estimation helps hotspot prediction, more automatically annotated training data improves results, and the predictions can benefit future-action classification.Source: §1 · pp. 1–2

2. Related Work

Prior video-anticipation work predicts either high-dimensional future frames or semantic action labels, neither of which directly conveys both where a person moves and what they will interact with. This paper situates its low-dimensional hand-trajectory and hotspot prediction alongside work on egocentric motion, affordance grounding, and Transformers.

Video anticipation

  • Future-frame prediction targets high-dimensional visual representations, while action anticipation predicts semantic labels that do not specify intended movement or interaction targets.The paper instead predicts hand trajectories and interaction hotspots as low-dimensional indicators of motion and future interaction.Source: §2 · p. 2

Human motion forecasting

  • Earlier egocentric hand-trajectory work is described as relying on manual annotation and predicting only one hand; this work aims to collect large-scale data automatically and learn both hands.The paper notes that many prior motion-forecasting approaches use third-person or fixed bird’s-eye views.Source: §2 · p. 2

Grounded affordance prediction

  • Affordance work distinguishes holistic next-active-object segmentation from hotspot heatmaps, while this paper predicts compact contact locations in cluttered egocentric scenes.The task must locate the next-active object before grounding the interaction hotspot.Source: §2 · p. 2

Transformers for video forecasting

  • OCT uses object-centric hand, object, and environment tokens, Transformer self-attention, decoder cross-attention with predicted hand locations, and a C-VAE to represent future uncertainty.The encoder and decoder jointly support video forecasting of trajectories and interaction hotspots.Source: §2 · pp. 2–3

3. Problem Setup

The task is to forecast future hand trajectories and object contact points from observed video frames, with future locations expressed in the last observation frame’s coordinates. The paper describes an automatic label-generation process and feature-token preparation for prediction.

3.1. Task Description

  • Given T observation frames, the task predicts future hand trajectories H and object contact points O over F future steps, represented in the last observation frame’s coordinate system.The hands’ contact frame occurs at T+F; O contains up to N future contact locations.Source: §3.1 · p. 3

3.2. Training Data Generation

  • Training labels are generated by detecting hands and objects in future frames and projecting their locations back to the last observation frame using chained homographies.For hotspots, skin segmentation and fingertip detection within intersecting hand-object boxes provide contact points, which are projected back similarly.Source: §3.2 · p. 3

3.3. Pre-processing

  • Observation-frame features include hand, object, and global context tokens; hand and object RoI features are combined with center coordinates through MLPs, while global features are average-pooled.The resulting features are supplied as independent Transformer input tokens, and absent hand or object detections are represented by zero vectors.Source: §3.3 · p. 3

4. Object-Centric Transformer

OCT uses an encoder-decoder Transformer to reason over hand, object, and global video features and predict future hand trajectories and object contact points. Its C-VAE heads sample predictions, while training and inference use separate procedures for the two output types.

4. Object-Centric Transformer

  • OCT stacks attention and feed-forward blocks, using encoder self-attention and decoder cross-attention between predictions and encoder outputs.Attention weights values by scaled query-key dot products; a mask excludes padded keys.Source: §4 · pp. 3–4

4.1. Encoder

  • Each frame contributes hand, object, and global tokens with spatial and temporal embeddings; the encoder applies self-attention across tokens and time, masking padded detections.The encoder produces representations for the input frames.Source: §4.1 · p. 4

4.2. Decoder

  • The decoder predicts future hand features one step at a time, conditioning on the last observed hand location and previously generated hand locations.Decoder blocks cross-attend to the last observation frame’s encoder output and mask subsequent positions.Source: §4.2 · p. 4

4.3. Head Networks

  • The hand C-VAE predicts future hand locations from decoder hand features, using reconstruction loss and KL regularization of its sampled latent space.Its loss aggregates reconstruction over future time steps.Source: §4.3 · p. 5
  • The object C-VAE predicts future contact points conditioned on the last-frame global feature and future hand trajectory.Training uses ground-truth future hand trajectories as input; inference uses predicted trajectories, and the object loss combines reconstruction and KL terms.Source: §4.3 · p. 5

4.4. Training and Inference

  • Training combines hand and object losses, while inference samples trajectories and contact points 20 times per video and converts contact predictions to heatmaps for affordance evaluation.The object loss is weighted by λ=1e−1; trajectory evaluation reports the minimum among 20 samples.Source: §4.4 · p. 5

5. Experiments

Experiments on three egocentric datasets evaluate OCT on trajectory and hotspot prediction, while ablations probe stochastic heads, task dependencies, data scale, inputs, and transfer to action anticipation.

5.2. Datasets

  • Experiments use EK55, EK100, and EGTEA Gaze+ validation sets; trajectory metrics are ADE/FDE, while hotspot maps are assessed with SIM, AUC-J, and NSS.The implementation samples 10 frames at 4 FPS for Epic-Kitchens and 9 frames at 6 FPS for EGTEA Gaze+, then draws 20 C-VAE samples at inference.Source: §5.2 · p. 5, §5.1 · p. 5, §5.3 · p. 6

5.4. Comparison to the state-of-the-art

  • On EK100, OCT improves ADE by 50% and FDE by 27.3% over each metric’s second-best method, and SIM by +5%, AUC-J by +3%, and NSS by +25%.The trajectory comparison is against the second-best method for each metric; the hotspots comparison is likewise against each metric’s second-best method.Source: Table 1 · p. 6, §5.4 · p. 6, Table 2 · p. 6
  • When trained on Epic-Kitchens and tested on EGTEA Gaze+, OCT significantly outperforms other approaches across metrics on both prediction tasks.Source: Table 3 · p. 6, Table 4 · p. 6, §5.4 · p. 7

5.5. Ablations and Analysis

  • Against MLP, C-VAE improves trajectory ADE by 75.0% and FDE by 45.5%, and hotspots SIM by +5%, AUC-J by +10%, and NCC by +29%.Stochastic models outperform deterministic models on both tasks, and C-VAE also outperforms the output-space Bivariate and MDN alternatives.Source: Table 5 · p. 7, Table 6 · p. 7, §5.5 · p. 7
  • Conditioning hotspots on future hand trajectory gives the best results on both tasks, improving SIM by +3%, AUC-J by +5%, and NCC by +18% over the inverse conditioning order.The compared conditions are no dependency, hand trajectory conditioned on object contact point, and object contact points conditioned on hand trajectory.Source: Table 7 · p. 7, §5.5 · p. 7
  • Training on EK100 rather than EK55 improves both tasks on the manually annotated EK100 validation split, despite noise in the automatically generated training labels.Source: Table 8 · p. 7, §5.5 · p. 7
  • Removing global features drops ADE metric 25.0%; removing hand features drops FDE metric 45.5%, while longer observation context improves trajectory performance.The global features encode environmental context; the study also reports improvement as more observation frames are incorporated.Source: Table 9 · p. 7, Figure 4 · p. 7, §5.5 · p. 8
  • Fine-tuning interaction-pretrained OCT beats training from scratch across anticipation metrics; visualizations show single- and two-hand cases and diverse future predictions.The authors caution that these action-anticipation results are not fully comparable with state-of-the-art models because training and evaluation use a subset of samples.Source: Table 10 · p. 8, §5.5 · p. 8, Figure 5 · p. 8

6. Discussion

The paper concludes that OCT jointly forecasts future hand trajectories and interaction hotspots, outperforming state-of-the-art approaches. It also identifies potential label noise from off-the-shelf tools and outlines the supplementary material’s additional details and experiments.

6. Discussion

  • OCT jointly predicts future hand trajectories and interaction hotspots, significantly outperforming state-of-the-art approaches in the reported experiments.The authors also report benefits from stochastic modeling and conditioning trajectory and hotspot predictions on each other.Source: §6 · p. 8
  • Training labels may inherit bias and errors from off-the-shelf hand-object detectors and skin segmentation; the authors propose self-supervision to improve robustness to label noise.Source: §6 · p. 8

Appendix

  • The supplementary material covers label generation, evaluation annotations, implementation and architecture, additional comparisons, visualizations, and generalization results.Source: Appendix · p. 12

A. Training Labels Generation Details

Training labels are generated by projecting detected future hand locations back to the final observation frame, with interpolation for missing detections. Hotspot labels handle moving active objects separately, while evaluation annotations are collected from human point placements.

A. Training Labels Generation Details

B. Evaluation Annotations Details

C. Implementation Details

Implementation details specify model settings, inference handling for invisible hands, and dataset-specific observation and forecasting configurations. The supplementary figures illustrate hotspot mapping and the evaluation annotation interface.

C. Implementation Details

  • The Transformer uses 512-dimensional embeddings and 0.1 dropout in both encoding and decoding blocks.Source: C. Implementation Details · p. 12
  • For moving active objects, detected contact points are shifted along the future object trajectory to their original locations in the last observation frame.Source: Figure B.1 · p. 12
  • The evaluation interface asks users to mark the same object location in the last observation frame that the hand touches in the contact frame.Source: Figure B.2 · p. 12
  • At inference, invisible hands receive fixed normalized locations; training uses cosine-annealed learning-rate decay starting at 1e−4.The implementation uses PyTorch.Source: C. Implementation Details · p. 12
  • Epic-Kitchens uses 2.5-second observations, 1-second forecasts, 4-fps sampling, and 35 training epochs including 5 warmup epochs.Source: C. Implementation Details · p. 12
  • EGTEA Gaze+ uses 1.5-second observations, 0.5-second anticipation, 6-fps sampling, and 25 training epochs including 5 warmup epochs.Source: C. Implementation Details · p. 12

D. Network Architectures

On EK100, the proposed Transformer outperforms the 3D CNN baseline across all reported trajectory and hotspot metrics. The architecture uses fused visual and location inputs and samples future hand trajectories and object contact points through separate C-VAE heads.

D. Network Architectures

  • ROIAlign crops global, hand, and object features; these are fused with detected hand and object box locations into Transformer input tokens.Table B.1 presents the architecture and corresponding dimensionalities.Source: D. Network Architectures · p. 12, Table B.1 · p. 13
  • On EK100, the proposed Transformer outperforms the 3D CNN with I3D and ResNet-50 in trajectory estimation and hotspot prediction across all metrics.Source: Table B.2 · p. 13
  • The hand C-VAE samples future trajectory points autoregressively, while the object C-VAE samples the final contact points.Source: D. Network Architectures · p. 13

E. Additional Experiments

Additional experiments compare OCT with a 3D CNN and examine end-to-end training. OCT improves trajectory and hotspot metrics over the 3D CNN, while end-to-end training offers comparable performance at greater training cost.

End-to-end training

  • End-to-end training gives a slight trajectory gain but comparable overall performance, so the experiments freeze the backbone to reduce training time.Source: Table B.3 · p. 13

Training labels visualization

  • Figure E.3 visualizes automatically generated training labels on Epic-Kitchens, with future hand trajectories and interaction hotspots.Source: Figure E.3 · p. 13

Comparison to 3D CNNs

F. Training Labels Visualization

The visualizations show automatically generated labels across kitchen environments and subjects, then compare OCT predictions with prior methods and illustrate predictions in unseen settings.

F. Training Labels Visualization

G. Qualitative Comparisons

H. Generalization Results Visualization

The visualizations show reasonable predictions in unseen EK100 kitchens and participants, as well as cross-dataset predictions when training on Epic-Kitchens and testing on EGTEA Gaze+.

Related papers

Loading 2204.01696v1…