PaperlayerView exact revision on arXiv

Source-linked AI summary

GIMO: Gaze-Informed Human Motion Prediction in Context

Yang Zheng, Yanchao Yang, Kaichun Mo, Jiaman Li, Tao Yu, Yebin Liu, C. Karen Liu, Leonidas J. Guibas

arXiv:2204.09443v2cs.CV

TL;DR

Human motion prediction needs to account for scene context and intent, but existing datasets offer limited ego-centric gaze data and diversity. GIMO combines motion, gaze, and scene data and introduces bidirectional multimodal fusion; the authors report improved predictions across architectures and top performance for their method.

  • Problem

    Gaze-informed motion prediction is difficult to study because existing datasets often lack ego-centric gaze paired with 3D poses and diverse scenes and motions.

    Source: §3 · p. 5

  • Method

    The paper builds GIMO, a real-world dataset of motion, gaze, ego-centric views, and 3D scenes, and proposes bidirectional fusion between motion and gaze features.

    Source: §1 · p. 3, §6 · p. 15

  • Results

    The authors report better human motion predictions across multiple architectures and top performance for their proposed architecture on GIMO.

    Source: §1 · p. 3, §6 · p. 15

  • Takeaways & Limitations

    GIMO supports studying gaze-informed motion prediction and is also presented as useful for intention-aware motion synthesis and gaze-guided ego-centric pose estimation.

    Source: §6 · p. 15

    The method relies on 3D inputs; incorporating visual information from ego-centric images is left to future work.

    Source: §6 · p. 15

Abstract

from arXiv · show

Predicting human motion is critical for assistive robots and AR/VR applications, where the interaction with humans needs to be safe and comfortable. Meanwhile, an accurate prediction depends on understanding both the scene context and human intentions. Even though many works study scene-aware human motion prediction, the latter is largely underexplored due to the lack of ego-centric views that disclose human intent and the limited diversity in motion and scenes. To reduce the gap, we propose a large-scale human motion dataset that delivers high-quality body pose sequences, scene scans, as well as ego-centric views with the eye gaze that serves as a surrogate for inferring human intent. By employing inertial sensors for motion capture, our data collection is not tied to specific scenes, which further boosts the motion dynamics observed from our subjects. We perform an extensive study of the benefits of leveraging the eye gaze for ego-centric human motion prediction with various state-of-the-art architectures. Moreover, to realize the full potential of the gaze, we propose a novel network architecture that enables bidirectional communication between the gaze and motion branches. Our network achieves the top performance in human motion prediction on the proposed dataset, thanks to the intent information from eye gaze and the denoised gaze feature modulated by the motion. Code and data can be found at https://github.com/y-zheng18/GIMO.

1 Introduction

The paper argues that gaze can help resolve uncertainty when early movements look similar, and introduces a diverse gaze-informed dataset and bidirectional architecture. Its experiments report improved predictions across architectures and top performance for the proposed architecture.

1 Introduction

  • Distinctive gaze patterns can reveal different intentions even when the first movements are difficult to distinguish.The paper motivates gaze as a way to reduce uncertainty in predicting future body movements.Source: §1 · pp. 1–2, Figure 1 · p. 2
  • Existing datasets lack the aligned ego-centric views, gaze, and 3D poses needed to evaluate gaze-informed prediction, while available datasets have limited scene and motion diversity.The paper identifies both dataset quality and modality-aware fusion as requirements for studying gaze's role.Source: §1 · p. 2
  • The capture approach records scene context, eye gaze, and high-fidelity 3D body poses using IMUs, enabling varied long-horizon activities in diverse daily-living environments.It avoids cumbersome motion-capture-device setups and records accurate poses without 2D-3D lifting.Source: §1 · p. 3
  • The proposed architecture achieves top performance under different criteria, while experiments find better predictions across multiple architectures when gaze is used.The bidirectional fusion scheme lets motion and gaze inform one another during prediction.Source: §1 · p. 3
  • The proposed architecture uses bidirectional multimodal fusion so gaze and observed motion can reinforce each other's relevance for future predictions.The authors note that gaze and motion can both be ambiguous, such as when gaze falls on an unrelated salient object.Source: §1 · pp. 2–3

2 Related Work

Prior work supplies motion data, scene-aware prediction, and ego-centric observations, but the paper distinguishes GIMO by combining diverse scenes and motions with gaze and long-term intent-focused prediction.

2 Related Work

  • Earlier motion datasets often omit 3D scene context, while ego-centric datasets and interaction datasets do not provide the scene and motion diversity with intentions targeted by GIMO.The paper cites Human3.6M, GTA-1M, PROX, EgoPose, Kinpoly, HPS, You2Me, and EgoBody [57].Source: §2 · pp. 3–4
  • Prior motion-prediction work models motion dynamics, scene constraints, or auxiliary modalities, whereas this work targets long-term sparse motions to understand human intentions.The paper contrasts its objective with dense motion prediction and cites MoGaze as gaze-based motion-prediction work in full-body manipulation.Source: §2 · p. 4
  • Ego-centric pose-estimation work uses camera observations or another visible person, while this work predicts motion from past motions in an ego-centric setting.The paper describes its dataset as also potentially useful for ego-centric motion estimation.Source: §2 · p. 4

3 GIMO Dataset: Gaze and Motion with Scene Context

GIMO addresses limitations in existing motion datasets by combining high-quality human motion, ego-centric views with gaze, and 3D environments. Its collection and preparation pipeline captures diverse goal-directed activities and aligns the recorded modalities to scanned scenes.

3 GIMO Dataset: Gaze and Motion with Scene Context

  • GIMO is designed to provide synchronized, spatially aligned ego-centric views, gaze, 3D poses, and real-world scenes amid limited diversity in existing datasets.The paper notes that prior datasets may use virtual environments or camera arrays with limited scene and motion diversity, and usually lack gaze.Source: §3 · p. 5

3.1 Hardware Setup

  • IMU-based motion capture records body and hand movement, Hololens2 captures 3D gaze and ego-centric video, and iPhone LiDAR scans the 3D environment.SMPL-X fitting is used for full-body pose and hand gesture; the authors state that the pipeline avoids pose-estimation errors from noisy detections and occlusions and needs less environmental setup than some alternatives.Source: §3.1 · pp. 5–6

3.2 Data Collection with Human in Action

  • The dataset focuses on semantically meaningful, long-term daily activities; it contains 129k ego-centric images, 11 subjects, 217 trajectories, and 19 scenes.Subjects begin at a distance from a goal object and then move to a destination to perform an activity.Source: §3.2 · p. 7

3.3 Data Preparation

  • A hand gesture synchronizes motion capture and headset recordings; SMPL-X poses are fitted from recorded joints and aligned with scanned scenes using ICP and annotated body shapes.The hand pose fitting uses body and hand joints, while the scene alignment uses the Hololens2 depth reconstruction and high-quality scene scans.Source: §3.3 · pp. 7–8

4 Gaze-Informed Human Motion Prediction

Given past motion, 3D scene, and eye gaze, the framework predicts future motion by fusing scene, motion, and gaze features through bidirectional communication. It uses scene geometry to inform motion features and motion to refine gaze features.

4 Gaze-Informed Human Motion Prediction

  • The framework uses bidirectional fusion to integrate 3D scene geometry, past motion, and gaze for intention-aware future motion prediction.Cross-modal transformers facilitate communication between modalities.Source: §4 · pp. 8–9

4.1 Problem Definition

  • Motion is represented as a parametric pose sequence, and the model predicts future motion from past poses, corresponding gaze points, and a scene point cloud.Each pose frame includes global translation and orientation, body pose, shape, and hand pose.Source: §4.1 · p. 9

4.2 Multi-modal Feature Extraction

  • PointNet++ extracts local and global scene features, while gaze points and SMPL-X vertices query scene features to form gaze and motion embeddings.The gaze feature is interpolated from per-point scene features; queried vertex features form ambient scene context for motion.Source: §4.2 · pp. 9–10

4.3 Attention-Aware Multi-modal Feature Fusion

  • Scene context queries motion features before the resulting scene-aware motion embedding is used with gaze features to integrate scene and intention information.The gaze feature is the query input to the next motion-gaze transformer.Source: §4.3 · pp. 10–11
  • The updated gaze and motion features, together with the global scene feature, are composed to predict future motion.The prediction uses concatenated multimodal features and temporal positional encodings.Source: §4.3 · p. 11

5 Experiments

Gaze-informed methods reduce motion-prediction ambiguity, and the proposed method achieves more accurate destination poses than MultimodalNet. The experiments examine gaze, scene features, and cross-modal transformer design across known and new scenes.

5.1 Experimental Setup

  • Experiments predict five seconds of motion from three seconds of input, using six input poses and ten output poses at 2 fps.The 217 trajectories are split into 180 for training and 37 for testing, including known scenes and two new environments.Source: §5.1 · pp. 11–12, §5.2 · p. 12

5.2 Evaluation

  • In qualitative examples, gaze helps the method reach the correct destination in both a known-scene sitting activity and a new-environment eraser task.Without gaze, the method misses the destination; MultimodalNet with gaze reaches toward the wrong location in the eraser example.Source: Figure 4 · p. 12, §5.2 · pp. 13–14
  • The proposed method outperforms MultimodalNet in recognizing intent from gaze and predicting destination poses.The evaluation includes path errors in Table 4, but the supplied passages do not report its cell values.Source: §5.2 · p. 13, Table 4 · p. 14

5.3 Ablation Study

  • Gaze improves RNN and MultimodalNet accuracy, and its attention weights emphasize ending poses, consistent with gaze indicating motion goals.The gaze ablation also finds the method confused about future destinations without gaze.Source: Figure 5 · p. 13, §5.3 · p. 14
  • PointNet scene features perform well on training scenes but lose competitiveness in new environments with different 3D structures.The proposed PointNet++ per-point scene features are compared with a global scene feature from PointNet.Source: §5.3 · p. 14
  • Replacing the cross-modal transformer with a vanilla transformer reduces accuracy, with new-scene path error exceeding the no-gaze result.The authors report that the cross-modal transformer is more effective for capturing correlations among multimodal inputs.Source: §5.3 · p. 14

6 Conclusion, Discussion and Future Work

The paper presents GIMO, a real-world dataset, and a gaze-informed motion-prediction framework. The framework uses 3D inputs and achieves promising long-term prediction results; visual ego-centric inputs are left for future work.

6 Conclusion, Discussion and Future Work

  • GIMO combines ego-centric images, 3D gaze, 3D scene context, and ground-truth human motions.The dataset is presented for studying gaze-informed human motion prediction.Source: §6 · p. 15
  • The proposed framework uses eye gaze to infer potential intent and reports promising results for long-term future motion prediction.The paper describes the task as using gaze to minimize ambiguities in motion prediction.Source: §6 · p. 15
  • The current method relies only on 3D inputs; incorporating ego-centric image information is proposed as future work to further boost accuracy.The dataset is also identified as potentially useful for intention-aware motion synthesis and gaze-guided ego-centric pose estimation.Source: §6 · p. 15

A.1 Implementation Details

The implementation combines scene, motion, and gaze features through cross-modal transformers to predict future motion parameters. Baselines use comparable network capacity, while the predicted body representation omits hand poses and shape.

A.1 Implementation Details

  • PointNet++ extracts global and per-point scene features, while cross-modal transformer stages exchange information between scene-aware motion and gaze features.The updated motion, gaze, and global scene features are concatenated into a 768D embedding for predicting future motion.Source: §A.1 · p. 19
  • The 32D motion parameter comprises global translation, axis-angle orientation, and a VPoser pose embedding; hand poses and body shape are omitted.The authors state that these parameters represent global body pose and leave hand poses and shape for future work.Source: §A.1 · p. 19
  • The reimplemented baselines include spatio-temporal transformer, RNN, and MultimodalNet models, with similar network capacity to the proposed method.The spatio-temporal transformer and RNN use 3D joint angles, while MultimodalNet uses the same 32D motion parameter as the proposed method.Source: §A.1 · pp. 19–20

A.2 Training Loss

Training uses an L1 loss over predicted motion parameters. The loss combines translation, orientation, and pose-embedding terms, with equal weights.

A.2 Training Loss

  • The training objective uses L1 loss between predicted motion parameters and ground truth, comprising translation, orientation, and pose-embedding losses.The supplied passages name the loss components but do not provide their displayed formulas.Source: §A.2 · pp. 20, 22
  • For translation loss, T denotes output-pose length, and each predicted global translation is compared with its ground-truth translation.The passage identifies the predicted translation for the k-th pose as btk and the ground truth as tk.Source: §A.2 · p. 21
  • The translation, orientation, and pose-embedding loss weights are each set to 1 during training.The objective also includes orientation and pose-embedding terms.Source: §A.2 · p. 22

B GIMO Dataset

GIMO comprises 217 motion trajectories collected across 19 scenes by 11 subjects, with additional qualitative results and a depicted noisy-gaze failure case.

B GIMO Dataset

  • The dataset contains 217 motion trajectories from 11 subjects across 19 scenes.The scenes include varied daily indoor environments such as living rooms, meeting rooms, libraries, and labs.Source: B GIMO Dataset · p. 22
  • Fig. 10 presents additional qualitative results for the baseline methods.Source: Figure 10 · p. 23
  • Fig. 11 depicts a failure case in which noisy gazes confuse the method's interpretation of the subject's intention.Source: Figure 11 · p. 23

C.1 Human Evaluation

The human evaluation tests whether gaze helps people infer a motion's final pose from ego-centric video. Participants were more accurate when gaze was shown.

C.1 Human Evaluation

  • Participants watched ego-centric video with or without gaze and inferred the trajectory's final pose.They selected a pose from a training-set pose database and placed it in the 3D scene as the motion's final position.Source: §C.1 · p. 24
  • Humans extracted intention clues from gaze and solved the prediction task accurately, while observers without gaze could be confused.Source: §C.1 · p. 24
  • The evaluation simplified prediction by asking participants to infer only the final pose rather than the full motion trajectory.Source: §C.1 · pp. 23–24

C.2 More Results of Baselines and Failure cases

The additional results support the method's superiority over baselines, but noisy gaze can impair intention interpretation when it dominates the sparse input.

C.2 More Results of Baselines and Failure cases

  • Additional baseline results further demonstrate the method's superiority in predicting future motion from gaze, motion, and scene information.Source: §C.2 · p. 24
  • When noisy, uninformative gaze accounts for a large portion of the 2-fps input, the method can misinterpret the subject's goal and predict inaccurately.Source: §C.2 · p. 24
  • Higher-frame-rate inputs might mitigate this issue, as recorded sequences show attention concentrated on objects related to the motion destination.Source: §C.2 · p. 24

Related papers

Loading 2204.09443v2…