PaperlayerView exact revision on arXiv

Source-linked AI summary

You2Me: Inferring Body Pose in Egocentric Video via First and Second Person Interactions

Evonne Ng, Donglai Xiang, Hanbyul Joo, Kristen Grauman

arXiv:1904.09882v2cs.CV

TL;DR

Wearable cameras often cannot see the wearer’s body, limiting direct estimation of their 3D pose from egocentric video. You2Me models interaction dynamics using the visible partner’s pose alongside camera motion and scene appearance. It improves estimates over camera-motion and scene-context cues alone and outperforms prior methods in the evaluated settings.

  • Problem

    The wearer’s body is often largely out of view in egocentric video, making full-body pose estimation difficult despite its relevance to wearable-camera applications.

    Source: §1 · p. 1

  • Method

    You2Me uses a recurrent model of the visible interactee’s pose, camera motion, and scene appearance to infer the wearer’s 3D pose sequence.

    Source: §1 · p. 1, §3.1 · p. 3, §3.5 · p. 4

  • Results

    You2Me significantly boosts estimates over camera motion and scene context alone and outperforms state-of-the-art ego-pose and adapted deep third-person pose methods across several interaction domains.

    Source: §1 · p. 2

  • Takeaways & Limitations

    Results across two capture scenarios and several activity domains demonstrate the promise of leveraging links between observed partner poses and the camera wearer’s pose.

    Source: §6 · p. 8

    Future work includes handling periods without second-person poses and sequences with multiple second people.

    Source: §6 · p. 8

Abstract

from arXiv · show

The body pose of a person wearing a camera is of great interest for applications in augmented reality, healthcare, and robotics, yet much of the person's body is out of view for a typical wearable camera. We propose a learning-based approach to estimate the camera wearer's 3D body pose from egocentric video sequences. Our key insight is to leverage interactions with another person---whose body pose we can directly observe---as a signal inherently linked to the body pose of the first-person subject. We show that since interactions between individuals often induce a well-ordered series of back-and-forth responses, it is possible to learn a temporal model of the interlinked poses even though one party is largely out of view. We demonstrate our idea on a variety of domains with dyadic interaction and show the substantial impact on egocentric body pose estimation, which improves the state of the art. Video results are available at http://vision.cs.utexas.edu/projects/you2me/

1. Introduction

Egocentric cameras support applications in AR, healthcare, and robotics, but often leave the wearer’s body largely out of view. You2Me uses interaction dynamics and visible partner poses to estimate the wearer’s pose, improving on existing approaches.

1. Introduction

  • Wearer body pose is relevant to wearable-camera applications in augmented reality, healthcare, and robotics.These include accessing virtual information, remote rehabilitation, and video-based learning from demonstration.Source: §1 · p. 1
  • Prior ego-pose methods restrict the task to static environments, relying on camera motion or coarse scene layout while the wearer’s body is often out of view.The hidden body also limits the suitability of third-person pose methods.Source: §1 · p. 1
  • You2Me uses action-reaction dynamics between interacting people, whose poses are linked by back-and-forth responses such as handshakes or conversational gestures.The visible partner’s pose can provide a cue to the wearer’s otherwise hidden pose.Source: §1 · pp. 1–2
  • The model uses an RNN to combine the observed second-person pose with camera motion and scene appearance to infer the wearer’s latent pose across a sequence.“Second person” refers to the person the wearer is interacting with.Source: §1 · pp. 1–2
  • You2Me outperforms state-of-the-art ego-pose estimation and an adapted deep third-person pose method across data from 10 subjects and several interaction domains.The study uses ground truth from Kinect sensors and Panoptic Studio.Source: §1 · p. 2

2. Related work

Related work studies visible-body pose, human interactions, and egocentric video, but You2Me targets pose estimation for the camera wearer, whose body is out of view. It differs from prior approaches by using the visible interactee’s pose from a single ego-video.

2. Related work

  • Existing methods model human-human interactions assuming all people are fully within camera view, while this approach addresses interactions between an in-view person and the out-of-view camera wearer.Related work uses interactions to predict pedestrian trajectories, analyze social behavior and group activities, and understand human-object interactions.Source: §2 · p. 2
  • The geometry-based inside-out mocap method reconstructs 16 joint-mounted cameras using structure from motion, while You2Me needs only one chest-mounted camera.Egocentric full-body 3D pose estimation had received limited attention.Source: §2 · p. 2
  • Prior ego-pose approaches include MotionGraph and a simulator-based control method, while You2Me learns from full video frames and interaction dynamics.MotionGraph uses hand-crafted costs; the simulator-based method is evaluated quantitatively only on synthetic sequences, and both prior learning-based methods focus on sweeping motions.Source: §2 · p. 2
  • Paired-egocentric-video work classifies dyadic actions using two videos at test time, while You2Me estimates body pose from one ego-video.Other egocentric vision work targets objects, activities, visible hands and arms, gaze, or future camera trajectories.Source: §2 · pp. 2–3

3. Our approach

You2Me estimates the camera wearer’s 3D pose sequence from egocentric video by combining scene appearance, camera motion, and the visible interactee’s pose. An LSTM models the temporal links between these cues and the wearer’s poses.

3.1. Problem formulation

  • You2Me predicts the camera wearer’s 3D joint pose for each frame from a monocular egocentric video sequence.The output is articulated pose, not action recognition.Source: §3.1 · p. 3, §3 · p. 3

3.2. Dynamic first-person motion features

  • The model represents motion with successive-frame homographies, using camera movement as a scene-independent cue to wearer articulation.A stack covers the preceding half-second for 30 fps video.Source: §3.2 · p. 3

3.3. Static first-person scene features

  • ResNet-152 scene features provide appearance cues when motion is ambiguous, while static objects can indicate coarse posture and a nearby interactee can reveal finer details.The full scene encoding contributes to more accurate pose learning than a standing-versus-sitting classifier.Source: §3.3 · p. 3, Figure 3 · p. 4

3.4. Second-person body pose interaction features

  • The interactee’s inferred 2D pose is the model’s most important input for capturing interaction-driven gestures and micro-actions.OpenPose estimates the visible partner’s pose; 2D keypoints are used because available 3D methods fail on the data’s extensive hand occlusions.Source: §3.4 · p. 4

3.5. Recurrent neural network for pose inference

  • An LSTM combines the video cues with an embedding of the wearer’s previous pose to predict fine-grained quantized poses over the sequence.K-means uses separate cluster granularities for lower-body and upper-body poses.Source: §3.5 · pp. 4–5, Figure 4 · p. 5

4. You2Me Video Datasets

The authors introduce a dataset of dyadic interactions recorded from a chest-mounted camera, with synchronized pose ground truth for both participants. It combines constrained studio capture with more mobile indoor capture.

Dataset overview

  • The dataset contains 42 two-minute sequences from 10 individuals interacting in hand games, tossing and catching, sports, and conversation.About half of the frames show no first-person body parts.Source: §4 · pp. 5–6

Panoptic Studio capture

  • The Panoptic Studio subset comprises 14 sequences with 19-joint ground truth and highly accurate poses, but has more constrained backgrounds.Six participants enacted all four activity classes.Source: §4 · p. 6

Kinect capture

  • The Kinect subset comprises 28 indoor sequences with 25-joint ground truth in varied locations, but its pose labels are noisier than Panoptic Studio’s.Four participants enacted the activity classes; Kinect mobility enables the less constrained capture setting.Source: §4 · p. 6

Dataset motivation

  • At test time, the method uses only egocentric video; the authors state that existing datasets lack the required combination of viewpoint, interaction, and body-pose labels.The dataset is publicly available.Source: §4 · p. 6

5. Experiments

You2Me is evaluated on Kinect and Panoptic Studio captures across interaction domains, with tests comparing it to competing methods and examining the effects of input features and pose sources. The results show consistent gains, especially for upper-body predictions, while extended loss of the interactee’s pose is a key failure condition.

5. Experiments

  • Evaluation covers conversation, sports, hand games, and ball tossing in activity-agnostic splits across Kinect and Panoptic Studio captures.Test clips contain people absent from training; Kinect test sequences were recorded at unseen locations.Source: §5 · p. 6
  • The proposed method consistently outperforms competing methods, with larger improvements for upper-body joints.The comparison reports average joint error in centimeters across both dataset captures.Source: Table 1 · p. 7, §5 · p. 7
  • The method has its greatest impact on conversation and its lowest on sports, where the interactee often moves out of view for long periods.The paper associates conversation’s lesser global motion with the value of second-person pose for upper-body predictions.Source: §5 · p. 7
  • Ablations show that models using the second-person pose outperform those without it, with the largest benefit for upper-body predictions.Scene features alone do not capture the information encoded in the explicit pose estimate.Source: Table 2 · p. 7, §5 · p. 8
  • The model makes more informed limb estimates than Sit and Stand, including 10 cm better average upper-body keypoints, and its classification objective beats regression.Stand beats other baselines but not You2Me; the paper also reports that You2Me outperforms DeconvNet.Source: §5 · p. 7
  • Linked gestures such as mutual reaches illustrate useful interaction cues, while failures occur when the camera points down or the interactee stays unseen for extended periods.The model can continue without interactee poses for a few frames, but prolonged absence is detrimental.Source: §5 · p. 7, Figure 5 · p. 7, Figure 6 · p. 8
  • More accurate ground-truth interactee poses can further improve results, while incorrect or unrelated pose inputs produce poorer results.2D OpenPose input outperforms predicted 3D poses, supporting its use as the second-person pose estimate.Source: Table 3 · p. 8, §5 · p. 8

6. Conclusions

You2Me estimates a chest-camera wearer’s pose by modeling the ties between the unobserved wearer and observed interaction partner. Results across two capture scenarios and multiple activity domains demonstrate promise and state-of-the-art ego-pose results; future work includes non-interaction periods and multiple partners.

6. Conclusions

  • You2Me predicts the camera wearer’s pose from a single chest-mounted camera by capturing ties between the unobserved wearer and observed partner poses.The approach was evaluated in two capture scenarios across several activity domains and obtained state-of-the-art ego-pose results.Source: §6 · p. 8
  • Future work includes handling absent second-person poses when interactions stop, sequences with multiple partners, and using ego-pose estimates to improve second-person pose estimates.Source: §6 · p. 8

A. Video example results

Video examples show You2Me estimating wearer actions across four interaction domains and both capture settings, while also revealing conditions where predictions fail. The supplementary preprocessing description scales OpenPose coordinates and zeroes missing joints.

A. Video example results

  • The method performs well across conversation, sports, hand games, and ball tossing in both Kinect and Panoptic Studio captures.Examples include predicting when the wearer will squat or sit, raise a hand to receive or catch an item, or gesture during conversation.Source: A. Video example results · p. 12
  • In hand games, You2Me detects clapping and distinguishes returning single-handed from double-handed claps, unlike MotionGraph and the OpenPose-feature ablation in the cited examples.Source: A. Video example results · p. 12
  • Predictions can fail when the partner is obscured for a long duration or interaction dynamics leave several wearer poses plausible, especially early in a sequence.The text highlights sports for extended obstruction and hand games with clasped hands as an ambiguous case.Source: A. Video example results · p. 12

B. Preprocessing of OpenPose 2D poses

C. Pose clusters

The pose-cluster representation uses fine-grained upper- and lower-body quantization intended to cover training poses and smaller or intermediate movements.

C. Pose clusters

  • The average joint distance to the nearest pose is 0.27 cm, indicating fine-grained pose quantization.Source: C. Pose clusters · p. 12
  • Using Kupp = 700 and Kbot = 100 clusters provides diversity intended to capture the poses in the training set.Source: C. Pose clusters · p. 12
  • The fine-grained clusters capture smaller arm and leg movements and intermediate poses in larger actions such as swinging, walking, and sitting.Source: C. Pose clusters · p. 12

D. Architecture details

The architecture predicts upper- and lower-body pose clusters with separate, identically structured LSTMs, then combines them into a full-body skeleton. Both networks share the same feature extraction and training setup.

D. Architecture details

  • Separate LSTMs predict upper- and lower-body pose cluster IDs, which are combined at each frame to construct the full-body skeleton.Source: D. Architecture details · p. 12
  • Two LSTMs were more effective than a single LSTM over unique 2D cluster IDs, which exceeded GPU memory, or reducing the pose set to frequent poses, which sacrificed precision.Source: D. Architecture details · p. 12
  • Each LSTM uses embedding dimension 256, hidden state dimension 512, batch size 32, and two recurrent layers, and trains for 20 epochs on 512-frame sequences.The learning rate is 0.001 for the first 10 epochs and 0.0001 for the remaining 10; preprocessed 30-fps video frames are resized to 224 x 224 x 3.Source: D. Architecture details · p. 12

E. Impact of different quantizations on results

Controlled upper/lower-body quantization improves performance, while results across quantization setups show that second-person pose information remains important. The supplemental autoencoder smooths LSTM pose trajectories for visualization; main-paper evaluation uses unsmoothed outputs.

E. Impact of different quantizations on results

F. Convolutional autoencoder details

  • For the supplemental video, a convolutional autoencoder converts quantized LSTM outputs into smoothed, non-quantized pose trajectories to reduce jitter.Main-paper evaluation uses the unsmoothed LSTM outputs.Source: F. Convolutional autoencoder details · p. 13

G. Per joint and per activity analysis

The per-joint and per-activity analyses report gains on interaction-relevant joints and find that second-person pose information helps across activities, including throw-and-catch. The activity breakdown reports joint errors in centimeters for Kinect and Panoptic captures.

Per-joint and per-activity analysis

  • The analyses cover errors against state-of-the-art baselines on two datasets, activity-type effects, influential second-person poses, successes and failures, and ablations.The paper reports separate errors for upper- and lower-body joints.Source: G. Per joint and per activity analysis · p. 13

Per-activity analysis

  • Second-person pose information helps even in non-symmetric activities such as throw-and-catch.Table 6 breaks down joint errors by activity for Ours and Ours w/o ot, with Kinect/Panoptic values reported separately.Source: G. Per joint and per activity analysis · p. 13

Per-joint analysis

  • Errors for MotionGraph [Jiang et al. CVPR 2017] are at least ∼50% higher than ours on hands and elbows on both datasets.The analysis reports significant gains for joints relevant to interactions.Source: G. Per joint and per activity analysis · p. 13

Related papers

Loading 1904.09882v2…