Source-linked AI summary
Seeing Invisible Poses: Estimating 3D Body Pose from Egocentric Video
Hao Jiang, Kristen Grauman
TL;DR
Egocentric video often cannot show the wearer’s body, while prior visible-part approaches provide only a partial pose estimate. The paper infers full-body pose from scene cues and longer-term pose dynamics, and reports better results than several alternatives, including CNN regression.
Problem
The wearer’s body is often invisible in egocentric video, and visible-part pose methods are restricted to arms and hands despite frequent occlusion of limbs.
Source: §1 · pp. 1–2Method
The method combines learned dynamic and static scene cues with soft per-frame pose predictions and a non-parametric pose-dynamics prior to infer a longer pose sequence.
Source: §1 · p. 3Results
The method greatly outperforms several alternatives, including a CNN regression approach, in evaluation on videos from ten camera wearers.
Source: §1 · p. 3Takeaways & Limitations
The authors say the approach could be useful for egocentric video logging, summarization, and information retrieval, and could facilitate action and movement understanding.
Source: §5 · p. 16Some poses are aliased in egocentric video, and failures also include unpredictable arm poses and sitting-versus-standing misclassification.
Source: §1 · p. 2, §4 · p. 15
Abstract
from arXiv · showhide
Understanding the camera wearer's activity is central to egocentric vision, yet one key facet of that activity is inherently invisible to the camera--the wearer's body pose. Prior work focuses on estimating the pose of hands and arms when they come into view, but this 1) gives an incomplete view of the full body posture, and 2) prevents any pose estimate at all in many frames, since the hands are only visible in a fraction of daily life activities. We propose to infer the "invisible pose" of a person behind the egocentric camera. Given a single video, our efficient learning-based approach returns the full body 3D joint positions for each frame. Our method exploits cues from the dynamic motion signatures of the surrounding scene--which changes predictably as a function of body pose--as well as static scene structures that reveal the viewpoint (e.g., sitting vs. standing). We further introduce a novel energy minimization scheme to infer the pose sequence. It uses soft predictions of the poses per time instant together with a non-parametric model of human pose dynamics over longer windows. Our method outperforms an array of possible alternatives, including deep learning approaches for direct pose regression from images.
1 Introduction
The paper addresses full-body 3D pose estimation from egocentric video, where the wearer is usually out of view and visible-part methods cover only arms and hands. It combines dynamic scene motion, static scene context, and longer-term pose dynamics to infer the wearer’s pose sequence.
1 Introduction
- Visible-part approaches estimate arms and hands, while arms and legs are visible in less than 10% of the cited daily-activity footage.A multi-camera structure-from-motion alternative requires obtrusive equipment and hours to days of processing per minute of video.Source: §1 · pp. 1–2
- The approach uses scene-independent motion signatures and static scene structure, such as a table’s distance and angle, as cues to pose.The authors note that some poses are aliased because different poses can produce the same visual signal.Source: §1 · p. 2
- A learned approach predicts probabilities over quantized poses per frame, then jointly infers poses over 1–3 minutes using a non-parametric pose-dynamics model and dynamic programming.The resulting method runs at about 0.5 seconds per frame.Source: §1 · p. 3
- The method greatly outperforms alternatives, including a CNN regression method adapted from DeepPose, in quantitative tests on ten camera wearers.The authors also report qualitative tests on challenging unconstrained videos.Source: §1 · p. 3
- The paper’s contributions are the new invisible-pose task, global optimization using learned dynamic and scene classifiers with long-span pose coupling, and benchmarks of handcrafted and CNN features.Source: §1 · p. 3
2 Related work
The paper distinguishes its full-body, out-of-view pose task from third-person pose estimation, visible-part egocentric pose methods, and egocentric activity recognition. It positions its approach as learning pose from scene cues and temporal constraints rather than directly regressing from images or localizing body-mounted cameras.
2 Related work
- Third-person pose methods assume the person is visible, unlike this task of estimating the wearer’s pose behind the camera.The paper says existing third-person methods are not applicable to its scenario.Source: §2 · p. 3
- Prior egocentric pose work estimates visible hands, arms, shoulders, or hands, whereas this paper targets full-body 3D pose even when the wearer is entirely out of view.The paper describes its pose representation as 25 joint positions.Source: §2 · p. 4
- Unlike multi-camera structure-from-motion capture, the proposed learning approach uses one chest-mounted camera and takes about 15 minutes rather than 1.5 days per minute of capture.Its trade-off is the need for representative training data, which the authors say is relatively easy to collect without manual annotations.Source: §2 · p. 4
- Egocentric activity methods recognize action classes, while this method produces detailed pose across an activity and could provide a mid-level representation for later activity recognition.The paper says direct image regression is inadequate because training data may not capture environmental and movement variation, and the signals are temporal.Source: §2 · pp. 3–4
3 Method
The method estimates 3D joint positions in the wearer’s local frame from chest-camera video. It combines clustered-pose probabilities from video segments with motion features and longer sequence reasoning.
3 Method
- The method estimates each body joint’s 3D position in the wearer’s local frame from chest-mounted-camera video.Source: §3 · p. 4
3.1 Pose parameterization and data collection
- Ground-truth poses comprise 25 Kinect V2 joint positions in a wearer-oriented coordinate frame, normalized by five times shoulder length.The dataset has 18 synchronized videos from ten subjects; three are for training and the rest for testing, with indoor capture and outdoor tests also demonstrated.Source: §3.1 · p. 5
3.2 Instantaneous pose estimation
- For instantaneous pose estimation, k-means clusters normalized training poses into K = 300 groups, and a classifier estimates the probability that a video segment matches each cluster.The pose probability is approximated by the probability of its cluster identity.Source: §3.2 · p. 5
Dynamic clues
- Dynamic clues use successive-frame homographies, which approximate scene-invariant camera rotation when translation is small, to represent camera motion over one second.The method extracts homographies using optical-flow correspondences and trains a random forest to estimate pose-cluster probabilities; static scene cues are added when video motion is limited. Fig. 2 pairs example poses with their one-second dynamic features.Source: Dynamic clues · pp. 5–6, Figure 2 · p. 6
Static scene structure clues
Static scene structure provides a cue to broad posture categories, especially sitting-like versus standing-like pose. The method classifies egocentric images into those two categories.
Static scene structure clues
- The static-scene classifier predicts whether an egocentric image corresponds to a sitting-like or standing-like pose.The training set contains 5,530 standing images and 2,946 sitting images; Fig. 3 shows samples from each class.Source: Static scene structure clues · p. 6, Figure 3 · p. 7
- The method focuses on standing-like and sitting-like poses; lying down is excluded because it is uncommon during the day.The paper notes that roughly 95% of frames in a cited dataset can be classified as standing-like or sitting-like.Source: Static scene structure clues · p. 6
- On 71,623 ground-truth frames, sitting-like and standing-like classification accuracy is 65.09% and 77.97%, respectively.The dataset contains 79.71% standing-like images.Source: Static scene structure clues · p. 7
Local cost of pose estimation
Dynamic and static classifier outputs provide local pose costs, but framewise estimates can be noisy and quantized. The method therefore optimizes across a longer sequence using a non-parametric pose-dynamics prior.
Local cost of pose estimation
- Dynamic and static classification outputs serve as unary costs for pose choices across the longer sequence.The energy also penalizes disagreement between the classifiers.Source: Local cost of pose estimation · p. 7
- Framewise costs can produce noisy predictions and staircase-like pose changes, motivating joint optimization over a long span.The global stage infers the pose path for the sequence.Source: Local cost of pose estimation · pp. 7–8
3.3 Non-parametric prior on pose dynamics
- The final sequence is optimized from the local costs together with a non-parametric prior on pose dynamics.The section introduces the prior before describing an efficient optimization approach.Source: §3.3 · p. 8
Pose paths in an implicit motion graph
The method represents a pose sequence as a path through poses from temporally ordered exemplar videos. Its implicit motion graph and energy terms constrain which transitions are favored while allowing the path to account for different movement speeds.
Pose paths in an implicit motion graph
- The implicit motion graph uses exemplar poses as nodes and possible pose transitions as edges; cluster adjacency and step regularizers constrain the path.Transition costs depend on traversal history, and the constraints still permit some longer jumps.Source: Pose paths in an implicit motion graph · p. 8
- Unlike motion synthesis, which generates movement from anchor poses, this graph jointly infers poses for a novel egocentric video and dynamically adjusts edge weights.The edge weights support the path regularizers.Source: Pose paths in an implicit motion graph · p. 8
- A pose path selects a sequence from the temporally ordered training poses, with one pose assignment at each time.The assignment is represented by binary variables, with one selected pose per time instant.Source: Pose paths in an implicit motion graph · p. 8
- The path objective combines a unary pose cost with terms controlling step size, transition speed, and stationary steps.The energy is minimized over pose sequences drawn from the exemplar set.Source: Pose paths in an implicit motion graph · pp. 8–9
- Step size links exemplar timing to test-video speed: zero holds a pose, one preserves exemplar speed, and larger steps move faster.The energy favors small steps while allowing occasional large jumps.Source: The step size term T · p. 9
The step size term T
The path regularizers favor small exemplar steps and smooth speed over short intervals, while allowing changes when needed.
The step size term T
- The step-size term penalizes backward moves and steps larger than two, and prohibits transitions between non-neighboring pose clusters.Longer jumps within the same or a directly neighboring cluster remain possible but incur a penalty.Source: The step size term T · p. 9
The speed smoothness of the path V
- Because human motion is usually uniform over one or two seconds, the speed term penalizes short-window changes in path speed.Its truncated-linear function encourages the path to maintain a constant speed.Source: The speed smoothness of the path V · p. 9
The stationary step penalty S in the path
The stationary-step penalty discourages extended pose holds that can arise when local costs and smoothness terms favor remaining still.
The stationary step penalty S in the path
- Flat local costs can make a pose path stationary even when step-size and speed-smoothness terms are minimized.The step-size penalty alone does not fully address this behavior.Source: The stationary step penalty S in the… · p. 9
- The penalty counts stationary steps and makes prolonged pose holds less likely, helping address temporal-resolution loss.It penalizes the pose path when the pose stops changing for a long time.Source: The stationary step penalty S in the… · p. 10
Optimizing the pose path using DP
The method finds a minimum-cost pose path through a trellis using dynamic programming. It restricts transitions to the same or neighboring pose clusters and can prune unlikely poses to speed computation.
Optimizing the pose path using DP
- Dynamic programming selects the minimum-cost pose path by updating each node’s energy and state from eligible predecessors.The state tracks stationary step count, step speed, and the previous optimal pose; the recursion is equivalent to minimizing the combined pose-path energy.Source: Optimizing the pose path using DP · p. 10
- Each trellis transition is limited to the same or a neighboring pose cluster, reducing the set of eligible paths.Fig. 4 illustrates the cluster-based edge connections between trellis layers.Source: Figure 4 · p. 10, Optimizing the pose path using DP · p. 10
- Keeping only poses with classifier probability above 0.01 makes the trellis sparse and typically adds 0.01 seconds per frame for the whole system.The method also reports complexity much lower than O(M^2N) because each stage connects only same- or neighboring-cluster nodes.Source: Optimizing the pose path using DP · pp. 10–11
4 Experimentation
The experiments evaluate 3D pose predictions across same-person and cross-person settings, using quantitative ground-truth tests and qualitative unconstrained videos. The proposed method outperforms competing methods overall, while comparisons also expose the effects of temporal coupling and learned features.
4 Experimentation
- Ground-truth evaluation spans 71,623 test frames from ten subjects, with same-subject and cross-subject settings plus 15 minutes of unconstrained video.The error analysis focuses on head, elbows, wrists, knees, and ankles, with raw errors converted to centimeters using a 30 cm shoulder-length reference.Source: §4 · pp. 11, 13, 14
- Qualitatively, the proposed method outperforms CNN-Regression and KdTree, which do not model long-term pose coupling.CNN-Regression adapts DeepPose, while KdTree predicts poses from nearby training segments.Source: §4 · pp. 12–13, Figure 6 · p. 12
- Path-Cluster produces lower-temporal-resolution, less-smooth predictions, and its cluster-restricted refinement cannot undo first-stage errors.Linear interpolation is not directly usable because it does not always produce valid poses.Source: Figure 7 · p. 12, §4 · p. 13
- Deeply learned dynamic and scene features do not improve results through either training from scratch or fine-tuning, while the proposed method performs well with a small dataset.The authors note that neural approaches need a large dataset to capture scene and pose variation.Source: §4 · p. 13, Figure 8 · p. 13
- The proposed method yields smaller errors than all competing methods in both benchmark settings.Tables 1 and 2 cover same-subject/different-environment and different-subject/different-environment tests.Source: Table 1 · p. 14, Table 2 · p. 14, §4 · p. 14
- The method improves on AlwaysStanding, including by 6 cm on average for ankles and knees, and all methods have lower absolute error when training includes the test subject.AlwaysSitting has much larger errors than any method.Source: §4 · pp. 14–15
- Failures arise when arm pose does not affect camera motion or viewing angle, and from standing-versus-sitting misclassification.The authors suggest that improving features and instantaneous pose-estimation accuracy may improve results.Source: Figure 9 · p. 14, §4 · p. 15
- On eight unconstrained sequences, the method works well in varied environments, including outdoors despite indoor-only training.The test videos cover three subjects; side-camera views are for display only and are not method inputs.Source: §4 · pp. 15–16, Figure 10 · p. 15
5 Conclusion
The paper introduces egocentric human-pose prediction and a global optimization method for pose sequences. It reports accurate predictions in same-person and cross-person tests, superior results to alternative approaches, and potential applications in video logging and movement understanding.
5 Conclusion
- The paper tackles the task of predicting human poses from egocentric video.The authors frame this as a new computer-vision problem.Source: §5 · p. 16
- Global optimization gives accurate pose predictions in same-person and cross-person tests and outperforms alternative approaches.Source: §5 · p. 16
- The authors identify video logging, summarization, information retrieval, and action and movement understanding as potential uses.Source: §5 · p. 16