PaperlayerView exact revision on arXiv

Source-linked AI summary

Recurrent 3D Pose Sequence Machines

Mude Lin, Liang Lin, Xiaodan Liang, Keze Wang, Hui Cheng

arXiv:1707.09695v1cs.CV

TL;DR

Monocular 3D pose recovery is challenging because images vary in appearance and viewpoint, visibility is obstructed, and 3D pose is geometrically ambiguous. RPSM learns spatial and temporal dependencies through multi-stage recurrent pose refinement, and evaluations report that it outperforms existing methods on Human3.6M and HumanEva-I.

  • Problem

    Monocular imagery makes 3D pose recovery difficult because of appearance, viewpoint, and visibility variation, as well as geometric ambiguity.

    Source: §1 · p. 1

  • Method

    RPSM uses multi-stage refinement to integrate spatial and temporal dependencies, with 2D pose features adapted for recurrent 3D pose prediction using prior-stage information.

    Source: §1 · p. 2

  • Results

    RPSM outperforms existing 3D human pose estimation methods on Human3.6M and HumanEva-I; on Human3.6M, mean error is reduced by 31.85% versus [40] and 21.52% versus.

    Source: §1 · p. 2, §5.2 · p. 6

  • Takeaways & Limitations

    Evaluations on two public 3D human pose datasets support the effectiveness and superior performance of the RPSM framework.

    Source: §6 · p. 8

Abstract

from arXiv · show

3D human articulated pose recovery from monocular image sequences is very challenging due to the diverse appearances, viewpoints, occlusions, and also the human 3D pose is inherently ambiguous from the monocular imagery. It is thus critical to exploit rich spatial and temporal long-range dependencies among body joints for accurate 3D pose sequence prediction. Existing approaches usually manually design some elaborate prior terms and human body kinematic constraints for capturing structures, which are often insufficient to exploit all intrinsic structures and not scalable for all scenarios. In contrast, this paper presents a Recurrent 3D Pose Sequence Machine(RPSM) to automatically learn the image-dependent structural constraint and sequence-dependent temporal context by using a multi-stage sequential refinement. At each stage, our RPSM is composed of three modules to predict the 3D pose sequences based on the previously learned 2D pose representations and 3D poses: (i) a 2D pose module extracting the image-dependent pose representations, (ii) a 3D pose recurrent module regressing 3D poses and (iii) a feature adaption module serving as a bridge between module (i) and (ii) to enable the representation transformation from 2D to 3D domain. These three modules are then assembled into a sequential prediction framework to refine the predicted poses with multiple recurrent stages. Extensive evaluations on the Human3.6M dataset and HumanEva-I dataset show that our RPSM outperforms all state-of-the-art approaches for 3D pose estimation.

1. Introduction

Monocular 3D pose recovery is difficult because of visual variation, occlusion, and geometric ambiguity, while existing approaches often rely on elaborate hand-designed priors. RPSM instead learns spatial and temporal dependencies through multi-stage refinement, and evaluations report improvements over existing methods on two datasets.

1. Introduction

  • Monocular 3D pose recovery is challenging because appearance, viewpoint, and visibility vary, while 3D pose is geometrically ambiguous from images.The introduction identifies 2D spatial relationships, 3D geometry, and temporal consistency as relevant constraints to model together.Source: §1 · p. 1
  • Prior methods use external pose galleries or elaborately designed kinematic constraints, making their models sophisticated and component effectiveness difficult to validate.RPSM is presented as a fully differentiable, data-driven alternative that integrates 2D spatial relationships, 3D geometry, and temporal smoothness.Source: §1 · p. 2
  • RPSM uses multi-stage training to capture long-range dependencies among body parts for 3D pose prediction and enforce temporal consistency across sequential frames, recursively refining pose sequences using prior-stage information.At each stage, a 2D pose module updates 2D representations, a feature adaption module transforms them to 3D, and an LSTM-based 3D pose recurrent module regresses poses using transformed representations, prior-stage joint predictions, and past-frame states.Source: §1 · p. 2
  • Evaluations on Human3.6M and HumanEva-I report that RPSM outperforms existing 3D human pose estimation methods by large margins.The paper presents this as an outcome of extensive evaluations on both public datasets.Source: §1 · p. 2

2. Related work

Earlier video-based approaches include pose tracking with Markov dependencies and framewise candidate detection followed by temporal post-processing. Deep-learning methods add image-based regression, motion cues, or explicit structure and priors; RPSM is presented as a unified recurrent refinement framework.

2. Related work

  • Video-based 3D pose methods include frame-to-frame tracking with Markov dependencies and framewise candidate detection followed by temporal-consistency post-processing.Tracking approaches can require an initialization pose and may be unable to recover from tracking failure.Source: §2 · p. 2
  • Deep-learning approaches include CNN regression from monocular images and methods combining image features with pose structure or motion information.Examples include joint-embedding subnetworks and networks that use consecutive-frame motion information.Source: §2 · pp. 2–3
  • RPSM differs by recurrently refining poses in multiple stages, with 2D pose features adapted to 3D and a recurrent module incorporating sequence dependence.Its three modules are combined in a unified end-to-end framework.Source: Figure 2 · p. 3

3. Recurrent 3D Pose Sequence Machines

RPSM predicts a sequence of 3D joint locations from monocular frames, recurrently refining its estimates through multiple stages. Each stage extracts 2D pose features, adapts them for 3D prediction, and uses recurrent temporal context together with prior pose estimates.

3. Recurrent 3D Pose Sequence Machines

  • RPSM combines 2D pose features, feature adaptation, and recurrent 3D pose prediction in a unified, end-to-end multi-stage framework.The image sequence passes through the stages to gradually refine predicted poses.Source: §3 · p. 3

3.1. Multi-stage Optimization

  • The final sequence is the set of 3D poses predicted by the last stage, after sequentially updating the mapping from image sequences to pose sequences.The target is a sequence of poses, each represented by P joint locations relative to a root joint.Source: §3.1 · pp. 3–4

3.2. 2D Pose Module

  • The 2D pose module encodes each monocular frame as a compact pose representation, combining shared convolutional features with previous-stage pose-aware features.Its final two convolution layers are specialized independently at each stage to support refinement.Source: §3.2 · p. 4

3.3. Feature Adaption Module

  • The feature adaptation module transforms 2D pose-aware features into a 1024-dimensional adapted feature vector for 3D prediction.It uses two convolutional layers followed by a fully connected layer.Source: §3.3 · pp. 4–5

3.4. 3D Pose Recurrent Module

  • The 3D pose recurrent module uses an LSTM to model temporal dependencies and predict poses from adapted features, prior hidden states, and previous 3D poses.Its output layer predicts 17 joint locations; incorporating previous-stage poses supports gradual refinement.Source: §3.4 · p. 5

4. Model Training and Testing

Training supervises every frame at every refinement stage with a weighted Euclidean pose loss, while the 2D module is pretrained on MPII before joint fine-tuning. At test time, frames pass through the stages sequentially and the final refinement supplies the prediction.

4. Model Training and Testing

  • The training loss measures Euclidean distances between predicted and ground-truth locations for all P joints, across every frame and stage.Each stage’s loss is weighted by α_k; K denotes the number of stages and T the sequence length.Source: §4 · p. 5
  • The 2D pose module is pretrained on MPII Human Pose data using joint-confidence heat maps before whole-framework optimization.The framework uses ADAM for parameter optimization.Source: §4 · p. 5
  • Training decomposes long monocular sequences into equal clips of C frames and fine-tunes the recurrent 3D module, feature adaption module, and stage-specific 2D layers.The feature adaption module learns representations for subsequent 3D pose estimation.Source: §4 · p. 5
  • At test time, each input frame is processed stage by stage, and the final refinement produces the 3D pose prediction.Source: §4 · p. 5

5. Experiments

RPSM outperforms compared methods on Human3.6M and HumanEva-I. Component analyses report gains from staged refinement and temporal context, alongside tradeoffs involving pre-training and layer sharing.

5.2. Comparisons with state-of-the-art methods

  • RPSM outperforms compared methods on Human3.6M, reducing mean error by 31.85% vs [40] and 21.52% vs, and achieves state-of-the-art results on all three HumanEva-I actions.Table 2 covers Human3.6M actions for subjects 9 and 11; Table 3 covers Walking, Jogging, and Boxing.Source: Table 2 · p. 6, Table 3 · p. 6, §5.2 · pp. 6–7

5.1. Experimental Settings

  • Experiments evaluate 3D pose error on Human3.6M and HumanEva-I, using the prior-work partition protocol for Human3.6M and the validation sequence for HumanEva-I.Human3.6M training uses five subjects and tests on two; HumanEva-I trains on subjects 1–3. Three refinement stages are used in the experiments.Source: §5.1 · pp. 5–6

5.2. Comparisons with state-of-the-art methods

  • RPSM processes an image in 50 ms, compared with around three minutes for and more than 25 seconds for [38].The cited comparisons are reported for HumanEva-I.Source: §5.2 · p. 7

5.3. Component Analysis

  • Without temporal information, RPSM-3stage seq 1 has 81.12 versus 90.83 3D pose error for RPSM-1stage seq 1; performance improves monotonically through three stages.Figure 5 shows joint predictions progressively corrected across stages.Source: §5.3 · pp. 7–8, Table 4 · p. 7, Figure 5 · p. 8
  • RPSM-3-stage no MPII performs worse than RPSM-3-stage; full 2D-module layer sharing is slightly better than partial sharing but significantly increases training time.The authors choose partial sharing.Source: §5.3 · p. 8
  • On Human3.6M, RPSM-1stage seq 5 and seq 10 reduce mean joint errors by about 10% versus seq 1, while the difference between lengths 5 and 10 is minor.The authors suggest temporal inconsistency may be more likely in longer clips.Source: §5.3 · p. 8

6. Conclusion

The conclusion presents RPSM as a unified architecture for monocular 3D pose sequences, combining three modules and multiple refinement stages. Evaluations on two public datasets validate its effectiveness and superior performance.

6. Conclusion

  • RPSM combines 2D pose, feature adaption, and 3D pose recurrent modules to integrate spatiotemporal long-range dependencies.The conclusion describes this integration as implicit and comprehensive.Source: §6 · p. 8
  • Multiple sequential stages refine pose estimates using 3D pose geometry information.Source: §6 · p. 8
  • Extensive evaluations on two public 3D human pose datasets validate the effectiveness and superior performance of Recurrent 3D Pose Sequence Machines (RPSM) for estimating 3D human pose from a sequence of monocular images.Source: §6 · p. 8
  • Future work will extend RPSM to human-centric sequence tasks such as action and activity recognition.Source: §6 · p. 8

Related papers

Loading 1707.09695v1…