PaperlayerView exact revision on arXiv

Source-linked AI summary

Single-Stage Multi-Person Pose Machines

Xuecheng Nie, Jianfeng Zhang, Shuicheng Yan, Jiashi Feng

arXiv:1908.09220v1cs.CV

TL;DR

Multi-person pose estimation is inefficient in existing two-stage pipelines that separately process instances and joints. The paper introduces SPR and the single-stage SPM model, which predicts multiple structured poses efficiently and performs competitively across 2D and 3D benchmarks.

  • Problem

    Existing multi-person pose estimators generally use redundant two-stage pipelines, limiting efficiency despite high accuracy.

  • Method

    SPM uses SPR, encoding each person with a root joint and body-joint displacements, and predicts these structured poses in one CNN-based stage.

  • Results

    SPM achieves high efficiency across four benchmarks, including 46.1% mAP on extended PASCAL-Person-Part and 8× faster speed than [28] on MSCOCO with 0.669 AP.

  • Takeaways & Limitations

    The proposed approach provides a compact single-stage solution for multi-person 2D and 3D pose estimation with competitive or state-of-the-art performance across benchmarks.

  • Takeaways & Limitations

    The 3D formulation sets the camera position as the origin of the coordinate system.

Abstract

from arXiv · show

Multi-person pose estimation is a challenging problem. Existing methods are mostly two-stage based--one stage for proposal generation and the other for allocating poses to corresponding persons. However, such two-stage methods generally suffer low efficiency. In this work, we present the first single-stage model, Single-stage multi-person Pose Machine (SPM), to simplify the pipeline and lift the efficiency for multi-person pose estimation. To achieve this, we propose a novel Structured Pose Representation (SPR) that unifies person instance and body joint position representations. Based on SPR, we develop the SPM model that can directly predict structured poses for multiple persons in a single stage, and thus offer a more compact pipeline and attractive efficiency advantage over two-stage methods. In particular, SPR introduces the root joints to indicate different person instances and human body joint positions are encoded into their displacements w.r.t. the roots. To better predict long-range displacements for some joints, SPR is further extended to hierarchical representations. Based on SPR, SPM can efficiently perform multi-person poses estimation by simultaneously predicting root joints (location of instances) and body joint displacements via CNNs. Moreover, to demonstrate the generality of SPM, we also apply it to multi-person 3D pose estimation. Comprehensive experiments on benchmarks MPII, extended PASCAL-Person-Part, MSCOCO and CMU Panoptic clearly demonstrate the state-of-the-art efficiency of SPM for multi-person 2D/3D pose estimation, together with outstanding accuracy.

1. Introduction

Multi-person pose estimation is challenging because existing accurate methods use redundant two-stage pipelines. SPM addresses this with SPR, enabling single-stage prediction and efficient multi-person 2D/3D pose estimation.

  • Motivation: Existing top-down and bottom-up methods use two-stage processing, separating person localization or joint detection from pose allocation.This conventional representation separates person-instance and body-joint position information, creating computational redundancy.
  • Proposed approach: SPR unifies person-instance and body-joint positions by assigning each person a root joint and encoding body joints as displacements from it.A hierarchical extension factorizes long-range displacements into shorter articulated-joint displacements.
  • Proposed approach: SPM directly predicts multiple structured poses in one stage by simultaneously regressing root-joint positions and body-joint displacements.The model is implemented with CNNs for a compact pipeline.
  • Results: SPM achieves high efficiency across MPII, extended PASCAL-Person-Part, MSCOCO, and CMU Panoptic benchmarks.The paper reports new state-of-the-art results on MPII and extended PASCAL-Person-Part, competitive MSCOCO performance, and promising 3D results.

2. Background

The background frames multi-person pose estimation as locating and assigning joints across people. Existing top-down and bottom-up methods rely on separate processing stages, motivating a unified single-stage representation.

  • Task definition: The task estimates all human poses in an image by inferring body-joint coordinates for multiple person instances.It has applications in action recognition, person re-identification, and pedestrian tracking.
  • Limitation of prior methods: Both strategies use two-stage pipelines because conventional absolute-coordinate representations separate person-instance positioning from body-joint localization.The paper identifies this separation as a source of computational redundancy and low efficiency.
  • Top-down methods: Top-down methods first detect person instances, then estimate body joints separately for each detected person.Person instances are commonly represented by bounding boxes.
  • Bottom-up methods: Bottom-up methods first localize all body joints, then assign them to corresponding persons using affinities or graph partitioning.Joint candidates and assignment affinities support the allocation step.
  • Motivation for SPR: SPR instead unifies person-instance and body-joint positions to enable a compact, efficient single-stage solution.This representation is presented as distinct from previous two-stage methods.

3. Structured pose representation

SPR represents each person through a root joint and body-joint displacements, while hierarchical SPR decomposes long-range displacements into shorter articulated-joint offsets.

  • Structured Pose Representation: SPR introduces a unique root joint that denotes each person instance’s position and identity.The same representation extends from 2D to 3D by replacing 2D coordinates with 3D coordinates.
  • Structured Pose Representation: Body-joint positions are encoded as displacements relative to the corresponding person’s root joint.This establishes a structured relationship between instance position and joint position and can be converted back to conventional coordinates.
  • Structured Pose Representation: The person centroid is used as the root joint because it is stable and robust for discriminating instances under extreme poses.The centroid-based root is illustrated in the proposed multi-person representation.
  • Hierarchical SPR: Hierarchical SPR factorizes long-range root-to-joint displacements into accumulated short-range displacements between neighboring articulated joints.An articulated path connects the root to each body joint, and the target joint is recovered by accumulating offsets along that path.
  • Hierarchical SPR: Hierarchical SPR retains the unified instance-and-joint representation while improving estimation of large displacements without extra complexity.It can also be converted to SPR and conventional pose representations.

4. Single-stage multi-person pose machine

SPM uses Structured Pose Representation to encode each person through a root joint and body-joint displacements, enabling single-stage multi-person pose prediction. Its Hourglass-based CNN jointly regresses these targets and reconstructs poses from detected roots and displacement maps.

  • Single-stage multi-person pose machine: SPM maps an input image to multiple human poses represented by SPR through a single learned CNN mapping.Unlike two-stage solutions, it predicts all poses in one stage.
  • Regression targets: SPR identifies each person with a root joint and encodes body-joint positions as displacements relative to that root.The representation unifies person-instance position and body-joint localization.
  • Regression targets: Root-joint locations are represented by an aggregated confidence map whose values are Gaussian peaks, combined by maximum to preserve nearby distinctions.The Gaussian variance parameter is set to σ=7 in the experiments.
  • Regression targets: Dense displacement maps average non-zero joint-to-root vectors across persons at each image location; hierarchical SPR instead uses neighboring hierarchy roots.The normalization factor uses image height H and width W, with neighborhood parameter τ=7.
  • Network architecture: The Hourglass backbone adds a displacement-regression branch to its confidence branch, producing hierarchical SPR in one forward pass.The network uses stacked Hourglass modules with downsampling, upsampling, and skip connections.
  • Training and inference: Training uses ℓ2 loss for root confidence and smooth ℓ1 loss for dense displacements, with intermediate supervision across Hourglass modules.The two losses are combined with a balance factor β=0.01.
  • Training and inference: During inference, SPM applies NMS to the predicted root confidence map, samples displacements at detected roots, and combines them into human poses.Hierarchical predictions are assembled sequentially according to joint hierarchies.

5. Experiments

The experiments evaluate SPM on three multi-person 2D benchmarks and one 3D benchmark using established pose metrics and specified training protocols. Reported comparisons include state-of-the-art evaluation on MPII and extended PASCAL-Person-Part, with runtime measured under stated hardware and testing conditions.

  • Experiment setup: SPM is evaluated on MPII, extended PASCAL-Person-Part, MSCOCO, and CMU Panoptic for multi-person 2D and 3D pose estimation.CMU Panoptic supplies the 3D evaluation setting.
  • Experiment setup: MPII contains 5,602 multi-person image groups and uses mAP for evaluation, while extended PASCAL-Person-Part provides 14-joint annotations and also uses mAP.MPII annotations contain 16 body joints per person.
  • Experiment setup: CMU Panoptic experiments use front-view captures from 17 multi-person videos, split into 65,552 training and 10,000 testing images, evaluated with 3D-PCK@150mm.The dataset provides multi-view 3D pose annotations and camera parameters.
  • Implementation: Training uses 384×384 cropped inputs with rotation, scaling, translation, and horizontal-flipping augmentation, while MSCOCO uses a wider scaling range.The reported augmentation ranges differ between MPII/PASCAL and MSCOCO.
  • Comparison with state-of-the-arts: MPII runtime is measured with single-scale testing on a TITAN X GPU and Intel I7-5820K CPU, excluding refinement by single-person pose estimation.Runtime references for comparison methods follow their authors’ available implementations or reported settings.

mAP

SPM combines strong pose accuracy with substantial efficiency gains across 2D and 3D multi-person benchmarks. Its hierarchical representation improves long-range joint estimation without added time cost, while qualitative results show robustness in challenging scenes.

  • MPII: 0.058s per image makes SPM about 5× faster than the bottom-up model while achieving 78.5% mAP on MPII.The result demonstrates both the efficiency and accuracy of single-stage multi-person pose estimation.
  • Ablation analysis: 77.7% mAP improves over SPM-Vanilla’s 75.3% mAP without increasing time cost through hierarchical SPR.Both models require 0.058s per image; hierarchical SPR has the same complexity as SPR.
  • PASCAL-Person-Part: 46.1% mAP establishes new state-of-the-art performance on the extended PASCAL-Person-Part dataset.SPM outperforms previous models for all body joints.
  • MSCOCO: 0.669 AP on MSCOCO is slightly below the state-of-the-art, while SPM is 8× faster than that model.The results indicate competitive performance with a substantial speed advantage.
  • CMU Panoptic: About 20 FPS is achieved on CMU Panoptic, where SPM provides qualitative multi-person 3D pose results without reported quantitative comparison baselines.Its single-stage design avoids intermediate 2D pose estimation or 3D reconstruction from multiple views.

6. Conclusion

The paper concludes that SPM is a first single-stage model for multi-person 2D/3D pose estimation, enabled by structured representations that unify instance and joint positions. Experiments support state-of-the-art speed with strong 2D performance and promising efficient 3D estimation.

  • 6. Conclusion: SPM is presented as the first single-stage model for multi-person 2D/3D pose estimation.It offers a more compact pipeline and an efficiency advantage over existing two-stage solutions.
  • 6. Conclusion: SPR unifies person-instance and body-joint position information, while hierarchical SPR factorizes long-range displacements into accumulative short-range ones without extra complexity.This representation underpins SPM’s compact single-stage design.
  • 6. Conclusion: SPM estimates multiple-person poses in a single-stage feed-forward manner and applies flexibly to both 2D and 3D scenarios.CNN-based end-to-end learning and inference support the model’s implementation.
  • 6. Conclusion: Extensive 2D experiments demonstrate state-of-the-art speed with superior pose-prediction performance, while 3D results show promising performance with attractive efficiency.The conclusion summarizes the reported outcomes across the evaluated benchmarks.
Loading 1908.09220v1…