PaperlayerView exact revision on arXiv

Source-linked AI summary

Single-Network Whole-Body Pose Estimation

Gines Hidalgo, Yaadhav Raaj, Haroon Idrees, Donglai Xiang, Hanbyul Joo, Tomas Simon, Yaser Sheikh

arXiv:1909.13423v1cs.CVcs.LG

TL;DR

Whole-body pose estimation must combine body, face, hand, and foot keypoints despite their differing scale needs and the runtime cost of prior multi-network methods. This paper uses multi-task learning and an improved single-network architecture; it reports faster inference and training than OpenPose, with slightly improved overall accuracy.

  • Problem

    Prior whole-body estimation uses person-dependent face and hand detectors, while body/foot and face/hand keypoints have different scale requirements.

    Source: §1 · pp. 1–2

  • Method

    The method trains a single whole-body network with multi-task learning and an architecture designed to handle the scale differences among keypoint tasks.

    Source: §1 · p. 2, §3.2 · p. 4

  • Results

    The method considerably outperforms OpenPose in training and testing speed and slightly improves accuracy overall, with higher accuracy especially for face and hand keypoints.

    Source: §1 · p. 2, §5 · p. 7

  • Takeaways & Limitations

    The single-network approach provides constant real-time inference regardless of the number of people in an image.

    Source: §1 · p. 2

    The initial shallow detector uses low input resolution, limiting face and hand pixel-localization precision.

    Source: §3.2 · p. 5

Abstract

from arXiv · show

We present the first single-network approach for 2D~whole-body pose estimation, which entails simultaneous localization of body, face, hands, and feet keypoints. Due to the bottom-up formulation, our method maintains constant real-time performance regardless of the number of people in the image. The network is trained in a single stage using multi-task learning, through an improved architecture which can handle scale differences between body/foot and face/hand keypoints. Our approach considerably improves upon OpenPose~\cite{cao2018openpose}, the only work so far capable of whole-body pose estimation, both in terms of speed and global accuracy. Unlike OpenPose, our method does not need to run an additional network for each hand and face candidate, making it substantially faster for multi-person scenarios. This work directly results in a reduction of computational complexity for applications that require 2D whole-body information (e.g., VR/AR, re-targeting). In addition, it yields higher accuracy, especially for occluded, blurry, and low resolution faces and hands. For code, trained models, and validation benchmarks, visit our project page: https://github.com/CMU-Perceptual-Computing-Lab/openpose_train.

1. Introduction

Whole-body pose estimation unifies body, face, hand, and foot keypoints, but prior approaches struggle with multi-person runtime and the different scales required by these keypoint groups. The paper proposes a single-network approach using multi-task learning and an improved architecture.

1. Introduction

  • The goal is to integrate body, face, hand, and foot keypoint detection within one whole-body estimation algorithm.Earlier work addressed these keypoint tasks separately.Source: §1 · p. 1
  • OpenPose’s multi-network approach depends on body detections and adds face and hand detectors per person, making runtime grow with the number of people.If body-only detection fails, the approach cannot recover from visible faces or hands.Source: §1 · pp. 1–2
  • Body and foot keypoints need large receptive fields, while face and hand keypoints need higher image resolution, and their datasets reflect these different scales.Body datasets often contain multiple people with low-resolution faces and hands; face datasets focus on single or cropped subjects, while hand datasets usually show a single full-body person.Source: §1 · p. 2
  • The method combines multi-task learning with an improved architecture to train one model across keypoint tasks with different scale characteristics.The paper notes that prior multi-task learning for body and foot does not generalize to whole-body estimation because of the scale problem.Source: §1 · p. 2
  • The authors report constant real-time inference regardless of people count, approximately n times faster inference than OpenPose for n people, and higher accuracy, especially for face and hand keypoints.They also report single-stage training rather than independent training for each task, reducing total training time by approximately one-half.Source: §1 · p. 2

2. Related Work

Related work spans face alignment, body and foot pose estimation, hand detection, and whole-body systems. The paper builds on multi-task learning to combine these previously separate keypoint tasks.

2. Related Work

  • Face alignment methods are broadly divided into template fitting and regression-based approaches, with CNN methods typically using convolutional heatmap regression.Template fitting commonly uses cascades of regression functions.Source: §2 · p. 2
  • Body pose estimation progressed from graphical models to CNN-based single-person methods and then to top-down and bottom-up multi-person approaches.The passage attributes the accuracy boost to CNNs and the release of large annotated datasets.Source: §2 · p. 2
  • Cao et al. [9] introduced a foot dataset from a COCO subset and a combined body-foot detector using naive multi-task learning.The paper describes its work as extending that detector to face and hand keypoints as well.Source: §2 · p. 2
  • Hand annotation is difficult because of self-occlusion, and available RGB methods include multi-view bootstrapping, though many methods have only shown results in controlled labs.The cited multi-view approach uses a manually labeled subset to train a detector, then adds annotations from a 30-camera dome using 3D reconstruction.Source: §2 · pp. 2–3
  • OpenPose [9] [10] [58] is described as the only known 2D whole-body system, detecting body and foot first and then running dedicated face and hand networks for candidates.The approach estimates face and hand bounding boxes from body keypoints.Source: §2 · p. 3
  • Multi-task learning can use soft or hard parameter sharing, and the paper aims to combine body, face, hand, and foot detection in one model.Soft sharing keeps separate models but regularizes their parameters; hard sharing is common in computer vision.Source: §2 · p. 3

3. Method

The method extends a Part Affinity Field (PAF) pose detector to whole-body estimation, combining body, face, hand, and foot keypoints in one network. It adapts training and architecture to differences among keypoint scales and datasets.

3.1. PAF-based Body Pose Estimation

  • PAF stages iteratively refine part-to-part association fields, then combine the final fields with image features to predict confidence maps and assemble poses.The pipeline applies per-stage ℓ2 losses, non-maximum suppression, and bipartite matching.Source: §3.1 · pp. 3–4

3.2. Whole-Body Pose Estimation

  • The unified model concatenates body, face, hand, and foot outputs, connects tasks through shared ankle, wrist, and eye keypoints, and makes only the final layer task-specific.This is an extreme form of hard parameter sharing.Source: §3.2 · p. 4
  • Because available datasets annotate different keypoint subsets, training samples datasets by probability ratio and masks losses for unlabeled keypoints; scale-specific augmentation exposes faces and hands at relevant sizes.Face augmentation includes smaller scales, while hand augmentation increases the maximum scale.Source: §3.2 · pp. 4–5
  • Dataset sampling is adjusted to address overfitting, while empty-image examples and masking unlabeled people help mitigate false positives and incomplete annotations.Face and lab-recorded datasets are downweighted when validation convergence is uneven; confidence-map Gaussian radii are reduced to improve face and hand localization precision.Source: §3.2 · p. 5
  • The shallow detector loses body/foot accuracy and has imprecise face/hand localization, so the final architecture raises input resolution, restores receptive field, widens late layers, and reduces PAF stages.The output grows from 25 to 135 keypoints, requiring about five times more information to be compressed with the same parameter count.Source: §3.2 · p. 5
  • The improved model is approximately n times faster than Cao et al. [9] for an image with n people and slightly improves global accuracy.The speedup is reported for the final deep whole-body model.Source: §3.2 · p. 5

4. Evaluation

The evaluation compares body, foot, face, hand, and whole-body keypoint results across their respective datasets using COCO-style AP and AR. It reports stronger in-the-wild face and hand results and runtime that stays constant as the number of people increases.

4.1. Experimental Setup

  • Evaluation uses separate datasets for body, foot, hand, face, and whole-body keypoints, reporting mean AP and mean AR for all keypoint sets.The evaluation includes COCO body data, OpenPose foot and hand datasets, custom face data, and Monocular Total Capture whole-body data.Source: §4.1 · pp. 5–6

4.2. Ablation Experiments

  • At matched inference time, the efficient configuration increases convolutional blocks and width while reducing refinement stages.The models were tuned for the same inference runtime; APs and ARs denote single-scale results.Source: §4.2 · p. 6, Table 1 · p. 6

4.3. Body and Foot Keypoint Detection Accuracy

  • The deeper body-foot model improves accuracy by 1.1% over Cao et al. [9], while deep whole-body trails deep body-foot by less than 1%.Adding face and hand keypoints reduces body accuracy by about 5% for the shallow model compared with [9].Source: §4.3 · p. 6, Table 2 · p. 6

4.4. Face Keypoint Detection Accuracy

  • On the in-the-wild i-bug dataset, the approach is about 2% more accurate, although both methods over-fit on Multi-PIE and FRGC.The Multi-PIE and FRGC sets contain controlled-lab images with frontal, unoccluded faces.Source: §4.4 · pp. 6–7, Table 3 · p. 7

4.5. Hand Keypoint Detection Accuracy

  • On Hand MPII, the deeper model outperforms Cao et al. [9] and the initial shallow model by about 5.5%.Hand MPII consists of manually annotated MPII images and is described as a more challenging in-the-wild set than Hand Dome.Source: §4.5 · p. 7, Table 4 · p. 7

4.6. Run-time Comparison

  • For images with 10 people, the approach is about 7 times faster than OpenPose [9], with inference time remaining constant as people increase.OpenPose runtime grows with the number of people and face and hand proposals; crowd occlusion slightly reduces the speedup.Source: §4.6 · p. 7, Figure 4 · p. 7

5. Conclusion

The unified single-network method considerably improves speed and slightly improves accuracy, while face and hand detection still has specific failure cases.

5. Conclusion

  • The method considerably outperforms OpenPose in training and testing speed while slightly improving its accuracy.Source: §5 · p. 7
  • Failures remain under substantial occlusion or cropping, and face and especially hand accuracy is limited for severe blur, small people, extreme gestures, or nearby hands.Source: §5 · p. 7, Figure 5 · p. 8
  • Face and hand detectors generalize better in-the-wild, including for gloves, small or blurry faces, extreme or occluded hands, and cropped arms.Source: §5 · p. 7, Figure 5 · p. 8
  • For simple, unoccluded face and hand poses, prior OpenPose can outperform because enlarged candidate crops provide higher pixel-localization precision when detection succeeds.Source: §5 · pp. 7–8

Related papers

Loading 1909.13423v1…