PaperlayerView exact revision on arXiv

Source-linked AI summary

DirectPose: Direct End-to-End Multi-Person Pose Estimation

Zhi Tian, Hao Chen, Chunhua Shen

arXiv:1911.07451v2cs.CV

TL;DR

Top-down and bottom-up pose-estimation methods rely on multiple stages, including person detection or heuristic keypoint grouping. DirectPose predicts instance-aware keypoints end to end and uses KPAlign to improve performance by aligning features with keypoints.

  • Problem

    Existing top-down and bottom-up pose-estimation methods use multiple steps, including person detection or heuristic grouping, rather than end-to-end instance-aware prediction.

    Source: §1 · p. 1

  • Method

    DirectPose directly maps an input image to instance-aware keypoints, using differentiable KPAlign to align convolutional features with target keypoints.

    Source: §1 · p. 2

  • Results

    KPAlign significantly improves keypoint detection performance in the end-to-end model.

    Source: §5 · p. 9

  • Takeaways & Limitations

    Direct regression of keypoint coordinates provides a shared framework for keypoint and bounding-box detection, according to the authors.

    Source: §1 · pp. 2–3

Abstract

from arXiv · show

We propose the first direct end-to-end multi-person pose estimation framework, termed DirectPose. Inspired by recent anchor-free object detectors, which directly regress the two corners of target bounding-boxes, the proposed framework directly predicts instance-aware keypoints for all the instances from a raw input image, eliminating the need for heuristic grouping in bottom-up methods or bounding-box detection and RoI operations in top-down ones. We also propose a novel Keypoint Alignment (KPAlign) mechanism, which overcomes the main difficulty: lack of the alignment between the convolutional features and predictions in this end-to-end framework. KPAlign improves the framework's performance by a large margin while still keeping the framework end-to-end trainable. With the only postprocessing non-maximum suppression (NMS), our proposed framework can detect multi-person keypoints with or without bounding-boxes in a single shot. Experiments demonstrate that the end-to-end paradigm can achieve competitive or better performance than previous strong baselines, in both bottom-up and top-down methods. We hope that our end-to-end approach can provide a new perspective for the human pose estimation task.

1. Introduction

Existing top-down and bottom-up pose-estimation pipelines use multiple steps, while a direct end-to-end approach could map images to instance-aware keypoints without detection or grouping. The paper proposes DirectPose with KPAlign to address the difficulty of directly encoding all keypoints for an instance.

1. Introduction

  • A naive direct framework asks one feature vector to encode all keypoint locations, but the paper reports that this yields inferior performance for keypoint detection.The authors contrast this information demand with bounding-box detection, which involves only two corner points.Source: §1 · p. 2
  • KPAlign aligns convolutional features with target keypoints so each feature need encode nearby information, and the differentiable module keeps the model end-to-end trainable.The paper also jointly learns keypoint regression with a heatmap-based task during training.Source: §1 · p. 2
  • DirectPose maps an input image directly to instance-aware keypoints, without bounding-box detection, RoI operations, or grouping post-processing.Its inference avoids top-down early commitment and instance-dependent computation while removing bottom-up heuristic assembly.Source: §1 · pp. 2–3
  • Directly regressing precise keypoint coordinates decouples localization precision from network output resolution, unlike heatmap-based approaches described by the authors.The authors present this as offering potential for detecting very dense keypoints and as a shared methodology for keypoint and bounding-box detection.Source: §1 · p. 3

1.1. Related Work

  • Top-down methods detect and crop each person before pose estimation, while bottom-up methods detect keypoints first and heuristically assemble them into instances.Top-down runtime depends on the number of instances; bottom-up methods involve grouping tricks and hyperparameters.Source: §1.1 · p. 3

2. Our Approach

The approach extends FCOS with direct keypoint regression, then addresses feature–prediction misalignment through KPAlign and uses an auxiliary heatmap task to regularize regression.

2. Our Approach

  • The framework extends FCOS with a keypoint branch, then adds KPAlign and auxiliary heatmap learning to improve keypoint localization.The approach section describes these as the framework’s key components.Source: §2 · p. 3

2.1. End-to-End Multi-Person Pose Estimation

  • Keypoints are represented as 2K coordinates relative to each pixel, treating an instance as a special bounding-box with K corner points.The FPN branch predicts these coordinates at each output level; minimum enclosing keypoint rectangles provide pseudo-boxes for training labels when box annotations are absent.Source: §2.1 · pp. 3–4

2.2. Keypoint Alignment (KPAlign) Module

  • The naive design struggles because one feature vector must encode all instance keypoints, motivating KPAlign’s alignment of features with predicted keypoint locations.The aligner locates rough keypoint positions and samples feature vectors there; the predictor uses the aligned vectors to make final predictions.Source: §2.2 · p. 4
  • For each keypoint, the predictor adds a learned coordinate offset to its sampled location to produce the final coordinate.KPAlign’s operations are differentiable, so the model remains trainable end-to-end with standard back-propagation.Source: §2.2 · p. 5
  • Grouped KPAlign lets nearby keypoints share a feature vector, reducing samples from K to G with similar performance; separate group feature maps can further improve performance.The separate maps use one-quarter as many channels per group; the sampler can also use lower-level features than the locator when available.Source: §2.2 · p. 5

2.3. Regularization from Heatmap Learning

  • An auxiliary heatmap task regularizes the harder regression task during training and is removed at test time.Its P3-based head uses K one-versus-all classifiers; nearest-keypoint locations receive a type label and other locations receive background, with focal loss used for class imbalance.Source: §2.3 · pp. 5–6

3. Experiments

Experiments use COCO human keypoint detection, with separate validation and test-dev splits and AP measured by OKS. The supplied passages describe the evaluation protocol and implementation, but provide no benchmark results.

3. Experiments

  • The experiments use COCO, which contains more than 250K person instances with 17 annotated keypoints, training on trainval35k and using minival for ablations.The trainval35k split contains 57K images, and minival contains 5K.Source: §3 · p. 6
  • Main results are evaluated on COCO test-dev, with performance measured as Average Precision (AP) based on Object Keypoint Similarity (OKS).The test-dev split contains 20K images.Source: §3 · p. 6
  • Unless specified, training uses ResNet-50 and keypoint annotations without boxes; inference applies NMS to detected boxes or keypoint-enclosing rectangles.A longer 100-epoch schedule is used for test-dev results, and test images are resized with shorter side 800 and longer side at most 1333.Source: §3 · p. 6

3.1. Ablation Experiments

The ablations identify feature–prediction alignment as the key improvement over the naive end-to-end framework. Grouped sampling, separate feature maps, finer sampling, and training-time heatmap learning further improve the design or its efficiency.

3.1.1 Baseline: the naive end-to-end framework

  • The naive framework obtains 43.4% in APkp, with its low performance attributed to misalignment between features and keypoint predictions.The baseline replaces FCOS’s bounding-box head with a keypoint head and uses pseudo-boxes to compute training labels on FPN feature maps.Source: §3.1.1 · p. 6

3.1.2 Keypoint alignment (KPAlign) module

  • More than 7 points in APkp are gained with full KPAlign, while disabling its aligner drops performance to 43.0% in APkp, nearly the naive framework’s level.The ablation attributes the improvement to retained alignment between features and keypoint predictions.Source: §3.1.2 · p. 6, Table 1 · p. 6

3.1.3 Grouped KPAlign

  • Grouped KPAlign reduces sampled feature vectors from 17 to 9 while performing slightly better than original KPAlign, so it is used in later experiments.Other tested ways of forming groups achieve similar performance.Source: §3.1.3 · pp. 6–7, Table 2 · p. 7

3.1.4 Using separate convolutional features

  • Separate feature maps improve APkp by 0.8%, from 50.6% to 51.4%, with similar computational complexity and a better speed–accuracy trade-off.The separate feature maps use 64 rather than 256 channels.Source: §3.1.4 · p. 7, Table 2 · p. 7

3.1.5 Where to sample features in KPAlign?

  • Sampling predictor features on finer maps raises performance to 52.2% without increasing computational complexity, with gains mainly at higher keypoint thresholds.The locator uses low-resolution, high-level maps, while the predictor prefers high-resolution, low-level maps.Source: §3.1.5 · p. 7

3.1.6 Regularization from heatmap learning

  • Heatmap learning raises performance from 52.2% to 58.0%; increasing training from 25 to 100 epochs adds 5.1% in APkp.Heatmaps are used only during training, and lower-resolution heatmaps yield similar performance, making the method relatively insensitive to heatmap design choices.Source: §3.1.6 · p. 7, Table 3 · p. 7

3.2. Combining with Bounding Box Detection

The framework adds person bounding-box detection to its end-to-end keypoint method. On COCO minival, its person detection result is comparable to the Faster R-CNN detector in Mask R-CNN.

3.2. Combining with Bounding Box Detection

  • The framework achieves 55.3% in AP for person detection, compared with 53.7% for the Faster R-CNN person detector in Mask R-CNN.The framework adds the original FCOS box-detection head and unifies bounding-box and keypoint detection in the same methodology.Source: §3.2 · pp. 7–8

3.3. Comparisons with State-of-the-art Methods

On COCO test-dev, DirectPose achieves competitive results against strong bottom-up and top-down methods, with performance varying by backbone and test setting. The paper also reports inference times that are slightly faster than Mask R-CNN in one matched-backbone comparison.

3.3. Comparisons with State-of-the-art Methods

  • DirectPose reaches 62.2% and 63.3% in APkp without multi-scale testing using ResNet-50 and ResNet-101, respectively, and 63.0% and 64.8% with multi-scale testing.These results are on the COCO test-dev split and use the best model from the ablations.Source: §3.3 · p. 8
  • With ResNet-50, DirectPose scores 62.2% versus CMU-Pose’s 61.8%; it also scores 63.3% versus AE’s 56.6%.Against PersonLab, using ResNet-101 and single-scale testing, DirectPose scores 63.3% versus 65.5%.Source: §3.3 · p. 8
  • With ResNet-50, DirectPose has similar performance to Mask R-CNN, 62.2% versus 62.7%, while the paper notes it remains behind other top-down methods.The paper contrasts its direct fully convolutional mapping with top-down pipelines that detect, crop, and separately process person instances.Source: §3.3 · p. 8
  • Inference takes 74 ms per image with ResNet-50 and 87 ms with ResNet-101; Mask R-CNN takes 78 ms with ResNet-50, and its runtime depends on instance count.DirectPose’s inference time is reported as nearly constant for any number of instances.Source: §3.3 · p. 8

4. More Discussions and Results

The section compares DirectPose with SPM and presents visualizations of KPAlign, training losses, and keypoint detections. It reports that heatmap learning lowers training loss and that KPAlign uses nearby features to predict keypoints.

4.3. Training Losses of using Heatmap Learning

  • Heatmap learning produces a much lower training loss and is reported to result in better performance.Figure 4 compares loss curves for training with and without heatmap learning.Source: Figure 4 · p. 9, §4.3 · p. 9

4.1. Comparison to SPM

  • Unlike SPM, which predicts instance-agnostic nodes and requires grouping, KPAlign uses final-keypoint coordinates as supervision and aligns features to predictions without grouping.SPM uses Hierarchical SPR to avoid learning long-range displacements between roots and keypoints.Source: §4.1 · pp. 8–9

4.2. Visualization of KPAlign

  • KPAlign uses features near keypoints rather than encoding distant keypoints, which is reported to improve performance.Figure 5 visualizes the feature use.Source: §4.2 · p. 9

4.4. Visualization of Keypoint Detections

  • DirectPose visualizations show instance-aware keypoint detections without grouping post-processing or bounding-box detection; results with simultaneous box detection are also shown.The detections are shown in Figures 6 and 7, respectively.Source: §4.4 · p. 9

5. Conclusion

DirectPose maps raw images directly to instance-aware keypoints, while KPAlign addresses feature-prediction misalignment and heatmap learning improves regression performance. The method achieves competitive or better performance than previous bottom-up and top-down methods.

5. Conclusion

  • DirectPose maps raw images to instance-aware keypoint detections with constant inference time, without grouping, box detection, or RoI operations.The framework is described as end-to-end trainable.Source: §5 · p. 9
  • KPAlign addresses misalignment between convolutional features and predictions, significantly improving keypoint detection performance.The conclusion also reports improved regression-task performance from joint learning with a heatmap-based task.Source: §5 · p. 9
  • DirectPose achieves competitive or better performance than previous bottom-up and top-down methods.Source: §5 · p. 9

Related papers

Loading 1911.07451v2…