Source-linked AI summary
Differentiable Hierarchical Graph Grouping for Multi-Person Pose Estimation
Sheng Jin, Wentao Liu, Enze Xie, Wenhai Wang, Chen Qian, Wanli Ouyang, Ping Luo
TL;DR
Bottom-up pose estimation is efficient, but its grouping stage is often separate from keypoint detection and is not directly supervised. HGG turns grouping into differentiable hierarchical graph clustering that can be trained jointly with detection. It reports comparable COCO performance to state-of-the-art bottom-up methods and 41.8/36.0 mAP on OCHuman validation/test.
Problem
Bottom-up grouping is often solved separately from keypoint detection, without direct supervision of grouping results, leaving the grouping loss potentially mismatched with inference parsing.
Source: §1 · pp. 2–3Method
HGG clusters keypoint proposals through differentiable hierarchical graph grouping, jointly trained end-to-end with the keypoint detection network.
Source: §1 · p. 3Results
41.8/36.0 mAP on OCHuman val/test, with performance comparable to state-of-the-art bottom-up methods on COCO and a large-margin gain over the baseline.
Source: §1 · p. 3Takeaways & Limitations
HGG is designed to embed in mainstream bottom-up methods and enables direct hierarchical supervision of grouping alongside detection.
Source: §1 · p. 3The model cannot recover false negatives or localization errors; tiny people may cause false negatives, while severe occlusion and atypical poses may cause localization errors.
Source: §4.3 · p. 12
Abstract
from arXiv · showhide
Multi-person pose estimation is challenging because it localizes body keypoints for multiple persons simultaneously. Previous methods can be divided into two streams, i.e. top-down and bottom-up methods. The top-down methods localize keypoints after human detection, while the bottom-up methods localize keypoints directly and then cluster/group them for different persons, which are generally more efficient than top-down methods. However, in existing bottom-up methods, the keypoint grouping is usually solved independently from keypoint detection, making them not end-to-end trainable and have sub-optimal performance. In this paper, we investigate a new perspective of human part grouping and reformulate it as a graph clustering task. Especially, we propose a novel differentiable Hierarchical Graph Grouping (HGG) method to learn the graph grouping in bottom-up multi-person pose estimation task. Moreover, HGG is easily embedded into main-stream bottom-up methods. It takes human keypoint candidates as graph nodes and clusters keypoints in a multi-layer graph neural network model. The modules of HGG can be trained end-to-end with the keypoint detection network and is able to supervise the grouping process in a hierarchical manner. To improve the discrimination of the clustering, we add a set of edge discriminators and macro-node discriminators. Extensive experiments on both COCO and OCHuman datasets demonstrate that the proposed method improves the performance of bottom-up pose estimation methods.
1 Introduction
Bottom-up pose estimation is efficient, but its grouping stage is often separate from detection and lacks direct supervision. HGG makes grouping a differentiable graph-clustering process, jointly trained with keypoint detection, and reports strong benchmark results.
1 Introduction
- Grouping is often post-processing without end-to-end training or direct supervision, and substitute indicators can mismatch the grouping accuracy at inference.For associative embedding, pairwise loss can disagree with the results of greedy parsing.Source: §1 · p. 2
- HGG reformulates grouping as graph clustering over keypoint proposals and uses Online Hierarchical Graph Clustering to merge related nodes iteratively.The graph adapts to each image and supports a variable number of human instances.Source: §1 · p. 3
- Differentiable hierarchical grouping lets grouping loss back-propagate to the keypoint detection network, while stepwise merges form a hierarchy from body parts to whole poses.This directly supervises grouping and can improve the detector's feature representation.Source: §1 · p. 3
- Edge and macro-node discriminators strengthen local keypoint relations and global consistency to improve grouping discrimination.The contributions include edge discriminators and macro-node discriminators for these complementary roles.Source: §1 · p. 3
- 41.8/36.0 mAP on OCHuman val/test accompanies comparable performance to state-of-the-art bottom-up methods on COCO and a large-margin gain over the baseline.The values are reported for OCHuman validation and test, respectively.Source: §1 · p. 3
2 Related Work
Related work contrasts efficient bottom-up methods with top-down approaches that repeatedly estimate poses within detected person boxes. It then situates HGG among graph-based pose methods and GNNs, emphasizing dynamic grouping for multi-person inputs.
2.1 Multi-person Pose Estimation in Images
- Bottom-up methods share convolutional computation and are usually faster, but their heuristic grouping is separate from pose estimation; top-down methods repeatedly estimate each detected person's pose.Prior bottom-up grouping uses graph partitioning, greedy decoding, or clustering; the paper also notes that its method retains higher precision under stricter AP75 metrics than cited single-stage methods.Source: §2.1 · pp. 4–5
2.2 Graph Representation for Pose Estimation
- For multi-person poses, graph partitioning by integer linear programming is offline and time-consuming, unlike the paper's direct graph-clustering approach.Single-person graph models need no grouping because their keypoints belong to one person and their graph structure is fixed.Source: §2.2 · p. 5
2.3 Graph Neural Networks
- Prior GNN pose applications address single-person settings with fixed graph structure, whereas multi-person images require dynamic interaction as keypoint and person counts vary.The paper develops hierarchical clustering that incorporates body-structure constraints and models the grouping process.Source: §2.3 · p. 5
3 Method
HGG builds a graph over keypoint candidates and uses online hierarchical clustering to group them, with grouping supervised jointly with keypoint detection. Edge and macro-node discriminators provide local and cluster-level supervision.
3 Method
- HGG builds a graph from keypoint candidates and relational features, then uses online hierarchical clustering to group candidates iteratively in an end-to-end trainable pipeline.The grouping network is trained with the keypoint detection network.Source: Overview · p. 6
3.1 Hierarchical Graph Grouping
- OHGC repeatedly aggregates features, updates edge proximity, clusters nodes, and prunes infeasible links, forming a hierarchy from isolated keypoints to whole-body clusters.Pruning prevents a keypoint from being assigned to multiple people and prevents a person from having multiple keypoints of the same type; clustering proceeds from easier pairs toward macro-nodes.Source: Online Hierarchical Graph Clustering… · p. 8
3.2 Grouping Discriminators
- The edge discriminator classifies whether linked nodes belong to the same person, while the macro-node discriminator supervises whether each cluster contains one person’s keypoints.Macro-node supervision applies to both final human-level groups and intermediate part-level groups.Source: Edge Discriminator · p. 9, Macro-node Discriminator · p. 9
3.3 Implementation Details
- The proposal network produces keypoint heatmaps and pairwise relational feature maps, and the proposal network, GNN, and discriminators are jointly trained end to end.Training includes keypoint detection, pairwise push/pull, and discriminator BCE losses; inference uses flip and multi-scale testing without single-person refinement.Source: Keypoint Proposal Network · p. 9, Training and Inference · p. 10
4 Experiments
Experiments evaluate HGG on COCO and OCHuman through benchmark comparisons, component ablations, graph-configuration tests, grouping visualizations, and runtime analysis. HGG establishes a new OCHuman state of the art and is slightly below PersonLab in overall COCO AP.
4.1 Datasets and Evaluation
- Models are trained on COCO; COCO ablations use validation and state-of-the-art comparisons use test-dev, while OCHuman has no training samples and is used for evaluation.The evaluation uses Average Precision (AP); the OCHuman models are trained on COCO.Source: §4.1 · p. 10
4.2 Ablation Study
- Grouping loss raises baseline AP by 0.6% and AR by 1.3%; training with grouping loss also improves the proposal network while allowing grouping to be removed at inference.The inference setting uses the original greedy grouping procedure and adds no computation overhead when the grouping module is removed.Source: §4.2 · p. 11
- 2.0 mAP from edge supervision, 0.5 mAP from final macro-node supervision, and another 0.3 mAP from intermediate supervision yield a 0.8 mAP full-supervision boost.The ablations evaluate HGG components on COCO validation; Table 2 identifies the FinalM, Edge, and IntermM components.Source: §4.2 · pp. 11–12, Table 2 · p. 12
- Graph-based models outperform Ours-FC, with EdgeConv best at 60.4 AP; denser graphs raise grouping accuracy from 56.1% to 60.4% mAP at nearly unchanged runtime, favoring Full.Figure 4a compares configurations on COCO validation and reports Full as best among them.Source: §4.2 · p. 12, Figure 4 · p. 13
4.3 Qualitative Analysis
- OHGC iteratively merges the easiest-to-group keypoints into clusters, forming a part-to-whole pose hierarchy that benefits from global supervision.Figure 5 visualizes clusters by iteration, using different colors for different clusters.Source: §4.3 · p. 12, Figure 5 · p. 14
- Tiny people can cause false negatives, while severe occlusion and non-typical poses can cause localization errors; the model cannot recover either error type.The authors note that multi-scale test-time augmentation may mitigate these issues.Source: §4.3 · p. 12
4.4 Comparisons with the State-of-the-art Methods
- 41.8% and 36.0% mAP on OCHuman validation and test establish a new state of the art; HGG also reaches 67.6 AP on COCO, slightly below PersonLab.PersonLab uses extra instance-segmentation annotations; HGG exceeds recent single-shot methods in AP75, though it is lower in AP50.Source: §4.4 · p. 13
4.5 Runtime Analysis
- On one GTX-1060 GPU, the grouping module accounts for only a small proportion of per-image runtime compared with the full method.The runtime analysis uses COCO validation, PyTorch, batch size 1, and a single thread; Figure 4b compares grouping and keypoint-proposal module runtime.Source: §4.5 · p. 13, Figure 4 · p. 13
5 Conclusion and Future Work
The paper reformulates human pose estimation with a graph model and presents HGG as an end-to-end learning framework. It introduces discriminators to supervise grouping and suggests possible applications to related tasks.
5 Conclusion and Future Work
- HGG reformulates human pose estimation using a graph model in a full end-to-end learning framework.The framework combines CNN feature learning with GNN long-range message passing and relational feature learning.Source: §5 · p. 14
- Edge and macro-node discriminators supervise the whole grouping process.Source: §5 · p. 14
- The authors envision applying the framework to related problems such as multi-object tracking and instance segmentation.Source: §5 · p. 14