Source-linked AI summary
Deep Dual Consecutive Network for Human Pose Estimation
Zhenguang Liu, Haoming Chen, Runyang Feng, Shuang Wu, Shouling Ji, Bailin Yang, Xun Wang
TL;DR
Static-image pose detectors lose performance on video, where blur, defocus, and occlusion make temporal context important. DCPose combines bidirectional temporal cues with pose refinement, and reports state-of-the-art results on PoseTrack2017 and PoseTrack2018.
Problem
Static-image pose methods lose performance on video, where they miss temporal dependencies and geometric consistency relevant to blur, defocus, and occlusion.
Source: §1 · p. 1Method
DCPose uses PTM to localize keypoint search scopes, PRF to compute weighted residuals across temporal directions, and PCN to refine estimates.
Source: §1 · p. 2Results
Takeaways & Limitations
The authors conclude that DCPose achieves notable improvements and can handle complex scenes.
Source: §5 · p. 8
Abstract
from arXiv · showhide
Multi-frame human pose estimation in complicated situations is challenging. Although state-of-the-art human joints detectors have demonstrated remarkable results for static images, their performances come short when we apply these models to video sequences. Prevalent shortcomings include the failure to handle motion blur, video defocus, or pose occlusions, arising from the inability in capturing the temporal dependency among video frames. On the other hand, directly employing conventional recurrent neural networks incurs empirical difficulties in modeling spatial contexts, especially for dealing with pose occlusions. In this paper, we propose a novel multi-frame human pose estimation framework, leveraging abundant temporal cues between video frames to facilitate keypoint detection. Three modular components are designed in our framework. A Pose Temporal Merger encodes keypoint spatiotemporal context to generate effective searching scopes while a Pose Residual Fusion module computes weighted pose residuals in dual directions. These are then processed via our Pose Correction Network for efficient refining of pose estimations. Our method ranks No.1 in the Multi-frame Person Pose Estimation Challenge on the large-scale benchmark datasets PoseTrack2017 and PoseTrack2018. We have released our code, hoping to inspire future research.
1. Introduction
Static-image pose methods lose performance on video, where blur, defocus, and occlusion make temporal cues important. DCPose combines bidirectional temporal context with keypoint refinement and reports state-of-the-art results on PoseTrack benchmarks.
1. Introduction
- Static-image pose methods have diminished performance on video, where missing temporal and geometric cues are important for challenging blur, defocus, and occlusion cases.The text notes that recurrent approaches can also struggle with occlusion in multi-person scenes and self-occlusion.Source: §1 · pp. 1–2
- The PTM illustration shows preliminary right-wrist heatmaps for a cropped person clip and merged heatmaps whose color intensity encodes spatial aggregation.The merged maps are generated by encoding keypoint spatial contexts.Source: Figure 1 · p. 2
- DCPose uses PTM to aggregate keypoints into localized search scopes, PRF to compute inter-frame residuals, and PCN to refine heatmaps within those scopes.PCN uses five parallel convolution layers with different dilation rates.Source: §1 · p. 2
- Unlike PoseWarper's label propagation, DCPose refines the current-frame estimate with motion context from unlabeled neighboring frames and fuses heatmaps from adjacent and current frames.It also uses information from both temporal directions and explicitly considers weighted inter-frame residuals.Source: §1 · p. 2
- DCPose achieves state-of-the-art results on the PoseTrack2017 and PoseTrack2018 Multi-frame Person Pose Estimation Challenge.The authors also report releasing the source code.Source: §1 · p. 2
2. Related Work
Image-based multi-person pose estimation developed from structured body models to neural heatmap methods, while video approaches add temporal information through flow, recurrent, warping, or temporal-convolution methods.
2.1. Imaged Based Multi Person Pose Estimation
- Earlier image-based methods represented bodies with tree- or forest-structured pictorial models, while neural approaches commonly use heatmaps and either bottom-up part assembly or top-down person detection.The text notes that top-down approaches estimate each detected person separately.Source: §2.1 · pp. 2–3
- HRNet retains high-resolution feature maps through multi-scale fusion, improving spatial precision in keypoint heatmaps and achieving strong image-benchmark results.The passage describes HRNet as a recent image-based method.Source: §2.1 · p. 3
2.2. Video Based Multi Person Pose Estimation
- Video methods add temporal cues through optical flow, convolutional LSTMs, pose warping, or temporal convolutions, but flow and recurrent approaches have stated limits in blur, defocus, or occlusion.3DHRNet extends HRNet with temporal convolutions and is described as successful for pose estimation and tracking jointly.Source: §2.2 · pp. 3–4
3. Our Approach
DCPose refines current-frame pose estimates by combining temporal context from prior and future frames. Its modules aggregate keypoint heatmaps and pose residuals before correcting the initial estimates.
3.1. Pose Temporal Merger
- PTM uses temporal consistency to create a confined search scope, weighting nearer frames more and applying group convolution to joint-specific heatmaps.Residual blocks process the grouped features to produce merged keypoint heatmaps.Source: §3.1 · p. 4
3.2. Pose Residual Fusion
- PRF computes pose residuals between adjacent frames and combines original and temporally weighted residuals into a pose residual feature.Residual blocks process the concatenated features.Source: §3.2 · pp. 4–5
3.3. Pose Correction Network
- PCN uses pose residual features to predict offsets at five dilation rates and merged heatmaps to predict masks, then combines the resulting pose predictions.The dilation rates provide receptive fields of different sizes, from local to broader spatial contexts.Source: §3.3 · p. 5
3.4. Implementation Details
- The implementation uses a PoseTrack-pretrained HRNet-W48 backbone, freezes its parameters throughout training, and backpropagates only through subsequent DCPose components while minimizing standard L2 distance between prediction and ground-truth heatmaps for 15 joints.Source: §3.4 · p. 6
4. Experiments
DCPose reports state-of-the-art test results on PoseTrack2017 and PoseTrack2018, with ablations examining its modules, temporal inputs, and frame spacing. Visual comparisons cover challenging motion, proximity, occlusion, and defocus cases.
4.1. Experimental Settings
- PoseTrack2017 has 514 clips and 16,219 pose annotations, while PoseTrack2018 has 1,138 clips and 153,615 annotations; evaluation uses AP on visible joints.Source: §4.1 · p. 6
4.2. Comparison with State-of-the-art Approaches
- DCPose achieves state-of-the-art results for multi-frame person pose estimation on the validation and test sets, with 79.2 mAP on the test set; on PoseTrack2018, test-set mAP is 79.0.Source: §4.2 · p. 6
4.3. Ablation Experiments
- Removing PTM lowers mAP from 82.8 to 82.0 or 82.2 and weakens the effective search range for subsequent joint localization.Source: §4.3 · p. 7
- Removing PRF drops mAP by 0.7 to 82.1, with the authors linking the loss to less accurate residual cues for offsets and keypoint localization.Source: §4.3 · p. 7
4.4. Comparison of Visual Results
- Across rapid motion, nearby-person, occlusion, and video-defocus cases, the visual comparison reports more robust detection by DCPose than the three comparator methods.Figure 4 marks inaccurate predictions with red circles.Source: Figure 4 · p. 8, §4.4 · p. 8
4.3. Ablation Experiments
- PCN mAP across expanding dilation settings is 81.7, 81.9, 82.7, 82.6, and 82.8, which the authors associate with broader local and global receptive fields.The complete setting uses dilation rates {3, 6, 9, 12, 15}.Source: §4.3 · pp. 7–8
- Removing the previous-frame input lowers mAP by 0.6, while removing the next-frame input lowers it by 0.5.Source: §4.3 · p. 8
- mAP falls from 82.8 at T=1 to 82.6, 82.1, and 81.7 at T=2, 3, and 4, respectively, as wider frame spacing weakens temporal cues.The authors attribute this pattern to reduced temporal consistency at larger intervals.Source: §4.3 · p. 8
5. Conclusion
The conclusion presents DCPose as a dual-consecutive framework that outperforms existing methods on benchmark datasets. Its temporal modules provide localized, residual-corrected search ranges, which the correction network refines for complex scenes.
5. Conclusion
- DCPose is a dual-consecutive multi-frame pose-estimation framework that significantly outperforms existing methods on benchmark datasets.Source: §5 · p. 8
- Pose Temporal Merger and Pose Residual Fusion draw auxiliary information from adjacent frames to provide localized, pose-residual-corrected search ranges for keypoint location.Source: §5 · p. 8
- Pose Correction Network uses multiple effective receptive fields to refine estimates in the search range and handle complex scenes.Source: §5 · p. 8