Combining detection and tracking for human pose estimation in videos

• The method propagates detected person boxes to neighboring frames and searches there for people missed by the detector.A Clip Tracking Network estimates and tracks body joints across the resulting spatial-temporal tube, including frames without a person detection.Source: §1 · pp. 1–2
• Top-down video methods can underperform because detectors miss people under occlusion, unusual viewpoints, motion blur, and atypical poses.Unlike image settings, video frames can contain these challenging conditions, including highly occluded people that are not localized.Source: §1 · p. 1
• Overlapping tracklets are associated and merged into arbitrary-length tracks using joint hypotheses that favor spatial agreement and temporal smoothness.The consensus-based merging can correct pose mistakes on highly entangled people.Source: §1 · p. 2
• The approach achieves state-of-the-art joint detection and tracking on PoseTrack 2017 and 2018, against both top-down and bottom-up methods.The introduction reports error reductions of 28% for joint detection on PoseTrack 2017 and 9% for joint tracking on PoseTrack 2018.Source: §1 · p. 2

2.1. Human pose estimation in images

• Top-down image methods avoid joint grouping and large scale variation, but cannot recover joints for people their detector fails to localize.Bottom-up methods avoid detector dependence but must handle joint detection across scale variation and grouping joints into people.Source: §2.1 · p. 2
• The paper extends HRNet, originally proposed for image pose estimation, to human pose estimation and tracking in videos.HRNet had also been adapted for object detection [31] and semantic segmentation [32].Source: §2.1 · p. 2

2.2. Human pose estimation and tracking in videos

• Prior video methods link bottom-up joints with spatial-temporal graphs or top-down person boxes with temporal graphs.Examples include SpatioTemporal Affinity Fields, Spatio-Temporal Embedding, optical-flow-based linking, and clip-based detection and tracking.Source: §2.2 · p. 2
• Compared with Detect-and-Track, this method propagates boxes to recover missed people, uses overlapping clips and multiple joint hypotheses, and needs 3D filters in only part of the network.Detect-and-Track uses non-overlapping clips, box-based tracking, and fully 3D convolutional networks.Source: §2.2 · p. 3

3. Methodology

• The method cuts videos into overlapping clips, detects people in each keyframe, estimates poses forward and backward, sews tracklets into longer tracks, and polishes joints using spatial-temporal consistency.The three components are the Clip Tracking Network, Video Tracking Pipeline, and Spatial-Temporal merging procedure.Source: §3 · p. 3

3.1. Clip Tracking Network

• A keyframe person box defines a tube processed by 3D HRNet to estimate that same person's poses across the clip, including when occluded or outside the tube frame.The network enlarges boxes by 25% and inflates early HRNet convolutions to 3D to support movement and tracking.Source: §3.1 · p. 3, Figure 2 · p. 4

3.2. Video Tracking Pipeline

• Overlapping fixed-length tracklets are associated by pose similarity on shared frames and merged into arbitrary-length tracks, allowing pose estimates where keyframe detections were missed.The pipeline models association as bipartite-graph energy minimization using Hungarian matching and overlap-frame OKS.Source: §3.2 · p. 4

3.3. Spatial-Temporal merging of pose hypotheses

• Selecting each joint's highest-confidence hypothesis can place it incorrectly when people are entangled or occluded.This is the Baseline Merge procedure.Source: §3.3 · pp. 4–5
• The refinement clusters joint hypotheses and uses a shortest-path optimization to favor locations consistent across candidates in a frame and across consecutive frames.Mean-shift clustering smooths hypothesis noise and reduces graph size; Dijkstra's algorithm solves the shortest-path problem.Source: §3.3 · p. 5, Figure 4 · p. 5

4.1. Datasets and Evaluation

• PoseTrack2017 and 2018 provide crowded video benchmarks with 15 annotated joints; evaluation averages per-joint AP and MOTA.The implementation uses 9-frame clips, step size 1, and a ResNet-101 SNIPER person detector; validation MOTA thresholds low-confidence predictions, and very short tracklets and tiny boxes are removed.Source: §4.1 · p. 5, §4.2 · p. 6

4.3. Comparisons with the state-of-the-art

• Our method reaches state of the art on both validation datasets and tasks, including +6.5 mAP on PoseTrack2017 and +3.0 MOTA on PoseTrack2018.For PoseTrack2017 tracking, it exceeds FlowTrack by +6.2 MOTA (71.6 vs 65.4).Source: Table 1 · p. 6, Table 4 · p. 6, Table 2 · p. 6
• On test sets, the method achieves the best tracking results on both datasets (+3 MOTA) and joint detection results on par with state of the art.The PoseTrack2018 model was trained on less data than competitors.Source: Table 5 · p. 6, Table 6 · p. 6, §4.3 · p. 6

4.4. Analysis of our approach

• 83.8 mAP and 71.6 MOTA: the full model combines spatial and temporal merging, improving on the component ablations.The 3D HRNet without tracking reaches 82.3 mAP versus 77.7 mAP for the framewise 2D baseline; tracking with the pipeline and baseline merge adds +0.8 mAP.Source: Table 7 · p. 7, §4.4 · p. 7
• With |C| = 3, all tested 3D-filter placements outperform 2D HRNet, and the Early design performs best on detection and tracking.Early places 3D convolutions in the first two stages; Last places them in the last stage, and All uses a fully 3D architecture.Source: Table 8 · p. 7, §4.4 · pp. 7–8
• Increasing clip length improves mAP and MOTA while reducing false negatives; the largest gain comes from |C| = 1 to 3.The paper attributes this early gain to temporal information compensating for person-detector failures.Source: §4.4 · p. 7, Figure 5 · p. 7
• Increasing step size S speeds inference linearly by S but lowers detection and tracking results; at S = 8, performance remains competitive.At S = 8, results are 78.9 mAP and 67.2 MOTA, above the 2D HRNet baseline’s 77.7 mAP and 65.6 MOTA, while the detector runs every eight frames.Source: §4.4 · p. 7, Figure 5 · p. 7
• On PoseTrack2018, the 3D approach recovers 4–7% recall and raises a weaker detector’s recall to 83, near the 2D ResNet-101’s 82.9.The comparison uses two backbones and the PoseTrack2018 validation set.Source: Table 9 · p. 8, §4.4 · p. 8

5. Conclusion

• The approach achieves state-of-the-art joint detection and tracking results on both PoseTrack 2017 and 2018, against top-down and bottom-up approaches.Source: §5 · p. 8
• The method recovers from person-detector failures by propagating known person locations through time and searching those regions for poses.Source: §5 · p. 8
• Clip Tracking Network estimates poses and tracks on clips, Video Tracking Pipeline merges tracklets, and Spatial-Temporal Merging refines joint locations through consensus.Source: §5 · p. 8
• The approach correctly predicts poses in difficult scenes with severe occlusion and entanglement.Source: §5 · p. 8