Source-linked AI summary
Pose Flow: Efficient Online Pose Tracking
Yuliang Xiu, Jiefeng Li, Haoyu Wang, Yinghong Fang, Cewu Lu
TL;DR
Multi-person pose tracking in unconstrained videos remains challenging because frame-level pose estimates can be unreliable under blur, truncation, and occlusion. The paper introduces an online pose-flow tracker using PF-Builder and PF-NMS, achieving 10 FPS tracking and improving results in ablation studies.
Problem
Frame-level pose estimates in top-down tracking can become unreliable under blur, truncation, or occlusion, requiring robust cross-frame pose association.
Method
The method builds pose flows through online optimization of cross-frame pose associations and applies pose-flow non-maximum suppression to merge redundant or temporally disjoint flows.
Results
The combined PF-Builder and PF-NMS improve tracking results by 1.9 mAP and 2.5 MOTA, while the tracker processes video at 10 FPS excluding pose estimation.
Takeaways & Limitations
The proposed tracker provides a scalable top-down approach that leverages spatio-temporal information for online pose tracking and has potential for realistic applications.
Abstract
from arXiv · showhide
Multi-person articulated pose tracking in unconstrained videos is an important while challenging problem. In this paper, going along the road of top-down approaches, we propose a decent and efficient pose tracker based on pose flows. First, we design an online optimization framework to build the association of cross-frame poses and form pose flows (PF-Builder). Second, a novel pose flow non-maximum suppression (PF-NMS) is designed to robustly reduce redundant pose flows and re-link temporal disjoint ones. Extensive experiments show that our method significantly outperforms best-reported results on two standard Pose Tracking datasets by 13 mAP 25 MOTA and 6 mAP 3 MOTA respectively. Moreover, in the case of working on detected poses in individual frames, the extra computation of pose tracker is very minor, guaranteeing online 10FPS tracking. Our source codes are made publicly available(https://github.com/YuliangXiu/PoseFlow).
1 Introduction
Multi-person pose tracking extends pose estimation to unconstrained videos, where frame-level errors and temporal association remain challenging. The paper proposes efficient pose-flow construction and suppression within a top-down pipeline, reporting strong benchmark results and online 10 FPS tracking.
- Pose trajectories support action recognition, person re-identification, human-object interaction, surveillance, and sports video analysis.
- Top-down methods detect people, estimate keypoints independently, then track human boxes using adjacent-frame similarity.
- Frame degeneration, truncation, and occlusion can make individual-frame pose estimates unreliable, motivating cross-frame association to reduce uncertainty.
- PF-Builder associates cross-frame poses by maximizing overall confidence, while PF-NMS reduces redundant pose flows and relinks disjoint ones.
- 58.3% MOTA and 66.5% mAP were achieved on the PoseTrack Challenge validation set, compared with 51.0% MOTA and 63.0% mAP on its testset.
2 Related Work
Prior work includes graph-based and multi-stage pose estimation and tracking, but graph optimization can be computationally heavy and difficult to scale. The paper positions pose-flow processing as an efficient alternative that uses temporal information to produce robust trajectories.
- Graph decomposition methods formulate multi-person pose estimation as graph partitioning and solve it with integer linear programming.
- Multi-stage pipelines separate human detection, single-person pose estimation, and post-processing, with frameworks differing mainly in detector and pose-estimator choices.
- PoseTrack and ArtTrack extend spatial joint graphs into spatio-temporal graphs, but hand-crafted graphical models are not scalable for long clips of unseen scenes.
- Top-down tracking links frame-level poses using appearance similarity and temporal relationships, while needing robust filtering, trajectories, and identity association.
- Unlike 3D Mask R-CNN, the proposed generator treats pose flows as units and combines Pose Flow Builder with Pose Flow NMS.
- The overall pipeline estimates poses, builds pose flows by maximizing confidence, and applies Pose Flow NMS to obtain multi-pose trajectories.
3 Our Proposed Approach
The approach builds pose flows by optimizing cross-frame associations, then applies pose-flow NMS to merge redundant or disjoint trajectories while preserving online operation.
- Pipeline: Pose Flow Building and Pose Flow NMS respectively construct pose flows and reduce redundant or disjoint pose flows.The framework first maximizes confidence along temporal sequences, then performs spatio-temporal suppression and merging.
- Pose Flow Building: Pose-flow building selects cross-frame pose associations by maximizing the summed confidence of candidate trajectories.The objective can be solved online with dynamic programming, allowing low-confidence frames caused by occlusion or blur to be compensated by other frames.
- Pose Flow Building: The tracker stops extending a pose flow only when confidence accumulated over following frames falls below a threshold, resisting sudden occlusion and motion blur.The stopping criterion examines multiple future frames rather than relying on confidence from a single frame.
- Pose Flow NMS: PF-NMS treats each pose flow as the suppression unit, grouping flows close to a high-confidence reference and merging them into a representative flow.Pose-flow distance uses temporal overlap and a median intra-frame pose distance, which is more robust to missed detections from occlusion and motion blur.
- Pose Flow NMS: The merging step reduces redundant pose flows and re-links temporally disjoint flows into longer completed trajectories.It computes representative keypoint coordinates and confidence scores from the pose flows in each group.
- Online Processing: The complete process runs in a sliding temporal window of length L = 20, making pose-flow construction online.The pipeline can use different human detectors and pose estimators and adds only minor tracking computation.
4 Experiments and Results
Experiments evaluate the tracker on PoseTrack datasets, ablate its components, and measure computational cost. The method achieves strong tracking results while maintaining online operation.
- Evaluation and datasets: The evaluation covers PoseTrack and PoseTrack Challenge datasets, including multi-person pose tracking and pose estimation tasks.The Challenge dataset reports validation and test-set results, while the PoseTrack dataset includes complex activities, occlusion, truncation, and motion blur.
- Time performance: The tracker requires 100ms per frame of extra computation and is not the bottleneck relative to pose estimation.It operates on poses detected independently in individual frames and can be applied with different multi-person pose estimators.
- Ablation studies: PF-NMS merges redundant pose flows and re-links temporally disjoint ones, improving results by 1.9 mAP and 2.5 MOTA.PF-Builder provides the pose-flow construction used before PF-NMS in the ablation comparison.
- Ablation studies: DMGP improves tracking by 4.6 MOTA and pose estimation by 4.3 mAP, while iDPG adds 1.1 mAP and 0.5 MOTA.DMGP recovers missing boxes through bidirectional propagation; iDPG targets difficult keypoints.
5 Conclusion
The paper presents a scalable, efficient top-down pose tracker that uses spatio-temporal information to build pose flows. Its tracker processes video at 10 FPS excluding pose estimation and is positioned for realistic applications.
- Conclusion: The tracker combines PF-Builder and PF-NMS with iDPG and DMGP to improve pose tracking performance.
- Conclusion: The pose tracker processes video frames at 10 FPS when pose estimation is excluded.
- Future work: The authors identify long-term action recognition and scene understanding as future applications of the proposed tracker.