Source-linked AI summary
Detect-and-Track: Efficient Pose Estimation in Videos
Rohit Girdhar, Georgia Gkioxari, Lorenzo Torresani, Manohar Paluri, Du Tran
TL;DR
The paper addresses human keypoint estimation and tracking in complex, multi-person videos. It combines short-clip 3D Mask R-CNN pose estimation with lightweight temporal matching, achieving state-of-the-art PoseTrack challenge performance, including 51.8 MOTA on the test set.
Problem
Human pose tracking must estimate and associate poses over time despite pose changes, occlusions, and overlapping people.
Method
A two-stage pipeline uses 3D Mask R-CNN on short clips for temporally informed pose estimation, followed by lightweight matching to link detections.
Results
51.8 MOTA on the full PoseTrack test set outperformed all competing submissions to the ICCV’17 challenge.
Takeaways & Limitations
The results support simple Hungarian matching on strong keypoint predictions as a strong baseline for video keypoint tracking.
Takeaways & Limitations
GPU memory limits led the experiments to use a lightweight ResNet-18 setup with 256-pixel images and three-frame clips.
Abstract
from arXiv · showhide
This paper addresses the problem of estimating and tracking human body keypoints in complex, multi-person video. We propose an extremely lightweight yet highly effective approach that builds upon the latest advancements in human detection and video understanding. Our method operates in two-stages: keypoint estimation in frames or short clips, followed by lightweight tracking to generate keypoint predictions linked over the entire video. For frame-level pose estimation we experiment with Mask R-CNN, as well as our own proposed 3D extension of this model, which leverages temporal information over small clips to generate more robust frame predictions. We conduct extensive ablative experiments on the newly released multi-person video pose estimation benchmark, PoseTrack, to validate various design choices of our model. Our approach achieves an accuracy of 55.2% on the validation and 51.8% on the test set using the Multi-Object Tracking Accuracy (MOTA) metric, and achieves state of the art performance on the ICCV 2017 PoseTrack keypoint tracking challenge.
1. Introduction
The paper targets human pose tracking in complex, multi-person videos, where pose changes, occlusions, and overlapping instances complicate identity-preserving estimation over time. It addresses limitations of frame-only prediction and computationally costly tracking optimization by combining short-clip temporal reasoning with lightweight linking.
- Human pose tracking must estimate and associate every person’s pose across time despite pose changes, occlusions, and overlapping instances.
- Prior systems commonly optimize links between frame-level keypoints, but hand-coded constraints and computational complexity limit scalability beyond short clips.
- Temporal information can improve poorly localized keypoints by using less ambiguous evidence from adjacent frames.
- The proposed two-stage design estimates poses from frames or short clips, then uses lightweight tracking to link predictions across arbitrarily long videos.
- The method is evaluated on PoseTrack using a Mask R-CNN baseline and a 3D Mask R-CNN extension for more robust frame predictions.
2. Related Work
Related work includes top-down and bottom-up image pose estimators, two-stage video pipelines, and multi-object tracking methods. The paper extends top-down estimation to spatiotemporal tubes while using a less expensive tracking stage than prior optimization-heavy approaches.
- Multi-person pose estimation in images: Top-down image methods detect person instances before estimating joints, whereas bottom-up methods detect keypoints and group them into instances.
- Multi-person pose estimation in images: The proposed approach extends top-down models to video by predicting spatiotemporal person tubes before estimating keypoints within them.
- Multi-person pose estimation in video: Dominant video pose systems first estimate frame-level keypoints and then connect them through space-time optimization.
- Multi-person pose estimation in video: Integer-programming approaches can jointly smooth poses and assign identities but are impractical for long videos because the optimization is NP-hard.
- Multi-object tracking in video: Multi-object tracking research spans hand-crafted pipelines and deep recurrent models that use cues such as appearance and motion.
3. Technical Approach
The method uses a two-stage pipeline: a 3D Mask R-CNN estimates person tubes and keypoints from short clips, while lightweight matching links detections into longer tracks. Its architecture integrates spatiotemporal features through 3D convolutions, tube proposals, and temporal RoIAlign.
- Stage 1: Spatiotemporal pose estimation over clips: The first stage extends Mask R-CNN with spatiotemporal operations so short video clips produce pose predictions for all people across their frames.
- 3D Mask R-CNN: Inflated 3D convolutions integrate spatial and temporal cues, while the input changes from a single frame to a clip of adjacent frames.
- 3D Mask R-CNN: A 3D ResNet creates feature blobs, a Tube Proposal Network generates person tubes, and spatiotemporal RoIAlign extracts tube-specific features.
- 3D Mask R-CNN: The model’s heads classify and regress tight tubes and predict keypoint heatmaps for each input frame.
- Stage 2: Linking keypoint predictions into tracks: The second stage represents person detections as graph nodes, connects adjacent-frame boxes, and uses pairwise likelihood costs for matching.
- Stage 2: Linking keypoint predictions into tracks: Sequential bipartite matching propagates identities frame by frame, while unmatched boxes initiate new tracks and support videos of arbitrary length.
4. Experiments and Results
Experiments on PoseTrack evaluate dataset setup, tracking design choices, upper bounds, final performance, runtime, and a lightweight 3D Mask R-CNN variant. The results support simple matching, substantial speed advantages, and further gains from temporal context, while indicating pose estimation remains the main bottleneck.
- Baseline design: The Hungarian algorithm slightly outperforms greedy matching, while tracking performance remains relatively stable across hand-crafted and learned cost criteria.The final system uses Hungarian matching with bounding-box overlap because of its strong performance and low computational cost.
- Upper-bound analysis: 55.2 →57.5 MOTA is the gain from perfect association, whereas perfect pose predictions raise performance from 55.2 to 78.4 MOTA.These upper bounds show that the simple association heuristic is near-optimal for combinatorial matching, while pose quality is the larger challenge.
- Final performance: 51.8 MOTA on the full test set outperforms all competing submissions in the ICCV’17 PoseTrack challenge.The reported test result was obtained from the evaluation server.
- Runtime: 237× speedup is achieved on a 256-frame video, with runtime growing linearly in the proposed method versus non-linearly for the IP-based comparison.The proposed method takes 5.2 minutes compared with 20 hours for the IP-based method.
- 3D Mask R-CNN: The 3D Mask R-CNN improves over a 2D baseline at the same lightweight resolution and network depth by integrating temporal information from short clips.The experiment uses a ResNet-18 architecture, 256-pixel resolution, and T = 3 frame clips, constrained by GPU memory.
5. Conclusion and Future Work
The paper presents an efficient human keypoint-tracking approach that combines strong frame-level pose estimation with fast person-level tracking. Experiments on PoseTrack support simple matching as a strong baseline, while the 3D extension improves frame-level results and the 2D variant remains advantageous at higher resolutions.
- The approach combines state-of-the-art frame-level pose estimation with a fast person-level tracking module to connect keypoints over time.
- A simple Hungarian matching algorithm on top of good keypoint predictions achieves strong performance for keypoint tracking.
- The proposed 3D Mask R-CNN outperforms the 2D baseline for frame-level pose estimation with the same base architecture and input resolution.
- The 2D baseline uses less GPU memory and supports higher image resolutions with high-capacity models, producing state-of-the-art PoseTrack results.
- Extensive ablative experiments explore model design choices and report strong results on the PoseTrack challenge benchmark.