Source-linked AI summary
PoseTrack: A Benchmark for Human Pose Estimation and Tracking
Mykhaylo Andriluka, Umar Iqbal, Eldar Insafutdinov, Leonid Pishchulin, Anton Milan, Juergen Gall, Bernt Schiele
TL;DR
Articulated multi-person pose tracking in monocular video lacked a broad benchmark covering realistic variation. PoseTrack introduces a large-scale dataset and three related evaluation tracks with standardized held-out-test assessment. The benchmark also analyzes existing approaches, training-data effects, and remaining evaluation and annotation challenges.
Problem
Articulated multi-person body-joint tracking in monocular video remains largely unaddressed, while existing training sets are limited in scenarios, activities, and body-motion variability.
Method
PoseTrack collects and annotates multi-person video sequences with poses and person tracks, organizes three benchmark tasks, and evaluates methods using standardized metrics and an online held-out-test server.
Results
The benchmark supports objective comparison across single-frame pose estimation, video pose estimation, and articulated tracking, while submitted methods commonly combine detection, pose estimation, and frame-to-frame association.
Takeaways & Limitations
PoseTrack provides a large, diverse platform for training and evaluating articulated people tracking in realistic scenes.
Takeaways & Limitations
Creating truly large annotated datasets of articulated pose sequences remains a major challenge, motivating transfer learning, propagated annotations, and synthetic data.
Abstract
from arXiv · showhide
Human poses and motions are important cues for analysis of videos with people and there is strong evidence that representations based on body pose are highly effective for a variety of tasks such as activity recognition, content retrieval and social signal processing. In this work, we aim to further advance the state of the art by establishing "PoseTrack", a new large-scale benchmark for video-based human pose estimation and articulated tracking, and bringing together the community of researchers working on visual human analysis. The benchmark encompasses three competition tracks focusing on i) single-frame multi-person pose estimation, ii) multi-person pose estimation in videos, and iii) multi-person articulated tracking. To facilitate the benchmark and challenge we collect, annotate and release a new %large-scale benchmark dataset that features videos with multiple people labeled with person tracks and articulated pose. A centralized evaluation server is provided to allow participants to evaluate on a held-out test set. We envision that the proposed benchmark will stimulate productive research both by providing a large and representative training dataset as well as providing a platform to objectively evaluate and compare the proposed methods. The benchmark is freely accessible at https://posetrack.net.
1. Introduction
PoseTrack addresses the largely unaddressed problem of articulated multi-person body-joint tracking in monocular video by establishing a large-scale benchmark. It organizes evaluation across single-frame pose estimation, video pose estimation, and articulated tracking, with centralized held-out-test evaluation.
- Existing benchmarks have advanced single-person and single-frame multi-person pose estimation, but articulated multi-person tracking in monocular video remains largely unaddressed.
- PoseTrack establishes a new large-scale, high-quality benchmark for video-based multi-person pose estimation and articulated tracking.
- The benchmark covers single-frame multi-person pose estimation, multi-person pose estimation in video, and multi-person articulated tracking.
- A centralized evaluation server supports scalable evaluation on a held-out test set and standardized comparison of methods.
- Feedback from twelve competition teams informed the paper, which also analyzes leading approaches and common difficulties.
2. Related Datasets
Existing pose datasets generally emphasize single-person or single-frame settings, while PoseTrack targets multi-person video pose estimation and tracking in unconstrained, crowded scenes. It combines dense video annotations, person identities, and unknown person scale and location in evaluation.
- Public pose benchmarks span single-person single-frame, single-person video, multi-person single-frame, and multi-person video datasets.
- Earlier single-person datasets differ in scale and scope, with LSP focused on sports and FLIC on frontal upright upper-body pose estimation.
- PoseTrack evaluates multiple highly articulated people whose number, locations, and scales are unknown.
- Unlike single-frame multi-person benchmarks, PoseTrack provides video pose and identity annotations for highly articulated people in dense crowds.
- The dataset uses an unconstrained evaluation setup and adds ignore regions for crowds that are unreasonably complex to annotate.
- Existing multi-person video datasets contain only 60 or 30 videos, whereas PoseTrack offers larger scale, broader variety, and open evaluation.
- Compared with isolated-person video datasets and controlled indoor 3D-pose datasets, PoseTrack covers multiple people in dynamic crowded real-world scenes with 2D poses.
3. The PoseTrack Dataset and Challenge
PoseTrack constructs a densely annotated benchmark from crowded, motion-rich videos and evaluates both pose accuracy and temporal identity consistency. It provides multiple challenge tracks, standardized metrics, and held-out-test evaluation through an online server.
- Data collection: PoseTrack extends MPII Human Pose videos with 41–298 neighboring frames selected for crowded scenes and substantial motion, pose, and appearance variation.
- Data annotation: Annotations include ignore regions, person locations, identities, body poses, head bounding boxes, and track IDs.
- Dataset statistics: The dataset contains 550 video sequences split into 292 training, 50 validation, and 208 testing videos.
- Dataset statistics: Around 23,000 labeled frames contain 153,615 pose annotations, with validation and test sequences densely annotated every four frames.
- Challenges: The benchmark evaluates single-frame pose estimation, video pose estimation, and articulated tracking, including temporal consistency through identity switches.
- Evaluation: An online server evaluates held-out test predictions with identical ground truth and scripts to prevent overfitting and enable meaningful comparison.
- Evaluation: PCKh uses 50% of head length for adaptive joint-localization thresholds, while tracking reports MOTA, MOTP, Precision, and Recall.
4. Analysis of the State of the Art
The analysis evaluates baseline and submitted methods for pose estimation and articulated tracking, identifying two-stage designs, training-data effects, dataset difficulty, and evaluation limitations.
- 4.1. Baseline Methods: Two baseline methods adapt prior articulated-tracking approaches to PoseTrack’s larger and longer sequences, while the challenge evaluates submitted methods online.The baselines modify graph-partitioning methods for computational feasibility, and the challenge ranks top submissions for pose estimation and tracking.
- 4.2. Main Observations: All submissions use tracking by detection: frame-level person detection and pose estimation are followed by linking pose estimates over time.Top tracking methods use different pose estimators, including Mask-RCNN, PAF, and DeeperCut.
- 4.2. Main Observations: Most methods assemble body parts into poses per frame and match whole poses across frames, despite significant overlap between people.Only BUTD and the PoseTrack baseline track people at the level of individual body parts; ProTracker uses simple frame-to-frame matching.
- 4.2. Main Observations: 55.5 vs. 68.7 mAP: adding MPII-Pose to ArtTrack training considerably improves pose estimation, while PoseTrack and MPII-Pose remain complementary.The combined training set also outperforms MPII-Pose alone, 66.4 vs. 68.7 mAP.
- 4.2. Main Observations: Test sequences vary greatly in difficulty, with some showing good pose estimation but failed tracking; MOTA can become negative when false positives greatly exceed targets.The analysis uses per-sequence MOTA, articulation complexity, and mAP–MOTA correlation to examine these differences.
- 4.2. Main Observations: 38.1 to 53.4 MOTA: raising the part-detection threshold from τ = 0.1 to τ = 0.8 improves ArtTrack tracking, exposing sensitivity to score filtering.The authors propose confidence scores for predicted tracks as a future evaluation-protocol improvement.
5. Dataset Analysis
The analysis ranks test sequences by average MOTA to estimate difficulty and examines performance across difficult scenes and pose complexity. High-scoring sequences are contrasted with sequences challenged by crowding, proximity, rare poses, and camera motion.
- Average MOTA across seven methods ranks sequences by estimated difficulty for current pose-tracking approaches.The ranking is based on each sequence’s mean MOTA over the evaluated methods.
- Sequences with ArtTrack-baseline MOTA above 75% provide examples of successful tracking predictions.
- Negative-average-MOTA sequences expose challenges from crowds, extreme person proximity, rare poses, and strong camera motions.
- Pose complexity is measured by average deviation from the sequence’s mean pose and used to sort sequences for mAP analysis.
6. Conclusion
The paper introduces a larger, more diverse benchmark with online held-out evaluation for objective articulated-tracking comparisons. Its analysis finds that current methods handle easy sequences but remain challenged by fast camera motion and complex articulation.
- The benchmark is significantly larger and more diverse in data variability and complexity than existing pose-tracking benchmarks.
- An online evaluation server supports objective comparison on a held-out test set while measures limit dataset overfitting.
- Most existing methods combine people detection, single-person pose estimation, and simple association between neighboring frames.
- Current methods perform well on easy sequences with separated upright people but are severely challenged by fast camera motions and complex articulations.