Source-linked AI summary
ArtTrack: Articulated Multi-person Tracking in the Wild
Eldar Insafutdinov, Mykhaylo Andriluka, Leonid Pishchulin, Siyu Tang, Evgeny Levinkov, Bjoern Andres, Bernt Schiele
TL;DR
ArtTrack addresses articulated multi-person pose tracking in unconstrained monocular videos, where occlusion, truncation, fast motion, and unknown subject counts complicate joint association. It combines feed-forward body-part proposals with sparse graph-based spatio-temporal grouping, achieving state-of-the-art performance with substantially faster inference and gains from temporal information in crowded scenes.
Problem
Articulated multi-person pose tracking must identify people, localize their joints, and associate those joints over time despite occlusion, truncation, fast motion, and appearance variability.
Method
The approach generates body-part proposals with convolutional detectors and jointly groups them within and across frames using sparse graph partitioning, temporal reasoning, and inter-person exclusion constraints.
Results
The model improves over state-of-the-art while being substantially faster, with a top-down/bottom-up variant running 24x faster than a fully connected model while exceeding its performance.
Takeaways & Limitations
Temporal grouping propagates strong observations through video, while the sparse formulation and feed-forward association enable efficient articulated tracking in crowded scenes.
Takeaways & Limitations
The current top-down/bottom-up implementation computes person-conditioned proposals sequentially for each person and processes the whole image.
Abstract
from arXiv · showhide
In this paper we propose an approach for articulated tracking of multiple people in unconstrained videos. Our starting point is a model that resembles existing architectures for single-frame pose estimation but is substantially faster. We achieve this in two ways: (1) by simplifying and sparsifying the body-part relationship graph and leveraging recent methods for faster inference, and (2) by offloading a substantial share of computation onto a feed-forward convolutional architecture that is able to detect and associate body joints of the same person even in clutter. We use this model to generate proposals for body joint locations and formulate articulated tracking as spatio-temporal grouping of such proposals. This allows to jointly solve the association problem for all people in the scene by propagating evidence from strong detections through time and enforcing constraints that each proposal can be assigned to one person only. We report results on a public MPII Human Pose benchmark and on a new MPII Video Pose dataset of image sequences with multiple people. We demonstrate that our model achieves state-of-the-art results while using only a fraction of time and is able to leverage temporal information to improve state-of-the-art for crowded scenes.
1. Introduction
ArtTrack targets articulated multi-person pose tracking in difficult monocular videos, combining efficient sparse modeling with learned body-part association and temporal grouping. It addresses occlusion, truncation, fast motion, and unknown numbers of people while introducing a crowded-scene evaluation dataset.
- The task requires counting people, localizing each person’s joints, and associating those joints over time in monocular video.
- Occlusion, image-boundary truncation, fast motion, and appearance variability can leave only subsets of joints visible.The paper motivates reasoning beyond geometric joint arrangements and incorporating image cues across multiple people.
- The tracking model groups bottom-up part detections within frames and across time while enforcing inter-person exclusion and propagating strong observations to neighboring frames.
- A sparse body-part graph and person-conditioned proposals reduce inference complexity, with feed-forward convolutional reasoning handling substantial body-part association.
- A top-down/bottom-up model exceeds a fully connected model while being 24x faster at inference time.
- The paper contributes a challenging MPII Video Pose dataset for articulated joint tracking with multiple overlapping people in crowded realistic environments.
2. Tracking by Spatio-temporal Grouping
The tracker represents joint detections and candidate associations as a graph, then selects a constrained subgraph whose connected components define person tracks. Node and edge variables, feature-based costs, and feasibility constraints jointly determine detections and identities.
- Each body-joint proposal records its video frame, image location, detection probability, and joint type.
- Graph nodes are joint detections, while edges connect detection pairs that could belong to the same target.
- The algorithm filters redundant or erroneous detections and retains edges linking detections assigned to the same target in an output subgraph.
- Binary node and edge variables specify the selected subgraph, with each track represented by a connected component.
- Feasibility constraints couple node and edge selections, including requiring both endpoints of every selected edge to be selected.
- The model estimates node and edge costs from image features and minimizes negative log-likelihood over feasible assignments.
- The resulting constraints and objective form a minimum cost subgraph multicut problem that enforces consistent connected components and person identities.
3. Articulated Multi-person Tracking
The model combines sparse within-frame body-part graphs with person-conditioned proposals and temporal grouping to track multiple articulated people. Its graph encodes part relationships, inter-person exclusion, and temporal continuity while convolutional predictions reduce association complexity.
- Graph formulation: The subgraph multicut formulation uses approximate local-search inference because the underlying optimization problem is NP-hard.Each track corresponds to a connected component, with constraints enforcing consistent assignments and unambiguous person identities.
- Graph formulation: The tracking graph couples body-part proposals within frames and across neighboring frames using cross-type, same-type, and temporal edges.Sparse connectivity reduces potentially uninformative dependencies while retaining temporal continuity and inter-person exclusion.
- Model variants: The BU and TD/BU variants share same-type and temporal terms but differ in proposal generation, cross-type terms, and graph connectivity.TD/BU conditions convolutional body-part predictions on person locations, addressing part association within the proposal mechanism.
- Bottom-Up model: Sparse connectivity omits edges between part types carrying little information about each other’s image locations, producing a simpler and faster model.The model can use either fully connected within-image graphs or sparse graphs over selected part types.
- Top-Down/Bottom-Up model: TD/BU uses person root detections and a convolutional network to generate conditional distributions for body-part proposals, while must-not-link constraints restrict proposals to one person.Proposals not connected to any root node are excluded from the final solution.
- Temporal model: Temporal edges connect same-type proposals in adjacent frames and receive logistic-regression costs based on motion and appearance features.The feature set combines Euclidean distance, SIFT descriptors, and DeepMatching correspondences, which are described as complementary across motion conditions.
4. Experiments
The experiments evaluate single-frame and video-based pose and tracking models on MPII benchmarks and the new MPII Video Pose dataset. The results show that sparse, temporally informed models improve accuracy while reducing graph-inference time, though long-term occlusions remain problematic.
- Datasets and evaluation: The MPII Video Pose dataset contains 28 challenging sequences with over 2,000 annotated poses, and evaluation uses AP, runtime, and MOTA.The sequences feature crowded scenes, articulated people, and dynamic activities.
- Single-frame models: 70.5% AP is achieved by BU-full, label at 3.06 s/f, while BU-full reduces runtime to 0.38 s/f and improves AP to 71.9%.Pre-labeling detection candidates based on detection score reduces graph variables and improves performance.
- Single-frame models: 70.6% AP is obtained by BU-sparse versus 71.9% AP for BU-full, while graph-partitioning runtime falls from 0.38 to 0.22 s/f.Sparsifying the graph reduces complexity at a performance cost.
- Single-frame models: 87.7% AP is reported for TD/BU on WAF versus 82.0% AP for the best published result, with +6.4% AP improvements on wrists and elbows.The comparison uses the AP evaluation measure across body parts.
- Multi-frame models: 73.1% AP is achieved by BU-sparse+temporal versus 71.6% AP for BU-sparse, with gains of +3.9% AP on ankles and +2.6% AP on wrists.Temporal reasoning helps localize ambiguous body parts when multiple people are close together.
- Multi-frame models: The tracker fails when a person’s head is occluded for multiple frames because it lacks long-range connectivity between target hypotheses.The authors leave handling long-term occlusions for future work.
5. Conclusion
The paper introduces an efficient approach that jointly groups body-part proposals within frames and across time using graph partitioning. It improves over state of the art while being substantially faster, with pose-estimation results also reported on WAF.
- Conclusion: The approach jointly groups body-part proposals within each video frame and across time through graph partitioning.Recent local search techniques support efficient inference.
- Conclusion: The model is designed to improve state-of-the-art performance while being substantially faster than related work.The conclusion characterizes the approach as both efficient and effective for monocular video.
A. Additional Results on the MPII Multi-Person Dataset
TD/BU handles highly articulated, overlapping people more reliably than BU-full in challenging single-frame scenes by explicitly modeling person identity and using feed-forward association reasoning.
- TD/BU resolves substantial overlap and high articulation cases that cause BU-full to fail.BU-full works when people are separated but fails when they overlap strongly or exhibit high articulation.
- Person-conditioned top-down/bottom-up reasoning offloads body-part association to a feed-forward convolutional architecture.This explicit identity modeling helps distinguish parts belonging to different people in cluttered scenes.
- TD/BU can recover lower limbs through partial occlusion in several challenging examples.The qualitative examples include people in the back whose lower limbs are partly occluded.
- TD/BU occasionally assembles body parts in kinematically implausible configurations.The described failure occurs because the model does not explicitly model geometric body-part relationships.
B. Results on the We Are Family dataset
On the We Are Family dataset, TD/BU surpasses the best published results across body parts, including larger gains on articulated joints such as wrists and elbows.
- 87.7 vs 82.0% AP: TD/BU outperforms the best published results across all body parts.The comparison uses the evaluation protocol from and reports AP on the We Are Family dataset.
- +6.4% AP on wrists and elbows: TD/BU improves especially on these articulated body parts.
- The authors attribute TD/BU’s gains to better learned part associations than explicit geometric pairwise modeling.The comparison is made against methods such as that explicitly model geometric pairwise relations.
C. Evaluation of temporal features.
Temporal features improve BU-sparse tracking on the Multi-Person Video dataset, with geometry, appearance, and SIFT-based cues contributing complementary gains.
- 73.1% AP: BU-sparse+temporal improves 1.5% AP over single-frame BU-sparse.The combined temporal model uses geometry-, appearance-, and SIFT-distance features.
- 72.1% AP: geometry-based det-distance features improve over the 71.6% AP single-frame baseline.These features enable propagation of information from neighboring frames.
- Appearance-based deepmatch features help link the same body part across time, especially during fast motion.They improve performance beyond geometry-only temporal features when det-distance may fail.
- 72.5% AP: combining geometry and appearance features demonstrates their complementarity.
- SIFT-distance features raise performance to 73.1% AP by handling sudden background and body-part orientation changes.