Source-linked AI summary
Learning to Detect and Track Visible and Occluded Body Joints in a Virtual World
Matteo Fabbri, Fabio Lanzi, Simone Calderara, Andrea Palazzi, Roberto Vezzani, Rita Cucchiara
TL;DR
Multi-person tracking needs precise detection and temporal continuity when clutter produces occluded targets. The paper proposes an end-to-end network that detects and associates body parts across short temporal spans, supported by a large synthetic dataset with occlusion and tracking annotations. The model transfers positively to real tracking scenarios when image resolution and sharpness are high enough, producing tracklets for later association or re-identification.
Problem
Multi-person tracking lacks precise detection and temporal continuity for occluded targets, while surveillance data with tracking, body-part, and occlusion annotations are limited.
Method
The paper proposes THOPA-net, an end-to-end architecture with visible- and occluded-joint heatmaps, part affinity fields, and temporal affinity fields for short-term body-part association.
Results
The model transfers positively from synthetic training to real tracking scenarios when image resolution and sharpness are high enough.
Takeaways & Limitations
The dataset and architecture provide a starting point for treating surveillance tracking as one process combining detection and temporal association, with tracklets usable for later association or re-identification.
Abstract
from arXiv · showhide
Multi-People Tracking in an open-world setting requires a special effort in precise detection. Moreover, temporal continuity in the detection phase gains more importance when scene cluttering introduces the challenging problems of occluded targets. For the purpose, we propose a deep network architecture that jointly extracts people body parts and associates them across short temporal spans. Our model explicitly deals with occluded body parts, by hallucinating plausible solutions of not visible joints. We propose a new end-to-end architecture composed by four branches (visible heatmaps, occluded heatmaps, part affinity fields and temporal affinity fields) fed by a time linker feature extractor. To overcome the lack of surveillance data with tracking, body part and occlusion annotations we created the vastest Computer Graphics dataset for people tracking in urban scenarios by exploiting a photorealistic videogame. It is up to now the vastest dataset (about 500.000 frames, almost 10 million body poses) of human body parts for people tracking in urban scenarios. Our architecture trained on virtual data exhibits good generalization capabilities also on public real tracking benchmarks, when image resolution and sharpness are high enough, producing reliable tracklets useful for further batch data association or re-id modules.
1 Introduction
The paper treats multi-person detection and short-term tracking as an integrated problem, particularly when clutter causes occlusions. It introduces THOPA-net, which jointly localizes and associates body parts while hallucinating plausible occluded joints, and reports transfer from synthetic training to real tracking scenarios.
- 1 Introduction: The approach frames detection and short-term tracking as two parts of a single end-to-end problem.The authors propose integration as a proxy for more complex tracking or re-identification association methods.
- 1 Introduction: THOPA-net jointly locates people body parts and associates them across short temporal spans.The architecture integrates a temporal coherency module with bottom-up human pose estimation.
- 1 Introduction: Occluded body parts are handled explicitly by hallucinating feasible solutions for joints that are not visible.The method exploits deep networks’ capacity to predict plausible occluded-part configurations.
- 1 Introduction: Experiments report encouraging precision in crowded scenes and positive transfer to real multi-person tracking datasets.The reported transfer is stronger when image quality and resolution are high, while realism or shape details matter less than image quality.
2 Related Works
Prior work separately addressed multi-person pose estimation, temporal pose refinement, and online tracking, leaving overlapping people and unified pose-tracking annotations as central challenges. The paper responds with a dataset strategy that combines precise pose and tracking annotations in realistic urban scenarios.
- 2 Related Works: Multi-person pose estimation extends single-person methods to images containing varying numbers of people who may occlude one another.The related-work discussion identifies occlusion between people as a key reason single-person techniques are insufficient.
- 2 Related Works: Video pose-estimation methods use temporal smoothing or optical flow but neglect multiple overlapping people.This distinguishes temporal pose refinement from the paper’s multi-person setting.
- 2 Related Works: Online multi-target tracking uses multiple cues beyond appearance, including detectors, deep appearance features, long-term temporal dependencies, and spatial-temporal attention.The cited approaches illustrate the range of existing online tracking mechanisms.
- 2 Related Works: The proposed dataset combines precise pose and tracking annotations in realistic urban scenarios using automatically computed ground truth from photorealistic computer-graphics environments.This strategy is presented as combining complementary annotation types for pose estimation and tracking.
3 JTA Dataset
JTA is a large synthetic dataset for pedestrian pose estimation and tracking in varied urban scenes. It provides precise visible, occluded, tracking, and 2D/3D body-part annotations generated from a photorealistic game environment.
- 3 JTA Dataset: JTA was collected from Grand Theft Auto V to provide pedestrian pose-estimation and tracking data in urban scenarios.The videos vary body poses, illumination conditions, viewpoints, and urban settings.
- 3 JTA Dataset: Each clip includes annotations for visible and occluded body parts, people tracking, and 2D and 3D coordinates.The dataset records both image-space and virtual-world spatial information.
- 3 JTA Dataset: The dataset uses manually directed game scenes and a synchronized modification of the game engine to generate natural pedestrian flows and actions.The setup recreates behaviors such as sitting, running, chatting, and talking on the phone.
- 3 JTA Dataset: JTA’s occlusion labels distinguish joints hidden by objects or other pedestrians from joints self-occluded by the same person.Occlusion annotations are captured through the game renderer.
4 THOPA-net
THOPA-net combines single-frame body-part estimation with temporal linking to jointly predict visible and occluded joints, spatial affinities, and short-term temporal associations. Its multi-stage architecture uses branch-specific predictions and temporal affinity fields to construct multi-person skeletons across frames.
- Architecture: THOPA-net extends a single-image pose model with an occluded-joint branch and a temporal network for joint pose estimation and short-term tracking.The single-image model predicts visible heatmaps, occluded heatmaps, and PAFs; the temporal model adds TAFs.
- Single Image Pose Prediction: The single-image CNN uses three branches for visible-part heatmaps, occluded-part heatmaps, and PAFs that link body parts within each frame.Each branch iteratively refines predictions through multiple stages with intermediate supervision.
- Single Image Pose Prediction: The branch losses are stage-wise masked SSE objectives, with the mask preventing occluded joints from being penalized in the visible branch.The generic branch output represents visible heatmaps, occluded heatmaps, or affinity fields.
- Temporal Consistency Branch: The temporal model processes N RGB frames with shared feature extraction and 3D convolutions, producing TAFs alongside heatmaps and PAFs.Temporal pooling operates along the temporal dimension, and temporal information flows to all branches after the first stage.
- Training Procedure: TAFs are 2D vector fields linking the same body-part types across adjacent frames, providing expected previous-frame locations for detection boosting and temporal association.PAFs connect different body-part types within a frame, whereas TAFs connect corresponding parts between frames.
- Spatio-Temporal Multi-Person Joints Association: Skeleton construction scores candidate limbs using both spatial PAF line integrals and temporal TAF consistency before maximizing anatomically plausible connections.The method first obtains discrete joint candidates from non-maximum-suppressed heatmaps and then optimizes allowed connections.
5 Experiments
Experiments evaluate joint detection and short-term tracking on the virtual JTA dataset, then assess transfer to real MOT-16 and PoseTrack data. Results support virtual-data training and show that image quality and resolution influence real-data generalization.
- Experiments on JTA evaluate both people-joint extraction and temporal association, while real-data experiments assess transfer and the need for fine-tuning.
- Detection experiment: The detection experiment measures joint localization and bounding-box accuracy, including PCKh, precision, recall, and F1-score at 50% intersection over union.
- Detection experiment: Ablation results show that adding the occlusion-estimation branch improves both joint-location accuracy and detection performance.
- Tracking experiment: The JTA tracking experiment processes one-second clips and associates targets frame by frame using temporal affinity field scores, comparing against Hungarian lower-midpoint baselines.
- MOT-16: On MOT-16, the fine-tuned network is compared with online trackers and ranks near the top by MOTA, while its intermediate tracklets retain high IDS and fragmentation.
- PoseTrack: On PoseTrack, THOPA-net produces satisfying MOTA and mAP results using solely synthetic training data, supporting transfer to real urban scenarios.
6 Conclusion
The paper combines a massive synthetic dataset with a network that handles occlusion and temporal association for short-term body-part tracking. Results indicate adaptation to real scenarios when image resolution and sharpness are sufficiently high.
- The JTA dataset provides realistic urban scenarios with precise visible and occluded-joint annotations for human pose estimation and tracking.
- The proposed network jointly locates body parts and associates them across short temporal spans by integrating occlusion handling and temporal coherency.
- Synthetic-data training transfers to real-world scenarios when image resolution and sharpness are high enough.
- The dataset and architecture provide reliable tracklets that can serve as input to batch optimization and re-identification techniques.
Supplementary Material
The supplementary material identifies an author, an institutional affiliation, and an email contact in Italy.
- The author list includes Rita Cucchiara.
- The listed institution is the Department of Engineering “Enzo Ferrari” at the University of Modena and Reggio Emilia, Italy.
- The supplementary contact uses the unimore.it email domain.
1 Per Joint Results on JTA
The JTA evaluation reports mean average precision separately for each body joint.
- JTA results are reported as mean average precision (mAP) per body joint.
2 Per Sequence Results on MOT-16
The MOT-16 evaluation reports results per sequence and compares the model with and without fine-tuning on real data. The authors report stronger transfer on high-resolution, sharp sequences and limited generalization as image quality decreases.
- MOT-16 results are reported separately for each sequence.
- The experiment compares MOTA with and without fine-tuning on real data.Figure 1 presents the comparison between the two training settings.
- High-resolution, sharp MOT16-09 and MOT16-11 sequences support transfer from virtual-world features to real images.
- Generalization is limited as image quality decreases.
- Limited fine-tuning enables adaptation using a self-annotated dataset with potential errors and inaccuracies.
3 JTA Dataset
The JTA Dataset includes statistics on camera distances and poses per frame, alongside varied examples spanning viewpoints, crowd sizes, illumination, and scenarios.
- Example images exhibit variety in viewpoints, number of people, illuminations, and scenarios.
- Dataset statistics describe annotated pose counts by camera distance in meters.
- Dataset statistics also relate frame counts to the number of annotated poses per frame.