PaperlayerView exact revision on arXiv

Source-linked AI summary

PoseTrack: Joint Multi-Person Pose Estimation and Tracking

Umar Iqbal, Anton Milan, Juergen Gall

arXiv:1611.07727v3cs.CV

TL;DR

The paper tackles joint pose estimation and identity association for an unknown number of people in unconstrained videos. It represents joint detections as a spatio-temporal graph and optimizes its partition with integer linear programming, while introducing a dataset and protocol for evaluation. Experiments quantify the approach on the new dataset, including the effect of temporal connection choices on tracking performance.

  • Problem

    Joint multi-person pose estimation and tracking in unconstrained videos requires estimating poses and associating people over time, a problem existing image-based methods do not directly solve.

  • Method

    The method jointly models pose estimation and tracking by partitioning a spatio-temporal graph of body-joint detections into plausible person trajectories using integer linear programming.

  • Results

    A MOTA of 28.2 is obtained when Neck and Shoulder detections are added to Head Tops for temporal connections, improving recall from 57.6% to 62.7%.

  • Takeaways & Limitations

    The dataset and unconstrained evaluation protocol enable quantitative benchmarking of multi-person pose estimation and tracking under occlusion and truncation.

Abstract

from arXiv · show

In this work, we introduce the challenging problem of joint multi-person pose estimation and tracking of an unknown number of persons in unconstrained videos. Existing methods for multi-person pose estimation in images cannot be applied directly to this problem, since it also requires to solve the problem of person association over time in addition to the pose estimation for each person. We therefore propose a novel method that jointly models multi-person pose estimation and tracking in a single formulation. To this end, we represent body joint detections in a video by a spatio-temporal graph and solve an integer linear program to partition the graph into sub-graphs that correspond to plausible body pose trajectories for each person. The proposed approach implicitly handles occlusion and truncation of persons. Since the problem has not been addressed quantitatively in the literature, we introduce a challenging "Multi-Person PoseTrack" dataset, and also propose a completely unconstrained evaluation protocol that does not make any assumptions about the scale, size, location or the number of persons. Finally, we evaluate the proposed approach and several baseline methods on our new dataset.

1. Introduction

The paper addresses joint pose estimation and tracking for multiple people in unconstrained videos, where pose, scale, motion, occlusion, truncation, and changing person counts complicate association over time. It proposes a spatio-temporal graph formulation and introduces a dataset and evaluation protocol for quantitative assessment.

  • Multi-person pose tracking in unconstrained videos must handle large pose and scale variations, fast motions, changing person counts, occlusion, and truncation.
  • The proposed method jointly solves pose estimation and person association by optimizing a densely connected spatio-temporal graph of joint candidates.
  • The Multi-Person PoseTrack dataset provides detailed, dense annotations for multiple people in each video to support quantitative evaluation.
  • The evaluation protocol accounts for occluded body joints and makes no assumptions about person scale, size, location, or number.

2. Related Work

Prior work largely addressed single-person video pose estimation or multi-person image pose estimation, while joint multi-person pose estimation and tracking remained quantitatively under-studied. The paper combines pose estimation and tracking in one objective and contributes a dataset designed to benchmark both.

  • Earlier multi-person image methods commonly detect people first and estimate each pose independently, which is limited when people are close together or occlude one another.
  • Single-person video methods use temporal smoothing or optical flow, but they are not directly applicable to videos with multiple potentially occluding people.
  • Prior approaches to joint multi-person pose estimation and tracking used multi-stage pipelines rather than reasoning jointly about pose and tracking.
  • The paper proposes a joint objective that simultaneously estimates multiple poses and tracks people over time.
  • Multi-Person PoseTrack provides pose annotations and unique person IDs, enabling separate evaluation of pose estimation and multi-person tracking.

3. Multi-Person Pose Tracking

The method jointly estimates and tracks multiple people by partitioning a spatio-temporal graph of joint detections with integer linear programming. Spatial, temporal, and spatio-temporal constraints enforce valid, consistent pose trajectories while accommodating short-term occlusion and missing detections.

  • 3.2. Graph Partitioning: The ILP selects nodes and edges whose graph partitions correspond to tracked poses for individual persons.Binary vectors indicate whether detections, spatial edges, and temporal edges are removed; minimizing the remaining cost yields the partitioning.
  • 3.1. Spatio-Temporal Graph: Joint detections become graph nodes, with spatial edges within frames and temporal edges linking same-joint candidates across frames.Temporal links extend beyond neighboring frames to connect detections up to τ frames apart.
  • 3.2. Graph Partitioning: Unary and pairwise costs use detection confidence, same-person spatial probability, and same-person temporal probability to score graph elements.High-confidence detections reduce the objective because their unary cost is negative when pd>0.5.
  • 3.2. Graph Partitioning: Transitivity constraints prevent inconsistent spatial and temporal associations, including multiple instances of one joint type within a pose.Spatio-temporal constraints additionally enforce consistency between associations across two frames.
  • 3.3. Optimization: The solver processes sequences in batches of k = 31 frames and removes partitions shorter than 7 frames or averaging fewer than 6 nodes per frame.Previously selected nodes and edges are fixed when processing subsequent batches.
  • 3.4. Potentials: The detector runs over four image-pyramid scales because the method does not assume person scale at test time.The scales are γ ∈ {0.6, 0.9, 1.2, 1.5}; highly overlapping lower-confidence boxes are removed using an IoU threshold over 0.7.

4. The Multi-Person PoseTrack Dataset

The paper introduces Multi-Person PoseTrack, a video dataset designed for quantitative evaluation of multi-person pose estimation and tracking, together with an evaluation protocol covering occluded joints and realistic variation.

  • Dataset contribution: Multi-Person PoseTrack was introduced because no dataset previously provided annotations for quantitative evaluation of joint multi-person pose estimation and tracking.The dataset provides detailed and dense annotations for multiple persons in each video.
  • Dataset composition: The dataset contains videos selected for multiple-person interactions and substantial variation in body-pose appearance.Selections include person-person or person-object interactions and challenging pose variation.
  • Dataset composition: PoseTrack comprises 60 videos with 41–151 frames each, 2–16 persons per video, over 16,000 annotated poses, and person heights from 100 to 1200 pixels.Training and testing sets contain equal numbers of videos.
  • Annotations: Each person has 14 annotated body joints, a head rectangle for scale estimation, a persistent video identity, and occlusion flags for body joints.Truncated joints outside image borders are not annotated, so the number of joints per person varies.
  • Evaluation protocol: The evaluation protocol uses PCKh with a 20% head-box-diagonal threshold and combines pose metrics with joint-trajectory tracking metrics.Tracking evaluation includes CLEAR MOT measures, while frame-wise pose accuracy uses mean average precision.

5. Experiments

Experiments evaluate joint multi-person pose estimation and tracking on Multi-Person PoseTrack using ablations, baselines, and runtime analysis. The results show that temporal connections and constraint types materially affect performance, while the proposed method outperforms the compared approaches.

  • Temporal connection ablations: Adding Neck and Shoulder temporal connections improves tracking MOTA from 27.2 to 28.2 and recall from 57.6% to 62.7%.Using additional joints also improves MT and reduces ID switches and fragments, whereas adding hip connections slightly decreases performance.
  • Temporal connection ablations: Temporal edges spanning at most three frames improve performance slightly, while extending them to five frames worsens it; the experiments therefore use τ = 3.The selected setting balances tracking performance with the tested temporal range.
  • Constraint ablations: All tested constraint types contribute to performance, with spatial transitivity most crucial because it enforces valid poses without duplicate joint types per person.Removing temporal transitivity or spatio-temporal constraints reduces recall and causes a drop in MOTA.
  • Frame-wise pose estimation: For frame-wise pose estimation, HT-only connections achieve 34.3% mAP, increasing to 37.9% with Neck and Shoulder connections.Adding hip connections improves mAP slightly, unlike the tracking result; τ = 3 and the tested constraints follow similar performance trends.
  • Baseline comparisons: The proposed method outperforms the baseline approaches on multi-person pose tracking and does not require a separate person detector.It jointly detects across scales and associates persons across frames.
  • Runtime analysis: The experiments also report mean and median graph-construction and optimization runtimes, together with graph size, for k = 31-frame sequences.These measurements use τ = 3 and HT:N:S on a single-threaded 3.3GHz CPU.
Loading 1611.07727v3…