Multi-person Articulated Tracking with Spatial and Temporal Embeddings

1. Introduction

• The authors identify varied interacting people, frequent occlusion, fast motion, pose and scale changes, and camera movement as challenges for multi-person tracking.They motivate the task by its relevance to applications such as video understanding and action recognition.Source: §1 · p. 1
• Appearance and geometric cues are complementary: SIE may split one pose, while KE may group distant body parts; the paper combines them for spatial grouping.KE better preserves within-person consistency but can struggle to separate instances without geometric constraints.Source: Figure 1 · p. 1, §1 · p. 2
• For temporal grouping, HE encodes holistic appearance and TIE encodes temporal geometric consistency, addressing different tracking errors under camera motion and pose change.The cited figure illustrates TIE's vulnerability to camera zooming and movement and HE's vulnerability to pose changes.Source: Figure 1 · p. 1, §1 · p. 2
• PGG makes detection and grouping differentiable and end-to-end trainable, directly supervising grouping and back-propagating its loss to low-level features.The paper says this directs learning toward mistakenly grouped body parts and helps produce accurate embeddings.Source: §1 · p. 2
• SpatialNet and TemporalNet share feature-extraction layers while handling, respectively, single-frame body-part detection and grouping and temporal grouping of people into trajectories.The framework presents pose estimation and tracking as part of one system.Source: §1 · p. 2

2.1. Multi-person Pose Estimation in Images

• Bottom-up methods detect body-part candidates and group them, but prior pipelines commonly use post-processing without back-propagating grouping errors.Top-down methods rely on person detectors, can struggle with clutter and occlusion, and have inference time proportional to the number of people.Source: §2.1 · pp. 2–3
• Unlike KE's indirect grouping supervision, PGG directly learns grouping end to end; the paper also contrasts its grouping losses with clustering or refinement used for SIE [22] [23].The paper argues that a low KE pairwise loss can still yield incorrect groupings.Source: §2.1 · p. 3

2.2. Multi-person Pose Tracking

• Offline tracking can use future frames or long-range consistency but has high computational complexity; online approaches associate current detections with existing trajectories.The cited methods include temporal-window processing and graph-partitioning formulations.Source: §2.2 · p. 3
• The paper characterizes online trackers as relying on part-level geometry while ignoring global appearance, then proposes HE appearance and TIE temporal coherence for improved tracking.Its approach uses online pose tracking and bipartite graph matching [6].Source: §2.2 · p. 3

3. Method

• SpatialNet processes one frame to detect body parts and spatially group them into pose proposals.It predicts heatmaps, KE, SIE, and geometric-ordinal maps; PGG refines KE and SIE before greedy decoding.Source: §3 · p. 3
• TemporalNet shares low-level features with SpatialNet and extracts HE appearance and TIE geometric embeddings for each pose proposal.The two embeddings provide pairwise potentials for association.Source: §3 · p. 3
• The framework solves bipartite graph matching between current-frame pose proposals and already tracked poses to generate trajectories.TemporalNet performs this human-level grouping online.Source: §3 · p. 3

3.1. SpatialNet: Part-level Spatial Grouping

• Body-part detection uses Gaussian confidence heatmaps and a weighted ℓ2 loss against ground-truth heatmaps.Source: §3.1 · pp. 3–4

3.1.1 Keypoint Embedding (KE) with auxiliary tasks

• Six auxiliary ordinal tasks impose left-right, top-bottom, and near-far ordering on KE maps to facilitate training and improve interpretation.The auxiliary training replaces the push loss with ordinal loss while retaining the pull loss; instances are ordered by centroid location or head size.Source: §3.1.1 · p. 4

3.1.2 Spatial Instance Embedding (SIE)

• SIE complements KE's difficulty separating instances geometrically by encoding each pixel with its person's center location, decoded from spatial vector fields.The ground-truth spatial vector is the displacement from the person center to each body part.Source: §3.1.2 · p. 4

3.2. Pose-Guided Grouping (PGG) Module

• PGG reformulates grouping as a differentiable module, allowing direct supervision of grouping results and end-to-end training.Grouping loss is back-propagated to low-level feature learning stages.Source: §3.2 · pp. 4–5
• Figure 3 reports that PGG produces more compact and accurate KE and SIE embeddings.Source: Figure 3 · p. 5
• A pose mask excludes irrelevant background regions from grouping, addressing noisy embeddings and reducing memory cost.The mask thresholds the maximum body-part confidence at τ = 0.2; selected pixels are far fewer than all image pixels.Source: §3.2 · pp. 4–5
• PGG refines concatenated KE and SIE embeddings through recurrent mean-shift updates using Gaussian affinities and normalized weighted means.The selected embeddings initialize X, and pairwise pull/push losses are applied over all grouping iterations during training.Source: §3.2 · p. 5

3.3. TemporalNet: Human Temporal Grouping

• TemporalNet combines human-level appearance features with temporally coherent geometric information to associate poses across adjacent frames.It groups current-frame proposals with poses already tracked in the previous frame.Source: §3.3 · p. 5

3.3.1 Human Embedding (HE)

• Human Embedding (HE) pools ROI-Align features from predicted-pose bounding boxes and uses triplet loss to separate identities.The region-specific features are mapped to a 3072-dimensional embedding.Source: §3.3.1 · p. 5

3.3.2 Temporal Instance Embedding (TIE)

• Temporal Instance Embedding (TIE) represents cross-frame keypoint-to-human-center displacement with forward and backward temporal vector fields.The fields are refined independently by Pose-Guided Grouping using pose masks and recurrent mean-shift grouping.Source: §3.3.2 · p. 6

3.3.3 Pose Tracking

• TemporalNet matches current- and previous-frame poses using appearance and temporal-smoothness potentials, then solves a bipartite matching problem to form trajectories.The matching uses the Munkres algorithm.Source: §3.3.3 · p. 6

3.4. Implementation Details

• SpatialNet uses a four-stage stacked-hourglass backbone and is first trained without Pose-Guided Grouping, then fine-tuned with it.Its initial training combines detection, KE, auxiliary, and SIE losses.Source: §3.4 · p. 7
• One Pose-Guided Grouping iteration was sufficient in practice; additional iterations yielded little gain.Source: §3.4 · p. 7
• TemporalNet uses a one-stage hourglass and is trained for 40 epochs while SpatialNet is fixed.Training pairs are sampled from within a five-frame temporal window.Source: §3.4 · p. 7

4.1. Datasets and Evaluation

• The ICCV’17 PoseTrack benchmark contains 250 training clips and 50 validation sequences; pose estimation is evaluated with AP and tracking with MOTA.The comparisons also use additional MPII-Pose images for training.Source: §4.1 · p. 7, §4.2 · p. 7

4.2. Comparisons with the State-of-the-art Methods

• Our method improves MOTA by 16.6% over ProTracker [8] and 6.4% over FlowTrack, with comparable single-frame pose accuracy to FlowTrack.The comparison is on the ICCV’17 PoseTrack validation set; Table 2 reports multi-person pose-tracking results.Source: Table 2 · p. 7, §4.2 · p. 7

4.4. Runtime Analysis

• Using instance-agnostic pose masks, PGG reduces memory consumption to about 1% of GBMS, while runtime comparisons show SpatialNet's limited computational-complexity increase over.Runtime analysis measures CNN processing on one GTX-1060 GPU and tracking on a single 2.4GHz CPU core.Source: Figure 6 · p. 7, §4.4 · p. 8

4.3. Ablation Study

• Combining KE and SIE yields 74.0% mAP versus 70.9%/71.3%, and adding PGG raises the combined model to 77.0% mAP.PGG improves pose-estimation accuracy by 2.3% for KE, 3.8% for SIE, and 2.7% for their combination.Source: §4.3 · pp. 7–8, Table 3 · p. 8
• TemporalNet improves tracking over the KE/SIE baseline, and spatial-temporal PGG raises performance from 69.2% to 71.8% MOTA.The reported gains are attributed to combining holistic appearance features in HE with temporal smoothness in TIE.Source: §4.3 · p. 8, Table 4 · p. 8
• Multi-scale SpatialNet inference gains 3% AP on MS-COCO over single-scale inference, using scales {0.5, 1, 1.5, 2}.Reported results use no model ensembling or pose refinement.Source: §4.3 · p. 8, Table 5 · p. 8

• SpatialNet detects body parts and groups them spatially, while TemporalNet groups human instances across frames; the framework extends KE and SIE with HE and TIE and uses PGG for end-to-end learning.HE provides appearance features and TIE provides temporally consistent geometric features for online tracking.Source: §5 · p. 8