PaperlayerView exact revision on arXiv

Source-linked AI summary

OpenPifPaf: Composite Fields for Semantic Keypoint Detection and Spatio-Temporal Association

Sven Kreiss, Lorenzo Bertoni, Alexandre Alahi

arXiv:2103.02440v2cs.CV

TL;DR

Semantic keypoint tracking must handle crowded, occluded scenes while meeting real-time demands, but prior approaches commonly separate pose detection from tracking. The paper jointly constructs spatio-temporal poses with Composite Fields, a Temporal Composite Association Field, and greedy decoding. It reports competitive or improved accuracy with substantially faster prediction across human pose benchmarks and generalization to car and animal poses.

  • Problem

    Real-time semantic keypoint detection, association, and tracking remain challenging in crowded and occluded scenes, while prior approaches treat pose detection and tracking as multi-stage processes.

  • Method

    The method uses Composite Fields, a Temporal Composite Association Field, and a greedy decoder to jointly form connected spatio-temporal poses from semantic keypoints.

  • Results

    The method achieves competitive or improved accuracy while predicting substantially faster on COCO, CrowdPose, and PoseTrack, and generalizes to car and animal poses.

  • Takeaways & Limitations

    A single framework can support real-time multi-person pose detection and tracking while extending semantic keypoint perception beyond humans.

Abstract

from arXiv · show

Many image-based perception tasks can be formulated as detecting, associating and tracking semantic keypoints, e.g., human body pose estimation and tracking. In this work, we present a general framework that jointly detects and forms spatio-temporal keypoint associations in a single stage, making this the first real-time pose detection and tracking algorithm. We present a generic neural network architecture that uses Composite Fields to detect and construct a spatio-temporal pose which is a single, connected graph whose nodes are the semantic keypoints (e.g., a person's body joints) in multiple frames. For the temporal associations, we introduce the Temporal Composite Association Field (TCAF) which requires an extended network architecture and training method beyond previous Composite Fields. Our experiments show competitive accuracy while being an order of magnitude faster on multiple publicly available datasets such as COCO, CrowdPose and the PoseTrack 2017 and 2018 datasets. We also show that our method generalizes to any class of semantic keypoints such as car and animal parts to provide a holistic perception framework that is well suited for urban mobility such as self-driving cars and delivery robots.

I. INTRODUCTION

The paper frames real-time semantic keypoint detection, association, and tracking as necessary for fine-grained perception in transportation settings. It introduces Composite Fields and a connected spatio-temporal pose representation to jointly detect and track poses efficiently.

  • Semantic keypoint tracking targets meaningful parts such as body joints and vehicle lights, unlike classical keypoints based on local image geometry.
  • Real-time pose tracking remains limited by accuracy and computational complexity, constraining transportation applications such as self-driving cars and delivery robots.
  • The task requires estimating and tracking multiple human, car, and animal poses despite occlusion, lighting, weather, and real-time response demands.
  • Earlier pose-tracking systems use graph matching or temporal flow fields but treat detection and tracking as separate stages.
  • The proposed method uses Composite Fields to form a spatio-temporal pose: one connected graph containing multiple single-frame poses and cross-frame connections.
  • A Temporal Composite Association Field and greedy decoder jointly detect and track poses in a single stage, with reported accuracy and speed advantages across several benchmarks.

B. Pose Tracking

Pose tracking is framed as forming associations among semantic keypoints across images, with Composite Fields providing continuous, detailed representations for detection and association. The method targets crowded, occluded scenes where top-down assumptions and multi-stage matching are problematic.

  • Pose tracking approaches can be top-down or bottom-up in both their pose and tracking components.
  • Composite Fields: Composite Fields jointly predict confidence, precise location, and size for semantic keypoints over feature-map locations.
  • Composite Intensity Fields: Composite Intensity Fields accumulate confidence-weighted, precisely regressed keypoint locations with predicted size to seed decoding and rescore associations.The accumulation uses a Gaussian kernel whose width is the predicted joint size.
  • Architecture: The architecture uses a shared ResNet or ShuffleNetV2 backbone, dataset-specific heads, and an additional TCAF head for tracking.
  • Composite Association Fields: Composite Association Fields extend association fields with joint-scale components and regress both source and target points for each connected joint pair.They represent confidence, two vectors, two spatial spreads, and two joint sizes.
  • Composite Association Fields: Unlike Part Affinity Fields, the Composite Association Field remains stable when two associated joints occupy the same pixel location.

IV. METHOD

The method is designed for efficient bottom-up keypoint detection, association, and tracking in crowded urban scenes. It uses a shared architecture with Composite Field heads and extends single-image pose estimation toward tracking, including suppression of self-hidden keypoints during training.

  • Motivation: The method targets urban and crowded scenes where overlapping bounding boxes make top-down methods difficult to apply.
  • Architecture: A shared ResNet or ShuffleNetV2 backbone uses shallow dataset-specific heads for joint intensities, associations, and optional temporal associations.
  • Tracking: Tracking is introduced as a direct extension of single-image pose estimation.
  • Training: Self-hidden keypoint suppression removes hidden keypoints near visible same-type keypoints from training associations.The authors report better pose reconstruction when one same-type keypoint hides another.

A. Single-Image Pose Estimation

Single-image pose estimation uses Composite Fields and a greedy frontier decoder to reconstruct poses robustly from keypoint detections and associations. Denser skeleton connections provide redundancy for occlusions and can merge fragments into a single pose.

  • Loss Functions for Composite Fields: The loss models confidence, localization, and scale, using masked feature cells and a Laplace localization loss with predicted uncertainty.Localization and scale supervision is restricted to a 4 × 4 window around each ground-truth keypoint, while confidence covers nearly the full image outside crowd regions.
  • Greedy Decoder with Frontier: The greedy decoder prioritizes possible associations using future joint scores that combine the current score with the best association confidence.It uses reverse matching and a weighted mixture of the best two associations to reduce jitter before committing the highest-priority connection.
  • Instance Score and Non-Maximum Suppression: Pose instances are ranked by a weighted mean of keypoint scores before keypoint-level NMS with a joint-size-dependent suppression radius.The three highest keypoint scores receive three times the weight of the others, and predictions are not refined after decoding.
  • Denser Pose Skeletons: Additional long-range associations add redundancy for occlusions, while the decoder prefers easier, more confident connections when available.The dense skeleton improves connectivity when sparse associations split a pose into separate instances.

B. Pose Tracking

Pose tracking extends single-image Composite Fields into a unified spatio-temporal representation with spatial and temporal associations decoded jointly. The approach forms tracks from previous poses or current-frame seeds using the same greedy frontier decoder.

  • Pose Tracking: The method jointly forms spatial and temporal associations, treating tracking as completion of a connected pose graph spanning multiple frames.The unified approach can also improve single-image poses by using additional temporal information.
  • Temporal Composite Association Field: The Temporal Composite Association Field processes independently created feature-map pairs to form temporal associations between frames.Its output structure matches a Composite Association Field while its input combines feature maps from different images.
  • Spatio-Temporal Poses: A spatio-temporal pose combines two frame-level poses with within-frame associations and cross-frame Temporal Composite Association Field connections.For a 17-joint pose, the representation contains 2×17 joints, 18 spatial associations, and 17 temporal associations.
  • From Spatio-Temporal Poses to Tracks: The decoder creates tracking poses from prior tracks or current-frame seeds and completes them with the same greedy frontier procedure.This bookkeeping connects pairwise temporal associations into image-sequence tracks.

V. EXPERIMENTS

Experiments evaluate the method on crowded-image pose estimation, standard person keypoint prediction, vehicle and animal keypoints, and video pose tracking.

  • Experiments: Experiments cover CrowdPose, COCO, ApolloCar3D, Animal-Pose, and PoseTrack 2017 and 2018.The evaluation spans single-image and video settings, human poses, and non-human semantic keypoints.

A. Datasets

The experiments use datasets spanning crowded human images, standard person keypoints, vehicle parts, animal parts, and annotated pose-tracking videos. Their scales and splits provide distinct settings for evaluating detection, generalization, and tracking.

  • CrowdPose: CrowdPose contains 10,000 training, 2,000 validation, and 8,000 test images with a uniform distribution of crowd density.The models are pretrained on ImageNet and then trained only on CrowdPose to avoid contamination from its composed sources.
  • COCO: COCO provides 56,599 diverse training images with person keypoint annotations, plus 5,000 validation and 20,288 test-dev images.Its private test set supports an active leaderboard through a protected challenge server.
  • ApolloCar3D: ApolloCar3D contains 5,277 4K driving images and more than 60K car instances annotated with 66 semantic keypoints.The evaluation uses a selected subset of 24 vehicle keypoints.
  • Animal Dataset: Animal-Pose contains 5,517 instances across more than 3,000 images, covering five animal categories and 20 keypoints.The categories are dog, cat, cow, horse, and sheep.
  • PoseTrack 2017 and 2018: PoseTrack 2018 contains 593 training, 170 validation, and 375 test scenes, while PoseTrack 2017 contains 292, 50, and 208 scenes respectively.Results are reported for both datasets because the 2018 leaderboard is frozen and recent methods commonly use the older 2017 benchmark.

B. Evaluation

The evaluation uses standard keypoint metrics for single-image pose estimation and MOTA for pose tracking. CrowdPose is additionally stratified by crowding, while PoseTrack comparisons use private test sets.

  • Single-image pose estimation: CrowdPose and COCO use object keypoint similarity to compute average precision and average recall from predicted and ground-truth keypoint overlaps.OKS assigns predictions to keypoints as a function of the person bounding-box area.
  • Single-image pose estimation: CrowdPose divides test images into easy, medium and hard subsets using crowd-index intervals [0, 0.1], [0.1, 0.8] and [0.8, 1.0].Because the crowd-index is uniformly distributed, most test images are medium.
  • Single-image pose estimation: COCO reports precision for medium instances measuring (32 px)^2 to (96 px)^2 and large instances exceeding (96 px)^2.Evaluators receive 17 keypoint locations and a total score for each pose, considering only the top 20 poses per image.
  • Pose tracking: PoseTrack evaluates human pose tracking primarily with MOTA, which combines false positives, false negatives and ID switches into one metric.The comparison covers leading methods submitted to the private PoseTrack 2017 and 2018 evaluation servers.

C. Implementation Details

The implementation combines convolutional backbones and shallow prediction heads with extensive image and tracking augmentations. Tracking is trained jointly on PoseTrack and COCO, with alternating batches and synthetic temporal variation.

  • Neural network configuration: Models use ResNet or ShuffleNetV2 backbones with the input max-pooling removed, stride 16 feature maps, and 2048 features per location.Custom ShuffleNetV2K16 and ShuffleNetV2K30 configurations are also used.
  • Neural network configuration: Each prediction head applies a 1×1 convolution followed by sub-pixel convolution, reducing total output stride to eight.Confidence uses sigmoid normalization, while joint-size scales use softplus to remain positive.
  • Augmentations: Training augments images with flipping, rescaling by r ∈[0.5, 2.0], cropping, padding, color jitter, grayscale conversion and JPEG artifacts.Color variation is 40% for brightness and saturation and 10% for hue.
  • Augmentations: Tracking augmentation varies image scale, cropping, orientation, camera shift, frame interval and negative-image replacement.Image pairs use intervals of 4, 8 or 12 frames, and 20% replace one image with a random image.
  • Optimization: Single-image models use SGD with Nesterov momentum, scheduled learning-rate decay and model averaging, with dataset-specific training schedules for CrowdPose and COCO.CrowdPose training runs for 300 epochs, while COCO training uses 150 epochs.
  • Tracking training: Tracking uses ShuffleNetV2k30 and jointly trains on PoseTrack and COCO, alternating datasets while retaining COCO single-image estimation as an auxiliary task.The datasets have similar but non-identical pose annotations.
  • Tracking training: Tracking training runs for 50 epochs, with 4994 batches per epoch and 55 minutes per epoch on two V100 GPUs.The optimizer uses synthetic camera shifts of up to 30px during training.

D. Results

OpenPifPaf performs competitively across crowded human pose estimation, COCO, pose tracking, vehicles and animals. It combines strong precision with real-time or online inference and uses temporal associations to complete difficult poses.

  • Crowded single-image pose estimation: 13.7 FPS is achieved on a single GTX1080Ti while exceeding previous top-performing methods across AP, AP0.50, AP0.75, APeasy, APmedium and APhard on CrowdPose.The qualitative examples come from the hard subset with crowd-index larger than 0.8.
  • COCO: On COCO, OpenPifPaf performs on par with the best existing bottom-up method using a single forward pass without flipping or multi-scale evaluation.Average processing time is 152 ms per image on a GTX1080Ti and 63 ms on a V100.
  • Pose tracking: PoseTrack results cover private 2017 and 2018 test sets, with single-pass online inference and tracking that can outperform methods with higher single-image AP.Reported MOTP, precision and recall are 84.5, 84.1 and 77.7 for 2017, and 84.9, 84.4 and 78.3 for 2018.
  • Pose tracking: Temporal connections allow the decoder to connect occluded or difficult limbs when spatial associations are unreliable.The greedy decoder selects spatial or temporal connections according to predicted association confidences.
  • Other semantic keypoints: 76.1% AP is achieved on car instances, while the method also predicts pedestrian, car and animal poses across the evaluated datasets.Table IV reports detection protocols for pedestrians, vehicles and animals.
  • Other semantic keypoints: 86.1% keypoint detection rate is achieved on ApolloCar3D, compared with 75.4% for CPM.A keypoint is counted as correct when its error is below 10 pixels.
  • Other semantic keypoints: 47.8% AP is achieved on animal instances, compared with 44.3% for WS-CDA.The paper attributes lower animal performance to the dataset's 4K training instances and leaves simultaneous human-animal training for future work.

E. Ablation Studies

Ablations expose trade-offs among memory, precision, decoding speed and input resolution. They also show that association-based keypoint discovery and temporal decoding are central to the reported performance.

  • Study design: Single-image studies compare backbones, keypoint criteria, decoder variants, memory-efficient decoding and input image size before tracking studies.The experiments use COCO validation for single images and PoseTrack validation for tracking.
  • Decoding assumptions: Forcing complete poses is common for COCO evaluation but is not used in most real-world settings.Without forcing completeness, decoding and total prediction time decrease by about 10ms.
  • Backbone: Larger backbones improve AP at a timing cost, with comparisons among ShuffleNetV2K16, ResNet50 and ShuffleNetV2K30.ShuffleNetV2K16 is the reference backbone.
  • Keypoint criterion and frontier decoder: Restricting associations to independently confident keypoints causes an 8.1% AP drop, while the frontier adds +0.3 AP for dense poses.OpenPifPaf can discover new keypoints during association from a strong seed keypoint.
  • Memory-efficient decoding: Removing both high-resolution seed and CAF rescoring reduces AP by 1.4% relative to the original configuration.CAF rescoring alone contributes about +1.0% AP, while seed rescoring changes AP by only 0.1 but substantially affects decoding time.
  • Tracking baselines: The tracking baselines first estimate single-frame poses and then match them frame to frame with Hungarian algorithms using Euclidean or OKS distances.These baselines provide comparisons against the joint detection-and-tracking approach.
  • Tracking input size: Reducing tracking input width to 513px lowers MOTA by 2.9 but increases speed by 82% to 22.2 FPS.The paper also reports a smaller 0.9 MOTA reduction for a +37% speed increase at an intermediate setting.

VI. CONCLUSIONS

The method provides fast bottom-up 2D pose tracking for humans and generalizes to car and animal pose estimation. It performs strongly on crowded and occluded scenes, outperforming prior methods on CrowdPose and PoseTrack2018 and matching state of the art on PoseTrack2017 at much higher speed.

  • The method demonstrates bottom-up tracking for 2D human poses in crowded and occluded scenes relevant to self-driving cars and social robots.
  • It outperforms previous state-of-the-art methods on CrowdPose and PoseTrack2018.
  • On PoseTrack2017, it matches state-of-the-art performance while running an order of magnitude faster.
  • The method generalizes from human poses to pose estimation for cars and animals.
  • The authors propose investigating shared backbone architectures to form a holistic perception framework for autonomous vehicles.
Loading 2103.02440v2…