PaperlayerView exact revision on arXiv

Source-linked AI summary

UniPose: Unified Human Pose Estimation in Single Images and Videos

Bruno Artacho, Andreas Savakis

arXiv:2001.08095v1cs.CV

TL;DR

Pose estimation must handle complex body configurations, occlusion, and the limited generalization of predefined anchor poses. UniPose uses a single-stage contextual architecture with Waterfall Atrous Spatial Pooling, extending it to UniPose-LSTM for video. Across LSP, MPII, Penn Action, and BBC Pose, the architectures demonstrated superior performance to state-of-the-art methods using various metrics.

  • Problem

    Pose estimation is difficult because body mechanics have many degrees of freedom, occlusion is frequent, and predefined anchor poses can limit generalization to unforeseen poses.

  • Method

    UniPose performs single-stage pose estimation using contextual multi-scale processing with a Waterfall atrous-convolution module, and UniPose-LSTM extends it to temporal video processing.

  • Results

    UniPose and UniPose-LSTM demonstrated superior performance to state-of-the-art methods across LSP, MPII, Penn Action, and BBC Pose using various metrics.

  • Takeaways & Limitations

    The framework provides a unified architecture for estimating poses in single images and videos without relying on statistical postprocessing.

  • Takeaways & Limitations

    The framework is presented as promising for broader applications, including multiple-person pose estimation, indicating that this scope remains future work.

Abstract

from arXiv · show

We propose UniPose, a unified framework for human pose estimation, based on our "Waterfall" Atrous Spatial Pooling architecture, that achieves state-of-art-results on several pose estimation metrics. Current pose estimation methods utilizing standard CNN architectures heavily rely on statistical postprocessing or predefined anchor poses for joint localization. UniPose incorporates contextual segmentation and joint localization to estimate the human pose in a single stage, with high accuracy, without relying on statistical postprocessing methods. The Waterfall module in UniPose leverages the efficiency of progressive filtering in the cascade architecture, while maintaining multi-scale fields-of-view comparable to spatial pyramid configurations. Additionally, our method is extended to UniPose-LSTM for multi-frame processing and achieves state-of-the-art results for temporal pose estimation in Video. Our results on multiple datasets demonstrate that UniPose, with a ResNet backbone and Waterfall module, is a robust and efficient architecture for pose estimation obtaining state-of-the-art results in single person pose detection for both single images and videos.

1. Introduction

UniPose addresses pose-estimation challenges from body-motion complexity, occlusion, and limited anchor-pose generalization with a single-stage contextual framework. Its Waterfall module combines cascaded atrous convolutions and multi-scale fields of view, while UniPose-LSTM extends the approach to video.

  • Pose estimation is challenging because human bodies have many degrees of freedom and body parts are frequently occluded.
  • Known anchor poses can limit generalization and learning for unforeseen poses.
  • UniPose performs single-stage contextual pose estimation without statistical postprocessing.The framework uses contextual information from a large field of view and multi-scale processing.
  • The Waterfall module combines cascaded atrous convolutions with multiple fields of view inspired by spatial pyramids.This design combines cascade efficiency with multi-scale receptive-field coverage.
  • UniPose jointly determines human bounding boxes and joint locations, eliminating separate network branches for these tasks.
  • UniPose-LSTM extends the Waterfall approach with a linear sequential LSTM configuration for temporal pose estimation in video.

2. Related Work

Prior pose-estimation methods use geometric reasoning, cascaded CNNs, multi-scale representations, or temporal modeling, but often add computational complexity or rely on separate detection branches and predefined pose anchors.

  • Image-based pose estimation: CNN-based methods improved pose estimation through cascaded refinement, multi-context features, multi-scale fusion, regression, and contextual aggregation.Examples include Convolutional Pose Machines, stacked hourglass networks, HRNet, DeepPose, CPF, and CFA.
  • Image-based pose estimation: Predefined anchor poses can limit generalization to unforeseen human poses.LCR-Net detects joints, matches them to anchor poses, and regresses 3D coordinates.
  • Image-based pose estimation: Some methods require separate branches or models for human bounding-box detection and joint arrangement.LightTrack uses YOLO before joint detection, while LCR-Net separates detection from joint classification and arrangement.
  • Temporal pose estimation: Video pose estimation methods often process frames independently, despite temporal information helping with motion blur and occlusion.Existing alternatives use adjacent color frames, optical flow, or recurrent architectures, with optical-flow approaches increasing computational cost.
  • Atrous convolution and ASPP: Pooling reduces feature-map resolution, while atrous convolutions enlarge receptive fields, avoid downsampling, and support multi-scale processing.ASPP combines parallel atrous branches with different rates and restores feature maps to the original resolution through interpolation.

3. UniPose Architecture

UniPose combines a Waterfall Atrous Spatial Pooling module with a decoder that jointly predicts joint and bounding-box heatmaps. UniPose-LSTM extends the same backbone, WASP, and decoder with recurrent temporal processing for video pose estimation.

  • UniPose: UniPose uses a WASP module and decoder to perform unified human bounding-box and joint detection without separate branches.The decoder produces joint and bounding-box heatmaps without postprocessing or independent parallel branches.
  • UniPose: The pipeline applies a ResNet-101 backbone, WASP feature processing, Softmax heatmaps, bilinear upsampling, and local-max joint localization.The decoder generates one heatmap per joint and recovers the original image resolution before localization.
  • WASP module: WASP combines large receptive fields, cascaded atrous convolutions, and multi-scale representations in a compact waterfall arrangement.Its branches use increasing dilation rates of 6, 12, 18, and 24, with smaller filters intended to improve efficiency and reduce parameters.
  • Decoder module: The decoder concatenates 256 WASP feature maps with 256 low-level ResNet feature maps before convolution, dropout, interpolation, and heatmap prediction.For a 1280×720 input, the resulting heatmaps represent joints and the bounding box.
  • UniPose-LSTM: UniPose-LSTM feeds current decoder heatmaps and previous final heatmaps into an LSTM, followed by convolutional layers that produce final joint heatmaps.The video configuration retains the same ResNet-101 backbone, WASP module, and decoder as the single-image model.

4. Datasets

The evaluation uses two single-image datasets and two video datasets spanning sports, everyday activities, temporal tracking, and sign language. Training uses Gaussian joint maps rather than single-point targets.

  • Dataset selection: Experiments use LSP and MPII for single-image pose estimation, and Penn Action and BBC Pose for video-based evaluation.The datasets cover single-person images, everyday activities, temporal tracking, and sign-language pose estimation.
  • Single-image datasets: LSP contains 2,000 sports images with 14 body keypoints, split evenly between training and testing.Its lower data variation provides an initial assessment of single-person pose estimation.
  • Single-image datasets: MPII contains approximately 25,000 images covering over 40,000 subjects and 410 everyday activities, with joint, orientation, and occlusion annotations.It also includes adjacent frames without labels for those frames.
  • Video datasets: Penn Action provides 2,326 video sequences across 15 activities for temporal pose estimation and joint tracking.The evaluation uses previous detections as context when estimating pose in a frame.
  • Video datasets: BBC Pose contains 20 sign-language videos with 610,115 training, 309,171 validation, and 309,260 testing images, but labels only seven upper-body keypoints.The restricted keypoint coverage is a stated dataset limitation.
  • Training targets: Training converts ground-truth joint locations into Gaussian maps, using σ = 3 for the reported results.Gaussian targets are intended to provide more effective training signals than single-point joint labels.

5. Experiments

UniPose and UniPose-LSTM are evaluated with dataset-specific pose metrics and training settings, and the reported experiments show superior performance across several single-frame, video, and sign-language benchmarks.

  • Reported results: UniPose and UniPose-LSTM achieved superior performance on several datasets for single-frame, video, and sign-language pose estimation.The claim covers both image and temporal pose-estimation configurations.
  • Metrics: PCP@0.5 counts a limb as detected when the distance between its predicted joints is below half the ground-truth limb length.The paper notes that PCP can penalize smaller limbs more strongly.
  • Metrics: PCK@0.2 uses 20% of torso diameter as its correctness threshold, while PCKh@0.5 uses 50% of head diameter.Both metrics classify a joint as correct when its predicted location lies within the specified ground-truth-relative distance.
  • Training settings: Training uses native image resolution rather than resizing, with batch sizes ranging down to 4 for datasets such as BBC Pose.The stated rationale is to preserve detail through the dense, large-FOV network.

6. Results

UniPose and UniPose-LSTM achieve strong single-image and video pose-estimation results across LSP, MPII, Penn Action, and BBC Pose. The evaluations also show effective handling of occlusion, motion, and temporal information.

  • LSP: 72.8% PCP and 94.5% PCK@0.2 were achieved on LSP, with gains over other approaches on both metrics.
  • LSP: UniPose identifies symmetric joints precisely in LSP examples, including challenging cases where limbs occlude one another.
  • MPII: 92.7% PCKh was achieved on MPII, outperforming other methods for single-person pose estimation and handling occlusion such as a horse rider.
  • Penn Action: 99.3% detection was achieved on Penn Action using five consecutive frames, improving 1.6% over the next-best result.The evaluation counts a joint as correct when its predicted location lies within the provided bounding box.
  • Penn Action: LSTM accuracy gains plateaued at five or more frames, while incorporating consecutive frames improved robustness against motion blur and occlusions.
  • BBC Pose: UniPose-LSTM achieved 98.9% PCKh on BBC Pose, exceeding CPM's 97.6% PCK comparison.The system also generated bounding boxes containing the signers' signing areas.

7. Conclusion

UniPose and UniPose-LSTM provide unified architectures for pose estimation in single images and videos. Across four datasets and multiple metrics, the methods showed superior performance to state-of-the-art methods, while broader multiple-person and 3D applications remain future directions.

  • Conclusion: UniPose uses a Waterfall module with cascaded atrous convolutions and multi-scale representations for single-image pose estimation.Its large field of view is intended to improve contextual interpretation of the frame.
  • Conclusion: UniPose and UniPose-LSTM demonstrated superior performance to state-of-the-art methods across LSP, MPII, Penn Action, and BBC Pose using various metrics.
  • Future directions: The framework is presented as promising for future multiple-person pose detection and 3D pose estimation.
Loading 2001.08095v1…