PaperlayerView exact revision on arXiv

Source-linked AI summary

A2J: Anchor-to-Joint Regression Network for 3D Articulated Pose Estimation from a Single Depth Image

Fu Xiong, Boshen Zhang, Yang Xiao, Zhiguo Cao, Taidong Yu, Joey Tianyi Zhou, Junsong Yuan

arXiv:1908.09999v1cs.CV

TL;DR

Single-depth-image 3D hand and body pose estimation is challenging, and existing approaches face representational or computational limitations. A2J addresses this with densely placed, adaptively weighted anchor regressors and an end-to-end 2D CNN architecture. Across three hand and two body datasets, it generally outperforms state-of-the-art methods in effectiveness and efficiency while running around 100 FPS.

  • Problem

    Single-depth-image 3D hand and body pose estimation remains difficult because of pose variation, joint similarity, self-occlusion, and limitations in existing architectures.

  • Method

    A2J densely places anchor points as local regressors, aggregates their weighted joint predictions, proposes informative anchors, and uses a ResNet-50 2D CNN backbone with three functional branches.

  • Results

    A2J generally outperforms state-of-the-art methods on three hand and two body datasets in effectiveness and efficiency, with around 100 FPS on one NVIDIA 1080Ti GPU.

  • Takeaways & Limitations

    The anchor-based paradigm provides an end-to-end approach that combines global-local spatial context and adaptive joint-specific anchor selection for efficient 3D pose estimation.

Abstract

from arXiv · show

For 3D hand and body pose estimation task in depth image, a novel anchor-based approach termed Anchor-to-Joint regression network (A2J) with the end-to-end learning ability is proposed. Within A2J, anchor points able to capture global-local spatial context information are densely set on depth image as local regressors for the joints. They contribute to predict the positions of the joints in ensemble way to enhance generalization ability. The proposed 3D articulated pose estimation paradigm is different from the state-of-the-art encoder-decoder based FCN, 3D CNN and point-set based manners. To discover informative anchor points towards certain joint, anchor proposal procedure is also proposed for A2J. Meanwhile 2D CNN (i.e., ResNet-50) is used as backbone network to drive A2J, without using time-consuming 3D convolutional or deconvolutional layers. The experiments on 3 hand datasets and 2 body datasets verify A2J's superiority. Meanwhile, A2J is of high running speed around 100 FPS on single NVIDIA 1080Ti GPU.

1. Introduction

A2J addresses single-depth-image 3D pose estimation with an end-to-end anchor-based network that aggregates adaptive local regressors, targeting accuracy, generalization, and efficiency.

  • Single-depth-image 3D hand and body pose estimation remains challenging because of pose variation, joint similarity, and self-occlusion.
  • A2J densely places anchor points that jointly capture global-local context and predict each joint through weighted aggregation.Each anchor acts as a local regressor, while shared anchors can preserve articulated relationships among joints.
  • The anchor proposal procedure selects informative anchors for each joint using estimation error and spatial layout, encouraging them to surround the joint.This design is intended to enhance generalization and adapt anchor selection to different joints.
  • A2J uses a ResNet-50 2D CNN backbone with offset, depth, and anchor-proposal branches, avoiding deconvolutional layers.The architecture is designed for end-to-end learning and high running efficiency.
  • A2J generally outperforms state-of-the-art methods on three hand and two body datasets in effectiveness and efficiency.It runs at around 100 FPS on a single NVIDIA 1080Ti GPU.

2. Related Works

Prior pose-estimation methods use hand-crafted features, global or local 2D deep learning, 3D CNNs, or point-set networks, each with notable representational or computational trade-offs.

  • Non-deep-learning methods typically extract hand-crafted features before classification or regression, but these features may be insufficiently representative.
  • Global-regression 2D methods can lose local spatial context through fully connected feature aggregation, while local approaches use different design choices.
  • 3D CNN methods voxelize depth images and require costly convolutional or deconvolutional processing with many parameters.Voxelization increases memory and runtime burden.
  • Point-set methods transform depth images into point clouds but require time-consuming preprocessing such as point sampling and KNN search.
  • A2J uses dense anchors with a 2D CNN backbone to combine global-local context without computationally expensive deconvolutional layers.

3. A2J: Anchor-to-Joint Regression Network

A2J densely places anchors on depth images and combines their joint estimates, using learned anchor weights and a 2D CNN backbone with three functional branches.

  • A2J densely sets anchor points on depth images, with each anchor acting as a local regressor for predicting all joint positions.The anchors capture global-local spatial context through their distribution on the input image.
  • Anchor proposal assigns joint-specific weights to identify informative anchors, whose normalized outputs are aggregated for joint localization.Anchors with normalized weight ˜Pj(a) > 0.02 are treated as informative, and training encourages them to surround the corresponding joint.
  • The offset and depth branches separately predict anchor-to-joint in-plane offsets and joint depth values using local context from backbone features.Each branch uses four 3 × 3 intermediate convolutional layers and produces outputs for 16 anchors corresponding to each feature-map location.
  • The anchor proposal branch predicts joint-specific anchor weights through four intermediate convolutional layers and one output convolutional layer.Its output has 16 × K × 1 channels for K joints.
  • A2J uses a modified ImageNet-pretrained ResNet-50 backbone, retaining finer spatial information with 16× rather than 32× in-plane downsampling.The backbone also uses dilation 2 in layer 4 to enlarge the receptive field.

3.2. Learning procedure of A2J

A2J learns joint positions through jointly supervised position estimation and informative-anchor surrounding losses. Depth targets are transformed for comparable scale, while surrounding constraints reduce viewpoint bias.

  • Target transformation: A2J transforms each joint’s ground-truth depth using scale and offset parameters before training.For hand pose, μ=1 and θ is the center-point depth; for body pose, μ=50 and θ=0.
  • Joint position estimation loss: The joint position loss supervises anchor-based in-plane offset and depth estimation with a smoothL1-like objective.The factor α=0.5 balances the in-plane offset and depth tasks, while τ1=1 and τ2=3 account for noisier depth values.
  • Informative anchor point surrounding loss: The informative-anchor surrounding loss encourages selected anchors to surround joints, helping A2J observe joints from multiple viewpoints.The authors state that this constraint alleviates viewpoint bias and improves generalization ability.
  • End-to-end training: The two losses jointly supervise A2J’s end-to-end learning procedure, with λ=3 balancing the surrounding and joint-position terms.The total loss is described as loss = λloss1 + loss2.

4.1. Experimental setting

Experiments evaluate A2J on three hand datasets and two body datasets using task-specific 3D pose metrics. The setup includes varied dataset sizes, joint annotations, preprocessing, augmentation, and optimization choices.

  • Datasets: The evaluation covers HANDS 2017, NYU, ICVL, ITOP, and K2HPD datasets with hand or body 3D joint annotations.The datasets contain 14–36 evaluated hand joints and 15–19 body joints, with training and test splits reported for each.
  • Evaluation metrics: Hand pose is evaluated with average 3D distance error and percentage of success frames, while body pose uses PDJ and mAP under a 10-cm rule.The metrics are selected separately for hand and body pose estimation.
  • Reported comparisons: HANDS 2017 and NYU results are presented in performance-comparison tables, including mean 3D distance error for NYU.The HANDS 2017 table distinguishes seen, unseen, average, and 10-model ensemble results.
  • Implementation: A2J training uses cropped and resized depth images, random geometric and Gaussian-noise augmentation, and Adam optimization with a 0.00035 learning rate.Input resolution is 176×176 for hand and 288×288 for body; weight decay is 0.0001.

4.2. Comparison with state-of-the-art methods

Across hand and body benchmarks, A2J generally outperforms state-of-the-art methods in effectiveness and efficiency, with specific exceptions and margins reported for ICVL, ITOP, K2HPD, and HANDS 2017.

  • K2HPD dataset: On the million-scale K2HPD dataset, A2J consistently outperforms other approaches in both effectiveness and efficiency.This dataset-level result is presented as evidence for the superiority of the proposed approach.
  • HANDS 2017 dataset: V2V∗ is the strongest competitor but uses a 10-model ensemble, making it much slower than A2J's single model.On HANDS 2017, A2J is reported as inferior to V2V∗, while the comparison highlights the efficiency difference.
  • NYU and ICVL datasets: A2J is superior to other methods in most NYU and ICVL cases on accuracy and efficiency, but is slightly less accurate than V2V∗ and P2P on ICVL.The ICVL accuracy exception comes with much higher running efficiency.
  • ITOP dataset: A2J is significantly superior on ITOP front-view and top-view tracks except V2V∗, with performance gaps of at least 3.1 and 5, respectively.These results support applicability to 3D body pose estimation as well as 3D hand pose estimation.
  • K2HPD dataset: On K2HPD, A2J outperforms other methods consistently across PDJ thresholds, with an average performance gap of at least 10.8.The advantage increases remarkably as the PDJ threshold decreases.

4.3. Ablation study

Ablation studies show that anchor adaptivity, spatially surrounding informative anchors, and separate offset-depth estimation contribute to A2J's performance, while the anchor paradigm outperforms alternatives and supports efficient transfer.

  • Component effectiveness analysis: Removing the anchor proposal branch causes a remarkable performance drop, especially for body pose, confirming that anchor points contribute unequally to each joint.Anchor point adaptivity is identified as an essential A2J property.
  • Component effectiveness analysis: Removing the informative anchor point surrounding loss reduces performance, especially for body pose, indicating that spatial layout matters for generalization ability.The loss encourages informative anchors to surround the corresponding joint and alleviates viewpoint bias.
  • Component effectiveness analysis: Combining in-plane offset and depth-value estimation in one branch reduces performance to some degree, consistent with their different physical characteristics.The ablation separately evaluates in-plane offset and depth estimation branches.
  • Effectiveness of anchor-based paradigm: Using the same ResNet-50 backbone, A2J significantly outperforms global regression and FCN paradigms on K2HPD.The comparison attributes the advantage to preserved local spatial context and anchor adaptivity toward each joint.
  • Effectiveness of anchor-based paradigm: A2J runs faster than FCN but slower than global regression, while providing a better effectiveness-efficiency tradeoff than global regression.The comparison uses the same ResNet-50 backbone for the evaluated paradigms.
  • Effectiveness of pre-training: ImageNet pre-training improves A2J performance for both hand and body pose estimation.The study compares A2J with and without ImageNet pre-training on NYU and ITOP.
  • Backbone network comparison: Deeper backbones achieve better results with relatively slower efficiency, but the performance gap among ResNet-18, ResNet-34, and ResNet-50 is not large.Even ResNet-18 achieves generally state-of-the-art performance at 192.25 FPS.

4.4. Qualitative evaluation

Qualitative results indicate that A2J generally works well for 3D hand and body pose estimation, while failures mainly arise under serious self-occlusion and dramatic pose variation.

  • Qualitative results: A2J generally works well on NYU hand and ITOP front-view body pose data, with failures mainly caused by serious self-occlusion and dramatic pose variation.The qualitative evaluation covers both 3D hand and body pose estimation.

4.5. Running speed analysis

A2J achieves near-real-time online inference on a single NVIDIA 1080Ti GPU, running at different speeds for hand and body pose estimation.

  • Running speed: 105.06 FPS is achieved for 3D hand pose estimation on a single NVIDIA 1080Ti GPU.This includes 1.5 ms for image reading and warping and 8.0 ms for network forward propagation and post-processing.
  • Running speed: 93.78 FPS is achieved for 3D body pose estimation on a single NVIDIA 1080Ti GPU.This includes 0.4 ms for image reading and warping and 10.2 ms for network forward propagation and post-processing.

5. Conclusions

A2J estimates 3D articulated pose from depth images using densely placed anchors that capture global-local context and predict joint positions in an ensemble. Its experiments demonstrate effectiveness and efficiency, while future work targets improved anchor fusion.

  • A2J densely places anchor points on depth images to capture global-local spatial context and predict joint positions in an ensemble.The method also extracts informative anchors that adapt to different joints.
  • Qualitative results compare ground-truth poses shown in red with predicted poses shown in yellow.
  • A2J is built on a 2D CNN without computationally expensive deconvolutional layers.
  • Wide-range experiments demonstrate A2J's superiority in both effectiveness and efficiency.
  • Future work will seek a more effective way to fuse the anchor points.
Loading 1908.09999v1…