PaperlayerView exact revision on arXiv

Source-linked AI summary

Spatial Attention Deep Net with Partial PSO for Hierarchical Hybrid Hand Pose Estimation

Qi Ye, Shanxin Yuan, Tae-Kyun Kim

arXiv:1604.03334v2cs.CV

TL;DR

3D hand pose estimation must handle high-dimensional configurations and kinematic implausibility, while existing hierarchical methods mainly decompose the output space. The paper combines spatial attention for hierarchical input, feature, and output transformations with hierarchical partial PSO for kinematic-constrained refinement, and reports significant improvements over prior methods across three public benchmarks.

  • Problem

    3D hand pose estimation is challenging in a high-dimensional space, while existing hierarchical methods mainly decompose the output space and leave input or feature spaces nearly unchanged.

  • Method

    The method uses spatial attention to transform input, feature, and output spaces hierarchically, and partial PSO to enforce hand kinematic constraints on CNN estimates.

  • Results

    The method significantly outperforms state-of-the-art methods on three public benchmarks, including larger gains under broad viewpoint and articulation variation.

  • Takeaways & Limitations

    Applying the kinematic hierarchy to both discriminative spaces and generative optimization yields good performance across three datasets, especially under large variations.

Abstract

from arXiv · show

Discriminative methods often generate hand poses kinematically implausible, then generative methods are used to correct (or verify) these results in a hybrid method. Estimating 3D hand pose in a hierarchy, where the high-dimensional output space is decomposed into smaller ones, has been shown effective. Existing hierarchical methods mainly focus on the decomposition of the output space while the input space remains almost the same along the hierarchy. In this paper, a hybrid hand pose estimation method is proposed by applying the kinematic hierarchy strategy to the input space (as well as the output space) of the discriminative method by a spatial attention mechanism and to the optimization of the generative method by hierarchical Particle Swarm Optimization (PSO). The spatial attention mechanism integrates cascaded and hierarchical regression into a CNN framework by transforming both the input(and feature space) and the output space, which greatly reduces the viewpoint and articulation variations. Between the levels in the hierarchy, the hierarchical PSO forces the kinematic constraints to the results of the CNNs. The experimental results show that our method significantly outperforms four state-of-the-art methods and three baselines on three public benchmarks.

1 Introduction

3D hand pose estimation is difficult because depth images must support high-dimensional configurations under viewpoint, articulation, occlusion, and shape variation. The proposed hybrid method applies hierarchy to both discriminative input/feature spaces and generative optimization, and outperforms prior methods on three benchmarks.

  • 3D hand pose estimation maps depth images to high-dimensional hand configurations affected by articulation, viewpoint, self-similarity, occlusion, and shape variation.
  • Hierarchical methods decompose output variables into smaller spaces, but prior approaches largely leave input or feature spaces unchanged across the hierarchy.Cascaded methods only partially update inputs or features, while one prior optimization performs brute-force energy search.
  • The method extends hierarchy to discriminative input, feature, and output spaces through spatial attention, while also organizing generative energy optimization hierarchically.The spatial attention mechanism transforms spaces to reduce viewpoint and articulation variation.
  • Partial PSO selects pose estimates satisfying hand kinematic constraints between hierarchical levels, using samples centered on discriminative results to reduce the generative search space.
  • The method significantly outperforms state-of-the-art methods on three public benchmarks.

2 Related Work

Prior attention and cascaded-regression methods mainly select or partially transform features, whereas this method jointly transforms feature and output spaces through hierarchical cascade refinement. Partial pose optimization further reduces the complexity of generative search.

  • Attention mechanisms have been used to focus models on relevant regions or generate samples under different transformations.
  • Feature Selection with Attention: The proposed attention mechanism breaks down viewpoint and articulation complexity hierarchically while refining estimates through a cascade.
  • Feature Selection with Attention: Cascaded refinement transforms feature and output spaces into a canonical space rather than only selecting features.
  • Earlier cascaded hand-pose methods used hand-crafted features or only partial spatial transformations, whereas the proposed method fully transforms feature and output spaces.
  • Partial pose optimization reduces each generative estimation's search complexity within the hierarchical discriminative-generative framework.

3 Method overview

The method estimates 21 hand joints across four hierarchical layers, sequentially refining CNN predictions with spatial attention and passing kinematically constrained partial poses between layers. Its hand model uses 51 degrees of freedom with fixed palm joints and finger bone lengths.

  • The hand model represents 21 joints divided into four layers, with joints indexed by layer and finger position.The wrist is indexed separately in the first layer, while finger joints are ordered from thumb to pinky.
  • Four layers are estimated sequentially, with spatial attention linking layers by transforming input, feature, and output spaces.Within each layer, cascaded stages learn residuals from previous-stage estimates using transformed attention features and output spaces.
  • Partial PSO enforces kinematic constraints on final cascaded predictions before the optimized pose is passed to the next layer.
  • The hand model has 51 degrees of freedom: six for global orientation and location, plus 15 bone-rotation degrees of freedom in each of layers 1–3.The model fixes six palm joints and preserves finger bone lengths.

4 Partial Pose Estimation by Spatial Attention Deep Net

The method transforms both CNN feature inputs and pose outputs into joint- and viewpoint-centered spaces, then performs cascaded refinement within each hierarchical layer. Spatial attention updates these spaces from previous estimates, while residual predictions are mapped back to original coordinates.

  • 4.1 Spatial Attention Mechanism for Hand Space: Spatial attention splits into rotation calculation and pixel mapping, transforming feature maps, estimates, and training ground truth into a new space.The inverse transformation returns locations to the original space.
  • 4.1 Spatial Attention Mechanism for Hand Space: The mapping uses joint-centered translation and rotation; b = 1 gives rotation and translation, while b < 1 additionally permits cropping.Coordinates are normalized by the corresponding input and output feature-map dimensions.
  • 4.2 Cascaded Regression within Each Hierarchical Layer: Within each layer, cascaded stages transform feature maps using the previous pose estimate, regress residual offsets, and transform updated predictions back.The process repeats through the final stage, with other joints refined in parallel.
  • 4.3 Hierarchical Regression: Layer 0 jointly learns all initial joints, while later hierarchical layers use multi-resolution images, prior-layer estimates, and a fixed viewpoint from Layer 0.The same process is applied separately to the joints in later layers.
  • 4.3 Hierarchical Regression: For later layers, spatial attention transforms multi-resolution inputs and feature maps according to previous-layer estimates, while output transformations use the estimated viewpoint and pose.Initial-stage and refinement-stage transformations differ in whether rotation is updated.

5 Partial PSO with Kinematic Constraints for Final Refinement

The generative refinement stage applies partial PSO after CNN estimation to search over hierarchical pose variables while explicitly enforcing palm structure and bone-length constraints. Different initialization procedures are used for the palm and subsequent bone layers.

  • 5. Partial PSO with Kinematic Constraints for Final Refinement: Partial PSO performs final refinement from each layer’s discriminative prediction using an energy function that explicitly incorporates kinematic constraints.The optimization is organized hierarchically to refine partial poses.
  • 5. Partial PSO with Kinematic Constraints for Final Refinement: The pose variables comprise a 7-dimensional global palm pose and three 15-dimensional partial poses defined by bone rotations.The global pose uses a unit quaternion and hand translation; later layers use five 3D Euler angles.
  • 5. Partial PSO with Kinematic Constraints for Final Refinement: The energy terms constrain palm spatial structure, bone lengths, silhouette inclusion, and joint depth-range compatibility.The sampling covariance is manually set so parameters remain within valid ranges.
  • 5. Partial PSO with Kinematic Constraints for Final Refinement: For layer 0, Kabsch alignment initializes global rotation and translation, and Gaussian samples around that pose preserve the palm’s spatial structure.This initialization is centered on the inferred pose rather than directly on the CNN locations.
  • 5. Partial PSO with Kinematic Constraints for Final Refinement: For later layers, bone rotations are inferred from CNN joint estimates across adjacent layers, then converted back to locations while enforcing bone lengths.The same optimization structure is applied to these partial poses.

6 Experiment

Experiments on ICVL, NYU, and MSRC evaluate the proposed method against baselines and prior work. Hierarchical spatial attention improves over baselines, while partial PSO further improves results across datasets.

  • Experimental setup: The evaluation uses three public benchmarks: ICVL, NYU, and MSRC, with differing views, articulations, frame counts, and joint annotations.ICVL provides 16 bone-centre locations, NYU 36 joints, and MSRC 22 joints; the study uses the annotation version from.
  • Self-comparison: Hier SA significantly outperforms the three baselines on all datasets under roughly equal 130MB parameter budgets.The baselines are Holi, Holi Derot, and Holi SA.
  • Self-comparison: At 20mm, Hier SA improves the percentage of frames over Holi by 5% on ICVL, 18% on NYU, and 30% on MSRC.The largest gain occurs on MSRC, which covers a full range of viewpoints and articulations.
  • Self-comparison: The self-comparisons attribute gains to hierarchical handling of articulations and spatial attention for reducing viewpoint and articulation complexity.A separate rotation-estimation network can instead deteriorate later estimation when training-set viewpoint variation is small.
  • Self-comparison: Hybr Hier SA further improves Hier SA by a large margin consistently across all datasets, supporting the effectiveness of partial PSO for enforcing kinematic constraints.The generative refinement is evaluated after the discriminative Hier SA model.
  • Comparison with Prior Works: Compared with prior work, the method is 26% better than HSO at D = 10mm on ICVL, 10% better than HandsDeep within D = 30mm on NYU, and roughly matches FeedLoop.On MSRC at D = 30mm, it exceeds HSO and Sharp et al. by 35% and 50%, respectively, even when those methods use ground-truth rotation.

7 Conclusion

The method applies kinematic hierarchy to both discriminative input/features and generative optimization, using spatial attention and partial PSO. It reports good performance across three datasets, especially under large variations, with improved finger discrimination and precision over prior methods.

  • Spatial attention transforms the input, feature, and output spaces interactively, reducing viewpoint and articulation complexity during refinement.
  • Partial PSO between hierarchy layers enforces kinematic constraints on discriminative estimates and reduces error accumulation from previous layers.
  • The method demonstrates good performance on three datasets, especially on the dataset under large variations.
  • Compared with other methods, it discriminates fingers well, achieves better precision, and maintains good estimation for many challenging viewpoints.
Loading 1604.03334v2…