KAD-Net: Kinematics-Aware Decoupled Learning for Robust 3D Hand Pose Estimation from a Single Depth Image

1 Introduction

• Regression methods can lose spatial structure when flattening feature maps, while detection methods preserve local articulation but may miss long-range joint relationships.The introduction frames these as complementary limitations of the two common approaches.Source: §1 · p. 2
• Prior GCN-based methods treat joints uniformly, overlooking kinematic dependencies that could regularize distal-joint predictions.The introduction notes that distal joints have higher degrees of freedom and greater articulation variability.Source: §1 · p. 2
• Shared feature spaces for UV localization and depth regression can cause task interference because the tasks require different kinds of geometric reasoning.The introduction characterizes UV prediction as 2D surface mapping and depth estimation as 3D scene-context reasoning.Source: §1 · p. 3
• The method decouples UV localization from depth regression and uses a hierarchical learning strategy in the depth branch to mitigate negative feature transfer.The separate branches learn representations tailored to their respective geometric objectives.Source: §1 · p. 3
• FTC aggregates three consecutive finger joints to enrich distal-joint features with proximal context and improve robustness to occlusion.Structural cues from visible joints help locate partially or fully occluded distal joints.Source: §1 · pp. 3–4
• KAD-Net achieves state-of-the-art performance on ICVL, NYU, and MSRA.The method is reported to outperform existing methods on all three public datasets.Source: §1 · p. 4

2.1 Graph convolutional neural networks

• Spectral GCNs filter graph signals in the frequency domain, whereas spatial GCNs update nodes by aggregating neighbor features.Examples cited include Cai et al. for spectral convolutions and Kulon et al. for a spatial GCN encoder-decoder.Source: §2.1 · p. 4
• KAD-Net uses three finger joints to form a local kinematic representation that complements distal-joint features and supports occlusion robustness.The passage distinguishes this approach from existing GCN methods by its use of local finger kinematics.Source: §2.1 · p. 4

2.2 Hierarchical multitask networks

• Hierarchical multitask methods divide estimation by finger groups, hand regions, correlated joints, or coarse-to-fine stages to capture task-specific structure.Examples include finger-specific streams, finger–palm branches, joint grouping, and progressive regression.Source: §2.2 · pp. 4–5
• Unlike prior unified UV-depth hierarchies, KAD-Net separates UV, attention, and depth branches, with six depth subtasks per finger and the palm.The depth branch uses a specialized hierarchical multitask architecture to mitigate negative feature transfer.Source: §2.2 · p. 5, Figure 1 · p. 5

3.1 Architecture overview

• Distinct UV, attention-enhancement, and depth branches use Hourglass features; UV heatmaps and unsupervised attention maps are fused to guide depth estimation.The depth branch uses a hierarchical multitask design to mitigate negative feature transfer.Source: §3.1 · pp. 6–7

3.2 Finger topological constraint module of the UV branch

• FTC maps image features into graph nodes, propagates relations, builds three-joint per-finger kinematic features, rectifies distal nodes, then remaps refined graph features to convolutional space.For NYU, which annotates two joints per finger, the sequence is supplemented with the corresponding palm joint.Source: §3.2 · pp. 7–9

3.3 Hierarchical multitask depth estimation

• Fused UV and unsupervised attention maps guide six dataset-adapted finger/palm depth subtasks; each predicts local depths, followed by global depth prediction for all joints.Joint groupings vary with each benchmark’s annotation scheme.Source: §3.3 · pp. 9–10

3.4 Loss function

• Training applies L1 supervision to FTC node features, UV outputs, each depth subtask, and global depth outputs.The total objective combines the listed losses; the experiment sets γ = 0.2.Source: §3.4 · p. 10

4.1 Implementation details

• Inputs are 128×128 depth images normalized to [−1, 1], augmented with rotations, scaling, translations, and PixDropout.Rotations span [−180, 180] degrees, scaling factors are [0.9, 1.1], translations are [−8, 8] mm, and PixDropout uses α = 0.15.Source: §4.1 · p. 10
• Training uses Adam with an initial learning rate of 10−5 and cosine annealing; weight decay is 10−4 and dropout is 0.6.Source: §4.1 · p. 10
• The model trains for 70 epochs on ICVL and NYU and 80 on MSRA, using one NVIDIA RTX 4090 GPU with 24 GB memory.Source: §4.1 · p. 10

4.2 Dataset and evaluation metrics

• ICVL provides 22,000 training and 1,600 testing images; the protocol excludes 308,000 augmented frames, and its skeletal model has three joints per finger plus a palm joint.Source: §4.2 · p. 11
• NYU has 72,000 training samples from one subject and 8,200 test samples from three; evaluation uses the 14-joint frontal-view subset.Source: §4.2 · p. 11
• MSRA contains 76,500 depth images from nine subjects, uses four joints per finger plus a palm joint, and follows leave-one-subject-out cross-validation.Source: §4.2 · p. 11
• MJE is average Euclidean joint-coordinate error in millimeters, while FSR is the share of test frames whose MJE falls below a predefined threshold.Source: §4.2 · p. 11

4.3.1 Components and structures

• 5.45 mm MJE is the best ICVL result among the tested component configurations, combining FTC and MMD (Table 1).FTC alone reduces MJE from 5.73 mm to 5.63 mm, while MMD alone reaches 5.61 mm.Source: §4.3.1 · p. 11, Table 1 · p. 12

4.3.2 Components of FTC module

• FG reduces errors across all joints; TC alone helps fingertips and intermediate joints but slightly worsens others, while its constraints act as a global kinematic-chain regularizer.The full FTC module adds only modest fingertip-error reductions over FG alone.Source: §4.3.2 · pp. 12–13

4.3.3 Combination strategy of FTC module

• 5.66 mm for Reversed-serial, 5.57 mm for Parallel, and 5.45 mm for the proposed FTC integration; the proposed strategy performs best (Table 3).The authors attribute the advantage to coordinating feature propagation and topological constraints.Source: Table 3 · p. 12, §4.3.3 · p. 13

4.3.4 Multi-task architecture designs

• 6.68 mm for Shared, 5.64 mm for Decoupled, and 5.61 mm for Guided-Decoupled MJE; UV-to-depth guidance improves on decoupling alone (Table 4).The passages attribute the remaining advantage to spatial cues that help refine joint-wise depth estimation.Source: §4.3.4 · pp. 13–14, Table 4 · p. 14

• On ICVL, errors fall for most joints, especially Thumb, Pinky, and Index, but rise slightly for Middle and Ring; FSR leads below 10 mm.The authors attribute the Middle and Ring pattern to similar motion and note that limited training data and annotation inaccuracies exacerbate it.Source: §4.4 · pp. 14, 16
• On NYU, KAD-Net leads per-joint results except palm and has the highest FSR at 30 mm; severe multi-joint occlusion can still degrade predictions.Qualitative comparisons show more anatomically plausible poses under self-similarity and self-occlusion, while substantial occlusion remains a failure case.Source: Figure 8 · p. 16, §4.4 · p. 17
• On MSRA, KAD-Net improves over SOTA across a wide range of FSR thresholds and most yaw and pitch variations, with robust per-joint results.Source: Figure 6 · p. 15, Figure 7 · p. 16, §4.4 · p. 17
• On ICVL, the lowest compared MJE comes with 87 FPS on one GPU and a moderate parameter count, balancing accuracy and computational cost.Source: §4.4 · p. 17

• KAD-Net separates UV and depth estimation; its UV branch uses three-joint finger kinematics and a topological constraint to capture hand-joint relationships.Source: §5 · p. 17
• The depth branch uses a hierarchical multi-branch structure intended to mitigate negative feature transfer.Source: §5 · p. 17
• Across ICVL, NYU, and MSRA, KAD-Net reports MJE values of 5.45 mm, 7.43 mm, and 6.93 mm, respectively.Source: §5 · p. 18