LISA: Learning Implicit Shape and Appearance of Hands

• MANO is widely used for hand modeling and tracking, but its low-resolution mesh lacks texture coordinates, complicating surface-color representation.Source: §1 · p. 1
• Implicit representations can capture finer geometry and appearance, but their applicability to articulated hands and generalization to unseen poses remained unexplored.Source: §1 · pp. 1–2
• LISA is introduced as a neural hand model for shape and appearance, arbitrary-subject generalization, dense surface correspondences, image-based reconstruction, and animation.Source: §1 · p. 2
• LISA is trained with shape and appearance losses on multi-view RGB sequences annotated with coarse 3D hand-skeleton poses.Source: §1 · p. 2
• LISA disentangles shape, color, and pose representations, and its evaluation reports better hand reconstruction than baselines for point clouds and RGB images.Source: §1 · p. 2

• MANO captures hand shape and pose through blend shapes but is limited by low mesh resolution; neural mesh refinements and GHUM address aspects of this limitation.Source: §2 · p. 2
• Implicit geometry methods commonly model shape with SDFs or occupancy fields, and part-based implicit templates have been studied to improve learning efficiency.Source: §2 · p. 2
• Articulated implicit models use approaches including inverse skinning, per-bone occupancy, point-cloud registration, and learned canonicalization; LISA takes inspiration from NASA and explicitly models skinning weights for blending shape and color.Source: §2 · p. 2
• Neural appearance methods render learned volumes from multi-view images; LISA predicts SDF and appearance independently within each bone and blends them using corresponding skinning weights.Source: §2 · p. 2
• LISA’s architecture trains on multi-view RGB sequences with coarse skeleton poses and combines per-bone signed-distance and color predictions using learned skinning weights.Source: Figure 2 · p. 3
• Prior disentanglement methods factor properties such as shape, pose, identity, deformation, and expression; LISA proposes a generative hand representation with disentangled shape, pose, and appearance parameters.Source: §2 · pp. 2–3

• MANO represents a hand using pose parameters θ and shape parameters β, with a 16-joint skeleton and a 10-dimensional shape vector controlling a triangular mesh.Source: §3 · p. 3
• Linear Blend Skinning deforms canonical hand vertices using bone transformations and weights, and LISA uses MANO’s pose parameters and bone transformations.Source: §3 · p. 3
• NeRF maps 3D locations and viewing directions to color and density, while VolSDF derives volume density from an SDF and uses separate MLPs for geometry and color.Source: §3 · p. 3

4.1. Model definition

• Per-bone MLPs predict SDF and color in unposed coordinates, while a weight MLP blends them; conditioning weights on per-bone SDFs improves geometry learning.The predicted weights are analogous to classic LBS skinning weights.Source: §4.1 · p. 4

4.2. Training

• Shape β+ and color γ are user-specific 128-dimensional codes; MANO pose is fixed for the first 10% of training, then optimized to address annotation errors.Source: §4.2 · p. 5
• Training combines rendered-color, Eikonal SDF, supervised skinning-weight, and regularization losses, sampling SDF points on-surface and uniformly in the scene.Source: §4.2 · p. 5
• A 3DH prior pretrained on ~13k scans from 183 users uses surface-zero and normal-gradient supervision, then initializes full optimization and boosts performance.Source: §4.2 · p. 5

4.3. Inference

• For point-cloud reconstruction, LISA is fit to the data by minimizing an objective similar to the one used to learn the shape prior.Source: §4.3 · pp. 5–6
• Image fitting minimizes color and shape-pose regularization losses plus a 2D joint reprojection penalty, using foreground masks and detected 2D joints; multiview fitting also uses camera extrinsics.Source: §4.3 · p. 6

5.1. Datasets and baselines

• Training uses multi-view sequences from 27 users and a 3DH geometry prior; baselines include adapted NASA and NARF, MANO, and a pose-, shape-, and color-conditioned VolSDF.The InterHand2.6M training set contains 5,804 multi-view frames and 131k images; evaluations also use 3DH, MANO, and DeepHandMesh.Source: §5.1 · p. 6

5.3. Shape and color reconstruction from images

• On DeepHandMesh images, LISA-im outperforms all other image-trained methods in shape reconstruction, and LISA-full adds a boost most noticeable with one view.Table 2 reports shape distances and novel-view color PSNR for reconstructions from 1, 2, or 4 views.Source: Table 2 · p. 7, §5.3 · p. 7

5.2. Shape reconstruction from point clouds

• LISA-im consistently outperforms other image-trained methods on point-cloud shape reconstruction, while adding the geometric prior significantly boosts performance.LISA-geom, trained only with the prior, has lower errors than LISA-full; the paper attributes this to wrist closure learned from segmented training images, which increases measured error.Source: Table 1 · p. 6, §5.2 · p. 7

5.3. Shape and color reconstruction from images

• On InterHand2.6M color reconstruction, methods are comparable across PSNR, SSIM, and LPIPS; the authors suspect noisy training hand masks contribute.For DeepHandMesh novel-view PSNR, LISA-im is slightly superior except in the single-image case, where it is on par with NASA.Source: §5.3 · pp. 7–8, Table 3 · p. 7
• LISA-full reconstructs hands from in-the-wild images, including cases with partial object occlusion.Figure 4 also displays color, shape, and predicted skinning weights for reconstructions using one input RGB and two held-out views as rendering ground truth.Source: §5.3 · p. 8, Figure 4 · p. 8

5.4. Inference speed

• Fitting LISA takes approximately 5 minutes; mesh extraction at resolution 128^3 takes around 5 seconds, while novel-view rendering takes about one minute.Measurements used 1024 × 667 px images and one Nvidia Tesla P100 GPU.Source: §5.4 · p. 8

6. Conclusion

• LISA combines volume rendering approaches with a hand geometric prior to represent textured hands.Source: §6 · p. 8
• Sub-millimeter error in point-cloud fitting and large margins over evaluated baselines in image reconstruction demonstrate accurate 3D shape reconstruction.Source: §6 · p. 8
• The reported accuracy is not achievable with low-resolution parametric meshes such as MANO or single-person models such as DeepHandMesh.Source: §6 · p. 8
• Future work includes temporal consistency, reducing inference inputs, improving runtime efficiency, and adding high-frequency texture detail while retaining generalization.Source: §6 · p. 8