What's in your hands? 3D Reconstruction of Generic Objects in Hands

• The goal is geometric understanding of everyday hand-object interaction beyond assigning labels such as “touching” or “holding.”Source: §1 · pp. 1–2
• The task is ill-posed because of limited data and mutual or self-occlusion, while template-based methods cannot reconstruct unknown objects and struggle with varied in-the-wild shapes.Source: §1 · p. 2
• Hand articulation provides cues to local object geometry, so the method estimates hand pose first and reconstructs object shape from pose and image in a normalized hand-centric frame.Source: §1 · p. 2
• Across three synthetic and real-world datasets, articulation-conditioned shape prediction consistently outperforms explicit and implicit HOI baselines and reconstructs diverse object shapes.Source: §1 · p. 2
• Test-time optimization that encourages interaction between predicted hand and object can further improve the initial hand pose estimate.Source: §1 · p. 2

• Hand pose methods either lift detected 2D keypoints to 3D joints or skeletons [10] [34] [45] [47] [52] [74], or use statistical models such as MANO; this work uses a model-based system [56].Source: §2 · p. 2
• Generic object reconstruction spans category-specific and cross-category methods with varied representations, including neural implicit fields [9] [43] [48], but typical systems assume isolated, unoccluded images.Source: §2 · pp. 2–3
• Prior hand-held-object methods commonly use instance-specific templates—typically around 10 objects—in controlled settings, reducing reconstruction to 6D pose estimation [17] [60] [62] [64].Source: §2 · p. 3
• Model-free methods for unknown objects predict meshes [31] or joint implicit fields, but infer hand pose and object shape independently; this work conditions shape on articulation in a normalized wrist frame.Source: §2 · p. 3

3. Method

• A wrist-frame implicit network predicts object signed distance from query-point visual features and articulation-aware coordinates, enabling shape inference without a known object template at inference.The predicted zero-level set defines the object surface, and the normalized frame supports learning relations invariant to global transformations.Source: §3 · p. 3, §3.2 · p. 3, §3.3 · p. 4

3.1. Preliminary: Hand Reconstruction

• The off-the-shelf hand system regresses 45-dimensional articulation, global rotation and translation, and a weak-perspective camera to project canonical wrist-frame points into the image.The method converts the predicted weak-perspective camera to full perspective to account for large perspective effects.Source: §3.1 · p. 3

3.2. Articulation-conditioned SDF

• The SDF network combines global and pixel-aligned multiscale image features with query-point positions relative to articulated joints, then decodes these inputs into signed distance.The global feature supplies context, while local features align predictions with visual observations; joint-relative encoding is motivated as more robust to hand-pose error than using articulation parameters directly.Source: §3.2 · pp. 3–4

3.3. Training

• Training uses known hand poses and object shapes to supervise SDF values at sampled points, with 95% sampled near object surfaces and the remainder uniformly in space.An eikonal term regularizes the SDF objective.Source: §3.3 · p. 4

3.4. Refining hand pose

• At inference, hand pose and object shape can be jointly optimized by encouraging contact and penalizing hand-surface points predicted inside the object.The SDF depends on articulation; the method fixes it during pose optimization and updates it once using the final optimized pose.Source: §3.4 · p. 5

4.1. Results on ObMan

• On ObMan, the method outperforms baselines by a large margin in F-score, especially at 5 mm, and has lower Chamfer distance than GF but not HO.The authors attribute HO’s Chamfer advantage as possibly reflecting its explicit Chamfer training and edge-length regularizer, which discourages large displacements from a sphere.Source: Table 1 · p. 6, §4.1 · p. 6
• On ObMan, the method better preserves sharp object details, aligns more closely with visible inputs, and reconstructs occluded surfaces than the baselines.Baselines often lose details such as phone corners or omit parts of an object surface.Source: Figure 3 · p. 5, §4.1 · p. 6

4.2. Evaluation on real-world datasets

• On real-world data, the method reconstructs varied shapes more accurately, while models trained on ObMan or MOW transfer to HO3D without finetuning and outperform HO3D-trained baselines.MOW has 350 training images versus 21K for the synthetic dataset; the authors point to diversity as important for in-the-wild training.Source: §4.2 · p. 6, Table 2 · p. 6

4.3. Ablations

• Removing explicit hand articulation degrades reconstruction and doubles intersection volume; conditioning on articulation better respects physical relations such as nonpenetration and fingertip contact.The comparison uses a variant conditioned only on pixel-aligned image features.Source: §4.3 · pp. 6–7, Table 3 · p. 7, Figure 6 · p. 7
• Concatenating query points with pose parameters performs comparably with ground-truth poses but degrades with predicted poses, even below the no-articulation variant.The authors attribute this to overfitting pose parameters that are constant within each example; their articulation-aware encoding generalizes better.Source: §4.3 · pp. 7–8, Table 4 · p. 8
• The method still outperforms baselines when predicted hand poses have twice the error, across Gaussian noise and structured prediction noise.Structured noise is produced by interpolating or extrapolating between true poses and off-the-shelf predictions.Source: §4.3 · p. 8, Table 5 · p. 8, Figure 7 · p. 8
• Test-time refinement improves both hand and object reconstruction, reduces simulation displacement alongside intersection volume, and improves object error and displacement further with ground-truth articulation.Simulation displacement measures how much the object slips from the hand in a fixed-duration simulation.Source: §4.3 · p. 8, Table 6 · p. 8, Figure 8 · p. 8

5. Conclusion

• The method infers implicit 3D shapes of generic hand-held objects using predicted hand pose as an object-inference cue.The authors report that articulation complements visual cues, especially when the object is occluded.Source: §5 · p. 8
• The method cannot be directly adapted to reconstruct dynamic grasps from video when object consistency across varying articulation is required.Source: §5 · p. 8
• Training requires 3D ground truth; the authors identify extending the method with differentiable rendering as future work.Source: §5 · p. 8

A.1. Camera conversion

• The off-the-shelf system predicts a weak-perspective camera with scale factor s and 2D translation tx, ty.Source: §A.1 · p. 12
• The method converts the predicted weak-perspective camera to a full-perspective camera to account for large perspective effects.It does so by translating the final mesh by offset (tx, ty, f/s).Source: §A.1 · p. 12
• The method projects a query point in the wrist frame to the image.Source: §A.1 · p. 12

A.2. Coordinate Transformation

• The articulation embedder takes articulation parameters and a wrist-frame point, then outputs an encoding concatenating coordinates relative to every joint.Source: §A.2 · p. 12
• Forward kinematics maps a wrist-frame point to each joint coordinate through transformation T(θA).Source: §A.2 · p. 12
• The wrist-to-joint transformation is the product of transformations along the kinematic chain, with bone lengths predefined in MANO models.Source: §A.2 · p. 12

A.3. Training

• The model is trained with Adam at learning rate 1e −4 on 8 GPUs, using batch size 64.Source: §A.3 · p. 12
• The model is trained on ObMan for 200 epochs, then fine-tuned on HO3D and RHOI for 50k iterations each.Source: §A.3 · p. 12
• The eikonal term coefficient is 0.1.Source: §A.3 · p. 12

A.4. HO3D dataset split

• HO-3D contains 103k annotated images of 10 subjects interacting with 10 common YCB objects.The original train-test split partitions interaction sequences.Source: §A.4 · p. 12
• The original test set covers four objects, three also seen in training, and all four are cuboidal.The three overlapping objects are bleach cleanser, mustard bottle, and meat can.Source: §A.4 · p. 12
• A custom split holds out one video sequence per object for testing, including to assess less common shapes such as the power drill.The sequences used for this split are listed in Table 7.Source: §A.4 · p. 12

B. Qualitative Results

• A figure compares reconstructions with and without explicit hand-pose conditioning on ObMan, HO3D, and RHOI.The passage identifies this comparison as Figure 13.Source: B. Qualitative Results · p. 12, Figure 14 · p. 19
• Reconstructions from models trained only on ObMan and RHOI are visualized.The passage identifies these as Figure 14.Source: B. Qualitative Results · p. 12
• Hand-object reconstructions before and after test-time refinement are shown in the image frame and two novel views.The visualizations are identified as Figure 15.Source: B. Qualitative Results · p. 12, Figure 15 · p. 20
• Reconstructions are visualized against two baselines on ObMan, HO3D, and RHOI from the image frame and a novel view.The visualizations are shown in Figures 9–12.Source: B. Qualitative Results · p. 12, Figure 9 · p. 13, Figure 10 · p. 14
• Cross-dataset results on HO3D are shown for models pretrained on ObMan and RHOI.The figure caption describes these as quantitative results.Source: Figure 13 · p. 18