Source-linked AI summary
Leveraging Photometric Consistency over Time for Sparsely Supervised Hand-Object Reconstruction
Yana Hasson, Bugra Tekin, Federica Bogo, Ivan Laptev, Marc Pollefeys, Cordelia Schmid
TL;DR
Joint 3D hand-object reconstruction from monocular RGB videos is difficult because interactions create mutual occlusions and 3D annotations are expensive to collect. The paper uses sparse-frame supervision with differentiable optical-flow warping and a self-supervised photometric loss, achieving state-of-the-art benchmark results and improved accuracy in low-data regimes.
Problem
Joint hand-object pose estimation is challenging under mutual occlusions, while existing methods require large fully supervised datasets with costly, difficult 3D annotations.
Method
The method jointly reconstructs hand and object pose and shape from color videos, using optical-flow warping between adjacent frames to impose photometric consistency from sparse annotations.
Results
The approach achieves state-of-the-art 3D hand-object reconstruction benchmark results and improves pose estimation accuracy when only a few percent of the dataset is annotated.
Takeaways & Limitations
Neighboring video frames provide additional supervision, reducing reliance on large labeled datasets for hand-object reconstruction.
Takeaways & Limitations
Photometric supervision can fail when the consistency assumption is violated, such as during fast motions or illumination changes.
Abstract
from arXiv · showhide
Modeling hand-object manipulations is essential for understanding how humans interact with their environment. While of practical importance, estimating the pose of hands and objects during interactions is challenging due to the large mutual occlusions that occur during manipulation. Recent efforts have been directed towards fully-supervised methods that require large amounts of labeled training samples. Collecting 3D ground-truth data for hand-object interactions, however, is costly, tedious, and error-prone. To overcome this challenge we present a method to leverage photometric consistency across time when annotations are only available for a sparse subset of frames in a video. Our model is trained end-to-end on color images to jointly reconstruct hands and objects in 3D by inferring their poses. Given our estimated reconstructions, we differentiably render the optical flow between pairs of adjacent images and use it within the network to warp one frame to another. We then apply a self-supervised photometric loss that relies on the visual consistency between nearby images. We achieve state-of-the-art results on 3D hand-object reconstruction benchmarks and demonstrate that our approach allows us to improve the pose estimation accuracy by leveraging information from neighboring frames in low-data regimes.
1. Introduction
The paper targets joint 3D hand-object reconstruction from monocular RGB videos when dense 3D supervision is costly and sparse annotations are available. It introduces temporal photometric supervision to propagate information from annotated frames and reports accurate reconstruction and state-of-the-art benchmark performance.
- Joint hand-object pose estimation is difficult because manipulation causes mutual occlusions, while unified 3D modeling remains important for augmented reality, robotics, and surveillance.
- Existing hand-object methods require fully supervised datasets with 3D annotations, whose collection is challenging, costly, and error-prone.
- The proposed weakly supervised approach jointly estimates hand and object pose and reconstructs their 3D shape using videos annotated in only sparse frames.
- The method introduces a photometric loss based on optical flow between adjacent images, warping one frame to another inside the network to exploit visual consistency.
- The approach reliably predicts interacting hand-object poses in 3D, densely reconstructs their shape, and achieves state-of-the-art results on 3D hand-object reconstruction benchmarks.
2. Related Work
Prior work largely estimates hand or object pose separately, or models hand-object interactions with substantial supervision and restricted capture settings. This paper relates its approach to motion- and photometric-cue methods while focusing on complex monocular hand-object interactions.
- Hand and object pose estimation: Object pose methods commonly predict 2D projections of 3D bounding-box points and recover 6D pose using PnP or iterative refinement, often requiring object models and labeled data.
- Hand and object pose estimation: Hand pose methods increasingly regress 3D surfaces or parametric MANO models, but many do not address hand-object interactions.
- Hand-object datasets: Recent hand-object datasets use manual annotation, depth tracking, motion capture, or multi-view setups, while monocular CNN methods remain fully supervised and do not exploit temporal information.
- Supervision using motion and photometric cues: Motion-based self-supervision has been used for human body pose through optical-flow correspondences, annotation propagation, and learned pose warping.
- Supervision using motion and photometric cues: The proposed method enforces photometric consistency between adjacent-frame pose estimates for articulated hand-object interactions rather than mostly rigid scenes.
3. Method
The method reconstructs hand-object shape and pose from monocular RGB videos using sparse annotations and photometric consistency across frames. A feed-forward network predicts per-frame reconstructions, while differentiable flow-based warping supplies self-supervision for unannotated frames.
- Input and output: The model takes monocular RGB videos with a known object model and sparse annotations, returning hand and object 3D vertices, shapes, and poses per frame.Hand reconstruction uses MANO, while the object is represented by a provided 3D mesh model.
- Photometric supervision: The photometric loss propagates supervision from annotated frames to unannotated intermediate frames by enforcing temporal consistency between neighboring images.This uses temporal continuity in videos as a self-supervisory signal for dense 3D hand-object reconstruction.
- Photometric supervision: Corresponding mesh-vertex displacements are differentiably rendered into optical flow, which warps the unannotated image into the reference frame for pixel-space consistency.The method back-projects mesh vertices using camera intrinsics, interpolates flow over visible mesh triangles, and samples the target image according to predicted displacements.
- Photometric supervision: A visibility mask restricts photometric supervision to pixels inside the reprojected silhouette that remain stable under a cyclic forward-and-backward flow check.The loss is minimized with respect to the estimated hand and object vertices in the unannotated frame.
- Training losses: The image-space loss can be combined with reconstruction losses because all operations are differentiable, targeting articulated and rigid hand-object motion without requiring UV parameterization.The reconstruction framework includes MANO pose and shape regularization to discourage unnatural rotations and extreme deformations.
- Network: A single feed-forward network regresses hand and object reconstructions independently for each frame, using an RGB encoder with separate hand and object prediction components.The network predicts MANO parameters for the hand and 6D object pose, using a computationally efficient ResNet-18 backbone.
4. Evaluation
The evaluation covers joint hand-object reconstruction on FPHAB and HO-3D, comparing single-frame accuracy, unified training, and photometric supervision under sparse annotation. The method achieves competitive or state-of-the-art pose results, with photometric consistency providing its clearest gains in low-data regimes.
- Datasets and metrics: The framework is evaluated on FPHAB and HO-3D using hand and object pose annotations with 3D and 2D error metrics.Reported metrics include mean 3D joint or vertex/corner errors, mean 2D reprojection errors, and PCK.
- Single-frame reconstruction: The single-frame model directly reconstructs dense meshes while inferring hand and object poses from color images.This enables evaluation against state-of-the-art pose-estimation methods on both datasets.
- Single-frame reconstruction: Object pose accuracy exceeds Tekin et al. [47] on FPHAB, while hand pose error averages 18 mm and remains 2.6 mm behind that baseline.The model additionally predicts detailed hand shape and global hand position in camera space.
- Single-frame reconstruction: The model outperforms on both hand and object pose estimation on the early HO-3D release.Figure 5 reports hand PCK and object mean 2D reprojection error.
- Unified training: Sharing the encoder for joint hand-object training incurs a minor performance cost in hand and object pose accuracy on FPHAB.The comparison considers jointly trained and separately trained pose-estimation models.
- Photometric supervision: 20% of FPHAB frames suffice to reach densely supervised performance, while using less than 1% yields a 7-pixel average improvement from photometric consistency.The benefit becomes most noticeable as annotated data become scarcer, although generalization error decreases at very low annotation fractions.
- Photometric supervision: Photometric consistency can refine inaccurate poses under large motions, but failures occur with important motion blur, large occlusions, fast motion, or violated photometric assumptions.HO-3D qualitative results also show improved 2D reprojection while some camera-plane rotation errors remain uncorrected or are introduced.
5. Conclusion
The paper concludes that photometric consistency enables dense hand-object reconstruction from monocular color images with sparse supervision. It improves pose estimation by exploiting similarities between annotated and neighboring unannotated frames, while future work targets broader scene constraints.
- Conclusion: The method performs dense 3D reconstruction of hands and objects from monocular color images.The framework is designed for joint hand-object reconstruction and pose estimation.
- Conclusion: Photometric consistency between sparsely annotated and neighboring unannotated frames provides additional supervision.The conclusion identifies this as the basis for the sparse-supervision approach.
- Future work: Future work will explore 3D interpenetration and scene-interaction constraints, with possible extension to the full human body and environment surfaces.These directions aim toward full human-centric scene understanding.
Appendix
The appendix supplements the main method with implementation details, dataset splits, and the cyclic consistency check used to obtain valid photometric-loss masks.
- Appendix contents: The appendix provides implementation details and specifies the HO-3D training and test splits used in the experiments.It also describes how valid masks are computed for the photometric consistency loss.
- Appendix contents: A cyclic consistency check is used to compute the valid mask for the photometric consistency loss.The check is presented as an appendix component of the loss implementation.
A. Implementation details
The implementation uses a ResNet18-based network with separate pose and shape branches, Adam optimization, sparse-data pretraining, and photometric-loss fine-tuning. The forward pass runs in real time at 34 frames per second on a Titan X GPU.
- Architecture: ResNet18 features feed separate branches for object translation and rotation, hand translation, and 28 MANO parameters.The MANO parameters represent global hand rotation, articulated pose, and shape deformation.
- Training: Training uses Adam with learning rate 5 · 10^-5 and ImageNet-initialized ResNet weights.Batch-normalization weights are frozen at their ImageNet initialization for improved training stability.
- Training: Models are pretrained without consistency loss on data fractions, then fine-tuned with consistency loss for 200 epochs.Smaller training subsets require more pretraining epochs to reach convergence.
- Runtime: 34 frames per second is achieved for the forward pass on a Titan X GPU.The reported runtime concerns inference through the network.
B. HO-3D subset
This section identifies a subset of released sequences and describes the regression-branch architecture, alongside a skeleton/no-skeleton deformation comparison.
- Dataset subset: The selected subset contains 14 sequences depicting manipulation of the mustard bottle and cracker box.The passage states that these sequences come from the earlier release of the dataset.
- Dataset subset: The listed training sequences include SM2, SM3, SM4, SM5, MC4, MC6, SS1, SS2, SS3, SM2, MC1, and MC5.
- Regression architecture: The hand and object branches regress pose and shape parameters from 512-dimensional features using fully connected linear layers.
- Skeleton adaptation: Figure 10 compares predicted shape deformations with and without the skeleton adaptation layer on FPHAB.
C. Cycle consistent visibility check
The visibility check filters photometric-loss locations by testing whether bidirectional flow warping returns pixels close to their starting positions.
- Bidirectional warping: The check warps each pixel from the estimated frame to the reference frame and back using opposite-direction flows.
- Visibility criterion: Pixels whose round-trip displacement exceeds 2 pixels are excluded from the loss.
- FHB effect: With 1% of FHB data used as reference frames, the check discards 3.3% of masked pixels.
D. Skeleton Adaptation
The skeleton adaptation layer addresses mismatches between FPHAB joint definitions and MANO while affecting joint accuracy and mesh realism differently.
- Motivation: FPHAB joint locations do not exactly match the corresponding locations in the MANO hand model.
- Joint predictions: The skeleton adaptation layer yields marginal improvements in average joint predictions.
- Mesh realism: Although MANO can deform for different skeleton conventions, those deformations reduce the realism of reconstructed meshes.