Source-linked AI summary
LatentFusion: End-to-End Differentiable Reconstruction and Rendering for Unseen Object Pose Estimation
Keunhong Park, Arsalan Mousavian, Yu Xiang, Dieter Fox
TL;DR
Existing 6D pose methods depend on object-specific 3D models and extra training, limiting their use for unseen objects. LatentFusion reconstructs a latent 3D representation from a few reference views and differentiably renders it for pose optimization. It reports competitive performance with supervised methods on LINEMOD while requiring no training on those objects.
Problem
6D pose estimation remains difficult to scale because existing methods require per-object 3D models and extensive training for unseen objects.
Method
LatentFusion reconstructs a latent 3D object representation from posed reference images, renders arbitrary views, and optimizes pose through differentiable rendering.
Results
The method performs on-par with supervised LINEMOD methods despite never seeing the evaluated objects during training.
Takeaways & Limitations
The framework supports zero-shot pose estimation with a small number of reference views and one universal network for unseen objects.
Takeaways & Limitations
Limited 128 × 128 image and 16×16×16 voxel resolutions can hinder pose accuracy for small or distant objects.
Abstract
from arXiv · showhide
Current 6D object pose estimation methods usually require a 3D model for each object. These methods also require additional training in order to incorporate new objects. As a result, they are difficult to scale to a large number of objects and cannot be directly applied to unseen objects. We propose a novel framework for 6D pose estimation of unseen objects. We present a network that reconstructs a latent 3D representation of an object using a small number of reference views at inference time. Our network is able to render the latent 3D representation from arbitrary views. Using this neural renderer, we directly optimize for pose given an input image. By training our network with a large number of 3D shapes for reconstruction and rendering, our network generalizes well to unseen objects. We present a new dataset for unseen object pose estimation--MOPED. We evaluate the performance of our method for unseen object pose estimation on MOPED as well as the ModelNet and LINEMOD datasets. Our method performs competitively to supervised methods that are trained on those objects. Code and data is available at https://keunhong.com/publications/latentfusion/.
1. Introduction
The paper targets 6D pose estimation for unseen objects without per-object 3D models or additional test-time training. It reconstructs a latent representation from a few posed reference images, renders arbitrary views, and introduces MOPED for zero-shot evaluation.
- Motivation: Current methods require a 3D model for each object, making model construction impractical at scale.High-quality scans require significant effort.
- Motivation: Existing methods require extensive training for lighting and occlusion variation, while multi-object networks lose accuracy as object count increases.Per-object networks improve this issue but do not scale, and unseen objects still require training.
- Method: The proposed network builds a 3D object representation from posed reference RGB images without 3D models or extra training for unseen objects.Pose is estimated by comparing the input image with rendered views and searching for the best pose.
- Method: A latent voxel representation is constructed from projected 2D features, then rotated and projected to render novel views.A decoder predicts depth and assigns color in the rendered view.
- Method: The network is trained on randomly textured ShapeNet objects under random lighting and directly optimizes pose through differentiable rendering.The method assumes the object is segmented.
- Benefits: The method emphasizes few-image capture, appearance robustness, and practicality through one universal model requiring no novel-object training.It uses latent representations rather than scanned models with baked appearance.
- Dataset: MOPED provides registered reference and test images for evaluating zero-shot pose estimation with casual real-world references.Existing benchmarks typically provide 3D models and rendered training image sequences instead.
2. Related Work
Prior pose-estimation methods use architectural prediction, image-feature recovery, or 3D-model rendering, but commonly depend on object-specific models and training. LatentFusion instead connects latent reconstruction, differentiable rendering, and multi-view reconstruction ideas for broader object generalization.
- Pose Estimation: Pose-estimation research includes direct architectural methods, 2D-feature prediction followed by pose recovery, and alignment of rendered 3D models to images.Rendering-based approaches include DeepIM and latent-object reconstruction methods.
- Pose Estimation: Latent reconstruction methods require separate auto-encoders for object categories and high-fidelity textured 3D models that are difficult to build.These requirements limit knowledge transfer and practical deployment.
- 3D Shape Learning: Category-level shape-learning methods represent geometry with voxels, SDFs, point clouds, or implicit functions but are limited to trained categories.The supplied passage introduces this limitation without describing LatentFusion’s response.
- Differentiable Rendering: Differentiable rendering propagates gradients through rendering to optimize parameters such as pose or appearance, but existing methods require a 3D mesh.They also face difficulty modeling complex appearance.
- Multi-View Reconstruction: Multi-view reconstruction methods motivate LatentFusion, which extends space carving into latent space while addressing learned representations from unaligned views.The cited passage situates the method among dense fusion and learned multi-view reconstruction approaches.
3. Overview
LatentFusion reconstructs a flexible latent representation from a small collection of reference images and uses a fully differentiable renderer to optimize camera parameters without additional training or codebook generation. Its multi-view-consistent representation avoids category-specific shape priors, supporting generalization beyond the training-object distribution.
- Overview: The reconstruction pipeline takes a small collection of reference images and produces a representation renderable from novel viewpoints.It uses multi-view consistency without category-specific shape priors.
- Overview: The pose-estimation stage directly optimizes camera parameters through fully differentiable rendering without additional training or codebook generation.This formulates 6D pose estimation using the learned neural renderer.
- Camera Model: The camera parameter set is θ = {R, t, c}, combining rotation, translation, and viewport cropping.The camera uses a perspective pinhole model with intrinsic and homogeneous extrinsic matrices.
4. Neural Reconstruction and Rendering
The reconstruction pipeline builds a canonical latent 3D voxel representation from posed reference views, then renders arbitrary views using depth prediction and image-based color synthesis.
- Modeling: The modeling pipeline predicts per-view feature volumes and transforms them into a canonical object-space volume using multi-view consistency.Each view is lifted from 2D features into a 3D camera-frustum volume before rigid transformation and fusion.
- Modeling: Each per-view feature volume represents a depth-bounded camera frustum defined by the crop parameter c and object-depth range.The crop parameter c determines the image-plane bounds, while depth determines the remaining frustum extent.
- Modeling: Per-view volumes are fused either by channel-wise average pooling or by sequential recurrent integration that can retain or ignore view-specific features.The recurrent fusion module slightly improves reconstruction accuracy over pooling by facilitating comparisons between views.
- Rendering: The renderer transforms the canonical volume to a query camera, projects it into 2D features, and predicts depth and a segmentation mask.The rendering network uses 3D and 2D U-Nets, with separate output branches for depth and mask prediction.
- Rendering: Color rendering first reprojects reference images using predicted depth, then uses neural blend weights based on depth, reprojected images, and view similarity.This image-based rendering design avoids blurry intermediate views and handles invalid pixels caused by occlusions.
- Training: Training uses textured ShapeNet models with randomized lighting and views, while inputs are cropped and scaled to 128 × 128 before optimization.The network is trained with 16 reference and 16 target poses per sampled object using Adam for 1.5M iterations.
5. Object Pose Estimation
Pose estimation uses the differentiable reconstruction-and-rendering pipeline to optimize camera pose from an RGB image, mask, and depth input. A coarse estimate initializes gradient-based refinement over rotation, translation, and viewport parameters.
- Pipeline: The pose-estimation pipeline first obtains a coarse pose through forward inference and then refines it with gradient optimization.The two-stage procedure addresses the non-convex pose space and reduces the risk of poor local minima.
- Formulation: Pose is parameterized by rotation, translation, and viewport crop, with rotation encoded as a quaternion and input consisting of RGB image, mask, and depth.The viewport parameter provides scale invariance while efficiently cropping the object for reconstruction.
- Losses: The objective combines depth L1, mask binary cross-entropy, IoU, and latent losses to measure agreement between predictions, input observations, and latent representations.The latent loss compares rendered and modeled features directly for a given image and pose without training on the object.
- Initialization: Coarse initialization estimates translation from the mask bounding box and depth, samples orientations with a Fibonacci lattice, and optimizes candidates using the cross entropy method.Random yaw sampling supplements evenly sampled azimuth and elevation angles to cover possible orientations.
- Qualitative analysis: Figure 5 compares target images, ground-truth depth, optimized predicted depth, and their L1 error, illustrating that low depth error can coexist with high angular error.The example highlights a mismatch between depth-based reconstruction quality and rotational pose accuracy.
- Qualitative analysis: Figure 6 presents qualitative results on the MOPED dataset.The figure is used to inspect visual pose-estimation behavior on MOPED examples.
- Optimization: End-to-end differentiability enables joint Adam optimization of rotation, translation, and viewport parameters using the rendering network alone.The refinement step does not use the modeling network or image-based rendering network.
6. Experiments
The experiments evaluate unseen-object pose estimation on LINEMOD, ModelNet, and the new MOPED dataset, including generalization, robustness, and design choices. The method performs competitively or better than supervised and model-based baselines, while performance depends on resolution, reference-view count, and loss design.
- LINEMOD: The method performs on-par with supervised methods on LINEMOD despite never seeing the evaluated objects during training.It uses 16 reference views per object at inference time and reports ADD recall below 10% of object diameter.
- ModelNet: The method achieves state-of-the-art average results across all reported ModelNet metrics for seven unseen object categories.The metrics are (5° , 5cm), ADD within 10% of object diameter, and Proj.2D within 5 pixels.
- Limitations: Limited 128 × 128 image and 16×16×16 voxel resolutions hinder performance for small or distant objects, especially under the (5° , 5cm) metric.Small depth changes can disproportionately affect estimated rotation relative to the losses.
- MOPED: MOPED contains 11 objects with registered RGB-D captures, multiple environments, and approximately 300 evaluation images per object.The dataset provides AUC for ADD, ADD-S, and Proj.2D over specified distance thresholds.
- MOPED: The method achieves superior ADD and ADD-S performance on MOPED compared with PoseRBPF, which uses object-specific encoders and reconstructed textured meshes.The combined depth and latent losses improve results on textured objects, while depth-only optimization performs better for geometrically asymmetric objects.
- Ablation Studies: Accuracy increases with more reference views, but using more than eight views yields only marginal gains.This indicates that good pose estimation does not require many reference images.
- Ablation Studies: ConvGRU view fusion performs better than average pooling for some objects, although average performance is similar.The comparison evaluates alternative strategies for aggregating latent representations across reference views.
7. Conclusion
The paper presents a framework that builds latent 3D object representations from a small number of reference images and renders them from arbitrary viewpoints. This supports zero-shot pose estimation for unseen objects without object-specific training or detailed 3D models.
- Conclusion: The framework builds 3D object representations at inference time from a small number of reference images and renders them from arbitrary 6D viewpoints.The reconstruction and neural-rendering components form the paper’s central framework.
- Conclusion: Training on thousands of randomly textured shapes under varied lighting enables generalization to unseen objects without additional training.The conclusion describes this as robust generalization across unseen objects.
- Conclusion: Zero-shot pose estimation requires only a small number of reference views and does not require a detailed 3D model or training a new network.A single universal network is used for all objects, including unseen ones.