Source-linked AI summary
ShapeR: Robust Conditional 3D Shape Generation from Casual Captures
Yawar Siddiqui, Duncan Frost, Samir Aroudj, Armen Avetisyan, Henry Howard-Jenkins, Daniel DeTone, Pierre Moulon, Qirui Wu, Zhengqin Li, Julian Straub, Richard Newcombe, Jakob Engel
TL;DR
Casual 3D capture contains occlusions, clutter, noise, and poor viewpoints that challenge methods developed for clean inputs. ShapeR combines sparse SLAM points, posed multiview images, and captions to generate complete metric shapes, outperforming existing approaches in these settings.
Problem
Existing 3D shape generators assume clean, well-segmented, and unoccluded inputs, whereas casual captures contain occlusions, clutter, noise, low resolution, and poor viewpoints.
Method
ShapeR conditions a rectified-flow model on sparse SLAM points, posed multiview images, and machine-generated captions to generate complete object shapes.
Results
ShapeR reconstructs complete, metric shapes without segmentation inputs and remains robust across casual captures.
Takeaways & Limitations
ShapeR provides automatic object-centric reconstruction and a benchmark for evaluating 3D reconstruction under realistic, in-the-wild capture conditions.
Takeaways & Limitations
Reconstructions may lack detail with low-fidelity or few-view inputs, include adjacent structures, and fail when upstream detection misses objects or mislocalizes them.
Abstract
from arXiv · showhide
Recent advances in 3D shape generation have achieved impressive results, but most existing methods rely on clean, unoccluded, and well-segmented inputs. Such conditions are rarely met in real-world scenarios. We present ShapeR, a novel approach for conditional 3D object shape generation from casually captured sequences. Given an image sequence, we leverage off-the-shelf visual-inertial SLAM, 3D detection algorithms, and vision-language models to extract, for each object, a set of sparse SLAM points, posed multi-view images, and machine-generated captions. A rectified flow transformer trained to effectively condition on these modalities then generates high-fidelity metric 3D shapes. To ensure robustness to the challenges of casually captured data, we employ a range of techniques including on-the-fly compositional augmentations, a curriculum training scheme spanning object- and scene-level datasets, and strategies to handle background clutter. Additionally, we introduce a new evaluation benchmark comprising 178 in-the-wild objects across 7 real-world scenes with geometry annotations. Experiments show that ShapeR significantly outperforms existing approaches in this challenging setting, achieving an improvement of 2.7x in Chamfer distance compared to state of the art.
1. Introduction
ShapeR addresses the sharp performance drop of object-level 3D generative models on casual captures by conditioning a rectified-flow model on complementary metric, visual, and language modalities. It improves robustness through multimodal augmentation and curriculum training, and introduces an in-the-wild benchmark for realistic evaluation.
- Motivation: Object-level generative models produce high-fidelity shapes from clean, segmented, unoccluded inputs but perform significantly worse on casual captures.Casual captures involve natural, non-scanning trajectories and challenges such as clutter, poor viewpoints, noise, motion blur, and occlusions.
- Method: ShapeR generates robust 3D shapes from casually captured sequences using sparse metric point clouds, posed multiview images, and machine-generated captions.Off-the-shelf SLAM provides sparse point clouds and camera poses, while 3D instance detection extracts object-centric information.
- Training: ShapeR improves robustness with on-the-fly augmentation across modalities, implicit image segmentation from 3D instance points, and a two-stage curriculum spanning isolated objects and scenes.The curriculum begins with large, diverse object-centric datasets and addresses limitations of contrived isolated-object settings.
- Evaluation: 178 objects across 7 diverse scenes form ShapeR’s benchmark of in-the-wild sequences with posed multiview images, SLAM point clouds, and complete 3D shape annotations.The dataset targets realistic occlusions, clutter, and variable resolution and viewpoints.
- Contributions: ShapeR aims to unify generative 3D shape modeling with metric 3D scene reconstruction by producing complete, high-fidelity object shapes while preserving real-world metric consistency.The authors state that code, model weights, and the evaluation dataset will be released.
2. Related Works
Prior work spans non-object-centric reconstruction, conditional object reconstruction, and object-centric scene reconstruction, but commonly prioritizes appearance, assumes clean inputs, or lacks metric grounding. ShapeR instead uses sparse metric point clouds, posed images, and captions for robust reconstruction under occlusion, clutter, and viewpoint variation.
- Non Object-centric Reconstruction: NeRF, 3DGS, and extensions achieve high-fidelity view synthesis but prioritize appearance over geometric accuracy.SDF-based implicit methods improve geometric faithfulness while maintaining view quality.
- Conditional Object Reconstruction: Conditional reconstruction progressed from class-specific image- or point-cloud-conditioned models to text-conditioned diffusion and large reconstruction models.These approaches moved beyond fixed classes and scaled image-to-3D reconstruction with integrated mesh generation.
- Conditional Object Reconstruction: Most conditional reconstruction methods require clean, well-segmented inputs, lack metric grounding from single images, and struggle in real-world scenarios.ShapeR addresses these limitations with multimodal conditioning from sparse metric point clouds, posed images, and captions.
- Object-centric Scene Reconstruction: Object-centric scene reconstruction evolved from joint detection and completion or CAD retrieval toward single-view object and scene-layout reconstruction using diffusion priors and open-vocabulary detection.Earlier approaches often produced incomplete or mismatched geometry, while later methods were typically limited to specific classes.
3. Method
ShapeR generates object-centric metric 3D shapes from casually captured image sequences by conditioning a rectified-flow model on multimodal geometric, visual, positional, and textual cues. Its robustness is developed through compositional corruptions and curriculum training from diverse object meshes to realistic scene crops.
- Generative model: A rectified-flow transformer denoises latent VecSets representations conditioned on posed images, SLAM points, captions, and projected-point masks, then decodes them into an object shape.The architecture uses a FLUX DiT design with modality-specific encoders, including sparse 3D ResNet point encoding, frozen DINOv2 image features, Plücker pose encodings, and a 2D convolutional projection-mask encoder.
- Generative model: The latent autoencoder represents meshes with uniformly sampled surface points and edge-salient points, producing variable-length latent codes with feature width d = 64.The decoder predicts signed distance values for 3D query points, supporting reconstruction of the final shape from the decoded implicit representation.
- Training strategy: Training first uses over 600K artist-created meshes with on-the-fly compositional augmentations that corrupt images and SLAM points to simulate casual captures.Image augmentations include background compositing, occlusions, fog, resolution degradation, and photometric perturbations, while point clouds receive trajectory truncation, dropout, Gaussian noise, and occlusion.
- Training strategy: A second curriculum stage fine-tunes ShapeR on object crops from realistic synthetic scenes containing image occlusions, SLAM noise, and inter-object interactions.This stage addresses capture complexity not fully represented by the large-scale object-centric pretraining data.
- Input processing: ShapeR extracts sparse metric SLAM points, posed images, projection masks, and vision-language captions for each detected object to form its conditioning set.The object conditioning set is normalized before generation, and the inputs capture both geometric and appearance information across the sequence.
- Inference: Flow sampling integrates the learned denoising dynamics from Gaussian noise and reconstructs metrically consistent meshes aligned with the input sequence’s real-world scale and placement.The final mesh is obtained by applying Marching Cubes to the decoded signed distance representation.
4. Experiments
ShapeR is evaluated against reconstruction and generation baselines on a dedicated dataset using complementary geometry metrics. Results show that it reconstructs complete metric shapes robustly from casual captures, while ablations identify SLAM points, augmentations, curriculum training, and 2D point prompting as important components.
- Posed Multi-view to 3D: ShapeR reconstructs complete metric shapes without segmentation inputs and remains robust across casual captures, unlike methods hindered by occlusions or imperfect masks.Monolithic scene methods produce incomplete objects under occlusion, while segmentation-based methods degrade with imperfect masks.
- Effect of SLAM Points: SLAM points significantly improve robustness by encoding aggregated shape information across the sequence, especially when visual cues are weak.Image-only inputs produce reasonable reconstructions, but the complementary geometric information from SLAM points improves performance under weak visual evidence.
- Effect of Augmentations: Point-cloud and image augmentations are both critical for robust real-world reconstruction under noise and partial observations.Removing either augmentation degrades reconstruction, while synthetic occlusion and background augmentation reduce dependence on noisy foreground masks.
- Effect of Two-stage Curriculum Training: Fine-tuning on realistic scene data after large-scale object-centric pretraining substantially improves robustness and generalization to casual captures.The two-stage curriculum combines object-centric pretraining with realistic scene fine-tuning.
- Effect of 2D Point Mask Prompting: 2D point mask prompting mitigates reconstruction of adjacent objects and produces cleaner reconstructions.The cues guide DINO features toward the intended object.
5. Conclusion
ShapeR is a multimodally conditioned rectified flow model for robust 3D shape generation from casually captured sequences. It combines posed images, sparse SLAM points, textual cues, curriculum training, and augmentations, and introduces an evaluation dataset for casual-capture reconstruction.
- 5. Conclusion: ShapeR uses posed images, sparse SLAM points, and textual cues to reconstruct objects accurately and completely without explicit segmentation.The model is a multimodally conditioned rectified flow approach for casually captured sequences.
- 5. Conclusion: Large-scale training, extensive augmentations, and a two-stage curriculum enable strong generalization to real-world scenarios.These techniques support robustness to natural, casually captured environments.
- 5. Conclusion: ShapeR introduces the ShapeR Evaluation Dataset as a benchmark for object-centric reconstruction under casual capture.The benchmark supports scalable and automatic 3D reconstruction in natural environments.
A. ShapeR Evaluation Dataset
The ShapeR evaluation dataset addresses limitations of existing real-world reconstruction datasets by focusing on casually captured indoor sequences and annotated object geometry. It contains 178 objects across 7 real indoor sequences, with data collected using Project Aria glasses and SLAM-based camera information.
- Dataset motivation: Existing datasets either provide complete geometry only under controlled, uncluttered, studio-like conditions or represent less object-focused real-world settings.Examples of controlled datasets include Digital Twin Catalog, StanfordORB, and Google Scanned Objects.
- Data collection: Sequences are captured with Project Aria Gen 1 or Gen 2 glasses while annotators casually walk through indoor scenes.RGB and CV camera images are processed by Aria Machine Perception Services to extract SLAM points and camera parameters.
- Dataset scale: 178 objects are annotated across 7 real indoor sequences spanning a range of categories.The dataset includes 3D shape annotations for selected objects.
B. Additional Experiments
Additional experiments show that ShapeR remains robust across increasingly casual and cluttered settings, outperforming single-image and baseline methods while supporting metric monocular reconstruction without retraining. Evaluations span SAM 3D Objects, ScanNet++ and Replica, DTC Active and Passive, robustness trends, and monocular image-to-3D reconstruction.
- Comparison against SegmentAnything 3D Object: ShapeR maintains metrically accurate shapes and robust object layouts in cluttered, multi-object scenes where SAM 3D Objects deteriorates.ShapeR uses posed multi-view images and SLAM points, whereas SAM 3D Objects relies on single images and lacks metric accuracy.
- Evaluation on Scannet++ and Replica: ShapeR is evaluated on six ScanNet++ scenes and seven Replica scenes using recall-based metrics because these datasets lack complete 3D geometry.The protocol follows DP-Recon, and ShapeR often surpasses the ground-truth scans in completeness.
- Evaluation on Digital Twin Catalog (DTC): On DTC Active and DTC Passive, ShapeR is compared with LIRM across approximately 100 tabletop sequences per dataset with minimal occlusion and clutter.DTC Passive permits freer user movement than the active variant, making it more casual.
- Robustness Trends: ShapeR shows significantly greater robustness than LIRM as capture conditions progress nonlinearly from controlled to complex and casual scenes.ShapeR maintains high reconstruction quality as scene casualness increases, while baseline metrics deteriorate.
- Monocular Image-to-3D: ShapeR can reconstruct metric 3D shapes from a monocular image without retraining by conditioning on the image and a metric point cloud from MapAnything.This configuration supports reconstruction of both individual objects and entire scenes.
C. Implementation Details
ShapeR uses a transformer-based 3D VAE with specified encoder, decoder, width, attention, and training settings. Qualitative comparisons show metrically accurate, well-placed, competitive, and complete reconstructions across several benchmarks.
- VAE architecture and training: The 3D VAE uses 8 transformer layers in its encoder and 16 in its decoder, with hidden width 768 and 12 attention heads.It is trained for 200K steps with an effective batch size of 640 across 64 NVIDIA H100 GPUs.
- Qualitative comparisons: ShapeR generates metrically accurate geometry and consistently well-placed objects from posed image sequences, unlike SAM 3D Objects’ non-metric outputs and occasional placement or hallucination errors.SAM 3D Objects uses a single image and interactive object segments.
- Qualitative comparisons: ShapeR performs competitively on Active sequences and surpasses LIRM on the more casual Passive sequences in DTC.Both sequence types contain tabletop objects without clutter or occlusions, while Passive sequences permit freer user movement.
- Qualitative comparisons: ShapeR produces complete reconstructions on ScanNet++ and Replica, often exceeding ground-truth scans in completeness because occluded regions lack geometry in those scans.The comparison is against DPRecon.
D. Limitations
ShapeR remains vulnerable when casual captures provide insufficient evidence, when target objects are closely attached to other structures, and when upstream 3D detection fails. These conditions can produce incomplete, less detailed, contaminated, or unrecoverable reconstructions.
- Capture quality and coverage: Low-fidelity images or very few views can yield incomplete reconstructions or missing fine detail because geometric and visual evidence is insufficient.This limitation arises when the capture provides too little information for reliable reconstruction.
- Object separation: Objects stacked on or closely attached to other items may produce meshes containing remnants of adjacent structures rather than cleanly isolated targets.The passage gives tables supporting other objects as an example of this failure mode.
- Upstream detection: ShapeR depends on upstream 3D instance detection, so missed detections or inaccurate bounding boxes propagate errors and prevent missed objects from being recovered.Detection failures directly affect the reconstruction stage because undetected objects never reach it.