PaperlayerView exact revision on arXiv

Source-linked AI summary

SAM 3D: 3Dfy Anything in Images

SAM 3D Team, Xingyu Chen, Fu-Jen Chu, Pierre Gleize, Kevin J Liang, Alexander Sax, Hao Tang, Weiyao Wang, Michelle Guo, Thibaut Hardin, Xiang Li, Aohan Lin, Jiawei Liu, Ziqi Ma, Anushka Sagar, Bowen Song, Xiaodong Wang, Jianing Yang, Bowen Zhang, Piotr Dollár, Georgia Gkioxari, Matt Feiszli, Jitendra Malik

arXiv:2511.16624v2cs.CVcs.AI

TL;DR

SAM 3D targets single-image 3D reconstruction in natural scenes, where scalable real-world image–3D data is limited and occlusion is common. It combines model- and human-in-the-loop annotation with staged synthetic-to-real training, achieving strong preference results over prior work and introducing a real-world benchmark. Its scope includes layout evaluation that requires ground-truth depth or point maps as a reference.

  • Problem

    Natural images paired with 3D ground truth are difficult to obtain at scale, while models trained on isolated objects struggle with distant or heavily occluded objects in natural scenes.

  • Method

    SAM 3D combines model-in-the-loop and human-artist annotation with synthetic pretraining, semi-synthetic mid-training, and real-image post-training.

  • Results

    SAM 3D achieves a 5 : 1 head-to-head win rate on real images and a 6 : 1 preference over prior SOTA on three scene-reconstruction evaluation sets.

  • Takeaways & Limitations

    SAM 3D provides full reconstruction of 3D shape, texture, and layout from natural images and releases a benchmark for real-world object reconstruction.

  • Takeaways & Limitations

    Layout evaluation in Table 3 requires ground-truth depth or point maps as a reference.

Abstract

from arXiv · show

We present SAM 3D, a generative model for visually grounded 3D object reconstruction, predicting geometry, texture, and layout from a single image. SAM 3D excels in natural images, where occlusion and scene clutter are common and visual recognition cues from context play a larger role. We achieve this with a human- and model-in-the-loop pipeline for annotating object shape, texture, and pose, providing visually grounded 3D reconstruction data at unprecedented scale. We learn from this data in a modern, multi-stage training framework that combines synthetic pretraining with real-world alignment, breaking the 3D "data barrier". We obtain significant gains over recent work, with at least a 5:1 win rate in human preference tests on real-world objects and scenes. We will release our code and model weights, an online demo, and a new challenging benchmark for in-the-wild 3D object reconstruction.

1 Introduction

SAM 3D addresses the difficulty of reconstructing 3D objects from cluttered, occluded natural images by combining scalable annotation with staged training. It introduces a foundation model, a large visually grounded dataset pipeline, and the SA-3DAO benchmark.

  • Motivation and model: SAM 3D reconstructs full 3D shape, texture, and camera-relative layout from a single image, including objects in cluttered and occluded scenes.Because it predicts full 3D shape rather than only the visible surface, the object can be re-rendered from new viewpoints.
  • Motivation and model: Natural images paired with 3D ground truth are difficult to obtain at scale, and existing models trained on isolated objects struggle with distant or heavily occluded scene objects.The paper identifies scalable association of image objects with 3D shape models as a central data challenge.
  • Data and training: The data engine combines synthetic scenes with human selection and pose alignment of candidate 3D models to produce visually grounded annotations.Human annotators choose among proposed meshes or route difficult instances to artists.
  • Data and training: Training proceeds from synthetic supervised pretraining to semi-synthetic mid-training and real-image post-training using model-in-the-loop annotations and human 3D artists.The staged recipe learns shape and texture priors before aligning the model to natural images and human preference.
  • Evaluation and contributions: The paper introduces SA-3DAO, a benchmark of 1,000 real-world image–3D pairs spanning structures, animals, household items, and rare objects.Professional 3D artists create the benchmark shapes, providing an expert human upper bound for visually grounded reconstruction.
  • Evaluation and contributions: The contributions include SAM 3D, a MITL annotation pipeline, synthetic-to-real training, and a challenging benchmark, with gains reported through metrics and large-scale human preference.The released system includes code, model weights, and a demo.

2 The SAM 3D Model

SAM 3D uses a staged generative architecture that first predicts coarse geometry and layout, then refines geometry and texture. Its conditioning combines object-focused information with full-scene context and can optionally incorporate point maps.

  • Formulation: SAM 3D models single-image reconstruction as a conditional distribution over shape, texture, rotation, translation, and scale.The learned generative model approximates the distribution of these 3D properties given the image and object mask.
  • Geometry model: The Geometry Model jointly predicts coarse shape and object layout, enabling coherent multi-object scene reconstruction rather than isolated-object reconstruction.Its coarse shape representation is O ∈R643, while layout comprises rotation, translation, and scale.
  • Input conditioning: The model encodes both a cropped object view and the full image, combining focused object detail with global scene context and recognition cues.The cropped input includes its binary mask, while the full-image input includes the full-image mask.
  • Input conditioning: Optional coarse scene point maps from sensors or monocular depth estimation provide additional conditioning for integration with other pipelines.Point maps can be obtained from hardware sensors such as LiDAR or from monocular depth estimation.
  • Texture and decoding: The Texture & Refinement Model receives Geometry-model voxels and adds higher-resolution geometric detail and object textures.It extracts active voxels from the coarse shape and uses a sparse latent flow transformer for refinement and texture synthesis.
  • Texture and decoding: The refined latent representation can be decoded into either meshes or 3D Gaussian splats using separately trained VAE decoders.The two decoders share the same VAE encoder and structured latent space.

3 Training SAM 3D

SAM 3D uses staged training and data collection to move from synthetic 3D priors toward natural-image reconstruction and human-preferred outputs. Its data engine combines model proposals with human selection, artist annotation for hard cases, and iterative model improvement.

  • Training strategy: SAM 3D stacks synthetic pretraining, semi-synthetic mid-training, and real-image post-training with supervised finetuning and preference optimization.The stages progressively build shape and texture generation, occlusion robustness, mask-following, layout estimation, and human-preferred behavior.
  • Mid-training: 61 million render-paste samples with 2.8 million unique meshes provide physically plausible semi-synthetic training data with accurate 3D ground truth.The dataset composites textured meshes into natural images, including occluder-occludee pairs and synthetic replacements for real objects.
  • Post-training: Real-image post-training closes the synthetic-to-natural domain gap and aligns shape quality with human preferences through an iterative data engine.The pipeline repeatedly collects data with the current model, updates the model, and feeds improved outputs back into collection.
  • Data engine: Human selection turns candidate preferences into training data because annotators can choose and grade plausible meshes even when they cannot create meshes directly.The pipeline uses N = 8 candidates, while difficult cases are routed to professional 3D artists and early cold-start proposals come from learned and retrieval-based models.
  • Data engine: The annotation pipeline collects object shape, texture, orientation, location, and scale by separating object selection, model selection, and scene posing.Stages use existing models and annotators, with model-in-the-loop support for candidate ranking and pose placement.

4 Experiments

SAM 3D is evaluated on real-world and isolated-object reconstruction using quantitative metrics and human preferences. It substantially outperforms prior methods on challenging real-world shape, texture, and scene reconstruction, with gains increasing through multi-stage training and continued data-engine iterations.

  • Evaluation setup: SAM 3D is evaluated on SA-3DAO, ISO3D, Aria Digital Twin, and curated human-preference sets covering shape, texture, layout, and scenes.SA-3DAO contains artist-created real-world meshes; ISO3D evaluates isolated objects, while Aria Digital Twin evaluates layout.
  • 3D shape and texture: 5:1 head-to-head human preference favors SAM 3D over prior state-of-the-art methods for single-object reconstruction on real images.The comparison evaluates textured meshes.
  • 3D shape and texture: SAM 3D matches or exceeds prior state-of-the-art on ISO3D and significantly outperforms all baselines on challenging SA-3DAO inputs.Qualitative examples show strong generalization under heavy occlusion.
  • 3D scene reconstruction: 6:1 user preference favors SAM 3D scene reconstructions over prior state-of-the-art across three evaluation sets.The model jointly generates shape and layout and remains effective with RGB-only inputs and provided pointmaps.
  • 3D scene reconstruction: 2% →77% ADD-S @ 0.1 measures the improvement in jointly generating shape and layout for real-world scene reconstruction.A sample-then-optimize approach can further improve layout performance.
  • Ablations and scaling: Near-monotonic 3D shape improvements arise as successive training stages are added, while longer data-engine operation yields steadily improving performance.The data engine shows near-linear Elo scaling across iterations, with simultaneous scaling of training stages identified as important.

5 Related Work

Related work spans classical multi-view and single-view reconstruction, scene layout estimation, 3D datasets, post-training alignment, and multi-stage pretraining. SAM 3D is positioned within generative single-image reconstruction and staged training approaches.

  • 3D reconstruction: Classical 3D reconstruction includes binocular stereopsis, structure-from-motion, SLAM, silhouette analysis, volume rendering, and explicit or implicit representations.These methods commonly use multiple views or analytical and synthesis-based representations.
  • Single-view reconstruction: Single-view reconstruction methods predict meshes, voxels, point clouds, or CAD-aligned geometry from direct 3D supervision and increasingly use generative object-asset corpora.The cited work addresses reconstruction from a single image.
  • Layout estimation: Layout estimation extends object reconstruction to scenes by estimating shape and pose for mask-indicated object instances.Methods include model-based pipelines that separate shape and posing and model-free approaches that estimate them jointly.
  • 3D datasets: Existing 3D datasets are difficult to source and often consist primarily of single synthetic objects without paired real-world images.This limits learning from rendered views when natural-image pairing is unavailable.
  • Post-training: Post-training research includes supervised fine-tuning, reinforcement-learning-from-human-feedback, direct preference optimization, and self-training methods.These approaches use alignment or model-generated supervision to steer pretrained models.
  • Multi-stage pretraining: Multi-stage pretraining builds on curriculum learning and synthetic/web data mixing, with additional mid-training stages used for capability injection.The related work emphasizes staged data mixtures and training curricula.

6 Conclusion

SAM 3D is presented as a foundation model for reconstructing 3D shape, texture, and layout from natural images. Its data engine and training recipe target robustness across diverse objects and scenes, supported by large-scale annotations and released resources.

  • Conclusion: SAM 3D reconstructs full 3D shape, texture, and layout of objects from natural images.The model is described as a foundation model for real-world 3D perception.
  • Data engine: 850,000 unique object instances from 360,000 images are sourced through adaptive sampling that expands coverage of long-tail categories.The curriculum progresses from simple rigid objects to complex, non-rigid, and highly deformable objects.
  • Data engine: The annotation pipeline converts difficult shape creation into verification by asking annotators to select and grade the best of N candidate 3D models.Candidates include retrieval, text-to-3D, and image-to-3D generations.
  • Annotation: Stage 2 uses pairwise comparisons among candidate objects, with annotators choosing whether the left, right, or neither candidate is better.Objects are automatically rotated on a turntable, while annotators may rotate or zoom them.
  • Data engine: 3.14 million trainable shapes, 1.23 million layout samples, 100K trainable textures, and over 7 million pairwise preferences are produced by the MITL data engine.These totals are reported over the project lifetime, including development.

A.6.2 Training Intuition

The data engine iteratively expands supervision from reliable model generations toward challenging long-tail inputs. Best-of-N search and reward ranking recover otherwise difficult annotations, while human preference and expert supervision remain important for coverage.

  • Alignment objective: The post-training goal is to align the model with human preference across the distribution of real-world objects.The data engine collects viable model samples and uses them to update the model repeatedly.
  • Distribution expansion: The data engine starts from regions where the model generates reliably and pushes toward less common, more challenging parts of the object distribution.Common simple objects form reliable regions, while tail inputs require more sampling or expert intervention.
  • Expert supervision: Human 3D artists provide supervision for inputs whose successful model generations are extremely unlikely, seeding new regions of usable training data.This addresses the chicken-and-egg problem in which improvement requires at least occasional successful generations.
  • Best-of-N trade-off: Around 50 seeds can make successful mesh generation reliable for some challenging categories such as food, but larger candidate sets increase annotation time and choice noise.The trade-off limits straightforward scaling of best-of-N search.
  • Reward-ranked search: 50-generation reward-ranked search raises recovered annotation yield from 0% at N = 2 to 86.8% at N = 50.Reward models rank candidates before human verification, reducing the number of options presented to annotators.
  • Reward-ranked search: The food category’s share of successful annotations increases 9×, from 4% in the original distribution to 36%.The reported experiments compare best-of-N scaling and reward-model variants.
  • Algorithmic interpretation: The data engine resembles online alignment and Expert Iteration but uses humans as verifiers, preference supervision, reward ranking, and DPO.The model alternates between collecting supervision and updating the current policy.

B.2 Render-and-Paste Data Pipeline

The render-and-paste pipeline creates natural-image training data by replacing or inserting objects with synthetic 3D meshes while modeling occlusion, depth, pose, and semantic alignment. It includes progressively more aligned variants and supports efficient training through staged objectives and flow-matching distillation.

  • Core Render-Paste Approach: RP-3DO replaces an object in a natural image with a synthetic 3D object, using masks and depth pointmaps to determine placement, visibility, and occlusion.The resulting data has precise 3D ground truth and pixel alignment.
  • Pipeline Variants: The pipeline includes Flying Occlusions, Object Swap–Random, and Object Swap–Annotated variants that differ in pose information and semantic relevance.FO is pose-unaware; OS-R uses random rotation with depth-aware placement; OS-A uses human-annotated pose and object matching.
  • Flying Occlusions: 55.1M Flying Occlusions samples use 2.87M unique meshes and 11.17M unique images, with one third treating the selected mesh as the occluder.This balancing prevents the model from always predicting the occluded object.
  • Object Swap–Random: OS-R replaces partially occluded or bottom-supported objects with randomly selected meshes fitted to estimated 3D bounds, preserving depth-ordering and visibility cues.Samples with less than 20% visibility are filtered, and pointmaps are updated from unoccluded mesh surfaces.
  • Object Swap–Annotated: OS-A uses human-annotated masks, meshes, placement, and target poses to create fully aligned replacements for Texture & Refinement training.Its purpose is to preserve pixel-aligned correspondence between rendered geometry and image appearance.
  • Efficient Inference: Flow-matching distillation reduces inference steps from fully trained checkpoints while retaining a switch to the original inference mode.The shortcut formulation targets online 3D perception applications without requiring shortcut models to be trained from scratch.

C.5 Texture & Refinement Training Details

Texture & Refinement training progresses from aesthetically curated synthetic data to contextual render-paste data and human-annotated real-world data. Depth-aware feature projection and visibility filtering improve correspondence between image features and visible 3D regions.

  • Pretraining: Texture pretraining on Iso-3DO-500K teaches plausible high-quality textures, while later stages target robustness to real-world complex images.Randomized rendering lighting encourages prediction of textures without baked-in illumination effects.
  • Mid-training: Mid-training on RP-3DO adds full-image conditioning so contextual cues can support texture prediction when objects are heavily occluded.The stage uses FO and OS-A data, with mask and blur augmentation for noisy masks, motion blur, and small objects.
  • Supervised Fine-tuning: Supervised fine-tuning uses MITL-3DO texture annotations, and scaling those annotations by 2x improves human preference rate by 14.2%.The annotations include aesthetically selected samples.
  • Preference Optimization: DPO provides a final texture-alignment stage using human preferences collected from the texture data engine.Its effect is evaluated in Table 4 and Figure 17.
  • Depth-VAE: Depth-VAE back-projects features only to view-visible voxels and normalizes KL regularization by active voxel count.This addresses reduced sharpness caused by projecting features to occluded voxels.
  • Depth-Aware Feature Aggregation: Depth-guided projection samples image features at projected coordinates, filters points using a depth buffer, and aggregates the visible features with normalized weights.Ground-truth depth can replace the constructed surface-depth map when available.

D.1 SA-3DAO: A New Benchmark for Real-World 3D Object Reconstruction

SA-3DAO is a 1,000-object benchmark designed to evaluate visually grounded 3D reconstruction in diverse natural scenes rather than isolated synthetic settings. It combines professionally created, image-aligned 3D ground truth with human preference evaluations and complementary geometric metrics.

  • Benchmark Construction: SA-3DAO contains 1,000 high-fidelity untextured 3D objects aligned to natural images spanning indoor and outdoor environments and diverse object categories.Examples include ski lifts, escalators, clothing, and culturally specific objects such as tribal face masks.
  • Ground Truth: Professional 3D artists reconstruct complete meshes from single images, including unseen regions, to provide an expert human upper bound.The meshes are required to align precisely with visible image pixels.
  • Human Preference Evaluation: Human preference datasets cover generalization, occlusion, dense layouts, and accurate-pointmap scenes across SA-1B, MetaCLIP, LVIS, and Aria Digital Twin.The evaluation uses structured pairwise comparisons to identify the preferred reconstruction.
  • Geometric Metrics: SA-3DAO shape evaluation applies independent normalization and ICP alignment before measuring F-score @ 0.01, Voxel-IoU, Chamfer Distance, and Earth Mover’s Distance.Together, the metrics assess local correspondence, volumetric agreement, fine-grained deviation, and global structural consistency.
  • Ground-Truth-Free Evaluation: ISO3D evaluation uses ULIP and Uni3D cross-modal similarity because its 101 in-the-wild synthetic images lack 3D ground truth.Generated meshes are represented by uniformly sampled surface points for comparison with image features.

E.1 Intermediate Training Stage Knockout

The intermediate-stage knockout study evaluates the contribution of real-world and artist-generated data during training. Removing any such stage causes a notable drop in shape performance.

  • Study Design: Table 7 measures the impact of MITL and 3D artist-generated data as intermediate training stages.It complements the cumulative training-stage analysis in Table 4.
  • Finding: Removing any intermediate real-world data stage produces a notable drop in shape performance.

E.2 Texture Evaluations

SAM 3D outperforms existing image-to-3D methods in holistic reconstruction and texture estimation, particularly on datasets containing occlusion and clutter. Ablations and optimization studies identify data, augmentation, representation, and test-time refinement choices that improve performance.

  • Texture Evaluation: SAM 3D outperforms existing methods on holistic image-to-3D reconstruction and texture estimation across multiple datasets.Evaluations include ISO3D, Preference Set, SA-3DAO, and LVIS, with challenging occluded and cluttered images.
  • Ablations: Lighting augmentation is the most critical augmentation component in Texture & Refinement ablations.Mask and Blur augmentations target specific difficult cases, so their effects are diluted in holistic evaluation.
  • Ablations: Post-training data, RP-3DO data, and DPO contribute substantial gains, with additional improvements from AES data and scaling.These findings support the effectiveness of the data engine and preference alignment.
  • Layout Optimization: Test-time render-and-compare optimization further improves pose metrics for accepted layouts on ADT.Optimization renders generated objects, compares masks and pixels, backpropagates layout gradients, and accepts proposals only when mask IoU improves.
  • Representation: 6D continuous rotation parameterization reduces oriented rotation error relative to quaternions, while normalization yields an additional flow-matching improvement.The ablation uses a held-out Objaverse test split and reports Chamfer distance and ICP rotation error.

E.7 Data Engine: Increasing Best-of-N Search with Reward Models

The data engine uses reward-model best-of-N search to recover additional training data for difficult examples. Including this data during supervised fine-tuning improves performance on challenging evaluation sets, while distilled inference reduces sampling cost.

  • Best-of-N Data: Reward-model-recovered data improves Chamfer Distance and F1 score on tail holdout, Epic Kitchens, and SA-3DAO.The data is included during supervised fine-tuning.
  • Best-of-N Data: Increasing best-of-N search amplifies the expert policy and improves robustness on challenging categories.Improvements are reported on the artist evaluation set, tail holdout set, and Epic Kitchens.
  • Reward Models: VLM-as-reward agrees with human preferences at 68.9%, while DPO agrees at approximately 65%.Around 80% of recovery data came from the DPO-as-reward model.
  • Distillation: 1-step and 4-step geometry distillation provide 38× and 10× inference speed improvements relative to 25-step flow matching.Shortcut mode distills classifier-free guidance into the model, reducing the number of function evaluations.
  • Input Assumption: SAM 3D requires an input segmentation mask to identify the target object and isolate its pixels for feature extraction.Available segmentation models provide a fast, promptable way to obtain masks, although automatic segmentation is imperfect.

F Limitations

SAM 3D has resolution and scene-layout limitations. Its coarse representation can lose fine details, while independent object prediction does not model physical interactions or pose-aware texture orientation.

  • Resolution: The coarse geometry resolution O ∈R643 and limited splat budget can distort complex shapes and thin structures.Artifacts are especially noticeable for fine human features such as hands and faces when reconstructing a whole body.
  • Resolution: SAM 3D reconstructs focused objects such as an isolated hand or head better because more relative resolution is available.The paper suggests architectural changes, superresolution, parts-based generation, or implicit representations as possible next steps.
  • Scene Layout: Objects are predicted one at a time without reasoning about contact, physical stability, interpenetration, or shared ground-plane alignment.Joint multi-object prediction with appropriate losses is proposed to address scene-layout limitations.
  • Texture Orientation: Texture prediction lacks the object pose, so rotationally symmetric objects can receive textures corresponding to an incorrect orientation.This limitation arises from separating texture prediction from pose information.
Loading 2511.16624v2…