Source-linked AI summary
CAST: Component-Aligned 3D Scene Reconstruction from an RGB Image
Kaixin Yao, Longwen Zhang, Xinhao Yan, Yan Zeng, Qixuan Zhang, Wei Yang, Lan Xu, Jiayuan Gu, Jingyi Yu
TL;DR
Single-image scene reconstruction must recover detailed geometry while preserving object relations and physical plausibility, despite occlusions and representation limits in prior approaches. CAST decomposes and analyzes the scene, generates and aligns objects independently, and applies physical correction; the paper reports improved visual quality and physical plausibility, with applications including real-to-simulation workflows. Its main stated limitation is dependence on an object-generation model whose detail and precision remain insufficient.
Problem
Single-image scene reconstruction remains limited by pose misalignment, occlusions, missing object dependencies, and the lack of editable, detailed scene representations.
Method
CAST combines scene decomposition, relation analysis, occlusion-aware object generation, generative alignment, and physics-aware pose correction to reconstruct coherent 3D scenes.
Results
CAST demonstrates significant improvements over state-of-the-art methods in visual quality and physical plausibility, including object- and scene-level geometry quality and realism.
Takeaways & Limitations
CAST supports open-vocabulary reconstruction from diverse images and can facilitate realistic, scalable real-to-simulation environments for robotics.
Takeaways & Limitations
CAST depends heavily on its underlying object-generation model, whose insufficient detail and precision can cause inconsistencies affecting object alignment and spatial relations.
Abstract
from arXiv · showhide
Recovering high-quality 3D scenes from a single RGB image is a challenging task in computer graphics. Current methods often struggle with domain-specific limitations or low-quality object generation. To address these, we propose CAST (Component-Aligned 3D Scene Reconstruction from a Single RGB Image), a novel method for 3D scene reconstruction and recovery. CAST starts by extracting object-level 2D segmentation and relative depth information from the input image, followed by using a GPT-based model to analyze inter-object spatial relationships. This enables the understanding of how objects relate to each other within the scene, ensuring more coherent reconstruction. CAST then employs an occlusion-aware large-scale 3D generation model to independently generate each object's full geometry, using MAE and point cloud conditioning to mitigate the effects of occlusions and partial object information, ensuring accurate alignment with the source image's geometry and texture. To align each object with the scene, the alignment generation model computes the necessary transformations, allowing the generated meshes to be accurately placed and integrated into the scene's point cloud. Finally, CAST incorporates a physics-aware correction step that leverages a fine-grained relation graph to generate a constraint graph. This graph guides the optimization of object poses, ensuring physical consistency and spatial coherence. By utilizing Signed Distance Fields (SDF), the model effectively addresses issues such as occlusions, object penetration, and floating objects, ensuring that the generated scene accurately reflects real-world physical interactions. CAST can be leveraged in robotics, enabling efficient real-to-simulation workflows and providing realistic, scalable simulation environments for robotic systems.
1 INTRODUCTION
CAST targets single-image 3D scene reconstruction by modeling objects together with their spatial and physical relations. It combines open-vocabulary scene analysis, independently generated object meshes, alignment, and physics-aware correction to improve reconstruction quality and realism.
- Motivation: Single-object generation methods struggle when assembling complete scenes because pose estimation and inter-object spatial relations remain inadequately modeled.Existing reconstructions can produce penetrations, floating objects, and missing contacts when physical constraints are absent.
- Method: CAST extracts open-vocabulary object segments, locations, and partial 3D information, including relative transformations and scales, from one RGB image.The pipeline uses 2D foundation models and monocular depth estimation to establish scene-level context.
- Method: CAST generates each object as a high-fidelity mesh and predicts transformations that align canonical object geometry with the reference image and scene point cloud.Its perceptive 3D instance generator combines occlusion-aware object generation with pose alignment generation.
- Method: A physics-aware correction process uses GPT-4v-derived physical relations to optimize object poses and reduce implausible interdependencies.The stated goal is realistic physical coherence for simulation, editing, and rendering.
- Results: CAST supports perceptually realistic reconstruction across indoor, outdoor, real-world, and AI-generated images, while outperforming strong baselines on geometry and realism evaluations.The paper reports gains over 3D-Front in object- and scene-level geometry quality, plus perceptual and physical realism across diverse images.
2 RELATED WORK
Related work spans multi-view capture, monocular reconstruction, retrieval, generative assets, and physics-aware modeling. CAST positions modular, independently generated and aligned meshes as a way to support open-vocabulary, physically coherent, editable scenes from a single image.
- Single Image Scene Reconstruction: Photogrammetry captures detailed physical scenes but requires tens to hundreds of views, whereas single-image approaches use one image and are more scalable.The comparison concerns acquisition cost, time, and scalability rather than reconstruction quality alone.
- Single Image Scene Reconstruction: Monocular reconstruction provides depth point clouds but struggles with occlusions, hidden scene portions, and detailed, precise representations.Related methods use radiance fields or 3D Gaussian representations to learn occlusion priors from 3D datasets.
- Reconstruction as Generation: Feed-forward scene methods often rely on small indoor datasets and produce geometries lacking sufficient detail, while retrieval methods replace scene objects with similar database assets.These approaches use annotated datasets or pre-existing object collections, constraining coverage and flexibility.
- Reconstruction as Generation: Video diffusion and 3D Gaussian approaches yield compelling visuals but lack editable meshes, UV mappings, and decomposable PBR materials needed by traditional production pipelines.The passage identifies these representation gaps as limitations for fully realized, editable 3D scenes.
- Reconstruction as Generation: CAST generates objects independently and aligns them into a cohesive scene while preserving geometry, textures, and spatial relations for more editable reconstructions.Its stated design emphasizes improved quality and flexibility through modular scene construction.
- Physics-Aware 3D Modeling: Compared with prior physics-aware methods, CAST uses one image, supports open-vocabulary reconstruction, and models complex inter-object relations such as support and contact.The comparison distinguishes CAST from methods limited to indoor scenes, closed-vocabulary retrieval, multi-view inputs, individual objects, or stability.
3 OVERVIEW
CAST reconstructs high-fidelity, contextually consistent 3D environments from a single unannotated RGB image by analyzing object relations before object-wise generation. Its pipeline combines semantic extraction, geometric cues, contextual relation analysis, and physical plausibility constraints.
- Overview: CAST addresses scene-level reconstruction from a single RGB image by emphasizing the arrangement and relations of multiple entities.The method is intended for images from real-world photography or synthetic data.
- Overview: The pipeline segments and localizes objects, obtains preliminary point clouds, and analyzes semantic and spatial relations before generating objects.This contextual backbone informs subsequent object-wise reconstruction and placement.
- Overview: Object relation analysis helps each reconstructed object retain geometric fidelity and correct placement within the broader scene.The approach then synthesizes a coherent environment with structurally sound layouts and realistic interactions.
- Overview: CAST pursues generative modeling of complex inter-object relations while integrating geometric and contextual information to maximize reconstruction accuracy and plausibility.The stated objectives connect context-driven generation with the gap between 2D imagery and immersive virtual environments.
4 PERCEPTIVE 3D INSTANCE GENERATION
CAST reconstructs scenes compositionally by generating individual objects from partial observations, aligning them to scene coordinates, and iteratively refining their geometry and transformations. Its perceptive instance-generation pipeline combines occlusion-aware image conditioning, point-cloud conditioning, and semantic alignment to produce geometrically coherent objects.
- Occlusion-aware 3D Object Generation: Object-wise generation addresses occlusions and limited sensor coverage by reconstructing complete 3D assets from partial image and point-cloud observations.The generated asset is aligned with the observed partial point cloud in canonical space and mapped back to the original scene space.
- Occlusion-aware 3D Object Generation: A large generative model produces holistic, high-fidelity object meshes from partial image and point-cloud observations.The base model is pretrained with textual and image conditioning following native 3D generative approaches.
- Occlusion-aware 3D Object Generation: DINOv2 image encoding with masked-autoencoder behavior infers latent features for occluded regions during object generation.The occlusion mask identifies missing pixels, while random masking during pretraining supports inference from visible regions.
- Canonical Point Cloud Conditioning: Canonical point-cloud conditioning improves pixel-aligned geometry by supplementing high-level image features with observed partial geometry.The geometry representation uses a VAE to encode surface point clouds and decode latent representations into SDFs for mesh extraction.
- Generative Alignment: The alignment generative model transforms scene-space partial point clouds into canonical space using generated geometry latents, addressing semantic misalignment from canonical poses.Unlike ICP, the model is conditioned on both the scene-space partial point cloud and the canonical-space geometry latent code.
- Iterative Generation Procedure: Iteratively alternating geometry generation and transformation estimation continues until transformation changes fall below a threshold or a maximum iteration count is reached.The resulting object is intended to be visually accurate and geometrically aligned with the input data.
5 PHYSICS-AWARE CORRECTION
CAST uses image-derived physical relations to correct object poses after independent reconstruction, targeting penetration, floating, and unsupported contacts. Its simplified optimization uses relation-specific constraints and SDF-based costs rather than full rigid-body dynamics.
- Motivation: Independent object reconstruction can yield physically implausible scenes, including intersecting objects and unsupported floating objects.The section illustrates a guitar intersecting a cooler and a surfboard floating without van support.
- Correction process: CAST optimizes object rotations and translations using physical constraints derived from an image-based inter-object relation graph.The correction process is formulated as an optimization problem over pairwise object constraints extracted from the image.
- Correction process: The method replaces full dynamics with customized pose optimization that seeks physical plausibility at the current time step.The authors note that an object may not remain stable over time, while optimized poses can initialize later physical simulations.
- Physical constraints: Relations are categorized as contact or support, with contact modeled bilaterally and support optimized in one direction for stacked objects.Support treats the supporting object as static while optimizing the supported object's pose.
- Physical constraints: Signed distance functions detect penetration and separation, while surface regularization promotes close contact with flat supports and partially reconstructed objects.The contact constraint enforces non-penetration and at least one contact point; regularization also handles cases such as a van reconstructed with only two wheels.
- Scene relation graph: Fine-grained relations such as Stack, Lean, Hang, Clamped, Contained, and Edge/Point are mapped into a directed constraint graph for pose optimization.Bidirectional edges represent Contact, directed edges represent Support, and the graph reduces optimized pairwise constraints relative to full simulation.
6 RESULT
CAST reconstructs diverse single-image scenes with high-fidelity geometry, realistic textures, accurate alignment, and physically coherent object interactions. Evaluations and ablations report improvements across qualitative, quantitative, alignment, occlusion, and physical-consistency assessments.
- Qualitative Results: CAST generates detailed indoor, close-up, and AI-generated scenes with high-fidelity geometry, realistic textures, and convincing compositions.
- Qualitative Results: CAST supports qualitative comparison across open-vocabulary, indoor, outdoor, close-up, and AI-generated scenarios against ACDC and Gen3DSR.The comparison includes reference and novel views, with datasets and input types arranged across the figure rows.
- Quantitative Results: CAST outperforms ACDC and Gen3DSR on CLIP score, GPT-4 ranking, visual quality, and physical plausibility.These four metrics evaluate visual similarity, semantic ranking, user-rated visual quality, and physical plausibility.
- Quantitative Results: On 3D-Front, CAST achieves higher object-level generation quality and better scene layout accuracy than existing approaches.The evaluation uses Chamfer Distance, F-Score, and IoU to assess shape accuracy, object-level quality, and scene-level overlap.
- Ablation Studies: MAE conditioning improves reconstruction of occluded regions, producing more complete and visually coherent object generations.Without the occlusion-aware module, examples such as a spaceship and cup appear fragmented or incomplete.
- Ablation Studies: Relational graph constraints improve physical plausibility, intended-scene alignment, and preservation of correct spatial relations.The physical-consistency evaluation addresses floating objects, penetration, and misaligned spatial relations; the ablation study reports significant contributions from each component.
7 CONCLUSIONS
CAST combines geometric fidelity, pixel-level alignment, and physically grounded constraints to reconstruct visually accurate and physically consistent 3D scenes. Its limitations include dependence on object-generation quality, weak material and lighting representation, and slight degradation in complex scenes.
- CAST combines geometric fidelity, pixel-level alignment, and physically grounded constraints in single-image 3D scene reconstruction.
- The method’s scene quality depends heavily on the underlying object-generation model, whose insufficient detail and precision can affect object alignment and spatial relations.
- Mesh representations struggle with textiles, glasses, fabrics, and transparent materials, which can appear unnatural.
- CAST lacks lighting estimation and background modeling, so generated environments may have less natural shading, illumination, and contextual depth.
- Performance may slightly degrade in complex scenes with intricate spatial layouts and dense object configurations.
A GPT-4V PROMPT
The GPT-4V prompt analyzes numbered scene objects and outputs their physical relationships in a strict JSON format. It defines six relationship types and restricts relationships to contacting objects.
- The prompt instructs GPT-4V to act as an object–relationship analyst for numbered scene images.
- For each related object pair, GPT-4V must output a JSON object containing the relationship details and all possible relationships.
- The prompt defines six relationship types: Stack, Lean, Hang, Clamped, Contained, and Edge/Point.
- Only objects that are in contact with each other should have a relationship.
- Each relationship requires the correct type and a clear explanation, with Stack used when contacting objects are difficult to classify.