Source-linked AI summary
CODA: Depth-Aligned Scene Completion and Object Decomposition from a Single RGB-D Image
Dongwon Son, Junhyek Han, Yoontae Cho, Minseok Lee, Hong-seok Choi, Jiwook Choi, Hyungjin Kim, Beomjoon Kim
TL;DR
Single-view RGB-D reconstruction must infer complete scene geometry despite occlusion, while keeping generated surfaces consistent with measured depth. CODA completes a scene before separating its objects and reports improved reconstruction accuracy and pose stability over evaluated object-first and scene-first baselines.
Problem
A single RGB-D view reveals only part of a cluttered scene, yet safe retrieval requires complete geometry for the target, neighboring objects, and support at the observed scale and pose.
Source: §I · p. 1Method
CODA completes scene geometry from an unsegmented RGB-D view, then decomposes it into environment and object meshes, grounding generation with observed-surface initialization and depth-guided feature matching.
Source: §I · p. 2Results
Takeaways & Limitations
The results support completing the scene before separating objects, with explicit depth grounding connecting generative shape priors to measured geometry.
Source: §V · p. 7
Abstract
from arXiv · showhide
Robots operating safely in cluttered everyday environments often need to infer scene geometry from partial observations. Methods that detect objects in 2D and reconstruct them independently struggle in such scenes: a missed object is never reconstructed, a merged detection can fuse two objects, and separately reconstructed meshes may overlap or fail to touch their supporting surfaces. We introduce CODA (Complete Once, Decompose Afterward), a generative model that instead reconstructs the complete scene geometry from a single unsegmented RGB-D image, then separates the surface into the surrounding environment and movable objects. Still, generated scene geometry can drift from the observed partial point cloud. To reduce this drift, CODA uses two explicit 3D grounding mechanisms to keep reconstructed geometry consistent with observed surfaces while completing unseen regions. Experiments on HomebrewedDB and our custom cluttered-scene dataset show more accurate reconstructions and a higher fraction of objects remaining in place under simulated gravity than both object-first and scene-first baselines.
I. INTRODUCTION
CODA addresses single-view reconstruction in clutter by completing a scene before separating its movable objects, while grounding generated geometry in measured depth. The paper reports improved reconstruction accuracy and pose stability over evaluated object-first and scene-first baselines.
I. INTRODUCTION
- The task requires complete geometry for the target, neighboring movable objects, and shelf at the correct scale and pose, despite occlusion in one RGB-D view.Hidden shape and even object count may be ambiguous.Source: §I · p. 1
- Object-first methods depend on segmentation: missed objects remain absent, merged masks can fuse objects, and independently reconstructed meshes may misalign, interpenetrate, or lack support..Source: §I · pp. 1–2
- Scene-first reconstruction uses neighboring surfaces and supports to infer hidden shapes and relative positions, but some methods lack separate movable-object geometry while others may lack manipulation-level surface detail.The cited examples include Oct-MAE and GenRecon [8], and SIMstack, CoReNet, and PaSCo.Source: §I · p. 2
- CODA completes one detailed scene before separating it into objects, avoiding reliance on an initial 2D instance mask or known object count.It fine-tunes TRELLIS.2 geometry and texture flows, repurposing texture prediction to represent object identities.Source: §I · p. 2
- CODA addresses drift from observed depth with observed-surface initialization for coarse and fine geometry generation and explicit image-to-3D feature matching.These mechanisms anchor generated surfaces to observed geometry while allowing different completions of hidden regions.Source: §I · p. 2
- CODA improves reconstruction accuracy and pose stability over object-first and scene-first baselines; it is demonstrated on real-world datasets including HomebrewedDB and a custom dataset featuring diverse environments and object shapes in cluttered arrangements.All 13 evaluated objects remain stable over 2 s in the predicted shelf in Fig. 1(d).Source: Figure 1 · p. 1, §I · p. 2
A. Scene Reconstruction and Object Decomposition
Prior scene-reconstruction approaches trade off joint scene reasoning, separate object geometry, and surface detail. CODA completes geometry jointly and then decomposes it, while grounding generation in measured depth through initialization and image-to-3D correspondence.
A. Scene Reconstruction and Object Decomposition
- Scene-first methods use neighboring objects and supports to infer hidden shapes and relative positions, but outputs range from joint reconstructions without separate meshes to object-level outputs that may lack fine detail.PaSCo produces voxel masks, whereas CODA uses a pretrained texture flow for scene-level ownership and detailed object meshes.Source: A. Scene Reconstruction and Object… · p. 2
- Physical refinement methods are complementary to CODA: it uses no explicit physics term and evaluates its meshes with a gravity-based settling test.Prior approaches include stability-aware completion, equilibrium constraints, and physics-based reconstruction or refinement.Source: A. Scene Reconstruction and Object… · p. 2
B. Grounding Generation in Measured Depth
- CODA initializes both geometry stages from the observed surface, grounding coarse scene structure and fine details that help distinguish nearby objects and preserve support contacts.Points-to-3D initializes the coarse latent from a partial point cloud but starts the fine latent from noise.Source: B. Grounding Generation in Measured… · p. 3
- CODA assigns image features to observed 3D voxels and adds them to matching voxel representations, supplementing cross-attention in all three stages.This adapts GenRecon's explicit spatial correspondence [8] to a single RGB-D view.Source: B. Grounding Generation in Measured… · p. 3
III. METHOD
CODA models single-view scene completion and decomposition from an RGB-D observation, using a three-stage pipeline to generate coarse occupancy, fine geometry, and object identities. The input includes camera calibration and a reconstruction region, with a stated pose-construction assumption when camera-to-world pose is unavailable.
A. Overview
- The input O comprises an RGB image, depth map, camera intrinsics, camera-to-world transformation, and reconstruction region; the world frame is gravity-aligned.If camera-to-world pose is unavailable, the method assumes an approximate pose based on the observed point-cloud centroid and an upward normal from a fitted horizontal support plane.Source: A. Overview · p. 3
- CODA first samples coarse occupancy Ω, then shape latent Z on the occupancy-selected grid, and finally material latent A conditioned on Z and the observation.The three stages perform coarse scene completion, fine surface reconstruction, and object decomposition, respectively.Source: A. Overview · p. 3
- The shape and material latents share active 3D voxel sites, and decoding them yields a scene mesh and surface attributes used to partition it by identity.The geometry decoder operates at an effective 512^3 resolution; flow models and encoders/decoders remain frozen while flows' LoRA and additional adapters are trained.Source: A. Overview · p. 4
B. Coarse-to-Fine Scene Reconstruction
CODA grounds its coarse and fine geometry flows in observed surfaces and adds depth-aligned image features to voxel tokens. Optional instance-mask conditioning is reserved for the object-first baseline; standard scene-first CODA reconstructs the whole scene without that signal.
B. Coarse-to-Fine Scene Reconstruction
- Both geometry stages initialize observed voxel locations with encoded partial geometry and unobserved locations with independent Gaussian noise.The observed locations are identified by a binary mask derived from the depth- and calibration-based point map.Source: B. Coarse-to-Fine Scene Reconstruction · p. 4
- In all three stages, projected DINO image features are assigned to voxels containing back-projected depth points and added to matched voxel tokens before every DiT block.Features outside the reconstruction region or in voxels absent from the active grid are excluded.Source: B. Coarse-to-Fine Scene Reconstruction · p. 4
- Optional mask tokens support single-object reconstruction for the object-first baseline, while standard scene-first CODA disables mask conditioning and reconstructs the entire scene.The geometry flows are trained separately with conditional flow matching; Stage 2 uses ground-truth sparse coordinates during training and Stage 1's predicted support at inference.Source: B. Coarse-to-Fine Scene Reconstruction · p. 4
C. Object Decomposition
CODA uses Stage 3 to decompose the fine scene mesh from Stage 2 into environment and individual object meshes without changing its geometry. The experiments test reconstruction accuracy and gravity stability, depth grounding, and diversity of hidden shapes.
C. Object Decomposition
- Stage 3 adapts a pretrained shape-conditioned texture flow to predict surface identity in the fine scene mesh from Stage 2.It predicts identity attributes at mesh vertices, then assigns each triangle to its nearest code to produce environment and object meshes without altering geometry.Source: C. Object Decomposition · p. 4
- A fixed codebook assigns one code to the environment and the remaining codes to movable-object identities, rather than specifying the predicted object count.Ground-truth surface labels are converted to six-channel codes and compressed into material latents for supervision.Source: C. Object Decomposition · p. 4
IV. EXPERIMENTS
- The experiments evaluate whether CODA improves reconstruction accuracy and gravity stability, whether depth grounding improves accuracy and observed-surface agreement, and whether stochastic generation diversifies hidden shapes while preserving observed geometry.Source: §IV · p. 4
A. Experimental Setup
The evaluation combines synthetic scenes with real-world benchmarks, including a collected dataset of cluttered arrangements. Methods receive one RGB-D view with known camera intrinsics, with object-first baselines tested under predicted and ground-truth masks.
A. Experimental Setup
- The synthetic dataset contains 28K physically settled scenes and 224K RGB-D frames across table, shelf, and open-container layouts.Each scene has 5–12 objects; randomized poses, scales, camera viewpoints and roll, lighting, and materials are rendered in simulation.Source: A. Experimental Setup · p. 4
- All methods receive one RGB-D view with known camera intrinsics, and object-first baselines are evaluated with SAM 3.1 or ground-truth instance masks.The evaluated object-first methods include RecGen, RGB-D-conditioned SAM 3D, ZeroGrasp, and an object-first CODA variant.Source: A. Experimental Setup · pp. 4–5
- CODA and ChatGPT Astra use raw predicted poses, while other external baselines are rigidly aligned to measured depth with ICP.PaSCo is the scene-first voxel-completion baseline, adapted to the workspace and trained on the synthetic dataset.Source: A. Experimental Setup · p. 5
- The collected real-world test set contains 50 RGB-D views across 27 cluttered scenes and 449 object-view targets.It supplements HomebrewedDB, which has limited coverage of constrained clutter such as shelves and open containers; examples include confined shelves, two-tier shelves, tables, and containers.Source: A. Experimental Setup · p. 5, Figure 4 · p. 5
- CODA uses a 1 × 1 × 1 m workspace with 64^3 coarse occupancy and 512^3 fine geometry, and each stage is trained on one GPU.The implementation uses LoRA rank 64 and α = 128, AdamW, and exponential-moving-average weights with decay 0.9999.Source: A. Experimental Setup · p. 5
B. Reconstruction Accuracy, Stability, and Efficiency
Across both evaluation datasets, CODA achieves the highest complete-surface F@5 and pose retention among the evaluated methods in the main comparison. The section also reports segmentation-related failure cases and runtime comparisons.
B. Reconstruction Accuracy, Stability, and Efficiency
- A missed SAM proposal leaves a severely occluded target without a RecGen reconstruction, while one proposal covering two objects leaves RecGen’s projection fused; CODA projects distinct instances.These examples illustrate segmentation-related failure cases for object-first methods on HomebrewedDB.Source: Figure 5 · p. 5, B. Reconstruction Accuracy, Stability,… · p. 6
- Evaluation reports surface F1 at a 5 mm threshold for complete-surface and input-view comparisons, plus pose retention after 2 s of gravity simulation.Pose retention requires objects to remain within 10 mm center-of-mass translation and 10 degrees of rotation; F@5 and retention are higher-is-better.Source: B. Reconstruction Accuracy, Stability,… · p. 5
- In two selected scenes, CODA retains all evaluated objects after 2 s under gravity: 5/5 and 10/10.The figure marks retained objects with green checks and failed objects with red crosses; baselines show initial overlap or inadequate support.Source: Figure 6 · p. 6
- CODA achieves the highest complete-surface F@5 and pose retention under gravity among evaluated methods on both datasets, including object-first and scene-first baselines.It improves complete-surface F@5 over RecGen by 14.5% on each dataset; pose retention is 0.785 on HomebrewedDB and 0.766 on the collected dataset, exceeding the strongest external baselines by 2.2 and 50.8 percentage points, respectively.Source: Table I · p. 5, B. Reconstruction Accuracy, Stability,… · p. 6
- 5.59 s per scene, CODA’s runtime on six custom views, compares with 100.15 s for RecGen and 85.08 s for SAM3D.PaSCo and ZeroGrasp are faster at 2.35 s and 1.54 s, but achieve lower complete-surface accuracy and pose retention; Astra takes 28.1 min per scene.Source: B. Reconstruction Accuracy, Stability,… · p. 6
C. Grounding and Object-Decomposition Ablations
CODA’s ablations test how explicit depth grounding and object decomposition affect reconstruction. The full grounding model outperforms variants that disable or reduce its mechanisms, while a learned-query decomposition variant shows separation errors.
C. Grounding and Object-Decomposition Ablations
- Complete-surface F@5 is 0.679 for the full model, versus 0.633 without matching, 0.450 without initialization, and 0.140 with both mechanisms disabled.Coarse-only initialization reaches 0.399.Source: C. Grounding and Object-Decomposition… · p. 6, Table III · p. 7
- The grounding ablation compares a TRELLIS.2 baseline fine-tuned for RGB-D conditioning with GenRecon-style and Points-to-3D-style configurations.Matching ablations apply to all three stages.Source: C. Grounding and Object-Decomposition… · p. 6
- The learned-query decoder reaches complete-surface F@5 of 0.3748 and pose retention of 0.2010, with most failures arising from incorrect object–environment separation.It partitions saved CODA geometry without texture-flow pretraining.Source: C. Grounding and Object-Decomposition… · p. 7
D. Occlusion Accuracy and Object-Wise Diversity
The evaluation compares CODA and RecGen across increasing occlusion, assessing reconstruction quality and variation in observed and hidden regions. CODA’s relative best-sample advantage is largest for hard-occluded objects.
D. Occlusion Accuracy and Object-Wise Diversity
- The study generates eight fixed-seed reconstructions per target and evaluates 274 instances grouped by silhouette visibility: 91 easy, 90 mid, and 93 hard.Easy objects retain 75–100% visibility, mid objects 40% to less than 75%, and hard objects 10% to less than 40%.Source: D. Occlusion Accuracy and Object-Wise… · p. 7
- CODA’s best-sample F@10 gain over RecGen rises from 16.0% for easy to 40.4% for hard objects; in the hard group, visible F@10 improves 23.8% and Dvis decreases 70.7%.Best-sample F@10 selects the highest ground-truth surface F-score among eight samples; visible F@10 averages scores against measured RGB-D points.Source: D. Occlusion Accuracy and Object-Wise… · p. 7
- CODA’s hidden-region diversity increases from 0.347 to 0.619 as occlusion rises from easy to hard, while RecGen’s higher Dhid accompanies greater visible-region variation and lower reconstruction accuracy.Diversity alone does not imply better completion.Source: D. Occlusion Accuracy and Object-Wise… · p. 7
- Figure 7 shows eight fixed-seed predictions over observed surface points and, beside them, each method’s oracle GT-best sample against the ground-truth silhouette.The examples are two hard-occluded YCB objects in simulation.Source: Figure 7 · p. 7
V. CONCLUSION
CODA completes a scene from one RGB-D observation before separating environment and object meshes, using depth grounding and texture-flow decomposition. Evaluations show improved reconstruction and pose stability over the tested baselines.
V. CONCLUSION
- CODA reconstructs a complete scene from a single RGB-D observation, then decomposes it into environment and object meshes without input instance masks.Observed-surface initialization and depth-grounded feature matching connect generative shape priors to measured geometry.Source: §V · p. 7
- Experiments on HomebrewedDB and collected cluttered scenes show improved reconstruction accuracy and pose stability over the evaluated baselines.Ablations support explicit depth grounding, and stochastic reconstruction yields varied hidden shapes with greater consistency in observed regions.Source: §V · p. 7
- The results support completing the scene before separating objects as an effective approach to reconstruction under clutter and occlusion.Future work will apply CODA to object manipulation tasks.Source: §V · p. 7