PaperlayerView exact revision on arXiv

Source-linked AI summary

Lucida: Parse, Generate, and Place for Composable Real-to-Sim Scene Modeling

Minghan Qin, Yuang Wang, Xiuyu Yang, Yushi Long, Yujian Zhang, Ruihuan Wang, Kai Ye, Yangang Zhang, Hang Li

arXiv:2608.30821v1cs.CVcs.AI

TL;DR

Composable scene modeling must recover editable object assets from cluttered captures despite missing precise geometry, unoccluded views, and matching assets. Lucida redistributes requirements across parsing, generation, and placement, using evidence-rich scene graphs, asset completion, and closed-loop GizmoAct refinement. It improves detection, pose estimation, and scene reconstruction, including a scene F-Score of 0.924 versus 0.794 for SAM 3D.

  • Problem

    Existing parse–generate–place pipelines assume accurate instance geometry, unoccluded views, and assets that match cluttered observations, which real indoor captures rarely provide.

  • Method

    Lucida builds a scene graph with per-instance multi-view evidence, generates complete assets from that evidence, and places them using GizmoAct’s closed-loop multi-turn gizmo interaction.

  • Results

    Lucida improves performance across scene-level 3D object detection, object pose estimation, and scene reconstruction, reaching a scene F-Score of 0.924 versus 0.794 for SAM 3D.

  • Takeaways & Limitations

    Deferring precision to closed-loop placement lets Lucida tolerate occlusion, clutter, coarse initial poses, and mismatched assets across the composable reconstruction pipeline.

  • Takeaways & Limitations

    Objects missing after scene parsing cannot be recovered by subsequent generation or grounding stages.

Abstract

from arXiv · show

Composable scene modeling aims to recover a real indoor scene as complete, editable object assets arranged as observed, giving robot simulation and embodied AI a simulation-ready replica of the real environment whose objects can be manipulated individually. Existing pipelines decompose the task into three steps---parse the observations into instances, generate an asset for each, and place each asset back---but every step presumes an input that a cluttered capture rarely provides: accurate instance geometry, unoccluded views, and assets that accurately match the observations. We propose Lucida, which keeps this order but redistributes the requirements, so each step consumes only what a real capture reliably provides and precision is reached at the end of the pipeline rather than demanded at its start. Lucida parses the video into a scene graph whose nodes carry per-instance multi-view evidence, generates a complete asset for each instance from its evidence, and places assets with GizmoAct, a VLM policy that casts placement as multi-turn GUI interaction, manipulating the object's gizmo in a closed loop and deciding itself when alignment is reached. Across scene-level 3D object detection, object pose estimation, and scene reconstruction, Lucida improves mAP over Boxer by 69% on R2S-Scene, raises ADD-SB@0.05 from 57.8% to 83.4% on CA-1M, and increases scene F-Score from 0.794 for SAM3D to 0.924.

1 Introduction

Lucida redesigns composable scene modeling so parsing, generation, and placement consume evidence available from cluttered captures, with precision deferred to closed-loop placement. The resulting system builds evidence-rich scene graphs, completes object assets, and achieves strong detection, pose, and reconstruction results.

  • 1 Introduction: Composable scene modeling seeks separable, editable object assets and spatial arrangements, unlike monolithic reconstructions or library-limited asset retrieval.These representations support robot simulation, embodied AI, AR/VR, and content creation.
  • 1 Introduction: Cluttered indoor captures lack the precise masks, clean point clouds, unoccluded views, and geometrically matching assets assumed by conventional parse–generate–place pipelines.Occlusion, repeated furniture, noisy depth, and asset mismatch make fixed-order processing especially demanding.
  • 1 Introduction: Lucida redistributes requirements across the pipeline: parsing consolidates multi-view evidence, generation completes occluded instances, and placement supplies final precision through closed-loop refinement.The design follows how a human modeler reviews footage, builds models, and nudges them against observations until they agree.
  • 1 Introduction: The parser builds a relation-aware scene graph whose nodes represent instances and carry multi-view evidence bundles for generation and placement.Geometry-aware keyframes, object discovery, full-sequence consolidation, and relation-aware refinement organize the evidence.
  • 1 Introduction: GizmoAct casts placement as multi-turn GUI interaction, reading rendered state and issuing incremental gizmo edits rather than regressing absolute pose or metric scale.The closed loop refines translation, rotation, and scale while accepting coarse initial states and mismatched assets.

2 Method

Lucida parses posed observations into evidence-bearing scene graphs, generates complete object assets from multi-view evidence, and places them through closed-loop gizmo manipulation. The resulting assets form an editable reconstruction.

  • 2 Method: Lucida converts posed RGB(-D) indoor observations into complete object assets organized in a scene graph with 9-DoF poses and scales.The pipeline retains the parse–generate–place order.
  • 2 Method: Parsing constructs an object-centric scene graph containing per-instance evidence bundles and coarse 3D estimates for later stages.The graph represents objects and provides per-object context.
  • 2 Method: Generation synthesizes an occlusion-free object-centric image from each evidence bundle and lifts it into a complete 3D asset.Multi-view evidence supplies complementary visual information when point-cloud observations are incomplete.
  • 2 Method: GizmoAct places each asset through closed-loop 9-DoF gizmo manipulation, iteratively refining rotation, translation, and anisotropic scale until alignment is reached.The placed assets form an editable reconstruction of the captured scene.

2.1 Multi-View Object-Centric Scene Parsing

Lucida parses posed RGB-D video into a scene graph that consolidates multi-view visual and geometric evidence for each object. Keyframe discovery, full-sequence validation, and relation-aware refinement improve object completeness and scene consistency before asset generation and placement.

  • Scene-graph representation: Lucida builds a scene graph whose nodes store multi-view observations, masks or boxes, partial point clouds, representative 3D boxes, and category descriptions.Edges encode relations including support, containment, and adjacency.
  • Geometry-Aware Keyframe Selection and Object Discovery: Keyframe selection uses covisibility and temporal separation to reduce redundant observations while distributing selected frames across long sequences.A frame is retained when its similarity to every selected keyframe falls below a threshold.
  • Geometry-Aware Keyframe Selection and Object Discovery: A VLM discovers objects on selected keyframes, a 3D detector predicts their boxes, and semantically and geometrically consistent observations are grouped in a common 3D frame.The grouped observations and per-frame boxes initialize candidate-object evidence.
  • Object-Centric Full-Sequence Evidence Consolidation: Full-sequence consolidation retrieves additional object observations by projecting representative 3D boxes into input frames and retaining observations consistent with the boxes and observed geometry.When keyframe boxes are inconsistent, nearby-frame propagation supports further object tracking before validation.
  • Relation-Aware Scene Refinement: Relation-aware refinement corrects erroneous object merges and splits, infers spatial relations, and adds missing supporting objects after multi-view geometric validation.The resulting per-object evidence guides amodal asset generation and provides object references and coarse placement initialization for GizmoAct.

2.2 Amodal Object Asset Generation

Lucida converts each multi-view evidence bundle into a complete standalone object asset. Because occlusion and noisy depth make per-object point clouds incomplete, it first synthesizes a complete object-centric image from complementary RGB views before lifting it into 3D.

  • Amodal Object Asset Generation: Lucida generates a complete standalone object asset from each multi-view evidence bundle.The approach uses complementary multi-view RGB observations because occlusion and noisy depth make per-object point-cloud observations incomplete and unreliable for direct 3D completion.

2.3 Agentic 3D Grounding with GizmoAct

GizmoAct reformulates 3D grounding as closed-loop GUI interaction: a VLM observes rendered evidence, emits one incremental edit, and stops when the asset is aligned. Its local-frame updates, auxiliary viewpoints, coarse-axis actions, and trajectory training accommodate arbitrary initial poses, occlusion, scale ambiguity, and imperfect assets.

  • 2.3.1 Formulation: 3D Grounding as Multi-turn GUI Interaction: GizmoAct casts 3D grounding as multi-turn GUI interaction, rendering the current state and applying one executable edit per turn.The VLM re-renders the edited state and continues until it predicts stop.
  • 2.3.1 Formulation: 3D Grounding as Multi-turn GUI Interaction: The manipulated state is a 9-DoF object pose comprising center, rotation, and anisotropic scale for the generated asset.The asset and evidence bundle provide the manipulated element and grounding evidence, while the rollout starts from a coarse pose with arbitrary orientation.
  • 2.3.3 GizmoAct Action Space: Auxiliary observations expose hidden geometry through multi-view and orthographic renderings, while an occlusion overlay clarifies the model’s depth relation to the point cloud.The core action space contains update_pose and stop; extended actions add viewpoint switching and axis permutation for difficult rotations.
  • 2.3.3 GizmoAct Action Space: Every update is expressed incrementally in the object’s gizmo frame, with translation and scale measured relative to the current object size rather than metric quantities.This avoids predicting an absolute pose and removes dependence on metric scale estimation.
  • 2.3.3 GizmoAct Action Space: For large rotation errors, switch_obs reveals six signed-axis views and permute_axis selects one of 24 axis-aligned reorientations before residual update_pose refinement.The permutation removes the dominant rotation residual in one action, leaving incremental updates to repair the remainder.
  • 2.3.4 Supervised Finetuning on Synthetic Trajectories: GizmoAct is finetuned on synthetic expert trajectories pairing evidence, assets, perturbed initial poses, rendered observations, and XML actions.Error injection supplies recoverable flawed actions, while the reported limitation is that scripted errors cover only part of the policy’s on-policy contexts.

2.4 Scene Composition and Postprocessing

After objects are generated and grounded, Lucida composes them into an object-centric indoor scene and keeps scene-level consistency operations separate from GizmoAct. Postprocessing can refine relations, collisions, contacts, and placement plausibility using rules, physics checks, or global verification.

  • 2.4 Scene Composition and Postprocessing: Lucida composes grounded objects into an object-centric scene graph containing assets, poses, scales, categories, and spatial relations.Optional postprocessing separately refines support relations, collision, contact consistency, and placement plausibility.

3 Experiments

Lucida is evaluated separately for scene parsing, pose refinement, and complete scene reconstruction, with ablations probing its parsing and GizmoAct training choices. Across these levels, it improves detection, alignment, reconstruction, and robustness to initialization.

  • 3.1 Scene-Level 3D Object Detection: Lucida achieves the highest mAP in all four scene-level 3D object detection protocols.On R2S-Scene, mAP rises from 0.351 to 0.592 under _all and from 0.355 to 0.597 under _filter.
  • 3.1 Scene-Level 3D Object Detection: On CA-1M, mAP improves from 0.171 to 0.180 under _all and from 0.373 to 0.390 under _filter.Lucida uses keyframe prompts while propagating, validating, and refining instances over the full sequence.
  • 3.2 Layout Refinement and Object Pose Estimation: GizmoAct improves pose metrics across R2S-Object, CA-1M, and ADT, with the best initializer depending on input geometry accuracy.The same policy adapts to heterogeneous initialization errors without retraining.
  • 3.2 Layout Refinement and Object Pose Estimation: On CA-1M, ADD-SB@0.05 increases from 57.8% to 83.4%, while ADD-SB decreases from 0.046 to 0.021 and 3D IoU rises from 0.434 to 0.607.The max-4-view variant improves strict alignment and oriented-box overlap.
  • 3.3 Scene Reconstruction: At scene level, F-Score increases from 0.794 for SAM 3D to 0.924, while CD decreases from 0.022 to 0.010 and BBox IoU rises from 0.396 to 0.495.The qualitative comparison also reports fewer intersections and more consistent relative placement across reference, side, and top views.
  • 3.4 Ablations: Uniform keyframe sampling reduces mAP from 0.597 to 0.516 and scene F-Score from 0.831 to 0.727, the largest ablation decrease.Removing evidence consolidation and relation-aware refinement also reduces detection mAP and scene F-Score.

4 Related Work

Related work covers monolithic and multi-stage composable reconstruction, scene-level object detection and grounding, and iterative pose refinement. Lucida differs by combining evidence-based parsing with VLM-driven GUI refinement that supports mismatched geometry and full 9-DoF state updates.

  • Composable Scene Modeling: Differentiable-rendering and object-compositional methods reproduce observed geometry but leave occluded surfaces unrecovered.Other pipelines retrieve or generate per-object assets, yet their fidelity and completeness remain constrained by database coverage or observed evidence.
  • Scene-Level 3D Object Detection: Scene-level 3D detection methods remain fragile to occlusion and viewpoint or lack persistent instance association across views.Cluttered captures also risk merging or duplicating repeated instances and missing occluded objects.
  • 3D Visual Grounding and Iterative Pose Refinement: Existing pose-refinement methods compare renderings with observations but commonly rely on external stopping rules and rarely estimate scale beyond SE(3).GizmoAct instead uses a VLM operating a GUI, with learned termination and anisotropic-scale refinement.
  • 3D Visual Grounding and Iterative Pose Refinement: GizmoAct inherits incremental decision making and imitation-then-reinforcement learning while operating through a pretrained VLM and object gizmo.It tolerates geometry–observation disagreement and decides termination from the rendering alone.

5 Conclusion

Lucida redistributes requirements across parsing, generation, and placement so real captures need not provide precise geometry, unoccluded views, or perfectly matching assets upfront. Its evaluations show gains across detection, pose estimation, and scene reconstruction, while missing parsed objects remain unrecoverable and agentic refinement is limited to placement.

  • Conclusion: Lucida keeps the parse–generate–place order but makes each stage consume only evidence reliably available from real captures.Parsing builds and refines an evidence-bearing scene graph, generation completes assets amodally, and GizmoAct performs closed-loop placement.
  • Conclusion: Lucida surpasses prompt-based detectors, improves strict pose alignment and 3D IoU across three datasets, and raises scene F-Score from 0.794 for SAM 3D to 0.924.These evaluations cover scene parsing, independent pose refinement, and complete scene reconstruction.
  • Conclusion: Objects missing after scene parsing cannot be recovered by subsequent generation or grounding stages.Extending closed-loop agentic treatment beyond final placement is identified as future work.

Contributions

The supplied passages identify additional qualitative comparisons and trajectory examples included in the paper’s evaluation materials.

  • Additional Evaluation Materials: Figure 10 provides additional qualitative object pose-estimation comparisons on CA-1M, R2S-Object, and ADT.Each method’s posed model is rendered with the blue ground-truth posed model from a main view and three axis-aligned close-ups.
  • Additional Evaluation Materials: Figures 11 and 12 provide further GizmoAct trajectory examples across CA-1M, R2S-Object, and ADT.They show closed-loop rollouts from Boxer, Any6D*, and SAM 3D initializations.
  • Additional Evaluation Materials: The additional trajectory figures use a reference image for each dataset followed by closed-loop rollouts from multiple initializations.Their layout and notation follow Figure 7.
Loading 2608.30821v1…