Source-linked AI summary
WorldAct: Activating Monolithic 3D Worlds into Interactive-Ready Object-Centric Scenes
Jichen Hu, Jiawei Guo, Jiazhong Cen, Chen Yang, Sikuang Li, Wei Shen
TL;DR
Monolithic generated 3D worlds are visually coherent but difficult to edit or physically interact with. WorldAct decomposes them into object-centric, interaction-ready scenes, and experiments show preserved visual appearance alongside object-level editing and embodied interaction.
Problem
Generated 3D worlds typically fuse objects into monolithic representations, limiting individual editing, manipulation, and suitability for embodied simulation.
Method
WorldAct uses an agent-guided pipeline to discover and segment objects, repair backgrounds, reconstruct object assets, and add collision geometry.
Results
WorldAct preserves the original scene’s visual appearance while enabling object-level editing, placement, physical interaction, and embodied simulation.
Takeaways & Limitations
The framework provides a practical representation for turning generated 3DGS worlds into executable environments for downstream editing and agent–scene interaction.
Takeaways & Limitations
The framework depends on input-world quality and does not yet support dynamic scenes, articulated objects, or physical properties such as mass and friction.
Abstract
from arXiv · showhide
Recent 3D world modeling systems based on generative scene synthesis, such as Marble, can create coherent and explorable 3D environments, yet their outputs are typically static monolithic assets with limited editability and physical interaction. This restricts their use in immersive content creation and embodied simulation, where generated worlds must be actively modified and manipulated. To tackle this challenge, we present WorldAct, a framework that converts static generated 3D worlds into editable and interaction-ready scenes. WorldAct uses a multimodal agent to guide scene decomposition, identify actionable objects, reconstruct geometrically aligned object-level meshes for interaction, and restore the residual background via 3D inpainting. The resulting scenes support object-level editing, collision-aware manipulation, and embodied task execution while preserving global scene coherence. Experiments show that WorldAct enables richer interaction scenarios than the original generated scenes, suggesting a practical path toward editable and interactive 3D world models.
1 Introduction
Generative models create immersive, spatially coherent 3D worlds, but their static monolithic representations limit object-level editing and physical interaction. WorldAct converts these scenes into decomposed, editable, and interaction-ready environments through agent-driven object identification, scene decomposition, background restoration, and object reconstruction.
- Motivation: Generative models synthesize large-scale, spatially coherent 3D environments from simple text or image prompts for virtual simulation and digital content creation.These worlds provide a foundation for simulation and digital content creation.
- Limitations: Static monolithic representations fuse objects into a single structure, preventing individual selection, movement, or replacement.This limits fine-grained editing in workflows such as game design and interior decoration and restricts embodied AI simulation.
- WorldAct: WorldAct converts monolithic 3D Gaussian Splatting scenes into decomposed, editable, and physically interactive worlds.The framework targets object-level editing and manipulation while preserving a structured scene representation.
- WorldAct: An agent-driven pipeline identifies operable objects, decomposes the scene, restores the background, and reconstructs object assets without manual annotation.The pipeline uses selected viewpoints, 2D segmentation, and projection back to 3D to separate individual objects.
- Applications: The resulting environment supports downstream simulation tasks including robotic manipulation and scene rearrangement.Individual objects can be edited, moved, and physically interacted with.
2 Related Works
Related work advances from efficient 3D representations and generative world synthesis toward compositional, object-centric scene construction. Existing decomposition, inpainting, and object-generation methods improve controllability but remain limited by fused geometry, incomplete reconstructions, constrained inputs, or costly optimization.
- 3D World Generation: NeRFs and 3DGS enable efficient, photorealistic rendering, while LucidDreamer, Text2Room, Marble, and HY-World synthesize complete 3D worlds from text or images.These generative methods typically produce static, monolithic representations that limit object-level editing and interaction.
- Compositional Scene Construction: Compositional methods assemble independently generated or jointly modeled objects, and agent-based methods use asset retrieval for scene construction.They enable object-level controllability and interaction but typically depend on limited-view inputs or predefined assets.
- Scene Decomposition and Inpainting: 2D segmentation and vision-language models support lifting masks into 3D, but extracted objects often lack occluded geometry and clean mesh representations.Removing objects also leaves background holes, which 3D inpainting methods can partially address.
- Object-Level 3D Generation: Object-level 3D generation spans SDS-based text-to-3D optimization and image-conditioned reconstruction, while native 3D models learn directly over point clouds, voxels, meshes, and 3D Gaussians.2D-prior-based methods often face limited 3D consistency or expensive optimization.
3 Preliminaries
The paper represents generated 3D worlds primarily as 3D Gaussian Splatting scenes, where colored Gaussian primitives form the renderable visual representation. Existing world models produce visually plausible, spatially coherent environments but lack explicit object entities, geometry, and physical properties needed for direct interaction.
- 3D Gaussian Splatting: 3DGS represents a continuous 3D scene as an explicit set of unstructured colored Gaussian primitives.Each primitive is parameterized by a 3D center, anisotropic covariance, opacity, and color feature.
- 3D Gaussian Splatting: Each 3DGS scene is rendered by projecting Gaussians onto the image plane and accumulating them through differentiable alpha blending.The rendering process uses depth-ordered Gaussians overlapping each pixel and their effective opacities.
- 3D World Models: Recent 3D world models generate large-scale, navigable, and spatially coherent 3D environments from sparse conditions such as text, images, videos, or panoramas.They can be abstracted as a conditional generator Φ : X →W, with the generated world generally represented as a 3DGS scene.
- Interactive-Ready Representation: Interactive-ready worlds require object-level entities and surface or proxy geometry, but standard 3D world models provide monolithic visual assets without primitive-to-entity assignments.For 3DGS, object entities can be represented as disjoint partitions of Gaussian primitives into background and independently editable objects.
- Interactive-Ready Representation: Raw 3DGS scenes lack explicit watertight surfaces, collision proxies, and physical properties such as mass, friction, and support relations.Consequently, visually plausible generated worlds are not directly suitable for downstream interaction-oriented use.
4 Method
WorldAct converts a monolithic 3DGS scene into an interaction-ready, object-decomposed environment through agent-guided discovery, 3D decomposition, background repair, object generation, and pose alignment. The resulting representation combines a completed background and collision mesh with generated object assets for editing, manipulation, and embodied task execution.
- Scene discovery and segmentation: WorldAct renders camera trajectories and uses a vision-language agent plus SAM3 to discover operable objects and produce multi-view segmentation masks.The agent generates object prompts from sparse keyframes, filters semantically irrelevant objects, and SAM3 propagates prompt-based masks across frames.
- 3D decomposition: WorldAct lifts multi-view masks into object-level Gaussian subsets by optimizing soft Gaussian assignments with mask inverse rendering, then defines the residual background.The 3DGS parameters remain fixed while assignment scores are optimized and thresholded; noisy or incomplete subsets serve as localization proxies for asset regeneration.
- Background completion and collision geometry: WorldAct repairs the residual background by fusing removal masks, video-inpainting masked frames, lifting inpainted keyframes into 3D, and constructing a simplified planarized collision mesh.DiffuEraser supplies inpainted frames, DepthLab estimates keyframe depths, and Poisson reconstruction with plane regularization produces the lightweight background proxy.
- Object asset generation: For each discovered object, the agent selects an RGB view and mask for SAM3D, which generates complete 3DGS and textured mesh assets despite incomplete original observations.The isolated object Gaussians are treated as insufficient for direct interaction because occlusion can make them incomplete.
- Object alignment and interaction-ready assembly: WorldAct aligns generated objects using ICP and DINOv2-based view selection, then differentiably refines translation, rotation, and scale with mask, contact, and penetration objectives.The final scene combines the completed background and collision mesh with placed generated object assets, supporting object-level editing, manipulation, and embodied task execution.
5 Experiments
WorldAct is evaluated on six Marble-generated indoor scenes using decomposition, perceptual-quality, qualitative, and user-study measures. Results indicate substantially improved object discovery and interactive editability while preserving scene-level visual quality and supporting embodied interaction.
- Evaluation Setup: WorldAct is evaluated on six diverse indoor scenes from the Marble-World-Model dataset, including kitchens, restrooms, and storage rooms.Marble is selected as the foundation model because of its generation quality and representative multimodal 3D-world modeling capabilities.
- Qualitative Results: Across viewpoints, reinserted interactive meshes remain spatially aligned with reliable multi-view consistency and preserved visual fidelity.Figure 4 shows decomposed outputs alongside original Marble scenes, with boundaries between reinserted objects and backgrounds remaining visually coherent.
- Quantitative Results: Interactable Object Recall increases from 25.40% to 83.98% on MWM-easy, exceeding the unguided baseline by more than threefold.The improvement is also maintained on the challenging MWM-hard subset, though its value is not included in the supplied passage.
- Quantitative Results: Object removal outperforms Gaussian Grouping on both ReMOVE and MANIQA, indicating cleaner background completion.For fair comparison, Gaussian Grouping receives the same masks because it cannot handle the scenes’ complex holes.
- Applications: WorldAct enables grasping and placement in embodied simulation and supports adding, removing, moving, scaling, texturing, and rematerializing scene objects.Object-level editing preserves visual coherence, and removal can restore previously occluded regions.
- Perceptual Evaluation: Converted scenes retain similar scene-level quality while improving separated-object quality relative to static scenes.The MOS study used 20 participants and GPT-5.5 ratings on a 5-point Likert scale across four visual-quality dimensions.
6 Conclusion
WorldAct converts monolithic 3DGS scenes into object-decomposed environments for editing and interaction while preserving the original scene’s visual appearance. The framework remains dependent on input-world quality and does not yet support dynamic scenes, articulated objects, or physical properties such as mass and friction.
- Conclusion: WorldAct decomposes generated scenes into objects, repairs the background, regenerates cleaner assets, and aligns them with simple collision geometry.The resulting environments support basic object-level editing and placement.
- Limitations: The framework depends on input 3D world-model quality and does not yet handle dynamic scenes, articulated objects, mass, or friction.Addressing these limitations is identified as an important direction for future work.
A Overview … B.5 Agent Workflow and Interface to Reconstruction
The supplementary material details WorldAct’s object- and view-selection agent, which separates portable objects from static scenes through semantic parsing, mask extraction, view selection, and video inpainting. Its outputs provide object-level supervision for reconstructing interactive 3D environments while reducing entanglement between object geometry, appearance, and background.
- A Overview: The supplementary material complements the main manuscript with agent design details, user-study implementation, and additional reconstruction experiments.These materials are organized into Sections B, C, and D.
- B Agent Design Details: The object- and view-selection agent identifies portable objects, extracts text-guided masks, and provides masks and inpainted videos for downstream reconstruction.It comprises vision, memory, and execution modules.
- B.1 Overview: The agent parses inputs into an object inventory, selects portable objects, generates segmentation prompts, aggregates SAM3 masks, selects views with a VLM, and inpaints selected objects.Its outputs include object-level representations, best views, aggregated masks, and inpainted background videos.
- B.2 Vision Module: The vision module uses Qwen3.6-Plus to enumerate visible objects and classify mobility and semantic recognizability, producing a deduplicated inventory.Objects are classified as portable or fixed and as precise, subtle, or unrecognizable.
- B.3 Memory Module: The memory module stores each discovered object’s state and supplies exact prompts while tracking processing states for logical consistency.It serves as the agent’s central representation for execution.
- B.4 Execution Module: The execution module prompts SAM3 with portable-object names and uses Qwen3.6-Plus to score candidate views from 0 to 100 by completeness, clarity, and centeredness.Candidate masks are aggregated, duplicates removed, and the resulting masks guide video inpainting to produce a clean background video.
- B.5 Agent Workflow and Interface to Reconstruction: Algorithm 1 returns best-view masks and inpainted background videos that separate portable objects from the static environment before reconstruction.This separation reduces entanglement between object geometry, appearance, and background, supporting interactive-scene decomposition.
C User Study and Auxiliary Agent Evaluation · C.1 Human User Study
The human user study assessed Marble and WorldAct renderings through a blind web questionnaire covering whole scenes and objects, using randomized sampling and four MOS-based perceptual criteria. Responses were anonymized, recorded by criterion, and aggregated across scene- and object-method pairs.
- C.1 Human User Study: 20 questionnaires each contained 20 anonymized single-image questions: 10 whole-scene renderings and 10 object-level renderings.The study evaluated all scenes in the MWM dataset, with participants seeing only one anonymized rendering per question.
- C.1 Human User Study: The backend recorded anonymous question identifiers, hidden image and method labels, hidden scene or object identifiers, and the four criterion scores.These records supported later analysis while keeping identifying metadata hidden from participants.
- C.1 Human User Study: Randomized sampling prevented same-scene method pairing within a questionnaire and separated whole-scene samples from the same scene by at least 10 frames.Each scene was assigned to either Marble or WorldAct within a questionnaire, reducing recognition bias and direct pairwise comparison effects.
- C.1 Human User Study: Participants rated each image for Overall Quality, Surface Completeness, Boundary Cleanliness, and Naturalness.The criteria covered visual impression, geometric continuity, boundary contamination, and coherence with real visual experience.
- C.1 Human User Study: All four criteria used a five-point Mean Opinion Score scale ranging from 1, bad quality that was difficult to accept, to 5, excellent quality with almost no visible distortion.Scores 2, 3, and 4 represented poor, fair, and good quality, respectively, with increasing distortion severity.
- C.1 Human User Study: Object-level questions provided local zoom and panning so participants could inspect object boundaries and surface details more reliably.The interface did not expose hidden metadata, and zoom was not used for scene-level questions.
- C.1 Human User Study: Scores were aggregated separately by criterion and mean score across scene-method pairs, object-method pairs, and all scene or object images grouped by method.The aggregation also retained sample counts for each scene-method and object-method pair.
C.2 Auxiliary Agent Evaluation
WorldAct’s auxiliary GPT-5.5 evaluation provided a reproducible qualitative audit alongside, rather than replacing, the human user study. It assessed paired scene and object renderings using the same four visual criteria and documented failure-mode rationales.
- Evaluation purpose: GPT-5.5 served as a reproducible qualitative audit to test whether the evaluation criteria captured meaningful visual failure modes and representative examples.The agent-based inspection was explicitly not used as a replacement for human ratings.
- Scoring criteria: GPT-5.5 scored Overall Quality, Surface Completeness, Boundary Cleanliness, and Naturalness on a 1-to-5 scale, with rationales tied to visible reconstruction artifacts.Recorded evidence included missing surfaces, boundary halos, smearing, local holes, unstable structure, and style inconsistency; scores remained an internal audit and were not merged with human ratings.
D More results
Additional qualitative results demonstrate the pipeline’s robustness and generalization on MWM-easy and highly cluttered MWM-hard scenes. The visualizations show complete scene processing from Marble inputs through decomposition, background repair, and final assembly.
- Additional qualitative results: Figures 7 and 8 visualize the full pipeline from input Marble scenes through segmentation, object removal, inpainting, and final assembly.The additional results cover both MWM-easy and MWM-hard datasets.
- MWM-easy results: On MWM-easy, the method detects object masks, removes objects, inpaints the background, and places generated objects back into the scene.Figure 7 presents this progression for each scene.
- MWM-hard results: Despite highly cluttered and occluded MWM-hard scenes, the method successfully decomposes objects, repairs the background, and reassembles high-quality assets.Figure 8 illustrates the method’s performance under increased scene complexity.