Source-linked AI summary
SAGE: Scalable Agentic 3D Scene Generation for Embodied AI
Hongchi Xia, Xuan Li, Zhaoshuo Li, Qianli Ma, Jiashu Xu, Ming-Yu Liu, Yin Cui, Tsung-Yi Lin, Wei-Chiu Ma, Shenlong Wang, Shuran Song, Fangyin Wei
TL;DR
Embodied-AI data collection is costly and unsafe, while existing scene-generation methods leave gaps in realism, physical validity, and simulator compatibility. SAGE adaptively combines generators, critics, augmentation, and action synthesis to create simulation-ready training environments; policies trained on them scale with scene diversity and demonstrations and generalize to unseen objects and layouts.
Problem
Real-world embodied-data collection is slow, costly, and constrained by interactive-environment requirements, while existing generators leave gaps in physical grounding and robot-simulator compatibility.
Method
SAGE uses an MCP-based agent to orchestrate layout and asset generators with visual and physics critics, then augments scenes and synthesizes action demonstrations for policy training.
Results
Policies trained purely on SAGE-generated data show clear scaling with scene diversity and demonstration count and improve generalization to unseen objects and layouts across Pick-and-Place and Mobile Manipulation.
Takeaways & Limitations
SAGE demonstrates that simulation-ready scene generation can provide scalable embodied-learning data with realistic, diverse, and directly deployable environments.
Takeaways & Limitations
The current scope emphasizes indoor scenes and rigid-body physics, while action generation targets compositions of pick, place, and navigation.
Abstract
from arXiv · showhide
Real-world data collection for embodied agents remains costly and unsafe, calling for scalable, realistic, and simulator-ready 3D environments. However, existing scene-generation systems often rely on rule-based or task-specific pipelines, yielding artifacts and physically invalid scenes. We present SAGE, an agentic framework that, given a user-specified embodied task (e.g., "pick up a bowl and place it on the table"), understands the intent and automatically generates simulation-ready environments at scale. The agent couples multiple generators for layout and object composition with critics that evaluate semantic plausibility, visual realism, and physical stability. Through iterative reasoning and adaptive tool selection, it self-refines the scenes until meeting user intent and physical validity. The resulting environments are realistic, diverse, and directly deployable in modern simulators for policy training. Policies trained purely on this data exhibit clear scaling trends and generalize to unseen objects and layouts, demonstrating the promise of simulation-driven scaling for embodied AI. Code, demos, and the SAGE-10k dataset can be found on the project page here: https://research.nvidia.com/labs/dir/sage/.
1. Introduction
SAGE addresses the need for scalable embodied-AI data by generating diverse, realistic, physically valid, simulation-ready scenes from user prompts. Its agentic loop combines generators, visual and physics critics, augmentation, and action synthesis to support policy learning and generalization.
- Real-world embodied-data collection is slow, costly, and constrained by the need for interactive environments, motivating scalable simulation.
- Simulation data must provide realism, diversity, simulation-readiness, and task alignment for reliable embodied-policy training and transfer.
- Existing approaches trade off scalability, flexibility, realism, physical grounding, or simulator compatibility, leaving major gaps in interaction and deployment.
- SAGE adaptively orchestrates layout and asset generators with visual and physics critics to produce simulation-ready scenes directly from arbitrary prompts.
- Multi-level augmentation and automatic action synthesis create diverse task-consistent training data, with policies showing scaling trends and stronger generalization to unseen objects and layouts.
- The generation loop iteratively uses visual feedback for semantic corrections and physics validation to refine scenes until they satisfy user requirements.
2. Related Works
Prior indoor-scene methods provide procedural scalability or learned visual realism, but their outputs are not consistently task-aligned, physically grounded, or simulator-validated. SAGE integrates open-vocabulary generation with physical-property attachment and in-simulator correction.
- Procedural systems scale and ground geometry, but fixed rules constrain open vocabulary, fine-grained control, self-improvement, and physics-rich attributes.
- Data-driven scene generators learn spatial priors and realistic layouts, yet physically attached and simulator-validated outputs are not their default artifact.
- SAGE extends semantically plausible synthesis to task-aligned, simulation-ready scenes by supporting open-vocabulary prompts, physical properties, collision checks, and stability validation.
- Existing simulation environments support safe, scalable robot-learning evaluation across tabletop and home-scale tasks, but procedurally retrieved assets can limit diversity.
- Unlike pipelines that enable simulation only after creation, SAGE incorporates simulation into generation for iterative self-correction and improvement.
3. Method
SAGE uses an MCP-based agent to compose scene generators and critics into an adaptive, self-correcting pipeline. It then augments validated scenes, synthesizes robot actions, and trains policies on diverse task-consistent environments.
- The agent receives a robot-task demand and uses critic feedback with multiple editing operations to generate diverse scenes ready for scalable policy learning.
- Through MCP, the agent dynamically invokes tools, incorporates returned results, and selects subsequent actions without a hard-coded tool order.
- Scene initialization creates an empty room and proposes task-relevant objects with descriptions, estimated physical attributes, and placement constraints.
- Asset placement generates text-described 3D objects, estimates height, mass, and rendering properties, and assigns floor, wall, or on-top placement categories.
- Asset movers relocate objects through removal and replanning, while asset removers delete critic-identified objects from the scene.
- Visual and physics critics address missing or misplaced objects, instability, and collisions through semantic feedback and simulator-in-the-loop validation.
- Scene augmentation varies task-relevant object configurations and categories while regenerating task-irrelevant layouts, preserving task semantics across environments.
- After augmentation, physics validation checks placement stability and physical plausibility so scenes remain deployable for policy training.
4. Experiments
SAGE is evaluated on scene quality, physical stability, open-vocabulary flexibility, dataset scale, and policy learning. Across these experiments, it generates stable and diverse scenes while supporting scalable policy training and generalization.
- Common scene types: SAGE achieves the best results across visual-quality and physical-stability metrics on Bedroom, Kitchen, and Living Room scenes.Scores are averaged across 10 generated scenes per room type.
- Open-vocabulary generation: SAGE generates diverse, semantically coherent open-vocabulary scenes that follow user prompts and distinctive visual styles, including long-tail spaces.Examples span Gym, Office, Cyberpunk game den, and Starry-night bedroom.
- Ablation study: The physics critic reduces collisions from 7.8% to 1.9% and raises stability to 99.6%, while combining visual and physics critics gives the best overall performance.The ablation identifies visual feedback and simulator-in-the-loop validation as complementary components.
- Dataset: SAGE-10k contains 10k scenes across 50 room types and 50 styles, including 565K uniquely generated 3D objects.The dataset is pre-generated to support community research at scale.
- Extensions: SAGE extends to multi-room, image-conditioned, and articulated-object generation without architectural changes or with modular asset integration.Image-conditioned scenes remain semantically coherent though not pixel-aligned, and multi-room generation supports parallel room updates.
- Policy learning: Policy success rates increase with scene count, approach the underlying motion planner, and outperform baselines that reach less than one-third of SAGE’s final success rate.SAGE-trained policies also achieve higher success on baseline-generated held-out scenes despite distribution shift.
5. Conclusion
SAGE converts open-vocabulary text prompts into simulation-ready environments and scales them for embodied learning. Policies trained on generated data scale with scene diversity and demonstrations while generalizing to unseen objects and layouts.
- SAGE turns open-vocabulary text prompts into simulation-ready indoor environments by orchestrating layout and asset generators with visual and physics critics via MCP.
- The same pipeline scales environments through multi-level scene augmentation and automatic action synthesis for embodied learning.
- Policies trained purely on generated data show clear scaling with scene diversity and demonstration count and improve generalization to unseen objects and layouts.
- The current scope emphasizes indoor scenes and rigid-body physics, while action generation focuses on pick, place, and navigation compositions.
A. Additional Experiment Results
Additional experiments diversify generated environments while preserving task semantics. Object variations change shapes and textures, whereas layout augmentation regenerates backgrounds and repositions task-relevant objects.
- The augmentation experiments include object category-level and scene layout-level variants of previously generated scenes.
- Object Category-level Augmentation: Category-level augmentation changes selected objects’ geometry and texture while maintaining their original categories.
- Scene Layout-level Augmentation: Layout-level augmentation regenerates room geometry and task-irrelevant objects while preserving task-relevant objects and their task specification.
- Robot Action Generation: Robot action data combines end-effector poses with multiple camera views for Pick-and-Place and Mobile Manipulation.
A.2.2. Policy Inference
Policy inference feeds observed camera views into the trained policy network and executes its inferred robot actions. Demonstrated failures mainly involve inference randomness, distant objects, or difficult-to-reach grasp locations.
- Policy inference uses observed camera views as input and executes the robot actions inferred by the trained policy.
- Failure cases are mostly attributed to policy-inference randomness, far-away objects, or difficult grasp locations.
A.3. Runtime Analysis
Runtime is dominated by object generation and simulation-in-the-loop stability validation, while parallel hardware reduces action-generation cost. A 20-object scene takes approximately 10 minutes, with scene-generation time scaling linearly with additional objects.
- Scene Generation: 15 seconds per object with TRELLIS falls to 2–3 seconds per object when object generation is parallelized across 8 GPUs.
- Scene Generation: A scene with 20 objects takes approximately 10 minutes, and generation time scales linearly for additional objects.
- Action Generation: 2–3 seconds per Pick-and-Place demonstration is achieved by parallelizing motion planning across 8 environments.
- Action Generation: Mobile Manipulation demonstrations require 8–10 seconds each under parallelized simulation.
- Policy Training: Diffusion Policy training takes several hours to converge on the generated robot action data.
B. Additional Implementation Details
This section details the proposed agent-driven scene generation, robot action generation, and policy learning components.
- The implementation details cover agent-driven scene generation.
- The section also covers robot action generation.
- Policy learning is included as a further implementation component.
B.1.1. Overview
SAGE uses an agent-driven, MCP-based system that dynamically composes scenes with generators and critic feedback. It augments objects and layouts, validates physics, and supports embodied-task policy training in parallel simulation environments.
- Overview: SAGE dynamically invokes MCP-based generator tools to construct scenes through iterative, adaptive operations.The tools initialize layouts, add assets, and adjust existing content based on agent reasoning and critic feedback.
- Generators: The scene initializer generates room layouts, connectivity, and floor and wall materials from language-based descriptions.Multi-room layouts receive generated connections and doors, while materials are produced from text descriptions.
- Generators: The asset placer generates 3D objects, post-processes their meshes, and places them using category-specific constraints and collision-aware search.Placement distinguishes floor, wall, and on-top objects, then uses grid sampling and depth-first search for positions and orientations.
- Critics: Visual and physics critics provide feedback on semantic composition, missing task objects, placement adjustments, and stability failures.The physics critic validates stability after additions, movements, and removals during physical simulation.
- Augmentation: SAGE scales scene diversity through object category-level and scene layout-level augmentation while preserving task-relevant structure.Category augmentation reuses supporting relationships and validates physics; layout augmentation regenerates background scenes while preserving task-relevant objects.
- Embodied Tasks: Pick-and-Place and Mobile Manipulation use grasping, motion planning, navigation, and simulator parallelism for embodied-policy training.Isaac Sim and Isaac Lab simulate 8 Pick-and-Place or 2 Mobile Manipulation environments per GPU.