Source-linked AI summary
PhyScene: Physically Interactable 3D Scene Synthesis for Embodied AI
Yandan Yang, Baoxiong Jia, Peiyuan Zhi, Siyuan Huang
TL;DR
Existing scene-synthesis methods prioritize realism while leaving physical plausibility and interactivity insufficiently explored, limiting their fit for embodied AI. PHYSCENE uses conditional diffusion with collision, room-layout, and reachability guidance to generate interactive scenes. It consistently improves physical plausibility and interactivity over state-of-the-art baselines, while remaining limited to certain room types and excluding small objects.
Problem
Existing scene-synthesis methods prioritize naturalness and realism, while physical plausibility and interactivity needed for embodied AI remain largely unexplored.
Method
PHYSCENE uses a conditional diffusion model with guidance functions for collision avoidance, room layout, reachability, and physical interaction constraints.
Results
PHYSCENE consistently improves state-of-the-art baseline performance on physical plausibility and interactivity metrics, with gains across collision and reachability evaluations.
Takeaways & Limitations
The generated scenes provide a pipeline for large-scale interactable scene synthesis that maintains visual realism and supports embodied AI research.
Takeaways & Limitations
Due to data limitations, PHYSCENE covers only limited room types and excludes small objects, constraining applications such as pick-and-place tasks.
Abstract
from arXiv · showhide
With recent developments in Embodied Artificial Intelligence (EAI) research, there has been a growing demand for high-quality, large-scale interactive scene generation. While prior methods in scene synthesis have prioritized the naturalness and realism of the generated scenes, the physical plausibility and interactivity of scenes have been largely left unexplored. To address this disparity, we introduce PhyScene, a novel method dedicated to generating interactive 3D scenes characterized by realistic layouts, articulated objects, and rich physical interactivity tailored for embodied agents. Based on a conditional diffusion model for capturing scene layouts, we devise novel physics- and interactivity-based guidance mechanisms that integrate constraints from object collision, room layout, and object reachability. Through extensive experiments, we demonstrate that PhyScene effectively leverages these guidance functions for physically interactable scene synthesis, outperforming existing state-of-the-art scene synthesis methods by a large margin. Our findings suggest that the scenes generated by PhyScene hold considerable potential for facilitating diverse skill acquisition among agents within interactive environments, thereby catalyzing further advancements in embodied AI research. Project website: http://physcene.github.io.
1. Introduction
Scene synthesis is increasingly important for embodied AI, but existing methods do not adequately support physical constraints and object interactivity. PHYSCENE addresses this gap with guided diffusion for physically plausible, interactive scenes.
- Embodied AI increases demand for scene synthesis that supports agents learning navigation and manipulation skills.
- Existing scene datasets and methods often use manually created scenes with non-interactable objects, limiting physical simulation and object reachability.
- PHYSCENE combines diffusion-based scene distribution learning with physical-commonsense guidance for functionally interactive and physically plausible scenes.
- PHYSCENE converts collision avoidance, room layout, and reachability constraints into guidance functions for scene generation.
2. Related Work
Prior scene-synthesis evaluation emphasizes visual realism, while physical plausibility and interactivity remain insufficiently addressed. PHYSCENE targets this gap by uniting realistic scene synthesis with constraints needed for embodied AI.
- Indoor scene synthesis commonly predicts object layouts using bounding boxes, semantic labels, shape features, scene graphs, and spatial or category priors.
- Common perceptual metrics such as FID and KID do not evaluate physical plausibility or interactivity in generated scenes.
- The 3D-FRONT dataset contains frequent physically implausible layouts, while object manipulation and reachability remain understudied.
- PHYSCENE aims to provide large-scale interactable scene synthesis while maintaining visual realism and naturalness.
- Optimization-based methods have addressed physical plausibility for pose and motion, but physical plausibility in scene generation has received limited study.
3. PHYSCENE
PHYSCENE models scene layouts with conditional diffusion and guides denoising using collision, room-layout, and reachability constraints. Its design also accommodates articulated objects and interaction constraints for physically interactive scenes.
- 3.3. Guidance for Physical Interactivity: The method guides diffusion with collision, room-layout, and reachability functions that are combined during constrained sampling.
- 3.1. Articulated Objects in Scene Synthesis: Each scene object is represented by a semantic label, size, orientation, location, and shape feature, with shape features supporting retrieval across asset libraries.
- 3.2. Conditional Diffusion for Layout Modeling: PHYSCENE uses conditional diffusion to learn scene layouts from noise while conditioning generation on the floor plan.
- 3.3. Guidance for Physical Interactivity: Constraint guidance tilts the original scene-layout distribution by perturbing the denoising Gaussian transition toward constraint satisfaction.
- 3.3. Guidance for Physical Interactivity: PHYSCENE approximates collision violations using 3D bounding-box overlap and object centers rather than directly computing mesh collisions.
- 3.3. Guidance for Physical Interactivity: The method can be extended with simple modifications to support articulated interactions such as grasping and opening, and rigid-object interactions such as sitting.
4. Experiment
Experiments evaluate PHYSCENE against ATISS and DiffuScene on perceptual quality, physical plausibility, floor-plan adherence, reachability, and articulated-object embedding. PHYSCENE achieves strong overall physical-interactivity results, while guidance ablations show a balance among competing constraints.
- Experiment Setup: Evaluation measures perceptual quality with FID, KID, SCA, and CKL, while physical interactivity uses collision, floor-plan, reachability, and walkability metrics.The experiments use 3D-FRONT rooms and retrieve objects from 3D-FUTURE and GAPartNet.
- Unconditioned Scene Synthesis: PHYSCENE achieves state-of-the-art results on almost all metrics, with especially significant improvements in physical plausibility metrics such as Colobj and Colscene.The comparison covers unconditional scene synthesis on 3D-FRONT and reports reduced collision rates alongside better visual plausibility.
- Floor-conditioned Scene Synthesis: PHYSCENE surpasses baselines on collision metrics and CKL, and consistently improves over DiffuScene across physical-interactability metrics.ATISS performs favorably on Rout and Rreach except in the Bedroom setting, which the authors attribute to prioritizing floor-plan constraints over collision avoidance.
- Scene Synthesis with Articulated Objects: Articulated-object guidance uses joint-expanded 3D bounding boxes to account for objects manipulated to their fullest extent during collision evaluation.The guided substitution improves over previous methods, although articulated-object collision rates are much higher than those for rigid objects.
- Ablation Study on Guidance: The final guidance configuration balances collision, floor-plan, and reachability objectives, improving each corresponding metric despite conflicts between Colobj and Rout.Collision guidance pushes objects apart, whereas floor-plan guidance pushes them closer to fit the scene; qualitative results visualize the resulting improvement.
5. Conclusion
PHYSCENE is a guided conditional diffusion model for physically interactable scene synthesis, with experiments showing consistent improvements over state-of-the-art baselines on physical plausibility and interactivity. The method remains limited by room-type and object coverage due to data limitations.
- PHYSCENE uses guidance functions for object collision, room layout, and interactivity within each diffusion inference step.These constraints are integrated into the denoising process to support physically plausible and interactive scenes.
- PHYSCENE consistently improves physical plausibility and interactivity over state-of-the-art baseline models.
- Data limitations restrict PHYSCENE to limited room types and exclude small objects.This constrains applicability to embodied-AI tasks involving small-object manipulation, such as pick-and-place.
Supplementary Material
The supplementary material describes the forward diffusion process, including repeated Gaussian-noise transitions and a reparameterized expression relating noisy samples to the original data.
- The forward diffusion process repeatedly adds Gaussian noise with variance β_t to obtain x_t from x_{t−1}.Each transition is modeled by q(x_t|x_{t−1}) as a Gaussian distribution.
- The full forward process factors into repeated transitions q(x_t|x_{t−1}) across timesteps 1 through T.
- Reparameterization with ˆα_t expresses x_t as a scaled x_0 plus Gaussian noise.The scaling terms are determined by the cumulative product of α values.
- Algorithm 2 provides the detailed procedure for calculating reachability guidance.
B. Data Processing
The data-processing pipeline prepares watertight and sequential articulated-object representations, aligns assets across 3D-FUTURE and GAPartNet, and supports reachability-guidance computation from scene layouts.
- 3D-FUTURE CAD models are re-meshed because their non-watertight geometry prevents direct collision-rate calculation.The re-meshed models fill hollow regions for collision computation while remaining perceptually similar to the originals.
- Reachability guidance takes a floor plan and 3D object bounding boxes as input, then uses walkability costs and shortest-path search.The procedure initializes walkable regions, adds Gaussian object costs, finds connected areas, and computes an agent path.
- Articulated objects are represented by integrated meshes recording states from closed to open for collision computation.The sequential representation is generated from GAPartNet CAD models and URDF files.
- Assets from 3D-FUTURE and GAPartNet are mapped across corresponding interactive categories for retrieval.Examples include mapping 3D-FUTURE wardrobes to GAPartNet StorageFurniture.
- GAPartNet includes articulated StorageFurniture and Table models with revolute and prismatic joints.StorageFurniture accounts for 324 of 1045 models in the dataset.
C. Additional Results
Additional results examine physical plausibility in existing scenes and compare collision rates across datasets and synthesis methods. The supplied passages also identify the category distribution used for cross-dataset retrieval.
- Existing 3D-FRONT scenes contain physical-plausibility violations including object collisions and objects outside rooms.
- StorageFurniture and Table comprise the largest proportions of GAPartNet and are used to composite interactable scenes with cross-dataset retrieval.
- PHYSCENE has a lower collision rate than 3D-FRONT in most cases, while ATISS and DiffuScene have higher collision rates than 3D-FRONT.
C.2. Guidance on Different Agent Size
Reachability guidance adapts to the embodied agent’s size, while physical guidance reduces collision-related implausibility inherited from training data.
- C.2. Guidance on Different Agent Size: Agent-size-specific guidance matters: guidance for size 0.2 leaves a size-0.5 agent able to reach only half the room, whereas size 0.5 makes the whole room reachable.Guidance and evaluation use agent sizes 0.2, 0.3, and 0.5 meters.
E. Guidance Details
The method applies guidance during late denoising steps, when the layout is being refined, and computes guidance over object locations and angles rather than sizes.
- E. Guidance Details: PhyScene visualizations cover Bedroom and Dining Room synthesis results on 3D-FRONT.
- E. Guidance Details: Adding guidance during the final denoising steps performs best because layouts stabilize after early shrinkage and middle-stage ordering.The layout changes only slightly at the final steps.
- E. Guidance Details: Guidance moves objects through location and angle gradients, while avoiding size gradients because they produce excessively small object thicknesses.The guidance objective uses bounding-box object size, location, and angle, but gradients are calculated only for location and angle.
F. Collision with Finer 3D Representations
Finer occupancy-field representations slightly improve collision granularity, but their repeated computation creates substantial overhead, motivating a speed–granularity trade-off.
- F. Collision with Finer 3D Representations: The collision guidance represents objects with bounding boxes while using occupancy fields as indicators for loss calculation.The objective penalizes bounding-box collisions only when corresponding occupancy fields collide.
- F. Collision with Finer 3D Representations: Figure A.10 compares different 3D representations used in collision guidance.
- F. Collision with Finer 3D Representations: Occupancy-field collision computation is 55 times slower than the bounding-box-based alternative, despite slightly finer collision granularity.The authors leave better fine-representation trade-offs as future work.
G. Agent Interaction
The interaction extension estimates reachability to articulated object endpoints and visualizes guided synthesis with articulated objects across layouts and room settings.
- G. Agent Interaction: The proposed extension enlarges articulated objects to their fully opened state, plans paths to interactable parts, and moves obstacles along those paths.The same planning idea can model rigid-object interactions such as sitting.
- G. Agent Interaction: A simplified articulated-object interaction estimate raises the interactiveness rate from 0.101 to 0.143.The rate measures whether robots can reach object-part endpoints when the parts are maximally interacted.
- G. Agent Interaction: The diffusion model generates object layouts jointly with a fixed number of vectors, embedding overall scene and inter-object information.This contrasts with ATISS’s autoregressive prediction process.
- G. Agent Interaction: PhyScene visualizations compare synthesis without and with guidance and show generated scenes containing articulated objects.Additional results use rectangular ProcTHOR floor plans and articulated objects.