PaperlayerView exact revision on arXiv

Source-linked AI summary

Diffusion-based Generation, Optimization, and Planning in 3D Scenes

Siyuan Huang, Zan Wang, Puhao Li, Baoxiong Jia, Tengyu Liu, Yixin Zhu, Wei Liang, Song-Chun Zhu

arXiv:2301.06015v1cs.CV

TL;DR

3D scene understanding requires generation, optimization, and planning that remain consistent with scenes, physics, and goals, while prior approaches exhibit posterior collapse and disconnected modules. SceneDiffuser unifies these tasks in a conditional diffusion model using iterative guided sampling. Across diverse pose, motion, grasp, navigation, and robot-arm tasks, it reports substantial improvements over previous models, while requiring slower computation and objective-dependent hyperparameter tuning.

  • Problem

    Existing 3D scene models suffer from posterior collapse and disconnected generation, optimization, and planning modules, limiting consistent scene-conditioned task solving.

  • Method

    SceneDiffuser uses conditional diffusion with iterative guided sampling to jointly perform scene-aware generation, physics-based optimization, and goal-oriented planning.

  • Results

    SceneDiffuser outperforms previous models by a large margin across diverse 3D scene understanding tasks, including generation and planning.

  • Takeaways & Limitations

    The unified model demonstrates efficacy and flexibility across human, grasping, navigation, and robot-arm scene-understanding tasks.

  • Takeaways & Limitations

    SceneDiffuser has slower training and test speed than previous scene-conditioned generative models, and optimization and planning depend strongly on objective design and hyperparameter tuning.

Abstract

from arXiv · show

We introduce SceneDiffuser, a conditional generative model for 3D scene understanding. SceneDiffuser provides a unified model for solving scene-conditioned generation, optimization, and planning. In contrast to prior works, SceneDiffuser is intrinsically scene-aware, physics-based, and goal-oriented. With an iterative sampling strategy, SceneDiffuser jointly formulates the scene-aware generation, physics-based optimization, and goal-oriented planning via a diffusion-based denoising process in a fully differentiable fashion. Such a design alleviates the discrepancies among different modules and the posterior collapse of previous scene-conditioned generative models. We evaluate SceneDiffuser with various 3D scene understanding tasks, including human pose and motion generation, dexterous grasp generation, path planning for 3D navigation, and motion planning for robot arms. The results show significant improvements compared with previous models, demonstrating the tremendous potential of SceneDiffuser for the broad community of 3D scene understanding.

1. Introduction

SceneDiffuser addresses limitations in 3D scene understanding by unifying scene-conditioned generation, physics-based optimization, and goal-oriented planning within a diffusion-based iterative sampling framework. It improves diversity, physical plausibility, and long-horizon generalization across several 3D tasks.

  • Motivation: Existing 3D scene models face posterior collapse in cVAE-based generation and discrepancies between separately trained generation, optimization, and planning modules.Posterior collapse limits diversity, while post-optimization and standalone planners can produce inconsistent outputs or gaps during inference.
  • Method: SceneDiffuser uses a conditional diffusion model with unified iterative guided sampling for scene-aware generation, physics-based optimization, and goal-oriented planning.The framework jointly solves these components during inference rather than treating optimization and planning as separate modules.
  • Generation: Forward diffusion acts as data augmentation that helps SceneDiffuser traverse more scene-conditioned distribution modes and alleviate posterior collapse.The proposed design targets generation diversity under complex 3D scene conditions.
  • Optimization: Physics-based objectives guide every sampling step, enabling differentiable optimization during learning and sampling and facilitating physically plausible generation.The physics objective is integrated as conditional guidance throughout the denoising process.
  • Planning: A scene-conditioned trajectory-level generator provides physics- and goal-aware planning that generalizes to long-horizon trajectories and novel 3D scenes.The planner operates globally over trajectories rather than relying only on isolated stepwise decisions.
  • Evaluation: Across human pose, motion, dexterous grasp, navigation path, and robot-arm motion planning, SceneDiffuser significantly improves results and produces plausible, diverse generations.The reported evaluations cover both generation and planning tasks under 3D scene and object conditions.

2. Related Work

Prior work addresses conditional generation, physical plausibility, and planning in separate streams, but commonly suffers from posterior collapse, inefficient post-optimization, and limited long-horizon or natural-scene generalization. SceneDiffuser combines these capabilities through diffusion-based trajectory and content generation.

  • Conditional Generation in 3D Scenes: Previous conditional generation methods commonly rely on cVAEs and suffer from posterior collapse, particularly for complex 3D scene conditions.The related applications include human pose, motion, and object-conditioned grasp generation.
  • Physics-based Optimization in 3D Scenes: Physics-based post-optimization can be inefficient and cannot be learned jointly with generative models, yielding inconsistent generation results.Prior differentiable approaches also integrate collision and contact constraints into generation frameworks.
  • Planning in 3D Scenes: Existing 3D planning methods often use single-step dynamic models, lacking trajectory-level optimization for long-horizon planning and explicit physical interactions.These deficiencies make generalization to natural scenes challenging when exploration is limited.
  • Diffusion-based Models: Diffusion models learn data distributions through iterative denoising and support flexible conditioning across image, text, video, and 3D generation.Their iterative sampling formulation provides the modeling basis used by SceneDiffuser.
  • Diffusion-based Models: SceneDiffuser is presented as the first framework to combine 3D scene-conditioned generation, optimization, and planning in a unified diffusion model.The related-work discussion positions this integration as the paper’s distinguishing contribution.

3. Background

The paper formulates scene-conditioned task solving as trajectory optimization in a 3D scene and introduces diffusion models as iterative denoising generative processes. Global trajectory optimization uses scene, goal, state, action, and dynamics information to address long-horizon tasks.

  • 3.1. Problem Definition: Given a 3D scene S and goal G, the objective is to generate an optimal trajectory for tasks such as navigation and manipulation.A trajectory contains interleaved states and actions over a discrete horizon N, with transitions defined by a dynamic model.
  • 3.1. Problem Definition: Scene-conditional trajectory optimization maximizes a task objective over trajectories conditioned on the scene and goal.The formulation expresses task solving as optimization rather than isolated action prediction.
  • 3.2. Planning with Trajectory Optimization: Known dynamics allow future states and actions to be optimized jointly and non-progressively over the entire trajectory.Trajectory-level optimization preserves awareness of history and future states for long-horizon tasks.
  • 3.3. Diffusion Model: Diffusion models generate data through a forward process that gradually adds Gaussian noise and a learned reverse process that denoises it.Training uses denoising score matching across multiple noise scales.

4. SceneDiffuser

SceneDiffuser uses diffusion-based guided sampling to unify scene-conditioned generation, physics-based optimization, and goal-oriented planning. Its trajectory-level generator supports diverse scene interactions, physical plausibility, and iterative planning toward specified goals.

  • Unified formulation: SceneDiffuser models goal-oriented planning as trajectory optimization through sampling from a scene-conditioned trajectory distribution.The goal is represented through an optimality objective, while physical and planning objectives guide sampling.
  • Unified formulation: The scene-conditioned generator uses an iterative diffusion denoising process to sample trajectories from pθ(τ_0|S).The model learns reverse-transition parameters and directly samples diverse trajectory modes conditioned on the 3D scene.
  • Objective design: Guidance scales the objective gradient during sampling, and increasing guidance during the final diffusion steps empirically enhances its effect.The design considers both trajectory-level and accumulated step-wise objectives for optimization and planning.
  • Optimization: Physics-based optimization incorporates differentiable scene objectives into guided sampling, producing physically plausible trajectories without a separate planning objective.The optimization objective can encode collision, contact, and intersection relationships with the scene.
  • Planning: Goal-oriented planning uses motion inpainting to preserve previous states while sampling the remaining trajectory toward a defined goal state.The procedure repeats planning steps until reaching the goal or the maximum planning step, then executes actions toward sampled next states.
  • Model architecture: Cross-attention relates the input trajectory to the encoded 3D scene, while optimizer and planner guidance impose physical and goal constraints.The scene supplies keys and values, the trajectory supplies queries, and a feed-forward layer estimates diffusion noise.

5. Experiments

SceneDiffuser is evaluated across five 3D scene understanding tasks, spanning scene-conditioned generation and long-horizon planning. Across these experiments, it improves physical plausibility, grasp success, and planning performance while preserving useful diversity.

  • Experimental scope: Experiments cover human pose and motion generation, dexterous grasp generation, 3D navigation path planning, and robot-arm motion planning.Comparisons include cVAE variants, physics-optimized cVAE models, imitation-based planning, and an L2 heuristic planner.
  • Human pose generation: SceneDiffuser generates more physically plausible human poses than cVAE-based baselines while maintaining comparable diversity, with optimization-guided sampling improving plausible rate by 25%.The improvement is reflected in plausible-rate and non-collision evaluations, alongside qualitative reductions in floating and severely colliding poses.
  • Human motion generation: SceneDiffuser produces high-quality human motions with better plausible rate and contact scores, although start-pose conditioning reduces diversity and optimization adds only marginal improvement.From the same start pose, it generates diverse behaviors such as sitting and walking in unseen 3D scenes.
  • Dexterous grasp generation: SceneDiffuser achieves higher dexterous-grasp success while balancing diversity and grasp success, and guided sampling reduces physically implausible grasping poses without additional training.The qualitative comparison shows fewer colliding or floating grasps than cVAE generation.
  • 3D navigation: SceneDiffuser outperforms behavior cloning and deterministic planning baselines for unseen-scene navigation, requiring fewer planning steps while maintaining a higher success rate.Its guided sampling combines trajectory-distribution knowledge with planning objectives to avoid obstacles and dead-ends in long-horizon tasks.
  • Robot-arm planning: For robot-arm motion planning, SceneDiffuser consistently outperforms both baselines, while successful-trial planning steps remain comparable to the deterministic planner.The evaluation considers reaching the goal pose within a limited number of steps.

6. Conclusion

SceneDiffuser is a general conditional generative model for 3D-scene generation, optimization, and planning, with scene-aware, physics-based, and goal-oriented properties. Future work includes richer 3D representations, outdoor scenes, and language-guided generation and planning.

  • SceneDiffuser unifies generation, optimization, and planning in a conditional generative model for 3D scenes.
  • Its design is scene-aware, physics-based, and goal-oriented, and it outperforms previous models across various tasks.
  • Future extensions include RGB-D, semantic, BEV, video, mesh, and NeRF scene representations, though these conditions require substantial 3D training data.
  • The authors also propose extending SceneDiffuser to outdoor autonomous-driving scenes and combining it with language models for natural-language-controlled generation and planning.

B. Model Architectures

SceneDiffuser uses task-specific scene and object encoders with attention-based trajectory processing, while optimization objectives encode contact, collision avoidance, smoothness, and goal-reaching behavior.

  • Human pose and navigation tasks use a PointTransformer scene encoder whose features serve as keys and values in cross-attention.
  • Trajectory features are fused with denoising-timestep embeddings, processed by self- and cross-attention, and used to estimate noise.
  • Dexterous grasp generation uses PointNet object features reshaped into tokens and feature dimensions before cross-attention.
  • Navigation objectives generate collision-free paths toward targets, while robot-arm planning uses an objective analogous to the navigation formulation.

D. Implementation Details

Implementation uses explicit body and robot representations, task-specific datasets and simulation settings, and scheduled guidance during diffusion sampling.

  • SMPL-X represents human bodies with 79 parameters and maps them to watertight meshes for collision and contact modeling.
  • Human-generation evaluation samples 1000 examples per test scene and reports diversity, non-collision, and contact metrics.
  • Dexterous grasp experiments use Shadowhand and split MultiDex into 48 seen and 10 unseen objects for training and testing.
  • Navigation data comprise 61 ScanNet scenes and about 6.3k shortest-path trajectories, with a fixed trajectory horizon of 32 during training.
  • During inference, guidance scales are task-specific, including optimization and planning coefficients for navigation and planning guidance of 0.2 for robot arms.
  • Robot-arm experiments use a seven-joint Franka Emika and 19,800 collision-free trajectories from 200 clustered tabletop scenes.
  • The guidance scale is scheduled by dividing it by Σ rather than keeping it constant, strengthening adaptation across denoising steps.

E. Additional Ablative Experiments

Ablations examine architecture, diffusion settings, optimization guidance, trajectory length, and planning objectives across SceneDiffuser experiments.

  • The ablations vary scene encoders, noise-prediction modules, diffusion steps, optimizer scale coefficients, fixed frames, and navigation planning objectives.

E.1. Model Architecture

The ablations examine scene encoders, attention architectures, and diffusion-step choices for dexterous grasp generation. Results show trade-offs between grasp success, diversity, and generation quality.

  • Scene Encoder: PointNet’s global features favor mean-pose learning, achieving higher grasping success, whereas PointNet++ provides more local scene information.The comparison evaluates PointNet and PointNet++ as object-feature extractors.
  • Model Architecture: Self-attention better captures the joint distribution of inputs and conditions, improving generation quality and success rate at slightly lower diversity.Cross-attention learns queries from the input and keys and values from scene conditions; self-attention concatenates input and scene features.
  • Diffusion Steps: Diffusion steps balance diversity and success: T = 30 gives the best grasp-pose diversity, while T = 1000 gives the best overall success rate.The ablation evaluates diffusion steps ranging from 30 to 1000.

E.3. Scale Coefficient

The ablations show that optimization and planning guidance require calibrated scale coefficients and inpainting horizons. Guidance strength trades off collision avoidance, stability, trajectory diversity, and obstacle escape.

  • Optimization Guidance: Larger optimization guidance reduces depth collisions but moves grasp poses away from objects, lowering grasp stability and success rate.The scale coefficient is varied from 0.0 to 1.0 in dexterous grasp generation.
  • Planning Guidance: Both too-small and too-large planning coefficients reduce navigation performance: weak guidance is insufficient, while strong guidance suppresses diversity and obstacle escape.The stated explanation links excessive guidance to dead-end trapping and reduced trajectory diversity.
  • Inpainting Horizon: Fixing the first 15 of 32 trajectory frames during inpainting produces the best path-planning performance.The comparison fixes 1, 7, 15, 23, or 31 initial frames.
  • Planning Objectives: The study compares four planning objectives based on final-frame distance, all-frame distance, a last-frame objective, and the closest-to-target frame.These alternatives are evaluated against the objective in Eq. (A5).
  • Trainable Scaling: A timestep-conditioned MLP can learn the optimization-guidance scale while the pretrained diffusion model remains fixed.The MLP maps each timestep embedding to a scalar scaling factor.

G. More Qualitative Results

Additional qualitative results cover pose and motion generation, navigation, dexterous grasping, and robot-arm motion planning. The examples extend evaluation across varied 3D scenes and unseen objects.

  • Pose Generation: Additional pose-generation examples are provided for human pose synthesis in 3D scenes.The corresponding appendix figure presents more qualitative results.
  • Motion Generation: Additional motion-generation samples show human motions from the same start pose across other scenes.Rendered animations are available in the supplemental demo video.
  • Path Planning: Qualitative navigation results visualize planned paths through 3D scenes, with red balls marking ordered trajectory points and a green ball marking the target.The red sequence runs from lightest to darkest red.
  • Dexterous Grasp Generation: Additional dexterous grasp results evaluate objects unseen during training.The examples concern grasp-pose generation for 3D objects.
  • Robot-Arm Motion Planning: Robot-arm motion-planning results are rendered as animations for qualitative visualization.The supplemental demo video contains the qualitative results.
Loading 2301.06015v1…