PaperlayerView exact revision on arXiv

Source-linked AI summary

Repurposing 3D Generative Model for Autoregressive Layout Generation

Haoran Feng, Yifan Niu, Zehuan Huang, Yang-Tian Sun, Yuxin Peng, Lu Sheng

arXiv:2604.16299v2cs.CV

TL;DR

Existing layout methods struggle to represent geometric relations and physical constraints directly, motivating native 3D layout generation. LaviGen repurposes a 3D generative model for autoregressive scene construction with dual-guidance self-rollout distillation, achieving higher physical plausibility and faster computation. The paper also identifies annotation quality and text-conditioning robustness as areas for future improvement.

  • Problem

    Prior layout methods have limited ability to model geometric relations and physical constraints directly, causing spatial inconsistencies or costly indirect supervision.

  • Method

    LaviGen uses an adapted autoregressive 3D diffusion model that integrates scene, object, and instruction information, with dual-guidance self-rollout distillation.

  • Results

    19% higher physical plausibility and roughly 65% lower computational time were achieved on the LayoutVLM benchmark than existing approaches.

  • Takeaways & Limitations

    Operating in native 3D space supports physically plausible, semantically coherent, efficient, and editable layout synthesis.

  • Takeaways & Limitations

    The authors plan to collect more high-quality labeled data and improve text-conditioning mechanisms to strengthen robustness and semantic reliability.

Abstract

from arXiv · show

We introduce LaviGen, a framework that repurposes 3D generative models for 3D layout generation. Unlike previous methods that infer object layouts from textual descriptions, LaviGen operates directly in the native 3D space, formulating layout generation as an autoregressive process that explicitly models geometric relations and physical constraints among objects, producing coherent and physically plausible 3D scenes. To further enhance this process, we propose an adapted 3D diffusion model that integrates scene, object, and instruction information and employs a dual-guidance self-rollout distillation mechanism to improve efficiency and spatial accuracy. Extensive experiments on the LayoutVLM benchmark show LaviGen achieves superior 3D layout generation performance, with 19% higher physical plausibility than the state of the art and 65% faster computation. Our code is publicly available at https://github.com/fenghora/LaviGen.

1. Introduction

3D layout generation must capture both semantic dependencies and geometric relationships while respecting physical constraints. LaviGen addresses this by generating layouts autoregressively in native 3D space, using 3D geometric priors and dual-guidance distillation to improve plausibility and efficiency.

  • Motivation: Coherent 3D layouts arrange objects in semantically consistent, physics-compliant configurations while encoding their geometric relationships.Such layouts are important for realistic and interactive VR/AR environments.
  • Limitations of Existing Methods: Language-based methods provide semantic priors but can produce collisions, inter-penetrations, and floating because they lack physical modeling.Vision-based supervision improves visual plausibility but is computationally costly and does not directly model 3D constraints.
  • LaviGen: LaviGen repurposes 3D generative models to learn directly from geometric distributions and place objects sequentially in native 3D space.The autoregressive formulation supports object addition and removal while providing greater controllability than monolithic generation.
  • LaviGen: Exposure bias in long autoregressive sequences can accumulate spatial errors, while the model must align global scene geometry with each object’s canonical space.These challenges motivate post-training designed for more stable and coherent generation.
  • LaviGen: Dual-guidance self-rollout distillation combines holistic scene guidance with step-wise scene-object alignment supervision to mitigate error accumulation.The framework integrates semantic information while generating updated scenes with semantically consistent object placement.
  • Results: 19% higher physical plausibility and roughly 65% lower computational time were achieved on the LayoutVLM benchmark versus existing approaches.LaviGen also supports layout completion and editing without relying on native-language or vision-based representations.

2. Related Work

Prior 3D layout methods use coordinate regression, language plans, or 2D visual optimization, each operating outside native 3D representations. LaviGen instead uses a native 3D autoregressive process backed by geometric priors to model spatial relations and physical constraints.

  • 3D Layout Generation: Early autoregressive transformers directly regressed object placements but often neglected geometric semantics, producing spatial inconsistencies.ATISS is identified as a representative early learning-based approach.
  • 3D Layout Generation: Foundation-model approaches treat layout as a language task, gaining semantic coherence through structured textual plans but struggling with explicit physical constraints.Reported artifacts include object collisions and floating.
  • 3D Layout Generation: LayoutVLM uses rendered images and differentiable optimization to refine poses, improving plausibility but introducing computationally expensive optimization.Its 2D supervision is also described as incomplete for complex 3D interactions.
  • 3D Generative Models: Modern 3D generative models learn rich spatial relationships from large-scale 3D data, providing geometric priors for physically consistent layout generation.These models exhibit high 3D fidelity and structural consistency.
  • Autoregressive Diffusion and Distillation: Sequential object placement makes layout generation a long-sequence autoregressive problem, where conventional bidirectional-attention diffusion models perform poorly and exposure bias accumulates errors.Autoregressive diffusion and distillation methods are motivated by this combination of modeling and training challenges.

3. Methodology

LaviGen repurposes structured 3D generative models for autoregressive layout synthesis directly in native 3D space. It combines scene–object diffusion conditioning, identity-aware embeddings, and dual-guidance self-rollout distillation to produce coherent layouts while addressing accumulated autoregressive errors.

  • Framework overview: LaviGen repurposes a pretrained 3D generative model for language-conditioned, editable layout synthesis in native 3D space.The framework uses structured 3D priors to support spatially coherent and physically plausible layouts.
  • Structured 3D prior: The structure-level 3D prior predicts sparse voxel occupancies, modeling object spatial organization and physically and semantically plausible relationships.LaviGen retains the structure-generation stage of TRELLIS rather than its fine-grained geometry and appearance refinement stage.
  • Autoregressive generation: At each autoregressive step, LaviGen encodes the current scene, target object, and instruction to generate an updated layout that becomes the next state.Aligned furniture meshes are placed into the generated layout after downsampling and Iterative Closest Point registration.
  • Diffusion architecture: The adapted diffusion model concatenates scene, object, and noisy target-scene latents, then uses textual conditioning and identity-aware embeddings for denoising.The denoised output is decoded into the updated scene state; identical positional encodings align scene and noisy-latent streams, while distinct encodings preserve object identity.
  • Dual-guidance distillation: Self-rollout post-training exposes the student to its own generated contexts and combines holistic scene-level with step-wise object-level supervision.Holistic guidance alone provides no intermediate correction and lacks object-placement guidance, motivating the dual-guidance objective.

4. Experiments

Experiments compare LaviGen with text-driven baselines using qualitative, quantitative, user-study, completion, editing, and ablation evaluations. Across these settings, LaviGen provides stronger physical plausibility and coherent native-3D manipulation.

  • Main Comparison: LaviGen models inter-object relations directly in 3D, producing physically plausible arrangements with fewer collisions and floating artifacts than competing baselines.LayoutGPT remains semantically coherent but often violates physical constraints, while LayoutVLM reduces boundary violations but retains collision and floating problems.
  • Main Comparison: LaviGen achieves the best CF and IB scores, indicating strong collision avoidance and compliance with scene boundaries.CF measures collision-free placement, while IB measures whether objects remain inside scene boundaries.
  • User Study: A user study with 43 participants and 430 responses finds LaviGen strongest in physical plausibility and overall quality, while semantic consistency remains comparable.Participants selected the best model across physical plausibility, semantic consistency, and overall quality.
  • Applications: LaviGen completes partially specified scenes by placing objects directly into the current 3D scene with physical plausibility and semantic coherence.This capability is presented as useful for settings such as robotic perception, AR/VR environment generation, and autonomous navigation.
  • Applications: LaviGen supports context-aware layout editing, including object insertion, removal, and replacement with spatially coherent and semantically consistent results.The editing formulation swaps autoregressive targets so the model can remove objects and regenerate them in context.
  • Ablation Study: The ablation shows progressive gains from identity-aware embeddings, holistic guidance, and step-wise guidance, with the full model avoiding collisions and small-object inversion artifacts.Identity-aware embeddings improve coherence but retain exposure-bias collisions; holistic distillation reduces generation time but harms object fitting before step-wise guidance is added.

5. Conclusion

LaviGen repurposes structured 3D generative priors for autoregressive layout generation directly in native 3D space. Its dual-guidance self-rollout distillation improves spatial accuracy and efficiency across baselines.

  • Conclusion: LaviGen generates layouts by modeling spatial relationships between the current scene and input objects in an adapted autoregressive 3D diffusion model.The framework targets physically plausible and semantically coherent layouts rather than language-only scene descriptions.
  • Conclusion: Dual-guidance self-rollout distillation addresses exposure bias in long-sequence generation and improves training stability and physical fidelity.The approach combines holistic and step-wise guidance during distillation.
  • Conclusion: Experiments show superior spatial accuracy and efficiency across baselines, supporting 3D generative models as a foundation for geometry-aware, semantically controllable scene generation.The conclusion frames the result as a principled 3D generative paradigm for layout generation.

6. Implementation Details

LaviGen uses a structured 3D generative prior, adapts it into an autoregressive teacher, and distills that teacher into an efficient student through dual guidance.

  • Base 3D Generative Model: The base model follows TRELLIS by reusing its structured variational autoencoder and predicting sparse voxel occupancies for spatial organization.The structure-level generation stage is retained to encode physically and semantically plausible spatial relationships.
  • Autoregressive Layout Generation: The teacher jointly receives the current scene and target object, enabling comprehensive modeling of their geometric relationships.The architecture minimizes changes to the pretrained 3D generative model while distinguishing scene and object information.
  • Self-Rollout Distillation: Dual-guidance self-rollout distillation mitigates exposure bias by distilling a pretrained few-step student generator from holistic and step-wise teachers.The student is trained to use scene-level and intermediate corrective supervision during sequential rollout.
  • Self-Rollout Distillation: The dual objective Ldual = Lholistic + Lstep minimizes reverse KL divergence using score differences between student and teacher distributions.The critic estimates the student score, while the fixed teachers provide holistic or step-wise score functions.

7. Additional Results

Additional evaluations test long-sequence generation and backbone transfer. LaviGen handles scenes with more than 20 objects and retains strong quality across different 3D generative backbones.

  • Long-Sequence Generation: LaviGen supports a “train short, test long” paradigm and generates scenes containing more than 20 objects.The reported evaluation compares methods on 8–10 objects, while the long-sequence demonstration extends beyond 20 objects.
  • Backbone Generalization: Applying LaviGen to TRELLIS with its original CLIP text encoder preserves high physical plausibility and semantic coherence.This transfer test omits the additionally trained Qwen encoder.

8. Limitations and Future Work.

The paper identifies challenges in training and deploying autoregressive layout generation, while outlining future improvements for difficult spatial arrangements and semantic reliability.

  • Method and implementation: LaviGen’s dual-guidance self-rollout distillation is organized around holistic and step-wise teacher guidance during autoregressive generation.The algorithm uses an initial state, an object sequence, a text prompt, and student-generator rollouts before step-wise processing.
  • Method and implementation: The rollout procedure repeatedly conditions on the current scene, object, and text prompt while generating updated scene states.It initializes noisy latent states, applies the student generator, and appends predicted scene states for later processing.
  • Generation diversity: The paper reports diverse layouts generated from the same input instruction, indicating variation across outputs under fixed instructional input.The supplied figure caption identifies this as Figure 10’s focus but does not provide a quantitative diversity measure.
  • Limitations and future work: The stated future-work boundary is handling complex spatial configurations or object arrangements more robustly.The authors plan to improve annotation quality and investigate more advanced text-conditioning mechanisms.
Loading 2604.16299v2…