PaperlayerView exact revision on arXiv

Source-linked AI summary

LayoutVLM: Differentiable Optimization of 3D Layout via Vision-Language Models

Fan-Yun Sun, Weiyu Liu, Siyi Gu, Dylan Lim, Goutam Bhat, Federico Tombari, Manling Li, Nick Haber, Jiajun Wu

arXiv:2412.02193v3cs.CVcs.AI

TL;DR

Open-ended 3D layout generation remains challenging when scenes are dense and physically constrained, because existing approaches can struggle to combine physical plausibility with semantic intent. LayoutVLM uses VLM-generated poses and spatial relations with self-consistent decoding and differentiable optimization. Across 11 room types, it improves PSA by 40.8 over the best-performing baseline, and the paper reports occasional failures from suboptimal VLM initializations.

  • Problem

    Existing layout-generation approaches can struggle to produce scenes that are both physically plausible and semantically coherent, limiting alignment with open-ended language instructions.

    Source: §1 · p. 2

  • Method

    LayoutVLM uses visually marked images to generate numerical object poses and spatial relations, then applies self-consistent decoding and differentiable optimization to balance physical and semantic objectives.

    Source: §1 · p. 2, §4.2 · p. 5

  • Results

    Averaged over 11 room types, LayoutVLM improves the PSA score by 40.8 compared to the best-performing baseline, I-Design.

    Source: §5.2 · p. 8

  • Takeaways & Limitations

    Fine-tuning open-source models on synthetic data using the scene representation yields substantial performance improvements, including for models that struggle with 3D layout generation.

    Source: §1 · p. 2

    The formulation assumes input 3D objects are upright and relies on an off-the-shelf VLM to determine their front-facing orientations.

    Source: §3 · p. 3

Abstract

from arXiv · show

Spatial reasoning is a fundamental aspect of human cognition, enabling intuitive understanding and manipulation of objects in three-dimensional space. While foundation models demonstrate remarkable performance on some benchmarks, they still struggle with 3D reasoning tasks like arranging objects in space according to open-ended language instructions, particularly in dense and physically constrained environments. We introduce LayoutVLM, a framework and scene layout representation that exploits the semantic knowledge of Vision-Language Models (VLMs) and supports differentiable optimization to ensure physical plausibility. LayoutVLM employs VLMs to generate two mutually reinforcing representations from visually marked images, and a self-consistent decoding process to improve VLMs spatial planning. Our experiments show that LayoutVLM addresses the limitations of existing LLM and constraint-based approaches, producing physically plausible 3D layouts better aligned with the semantic intent of input language instructions. We also demonstrate that fine-tuning VLMs with the proposed scene layout representation extracted from existing scene datasets can improve their reasoning performance.

1. Introduction

Open-universe layout generation must reconcile semantic intent with physical plausibility, but existing approaches often fail at one or the other. LayoutVLM combines VLM-generated poses and spatial relations with differentiable optimization and self-consistent decoding.

1. Introduction

  • Existing layout methods struggle to achieve physical plausibility and semantic coherence together: pose-prediction approaches can cause collisions, while constraint methods can miss language nuance.Traditional synthesis is also limited by predefined categories and learned placement patterns.Source: §1 · p. 2
  • LayoutVLM initializes object poses, then jointly optimizes physics-based objectives and spatial relations to preserve physical plausibility and layout semantics.Visually marked images help VLMs estimate object placements, and self-consistent decoding focuses optimization on semantically meaningful relations.Source: §1 · p. 2
  • The proposed scene representation combines numerical pose estimates with spatial relations and their matching differentiable objectives to generate diverse layouts.The VLMs generate the representation from visually marked scene and asset renderings.Source: §1 · p. 2
  • Evaluation across 11 room types reports significant improvement over the current best-performing method, and synthetic-data fine-tuning improves open-source VLM performance.The reported fine-tuning gains include models that struggle with 3D layout generation.Source: §1 · p. 2

2. Related Work

Prior scene-synthesis work spans image-generation methods and asset-based layouts, while VLM spatial-reasoning studies have largely focused on perception. LayoutVLM uses VLMs for layout generation and differentiable optimization.

2.1. Layout Generation & Indoor Scene Synthesis

  • Image-generation scenes using NeRFs or Gaussian splats lack separable, manipulable objects and surfaces [6] [7] [8] [9]; LMMs enable open-vocabulary 3D scene synthesis without predefined labels or categories [13] [14].LayoutGPT [4] prompts language models to directly generate 3D layouts for indoor scenes.Source: §2.1 · p. 2
  • LayoutVLM generates scene-layout representations from image and text inputs with VLMs, then uses differentiable optimization rather than search-based constraint solving.Its representation combines numerical object poses and spatial relations amenable to optimization.Source: §2.1 · p. 3, Figure 2 · p. 3

2.2. Vision-Language Models for 3D Reasoning

  • Earlier VLM spatial-reasoning work mainly targets perception or scene reconstruction, whereas LayoutVLM applies 2D VLMs to 3D layout generation and investigates fine-tuning for that task.It repurposes visual-reasoning techniques such as multiple scene views and visual markers for spatial planning.Source: §2.2 · p. 3

3. Problem Formulation

Open-universe 3D layout generation arranges arbitrary assets in a 3D environment to match a natural-language instruction, under stated object-orientation assumptions.

3. Problem Formulation

  • The task is to arrange any assets in a 3D environment so the scene faithfully represents a natural-language layout criterion.The environment is defined by four cardinally oriented walls and a set of N 3D meshes.Source: §3 · p. 3
  • The formulation assumes input objects are upright and uses an off-the-shelf VLM to determine their front-facing orientations.The VLM also assigns each object a short textual description and dimensions are represented by an axis-aligned bounding box after rotation.Source: §3 · p. 3
  • The target output is each object's pose: its 3D position and rotation about the z-axis.The pose is denoted pi = (xi, yi, zi, θi).Source: §3 · p. 3

4. LAYOUTVLM

LAYOUTVLM uses VLM-generated pose estimates and spatial relations in a differentiable optimization framework to generate layouts. It combines visual prompting and self-consistent decoding with physical objectives, and also fine-tunes VLMs using representations extracted from scene datasets.

4. LAYOUTVLM

  • VLMs generate code for a representation of initial object poses and spatial relations, which optimization uses to produce final placements.The semantic objective comes from the representation, while a physics objective promotes physical plausibility.Source: §4 · pp. 3–4

4.1. Scene Layout Representation

  • The representation combines initial numerical pose estimates with differentiable spatial relations, preserving instruction semantics as optimization adjusts poses for physical plausibility.Its five relation types cover distance, on_top_of, align_with, point_towards, and against_wall; poor initialization can yield suboptimal layouts.Source: §4.1 · pp. 4–5

4.2. Generating Scene Layout Representation with Vision-Language Models

  • Visual prompting marks scenes with 2-meter coordinate points, coordinate frames, and object-facing arrows; self-consistent decoding retains relations satisfied by predicted poses.Assets are grouped and placed a group at a time, with scenes rerendered between groups to identify unoccupied areas.Source: §4.2 · p. 5

4.3. Differentiable Optimization

  • Joint pose optimization applies VLM-generated spatial relations and Distance-IoU collision loss, using projected gradient descent and projecting assets within physical boundaries.The combined objective is used to confine objects and avoid unwanted intersections.Source: §4.3 · p. 5

4.4. Finetuning VLMs with Scene Datasets

  • The representation can be extracted automatically from scene layouts and used to fine-tune VLMs; the implementation extracts training data from around 9,000 3D-Front rooms.The two investigated models are GPT-4o and LLaVA-NeXT-Interleave.Source: §4.4 · pp. 5–6
  • Across 11 room types, LAYOUTVLM outperformed existing open-universe layout methods; qualitative examples show layouts following detailed instructions while respecting physical constraints.The benchmark caption reports the scope of the comparison; the qualitative-comparison caption describes instruction-following and physical plausibility.Source: Table 2 · p. 6, Figure 4 · p. 6

5. Experiments

Experiments evaluate LayoutVLM’s layout quality, metric validity, component ablations, and fine-tuning. Results report stronger benchmark performance, contributions from its design components, and benefits from fine-tuning its representation.

5.1. Experimental Setup

  • Evaluation covers open-vocabulary instructions, objects beyond predefined categories, and accurate, bounded, collision-avoiding placements across 11 room types.Physical plausibility is measured with Collision-Free and In-Boundary scores; GPT-4o rates semantic coherence from rendered views, and PSA combines that rating with physical plausibility.Source: §5.1 · p. 7

5.2. Benchmark Performance

  • Averaged across 11 room types, LayoutVLM improves PSA by 40.8 over I-Design, while reducing out-of-bound objects and maintaining positional and rotational coherence.Qualitative examples also show valid placements for many assets and close adherence to detailed, including unconventional, instructions.Source: §5.2 · p. 8

5.3. Human Eval to validate our metrics

  • User rankings and GPT-4o rankings show strong user-user and user-GPT agreement, supporting the reported alignment of evaluator rankings with the proposed metrics.The study collected 495 ratings for each method-metric pair.Source: §5.3 · p. 8, Table 3 · p. 8, Table 4 · p. 8

5.4. Ablation Study

  • Ablations indicate that visual input aids spatial planning, while the combined pose-and-relation representation and self-consistency support practical layouts meeting physical and semantic requirements.Spatial constraints are especially important for physical validity and preventing overlap; good numerical initialization contributes to semantic scores.Source: §5.4 · pp. 8–9, Table 5 · p. 8

5.5. Finetuning LayoutVLM

  • Fine-tuning with the scene representation outperforms fine-tuning to predict numerical poses and significantly improves the open-source model on residential layouts with unseen objects.The reported improvement is more limited for the closed-source model, which the authors suggest may already use the representation through prompts.Source: §5.5 · p. 9, Table 6 · p. 9

6. Conclusion

LayoutVLM combines pose estimates and spatial relations from visually marked images with self-consistent decoding and differentiable optimization. It improves over existing approaches, though suboptimal VLM initializations can still occasionally yield invalid layouts.

6. Conclusion

  • LayoutVLM represents layouts with numerical pose estimates and spatial relations derived from visually marked images.The framework combines this representation with self-consistent decoding, visual prompting, and differentiable optimization.Source: §6 · p. 9
  • The framework demonstrates improvements over existing LLM and constraint-based approaches in open-universe 3D layout generation.Source: §6 · p. 9
  • Occasional invalid layouts remain when VLM initializations are suboptimal.Source: §6 · p. 9

Supplementary Material

The supplementary material describes prompts for grouping assets and organizing groups into a structured layout plan.

A.1. Grouping

  • Semantic asset groups collect logically related assets using functional, semantic, and geometric relationships, including assets that are not physically close.The prompt asks the model to describe the scene, identify relationships, and explain each group’s rationale.Source: §A.1 · p. 12
  • The prompt orders groups by placement sequence, with prominent assets such as a bed, nightstand, and lamp placed first to establish a focal area.Source: §A.1 · p. 12
  • The output format records each group’s name, assets, rationale, and key relations within and across groups.Source: §A.1 · p. 13

A.2. Differentiable Spatial constraint

The method defines differentiable objectives for spatial relations using asset poses, orientation vectors, and bounding-box sizes. These objectives combine geometric constraints with VLM-generated placement plans that are optimized for physical feasibility.

A.2. Differentiable Spatial constraint

  • Asset poses are represented by position and rotation, with orientation vectors and 3D bounding-box sizes supplying notation for spatial-relation objectives.The pose is pi = (xi, yi, zi, θi), and the orientation vector is vi = (cos θi, sin θi).Source: §A.2 · p. 13
  • Planar distance uses the x-y Euclidean distance between objects, while clamp bounds values to a specified interval.clamp(x, a, b) = min(max(x, a), b).Source: §A.2 · p. 13
  • The On-Top-Of objective uses bounding-box IoU and directly sets one object's z-coordinate above the other rather than using a z-axis loss.Source: §A.2 · p. 13
  • Point-towards constraints compare an object's orientation vector with a direction vector rotated by ϕ around the z-axis.The direction vector points from one object to the other.Source: §A.2 · p. 13
  • The Against-Wall objective combines object-corner distances to the wall with a term encouraging the object to point away from it.The description defines object corners from position, rotation, and bounding-box size, and identifies the wall normal as perpendicular to the wall.Source: §A.2 · p. 14
  • The VLM receives goals and scene and asset views, then specifies placements and semantic constraints; optimizing 40 assets takes 1–5 minutes and 5–10 GPT-4o calls.Source: §A.2 · pp. 14–15

A.3. Self-Consistent Decoding

Self-consistent decoding filters VLM-generated spatial relations to preserve coherent semantics during optimization. The asset annotation process uses GPT-4o to assign descriptive and placement-related attributes to unlabeled 3D assets.

A.3. Self-Consistent Decoding

  • Self-consistent decoding keeps at most one orientational constraint per asset and excludes on-top-of relations because the model already predicts them accurately and reasonably.Source: §A.3 · p. 15

A.4. Annotating Unlabeled 3D Assets

  • GPT-4o annotates assets from four orthogonal views with category, variable name, front view, description, materials, and Boolean placement attributes.The placement attributes indicate typical locations such as ceiling, wall, floor, or on-object placement.Source: §A.4 · p. 15

B. Details of our Experiments

This section describes how open-vocabulary 3D layout test cases are created, candidate assets retrieved, and their suitability for the described room or tabletop arrangement evaluated. Human verification removes assets that do not fit the room type or layout instruction.

B.1. Generating Test Cases

  • Test cases pair room or tabletop layout instructions with object lists generated from the description and room dimensions.GPT-4o generates layout criteria, followed by a prompt to specify plausible objects and their counts and type variety.Source: §B.1 · pp. 15–16
  • Object lists favor simple, relevant, singular object names and aim for at least 10 items, with room and tabletop contents distinguished.The prompt also calls for seating and a surface for setting things down in rooms, and corresponding chairs or stools for desks and workbenches.Source: §B.1 · pp. 15–16
  • Using CLIP [36] embeddings of generated asset descriptions, the method retrieves 3D assets from Objaverse and prompts a verification check to determine whether each retrieved object belongs in the described room.The prompt also asks whether the object's style and color match the room type.Source: §B.1 · p. 16
  • Test cases ask whether an object should be placed in the room or tabletop scene, with “False” used when it should not be.Source: §B.1 · p. 17
  • Humans verify the assets and remove those deemed unsuitable for the room type or layout instruction, such as an entire city asset in an indoor scene.Source: §B.1 · p. 17

B.2. Evaluation

The evaluation uses VLM prompts to assess both physical plausibility and semantic coherence in generated layouts. Its measures cover positional and rotational coherence as well as collision-free, in-boundary, and prompt-alignment criteria.

B.2. Evaluation

  • The evaluation framework uses VLM prompts to assess physical plausibility and semantic coherence in generated 3D layouts.Source: §B.2 · p. 17
  • Semantic coherence is assessed through prompts that rate whether related objects are positioned and oriented sensibly for their use.Source: §B.2 · p. 17
  • The Physically-grounded Semantic Alignment Score combines Collision-Free Score, In-Boundary Score, and overall prompt alignment.Source: §B.2 · p. 17
  • The layout-criteria prompt rates how well relative object positions and poses match the target criteria, using anchored score descriptions from 20 to 100.Source: §B.2 · p. 18

C. More Qualitative Comparison

Qualitative examples show LayoutVLM outperforming baselines across room types in physical plausibility and semantic coherence, with room-specific examples of asset grouping and stacking.

C. More Qualitative Comparison

Loading 2412.02193v3…