PaperlayerView exact revision on arXiv

Source-linked AI summary

3D-Layout-R1: Structured Reasoning for Language-Instructed Spatial Editing

Haoyu Zhen, Xiaolong Li, Yilin Zhao, Han Zhang, Sifei Liu, Kaichun Mo, Chuang Gan, Subhashree Radhakrishnan

arXiv:2603.22279v1cs.CVcs.AI

TL;DR

Few methods execute structured, multi-step 3D layout edits from natural-language instructions, while existing VLM spatial-reasoning systems primarily answer queries without modifying scenes. 3D-Layout-R1 performs structured, language-directed 3D layout editing through explicit, iterative updates to a 3D bounding-box-based scene graph. 3D-Layout-R1 improves mean IoU by 15% and reduces center-distance errors by 25–30% versus CoT-SFT and GRPO baselines, and achieves up to 20% higher IoU than leading zero-shot LLMs and prior systems.

  • Problem

    Few methods execute structured, multi-step 3D layout edits from natural-language instructions, while existing VLM spatial-reasoning systems primarily answer queries without modifying scenes.

    Source: §1 · p. 2

  • Method

    3D-Layout-R1 performs structured, language-directed 3D layout editing through explicit, iterative updates to a 3D bounding-box-based scene graph.

    Source: §1 · p. 2

  • Results

    3D-Layout-R1 improves mean IoU by 15% and reduces center-distance errors by 25–30% versus CoT-SFT and GRPO baselines, and achieves up to 20% higher IoU than leading zero-shot LLMs and prior systems.

    Source: §1 · p. 2

  • Takeaways & Limitations

    Across benchmarks, the framework reduces layout error and outperforms zero-shot and CoT-based baselines, particularly in demanding spatial-alignment and room-editing settings.

    Source: §5 · p. 9

    Although large VLMs outperform their LLM counterparts by over 10% on spatial alignment, post-training on VLM backbones shows no comparable gains, suggesting specialized visual prompting or representations may be needed.

    Source: §4.3 · p. 9

Abstract

from arXiv · show

Large Language Models (LLMs) and Vision Language Models (VLMs) have shown impressive reasoning abilities, yet they struggle with spatial understanding and layout consistency when performing fine-grained visual editing. We introduce a Structured Reasoning framework that performs text-conditioned spatial layout editing via scene-graph reasoning. Given an input scene graph and a natural-language instruction, the model reasons over the graph to generate an updated scene graph that satisfies the text condition while maintaining spatial coherence. By explicitly guiding the reasoning process through structured relational representations, our approach improves both interpretability and control over spatial relationships. We evaluate our method on a new text-guided layout editing benchmark encompassing sorting, spatial alignment, and room-editing tasks. Our training paradigm yields an average 15% improvement in IoU and 25% reduction in center-distance error compared to Chain of Thought Fine-tuning (CoT-SFT) and vanilla GRPO baselines. Compared to SOTA zero-shot LLMs, our best models achieve up to 20% higher mIoU, demonstrating markedly improved spatial precision.

1. Introduction

Language-directed 3D layout editing requires compositional spatial reasoning and physical constraints, yet few methods execute structured, multi-step edits. 3D-Layout-R1 reasons through explicit scene-graph updates and reports improvements over CoT-SFT, GRPO, and zero-shot baselines.

1. Introduction

  • Few methods execute structured, multi-step 3D layout edits from natural-language instructions, despite advances in VLM spatial question answering.Existing VLM spatial-reasoning systems primarily answer queries without modifying the underlying scene.Source: §1 · p. 2
  • Prior language-driven editing pipelines use external solvers or optimizers, but face limited flexibility, difficulty with long-horizon edits, and opaque reasoning with little opportunity for correction.One-shot prediction and single-object placement methods do not generalize to multi-object rearrangement or sequential editing of existing scenes.Source: §1 · p. 2
  • 3D-Layout-R1 treats a 3D bounding-box scene graph as an iterative canvas, producing explicit, verifiable graph edits for interpretable, geometrically coherent multi-step rearrangements.The framework also integrates GRPO with dense 3D IoU rewards and collision-aware penalties to optimize layout accuracy.Source: §1 · p. 2
  • 3D-Layout-R1 improves mean IoU by 15% and reduces center-distance errors by 25–30% compared to CoT-SFT and GRPO baselines, with up to 20% higher IoU than leading zero-shot LLMs and prior systems.The framework addresses Sorting, Spatial Alignment, and Room Editing.Source: §1 · p. 2
  • The paper introduces a benchmark dataset of 15k 3D scenes with instructions, intermediate graph edits, and target layouts for multi-step editing of existing scenes.Its contributions also include geometric reward functions based on 3D IoU for policy optimization and evaluation.Source: §1 · p. 2

2. Related Works

Prior spatial-reasoning work largely addresses question answering through visual, coordinate-based, or implicit representations, while language-driven 3D editing uses varied intermediates and often external optimization. The paper positions direct, iterative reasoning over structured 3D scene graphs as a less-explored direction.

Spatial Reasoning

  • Spatial reasoning approaches primarily target spatial VQA using 3D cues, implicit maps, or coordinates; this work instead investigates updating structured 3D scene graphs.Examples include SpatialVLM [4], SpatialRGPT, and coordinate-based approaches [28] and [26].Source: §2 · p. 2

Language-Driven 3D Layout Editing

  • Figure 2 illustrates sorting by shape and height through concise JSON scene-graph updates rather than a long, ambiguous free-form reasoning path.The example describes grouping and sorting objects to transform the initial scene into the final layout.Source: Figure 2 · p. 3
  • Most natural-language 3D editing approaches rely on external solvers and struggle with complex compositional instructions; this work enables an LLM to iteratively update a structured 3D scene graph for long-horizon compositional editing without external optimization.Source: §2 · p. 3

Structured Chain-of-Thought

  • Directly integrating structured 3D information into LLM/VLM reasoning is less explored; the authors posit that embedding a 3D bounding-box scene graph in the reasoning trace guides more faithful spatial logic and mitigates multimodal hallucination.Source: §2 · p. 3

3. Method

The method represents 3D layouts as scene graphs and trains structured, stepwise reasoning over edits. It builds task-specific instruction and target-layout data, then refines model outputs with CoT-SFT and GRPO rewards for alignment, physical feasibility, and format.

3.1. Preliminaries

  • Each scene graph represents objects and supporting regions as nodes, with contact or containment relations as edges and spatial attributes stored per node.The serialized JSON stores node type, center, dimensions, rotation, and optionally a natural-language caption.Source: §3.1 · p. 3

3.2. Dataset Creation

  • Training pairs rendered layouts and initial and target graphs with instructions and stepwise graph-edit traces generated by DeepSeek-R1.The traces map textual constraints to concrete object-level operations and are used for Structured CoT-SFT.Source: §3.2 · p. 4, §3.1 · p. 4
  • The 10k-scene sorting benchmark combines semantic grouping, within-group attribute sorting, group ordering, and prescribed spans and gaps.Objects are snapped to a common support surface, with vertical positions set by object height and the table plane.Source: §3.2 · p. 5
  • The spatial-alignment task perturbs objects in 1k rendered scenes and asks the model to restore displaced objects to their category-grouped grid positions and orientations.Correctly placed objects should remain unchanged.Source: §3.2 · p. 5
  • Room-editing instructions gain distances to two or three reference objects, narrowing feasible placements toward an almost uniquely determined target graph.The source instructions come from InstructScene [22], where relational descriptions alone may allow multiple valid layouts.Source: §3.2 · p. 5

3.3. 3D Layout Reinforcement Learning

  • CoT-SFT provides a cold start for structured reasoning, after which GRPO refines layout accuracy, physical plausibility, and output formatting.The RL stage uses task-specific rewards and benefits from structured behaviors acquired during CoT-SFT.Source: §3.3 · p. 5
  • The GRPO objective combines graph-alignment, collision, and format rewards to balance geometric fidelity, physical feasibility, and parseable structured outputs.The IoU reward averages overlap over matched predicted and target boxes; the collision score penalizes interpenetrations among non-container objects.Source: §3.3 · p. 5
  • The format reward favors the required reasoning-and-answer pattern, including valid JSON in both the reasoning section and final answer.Unmatched tags, malformed braces, or misplaced tags reduce the score.Source: §3.3 · p. 6

4. Experiments

Experiments evaluate sorting, spatial alignment, and room editing, alongside qualitative warehouse and tabletop demonstrations. Structured scene-graph reasoning and RL improve results in the reported benchmark comparisons.

4.1. Experiment Setup

  • The setup compares against open-source and proprietary LLM/VLM baselines and evaluates geometric accuracy, physical plausibility, and sorting order.Metrics include Mean IoU, IoU@x, Center Distance, Collision-Free Score, and sorting Edit Distance.Source: §4.1 · p. 6

4.2. Sorting

  • Structured scene-graph reasoning consistently outperforms zero-shot LLM/VLM baselines and standard CoT-SFT on Sorting; adding RL further improves performance over CoT-SFT.Skipping structured CoT-SFT causes a 7.4% drop in mIoU gain.Source: Table 3 · p. 6, §4.2 · p. 7

4.3. Spatial Alignment

  • Spatial alignment requires inferring an N × M group structure and finding misaligned objects; structured reasoning helps smaller 7B/8B models match or exceed large commercial models.The model infers a grid, detects center-position deviations, and locates a target by extending an end node or interpolating along a grid line.Source: §4.3 · pp. 8–9

4.4. Room Layout Editing

  • RL on structured CoT-SFT achieves the strongest overall Room Editing performance, though its advantage over CoT-SFT is smaller than in Sorting.The authors attribute the smaller gap to simpler edits that modify only a small part of the scene.Source: Table 4 · p. 8, §4.4 · p. 9

4.5. Qualitative Results

  • Sorting traces group objects by attributes, allocate spans along a target axis, and place items with required gaps; room-editing traces update poses and verify distance constraints.The room-editing example produces a layout described as visually plausible and quantitatively consistent with the ground-truth scene graph.Source: §4.5 · p. 9
  • Without warehouse-data retraining, the model follows unseen rearrangement instructions; tabletop demonstrations use detected boxes to generate goals for a robot with known grasping skills.The model does not represent continuous dynamics or collisions; robot execution is handled by a rule-based, goal-conditioned system.Source: Figure 5 · p. 8, §4.5 · p. 9, Figure 6 · p. 9

5. Conclusion

The conclusion presents 3D-Layout-R1 as a structured reasoning framework for language-guided 3D layout editing. It reports improved accuracy and interpretability, with lower layout error and stronger benchmark performance than zero-shot and CoT-based baselines.

5. Conclusion

  • Across benchmarks, 3D-Layout-R1 reduces layout error and outperforms zero-shot and CoT-based baselines, especially on spatial alignment and room editing.The authors describe the framework as improving accuracy and interpretability through structured multi-step reasoning refined with GRPO.Source: §5 · p. 9

1. Implementation Details

Implementation uses instruction-tuned text and vision-language backbones, followed by CoT supervised fine-tuning and GRPO reinforcement learning. Evaluation uses deterministic decoding, and a figure shows the shared prompt-template design.

Base Models

  • The implementation starts from Qwen2.5-VL-7B-Instruct and text-only Qwen3-Instruct.For the VLM, the vision backbone is frozen while multimodal fusion and language components are updated.Source: §1 · p. 13

CoT Supervised Fine-tuning

  • CoT-SFT trains on reasoning-augmented data for five epochs, using a 10,240-token maximum sequence length and 16 A100 GPUs.The VLM vision backbone remains frozen; the text backbone is trained for Qwen3.Source: §1 · p. 13

Reinforcement Learning Fine-tuning

  • GRPO starts from the CoT-SFT checkpoint and refines the policy for 15 epochs with a KL-regularized objective.Training uses eight GPUs and a global batch size of 32.Source: §1 · p. 13

Inference

  • Evaluation uses greedy deterministic decoding with temperature 0.0, top-p 1.0, and best-of set to 1 for both model families.Source: §1 · p. 13

Implementation Details

  • Figure 7 shows the prompt template used to generate reasoning traces over scene graphs for spatial tasks.Source: Figure 7 · p. 13

2. Reasoning Prompt Design for Data Creation

The prompts cast spatial tasks as inferring a hidden goal scene graph from an instruction and initial graph. Shared rules enforce relevant-object coverage, collision-free placements, and coordinate consistency, while task-specific instructions vary by setting.

Reasoning Prompt Design for Data Creation

  • Three related prompts cover generic spatial reasoning, distance-constrained InstructScene reasoning, and grid-based arrangement.Source: §2 · p. 13
  • Each task provides a natural-language question, initial JSON scene graph, and goal-format schema while withholding the goal contents.The model produces a multi-step trace with explanations and updated or partial JSON subgraphs.Source: §2 · pp. 13–14
  • Reasoning-based training improves Sorting performance consistently across model scales from 1.7B to 8B.Source: Figure 8 · p. 14
  • Task-specific prompts add explicit distance calculations for InstructScene or restrict grid updates to mispositioned items; all require relevant objects and collision-free, coordinate-consistent placements.Source: §2 · p. 14

3. More Experiments

RL consistently outperforms the other training paradigms across model sizes, while reward ablations identify Format and IoU as crucial and CF as a smaller, consistent contributor.

3. More Experiments

  • RL consistently outperforms the other training paradigms across model sizes and shows more favorable scaling behavior overall.The passage reports no specific numeric values for these trends.Source: §3 · p. 14
  • CF reward provides a comparatively modest but consistent performance gain.This finding is part of the reward-component ablation.Source: §3 · p. 14, Table 5 · p. 14
  • Format and IoU rewards are crucial for strong performance.The reward-component ablation identifies both as essential.Source: Table 5 · p. 14, §3 · p. 14

4. More Qualitative Results

In qualitative sorting and alignment examples, Qwen3-235B and DeepSeek often fail to follow instructions, whereas 3D-Layout-R1 matches the ground-truth layouts.

4. More Qualitative Results

  • Qwen3 and DeepSeek often misinterpret multi-constraint instructions, producing incorrect group ordering, violating ascending criteria, or drifting objects off the intended axis or grid.Examples include grouping by shape, sorting by geometric attributes, preserving gaps, and snapping to a latent grid.Source: §4 · p. 14
  • Their outputs also frequently contain collisions or overlaps and ignore spacing constraints, yielding physically inconsistent layouts.These are described as typical failure modes in the qualitative comparison.Source: §4 · p. 14, Figure 9 · p. 15
  • 3D-Layout-R1 matches the ground-truth targets in qualitative sorting and alignment comparisons where Qwen3-235B and DeepSeek often fail.The compared outputs include the two baselines, the proposed method, and the target layout.Source: §4 · p. 14, Figure 9 · p. 15

Related papers

Loading 2603.22279v1…