PaperlayerView exact revision on arXiv

Source-linked AI summary

Scene-Designer: Learning Design Intelligence for Agentic 3D Scene Generation

Xuanyu Liu, Hongyan Feng, Miao Pan, Sunlai Chen, Lizhou Liang, Bing Sun, Wenqi Zhang, Xuhong Zhang

arXiv:2610.23121v1cs.CV

TL;DR

Scene generation needs models that can both design scenes and critique their quality, while prior approaches rely on general-purpose models or static-scene data. Scene-Designer learns from complementary design and critique experiences and uses Critic-guided optimization. It improves scene generation across three quality dimensions, and its learned Critic substantially outperforms GPT-5.2 in scene ranking.

  • Problem

    Existing approaches rely on general-purpose models lacking domain-specific design experience or on static scenes that do not capture design and critique experience.

    Source: §1 · pp. 1–2

  • Method

    Scene-Designer combines Scene-OTA-35K design supervision and Scene-OTC-15K critique preferences with Scene-GRPO, which uses a frozen Critic to guide candidate edits and assess scene improvement.

    Source: §1 · pp. 3–4

  • Results

    Scene-Designer improves physical validity, functional correctness, and layout coherence; its Scene-Critic achieves 89% pairwise comparison accuracy versus 72% for GPT-5.2.

    Source: §1 · p. 4

  • Takeaways & Limitations

    The learned Critic also guides editing decisions at test time, further improving scene quality beyond its use during training.

    Source: §1 · p. 4

Abstract

from arXiv · show

Generating complete, coherent, and usable 3D indoor scenes from natural language requires strong \textbf{design intelligence}: the ability to design high-quality scenes and critique their quality. However, existing approaches either rely on general-purpose models lacking domain-specific design experience or learn from static scenes. In this work, we introduce \textbf{Scene-Designer}, an agentic framework that learns design intelligence through complementary design and critique experiences. We construct two datasets: \textbf{Scene-OTA-35K}, which provides Observation--Thought--Action supervision for learning scene design and efficient error correction, and \textbf{Scene-OTC-15K}, which provides Observation--Thought--Critic supervision for learning scene quality critique across physical validity, functional correctness, and layout coherence. Based on these experiences, we propose \textbf{Scene-GRPO}, a Critic-guided GRPO framework that improves design decisions and reduces unnecessary trial-and-error corrections. Extensive experiments demonstrate that Scene-Designer achieves superior performance in physical validity, functional correctness, and layout coherence, while the learned Scene-Critic significantly outperforms general-purpose models in scene critique. Moreover, test-time Critic guidance further improves scene generation, showing that the Critic can provide useful guidance beyond training. Additional analysis suggests a trend of mutual enhancement between scene design and critique abilities.

1 Introduction

High-quality 3D scene generation requires models to make design decisions and assess scene quality, but general-purpose models show weaknesses in efficient correction and scene critique. Scene-Designer addresses these gaps with complementary design and critique datasets and Critic-guided training, improving scene quality and correction decisions.

1 Introduction

  • Across 100 scenes generated by GPT-5.2 with a strong scene-construction scaffold, the model often made unnecessary corrective attempts, even for errors resolvable with only one or two simple operations.Approximately 18% of window accessibility blockages still required more than five corrective moves.Source: §1 · p. 2
  • Reliable scene critique must assess physical validity, functional correctness, and layout coherence, yet general-purpose models struggle to rank scene quality across these dimensions.The paper identifies static scene datasets and the lack of design-process and critique experience as data challenges.Source: §1 · pp. 2–3
  • Scene-OTA-35K provides Observation–Thought–Action supervision, while Scene-OTC-15K provides preference comparisons for learning scene critique.Causal Trajectory Consolidation preserves important correction decisions while removing ineffective corrections; OTC preferences use quality degradations from scene perturbations with rankings determined by perturbation strength.Source: §1 · pp. 3–4
  • Joint SFT produces Scene-Builder, then Scene-GRPO uses a frozen Critic to compare candidate edits and assess improvement over parent scenes.These signals guide design choices and reduce unnecessary trial and error.Source: §1 · p. 4
  • Scene-Designer improves physical validity, functional correctness, and layout coherence while making more effective corrections; Scene-Critic also outperforms general-purpose models in critique.For example, interaction-orientation errors requiring at least five corrective moves fall from 40% to 7%, and window-accessibility blockages from 21% to 5%, versus Scene-Builder.Source: §1 · p. 4

2 Related Work

Prior scene-generation work uses foundation models, static-scene learning, and evaluation methods, but the described approaches leave gaps in design-process supervision and holistic scene critique. Scene-Designer combines construction experience and scene preference comparisons to learn design and critique abilities.

2 Related Work

  • Scene-Designer learns design and critique from complementary construction trajectories and scene preference comparisons.Its overview describes Scene-OTA Construction as extracting design experience and Scene-OTC Construction as generating preference comparisons.Source: Figure 4 · p. 5
  • Foundation-model scene generators and agentic frameworks support language-driven layout generation and iterative construction, but rely largely on general-purpose models for design decisions.This reliance can lead to high interaction costs and inefficient error correction.Source: §2 · pp. 4–5
  • Learning-based scene-generation datasets mainly provide static layouts, without supervision on construction decisions, error corrections, or their effects on scene quality.The cited methods learn plausible spatial arrangements from scene collections.Source: §2 · p. 5
  • Existing evaluation approaches use geometric metrics, human preferences, VLM judgments, or structured criteria, but predefined criteria remain limited in capturing holistic functional organization and layout coherence.SceneEval evaluates instruction fidelity and physical plausibility, while SceneCritic checks spatial priors such as scale, co-occurrence, orientation, and overlap.Source: §2 · p. 5

3 Method

Scene-Designer combines complementary design and critique supervision with Critic-guided optimization. Its training data and rewards target effective scene construction, quality assessment, and reduced trial and error.

3 Method

  • Scene-Designer is formed by applying Critic-guided optimization to Scene-Builder, with the Critic comparing candidate scenes and child scenes against their parents.The framework is intended to improve design decisions and reduce unnecessary trial and error.Source: §3 · pp. 5–6, §3.2 · p. 7

3.1 Learning Design Intelligence: From General VLM to Scene-Builder

  • Scene-OTA-35K contains 1K scenes and 35K Observation–Thought–Action steps, built from consolidated trajectories that shorten construction while preserving effective corrections and final scene states.Consolidation identifies potentially ineffective operations from scene-state changes and remaining errors, then uses GPT-5.2 to assess correction effects and removes only operations without downstream contributions.Source: §3.1 · p. 6
  • Scene-OTC-15K provides ranked scene critiques across physical validity, functional correctness, and layout coherence using examples generated from progressively degraded scenes.Its perturbation strategies support automatic preference ranking by severity in some cases, while comparisons across objects require manual annotation because impact depends on functional role.Source: §3.1 · p. 6
  • Joint SFT on Scene-OTA and Scene-OTC teaches the model to construct physically valid scenes, make effective corrections, and evaluate scene quality, producing Scene-Builder.The datasets provide complementary supervision for design and critique, addressing the lack of task-specific experience in general-purpose vision-language models.Source: §3.1 · pp. 6–7

3.2 Critic-Guided Optimization: From Scene-Builder to Scene-Designer

  • Scene-GRPO samples alternative actions from a parent scene, uses a frozen Critic to rank valid child scenes, and continues with the highest-ranked branch.Branch-ranking compares sibling scenes, while improvement reward checks whether an edit improves its parent; global rewards encourage efficient and complete construction.Source: §3.2 · pp. 7–8
  • Scene-GRPO combines local rewards for choosing better sibling actions and improving parent scenes with global rewards for efficient, complete scene construction.Auxiliary format rewards encourage properly formatted reasoning and valid tool calls; the policy is optimized with the standard GRPO objective.Source: §3.2 · pp. 7–8

4 Experiment

Experiments report fewer prolonged correction attempts with Scene-Designer and stronger scene-ranking performance from joint design-and-critique training. Analyses also indicate complementary reward effects and a trend toward design and critique supporting one another.

4.1 Scene Design for Common Room Types

  • For door clearance, window accessibility, interaction orientation, and scale mismatch, Scene-Designer reduced cases needing at least five corrective moves from 15% to 13%, 21% to 5%, 40% to 7%, and 50% to 23%, respectively.The authors report that test-time Critic guidance further reduces unnecessary corrections by filtering inferior editing decisions.Source: §4.1 · pp. 9–10, Figure 6 · p. 9

4.2 Scene-GRPO Analysis

  • During Scene-GRPO training, branch-ranking, improvement, and global length rewards rose overall, which the authors interpret as progress in decision quality and construction efficiency.Figure 8 plots the evolution of these reward components during optimization.Source: §4.2 · p. 10, Figure 8 · p. 10
  • Removing any reward component degraded performance; removing branch-ranking caused a substantial drop, while removing improvement reduced correction effectiveness.The analysis attributes complementary roles to the two local rewards: comparing alternatives and assessing whether edits improve parent scenes.Source: §4.2 · p. 10, Figure 9 · p. 10

4.3 Design and Critique: Two Sides of the Same Coin

  • Joint Scene-OTA+OTC training reached 89% pairwise and 76% four-scene preference accuracy, exceeding OTC-only fine-tuning at 86% and 51%, respectively.The reported comparison includes the base model at 72% and 2%, and GPT-5.2 at 72% and 15%; Figure 10 presents overall preference accuracy.Source: §4.3 · p. 10, Figure 10 · p. 10
  • With image-label order shuffled five times, the Critic achieved 90.00 ± 2.07% on n = 2 and 75.64 ± 6.88% on n = 4, closely matching original results.Figure 11 reports accuracy across randomized runs and ranking accuracy on n4.Source: §4.3 · p. 11, Figure 11 · p. 11
  • Scene-OTA+OTC achieved 82% Top-1 and 76% Top-4 accuracy on n4, and the authors argue reliable critique requires understanding scene-design decisions.The passage frames design knowledge as an essential complement to critique ability.Source: Figure 10 · p. 10, §4.3 · p. 11
  • Qualitative examples show Scene-OTA+OTC policies organizing dining and reading areas into more coherent functional compositions than Scene-OTA-only policies.The authors describe these examples as a trend suggesting critique can provide complementary guidance for scene generation.Source: §4.3 · p. 11, Figure 12 · p. 11

5 Conclusion

Scene-Designer combines complementary design and critique datasets with Critic-guided optimization. Experiments report superior scene-generation performance and suggest design and critique are complementary capabilities.

5 Conclusion

  • Scene-Designer learns design intelligence through complementary design and critique supervision and Critic-guided optimization.The framework uses Scene-OTA and Scene-OTC and proposes Scene-GRPO to guide design decisions.Source: §5 · p. 11
  • Experiments report superior scene-generation performance and suggest that design and critique are complementary capabilities.Source: §5 · p. 12

A Related Work

Existing agentic methods and static-scene datasets lack learned experience of effective design and critique; the section formulates generation as tool use and details Scene-GRPO’s local and global rewards.

A Related Work

  • Agentic systems lack learned design experience, static datasets omit construction trajectories, and symbolic evaluators miss holistic qualities; Scene-Designer combines trajectory supervision with a learned Critic.Source: A Related Work · pp. 16–17

B Problem Formulation

  • Given instruction U, the policy iteratively observes the scene and interaction history, produces reasoning and tool actions, and ultimately generates a physically valid, functional, layout-coherent scene.Source: B Problem Formulation · p. 17

C.1 Local Rewards

  • Valid candidate branches are ranked, while each renderable child is compared with its parent; positive and negative rewards favor improvements over degrading or unnecessary edits.Source: §C.1 · pp. 17–18

C Scene-GRPO Reward

  • Scene-GRPO’s rewards combine local feedback on editing decisions with trajectory-level rewards for efficient, complete scene construction.Source: §C.1 · p. 18, §C.2 · p. 18

C.2 Global Rewards

  • The global rewards favor fewer interaction steps and reward final scenes according to object count; training uses 25 steps and 25 objects as respective targets.Source: §C.2 · p. 18

C.3 Advantage Estimation

  • Local advantages normalize candidate branches within a parent-scene group, while the trajectory-level global advantage is shared across that trajectory’s editing steps.Source: §C.3 · p. 19

D.1 Analysis of Scene Construction Errors, Correction Processes, and Editing Behaviors

The analysis separates construction errors into physical and semantic/layout failures, then examines how often they occur and how efficiently GPT-5.2 corrects them. Repeated asset edits reveal a major source of unnecessary trajectory length.

Error categories

  • Physical errors account for 60.7% of observed errors, while semantic and layout errors account for 39.3%.Physical errors include collisions, boundary violations, door-clearance occupation, and window-accessibility blockage; semantic and layout errors include seven functional and compositional categories.Source: §D.1 · pp. 19–20

Error frequency

  • Object collision, boundary violation, and door-clearance occupation appear in 91.8%, 91.8%, and 90.4% of trajectories, respectively; window-accessibility blockage appears in 79.5%.Semantic and layout errors occur less often, including circulation and accessibility failure (23.3%), wall-facing orientation error (21.9%), spatial imbalance (20.5%), and interaction orientation error (19.2%).Source: §D.1 · p. 21

Implications

  • Scene design permits iterative refinement, but frequently occurring errors combined with ineffective corrections create unnecessary interaction and increase inference cost and latency for closed-source APIs.The analysis therefore treats efficient correction—not complete avoidance of all construction errors—as a key design requirement.Source: §D.1 · pp. 21–22

Correction efficiency

  • Only 33% of door-clearance cases are fixed in one move, while 20% require at least five; 18% of window-accessibility cases also require five or more moves.For orientation errors, only 12% of wall-facing and 7.5% of interaction cases are resolved in one move, while 50% and 62% require at least five moves; five or more moves are also needed in 42% of scale mismatches, 69% of circulation/accessibility failures, and 47% of spatial-imbalance cases.Source: §D.1 · p. 21

Editing behavior

  • GPT-5.2 often revisits the same assets through exploratory edits instead of applying targeted corrections, extending construction trajectories.Across trajectories, 34% of assets are placed once and never modified, while 24% require five or more corrective moves and some are edited up to 18 times.Source: §D.1 · pp. 22–23

D.2 Scene-OTA Construction

Scene-OTA consolidates construction trajectories by removing operations judged ineffective while preserving useful correction decisions and the final scene. Scene-OTC creates ranked scene comparisons from controlled perturbations to supervise fine-grained critique.

D.2 Scene-OTA Construction

  • Candidate operations are screened by progress toward the accepted final scene and reduction of construction errors, then a causal verifier judges whether they are redundant, useful, or uncertain.Potentially redundant operations are removed only above a confidence threshold, while asset additions/removals, useful intermediate states, and coordinated corrections are preserved; replay verifies that the final scene remains unchanged.Source: §D.2 · pp. 23–25
  • 57% fewer tool calls per trajectory result from removing read-only calls and repeated retries while preserving essential placement and effective corrections.Read-only calls account for 41.9% of raw trajectory steps; consolidation selectively removes ineffective operations rather than simply shortening trajectories.Source: Figure 20 · p. 25, §D.2 · pp. 25–26

D.3 Scene-OTC Construction

  • Scene-OTC applies controlled perturbations to high-quality scenes to create graded variants and preference supervision rather than binary quality labels.The perturbations include cumulative degradation, increasing severity of the same change, and the same change applied to objects with different functional roles.Source: §D.3 · pp. 26–27
  • Preference orders are automatic for cumulative and different-strength perturbations, but manually annotated when the same perturbation affects objects with different functional roles.GPT-5.2 generates reasoning traces and complete rankings from renderings and instructions; the critique dimensions include physical validity, functional correctness, and layout coherence.Source: §D.3 · p. 26

B. Remove ottoman

Scene-OTC uses object-role-sensitive perturbations to teach scene critique, and the accompanying case-study section presents qualitative comparisons of scene-generation methods.

B. Remove ottoman

  • Removing a coffee table weakens a conversation zone, but removing the sofa eliminates the primary seating and visual anchor, so the coffee-table removal ranks higher.The comparison illustrates why the same removal operation can have different quality impacts depending on the affected object's functional role.Source: B. Remove ottoman · p. 28, Figure 22 · p. 28
  • Scene-OTC-15K contains approximately 15K critique examples, with a 94-example two-scene comparison test set and a 55-example four-scene ranking test set.The perturbation hierarchy covers physical validity, functional correctness, and layout coherence.Source: B. Remove ottoman · pp. 28–29

E Case Study

  • The case-study section compares Scene-Builder, Scene-Designer, and Scene-Designer with Critic across representative room types and examines Critic-guided branch selection.Its qualitative examples are intended to illustrate effects of SFT, Scene-GRPO, and test-time Critic guidance on scene quality, construction behavior, and intermediate decisions.Source: E Case Study · p. 29

E.1 Examples of Scene-Builder, Scene-Designer, and Scene-Designer with Critic

Qualitative examples compare Scene-Builder, Scene-Designer, and Scene-Designer with Critic across several room instructions. The examples describe more coherent layouts and fewer unnecessary modifications with the learned design process and additional Critic guidance.

Bedroom example

  • For the bedroom instruction, Scene-Designer and Scene-Designer with Critic produce scenes that better satisfy physical validity, functional correctness, and layout coherence than Scene-Builder.The trajectories describe more deliberate modifications and Critic filtering of inferior editing alternatives before execution.Source: §E.1 · pp. 29–30, Figure 23 · p. 29

Trajectory analysis

  • In 100 generated scenes, Scene-Designer raises untouched objects from 46% to 56% and cuts objects with more than five corrections from 15% to 10%; with Critic, these become 58% and 5%.The analysis attributes these trends to more reliable design decisions before execution, rather than simply shorter trajectories.Source: Figure 23 · p. 29, §E.1 · p. 30

Dining-room example

  • Scene-Designer improves the dining-room composition around the meal activity, while test-time Critic guidance favors stronger functional connections and balanced spatial distribution.The described Critic-guided trajectory requires fewer unnecessary modifications while maintaining a more consistent design process.Source: §E.1 · p. 30, Figure 25 · p. 30

Bathroom example

  • In the bathroom example, Scene-Designer separates the shower and vanity zones, while Critic guidance further filters alternatives to produce a more balanced and coherent layout.The comparison describes Critic guidance as useful for subtle functional and aesthetic decisions as well as avoiding physical errors.Source: §E.1 · p. 31, Figure 26 · p. 31

E.2 Test-Time Critic Guidance Examples

The Scene-Critic evaluates multiple construction branches across physical validity, functional correctness, and layout coherence, selecting intermediate layouts that better meet scene requirements. The example shows how this guidance supports deliberate refinement and avoids unnecessary corrections.

E.2 Test-Time Critic Guidance Examples

  • The Critic ranks alternative layouts by physical validity, functional correctness, and layout coherence, then selects the strongest branch for continued construction.At the first selection, it prefers Scene A for its coherent seating, viewing, and circulation relationships.Source: §E.2 · p. 31
  • Across later edits, the Critic favors additions and removals that preserve essential functional groups and circulation rather than simply increasing or reducing object count.It rejects redundant additions and removals that would disrupt balanced seating or create circulation conflicts.Source: §E.2 · pp. 31, 33
  • After several Critic-guided rounds, the example forms a reading-oriented living room with both a conversation area and a secondary reading corner, avoiding unnecessary corrections.The sofa, coffee table, accent chair, floor lamp, and bookshelf form meaningful spatial relationships.Source: §E.2 · p. 33
  • The learned Critic acts as an inference-time decision assistant, applying transferable scene preferences to compare branches and guide the policy toward better intermediate layouts.The passage distinguishes this role from serving only as a reward estimator during GRPO optimization.Source: §E.2 · p. 33
  • GPT-5.2 can offer plausible rationales yet overlook collisions and functional relationships, producing branch preferences inconsistent with scene quality.Examples include selecting a layout with a table–lamp collision and favoring locally appealing cues over global organization.Source: §E.2 · p. 33

E.3 Examples of Scene-Designer and Other Methods

Qualitative comparisons show that Scene-Designer improves scene completeness and organization beyond existing methods and Scene-Builder, with further gains from inference-time Critic guidance. The supplementary material also describes staged prompts, compact policy memory, and structured correction-effect evaluation.

E.3 Examples of Scene-Designer and Other Methods

  • GPT-5.2 may give detailed rationales yet select branches inconsistently with scene quality, including overlooking physical violations and functional relationships.The figure caption characterizes this as a representative failure case during test-time critique.Source: Figure 28 · p. 34
  • Scene-Designer improves functional organization and layout coherence beyond Scene-Builder, and inference-time Critic guidance further enhances scene quality across physical validity, functional correctness, and layout coherence.Existing methods often produce incomplete scenes or physical and functional problems, while Scene-Builder provides valid structures and basic physical constraints.Source: §E.3 · pp. 33–35, Figure 29 · p. 35

F.1 System Prompts

  • Construction uses four stages—room structure, floor-standing assets, wall-mounted assets, and ceiling-mounted assets—while recording them as one unified Observation–Thought–Action trajectory.Stage-specific prompts define the current objective and available operations; a reasoning trace without an executable tool call marks stage completion.Source: §F.1 · pp. 35–36

F.2 Memory Snapshot

  • At each reasoning step, the policy receives a memory snapshot of the current stage state, physical-validation status, and recent tool interactions instead of the full trajectory.The memory is updated from the result of a successful tool execution.Source: §F.2 · p. 36

F.3 Correction-Effect Evaluation Prompt

  • Correction-effect evaluation gives GPT-5.2 structured trajectory and operation evidence and requests operation-level labels, confidence scores, and brief causal explanations.The input includes scene states, progress signals, validation or revision feedback, and object lifecycle context.Source: §F.3 · pp. 36–37

G Additional Experiments

  • The supplementary section presents additional experiments and analyses to evaluate the method from complementary perspectives.This passage introduces the additional analyses without specifying their findings.Source: G Additional Experiments · p. 38

G.1 Quantitative Evaluation of Scene Design Quality

Geometry-based evaluation indicates that Scene-Designer improves spatial organization over Scene-Builder, with Critic guidance yielding the strongest reported reachability and navigable-area results. The metrics provide complementary evidence about accessibility while NAR is interpreted as congestion, not overall scene quality.

G.1 Quantitative Evaluation of Scene Design Quality

  • Reachability Ratio measures how much free space belongs to the largest connected region, while Full Reachability tests whether all free space forms one connected region.The reported mRR averages RR at agent widths 0.2, 0.3, and 0.5 to reduce sensitivity to a single assumed movement width.Source: §G.1 · p. 38
  • Navigable Area Ratio characterizes the room area available for movement after accounting for furniture and agent width, but is not a monotonic measure of scene quality.An under-furnished or nearly empty room may have high NAR despite failing the design instruction, so the authors use it mainly to characterize congestion.Source: §G.1 · pp. 38–39
  • Door Accessibility Rate measures doors with unobstructed clearance, and Window Accessibility Rate measures windows not blocked by furniture extending above the sill.Furniture below a window sill is not counted as an obstruction; all methods achieve perfect DAR and Critic guidance achieves the highest WAR.Source: §G.1 · p. 39
  • The geometry-based metrics are computed per scene and macro-averaged, using the same geometric rules, thresholds, and agent widths for all methods.They are derived from room geometry and final asset placements without GPT or rendered images.Source: §G.1 · pp. 38–39
  • mRR reaches 0.98 and FR@0.3 reaches 1.0 for Scene-Designer w/ Critic, which also achieves the highest RR across agent widths and the highest navigable area ratio.Scene-Designer also exceeds Scene-Builder on RR@0.3, RR@0.5, and FR@0.3; all methods achieve perfect DAR, while Scene-Designer w/ Critic has the highest WAR.Source: Table 6 · p. 38, §G.1 · p. 39

Related papers

Loading 2610.23121v1…