PaperlayerView exact revision on arXiv

Source-linked AI summary

Apples on the Table? Evaluating Text-Guided 3D Scene Synthesis via Fine-Grained Constraint Verification

Minseok Kang, Dongwook Choi, Gyeom Hwangbo, Seungwon Lim, Kai Tzu-iunn Ong, Jinyoung Yeo

arXiv:2511.03001v3cs.CL

TL;DR

Existing text-guided 3D scene evaluators capture coarse similarity or miss spatial reasoning needed to verify fine-grained constraints. The paper introduces LEGO, a dataset pairing descriptions with annotated constraints and reference scenes, and LEGO-Eval, which checks constraints with tools. LEGO-Eval is reported to outperform existing evaluators, while current synthesis methods achieve holistic success rates below 10%.

  • Problem

    Existing evaluators capture only coarse text-scene similarity or lack the spatial reasoning to verify object placement, while no paired dataset supplies fine-grained constraints for evaluation.

    Source: §1 · p. 2

  • Method

    LEGO pairs descriptions with human-annotated constraints and reference scenes, while LEGO-Eval decomposes descriptions into constraints and verifies them against synthesized scenes using tools.

    Source: §1 · p. 2

  • Results

    Current scene synthesis methods achieve holistic success rates below 10%, and LEGO-Eval substantially outperforms existing evaluators.

    Source: §8 · p. 9

  • Takeaways & Limitations

    LEGO and LEGO-Eval provide a foundation for progress toward more faithful text-guided synthesis and reliable 3D environments for embodied-agent training.

    Source: §8 · p. 9

    LEGO currently covers rectangular or square rooms, LEGO-Eval tools are simulator-dependent, and evaluating the benchmark takes approximately two hours.

    Source: Limitations · p. 10

Abstract

from arXiv · show

Accurately synthesizing 3D scenes from user-provided text descriptions is crucial for developing embodied agents. Despite the importance of scene-description alignment, existing evaluation methods for such text-guided 3D scene synthesis either capture only coarse similarity between the synthesized scene and the user description, or ignore the spatial reasoning for verifying object placement. None of them addressed the fine-grained constraints (e.g., X needs to be in the scene in a Y manner) implied by the description from users. To address this, we introduce LEGO, a benchmark dataset that pairs each user description with human-annotated constraints and a reference scene, and LEGO-Eval, an evaluation framework that decomposes a description into atomic constraints and verifies each one using tools that ground textual references to 3D objects and reason about their spatial relationships. We show that (i) LEGO-Eval evaluates misalignment far more accurately than existing methods and (ii) current scene synthesis approaches achieve at most 10% success rate in LEGO-Eval.

1 Introduction

Text-guided 3D synthesis must satisfy fine-grained scene constraints, but existing evaluation misses either detailed alignment or spatial reasoning. The paper introduces LEGO and LEGO-Eval to support constraint-level diagnosis and reports more accurate evaluation than existing methods.

1 Introduction

  • Existing evaluators capture only coarse text-scene similarity or lack the 3D spatial reasoning to verify object placement, and no paired dataset provides fine-grained constraints.Misalignment can introduce misleading supervision and degrade agent performance, motivating fine-grained diagnosis of where scenes fail to align.Source: §1 · pp. 1–2
  • LEGO pairs natural-language descriptions with fine-grained annotated constraints and manually constructed reference scenes, plus metadata on scene components.The metadata includes component positions, orientations, scales, and material properties.Source: §1 · p. 2
  • LEGO-Eval decomposes descriptions into individual constraints and verifies them against synthesized scenes with tools, enabling constraint-level diagnosis.Its tool-based design goes beyond vision-only judgments.Source: §1 · p. 2
  • LEGO-Eval evaluates misalignment substantially more accurately than existing methods, while current synthesis approaches fall short of satisfying fine-grained descriptions.The conclusion reports holistic success rates below 10% for current scene synthesis methods.Source: §1 · p. 2

2 Background

Text-guided synthesis aims to construct scenes that reflect user intent, because scene-grounded tasks depend on those environments. Evaluating alignment requires identifying relevant objects and checking their properties and spatial relationships.

2 Background

  • Text-guided synthesis maps a user's natural-language description to a structured scene representation intended to reflect that user's intent.The paper denotes the description as l and the synthesized scene as S = F(l).Source: §2 · p. 2
  • Scene misalignment shifts the task distribution embodied agents encounter, undermining learning of intended behaviors and making observed successes or failures hard to interpret.Failures may reflect environment errors rather than agent behavior.Source: §2 · p. 2
  • Checking alignment requires grounding each textual constraint to scene components and reasoning over their properties and relationships, such as object identity, color, and placement.The paper uses a blue chair next to a black desk as an example of multi-hop grounding.Source: §2 · p. 2

3 LEGO: Dataset for Text-Guided 3D Scene Synthesis Evaluation

LEGO is a human-curated dataset for fine-grained evaluation of text-guided 3D scene synthesis. It pairs descriptions and annotated constraints with manually constructed scenes and metadata, built through human annotation, scene refinement, and cross-validation.

3.1 Dataset Overview

  • Each LEGO instance includes a fine-grained description with annotated constraints, a manually constructed scene, and rich scene metadata.The dataset contains 130 descriptions and 1,250 constraints, averaging 9.6 constraints per description, and spans diverse rooms, receptacles, objects, and structural elements.Source: §3.1 · p. 3, Figure 2 · p. 3

3.2 Dataset Construction

  • Four annotators construct descriptions from observations and indoor images, identify embedded constraints, and classify and map them to text segments.Dependent constraints are restructured as conditional expressions, and attributes of the same entity are merged.Source: §3.2 · p. 3
  • Annotators generate initial scenes with Holodeck, refine them to satisfy all constraints, and cross-validate annotation accuracy and scene-description alignment over two iterations.Scenes use Objaverse objects and structural assets from ProcTHOR and AI2-THOR libraries.Source: §3.2 · p. 3

4 LEGO-EVAL: Evaluation with Tool-Augmented VLMs

LEGO-EVAL uses a VLM judge and tools to decompose scene descriptions into constraints, verify them individually, and produce an overall judgment with a diagnostic report.

4 LEGO-EVAL: Evaluation with Tool-Augmented VLMs

  • LEGO-EVAL decomposes descriptions into constraints and verifies each against a generated scene with tools, enabling fine-grained diagnosis of scene-description alignment.Each constraint receives a binary judgment and explanation, which contribute to the final evaluation report.Source: §4 · p. 3, §4.1 · p. 4

4.1 LEGO-EVAL

  • The VLM assigns each constraint a type, plans and executes relevant tools sequentially, reuses prior retrievals, and aggregates the results into a final judgment and report.The final judgment is valid only when all constraint judgments are valid.Source: §4.1 · pp. 4–5

4.2 Implementation Details

  • LEGO-EVAL runs in AI2THOR and uses 21 tools spanning environment interaction, textual reasoning, and multimodal reasoning.These tools retrieve visual information, structured scene attributes, and text descriptions from images.Source: §4.2 · p. 5

5 Experiments

The experiments compare evaluation methods with human judgments, ablate LEGO-EVAL’s tools, and benchmark current scene synthesis methods on LEGO.

5.1.1 Setup

  • The evaluation-method comparison uses LEGO descriptions, constraints, and scenes, plus manually crafted violating scenes, for 260 pairs from 130 original descriptions.Baselines include CLIPScore, VLM-as-a-judge, and SceneEval; SceneEval cannot assess 41% of LEGO constraints, so it is evaluated only on those it can handle.Source: §5.1.1 · p. 5

5.1.2 Results

  • LEGO-EVAL achieves the highest F1 and Cohen’s κ among baselines at both holistic and partial levels, including with the Qwen3-VL-8B backbone.In the Figure 5 case study, LEGO-EVAL recognizes that two referenced objects are absent, unlike the other judges.Source: §5.1.2 · p. 6, Figure 5 · p. 6

5.1.3 Ablation Study

  • Disabling any tool type degrades evaluation, with the largest performance drop when textual tools are disabled.Textual tools capture subtle attributes such as small-object colors and convey information extracted from retrieved images.Source: §5.1.3 · pp. 6–7, Table 3 · p. 6

5.2.1 Setup

  • Synthesis performance is measured with Holistic SR, the share of descriptions satisfying every constraint, and Partial SR, the share of valid constraints.Partial SR is also reported by constraint type.Source: §5.2.1 · p. 7

5.2.2 Results

  • No current synthesis method exceeds 10% Holistic SR; most exceed 50% average Partial SR, but object selection and placement fall below that level.The results indicate that choosing appropriate objects and placing them appropriately remain difficult.Source: §5.2.2 · p. 7, Table 4 · p. 7
  • Existing synthesis methods consistently fail on complex descriptions, which average 18.2 constraints per room in LEGO.Descriptions are grouped as simple (2–7 constraints), moderate (8–12), or complex (more than 12).Source: §5.2.2 · p. 7, Figure 6 · p. 7

5.2.3 Human Evaluation

  • LEGO-EVAL agrees with human judgments at 72.6–89.8% across synthesis methods, and human evaluation preserves similar relative trends.Inter-evaluator agreement is 80.3%; the low absolute scores reflect the difficulty of satisfying multiple scene constraints, not an overly strict evaluator.Source: §5.2.3 · pp. 7–8, Table 5 · p. 8

6 Exploring the Reliability and Utility of LEGO-EVAL

The section tests whether LEGO-EVAL’s reliability comes from access to scene information or from selecting constraint-relevant information, and evaluates its use as refinement feedback.

6.1 Examining the Source of LEGO-EVAL’s Reliability

  • The reliability analysis contrasts LEGO-EVAL’s selective tool retrieval with baselines that infer scene information from rendered images.It tests whether more scene access or better identification alone explains the reliability gap.Source: §6.1 · p. 8

6.1.1 Does More Access to Scene Information Improve Evaluation?

  • Providing VLM-as-a-judge with the full scene graph does not improve its evaluation, and SceneEval remains below LEGO-EVAL despite structured scene access.The results point to selecting information relevant to each constraint, rather than simply having more scene information.Source: §6.1.1 · p. 8, Table 6 · p. 8

6.1.2 Does Better Access to Scene Information Improve Evaluation?

  • Adding identification modules increases total errors, especially identification errors, when the judge cannot determine whether retrieved information is relevant to the constraint.The tested modules support bounding-box detection, scene-graph generation, and 3D language-model identification.Source: §6.1.2 · pp. 8–9, Table 7 · p. 9

6.2 Applying LEGO-EVAL to Improve Scene Synthesis

  • LEGO-EVAL’s rationale improves Holodeck more than VLM-as-a-judge feedback over three refinement iterations.The authors attribute the difference to constraint-level grounding and detailed, interpretable feedback.Source: §6.2 · p. 9

7 Related Work

Related work covers text-guided scene synthesis approaches and automatic methods for evaluating how well synthesized scenes match descriptions.

7 Related Work

  • Text-guided synthesis methods generate complete environments, select and place objects, or optimize layouts of predefined assets.Examples include Feng et al. (2023), Yang et al. (2024b), and Bucher and Armeni (2025) for complete environments; Yang et al. (2024a) and Çelen et al. (2025) for object selection and placement; and Sun et al. (2025) and Ran et al. (2025) for layout optimization.Source: §7 · p. 9
  • Prior evaluation uses CLIPScore or multi-view VLM judgments, while SceneEval applies predefined criteria with limited relation and architectural-attribute coverage.LEGO-EVAL is described as supporting spatial reasoning across scene components and diverse natural-language relationships through tools.Source: §7 · p. 9

8 Conclusion

LEGO pairs fine-grained scene descriptions with annotated constraints and manually constructed reference scenes; LEGO-EVAL evaluates misalignment at the constraint level. The authors report that LEGO-EVAL outperforms existing evaluators and that current synthesis methods have holistic success rates below 10%.

8 Conclusion

  • LEGO pairs natural-language scene descriptions with fine-grained annotated constraints and manually constructed reference scenes.LEGO-EVAL evaluates scene-description misalignment at the constraint level.Source: §8 · p. 9
  • Current scene synthesis methods achieve holistic success rates below 10%, while LEGO-EVAL substantially outperforms existing evaluators.The authors frame this as a gap between current capabilities and real-world demands.Source: §8 · p. 9

Limitations

The limitations section identifies boundaries in LEGO’s room geometry, LEGO-EVAL’s tool dependencies, and evaluation time. The annotation process combines dedicated tools, human constraint labeling, scene construction, and repeated verification.

Limitations

  • LEGO covers rectangular or square rooms, depends on simulator-specific tools for environment interaction and multimodal reasoning, and takes about two hours to evaluate.The authors identify broader geometries, simulator-independent evaluation, and faster execution as directions for future work.Source: Limitations · p. 10

A.1 Data Annotation Tool

  • The annotation tool tracks constraint identification, while Label Studio records constraint types and maps them to description spans.The paper illustrates the annotation interfaces in Figures 8 and 9.Source: §A.1 · p. 13, Figure 8 · p. 13, Figure 9 · p. 14

A.2 Details of Human Annotation

  • Annotators receive two hours of training, write 30–50 descriptions, specify viewpoints, and label constraints, including prerequisite-dependent constraints and unified attributes.Constraints are classified as floor layout, material selection, object selection, or object placement, then mapped to corresponding description spans.Source: §A.2 · p. 13
  • Annotators use Holodeck to draft scenes, manually revise them to satisfy descriptions, and mutually review annotations and scenes for two iterations.The review checks constraint identification and scene alignment, with re-annotation or scene modification when needed.Source: §A.2 · pp. 13–14

A.3 Statistics

LEGO contains diverse indoor-scene descriptions and annotated constraints. The implementation uses AI2-THOR and a mixture of closed- and open-source language and vision-language models.

A.3 Statistics

  • LEGO contains 130 descriptions and 1,250 constraints, averaging 9.6 constraints per description; 55% concern objects and 39% architectural components.About 40% concern material or object selection, while 60% concern floor layout or object placement; the paper characterizes the data as diverse and complex.Source: §A.3 · p. 14

B.1 Environment Setting

  • Experiments use AI2-THOR with assets from Objaverse, ProcTHOR, and AI2-THOR.The simulator is built on Unity and serves as the experimental environment.Source: §B.1 · p. 14

B.2 Model Inference

  • Closed-source inference uses Gemini 2.5 Pro, GPT-4.1, GPT-4.1 Mini, and GPT-5.1; open-source inference includes Gemma3-27B, Qwen2.5VL-32B, Qwen3-8B, and Qwen3-32B.Open-source models use temperature 0; input images are resized to 335 pixels, versus 1200 pixels for closed-source models.Source: §B.2 · pp. 14–15

B.3 Metric Details

The additional results examine automated constraint extraction, tool planning and use, argument selection, and evaluator errors. They report close agreement between automatic- and human-constraint evaluation, alongside findings about tool use and component performance.

B.3 Metric Details

  • F1 balances precision and recall, while Cohen’s κ measures agreement beyond chance; κ uses observed and independence-expected agreement.The evaluation measures agreement with human judgments at holistic and per-constraint levels.Source: §B.3 · p. 15

B.4 Tool Set

  • LEGO-EVAL uses 21 tools, combining scene views and spatial-relation displays with multimodal attribute checks and textual scene metadata.Examples include front views, wall views, spatial-relation visualizations, property verification, and object or room information.Source: §B.4 · p. 15, §B.4.1 · pp. 15–16, §B.4.2 · p. 16

C.1 LEGO-EVAL Provides End-to-End Automated Evaluation

  • Across four synthesis methods, LEGO-EVAL shows only minor performance differences between automatic and human-provided constraints, supporting end-to-end automated evaluation.Constraints are extracted and categorized with GPT-4.1, and the evaluations use Qwen2.5-VL-32B; Table 8 summarizes the comparison.Source: §C.1 · p. 17, Table 8 · p. 17

C.2 LEGO-EVAL Plans Tool Execution Effectively

  • Tool execution planning correlates more strongly with overall evaluation performance than argument selection, while argument selection correlates more strongly when plans are fixed to ground truth.The authors interpret planning as an orchestrating component.Source: §C.2 · pp. 17–18

C.3 Argument Selection Plays a Critical Role in Evaluation Performance

  • Qwen2.5-VL-32B achieves the highest Argument F1 among the three tested open-source models, aligning with stronger holistic and partial evaluation performance.Qwen3-32B is competitive, while Gemma3-27B lags across metrics; the authors state that argument selection affects evidence retrieved by tools.Source: §C.3 · p. 18

C.5 Error Analysis of VLM-as-a-Judge with Different Modules

  • On 100 sampled instruction–scene pairs, vanilla VLM-as-a-judge has the most identification and attribute errors; added perception modules do not consistently reduce errors.Grounding DINO increases identification errors, LASER produces more errors across categories, and Point-LLM slightly reduces spatial errors while increasing identification and attribute errors.Source: §C.5 · p. 18

C.6 Effect of LEGO-EVAL on Embodied Agent Training

  • The embodied-agent experiment uses a pickup task with 300 generated scenes, including 200 aligned and 100 intentionally misaligned scenes, to compare training-data settings.The study splits aligned scenes into training and test sets and compares zero-shot performance with fine-tuning on misaligned, mixed, or LEGO-EVAL-selected aligned data.Source: §C.6 · p. 18, §C.6.1 · pp. 18–19

C.4 Tool Usage Distributions Differ Across Constraint Types

  • Environment-interaction and textual-reasoning tools make up most tool use, while multimodal reasoning accounts for about 40% of calls for object-placement constraints.The authors attribute the increase to object identification and attribute verification that align descriptions with visual observations.Source: §C.4 · p. 18, Figure 12 · p. 19

C.6.2 Results

Training with more instruction-aligned scenes is associated with higher downstream task success, and LEGO-EVAL-selected scenes reach 10%. A survey also found that room descriptions averaged 18.2 constraints; the methods appendix introduces its method details.

C.6.2 Results

  • Success rises from 4% with only misaligned scenes to 9% with 50 aligned scenes and 10% with scenes selected by LEGO-EVAL.The reported results suggest instruction-scene alignment affects embodied-agent training and that LEGO-EVAL identifies scenes beneficial for downstream policy learning.Source: §C.6.2 · pp. 19–20

C.7 Humans Usually Provide Detailed Descriptions of the Room

  • Surveyed room descriptions average 18.2 constraints, indicating that people provide complex descriptions.Participants described room photos in response to a survey prompt.Source: §C.7 · p. 20

D Method Details

  • The methods appendix describes methods used for scene evaluation and scene generation in the experiments.Source: D Method Details · p. 20

D.1.1 Overview of Existing Evaluation Methods

The appendix reviews evaluation and synthesis methods, describes implementation choices, and provides examples of tool outputs and prompts. It also situates the work within research on realistic simulation scenes for embodied tasks.

D.1.1 Overview of Existing Evaluation Methods

  • VLM-as-a-judge receives scene images from four perspectives, while CLIPScore makes binary instruction/top-down-image judgments at thresholds 15, 20, and 25; SceneEval evaluates explicit and implicit properties.Source: §D.1.1 · pp. 20–21

D.1.2 VLM-as-a-Judge with Different Modules

  • Grounding DINO localizes text-referenced elements, while LASER scene graphs and Point-LLM point clouds add structured or 3D inputs to VLM judgment.LASER generates graphs for rooms in a multi-room environment; the scene graphs are converted to downsampled point clouds for Point-LLM evaluation.Source: §D.1.2 · p. 21

D.2.1 Overview of Scene Synthesis Methods

  • The surveyed generators span layout planning, personalized agent-based design, simulation platforms, open-universe layout optimization, and multi-floor scene generation.Source: §D.2.1 · pp. 21–22

D.2.2 Implementation of Scene Synthesis Methods

  • For comparisons, Holodeck supplies room structure; LayoutVLM receives Holodeck-selected assets, LayoutGPT's choices are category-restricted, and MANSION is limited to rectangular or square layouts.Source: §D.2.2 · p. 22

E Extended Related Work

  • The related-work discussion connects diverse, realistic scenes to embodied tasks and describes the sim-to-real gap arising from differences between simulated and real environments.Source: E Extended Related Work · pp. 22–23

F Examples of Tool Image Outputs

G Prompts Used in Our Work

Related papers

Loading 2511.03001v3…