PaperlayerView exact revision on arXiv

Source-linked AI summary

SceneEval: Evaluating Semantic Coherence in Text-Conditioned 3D Indoor Scene Synthesis

Hou In Ivan Tam, Hou In Derek Pun, Austin T. Wang, Angel X. Chang, Manolis Savva

arXiv:2503.14756v3cs.GRcs.CV

TL;DR

Text-conditioned 3D indoor scene evaluation often misses whether generated scenes meet specific text requirements and implicit expectations of plausibility. SceneEval combines fine-grained explicit and implicit metrics with SceneEval-500, then evaluates six recent methods, finding persistent shortcomings and an execution-time cost that grows with scene object count.

  • Problem

    Existing metrics do not provide fine-grained assessment of text constraints and broader implicit plausibility, while prior benchmarks often lack annotated ground truth.

    Source: §1 · p. 2

  • Method

    SceneEval combines metrics for explicit requirements and implicit plausibility with SceneEval-500, a benchmark of 500 annotated indoor scene descriptions.

    Source: §1 · p. 2

  • Results

    Experiments on six recent methods reveal consistent shortcomings in satisfying detailed user constraints and ensuring physical plausibility.

    Source: §1 · p. 2, §5 · p. 8

  • Takeaways & Limitations

    SceneEval and SceneEval-500 establish a shared reference for systematic evaluation of text-conditioned 3D indoor scene generation.

    Source: §1 · p. 2, §5 · p. 8

    SceneEval’s execution time scales with scene object count, increasing as scenes become more complex.

    Source: §D.1 · p. 16

Abstract

from arXiv · show

Despite recent advances in text-conditioned 3D indoor scene generation, there remain gaps in the evaluation of these methods. Existing metrics often measure realism by comparing generated scenes to a set of ground-truth scenes, but they overlook how well scenes follow the input text and capture implicit expectations of plausibility. We present SceneEval, an evaluation framework designed to address these limitations. SceneEval introduces fine-grained metrics for explicit user requirements-including object counts, attributes, and spatial relationships-and complementary metrics for implicit expectations such as support, collisions, and navigability. Together, these provide interpretable and comprehensive assessments of scene quality. To ground evaluation, we curate SceneEval-500, a benchmark of 500 text descriptions with detailed annotations of expected scene properties. This dataset establishes a common reference for reproducible and systematic comparison across scene generation methods. We evaluate six recent scene generation approaches using SceneEval and demonstrate its ability to provide detailed assessments of the generated scenes, highlighting strengths and areas for improvement across multiple dimensions. Our results identify significant gaps in current methods, underscoring the need for further research toward practical and controllable scene synthesis.

1. Introduction

Text-conditioned 3D scene synthesis must satisfy both explicit user requirements and implicit expectations of physical plausibility, but existing evaluation does not assess these dimensions comprehensively. SceneEval pairs fine-grained metrics with SceneEval-500, then evaluates six recent methods and finds significant remaining gaps.

1. Introduction

  • Practical scene synthesis requires both stated text requirements and assumed expectations such as physical plausibility.Manual scene authoring is laborious, and natural-language descriptions can vary from minimal to detailed.Source: §1 · p. 1
  • Existing metrics either depend on ground-truth scenes, provide coarse text–scene correspondence, or assess only isolated aspects of plausibility.These approaches do not show which textual constraints are satisfied or violated, and isolated checks can miss problems such as unnavigable scenes without collisions.Source: §1 · pp. 1–2
  • SceneEval-500 contains 500 indoor scene descriptions annotated with verifiable object counts, attributes, spatial and architectural relationships.The annotations provide a standardized reference for evaluating explicit user requirements.Source: §1 · p. 2
  • SceneEval evaluates four explicit-constraint categories and five implicit-plausibility categories in generated scenes.Explicit metrics cover counts, attributes, object–object and object–architecture relationships; implicit metrics cover collisions, support, accessibility, out-of-bounds, and navigability.Source: §1 · p. 2
  • SceneEval first matches generated-scene object instances to annotated categories, then evaluates fidelity and plausibility metrics.The overview depicts the evaluation pipeline applied to a generated scene and its annotated properties.Source: Figure 2 · p. 2
  • Six recent methods show significant gaps in satisfying explicit user requirements and implicit expectations in generated scenes.The evaluation highlights strengths and weaknesses across methods, motivating further research.Source: §1 · p. 2

2. Related Work

Prior work spans rule-based, data-driven, and deep-learning scene generation, while text-conditioned generation and evaluation have grown across modalities. For 3D indoor scenes, existing approaches offer limited fine-grained assessment of text fidelity and implicit plausibility.

3D Indoor Scene Generation

  • Scene-generation research has progressed from manual placement systems to rule-based, data-driven, and deep-learning methods, with recent works using LLMs and VLMs as text parsers and spatial priors.Text-conditioned generation is one of several input paradigms, alongside room types, floor plans, and scene graphs.Source: §2 · p. 2

Evaluation of Text-conditioned Generation

  • Evaluation methods across text, images, video, and 3D shapes assess alignment with input text, but 3D scene descriptions add viewpoint ambiguity and spatial constraints.Holistic 3D shape-generation evaluators lack the precision needed for fine-grained scene fidelity, and 2D metrics do not transfer readily to 3D scene generation.Source: §2 · pp. 2–3

Evaluation of Text-conditioned Scene Generation

  • SceneEval evaluates generated scenes against fine-grained properties in the input text and includes implicit expectations such as accessibility and navigability.This contrasts with prior approaches that provide overall correspondence but little insight into which text aspects are preserved or lost.Source: §2 · p. 3
  • Recent 3D scene evaluation uses distributional metrics, text-image similarity, and language-model evaluators, but these provide little fine-grained insight into text fidelity.Other directions annotate existing datasets with descriptions and relationships or use scene-graph-based evaluations.Source: §2 · p. 3

3. SceneEval

SceneEval pairs annotated text constraints with object matching to measure both fidelity to a description and physical plausibility. Its metrics cover explicit scene properties and implicit expectations, from object counts and relationships to support and navigability.

3. SceneEval

  • SceneEval evaluates fidelity to explicit scene requirements and plausibility against implicit expectations, using direct geometric checks where possible to improve interpretability.It establishes object-to-category correspondences from rendered object images before computing fidelity and plausibility metrics.Source: §3 · p. 3

3.1. SceneEval-500

  • SceneEval-500 contains 500 annotated descriptions across ten room types and three difficulty levels, with a textual scene graph specifying objects, attributes, and relationships.The dataset combines 100 manually written descriptions with 400 LLM-generated entries that were manually validated; most generated entries needed at least one edit.Source: §3.1 · pp. 3–4

3.2. Object Matching

  • SceneEval matches scene objects to annotated categories using VLM judgments on rendered front views, leaving categories without matches unmatched.This correspondence step addresses incomplete or unreliable object metadata and underpins the fidelity metrics.Source: §3.2 · p. 4

3.3. Text Fidelity Metrics

  • Four fidelity metrics assess object counts, attributes, object–object relationships, and object–architecture relationships against the description.OOR maps open-vocabulary relations to 13 categories and verifies them geometrically; OAR checks relations such as objects against walls or in room corners.Source: §3.3 · p. 5

3.4. Plausibility Metrics

  • Five plausibility metrics assess collisions, support, navigability, accessibility, and whether objects remain within the room boundaries.Collision and support checks cover physical feasibility; navigability measures connected free space, accessibility checks functional sides, and out-of-bounds flags objects when fewer than 99% of sampled surface points intersect the floor.Source: §3.4 · pp. 5–6

4. Experiments

The experiments evaluate six scene-generation approaches and examine how SceneEval assesses fidelity, plausibility, and agreement with human judgments. Results show strong overall fidelity for Holodeck but persistent difficulty satisfying detailed constraints.

4.1. Evaluated Methods

  • The evaluation covers ATISS, DiffuScene, InstructScene, LayoutGPT, LayoutVLM, and Holodeck, with differences in training data, assets, and conditioning capabilities.ATISS is included as a non-text-conditioned baseline; Holodeck and LayoutVLM use Objaverse assets, while several other methods use 3D-FRONT.Source: §4.1 · p. 6

4.2. Analysis

  • SceneEval’s fidelity judgments show agreement with human evaluations, with reported agreement rates from 83.5% to 94.6% across four constraint types in the manual evaluation.A separate user study on 125 scenes also reports agreement rates from 88.1% to 91.7%.Source: §4.2 · p. 6
  • Image-based metrics offer relative comparisons but give little diagnostic insight and can be sensitive to rendering or unavailable reference datasets.SceneEval instead evaluates generated scenes against text descriptions and separates counts, attributes, relationships, and plausibility.Source: §4.2 · p. 6
  • Holodeck achieves the strongest overall fidelity, although LayoutVLM slightly surpasses it on object counts; LayoutGPT’s apparent plausibility is qualified by high out-of-bounds rates.LayoutGPT also scores best on collisions and navigability but worst on out-of-bounds and support, illustrating why plausibility requires complementary checks.Source: Table 4 · p. 8, §4.2 · p. 8
  • Fewer than 30% of attribute requirements are satisfied even by the best method, and no method satisfies more than 20% of object-object relationships.The authors hypothesize that retrieval limitations and limited dataset diversity contribute to weak attribute performance.Source: §4.2 · p. 8

5. Conclusion

The paper presents SceneEval and SceneEval-500 for evaluating text-conditioned 3D indoor scene generation. Across six recent methods, experiments reveal shortcomings in satisfying detailed constraints and ensuring physical plausibility.

5. Conclusion

  • SceneEval separately assesses explicit user requirements and implicit expectations, enabling more interpretable and diagnostic evaluation.The framework is designed for text-conditioned 3D indoor scene generation.Source: §5 · p. 8
  • SceneEval-500 provides a shared benchmark of 500 scene descriptions with fine-grained annotations for systematic evaluation.The benchmark is presented as a common reference point for future research.Source: §5 · p. 8
  • Experiments on six recent methods reveal consistent shortcomings in satisfying detailed user constraints and ensuring physical plausibility.The authors describe SceneEval and SceneEval-500 as tools for research toward methods better aligned with user needs and expectations.Source: §5 · p. 8

Appendices

The appendices provide dataset, metric, evaluation, and study details. SceneEval-500 annotations represent counts, attributes, and spatial relationships using structured fields.

Appendices

  • The appendices cover SceneEval-500, metric implementation, an open-source VLM evaluation, work limitations, user-study details, VLM prompts, and artifact information.Dataset details include the annotation schema and data-collection process.Source: Appendices · p. 12, A. Dataset Details · p. 12

A.1. Annotation Schema

  • SceneEval-500 annotations use four fields: object count, object attribute, object-object relationships, and object-architecture relationships.The schema defines quantifiers and quantities for counts, open-vocabulary category and attribute labels, and relationships expressed with prepositions or verbs.Source: §A.1 · p. 12

A.2. Manual Data Collection Process

The dataset’s initial 100 descriptions and annotations were manually created and validated. Additional results report how scene-generation performance varies with description difficulty.

A.2. Manual Data Collection Process

  • The initial 100 English scene descriptions and annotations were authored by annotators primarily from Asia, aged 20–30, with computer-science backgrounds and scene-generation familiarity.They had lived in Western countries and spoke English as a second language.Source: §A.2 · p. 12
  • Annotators received difficulty-level definitions, the annotation schema, and target scene counts, then wrote diverse descriptions and annotated them.They drew inspiration from daily life or online sources.Source: §A.2 · p. 12
  • As description difficulty increases, methods’ performance in generating the correct number of objects decreases; Holodeck performs best overall across difficulty levels.The appendix reports results by difficulty for six methods.Source: Table 6 · p. 13
  • Authors validated descriptions and annotations for quality and consistency, and a VLM checked descriptions for grammatical errors and typos.No personally identifiable information was collected.Source: §A.2 · p. 13

B. Metric Implementation Details

The appendix specifies how SceneEval renders objects, operationalizes support, and scores predefined spatial relationships. Its implementation combines VLM-based interpretation with geometric tests on scene objects.

B.1. Object Renderings

  • Front View, Size Reference, and Surrounding Context renderings respectively emphasize appearance, size relative to a 170 cm figure, and surrounding context for LLM evaluation.Figure 7 shows examples of the three rendering types.Source: §B.1 · p. 13, Figure 7 · p. 14

B.2.1. Object-Object Relationship

  • Object-object annotations map to one or more of 13 predefined relationships; each matching object combination is scored, and all mapped relationships must hold.The implementation evaluates every possible combination of objects in the specified categories.Source: §B.2.1 · p. 14

B.2.2. Object Support

  • Support evaluation uses front and contextual views to classify support type, then checks directional ray contacts and, for ground or object support, whether the projected centroid falls within the contact-point hull.Wall and ceiling objects count as supported when they have any valid contact points; the reported score is the percentage of supported instances.Source: §B.2.2 · p. 14

B.3.1. Object-Object Relationships

  • The 13 object-object relationships use a 0.5 positive/negative threshold unless specified otherwise; inside/outside and face scores use bounding-box samples and front-facing ray intersections.Face scores decline from 1.0 at 0 degrees to 0.0 near 30 degrees.Source: §B.3.1 · p. 15
  • Side-of and side-region scores use bounding-box samples, with side-of excluding interior points and extending the reference box by 25%, unlike side-region.Long/short-side scores use the reference object's bounding-box dimensions.Source: §B.3.1 · p. 15
  • On-top uses the side-of procedure without box extension; middle-of scores 2D centroid distance, while surround scores normalized distance and angle deviations from uniform placement.Surround scores are computed from the deviations for each surrounding object.Source: §B.3.1 · p. 15
  • Next to, near, across, and far use closest-point distances with respective ranges of 0–0.5, 0.5–1.5, 1.5–4.0, and at least 4.0 meters.Scores are 1 within the specified range and decrease as distance deviates from it.Source: §B.3.1 · p. 15

B.3.2. Object-Architecture Relationships

Object-architecture relationships are scored using predefined spatial tests, including room-relative placement and proximity to architectural elements.

B.3.2. Object-Architecture Relationships

  • The object-architecture metric defines 10 relationships and uses a 0.5 threshold; next to, near, across, and far follow the object-object distance definitions.Inside-room scores use sampled object-box points intersecting the floor plane; middle-room uses 2D centroid distance and accounts for object and room size.Source: §B.3.2 · p. 15
  • Corner-room scores combine proximity to two perpendicular walls, while on-wall and against-wall combine a front-of-wall score with distance to the wall.The distance ranges are 0–0.8 m for corner walls, 0–0.01 m for on-wall, and 0–0.3 m for against-wall.Source: §B.3.2 · p. 16
  • Hang-ceiling evaluates distance to the ceiling using a 0–0.01 m range and a Gaussian standard deviation of 0.03.The passage describes this as similar to the next-to relationship, with the ceiling as the reference element.Source: §B.3.2 · p. 16

C. SceneEval with Open-Source VLM

The authors tested SceneEval with a locally run open-source VLM and found consistent evaluation trends, though agreement was lower than with GPT-4o. The paper also notes that its metrics and dataset remain imperfect.

C. SceneEval with Open-Source VLM

  • Across 500 scenes generated from 100 manually written descriptions, Qwen2.5-VL-7B-Instruct preserved the overall trends obtained with GPT-4o.Agreement was lower than with GPT-4o, particularly for Attribute, which relies solely on the VLM; the authors describe the pipeline as modular and not overly dependent on one model.Source: C. SceneEval with Open-Source VLM · p. 16
  • Manual-evaluation agreement with Qwen2.5-VL-7B-Instruct was 80.65% for Object Count, 76.64% for Attribute, 91.11% for Object-Object Relationship, and 87.72% for Object-Architecture Relationship.The corresponding Cohen’s kappas were 0.50, 0.26, 0.43, and 0.47; on the overlapping user-study subset, agreement was 77.55%, 75.00%, 78.67%, and 83.48%, respectively, with kappas of 0.48, 0.12, 0.32, and 0.48.Source: C. SceneEval with Open-Source VLM · p. 16

D. Limitations

  • The authors state that SceneEval and SceneEval-500 provide broader coverage than existing metrics but are not perfect.The passage introduces further discussion of limitations in the metrics, dataset, and semi-automatic data generation process.Source: D. Limitations · p. 16

D.1. Limitations in SceneEval

The stated limitations concern unmodeled commonsense placement expectations and runtime growth with scene size.

D.1. Limitations in SceneEval

  • SceneEval does not assess unstated commonsense placement expectations, such as bookshelves typically being against walls except when dividing spaces.The authors say these expectations are less well-defined, making metric integration challenging and a subject for further research.Source: §D.1 · p. 16
  • SceneEval’s execution time scales with the number of scene objects, so more complex scenes require more time for matching and metric computation.The authors identify parallelization and other optimization techniques as future-work directions.Source: §D.1 · p. 16

D.2. Limitations in SceneEval-500

SceneEval-500 remains limited in scale, and its annotations may not represent expectations universally. The authors identify both dataset size and annotator backgrounds as limitations.

D.2. Limitations in SceneEval-500

  • SceneEval-500’s scale remains limited compared with datasets in domains such as image generation.The authors say expansion, especially through scalable automatic methods, would support more comprehensive evaluation and deeper analysis of model capabilities.Source: §D.2 · p. 16
  • Scalable expansion is proposed as a way to enable more comprehensive evaluations and deeper analysis of model capabilities.The authors identify automatic methods as a possible way to expand the dataset.Source: §D.2 · p. 16
  • The dataset’s creators and validators are primarily from Asia and have lived in Western countries, and none are trained interior designers or architects.The authors caution that the dataset may reflect cultural assumptions and expectations that are not universally representative.Source: §D.2 · p. 16

D.3. Semi-Automatic Data Generation Limitations

This section documents the limitations and procedures involved in semi-automatic data generation and annotation. It covers VLM-assisted prompts and mapping rules, scene-description and annotation instructions, the user-study interface, and associated licenses and AI tools.

D.3. Semi-Automatic Data Generation Limitations

  • Semi-automatic data generation encounters four failure modes that limit scalability and prevent fully automatic generation.These are missing attributes or relationships, hallucinated annotations and anchor indices, merged non-atomic attributes, and decreasing scene diversity; 101 missing attributes and 154 missing relationships were manually added across 210 medium and hard descriptions. Improving generation to reduce manual effort and support larger, more diverse datasets is future work.Source: §D.3 · p. 17

E. User Study Details

  • User-study participants inspect a 3D scene in an interactive viewer and mark each listed property True or False after examining the scene and expected properties.Participants save their completed responses to a dedicated cloud storage location.Source: E. User Study Details · p. 17

F. VLM Prompts

  • SceneEval uses a VLM in six tasks, with prompts supplied for evaluation and semi-automatic data generation; its system prompt establishes task context and the VLM’s role.The system prompt frames the VLM as an interior-design expert evaluating spatial arrangements according to task instructions.Source: F. VLM Prompts · p. 17, §F.2 · p. 17, §F.1 · p. 17

F.2.1. Object Matching

  • Object matching uses a front-view image to ask whether an object belongs to any ground-truth annotated category, with a justification.The object-attribute task uses front-view and human-scale images to assess whether objects satisfy annotated attributes and asks for reasons.Source: §F.2.1 · p. 17, §F.2.2 · p. 18

F.2.3. Object Support Type

  • The support-type task uses front and zoomed-out images to classify each object’s support as ground, wall, ceiling, or another object.The VLM provides a reason for its selection.Source: §F.2.3 · pp. 18–19

F.2.4. Object Functional Sides

  • The functional-sides task identifies accessible sides needed for proper use, considering front, back, left, and right; at least one must be accessible.Small objects such as cups and books placed on a surface receive no functional sides.Source: §F.2.4 · p. 19

F.2.5. Object Relationship Mapping

  • The relationship-mapping task maps annotated open-vocabulary object relationships to predefined spatial types, allowing multiple types when needed to fully describe a relationship.The prompt supplies type definitions, examples, and mapping guidelines, and asks for the mapped types with necessary information.Source: §F.2.5 · p. 19
  • Object relationships are mapped to predefined spatial types, with multiple types and ordered side values used when needed to represent one relation.The guidelines distinguish relations such as on_top from inside_of, define distance-based categories, and allow None when no type fits.Source: §F.2.5 · p. 20

F.2.6. Architectural Relationship Mapping

  • Architectural relationships are mapped to predefined types for objects relative to building elements, with element selection and floor-identification guidance.The mapping uses architectural categories including wall, floor, ceiling, room, window, and door; when floor IDs are inconclusive, annotators select all floors.Source: §F.2.6 · pp. 20–21

F.3. Semi-Automatic Data Generation

  • The semi-automatic data-generation process uses a VLM to create scene descriptions and annotations from manually authored in-context examples.Source: §F.3 · p. 21

F.3.1. System Prompt

  • The system prompt supplies the task, annotation schema, and guidelines, including object counts encoded with comparison operators and category names separated from descriptive attributes.For example, exactly three chairs are represented as eq,3,chair; the category is chair rather than an instance name, and descriptive qualifiers belong in attributes.Source: §F.3.1 · p. 21
  • The system prompt defines structured annotations for object counts, attributes, object-object relations, and object-architecture relations, distinguishing relations between objects from those involving architectural elements.Relationship records use quantifiers and references; architectural references include elements such as walls, floors, and windows.Source: §F.3.1 · p. 22

F.3.2. In-Content Example

  • The annotation prompt supplies manually annotated in-context examples and instructs the VLM to learn their format and exhaustively cover the scene’s objects.The examples are presented as scene descriptions paired with annotations before the VLM annotates a new description.Source: §F.3.2 · p. 22

F.3.3. Generate Scene Description

  • Scene-description prompts offer single- and multiple-description versions, set easy, medium, and hard object-count ranges, and request clear arrangements without viewpoint-dependent directions.Easy scenes have at most four objects, medium scenes have five to eight, and hard scenes have at least nine; descriptions should include object styles and relationships as needed.Source: §F.3.3 · p. 22

F.3.4. Generate Annotations

  • The annotation-generation prompt has single- and multiple-scene versions and asks the VLM to annotate the final scene description using the examples and the four annotation definitions.The description may have been edited by the user before annotation.Source: §F.3.4 · p. 23
  • Annotations should exhaustively cover scene objects, follow the specified format, use underscores, and record explicitly absent objects, such as with “eq,0,chair”.The instructions also allow “facing” as an alternative to “front.”Source: §F.3.4 · p. 24

G. Scientific Artifacts

  • The paper lists licenses or terms of use for its datasets, code, and models, and states that their use complies with those terms.It notes that LayoutVLM [48] has no license specified in its repository.Source: G. Scientific Artifacts · p. 24

H. AI Assistant Usage

  • GPT-4o [1] supported parts of the evaluation framework, o4-mini supported data generation, and Qwen2.5-VL [4] was used in open-source-model experiments.GitHub Copilot6 and ChatGPT7 assisted with code and proofreading.Source: H. AI Assistant Usage · p. 24
Loading 2503.14756v3…