Source-linked AI summary
Text to 3D Scene Generation with Rich Lexical Grounding
Angel Chang, Will Monroe, Manolis Savva, Christopher Potts, Christopher D. Manning
TL;DR
Prior text-to-3D systems used manually specified mappings between language and objects, while generating plausible scenes from descriptions remains challenging. This paper learns lexical grounding from paired scenes and descriptions and combines it with rule-based generation; the combined approach improves scene generation over the individual approaches and prior baselines, and its similarity metric correlates strongly with human judgments.
Problem
Method
Results
Takeaways & Limitations
Rich lexical grounding can be learned from an unaligned corpus of scenes and descriptions and can improve scene generation over prior-work baselines.
Source: §9 · p. 9The work leaves pragmatic inference for omitted information and the role of context and discourse in interpreting scene descriptions unaddressed.
Source: §8 · p. 9
Abstract
from arXiv · showhide
The ability to map descriptions of scenes to 3D geometric representations has many applications in areas such as art, education, and robotics. However, prior work on the text to 3D scene generation task has used manually specified object categories and language that identifies them. We introduce a dataset of 3D scenes annotated with natural language descriptions and learn from this data how to ground textual descriptions to physical objects. Our method successfully grounds a variety of lexical terms to concrete referents, and we show quantitatively that our method improves 3D scene generation over previous work using purely rule-based methods. We evaluate the fidelity and plausibility of 3D scenes generated with our grounding approach through human judgments. To ease evaluation on this task, we also introduce an automated metric that strongly correlates with human judgments.
1 Introduction
Text-to-3D scene generation could help people express and visualize scenes, but creating them through existing interfaces is difficult for non-experts. The paper learns lexical grounding from paired scenes and descriptions, combines it with rule-based generation, and reports improved generation and a human-correlated evaluation metric.
Motivation
- Text-to-3D generation has creative, robotics, and educational applications, while scene-design interfaces remain prohibitively complex for novice users.Prior work found the task challenging and time intensive for non-experts even with simplified interfaces.Source: §1 · p. 1
Contributions
- The learned-grounding and rule-based combination outperforms either approach alone in human judgments, and the introduced scene-similarity metric correlates strongly with those judgments.The approach targets varied lexical references to concrete 3D models.Source: §1 · p. 1
Approach
- The pipeline parses scene text into an abstract template of objects and relations, then retrieves and arranges 3D models to create a concrete scene.The paper focuses on learning textual-term-to-model mappings and integrates those groundings into rule-based generation.Source: §1 · p. 1, Figure 2 · p. 2
2 Task Description
Text-to-3D generation maps a natural-language description to a plausible 3D scene representation. This paper focuses on choosing the 3D models that represent mentioned objects, a choice that affects what later stages can arrange.
Task and scope
- Lexical grounding selects the 3D models that represent objects mentioned in the input, within a pipeline that parses a scene template before generating the scene.The template captures physical objects and constraints between them.Source: §2 · p. 2
Challenges
- Correct object selection matters because later scene-generation stages rely on that set, while language may omit common physical properties and scene structure.Prior systems relied on manually annotated object properties and relations to drive rule-based generation.Source: §2 · p. 2
3 Related Work
The related work contrasts 3D scene generation with image retrieval and prior rule-based text-to-scene systems, then distinguishes the paper’s grounding task from related grounding and image-language tasks. Its approach learns object mappings from described 3D scenes rather than relying solely on manually defined lexicons.
3 Related Work
- Unlike image retrieval, 3D scene synthesis searches a higher-dimensional space of configurations and grounds to 12,490 models from roughly 270 categories.Scene retrieval can approximate generation, but the paper emphasizes the difficulty of finding plausible and desirable 3D configurations.Source: §3 · p. 2
3.1 Text to Scene Systems
- Prior text-to-scene systems use manual language mappings that limit generalization to varied descriptions and can force unnatural phrasing; this work learns mappings from scene-description pairs.Learned lexical grounding alone is insufficient for high-quality scenes, so it is combined with rule-based handling of compositional knowledge.Source: §3.1 · p. 3
3.2 Related Tasks
- Related grounding work targets simple 3D shapes, image attributes, or spatial relations, whereas this paper grounds unrestricted free-form descriptions to a broad range of objects.The paper also contrasts 3D scene representations with image-based approaches, describing them as an intermediate structure between pixels and simpler microworlds.Source: §3.2 · p. 3
4 Dataset
The dataset pairs participant-created 3D scenes with free-form descriptions, using seed sentences to prompt scene creation and other participants to describe the results. Despite the small seed set, the scenes and descriptions show substantial variety.
Collection
- The dataset contains 1,128 scenes and 4,284 free-form descriptions, collected by having workers build scenes from 60 seed sentences and other workers describe them.The descriptions were manually sampled for quality; the authors estimate fewer than 3% were spam or unacceptably incoherent.Source: §4 · p. 4
Dataset characteristics
- More than 600 distinct 3D models appear, and over 40% of nonroom objects are rotated from their default orientation despite the extra interface step.The descriptions also typically differ substantially in length and content from the seed sentences.Source: §4 · p. 4
5 Model
The model learns lexical groundings from scene–description data and combines them with rule-based parsing, which supplies compositional information such as relationships and coreference. Its components use learned feature weights to identify object categories and select 3D models.
5 Model
- The paper combines learned lexical grounding with rule-based generation because the latter handles compositional knowledge while the learned component helps select models.The rule-based component includes spatial relationships and coreference.Source: §5 · p. 4, §5.2 · p. 4, §5.3 · p. 5
5.1 Learning lexical groundings
- The learned model is trained to discriminate the described scene from four randomly sampled distractor scenes, using unigram and bigram co-occurrence features tied to object categories or model IDs.The objective is L2-regularized one-vs-all logistic regression.Source: §5.1 · p. 4
- 0.833 test discrimination accuracy with category-plus-model-ID features, versus 0.715 using model-ID features alone.The classifier is trained on a scene-discrimination task using true scenes and randomly sampled distractor scenes.Source: §5.1 · p. 4
5.2 Rule-based Model
- The rule-based parser splits and parses text, resolves coreference, and attaches noun-phrase properties for querying the 3D model database.It uses Stanford CoreNLP, filters candidate object heads with WordNet, and extracts adjectives and nouns as properties.Source: §5.2 · pp. 4–5
5.3 Combined Model
- For each noun phrase, the combined model sums learned feature weights to score categories, selects the best category above 0.5, and otherwise uses the phrase head.The category score uses weights associating features with categories.Source: §5.3 · p. 5
- For model selection, the combined system weights terms associated with the object at 0.75 and the full description at 0.25, choosing the highest-scoring positive model or omitting the object.The full-description score supplements the object-term score because rule-based parsing may miss the correct descriptive terms.Source: §5.3 · p. 5
6 Learned lexical groundings
The learned model grounds lexical terms to 3D models in the scene corpus, recovering identity mappings and variants while also producing some co-occurrence errors. Figure 5 presents qualitative scene-generation comparisons across four methods.
6 Learned lexical groundings
- The learned model recovers simple identity mappings and lexical variants such as “sofa” for Couch, but also maps “fruit” to Bowl because they commonly co-occur in the dataset.High-weight features visualize lexical-term-to-model groundings.Source: §6 · p. 5
- Figure 5 compares scenes generated for three descriptions using random, learned, rule-based, and combined methods.The examples include one Seed description and two MTurk descriptions.Source: Figure 5 · p. 6
7 Experimental Results
Human ratings show that the combined model is closest to human-created scenes for complex MTurk descriptions and outperforms the rule-based and learned models alone. ASTS scores strongly correlate with human ratings.
7 Experimental Results
- The evaluation compares random, learned, rule-based, combined, and human-created scenes on simple Seeds and more varied MTurk descriptions.The human-judgment study samples 100 MTurk descriptions from the development split.Source: §7 · p. 6, §7.2 · p. 7
7.1 Qualitative Evaluation
- In qualitative examples, the rule-based method selects a CPU chassis for “computer,” while combo and learned select a monitor; combo also correctly places newspapers on a coffee table.The learned model is limited to four objects and lacks object identity, so it often duplicates objects.Source: §7.1 · p. 6
7.2 Human Evaluation
- Participants rated scene-description match on a seven-point scale and separately rated plausibility without a reference description.The study collected 5,040 match ratings from 168 participants and plausibility ratings from 63 participants; mixed-effects analysis accounted for participant and scene variation.Source: §7.2 · p. 7
- For complex MTurk descriptions, the combined model is closest to human-created scenes and significantly outperforms the rule-based and learned models alone.All methods receive significantly lower ratings on complex MTurk sentences than on simple seed sentences.Source: Table 2 · p. 7, §7.2 · p. 7
7.3 Error Analysis
- Error cases include wrong object-category selection from incorrect head identification, duplicated desks and lamps from top-four selection, spatial-constraint failures, and duplicate tables from coreference errors.The system does not understand the grounding of “room corner”; incorrect coreference also produces two tables in the table-and-cup example.Source: §7.3 · pp. 7–8
7.4 Scene Similarity Metric
- ASTS and human ratings are strongly correlated (r = 0.70, rτ = 0.49, p < 0.001), supporting ASTS as a potential algorithmic evaluation metric.ASTS scores the best alignment between scene-template nodes and scene objects, rewarding model matches more than category-only matches and penalizing unaligned items.Source: §7.4 · p. 8
8 Future Work
Future work targets limitations in spatial-relation handling, object-category coverage, cross-language validation, and interpretation of omitted information and discourse context.
8 Future Work
- Spatial relations remain a source of scene errors, including relations such as “around,” “between,” and “on the east side” that dependency patterns do not handle.The authors propose adding spatial constraints to scene similarity and using the improved metric to train a semantic parser.Source: §8 · p. 8
- Object choice currently relies on language–object cooccurrences and sparse manually annotated categories; label propagation could expand coverage to unlabeled objects.The authors also suggest extracting novel categories from descriptions and using new labels to broaden the object types available for scene generation.Source: §8 · p. 8
- Cross-language lexical grounding remains unconfirmed: additional data collection and experiments are needed to establish its applicability and identify language-specific challenges.The approach uses neither lexical similarity features nor a manually assembled lexicon.Source: §8 · p. 9
- The paper identifies omitted information and context or discourse as unresolved factors in interpreting scene descriptions.Some descriptions use embodied perspective, such as locating a table relative to someone entering through the doors.Source: §8 · p. 9
9 Conclusion
The paper introduces a dataset of 3D scenes with natural-language descriptions and learns to ground textual descriptions to objects. It reports improved scene generation over prior-work baselines and a metric that correlates strongly with human judgments.
9 Conclusion
- The paper introduces a dataset of 3D scenes annotated with natural-language descriptions and learns object grounding from that corpus.This approach addresses prior systems’ reliance on purely rule-based mappings from object references to concrete 3D objects.Source: §9 · p. 9
- The proposed automatic metric for comparing generated scene templates to scenes correlates strongly with human judgments.The authors also collect human judgments to evaluate the effect of grounding on generated-scene quality.Source: §9 · p. 9
- The learned grounding model improves scene generation over baselines adapted from prior work.The authors demonstrate learning rich lexical grounding from an unaligned corpus of 3D scenes and natural-language descriptions.Source: §9 · p. 9