Grounded Semantic Composition for Visual Scenes

P. Gorniak; D. Roy

Source-linked AI summary

Grounded Semantic Composition for Visual Scenes

P. Gorniak, D. Roy

arXiv:1107.0031v1Jun 30, 2011cs.AI

TL;DR

The paper addresses how visually grounded word meanings and their compositions support understanding of complex spatial referring expressions. It builds a compositional parsing system from human description strategies and achieves 58.7% test-set accuracy while identifying context and grouping limitations.

Problem
The paper asks how visual features connect to language and combine into utterance meanings, including whether word meanings and composition depend on visual context.
Method
The authors study human spatial descriptions and implement grounded visual features, a grounded lexicon, robust parsing, and compositional mechanisms corresponding to common descriptive strategies.
Results
58.7% of test-set utterances received correct object selections, and the system resolved a broad range of spatial referring expressions.
Takeaways & Limitations
The analysis indicates that visual context can shape word meanings and composition, motivating contextual interpretation, backtracking, improved grouping, and interactive disambiguation.
Takeaways & Limitations
The current grouping composers are simplistic and fail on scenes requiring richer grouping criteria such as colinearity.

Abstract

from arXiv · show

We present a visually-grounded language understanding model based on a study of how people verbally describe objects in scenes. The emphasis of the model is on the combination of individual word meanings to produce meanings for complex referring expressions. The model has been implemented, and it is able to understand a broad range of spatial referring expressions. We describe our implementation of word level visually-grounded semantics and their embedding in a compositional parsing framework. The implemented system selects the correct referents in response to natural language expressions for a large percentage of test cases. In an analysis of the system's successes and failures we reveal how visual context influences the semantics of utterances and propose future extensions to the model that take such context into account.

1. Introduction

The paper develops a visually grounded model that composes word meanings to understand complex spatial referring expressions. Its evaluation succeeds on many expressions while exposing context dependence and the need for non-incremental processing.

1. Introduction: The study analyzes how speakers use visual features and linguistic devices as compositional descriptive strategies to identify unique objects.The strategies combine colour, spatial relations, and spatial grouping.
1. Introduction: The model combines visual feature extraction, a visually grounded lexicon, robust parsing, and a compositional engine for lexical groundings.The implementation was designed around the descriptive strategies found in the human study.
1. Introduction: 59% of expressions had their visual referents correctly understood, compared with 13% chance performance.The evaluation used verbatim spoken utterances, including speech repairs and other spoken-language ungrammaticalities.
1. Introduction: The results show that word meanings can depend strongly on visual context, with four distinct interpretations found for “middle”.These interpretations were linguistically indistinguishable but depended on different visual contexts.
1. Introduction: Some utterance meanings depend on visual context beyond the meanings of their parts, and composition may require backtracking rather than purely incremental processing.The current feed-forward implementation fails when later processing requires reinterpretation or ambiguity resolution.
1. Introduction: The framework provides a step toward grounded compositional semantics, although it does not yet address context-dependent interpretation and backtracking.The authors identify these issues as improvements needed to reach human-level performance.

2. A Spatial Description Task

The Bishop task elicits spatial descriptions by asking people to describe objects in computer-generated scenes with many visually similar objects.

2. A Spatial Description Task: The Bishop task uses computer-generated scenes with up to 30 objects randomly positioned on a virtual surface.Objects share identical shapes and sizes and are either green or purple.

2.1 Motivation for Task Design

The task was designed to make spatial relations and grouping useful by increasing scene clutter and reducing distinctive object attributes. Its listener-selected setup eased reference resolution, while 3D rendering introduced limited realism-related ambiguity.

2.1 Motivation for Task Design: The designers increased the number of objects and reduced distinctive attributes to encourage speakers to use spatial descriptions.The number of objects also varied to include both cluttered and sparse scenes.
2.1 Motivation for Task Design: Randomly selecting targets made descriptions difficult and unnatural because chosen objects often lacked visually salient arrangements.Listeners made more errors in this task variation.
2.1 Motivation for Task Design: Allowing listeners to choose concise, nontrivially describable objects eased the understanding task and encouraged simpler descriptions when needed.The authors acknowledge that this design affects task difficulty.
2.1 Motivation for Task Design: The 3D scenes introduced occlusion, shadows, and ambiguity, but contained no interesting 3D spatial relations.The design anticipated eventual transition to a camera-driven vision system.
2.1 Motivation for Task Design: The experiment used a speaker-listener dyad with no spoken listener feedback, so it was not designed as a collaboration study.Listener feedback was limited to successful or unsuccessful object selection.

2.2 Data Collection

The study recorded paired speaker-listener descriptions with scene and target annotations, then used development data to build the system and additional participants to assess generalization.

2.2 Data Collection: Participants worked in pairs facing identical displays, with one describing a selected object and the other listening.Descriptions were recorded by microphone while scene and selection events were collected.
2.2 Data Collection: The initial development corpus contained 268 spoken descriptions from six participants, with verbatim transcription retaining speech errors.Audio was segmented using pause structure, and scene layouts and target identities were recorded.
2.2 Data Collection: An algorithm paired utterances with object selections by collecting relevant speech between successive removal events within a four-second interval.The collected speech was fused into a single utterance for processing.
2.2 Data Collection: After analyzing the development corpus to catalogue referring strategies, the researchers collected 179 additional descriptions from three participants to evaluate generalization and coverage.The implementation was first brought to acceptable performance on the development corpus.

2.3 Descriptive Strategies for Achieving Joint Reference

Participants described scene objects using colour, spatial extrema and regions, grouping, spatial relations, and anaphoric references. The analysis distinguishes clean data from errors and uncovered strategies, while reporting how frequently these devices occurred.

2.3.1 Colour: Colour adjectives directly preceded nouns, while elliptical forms such as “the leftmost purple” occurred in 7% of the data.
2.3.1 Colour: Colour identified one or more objects in 96% of the data and 95% of the clean data.
2.3.2 Spatial Regions and Extrema: Multiple spatial specifications were implemented sequentially in input order because human choices were not always explained by an unordered conjunction of extrema.
2.3.2 Spatial Regions and Extrema: 72% of the data used single spatial extrema, while spatial regions occurred in 20% and combined extrema or regions in 28%.
2.3.3 Grouping: Grouping identified objects in 12% of the data, with 7.5% selecting objects within groups and 8.5% specifying groups by number.
2.3.6 Other: Other descriptive strategies accounted for 13% of the data, while 15% contained errors from repairs, segmentation, or scene-utterance misalignment.

2.4 Summary

A computational system for human-like spatial descriptions must combine visual scene analysis, robust parsing, grounded lexical semantics, incremental composition, and referent selection.

The system must compute visual features such as natural groupings, inter-object distances, orderings, and spatial relations.
A robust parser must discover grammatical patterns associated with descriptive strategies.
A visually grounded lexicon links words to descriptive strategies and specifies how those strategies compose.
Semantic interpretation and composition must occur within parsing rather than as a separate process.
The system must interpret complete parses and select the object best described by the utterance.
The framework consists of a visual system, grounded lexical entries, a parser, and implementations of descriptive strategies.

3. The Understanding Framework

Bishop integrates synthetic vision, grounded lexical entries, and a partial chart parser to interpret referring expressions from segmented virtual scenes.

Vision system: Bishop derives object features from segmented rendered images rather than using the scene-rendering coordinates directly.Its synthetic vision algorithm maps image pixels to objects or background and uses pixel colour information.
Vision system: The vision system computes colour, centre-of-mass, distance, grouping, pair, triplet, convex-hull, and attentional-vector features.Groups are based on distance thresholds; pairs and triplets restrict group size, while convex hulls enclose object sets.
Representation: Objects and groups receive integer IDs, with groups retaining constituent IDs so they can be treated as objects or decomposed.
Lexicon: Lexical entries encode grammatical type, compositional arity, reference behaviour, and parameters for semantic composers.Concepts carry possible referents and numeric rankings produced by composition.
Parser: Context-free grammar rules support local compositional constraints, while a bottom-up chart parser incrementally reuses computed subtrees.Partial parsing retains the longest referring segments and allows unknown words to remain invisible to the understanding process.
Post-parse filtering: The filtering algorithm diagnoses inconsistencies such as ambiguity, contradiction, and descriptions with no fulfilling object.

4. Semantic Composition

Bishop composes visually grounded concepts incrementally, using specialized composers for attributes, extrema, regions, groups, spatial relations, and anaphora.

Composition framework: Composers accept concepts representing possible referents with ranked strengths and produce concepts that may refer to different object sets.Concepts can also be marked as non-referring, single-object, or group references to constrain composition.
Attributes: The probabilistic attribute composer assigns each candidate object a density based on its measured average colour.
Extrema and regions: Ordering composers select extrema or absolute spatial regions using exponentially weighted reference strengths.The extremum version uses γ = 0.38 in the reported results, while the absolute-position version supports terms such as “middle”.
Grouping: Grouping composers find nearby object clusters, enforce cardinality for numbered groups, and rank groups by within-group distance.They use a hand-set distance threshold and can create composite objects from group convex hulls.
Grouping: The “of” composer can split group concepts so expressions select an object within a selected group.
Spatial and anaphoric relations: The spatial composer evaluates AVS relations between possible landmarks and targets, using a lexical reference vector and distance normalization.The anaphoric composer instead anchors a concept to the last object removed from the previous visual scene.

5. Example: Understanding a Description

The worked example shows Bishop combining lexical alternatives and compositional operations to interpret “the purple one on the left” as a referring expression.

Lexical analysis: The example begins with the parser finding lexical entries for “the” and two senses of “purple”.
Colour composition: Colour composition narrows the candidate set to purple objects, while article composition marks the phrase as referring to an object.
Spatial composition: The ordering composer interprets “left” along the x axis and selects the leftmost purple objects before later single-object marking.The chart denotes this operation as O.x.min(NP).

6. Results and Discussion

The system performs well across several spatial referring-expression strategies, but errors arise when interpretation depends on context, grouping, parsing, or composition order. Detailed analyses identify both strong components and concrete extensions needed for broader coverage.

Overall performance: The system understands the vast majority of targeted utterances and significantly outperforms a uniform random-selection baseline.Figure 11 reports mean performance for complete and clean development and test corpora, with error bars showing one standard deviation.
Overall performance: A feedforward filtering interpretation can fail when intermediate composers exclude target objects or retain too many candidates.The authors identify backtracking and composers sensitive to broader visual context as the real solution beyond threshold adjustment.
Spatial regions and extrema: 100% accuracy was achieved for descriptions using only colour and a single spatial extremum.This strategy appeared in 38% of the clean data, supporting the interpretation that participants order objects along a spatial dimension and select an extreme candidate.
Spatial regions and extrema: Meanings of “middle” were linguistically indistinguishable but depended on visual, historical, and candidate-related context.The model’s fixed treatment of “middle” as a region and “left” or “top” as extrema does not capture these subtler conditions.
Spatial regions and extrema: 85% of cases containing one or more spatial region or extremum references were correctly interpreted, although errors reflected overcommitment and faulty ordering.These references occurred in 30% of the clean data, and the preferred composer order could conflict with an individual example.
Grouping: Grouping composers failed when target groups were not visually isolated or when alignment and competing groups required richer visual criteria.The current algorithm uses distance thresholding but does not incorporate grouping hints such as colinearity or alignment with the vertical axis.
Spatial relations: Spatial-relation errors occurred when grouping or region composers failed to identify landmarks or targets, while the relation composer succeeded when those referents were correct.Distance-based and combined spatial relations were outside the implementation’s coverage and were left for future work.
Anaphora: Single-object anaphora was resolved perfectly in the clean data, but references back to groups of objects were not covered.Because spatial relations also performed well, all development-data anaphora cases were covered.

7. Future Directions

Future work targets limitations in Bishop’s spatial reasoning, including restricted composers, no backtracking, weak context sensitivity, and manually constructed lexical models.

7. Future Directions: Future implementations should cover additional descriptive strategies, including connectivity, symmetry, numbered grouping, and in-group numbering.Proposed mechanisms include local connectivity graphs, mirrored landmark positions, higher-number grouping, and ordering distributions with arbitrary peaks.
7. Future Directions: The system’s spatial and grouping composers simplify problems that have more sophisticated solutions in prior work.The paper identifies individual composers as hard problems outside its main emphasis, with improvements expected to improve the overall system.
7. Future Directions: Backtracking should revise interpretation decisions when parsing fails to produce a referent.The proposed revision process would loosen decisions made at earlier interpretation stages until a referent is produced.
7. Future Directions: Future composition should incorporate local and global visual context because human interpretations can depend on scene structure rather than only absolute measurements.For example, “middle” may select an isolated object between left and right groups even when another object is closer to the screen centre.
7. Future Directions: The lexical treatment should move beyond isolated word-like entries toward phrase-level entries for expressions such as “to the left of”.The paper notes that the meaning of “of” can require grammar-specific handling and suggests treating whole spatial phrases as lexical entries.
7. Future Directions: Future systems could use semantic fragments for clarification dialogue, while machine learning could replace hand-set parameters and help construct new composers.The paper also describes applying Bishop to voice-based referent resolution in Blender and plans an interactive conversational robot.

8. Summary

The paper combines visually grounded lexical items, robust parsing, and semantic composition to resolve spatial references in spoken language. Bishop handled ill-formed speech and selected correct objects on both development and test data, while motivating several extensions.

8. Summary: Bishop grounds lexical items in visual features and grouping properties, then uses parsing and semantic composers to determine joint reference.The parser identifies syntactically coherent chunks and activates composers that combine word meanings into phrase-level semantics.
8. Summary: The robust parser processes grammatically ill-formed spoken transcripts while supporting compositional spatial-language interpretation.The evaluation used natural spoken utterances containing speech and processing irregularities.
8. Summary: 76.5% of development expressions and 58.7% of test expressions received correct object selections.Performance was higher on clean datasets with speech and processing errors held out.
8. Summary: The paper proposes better spatial grouping, semantically guided backtracking, learned models, and interactive dialogue as routes to improve performance.It also identifies transplanting Bishop into an interactive conversational robot as a near-term direction.

Appendix A. Utterances in the Test Data Set

Appendix A lists the 179 spoken utterances used as test data, preserving their original order and the speech-segmentation and stitching errors presented to the system.

Appendix A. Utterances in the Test Data Set: The test set contains 179 utterances presented in the order received by the understanding system.The utterances are transcribed examples from the spatial description task.
Appendix A. Utterances in the Test Data Set: The test inputs retain errors from faulty speech segmentation and algorithmic stitching of oversegmented utterances.The appendix therefore reflects the noisy spoken-language input used in evaluation.
Appendix A. Utterances in the Test Data Set: The utterances use color, depth, position, grouping, ordering, and relational descriptions to identify cones and other scene objects.Examples include frontmost, backmost, leftmost, middle, closest, isolated, paired, and between-object descriptions.

Appendix B

Appendix B documents the complete XML lexicon used in Bishop and directs readers to the online appendix file containing it.

Appendix B: The complete Bishop lexicon is specified in XML, with an initial comment describing lexical-entry attributes.The online appendix provides the file lexicon.xml.