Source-linked AI summary
A Joint Model of Language and Perception for Grounded Attribute Learning
Cynthia Matuszek, Nicholas FitzGerald, Luke Zettlemoyer, Liefeng Bo, Dieter Fox
TL;DR
The paper asks how robots can learn grounded language and perception for selecting objects from physical scenes with limited supervision. It jointly models semantic parsing and visual attributes, aligning latent meanings with classifiers through online EM-like learning. Experiments show accurate object-selection models and effective induction of novel attributes and synonyms, with performance depending on sufficient initialization data.
Problem
Robots need a joint language–vision model that maps descriptions to referred objects while learning new attribute meanings with little guidance.
Method
The approach jointly learns probabilistic compositional language representations and visual attribute classifiers from sentences, raw percepts, and target objects using online EM-like updates.
Results
The system accurately performs object set selection, extends grounded concepts, identifies novel words as color, shape, or non-attributes, and learns synonyms.
Takeaways & Limitations
Joint language–perception learning provides a simple mechanism for acquiring vocabulary in a physical environment from weakly supervised scenes.
Takeaways & Limitations
With fewer than 150 sentences, the learned grammar lacks sufficient coverage for modeling unknown words during joint learning.
Abstract
from arXiv · showhide
As robots become more ubiquitous and capable, it becomes ever more important to enable untrained users to easily interact with them. Recently, this has led to study of the language grounding problem, where the goal is to extract representations of the meanings of natural language tied to perception and actuation in the physical world. In this paper, we present an approach for joint learning of language and perception models for grounded attribute induction. Our perception model includes attribute classifiers, for example to detect object color and shape, and the language model is based on a probabilistic categorial grammar that enables the construction of rich, compositional meaning representations. The approach is evaluated on the task of interpreting sentences that describe sets of objects in a physical workspace. We demonstrate accurate task performance and effective latent-variable concept induction in physical grounded scenes.
1. Introduction
The paper addresses how robots can jointly learn language and visual meanings from lightly supervised physical scenes. It evaluates this approach on object descriptions and finds effective grounded concept extension.
- Robots must jointly reason over language and vision while inducing rich associations with minimal guidance.
- Grounded object selection requires mapping words such as “yellow” and “blocks” to perceptual attributes that identify the referred objects.
- The approach jointly learns visual classifiers, word meanings grounded in those classifiers, and compositional sentence semantics.
- Learning uses scenes containing sentences, images, and referred-object indications, with online EM-like updates and no explicit logical-form or classifier-output labels.
- Experiments on people describing object sets show that joint learning extends grounded concepts in an initially incomplete model.
2. Overview of the Approach
The model maps sentences and segmented scene objects to selected object sets by combining semantic parsing, visual classifiers, alignment, and uncertain execution. It learns new grounded words from sentence–scene–selection examples using online EM-like updates.
- Problem: The task is to map a natural-language sentence x and scene objects O to the described subset G ⊆ O.
- Model Components: A semantic parser assigns probabilities P(z|x) to logical meaning representations, such as λx.color(x, yellow).
- Model Components: Visual attribute classifiers assign probabilities that each classifier is true for each segmented object, using color and shape features from Kinect depth-camera segments.
- Joint Model: An alignment model pairs logical constants with visual classifiers so meanings such as yellow can be grounded in perception.
- Joint Model: An execution model combines parser uncertainty, classifier confidence, and a groundtruth constraint to determine which objects a logical expression selects.
- Model Learning: Online EM-like learning estimates latent logical forms and classifier outputs before updating language and visual models, after supervised bootstrapping.
3. Related Work
The paper combines established work in visual attributes, semantic parsing, and grounded learning while targeting joint induction of novel grounded language–perception concepts. Related grounded approaches generally use predefined language formalisms rather than extending them to entirely novel input.
- The paper claims to be the first approach jointly learning visual classifiers and semantic parsers into rich models spanning sensors to meaning.
- Vision: Visual recognition research uses local image and 3D descriptors, while visual attributes provide richer object descriptions.
- Semantic Parsing: Semantic parsing includes supervised CCG learning, alternate supervision from database answers, and unsupervised learning.
- Grounding: Grounded-learning research has addressed grounding words in visual scenes and learning probabilistic language models with visual components.
- Grounding: Closely related grounded approaches map language into predefined formalisms rather than extending models for entirely novel input.
4. Background on Semantic Parsing
The language component builds on FUBL, which learns factored CCG lexicons and log-linear parse probabilities. The paper initializes this parser and then induces lexemes linked to new visual attributes.
- FUBL learns factored CCG lexicons consisting of lexemes and lexical templates that produce logical forms from sentences.
- A lexeme combines with a template to form a lexical item, illustrated by pairing “half-pipe” with the logical constant arch.
- FUBL assigns probabilities to parses yielding logical form z given sentence x with a log-linear model.
- The feature vector φ(x, y, z) represents lexemes and lexical templates used to generate a parse, among other features.
- The paper initializes its parse model with FUBL, then automatically induces lexemes paired with new visual attributes absent from the initial training set.
5. Joint Language/Perception Model
The joint model combines latent semantic parsing, perceptual classifier outputs, and grounded execution to infer which scene objects a sentence names. Language and vision are coupled through agreement between logical forms and selected objects.
- Joint probabilistic model: The model factors joint probability into language, vision, and grounded execution: P(G, z, w | x, O) = P(z | x)P(w | O)P(G | z, w).The latent logical form z captures linguistic uncertainty, while perceptual assignment w captures uncertainty about classifier outputs.
- Grounded execution: The agreement term P(G|z, w) selects objects by applying a logical form to classifier assignments and requiring the resulting set to match G.For example, conjunctions of shape and color predicates select objects whose corresponding classifiers are true.
- Perceptual model: Perceptual worlds are modeled by assuming each classifier acts independently, so P(w|O) decomposes into probabilities of individual classifier assignments.Each classifier is a logistic regression model over object features.
- Inference: Training marginalizes over latent logical forms and perceptual assignments, while test-time inference chooses arg maxG P(G|x, O).Marginal computation involves checking which latent world/logical-form pairs name G and is formulated as weighted model counting under the general case.
6. Model Learning
Learning extends a partially supervised language-and-perception model by inducing new word–classifier alignments and optimizing parameters with latent-variable expectations. The online procedure is efficient and works well for the considered object-selection task, although convergence does not guarantee an optimal solution.
- Learning setup: The learner starts from a partial CCG parser and a small set of attribute classifiers, then aims to induce new classifiers tied to previously unseen words.The target model is P(G|x, O), trained from sentences, scenes, and selected object sets without labeling every latent structure.
- Aligning Words to Classifiers: New attribute classifiers are initialized uniformly, paired with new logical constants, and aligned exhaustively with previously unknown sentence words.This procedure creates candidate grounded concepts from the available training examples.
- Parameter Estimation: The marginal log-likelihood objective is non-convex because it sums over latent logical forms z and classifier outputs w.With labeled latent variables, the problem reduces to training the semantic parser and attribute classifiers as log-linear models.
- Parameter Estimation: An online EM-style algorithm estimates latent-variable marginals and updates language and perception parameters using expected gradients one example at a time.The implementation uses beam search for the top-N parses because enumerating all parses is exponential in sentence length.
- Discussion: The decayed online learning rate guarantees convergence, but the authors state that little can be said about the optimality of the resulting solution.They nevertheless report that the approach works well in practice for the object set selection task considered.
- Bootstrapping: A small supervised dataset initializes the language and perceptual models, while future work would explore replacing it with interactive human-teacher dialogue.The supervised data labels logical forms and classifier outputs, and need not contain most words or attributes learned by the full approach.
7. Experimental Setup
The experiments use toy-object scenes with RGB-D video, human gestures, and Mechanical Turk descriptions of referenced object sets. The setup covers shape and color attributes and illustrates both single-attribute and compositional language–meaning pairs.
- Data Set: Scenes contain wooden blocks, plastic food, and building bricks recorded as short RGB-D videos showing a person gesturing toward a subset of objects.Natural-language annotations were collected by asking Mechanical Turk workers to describe the objects indicated in each video.
- Data Set: Figure 3 contrasts descriptions of red object sets with a compositional example combining orange color and cube shape.The two red descriptions share λx.color(x, red), while the compositional example uses λx.color(x, orange) ∧shape(x, cube).
- Data Set: The dataset contains 142 scenes and 1003 sentence/annotation pairs describing 12 attributes divided evenly between shapes and colors.Referenced objects were marked as the positive set G for each scene.
- Perceptual Features: Objects are segmented by fitting the table plane with RANSAC and extracting connected components above it, then represented with kernel descriptors for depth and RGB.Depth features correspond to shape attributes, while RGB features correspond to color attributes.
- Language Features: The language model uses binary indicators for lexical entries and logical-form properties, allowing lexical selection to be weighted against compositional and visual evidence.These features define the log-linear model P(z|x; ΘL).
8. Results
The joint model performs well on selecting described object sets and learning grounded concepts, while sharply limited language or vision components substantially reduce performance. Learned classifiers and lexeme associations support concept induction, synonym learning, and stable performance with sufficient initialization data.
- Object Set Selection: 82% average precision, 71% recall, and 76% F1-score were achieved on object set selection across 10 randomized training runs.The evaluation used unseen scenes and unknown words for new attributes, with only the referred object sets labeled.
- Ablation Studies: The vision-only ablation achieved Precision=0.92, Recall=0.41, and F1-score=0.55, notably below the jointly trained model.This baseline used synonym sets and separately trained classifiers with a simple language model.
- Ablation Studies: The perception-ablated parsing baseline achieved Precision=0.52, Recall=0.09, and F1-score=0.14.Unknown concept words could be skipped, but the parser had no way to determine their meanings, causing many parses to select no objects.
- Learned Models: Newly learned color and shape classifiers reached average accuracies of 97% and 74%, although cube and cylinder were sometimes difficult to differentiate.Classifier quality affects the system’s ability to perform set selection.
- Learned Models: Novel attributes were most strongly associated with newly created classifiers, while irrelevant words such as “thing” tended to map to null.The parser must identify the classifier type appropriate for each novel word.
- Learned Models: The approach easily learned lexemes pairing new synonymous words with appropriate classifiers, with performance comparable to the main experiments.The synonym test used attributes known during initialization but described with new words.
- Data Requirements: With fewer than 150 initialization sentences, grammar coverage appeared insufficient for modeling unknown words; beyond that, performance was quite stable.The reduced-data result was illustrated using one particular split whose F1-score peaked at roughly 73%.
9. Conclusion
The joint model learns language and attribute representations grounded in physical perception, including whether novel words denote color, shape, or no attributes. Its modular design is intended to support future scaling to more advanced classifiers, richer language, and more complex physical scenes.
- The system jointly learns accurate language and attribute models from language, raw percepts, and target objects for object-set selection.
- Joint learning identifies whether novel words are color attributes, shape attributes, or no attributes at all.
- The modular framework was designed to incorporate future advances in visual classification and semantic parsing.
- The authors are working to scale language and physical-scene complexity toward learning in completely unconstrained environments.