Source-linked AI summary
Interactive Visual Grounding of Referring Expressions for Human-Robot Interaction
Mohit Shridhar, David Hsu
TL;DR
INGRESS addresses grounding natural-language referring expressions for robots handling varied objects and ambiguous instructions. It uses generated visual descriptions in a two-stage neural model, then asks object-specific questions when needed; experiments report improved performance against UMD Refexp and faster task completion through disambiguation.
Problem
Grounding referring expressions requires locating objects from images and language while supporting unconstrained object categories, free-form expressions, and ambiguity.
Method
INGRESS generates descriptions of regions and region pairs, compares them with the input expression in two grounding stages, and reuses the networks to generate object-specific questions.
Results
INGRESS outperformed UMD Refexp on RefCOCO and in human robot experiments, while interactive object-specific questions sped task completion by 1.6 times on average.
Takeaways & Limitations
The system supports grounding across varied objects and unconstrained language while enabling interactive disambiguation during robot manipulation.
Takeaways & Limitations
INGRESS handles only binary relations, relies on keyword matching for perspectives, and uses hard-coded clustering components.
Abstract
from arXiv · showhide
This paper presents INGRESS, a robot system that follows human natural language instructions to pick and place everyday objects. The core issue here is the grounding of referring expressions: infer objects and their relationships from input images and language expressions. INGRESS allows for unconstrained object categories and unconstrained language expressions. Further, it asks questions to disambiguate referring expressions interactively. To achieve these, we take the approach of grounding by generation and propose a two-stage neural network model for grounding. The first stage uses a neural network to generate visual descriptions of objects, compares them with the input language expression, and identifies a set of candidate objects. The second stage uses another neural network to examine all pairwise relations between the candidates and infers the most likely referred object. The same neural networks are used for both grounding and question generation for disambiguation. Experiments show that INGRESS outperformed a state-of-the-art method on the RefCOCO dataset and in robot experiments with humans.
I. INTRODUCTION
INGRESS targets natural-language grounding for robots operating on varied everyday objects, including ambiguous expressions that require interactive clarification. Its approach combines generated visual descriptions, compositional grounding, and object-specific questions.
- The system grounds referring expressions by locating objects from input images and language expressions.
- INGRESS supports unconstrained object categories and free-form language expressions, including objects not previously seen.
- When referring expressions are ambiguous, the robot asks interactive questions to disambiguate the intended object.
- The method uses grounding by generation: neural networks generate language from images and compare it with the input expression.
- INGRESS decomposes grounding into self-referential and relational stages, first identifying candidate objects and then examining relations among them.
- Experiments report that INGRESS outperformed UMD Refexp on RefCOCO and in robot experiments with humans.
III. INTERACTIVE VISUAL GROUNDING
INGRESS uses grounding by generation in two sequential stages and reuses the same LSTM-based models to produce object-specific clarification questions. This design links object selection and interactive disambiguation through generated descriptions.
- INGRESS sequentially trains separate S-LSTM and R-LSTM networks for self-referential and relational expressions.
- Each stage generates a textual description of an image region or region pair and compares it with the input expression to identify the referred object.
- Grounding by generation enables the networks to generate clarification questions when the referring expression remains ambiguous.
- For disambiguation, INGRESS describes each candidate object, fits the description into a “Do you mean ...?” template, and lets the user correct the choice.
B. Grounding Self-Referential Expressions
The self-referential stage scores proposed image regions by how well their generated descriptions match the input expression. It uses region proposals, LSTM-generated word distributions, and language-similarity measures before clustering candidates.
- The first stage identifies candidate image regions from an image and self-referential expression by maximizing the probability of the expression given a region.
- A Faster R-CNN localization module produces object proposals, and a fully connected layer converts each proposal into a 4096-dimensional feature vector.
- Fig. 3 illustrates relevancy clustering, which separates relevant and irrelevant object proposals before candidates continue to relational grounding.
- S-LSTM processes each region feature to generate a word-probability sequence describing the region, which is compared with the input expression using average cross-entropy loss.
- The implementation uses a DenseCap captioning network trained on Visual Genome, containing around 100,000 images and 4,300,000 expressions.
C. Relevancy Clustering
INGRESS combines language-generation similarity measures to cluster candidate regions, improving robustness to visual ambiguity and lexical variation.
- C. Relevancy Clustering: CEL compares the input expression with generated word-probability vectors but can be misled by lighting, sensor noise, detection failures, and visual similarity.A partially visible Pringles can may resemble a red cup, producing an incorrectly low CEL.
- C. Relevancy Clustering: METEOR adds normalized semantic similarity and automatically handles paraphrases and synonyms.It compares the generated expression with the input expression, but treats all words equally and ignores visual context.
- C. Relevancy Clustering: INGRESS computes both CEL and METEOR, then applies K-means with K = 2 to separate relevant from irrelevant candidate image regions.The relevant cluster is passed to the second grounding stage when it contains multiple candidates.
D. Grounding Relational Expressions
For relational expressions, INGRESS assumes binary relations and scores pairwise candidate regions with a relation-generating LSTM before interactive disambiguation resolves remaining ambiguity.
- D. Grounding Relational Expressions: The relational stage assumes binary relations, including cases such as “the leftmost cup” represented against all other objects.The paper characterizes binary relations as common in everyday expressions.
- D. Grounding Relational Expressions: INGRESS forms pairwise permutations of candidate regions, including the whole image, and feeds them to R-LSTM to predict relational expressions.This directly connects image-region pairs with language instead of relying on predefined relation templates.
- D. Grounding Relational Expressions: R-LSTM is trained on RefCOCO, which contains around 19,000 images and 85,000 referring expressions describing visual relations.Training uses the Multi-Instance Learning Negative Bag Margin loss with stochastic gradient descent.
- D. Grounding Relational Expressions: When grounding leaves multiple candidates, INGRESS asks “Do you mean ...?” questions while pointing to each candidate object.Users can answer yes, no, or provide a correcting expression that triggers rerunning the system on the identified candidates.
- D. Grounding Relational Expressions: The system generates candidate-specific expressions with S-LSTM or R-LSTM and selects informative questions using METEOR similarity.It begins with self-referential expressions because most referring expressions primarily rely on visual information.
IV. SYSTEM IMPLEMENTATION
INGRESS combines RGB-D perception, neural grounding, voice input, and manipulation planning to execute pickup and placement actions on a robot arm.
- IV. SYSTEM IMPLEMENTATION: The system was implemented on a robot manipulator with voice input and RGB-D sensing.The implementation setup is introduced as the basis for evaluating the approach.
- IV. SYSTEM IMPLEMENTATION: The grounding model maps an RGB image and textual referring expression to a 2D bounding box containing the referred object.A Kinect2 camera supplies visual perception and an Amazon Echo Dot supplies voice-based input.
- IV. SYSTEM IMPLEMENTATION: Object detection uses a nonmaximum suppression threshold of 0.7 and a final output threshold of 0.05 for minimal overlap in uncluttered scenes.
- IV. SYSTEM IMPLEMENTATION: S-LSTM and R-LSTM use vocabulary sizes of 10,497 and 2,020, respectively, with a maximum sequence length of 15 words.
- IV. SYSTEM IMPLEMENTATION: The 6-DOF Kinova MICO arm uses depth data to localize selected objects in 3D, plan grasp poses and paths, and identify placement locations.Its two high-level actions are PICKUP and PUTIT.
D. Software and Hardware Platform
The platform integrates visual perception, grounding, perspective handling, and manipulation planning, while experiments evaluate accuracy, generalization, and interactive disambiguation.
- D. Software and Hardware Platform: The complete system runs under ROS on an Intel i7 Quad Core PC with an NVIDIA Titan X GPU, with grounding executed on the GPU.
- D. Software and Hardware Platform: Object-centric perspective expressions are handled directly, while user-centric and robot-centric expressions receive limited treatment for simple common cases.
- D. Software and Hardware Platform: Perspective correction computes a 3D centroid, projects it onto the selected viewpoint’s image plane, and scales the box linearly with viewpoint distance.The original aspect ratio is maintained.
- D. Software and Hardware Platform: The evaluation compares UMD Refexp and INGRESS on RefCOCO using both human-annotated ground-truth and automatically generated MCG object proposals.
- D. Software and Hardware Platform: Across RefCOCO, human-robot experiments, and disambiguation tests, INGRESS outperformed UMD RefExp and sped task completion by 1.6 times on average.The overall voice-to-action cycle takes 2–5 seconds, while grounding takes approximately 0.15 seconds.
A. RefCOCO Benchmark
The RefCOCO benchmark evaluates grounding unambiguous referring expressions across varied objects and relations. INGRESS outperformed UMD Refexp, although gains were small because scenes contained few proposals.
- The dataset tests generalization across self-referential and relational expressions and unconstrained object categories.
- RefCOCO measures whether predicted image regions overlap ground-truth regions by more than 0.5 IoU.
- INGRESS outperformed UMD Refexp in most RefCOCO cases, but improvements were small.
- The two-stage grounding process offered limited benefit because validation and test images averaged only 10.2 ground-truth or 7.4 MCG proposals.
- Images containing people showed greater accuracy improvement than images containing only objects, likely reflecting Visual Genome’s dataset bias.
C. Disambiguation
INGRESS interactively resolves ambiguous referring expressions with object-specific questions and user corrections. It required fewer questions than the baseline and was rated more effective at communicating needed information.
- INGRESS asks object-specific questions and allows users to correct ambiguous expressions with additional descriptive information.
- Across 160 trials, participants eventually corrected the robot to identify the required object in every trial.
- INGRESS averaged 1.16 disambiguation questions versus 1.91 for the baseline, a statistically significant difference.
- Participants rated INGRESS higher than the baseline for communicating required disambiguation information, 4.4 versus 1.6.
- Approximately 79% of correcting responses mirrored the language used in the robot’s questions.
D. Examples
INGRESS produces interactive grounding results that include both successful disambiguation and recurring failures. Its failures arise especially with clutter, occlusion, counting, grouping, and ambiguous expressions.
- INGRESS generates clear, discriminative disambiguation questions in many successful grounding cases.The examples include successful results alongside generated questions.
- The system supports user-centric and robot-centric perspective corrections, including intuitive expressions such as “my left”.
- INGRESS struggles with cluttered environments and partially occluded objects, which can produce false positives.
- Complex relationships requiring counting or grouping, such as “third” or “all four,” remain unsupported.
- When a generated question is nondiscriminative, the robot’s pointing gesture can still help identify the intended object after two questions.
VI. DISCUSSION
The discussion identifies limits in INGRESS’s relational reasoning, perspective handling, grouping components, and robustness to clutter and occlusion. The conclusion presents the system as a step toward shared human-robot understanding while pointing to grounding verbs as future work.
- Limitations: INGRESS handles only binary relations and does not readily scale to tertiary or more complex relationships.
- Limitations: The system relies on keyword matching for perspectives, hard-coded clustering components, and cannot handle cluttered scenes with partially occluded objects.
- Possible improvements: Relational networks, perspective-bearing training expressions, learned clustering modules, non-verbal cues, and moving objects are proposed as possible improvements.
- Conclusion: INGRESS is presented as a step toward shared human-robot understanding, with grounding verbs identified as a distinct direction for expanding robot actions.