Source-linked AI summary
Modeling Context Between Objects for Referring Expression Understanding
Varun K. Nagaraja, Vlad I. Morariu, Larry S. Davis
TL;DR
Referring-expression comprehension must identify an object using both its properties and relationships to other objects, despite missing context-object annotations. The paper uses an LSTM over region–context-region pairs and trains it with max-margin multiple-instance learning. On Google RefExp and UNC RefExp, modeling object context outperforms the cited max-margin model, while qualitative results ground both the referred and supporting context regions.
Problem
Referring expressions use relationships between an object and other regions, but training typically annotates only the referred object, leaving context relationships weakly supervised.
Method
An LSTM scores referring expressions from region–context-region pairs, while max-margin multiple-instance-learning objectives discover context regions without their bounding-box annotations.
Results
The technique performs better than Mao et al.’s max-margin model on Google RefExp and UNC RefExp and qualitatively identifies referred regions with supporting context regions.
Takeaways & Limitations
Modeling context between objects supports referring-expression comprehension and can resolve ambiguity among multiple instances or infer relations when context objects are not explicitly mentioned.
Takeaways & Limitations
On UNC RefExp TestA, adding context did not improve performance because terse people-only expressions may not uniquely refer to a region.
Abstract
from arXiv · showhide
Referring expressions usually describe an object using properties of the object and relationships of the object with other objects. We propose a technique that integrates context between objects to understand referring expressions. Our approach uses an LSTM to learn the probability of a referring expression, with input features from a region and a context region. The context regions are discovered using multiple-instance learning (MIL) since annotations for context objects are generally not available for training. We utilize max-margin based MIL objective functions for training the LSTM. Experiments on the Google RefExp and UNC RefExp datasets show that modeling context between objects provides better performance than modeling only object properties. We also qualitatively show that our technique can ground a referring expression to its referred region along with the supporting context region.
1 Introduction
Referring expressions localize an image region by combining object properties with relationships to other regions. This work models those relationships despite training annotations typically identifying only the referred object.
- 1 Introduction: Referring expressions can encode category, attributes, pose, action, and scene context to identify queried objects.The comprehension task localizes a region from a referring expression; this work focuses on comprehension.
- 1 Introduction: Training usually annotates only the referred object, making relationships between regions difficult to learn directly.Such supervision can support object attributes but not reliably identify context objects.
- 1 Introduction: Context is especially useful when multiple instances share an object type, because relationships distinguish the intended instance.Examples include spatial relations such as “above” and interactions such as “holding.”
- 1 Introduction: The proposed method maps a referring expression to a region and supporting context region using weak supervision because context bounding boxes are unavailable.It learns relationships between regions rather than ignoring them as previous approaches generally do.
- 1 Introduction: An LSTM scores referring expressions for region–context-region pairs, with image context included and pair scores pooled using max or noisy-or.The highest-scoring region is selected as the referred region.
2 Related Work
Prior referring-expression work largely grounds expressions using single-region properties or image-level context. This paper extends max-margin LSTM training with multiple-instance learning to model context between objects.
- 2 Related Work: Related work includes weakly supervised visual grounding because caption datasets often lack phrase-to-bounding-box mappings.Multiple-instance learning is used in related captioning work to learn region correspondences.
- 2 Related Work: Attention models weigh image regions during caption generation, but they are not well suited to localizing a single region for a referring expression.Other approaches use two-stage grounding or supervised scene representations.
- 2 Related Work: Most referring-expression methods model object properties and image-level context while ignoring relationships between regions.Rule-based generation is restricted in the properties it can model.
- 2 Related Work: The paper extends max-margin LSTM training to multiple-instance-learning objectives for referring-expression comprehension.The objectives address context-region learning when context annotations are unavailable.
3 Modeling context between objects
The method models referring expressions using both a candidate region and supporting context regions, while learning context relationships from weak supervision. It combines LSTM scoring, context pooling, and multiple-instance max-margin training to select the referred region.
- Weakly supervised context: Context relationships are learned weakly because training annotations typically identify only the referred object, not its context objects.The approach addresses this missing supervision through multiple-instance learning.
- Context-aware region scoring: The model scores region–context pairs with an LSTM using CNN, bounding-box, and word-embedding features.The entire image is included as a possible context region, and LSTMs share weights.
- Context pooling: For each candidate region, probabilities across possible context regions are pooled with either max or noisy-or, and the highest-scoring region is selected.Noisy-or is described as more robust to noise than max pooling.
- MIL bag construction: MIL constructs positive bags from pairs whose first region is the referred object and negative bags from pairs whose first region is another candidate.Positive bags contain (Rt, Ri) pairs, while negative bags contain (Ri, Rj) pairs.
- MIL training objectives: Max-margin MIL objectives train the LSTM, with MI-SVM-style variants differing in whether only selected positive instances or latent labels for all positive-bag instances are used.The latent-label formulation assumes one positive pair and assigns negative labels to the remaining positive-bag pairs.
- Dataset-dependent assumptions: The one-positive-pair assumption fits uniquely identifying expressions in Google RefExp, whereas UNC RefExp also contains expressions that do not uniquely identify an object with its context.The paper uses two formulations to accommodate these dataset characteristics.
4 Experiments
Experiments evaluate MIL-based context modeling on Google RefExp and UNC RefExp, showing that performance depends on dataset characteristics, loss design, and context pooling. Qualitative results show improved localization through supporting context, while terse and ambiguous expressions remain challenging.
- Experimental setup: Experiments use Google RefExp and UNC RefExp, evaluating referred-region localization with Precision@1 based on IoU >0.5.The datasets contain referring expressions for MS-COCO images, and evaluation averages true-positive predictions across expressions.
- Comparison of techniques: MIL loss functions outperform Mao et al.’s max-margin technique on both validation partitions and UNC RefExp TestB, but improve only slightly on TestA.The comparison includes ground-truth and MCG proposals to assess behavior with and without proposal false positives.
- Loss functions: Google RefExp favors positive-and-negative bag margins because its context-referencing expressions usually identify a unique object-context pair.This uniqueness satisfies the assumption underlying a single positive instance in the positive bag.
- Loss functions: UNC RefExp favors or matches negative-only bag margins because context objects are often implicit, so the single-positive-instance assumption does not always hold.The elephant example illustrates context that is implied rather than explicitly mentioned.
- Context pooling: Noisy-or pooling performs best on every partition except UNC RefExp TestA and is more robust than max pooling.Image-only context also beats Mao et al.’s image-context max-margin model because MIL training mines negative context samples.
- Qualitative results: Supporting context helps localize unique instances and resolve ambiguity among multiple same-category objects, including cases where the context object is implicit.The method evaluates region likelihood relative to another region, enabling relational distinctions such as which object is farther left.
- Qualitative results: Adding context does not improve UNC RefExp TestA, whose terse person-centric expressions often fail to uniquely identify a region or explicitly mention the person.This partition forms the clearest scope boundary for the observed gains.
5 Conclusions
The paper models referring-expression probabilities from object regions and supporting context regions using an LSTM trained with multiple-instance learning. Results on Google RefExp and UNC RefExp outperform Mao et al.’s max-margin model, while qualitative examples identify both referred and supporting regions.
- 5 Conclusions: The method models a referring expression as a function of a region and its context region using an LSTM.Multiple-instance learning handles the lack of context-object annotations during training.
- 5 Conclusions: Multiple-instance learning objective functions enable LSTM training when context-object annotations are unavailable.The formulations are conceptually similar to MISVM and mi-SVM.
- 5 Conclusions: Results on Google RefExp and UNC RefExp show better performance than Mao et al.’s max-margin model.Qualitative results also identify a referred region together with its supporting context region.