Source-linked AI summary
Neighbourhood Watch: Referring Expression Comprehension via Language-guided Graph Attention Networks
Peng Wang, Qi Wu, Jiewei Cao, Chunhua Shen, Lianli Gao, Anton van den Hengel
TL;DR
Referring expression comprehension must localise an image object from language while handling expressions that describe relationships with neighbouring objects. The paper proposes language-guided graph attention over object nodes and edges, and reports stronger performance across three datasets with visualisable, explainable decisions.
Problem
Referring expression comprehension requires a discriminative region representation that adapts to language and captures relationships needed by complex expressions.
Method
LGRAN builds a directed object graph and applies language-guided node attention and intra- and inter-class edge attention to enrich referent representations.
Results
The method outperforms previous best results on almost all splits across RefCOCO, RefCOCO+ and RefCOCOg under different settings.
Takeaways & Limitations
The attended graph supports more expression-adaptive object representations and makes the comprehension decision visualisable and explainable.
Takeaways & Limitations
The candidate object set is either ground truth or generated by an object proposal method, so evaluation covers both candidate-generation settings.
Abstract
from arXiv · showhide
The task in referring expression comprehension is to localise the object instance in an image described by a referring expression phrased in natural language. As a language-to-vision matching task, the key to this problem is to learn a discriminative object feature that can adapt to the expression used. To avoid ambiguity, the expression normally tends to describe not only the properties of the referent itself, but also its relationships to its neighbourhood. To capture and exploit this important information we propose a graph-based, language-guided attention mechanism. Being composed of node attention component and edge attention component, the proposed graph attention mechanism explicitly represents inter-object relationships, and properties with a flexibility and power impossible with competing approaches. Furthermore, the proposed graph attention mechanism enables the comprehension decision to be visualisable and explainable. Experiments on three referring expression comprehension datasets show the advantage of the proposed approach.
1. Introduction
Referring expression comprehension matches language to image regions, but complex expressions require modelling both the referent and its relationships with surrounding objects. LGRAN addresses this with language-guided node and edge attention over an object graph, improving performance and interpretability.
- Problem: Referring expression comprehension selects the image region best matching a natural-language expression.The central challenge is jointly understanding textual and visual information.
- Problem: Monolithic region and language representations overlook complex structures and adapt poorly when the expression is complex.Existing approaches often learn language and region features independently, limiting cross-modal adaptation.
- Motivation: Compound expressions identify referents through relationships with neighbouring objects, not only through the referent’s own properties.The example “the child held by a woman beside a table” requires using clues from the child, woman, and table.
- Method: LGRAN builds a directed object graph and uses language-guided node attention for relevant objects plus edge attention for intra- and inter-class relationships.Three language parts guide the corresponding attention types, and the attended sub-graph enriches the candidate representation.
- Results: Experiments on RefCOCO, RefCOCO+ and RefCOCOg outperform previous best results on almost all splits under different settings.The graph attention mechanism also grounds the referent and supporting relationships, making decisions visualisable and explainable.
2. Related Work
Prior work approaches referring expression comprehension with recurrent visual-language models or metric learning, generally relying on compact representations and compatibility objectives.
- CNN/LSTM approaches: Conventional CNN/LSTM models combine region-level visual features with word vectors while differing in how they incorporate visual context.Examples use whole-image features, learned context regions, or visual differences between objects.
- Metric learning: Metric-learning approaches embed expression and region features in a common space to measure compatibility.They use objectives including softmax, max-margin, and Maximum Mutual Information losses.
3. Language-guided Graph Attention Networks (LGRANs)
LGRANs decomposes referring expressions into subject and relationship representations, then uses language-guided graph attention to adapt object representations to relevant nodes and edges.
- Architecture: LGRANs contains language self-attention, language-guided graph attention, and matching modules.The language module decomposes the expression, while the graph module processes candidate objects and relationships.
- Language Self-Attention: Self-attention decomposes an expression into subject, intra-class relationship, and inter-class relationship components.Word embeddings are processed by a Bi-LSTM and three attention mechanisms to derive the corresponding representations.
- Graph Construction: The graph represents candidate objects as nodes and object relationships as directed edges divided into intra-class and inter-class sets.Edges connect each object to its top k nearby same-category and other-category objects, with k = 5.
- Language-guided Graph Attention: Graph attention uses language features to assign attention to objects, intra-class edges, and inter-class edges.These attention values identify expression-relevant content and produce object features adapted to the expression.
- Language-guided Graph Attention: Inter-class relationships concatenate relative spatial edge features with the neighboring object's feature, unlike intra-class relationships that use relative spatial features alone.This explicitly models the other object when connected objects belong to different categories.
4. Experiments
Experiments evaluate LGRANs across three datasets, compare it with existing methods under ground-truth and detected regions, and test its components and interpretability.
- Datasets: Experiments cover RefCOCO, RefCOCO+ and RefCOCOg, with dataset-specific splits and expression characteristics.RefCOCO and RefCOCO+ use short game-collected phrases, while RefCOCOg uses longer declarative sentences; RefCOCO+ excludes absolute location words.
- Evaluation: Accuracy counts predictions whose intersection-over-union with the ground-truth referent exceeds 0.5.The evaluation first uses given MSCOCO regions and then automatically detected regions.
- Overall Results: On ground-truth regions, LGRANs outperforms other methods on almost all splits.The authors attribute its advantage to dynamically identifying language-relevant content while modeling relationships between objects.
- Ablation Study: Ablation results show progressive gains from graph representation, node attention and edge attention, with EdgeAttn improving performance by up to 3.4%.The full GraphAttn model consistently outperforms the incomplete solutions.
- Automatically detected regions: With automatically detected regions, all methods lose performance because of region quality, but LGRANs remains consistently better than comparison methods.This setting evaluates the model in fully automatic referring expression comprehension.
- Visualisation: Unlike attention schemes over isolated regions, LGRANs predicts attention over both objects and inter-object relationships.Visualisations show intra-class attention distinguishing among similar objects and inter-class attention selecting relevant relationships such as an umbrella or motorcycle.
5. Conclusion
LGRANs uses language-guided node and edge attention over an object graph to localise referents through relevant objects and relationships. The attended graph enriches object representations while making comprehension decisions visualisable and explainable.
- LGRANs builds a graph whose nodes are image objects and whose edges represent relationships between them.
- Language-guided node attention highlights referent candidates and narrows the search space for localisation.
- Language-guided edge attention identifies relevant relationships between the referent and its neighbourhood as informative clues.
- The attended graph dynamically enriches object representations so they better adapt to the referring expression.
- LGRANs makes referring-expression comprehension decisions visualisable and explainable.