PaperlayerView exact revision on arXiv

Source-linked AI summary

Referring Expression Comprehension: A Survey of Methods and Datasets

Yanyuan Qiao, Chaorui Deng, Qi Wu

arXiv:2007.09554v2cs.CVcs.CL

TL;DR

REC asks models to locate an object from a natural-language expression rather than a predefined label. This survey reviews methods and datasets, compares results across settings, and identifies limitations and future directions. It reports competitive performance from faster one-stage approaches and notes that current REC systems can lack interpretability.

  • Problem

    REC must match natural-language expressions to image regions, distinguishing objects through language and visual information rather than relying on predefined category labels.

    Source: §I · pp. 1–2

  • Method

    The survey reviews REC methods, datasets, and evaluation, classifies approaches by design, and groups results by dataset, backbone, and setting for comparison.

    Source: §I · p. 2, §V · p. 13

  • Results

    One-stage approaches are much faster yet remain competitive with many complex two-stage methods, while modular models can outperform graph-based approaches under limited existing datasets.

    Source: B. Modular-based Models · p. 5, §III · p. 12

  • Takeaways & Limitations

    The survey identifies compositional REC requiring longer reasoning chains as a promising future direction.

    Source: §I · p. 2

    Most reviewed methods provide limited interpretability: joint embedding often behaves as a black box, and attention maps do not show the reasoning steps.

    Source: A. What are current limitations of… · p. 12

Abstract

from arXiv · show

Referring expression comprehension (REC) aims to localize a target object in an image described by a referring expression phrased in natural language. Different from the object detection task that queried object labels have been pre-defined, the REC problem only can observe the queries during the test. It thus more challenging than a conventional computer vision problem. This task has attracted a lot of attention from both computer vision and natural language processing community, and several lines of work have been proposed, from CNN-RNN model, modular network to complex graph-based model. In this survey, we first examine the state of the art by comparing modern approaches to the problem. We classify methods by their mechanism to encode the visual and textual modalities. In particular, we examine the common approach of joint embedding images and expressions to a common feature space. We also discuss modular architectures and graph-based models that interface with structured graph representation. In the second part of this survey, we review the datasets available for training and evaluating REC systems. We then group results according to the datasets, backbone models, settings so that they can be fairly compared. Finally, we discuss promising future directions for the field, in particular the compositional referring expression comprehension that requires longer reasoning chain to address.

I. INTRODUCTION

Referring expression comprehension (REC) localizes an image region from a natural-language description, requiring language and visual information to distinguish the intended object. This survey reviews REC methods, datasets, evaluation, and future directions.

I. INTRODUCTION

  • REC selects the best region proposal for a referring expression, whose attributes and context can distinguish objects of the same category.The task combines natural-language descriptions with visual information about objects and their relationships.Source: §I · p. 1
  • Longer expressions may require more reasoning, while images lack language-like structure and convenient analysis tools.REC must process complex semantics alongside visual relationships, attributes, and regions.Source: §I · pp. 1–2
  • Unlike object detection with predefined category labels, REC uses variable natural-language descriptions to find the best matching region.The survey notes that language can distinguish same-category objects and can include attributes, spatial configuration, and interactions.Source: §I · pp. 1–2
  • REC is also an important first step for vision-language tasks such as visual question answering and visual dialogue, which require matching language to objects.The survey focuses on REC because simple detection cannot use textual information as a separate label.Source: §I · p. 2
  • The survey reviews REC methods, datasets, and evaluation metrics, then discusses possible future directions.Its methods review covers seven categories, including joint embedding, modular, graph-based, weakly supervised, one-stage, and pre-training approaches.Source: §I · p. 2

II. METHODS FOR REFERRING EXPRESSION COMPREHENSION

Joint-embedding methods encode language and image regions for matching, with approaches adding context or attention to improve representation. The surveyed results report gains for context- and attention-based methods, while noting limits in monolithic representations and attention supervision.

A. Joint embedding approaches

  • The CNN-LSTM approach feeds region-level CNN features and word vectors to an LSTM to maximize expression likelihood for a referred region.Its monolithic vectors ignore complex language and image structure, including expression dependencies.Source: A. Joint embedding approaches · pp. 2, 4
  • Mao et al. introduced a deep-learning approach combining CNN visual features and LSTM expression modeling, with maximum mutual information penalizing descriptions that fit other objects.The work also addressed the inverse task of localizing an object from an expression.Source: A. Joint embedding approaches · p. 2
  • VisDif uses visual comparisons with surrounding objects, while Nagaraja et al. use multiple-instance learning to model context regions without context-object annotations.Nagaraja et al. treat the entire image as a potential context region and model an expression as a function of a region and context region.Source: A. Joint embedding approaches · pp. 2–3
  • SCRC scores candidate regions using language, local descriptors, spatial configurations, and global scene context; its pre-training and adaptation process improves performance and avoids over-fitting.It uses a two-layer LSTM and transfers visual-language knowledge from the image-caption domain.Source: A. Joint embedding approaches · p. 3
  • Luo et al. use a comprehension model to rerank generated expressions, while Zhang et al. use variational inference to refine referent localization through context.The variational context method approximates context configurations with weak supervision.Source: A. Joint embedding approaches · p. 3
  • PLAN attends to image patches for global context and to object proposals using language, then matches the resulting representations to score proposals.AccumAtt jointly refines attention to the query, image concepts, and target object through an accumulating process.Source: A. Joint embedding approaches · p. 4
  • Attention-based PLAN and AccumAtt improve performance on RefCOCO, RefCOCO+, RefCOCOg, and GuessWhat?!, while SCRC improves over previous baselines on ReferItGame and Flickr30K Entities.PLAN experiments support attention at both image and region levels; AccumAtt ablations support multiple attention modules and accumulated attention scores.Source: A. Joint embedding approaches · p. 4
  • Attention-based methods cannot guarantee correct attention assignments because datasets usually lack corresponding annotations.The survey also notes that attention visualizations show attended regions but do not fully explain reasoning steps.Source: A. Joint embedding approaches · p. 4

B. Modular-based Models

Modular REC models decompose expressions into components and match those components to visual regions. The survey reports strong modular-model performance, while noting limits in handling complex language and visual relationships.

Motivation

  • Modular networks decompose expressions and match components to corresponding visual regions, addressing the limits of single-LSTM encoding or decoding.Early methods used an external parser, while later methods learned decomposition end-to-end.Source: B. Modular-based Models · p. 4

Methods

  • CMNs combine unary localization scores for subject or object regions with pairwise relationship scores after parsing expressions into subject, relationship, and object.CMNs learn linguistic analysis and visual inference end-to-end.Source: B. Modular-based Models · p. 5
  • MAttNet separates subject appearance, location, and relationships into modules, while CM-Att-Erase adds global image features and sentence-aware attention to its modular design.MAttNet learns expression parsing through soft attention; CM-Att-Erase erases the most attended textual or visual part during training.Source: B. Modular-based Models · p. 5

Performance and limitations

  • Modular models such as CM-Att-Erase outperform graph-based approaches under limited existing datasets, suggesting good performance without complex reasoning.CM-Att-Erase erases dominant textual or visual information during training to encourage discovery of complementary cross-modal correspondences.Source: B. Modular-based Models · p. 5
  • Modular networks oversimplify language structure and ignore relationships between visual objects, making them unsuitable for overly complex expressions.The independently learned or designed language and regional features may not adapt well to one another, especially for complex expressions.Source: B. Modular-based Models · p. 5

C. Graph based models

Graph-based REC represents candidate objects and their relationships explicitly, then uses language-guided graph reasoning to match an expression to an object. The survey reports strong results for several graph approaches, including state-of-the-art performance for CMCC on Flickr30K Entities.

Methods

  • LGRAN uses language self-attention to represent subject and intra- and inter-class relations, then graph attention to construct expression-relevant object representations for matching.Its graph contains candidate-object nodes and intra-class and inter-class edges.Source: C. Graph based models · pp. 5–6
  • DGA performs step-by-step graph inference guided by a predicted visual reasoning process, while CMCC matches phrase graphs with visual object graphs and CMRIN fuses cross-modal information through gated graph convolution.The survey reports that DGA surpasses existing algorithms and provides visual evidence for gradually locating objects in complex descriptions; LGRAN decisions are described as visual and interpretable.Source: C. Graph based models · p. 6

Performance and limitations

  • CMCC achieved state-of-the-art performance on Flickr30K Entities, surpassing other methods by a large advantage.The graph neural network is trained with a two-stage strategy.Source: C. Graph based models · p. 6

D. External parser

External-parser methods use syntactic structure to guide REC and can link phrases to objects or auxiliary supporting objects. The survey describes interpretability and fine-grained reasoning benefits, alongside a parser-related vocabulary caveat.

Motivation and methods

  • GroundNet maps syntactic components and relationships from an externally parsed expression to a composed graph of neural modules for localization of target and auxiliary supporting objects.The survey describes grammar-based methods as a way to use linguistic structure to locate target and supporting objects.Source: D. External parser · p. 6

Methods

  • NMTree converts a dependency parse into a neural module tree and accumulates grounding scores from leaves toward the root.Its Single module handles leaves and the root, while internal nodes use Sum and Comp modules.Source: D. External parser · p. 6

Performance and limitations

  • GroundNet offers phrase-to-object and localization-process interpretability, while NMTree is reported to provide more fine-grained interpretable compound reasoning and better performance.GroundNet maintains comparable target-object localization performance; parser processing may remove determiners that help identify an object.Source: D. External parser · p. 7

E. Vision-Language pre-training

Vision-language pretraining jointly models image and text features for REC, while one-stage methods avoid proposal-based pipelines. The survey reports strong RefCOCO+ performance for pretrained models and faster inference for one-stage approaches, with competitive or better performance for FAOA and MCN against strong two-stage baselines.

E. Vision-Language pre-training

  • ViLBERT uses separate vision and language streams connected by co-attention, then scores each image region using its final representation.It is pretrained on Conceptual Captioning with masked multimodal learning and multimodal alignment prediction.Source: E. Vision-Language pre-training · p. 7
  • ViLBERT and VL-BERT achieve the best performance on RefCOCO+ and significantly outperform previous state-of-the-art results, but require high computational budgets.VL-BERT uses one transformer stream for image and text tokens, unlike ViLBERT’s separate modality branches.Source: E. Vision-Language pre-training · p. 7

F. One stage approaches

  • Two-stage REC first generates object proposals and then matches them to the expression, incurring extra computation and dependence on proposal quality.A detector can miss the target, making the subsequent matching stage unable to succeed.Source: F. One stage approaches · pp. 7–8
  • FAOA combines image-pyramid and BERT language features to predict a target box, while ZSG combines detection and grounding using ResNet/FPN visual features and Bi-LSTM language features.FAOA also supplies normalized pixel coordinates to its prediction head; ZSG uses Focal Loss to classify regions inside or outside the target.Source: F. One stage approaches · p. 8
  • MCN jointly solves comprehension and segmentation in a one-stage pipeline, using consistency energy maximization and adaptive soft non-located suppression to address prediction conflicts.Source: F. One stage approaches · p. 8
  • One-stage methods achieve an order of magnitude speedup over two-stage approaches, and FAOA and MCN perform competitively or better than strong two-stage baselines including MAttNet.They avoid proposal generation and proposal feature extraction, the most time-consuming parts of two-stage approaches.Source: F. One stage approaches · p. 8

G. Weakly supervised

The survey reviews weakly supervised REC, where training lacks known expression-to-object mappings, and compares approaches that reconstruct language, align visual information, or locate phrases without bounding-box supervision.

G. Weakly supervised

  • Weakly supervised REC trains without known mappings between referring expressions and their target objects, addressing the annotation demands of supervised training.GroundR selects relevant regions and reconstructs the phrase during training, then uses only its grounding stage at test time.Source: G. Weakly supervised · p. 8
  • KAC Net reconstructs both expression and region information through visual- and language-consistency branches, with a knowledge-based pooling gate.Its branches predict and align region location parameters and reconstruct the input expression from regions.Source: G. Weakly supervised · p. 8
  • MATN locates freeform phrases without bounding-box supervision, using multi-scale correspondence and an anchor constraint from candidate regions.Training combines contrast reconstruction errors among phrases in one image with triple losses across images containing similar phrases.Source: G. Weakly supervised · p. 8
  • 4.5% and 10.6% higher performance for GroundR, and 9.78% and 5.13% higher accuracy for KAC Net, respectively, are reported on Flickr 30k Entities and ReferItGame.The survey also reports that MATN outperforms state-of-the-art weakly supervised phrase-localization methods by a significant margin.Source: G. Weakly supervised · p. 9
  • ARN combines adaptive grounding with collaborative reconstruction to match proposals and queries, and its qualitative results better handle multiple same-category objects.It encodes subject, location, and context features; reconstruction uses language, adaptive, and attribute-classification losses.Source: G. Weakly supervised · pp. 8–9

III. DATASETS AND EVALUATION

The survey reviews REC datasets and evaluation, then compares results across supervision, model families, and proposal settings. Its datasets range from real-world referring expressions to synthetic compositional reasoning and embodied navigation tasks.

III. DATASETS AND EVALUATION

  • ReferItGame contains real-world referring expressions collected through a two-player game; RefCOCO has 142,209 expressions, while RefCOCO+ has 141,564 and disallows location words.RefCOCO has 50,000 objects in 19,994 images and RefCOCO+ has 49,856 objects in 19,992 images; Test A contains multiple people, while Test B contains multiple instances of all other objects.Source: §III · pp. 9–10
  • RefCOCOg supplies longer, more complex expressions, while Flickr 30k Entities supports phrase localization and links entity mentions into coreference chains.RefCOCOg averages 8.43 words per expression; Flickr annotation includes coreference resolution and bounding-box annotation.Source: §III · p. 10
  • GuessWhat?! uses question-answer dialogues to locate objects; embodied datasets extend referring tasks to navigation, including Touchdown-SDR, Refer360°, and REVERIE.Refer360° targets can be any point in a panoramic scene, while REVERIE combines navigation with identifying a remote indoor object.Source: §III · pp. 10–12
  • CLEVR-Ref+ uses synthetic scenes, templated expressions, and uniform sampling to mitigate bias and probe reasoning, while Cops-Ref varies compositionality and tests fuller reasoning chains.CLEVR-Ref+ associates each image with 10 expressions; Cops-Ref contains 148,712 expressions across 75,299 images.Source: §III · pp. 11–12
  • REC evaluation averages IoU between the true and top-predicted boxes, counting a prediction as correct when IoU exceeds 0.5.The section summarizes dataset characteristics in Table I and results in Tables II–VII.Source: §III · pp. 9, 12
  • Supervised methods significantly outperform weakly supervised methods on ReferItGame, with similar conclusions on Flickr 30k Entities and Cops-Ref; weak supervision may be more practical given annotation costs.The comparisons are summarized in Tables II–III, VI, and VII.Source: Table II · p. 9, Table III · p. 9, Table VI · p. 11
  • Pretraining-based approaches, consistently achieve the best performance on RefCOCO and RefCOCO+, surpassing other approaches by a significant margin.The survey attributes improved generalization to pretraining on large-scale vision-and-language data.Source: Table IV · p. 10, §III · p. 12
  • Attention-, graph-, and modular-based approaches,,,,, outperform most early CNN/LSTM or joint-embedding methods; detected proposals perform worse than human-labeled proposals.One-stage methods [16], are faster yet competitive with many complex two-stage methods.Source: Table IV · p. 10, Table V · p. 11, §III · p. 12

IV. DISCUSSION AND FUTURE DIRECTIONS

The survey situates REC as a growing research area that remains challenging because it requires joint reasoning over visual and textual information. It discusses limitations and future directions for the field.

IV. DISCUSSION AND FUTURE DIRECTIONS

  • REC research has grown alongside advances in computer vision and natural language processing.The survey frames its discussion around REC's limitations and future directions.Source: §IV · p. 12
  • REC remains challenging because it requires joint reasoning over visual and textual information.Source: §IV · p. 12
  • The discussion addresses limitations of REC techniques and future directions for the field.Source: §IV · p. 12

A. What are current limitations of existing REC techniques?

The survey identifies interpretability and dataset bias as limitations of existing REC techniques, alongside evaluation that focuses on final predictions rather than intermediate reasoning.

A. What are current limitations of existing REC techniques?

  • Joint-embedding methods implicitly fuse expressions and candidate objects, leaving matching as a black box; attention highlights regions but not the reasoning steps.The survey identifies explaining the reasoning process as a continuing challenge.Source: A. What are current limitations of… · p. 12
  • Evaluation considers final predictions rather than intermediate reasoning, making models’ reasoning capabilities and defects difficult to assess.Source: A. What are current limitations of… · p. 12
  • Dataset imbalance can lead models to learn shallow correlations rather than joint image-text understanding, and some may rely on image information while ignoring the expression.The survey says such models are not applicable in real scenarios.Source: A. What are current limitations of… · pp. 12–13

B. Have we reached the upper bound of performance?

The survey argues that limited attribute coverage in existing datasets constrains performance gains from more sophisticated REC models. It points to multi-step reasoning benchmarks and compositional reasoning as directions for assessing and advancing the field.

B. Have we reached the upper bound of performance?

  • Limited attribute coverage in existing datasets accompanies good REC performance, while more sophisticated models have not yielded further gains on those datasets.Graph-based models introduce more comprehensive relationship encodings but show no significant improvement on existing referring-expression datasets.Source: B. Have we reached the upper bound of… · p. 13
  • Future REC work should combine language pre-training with the interpretability offered to some extent by graph-based and modular-based methods.The survey also identifies better exploitation of reasoning chains from spatial and semantic perspectives as a promising direction.Source: C. What is likely to be the next big… · p. 13
  • RefCOCO examples require two reasoning steps, whereas CLEVR-Ref+ [62] examples require multiple steps to locate the target.The example reasoning chain for “Man to the right of the green bag” finds bags, identifies the green one, then locates the man to its right.Source: C. What is likely to be the next big… · p. 13, Figure 10 · p. 13
  • State-of-the-art REC models show a significant drop on Cops-Ref [63] relative to conventional datasets, suggesting they may fit dataset bias rather than reason.Cops-Ref covers flexibly combined reasoning logics and rich visual properties; the survey identifies reasoning over long, semantically rich expressions as a challenge.Source: C. What is likely to be the next big… · p. 13

V. CONCLUSION

The paper reviews REC methods and datasets, organizes comparative results, and identifies future research directions, especially complex compositional reasoning.

V. CONCLUSION

  • The survey reviews common-feature-space embedding approaches alongside modular, graph-based, one-stage, and pre-training REC methods.It describes modular and graph-based models as more advanced approaches built upon common-space embedding methods.Source: §V · p. 13
  • The survey reviews REC datasets and groups results by dataset, backbone model, and setting to support fair comparison.Source: §V · p. 13
  • The paper identifies complex compositional reasoning in referring expressions as a promising research direction that may benefit other vision-and-language tasks.Source: §V · p. 13
Loading 2007.09554v2…