PaperlayerView exact revision on arXiv

Source-linked AI summary

Real-Time Referring Expression Comprehension by Single-Stage Grounding Network

Xinpeng Chen, Lin Ma, Jingyuan Chen, Zequn Jie, Wei Liu, Jiebo Luo

arXiv:1812.03426v1cs.CV

TL;DR

Referring-expression comprehension must localize a described referent while interpreting semantic and spatial relationships, and proposal-based multi-stage models can be computationally expensive. SSG instead uses a single-stage multimodal model with guided attention and attribute prediction to directly ground expressions. It reports comparable performance with state-of-the-art models, state-of-the-art performance on ReferItGame, and an average 40 referents per second on RefCOCO.

  • Problem

    Referring-expression comprehension requires locating a described referent while understanding semantic and spatial relationships, and multi-stage models can be computationally expensive.

    Source: §1 · pp. 1–2

  • Method

    SSG encodes the image and expression, attentively summarizes useful image regions, and directly predicts the referent’s bounding box, using guided attention and attribute prediction.

    Source: §1 · p. 2, §3 · p. 3

  • Results

    SSG achieves comparable results with state-of-the-art models, state-of-the-art performance on ReferItGame, and an average 40 referents per second on RefCOCO.

    Source: §1 · p. 2, §5 · p. 9

  • Takeaways & Limitations

    Across four public datasets, SSG achieves favorable performance and runs at an average 40 referents per second on RefCOCO.

    Source: §5 · p. 9

Abstract

from arXiv · show

In this paper, we propose a novel end-to-end model, namely Single-Stage Grounding network (SSG), to localize the referent given a referring expression within an image. Different from previous multi-stage models which rely on object proposals or detected regions, our proposed model aims to comprehend a referring expression through one single stage without resorting to region proposals as well as the subsequent region-wise feature extraction. Specifically, a multimodal interactor is proposed to summarize the local region features regarding the referring expression attentively. Subsequently, a grounder is proposed to localize the referring expression within the given image directly. For further improving the localization accuracy, a guided attention mechanism is proposed to enforce the grounder to focus on the central region of the referent. Moreover, by exploiting and predicting visual attribute information, the grounder can further distinguish the referent objects within an image and thereby improve the model performance. Experiments on RefCOCO, RefCOCO+, and RefCOCOg datasets demonstrate that our proposed SSG without relying on any region proposals can achieve comparable performance with other advanced models. Furthermore, our SSG outperforms the previous models and achieves the state-of-art performance on the ReferItGame dataset. More importantly, our SSG is time efficient and can ground a referring expression in a 416*416 image from the RefCOCO dataset in 25ms (40 referents per second) on average with a Nvidia Tesla P40, accomplishing more than 9* speedups over the existing multi-stage models.

1. Introduction

SSG targets referring-expression grounding without the proposal-generation and region-wise extraction stages used by earlier multi-stage methods. It directly predicts a bounding box, adds guided attention and visual-attribute prediction, and reports comparable performance with more than 9× speedups over existing multi-stage models.

1. Introduction

  • Referring-expression comprehension requires both localizing the described referent and understanding semantic and spatial relationships that distinguish it from other image objects.Source: §1 · p. 1
  • Earlier multi-stage approaches generate candidate regions, encode image regions and expressions, then rank regions; their stages, especially proposal generation and region-wise extraction, are computationally expensive.Source: §1 · pp. 1–2
  • SSG uses a multimodal encoder, an expression-conditioned interactor to summarize image regions, and a grounder that directly predicts bounding-box coordinates.It avoids region proposals and subsequent region-wise feature extraction.Source: §1 · p. 2
  • Guided attention uses an object center-bias to focus on the referent’s central region, while visual-attribute prediction helps distinguish referent objects.Source: §1 · p. 2
  • More than 9× faster than existing multi-stage models, SSG achieves comparable results with state-of-the-art models under the same hardware environment.Source: §1 · p. 2

2. Related Work

Prior referring-expression comprehension methods commonly rank image-region proposals, while object-detection work supplies related mechanisms for predicting boxes from image grids.

2.1. Referring Expression Comprehension

  • Earlier comprehension models score region proposals using visual and spatial features, with later work modeling region relationships, jointly handling comprehension and generation, or exploiting referent-context reciprocity.The review notes that multi-stage methods can be computationally expensive for practical applications.Source: §2.1 · p. 2

2.2. Object Detection

  • YOLO predicts bounding boxes and category confidence values from grid cells; YOLO-v2 adds five anchor boxes per cell and predicts their corrections.Source: §2.2 · p. 2
  • YOLO-v3 uses a 53-layer convolutional backbone and adds a pass-through layer to obtain finer features for small-object localization.Source: §2.2 · p. 3

3. Architecture

SSG encodes an image and referring expression, attends to relevant visual regions, and predicts a bounding box in a single stage without region proposals. Auxiliary guided-attention and attribute-prediction modules support training, while inference uses the localization module.

3. Architecture

  • SSG predicts the referent’s bounding box directly from an image and referring expression, without relying on region proposals.Its architecture comprises multimodal encoders, an interactor, and a grounder.Source: §3 · p. 3

3.1. Multimodal Encoder

  • The multimodal encoders represent images with local CNN features and expressions with semantic vectors from a recurrent text encoder.The image features cover local regions; the text encoder uses stacked bidirectional LSTM layers.Source: §3.1 · p. 3, §3.1.1 · p. 3, §3.1.2 · p. 3

3.2. Multimodal Interactor

  • Expression-conditioned attention weights and aggregates local visual features, then concatenates the aggregate with the text feature into a joint representation.The grounder uses this joint representation to localize the referring region.Source: §3.2 · pp. 3–4

3.3.1 Localization

  • The localization module predicts box coordinates and a confidence score from the joint representation; coordinate training uses mean squared error, while confidence is treated as binary classification.Box coordinates encode the top-left position and square-root width and height; confidence reflects predicted-box accuracy, with IoU threshold η usually set to 0.5.Source: §3.3.1 · pp. 4–5

3.3.2 Guided Attention

  • Guided attention trains the model to assign the referent’s central region a positive label and other regions negative labels, using an auxiliary cross-entropy loss.The 13 × 13 attention map contains 169 weights.Source: §3.3.2 · p. 5, Figure 3 · p. 5

3.3.3 Attribute Prediction

  • Attribute prediction is a multi-label auxiliary task intended to help distinguish referents, including objects of the same category, using frequent attribute words.The model predicts 50 attributes; its attribute loss is zero when the expression contains no attribute word.Source: §3.3.3 · pp. 5–6

3.4. Training Objective

  • The training objective combines localization, confidence, guided-attention, and attribute-prediction losses with weights balancing their contributions.The four weights are λloc, λconf, λatt, and λattr.Source: §3.4 · p. 6

3.5. Inference

  • At inference, SSG deactivates guided attention and attribute prediction and uses the localization module to predict the box in one stage.The encoders, interactor, and localization module jointly produce the prediction.Source: Figure 2 · p. 4, §3.5 · p. 6

4. Experiments

Experiments compare SSG across four referring-expression datasets and report competitive results, with the strongest comparison on ReferItGame and a substantial inference-speed advantage. Ablations and qualitative examples examine the contributions of the model’s losses and attention and attribute predictions.

4.1. Datasets

  • The evaluation covers RefCOCO, RefCOCO+, RefCOCOg, and ReferItGame, whose expression styles and dataset splits differ.RefCOCO+ omits location words, while RefCOCOg has longer expressions on average than RefCOCO and RefCOCO+ (8.4 vs. 3.5); both RefCOCOg splits are evaluated.Source: §4.1 · pp. 6–7

4.3. Performance Comparisons

  • SSG outperforms previous models on ReferItGame, while achieving favorable or competitive results on RefCOCO, RefCOCO+, and RefCOCOg.On RefCOCOg umd, SSG trails MAttNet but outperforms [16].Source: Table 2 · p. 7, §4.3 · p. 7

4.2. Experiment Settings

  • Expressions are truncated to 15 words, and a prediction counts as correct when its IoU with the ground truth exceeds 0.5.ELMo initializes the word embeddings.Source: §4.2 · p. 7

4.5. Efficiency

  • 0.025 seconds per referent, or 40 referents per second, makes SSG more than 9× faster than MAttNet on RefCOCO with GPU acceleration.The comparison uses the same hardware environment.Source: Table 3 · p. 7, §4.5 · p. 8

4.3. Performance Comparisons

  • Qualitative examples pair predicted and ground-truth boxes with attention heat maps and top-five predicted attributes; the authors report accurate grounding and meaningful attention.The examples show predicted attributes such as “blue,” “white,” and “food.”Source: Figure 4 · p. 8, §4.3 · p. 8

4.4. Ablation Study

  • Adding confidence-score loss, guided-attention loss, and attribute-prediction loss successively improves performance over the localization-loss-only baseline.The reported ablations cover RefCOCO, RefCOCO+, RefCOCOg, and ReferItGame.Source: §4.4 · p. 8

5. Conclusion

SSG directly grounds referring expressions without region proposals, using guided attention and attribute prediction. Experiments show favorable performance across four datasets, state-of-the-art results on ReferItGame, and an average speed of 40 referents per second on RefCOCO.

5. Conclusion

  • SSG directly localizes referring-expression referents without region proposals.The model uses guided attention based on object center-bias and attribute-prediction loss.Source: §5 · p. 8
  • Across four public datasets, SSG achieves state-of-the-art performance on ReferItGame.The paper reports favorable performance across the datasets overall.Source: §5 · p. 9
  • 40 referents per second is SSG’s average speed on RefCOCO.Source: §5 · p. 9

A.1. Datasets

The experiments use four commonly used datasets for referring expression comprehension, and Table 4 summarizes the datasets. RefCOCOg has two types of data splits.

A.1. Datasets

  • The experiments use RefCOCO, RefCOCO+, RefCOCOg, and ReferItGame.These four commonly used datasets are summarized in Table 4.Source: §A.1 · p. 10, Table 4 · p. 10
  • RefCOCOg has two types of data splits.Source: §A.1 · p. 10
  • Table 4 summarizes the datasets used for referring expression comprehension.Source: Table 4 · p. 10

A.2. Effect of End-to-end Training

The authors compare freezing the image encoder with fine-tuning it and report consistent performance improvements from fine-tuning. SSG is end-to-end, with all component parameters jointly optimized by stochastic gradient descent.

A.2. Effect of End-to-end Training

  • Fine-tuning the image encoder consistently improves performance over freezing its parameters.Table 5 reports the comparison across RefCOCO, RefCOCO+, RefCOCOg, and ReferItGame; no values are provided in the supplied passage.Source: §A.2 · p. 10, Table 5 · p. 10
  • SSG is an end-to-end model whose component parameters can be optimized jointly by stochastic gradient descent.Source: §A.2 · p. 10
  • The comparison evaluates freezing the image encoder against fine-tuning it on four datasets.The datasets are RefCOCO, RefCOCO+, RefCOCOg, and ReferItGame.Source: §A.2 · p. 10, Table 5 · p. 10

A.3. More Examples

The paper presents additional qualitative SSG examples and contrasts them with failure cases.

A.3. More Examples

  • Additional qualitative examples show SSG with λloc+conf+att+attr.Source: §A.3 · p. 11
  • Figures 9 and 10 show failure cases for comparison with the additional qualitative examples.Source: §A.3 · p. 11

A.3.1 Qualitative Results

Figures 5–8 present qualitative referring-expression comprehension results alongside visual attention heat maps and top-five predicted attributes. They mark ground-truth and predicted boxes differently and show ground-truth box centers.

A.3.1 Qualitative Results

A.3.2 Failure Cases

Figures 9 and 10 present failure cases for referring-expression comprehension alongside visual attention heat maps and predicted attributes.

A.3.2 Failure Cases

Related papers

Loading 1812.03426v1…