PaperlayerView exact revision on arXiv

Source-linked AI summary

A Real-Time Cross-modality Correlation Filtering Method for Referring Expression Comprehension

Yue Liao, Si Liu, Guanbin Li, Fei Wang, Yanjie Chen, Chen Qian, Bo Li

arXiv:1909.07072v4cs.CV

TL;DR

Existing two-stage referring-expression methods are computationally costly and fall short of real-time inference. RCCF uses cross-modality correlation filtering to localize targets, achieving state-of-the-art performance across four datasets and raising RefClef performance from 34.70% to 63.79%.

  • Problem

    Existing two-stage methods depend on object proposals and are computationally costly, leaving inference short of real-time speed.

    Source: §1 · pp. 1–2

  • Method

    RCCF reformulates referring expression comprehension as cross-modality template matching, using language as a filter over image features to localize the target.

    Source: §1 · p. 2

  • Results

    RCCF achieves state-of-the-art performance on RefClef, RefCOCO, RefCOCO+ and RefCOCOg; on RefClef, performance rises from 34.70% to 63.79%.

    Source: §1 · p. 2

  • Takeaways & Limitations

    The findings support correlation filtering as a one-stage framework for real-time referring expression comprehension.

    Source: §1 · p. 2, §5 · p. 8

    Reported failure cases include inaccurate bounding-box localization, heavily occluded targets, and very complex or long expressions.

    Source: §4.6 · p. 8

Abstract

from arXiv · show

Referring expression comprehension aims to localize the object instance described by a natural language expression. Current referring expression methods have achieved good performance. However, none of them is able to achieve real-time inference without accuracy drop. The reason for the relatively slow inference speed is that these methods artificially split the referring expression comprehension into two sequential stages including proposal generation and proposal ranking. It does not exactly conform to the habit of human cognition. To this end, we propose a novel Realtime Cross-modality Correlation Filtering method (RCCF). RCCF reformulates the referring expression comprehension as a correlation filtering process. The expression is first mapped from the language domain to the visual domain and then treated as a template (kernel) to perform correlation filtering on the image feature map. The peak value in the correlation heatmap indicates the center points of the target box. In addition, RCCF also regresses a 2-D object size and 2-D offset. The center point coordinates, object size and center point offset together to form the target bounding box. Our method runs at 40 FPS while achieving leading performance in RefClef, RefCOCO, RefCOCO+ and RefCOCOg benchmarks. In the challenging RefClef dataset, our methods almost double the state-of-the-art performance (34.70% increased to 63.79%). We hope this work can arouse more attention and studies to the new cross-modality correlation filtering framework as well as the one-stage framework for referring expression comprehension.

1. Introduction

RCCF reframes referring expression comprehension as cross-modal template matching, using language as a filter over image features to localize the described object. It aims to avoid the proposal-generation and ranking stages used by conventional methods.

1. Introduction

  • Conventional two-stage methods depend on proposal quality and extra detection data, and are computationally costly because they extract features and compute similarity for each proposal.The final selection uses only the proposal with the highest similarity.Source: §1 · pp. 1–2
  • RCCF uses the language embedding as a correlation filter over image feature maps, fuses multi-level center heatmaps, and regresses object size and center offset.The peak of the center heatmap supplies the target center; the corresponding width, height, and offset form the bounding box.Source: §1 · p. 2
  • 40 FPS with a single GPU, 12 times faster than two-stage methods, while RCCF's precision outperforms state-of-the-art methods.Figure 1 reports 40 fps (0.25 ms per image) on RefCOCO testA using a single Titan Xp GPU.Source: Figure 1 · p. 1, §1 · p. 2
  • RCCF can be trained using referring-expression data alone, without additional object-detection data, and avoids error accumulation from the object detector.Source: §1 · p. 2
  • 34.70% to 63.79% on RefClef, while RCCF achieves state-of-the-art performance on RefCOCO, RefCOCO+, and RefCOCOg.The paper describes the RefClef result as almost double the state-of-the-art performance.Source: §1 · p. 2

2. Related Work

Prior referring-expression methods largely rely on detector-generated proposals followed by ranking, while correlation filtering provides a template-based visual-matching precedent. The paper motivates a single-stage cross-modality filtering formulation for direct localization.

2.1. Referring Expression Comprehension

  • Two-stage referring-expression methods generate object proposals before ranking them, with much prior work improving ranking through context and referent relationships, including MAttNet [32].These methods explore language-image context and relationships between referents.Source: §2.1 · p. 2
  • Two-stage methods are limited by object detectors and spend time generating proposals and extracting proposal features, motivating direct expression-conditioned localization.Source: §2.1 · p. 2

2.2. Correlation Filtering

  • Correlation filtering has been used in computer vision for classification [7] [26] and tracking, including Fourier-domain methods [8] and Siamese FC [1].Source: §2.2 · p. 2
  • Inspired by global sentence understanding and template matching to the highest-response image region, the authors formulate grounding as cross-modality correlation filtering with single-stage joint optimization.Source: §2.2 · pp. 2–3

3. Method

RCCF localizes a referring-expression target without proposal generation by treating the expression as a visual-domain filter kernel, then combining the heatmap peak with regressed size and center offset to form the box.

3.1. Framework

  • RCCF maps the query and image to the target center, size, and local center offset, using the heatmap peak as the center estimate.This reframes referring-expression comprehension as cross-modality template matching rather than proposal generation followed by ranking.Source: §3.1 · p. 3, §3 · p. 3

3.2. Expression and Image Encoder

  • A Bi-LSTM encodes expressions as 512-D features, while modified 34-layer DLA with deformable convolutions extracts three visual feature levels.All three levels contribute to correlation; only the highest-resolution feature level is used for regression for computational efficiency.Source: §3.2 · p. 3

3.3. Cross-modality Correlation Filtering

  • Three learned 64-D kernels map the expression into visual space, correlate with corresponding image features, and produce maps whose pixel-wise average yields a heatmap peak for the target center.The three maps are fused by pixel-wise averaging and passed through a sigmoid activation.Source: §3.3 · p. 3, Figure 2 · p. 4

3.4. Size and Offset Regression

  • Parallel size and offset branches regress box dimensions and center-point offsets from visual features alone, preserving spatial information without expression features.Both branches use a 3 × 3 convolution with ReLU followed by a 1 × 1 convolution; regression loss is applied only at the ground-truth center.Source: §3.4 · p. 4, Figure 2 · p. 4

3.5. Loss and Inference

  • At inference, RCCF selects the highest-confidence heatmap location and reads the corresponding size and offset predictions to construct the target box.Training combines center-point focal loss with size and offset L1 losses; the size and offset weights are 0.1 and 1, respectively.Source: §3.5 · pp. 4–5, §3.3 · p. 4, §3.4 · p. 4

4. Experiments

Experiments evaluate RCCF on four benchmarks, analyze its components through ablations, measure efficiency, and examine qualitative successes and failures.

4.1. Experimental Setting

  • Evaluation covers RefClef, RefCOCO, RefCOCO+, and RefCOCOg using Prec@0.5, with RefCOCO+ excluding absolute-location words and RefCOCOg collected for longer expressions.RefCOCO and RefCOCO+ have train, val, testA, and testB splits; testA contains multiple people and testB multiple objects, while RefClef and RefCOCOg follow specified prior-work splits.Source: §4.1 · p. 5

4.2. Implementation Details

  • RCCF is trained on eight GPUs with Adam for 80 epochs at 512 × 512 input resolution and output stride 4, using random shifts and scaling for augmentation.The visual encoder is COCO-pretrained, while language encoder and output heads are randomly initialized; inference uses no augmentation, and ImageNet visual pretraining is tested in ablations.Source: §4.2 · p. 5

4.3. Comparison to the State-of-the-art

  • RCCF’s gains are attributed to its framework rather than a larger or more complex backbone, and avoiding explicit object detection reduces dependence on detector quality and training data.The compared methods mainly generate proposals with Faster R-CNN pretrained on COCO and Visual Genome; the authors note that this dependence particularly limits two-stage methods on RefClef.Source: §4.3 · p. 6
  • 63.79% from 34.70%: RCCF nearly doubles RefClef precision, outperforms prior methods on every RefCOCO and RefCOCO+ evaluation set, and is comparable but slightly below MAttNet on RefCOCOg.The RefCOCOg comparison is partly qualified by MAttNet’s additional attribute and region-class supervision and its ResNet-101 backbone; RCCF uses language-image pairs and DLA-34.Source: Table 2 · p. 5, §4.3 · p. 6, Table 3 · p. 7

4.4. Ablation Studies

  • Average fusion outperforms maximum and concatenation fusion, while one shared language filter across visual levels reduces precision by about 3 points and level-specific filters are sensitive to their visual features.A 3 × 3 language kernel performs almost the same as the 1 × 1 kernel, so the lower-computation 1 × 1 kernel is selected.Source: §4.4 · p. 6, Table 4 · p. 8
  • Language-guided features for size and offset regression reduce precision by about 6 points; ImageNet pretraining costs about 2 points, GloVe changes little, and Hourglass-104 is slower with little improvement.The ImageNet-pretrained setting remains comparable to the state of the art, showing RCCF can work without object-detection pretraining.Source: §4.4 · pp. 6–7, Table 4 · p. 8

4.5. Efficiency Analysis

  • 25 ms per image makes RCCF the only real-time method described and 12 times faster than MAttNet’s 314 ms; training on RefCOCO takes 35 hours on eight GPUs.MAttNet’s first and second stages take 262 ms and 52 ms per image, respectively.Source: §4.5 · pp. 7–8

4.6. Qualitative Results Analyses

  • Correlation maps respond strongly near the expression-specified object center, and qualitative examples show RCCF can localize abstract objects beyond common detector categories.The reported failure cases include inaccurate boxes, heavy occlusion, ambiguous ground truth, background targets, and long complex expressions.Source: Figure 3 · p. 7, §4.6 · p. 8, Figure 4 · p. 8

5. Conclusion and Future Works

RCCF directly localizes an object by predicting its center from a correlation map between the referent and image, achieving state-of-the-art performance on four referring-expression datasets at real-time speed. Future work will explore richer context and training with easier-to-annotate datasets.

5. Conclusion and Future Works

  • RCCF achieves state-of-the-art performance on four referring-expression datasets at real-time speed.Source: §5 · p. 8
  • Unlike previous two-stage methods, RCCF directly localizes the object by predicting its center from a correlation map between the referent and image.Source: §5 · p. 8
  • Future work will explore richer context from expressions and images and use easier-to-annotate datasets, such as object detection and image caption data, to train RCCF.Source: §5 · p. 8

Related papers

Loading 1909.07072v4…