Source-linked AI summary
A Real-Time Cross-modality Correlation Filtering Method for Referring Expression Comprehension
Yue Liao, Si Liu, Guanbin Li, Fei Wang, Yanjie Chen, Chen Qian, Bo Li
TL;DR
Referring expression comprehension needs to localize language-described objects, but two-stage proposal generation and ranking hinder real-time inference and depend on proposals and extra detection data. RCCF instead uses language-guided cross-modality correlation filtering to predict object centers, sizes, and offsets directly. It reaches 40 FPS with leading performance across four benchmarks, including an increase from 34.70% to 63.79% on RefClef.
Problem
Two-stage referring expression methods rely on proposal quality, extra detection data, and costly per-proposal computation, limiting real-time inference.
Method
RCCF maps language into visual space, uses it as a correlation filter over image features, and regresses object size and center offset.
Results
RCCF achieves leading performance across RefClef, RefCOCO, RefCOCO+ and RefCOCOg while reaching 40 FPS; RefClef performance increases from 34.70% to 63.79%.
Takeaways & Limitations
The results support a real-time one-stage framework for referring expression comprehension that does not require additional object detection data.
Takeaways & Limitations
RCCF can fail with heavily occluded targets, ambiguous ground truth, background objects, or very complex and long expressions.
Abstract
from arXiv · showhide
Referring expression comprehension aims to localize the object instance described by a natural language expression. Current referring expression methods have achieved good performance. However, none of them is able to achieve real-time inference without accuracy drop. The reason for the relatively slow inference speed is that these methods artificially split the referring expression comprehension into two sequential stages including proposal generation and proposal ranking. It does not exactly conform to the habit of human cognition. To this end, we propose a novel Realtime Cross-modality Correlation Filtering method (RCCF). RCCF reformulates the referring expression comprehension as a correlation filtering process. The expression is first mapped from the language domain to the visual domain and then treated as a template (kernel) to perform correlation filtering on the image feature map. The peak value in the correlation heatmap indicates the center points of the target box. In addition, RCCF also regresses a 2-D object size and 2-D offset. The center point coordinates, object size and center point offset together to form the target bounding box. Our method runs at 40 FPS while achieving leading performance in RefClef, RefCOCO, RefCOCO+ and RefCOCOg benchmarks. In the challenging RefClef dataset, our methods almost double the state-of-the-art performance (34.70% increased to 63.79%). We hope this work can arouse more attention and studies to the new cross-modality correlation filtering framework as well as the one-stage framework for referring expression comprehension.
1 School of Computer Science and Engineering, Beihang University 2 Sun Yat-sen University 3 SenseTime Research
The passage provides contact information for the paper’s authors.
- The listed contact email is wangfei, chenyanjie, and qianchen at sensetime.com.
- The contact information uses three named local parts: wangfei, chenyanjie, and qianchen.
1. Introduction
RCCF addresses the speed and proposal-dependence of two-stage referring expression comprehension by directly correlating language with image features. It achieves real-time inference and strong benchmark performance, including a large RefClef improvement.
- Two-stage methods depend on proposal quality, additional detection data, and costly per-proposal computation, limiting real-time inference.If the target is missed during proposal generation, later language-region matching cannot recover it.
- RCCF reformulates referring expression comprehension as cross-modality template matching, using language as a filter kernel over image feature maps.It produces a center-point heatmap, fuses multi-level correlation maps, and regresses object size and center offset.
- 40 FPS makes RCCF real-time on a single GPU and 12-times faster than two-stage methods.
- RCCF can be trained using referring expression data alone, without additional object detection data.The one-stage design also avoids error accumulation from the object detector.
- 34.70% to 63.79%: RCCF almost doubles state-of-the-art performance on RefClef.
2. Related Work
Related work frames referring expression comprehension as a two-stage proposal-and-ranking problem, while RCCF introduces cross-modality correlation filtering as a direct one-stage alternative.
- Referring Expression Comprehension: Conventional methods generate object proposals first and then select the region best matching the referring expression.
- Referring Expression Comprehension: Two-stage methods are constrained by object detectors and spend substantial time generating proposals and extracting features for each proposal.
- Correlation Filtering: Correlation filtering uses a template to discriminate or localize matching visual content and has been applied across computer vision tasks.
- Cross-modality Correlation Filtering: RCCF analogizes language-based grounding to forming a sentence template and activating corresponding visual regions through filter-based matching.
- Cross-modality Correlation Filtering: RCCF solves referring expression comprehension with cross-modality correlation filtering in a single-stage joint optimization framework.
3. Method
RCCF reformulates referring expression comprehension as one-stage cross-modality template matching: language generates filters that localize an object center, while regression predicts its size and offset. The framework combines encoded language and multi-level visual features for correlation filtering, then forms the bounding box from the resulting center, size, and offset predictions.
- Framework: RCCF directly localizes the described object without proposal generation by treating the language feature as a visual-domain filter kernel.The method first identifies the target center through correlation filtering, then regresses object size and center-point offset.
- Size and Offset Regression: The target box is represented by center coordinates, object size, and local offset, which together recover the object region.The predicted local offset compensates for discretization error caused by the output stride.
- Expression and Image Encoder: The framework encodes expressions and images, maps the language feature into the visual domain, and applies correlation filtering to image features.The encoder uses a Bi-LSTM for expression features and a DLA-based image encoder with deformable convolution.
- Size and Offset Regression: Size and offset regression uses only the highest-resolution visual feature, with parallel convolutional branches and losses applied at the ground-truth center.The branches omit expression features to preserve spatial information; training uses L1 losses for size and offset.
- Cross-modality Correlation Filtering: Three language-generated 64-D kernels filter three visual-feature levels, whose correlation maps are pixel-wise averaged before center-point selection.The highest-scoring location in the fused heatmap is taken as the target object center.
- Loss and Inference: During inference, the highest-confidence heatmap location supplies the center, while corresponding size and offset predictions determine the box corners.The final loss is a weighted sum of center, size, and offset losses, with λsize = 0.1 and λoff = 1.
4. Experiments
RCCF is evaluated across four referring-expression benchmarks, with comparisons covering accuracy, component effectiveness, inference efficiency, and qualitative behavior. It achieves strong benchmark performance and real-time inference, while ablations identify trade-offs among backbone depth, fusion, and language-filter design.
- Experimental setting: Experiments evaluate RCCF on RefClef, RefCOCO, RefCOCO+ and RefCOCOg using Prec@0.5, where predictions require IoU greater than 0.5.RefCOCO and RefCOCO+ include train, val, testA, and testB subsets with different object compositions; RefCOCOg follows its released split.
- Comparison to the state-of-the-art: 63.79% versus 34.70%: RCCF almost doubles state-of-the-art precision on RefClef.The comparison uses the RefClef benchmark results reported in Table 2.
- Comparison to the state-of-the-art: RCCF outperforms existing methods on all RefCOCO and RefCOCO+ evaluation sets and achieves comparable performance to the state of the art on RefCOCOg.It is slightly inferior to MAttNet on RefCOCOg, whose advantage is partly attributed to additional proposal supervision and a more complex backbone.
- Ablation studies: Average fusion outperforms maximum fusion and concatenation, while diverse language kernels improve matching across different visual-feature levels.A 3×3 language filter performs similarly to the 1×1 filter but adds computational cost; a single language filter reduces precision by about 3 points.
- Efficiency analysis: A deeper Hourglass-104 backbone improves performance only slightly but is much slower to train and infer than the DLA-34 basic setting.More than 100 hours are needed for training with the deeper network.
- Efficiency analysis: 25ms per image: RCCF is the only real-time method reported and is 12 times faster than MAttNet’s 314ms inference time.MAttNet’s first and second stages take 262ms and 52ms respectively, each longer than RCCF’s total inference time.
5. Conclusion and Future Works
RCCF provides a real-time, high-performance framework that directly localizes objects from expressions and reaches state-of-the-art performance across four referring expression datasets.
- RCCF directly localizes the object by predicting its center from a correlation map between the referring expression and image.
- RCCF achieves state-of-the-art performance on RefClef, RefCOCO, RefCOCO+ and RefCOCOg at real-time speed.
- Future work will seek to capture more expression and image context and use easier-to-annotate datasets, such as object detection and image captioning data.