Source-linked AI summary
Locate then Segment: A Strong Pipeline for Referring Image Segmentation
Ya Jing, Tao Kong, Wei Wang, Liang Wang, Lei Li, Tieniu Tan
TL;DR
Referring image segmentation must both locate the region described by language and generate an accurate object mask. LTS decouples these steps by predicting a position prior before lightweight segmentation, and it outperforms prior state-of-the-art methods across three benchmarks. The authors conclude that explicit position modeling provides a strong baseline, although mask quality remains below ground truth.
Problem
Referring image segmentation requires identifying the language-referred image region and generating a fine mask, while prior methods do not explicitly model referent localization.
Method
LTS fuses visual and linguistic features, predicts an object position prior through cross-modal interaction, and uses that prior with the fused features to generate the segmentation mask.
Results
+3.2% on RefCOCO+ and +3.4% on RefCOCOg; LTS outperforms previous state-of-the-art methods on three challenging benchmarks.
Takeaways & Limitations
Explicitly modeling the position prior yields higher segmentation performance and supports a simple, effective referring image segmentation baseline.
Takeaways & Limitations
The predicted mask quality remains far from ground truth, and the method uses only simple visual and linguistic feature-extraction backbones.
Abstract
from arXiv · showhide
Referring image segmentation aims to segment the objects referred by a natural language expression. Previous methods usually focus on designing an implicit and recurrent feature interaction mechanism to fuse the visual-linguistic features to directly generate the final segmentation mask without explicitly modeling the localization information of the referent instances. To tackle these problems, we view this task from another perspective by decoupling it into a "Locate-Then-Segment" (LTS) scheme. Given a language expression, people generally first perform attention to the corresponding target image regions, then generate a fine segmentation mask about the object based on its context. The LTS first extracts and fuses both visual and textual features to get a cross-modal representation, then applies a cross-model interaction on the visual-textual features to locate the referred object with position prior, and finally generates the segmentation result with a light-weight segmentation network. Our LTS is simple but surprisingly effective. On three popular benchmark datasets, the LTS outperforms all the previous state-of-the-art methods by a large margin (e.g., +3.2% on RefCOCO+ and +3.4% on RefCOCOg). In addition, our model is more interpretable with explicitly locating the object, which is also proved by visualization experiments. We believe this framework is promising to serve as a strong baseline for referring image segmentation.
1. Introduction
Referring image segmentation must identify the image region described by language and produce a fine mask, despite expressions containing attributes, actions, and spatial relations. LTS addresses these needs by explicitly predicting an object position prior before segmentation and achieves strong benchmark results.
- Motivation: Referring image segmentation generates a mask for the object referred to by a natural-language expression, supporting applications such as interactive image editing and language-guided human-robot interaction.The task is harder than traditional semantic segmentation because image and language have a semantic gap, while expressions may specify properties, actions, and positions.
- Problem: The model must both identify the most discriminative image region corresponding to the language and generate a fine segmentation result.Existing approaches commonly fuse visual and textual features to produce a coarse mask, then apply post-processing for refinement.
- Approach: LTS decouples the task into referring-object position prediction followed by object segmentation-mask generation.The predicted object prior provides visual positional guidance to the subsequent segmentation module.
- Approach: The method first fuses visual and linguistic features, obtains an object prior through localization, and concatenates that prior with cross-modal features for lightweight convolutional segmentation.The architecture is illustrated in Figure 2, while the supplied figure passage describes the feature extraction, fusion, localization, and refinement stages.
- Results: +3.2% on RefCOCO+ and +3.4% on RefCOCOg; LTS outperforms previous state-of-the-art methods on three challenging benchmarks.The authors also report that ablation studies verify the effectiveness of each component.
2. Related Work
Prior work develops object segmentation, referring localization, and cross-modal interaction separately, while this paper connects visual-language relevance modeling to referring segmentation. Its related methods motivate the proposed use of filtering and transformer-based cross-modal interaction.
- Object Segmentation: Object segmentation research includes FCN-based end-to-end models, atrous convolution for larger receptive fields, and pyramid pooling for multi-scale semantic information.These methods provide general segmentation components relevant to producing object masks.
- Referring Localization and Segmentation: Referring image localization predicts a referred object with a bounding box, whereas referring image segmentation localizes it with a segmentation mask.MAttNet is described as decomposing expressions into subject, location, and relationship for matching.
- Proposed Architecture: The architecture extracts visual and linguistic features with a ConvNet and bi-GRU, fuses them, predicts an object position prior, and refines the final mask convolutionally.The figure description presents this as the method’s sequence from feature extraction through position-prior generation to segmentation.
- Cross-Modal Interaction: Attention mechanisms extract visual contents corresponding to a language expression, and relevance filtering provides a simple attention mechanism used across computer-vision tasks.The related discussion connects cross-modal attention with image-language relationship modeling.
- Cross-Modal Interaction: The proposed interaction uses the unified attention-based transformer building block to obtain cross-modal relevance without designing complex attention models.This differs from prior cross-modal attention models that adaptively focus on image regions and informative language keywords.
3. Proposed Approach
LTS extracts visual and linguistic features, fuses them into multimodal representations, localizes the referred object through a position prior, and refines that prior into a fine segmentation mask.
- Locate-Then-Segment: LTS separates referring object position prediction from object segmentation mask generation.The localization module supplies visual positional guidance to the subsequent segmentation module.
- Feature Extraction and Fusion: The model extracts image features with ConvNets and sentence features with GRUs before constructing multimodal representations.
- Localization: Relevance filtering generates a language-guided kernel and applies it to the fusion feature to produce a heatmap whose higher responses indicate regions more likely referred by the expression.
- Localization: The transformer alternative converts the multimodal feature into a response map using multi-headed attention and fixed positional encodings.
- Segmentation: The segmentation module concatenates the multimodal feature with the position prior, applies ASPP, and upsamples the resulting mask by deconvolution.ASPP captures image context at multiple scales, while deconvolution increases the feature-map resolution.
4. Experiments
Experiments evaluate LTS on three referring-segmentation benchmarks using IoU and prec@X, comparing it with state-of-the-art methods and component variants. Results show strong quantitative performance, effective localization and refinement, and benefits from the full pipeline and several design extensions.
- Experimental Setup: The model is evaluated on RefCOCO, RefCOCO+ and RefCOCOg using IoU and prec@X.prec@X measures the percentage of test images whose IoU exceeds thresholds from 0.5 to 0.9.
- Main Results: The model achieves the best IoU performances across the three datasets without time-consuming DenseCRF or ASNLS post-processing.Using relevance filtering once, it exceeds CGAN by about 3% absolute IoU on RefCOCO+ and RefCOCOg.
- Main Results: The pipeline outperforms CMPC and LSCM by explicitly modeling object position before segmentation.Compared with DMN and Lang2seg, it uses one sentence-generated kernel and one filtering operation, while the heatmap guides mask generation.
- Qualitative Results: 81.74% of localization heatmaps place their maximum inside the ground-truth mask, while the segmentation module refines these coarse locations into precise masks.The heatmap is not itself an accurate segmentation mask; refinement uses objects and image context at multiple scales.
- Qualitative Results: Removing segmentation, fusion, or filtering produces weaker masks, while the full model yields more obvious object shapes and finer outlines than MCN.The model without segmentation localizes the referent but does not produce a fine object mask.
- Ablation Experiments: Transformer localization, larger input resolution, and the combined enhanced setting each improve performance over the baseline configuration.The paper retains 416×416 inputs for fair comparison with prior methods.
- Ablation Experiments: +0.61 IoU is obtained when relevance filtering is applied twice instead of once.The standard experiments use one filtering operation for simplicity.
5. Conclusion
The method addresses referring image segmentation by explicitly predicting the referred object's position before generating its fine mask. It achieves higher segmentation performance than previous best results, while mask quality remains below ground truth.
- The task requires both discovering the language-relevant image area and generating an accurate object mask.
- The approach decouples segmentation into referring object prior prediction followed by fine object segmentation mask generation.
- Explicitly modeling position prior produces much higher segmentation performance than previous best results.
- Extensive ablation studies verify the effectiveness of each method component.
- Mask quality remains far from ground truth despite higher IoUs than previous works.