Source-linked AI summary
Cross-Modal Self-Attention Network for Referring Image Segmentation
Linwei Ye, Mrigank Rochan, Zhi Liu, Yang Wang
TL;DR
Referring image segmentation must map an image and free-form language expression to the referred entity’s segmentation mask, while prior approaches insufficiently capture long-range cross-modal correlations. The paper proposes cross-modal self-attention with gated multi-level fusion, and reports state-of-the-art performance on all four benchmark datasets. Its failure cases remain associated with language ambiguity, similar object appearance, and occlusion.
Problem
Prior methods may miss detailed information from individual words and global interactions between linguistic and visual features needed for referring image segmentation.
Method
The model combines cross-modal self-attention for long-range visual-linguistic dependencies with gated multi-level fusion that selectively integrates features from different image levels.
Results
The proposed network achieves state-of-the-art results on all four benchmark datasets.
Takeaways & Limitations
The approach adaptively focuses on informative words and important image regions while using multi-level features for precise referring segmentation masks.
Takeaways & Limitations
Failures arise from ambiguous language, similar object appearance, and occlusion; some may potentially be addressed with object detectors.
Abstract
from arXiv · showhide
We consider the problem of referring image segmentation. Given an input image and a natural language expression, the goal is to segment the object referred by the language expression in the image. Existing works in this area treat the language expression and the input image separately in their representations. They do not sufficiently capture long-range correlations between these two modalities. In this paper, we propose a cross-modal self-attention (CMSA) module that effectively captures the long-range dependencies between linguistic and visual features. Our model can adaptively focus on informative words in the referring expression and important regions in the input image. In addition, we propose a gated multi-level fusion module to selectively integrate self-attentive cross-modal features corresponding to different levels in the image. This module controls the information flow of features at different levels. We validate the proposed approach on four evaluation datasets. Our proposed approach consistently outperforms existing state-of-the-art methods.
1. Introduction
Referring image segmentation combines vision and language to produce a mask for an entity described by free-form text, but prior methods insufficiently model global multimodal interactions. The paper introduces cross-modal self-attention and gated multi-level fusion to improve feature interaction and segmentation.
- Motivation: Referring image segmentation produces a pixel-level mask for entities described by free-form expressions containing attributes, actions, and spatial relationships.The task lies at the intersection of computer vision and natural language processing and supports applications such as interactive photo editing and human-robot interaction.
- Limitations of prior work: A common CNN-RNN approach separately represents the image and expression before concatenation, potentially losing fine details from individual words.Such details can be important for producing an accurate segmentation mask.
- Limitations of prior work: Sequential multimodal methods merge visual features with language representations locally at each step, which may miss global interaction information needed for semantic understanding and segmentation.Their interaction considers linguistic and visual information individually within local contexts.
- Proposed approach: The CMSA module learns long-range dependencies across visual and linguistic features while focusing on important image regions and informative language keywords.Its illustrated operations include self-attention over language, self-attention over image representations, and cross-modal attention between modalities.
- Proposed approach: A gated multi-level fusion module selectively integrates self-attentive features from different image levels to refine the referred entity’s segmentation mask.The gates control how information from multiple feature levels is leveraged.
2. Related Work
Related work spans semantic segmentation, referring expression understanding, and self-attention. This paper differs by using cross-modal self-attention to connect language and vision directly.
- Semantic segmentation: Semantic segmentation research uses fully convolutional networks for end-to-end training and methods such as dilated convolution and atrous spatial pyramid pooling for broader context.These approaches address down-sampling and semantic-context limitations in segmentation.
- Referring image segmentation: Referring image segmentation extends referring expression tasks from bounding-box retrieval to pixel-level masks using CNN and LSTM features.Later work improves word-to-image interaction by combining visual features with each language-LSTM word feature.
- Self-attention: Self-attention models relations among words or long-term dependencies across video frames, whereas CMSA bridges attention across language and vision.The distinction is the explicit cross-modal connection between linguistic and visual representations.
3. Our Model
The model builds multimodal features for each word and image location, uses CMSA to capture long-range linguistic–visual dependencies, and applies gated multi-level fusion for segmentation.
- Our Model: The architecture combines multimodal features, cross-modal self-attention, and gated multi-level fusion to produce a final segmentation mask.Multimodal features combine image, spatial-coordinate, and per-word language information; attention is applied at each level before fusion.
- 3.1. Multimodal Features: Each multimodal feature concatenates visual features, a word vector, and an 8-D spatial-coordinate feature for one image location and word.The resulting collection has dimension N × H × W × (Cv + Cl + 8).
- 3.2. Cross-Modal Self-Attention: CMSA learns correlations between every word–location pair and other word–location pairs, capturing long-range dependencies while producing an output independent of expression length.Queries, keys, and values are generated by learned linear transformations, and attention scores describe correlations across spatial locations and words.
- 3.2. Cross-Modal Self-Attention: The CMSA output is transformed, added through a residual connection, and average-pooled across words to form a spatial feature map.This preserves compatibility with the backbone while removing the word dimension from the final representation.
- 3.3. Gated Multi-Level Fusion: Gated fusion integrates cross-modal self-attentive features from three ResNet levels after aligning their channel dimensions.The levels correspond to Res3, Res4, and Res5, and 1 × 1 convolutions make their channel dimensions consistent before fusion.
4. Experiments
Experiments evaluate the model on four datasets, compare attention and fusion components through UNC ablations, and use visualizations to inspect behavior and failures.
- Datasets and setup: The evaluation uses IoU and prec@X, with datasets differing in expression restrictions, length, sources, and annotated object or stuff classes.UNC includes location-capable expressions; UNC+ excludes location words; G-Ref has longer expressions; ReferIt includes stuff annotations.
- Quantitative results: The proposed method consistently outperforms existing methods across all four datasets, with especially significant improvements on UNC+ and G-Ref.These datasets respectively exclude location words and contain longer, richer query expressions.
- Ablation study: Cross-modal self-attention significantly outperforms no, word, pixel, and word-pixel pair attention methods on the UNC ablation.The ablation compares attention variants by zeroing respective components in Eq. 2.
- Ablation study: Cross-modal self-attentive feature approaches outperform other multimodal representations, while CMSA-W outperforms the sentence-based CMSA-S variant.The comparison uses the Res5 output without multi-level fusion for multimodal representation testing.
- Ablation study: The gated multi-level fusion module outperforms alternative multi-scale fusion methods in the UNC ablation.Table 3 compares the proposed fusion against Deconv, PPM, ConvLSTM, and a self-gated method under a shared base model and DenseCRF postprocessing.
- Visualization and failure cases: Visualizations show query-responsive spatial activations and word attention across levels, while failures arise from language ambiguity, similar appearance, and occlusion.The model responds to categories, locations, relationships, and appearance phrases, but some failures may potentially be addressed with object detectors.
5. Conclusion
The proposed network combines cross-modal self-attention with gated multi-level fusion to address key challenges in referring image segmentation. It achieves state-of-the-art results on all four benchmark datasets.
- Cross-modal self-attention captures long-range dependencies between visual and linguistic modalities, improving multimodal representations for referred entities.
- Gated multi-level fusion adaptively integrates features from different image levels through learnable gates for each level.This selectively combines multi-level features for precise segmentation masks.
- The proposed network achieves state-of-the-art results on all four benchmark datasets.