Source-linked AI summary
Referring Image Segmentation via Cross-Modal Progressive Comprehension
Shaofei Huang, Tianrui Hui, Si Liu, Guanbin Li, Yunchao Wei, Jizhong Han, Luoqi Liu, Bo Li
TL;DR
Referring image segmentation must align visual features with expressions containing informative entity, attribute, and relational words to identify the correct entity. The paper proposes CMPC for progressive perception and relation-aware graph reasoning, and TGFE for text-guided multi-level exchange; it reports consistent superiority over prior state-of-the-art methods on four benchmarks.
Problem
Prior referring segmentation methods use implicit visual-linguistic interaction but often fail to exploit different informative word types for accurate feature alignment and entity identification.
Method
CMPC perceives candidate entities from entity and attribute words, then uses relational words for graph-based referent highlighting; TGFE selectively integrates multi-level features using textual guidance.
Results
The model consistently outperforms previous state-of-the-art methods on four referring segmentation benchmarks.
Takeaways & Limitations
Progressively exploiting entity, attribute, and relationship cues yields discriminative referent representations, while textual guidance refines multi-level features for mask prediction.
Abstract
from arXiv · showhide
Referring image segmentation aims at segmenting the foreground masks of the entities that can well match the description given in the natural language expression. Previous approaches tackle this problem using implicit feature interaction and fusion between visual and linguistic modalities, but usually fail to explore informative words of the expression to well align features from the two modalities for accurately identifying the referred entity. In this paper, we propose a Cross-Modal Progressive Comprehension (CMPC) module and a Text-Guided Feature Exchange (TGFE) module to effectively address the challenging task. Concretely, the CMPC module first employs entity and attribute words to perceive all the related entities that might be considered by the expression. Then, the relational words are adopted to highlight the correct entity as well as suppress other irrelevant ones by multimodal graph reasoning. In addition to the CMPC module, we further leverage a simple yet effective TGFE module to integrate the reasoned multimodal features from different levels with the guidance of textual information. In this way, features from multi-levels could communicate with each other and be refined based on the textual context. We conduct extensive experiments on four popular referring segmentation benchmarks and achieve new state-of-the-art performances.
1. Introduction
Referring image segmentation must identify entities described by varied expressions containing entities, attributes, and relationships. The paper addresses limited word-aware visual-linguistic alignment with progressive comprehension and text-guided multi-level feature exchange.
- 1. Introduction: Referring image segmentation segments entities described by natural language expressions, which may contain entities, attributes, and relationships across object or stuff categories.The task supports applications including interactive image editing and language-based robot controlling.
- 1. Introduction: Previous methods use concatenation, convolution, inter-modality attention, or self-attention but do not adequately exploit different informative word types for feature alignment.The paper identifies word-aware alignment as important for comprehending both the expression and image.
- 1. Introduction: CMPC first fuses entity and attribute words with visual features to perceive candidate entities, then uses relational cues in graph reasoning to highlight the referent and suppress irrelevant entities.The motivating example uses “man,” “white frisbee,” and “holding” to progressively narrow the prediction.
- 1. Introduction: TGFE uses linguistic guidance to exchange useful information among multimodal features from different levels before ConvLSTM fusion.This communication combines low-level visual details and high-level semantics for precise mask prediction.
- 1. Introduction: The method reports new state-of-the-art results on four referring segmentation benchmarks.The contribution statement presents this as evidence of the model’s effectiveness.
2. Related Work
Related work covers semantic segmentation architectures, referring-expression comprehension and segmentation, and graph-based context reasoning. The proposed overview combines progressive cross-modal alignment, multi-level feature exchange, and ConvLSTM-based fusion.
- 2. Related Work: Semantic segmentation architectures use fully convolutional networks, atrous convolution, pyramid pooling, and self-attention to capture multi-scale or contextual visual information.These approaches provide the visual segmentation foundations discussed in related work.
- 2. Related Work: Referring expression comprehension localizes described entities, with prior methods modeling cross-modal correlations, relationships, or decomposed subject, location, and relationship components.These methods commonly target bounding-box-level localization.
- 2. Related Work: The proposed architecture progressively aligns visual and linguistic features with CMPC, exchanges multi-level multimodal information with TGFE, and fuses features with ConvLSTM for prediction.Figure 2 presents this sequence as the model overview.
- 2. Related Work: Referring segmentation extends localization to precise masks using CNN-LSTM fusion, multimodal LSTMs, word-conditioned dynamic filters, and recurrent multi-level visual refinement.Prior work progressively adds sequential fusion, word-specific enhancement, and multi-level detail refinement.
- 2. Related Work: Graph-based models support context reasoning through graph structures such as dense CRF, spatial-temporal graphs, and global reasoning modules with graph convolution.These approaches motivate graph reasoning for contextual visual representations.
3. Method
The method progressively aligns linguistic cues with visual regions to identify the referred entity, then exchanges textual guidance across feature levels for prediction.
- Visual and Linguistic Feature Extraction: The model extracts multi-level visual features with spatial coordinates and word-level linguistic features using CNN and language encoders.Visual levels are transformed to a common spatial and channel size before cross-modal processing.
- Cross-Modal Progressive Comprehension: CMPC first fuses entity and attribute words with visual regions to perceive all candidate entities in the expression.Entity and attribute words are selected through predicted word-type probabilities and combined into linguistic context for bilinear fusion.
- Cross-Modal Progressive Comprehension: CMPC then constructs a spatial multimodal graph whose relational-word affinities define adaptive edges between image regions.The adjacency matrix is formed from normalized region–relation affinities, allowing relational words to route information between graph vertices.
- Cross-Modal Progressive Comprehension: Graph convolution reasons across regions to highlight the referent according to relationship cues while suppressing irrelevant regions.The enhanced graph features are reshaped back into spatial feature maps and fused with visual and linguistic context.
- Text-Guided Feature Exchange: TGFE repeatedly communicates information among multi-level multimodal features under linguistic guidance before ConvLSTM fusion produces the final prediction.Each level selects useful channels from the other levels, mutually refining features to fit the expression context.
4. Experiments
Experiments on four referring segmentation benchmarks show that the proposed method consistently outperforms prior state-of-the-art methods, while ablations validate CMPC, TGFE, and graph-reasoning design choices. Qualitative results further show improved handling of relationships, attributes, locations, and modified expressions.
- Experimental Setup: UNC, UNC+, G-Ref, and ReferIt provide four benchmark datasets with differing expression lengths, vocabulary constraints, and image sources.UNC+ excludes location words, while G-Ref uses longer expressions averaging 8.4 words than the other datasets.
- Comparison with State-of-the-arts: The method outperforms previous state-of-the-art methods on all four benchmarks, with 1.40%-2.82% performance gains over STEP.It achieves a 2.65% IoU boost over STEP on the G-Ref validation set and a 1.40% gain on the ReferIt test set.
- Ablation Studies: Entity perception improves baseline IoU by 1.70%, while relation-aware reasoning alone improves it by 6.04% on the UNC validation set.Combining entity perception and relation-aware reasoning reaches 55.38% IoU with single-level features.
- Ablation Studies: TGFE improves IoU from 59.85% to 60.72% after one exchange round, and increasing exchange rounds further increases IoU.On the baseline model, one TGFE exchange round improves IoU from 56.38% to 58.81%.
- Ablation Studies: Graph-convolution depth has dataset-dependent effects: additional layers can help longer G-Ref expressions but too many layers introduce noise and degrade performance.On UNC, more layers degrade performance; on G-Ref, two layers outperform one, while three layers reduce performance.
- Qualitative Results: Qualitative comparisons show that the full model distinguishes relationally specified entities and handles attribute and location relationships better than the multi-level baseline.Affinity maps become more concentrated on the referent as expressions add information, and modified expressions shift responses to the newly specified entity.
5. Conclusion and Future Work
The paper addresses referring image segmentation with CMPC for progressive entity selection and relation-aware reasoning, and TGFE for text-guided multi-level feature integration. It reports consistent improvements over prior state-of-the-art methods on four benchmarks and identifies structured linguistic analysis and compact graph formulation as future directions.
- Conclusion: CMPC first perceives candidate entities using entity and attribute words, then uses relational words for graph-based reasoning to highlight the referent and suppress others.This produces discriminative representations for the referred entity.
- Conclusion: TGFE selectively integrates multi-level features using textual information to refine mask prediction.
- Conclusion: The model consistently outperforms previous state-of-the-art methods on four referring segmentation benchmarks.
- Future Work: Future work will analyze linguistic information more structurally and explore more compact graph formulations.