Source-linked AI summary
Linguistic Structure Guided Context Modeling for Referring Image Segmentation
Tianrui Hui, Si Liu, Shaofei Huang, Guanbin Li, Sansi Yu, Faxi Zhang, Jizhong Han
TL;DR
Referring image segmentation needs multimodal context that distinguishes the referred object from background, but existing methods may model that context insufficiently or redundantly. The paper introduces LSCM, which uses a dependency-tree-suppressed word graph and gather-propagate-distribute processing to select context. It reports state-of-the-art performance on four benchmarks, outperforming prior methods.
Problem
Existing referring-segmentation methods either insufficiently or redundantly model multimodal context needed to distinguish the referent from the background.
Method
LSCM builds a Dependency Parsing Tree suppressed Word Graph to guide gathering, constrained propagation, and distribution of valid multimodal context through cross-modal interaction.
Results
The method outperforms previous state-of-the-art methods on UNC, UNC+, G-Ref, and ReferIt, with reported gains of +1.58%, +3.09%, +1.65%, and +2.44%, respectively.
Takeaways & Limitations
LSCM effectively guides words to include valid sentence context while excluding disturbing context, producing more discriminative multimodal features for the referent.
Abstract
from arXiv · showhide
Referring image segmentation aims to predict the foreground mask of the object referred by a natural language sentence. Multimodal context of the sentence is crucial to distinguish the referent from the background. Existing methods either insufficiently or redundantly model the multimodal context. To tackle this problem, we propose a "gather-propagate-distribute" scheme to model multimodal context by cross-modal interaction and implement this scheme as a novel Linguistic Structure guided Context Modeling (LSCM) module. Our LSCM module builds a Dependency Parsing Tree suppressed Word Graph (DPT-WG) which guides all the words to include valid multimodal context of the sentence while excluding disturbing ones through three steps over the multimodal feature, i.e., gathering, constrained propagation and distributing. Extensive experiments on four benchmarks demonstrate that our method outperforms all the previous state-of-the-arts.
1 Introduction
Referring image segmentation must align diverse linguistic concepts with visual content while separating the referent from noisy background features. The paper proposes LSCM, using a DPT-WG and gather-propagate-distribute processing to select valid multimodal context, achieving state-of-the-art results on four benchmarks.
- Referring image segmentation predicts the foreground mask matching a natural-language description and must align entities, attributes, and relationships across language and vision.
- Existing concatenation, recurrent refinement, dynamic-filter, and cross-modal self-attention methods model multimodal context insufficiently or redundantly.
- LSCM uses linguistic structure to selectively model valid multimodal context relevant to the sentence.
- The DPT-WG guides three steps: gather word-specific context through cross-modal attention, propagate context along suppressed dependency-tree routes, and distribute word features back to the image.
- The paper also proposes Dual-Path Multi-Level Fusion, combining low-level spatial details and high-level semantic information through bottom-up and top-down paths.
- +1.58% on UNC, +3.09% on UNC+, +1.65% on G-Ref, and +2.44% on ReferIt over prior state-of-the-art methods.
2 Related Work
Related work spans semantic-segmentation context aggregation, referring-image localization and segmentation, and graph-based context modeling for vision-language tasks.
- Semantic-segmentation methods use atrous convolutions, ASPP, or pyramid pooling to enlarge receptive fields and aggregate multi-scale context.
- Referring image localization models relationships between multimodal features, while MAttNet decomposes expressions into subject, location, and relationship components.
- Cross-modal self-attention captures long-range image-region and word dependencies but introduces redundant context through dense computation.
- Graph-based approaches use structured or latent graphs to model context within visual, linguistic, or referring-expression tasks.
3 Method
The model fuses multi-level visual, linguistic, and coordinate features, then uses LSCM to gather, constrain, and redistribute multimodal context through a dependency-guided word graph before mask prediction.
- 3.1 Multimodal Feature Extraction: Multi-level visual features, word features, and coordinate features are fused into multimodal features for each visual level.The model uses CNN-extracted visual features, LSTM word features, and spatial coordinates before context modeling.
- 3.2 Linguistic Structure Guided Context Modeling: Cross-modal attention gathers spatial multimodal features into one word node for each sentence word.The normalized attention map assigns multimodal feature information to T word nodes.
- 3.2 Linguistic Structure Guided Context Modeling: A fully connected word graph is constrained with a dependency parsing tree to suppress disturbing edges during information propagation.Non-tree edges receive a reduced weight through a tree mask, with α set to 0.1.
- 3.2 Linguistic Structure Guided Context Modeling: One graph convolution layer propagates and updates word-node features, producing context aligned with the whole sentence.The graph update includes an identity shortcut connection and learned feature-update parameters.
- 3.2 Linguistic Structure Guided Context Modeling: Updated word-node features are distributed back to spatial locations and combined with visual, sentence, and coordinate features for prediction.The distributed features are concatenated with V, tiled sentence features, and P before a 1 × 1 convolution; multi-level outputs are further fused.
4 Experiments
Experiments on four benchmarks show that the proposed method consistently outperforms prior state-of-the-art methods, while ablations validate LSCM, multi-level fusion, and constrained graph propagation. Qualitative results further indicate improved expression understanding and flexible localization.
- Comparison with State-of-the-arts: The method consistently outperforms bottom-up and top-down state-of-the-art methods in Overall IoU across four benchmark datasets.The comparison uses UNC, UNC+, G-Ref, and ReferIt.
- Comparison with State-of-the-arts: Our method exceeds STEP on all splits while using fewer multimodal feature fusions, indicating stronger context modeling efficiency.The authors attribute this to capturing more valid multimodal context for visual-linguistic alignment.
- Comparison with State-of-the-arts: On ReferIt, the performance gain reaches 2.94% IoU over RRN, while on UNC+ testB the method achieves 3.09% over STEP.These results are reported for challenging settings with marginal prior improvements or no location words.
- Ablation Studies: Adding LSCM to the single-level baseline yields a 7.61% IoU gain on UNC val, validating linguistic-structure-guided multimodal context modeling.GloVe and Mutan fusion further improve performance based on the LSCM module.
- Ablation Studies: Multi-level models show consistent improvements, while Dual-Path Fusion performs best by integrating high-level semantics with low-level details.The multi-level ablations use the proposed Dual-Path Fusion module to combine four feature levels.
- Ablation Studies: One graph-convolution layer is sufficient: additional propagation muddles word contexts, whereas unconstrained propagation introduces disturbing information.The one-layer setting also outperforms zero layers, showing that communication among words remains necessary after gathering context.
5 Conclusion and Future Work
The paper introduces LSCM, a gather-propagate-distribute scheme that uses linguistic structure to model multimodal context and highlight the referent while excluding disturbing information.
- LSCM implements a gather-propagate-distribute scheme for modeling multimodal context in referring image segmentation.
- The module constructs a Dependency Parsing Tree suppressed Word Graph with one node for each word.
- The DPT-WG guides words to include valid sentence context while excluding disturbing context, highlighting multimodal features of the referent.