Source-linked AI summary
Encoder Fusion Network with Co-Attention Embedding for Referring Image Segmentation
Guang Feng, Zhiwei Hu, Lihe Zhang, Huchuan Lu
TL;DR
Referring image segmentation must match language with visual features, but prior decoder-side fusion does not provide continuous guidance across multi-scale features. CEFNet performs progressive encoder fusion with co-attention and boundary enhancement, and reports state-of-the-art performance on four datasets, with author-identified failures for ambiguous, rare-word, and small-object queries.
Problem
Prior methods fuse language and vision in the decoder and either use limited or independently scaled interactions, leaving deep guidance of multi-modal fused features underexplored.
Method
CEFNet transforms the visual encoder into a multi-modal encoder, progressively guides features with language, updates modalities in parallel through co-attention, and recovers detail with boundary enhancement.
Results
The approach achieves state-of-the-art performance on UNC, UNC+, Google-Ref, and ReferIt, with reported inference speed of 50 FPS on an Nvidia GTX 1080Ti GPU.
Takeaways & Limitations
Encoder-side fusion, co-attention, and boundary enhancement provide the paper’s integrated strategy for cross-modal matching and detailed referring segmentation.
Takeaways & Limitations
The method sometimes fails on ambiguous queries, low-frequency or new words, and incomplete segmentation of small objects.
Abstract
from arXiv · showhide
Recently, referring image segmentation has aroused widespread interest. Previous methods perform the multi-modal fusion between language and vision at the decoding side of the network. And, linguistic feature interacts with visual feature of each scale separately, which ignores the continuous guidance of language to multi-scale visual features. In this work, we propose an encoder fusion network (EFN), which transforms the visual encoder into a multi-modal feature learning network, and uses language to refine the multi-modal features progressively. Moreover, a co-attention mechanism is embedded in the EFN to realize the parallel update of multi-modal features, which can promote the consistent of the cross-modal information representation in the semantic space. Finally, we propose a boundary enhancement module (BEM) to make the network pay more attention to the fine structure. The experiment results on four benchmark datasets demonstrate that the proposed approach achieves the state-of-the-art performance under different evaluation metrics without any post-processing.
1. Introduction
Referring image segmentation extracts a language-specified region through pixel-wise visual-linguistic matching. CEFNet moves fusion into the encoder, progressively guides multi-level features with language, applies co-attention, and enhances boundaries, achieving state-of-the-art results on four datasets at 50 FPS.
- Task motivation: Referring image segmentation finds a particular image region from a referring expression rather than segmenting predefined categories or all instances.The task is framed as pixel-wise foreground/background segmentation guided by a language query.
- Research gap: Previous methods fuse language and vision in the decoder, either using only the highest-level visual features or independently combining language with each visual scale.These strategies do not investigate deep language guidance of multi-modal fused features.
- Proposed approach: CEFNet transforms the visual encoder into a multi-modal encoder that progressively uses language to guide multi-level cross-modal feature learning.The modalities are deeply interleaved within the CNN encoder rather than fused only at the output side.
- Proposed approach: Its co-attention mechanism simultaneously updates both modalities through a shared affinity matrix and common feature subspace to improve cross-modal matching.Vanilla and asymmetric co-attention mechanisms are implemented.
- Proposed approach: The boundary enhancement module captures contour cues and uses them to progressively recover finer details during decoding.This module directs attention toward boundary representations.
2. Related Work
Related work spans semantic and instance segmentation, referring image comprehension, and referring image segmentation. Existing referring-segmentation approaches range from direct feature concatenation to attention-based cross-modal modeling, while the proposed architecture uses encoder fusion and boundary-aware decoding.
- Segmentation paradigms: Semantic segmentation groups pixels by meaning, whereas instance segmentation separates individual object instances; encoder-decoder and boundary-aware methods address detail loss.These paradigms provide context for referring image segmentation.
- Referring image comprehension: Referring image comprehension includes localization and segmentation, with localization methods evolving from detector-based ranking to one-stage coordinate prediction.The cited approaches differ in whether candidate regions are generated before ranking or predicted directly.
- Referring image segmentation: Early referring-segmentation methods concatenated language and visual features and relied on fully convolutional networks for pixel-wise masks.These methods did not explicitly formulate intra-modal and inter-modal relationships.
- Proposed architecture: The proposed architecture combines a Bi-GRU encoder, ResNet-101 multi-scale encoder, co-attention module, decoder blocks, and boundary enhancement module.The co-attention module matches multi-modal features, while the boundary module supports detail recovery.
- Referring image segmentation: Recent methods introduce self-attention and cross-attention to model linguistic-visual relationships, including adaptive linguistic context, non-local mixed-feature updates, and bidirectional inference.The cited work explores increasingly explicit cross-modal relationship modeling.
3. Proposed Method
The method fuses language and vision inside the encoder, progressively guiding multi-level visual features through co-attention, then refines fine structures with a boundary-enhanced decoder.
- Encoder Fusion Network: EFN inserts language features after res3, res4, and res5, converting ResNet into a multi-modal feature extractor with deeply interleaved cross-modal features.This differs from decoder-only fusion by progressively guiding multi-level features during encoding.
- Language Guidance: Position-specific linguistic contexts are generated from visual features and spatial coordinates, allowing different words to be treated differently.The resulting context can suppress language noise and highlight the desired region.
- Co-Attention Fusion: Co-attention flattens visual and linguistic feature maps, computes cross-modal affinities, normalizes them, and updates both modalities through weighted summation.The affinity value represents similarity between visual and linguistic positions, while the updated features are concatenated and projected into multi-modal features.
- Co-Attention Fusion: The asymmetric co-attention module uses pyramid pooling to reduce computational cost while producing modality-specific similarity matrices with fixed dimensions.Its updated modality features are concatenated to form the final multi-modal output.
- Boundary Enhancement Module: BEM uses boundary features to guide progressive mask refinement after repeated downsampling has weakened fine structure information.The final prediction is the refined mask from the last decoder block, supervised by the ground-truth segmentation.
4. Experiments
Experiments evaluate CEFNet on four referring-image-segmentation datasets, compare it with prior methods, and test its encoder fusion, co-attention, and boundary enhancement components. The method achieves strong accuracy and 50 FPS inference while remaining post-processing free.
- Experimental Setup: The evaluation covers UNC, UNC+, Google-Ref, and ReferIt, using Overall IoU and Prec@X for segmentation accuracy.The study compares the proposed model with thirteen previous methods.
- Performance Comparison: The proposed model consistently outperforms competitors on most datasets, except UNC+ testB.It gains 5.9%, 3.4%, and 3.9% over CMPC on G-Ref, UNC+ testA, and UNC+ val, respectively, without post-processing.
- Runtime and Memory: 50 FPS is the fastest reported speed for the method on an NVIDIA GTX 1080 Ti GPU.Runtime comparisons ignore post-processing time.
- Ablation Study: EFN is significantly better than the decoder-fusion baseline DFN on the UNC dataset.The comparison isolates encoder fusion while using an FPN decoder.
- Ablation Study: The asymmetric co-attention module improves Overall IoU by 7.6%, 7.6%, and 8.8% on UNC-val, UNC-testA, and UNC-testB over EFN.It performs slightly better than vanilla co-attention, attributed to modality-specific affinity learning and improved contextual understanding.
- Ablation Study: BEM improves Overall IoU by about 2%∼3% and makes prediction masks fit object boundaries more closely.The module emphasizes contour representation and helps recover finer details.
5. Conclusion
CEFNet fuses language and vision in the encoder, using co-attention and boundary enhancement to improve multimodal matching and detail attention. Evaluations on four datasets report better performance and speed than previous state-of-the-art methods.
- CEFNet uses encoder fusion to integrate multimodal information for referring image segmentation.
- The strategy uses language to guide multimodal feature learning without increasing computational complexity.
- The co-attention module strengthens matching between multimodal features and their targeting ability.
- A boundary enhancement module directs attention toward fine-grained details.
- Evaluations on four datasets report improvements over previous state-of-the-art methods in performance and speed.