Source-linked AI summary
LAVT: Language-Aware Vision Transformer for Referring Image Segmentation
Zhao Yang, Jiaqi Wang, Yansong Tang, Kai Chen, Hengshuang Zhao, Philip H. S. Torr
TL;DR
Referring image segmentation must align free-form language with relevant image positions to produce an object mask. LAVT performs early language-visual fusion inside a hierarchical vision Transformer encoder, enabling a lightweight mask predictor. It reports new state-of-the-art results across RefCOCO, RefCOCO+, and G-Ref, including overall IoU gains of up to 8.57 absolute points.
Problem
Referring image segmentation must exploit visual features relevant to free-form text conditions when predicting the described object's pixel-wise mask.
Method
LAVT jointly encodes linguistic and visual features through pixel-word attention and language pathways within a hierarchical vision Transformer encoder.
Results
LAVT achieves new state-of-the-art results on RefCOCO, RefCOCO+, and G-Ref, with overall IoU improvements of 7.08%, 6.64%, 6.84%, and 8.57% across four validation settings.
Takeaways & Limitations
Language-aware visual encoding can replace complicated post-encoding cross-modal decoding with a lightweight mask predictor for referring image segmentation.
Abstract
from arXiv · showhide
Referring image segmentation is a fundamental vision-language task that aims to segment out an object referred to by a natural language expression from an image. One of the key challenges behind this task is leveraging the referring expression for highlighting relevant positions in the image. A paradigm for tackling this problem is to leverage a powerful vision-language ("cross-modal") decoder to fuse features independently extracted from a vision encoder and a language encoder. Recent methods have made remarkable advancements in this paradigm by exploiting Transformers as cross-modal decoders, concurrent to the Transformer's overwhelming success in many other vision-language tasks. Adopting a different approach in this work, we show that significantly better cross-modal alignments can be achieved through the early fusion of linguistic and visual features in intermediate layers of a vision Transformer encoder network. By conducting cross-modal feature fusion in the visual feature encoding stage, we can leverage the well-proven correlation modeling power of a Transformer encoder for excavating helpful multi-modal context. This way, accurate segmentation results are readily harvested with a light-weight mask predictor. Without bells and whistles, our method surpasses the previous state-of-the-art methods on RefCOCO, RefCOCO+, and G-Ref by large margins.
1. Introduction
Referring image segmentation predicts a pixel-wise mask for an object described in natural language, requiring visual features to exploit text-relevant positions. LAVT addresses limitations of post-encoding cross-modal fusion through language-aware visual encoding and achieves state-of-the-art results on three datasets.
- Task and challenge: Referring image segmentation predicts a pixel-wise mask for an object specified by a natural-language expression.The task handles free-form vocabulary and syntactic variety, including entities, actions, attributes, and positions.
- Task and challenge: The central challenge is exploiting visual features relevant to the given text conditions.The target object is inferred from a free-form expression containing concepts organized by syntactic rules.
- Motivation: Previous methods independently encode vision and language, then use a cross-modal decoder to fuse features after encoding.Transformer decoders improved cross-modal alignment, but fusion remained confined to the post-encoding stage.
- Proposed approach: LAVT integrates linguistic features into visual features at intermediate vision-Transformer stages for hierarchical language-aware visual encoding.Pixel-word attention densely incorporates linguistic information, which subsequent Transformer blocks exploit as multi-modal context.
- Proposed approach: A lightweight mask predictor replaces the complicated cross-modal decoder because LAVT produces language-aware visual features suitable for segmentation.The approach jointly embeds linguistic and visual features during visual encoding rather than relying solely on a decoder for alignment.
- Results: 72.73%, 62.14%, 61.24%, and 60.50% overall IoU were achieved on RefCOCO, RefCOCO+, G-Ref UMD, and G-Ref Google validation sets, respectively.The corresponding improvements over the state of the art were 7.08%, 6.64%, 6.84%, and 8.57% absolute, respectively.
2. Related work
Related work primarily separates visual and language encoding from subsequent multi-modal fusion, using concatenation, recurrence, attention, graph reasoning, and syntax-guided modeling. LAVT instead performs early fusion within a hierarchical vision Transformer encoder and avoids a complicated cross-modal decoder.
- Conventional pipelines: Conventional referring-segmentation pipelines separately encode text and images before fusing multi-modal features to predict a mask.Prior visual encoders include convolutional networks, DeeplabV3, and DarkNet, while language encoders include recurrent networks and language Transformers.
- Multi-modal fusion: Prior fusion methods use concatenation, recurrent interaction, attention mechanisms, sentence-structure reasoning, and syntactic structure-guided context aggregation.These methods model cross-modal relations or capture concepts such as categories, attributes, and relations.
- Related methods: VLT uses a Transformer decoder for linguistic-visual fusion, whereas EFN uses a convolutional vision backbone to encode language information.Both are identified as methods most closely related to LAVT.
- LAVT distinction: LAVT differs from VLT by using early fusion to exploit the Transformer encoder for multi-modal context modeling.It differs from EFN by avoiding reliance on a complicated cross-modal decoder.
- Transformers in vision-language tasks: Transformer research spans vision-language applications including contrastive pretraining, unified multi-task learning, visual question answering, and text-to-video retrieval.The related-work discussion notes few attempts to design a unified Transformer model specifically for referring image segmentation.
3. Method
LAVT performs language-aware visual encoding by integrating linguistic features into a hierarchical vision Transformer through pixel-word attention and gated information flow. A lightweight top-down predictor then combines multi-scale multimodal features to produce the segmentation mask.
- Language-aware visual encoding: LAVT jointly embeds language and vision within a hierarchical vision Transformer instead of relying on post-encoding cross-modal fusion.The backbone has four stages, each containing Transformer layers, multimodal fusion, and a learnable gating unit.
- Pixel-word attention module: PWAM uses visual features as queries and linguistic features as keys and values to generate position-specific linguistic maps aligned with visual locations.The resulting Gi maps have the same spatial dimensions as Vi and aggregate language across the word dimension.
- Pixel-word attention module: PWAM forms multimodal feature maps by element-wise multiplying the linguistic maps with a projection of the visual features, followed by a final projection.The visual and final multimodal projections use 1×1 convolutions and ReLU nonlinearities.
- Language pathway: The language pathway uses a gate to rescale multimodal features before adding them to Transformer features, allowing adaptive linguistic flow without overwhelming visual signals.The gate is a two-layer perceptron with a 1×1 convolution, ReLU, a second 1×1 convolution, and tanh; summation preserves pretrained visual initialization more effectively than replacement or concatenation.
- Implementation: The implementation uses a BERT language encoder and Swin Transformer layers initialized with ImageNet-22K classification weights.The model is implemented in PyTorch and optimized with cross-entropy loss using AdamW.
4. Experiments
Experiments evaluate LAVT on three referring image segmentation benchmarks using standard IoU and precision metrics, including dataset comparisons, ablations, and qualitative analyses. LAVT consistently outperforms prior methods, while component studies support language-aware encoding and pixel-word attention.
- Datasets and metrics: LAVT is evaluated on RefCOCO, RefCOCO+, and G-Ref, whose expressions range from succinct descriptions to more complex language.The datasets contain 19,994, 19,992, and 26,711 images, respectively; RefCOCO and RefCOCO+ expressions average 3.5 words, whereas G-Ref expressions average 8.4 words.
- Datasets and metrics: The experiments report overall IoU, mean IoU, and precision at IoU thresholds of 0.5, 0.7, and 0.9.Overall IoU favors large objects, mean IoU weights samples equally, and precision measures the share of samples exceeding a threshold.
- Comparison with others: LAVT outperforms previous methods across all evaluation subsets of RefCOCO, RefCOCO+, and G-Ref using overall IoU.Against the second-best methods, margins include 7.08%, 7.53%, and 6.06% on RefCOCO; 6.64%, 9.18%, and 5.74% on RefCOCO+; and 6.84%, 5.44%, and 8.57% on reported G-Ref subsets.
- Ablation study: PWAM outperforms grouped attention and bidirectional cross-modal attention across all reported metrics under matched backbone, language model, and training settings.The comparison includes a computationally heavy attention module and a recent top-performing grouped-attention module.
- Ablation study: Qualitative visualizations show higher-level feature maps locating text-relevant concepts, while low-level maps retain boundary information for binary segmentation.Removing LP or PWAM causes false negatives on the target bus’s front window; removing LP also produces a false positive on the middle bus.
- Ablation study: Under matched experimental settings, adding VLT’s cross-modal Transformer decoder after LAVT encoding generally provides no additional gain.The only reported exception is a marginal 0.11% improvement in P@0.5.
5. Conclusion
LAVT jointly encodes multimodal inputs by leveraging the multi-stage design of a vision Transformer, and experiments demonstrate advantages over the state of the art on three benchmarks.
- LAVT jointly encodes multimodal inputs using the multi-stage design of a vision Transformer.
- Experimental results on three benchmarks demonstrate LAVT’s advantage over the state of the art.
A. Potential biases of the language model
The employed pre-trained language model BERT has been reported to contain ethnic biases that may raise societal concerns.
- BERT, the pre-trained language model used by LAVT, has been reported to contain ethnic biases of potential societal concern.Referenced studies analyze racial, gender, geological, and other forms of ethnic bias and propose mitigation methods.
B. The language pathway
The language pathway is designed to help vision Transformer layers embed multimodal information effectively through a residual connection, with a language gate inspired by learnable information-flow regulation.
- The language pathway uses a residual connection to combine features containing different types of information.The design choice is motivated by prior evidence that residual connections are effective for combining heterogeneous features in deep neural networks.
- The language gate design is inspired by learnable gates that regulate information flow in deep neural networks.
- Table 5 identifies LG as shorthand for language gate and uses an en dash to mark extremely slow training convergence.
C. Precision-recall analysis
The precision-recall analysis evaluates LAVT and two ablated models across thresholds on RefCOCO validation data; the full model achieves the best trade-off, while LP slightly exceeds PWAM in precision up to about 0.8 recall.
- Average precision and recall are computed for all validation samples at 100 evenly spaced thresholds from 0 to 1.A pixel is classified positive when its softmax-normalized object-class score exceeds the threshold.
- The full model obtains the best precision-recall trade-off compared with the ablated models.
- LP maintains a slight precision advantage over PWAM up to approximately 0.8 recall.
D. Mean IoU
Mean IoU evaluates LAVT across the three benchmark datasets while weighting each object equally rather than favoring large objects. Supplementary visualizations show predicted masks and intermediate feature maps, including successful and failed RefCOCO validation examples.
- Mean IoU: Mean IoU treats each object equally and does not favor large objects, complementing the overall IoU reported in the main paper.The paper recommends greater use of mean IoU for future evaluation of referring image segmentation.
- Feature visualizations: Figure 8 visualizes predicted masks and the Y4, Y3, Y2, and Y1 feature maps used for final classification on RefCOCO validation examples.The visualizations also identify the language pathway and pixel-word attention module.
- Prediction visualizations: Figure 9 compares predicted and ground-truth masks on RefCOCO validation examples, marking successful cases in green and failed cases in red.Some successful predictions are nearly identical to, or more accurate than, the ground-truth annotation.