PaperlayerView exact revision on arXiv

Source-linked AI summary

VLT: Vision-Language Transformer and Query Generation for Referring Segmentation

Henghui Ding, Chang Liu, Suchen Wang, Xudong Jiang

arXiv:2210.15871v1cs.CV

TL;DR

Referring segmentation requires grounding diverse language expressions in complex images, while fixed transformer queries cannot adapt to this variability. VLT generates input-specific queries, balances their decoder responses, and applies masked contrastive learning to improve multimodal understanding. The lightweight framework achieves state-of-the-art referring segmentation results consistently across five image and video datasets.

  • Problem

    Referring segmentation must handle complex visual relationships and highly diverse language expressions, but existing transformer queries remain fixed after training.

  • Method

    VLT builds global vision-language interactions, dynamically generates multiple input-specific queries, selectively balances their responses, and applies masked contrastive learning across expressions and objects.

  • Results

    The lightweight approach achieves new state-of-the-art referring segmentation results consistently on RefCOCO, RefCOCO+, G-Ref, YouTube-RVOS, and Ref-DAVIS17.

  • Takeaways & Limitations

    VLT supports diverse interpretations of language expressions and robustness to unconstrained language by combining query diversity, response selection, and inter-sample representation learning.

Abstract

from arXiv · show

We propose a Vision-Language Transformer (VLT) framework for referring segmentation to facilitate deep interactions among multi-modal information and enhance the holistic understanding to vision-language features. There are different ways to understand the dynamic emphasis of a language expression, especially when interacting with the image. However, the learned queries in existing transformer works are fixed after training, which cannot cope with the randomness and huge diversity of the language expressions. To address this issue, we propose a Query Generation Module, which dynamically produces multiple sets of input-specific queries to represent the diverse comprehensions of language expression. To find the best among these diverse comprehensions, so as to generate a better mask, we propose a Query Balance Module to selectively fuse the corresponding responses of the set of queries. Furthermore, to enhance the model's ability in dealing with diverse language expressions, we consider inter-sample learning to explicitly endow the model with knowledge of understanding different language expressions to the same object. We introduce masked contrastive learning to narrow down the features of different expressions for the same target object while distinguishing the features of different objects. The proposed approach is lightweight and achieves new state-of-the-art referring segmentation results consistently on five datasets.

1 INTRODUCTION

Referring segmentation must connect complex visual relationships with varied language expressions, but conventional approaches struggle with global multimodal interactions and fixed or image-independent language emphasis. VLT addresses these issues with global vision-language interactions, input-specific query generation and balancing, masked contrastive learning, and a lightweight architecture that achieves state-of-the-art results across five datasets.

  • Motivation: Referring segmentation generates a mask for an object described by natural language, but complex object relationships make the multimodal task challenging.The task combines language understanding and computer vision and is demanded in applications such as image and video editing.
  • Vision-Language Transformer: VLT uses global operations to build direct interactions among words, pixels, and word-pixel pairs, addressing inefficient long-range modeling in convolutional approaches.Regular convolutions require stacked small kernels to obtain large receptive fields, making distant information exchange inefficient.
  • Inter-Sample Learning: Masked contrastive learning narrows representations of different expressions for the same object while distinguishing representations of different objects.The method explicitly models inter-sample relationships to improve robustness to unconstrained language expressions.
  • Query Generation and Balance: QGM dynamically generates multiple input-specific queries, while QBM selectively fuses their responses to represent and prioritize different language comprehensions.The queries incorporate visual information, and responses corresponding to better comprehensions are highlighted.
  • Results: The lightweight approach achieves new state-of-the-art performance consistently on RefCOCO, RefCOCO+, G-Ref, YouTube-RVOS, and Ref-DAVIS17.Its parameter scale is described as roughly equivalent to seven convolution layers.

2 RELATED WORKS

Prior work spans referring segmentation, referring comprehension, and transformer architectures, while VLT combines fully attention-based processing with input-conditional decoder queries. Its design also incorporates spatial-dynamic multimodal fusion and masked contrastive learning within the vision-language transformer framework.

  • Referring Segmentation: Related referring segmentation methods commonly use attention or FCN-like architectures to combine visual and linguistic information for mask prediction.The cited works include cross-modal self-attention, bidirectional attention, and vision-language fusion approaches.
  • Proposed Framework: VLT employs a fully attention-based architecture to model long-range image dependencies and generates input-conditional decoder queries for varied language expressions.This differs from approaches that use fixed or language-only query representations.
  • Referring Comprehension: Referring comprehension identifies a language-referred object with a grounding box, whereas referring segmentation outputs a segmentation mask.Earlier referring comprehension systems often generate language-independent instance proposals before matching them to the expression.
  • Transformer: Transformer architectures use attention for sequence-to-sequence processing and have been applied broadly across natural-language, vision, and vision-language tasks.Examples include machine translation, visual question answering, image recognition, semantic segmentation, and image-text pretraining.
  • Proposed Framework: The proposed framework adds spatial-dynamic multimodal fusion and masked contrastive learning to improve position-specific feature interaction and robustness to unconstrained expressions.Its architecture projects image and language inputs, fuses them, generates queries, balances decoder responses, and decodes the target mask.

3 METHODOLOGY

VLT integrates vision and language through spatial-dynamic fusion, input-specific query generation, query balancing, and masked contrastive learning. These components address limitations of tiled fusion and fixed queries while supporting diverse language expressions and robust mask prediction.

  • Spatial-Dynamic Multi-Modal Fusion: Spatial-Dynamic Fusion derives a tailored language feature vector for each image pixel instead of identically tiling language features across the map.It uses word-level and sentence-level language features with image features to generate spatially varying attention.
  • Query Generation Module: Fixed transformer queries cannot represent the stochastic properties of target objects expressed through unconstrained language and visual scenes.Relevant clues include expression keywords such as “blue/yellow,” “small/large,” and “right/left.”
  • Query Generation Module: The Query Generation Module produces multiple input-specific language queries with different attention weights, guided by sequential vision and language features.These query vectors are sent to the transformer decoder to represent different understandings of the expression.
  • Query Balance Module: The Query Balance Module computes confidence levels for query-response pairs and weights responses before mask decoding.Each confidence reflects how well a query fits its prediction context and controls its response’s influence.
  • Overall Framework: The proposed modules jointly improve diverse image-language understanding and robustness to highly stochastic inputs while keeping the network lightweight.The parameter scale is described as roughly equivalent to seven convolution layers.
  • Masked Contrastive Learning: Masked contrastive learning brings features of different expressions for the same object closer while separating features of different objects.The method adds randomly masked expressions as positive samples and samples different-image, same-image-different-object, and same-image-same-object relationships.

4 EXPERIMENTS

Experiments show that VLT’s transformer modules, dynamic query mechanisms, spatial-dynamic fusion, and masked contrastive learning improve referring segmentation across image and video benchmarks. The model also handles diverse expressions while remaining lightweight and incurring limited training overhead from MCL.

  • Ablation Study: The Query Generation Module improves performance by about 5%–7% IoU and 4%–6% Pr@0.5 over alternative query-generation methods.Compared methods use language features or learned parameters as queries; QGM produces attended language features under visual guidance.
  • Ablation Study: Increasing the query count from one to 16 yields approximately 8% performance gain, while removing QBM causes a 1.44% IoU drop.Performance gains slow beyond eight queries, so Nq = 16 is selected as the default setting.
  • Ablation Study: MCL improves robustness to partially specified expressions, while increasing training memory by less than 4% and training time by less than 2%.With MCL, memory and speed are 18496MB and 0.479s/iter, versus 17842MB and 0.471s/iter without it.
  • Benchmark Results: VLT outperforms previous state-of-the-art methods on RefCOCO, RefCOCO+, and G-Ref benchmarks.On RefCOCO, it achieves approximately 2% IoU gains over LTS across three testing splits.
  • Referring Video Object Segmentation: VLT achieves new state-of-the-art results on both YouTube-RVOS and Ref-DAVIS17 without video-specific modules or training losses.The comparison uses the V-Swin-B backbone, matching ReferFormer’s base-model setting.

5 CONCLUSION

The conclusion presents VLT as a transformer-based referring segmentation framework that improves long-range multimodal interaction and handles ambiguous expressions through dynamic queries and inter-sample learning. It combines spatial-dynamic fusion, query balancing, and masked contrastive learning in a lightweight network.

  • 5 CONCLUSION: VLT reformulates referring segmentation as a direct attention problem to facilitate long-range information exchange.The framework uses transformer attention operations instead of relying on conventional convolutional networks.
  • 5 CONCLUSION: Spatial-dynamic fusion produces a position-specific language feature vector from interactions between language information and image pixels.This mechanism emphasizes differences among pixels and objects during multimodal fusion.
  • 5 CONCLUSION: QGM and QBM address ambiguous expressions by generating diverse comprehensions and selecting more suitable query responses with image information.The two modules increase the diversity of ways to understand the image and query language.
  • 5 CONCLUSION: Inter-sample learning uses masked contrastive learning to improve understanding of different expressions for the same object.The conclusion identifies this learning strategy as part of VLT’s approach to diverse language expressions.
  • 5 CONCLUSION: The proposed network is lightweight, with a parameter scale roughly equivalent to seven convolutional layers.
Loading 2210.15871v1…