PaperlayerView exact revision on arXiv

Source-linked AI summary

ReSTR: Convolution-free Referring Image Segmentation Using Transformers

Namyup Kim, Dongwon Kim, Cuiling Lan, Wenjun Zeng, Suha Kwak

arXiv:2203.16768v1cs.CVcs.AI

TL;DR

Referring image segmentation must identify entities described by language, while prior CNN/RNN-based systems struggle with long-range dependencies and flexible cross-modal interactions. ReSTR is a convolution-free transformer model with adaptive multimodal fusion and coarse-to-fine decoding, and it outperforms existing techniques on public benchmarks.

  • Problem

    Referring image segmentation requires modeling long-range relations within modalities and sophisticated interactions between visual and linguistic features.

  • Method

    ReSTR uses transformer encoders for both modalities, self-attention fusion with a class seed adaptive classifier, and coarse-to-fine segmentation decoding.

  • Results

    ReSTR outperformed existing referring image segmentation techniques on all public benchmarks.

  • Takeaways & Limitations

    A convolution-free transformer architecture can perform referring image segmentation while capturing global context and flexible cross-modal interactions.

  • Takeaways & Limitations

    Semantic segmentation architectures remain limited by predefined classes and convolutional layers that constrain local context encoding.

Abstract

from arXiv · show

Referring image segmentation is an advanced semantic segmentation task where target is not a predefined class but is described in natural language. Most of existing methods for this task rely heavily on convolutional neural networks, which however have trouble capturing long-range dependencies between entities in the language expression and are not flexible enough for modeling interactions between the two different modalities. To address these issues, we present the first convolution-free model for referring image segmentation using transformers, dubbed ReSTR. Since it extracts features of both modalities through transformer encoders, it can capture long-range dependencies between entities within each modality. Also, ReSTR fuses features of the two modalities by a self-attention encoder, which enables flexible and adaptive interactions between the two modalities in the fusion process. The fused features are fed to a segmentation module, which works adaptively according to the image and language expression in hand. ReSTR is evaluated and compared with previous work on all public benchmarks, where it outperforms all existing models.

1. Introduction

Referring image segmentation addresses natural-language descriptions of specific entities, but existing CNN/RNN-based methods struggle with long-range within-modality relations and flexible cross-modal fusion. ReSTR replaces these components with transformers and an adaptive, coarse-to-fine segmentation pipeline, achieving state-of-the-art performance on public benchmarks.

  • Motivation: Referring image segmentation segments regions specified by natural-language expressions rather than a predefined class set.This supports queries describing specific entities such as a red Ferrari or a person wearing a blue hat.
  • Motivation: The task requires modeling relations among language entities and jointly reasoning over visual and linguistic modalities.Expressions may encode structured relations such as spatial arrangements among multiple entities.
  • Limitations: Existing CNN/RNN-based methods have limited long-range interaction modeling and use fixed concatenation-convolution fusion between modalities.Their local building blocks constrain within-modality dependencies, while handcrafted fusion is insufficiently flexible across scenarios.
  • ReSTR: ReSTR independently extracts image-patch and word features with transformer encoders, capturing long-range interactions and using a unified topology across modalities.The encoders take non-overlapping image patches and word embeddings as inputs.
  • ReSTR: A self-attention fusion encoder combines both modalities with a class seed embedding transformed into an adaptive classifier for the target entity.The classifier evaluates whether each image patch contains part of the referred entity.
  • Results: ReSTR achieves state-of-the-art performance on four public benchmarks without additional postprocessing.The reported comparison excludes only the UNC+ testB set from the stated across-benchmark superiority claim.

2. Related Work

Prior segmentation work relies on convolutional architectures and predefined classes, limiting contextual encoding and real-world applicability. ReSTR extends convolution-free transformer processing to referring image segmentation and uses coarse-to-fine decoding for high-resolution masks.

  • 2.1. Semantic Segmentation: Semantic segmentation methods based on FCN variants use convolutional layers and predict masks for predefined classes.Earlier work addressed resolution and context through high-resolution prediction, dilated convolutions, and feature pyramid pooling.
  • 2.1. Semantic Segmentation: These convolutional architectures limit local context encoding and constrain application to real-world tasks involving undefined or user-specified entities.Referring image segmentation removes the predefined-class restriction by conditioning the mask on a natural-language expression.
  • 2.2. Referring Image Segmentation: Referring image segmentation groups pixels into a mask corresponding to a given natural-language expression.Existing approaches commonly extract visual and linguistic features with CNNs and RNNs before multimodal fusion.
  • 2.2. Referring Image Segmentation: Transformer-based studies have been used for multimodal fusion to capture long-range visual-linguistic interactions.This line of work motivates transformer processing for the referring segmentation problem.
  • 2.2. Referring Image Segmentation: ReSTR introduces a convolution-free architecture that encodes contextual information throughout the model and converts patch predictions into high-resolution masks coarsely to finely.Its design uses transformers for referring image segmentation rather than only as an attention component within CNNs.

3. Proposed Method

ReSTR uses transformer encoders to model long-range dependencies within vision and language, then fuses both modalities with self-attention and generates an adaptive classifier for segmentation.

  • Visual and Linguistic Feature Extraction: Transformer encoder layers combine multiheaded self-attention, layer normalization, and multilayer perceptron blocks.The encoder applies sequential transformer blocks to modality-specific features.
  • Visual and Linguistic Feature Extraction: ReSTR independently encodes image patches and word embeddings with transformer encoders to capture long-range interactions within each modality.The vision encoder processes non-overlapping image patches, while the language encoder processes word embeddings with positional encoding.
  • Multimodal Fusion Encoder: The multimodal fusion encoder uses visual-linguistic and linguistic-seed transformer encoders to interactively combine modalities and process a trainable class seed embedding.Visual and linguistic features are projected to a common channel dimension before fusion, while the seed embedding is initialized randomly.
  • Multimodal Fusion Encoder: The linguistic-seed encoder transforms the class seed embedding into an adaptive classifier for the target entity described by the language expression.A fixed classifier is insufficient because the target mask varies with the referring expression.
  • Multimodal Fusion Encoder: Patch-level predictions are computed from inner products between multimodal patch features and the adaptive classifier, followed by sigmoid normalization.The prediction uses the normalized product z′_v e′_s^T / sqrt(D).
  • Coarse-to-Fine Segmentation Decoder: The decoder masks multimodal features with patch predictions, concatenates them with visual features, and progressively upsamples to produce the final pixel-level map.Its sequential blocks upsample by factor 2, reduce channels by 1/2, and apply activation functions; inference uses only the pixel-level prediction.
  • Training: Training combines binary cross-entropy losses for patch-level and pixel-level predictions using a balancing hyperparameter.The total loss is λL_b(ŷ_p,y_p) + L_b(Ŷ_m,Y_m).

4. Experiments

Experiments evaluate ReSTR across four referring image segmentation benchmarks and analyze fusion variants, decoder depth, weight sharing, qualitative outputs, and computational cost. ReSTR generally outperforms prior methods while maintaining efficient computation, with CME providing the strongest fusion design among tested variants.

  • Benchmark evaluation: Experiments use ReferIt, UNC, UNC+, and Gref, covering four widely used referring image segmentation benchmarks.The datasets contain images, language expressions, and segmentation masks collected from IAPR TC-12 or COCO.
  • Benchmark evaluation: ReSTR outperforms previous methods on all public benchmarks except the UNC+ testB set, without inefficient DenseCRF postprocessing.On Gref, its performance also declines less as referring expressions become longer than ACM's, dropping 6.81%p versus 13.71%p.
  • Fusion encoder analysis: CME surpasses VME and IME in performance and efficiency, while weight-shared CME retains better performance with fewer parameters.VME's class-seed attention is biased toward visual features because visual and linguistic sequence lengths differ substantially, whereas CME mediates interactions through linguistic features.
  • Ablation studies: Four fusion-encoder transformer layers provide a substantial performance improvement, while six layers yield only a marginal additional gain.The ablation varies the multimodal fusion encoder between 2, 4, and 6 transformer layers.
  • Ablation studies: The segmentation decoder improves IoU by 1.67%p with four fusion layers but only 0.31%p with two layers.The decoder refines positive patch-level predictions into fine pixel-level masks in a coarse-to-fine process; the six-layer case was omitted because of memory shortage.
  • Efficiency and qualitative analysis: Weight sharing causes only marginal performance degradation, enabling more efficient ReSTR operation with little loss of performance.ReSTR also achieves the best accuracy with the least computation among compared models, using a smaller visual feature size than prior work.
  • Efficiency and qualitative analysis: Patch-level predictions roughly localize target patches and relational-object boundaries before the decoder produces fine pixel-level predictions for different language queries.Qualitative visualizations show masks corresponding to varying expressions on the same image.

5. Conclusion

ReSTR is a convolution-free transformer model for referring image segmentation that models global context and cross-modal interactions, then refines patch predictions into pixel-level masks. The paper reports that it outperforms existing techniques on all public benchmarks, while identifying a patch-size trade-off between performance and computational cost.

  • Conclusion: ReSTR uses transformer encoders for visual and linguistic features, a multimodal fusion encoder, an adaptive patch classifier, and a coarse-to-fine segmentation decoder.The architecture is convolution-free and uses transformers to encode global context and flexible interactions between modalities.
  • Conclusion: ReSTR outperforms existing referring image segmentation techniques on all public benchmarks.The conclusion presents this as the paper's overall empirical result.
  • Conclusion: Computational cost increases quadratically as patch size decreases, creating a trade-off between dense-prediction performance and efficiency.The paper suggests integrating linear-complexity transformer architectures as future work.

Appendix

The supplementary material reports omitted experiments on language-expression length, hyperparameter sensitivity, and additional qualitative results.

  • Supplementary analyses: Section A analyzes performance across language-expression lengths compared with previous methods.
  • Supplementary analyses: Section B investigates sensitivity to the loss-balancing weights λ and thresholding value τ.
  • Supplementary analyses: Section C presents additional qualitative results of ReSTR on the Gref dataset.

A. Impact of the length of language expression

ReSTR maintains stronger performance across language-expression lengths than prior methods, with less degradation on longer sentences and improvements across most groups.

  • A. Impact of the length of language expression: ReSTR outperforms most previous methods across language-expression length groups on four datasets.The exception is the 1–5 length group of Gref, where the gap is marginally 1.2%p.
  • A. Impact of the length of language expression: ReSTR shows less performance degradation from the shortest to longest sentence groups than ACM on four datasets.ACM’s performance improvement mostly comes from short sentence groups despite being designed to capture long-range dependencies.
  • A. Impact of the length of language expression: ReSTR improves performance on most sentence-length groups, suggesting better capture of long-range interactions in language expressions.The analysis links this pattern to understanding complex interactions between words in longer expressions.

B. Sensitivity to hyperparameters

ReSTR is relatively insensitive to its loss-balancing and patch-label threshold hyperparameters within a central range of loss weights.

  • B. Sensitivity to hyperparameters: The analysis varies λ across {0.01, 0.05, 0.1, 0.5, 1} and τ across {0.5, 0.6, 0.7, 0.8, 0.9}.These hyperparameters control loss balancing and the threshold used to generate patch-level labels.
  • B. Sensitivity to hyperparameters: ReSTR performance is high and stable when λ is between 0.05 and 0.5.Within this interval, performance is reported as insensitive to the hyperparameter setting.
  • B. Sensitivity to hyperparameters: The hyperparameter study examines IoU while jointly varying the loss-balancing weight λ and threshold τ.The results are summarized in Fig. A2 for the tested combinations.

C. More qualitative results

Qualitative evaluations show ReSTR producing accurate target masks across varied language queries, including non-human, partially visible, and occluded objects.

  • C. More qualitative results: Figures A3 and A4 present ReSTR qualitative results on the Gref dataset, including pixel-level predictions and DenseCRF-post-processed outputs.The displayed comparisons include patch-level prediction, pixel-level prediction, ReSTR, ReSTR+DCRF, and ground truth.
  • C. More qualitative results: Figure A1 compares IoU(%) against sentence-length groups across four datasets, while Figure A2 compares IoU across λ and τ.These figures provide additional analyses of language length and hyperparameter settings.
  • C. More qualitative results: ReSTR accurately segments non-human, partially appeared, and occluded target objects described by language expressions.Partially appeared examples are shown in rows 1–3 of Fig. A4, while occluded examples appear in rows 5–7.
  • C. More qualitative results: The pixel-level segmentation decoder produces fine-grained predictions and removes false positives from patch-level predictions.This comparison is included alongside qualitative outputs for the Gref dataset.
  • C. More qualitative results: Figure A5 shows qualitative ReSTR results for different language-expression queries applied to each image in the Gref validation set.The examples include queries such as “Man in grey in back of table” and “baby sitting on mans lap.”
Loading 2203.16768v1…