Source-linked AI summary
Recurrent Multimodal Interaction for Referring Image Segmentation
Chenxi Liu, Zhe Lin, Xiaohui Shen, Jimei Yang, Xin Lu, Alan Yuille
TL;DR
Referring image segmentation asks a model to segment the region described by natural language, while prior methods often model images and sentences independently. The paper introduces convolutional multimodal LSTM to encode sequential word-to-image interactions with visual and spatial information, and reports superior benchmark performance, state-of-the-art results on large-scale datasets, and interpretable intermediate features.
Problem
Referring image segmentation seeks to identify the image region corresponding to a natural-language expression, but existing methods combine independently modeled image and sentence representations.
Method
The paper proposes a two-layer convolutional multimodal LSTM that fuses language, visual, and spatial information after each word and retains it in a recurrent hidden state.
Results
The model outperforms the baseline on benchmark datasets and achieves new state-of-the-art results on all large-scale benchmark datasets.
Takeaways & Limitations
The learned multimodal hidden features are human-interpretable and facilitate segmentation, supporting word-to-image interaction as an effective modeling scheme.
Abstract
from arXiv · showhide
In this paper we are interested in the problem of image segmentation given natural language descriptions, i.e. referring expressions. Existing works tackle this problem by first modeling images and sentences independently and then segment images by combining these two types of representations. We argue that learning word-to-image interaction is more native in the sense of jointly modeling two modalities for the image segmentation task, and we propose convolutional multimodal LSTM to encode the sequential interactions between individual words, visual information, and spatial information. We show that our proposed model outperforms the baseline model on benchmark datasets. In addition, we analyze the intermediate output of the proposed multimodal LSTM approach and empirically explain how this approach enforces a more effective word-to-image interaction.
1. Introduction
The paper frames referring image segmentation as a language-guided pixel-labeling problem and argues that interaction between words and images should unfold throughout the expression. It proposes convolutional multimodal LSTM to model this sequential interaction and reports stronger benchmark performance than a language-only baseline.
- Problem: Referring image segmentation uses an image and natural-language expression to identify the corresponding region.The task may support interactive selection through typed or spoken descriptions.
- Prior approach: Existing approaches combine independently modeled image features and sentence representations, typically segmenting after the full expression is encoded.The baseline augments convolutional segmentation networks with a sentence-encoding LSTM.
- Motivation: Human sentence-picture processing motivates updating image-related beliefs throughout the expression rather than only after reading the final word.The paper describes reasoning from “man” to “on right” to “wear blue” as progressive pixel pruning.
- Method: The convolutional multimodal LSTM receives visual features and language representations, retaining spatial and semantic information in a hidden state across word-reading steps.A final convolution produces the segmentation result after the last word.
- Contributions: The proposed word-to-image multimodal LSTM approach achieves superior performance over the baseline on benchmark datasets.The paper also analyzes intermediate outputs to explain how the approach enforces more effective word-to-image interaction.
2. Related Work
Related work spans semantic segmentation, referring-expression localization, and multimodal interaction representation. The paper distinguishes its approach from localization methods that produce bounding boxes and from prior segmentation methods by modeling sequential language–visual–spatial interaction.
- Semantic Segmentation: Semantic segmentation work commonly uses fully convolutional networks for dense, downsampled per-pixel labeling.The related-work discussion notes that excessive pooling can prohibit the network from preserving needed spatial detail.
- Referring Expression Localization: Referring-expression localization methods score or align image regions but generally target bounding boxes rather than segmentation masks.The paper identifies prior referring image segmentation as the closest work and contrasts its sequential interaction modeling with that approach.
- Multimodal Interaction Representation: This paper updates segmentation belief after each word by modeling interactions among natural language, visual information, and spatial information.This sequential property differentiates the proposed approach from methods that encode modalities independently or interact only after the sentence.
3. Models
The models address referring image segmentation by combining image and language representations, with the RMI model instead modeling multimodal interactions sequentially at each word and spatial location.
- 3.2. Baseline Model: The baseline encodes the entire sentence with a language-only LSTM, then concatenates its fixed vector with image-spatial features to predict the segmentation mask.Image features are extracted by ResNet-101 with atrous convolution, reducing the downsampling rate from 32 to 8.
- 3.3. Recurrent Multimodal Interaction Model: The baseline performs segmentation only after memorizing the complete referring expression, requiring all sentence attributes to be jointly retained.The paper motivates an alternative because semantics are not grounded onto the image during the earlier word-reading steps.
- 3.3. Recurrent Multimodal Interaction Model: At each recurrent step, the multimodal LSTM combines language representation with the visual feature at each spatial location, retaining language, image, and spatial information.This explicitly models the progression of multimodal features over word-reading order.
- 3.3. Recurrent Multimodal Interaction Model: The same mLSTM operation is shared across image locations, making it equivalent to a 1 × 1 convolution that shares weights across spatial locations and time steps.The shared operation is applied over the feature map at every image location.
- 3.3. Recurrent Multimodal Interaction Model: The multimodal LSTM can retain either semantic or visual information, while jointly forcing word-visual interaction and producing multimodal features at every recurrent step.If visual inputs are ignored, it reduces to semantic encoding; if language is ignored, the repeated visual input can be retained.
- 3.3. Recurrent Multimodal Interaction Model: The RMI model uses a two-layer recurrent structure whose lower LSTM encodes semantics and whose upper multimodal LSTM preserves spatial resolution while generating multimodal features.The lower language-only LSTM is spatially agnostic, whereas the upper LSTM preserves feature resolution H′ × W′.
4. Experiments
Experiments evaluate referring image segmentation across four datasets, compare visual backbones and multimodal interaction, and analyze performance by expression length and qualitative intermediate behavior.
- Datasets: The evaluation uses Google-Ref, UNC, UNC+, and ReferItGame, with differences in expression content, length, and segmentation targets.Google-Ref contains longer, richer descriptions; UNC+ excludes location words; ReferItGame includes stuff masks and generally shorter expressions.
- Quantitative Results: ResNet consistently improves performance over VGG-based FCN-32s, while DeepLab provides an even higher baseline after segmentation-specific fine-tuning.The authors attribute the ResNet improvement to stronger features and higher spatial resolution, and the DeepLab advantage to easier knowledge transfer.
- Quantitative Results: RMI with multimodal LSTM consistently outperforms language-only LSTM across image extractors and datasets, including nearly 3.5% IOU improvement on Google-Ref with ResNet.The reported gain is observed regardless of the image feature extractor and dataset.
- Quantitative Results: RMI’s relative advantage generally grows with referring-expression length, suggesting that multimodal LSTM better fuses features over longer sequences.The RMI model outperforms the baseline in every expression-length group.
- Qualitative Results: DenseCRF consistently improves IOU, with usually larger gains for RMI, and can refine boundaries or suppress scattered false predictions.The authors interpret the larger RMI refinement gain as evidence of better localization ability.
- Qualitative Results: The mLSTM progressively updates spatial belief as words arrive, while qualitative examples show language-only LSTM can be distracted by later words.Responses can focus on candidate objects, incorporate attributes or locations, and suppress mismatching regions before final binary segmentation.
5. Conclusion
The paper introduces Recurrent Multimodal Interaction, which fuses language, visual, and spatial information sequentially during referring image segmentation. The word-to-image scheme outperforms the sentence-to-image baseline and achieves state-of-the-art results on large-scale benchmarks.
- Recurrent Multimodal Interaction is a two-layer recurrent architecture that encodes interactions among individual words, visual information, and spatial information.
- The model performs multimodal feature fusion after every word rather than independently encoding the referring expression and image into vectors.
- The word-to-image scheme outperforms the sentence-to-image scheme used by previous approaches.
- The model achieves new state-of-the-art performance on all large-scale benchmark datasets.
- Visualized mLSTM hidden states are human-interpretable and facilitate segmentation.