Source-linked AI summary
Segmentation from Natural Language Expressions
Ronghang Hu, Marcus Rohrbach, Trevor Darrell
TL;DR
The paper addresses pixelwise segmentation of image regions described by natural-language expressions, beyond fixed categories and rectangular localization. It combines an LSTM language encoder with fully convolutional visual processing and reports quality predictions that outperform baselines by a large margin on a benchmark dataset, while failure cases show imprecise object boundaries.
Problem
The task is to segment the visual entities described by a natural-language expression, which traditional semantic segmentation and prior localization methods cannot handle beyond fixed categories or rectangular regions.
Method
An end-to-end recurrent-convolutional model encodes the expression with an LSTM, extracts a spatial image feature map, and produces a pixelwise mask through fully convolutional processing and upsampling.
Results
The model produces quality segmentation predictions and outperforms baseline methods by a large margin on the ReferIt benchmark dataset.
Takeaways & Limitations
The approach can efficiently predict segmentation for referential expressions describing single or multiple objects or stuffs.
Takeaways & Limitations
Evaluation on the Google-RefExp dataset was left for future work because its test split had not been released.
Abstract
from arXiv · showhide
In this paper we approach the novel problem of segmenting an image based on a natural language expression. This is different from traditional semantic segmentation over a predefined set of semantic classes, as e.g., the phrase "two men sitting on the right bench" requires segmenting only the two people on the right bench and no one standing or sitting on another bench. Previous approaches suitable for this task were limited to a fixed set of categories and/or rectangular regions. To produce pixelwise segmentation for the language expression, we propose an end-to-end trainable recurrent and convolutional network model that jointly learns to process visual and linguistic information. In our model, a recurrent LSTM network is used to encode the referential expression into a vector representation, and a fully convolutional network is used to a extract a spatial feature map from the image and output a spatial response map for the target object. We demonstrate on a benchmark dataset that our model can produce quality segmentation output from the natural language expression, and outperforms baseline methods by a large margin.
1 Introduction
The paper introduces segmentation from natural language expressions, targeting precise pixelwise masks for the entities described by a phrase rather than predefined categories or generic foreground. It proposes an end-to-end recurrent-convolutional model that combines language and image features to produce the mask.
- Problem: Natural-language segmentation targets only the visual entities described by an expression, such as selected people, rather than all instances or a predefined category.The task differs from semantic segmentation, instance segmentation, and language-independent foreground segmentation.
- Motivation: Precise segmentation is useful for language-based robot instructions and interactive image editing, where bounding boxes may be insufficient for irregular objects.The paper also mentions referring to meal parts for nutrition estimation.
- Prior limitations: Prior suitable methods were limited to bounding boxes or fixed, predefined categories, leaving direct pixelwise segmentation from natural-language queries unaddressed.These limitations motivate a model that handles multi-word expressions and specific object instances.
- Method: The proposed model jointly processes language and vision using an LSTM expression encoder, a convolutional image feature extractor, and a fully convolutional classifier.The classifier produces a coarse response map that is upsampled with deconvolution into a pixel-level mask.
- Method: The method is presented as an overview for producing segmentation from natural language expressions.Figure 2 illustrates the model’s overall processing pipeline.
2 Related work
Related work covers language-based object localization, fully convolutional segmentation, and attention models. The paper distinguishes its approach by directly producing precise segmentation masks from natural-language descriptions rather than bounding boxes or coarse attention outputs.
- Localizing objects with natural language: Earlier language-based localization methods select a bounding box for the object whose description receives the highest probability.These methods build on image-captioning frameworks and differ from this paper’s direct segmentation approach.
- Localizing objects with natural language: The paper states that prior localization methods did not directly output segmentation masks for objects given natural-language queries.The authors compare against foreground segmentation applied to bounding-box predictions.
- Fully convolutional network for segmentation: Fully convolutional networks preserve spatial information and are suited to segmentation outputs arranged on a spatial grid.The paper uses them for feature extraction and segmentation output, and includes per-word segmentation as a baseline.
- Attention and visual question answering: Attention models generate spatial attention maps for tasks such as image captioning, visual question answering, and object-reference questions.The cited object-reference method generates attention maps for individual words such as “black” and “cat”.
- Attention and visual question answering: Unlike the paper’s precise segmentation goal, related attention models produce coarse spatial outputs to facilitate other tasks rather than segmenting objects.This distinction concerns both output resolution and the purpose of the spatial map.
3 Our model
The model jointly processes an image and a natural-language expression to produce a pixelwise segmentation mask. It encodes language with an LSTM, preserves spatial visual features, combines both modalities convolutionally, and upsamples the resulting response map.
- The model has three components: an LSTM expression encoder, a fully convolutional spatial feature extractor, and a classification-and-upsampling network.The architecture jointly processes linguistic and visual information for segmentation.
- The image network produces an L2-normalized w×h×(Dim+2) feature map containing local descriptors and relative x,y coordinates.Coordinate channels support reasoning about spatial relationships such as “right woman.”
- The expression encoder embeds each word, processes the sequence with an LSTM, and uses the final normalized hidden state h_T as its fixed-length representation.The implementation uses a 1000-dimensional LSTM hidden state.
- The encoded expression is tiled with local image descriptors and classified by two 1×1 convolutional layers to produce a coarse spatial response map.The coarse map acts as a low-resolution segmentation of the referential expression.
- Deconvolution upsamples the coarse response map to the input image size, with each high-resolution value representing confidence that a pixel belongs to the target.Pixels with response values greater than 0 form the final segmentation prediction.
- Training uses image-expression-mask tuples and an average pixelwise weighted logistic loss, with foreground weight α_f = 3 and background weight α_b = 1.The network is trained end-to-end with standard back-propagation and SGD with momentum.
4 Experiments
Experiments evaluate the model on ReferIt using natural-language expressions paired with segmented object and stuff regions. The model is compared with several baselines on segmentation quality and inference speed, and shows stronger quality with faster prediction than proposal-based methods.
- Dataset: ReferIt provides 20,000 images with 130,525 expressions annotated on 96,654 segmented regions, including both object and stuff regions.The experiments use the same trainval/test split as prior work, with 10,000 images for training and validation and 10,000 for testing.
- Baselines: The evaluation compares the model with per-word segmentation, bounding-box foreground segmentation, segmentation-proposal classification, and whole-image baselines.The baselines use combinations of LSTM-free per-word predictions, SCRC or GroundeR localization with GrabCut, MCG proposals, or the entire image.
- Baseline analysis: Per-word score averaging gives reasonable overall IoU, while bounding-box methods have comparable precision but lower overall IoU.MCG classification yields slightly higher precision than the bounding-box methods, and GrabCut improves precision for SCRC and GroundeR.
- Segmentation quality: The model outperforms all baseline methods by a large margin on both precision and overall IoU.The full high-resolution model improves over directly bilinearly upsampling the low-resolution response map and over the evaluated baselines.
- Qualitative results: The model produces reasonable segmentations for both object expressions and stuff expressions using the same approach.Examples include object descriptions such as “bird on the left” and stuff descriptions such as “sky above the bridge.”
- Failure cases: Failure cases include predictions with IoU below 50%, often because response maps cover the target but object or stuff boundaries are not precisely segmented.Figure 7 presents these low-overlap cases on ReferIt.
- Speed: The method is slower than per-word segmentation but significantly faster than proposal-based methods such as SCRC grabcut and MCG classification.Speed is measured as average test-time prediction on a single machine with an NVIDIA Tesla K40 GPU.
5 Conclusion
The paper addresses natural-language expression segmentation with an end-to-end recurrent convolutional model that produces pixelwise masks and outperforms baseline methods by a large margin.
- The model segments image regions described by natural-language referential expressions into pixelwise outputs.
- An end-to-end recurrent convolutional network encodes the expression, extracts image features, and predicts segmentation through fully convolutional classification and upsampling.
- The approach handles referential expressions describing single or multiple objects or stuff regions.
- The model outperforms baseline methods by a large margin on a benchmark dataset.