Source-linked AI summary
Dynamic Multimodal Instance Segmentation guided by natural language queries
Edgar Margffoy-Tuay, Juan C. Pérez, Emilio Botero, Pablo Arbeláez
TL;DR
The paper addresses segmenting a specific object from a natural-language expression, a task requiring visual and sequential linguistic information to be combined. It proposes the modular Dynamic Multimodal Network, which integrates these signals and uses intermediate visual features during upsampling. DMN outperforms previous methods in six of eight standard dataset splits.
Problem
Natural-language instance segmentation must associate an expression with the correct image instance while combining spatial visual information with sequential linguistic information.
Method
DMN uses separate visual and language modules, a synthesis module that jointly processes their outputs, and an upsampling module that reuses intermediate visual features.
Results
DMN outperforms state-of-the-art methods in six of eight standard dataset splits for referring-expression instance segmentation.
Takeaways & Limitations
The modular design combines domain-specific processing with multimodal synthesis and detailed mask reconstruction for natural-language-guided instance segmentation.
Takeaways & Limitations
Performance gains are smaller on testB splits containing more varied queries and objects than the person-biased testA splits, and ambiguous expressions can lead to incorrect or weak segmentations.
Abstract
from arXiv · showhide
We address the problem of segmenting an object given a natural language expression that describes it. Current techniques tackle this task by either (\textit{i}) directly or recursively merging linguistic and visual information in the channel dimension and then performing convolutions; or by (\textit{ii}) mapping the expression to a space in which it can be thought of as a filter, whose response is directly related to the presence of the object at a given spatial coordinate in the image, so that a convolution can be applied to look for the object. We propose a novel method that integrates these two insights in order to fully exploit the recursive nature of language. Additionally, during the upsampling process, we take advantage of the intermediate information generated when downsampling the image, so that detailed segmentations can be obtained. We compare our method against the state-of-the-art approaches in four standard datasets, in which it surpasses all previous methods in six of eight of the splits for this task.
1 Introduction
The paper introduces DMN, a modular end-to-end network that combines visual and language processing for segmenting instances described by natural language, while using intermediate visual features to recover detailed masks.
- Problem: Natural-language instance segmentation retrieves a specific image object from a referring expression rather than labeling instances within predefined semantic classes.The output distinguishes only the query member from background, allowing masks corresponding to objects referred to through varied language.
- Problem: Naive CNN-based approaches fall short because they do not naturally model the sequential information required by language or fully combine visual and linguistic domains.The paper argues that CNNs and RNNs remain useful components, but require an architecture exploiting their complementary properties.
- Approach: DMN divides the task into Visual, Language, Synthesis, and Upsampling Modules within a fully differentiable, end-to-end trainable architecture.The VM represents the image, the LM represents query meaning up to each word, the SM merges both streams, and the UM incrementally upsamples using VM feature maps.
- Approach: The method uses SRUs for language and multimodal processing and generates visual-space scores by merging linguistic and visual information in the Synthesis Module.This design processes multimodal information jointly while exploiting the sequential structure of referring expressions.
- Approach: The Upsampling Module uses intermediate downsampling features to recover fine-scale details in the final segmentation maps.The contribution explicitly links these visual features to more detailed masks during upsampling.
- Evaluation: DMN outperforms previous methods across various standard-dataset splits and achieves state-of-the-art results for referring-expression instance segmentation.The paper reports validation on all standard datasets and provides the implementation and training routines for reproducibility.
2 Related Work
Prior work combines visual and linguistic information for referring-expression segmentation, but differs in how it models language, fuses modalities, and restores resolution. The proposed direction combines recursive multimodal processing with multi-scale visual information for finer masks.
- Task formulation: Natural-language instance segmentation retrieves a specific object instance from an image using a referring expression rather than predefined semantic categories.The task requires associating expressions such as “Man on the right” with observed objects.
- Multimodal fusion: Existing methods separately extract image and query representations, then combine them through convolutions, recurrent multimodal interaction, or language-conditioned spatial filters.These approaches include CNN–LSTM pipelines, recurrent segmentation, and transformations of language into filters responsive to referred-object locations.
- Resolution recovery: Earlier systems commonly produce low-resolution outputs and use bilinear interpolation or deconvolution for upsampling, with some requiring DenseCRF post-processing.The related-work comparison identifies coarse interpolation and non-end-to-end post-processing as limitations of earlier designs.
- Language modeling: Recursive multimodal interaction exploits language sequence information by performing segmentation multiple times, whereas other approaches rely more heavily on a final sentence representation or non-sequential filtering.The related methods differ in whether linguistic information is repeatedly incorporated during segmentation.
- Proposed direction: The proposed approach combines multimodal concatenation, sequential dynamic filters, and skip connections that reuse downsampling features during upsampling.It also uses multi-scale visual feature maps and SRUs; the visual module outputs N scales for segmentation and upsampling.
3 Dynamic Multimodal Network
The Dynamic Multimodal Network modularizes visual, language, multimodal synthesis, and upsampling processes for referring-expression segmentation. It recurrently combines language-conditioned visual responses with spatial and visual features, then uses intermediate feature maps to recover detailed masks.
- Overall architecture: The architecture comprises Visual, Language, Synthesis, and Upsampling Modules that transform an image-query pair into a final segmentation mask.The Visual Module represents the image, the Language Module represents the query, the Synthesis Module merges them, and the Upsampling Module produces the mask.
- Visual Module: The Visual Module uses DPN92 to extract feature maps at N scales, with N = 5 in the experiments.The architecture is fully convolutional and uses downsampled feature maps whose dimensions are 1/2^n of the original image dimensions.
- Language Module: The Language Module embeds each query word, processes the sequence with an SRU, and concatenates word embeddings with hidden states to form enriched language representations.It also generates K dynamic filters from each enriched representation; Fig. 4 describes SRU-based query features and filters.
- Synthesis Module: The Synthesis Module applies language-derived filters to visual features, combines their response maps with spatial coordinates and language features, and recurrently processes the resulting multimodal representations with an mSRU.The mSRU uses the sequence of per-time-step multimodal maps and produces a single low-resolution feature map for the expression.
- Upsampling Module: The Upsampling Module concatenates synthesized features with visual feature maps at successive stages, applies 3×3 convolutions, and doubles spatial resolution by bilinear interpolation.After log2(N) stages, a 1×1 convolution and sigmoid produce a single-channel mask with scores between 0 and 1.
4 Experimental Setup
The experiments evaluate the proposed method on four standard referring-expression segmentation datasets using established metrics and dataset-specific validation or test splits. The datasets vary in object types, expression lengths, object sizes, ambiguity, and location-word restrictions.
- Datasets: Experiments use the four standard datasets ReferIt, UNC, UNC+, and GRef, whose characteristics differ in object types, expression lengths, and referred-object sizes.This variability is used to assess generalization across task settings.
- Datasets: UNC and GRef contain multiple same-category objects, requiring expressions to distinguish the referred instance beyond semantic class cues.UNC has two or more same-category objects, while GRef contains two to four objects of the same class.
- Datasets: UNC+ restricts expressions to appearance-based descriptions rather than location, making interpretation depend on scene perspective and object semantics.This restriction differs from UNC’s broader referring expressions.
- Evaluation: Performance is evaluated with standard literature metrics on validation or test sets after training on each dataset’s training set.The reported metrics include mean Intersection over Union and precision-based evaluation as described in the experimental protocol.
- Implementation: The implementation uses DPN92 as backbone, five visual scales, SRU and mSRU modules, ten dynamic filters, and Adam optimization with an initial learning rate of 1 × 10^-5.The principal hyperparameters were optimized on the UNC+ validation set.
5 Results
Ablations show that DMN’s linguistic, dynamic-filter, multimodal, and skip-connection components jointly support performance and detailed segmentation. Against prior methods, DMN performs best in six of eight dataset splits, while qualitative failures remain on ambiguous or long expressions.
- Control Experiments: Learned embeddings provide a small gain over hidden states alone, particularly at stricter overlap thresholds.The ablation replaces the concatenation of h_t and e_t with r_t = h_t.
- Control Experiments: Large performance drops after removing upsampling skip connections show that downsampling features provide finer details for segmentation.The authors attribute the benefit to convolutions followed by bilinear interpolation rather than deconvolutions.
- Control Experiments: Dynamic-filter responses and linguistic information must be combined; using either alone does not achieve high performance.Removing dynamic-filter responses is harmful, while reusing language-module features in the Synthesis Module adds little.
- Comparison with the State-of-the-Art: Six of eight dataset splits show DMN outperforming all previous methods in mIoU.The comparison covers the standard datasets and evaluates mIoU against state-of-the-art approaches.
- Control Experiments: SRUs achieve similar accuracy to LSTMs with fewer parameters and approximately threefold shorter training time.The SRU-based LM and SM use 9M and 10M parameters, compared with 24M and 24.2M for LSTMs.
- Qualitative Results: Qualitative failures include selecting the wrong person under ambiguous queries, weak masks for long expressions, and confusion between objects sharing attributes.One failure appears to focus mainly on the word “right,” while another ignores the relational cue “standing.”
6 Conclusions
DMN is a modular architecture for natural-language instance segmentation that combines multimodal synthesis with feature reuse during upsampling. It produces detailed masks and outperforms state-of-the-art methods in six of eight standard dataset splits.
- Conclusions: DMN modularizes visual, language, synthesis, and upsampling processing for instance segmentation from natural-language expressions.The Synthesis Module combines outputs from earlier modules, while the Upsampling Module uses encoding feature maps.
- Conclusions: Feature maps from the encoding stage help the Upsampling Module deliver detailed final segmentations.
- Conclusions: DMN outperforms state-of-the-art methods in six of the eight standard dataset splits.