Source-linked AI summary
MAttNet: Modular Attention Network for Referring Expression Comprehension
Licheng Yu, Zhe Lin, Xiaohui Shen, Jimei Yang, Xin Lu, Mohit Bansal, Tamara L. Berg
TL;DR
Referring expression comprehension requires localizing regions from language even though expressions emphasize different kinds of information. MAttNet softly decomposes expressions into subject, location, and relationship modules with learned language and visual attention, and it substantially outperforms prior methods on bounding-box and pixel-level comprehension.
Problem
Most recent referring-expression models treat expressions as a single unit despite variation in whether appearance, location, or relationships distinguish the target.
Method
MAttNet softly decomposes expressions into subject, location, and relationship phrases, applies module-specific language and visual attention, and combines module scores using adaptive weights.
Results
MAttNet reports ∼10% improvements on bounding-box localization and almost doubling precision on pixel segmentation over state-of-the-art methods.
Takeaways & Limitations
The modular attention framework supports general referring expressions by adapting processing to the information present in each expression.
Takeaways & Limitations
Evaluation with res101-frcn features cannot use RefCOCOg’s val* split because the detector was pretrained on COCO training images.
Abstract
from arXiv · showhide
In this paper, we address referring expression comprehension: localizing an image region described by a natural language expression. While most recent work treats expressions as a single unit, we propose to decompose them into three modular components related to subject appearance, location, and relationship to other objects. This allows us to flexibly adapt to expressions containing different types of information in an end-to-end framework. In our model, which we call the Modular Attention Network (MAttNet), two types of attention are utilized: language-based attention that learns the module weights as well as the word/phrase attention that each module should focus on; and visual attention that allows the subject and relationship modules to focus on relevant image components. Module weights combine scores from all three modules dynamically to output an overall score. Experiments show that MAttNet outperforms previous state-of-art methods by a large margin on both bounding-box-level and pixel-level comprehension tasks. Demo and code are provided.
1. Introduction
MAttNet addresses the varied information used in referring expressions with three specialized modules, learned attention, and adaptive score weighting. It reports substantially better comprehension on bounding-box and pixel-level tasks.
- Referring expression comprehension selects an image region described by natural language, supporting natural interaction between intelligent agents and people.
- Existing models often encode the whole expression with one LSTM and concatenate features, despite expressions emphasizing appearance, location, or relationships differently.Examples shift from “red ball” to location- or relationship-based descriptions as scene ambiguity changes.
- MAttNet softly decomposes each expression into subject, location, and relationship phrases that trigger separate visual modules and produce a weighted overall region score.The three modules cover appearance attributes, absolute or relative location, and subject-object relations.
- The model learns word attention and adaptive module weights from the input expression instead of relying on an external parser.Subject-heavy phrases such as “red cat” receive larger subject weights, while “woman on left” emphasizes subject and location.
- Subject and relationship modules use different visual attention mechanisms: soft in-box attention for object parts and hard out-of-box attention for relational objects.The paper reports that both attention types improve comprehension accuracy.
- Training uses only object-proposal and referring-expression pairs while learning word attention, module weights, and visual attention end to end.
- ∼10% improvements on bounding-box localization and almost doubling precision on pixel segmentation are reported over state-of-the-art methods.
2. Related Work
Related work models referring expressions with probabilistic, embedding, or modular approaches, while MAttNet targets richer expressions than fixed templates. The paper also evaluates pixel-level segmentation alongside bounding-box localization.
- Prior referring-expression systems model P(r|o), P(o|r), or joint vision-language relationships to select a region from candidate objects.
- Most earlier work focuses on bounding-box localization, while other methods use FCN-style models for expression-driven object segmentation.
- Modular networks decompose inputs into components and have been applied to visual question answering, reasoning, relationship modeling, and multitask reinforcement learning.
- Soft attention outperforms hard decisions predicted by an external parser in the reported comparison.
- MAttNet extends modular referring-expression comprehension beyond fixed Subject-Preposition/Verb-Object templates to richer forms such as “left dog” and “man in red”.It assigns word-level attention and module-level weights adaptively to the input expression.
3. Model
MAttNet decomposes referring expressions into subject, location, and relationship components, then combines module-specific visual matching scores using learned language attention and adaptive module weights. Its visual modules use distinct representations and attention mechanisms, trained end-to-end with proposal–expression pairs and ranking-based supervision.
- Language Attention Network: MAttNet softly parses each expression into three phrase embeddings that trigger subject, location, and relationship modules, whose scores are combined into an overall region score.The language attention network replaces an external parser and produces module-specific phrase representations.
- Language Attention Network: A bidirectional LSTM encodes contextual word representations, while trainable module-specific vectors attend over words to form modular phrase embeddings.The final hidden representation concatenates forward and backward hidden states before attention pooling.
- Language Attention Network: Adaptive module weights reflect which information is distinctive in each expression, allowing subject-only, subject-location, or subject-relationship descriptions to contribute differently.The weights are computed from a representation of the whole expression and normalized across the three modules.
- Subject Module: The subject module combines attribute prediction with phrase-guided in-box attention to emphasize relevant appearance regions inside a candidate bounding box.Attribute and C4 features are fused into a spatial subject representation, which is pooled according to the subject phrase.
- Location and Relationship Modules: The location module represents absolute and relative position, while the relationship module scores up to five surrounding objects using their appearance and offsets.The relationship score selects the highest-scoring supporting object, corresponding to weakly supervised multiple-instance learning.
- Scoring and Training: The final candidate score is S(oi|r) = wsubjS(oi|qsubj) + wlocS(oi|qloc) + wrelS(oi|qrel), trained with positive pairs, sampled negatives, attribute loss, and ranking loss.Faster R-CNN supplies candidate-region features, with ResNet or VGGNet used as feature extractors depending on the comparison.
4. Experiments
Experiments evaluate MAttNet on referring-expression comprehension and referential segmentation using RefCOCO-family datasets, ground-truth regions, and automatically detected objects. The results compare modular components, visual features, parser choices, and qualitative behavior against prior methods.
- Datasets: Evaluation uses RefCOCO, RefCOCO+, and RefCOCOg, whose collection settings, expression lengths, object composition, and location-word constraints differ.RefCOCOg expressions average 8.4 words versus 3.5 for RefCOCO and RefCOCO+, while RefCOCO+ forbids absolute location words.
- Evaluation setup: The comprehension task selects the highest-scoring proposal, counting localization as correct when its IoU with the ground-truth box exceeds 0.5.Testing includes person-heavy testA and object-heavy testB splits for RefCOCO and RefCOCO+; RefCOCOg is commonly evaluated on val*.
- Ground-truth regions: MAttNet achieves the highest accuracy by a large margin on ground-truth MS COCO regions, with res101-based Faster R-CNN features improving accuracy by another ∼3%.The comparison first uses vgg16 features for fairness, then evaluates res101-frcn and a full model with phrase-guided attentional pooling.
- Ablations: Ablations show that modular learning, location encoding, relationship modeling, attribute handling, and phrase-guided attentional pooling each improve performance over simpler combinations.Phrase-guided attentional pooling especially improves person-category performance on RefCOCO and RefCOCO+ testA.
- Fully automatic comprehension: The end-to-end learned parser is ∼5% better than training with three hard-coded template phrases, because the external parser makes errors on referring expressions.Fully automatic comprehension loses performance because of detection errors, but module improvements remain consistent and outperform prior state-of-the-art results; res101-mrcn performs better than res101-frcn.
- Referential segmentation: For referential segmentation, MAttNet uses the highest-scoring detected object’s box to obtain a pixel-wise mask and outperforms prior results by a large margin under all reported metrics.The evaluation reports Precision@X for X ∈ {0.5, 0.6, 0.7, 0.8, 0.9} and overall IoU; the authors attribute gains partly to decoupling box localization from segmentation.
5. Conclusion
MAttNet addresses variation in referring expressions by combining modular language and visual attention, and it improves comprehension across bounding-box and pixel-level tasks.
- MAttNet attends to relevant words and visual regions within a modular framework.It dynamically computes an overall matching score.
- The network addresses variance in referring expressions through modular attention.
- MAttNet significantly outperforms state-of-the-art methods on bounding-box-level and pixel-level comprehension.
A.1. Training Details
The model uses Adam optimization with scheduled learning-rate decay and 512-dimensional representations throughout its language and multilayer components.
- Adam optimization uses an initial learning rate of 0.0004 and a batch size of 15 images.
- The learning rate is halved every 8,000 iterations after an 8,000-iteration warm-up.
- The LSTM word embedding and hidden state sizes are both 512.
- All MLP and fully connected layer outputs are 512-dimensional.
A.2. Computational Efficiency
MAttNet converges in about half a day on a single Titan-X(Pascal) and adds little inference time beyond Mask R-CNN.
- The full MAttNet model converges at around 30,000 iterations.
- Training takes around half a day using a single Titan-X(Pascal).
- Inference takes 0.33 seconds on average, including 0.31 seconds for Mask R-CNN and 0.02 seconds for MAttNet.
A.3. Attribute Prediction
MAttNet predicts object attributes during testing using extracted attribute labels, with frequency filtering and quantitative evaluation of multi-attribute prediction.
- The full model predicts attributes during testing.
- A template parser extracts object names, colors, and generic attribute words from expressions.
- Low-frequency words are removed, and the 50 most frequent attribute words are used for training.
- Top-20 attribute histograms and quantitative multi-attribute prediction results are reported.
A.4. MAttNet + Grabcut
MAttNet can be extended to referential segmentation by replacing its mask branch with a foreground-background decomposition method such as GrabCut. Even with this inferior segmentation method, MAttNet+GrabCut outperforms prior state-of-the-art results.
- MAttNet can use a foreground-background decomposition method as its mask branch for referential segmentation.The paper identifies GrabCut as the simplest replacement.
- MAttNet+GrabCut outperforms previous state-of-the-art referential segmentation results despite GrabCut being an inferior segmentation method.
- The authors argue that decoupling box localization from segmentation is more suitable for instance-level referential segmentation.
A.5. Mask R-CNN Implementation
The implementation uses a Mask R-CNN mask branch and modifies its training setup for single-GPU constraints. Despite fewer training images and lower instance-segmentation AP than the original implementation, the resulting pixel-wise comprehension outperforms prior state-of-the-art results by a large margin.
- Implementation: The Mask R-CNN implementation follows the original mask-branch structure with several implementation differences.It samples 256 regions from one image per propagation, resizes images to a 600-pixel shorter edge, and trains on COCO trainval35k excluding RefCOCO validation/test images.
- Implementation: Mask R-CNN achieves higher object-detection AP than Faster R-CNN under the same ResNet101 training setting.The paper attributes this difference to multitask training with additional mask supervision.
- Implementation: The implementation is not strictly comparable with the original Mask R-CNN because it was trained with fewer images.
- Implementation: ∼2 points lower AP is reported for the implementation than for the original Mask R-CNN.The shorter 600-pixel edge setting and smaller training batch size are suggested as possible reasons.
- Results: Pixel-wise comprehension results already outperform state-of-the-art results by a huge margin.Examples across RefCOCO, RefCOCO+, and RefCOCOg show module weights, word attention, subject attention, box comprehension, and segmentation outputs, including incorrect cases.