Source-linked AI summary
Improving Referring Expression Grounding with Cross-modal Attention-guided Erasing
Xihui Liu, Zihao Wang, Jing Shao, Xiaogang Wang, Hongsheng Li
TL;DR
Referring expression grounding must align diverse textual and visual information, but attention models may focus on only dominant correspondences. The paper introduces cross-modal attention-guided erasing to generate hard training samples that encourage complementary alignments, achieving state-of-the-art performance on three datasets.
Problem
Referring expression grounding requires aligning complex language with objects, attributes, locations, and relationships, while attention models can neglect complementary cross-modal correspondences.
Method
The method erases high-attention textual or visual information to generate hard training samples and encourage discovery of complementary textual-visual alignments.
Results
The approach achieves state-of-the-art performance on three referring expression grounding datasets.
Takeaways & Limitations
Cross-modal attention-guided erasing helps models learn more comprehensive textual-visual correspondences without increasing inference complexity.
Takeaways & Limitations
The theoretical analysis assumes the matching score receives a positive gradient during back-propagation.
Abstract
from arXiv · showhide
Referring expression grounding aims at locating certain objects or persons in an image with a referring expression, where the key challenge is to comprehend and align various types of information from visual and textual domain, such as visual attributes, location and interactions with surrounding regions. Although the attention mechanism has been successfully applied for cross-modal alignments, previous attention models focus on only the most dominant features of both modalities, and neglect the fact that there could be multiple comprehensive textual-visual correspondences between images and referring expressions. To tackle this issue, we design a novel cross-modal attention-guided erasing approach, where we discard the most dominant information from either textual or visual domains to generate difficult training samples online, and to drive the model to discover complementary textual-visual correspondences. Extensive experiments demonstrate the effectiveness of our proposed method, which achieves state-of-the-art performance on three referring expression grounding datasets.
1. Introduction
Referring expression grounding must align complex language with diverse visual information, but attention models can overfocus on dominant cross-modal cues. The paper addresses this by erasing dominant textual or visual information during training to encourage complementary correspondences.
- Referring expression grounding retrieves image regions matching natural-language descriptions while requiring understanding of objects, attributes, and relationships.
- Attention models improve alignment by focusing on important words, phrases, and visual regions.
- Dominant-feature attention can neglect complementary correspondences, causing reliance on specific words or visual concepts and underuse of latent training-pair information.
- Cross-modal erasing discards high-attention textual or visual information to generate difficult training pairs and drive discovery of complementary evidence.The method uses erased images with original queries or erased queries with original images, without increasing inference complexity.
- The approach introduces attention-guided erasing across textual and visual domains to learn comprehensive latent alignments.
- State-of-the-art performance is reported on three referring expression grounding datasets.
2. Related Work
Prior work uses modular networks and attention to decompose and align referring expressions, while this paper extends visual erasing to both language and vision. The backbone follows MattNet's modular reasoning structure.
- Referring expression grounding is commonly formulated as retrieving the image region best matching a natural-language expression.
- Prior methods model context, multi-step reasoning, modular subject-relationship reasoning, and joint generation-grounding objectives.
- MattNet decomposes cross-modal reasoning into subject, location, and relationship modules with language- and vision-based attention.
- The proposed work extends prior visual-only erasing by erasing information in both images and sentences.
- Erasing is used only to create training samples, so the approach does not increase inference complexity.
3. Cross-modal Attention-guided Erasing
The approach uses image-aware attention to erase dominant textual or visual information during training, encouraging complementary cross-modal alignments. It combines query erasing, module-selected visual erasing, and an erasing loss within a modular grounding architecture.
- Overview: The backbone formulates grounding as retrieval and uses MattNet’s subject, location, and relationship modules with fused module-level attention.Each candidate region receives a matching score against the query, and the highest-scoring proposal is selected.
- Overview: The method erases dominant information from either queries or visual features to generate challenging samples and discover complementary textual-visual alignments.Query words are replaced with an “unknown” token, while visual erasing targets a sampled module according to module-level attention weights.
- Visual erasing: Visual erasing selects a module from attention weights, then removes salient subject-region or context-object features before applying the erasing loss alongside original samples.The erasing loss enforces matching between erased visual features and original queries, and between erased queries and original visual features.
- Image-aware attention: Image-aware language attention combines whole-image features with word embeddings to produce module-level and word-level attention weights and module-specific query embeddings.The three modules are subject, location, and relationship; word-level weights indicate each word’s importance for a module.
- Query sentence erasing: Attention-guided query erasing samples an important word and replaces it with an “unknown” token, preserving sentence structure while encouraging alternative evidence.The word is sampled from a distribution defined by overall word-level significance, and erased queries participate in ranking-based alignment training.
- Analysis: The theoretical analysis argues that attention amplifies gradients for salient aligned features while suppressing updates for low-attention correspondences, motivating selective erasure.The method is presented as selective regularization that combines salient-feature removal with attention-weighted randomness, rather than random dropping alone.
4. Experiments
Experiments evaluate the method on three referring expression grounding datasets under ground-truth and detection-proposal settings. Results and visualizations show that cross-modal attention-guided erasing improves grounding and helps discover complementary textual-visual correspondences.
- Datasets and Evaluation Metrics: Experiments use RefCOCO, RefCOCO+, and RefCOCOg, with ground-truth and detection-proposal evaluation settings.For detection proposals, a prediction is correct when its intersection over union with the ground truth exceeds 0.5.
- Datasets and Evaluation Metrics: The ground-truth setting retrieves the best-matching region from ground-truth boxes, while the detection-proposal setting retrieves it from detector-generated proposals.The analysis and ablation study use the ground-truth setting to focus on textual-visual correspondence rather than detection performance.
- Results: CM-Att-Erase combines original and attention-guided erased training samples, improving on the CM-Att baseline and outperforming previous methods without increasing inference complexity.The authors report this pattern under both ground-truth and detection-proposal settings in Table 1.
- Qualitative Results: Qualitative results indicate that CM-Att-Erase handles complex cases better, especially when multiple visual and textual cues must be considered.Figure 7 compares CM-Att and CM-Att-Erase grounding results using red and green bounding boxes, respectively.
- Visualization of Attention and Erasing: Erasing a highly attended dress region redirects attention toward the girl’s tennis-playing action, while erasing “glasses” encourages alignment with “black phone.”These examples illustrate visual-region erasing and query-sentence erasing as complementary training interventions.
- Ablation Study: Ablations show that visual and query-sentence erasing are complementary, and cross-modal erasing is superior to self-attention-guided erasing.Iterative erasing is unsuitable because removing multiple key words or regions can eliminate sentence meaning or make the referred object unrecognizable; inference-time erasing does not improve performance.
5. Conclusion and Future Work
The paper addresses comprehensive cross-modal comprehension in referring expression grounding with cross-modal attention-guided erasing. By generating hard samples that remove dominant information, the method drives discovery of complementary alignments and achieves state-of-the-art performance on three datasets.
- Conclusion: Cross-modal attention-guided erasing generates hard training samples by discarding dominant textual or visual information to encourage complementary alignments.The approach targets over-concentration on the most significant cues during referring expression grounding.
- Conclusion: The proposed models achieve state-of-the-art performance on three referring expression grounding datasets.The conclusion presents this result as evidence of the approach’s effectiveness.