Source-linked AI summary
Discriminative Triad Matching and Reconstruction for Weakly Referring Expression Grounding
Mingjie Sun, Jimin Xiao, Eng Gee Lim, Si Liu, John Y. Goulermas
TL;DR
Weakly supervised referring expression grounding must localize image referents without region-query mappings, while existing matching and reconstruction procedures are approximate. The paper introduces scalable discriminative triads with triad-level matching and reconstruction, achieving higher accuracy while reducing computational cost. It reports new state-of-the-art results across three RefCOCO benchmarks.
Problem
Weakly supervised referring expression grounding lacks region-query mappings during training, making existing matching and reconstruction procedures approximate.
Method
The method parses queries into one or multiple discriminative triads and uses triad-level matching and reconstruction for weakly supervised training.
Results
4.17%, 4.08%, and 7.8% accuracy gains over the previous SOTA are reported on RefCOCO, RefCOCO+, and RefCOCOg, respectively.
Takeaways & Limitations
The proposed network provides a lightweight, fast, and accurate WREG baseline that can be extended and adapted in future work.
Abstract
from arXiv · showhide
In this paper, we are tackling the weakly-supervised referring expression grounding task, for the localization of a referent object in an image according to a query sentence, where the mapping between image regions and queries are not available during the training stage. In traditional methods, an object region that best matches the referring expression is picked out, and then the query sentence is reconstructed from the selected region, where the reconstruction difference serves as the loss for back-propagation. The existing methods, however, conduct both the matching and the reconstruction approximately as they ignore the fact that the matching correctness is unknown. To overcome this limitation, a discriminative triad is designed here as the basis to the solution, through which a query can be converted into one or multiple discriminative triads in a very scalable way. Based on the discriminative triad, we further propose the triad-level matching and reconstruction modules which are lightweight yet effective for the weakly-supervised training, making it three times lighter and faster than the previous state-of-the-art methods. One important merit of our work is its superior performance despite the simple and neat design. Specifically, the proposed method achieves a new state-of-the-art accuracy when evaluated on RefCOCO (39.21%), RefCOCO+ (39.18%) and RefCOCOg (43.24%) datasets, that is 4.17%, 4.08% and 7.8% higher than the previous one, respectively.
1 INTRODUCTION
Weakly supervised referring expression grounding avoids costly region-query annotations but existing matching and reconstruction strategies are approximate and difficult to train. The paper introduces discriminative triads with triad-level matching and reconstruction to improve scalability, efficiency, and accuracy.
- Motivation: Weakly supervised training removes the costly region-query mapping required by supervised referring expression grounding.The task localizes a target object from a query sentence without proposal-query correspondences during training.
- Limitations of Existing Methods: Existing WREG methods either oversimplify query matching or attach multiple language-processing subnetworks to handle complex descriptions.Their reconstruction also relies on word-by-word sentence generation, producing unreliable losses and a heavy network.
- Proposed Representation: A discriminative triad represents a target object, a reference object, and their discriminative relation, allowing simple or complex queries to be parsed uniformly.For example, “the left man in black” becomes triads for the attributes black and left.
- Proposed Method: Triad-level matching scores triad-proposal compatibility and handles complex queries by considering the matching scores of multiple triads.This replaces traditional sentence-level matching with a formulation designed for weak supervision.
- Proposed Method: Triad-level reconstruction replaces full sentence generation with reconstruction of three triad units, improving loss reliability for weakly supervised training.The figure contrasts a 300-D MSE loss with the traditional 10,000-D CE loss.
- Results: 4.17%, 4.08%, and 7.8% accuracy gains over the previous SOTA are reported on RefCOCO, RefCOCO+, and RefCOCOg, respectively.The proposed network is also described as three times lighter and faster than the previous SOTA method.
2 RELATED WORK
Related work contrasts supervised REG, which uses proposal-query mappings and linguistic decomposition, with WREG approaches that must learn without region-query correspondences. Existing WREG methods reconstruct query or region information to create a training signal.
- Supervised REG: Supervised REG uses proposal-query mappings during training and can decompose queries into linguistic components for proposal matching.The components describe subject appearance, location, and relationships to other objects.
- Weakly Supervised REG: WREG removes region-query mappings and instead uses attention between queries and candidate proposals to reconstruct query information.Reconstruction-based training can also target candidate-region location parameters.
- Weakly Supervised REG: Existing WREG work includes query reconstruction from weighted visual features, location reconstruction, and matching based on decomposed linguistic components.These approaches use differences between original and reconstructed information as training signals.
3 METHODOLOGY
The methodology represents each query as one or more discriminative triads, matches triads to proposal pairs, and reconstructs triad-unit features for weakly supervised training.
- Region-level retrieval: WREG retrieves a target region by maximizing similarity between the query and candidate image proposals.The image contains proposal regions, and the target is selected through a scoring mechanism trained without region-query connection annotations.
- Discriminative triad generation: A query is converted into one or multiple discriminative triads, each comprising target, reference, and discriminative units.This representation supports unary, fragmentary, and complex referring forms through a shared formulation.
- Discriminative triad generation: Triads are generated by NLP analysis of sentence trees, part-of-speech tags, and dependency labels, with parsing patterns varying by phrase formation.The target unit is identified first, after which reference and discriminative units are generated for each triad.
- Triad-level matching: Triad-level matching scores proposal pairs using separate target, reference, and discriminative attention modules.The proposal pair supplies features for matching the target unit, reference unit, and their discriminative relation.
- Triad-level reconstruction: The hard reconstruction method replaces nondifferentiable argmax selection with differentiable Gumbel-Softmax.Its temperature parameter allows softmax to approach argmax as τ →0.
- Triad-level reconstruction: The reconstruction module rebuilds each triad unit's linguistic feature instead of reconstructing the entire query word by word.Three unit-specific reconstruction modules process target, reference, and discriminative units; inputs can use soft aggregation or a hard selection approximated with Gumbel-Softmax.
4 EXPERIMENTS
Experiments evaluate the proposed method across four datasets, comparing accuracy, efficiency, robustness, and ablated components. The method improves accuracy over prior WREG systems while using fewer parameters and faster inference.
- Datasets and metric: The method is evaluated on RefCOCO, RefCOCO+, RefCOCOg, and RefCLEF, with IoU above 0.5 counted as correct.RefCOCO+ excludes absolute location words, RefCOCOg contains longer expressions, and RefCLEF provides an auxiliary validation setting.
- Accuracy comparison: 4.17%, 4.08%, and 7.8% accuracy gains over KPRN are achieved on RefCOCO, RefCOCO+, and RefCOCOg, respectively.These comparisons use candidate proposals provided from bounding-box ground truth.
- Robustness: The method performs well across datasets under one fixed setting, whereas KPRN requires different settings to maximize performance on different datasets.This supports the reported robustness of the proposed approach across evaluation conditions.
- Accuracy comparison: 34.03% accuracy is obtained on RefCLEF, exceeding KPRN by 0.17% despite ambiguous and mistakenly annotated expressions.The proposed method also performs well when candidate proposals are generated by Faster RCNN.
- Efficiency: 7 million parameters and 0.04 seconds per image make the method approximately one-third as large and three times faster than the compared systems.ARN and KPRN use 24 and 20 million parameters and require 0.04 and 0.03 seconds per image, respectively, according to the reported comparison.
- Ablation studies: Removing any unit-level loss component reduces accuracy, with the largest decline occurring when the target-unit loss is removed.The target unit directly contributes to localization, while discriminative and reference units provide auxiliary information.
- Ablation studies: The triad-level reconstruction module improves performance, and the hard reconstruction setting outperforms the soft setting.The no-reconstruction setting performs worst, while Gumbel-Softmax helps focus reconstruction on the highest-attention visual feature.
- Ablation studies: Using all candidate triads during inference outperforms randomly selecting a single triad.The ablation indicates that exploiting the full set of parsed triads is important for final inference.
5 CONCLUSIONS
The paper introduces scalable discriminative-triad parsing and triad-level matching and reconstruction for weakly supervised referring-expression grounding. The resulting framework is designed as a lightweight, effective baseline that can be extended in future work.
- Conclusion: A query sentence can be parsed into one or multiple discriminative triads using a scalable representation.The triad-level matching and reconstruction modules provide lightweight weakly supervised training.
- Conclusion: The proposed framework is intended to provide a fast and accurate WREG baseline that can be extended and adapted in future work.