Source-linked AI summary
Multi-task Collaborative Network for Joint Referring Expression Comprehension and Segmentation
Gen Luo, Yiyi Zhou, Xiaoshuai Sun, Liujuan Cao, Chenglin Wu, Cheng Deng, Rongrong Ji
TL;DR
REC and RES identify referents from language but differ in their outputs, creating a case for joint learning while introducing prediction conflict. MCN jointly learns both tasks with shared visual-language components and uses CEM and ASNLS to manage that conflict. Across RefCOCO, RefCOCO+ and RefCOCOg, it reports gains of up to +7.13% for REC and +11.50% for RES over SOTA.
Problem
REC and RES are related referent-identification tasks whose joint learning is difficult because their predictions can conflict, especially when only a few instances are correct referents.
Method
MCN jointly learns REC and RES in one stage using shared visual and language components, with CEM aligning task regions and ASNLS suppressing unrelated RES responses from REC predictions.
Results
+7.13% for REC and +11.50% for RES are MCN's reported maximum gains over SOTA on RefCOCO, RefCOCO+ and RefCOCOg.
Takeaways & Limitations
Experiments support reinforcing REC and RES in a joint framework while reducing prediction conflict.
Abstract
from arXiv · showhide
Referring expression comprehension (REC) and segmentation (RES) are two highly-related tasks, which both aim at identifying the referent according to a natural language expression. In this paper, we propose a novel Multi-task Collaborative Network (MCN) to achieve a joint learning of REC and RES for the first time. In MCN, RES can help REC to achieve better language-vision alignment, while REC can help RES to better locate the referent. In addition, we address a key challenge in this multi-task setup, i.e., the prediction conflict, with two innovative designs namely, Consistency Energy Maximization (CEM) and Adaptive Soft Non-Located Suppression (ASNLS). Specifically, CEM enables REC and RES to focus on similar visual regions by maximizing the consistency energy between two tasks. ASNLS supresses the response of unrelated regions in RES based on the prediction of REC. To validate our model, we conduct extensive experiments on three benchmark datasets of REC and RES, i.e., RefCOCO, RefCOCO+ and RefCOCOg. The experimental results report the significant performance gains of MCN over all existing methods, i.e., up to +7.13% for REC and +11.50% for RES over SOTA, which well confirm the validity of our model for joint REC and RES learning.
1. Introduction
REC and RES identify referents from language but use different outputs, motivating joint learning that lets each task compensate for the other. MCN addresses their prediction conflict with collaborative architecture and two specialized designs, achieving strong benchmark gains.
- REC grounds referents with bounding boxes, whereas RES produces segmentation masks, despite both identifying visual instances from linguistic expressions.
- Joint learning is motivated because REC can improve RES referent localization, while RES pixel-level supervision can improve REC language-vision alignment.
- MCN jointly learns REC and RES in a one-stage framework with shared visual and language encoders but relatively separated multimodal inference branches.The separation accounts for task differences such as RES requiring higher-resolution feature maps.
- CEM maximizes consistency between task branches, while ASNLS softly suppresses unrelated RES regions using REC predictions to reduce prediction conflict.ASNLS is designed to tolerate errors in detection results better than hard processing methods.
- +7.13% in REC and +11.50% in RES are MCN's reported maximum gains over SOTA across RefCOCO, RefCOCO+ and RefCOCOg.The experiments also report that prediction conflict is effectively reduced by the proposed designs.
2. Related Work
Prior work develops REC and RES separately, with REC relying largely on multi-stage detection and ranking while RES embeds language into segmentation networks. Multi-task approaches and multimodal refinements improve efficiency or performance, but referent-selection ambiguity remains a challenge for RES.
- REC commonly detects salient image regions and ranks their multimodal matches, but this multi-stage design is computationally inefficient and constrained by the pre-trained detector.
- One-stage REC methods directly predict bounding boxes faster, yet their performance remains below popular two-stage approaches.
- RES typically embeds a language encoder into a segmentation network that learns multimodal features for decoding the referent mask.
- RES efficiency has been improved through multi-scale feature fusion, pyramidal refinement, and query- or transformer-based attention modules.
- Pixel-wise RES predictions can include incorrect regions such as overlapping people, motivating REC-based suppression of unrelated responses.
- Unlike conventional multi-task vision systems, MCN centers multi-task learning on language information, while target-instance selection intensifies prediction conflicts.
3. Multi-task Collaborative Network
MCN jointly learns REC and RES through shared visual-language representations, task-specific inference branches, and connections that promote collaboration while preserving distinct task requirements. CEM aligns spatial responses between tasks, while ASNLS refines RES masks using REC predictions with confidence-adaptive soft suppression.
- Framework: The REC and RES branches remain relatively independent because their distinct objectives and feature-resolution requirements make full sharing potentially counterproductive.REC uses a 13 × 13 feature map, whereas RES uses a 52 × 52 feature map.
- Framework: MCN extracts multi-scale visual features and a self-guided-attention textual feature, then fuses them into multimodal tensors for REC and RES.The visual backbone produces three scales, while the bi-GRU encoder produces the textual feature; multimodal fusion propagates language information across scales.
- Framework: REC receives bottom-up semantic information from RES to improve language-vision alignment during collaborative learning.The connection introduces semantics supervised by RES pixel-level labels into the REC branch.
- Consistency Energy Maximization: CEM maximizes consistency energy between REC and RES attention tensors so both tasks focus on similar visual regions and their learning processes remain connected.The attention tensors are converted into spatial energy distributions and inter-task correlations before the consistency objective is applied.
- Adaptive Soft Non-Located Suppression: ASNLS updates RES responses by enhancing regions inside REC’s predicted box and decaying responses outside it, then binarizes the refined mask.Its enhancement and decay factors are adaptively determined by REC confidence; lower confidence produces stronger suppression effects.
4. Experiments
MCN is evaluated on RefCOCO, RefCOCO+ and RefCOCOg for REC and RES, including benchmark comparisons, ablations, processing methods, speed, and prediction-conflict analysis. Results show gains from collaborative structure, CEM, and ASNLS across both tasks.
- Comparisons of different network structures: MCN significantly improves both tasks, while fully shared or completely independent multi-task structures do not maximize joint-learning benefits.The optimal multimodal tensor scales differ: 13 × 13 for REC and 52 × 52 for RES.
- Comparison of ASNLS and different post-processing methods: ASNLS outperforms Soft-NLS and avoids the RES degradation observed with hard processing on some metrics, including IoU and Acc@0.9.ASNLS suppresses unrelated RES regions using REC predictions, while soft processing provides greater error tolerance than direct feature cropping.
- Ablation study: CEM and ASNLS each provide performance gains, while CEM also reduces Inconsistency Error from 17.12% to 13.51%.The ablation study reports gains up to 7.04% for REC and 14.84% for RES.
- Comparison with the State-of-the-arts: MCN establishes state-of-the-art performance for both REC and RES on RefCOCO, RefCOCO+ and RefCOCOg.Its inference speed is 6 times faster than most existing multi-stage REC methods.
- Comparison with the State-of-the-arts: +7.13% REC improvement is reported on RefCOCO testB, while RES gains reach +8.39% on RefCOCO, +11.50% on RefCOCO+, and +3.32% on RefCOCOg.These comparisons are against advanced existing methods in the reported benchmark evaluations.
- Visualization analysis: MCN produces higher-quality boxes and masks in complex backgrounds, while CEM helps REC and RES focus on similar-semantic instances.The visualizations compare MCN with alternative multi-task structures and show effects of CEM and ASNLS.
5. Conclusion
The paper presents MCN as a first joint REC–RES framework that uses the complementary properties of both tasks. CEM and ASNLS address prediction conflict, and experiments show gains over SOTAs on three datasets.
- Conclusion: MCN jointly learns REC and RES to exploit the complementary properties of the two tasks.The paper describes this as the first attempt at joint REC and RES learning.
- Conclusion: CEM and ASNLS are introduced to address prediction conflict in the multi-task setting.The conclusion states that experiments show this conflict is well addressed.
- Conclusion: Experiments on three datasets report distinct performance gains over state-of-the-art REC and RES methods.The conclusion identifies these results as evidence for the proposed framework and designs.