Source-linked AI summary
Referring Transformer: A One-step Approach to Multi-task Visual Grounding
Muchen Li, Leonid Sigal
TL;DR
Visual grounding requires associating free-form language with image regions, while prior methods face proposal, anchor, or task-specific architecture limitations. Referring Transformer uses a one-stage transformer with visual-lingual encoding and contextualized queries to jointly produce bounding boxes and segmentation masks. It reports large-margin improvements on both REC and RES, with further gains from pretraining and benefits from multi-task training.
Problem
Prior visual grounding methods either use two-stage proposals with limited performance or require complex task-specific one-stage architectures.
Method
Referring Transformer is an end-to-end one-stage transformer that fuses visual and linguistic features, decodes contextualized phrase queries, and directly outputs bounding boxes and segmentation masks for REC and RES.
Results
The model outperforms state-of-the-art methods by large margins on REC and RES across five and three datasets respectively, while achieving real-time runtime.
Takeaways & Limitations
Joint REC and RES training improves both tasks, while contextualized information and external-dataset pretraining further improve performance.
Takeaways & Limitations
Two-stage methods cannot condition proposal generation on the query phrase, constraining their performance upper bound.
Abstract
from arXiv · showhide
As an important step towards visual reasoning, visual grounding (e.g., phrase localization, referring expression comprehension/segmentation) has been widely explored Previous approaches to referring expression comprehension (REC) or segmentation (RES) either suffer from limited performance, due to a two-stage setup, or require the designing of complex task-specific one-stage architectures. In this paper, we propose a simple one-stage multi-task framework for visual grounding tasks. Specifically, we leverage a transformer architecture, where two modalities are fused in a visual-lingual encoder. In the decoder, the model learns to generate contextualized lingual queries which are then decoded and used to directly regress the bounding box and produce a segmentation mask for the corresponding referred regions. With this simple but highly contextualized model, we outperform state-of-the-arts methods by a large margin on both REC and RES tasks. We also show that a simple pre-training schedule (on an external dataset) further improves the performance. Extensive experiments and ablations illustrate that our model benefits greatly from contextualized information and multi-task training.
1 Introduction
Visual grounding links free-form language to image regions, but prior two-stage and one-stage approaches impose proposal, anchor, or contextualization limitations. Referring Transformer addresses these issues with a one-stage transformer that jointly performs REC and RES and achieves large-margin improvements.
- Visual grounding localizes image regions corresponding to free-form linguistic expressions, supporting applications including image captioning, visual question answering, and visual reasoning.
- Two-stage architectures first generate image-region proposals and then associate regions with query phrases.Their proposal mechanism cannot condition on the query phrase, limiting the attainable performance.
- One-stage approaches avoid proposal stages but often require ad hoc dense anchors and process each query phrase independently, limiting contextualized decisions.
- Referring Transformer is an end-to-end one-stage DETR-inspired architecture that directly produces bounding boxes and segmentation masks without dense anchor definitions.A visual-lingual encoder and contextualized transformer decoder jointly support detection and segmentation outputs.
- The model jointly trains REC and RES with contextualized phrase queries, enabling simultaneous grounding across both tasks.Its implicit one-to-one correspondence between referring phrases and outputs avoids Hungarian matching for the loss.
- Up to 8.5% REC and 19.4% RES improvements are reported on RefCOCO, while vanilla and pretrained models outperform state-of-the-art methods on both tasks.
2 Related works
Prior work developed separate or partially integrated approaches for REC, RES, multi-task learning, pretrained multimodal transformers, and transformer detection. Referring Transformer combines these directions in a contextualized multi-task grounding framework.
- REC predicts a bounding box tightly enclosing a language query, while RES predicts a segmentation mask describing the referred region’s shape.
- Two-stage REC methods are limited by the speed and accuracy of their pretrained region proposals, motivating one-stage alternatives.
- RES methods commonly fuse multimodal information with segmentation networks, with later work emphasizing improved multimodal interactions through progressive fusion or cross-modal attention.
- Earlier segmentation approaches can produce noisy, irregular masks, whereas the described model produces fine-grained shapes under challenging occlusions or shadows.
- Prior multi-task work jointly addressed REC and RES using shared architectures and an explicit consistency constraint between task feature activations.
- Pretrained multimodal transformers derive cross-modal representations through multi-task pretraining on large aligned image-text datasets.
- DETR reformulates detection as set prediction with transformer-decoded learnable queries, but is disadvantaged by optimization difficulty and long training times.Referring Transformer adopts a similar pipeline while focusing on multimodal alignment and contextualized expressions.
- Contemporaneous transformer-based referring methods differed in scope: some focused specifically on REC, whereas this approach addressed REC and RES simultaneously.
3 Approach
The approach is an end-to-end, one-stage transformer that jointly grounds referring expressions as bounding boxes and segmentation masks. It fuses image and language features, generates contextualized phrase queries, and decodes them through jointly trained REC and RES heads.
- Overview: Given an image and query phrases, the model predicts one bounding box and segmentation mask for each phrase.It can also use an optional contextual text source, such as the sentence from which phrases were parsed.
- Feature Extraction: A visual-lingual transformer encoder fuses projected image and text features with positional and modality-label embeddings.The encoder produces a joint multi-modal feature sequence from convolutional image features and BERT text representations.
- Referring Decoder: Phrase-specific queries combine BERT phrase embeddings, pooled visual-lingual context, and a learnable embedding through an MLP.The pooled context is extracted using the phrase’s left and right bounds in the original text.
- Referring Decoder: An attention graph convolution contextualizes phrase queries before non-causal cross-attention decodes visual-lingual information for each query.The decoder allows information flow among all phrase queries and then attends to encoder features.
- Multi-task Training: Decoded query features feed separate detection and segmentation heads, while joint training optimizes L = Lseg + Ldet.REC uses a weighted L1 and Generalized IOU objective; RES uses focal and DICE losses with an FPN-like mask decoder.
- Pretraining: Simple pretraining on Visual Genome region-description splits can produce performance comparable to or better than some state-of-the-art pretrained models.The paper motivates this schedule by noting that transformers generally require substantial data for training.
4 Experiments
Experiments evaluate joint REC and RES performance across RefCOCO-family, Flickr30k Entities, and ReferIt datasets, using standard task metrics and comparisons with prior methods. Results show consistent gains, benefits from pretraining and multi-task learning, faster parallel inference, and improved handling of attributes, shapes, shadows, and occlusions.
- Evaluation setup: REC uses precision with IoU > 0.5, while RES uses Mean IoU between predicted and ground-truth masks.These metrics are applied to bounding-box localization and segmentation-mask quality, respectively.
- RefCOCO-family results: +2.7%/+4%/+2.1% on RefCOCO, +6.6%/+4.3%/+8.5% on RefCOCO+, and +4.4%/+5.1% on RefCOCOg were reported for REC over prior approaches.The three RefCOCO and RefCOCO+ values correspond to the reported benchmark splits; RefCOCOg reports two improvements.
- RefCOCO-family results: The model achieved the best reported RES performance on RefCOCO, RefCOCO+, and RefCOCOg, substantially exceeding the state of the art.Pretraining on REC also gave a large boost to RES despite using no segmentation masks during pretraining.
- Additional REC datasets: On ReferIt, the smaller model performed better by a large margin than recent one-stage methods, with a consistent boost from pretraining.The model also predicts all referring queries in parallel, enabling real-time performance while jointly producing detection boxes and segmentation masks.
- Qualitative analysis: Qualitative comparisons found better attribute modeling in REC and more detailed, smoother RES masks under shadows, occlusions, and difficult texture boundaries.The model still showed failures on number recognition and visually ambiguous spatial or relational cases.
- Ablation studies: Multi-task training improved both REC and RES, reduced cross-task inconsistency to an IE score of 7.54%(-40%), and supported collaborative learning.REC loss aided localization and early convergence, while RES loss supplied finer-grained shape information.
- Ablation studies: Context encoders, query decoders, context features, phrase features, and learnable embeddings each contributed to performance in component ablations.Removing context or phrase guidance caused considerable degradation and prevented correspondence between multiple queries and outputs.
5 Conclusions and Future Work
Referring Transformer jointly addresses REC and RES with contextualized multi-expression references, outperforming state of the art across both task families while achieving real-time runtime. The authors note a single-region-per-expression assumption as a limitation and identify multi-region prediction as future work.
- The model jointly trains REC and RES while supporting contextualized multi-expression references.
- It outperforms state of the art by a large margin on five REC datasets and three RES datasets while achieving real-time runtime.
- A limitation is the assumption that each expression refers to only one region.The authors plan to explore predicting multiple regions for each referring entity when necessary.
A More Details for Referring Expression Segmentation (RES)
The RES task head combines decoded phrase queries with visual-lingual features to compute attention, upsample features, and refine segmentation masks using backbone connections.
- The RES head computes query attention scores by taking dot products between decoded query embeddings and visual features.The attention score matrix has M attention heads, with M=8 in the implementation.
- It concatenates attention scores with visual features before sending them through convolutional up-sampling blocks.The blocks use convolutions with stride 2.
- Residual connections from multiple ResNet stages refine the up-sampled features.
B Additional Implementation Details
The appendix describes dataset-specific training schedules, Visual Genome pretraining, and planned code release for reproducing reported scores.
- Pretraining: Visual Genome pretraining uses 100k images with an average of 40 region descriptions per image for six REC epochs.The learning rate starts at 1e-4 and decays by 10x after four epochs; the resulting model initializes dataset-specific fine-tuning.
- RefCOCO Training: RefCOCO experiments first train REC with auxiliary loss for 60 epochs, then jointly train RES and REC without auxiliary loss for 30 epochs.Both stages use a learning rate of 1e-4, with decay in the second stage at epoch 10.
- ReferItGame / Flickr30k Training: ReferItGame and Flickr30k Entities training lasts 90 and 60 epochs, respectively.Learning-rate decay occurs at epochs 60 and 40, respectively.
- Source Code: The authors provide core model code in the supplement and plan to release complete code with checkpoints after acceptance.
C Additional Results
Additional experiments compare qualitative outputs, input-resolution trade-offs, and contemporaneous methods. The results emphasize strong REC performance, faster multi-query inference, and a broader multi-task formulation despite some pretrained REC comparisons being marginally worse.
- Additional RES Results: RES mask comparisons control for REC localization by evaluating cases where both MCN and the model localize correctly.The additional RES results use RefCOCO+ testA and testB images.
- Different Input Resolution: Input resolution presents a performance-speed trade-off, although the model is designed to process 640 × 640 images in real time.
- Additional Qualitative REC Results: Figure 4 compares MCN outputs, the model’s outputs, and ground truth across RefCOCO+ testA, testB, and RefCOCOg.Orange, blue, and red boxes represent MCN, the model, and ground truth; rows correspond to the three listed splits.
- Comparison with Contemporaneous Work: 10.4% better on RefCOCO+ Val, the largest reported improvement over the closest concurrent work, [7].The passage also reports a 9.2% improvement on RefCOCO+ testB.
- Comparison with Contemporaneous Work: Against [23] with pretraining, the model performs similarly on RefCOCO and marginally worse on RefCOCO+ and RefCOCOg.The authors attribute differences to larger pretraining data and a more sophisticated language model used by [23], while noting their own multi-task formulation is more general overall.