Source-linked AI summary
SeqTR: A Simple yet Universal Network for Visual Grounding
Chaoyang Zhu, Yiyi Zhou, Yunhang Shen, Gen Luo, Xingjia Pan, Mingbao Lin, Chao Chen, Liujuan Cao, Xiaoshuai Sun, Rongrong Ji
TL;DR
Visual grounding methods often require complex task-specific architectures and loss functions, limiting unified modeling across REC, phrase localization, and RES. SeqTR instead serializes grounding outputs as coordinate-token sequences and predicts them with one transformer and cross-entropy loss, achieving or matching state-of-the-art performance on five benchmarks. The paper concludes that a simple, universal approach is feasible.
Problem
Existing visual grounding approaches require substantial expertise in task-specific architectures and customized loss functions, while transformer methods still retain task-specific branches or heads.
Method
SeqTR casts grounding as image-and-text-conditioned point prediction, serializes boxes or masks into discrete coordinate tokens, and uses one transformer encoder-decoder with cross-entropy loss.
Results
SeqTR achieves or matches state-of-the-art performance on five benchmark datasets and outperforms large-scale BERT-style models with less pre-training expenditure.
Takeaways & Limitations
A simple universal point-prediction framework can unify visual grounding tasks without task-specific branches or heads.
Takeaways & Limitations
Uniformly sampling 36 points from ground-truth masks reaches 95.63 mIoU on RefCOCO validation, restricting RES performance below a 100 mIoU upper bound.
Abstract
from arXiv · showhide
In this paper, we propose a simple yet universal network termed SeqTR for visual grounding tasks, e.g., phrase localization, referring expression comprehension (REC) and segmentation (RES). The canonical paradigms for visual grounding often require substantial expertise in designing network architectures and loss functions, making them hard to generalize across tasks. To simplify and unify the modeling, we cast visual grounding as a point prediction problem conditioned on image and text inputs, where either the bounding box or binary mask is represented as a sequence of discrete coordinate tokens. Under this paradigm, visual grounding tasks are unified in our SeqTR network without task-specific branches or heads, e.g., the convolutional mask decoder for RES, which greatly reduces the complexity of multi-task modeling. In addition, SeqTR also shares the same optimization objective for all tasks with a simple cross-entropy loss, further reducing the complexity of deploying hand-crafted loss functions. Experiments on five benchmark datasets demonstrate that the proposed SeqTR outperforms (or is on par with) the existing state-of-the-arts, proving that a simple yet universal approach for visual grounding is indeed feasible. Source code is available at https://github.com/sean-zhuh/SeqTR.
1 Introduction
SeqTR reframes visual grounding as sequence prediction, unifying bounding-box and mask tasks with one transformer and cross-entropy objective. It achieves state-of-the-art or comparable performance across five benchmark datasets while reducing architectural and optimization complexity.
- Motivation: Existing visual grounding methods require substantial expertise in task-specific architectures, multimodal reasoning modules, and customized losses.Transformer-based approaches simplify some architecture components but still retain task-specific losses and branches or heads.
- Approach: SeqTR reformulates grounding as point prediction, serializing bounding boxes or masks into discrete coordinate-token sequences conditioned on image and text.A standard transformer encoder-decoder updates multimodal features and autoregressively predicts the grounding sequence.
- Approach: SeqTR unifies grounding tasks with a standard transformer architecture and one cross-entropy loss, without task-specific branches or heads.This design reduces the complexity of both architecture design and optimization.
- Approach: RES is integrated by clockwise sampling of the mask contour into a sequence of N points, eliminating the additional convolutional mask decoder.The sampling scheme enables segmentation within the same sequential modeling framework without network architecture modifications.
- Results: SeqTR achieves or maintains on-par state-of-the-art performance on five benchmark datasets and outperforms large-scale BERT-style models with less pre-training expenditure.The reported benchmarks are RefCOCO, RefCOCO+, RefCOCOg, ReferItGame, and Flickr30K Entities.
2 Related Work
Prior work addresses visual grounding through two-stage or one-stage REC pipelines, specialized RES alignment mechanisms, and multi-task architectures. These approaches commonly rely on task-specific components for coordinating REC and RES.
- REC: REC methods use either two-stage region-proposal ranking or faster one-stage pipelines based on dense anchors.The two-stage approach ranks extracted proposals by similarity to the language query, while one-stage methods predict from dense anchors.
- RES: RES methods perform fine-grained language grounding with attention mechanisms for cross-modal alignment and pixel-wise mask prediction.Examples include asymmetric co-attention, cascaded attention reasoning, and instance-level attention supervision.
- Multi-task grounding: Multi-task methods jointly address REC and RES by sharing region information or transformer features across task-specific branches.MCN links the branches through consistent energy maximization, while RefTR still requires an additional convolutional mask decoder for RES.
3 Method
SeqTR unifies visual grounding by representing bounding boxes and masks as discrete coordinate-token sequences predicted from image and text. It shares a compact transformer architecture and cross-entropy objective across tasks, using contour sampling to serialize masks.
- Problem formulation: SeqTR formulates visual grounding as point prediction conditioned on image and text, serializing grounding information into discrete coordinate tokens.The same formulation covers bounding boxes and binary masks.
- Objective: A shared cross-entropy objective predicts each target token from multimodal features and preceding coordinate tokens using causal attention.The loss applies per-token weights and conditions prediction only on preceding coordinate tokens.
- Sequence construction: Task-specific [TASK] tokens identify the grounding task, while [EOS] marks sequence termination and learnable embeddings mark sequence boundaries.Different task tokens enable multi-task grounding within the same network.
- Sequence construction: Bounding-box corners are normalized by image width and height, quantized into M integer bins, and represented with a shared coordinate embedding vocabulary.The bounding box uses two corner points before quantization.
- Sequence construction: For RES, SeqTR samples N clockwise points from the referred mask contour and quantizes them as coordinate tokens instead of predicting pixels directly.Uniform sampling distributes points more evenly than center-based sampling and preserves irregular contours more precisely.
- Architecture: The architecture uses a one-layer bidirectional GRU, coarsest-scale visual features, Hadamard multimodal fusion, and a six-layer encoder plus three-layer decoder.The decoder generates coordinates autoregressively, and predicted tokens are mapped back to image coordinates.
4 Experiments
SeqTR is evaluated across five visual-grounding datasets and REC, phrase localization, and RES tasks, with ablations examining sampling, token weighting, and multi-task training. It achieves competitive or superior results while retaining real-time inference and a simple architecture.
- Comparisons with State-of-the-Arts: SeqTR achieves new state-of-the-art performance on RefCOCO, RefCOCO+, and RefCOCOg, while reaching 69.66 on ReferItGame and 81.23 on Flickr30K Entities.Its inference speed is 50ms on a GTX 1080 Ti with batch size 1.
- Comparisons with State-of-the-Arts: SeqTR extends to RES without architecture modifications, outperforming various sophisticated models and matching VLT.Large-corpus pre-training yields up to 10.78% absolute improvement.
- Comparisons with State-of-the-Arts: With large-corpus pre-training, SeqTR uses 7.9M transformer parameters and surpasses MDETR, especially on RefCOCOg with up to 2.48% improvement.The reported parameter count is twice as few as MDETR’s.
- Ablation Studies: Increasing the first token weight to 1.5 improves performance over weighting later tokens, while multi-task supervision slightly degenerates performance despite reducing inconsistency error.The first token is the [TASK] token, followed by coordinate tokens.
- Ablation Studies: Uniform mask-point sampling provides higher practical performance and preserves more details than center-based sampling.The reported upper bounds are 95.57 for uniform sampling and 91.58 for center-based sampling.
- Ablation Studies: The number of sampled points trades off inference speed and performance, with 18 points best for RefCOCO and 12 points best for RefCOCO+ and RefCOCOg.No shuffling or 0.2 shuffling performs best across the reported dataset groups.
- Ablation Studies: Cross-attention maps show axial attention: boundary-relevant regions are attended to when predicting successive bounding-box coordinates.The decoder attends left, top, and corresponding boundary sides when generating x1, y1, x2, and y2.
5 Conclusions
The paper concludes that SeqTR reformulates visual grounding as point prediction and unifies different tasks with a general network. Experimental results support the feasibility of a simple, universal approach.
- Conclusions: SeqTR reformulates visual grounding as point prediction and uses a standard transformer encoder-decoder with cross-entropy loss across tasks.The formulation represents grounding information as coordinate sequences.
- Conclusions: SeqTR unifies different visual grounding tasks without architectural modifications or task-specific branches and heads.The paper presents this as reducing architecture and optimization complexity.
- Conclusions: The reported experiments show that SeqTR can ground language queries onto corresponding regions, supporting the feasibility of a simple yet universal approach.The paper evaluates this conclusion across the visual-grounding setting described in the work.
A.1 More implementation details
The implementation uses EMA and large-scale jittering in standard training, while disabling both during pre-training and ablation studies.
- More implementation details: EMA with decay rate 0.999 is used to accelerate training convergence.The procedure follows prior work.
- More implementation details: Training uses large-scale jittering with strength ranging from 0.3 to 1.4, without random color distortion, affine transformation, or horizontal flipping.EMA and LSJ are disabled during pre-training and ablation studies.
A.2 Sequence construction for Multi-task grounding
Multi-task grounding constructs decoder sequences with separate task tokens for box-level and pixel-level grounding. Bounding boxes and masks are represented through coordinate sequences.
- Sequence construction for Multi-task grounding: Multi-task input and target sequences use distinct randomly initialized [TASK] tokens for REC or phrase localization versus RES.The box-level task uses bounding-box coordinates, while the pixel-level task uses mask coordinates.
- Sequence construction for Multi-task grounding: Coordinates are marked by superscripts b for bounding boxes and m for binary masks in the sequence construction.This distinguishes the two grounding representations within multi-task sequences.
A.3 Nucleus sampling for RES
The study evaluates nucleus sampling for generating mask-coordinate sequences in RES and finds no quality benefit, so SeqTR uses argmax inference.
- Nucleus sampling does not improve the quality of generated sequences representing predicted binary masks.The method samples from a truncated ranked list of discrete coordinate tokens; setting p=0 is equivalent to argmax selection.
- SeqTR uses argmax because nucleus sampling adds the hyper-parameter p without improving mask-sequence quality.
A.4 More qualitative results
Qualitative errors arise from confusing the referent with another same-category object or from representing only the largest segment of a multi-segment mask.
- Predictions may shift to a same-category object that the language query does not reference.The paper suggests better multi-modal fusion could suppress such salient-object confusions, but does not add that module.
- When a referent mask has multiple segments, SeqTR samples only the largest contour segment and discards the others.This preprocessing choice causes the output to ground the query onto the largest segment.
- The qualitative visualizations show predicted masks alongside ground-truth binary masks inferred from the language query.