UniTAB: Unifying Text and Box Outputs for Grounded Vision-Language Modeling

1 Introduction

• Text describes image-level content but misses dense regions, whereas boxes locate regions but provide limited semantic description; grounding caption nouns can combine these strengths.The paper frames this unified output as a way to produce more comprehensive and interpretable image descriptions.Source: §1 · pp. 1–2
• UniTAB generates text and discrete box tokens autoregressively in one shared encoder-decoder, avoiding separate task-specific modules and alignment predictions.The paper motivates this design by noting that prior systems combine detection and vision-language modules, increasing complexity and running time while potentially limiting generalization.Source: §1 · pp. 2–4
• The special <obj> token brackets each grounded word and its box, signaling code-switching and representing their alignment directly in the output sequence.Words and boxes within a pair of <obj> tokens refer to the same entity.Source: Figure 1 · p. 2, §1 · p. 3
• UniTAB’s shared architecture supports multi-task training without task-specific heads, while its task-agnostic text-plus-box sequence can be reformatted for unseen tasks.Training one parameter set avoids storing task-specific model copies and allows data from different tasks to be used together.Source: §1 · pp. 3–4
• UniTAB significantly outperforms prior art, raising captioning CIDEr from 62.5 to 69.7 and grounding F1 from 8.44 to 12.95.The model is evaluated on seven vision-language benchmarks and achieves better or comparable performance to task-specific state of the art.Source: §1 · pp. 3–4

Grounded captioning

• Grounded captioning needs caption text, noun-phrase boxes, and alignments; prior methods use separate heads, often with offline detectors, unlike UniTAB’s single sequence.Source: §2 · p. 4

Vision-language pre-training (VLP)

• VLP shifted from detector-derived region features [41]—costly in computation and runtime—to raw-image grid features [32], but common output heads or decoders often remain task-specific and omit box prediction.Source: §2 · pp. 4–5

Unified VL framework

• VL-T5 and GPV [24] classify over detector-generated region candidates, while MDETR and UniT retain task-specific heads; UniTAB aims to take raw image and language inputs and generate structured text and boxes without format-specific modules.Source: §2 · p. 5

3.1 Architecture Overview

• A 6-layer transformer encoder processes concatenated image and text features, and a 6-layer autoregressive decoder generates tokens from both vocabularies.The architecture uses ResNet-101 for image features and RoBERTaBASE for text features, projected into a shared embedding space.Source: §3.1 · pp. 5–6

3.2 UniTAB Target Output Sequence

• UniTAB represents each box as four sequential discrete coordinate tokens and uses one vocabulary spanning text and box tokens for decoding.The unified vocabulary has size ntext+nbins+2, and the same nbins coordinate tokens serve all four box coordinates.Source: §3.2 · p. 6
• <obj> and <\\obj> bracket grounded words and their quantized box tokens, making word-box alignments directly recoverable from the generated sequence.For example, “a donut” is paired with four quantized coordinates inside the markers.Source: §3.2 · p. 6

3.3 UniTAB Training

• A single language-modeling objective trains UniTAB across up to three stages: vision-language pre-training, multi-task finetuning, and optional task-specific finetuning.The stages use different corpora and input-output designs while sharing the same objective.Source: §3.3 · pp. 6–7
• Pre-training alternates empty-string and image-description text inputs, while multi-task finetuning trains one model on supervised data from all 7 VL tasks.The multi-task model can initialize a second, task-specific finetuning stage; at inference, text and box outputs are extracted from the predicted sequence.Source: §3.3 · p. 7

4.1 Experiment Overview

• Experiments cover seven VL benchmarks, with pretraining on around 200K images and 1.3M grounded image-text pairs and multi-task finetuning on all seven downstream datasets.The pretraining corpus aggregates Flickr30k Entities, COCO, and Visual Genome data; Conceptual Captions and SBU image-text pairs without boxes can optionally be added.Source: §4.1 · p. 8

4.2 Comparison with Prior Arts

• UniTAB outperforms state of the art on referring-expression comprehension and phrase grounding, with multi-task finetuning improving all splits over UniTABSeparate.Its unified text-and-box output represents word-box alignments with the <obj> token; for grounded captioning, it also avoids pre-generated object regions and multiple output heads.Source: §4.2 · pp. 9–10
• On COCO captioning, CIDEr improves from 117.3 to 119.8 over prior methods using similar pretraining image counts, and reaches 123.1 with extra image-text pairs.UniTAB requires neither input region proposals nor object tags.Source: §4.2 · p. 10, Table 4 · p. 11
• UniTAB achieves competitive VQAv2 results and performs better than the discriminative approach on the Karpathy out-of-domain subset.The reported evaluation includes the official test-dev/test-std splits and the Karpathy split, which is divided into in-domain and out-of-domain subsets.Source: Table 5 · p. 11, §4.2 · p. 11
• UniTABPre-finetuning is better or comparable to state of the art across experimented tasks; grounded captioning raises F1all from 8.44 to 12.95, F1loc from 22.78 to 34.79, and CIDEr from 62.5 to 69.7.Adding image-text data without box annotations further raises grounded-captioning CIDEr from 69.7 to 74.2.Source: Table 2 · p. 9, §4.2 · p. 9, Table 6 · p. 11

4.3 Ablation and Analysis

• Pretraining consistently improves UniTABSeparate over its scratch variant, while UniTABShared matches or exceeds UniTABSeparate with seven times fewer model parameters; UniTABPre-finetuning further improves results.Adding task-specific input prefixes has no major performance effect.Source: §4.3 · pp. 11–12
• Without ImageNet images or annotations, UniTABShared outperforms prior art on ImageNet localization; adding 4M ungrounded image-text pairs yields +8.6 grounded-captioning CIDEr, +3.8 COCO CIDEr, and +2.5% absolute VQA accuracy.The <obj> token represents word-box alignment and provides text-box code-switching hints that simplify sequence prediction; the paper reports further design ablations in Appendix B.Source: §4.3 · pp. 12–13, Table 7 · p. 13, Table 8 · p. 13

4.4 Qualitative Results

• Qualitative examples show the shared output sequence supporting captioning, phrase grounding, and referring-expression comprehension, as well as transfer to COCO and zero-shot ImageNet localization and grounded captioning.In one referring-expression example, UniTAB localizes the cat in the mirror and aligns the predicted box with “the cat.”Source: §4.4 · pp. 13–14, Figure 3 · p. 14

• UniTAB uses <obj> to generate text and box predictions together, with word–box alignments represented naturally in the output sequence.Source: §5 · p. 14
• Unifying text and box outputs helps UniTAB better address grounded vision-language tasks such as grounded captioning.Source: §5 · p. 14
• UniTAB’s unified multi-task network and task-agnostic output sequence design make it parameter efficient and generalizable to new tasks.Source: §5 · p. 14
• The authors see potential for UniTAB to support vision systems with stronger intelligence, including in-context learning and instruction tuning.Source: §5 · p. 14

A Experiment Details

• The decoder embeds previous target tokens with token and position embeddings, without type embeddings to distinguish text from box tokens.Source: A Experiment Details · p. 20
• The 180K corpus combines COCO and Visual Genome images and annotations; the 200K corpus contains image-text pairs and grounded box annotations.Source: A Experiment Details · p. 20
• The 3M corpus contains Conceptual Captions image-text pairs, while the 4M corpus adds COCO, Visual Genome, and SBU Captions.Source: A Experiment Details · p. 20
• UniTAB extracts text, boxes, and word-box alignments from its unified output sequence for downstream evaluation; grounded captioning follows the standard benchmark.Source: A Experiment Details · p. 20
• For phrase grounding, UniTAB repeats the query and uses <obj> tokens to extract alignments; referring-expression comprehension uses the first predicted box, while COCO captioning and VQA evaluate text outputs.Incorrect repeated query text can disarray phrase-grounding alignments; COCO captioning uses caption metrics and VQA uses exact match.Source: A Experiment Details · p. 20

<obj> token

• Adding <obj> tokens yields around 1% accuracy improvement on Refcocog, where word-box alignment is not required.The ablation compares UniTABSeparate with variants removing <\obj> or both special tokens.Source: B Ablation Studies on Decoding Design · p. 21, Table 10 · p. 22

Decoder type embedding

• Decoder type embeddings had no major influence on performance, so UniTAB does not use them.The ablation compares no type embedding, text/box types, and a third type for <obj> and <\obj> on Refcocog and Flickr30k Entities grounding.Source: B Ablation Studies on Decoding Design · pp. 21–22, Table 11 · p. 22

Number of object tokens

• Grounding performance stays steady once object-token counts are large enough to avoid quantization error, at around 200 tokens for the experimented datasets.The ablation covers Refcocog and Flickr grounding.Source: B Ablation Studies on Decoding Design · p. 22, Figure 4 · p. 22

Decoding sampling method

• Nucleus sampling gives a marginal improvement over argmax decoding on Flickr phrase grounding, with smaller gains than reported for object detection.The paper attributes the difference to object detection allowing multiple correct output sequences, while UniTAB uses one fixed text+box target sequence.Source: B Ablation Studies on Decoding Design · p. 22, Figure 5 · p. 22

Decoding syntactic restriction

• UniTAB predictions had zero scanned syntactic failures on Refcocog grounding, Flickr grounded captioning, and COCO captions generated by UniTABShared.The scanned failures involved malformed box-token runs or unpaired/misplaced <obj> and <\obj> tokens.Source: B Ablation Studies on Decoding Design · pp. 22–23
• Syntactic restrictions changed Refcoco accuracy by +0.9 and Flickr grounded-captioning results by −0.2 CIDEr and −0.15 F1all.The restrictions mask invalid token logits during training; they have no major influence on performance.Source: B Ablation Studies on Decoding Design · p. 23

Pre-training with additional box annotations

• Additional box-only pretraining improves grounding accuracy and grounded-captioning F1; CIDEr rises 65.6 to 70.0 for grounded captioning and 119.3 to 120.7 for COCO captioning, while VQA accuracy rises 66.6% to 68.4%.Separatebox uses object boxes and class annotations without captions, training the model to generate text-and-box sequences for up to 32 shuffled objects per image.Source: C Discussions · p. 23

Multi-task finetuning with prefix

• UniTABPrefix achieves comparable performance with and without task-specific input prefixes across the experimented tasks and datasets.Prefixes identify tasks with strings such as “visual grounding:” and are added during multitask finetuning.Source: Table 13 · p. 24, C Discussions · p. 24

Robustness and bias analyses

• Following VQA-CE, UniTAB’s gain over UpDown was larger on “counterexamples” (+9.76%) than on “easy” (+6.17%), indicating better robustness against shortcuts than the UpDown baseline.Source: C Discussions · p. 24
• For gender error rate, UniTAB scores 9.21% versus Baseline-FT’s 19.30%, but is slightly worse than Equalizer.The discussion notes that robust and unbiased modeling remains challenging and could be improved.Source: Table 15 · p. 24, C Discussions · pp. 24–25

Multi-task finetuning with prefix

• UniTABShared performs well across tasks while remaining parameter efficient and showing promise for zero-shot generalization.Source: C Discussions · p. 24

D Qualitative Results

• On Flickr30k Entities, UniTAB produces smooth, informative captions with attributes and scene details while grounding noun phrases, including tiny objects, to image regions.Examples include “young boy” and “blue shirt,” as well as small objects such as a bat, baseball, and red ball.Source: D Qualitative Results · pp. 25–26
• On MSCOCO, UniTAB transfers Flickr30k Entities grounding ability despite no COCO grounding annotations, achieves caption performance comparable to the state of the art, and grounds every caption noun phrase.The paper notes that this grounding ability is important for reducing object hallucination and boosting interpretability and fairness.Source: D Qualitative Results · pp. 26–27
• On ImageNet, UniTAB’s captioning and grounding predictions are reported as similar in quality to those on Flickr30k Entities and MSCOCO.The authors describe this as evidence that UniTAB generalizes well to ImageNet images.Source: D Qualitative Results · p. 27
• On RefCOCO and RefCOCOg, UniTAB distinguishes query-selected objects, localizes the head noun in long expressions, and usually emits one box for the referring task.The authors conjecture it infers task type from input form, such as a short query versus a complete sentence.Source: D Qualitative Results · p. 27