PaperlayerView exact revision on arXiv

Source-linked AI summary

UniTAB: Unifying Text and Box Outputs for Grounded Vision-Language Modeling

Zhengyuan Yang, Zhe Gan, Jianfeng Wang, Xiaowei Hu, Faisal Ahmed, Zicheng Liu, Yumao Lu, Lijuan Wang

arXiv:2111.12085v2cs.CV

TL;DR

Grounded vision-language tasks need text descriptions, object boxes, and explicit word-box alignments, but prior systems use separate modules for these outputs. UniTAB generates them in one shared task-agnostic sequence with <obj> alignment tokens, achieving better or comparable results across seven VL benchmarks and strong grounded-captioning gains.

  • Problem

    Grounded VL requires generating text and boxes together while aligning words with object regions, whereas existing systems separately predict these outputs and use multiple modules.

  • Method

    UniTAB uses a shared autoregressive decoder for text and quantized box tokens, with <obj> delimiters representing word-box alignments inline.

  • Results

    UniTAB achieves better or comparable performance to task-specific state of the art across 7 VL benchmarks, including grounded-captioning improvements from 62.5 to 69.7 CIDEr and 8.44 to 12.95 grounding F1.

  • Takeaways & Limitations

    A unified multi-task network and task-agnostic output sequence make UniTAB parameter efficient and potentially generalizable to new VL tasks.

Abstract

from arXiv · show

We propose UniTAB that Unifies Text And Box outputs for grounded vision-language (VL) modeling. Grounded VL tasks such as grounded captioning require the model to generate a text description and align predicted words with object regions. To achieve this, models must generate desired text and box outputs together, and meanwhile indicate the alignments between words and boxes. In contrast to existing solutions that use multiple separate modules for different outputs, UniTAB represents both text and box outputs with a shared token sequence, and introduces a special <obj> token to naturally indicate word-box alignments in the sequence. UniTAB thus could provide a more comprehensive and interpretable image description, by freely grounding generated words to object regions. On grounded captioning, UniTAB presents a simpler solution with a single output head, and significantly outperforms state of the art in both grounding and captioning evaluations. On general VL tasks that have different desired output formats (i.e., text, box, or their combination), UniTAB with a single network achieves better or comparable performance than task-specific state of the art. Experiments cover 7 VL benchmarks, including grounded captioning, visual grounding, image captioning, and visual question answering. Furthermore, UniTAB's unified multi-task network and the task-agnostic output sequence design make the model parameter efficient and generalizable to new tasks.

1 Introduction

UniTAB unifies text, box, and word-box alignment generation in one task-agnostic sequence, replacing separate output modules with a shared decoder. Across seven VL benchmarks, it achieves better or comparable performance to task-specific state of the art, including large grounded-captioning gains.

  • Motivation: Text provides semantically rich image-level descriptions but cannot densely refer to regions, while boxes localize areas but lack rich semantics.Grounded captioning combines these complementary output formats by aligning mentioned noun entities with image regions.
  • Motivation: Prior grounded VL systems commonly combine detection, vision-language generation, and separate alignment predictions, weakening simplicity and parameter efficiency.UniTAB instead jointly predicts the outputs through a shared decoder.
  • Method: UniTAB represents text and quantized box outputs in one autoregressive sequence, using <obj> delimiters to encode word-box alignments inline.The model switches to box tokens after words to be grounded and returns to text after the box tokens.
  • Method: UniTAB approaches grounded captioning and other VL tasks without separate format-specific modules or task-specific output heads.The unified decoder covers text, box, and combined output formats within one architecture.
  • Implications: The task-agnostic output sequence and unified multi-task network support parameter-efficient training and generalization to unseen tasks.New tasks can be handled by reformatting their desired outputs into seen text-plus-box sequences.
  • Results: From 62.5 to 69.7 in captioning CIDEr and from 8.44 to 12.95 in grounding F1, UniTAB significantly outperforms prior grounded-captioning methods.It uses a single decoding sequence instead of separate text, box, and alignment heads.

2 Related Work

Prior grounded VL systems combine separate detection, language-generation, and alignment components, while unified frameworks still require task-specific output heads or do not support open-ended text generation. UniTAB targets a single framework that accepts structured image-language inputs and generates text and boxes without format-specific modules.

  • Grounded captioning: Grounded captioning requires captions, multiple object boxes, and word-box alignments, whereas UniTAB represents all three outputs in one decoding sequence.
  • Unified VL framework: Prior unified VL frameworks simplify boxes through detector-generated region indices or task-specific heads, but do not provide a common solution for open-ended text and box generation.The stated goal is a single framework that maps raw image and language inputs to structured text-and-box outputs.

3 The UniTAB Framework

UniTAB uses a transformer encoder-decoder with shared text and box vocabularies, representing boxes as discrete coordinate tokens and alignments inline with special delimiters. A single language-modeling objective supports pre-training, multi-task fine-tuning, and optional task-specific adaptation.

  • Architecture: UniTAB’s transformer decoder autoregressively generates tokens from both text and box vocabularies after image and text features are projected into a shared space.The architecture uses a six-layer transformer encoder and decoder.
  • Box token sequence: Box prediction is converted into multi-step classification by quantizing each rectangle into four discrete coordinate tokens.This representation follows the established Pix2seq-style ordering of box coordinates.
  • Target output sequence: UniTAB inserts <obj> delimiters around grounded words and their quantized box tokens, making word-box alignments directly recoverable from one output sequence.For example, “a donut” is paired with four quantized coordinates between the delimiters.
  • Objective: The model is trained with one autoregressive language-modeling objective that maximizes each target token’s likelihood conditioned on the image, input text, and previous target tokens.The objective is expressed as the negative summed log-likelihood over the target sequence.
  • Training stages: UniTAB supports up to three training stages—vision-language pre-training, multi-task fine-tuning, and optional task-specific fine-tuning—while retaining the same objective.The stages differ in training corpora and input-output designs.

4 Experiments

UniTAB evaluates a single unified architecture across seven vision-language benchmarks, covering grounded captioning, grounding, captioning, and VQA. It achieves strong task-specific comparisons while enabling parameter sharing, transfer, and zero-shot generalization.

  • Downstream tasks: UniTAB achieves better or comparable performance to task-specific state of the art across seven VL benchmarks with one unified architecture.The evaluation covers grounded captioning, visual grounding, COCO image captioning, and VQA.
  • Grounded captioning: F1all improves from 8.44 to 12.95 and F1loc from 22.78 to 34.79 on Flickr30k Entities grounded captioning, while CIDEr rises from 62.5 to 69.7.Additional image-text pre-training further raises CIDEr to 74.2.
  • Comparison with prior arts: UniTAB outperforms prior referring-expression and phrase-grounding methods, including systems trained on larger VL corpora or using task-specific architectures.Its multi-task finetuning improves performance across all reported splits compared with UniTABSeparate.
  • Comparison with prior arts: COCO captioning CIDEr improves from 117.3 to 119.8 over similarly pretrained prior methods and reaches 123.1 with extra image-text pairs.UniTAB does not require input region proposals or object tags.
  • Comparison with prior arts: UniTAB obtains competitive VQA results and performs better on the Karpathy out-of-domain subset than a discriminative approach.Results are reported on both official VQAv2 splits and the Karpathy split.
  • Ablation and analysis: The shared model performs comparably or better than separate task-specific models while using seven times fewer parameters, and pretraining consistently improves results over training from scratch.Further task-specific finetuning improves or matches state-of-the-art performance, while the task-agnostic sequence supports zero-shot ImageNet localization.

5 Conclusion

UniTAB unifies text and box outputs in one sequence, using a special <obj> token to represent word-box alignments. The resulting unified network is positioned as parameter efficient and generalizable for grounded VL tasks.

  • Conclusion: UniTAB generates text and box predictions together, with word-box alignments naturally represented by the special <obj> token.The unified representation targets grounded VL tasks such as grounded captioning.
  • Conclusion: Its unified multi-task network and task-agnostic output sequence make the model parameter efficient and generalizable to new tasks.

A Experiment Details

The appendix details UniTAB’s decoder settings, training corpora, and shared post-processing used to extract outputs for downstream task evaluation.

  • Hyper-parameter: UniTAB encodes previous target tokens with token and position embeddings and does not use type embeddings to distinguish text from box tokens.
  • Downstream task post-processing and evaluation: At inference, UniTAB first extracts text, boxes, and word-box alignments from the unified output sequence, then reuses them for task-specific evaluation.The same extracted outputs support grounded captioning and phrase grounding evaluation.

B Ablation Studies on Decoding Design

Ablations examine UniTAB’s output-sequence design, decoder embeddings, object-token budget, sampling, and training restrictions. The reported results support a compact design with roughly 200 object tokens and minimal dependence on additional decoder mechanisms.

  • <obj> token: Removing the <obj> token remains possible for referring expression comprehension, which requires only a single box and no word-box alignment.The ablation tests whether the token also simplifies sequence prediction and improves performance.
  • Number of object tokens: Around 200 object tokens are sufficient for the experimented VL datasets, avoiding quantization error while maintaining stable grounding performance.The object-token budget is evaluated on Refcocog and Flickr grounding tasks.
  • Decoder type embedding: Decoder type embeddings have no major influence on performance, so UniTAB does not use them to distinguish text and box tokens.Variants include separate text/box types and treating <obj> as a third type.
  • Decoding design: The decoding token-type pattern produces zero syntactic failures in scanned UniTABShared captions and COCO outputs.
  • Syntactic restrictions: Restricting the softmax loss to syntactically applicable tokens has no major influence on performance in the tested grounded-captioning settings.The experiment masks box logits after <obj> and is conducted on Refcoco and Flickr grounded captioning.

C Discussions

Additional experiments show that extra box annotations can improve both grounding and text outputs, while prefix-based multitask finetuning offers comparable performance without prefixes. Robustness and bias analyses remain limited, leaving room for improvement.

  • Additional box annotations: CIDEr improves from 65.6 to 70.0 for grounded captioning, from 119.3 to 120.7 for COCO captioning, and VQA accuracy from 66.6% to 68.4% with extra box annotations.The same variant also improves grounding accuracy and grounded captioning F1 on tasks requiring box prediction.
  • Additional box annotations: Extra box annotations consistently improve grounding accuracy and grounded captioning F1 for tasks requiring box prediction.
  • Multi-task finetuning: UniTABShared uses one model across downstream tasks, supporting parameter efficiency and showing promise for zero-shot generalization.
  • Multi-task finetuning with prefix: UniTABPrefix achieves comparable performance to UniTABShared across the experimented tasks and datasets.UniTABPrefix adds a task-specific task-identifying string to each input before multitask finetuning.
  • Robustness and bias analyses: Building robust and unbiased models remains challenging despite reasonable performance on standard robustness and bias analyses.

D Qualitative Results

Qualitative results show UniTAB generating detailed captions while grounding words and phrases to relevant image regions. The model also transfers grounded captioning ability across datasets and handles varied grounding targets, including small objects and scene regions.

  • Captioning tasks: UniTAB generates smooth, accurate captions and grounds noun phrases to image regions, including attributes, scenes, nearby objects, and tiny objects.Examples include a man’s clothing, a red door, a dog, a bat, a baseball, and a red ball.
  • Captioning tasks: UniTAB transfers grounded captioning ability from Flickr30k Entities to MSCOCO despite COCO captioning lacking grounding annotations.Text tokens are evaluated with standard COCO captioning metrics.
  • Captioning tasks: UniTAB generalizes to ImageNet images, producing caption and grounding predictions of similar quality to those on Flickr30k Entities and MSCOCO.
  • Grounding tasks: On referring expression comprehension, UniTAB identifies different objects in the same image under different input queries.The qualitative examples distinguish targets such as a yellow-sleeved person and a blue skier.
  • Grounding tasks: On phrase grounding, UniTAB grounds foreground objects, small background objects, and scene regions to their corresponding image areas.Examples include persons, women, skies, another man, snow, and a lake.
Loading 2111.12085v2…