PaperlayerView exact revision on arXiv

Source-linked AI summary

Unifying Vision-and-Language Tasks via Text Generation

Jaemin Cho, Jie Lei, Hao Tan, Mohit Bansal

arXiv:2102.02779v2cs.CLcs.AIcs.CVcs.LG

TL;DR

Vision-and-language learning commonly relies on task-specific architectures and objectives, even when different tasks share reasoning skills and text-expressible labels. The paper introduces VL-T5 and VL-BART, which generate text labels from multimodal inputs using one language-modeling objective; across seven benchmarks, they achieve comparable performance to state-of-the-art vision-and-language transformers and support joint training of seven tasks with one parameter set.

  • Problem

    Existing vision-and-language methods require separately designed architectures and objectives for different tasks, despite overlapping reasoning skills and text-expressible labels.

  • Method

    VL-T5 and VL-BART extend pretrained T5 and BART into multimodal encoder-decoder models that generate task labels as text with a shared language-modeling head.

  • Results

    Comparable performance to state-of-the-art vision-and-language transformers was achieved across 7 diverse benchmarks, while seven tasks could be trained jointly with one architecture and parameter set.

  • Takeaways & Limitations

    The generative framework is especially suited to open-ended visual question answering and avoids hand-crafted architectures and objectives across tasks.

  • Takeaways & Limitations

    The simple task prefixes may leave accuracy improvements from prompt engineering unexplored.

Abstract

from arXiv · show

Existing methods for vision-and-language learning typically require designing task-specific architectures and objectives for each task. For example, a multi-label answer classifier for visual question answering, a region scorer for referring expression comprehension, and a language decoder for image captioning, etc. To alleviate these hassles, in this work, we propose a unified framework that learns different tasks in a single architecture with the same language modeling objective, i.e., multimodal conditional text generation, where our models learn to generate labels in text based on the visual and textual inputs. On 7 popular vision-and-language benchmarks, including visual question answering, referring expression comprehension, visual commonsense reasoning, most of which have been previously modeled as discriminative tasks, our generative approach (with a single unified architecture) reaches comparable performance to recent task-specific state-of-the-art vision-and-language models. Moreover, our generative approach shows better generalization ability on questions that have rare answers. Also, we show that our framework allows multi-task learning in a single architecture with a single set of parameters, achieving similar performance to separately optimized single-task models. Our code is publicly available at: https://github.com/j-min/VL-T5

1. Introduction

Vision-and-language transformers often use separately parameterized task-specific architectures despite substantial overlap in required reasoning and text-expressible labels. The paper proposes one multimodal text-generation framework and evaluates it across seven benchmarks.

  • Motivation: Existing vision-and-language transformers typically require separately parameterized architectures for each pretraining or downstream task.Examples include multi-label sigmoid classification for visual question answering and softmax classification for referring expression comprehension.
  • Motivation: Visual question answering and referring expression comprehension share object-recognition requirements, while their labels can be expressed as text.A region identifier such as “<vis 3>” can represent the target region for grounding.
  • Approach: The proposed framework extends pretrained T5 and BART models so all tasks use one language-modeling head and text-formatted inputs and outputs.This avoids adding task-specific parameters, architectures, or objectives when learning a new task.
  • Approach: The generative formulation can produce open-ended natural-language answers, unlike discriminative methods restricted to predefined frequent candidates.The paper identifies this as especially useful for open-ended questions requiring non-trivial answers.
  • Evaluation: The framework is evaluated on 7 downstream benchmarks spanning question answering, grounding, reasoning, commonsense reasoning, captioning, and multimodal translation.The reported comparison is against recent vision-and-language transformers.

2. Related Works

Prior work applies pretrained transformer language models and image-text transformers to many vision-and-language tasks. Related unified frameworks motivate representing multiple tasks in a common format, which this paper extends to multimodal text generation.

  • Vision-and-Language Pretraining: Pretrained transformer language models have achieved strong results across natural-language understanding tasks.Vision-and-language transformers follow this pretraining-finetuning paradigm using large image-text corpora before downstream adaptation.
  • Vision-and-Language Pretraining: Existing image-text pretraining models apply transformer-based representations to a broad range of vision-and-language tasks.These models are part of the broader pretraining literature that motivates the paper’s architecture.
  • Unified Frameworks: Unified task frameworks in natural language processing express question answering, span prediction, or other tasks in common formats.Such frameworks support knowledge sharing and reuse of pretrained language models.
  • Unified Frameworks: This paper extends unified-format learning to previously separate vision-and-language tasks through text generation conditioned on multimodal inputs.The proposed direction connects unified NLP task formulations with visual and textual inputs.

3. Model

The model extends pretrained T5 and BART encoder-decoder systems with visual region embeddings and task-specific text prefixes, then generates label text from joint multimodal representations. Unlike conventional task-specific classifiers, the framework trains VQA and grounding through the same language-modeling objective.

  • Model: VL-T5 and VL-BART extend pretrained T5Base and BARTBase into multimodal conditional text-generation models.Image region embeddings are incorporated as additional inputs to the text encoders.
  • Visual Embeddings: Each image is represented with 36 Faster R-CNN object regions containing object features, bounding-box coordinates, image ids, and region ids.RoI features and coordinates use linear layers, while image and region ids use learned embeddings.
  • Text Embeddings: Task-specific text prefixes adapt the shared model to different tasks without designing separate architectures.The augmented text is tokenized and processed with shared embedding parameters across the encoder, decoder, and language-modeling head.
  • Visual Embeddings: Visual sentinel tokens such as <vis 3> provide text embeddings reused as region-id embeddings for grounding tasks.This shared embedding connects query text, label text, and image objects.
  • Text Embeddings: The text-prefix strategy may leave accuracy gains from prompt engineering unexplored.The authors explicitly identify prompt engineering as future work.
  • Encoder-Decoder Architecture: The encoder jointly contextualizes text and visual embeddings, while the decoder uses self-attention and cross-attention to generate label tokens.The decoder predicts each token conditioned on previously generated tokens and the encoder’s multimodal representations.
  • Task-Specific Methods vs. Our Unified Framework: VQA and referring expression comprehension are formulated as predicting text labels with the same language-modeling objective.This replaces VQA’s classifier head and grounding’s region-scoring classifier with a shared generative formulation.

4. Pretraining

The paper pretrains VL-T5 and VL-BART on diverse vision-and-language tasks, formulating them as multimodal conditional text generation. The tasks use a shared language-modeling objective across visual and textual inputs.

  • The models aggregate pretraining data from MS COCO and Visual Genome images, using captions, visual questions, and other multimodal resources.
  • Diverse pretraining tasks include multimodal language modeling, visual question answering, image-text matching, visual grounding, and grounded captioning.
  • Multimodal language modeling masks input text tokens and trains the model to predict the masked text.
  • Visual question answering is formulated as directly generating answers in their original text format rather than classifying over predefined answer candidates.
  • Image-text matching creates positive image-caption pairs and randomly sampled negative pairs, then predicts correspondence with “true” or “false”.
  • Visual grounding predicts the related object-region id from a region description, while grounded captioning generates a region description from a visual sentinel token.

5. Downstream Tasks and Results

Across seven downstream vision-and-language tasks, the unified generative approach achieves performance close to task-specific models while using a common text-generation formulation. Results also show stronger out-of-domain VQA generalization and competitive performance under several task-specific settings.

  • Overall results: The unified generative approach achieves performance close to task-specific models across seven diverse downstream benchmarks.The evaluated tasks include VQA, GQA, RefCOCOg, NLVR2, VCR, COCO Caption, and Multi30K.
  • Visual Question Answering: 6 and 6.2 points: generative VL-T5 and VL-BART improve over same-backbone discriminative counterparts on out-of-domain VQA questions.The models also improve across all in-domain and out-of-domain subsets, and outperform UNITERBase by about 3 points on the out-of-domain subset.
  • Visual Question Answering: 67.9 →69.3 on VQA Karpathy-test and 60.0 →60.2 on GQA test-dev: one shared “vqa” prefix handles both tasks without dataset-specific prefixes.These changes are slight performance increases over the original dataset-specific prefixes.
  • Natural Language Visual Reasoning: Triplet: VL-T5 shows comparable NLVR2 performance to UNITER at lower computational cost than Pair and Pair-biattn settings.Pair and Pair-biattn approximately double the computational cost over Triplet.
  • Referring Expression Comprehension: VL-T5 significantly outperforms MAttNet on RefCOCOg with pretraining, while remaining below state-of-the-art performance and reasonably comparable to UNITER.The task is formulated as generating the visual sentinel token for the selected region rather than classifying among regions.
  • Visual Commonsense Reasoning: VL-T5 achieves comparable or better VCR performance than UNITER and significantly higher performance than ViLBERT on the test split after staged pretraining.Both generic-domain and in-domain second-stage pretraining improve VCR performance on the validation split.
  • Image Captioning: Object tags give comparable or slightly improved captioning performance, including a significant 2.5 CIDEr improvement for VL-BART.Without and with vision-and-language pretraining, the models show comparable performance to baselines.
  • Multimodal Machine Translation: T5-based models outperform multimodal translation baselines on all three Multi30K test splits, although vision-and-language pretraining does not improve translation.The models improve their text-only backbones, while the source text may already contain sufficient information.

6. Conclusion

VL-T5 and VL-BART unify vision-and-language tasks under a text-generation objective. Across diverse tasks, they retain comparable performance while supporting joint training with shared parameters.

  • VL-T5 and VL-BART tackle vision-and-language tasks with a unified text-generation objective.
  • Comparable performance to state-of-the-art vision-and-language transformers is achieved without hand-crafted architectures and objectives.
  • Seven different tasks can be trained simultaneously with a single architecture and parameter set without much performance loss.

A. Comparison with Baselines

Table 11 provides a detailed comparison between baseline vision-and-language transformers and VL-T5 and VL-BART. The comparison covers pretraining datasets and model architectures.

  • Table 11 compares baseline vision-and-language transformers with VL-T5 and VL-BART.
  • The comparison includes the pretraining datasets used by the models.
  • The comparison also details model architectures.

B. Implementation Details

The paper reports dataset and task statistics, experimental hyperparameters, and download links for pretraining and downstream data.

  • Tables 12 and 13 provide detailed statistics for the pretraining and downstream datasets and tasks.
  • Table 14 lists the hyperparameters used in pretraining and downstream-task experiments.
  • Links are provided to download the pretraining and downstream datasets.

B.1. Pretraining Data

The pretraining data contains 9.18M image-text pairs across 180K distinct images, with splits designed to prevent downstream evaluation overlap and a 5,000-image validation set.

  • 9.18M image-text pairs on 180K distinct images comprise the pretraining dataset.
  • The pretraining data is split to avoid intersection with downstream validation and test sets.
  • 5,000 COCO Karpathy validation images are used to monitor pretraining performance.

B.2. Downstream Tasks

The downstream evaluation uses established dataset splits across VQA, captioning, GQA, NLVR2, VCR, RefCOCOg, and Multi30K. Training and validation protocols are specified per task, with RefCOCOg additionally distinguishing ground-truth training boxes from detected inference candidates.

  • VQA and COCO captioning use the Karpathy split with 113,287 / 5,000 / 5,000 train / validation / test images.
  • GQA-balanced uses 943,000 / 132,062 / 12,578 train / validation / test-dev questions, with train and validation data used for training and test-dev for validation.
  • NLVR2 and VCR use train and validation splits for training and validation, containing 86,373 / 6,982 / 6,967 sentences and 212,923 / 26,534 / 25,263 questions, respectively.
  • RefCOCOg uses the UMD split with 42,226 / 2,573 / 5,023 train / validation / test sentences.
  • RefCOCOg training uses ground-truth COCO boxes, while inference uses detected boxes from an off-the-shelf Mask R-CNN as candidates.
  • Multi30K English-German contains 29,000 / 1,014 / 1,000 / 1,000 / 1,017 train / validation / test2016 / test2017 / test2018 sentence pairs.
Loading 2102.02779v2…