PaperlayerView exact revision on arXiv

Source-linked AI summary

GIT: A Generative Image-to-text Transformer for Vision and Language

Jianfeng Wang, Zhengyuan Yang, Xiaowei Hu, Linjie Li, Kevin Lin, Zhe Gan, Zicheng Liu, Ce Liu, Lijuan Wang

arXiv:2205.14100v5cs.CV

TL;DR

Vision-language systems often use mismatched pre-training objectives, complex architectures, and external detectors or OCR. GIT addresses this with one image encoder and one text decoder trained through language modeling, then scales data and model size. It achieves state-of-the-art results across captioning and question answering, surpasses human TextCaps performance, and supports generation-based classification, while leaving caption control and parameter-free in-context learning unresolved.

  • Problem

    Existing vision-language methods use pre-training losses that differ from downstream tasks and often depend on complex structures or external modules such as object detectors, tags, and OCR.

  • Method

    GIT uses one image encoder and one text decoder trained on image-text pairs with language modeling, generating answers, captions, and class labels directly.

  • Results

    GIT achieves new state-of-the-art performance across numerous image/video captioning and question-answering benchmarks, including 123.0 CIDEr on nocaps versus 120.6 for CoCa.

  • Takeaways & Limitations

    Generation-based prediction enables image classification without a pre-defined vocabulary and is beneficial when new categories or data are added.

  • Takeaways & Limitations

    The authors leave control of generated captions and in-context learning without parameter updates for future work.

Abstract

from arXiv · show

In this paper, we design and train a Generative Image-to-text Transformer, GIT, to unify vision-language tasks such as image/video captioning and question answering. While generative models provide a consistent network architecture between pre-training and fine-tuning, existing work typically contains complex structures (uni/multi-modal encoder/decoder) and depends on external modules such as object detectors/taggers and optical character recognition (OCR). In GIT, we simplify the architecture as one image encoder and one text decoder under a single language modeling task. We also scale up the pre-training data and the model size to boost the model performance. Without bells and whistles, our GIT establishes new state of the arts on 12 challenging benchmarks with a large margin. For instance, our model surpasses the human performance for the first time on TextCaps (138.2 vs. 125.5 in CIDEr). Furthermore, we present a new scheme of generation-based image classification and scene text recognition, achieving decent performance on standard benchmarks. Codes are released at \url{https://github.com/microsoft/GenerativeImage2Text}.

1 Introduction

GIT unifies vision-language tasks with one image encoder and one text decoder trained under language modeling, avoiding object detectors, tags, and OCR. Scaling data and model size yields strong results across image/video captioning, question answering, and generation-based classification.

  • Results: GIT captions demonstrate recognition and description of scene text, tables, charts, food, banknotes, logos, landmarks, characters, and products.These examples are presented as evidence of rich multimodal knowledge about the visual world.
  • Method: GIT uses one image encoder and one text decoder, pre-trained on 0.8 billion image-text pairs with language modeling.The model extends to video by concatenating features from sampled frames, and treats VQA questions as text prefixes whose answers are generated autoregressively.
  • Results: GIT avoids dependencies on object detectors, object tags, and OCR while achieving new state-of-the-art performance across image/video captioning and question answering.The paper reports this pattern across numerous tasks and benchmarks.
  • Results: GIT surpasses human performance on TextCaps for the first time, demonstrating that a simple architecture can achieve strong performance with scaling.The supplied passages state the human-performance comparison and connect the result to scaling data and model size.
  • Results: GIT pre-trained on image-text pairs achieves new state-of-the-art performance on video tasks without video-dedicated encoders.Multiple sampled-frame features are concatenated as the video representation.
  • Contributions: GIT achieves 88.79% top-1 accuracy on ImageNet-1K using generation-based image classification.Predicted labels come directly from the generative model rather than a pre-defined vocabulary.

2 Related Work

Prior vision-language pre-training uses multiple objectives and often complex multimodal architectures, while image representations commonly rely on detector-based or dense features. Recent generative approaches seek a unified text-generation formulation, and scaling studies examine model and data size together.

  • Pre-training objectives: Masked Language Modeling and Image-Text Matching are widely used pre-training tasks, but their losses differ from downstream objectives and require task-specific adaptation.Examples include removing ITM for image captioning and adding a randomly initialized multilayer perceptron for VQA.
  • Generative models: Recent vision-language methods cast many tasks as text generation, using designed input-output text formats and multimodal transformer encoders.Examples include masked-span recovery and randomly splitting a sentence into input and target output.
  • Image representation: Faster R-CNN region features have been common for image representation, while newer methods increasingly use dense feature-map representations without bounding-box annotations.The passage contrasts detector-based region features with dense representations.
  • Scaling studies: The paper studies three model sizes and three pre-training data scales across nine benchmarks covering image/video captioning and question answering.This yields nine data points for each benchmark in the reported scaling study.

3 Generative Image-to-text Transformer

GIT uses one image encoder and one text decoder under a language-modeling objective, with the same generative framework extended across image, video, captioning, VQA, classification, and scene-text tasks. The design scales image-text pre-training and adapts inputs or frame representations rather than adding task-specific external modules.

  • 3.1 Network Architecture: GIT consists of one image encoder and one text decoder, using image features and text embeddings as inputs to a transformer decoder.The image encoder projects flattened visual features into D dimensions, while text is decoded autoregressively from [BOS] to [EOS].
  • 3.1 Network Architecture: The image encoder is contrastively pre-trained, removes object-detector dependence, and can represent video by independently encoding sampled frames with temporal embeddings.Frame features are concatenated after adding a learnable temporal embedding initialized to zero.
  • 3.2 Pre-training: A seq2seq attention mask controls which inputs each output can depend on, while self-attention-based decoding performs better than cross-attention with large-scale pre-training.The cross-attention alternative performs better in the small-scale setting, whereas self-attention performs better overall after large-scale pre-training.
  • 3.2 Pre-training: GIT trains on image-text pairs with language modeling, predicting each token from the image and preceding text tokens.The authors choose language modeling because each iteration predicts all tokens, unlike masked language modeling, which typically predicts only 15%.
  • 3.3 Fine-tuning: Fine-tuning preserves the language-modeling formulation for captioning and uses the question as a prefix while applying loss only to answers and [EOS] in VQA.The generative VQA formulation avoids predefined answer candidates but is more challenging because it must generate both the answer and termination token.
  • 3.3 Fine-tuning: GIT also supports generation-based image classification, where class names are treated as captions and new categories can be added without introducing new parameters.The model predicts class names autoregressively instead of using a fixed vocabulary and linear softmax classifier.

4 Experiments

GIT is evaluated across image and video captioning, visual question answering, image classification, scene-text recognition, and scaling studies. The results show strong performance across many benchmarks, while also revealing task-specific effects of vocabulary constraints, generative prediction, and model/data scale.

  • Image captioning: GIT achieves new state-of-the-art captioning performance on most evaluated image-captioning benchmarks, including nocaps and TextCaps.On nocaps, GIT reaches 123.0 versus CoCa’s 120.6 CIDEr despite using fewer parameters; on TextCaps, it exceeds the prior state of the art by 28.5 CIDEr points and surpasses human performance.
  • Question answering: GIT reaches new state-of-the-art results on several visual question-answering benchmarks and outperforms Flamingo on TextVQA while using substantially fewer parameters.Compared with Flamingo, GIT is 5.4 points higher on TextVQA but 3.29 points lower on VQAv2; its model has 0.7B parameters versus Flamingo’s 80B.
  • Video captioning and QA: GIT achieves new state-of-the-art video captioning results on MSVD, MSRVTT, and VATEX, and on MSVD-QA and TGIF-Frame for video question answering.On the VATEX private test, GIT scores 93.8 versus 86.5 for CLIP4Caption++, and exceeds Flamingo’s 84.2 despite not being specifically designed for video.
  • Image classification: GIT supports generation-based image classification without a predefined vocabulary, but exact-match accuracy is sensitive to unrestricted generation and vocabulary constraints.Without vocabulary knowledge, exact-match accuracy is 1.93%, containment accuracy is 40.88%, and vocabulary-prior accuracy is 33.48%; few-shot fine-tuning substantially improves accuracy.
  • Scene-text recognition: GIT demonstrates strong scene-text recognition using captioning or dedicated scene-text fine-tuning, reaching 89.9 and 92.9 average accuracy respectively.The dedicated MJ+ST result of 92.9 surpasses prior art at 91.9 across six standard benchmarks.
  • Model and data scaling: Scaling benefits depend on both task and capacity: more data helps TextCaps and VizWiz-QA, while larger backbones benefit especially from the 0.8B-pair setting.On COCO, the Base model improves from 4M to 14M pairs but drops with 0.8B noisy data, whereas larger models benefit more from the largest scale.

5 Conclusion

GIT unifies vision-language tasks with a simple generative architecture and achieves strong results across captioning, question answering, and classification. The paper also identifies unresolved issues in controllable generation and parameter-free in-context learning, alongside deployment concerns from large-scale pretraining data.

  • GIT maps images to associated text descriptions using a simple generative model trained on large-scale image-text pairs.
  • The model achieves new state-of-the-art performance across numerous image/video captioning and question-answering benchmarks, surpassing human performance on TextCaps for the first time.
  • Generation-based image classification predicts label names directly rather than using a predefined fixed vocabulary, supporting continued training when new categories are added.
  • The authors leave controllable caption generation and in-context learning without parameter updates for future work.
  • Large-scale pretraining data may contain toxic language, so deployment requires special care to control outputs.
  • GIT variants scale from smaller configurations to GIT2, a 5.1B-parameter model trained with 12.9B image-text pairs.

A.4 Implementation of the Data Loader

The data loader addresses a 39TB pretraining corpus that exceeds local disk capacity by distributing, shuffling, prefetching, and rotating data trunks across compute nodes. The authors report that preprocessing adds almost no training-time cost because it overlaps GPU training.

  • The loader handles 39TB of data on local disks of about 7TB by downloading data before reading it and keeping operations independent of dataset size.
  • Image-text pairs are evenly split across compute nodes, and each node consumes the data trunk by trunk.
  • Each trunk is shuffled independently, making shuffling cost independent of the total dataset size.
  • Shuffled trunk data are divided among GPUs within each node, while one process prefetches up to seven future trunks.
  • Local storage retains at most 12 trunks and removes the oldest data as new trunks arrive.
  • The authors observe almost no data-loading time cost because preprocessing overlaps with GPU training.

B Results on Image Captioning

Across image-captioning benchmarks, GIT delivers competitive or state-of-the-art results with simplified processing and demonstrates recognition of novel objects, scene text, and challenging real-world imagery. Qualitative examples cover diverse domains including products, landmarks, foods, documents, and low-quality images.

  • COCO: COCO results show competitive performance for base and large GIT models despite their simplified architecture.
  • Benchmark results: GIT achieves new state-of-the-art performance across COCO, Flickr30K, nocaps, TextCaps, and VizWiz-Captions captioning benchmarks.The results are evaluated with benchmark-specific captioning metrics, including CIDEr and CIDEr-D.
  • nocaps: GIT identifies novel objects in nocaps without object tags as network inputs.
  • TextCaps: TextCaps examples show reading of screen times, multiple languages, stylized and curved text, jerseys, handwriting, long text, and book pages.
  • TextCaps: TextCaps-fine-tuned GIT produces diverse captions about logos, landmarks, foods, characters, celebrities, and products using tags or packaging information.
  • VizWiz-Captions: VizWiz-Captions examples cover blurry or occluded images, banknotes, bottle and can text, menus, pages, and screens.

C Results on Visual Question Answering

The supplied passage specifies fine-tuning settings for visual question-answering experiments and points to full results and qualitative visualizations across VizWiz-VQA and ST-VQA.

  • Visual question-answering experiments use different fine-tuning schedules for VizWiz-QA and other tasks, with input sizes of 384 and 576.
  • The paper reports full results and shows correct predictions on randomly selected VizWiz-VQA and ST-VQA images.

D Results on Video Captioning and Question Answering

GIT is evaluated on video captioning and video question answering using a simple frame-sampling procedure rather than video-specific architectural components.

  • Video evaluation: During training, GIT randomly samples 6 frames at equal intervals and applies the same random crop to all frames.At inference, it uniformly samples 6 frames and applies a center crop.

E Results on Image Classification

GIT performs generation-based ImageNet-1K classification by mapping labels to unique names and autoregressively generating class names without a predefined vocabulary.

  • Label representation: Each ImageNet-1K label is mapped to a unique name using its WordNet offset.The mapping supports readable generated class names.
  • Unrestricted generation: Only 13 predictions, or 0.026%, fall outside the 1K categories when output is unconstrained.Some out-of-category predictions are semantically reasonable, such as “ipad” for the ground-truth class “hand-held computer.”

F Results on Scene Text Recognition

GIT is also applied to scene text recognition and achieves strong validation performance while producing text in natural-language form.

  • Scene text recognition: GITTextCaps visually recognizes scene text almost as well as GITMJSJ while producing natural-language outputs.The comparison covers six benchmarks: IC13, SVT, IIIT, IC15, SVTP, and CUTE.
  • Scene text recognition: 3.9% absolute accuracy improvement is obtained on the irregular-text CUTE80 benchmark.The paper reports especially strong visual performance on images resembling natural images.
  • TextOCR: 81.27% accuracy is achieved on the TextOCR validation set because test annotations are unavailable publicly.The evaluation is therefore performed on validation data.

G.1 Model and data scaling

Scaling pre-training data and model size generally improves scene-text-related question answering and video captioning, while gains vary by task and initialization choice.

  • Model and data scaling: Larger models and more pre-training data significantly improve scene-text-related QA tasks and video captioning.For VQAv2, 0.8B pre-training data help little or slightly worsen performance.
  • Decoder architecture: The pure self-attention decoder is used in all experiments instead of a cross-attention decoder.Self-attention allows image tokens to attend to one another during decoding, refining their representation.
  • Text decoder initialization: Text-corpus-pretrained decoder weights provide no improvement or worse performance than random initialization for both Base and Large decoders.The observation applies when decoder weights are learnable and may not apply to frozen-parameter settings.
  • Image encoder initialization: CLIP-pretrained image-encoder weights produce higher performance than supervised, self-supervised, or random initialization.Random initialization yields significantly lower performance in the reported GITB setting.
  • Intermediate fine-tuning: Intermediate fine-tuning improves performance on all evaluated VQA tasks, with larger gains when target training data are limited.The result is reported for combining multiple VQA datasets before target-task fine-tuning.
  • Bias analysis: Normalized performance differences for gender and skin groups range from 0.7% to 5.3% across all reported metrics.The analysis evaluates captioning-model bias using annotations for gender and skin type.

G.7 Scene text in pre-training data

The pre-training data contains substantial scene-text content, including natural-language descriptions, long text passages, and stylized fonts. These examples align with TextCaps and are associated with robust scene text recognition.

  • A considerable amount of pre-training samples contains scene text descriptions.
  • Natural-language scene-text descriptions in the pre-training samples align with the requirements of TextCaps.
  • Stylized-font examples in pre-training data lead to GIT’s ability in robust scene text recognition.
Loading 2205.14100v5…