Source-linked AI summary
ViLT: Vision-and-Language Transformer Without Convolution or Region Supervision
Wonjae Kim, Bokyung Son, Ildoo Kim
TL;DR
Existing VLP systems depend on costly visual embedding processes, creating efficiency concerns and tying visual processing to dedicated embedders. ViLT uses a unified, shallow, convolution-free transformer-based architecture for visual and textual inputs, achieving competitive downstream performance while substantially reducing runtime. The paper presents this as evidence that efficient VLP can remain competent without convolutional visual embedders or region supervision.
Problem
Existing VLP models rely on costly visual embedding pipelines, while visual embedding is commonly a computational bottleneck in real-world applications.
Method
ViLT uses a unified transformer to process visual inputs through shallow, convolution-free pixel embedding instead of a separate deep visual embedder.
Results
ViLT achieves competent downstream performance without region features or deep convolutional visual embedders, while reported comparisons show it is tens of times faster than region-feature models and at least four times faster than grid-feature models.
Takeaways & Limitations
Efficient VLP can remain competent without convolutional visual embedding networks or region supervision, shifting attention toward multimodal interaction inside the transformer.
Takeaways & Limitations
ViLT-B/32 is described as more of a proof of concept than a mature efficient VLP model.
Abstract
from arXiv · showhide
Vision-and-Language Pre-training (VLP) has improved performance on various joint vision-and-language downstream tasks. Current approaches to VLP heavily rely on image feature extraction processes, most of which involve region supervision (e.g., object detection) and the convolutional architecture (e.g., ResNet). Although disregarded in the literature, we find it problematic in terms of both (1) efficiency/speed, that simply extracting input features requires much more computation than the multimodal interaction steps; and (2) expressive power, as it is upper bounded to the expressive power of the visual embedder and its predefined visual vocabulary. In this paper, we present a minimal VLP model, Vision-and-Language Transformer (ViLT), monolithic in the sense that the processing of visual inputs is drastically simplified to just the same convolution-free manner that we process textual inputs. We show that ViLT is up to tens of times faster than previous VLP models, yet with competitive or better downstream task performance. Our code and pre-trained weights are available at https://github.com/dandelin/vilt.
1. Introduction
Existing vision-and-language pre-training models rely on costly visual embedding pipelines, often using convolutional networks and region supervision. ViLT simplifies visual processing with a convolution-free transformer-based design while retaining competent downstream performance and improving efficiency.
- Motivation: Most VLP models use convolutional or object-detection-based visual embedding, often relying on Visual Genome supervision with 1,600 object and 400 attribute classes.These image features must be extracted before multimodal processing.
- Motivation: Cached region features obscure the cost of visual embedding during training, but real-world queries still require slow feature extraction.The paper identifies visual embedding as a practical bottleneck for deployed applications.
- ViLT: ViLT replaces a separate deep visual embedder with the transformer module, processing visual inputs in a unified, convolution-free manner.Its visual representation uses shallow pixel-level embedding rather than region or grid features.
- ViLT: ViLT cuts model size and runtime by removing deep components dedicated solely to visual inputs.The architecture commissions the transformer to extract and process visual features.
- Results: ViLT is tens of times faster than region-feature VLP models and at least four times faster than grid-feature models while achieving similar or better downstream performance.The comparison is reported across vision-and-language downstream tasks.
- Results: The paper reports competent performance without region features or deep convolutional visual embedders, and finds that whole word masking and image augmentations further drive downstream performance.These are presented as contributions of the proposed training and architecture choices.
2. Background
ViLT places vision-and-language models in a taxonomy based on modality expressiveness and interaction depth, using shallow pixel embeddings and deep transformer-based interaction. This design replaces heavy visual feature extraction with lightweight patch projection, targeting the computation and runtime burdens of conventional region- and grid-feature pipelines.
- The taxonomy classifies vision-and-language models by modality expressiveness and interaction depth, yielding four archetypes.
- Deep-interaction VLP models still rely on convolutional networks to extract image features, accounting for most computation.
- ViLT is the first Figure 2d model with shallow, computationally light raw-pixel embeddings that concentrate computation on modality interactions.
- Existing VLP models commonly use separate visual and textual embedders, with visual embedding remaining the computational bottleneck.
- Region Feature: Region features are produced through region proposals, non-maximum suppression, pooling, and RoI heads, with backbone, NMS, and head choices affecting runtime.
- Patch Projection: Patch projection uses linear mappings over image patches, simplifying visual embedding to the level of textual embedding.
3. Vision-and-Language Transformer
ViLT uses a unified, single-stream transformer with shallow patch-based visual embedding instead of a separate deep visual embedder. It combines image and text representations and trains with multimodal objectives, including alignment and whole-word masking.
- Model Overview: ViLT follows a single-stream architecture with a minimal visual embedding pipeline.The model processes concatenated image and text inputs through one interaction transformer.
- Initialization: The interaction transformer is initialized from pre-trained ViT weights rather than BERT weights.This initialization is intended to let the interaction layers process visual features without a separate deep visual embedder.
- Visual Embedding: Image pixels are sliced into patches, flattened, linearly projected, and position-embedded into visual tokens.For an image of size H × W with patch resolution P, the number of patches is N = HW/P^2.
- Multimodal Transformer: Text and image embeddings receive modal-type embeddings, are concatenated into z0, and pass through D transformer layers to produce zD.The pooled representation p comes from the first position of zD after a linear projection and tanh.
- Pre-training Objectives: ViLT is trained with image-text matching and masked language modeling, with word-patch alignment added to the ITM loss.The word-patch alignment objective estimates alignment between textual and visual subsets using optimal transport.
- Whole Word Masking: Whole-word masking masks all consecutive subword tokens composing a word, using a 0.15 masking probability during pre-training.The paper motivates this choice as encouraging use of image information when predicting masked words.
4. Experiments
ViLT is evaluated on classification and retrieval tasks using multiple vision-language datasets, standardized training settings, and ablations of training duration and objectives. It maintains competitive classification performance and strong retrieval results while showing benefits from longer training, whole-word masking, and image augmentation, but falls short on VQAv2 relative to heavier visual embedders.
- Evaluation setup: ViLT-B/32 is evaluated on VQAv2 and NLVR2 for classification, and MSCOCO and Flickr30K for image-text retrieval.Classification scores are averaged across three fine-tuning runs, whereas retrieval tasks use one run.
- Implementation details: 384 × 640 image inputs yield 240 patches, with at most 200 sampled during pre-training, versus 800 × 1,333 inputs used by other VLP models.The resizing preserves aspect ratio and uses padding for batch training.
- Classification tasks: ViLT falls short of heavier visual-embedder models on VQAv2, possibly because detector-derived object representations ease questions that typically ask about objects.The paper reports this as a suspected explanation rather than a demonstrated causal mechanism.
- Classification tasks: ViLT-B/32 maintains competitive performance on VQAv2 and NLVR2 considering its inference speed.The comparison is reported in Table 2 against other downstream classification models.
- Retrieval tasks: ViLT-B/32 generally outperforms ImageBERT in zero-shot retrieval and has higher fine-tuned recalls than the second-fastest model, Pixel-BERT-R50.The zero-shot comparison holds despite ImageBERT using a larger 14M pre-training dataset.
- Ablation study: Performance increases with longer training, while whole-word masking and image augmentation improve results; extending training to 200K steps helps several tasks but later reduces fine-tuned text retrieval performance.The 200K-step improvement is reported for VQAv2, NLVR2, and zero-shot retrieval.
5. Conclusion and Future Work
ViLT establishes that a minimal architecture without convolutional visual embedders or region supervision can remain competent, while suggesting several directions for improving the model family. These include scaling, better visual masking objectives, and improved augmentation strategies.
- ViLT demonstrates that a minimal architecture can remain competent without convolutional visual embedders or region supervision.
- The paper encourages future VLP work to emphasize modality interactions inside the transformer rather than simply increasing unimodal embedder power.
- Scalability: Larger ViLT variants are a future direction because aligned vision-and-language datasets remain scarce.The paper notes that pre-trained transformers can scale with an appropriate amount of data.
- Masked Modeling for Visual Inputs: A naive masked patch prediction objective fails, motivating more sophisticated visual masking objectives for models without region supervision.
- Augmentation Strategies: Exploring appropriate augmentation strategies for textual and visual inputs is identified as a valuable addition.Prior contrastive visual representation work found gains from Gaussian blur relative to a simpler strategy.