PaperlayerView exact revision on arXiv

Source-linked AI summary

Seeing Out of tHe bOx: End-to-End Pre-training for Vision-Language Representation Learning

Zhicheng Huang, Zhaoyang Zeng, Yupan Huang, Bei Liu, Dongmei Fu, Jianlong Fu

arXiv:2104.03135v2cs.CV

TL;DR

Region-based visual features can miss contextual information needed for vision-language understanding, motivating an end-to-end alternative. SOHO jointly learns whole-image and language representations with a dynamic visual dictionary and multimodal Transformer, and reports absolute gains across multiple downstream tasks while avoiding box annotations.

  • Problem

    Region-based visual features focus on bounding-box contents and can neglect contextual information important for vision-language understanding.

  • Method

    SOHO uses a trainable CNN encoder, dynamic visual dictionary embedding, and multimodal Transformer trained with MVM, MLM, and ITM.

  • Results

    SOHO improves performance across four downstream tasks, including a 2.0% R@1 gain on MSCOCO text retrieval 5k and a 6.7% accuracy gain on SNLI-VE.

  • Takeaways & Limitations

    SOHO learns comprehensive yet compact visual representations directly from image-text pairs without relying on bounding-box annotations.

  • Takeaways & Limitations

    The visual dictionary requires freezing the ResNet parameters for the first 10 training epochs to avoid cold-start mode collapse.

Abstract

from arXiv · show

We study joint learning of Convolutional Neural Network (CNN) and Transformer for vision-language pre-training (VLPT) which aims to learn cross-modal alignments from millions of image-text pairs. State-of-the-art approaches extract salient image regions and align regions with words step-by-step. As region-based visual features usually represent parts of an image, it is challenging for existing vision-language models to fully understand the semantics from paired natural languages. In this paper, we propose SOHO to "See Out of tHe bOx" that takes a whole image as input, and learns vision-language representation in an end-to-end manner. SOHO does not require bounding box annotations which enables inference 10 times faster than region-based approaches. In particular, SOHO learns to extract comprehensive yet compact image features through a visual dictionary (VD) that facilitates cross-modal understanding. VD is designed to represent consistent visual abstractions of similar semantics. It is updated on-the-fly and utilized in our proposed pre-training task Masked Visual Modeling (MVM). We conduct experiments on four well-established vision-language tasks by following standard VLPT settings. In particular, SOHO achieves absolute gains of 2.0% R@1 score on MSCOCO text retrieval 5k test split, 1.5% accuracy on NLVR$^2$ test-P split, 6.7% accuracy on SNLI-VE test split, respectively.

1. Introduction

VLPT learns cross-modal representations from large-scale image-text pairs, but region-based features can omit contextual information outside bounding boxes. SOHO addresses this with end-to-end whole-image learning and a visual dictionary, reporting gains across downstream tasks.

  • VLPT models learn cross-modal representations from large-scale image-text pairs for tasks including VQA, image-text retrieval, and NLVR.
  • Region-based features focus on objects inside bounding boxes and can neglect contextual information needed for relation understanding and reasoning.
  • SOHO directly learns image embeddings, language embeddings, and their semantic alignment from image-text pairs in an end-to-end framework.
  • A visual dictionary represents comprehensive and compact visual semantics by grouping visual pixels with similar semantics through a moving-averaged encoder.
  • 2.0% R@1 score, 1.5% accuracy, and 6.7% accuracy are SOHO's absolute gains on MSCOCO text retrieval 5k, NLVR2 test-P, and SNLI-VE test splits, respectively.The reported results cover three downstream evaluation settings.

2. Related Work

Prior vision-language work commonly uses region-based or grid visual features, while SOHO applies an end-to-end VLPT pipeline without box annotations. Its framework combines a CNN encoder, visual dictionary embedding, and Transformer-based multimodal learning.

  • Earlier vision-language systems used ImageNet CNN features, while later methods adopted Visual Genome region features and task-specific grid features.
  • Vector-quantization approaches represent visual information, but SOHO dynamically updates its visual dictionary embedding from a trainable visual encoder.This dynamic update is described as capturing text-guided semantics from vision-language data.
  • VLPT models are categorized as two-stream or single-stream systems that process or fuse visual and language information in different ways.
  • SOHO uses a simple pipeline, relies on ImageNet-pre-trained visual parameters, and uses fewer annotations while targeting state-of-the-art performance.
  • SOHO combines a trainable CNN visual encoder, visual dictionary embedding, and multi-layer Transformer with masked visual, masked language, and image-text matching tasks.

3. Approach

SOHO is an end-to-end vision-language pre-training framework that replaces region-based inputs with whole-image visual features, a visual dictionary, and a cross-modal Transformer. It jointly trains these components using masked visual, masked language, and image-text matching objectives.

  • Architecture: SOHO combines a trainable CNN visual encoder, visual dictionary embedding module, and multi-layer Transformer for end-to-end vision-language pre-training.The framework produces visual features from images, aggregates them into visual tokens, and fuses them with language features.
  • Trainable Visual Encoder: Region-based features are limited by predefined object and attribute categories and can neglect important contextual information outside detected regions.The cited discussion identifies 1,600 object and 400 attribute categories as a constraint and notes that global-region quality is not guaranteed.
  • Trainable Visual Encoder: SOHO uses a whole-image trainable visual encoder to produce image-level features that can be optimized directly by pre-training and downstream losses.The encoder maps an input image I to visual features V, with ResNet followed by 1 × 1 convolution and 2 × 2 max pooling.
  • Visual Dictionary: The visual dictionary maps each visual feature to its nearest embedding and aggregates similar visual semantics into compact visual tokens.Its embeddings are updated online by moving averages, while stop-gradient handling keeps the visual encoder trainable despite the nondifferentiable nearest-neighbor assignment.
  • Visual Dictionary: The visual dictionary can capture semantics suited to cross-modal alignment because its clustering is influenced by vision-language learning tasks.The dictionary groups similar feature vectors into clusters whose indices act as virtual visual semantic labels.
  • Pre-training Pipeline: SOHO pre-trains with Masked Language Modeling, Masked Visual Modeling, and Image-Text Matching using a jointly trainable visual encoder, dictionary module, and Transformer.MVM predicts masked image features from surrounding visual features and all language tokens, while the three objectives receive equal loss weights.

4. Experiment

SOHO is evaluated on four downstream vision-language tasks using standard datasets and comparisons. It generally outperforms recent vision-language pre-training methods, with gains reported for retrieval, VQA, NLVR2, and visual entailment.

  • Downstream Tasks: SOHO is evaluated on image-text retrieval, VQA, NLVR, and visual entailment across MSCOCO, Flickr30K, VQA 2.0, NLVR2, and SNLI-VE.Image-text retrieval includes image-to-text and text-to-image subtasks.
  • Image-Text Retrieval: SOHO outperforms most recent VLPT works under most metrics on both MSCOCO and Flickr30K retrieval.The authors attribute the improvements to end-to-end pre-training and comprehensive yet compact visual semantic abstraction from the visual dictionary.
  • Visual Question Answering: 0.83% and 0.93% absolute improvements are obtained over LXMERT on VQA test-dev and test-std, respectively.SOHO also outperforms UNITER despite UNITER using out-of-domain pre-training data.
  • Visual Reasoning: 0.52% and 1.52% absolute gains over UNITER are reported on NLVR2 dev and test-P splits, respectively.The comparisons use the same settings as the cited LXMERT and UNITER results.
  • Visual Entailment: 85.00% and 84.95% accuracy are achieved on SNLI-VE val and test splits, with 6.41% and 6.67% absolute gains over UNITER.The authors connect these results to refining the CNN backbone together with the cross-modal Transformer for vision-language alignment.

4.3. Ablation Study

The ablation studies show that the visual dictionary generally improves performance across tasks, with k = 2048 usually providing the best balance of semantic detail and abstraction. SOHO also uses fewer inference stages than BUTD-based methods, while its visual dictionary groups image patches into consistent semantic abstractions.

  • Visual Dictionary Size: Visual dictionary size trades off fine-grained semantic grouping against coherent visual abstraction for vision-language alignment.Larger dictionaries can capture more complete semantics, but overly fine-grained grouping may harm abstraction and alignment.
  • Visual Dictionary Size: k = 2048 achieves the best results in most cases and is adopted as the default setting.Across VD sizes from 1024 to 8192, 2048 or 4096 generally performs best, with k = 2048 working best in most cases.
  • Visual Dictionary Effectiveness: A visual dictionary improves performance under almost all metrics across a broad range of downstream tasks.Compared with the baseline without VD, improvements are observed for k = 1024, 2048, and 4096.
  • Visual Dictionary Visualization: The visual dictionary groups meaningful and consistent image patches into indices that abstract visual semantics.Each index correlates with multiple visual features, with each feature corresponding to an image patch.
  • Inference Efficiency: SOHO has two inference stages, CNN and Transformer forwarding, whereas BUTD-based methods add region feature generation.The efficiency comparison uses matched hardware and model settings, including a V100 GPU, ResNet-101, a 12-layer Transformer, and 100 boxes.

5. Conclusion

SOHO presents an end-to-end vision-language pre-training design that learns compact visual representations and aligns them with language through a visual dictionary and three pre-training tasks. Across four downstream tasks, it outperforms models using region-based image features while reducing dependence on bounding-box annotations and accelerating inference.

  • 5. Conclusion: SOHO is an end-to-end vision-language pre-training model that learns comprehensive yet compact visual representations for cross-modal understanding.It transforms images into concrete semantics that can be fused with language tokens.
  • 5. Conclusion: A visual dictionary and three pre-training tasks build connections between images and languages.The visual dictionary transforms an image into concrete semantics for fusion with language tokens.
  • 5. Conclusion: SOHO outperforms pre-training models with region-based image features on four downstream tasks.The conclusion reports this as the aggregate outcome of the downstream evaluations.
  • 5. Conclusion: SOHO removes the requirement for bounding-box annotations and reduces inference time in vision-language tasks by about 10 times.The conclusion connects this efficiency to reduced human labeling costs and more online applications.

A.1. Dataset Statistics

The appendix summarizes dataset usage and distinguishes in-domain from out-of-domain pre-training data. SOHO focuses on in-domain datasets, which affects direct comparability with settings that additionally use out-of-domain data.

  • A.1. Dataset Statistics: The appendix reports detailed train/test image and text counts for pre-training and downstream datasets.It also compares pre-training dataset usage across recent VLPT works.
  • A.1. Dataset Statistics: MSCOCO and Visual Genome are treated as in-domain datasets, while Conceptual Captions and SBU Captions are out-of-domain and noisier.The classification follows UNITER's distinction between the two dataset classes.
  • A.1. Dataset Statistics: SOHO focuses on in-domain datasets because they are commonly used in vision-language tasks and VLPT works.This setting is used for fair comparison with UNITER's in-domain results when available.
  • A.1. Dataset Statistics: Results are not directly comparable when UNITER uses in-domain plus out-of-domain pre-training data and SOHO uses only in-domain data.The appendix states that SOHO's in-domain setting is inferior to the combined setting in such comparisons.
  • A.1. Dataset Statistics: Including out-of-domain data in pre-training is identified as future work.The statement marks the current data scope rather than reporting an evaluated configuration.

A.2. Implementation Details

SOHO speeds training by combining mixed-precision computation with mini-batch organization that avoids repeatedly forwarding the same image through the visual backbone.

  • A.2. Implementation Details: Mixed-precision training reduces memory cost and speeds up the training procedure.It is the first of two strategies used to improve training efficiency.
  • A.2. Implementation Details: Within a mini-batch, SOHO forwards an image through the visual backbone once when it has multiple corresponding texts.The image is then concatenated with each text into the cross-modal Transformers; an example pairs one image with four texts.

A.3. Visualization of Visual Dictionary

The visualizations show that visual-dictionary items group image patches with meaningful, consistent semantics, despite the absence of object bounding-box supervision.

  • A.3. Visualization of Visual Dictionary: Table 9 reports statistics for different datasets, with “*” denoting the Karpathy split.
  • A.3. Visualization of Visual Dictionary: Visual-dictionary items group image patches with meaningful and consistent semantics.The paper visualizes two examples and randomly selects ten additional indices in supplementary material.
  • A.3. Visualization of Visual Dictionary: The model learns unified representations for different image semantics without object bounding-box annotations.

A.4. Discussion

Image-text retrieval methods either compare separately projected image and text representations through late fusion or jointly encode them through Transformer-based early fusion.

  • A.4. Discussion: Traditional retrieval approaches project image and text into a common representation space before correlating them through late fusion.A widely used late-fusion method calculates cosine similarity with a dot-product operation.
  • A.4. Discussion: Cosine similarity based on a dot-product operation is described as simple and fast.
  • A.4. Discussion: Transformer-based approaches early fuse image and text through a multi-layer Transformer to obtain a unified representation.
Loading 2104.03135v2…