PaperlayerView exact revision on arXiv

Source-linked AI summary

Seeing Out of tHe bOx: End-to-End Pre-training for Vision-Language Representation Learning

Zhicheng Huang, Zhaoyang Zeng, Yupan Huang, Bei Liu, Dongmei Fu, Jianlong Fu

arXiv:2104.03135v2cs.CV

TL;DR

Region-based visual features can miss contextual information needed for vision-language understanding, motivating an end-to-end alternative. SOHO jointly learns whole-image and language representations with a dynamic visual dictionary and multimodal Transformer, and reports absolute gains across multiple downstream tasks while avoiding box annotations.

  • Problem

    Region-based visual features focus on bounding-box contents and can neglect contextual information important for vision-language understanding.

    Source: §1 · p. 1, §3.1 · p. 3

  • Method

    SOHO uses a trainable CNN encoder, dynamic visual dictionary embedding, and multimodal Transformer trained with MVM, MLM, and ITM.

    Source: §3 · p. 3, §3.3 · p. 4

  • Results

    SOHO improves performance across four downstream tasks, including a 2.0% R@1 gain on MSCOCO text retrieval 5k and a 6.7% accuracy gain on SNLI-VE.

    Source: §1 · p. 2, §4.2.4 · p. 7, §5 · p. 8

  • Takeaways & Limitations

    SOHO learns comprehensive yet compact visual representations directly from image-text pairs without relying on bounding-box annotations.

    Source: §1 · p. 2, §5 · p. 8

    The visual dictionary requires freezing the ResNet parameters for the first 10 training epochs to avoid cold-start mode collapse.

    Source: §3.2 · p. 4

Abstract

from arXiv · show

We study joint learning of Convolutional Neural Network (CNN) and Transformer for vision-language pre-training (VLPT) which aims to learn cross-modal alignments from millions of image-text pairs. State-of-the-art approaches extract salient image regions and align regions with words step-by-step. As region-based visual features usually represent parts of an image, it is challenging for existing vision-language models to fully understand the semantics from paired natural languages. In this paper, we propose SOHO to "See Out of tHe bOx" that takes a whole image as input, and learns vision-language representation in an end-to-end manner. SOHO does not require bounding box annotations which enables inference 10 times faster than region-based approaches. In particular, SOHO learns to extract comprehensive yet compact image features through a visual dictionary (VD) that facilitates cross-modal understanding. VD is designed to represent consistent visual abstractions of similar semantics. It is updated on-the-fly and utilized in our proposed pre-training task Masked Visual Modeling (MVM). We conduct experiments on four well-established vision-language tasks by following standard VLPT settings. In particular, SOHO achieves absolute gains of 2.0% R@1 score on MSCOCO text retrieval 5k test split, 1.5% accuracy on NLVR$^2$ test-P split, 6.7% accuracy on SNLI-VE test split, respectively.

1. Introduction

VLPT learns cross-modal representations from large-scale image-text pairs, but region-based features can omit contextual information outside bounding boxes. SOHO addresses this with end-to-end whole-image learning and a visual dictionary, reporting gains across downstream tasks.

  • VLPT models learn cross-modal representations from large-scale image-text pairs for tasks including VQA, image-text retrieval, and NLVR.Source: §1 · p. 1
  • Region-based features focus on objects inside bounding boxes and can neglect contextual information needed for relation understanding and reasoning.Source: §1 · p. 1
  • SOHO directly learns image embeddings, language embeddings, and their semantic alignment from image-text pairs in an end-to-end framework.Source: §1 · p. 2
  • A visual dictionary represents comprehensive and compact visual semantics by grouping visual pixels with similar semantics through a moving-averaged encoder.Source: §1 · p. 2
  • 2.0% R@1 score, 1.5% accuracy, and 6.7% accuracy are SOHO's absolute gains on MSCOCO text retrieval 5k, NLVR2 test-P, and SNLI-VE test splits, respectively.The reported results cover three downstream evaluation settings.Source: §1 · p. 2

2. Related Work

Prior vision-language work commonly uses region-based or grid visual features, while SOHO applies an end-to-end VLPT pipeline without box annotations. Its framework combines a CNN encoder, visual dictionary embedding, and Transformer-based multimodal learning.

  • Earlier vision-language systems used ImageNet CNN features, while later methods adopted Visual Genome region features and task-specific grid features.Source: §2.1 · p. 2
  • Vector-quantization approaches represent visual information, but SOHO dynamically updates its visual dictionary embedding from a trainable visual encoder.This dynamic update is described as capturing text-guided semantics from vision-language data.Source: §2.1 · p. 2
  • VLPT models are categorized as two-stream or single-stream systems that process or fuse visual and language information in different ways.Source: §2.2 · p. 2
  • SOHO uses a simple pipeline, relies on ImageNet-pre-trained visual parameters, and uses fewer annotations while targeting state-of-the-art performance.Source: §2.2 · pp. 2–3
  • SOHO combines a trainable CNN visual encoder, visual dictionary embedding, and multi-layer Transformer with masked visual, masked language, and image-text matching tasks.Source: Figure 2 · p. 3

3. Approach

SOHO is an end-to-end vision-language pre-training framework that replaces region-based inputs with whole-image visual features, a visual dictionary, and a cross-modal Transformer. It jointly trains these components using masked visual, masked language, and image-text matching objectives.

Architecture

  • SOHO combines a trainable CNN visual encoder, visual dictionary embedding module, and multi-layer Transformer for end-to-end vision-language pre-training.The framework produces visual features from images, aggregates them into visual tokens, and fuses them with language features.Source: §3 · p. 3

Trainable Visual Encoder

  • Region-based features are limited by predefined object and attribute categories and can neglect important contextual information outside detected regions.The cited discussion identifies 1,600 object and 400 attribute categories as a constraint and notes that global-region quality is not guaranteed.Source: §3.1 · p. 3
  • SOHO uses a whole-image trainable visual encoder to produce image-level features that can be optimized directly by pre-training and downstream losses.The encoder maps an input image I to visual features V, with ResNet followed by 1 × 1 convolution and 2 × 2 max pooling.Source: §3.1 · pp. 3–4

Visual Dictionary

  • The visual dictionary maps each visual feature to its nearest embedding and aggregates similar visual semantics into compact visual tokens.Its embeddings are updated online by moving averages, while stop-gradient handling keeps the visual encoder trainable despite the nondifferentiable nearest-neighbor assignment.Source: §3.2 · p. 4
  • The visual dictionary can capture semantics suited to cross-modal alignment because its clustering is influenced by vision-language learning tasks.The dictionary groups similar feature vectors into clusters whose indices act as virtual visual semantic labels.Source: §3.2 · p. 4

Pre-training Pipeline

  • SOHO pre-trains with Masked Language Modeling, Masked Visual Modeling, and Image-Text Matching using a jointly trainable visual encoder, dictionary module, and Transformer.MVM predicts masked image features from surrounding visual features and all language tokens, while the three objectives receive equal loss weights.Source: §3.3 · pp. 4–5

4. Experiment

SOHO is evaluated on four downstream vision-language tasks using standard datasets and comparisons. It generally outperforms recent vision-language pre-training methods, with gains reported for retrieval, VQA, NLVR2, and visual entailment.

Downstream Tasks

  • SOHO is evaluated on image-text retrieval, VQA, NLVR, and visual entailment across MSCOCO, Flickr30K, VQA 2.0, NLVR2, and SNLI-VE.Image-text retrieval includes image-to-text and text-to-image subtasks.Source: §4.2 · pp. 5–6

Image-Text Retrieval

  • SOHO outperforms most recent VLPT works under most metrics on both MSCOCO and Flickr30K retrieval.The authors attribute the improvements to end-to-end pre-training and comprehensive yet compact visual semantic abstraction from the visual dictionary.Source: §4.2.1 · p. 6

Visual Question Answering

  • 0.83% and 0.93% absolute improvements are obtained over LXMERT on VQA test-dev and test-std, respectively.SOHO also outperforms UNITER despite UNITER using out-of-domain pre-training data.Source: §4.2.2 · p. 7

Visual Reasoning

  • 0.52% and 1.52% absolute gains over UNITER are reported on NLVR2 dev and test-P splits, respectively.The comparisons use the same settings as the cited LXMERT and UNITER results.Source: §4.2.3 · p. 7

Visual Entailment

  • 85.00% and 84.95% accuracy are achieved on SNLI-VE val and test splits, with 6.41% and 6.67% absolute gains over UNITER.The authors connect these results to refining the CNN backbone together with the cross-modal Transformer for vision-language alignment.Source: §4.2.4 · p. 7

4.3. Ablation Study

The ablation studies show that the visual dictionary generally improves performance across tasks, with k = 2048 usually providing the best balance of semantic detail and abstraction. SOHO also uses fewer inference stages than BUTD-based methods, while its visual dictionary groups image patches into consistent semantic abstractions.

Visual Dictionary Size

  • Visual dictionary size trades off fine-grained semantic grouping against coherent visual abstraction for vision-language alignment.Larger dictionaries can capture more complete semantics, but overly fine-grained grouping may harm abstraction and alignment.Source: §4.3 · p. 7
  • k = 2048 achieves the best results in most cases and is adopted as the default setting.Across VD sizes from 1024 to 8192, 2048 or 4096 generally performs best, with k = 2048 working best in most cases.Source: §4.3 · pp. 7–8

Visual Dictionary Effectiveness

  • A visual dictionary improves performance under almost all metrics across a broad range of downstream tasks.Compared with the baseline without VD, improvements are observed for k = 1024, 2048, and 4096.Source: §4.3 · p. 8

Visual Dictionary Visualization

  • The visual dictionary groups meaningful and consistent image patches into indices that abstract visual semantics.Each index correlates with multiple visual features, with each feature corresponding to an image patch.Source: §4.4 · p. 8, Figure 3 · p. 8

Inference Efficiency

  • SOHO has two inference stages, CNN and Transformer forwarding, whereas BUTD-based methods add region feature generation.The efficiency comparison uses matched hardware and model settings, including a V100 GPU, ResNet-101, a 12-layer Transformer, and 100 boxes.Source: §4.5 · p. 8

5. Conclusion

SOHO presents an end-to-end vision-language pre-training design that learns compact visual representations and aligns them with language through a visual dictionary and three pre-training tasks. Across four downstream tasks, it outperforms models using region-based image features while reducing dependence on bounding-box annotations and accelerating inference.

5. Conclusion

  • SOHO is an end-to-end vision-language pre-training model that learns comprehensive yet compact visual representations for cross-modal understanding.It transforms images into concrete semantics that can be fused with language tokens.Source: §5 · p. 8
  • A visual dictionary and three pre-training tasks build connections between images and languages.The visual dictionary transforms an image into concrete semantics for fusion with language tokens.Source: §5 · p. 8
  • SOHO outperforms pre-training models with region-based image features on four downstream tasks.The conclusion reports this as the aggregate outcome of the downstream evaluations.Source: §5 · p. 8
  • SOHO removes the requirement for bounding-box annotations and reduces inference time in vision-language tasks by about 10 times.The conclusion connects this efficiency to reduced human labeling costs and more online applications.Source: §5 · p. 8

A.1. Dataset Statistics

The appendix summarizes dataset usage and distinguishes in-domain from out-of-domain pre-training data. SOHO focuses on in-domain datasets, which affects direct comparability with settings that additionally use out-of-domain data.

A.1. Dataset Statistics

  • The appendix reports detailed train/test image and text counts for pre-training and downstream datasets.It also compares pre-training dataset usage across recent VLPT works.Source: §A.1 · p. 9
  • MSCOCO and Visual Genome are treated as in-domain datasets, while Conceptual Captions and SBU Captions are out-of-domain and noisier.The classification follows UNITER's distinction between the two dataset classes.Source: §A.1 · p. 9
  • SOHO focuses on in-domain datasets because they are commonly used in vision-language tasks and VLPT works.This setting is used for fair comparison with UNITER's in-domain results when available.Source: §A.1 · p. 9
  • Results are not directly comparable when UNITER uses in-domain plus out-of-domain pre-training data and SOHO uses only in-domain data.The appendix states that SOHO's in-domain setting is inferior to the combined setting in such comparisons.Source: §A.1 · p. 9
  • Including out-of-domain data in pre-training is identified as future work.The statement marks the current data scope rather than reporting an evaluated configuration.Source: §A.1 · p. 9

A.2. Implementation Details

SOHO speeds training by combining mixed-precision computation with mini-batch organization that avoids repeatedly forwarding the same image through the visual backbone.

A.2. Implementation Details

  • Mixed-precision training reduces memory cost and speeds up the training procedure.It is the first of two strategies used to improve training efficiency.Source: §A.2 · p. 9
  • Within a mini-batch, SOHO forwards an image through the visual backbone once when it has multiple corresponding texts.The image is then concatenated with each text into the cross-modal Transformers; an example pairs one image with four texts.Source: §A.2 · p. 9

A.3. Visualization of Visual Dictionary

The visualizations show that visual-dictionary items group image patches with meaningful, consistent semantics, despite the absence of object bounding-box supervision.

A.3. Visualization of Visual Dictionary

  • Table 9 reports statistics for different datasets, with “*” denoting the Karpathy split.Source: Table 9 · p. 9
  • Visual-dictionary items group image patches with meaningful and consistent semantics.The paper visualizes two examples and randomly selects ten additional indices in supplementary material.Source: §A.3 · p. 9
  • The model learns unified representations for different image semantics without object bounding-box annotations.Source: §A.3 · p. 9

A.4. Discussion

Image-text retrieval methods either compare separately projected image and text representations through late fusion or jointly encode them through Transformer-based early fusion.

A.4. Discussion

  • Traditional retrieval approaches project image and text into a common representation space before correlating them through late fusion.A widely used late-fusion method calculates cosine similarity with a dot-product operation.Source: §A.4 · p. 9
  • Cosine similarity based on a dot-product operation is described as simple and fast.Source: §A.4 · p. 9
  • Transformer-based approaches early fuse image and text through a multi-layer Transformer to obtain a unified representation.Source: §A.4 · p. 9

Related papers

Loading 2104.03135v2…