PaperlayerView exact revision on arXiv

Source-linked AI summary

VinVL: Revisiting Visual Representations in Vision-Language Models

Pengchuan Zhang, Xiujun Li, Xiaowei Hu, Jianwei Yang, Lei Zhang, Lijuan Wang, Yejin Choi, Jianfeng Gao

arXiv:2101.00529v2cs.CVcs.AIcs.CLcs.LG

TL;DR

Prior vision-language pre-training research has focused mainly on cross-modal fusion, while object-centric visual representations have received less attention. This paper develops an improved object detector and evaluates its features with OSCAR+ across seven VL tasks, reporting new state-of-the-art results on all seven.

  • Problem

    Prior VLP research mainly improved cross-modal fusion while leaving object-centric visual representations less studied, despite evidence that visual features matter in VL models.

    Source: §1 · p. 1, §2 · p. 3

  • Method

    The paper develops an improved object-detection model for VL visual features, uses those features to pre-train OSCAR+ on paired image-text data, and fine-tunes across seven VL tasks.

    Source: §3 · p. 6, §6 · p. 14

  • Results

    Across all seven VL tasks, the new detector substantially uplifted state-of-the-art results; on VQA, OSCAR+B with X152-C4 reached 74.90 accuracy versus 72.38 for OSCARB with R101-C4.

    Source: §5.2 · p. 11, §6 · p. 14

  • Takeaways & Limitations

    The ablation attributes 95% of the VQA accuracy gain to improved visual features, indicating visual representations matter significantly in downstream VL tasks.

    Source: §5.2 · p. 11

    The reported across-task gains have one stated exception: B@4 on image captioning.

    Source: §5.1 · p. 9

Abstract

from arXiv · show

This paper presents a detailed study of improving visual representations for vision language (VL) tasks and develops an improved object detection model to provide object-centric representations of images. Compared to the most widely used \emph{bottom-up and top-down} model \cite{anderson2018bottom}, the new model is bigger, better-designed for VL tasks, and pre-trained on much larger training corpora that combine multiple public annotated object detection datasets. Therefore, it can generate representations of a richer collection of visual objects and concepts. While previous VL research focuses mainly on improving the vision-language fusion model and leaves the object detection model improvement untouched, we show that visual features matter significantly in VL models. In our experiments we feed the visual features generated by the new object detection model into a Transformer-based VL fusion model \oscar \cite{li2020oscar}, and utilize an improved approach \short\ to pre-train the VL model and fine-tune it on a wide range of downstream VL tasks. Our results show that the new visual features significantly improve the performance across all VL tasks, creating new state-of-the-art results on seven public benchmarks. We will release the new object detection model to public.

1 Introduction

This paper shifts attention from cross-modal fusion to object-centric visual representations, developing a new object-attribute detector and studying its effect on vision-language tasks. It reports improved results across tasks using features from the detector with OSCAR+.

1 Introduction

  • The X152-C4 object-attribute detector is larger and better-designed for VL than the classical OD model [2], and combines COCO [25], OpenImages [17], Objects365, and Visual Genome [16].It produces representations for 1848 object categories and 524 attribute categories, and the authors say they will release it to the research community.Source: §1 · pp. 1–3
  • The new visual features improve all seven VL tasks and yield state-of-the-art results across tasks, including GQA, where published pre-trained models had not surpassed NSM [12].Table 1 describes the comparison as replacing features from Anderson et al. [2]; the paper reports that its NoCaps results directly replace those features, while other tasks also use OSCAR+ pre-training.Source: Table 1 · p. 2, §1 · p. 3
  • Compared with an OpenImages-trained X152-FPN, the four-dataset X152-C4 detects richer concepts and attributes and covers nearly all semantically meaningful regions.Figure 1 illustrates attributes for “boy” and concepts such as fin, wave, and beach that the OpenImages model misses.Source: Figure 1 · p. 2
  • The authors pre-train OSCAR+ [21] on 8.85 million text-image pairs with fixed features from the new detector, then fine-tune it for VL understanding, retrieval, and generation tasks.The tasks include VQA [8], GQA [13], NLVR2 [35], COCO text-image retrieval [25], COCO image captioning [25], and NoCaps [1].Source: §1 · pp. 2–3
  • The paper’s ablations examine how object-category diversity, visual-attribute training, training-data scale, model size, and architecture contribute to performance gains.Source: §1 · p. 3

2 Improving Vision (V) in Vision Language (VL)

The section argues that VL research has left visual features largely untouched and develops an improved object detector to provide richer object and attribute representations for VL tasks.

2 Improving Vision (V) in Vision Language (VL)

  • VL models combine image understanding with cross-modal understanding; OSCAR [21] shows detected-object semantics can anchor joint representations, while most VLP work leaves vision features unchanged.The vision module provides semantic representations q and distributional region features v.Source: §2 · p. 3

2.1 Object Detection Pre-training

  • Object-detection pre-training combines COCO, OpenImagesV5, Objects365, and VG through class-aware sampling and vocabulary mapping, producing a unified dataset with 1,848 classes.The approach pre-trains on four public datasets and then fine-tunes on VG with an attribute branch.Source: §2.1 · p. 4
  • For VL region features, the authors choose C4 over FPN for its ImageNet-pretrained convolutional features; X152-C4 also outperforms X152-FPN in object detection under the same training process.They attribute C4's VL advantage to pre-trained layers and the convolutional head's inductive bias; X152-C4 is initialized from ImageNet-5K and trained for 1.8M iterations.Source: §2.1 · p. 5, §2.3 · p. 5

2.2 Injecting attribute information into the model

  • On VG, attribute-branch fine-tuning with an attribute-loss weight of 1.25 significantly outperforms prior models [2] [14] in detecting objects and attributes.The attribute branch covers 524 classes; the authors use a larger attribute-loss weight than the 0.5 used in [2] [14].Source: §2.2 · p. 5

2.3 Efficient region feature extractor for VL tasks

  • Class-agnostic NMS and non-dilated convolutions speed region extraction without downstream VL accuracy loss; the detector outputs object-name tags q and 2048-D region features with 6-D position encodings.Each region feature pairs a 2048-dimensional representation with a 6-dimensional position encoding.Source: §2.3 · p. 5

3 OSCAR+ Pre-training

OSCAR+ pre-trains an improved OSCAR model to learn joint image-text representations, using image tags as anchors for alignment. The section describes its corpus, objectives, and two model variants.

3.1 Pre-training corpus

  • The pre-training corpus combines captioning, visual QA, and image-tagging datasets into 5.65 million unique images and 8.85 million text-tag-image triples.The authors leave use of larger corpora, including full OpenImages and YFCC, to future work.Source: §3.1 · p. 6

3.2 Pre-training Objectives

  • The proposed 3-way contrastive loss transfers well to both VQA and text-image retrieval, unlike answer-polluted triplets alone, which adapt effectively to VQA but not retrieval.The comparison is reported for the pre-training contrastive-loss experiments in Table 3.Source: Table 3 · p. 6, §3.2 · p. 7
  • OSCAR+ uses masked token loss on text and a novel 3-way contrastive loss, differing from OSCAR’s binary contrastive loss [21].The contrastive loss is designed for training objectives used in VQA and text-image matching.Source: §3.2 · p. 6
  • For contrastive training, a 3-way classifier labels triplets as matched, caption-polluted, or answer-polluted, using negatives formed by corrupting either text input.The training set contains 50% matched, 25% caption-polluted, and 25% answer-polluted triplets; negatives are uniformly sampled from corpus captions/questions or tags/answers.Source: §3.2 · p. 7

3.3 Pre-trained models

  • OSCAR+B and OSCAR+L use BERT base and large initializations, respectively, and each is trained for at least 1M steps with batch size 1024.Their learning rates are 1e-4 and 3e-5, respectively; the text and region-feature sequence lengths are 35 and 50.Source: §3.3 · p. 7

4 Adapting to VL Tasks

The section adapts pretrained models to seven downstream VL tasks—five understanding and two generation—with task-specific fine-tuning strategies.

VQA & GQA

  • For VQA v2.0 [8] and GQA [13], OSCAR+ combines each question with object tags and region features, then classifies its [CLS] output to predict an answer.Following [2], the answer sets contain 3, 129 candidates for VQA and 1, 852 for GQA.Source: §4 · pp. 7–8

Image Captioning & NoCaps

  • For image captioning, OSCAR+ is fine-tuned with a seq2seq objective that predicts masked caption tokens from image regions, object tags, and remaining caption context.Training masks 15% of caption tokens and constrains caption-token attention to preceding tokens, while allowing caption tokens to attend to image regions and tags.Source: §4 · p. 8
  • NoCaps [1] evaluates descriptions of novel Open Images [17] objects unseen during training; replacing VIVO’s vision model with the new model significantly improved its performance.Following NoCaps restrictions, OSCAR+ is trained directly on COCO without pretraining initialization, using predicted Visual Genome and Open Images labels; the VIVO change is reported in Table 9.Source: §4 · p. 8

Image(-to-Text) Retrieval & Text(-to-Image) Retrieval

  • Image-to-text and text-to-image retrieval rank candidate pairs by a classifier’s predicted match score, with top-K results reported on the 1K and 5K COCO test sets.Following [21], the task is formulated as binary classification of matched and randomly mismatched image-text pairs; evaluation follows [19].Source: §4 · p. 8

NLVR2

  • For NLVR2 [35], OSCAR+ encodes the description with each image separately and concatenates the two [CLS] outputs for binary prediction.The task determines whether a text description is true of a pair of images.Source: §4 · pp. 7–8

5 Experiments & Analysis

Experiments show that VINVL’s visual features improve results across VL tasks, while ablations find that visual pre-training, training data and model scale, architecture, and richer visual semantics all affect performance. The reported task results include exceptions and evaluation conditions that qualify the broad gains.

5.1 Main Results

  • Across seven VL tasks, OSCAR+ with VINVL surpasses prior SoTA on all except image-captioning BLEU@4; OSCAR+B matches or beats large-model SoTA, while OSCAR+L substantially uplifts it.The overall comparison is reported in Table 4; the paper identifies B@4 on image captioning as the sole exception.Source: Table 4 · p. 9, §5.1 · pp. 9, 11
  • A single OSCAR+B beats the best VQA leaderboard ensemble, and OSCAR+ with VINVL is the first VLP model reported to outperform NSM on GQA.The VQA comparison is dated Dec. 12, 2020; NSM includes reasoning components designed for GQA.Source: §5.1 · p. 9, Table 5 · p. 9, Table 6 · p. 9
  • On the COCO online captioning leaderboard, one model ranked No. 1 among 263 models; on NoCaps, VinVL+VIVO improved VIVO by 6 CIDEr points.Without VLP, the BERT-based captioning model using VinVL features also surpassed human performance in CIDEr; the NoCaps models were trained on COCO without extra image-caption pairs.Source: Table 8 · p. 10, Table 9 · p. 10, §5.1 · p. 11

5.2 Ablation Analysis

  • On VQA, OSCAR+B with X152-C4 features raised accuracy from 72.38 to 74.90; the paper attributes 5% of the gain to OSCAR+ pre-training and 95% to improved visual features.The reported gains from VinVL and VLP over the no-VLP R101-C4 baseline are additive: 2.82 and 3.94, respectively.Source: §5.2 · p. 11, Table 12 · p. 11
  • Vision models trained on the merged four detection datasets outperform VG-only models, with larger improvements as model size increases; the paper also reports that architecture gaps diminish on the merged data.The comparison isolates data and model size without VLP; R50-FPN underperforms R50-C5 on VG-only training, but their gap narrows on the merged dataset.Source: §5.2 · p. 12, Table 13 · p. 12
  • OD pre-training benefits object and attribute detection, while FPN models perform worse than C4 models in VG attribute detection and show no VG object-detection advantage.VG mAP is lower than on typical OD datasets because it has many object classes with limited, highly imbalanced annotations and missing evaluation annotations.Source: §5.2 · pp. 12–13, Table 14 · p. 12
  • Richer object vocabularies generally improve VQA, and models trained with attributes outperform those without them.The reported vocabulary ordering is VG-obj < ImageNet < VG w/o attr; the paper says VG-obj omits important concepts such as sky, water, and mountains.Source: §5.2 · p. 13, Table 15 · p. 13
  • For VQA, COCO ground-truth boxes perform worse than proposals from VG-trained models, indicating that VL benefits from richer visual semantics even when localization is imperfect.The comparison includes object-only and object-and-stuff ground-truth regions with limited vocabularies, versus larger-vocabulary regions proposed by VG-trained models.Source: §5.2 · p. 13, Table 16 · p. 14

6 Conclusion

A larger, VL-focused object-detection model trained on broader text-image corpora produces richer visual features and substantially uplifts results across seven VL tasks. The study attributes these gains mainly to choices in category diversity, visual-attribute training, data scale, model size, and architecture.

6 Conclusion

  • The new object-detection model substantially uplifts state-of-the-art results across all seven VL tasks on multiple public benchmarks.Its features were evaluated in a VL fusion model that was pre-trained on a large-scale paired text-image corpus and fine-tuned on seven VL tasks.Source: §6 · p. 14
  • Compared with the widely used bottom-up and top-down model [2], the new model is larger, better designed for VL, and trained on much larger text-image corpora to represent a richer range of objects and concepts.Source: §6 · p. 14
  • Ablations show that gains mainly come from object-category diversity, visual-attribute training, training-data scale, model size, and model architecture.Source: §6 · p. 14

A Qualitative study of three pre-trained vision models · B OSCAR+ pre-training

The qualitative comparison shows that the four-dataset-pretrained X152-C4 model fine-tuned on VG has some repetitive detections but no obvious wrong detections, unlike the VG-trained R101-C4 model. The passages also report detections from X152-FPN trained on Open Images V5.

A Qualitative study of three pre-trained vision models

  • The study applies three pretrained object-detection models to the image in Figure 1 for a detailed comparison.Source: A Qualitative study of three… · p. 17
  • The Open Images V5-trained X152-FPN detections include surfboards, a man, human limbs, swimwear, shorts, and a boy.These detections are listed for Figure 2.Source: A Qualitative study of three… · p. 17
  • The VG-trained R101-C4 model produced obviously wrong detections, while the four-dataset-pretrained X152-C4 model fine-tuned on VG had repetitive detections but no obvious errors.The examples are shown in Figure 3, with R101-C4 predictions on top and X152-C4 predictions on the bottom.Source: A Qualitative study of three… · pp. 17, 19, Figure 3 · p. 18
  • The R101-C4 model trained on VG by Anderson et al. [2] generated the red-marked wrong detections shown in Figure 3.Its listed predictions include repeated descriptions of the boy, surfboards, and water.Source: A Qualitative study of three… · p. 17, Figure 3 · p. 18
  • The four-dataset-pretrained X152-C4 model fine-tuned on VG produced detections spanning people, surfboards, waves, ocean, beach, and other image details.The passage notes some repetitive detections but no obvious wrong detections.Source: A Qualitative study of three… · p. 19

B.1 Pre-training Corpus · B.2 OSCAR+ pre-training objectives

The corpus study compares three corpus sizes and expands OSCAR+ pre-training with image-tagging data converted into caption–tag–feature triplets. Its objectives combine masked-token prediction over text and tags with a 3-way contrastive loss intended to support retrieval and visual question answering.

B.1 Pre-training Corpus

  • OSCAR+ pre-training uses Small, Medium, and Large corpora, whose image and text statistics are reported in Table 17.Source: §B.1 · p. 19, Table 17 · p. 20
  • Unlike OSCAR [21], the corpus incorporates OpenImages tagging data by generating captions to form caption–tag–image-feature triplets.Source: §B.1 · p. 19
  • Self-training scales the pre-training corpus using large image-tagging datasets, including OpenImages (9M) and YFCC (92M).Source: §B.1 · p. 19

B.2 OSCAR+ pre-training objectives

  • Masked Token Loss (MTL) is applied to both caption or question tokens and object-tag answer tokens, treated together as sequence h = [w, q].Source: §B.2 · p. 20
  • At each iteration, MTL masks 15% of input tokens and trains the model to recover them from surrounding tokens and image features.Source: §B.2 · p. 20
  • MTL matches OSCAR [21] and resembles BERT’s masked language modeling, with image information helping ground word embeddings in visual context.Source: §B.2 · p. 20
  • The 3-way contrastive loss is presented as an objective that simultaneously mimics text–image retrieval and visual question answering.The passage points to Section 3.2 of the main paper for its presentation.Source: §B.2 · p. 20

B.3 Ablation of the two new techniques

The ablations show that adding tagged-image data to OSCAR+ improves VQA, while the new 3-way contrastive loss primarily improves Text-Image Retrieval. They also find only minor overall gains from OSCAR+ over OSCAR when both use features from [2].

Effect of the new 3-way contrastive loss

  • The 3-way contrastive loss leaves VQA performance the same as OSCAR pre-training but significantly improves Text-Image Retrieval.Source: §B.3 · p. 20

Effect of self-training: Leveraging Image Tagging data

  • Adding 1.7M OpenImages tagging data improves OSCAR+ VQA, and adding tagged Conceptual Captions and SBU improves it further; VinVL features outperform [2] features.Figure 4 reports VQA-dev accuracy for checkpoints fine-tuned with a fixed scheme.Source: §B.3 · p. 20, Figure 4 · p. 21

Overall improvement from OSCAR to OSCAR+

  • With features from [2], OSCAR+ improves only modestly over OSCAR because it adds just 1.7M tagging images to a much larger original corpus.The original corpus is 3.98M images and 7.18M image-caption pairs; the new loss contributes more to retrieval than to VQA.Source: §B.3 · pp. 20–22
  • The authors expect larger improvements by scaling OSCAR+ with OpenImages (9M) and YFCC (92M) image-tagging datasets.Source: §B.3 · p. 22

C Downstream Tasks Fine-tuning … D More on the Effect of the Object-Attribute Vocabulary Size: disentangling the effects of region proposals and model weights

The section describes OSCAR+-based fine-tuning procedures for downstream vision-language tasks and examines how region proposals and model weights affect visual features. Its disentanglement experiments find that both contribute to performance on VL tasks.

C Downstream Tasks Fine-tuning

C.1 VQA

  • VQA fine-tuning concatenates the question, object tags, and region features, then uses OSCAR+’s [CLS] output to predict soft-target answer scores.The model minimizes cross-entropy against the soft targets and uses Softmax for inference.Source: §C.1 · p. 22

C.2 GQA

  • GQA fine-tuning uses the 1,852-answer candidate set, training first on the unbalanced all-split for 5 epochs and then on the balanced split for 2 epochs.The first stage uses a learning rate of 5e−5 and batch size 128.Source: §C.2 · p. 22

C.3 Image Captioning

  • For image captioning, OSCAR+ uses a seq2seq objective with causal caption-token attention while caption tokens can attend to image regions and object tags.The model is fine-tuned directly on COCO without additional Conceptual Captions pre-training to test generation generalization.Source: §C.3 · pp. 22–23

C.4 NoCaps

  • For NoCaps, OSCAR+ is trained on COCO without pre-training initialization or additional image-text pairs, and an Open Images-trained detector generates tags.This setup follows the task’s restriction against using training data beyond COCO.Source: §C.4 · p. 23

C.5 Image-Text Retrieval

  • Image-text retrieval is formulated as binary classification of aligned versus randomly mismatched image-text pairs, using [CLS] to score alignment and rank candidates.The section reports top-K retrieval on the 1K and 5K COCO test sets using the Karpathy split.Source: §C.5 · p. 23

C.6 NLVR2

  • For NLVR2, the model encodes the statement with each image separately, concatenates the two [CLS] outputs, and feeds them to an MLP binary classifier.Source: §C.6 · p. 24

D More on the Effect of the Object-Attribute Vocabulary Size: disentangling the effects of region proposals and model weights

  • Both region proposals and model weights matter for VL tasks, according to synthetic experiments that combine proposals from one vision model with weights from another.Figure 6 reports VQA accuracy across vocabulary and feature-extraction comparisons.Source: D More on the Effect of the… · p. 24, Figure 6 · p. 25

D.1 Disentangling the effects of region proposals and model weights on R50-C4 · D.2 Disentangling the effects of region proposals and model weights on the SoTA model · E More on FPN and Comparison of C4 and FPN

The analyses show that pretrained model weights and visually rich region semantics are important for VL performance; attribute information, broader object vocabularies, and vision pretraining also improve features, while grid features incur a speed and memory tradeoff.

D.1 Disentangling the effects of region proposals and model weights on R50-C4

  • For the R50 backbone, vision pretraining on 4Sets→VG:O1594A524 yields better region and grid features than the other tested training datasets.The passage reports that this setting outperforms all other columns in the comparison.Source: §D.1 · p. 25
  • Features trained with VG:O1594A524 attributes perform significantly better than features trained without attributes.Source: §D.1 · p. 25
  • VQA performance improves with richer object vocabularies in the reported ordering OI:500 ≈VG-obj:O317 < ImageNet:O1000 < VG:O1594.Source: §D.1 · p. 25
  • Regions proposed by OI and VG-obj models perform significantly worse than other regions and even worse than grid features.Source: §D.1 · p. 25
  • Grid-273 features perform worse than features from VG-proposed regions, and training Grid-273 models takes three times longer.The passage notes that closing the performance gap may be possible at the cost of greater hardware memory and computation in cross-modal models.Source: §D.1 · p. 26
  • Mean evaluation curves across training trajectories further confirm the conclusions drawn in Section 5.2.The curves are presented as more robust evidence for the ranking than a single final number.Source: §D.1 · p. 26

D.2 Disentangling the effects of region proposals and model weights on the SoTA model

  • Using R101-C4 boxes with X152-C4 features improves VQA more than using X152-C4 boxes with R101-C4 features, indicating pretrained weights matter more than regions.This analysis motivates class-agnostic NMS, which the passage says does not sacrifice VQA performance and improves inference speed.Source: Table 18 · p. 26, §D.2 · p. 26
  • COCO ground-truth boxes perform much worse on VQA than proposals from VG-trained models, highlighting the need for richer visual semantics in VL.The passage contrasts VL’s semantic alignment needs with typical object-detection tasks.Source: Table 18 · p. 26, §D.2 · p. 26

E.1 Two reasons why FPN performs worse than C4 on VL tasks. · E.2 Effect of pooling methods in FPN on VQA performance.

The analysis attributes FPN’s weaker VL performance than C4 to pre-training and box-head architecture, not to its multi-scale pooling method. Different pooling methods show no obvious performance gap, while mismatched feature dimensions hurt performance.

E.1 Two reasons why FPN performs worse than C4 on VL tasks.

  • C4’s feature-extraction layers are ImageNet-pretrained, unlike FPN’s MLP head, and the authors find VG too small to train strong VL features alone.The analysis notes that ImageNet-pretrained weights benefit C4.Source: §E.1 · p. 26
  • After four-dataset vision pre-training, C4 and FPN perform similarly (68.3 vs. 68.2), while randomly initializing C4’s box head makes its performance match FPN.Source: §E.1 · p. 26
  • Experiments confirm that FPN does not provide better region features than C4 for VL tasks.The comparison is reported for the R50C4 and R50FPN columns on VQA.Source: §E.1 · p. 26, Table 19 · p. 27
  • Random-initialized C4 features outperform FPN and nearly match ImageNet-pretrained C4, supporting a stronger convolutional-head inductive bias; random FPN approaches no-image performance.Source: §E.1 · p. 27

E.2 Effect of pooling methods in FPN on VQA performance.

  • Across eight vision models and four pooling methods, adapt and concat perform slightly better than max and avg, with no obvious overall pooling-method difference.The study evaluates all combinations of the vision models and pooling methods on VQA.Source: §E.2 · p. 27
  • Performance is significantly worse when FPN model training and VL feature extraction use mismatched concat versus non-concat dimensions, but pooling is not the root cause of FPN’s weaker VQA performance.Source: §E.2 · p. 27

E.3 Large-scale object-detection pre-training of C4 and FPN models · F Grid feature · G End-to-end inference efficiency

R152-C4 outperformed R152-FPN across all four validation datasets, while R50-FPN’s results varied by dataset. The sections also report that vision pre-training improved both grid- and region-feature VQA models and compare their inference efficiency.

E.3 Large-scale object-detection pre-training of C4 and FPN models

  • The study trained R50-C4, R50-FPN, R152-C4, and R152-FPN on merged object-detection datasets and compared checkpoint mAP 50 across four validation sets.The sets were COCO with stuff, Objects365, OpenImages, and Visual Genome.Source: §E.3 · p. 27, Figure 8 · p. 28
  • Across four validation datasets, R152-FPN underperformed R152-C4 throughout, while R50-FPN beat C4 on COCO and Objects365 but lost on Visual Genome, leading the authors to use R152-C4 downstream.Figure 8 reports mAP 50 on COCO with stuff, Objects365, OpenImages, and Visual Genome.Source: Figure 8 · p. 28, §E.3 · pp. 28–29
  • The other run failed, leaving no standard deviation for that experiment.Source: §E.3 · p. 29

F Grid feature

  • X152 vision pre-training from 4Sets→VG improved VQA performance for both grid- and region-feature models, with the larger backbone benefiting more than R50.The models used vision pre-training initialized from the same ImageNet-5k checkpoint; region features used boxes proposed by the best X152-C4 model.Source: F Grid feature · p. 29, Table 20 · p. 29
  • For grid-feature models, ImageNet-5k outperformed 4Sets and VG with Attr, unlike region-feature models, suggesting that vision pre-training effects differ by feature type.The passage notes that training the vision model grid-feature-wise or region-feature-wise may substantially affect downstream VL tasks.Source: F Grid feature · p. 29

G End-to-end inference efficiency

  • On GPU, grid-feature VQA models were faster than region-feature models, and the proposed fast inference brought region-feature inference to within 3× of grid-feature models.The inference comparison used batch size 1 and 800 × 1333 input images.Source: G End-to-end inference efficiency · p. 29
  • Efficient X152-C4 region-feature extraction preserved SOTA performance and improved GPU efficiency, but saved no single-thread CPU time because backbone and C4 costs dominated NMS.CPU evaluation used one thread; Table 21 reports the end-to-end VQA inference-time comparison.Source: G End-to-end inference efficiency · p. 29, Table 21 · p. 30

Related papers

Loading 2101.00529v2…