PaperlayerView exact revision on arXiv

Source-linked AI summary

Plug-and-Play VQA: Zero-shot VQA by Conjoining Large Pretrained Models with Zero Training

Anthony Meng Huat Tiong, Junnan Li, Boyang Li, Silvio Savarese, Steven C. H. Hoi

arXiv:2210.08773v3cs.CV

TL;DR

Zero-shot VQA remains challenging, while many approaches adapt pretrained language models with vision-specific components or training. PNP-VQA connects pretrained models using question-guided captions and network interpretation, without additional training. It reports state-of-the-art results on multiple benchmarks, including improvements over Flamingo80B on VQAv2 and FewVLMlarge on GQA.

  • Problem

    Zero-shot VQA is challenging, and most methods incorporating vision into pretrained language models require substantial adaptation, new components, or training.

    Source: §1 · p. 1

  • Method

    PNP-VQA uses network interpretation to select question-relevant image patches, generates stochastic question-guided captions, and passes them with the question to a pretrained model for answering, without additional training.

    Source: §1 · p. 1, §3 · p. 3

  • Results

    PNP-VQA reports state-of-the-art zero-shot VQA results on multiple benchmarks, improving over Flamingo80B by 8.5% on VQAv2 and FewVLMlarge by 9.1% on GQA.

    Source: §1 · p. 2

  • Takeaways & Limitations

    The results demonstrate that pretrained models can be conjoined for zero-shot VQA without additional training.

    Source: §6 · p. 9

    The modular zero-training design inherits pretrained-model biases and adds inference cost through its multi-step process.

    Source: §7 · p. 9

Abstract

from arXiv · show

Visual question answering (VQA) is a hallmark of vision and language reasoning and a challenging task under the zero-shot setting. We propose Plug-and-Play VQA (PNP-VQA), a modular framework for zero-shot VQA. In contrast to most existing works, which require substantial adaptation of pretrained language models (PLMs) for the vision modality, PNP-VQA requires no additional training of the PLMs. Instead, we propose to use natural language and network interpretation as an intermediate representation that glues pretrained models together. We first generate question-guided informative image captions, and pass the captions to a PLM as context for question answering. Surpassing end-to-end trained baselines, PNP-VQA achieves state-of-the-art results on zero-shot VQAv2 and GQA. With 11B parameters, it outperforms the 80B-parameter Flamingo model by 8.5% on VQAv2. With 738M PLM parameters, PNP-VQA achieves an improvement of 9.1% on GQA over FewVLM with 740M PLM parameters. Code is released at https://github.com/salesforce/LAVIS/tree/main/projects/pnp-vqa

1 Introduction

PNP-VQA combines pretrained models for zero-shot VQA without additional training, using question-guided captions and network interpretation to connect vision and language. It reports state-of-the-art results on multiple benchmarks, including improvements over Flamingo80B on VQAv2 and FewVLMlarge on GQA.

1 Introduction

  • Most methods adapting pretrained language models for vision-language tasks require new components or training, while recombining pretrained models without changes or extra training is desirable but challenging.The cited examples include layers trained from scratch, trained vision encoders, and vision-language pretraining objectives.Source: §1 · p. 1
  • PNP-VQA connects pretrained vision-language and language models through question-relevant image patches, stochastic captions, and question answering from those captions, with zero additional training.The framework uses a pretrained vision-language model for captions and a pretrained language model to answer from them.Source: §1 · p. 1
  • Network interpretation and natural language jointly serve as the interface: question-relevant patches guide captions intended to cover information useful for accurate question answering.The framework is modular and can evolve as pretrained models advance.Source: §1 · p. 2
  • 8.5% improvement over Flamingo80B on VQAv2 and 9.1% over FewVLMlarge on GQA establish PNP-VQA's state-of-the-art zero-shot results across multiple benchmarks.The comparison is reported for PNP-VQA11B on VQAv2 and PNP-VQAlarge on GQA.Source: §1 · p. 2

2 Related Work

Prior zero-shot VQA approaches commonly train or adapt models to incorporate vision, while PNP-VQA uses pretrained models without architectural modifications or additional training. Related approaches differ in how they represent images and connect models.

2 Related Work

  • Zero-shot VQA remains challenging because it requires substantial reasoning ability, despite progress in large-scale image-text pretraining.Image-text pretraining has supported zero-shot image-text retrieval and image captioning.Source: §2 · p. 2
  • Many zero-shot VQA methods add vision-language training, train vision encoders or inserted layers, or adapt language models using multimodal objectives.Examples include Frozen, FewVLM, VLKD, and Flamingo.Source: §2 · p. 2
  • PNP-VQA instead directly uses pretrained models without architectural modifications or additional training.This distinguishes its approach from the adaptation methods described in the related work.Source: §2 · p. 2
  • Unlike PICa's single image caption, PNP-VQA generates multiple question-guided captions and fuses them after encoding to use many captions.The passage reports that this design yields considerable performance gains.Source: §2 · p. 2
  • The system chains image-question matching, diverse caption generation, and question answering, using BLIP for the first two modules and UnifiedQAv2 for the last.Image-question matching identifies patches relevant to the question; the answerer uses the question and captions.Source: Figure 1 · p. 3
  • Natural language interfaces connect models or reasoning steps, and PNP-VQA combines language with network interpretation as its interface between pretrained models.The cited work by Zeng et al. (2022a) evaluates captioning and retrieval, but not VQA.Source: §2 · pp. 2–3

3 Method

PNP-VQA connects pretrained vision-language and language models without training by turning question-relevant image regions into diverse captions for question answering. It selects patches using question-conditioned network interpretation, then combines caption evidence with Fusion-in-Decoder.

3.1 Matching Image Patches and Questions

  • PNP-VQA identifies question-relevant patches with gradient-weighted cross-attention, then samples patches in proportion to their relevance for captioning.The method uses a GradCAM variation to filter attention scores, aiming to retain scores that contribute to image-text similarity.Source: §3.1 · pp. 3–4
  • 100 question-guided captions outperform five MS COCO human-written captions by 5.2% on VQAv2 and 6.0% on OK-VQA.Question-guided patch sampling substantially outperforms generic all-patch captioning and random sampling, especially with many captions.Source: §3.1 · p. 5, Table 1 · p. 5

3.2 Informative Image Captioning

  • Stochastic top-k caption generation from relevant patches produces diverse descriptions intended to cover possible answers and visual content.The method generates multiple captions and discards a caption when it is subsumed as an exact substring by an earlier one.Source: §3.2 · p. 5

3.3 Answering the Question

  • FiD encodes each caption alongside the question separately, then supplies the combined representations to the decoder’s cross-attention.Unlike FiE’s concatenated long encoder input, FiD avoids the encoder’s maximum-input-length constraint and is more efficient for many captions.Source: §3.3 · pp. 5–6, Figure 3 · p. 5
  • FiE performance caps at around 40 captions when the encoder’s maximum input length is exceeded, while FiD performance continues to rise.Both methods initially improve as the number of captions increases.Source: §3.3 · p. 6, Figure 4 · p. 6

4 Experiments

The experiments evaluate zero-shot VQA across VQAv2, OK-VQA, and GQA, using open-ended answer generation and dataset-specific accuracy measures. PNP-VQA outperforms strong baselines on VQAv2 and GQA, while Flamingo performs better on OK-VQA.

4.1 Datasets and Evaluation

  • The evaluation covers VQAv2 validation and test-dev, OK-VQA test, and GQA-balanced test-dev, using open-ended generation and exact matching.The study reports soft accuracy for VQAv2 and OK-VQA and standard accuracy for GQA.Source: §4.1 · p. 6

4.2 Implementation Details

  • To avoid overlap with VQA evaluation data, BLIP captioning and image-text matching models are finetuned on COCO 2014 rather than the overlapping COCO 2017 split.The question-answering model is trained on text-only QA data and is unaware of vision during training.Source: §4.2 · p. 6
  • The setup uses 100 captions per question, with question-caption encoding grouped in fives for GQA and one caption at a time for VQAv2 and OK-VQA.The authors motivate the larger GQA context by its compositional visual reasoning demands.Source: §4.2 · p. 6

4.3 Comparison with State of the Arts

  • PNP-VQA11B outperforms Flamingo80B by 8.5% on VQAv2 test-dev, and PNP-VQAlarge outperforms FewVLMlarge by 9.1% on GQA.Flamingo is the only method reported to perform better than PNP-VQA on OK-VQA; PNP-VQA still exceeds baselines not trained on Flamingo’s gigantic data.Source: §4.3 · p. 7, Table 2 · p. 7
  • On OK-VQA, PNP-VQA11B achieves an 18.2% improvement over language-conjoined PICa (175B parameters); on VQAv2 test-dev, PNP-VQA11B outperforms second-best Flamingo80B by 8.5%.On VQAv2, PNP-VQA3B outperforms Flamingo80B by 7.2% and similarly sized Flamingo3B by 14.3%.Source: §4.3 · p. 7

5 Analysis

The analysis examines how caption information, decoding choices, and the textual QA model relate to PNP-VQA performance. Question-guided captions have the highest answer hit rate, while decoding and QA-model choice also distinguish performance.

5.1 Are PNP-VQA captions informative?

  • AHR excludes yes/no questions because their answers are contextual and the words “yes” and “no” rarely appear in captions.AHR is the proportion of questions for which at least one caption contains the ground-truth answer verbatim.Source: §5.1 · pp. 7–8
  • Within each sampling method, VQA accuracy increases with AHR, consistent with the authors’ hypothesis that answer-containing captions facilitate correct answers.Figure 5(a) presents the positive relationship between AHR and VQA accuracy.Source: §5.1 · p. 8, Figure 5 · p. 8
  • AHR does not capture answer position or occurrence count, so its correlation with answer accuracy is imperfect despite being a useful caption-quality measure.The authors describe AHR as an easy-to-compute measure of caption information quality.Source: §5.1 · p. 8
  • 71.8% to 84.0%: question-guided AHR rises as captions per question increase from 20 to 100, and this method has the highest AHR.The authors attribute some of PNP-VQA’s performance to informative captions that directly contain the correct answer.Source: §5.1 · p. 8, Figure 5 · p. 8

5.2 How sensitive is PNP-VQA to the caption decoding method?

  • Stochastic caption decoding methods perform similarly, while beam search produces a noticeable performance drop and repetitive captions that cover fewer image aspects.Table 3 reports the ablation on caption decoding methods.Source: §5.2 · p. 8, Table 3 · p. 8

5.3 Can PNP-VQA work with other textual QA models?

  • UnifiedQAv2 performs better on VQA tasks than T0 and GPT-J; the authors attribute this to its task-specific question-answering pretraining.The result indicates that textual QA model choice matters in zero-shot VQA without training.Source: Table 4 · p. 8, §5.3 · p. 9

6 Conclusion

PNP-VQA combines pretrained models for zero-shot VQA without additional training. Its image-question matching, question-guided captioning, and question-answering modules produce state-of-the-art performance on multiple VQA benchmarks.

6 Conclusion

  • PNP-VQA uses image-question matching to select relevant patches, generates question-guided captions from them, and feeds those captions to a question-answering module.The framework combines off-the-shelf pretrained models without additional training.Source: §6 · p. 9
  • PNP-VQA achieves state-of-the-art performance on multiple VQA benchmarks.Source: §6 · p. 9

7 Limitations

The authors identify inherited pretrained-model bias and added inference cost from PNP-VQA’s multi-step process as limitations of its zero-training modular design.

7 Limitations

  • PNP-VQA inherits bias from its pretrained models.Source: §7 · p. 9
  • Its multi-step process introduces additional inference cost despite the efficiency of zero training.Source: §7 · p. 9
  • The authors frame both limitations as consequences of the zero-training modular system design.Source: §7 · p. 9

A Visualization

The appendix includes visualizations of GradCAM heatmaps and generated captions for VQAv2, OK-VQA, and GQA.

A Visualization

B Hyperparameter sensitivity

GradCAM layer choice shows no clear relationship with VQAv2 accuracy, while sampling more image patches correlates with lower accuracy as the patches become less question-relevant.

B Hyperparameter sensitivity

Related papers

Loading 2210.08773v3…