Investigating Prompting Techniques for Zero- and Few-Shot Visual Question Answering

Background

• VQA requires models to understand visual and textual inputs, while prompting can draw on pretrained VLM knowledge without task-specific fine-tuning [9] [2] [25] [19].Prompting ranges from task descriptions in zero-shot settings to image-question-answer examples in few-shot settings.Source: §1 · p. 1

Research gap

• Prior work had not systematically compared prompting techniques for zero- and few-shot generative VLMs, leaving their relative effectiveness unclear.The paper frames this as a gap in knowledge about which techniques are more effective.Source: §1 · pp. 1–2

Study contributions

• The study investigates question templates, caption cues, chain-of-thought reasoning, text-only few-shot examples, and LLM-based pre-processing for generative VQA responses.The pre-processing step adapts verbose responses to conventional string-matching VQA metrics.Source: §1 · p. 2

Study scope

• Evaluations span four state-of-the-art VLMs and six benchmarks, including compositional and knowledge-based tasks and a VQA adaptation of Winoground.The evaluated models include BLIP2, LLaVa, OpenFlamingo, and Kosmos2.Source: §1 · p. 2

Key findings

• Across four VLMs and six benchmarks, template preferences differ by model, captions improve performance variably, and self-consistency offers a route to better CoT results.Text-only few-shot examples improve task-format alignment, but their benefits diminish with LLM-based pre-processing; Winoground-VQA remains challenging for most VLMs, and pre-processing improves evaluation reliability.Source: §1 · pp. 2–3

VQA tasks and datasets

• VQA benchmarks cover diverse visual and reasoning demands, including compositional reasoning in GQA and external-knowledge questions in OK-VQA and AOKVQA [14] [27] [31].VQAv2 includes diverse questions about images, while CLEVR focuses on synthetic visual reasoning [10] [16].Source: §2 · p. 3
• The paper evaluates complex reasoning settings and adapts Winoground into Winoground-VQA, a more controlled test of visio-linguistic compositionality.GQA targets compositional reasoning, while OK-VQA and AOKVQA test knowledge-based reasoning.Source: §2 · p. 3

Prompting in LLMs

• Prior NLP work uses instructions or labeled examples for prompting and finds chain-of-thought prompts useful for reasoning [18] [17] [39].The paper investigates whether such prompting techniques transfer to multimodal VQA.Source: §2 · p. 3

Multimodal prompting

• Few-shot in-context learning and zero-shot prompting have been explored in multimodal models, but the effectiveness of newer VLMs in zero-shot applications remains underexplored [36] [2] [25] [19] [20] [43] [29].Flamingo’s reliance on interleaved image-text pretraining poses data-curation challenges, while MAPL’s limited training resources constrain its VQA performance relative to state-of-the-art methods.Source: §2 · p. 3
• Language-mediated VQA approaches convert queries into programs or images into text, but some rely on GPT-3 APIs or less capable LLMs [33] [12] [15] [13] [11].The paper extends language-mediated VQA to VLMs that use both text and images, and targets fine-tuning-free prompting because prior multimodal CoT gains relied on fine-tuning [41] [20] [23].Source: §2 · pp. 3–4

VQA evaluation

• Conventional string-matching VQA metrics struggle with verbose generative answers [1]; this work adds LLM-based pre-processing rather than replacing the metric with a fully LLM-based alternative [26].The approach aims to preserve conventional evaluation while making it compatible with generative outputs.Source: §2 · p. 4

Prompting Technique 1: Varying the Question Template

• Question-template prompting varies phrasing and response format across five Standard VQA templates: Null, follow-qa, qa, short-qa, and instruct-qa.Null and follow-qa depart from the “Question: Answer” format, short-qa requests concise answers, and instruct-qa gives task-specific directions.Source: §3 · p. 4

Prompting technique 2: Feed caption as additional input

• Caption VQA adds image captions to VLM inputs, using general or question-guided templates and captions from BLIP2, Kosmos-2, or PromptCap.The captioning strategies include dense, grounded, and question-guided descriptions.Source: §3 · pp. 4–5

Prompting technique 3: CoT reasoning

• CoT VQA prompts VLMs to produce both a rationale and an answer, and also tests self-consistency across multiple reasoning paths.The study uses two CoT templates and examines whether current VLM capabilities support reasoning on complex VQA tasks.Source: §3 · p. 5

Prompting technique 4: Providing text-only few-shot examples

• Text-only few-shot exemplars guide the desired answer format and can be combined with other prompting techniques; examples are selected from training data for each test question.The method avoids overly similar examples and is used because, apart from OpenFlamingo, the evaluated VLMs do not benefit from image-text few-shot examples.Source: §3 · p. 5

Mitigating VQA Metric Challenges Using LLM

• LLM-based pre-processing parses concise answers from verbose outputs before conventional string-matching VQA evaluation.The paper describes this as simpler and more accessible than a fully LLM-based metric, and reports that a publicly available 7B LLM can perform the parsing.Source: §3 · p. 5

Vision-language Models

• The evaluated models include four BLIP2 variants, standard and instruction-tuned OpenFlamingo, LLaVa with Vicuna, and Kosmos2.The models differ in language-model pairings, parameter sizes, training backgrounds, or pretraining data.Source: §4 · p. 5

Datasets

• Six datasets cover real-world, object-level, knowledge-based, commonsense, and compositional VQA, including Winoground-QA, a yes/no adaptation of Winoground.Winoground-QA asks two yes/no questions for each of two images based on image-caption matching.Source: §4 · pp. 5–6

Evaluation Metrics

• The evaluation includes open-ended and multiple-choice settings and uses VQA accuracy or binary accuracy according to the dataset and task.Generated answers are preprocessed with Zephyr-7B before string matching; Winoground-QA uses binary accuracy over four yes/no questions.Source: §4 · p. 6

5.1 Is VQA performance sensitive to the choice of the question template?

• Template choice shifted zero-shot accuracy by nearly 2 to 3% across models; BO and OF fell to 0% with non-standard templates, while Null consistently underperformed.Kosmos2 showed an average performance gap of approximately 5%, and larger BLIP2 models were less sensitive to template changes.Source: Figure 2 · p. 6, §5.1 · p. 6

5.2 Augmenting VLM’s context with image captions and LLM-only VQA results

• Adding captions improved zero-shot VQA in many settings, with PromptCap leading, but gains varied by model-dataset combination and were less pronounced on compositional GQA.On OKVQA, PromptCap improved BO (6.7B) accuracy by 4.54%; on AOKVQA, the improvement was 3.82%.Source: Table 2 · p. 7, §5.2 · pp. 7–8
• On AOKVQA, LLaVa accuracy fell from 51.22% with image features to 37.96% without; on GQA, PromptCap-augmented VLMs reached 40.32%, versus 36.46% for LLM-only PromptCap.Most tested models achieved near-zero Winoground-VQA accuracy, even LLaVa; Caption VQA slightly improved BLIP2.Source: §5.2 · p. 8

5.3 Do VLMs showcase CoT reasoning in VQA?

• Zero-shot CoT rationales underperformed standard VQA, with LLaVa producing lengthy, inconsistent rationales and hallucinations; 30-path self-consistency matched standard VQA performance.Reordering the rationale before the question slightly improved performance, while trimming rationales to one sentence was also tested.Source: Table 5 · p. 8, Table 4 · p. 8, §5.3 · p. 9

5.4 Do text-only few-shot exemplars help?

• Text-only exemplars improved task-format alignment most for verbose LLaVa and Kosmos2, especially with captions or CoT, while already-concise zero-shot BLIP2 gained little.OF instead received image-text exemplars and performed better across prompting settings.Source: §5.4 · p. 9

5.5 Does LLM pre-processing mitigate the challenges associated with the VQA metric?

• LLM-guided preprocessing corrected accuracy values for all tested models, especially addressing unusable LLaVa metrics and OPT misreporting.When preprocessing was applied, the few-shot performance gain diminished for all models except OF; concise answers from exemplars and answer-format alignment were also achievable through preprocessing.Source: Table 8 · p. 9, §5.5 · pp. 9–10

• Question templates, caption prefixes, and few-shot examples are highlighted as strategies for fine-tuning-free VQA.Source: §6 · p. 10
• CoT reasoning has mixed results, while self-consistency helps bridge the gap.Source: §6 · p. 10
• The study presents these strategies as practical techniques for using large pre-trained VLMs in VQA without fine-tuning.Source: §6 · p. 10

• Winoground statements are converted into yes/no questions for a binary VQA task, with manual verification and regeneration of questions that fail quality standards.Source: A Dataset: Winoground-VQA · p. 13
• Evaluation uses prompts based on the original Winoground text as well as prompts using the converted questions.The table is described as showing original statements alongside converted questions.Source: A Dataset: Winoground-VQA · p. 13, Table 9 · p. 13
• The original-text evaluation asks whether a statement describes the image, while the converted-question format asks a yes/no question about the same content.Source: A Dataset: Winoground-VQA · p. 13

B.1 Model Description

• BLIP2 combines frozen image encoders and language models, with a lightweight 12-layer Q-Former as its only trainable component.Source: §B.1 · p. 13
• Kosmos-2, LLaVa, and OpenFlamingo represent grounded image-text training, visual instruction tuning, and interleaved image-text processing, respectively.Source: §B.1 · p. 13

B.2 Caption Generation Strategies (with examples)

• Caption generation uses BLIP2 captions refined by Zephyr, grounded captions from Kosmos-2, and question-guided captions from PromptCap.The approach also describes the three captioning models as specialized for enhancing visual comprehension through text.Source: §B.2 · p. 14

B.3 Few-shot exemplars selection

• Few-shot selection retrieves five training exemplars by SBERT question embeddings and cosine similarity, using a 0.6 threshold to avoid overly similar examples.The authors report that high similarity can lead models to replicate in-context answers rather than produce unique test responses.Source: §B.3 · p. 14
• Few-shot exemplar content depends on the task: question-answer pairs for Standard VQA, captions and answers for Caption VQA, and rationales and answers for CoT VQA.Source: §B.3 · p. 14

B.4 Samples Illustrating VQA Metric Failure Modes

• Verbose answers can be correct yet fail string matching against ground-truth references, motivating LLM-guided parsing into reference-style answers.Table 10 is described as showing examples of these mismatches.Source: §B.4 · p. 14, Table 10 · p. 14

B.5 Inference details

• Answer generation uses beam search with beam size 3; maximum output length is 10 tokens for BLIP2 and 50 for verbose models like Kosmos2.Captions and rationales instead use a 128-token maximum and length penalty 1.Source: §B.5 · p. 15

C.1 Example prompt template: LLM-guided answer parsing

• The answer-parsing prompt maps a question and long answer to a short answer of at most one to three words or a short phrase.Its examples demonstrate extracting concise responses such as a sport, location, or object name.Source: §C.1 · p. 15

C.2 Example prompt template: Few-shot exemplars

• The AOKVQA few-shot prompt places unrelated caption-question-answer examples before the image-specific question, using them as context for the response.Source: §C.2 · pp. 15–16

D.1 Analysis on Quality of Generated CoT rationales

• LLaVA is prone to producing longer rationales than human-authored ground truths, potentially reflecting training on detailed narrative datasets.The analysis compares BF (XXL) and LLaVA explanations on AOKVQA.Source: §D.1 · p. 16
• The examples include object hallucinations, incorrect object associations, irrelevant details, and language-prior references such as Federer in an explanation about John McEnroe.The paper says these issues collectively hinder the effectiveness of models in CoT tasks.Source: §D.1 · pp. 16–17
• Automated rationale-quality scores use Rouge-1, Rouge-L, and BERTScore, but may not fully capture the quality of generated explanations.The paper calls for more nuanced analysis, including comparison of lengthy model explanations with shorter human-written ones.Source: §D.1 · p. 17

D.2 Qualitative Samples: Model predictions across different prompt settings

• Each illustrated image is paired with a question, ground-truth response, generated caption, and predictions from the three prompt settings.Source: §D.2 · p. 17
• Caption-VQA gives more accurate responses than Standard-VQA in some examples, while caption-only information can be insufficient when OCR is needed.The cited examples are (c), (d), and (e), respectively.Source: §D.2 · p. 17
• CoT-VQA examples include hallucinations and lengthy rationales, and the section reports lower performance than Standard-VQA.The VQA accuracy metric may also penalize answers that are correct but do not exactly match the ground truth.Source: §D.2 · p. 17