Modular Visual Question Answering via Code Generation

• VQA spans skills from visual grounding to spatial and knowledge-based reasoning, making it costly to train for every combination of skills.Modular neural networks still require joint training and a parser that must be modified when modules change; replacing a module requires retraining.Source: §1 · p. 1
• CodeVQA uses generated Python programs to compose visual APIs and reason over their outputs with arithmetic and conditional logic.Its modules can probe for captions, object locations, or image-text similarity; Figure 1 illustrates a question combining a visual query and spatial comparison.Source: §1 · p. 1, Figure 1 · p. 2
• CodeVQA outperforms previous approaches by at least 3% on COVR and roughly 2% on GQA.The evaluation is in the few-shot VQA setting.Source: §1 · p. 1

• CodeVQA is compared against a strong LM-based few-shot baseline using the same in-context example selection to isolate program synthesis effects.The authors note that concurrent program-synthesis frameworks combine synthesis with the effects of the LM, examples, and visual primitives.Source: §2 · p. 2
• Related systems apply LM-written programs to robotics and reasoning over tables, text, and caption-represented images (Liang et al., 2022; Cheng et al., 2023).Source: §2 · p. 2
• CodeVQA obtains performance comparable to supervised-model approaches on GQA using an LM and models pre-trained on image-text pairs.The related-work discussion contrasts this with frameworks relying on supervised VQA or object-detection models.Source: §2 · p. 2

3 Few-shot VQA via Code Generation

• In few-shot VQA, CodeVQA generates a Python program from the question, then executes it with the image as needed to predict an answer.The setting provides 50 human-annotated VQA instances.Source: §3 · p. 2

3.1 Code Primitives

• Three primitives answer image questions, retrieve text-conditioned locations, and select the most text-relevant image using ITC.For query, GradCAM selects image patches for captioning before an LM predicts an answer; get_pos returns the (x, y) location maximizing GradCAM.Source: §3.1 · pp. 2–3

3.2 Code generation

• The code-generation prompt supplies Python API documentation and retrieved, semantically similar annotated programs as in-context examples.Python supports arithmetic, loops, and conditionals; if execution fails, the system falls back to querying the image with the original question.Source: §3.2 · p. 3

3.3 Component models

• The implementation uses a code-generation model, ITM and ITC models, an image-captioning model, and an LM for answering from captions.The specified code-davinci-002 model serves for program generation and caption-based question answering; BLIP models provide ITM, ITC, and captioning.Source: §3.3 · p. 3

4.4 Results

• CodeVQA has the highest accuracy among few-shot techniques and markedly outperforms Few-shot PnP-VQA on COVR, with its greatest advantage on 4- or 5-image instances.Table 1 compares results across GQA, COVR, and NLVR2; Table 3 breaks COVR validation accuracy down by image count.Source: Table 1 · p. 4, §4.4 · p. 4, Table 3 · p. 5

4.2 Datasets

• Experiments use 50 randomly sampled training questions per dataset, 12 in-context examples for single-image tasks and 6 for multi-image tasks, and exact-match accuracy on lower-cased answers.Implementation details are deferred to Appendix C; GQA, COVR, and NLVR2 have distinct image or statement settings.Source: §4.2 · pp. 3–4, §4.1 · p. 3

4.3 Baseline

• The baseline adapts PnP-VQA to few-shot prompting and runs the five-step query procedure for every question.Source: §4.3 · p. 4

4.6 Analysis

• In a COVR example, CodeVQA answers correctly by answering a simpler question for each image and comparing the answers, while Few-shot PnP-VQA fails despite captions containing the needed information.Source: §4.4 · p. 4, Figure 2 · p. 5

4.5 Ablations

• Embedding-based retrieval yields a greater CodeVQA improvement over Few-shot PnP-VQA than random retrieval does.The authors describe embedding retrieval as a systematic way to find relevant examples rather than curate one fixed set.Source: §4.5 · p. 4, Table 2 · p. 5
• CodeVQA improves over Few-shot PnP-VQA with either code-davinci-002 or text-davinci-003 as the question-answering LM, while accuracy stays roughly constant as shot count varies.Source: §4.5 · p. 4

4.6 Analysis

• On GQA, CodeVQA’s greatest improvement is roughly 30% for spatial questions, while text-davinci-003 closes its gap with Few-shot PnP-VQA on “and” questions.The figure also reports CodeVQA outperforming the baseline on “and” and “or” questions.Source: §4.6 · pp. 4–5, Figure 3 · p. 5
• Among 100 incorrect COVR validation answers, error labels overlap, with 13 examples receiving multiple categories; the authors find more errors from module execution than program generation.Irrelevant captions were marked in 31%, question-answering errors in 25%, and program-generation errors in 14%.Source: §4.6 · p. 5

• The framework prompts a language model to generate Python programs that invoke pre-trained visual modules and compose their outputs to predict answers.Source: §5 · p. 5
• Unlike previous modular VQA techniques, the framework requires neither module retraining nor a parser, and its frozen modules are interpretable.Source: §5 · p. 5
• The authors frame CodeVQA as extending pipelined systems to the full expression of code, with empirical gains motivating future work on modular few-shot VQA.Source: §5 · p. 5

6 Limitations

• CodeVQA remains below human VQA accuracy and models fine-tuned on VQA datasets; practical use may require further progress, and capable LMs bring compute or API costs.The evaluation focuses on English as the primary language; extending it to other languages is left for future work.Source: §6 · pp. 5–6

A.1 GQA

• GQA prompts use 12 in-context examples and provide coordinate constants and visual APIs in a preamble, with a bench-appearance query as an example.The displayed example invokes image opening, image querying, and position-finding functions.Source: §A.1 · p. 8

A.2 COVR

• The COVR example opens an image set, loops over its images, accumulates ladies' and men's black-shirt counts, then compares the totals for a yes/no answer.The prompt uses six in-context examples per evaluation example.Source: §A.2 · pp. 8–9

B GradCAM

• GradCAM maps use cross-attention and score gradients; query averages maps across question tokens, while get_pos averages across input-text tokens.The described GradCAM computation uses layer 6.Source: B GradCAM · p. 9

C Implementation Details

• GQA uses 7 captions per image and 12 selected examples; COVR uses 3 captions per image and 6 examples, while NLVR2 uses 7 captions per image and 6 examples.Examples are selected using question embeddings; statement inputs are converted into yes/no questions, and each reported accuracy is a single evaluation run.Source: C Implementation Details · p. 9

D Details on Baselines

• FewVLM samples 16 GQA examples; VisProg samples 24 for GQA and 16 for NLVR2, while ViperGPT uses 8 for GQA.VisProg uses text-davinci-002 or text-davinci-003 for code generation; ViperGPT uses code-davinci-002.Source: D Details on Baselines · p. 9

E Qualitative Comparisons

• Few-shot PnP-VQA captions are described as irrelevant to the question in the examples, while CodeVQA uses Python conditionals and visual primitives to focus on relevant regions and answer correctly.The primitives named are get_pos(image, text) and query(image, text).Source: E Qualitative Comparisons · pp. 9–10
• The COVR prompt's caption count was chosen as the maximum possible under the shot count and the davinci model's context-size constraint.The choice was based on preliminary OK-VQA experiments.Source: E Qualitative Comparisons · p. 9
• In 2,000 GQA validation examples, CodeVQA outperforms Few-shot PnP-VQA on spatial and or questions; spatial questions concern relations or object positions.Figure 4 encodes CodeVQA in blue and Few-shot PnP-VQA in orange.Source: Figure 4 · p. 10
• NLVR-2 examples show CodeVQA counting pandas across images in one case and using three simpler queries with an if-else statement in another.Both examples are reported as answered correctly.Source: E Qualitative Comparisons · p. 10
• A COVR multireference example counts images meeting two separate plate-and-food criteria, then checks whether the counts match.Another example uses for loops and nested conditional logic to locate images satisfying two pillow-and-furniture criteria and count occurrences.Source: E Qualitative Comparisons · p. 10

F Additional Quantitative Results

• Table 4 reports validation-set accuracies for CodeVQA and Few-shot PnP-VQA, comparing results when code-davinci-002 or text-davinci-003 is used as the question-answering LM.Source: F Additional Quantitative Results · p. 10
• Table 5 reports CodeVQA and Few-shot PnP-VQA accuracy at different prompt shot counts on 2,000 GQA validation examples.The supplied text describes the comparison but gives no accuracy values or trend.Source: F Additional Quantitative Results · p. 10, Table 5 · p. 10
• On the 2,000-example GQA validation sample, Few-shot PnP-VQA has an advantage on and questions.Figure 4 breaks down accuracy by question type.Source: F Additional Quantitative Results · p. 10
• Figure 5 presents GQA examples in which CodeVQA outperforms Few-shot PnP-VQA.Source: Figure 5 · p. 11

G Experiments with Additional Primitives

• Using find_object for counting produces mixed results on VQAv2, COVR, and NLVR2.The evaluation uses Grounding DINO; Table 6 covers a VQAv2 validation sample, COVR validation, and NLVR2 test-public.Source: G Experiments with Additional Primitives · p. 11
• Counting errors arise from detector mistakes, omitted question details, or invoking the detector for unsuitable objects such as blades of grass.Captions can convey counts when object numbers are small, so query can be effective for counting in these datasets.Source: G Experiments with Additional Primitives · p. 11
• Breaking out visual and general-knowledge questions does not improve accuracy on OK-VQA questions that require both.The comparison uses knowledge_query and query; Table 7 reports results on the OK-VQA validation set.Source: G Experiments with Additional Primitives · p. 12

H Licenses and Other Dataset Details

• GQA, COVR, NLVR2, and VQAv2 annotations are described as CC-BY-4.0 licensed, while COVR is MIT-licensed and NLVR2 images are unlicensed.The passage notes that imSitu images may not be licensed.Source: H Licenses and Other Dataset Details · pp. 12, 14
• Appendix examples show programs counting images that meet visual conditions and comparing counts across image sets.One example counts images with girls wearing skirts and holding rackets; another compares images containing cake or lemon on white plates.Source: H Licenses and Other Dataset Details · p. 13
• The listed evaluation sets range from 4,000 sampled VQAv2 validation examples to 12,578 GQA testdev instances.The passage also lists COVR validation and test, NLVR2 public test, and OK-VQA validation sizes.Source: H Licenses and Other Dataset Details · p. 14

I Ethics and Impact Statement

• CodeVQA may reduce retraining needs, but its high inference cost and environmental impact should factor into deployment decisions.The passage identifies use of large language models as the source of high inference cost.Source: I Ethics and Impact Statement · p. 14
• Generated programs can help Python-familiar readers inspect which visual tasks are used and how their outputs combine into an answer.Source: I Ethics and Impact Statement · p. 14
• Because CodeVQA relies on pre-trained vision-language models, it may exhibit social biases reported in image-caption-trained models.The paper advises practitioners to consider mitigation when deploying the system in ways that can affect human lives.Source: I Ethics and Impact Statement · p. 14