PaperlayerView exact revision on arXiv

Source-linked AI summary

Visual Programming: Compositional visual reasoning without training

Tanmay Gupta, Aniruddha Kembhavi

arXiv:2211.11559v1cs.CVcs.AIcs.CL

TL;DR

Complex visual tasks often require composing multiple specialized capabilities beyond the scope of a single end-to-end model. VISPROG uses a large language model’s in-context learning to generate and execute modular visual programs without task-specific gradient-based training. Across four tasks, it reports a 2.7-point compositional-VQA gain, 62.4% zero-shot NLVR accuracy, and qualitative and quantitative results for knowledge tagging and image editing.

  • Problem

    End-to-end systems require curated task-specific datasets, making complex natural-language visual tasks such as multi-step knowledge tagging difficult to scale.

    Source: §1 · p. 2

  • Method

    VISPROG uses in-context examples to generate Python-like visual programs that execute specialized models and image-processing or Python routines in compositional steps.

    Source: §1 · p. 2

  • Results

    Across four tasks, VISPROG reports a 2.7-point gain over a base VQA model, 62.4% zero-shot NLVR accuracy, and qualitative and quantitative results for knowledge tagging and image editing.

    Source: §1 · p. 2

  • Takeaways & Limitations

    VISPROG provides a flexible neuro-symbolic system whose executed programs produce interpretable visual rationales for complex visual tasks.

    Source: §1 · p. 2, §6 · p. 9

    The paper defines “training” as gradient-based learning, distinguishing it from in-context learning that uses a feedforward pass.

    Source: §1 · p. 2

Abstract

from arXiv · show

We present VISPROG, a neuro-symbolic approach to solving complex and compositional visual tasks given natural language instructions. VISPROG avoids the need for any task-specific training. Instead, it uses the in-context learning ability of large language models to generate python-like modular programs, which are then executed to get both the solution and a comprehensive and interpretable rationale. Each line of the generated program may invoke one of several off-the-shelf computer vision models, image processing routines, or python functions to produce intermediate outputs that may be consumed by subsequent parts of the program. We demonstrate the flexibility of VISPROG on 4 diverse tasks - compositional visual question answering, zero-shot reasoning on image pairs, factual knowledge object tagging, and language-guided image editing. We believe neuro-symbolic approaches like VISPROG are an exciting avenue to easily and effectively expand the scope of AI systems to serve the long tail of complex tasks that people may wish to perform.

1. Introduction

VISPROG addresses the long tail of complex visual tasks by using a language model to generate executable visual programs from natural-language instructions, without task-specific gradient-based training. Its modular programs combine specialized models and Python routines, while intermediate outputs provide interpretable rationales; across four tasks, it reports gains on compositional VQA and zero-shot NLVR.

Motivation

  • The system targets complex tasks that end-to-end systems struggle to execute, such as detecting faces, retrieving factual knowledge, classifying identities, and tagging an image.Source: §1 · p. 2

Approach

  • VISPROG decomposes natural-language visual instructions into executable programs whose modular steps consume and produce intermediate results.Modules can include off-the-shelf vision or language models, OpenCV routines, and arithmetic or logical operators.Source: §1 · p. 2
  • VISPROG generates complex programs with GPT-3 and a small number of in-context examples without requiring gradient-based training.The paper distinguishes this in-context process from training that uses gradient-based learning.Source: §1 · p. 2

Interpretability

  • Executed programs expose intermediate text, bounding boxes, masks, and generated images that users can inspect, verify, and use for error diagnosis or intervention.Source: §1 · p. 2

Evaluation

  • VISPROG is evaluated on four tasks and reports a 2.7-point gain over a base VQA model and 62.4% zero-shot NLVR accuracy without training on image pairs.The other demonstrated tasks are factual knowledge object tagging and language-guided image editing.Source: §1 · p. 2

Contributions

  • The paper contributes a flexible visual-programming system, applications to difficult visual tasks, and visual rationales used for error analysis and user-driven instruction tuning.Source: §1 · p. 3

2. Related Work

Prior visual program-generation systems compose specialized modules, while language-model prompting has enabled reasoning through examples and intermediate steps. VISPROG differs by generating instance-specific compositions for visual inputs and combining diverse modules, modalities, and intermediate representations.

Program generation for visual tasks

  • Neural Module Networks compose specialized differentiable modules for VQA, using parsers or learned layout generation with weak answer supervision.Source: §2 · p. 3
  • VISPROG generates higher-level programs that invoke trained state-of-the-art models and non-neural Python subroutines.Source: §2 · p. 3

LLMs for visual tasks

  • PICa feeds textual image representations to GPT-3 for knowledge-based VQA, while Socratic Models compose pretrained models across modalities for zero-shot tasks.Source: §2 · p. 3
  • Unlike Socratic Models’ fixed task compositions, VISPROG generates a composition for each instance from the instruction, question, or statement.Source: §2 · p. 3
  • VISPROG handles complex instructions across 20 modules, varied text and image inputs, multiple intermediate representations, and text or image outputs.Source: §2 · p. 3

Reasoning via prompting

  • Prompting research in NLP uses in-context examples and intermediate reasoning steps, including Chain-of-Thought and decomposer-based subtask handling.Source: §2 · p. 3

3. Visual Programming

VISPROG addresses broad, loosely defined visual tasks by generating and executing modular visual programs from natural-language instructions without task-specific training. Its interpreter executes reusable modules step by step and assembles intermediate outputs into an interpretable visual rationale.

Motivation

  • VISPROG targets real-world visual tasks that are broader than the narrow problems solved by existing task-specific models.Manual program construction is slow, labor intensive, and insufficient for the long tail of complex tasks.Source: §3 · pp. 3–4

Program Generation

  • GPT-3 uses instruction-program examples in context to generate a visual program for a new instruction without observing the input image.Programs use descriptive module, argument, and variable names, with outputs from earlier steps serving as later inputs.Source: §3 · p. 4

Modules

  • Each VISPROG module can invoke trained neural models, image-processing routines, knowledge retrieval, or arithmetic and logical operations.The system supports 20 modules implemented as Python classes that parse, execute, update program state, and summarize computations.Source: §3 · p. 4

Program Execution

  • An interpreter initializes program state with the inputs, executes program lines sequentially, and stores each step’s output for subsequent modules.The state is a dictionary mapping variable names to values.Source: §3 · p. 5

Visual Rationale

  • VISPROG stitches HTML summaries of module inputs and outputs into visual rationales that support logical verification, intermediate-output inspection, error analysis, and instruction adjustment.These rationales depict the flow of information through the executed program.Source: §3 · p. 5, Figure 4 · p. 5

Framework Scope

  • The framework evaluates diverse inputs, outputs, and reusable modules across several visual tasks.Figure 5 highlights reuse of Loc, FaceDet, and VQA modules whenever possible.Source: Figure 5 · p. 5

4. Tasks

VISPROG is evaluated on four tasks spanning spatial reasoning, multiple-image reasoning, knowledge retrieval, and image generation or manipulation. The task designs decompose complex instructions into reusable visual and language operations, while performance increases with more in-context examples on GQA and NLVRV2.

Task Suite

  • VISPROG is evaluated on compositional VQA, zero-shot reasoning over image pairs, factual knowledge object tagging, and language-guided image editing.These tasks span spatial reasoning, multiple images, knowledge retrieval, and image generation or manipulation.Source: §4 · p. 5

Compositional VQA

  • On GQA, VISPROG decomposes multi-step questions into localization, cropping, simpler VQA queries, counting, and Python-expression evaluation.This decomposition makes the resulting system more interpretable and more accurate than ViLT according to the cited passage.Source: §4.1 · p. 5

Image-Pair Reasoning

  • On NLVRV2, VISPROG verifies image-pair statements by asking simpler questions about individual images and combining their answers with arithmetic and logical operators.This avoids training a custom image-pair architecture on the benchmark’s training set.Source: §4.2 · p. 6

Knowledge Tagging

  • Knowledge Tagging combines open-vocabulary localization or face detection, knowledge-base category generation, and CLIP classification of image regions.The retrieved category-list size can be controlled through the instruction, including a specified maximum or an automatically determined list capped at 20.Source: §4.3 · pp. 6–7

Image Editing

  • Image editing supports face manipulation, object highlighting through color popping or background blur, and scene-context changes such as replacing a desert.The task uses language-guided combinations of visual modules rather than relying solely on text-to-image generation.Source: §4.4 · p. 7

Prompting

  • Validation performance increases progressively with more in-context examples on both GQA and NLVRV2, while majority voting across random seeds consistently outperforms average performance across runs.Performance saturates with fewer prompts on NLVR than on GQA.Source: Figure 7 · p. 7

5. Experiments and Analysis

The experiments assess VISPROG across four tasks, prompting strategies, error sources, and whether visual rationales enable instruction tuning. Results show gains from more effective prompting and user-guided modifications, while errors remain tied to program generation and module quality.

Experimental setup

  • Experiments evaluate prompt scaling, prompting strategies, four-task generalization, error sources, and instruction tuning through visual rationales.The evaluated tasks include GQA, NLVR, knowledge tagging, and image editing.Source: §5 · p. 7

Prompting

  • Validation performance increases with more in-context examples for both GQA and NLVR, while NLVR saturates with fewer prompts.Majority voting across random seeds consistently outperforms average performance across runs.Source: §5.1 · p. 7
  • A curated 20-example prompt matches voting performance while using 5× less compute, whereas random prompts only slightly outperform VILT-VQA on GQA.Voting produces a significant gain of 2.7 points over the random prompting comparison.Source: §5.2 · pp. 7–8

Task results

  • VISPROG performs NLVR zero-shot without training on image pairs, using a single-image VQA model for visual understanding and an LLM for reasoning.VILT-NLVR, finetuned on NLVRV2, is reported as an upper bound, and VISPROG remains several points behind it.Source: §5.2 · p. 8, Table 2 · p. 8
  • Knowledge tagging reaches 63.7% F1 for tagging and 80.6% F1 for localization with original instructions, while image editing supports diverse manipulations.Knowledge tagging combines open-vocabulary localization with knowledge-base retrieval; editing includes face manipulation, object highlighting, and scene-context replacement.Source: §5.2 · p. 8, Table 3 · p. 8, Table 4 · p. 8

Error analysis

  • Incorrect programs affect 16% of GQA samples, while replacing high-error modules could improve NLVR by up to 24% and reduce errors in tagging and editing.The analysis identifies “List” and “Select” as major error sources for knowledge tagging and image editing.Source: §5.3 · pp. 8–9

Instruction tuning

  • Visual-rationale-guided instruction tuning improves knowledge tagging and image editing by refining localization, retrieval, selection, and classification instructions.Examples include more specific localization and retrieval queries, category restrictions, and controlling classification categories with the List module.Source: §5.3 · p. 9, Figure 9 · p. 9

6. Conclusion

VISPROG uses visual programming to apply LLM reasoning to complex visual tasks without task-specific training. Its strong performance and interpretable visual rationales motivate better prompting and user-feedback methods.

6. Conclusion

  • VISPROG brings LLM reasoning to complex visual tasks through visual programming without task-specific training.The conclusion describes this approach as simple and effective.Source: §6 · p. 9
  • The system demonstrates strong performance while generating highly interpretable visual rationales.The authors identify improved prompting and new ways to incorporate user feedback as future directions.Source: §6 · p. 9

A. Appendix

The appendix provides implementation-related materials and directs readers to additional qualitative visual-rationale examples.

Appendix

  • The appendix includes task prompts and module implementation details for VISPROG.These materials are listed as Sections A.1 and A.2.Source: A. Appendix · p. 12
  • Additional qualitative results cover successful and failure cases with visual rationales.The paper points readers to the VISPROG project webpage for these examples.Source: A. Appendix · p. 12

A.1. Task Prompts

The appendix shows prompt structures for each of VISPROG’s four tasks using three in-context examples.

A.1. Task Prompts

  • Prompt structures are provided for GQA, NLVR, knowledge tagging, and language-guided image editing.Each task prompt structure uses 3 in-context examples.Source: §A.1 · p. 12, Figure 10 · p. 12

A.2. Module Details

VISPROG’s modules implement specialized operations for selecting regions, classifying them against category lists, and retrieving factual lists. Prompt configurations expose defaults and argument values for flexible list retrieval and image-editing or tagging tasks.

Select

  • The Select module uses CLIP-ViT region-text scoring to choose the highest-scoring region for a query, optionally restricting selection by category.Comma-separated queries select one highest-scoring region per phrase.Source: §A.2 · p. 12

Prompt configuration

  • Prompt examples configure module arguments, including emoji-name mappings for Emoji, category values for Select, and list_max defaults for List.Natural instructions may infer arguments, while users can minimally provide missing values.Source: Figure 12 · p. 12, Figure 13 · p. 13, Figure 14 · p. 13

Classify

  • The Classify module assigns object regions to categories using CLIP scores and assumes tagging images contain at most one instance of each category.With multiple categories, it de-duplicates assignments by retaining the maximum-scoring region for each assigned category.Source: §A.2 · p. 12

List

  • The List module uses GPT-3 to retrieve factual knowledge and create comma-separated lists from a query.Its prompt includes a configurable default maximum list length.Source: Figure 14 · p. 13

Related papers

Loading 2211.11559v1…