Source-linked AI summary
ViperGPT: Visual Inference via Python Execution for Reasoning
Dídac Surís, Sachit Menon, Carl Vondrick
TL;DR
Complex visual queries require both perception and reasoning, but end-to-end models do not explicitly separate these processes, limiting interpretability and generalization. ViperGPT generates Python programs that compose API-accessible pretrained modules, achieving state-of-the-art zero-shot results across visual tasks. Its approach remains dependent on the quality of its end-to-end perception modules.
Problem
End-to-end visual models implicitly perform multiple subtasks without exposing compositional reasoning or enabling reliable step-by-step auditing and systematic generalization.
Method
ViperGPT uses code-generating language models to produce Python functions that compose vision and language modules exposed through an API, then compiles and executes them.
Results
ViperGPT achieves state-of-the-art zero-shot results across visual grounding, image question answering, and video question-answering.
Takeaways & Limitations
Programmatic composition connects specialized vision, language, math, and logic functions and enables capabilities beyond what any individual model can provide alone.
Takeaways & Limitations
ViperGPT strongly relies on end-to-end perception modules when they are used in the system.
Abstract
from arXiv · showhide
Answering visual queries is a complex task that requires both visual processing and reasoning. End-to-end models, the dominant approach for this task, do not explicitly differentiate between the two, limiting interpretability and generalization. Learning modular programs presents a promising alternative, but has proven challenging due to the difficulty of learning both the programs and modules simultaneously. We introduce ViperGPT, a framework that leverages code-generation models to compose vision-and-language models into subroutines to produce a result for any query. ViperGPT utilizes a provided API to access the available modules, and composes them by generating Python code that is later executed. This simple approach requires no further training, and achieves state-of-the-art results across various complex visual tasks.
1. Introduction
ViperGPT addresses the limits of end-to-end visual reasoning by generating and executing Python programs that compose pretrained vision and language modules. This produces explicit, modular, training-free reasoning while achieving state-of-the-art zero-shot results across several visual tasks.
- Motivation: End-to-end models must implicitly perform all subtasks, limiting systematic generalization, reuse of vision advances, mathematical operations, and step-by-step auditing.The paper motivates modular reasoning with examples such as counting children and muffins before dividing.
- Method: ViperGPT uses code-generating language models to create customized Python programs that compose vision capabilities exposed through an API.The generated function receives an image or video and returns the answer to the query.
- Properties: The framework makes reasoning interpretable because its steps are explicit code function calls with inspectable intermediate values.The Python interpreter handles logical and mathematical operations, while pretrained models provide perceptual capabilities.
- Properties: ViperGPT supports compositional, flexible, adaptable, training-free, and general visual reasoning by recombining modules without retraining for each task.Adding a module requires specifying its associated API interface.
- Results: ViperGPT achieves state-of-the-art zero-shot results across visual grounding, image question answering, and video question-answering.The authors report that interpretability aids performance rather than hindering it.
2. Related Work
Prior modular vision systems sought compositional reasoning but often required program supervision, end-to-end training, or hand-specified module combinations. ViperGPT instead generates unrestricted Python code, enabling more flexible composition and control flow while leveraging pretrained vision and language modules.
- Modular Vision: Neural Module Networks decompose complex vision tasks into atomic perceptual units, but related systems require expensive program supervision and end-to-end training.These approaches separate reasoning from perception and connect to neuro-symbolic methods.
- Modular Vision: Because modular methods were difficult to use in practice, the field largely shifted toward end-to-end all-in-one models.Other approaches combine large pretrained models but hand-specify how those models are combined.
- Novelty: Unlike concurrent pseudocode-based work, ViperGPT directly generates unrestricted Python code for greater flexibility, including control flow and mathematical operations.Python also leverages Codex’s prior knowledge from large-scale Internet code training.
- Interpretability: ViperGPT’s interpretability differs from post-hoc visual explanations by exposing step-by-step reasoning through image crops and text.The comparison is made with methods such as pixel-importance explanations and hard-attention approaches.
- Pretrained Models: ViperGPT combines pretrained modules for object detection, text-image similarity, depth estimation, external knowledge, and simple visual queries.The cited implementations include GLIP, X-VLM, MiDaS, GPT-3, and BLIP-2.
3. Method
ViperGPT generates Python programs that compose vision and language modules for each visual query, then executes them with a standard Python interpreter. This design avoids task-specific program-generation training while supporting inspectable, compositional reasoning across visual inputs.
- Program formulation: Given a visual input x and textual query q, ViperGPT synthesizes a program z = π(q) and executes it to produce result r = φ(x, z).Programs can accept images or videos and return outputs such as text or image crops.
- Program representation and execution: Python code represents programs directly, providing built-in logical operators, mathematical functions, control flow, and compatibility with existing tools.This replaces graph-based program representations and avoids requiring a custom interpreter.
- Program generation: Replacing a learned program generator with an LLM removes the need for task-specific training for program generation.Earlier approaches required supervised or reinforcement learning and struggled with unavailable program supervision or prohibitive graph-search optimization.
- Program generation: The program generator uses a code-generating LLM with an API specification and query as input, producing a Python function that is compiled and executed.The API exposes perceptual and knowledge modules through signatures, docstrings, and examples rather than full implementations.
- Modules and execution: The framework relies on pretrained end-to-end perception modules while using generated programs and Python execution for sequential reasoning.Its stated design combines pattern-recognition modules with analytic, multi-step processing.
- Scope and implementation: ViperGPT supports visual tasks with arbitrary output types and evaluates visual grounding, compositional image question answering, external-knowledge image question answering, and video reasoning without additional training.Programs are batch-executed with multiprocessing and a producer-consumer design for GPU batching.
4. Evaluation
ViperGPT is evaluated across visual grounding, compositional and knowledge-dependent image question answering, and video causal-temporal reasoning. It achieves strong zero-shot performance while generating programs that combine perception, external knowledge, and temporal frame selection.
- ViperGPT evaluates four settings: visual grounding, compositional image question answering, external knowledge-dependent image question answering, and video causal-temporal reasoning.
- Visual Grounding: On RefCOCO and RefCOCO+, ViperGPT outperforms other zero-shot methods for visual grounding.RefCOCO includes spatial relations, whereas RefCOCO+ does not.
- External Knowledge-dependent Image Question Answering: For OK-VQA, ViperGPT combines step-by-step perception with external knowledge, outperforming all zero-shot methods and surpassing the best prior publicly resourced model by 6%.
- Video Causal/Temporal Reasoning: For video queries, generated programs identify relevant frames and reason over extracted information and frame numbers to answer causal and temporal questions.The evaluation uses the multiple-choice version of NExT-QA.
- Video Causal/Temporal Reasoning: Despite seeing no video data, ViperGPT matches the best supervised model and surpasses it on the NExT-QA hard split for both temporal and causal queries.
5. Exploring New Capabilities
ViperGPT supports capabilities beyond standard benchmarks by composing general-purpose modules and enabling diagnosis and contextual adaptation. Its programs can incorporate additional knowledge to change their logic for different settings.
- Extensible modules: The framework permits adding modules such as OCR, surface-normal estimation, and segmentation when new capabilities are required.Its included modules cover a wide range of tasks, while the API can be extended.
- Interventional explainability: ViperGPT’s programmatic structure enables automatic diagnosis of which modules are responsible for prediction errors.Interventions can evaluate module importance without requiring ground-truth labels for every intermediate output.
- Interventional explainability: On RefCOCO visual grounding, perception modules and Python operations show a similar level of importance.The analysis indicates that both types of components are tightly integrated in the approach.
- Conditioning on additional information: Additional context can cause ViperGPT to generate different logic for the same query, such as adapting road-side reasoning to the photo’s country.The context is supplied as a comment before code generation.
6. Conclusions
ViperGPT composes specialized vision, language, math, and logic functions into programs for complex visual queries. The framework is presented as connecting advances across these capabilities, with results expected to improve as its component models improve.
- Conclusions: ViperGPT programmatically composes specialized vision, language, math, and logic functions for complex visual queries.The framework connects individual advances across these component capabilities.
- Conclusions: ViperGPT’s results are expected to improve in tandem with improvements to the models implementing its composed functions.This conclusion concerns the framework’s dependence on improving component models.
A. Pretrained Models
The implementation uses several pretrained models for perception, visual-language matching, depth estimation, language querying, and code generation. The appendix identifies the model versions, repositories, and API choices used.
- Pretrained models: GLIP-L is used for object detection, and MiDaS DPT_Large is used for depth estimation.GLIP required CUDA modifications for newer PyTorch versions, while MiDaS came from PyTorch Hub.
- Pretrained models: BLIP-2 uses the Flan-T5 XXL version, with the official implementation slightly more performant and the Hugging Face implementation faster.The authors report little difference between the two implementations.
- Pretrained models: X-VLM is the official implementation fine-tuned for retrieval on MSCOCO.It is listed among the pretrained models used in the system.
- Language querying: GPT-3 text-davinci-003 is used for the llm_query function through the official OpenAI Python API.The appendix separately specifies the API used to access this model.
- Code generation: GPT-3 code-davinci-002 is used for Codex code generation.The appendix identifies this model as the code-generation model.
B. API
The API exposes image patches, detection and verification operations, visual queries, depth estimation, cropping, and matching functions. These primitives provide the operations used to construct executable visual programs.
- ImagePatch: The API defines ImagePatch as a crop of an image centered around an object and carrying relevant information.ImagePatch can represent either an unmodified image or a crop with stored coordinates.
- Object operations: find returns image patches matching an object name, while exists returns whether that object is present.find returns an empty list when no matching objects are found; exists returns a boolean.
- Property verification: verify_property checks whether a named object has a specified property, presupposing that the object exists.It differs from exists by testing an attribute rather than object presence.
- Visual-language queries: best_text_match selects the option that best matches the image, while simple_query answers a basic question about the image.The API also supports prefixes for text matching and an optional question for simple_query.
- Geometry and depth: compute_depth returns the median depth of an image crop, and crop returns a new ImagePatch for specified coordinates.The API documents the crop boundaries and the median-depth output.