Zero-Shot Video Question Answering with Procedural Programs

1. Introduction

• Procedural decomposition is motivated by how people solve video questions through concrete steps, and the authors hypothesize it can improve zero-shot video QA.Existing zero-shot approaches are described as unable to carry out procedural reasoning explicitly; ViperGPT's image-centric approach was limited in generalizing to diverse video benchmarks.Source: §1 · p. 1
• ProViQ uses an LLM to generate Python programs that break a video query into steps using an API of visual modules for detection, QA, retrieval, captioning, transcription, tracking, and summarization.The modules support reasoning over individual frames, clips, and whole videos.Source: §1 · pp. 1–2
• The generated programs require no further training, expose module-level error attribution, and can compose modules for tasks such as query-based tracking and basic video editing.The paper also describes adding task-specific modules without fine-tuning.Source: §1 · p. 2
• Up to 25% improvement is reported across six zero-shot QA benchmarks, with additional gains on EgoSchema and state-of-the-art performance on TVQA.The reported results include surpassing supervised state-of-the-art on some datasets.Source: §1 · p. 2

2.1. Video Question Answering

• Video QA commonly relies on pretraining and fine-tuning, while zero-shot methods such as FrozenBiLM and transfer from image QA have limitations in training requirements or benchmark performance.Other video-conversation approaches using language models and visual inputs [24] [42] do not achieve strong quantitative results on standard benchmarks.Source: §2.1 · p. 2

2.2. Modular Vision

• Modular VQA methods compose visual modules, but many train the layout and modules together or cannot freely compose them [59]; ProViQ extends code-generated modular reasoning to video.CodeVQA, VISPROG, and ViperGPT generate short Python programs, while ViperGPT's image-centric approach was limited in temporal reasoning and broader video experiments.Source: §2.2 · p. 2, Figure 2 · p. 3

2.3. Prompting and Tool Use

• ProViQ uses pretrained visual components including GroundingDINO [29], BLIP-2 [23], LaViLa, GPT3.5, and ByteTrack within its program-generation framework.Prior multimodal language models such as Flamingo, GPT-4, LLAVA, X-GPT, and MiniGPT-4 [62] are described as not incorporating video successfully.Source: §2.3 · p. 3

3.1. Program Generation

• The LLM receives the video query, a documented visual API, and example programs, then generates a short Python program that calls modules and is executed on the video to produce an answer or modified video.The input can include multiple-choice options and other relevant information.Source: §3.1 · p. 3

3.2. Video Modules

• The API provides frame- and clip-level tools such as filtering frames, detecting objects, finding object crops, and voting over frame-wise QA answers; it also includes transcript, caption, tracking, and option-selection modules.The API is intended as a flexible toolbox for a wide range of questions, though it is not necessarily exhaustive.Source: §3.2 · pp. 3–4

3.3. Long Video Summarization

• For long-video narrative questions, get summary applies a video-to-text model to one-second chunks and uses an LLM to aggregate timestamped captions into a paragraph with sentences describing five-second intervals.Its effectiveness depends on caption quality; the paper reports stronger suitability for EgoSchema because Ego4D provides dense annotated narrations.Source: Figure 3 · p. 4, §3.3 · p. 5

3.4. Prompting and Examples

• Benchmark-specific example programs are selected for the query, with examples sampled from training splits to avoid evaluation contamination.Prompts for LLM-based modules such as choose option and get summary remain fixed, using static example inputs.Source: §3.4 · p. 5

3.5. Open-Ended and Multiple-Choice Benchmarks

• Multiple-choice answers are constrained to listed options, while open-ended outputs are mapped to the closest FastText vocabulary answer and may be restricted by question category.The embedding-based mapping is performed outside program execution to reduce calls.Source: §3.5 · pp. 5–6
• Table 1 reports state-of-the-art zero-shot accuracy gains of up to 26% on open-ended and multiple-choice benchmarks, with performance competitive with supervised methods.Table 2 reports a large gain over end-to-end models on the held-out zero-shot EgoSchema test split for long-video understanding.Source: Table 1 · p. 6, Table 2 · p. 6

4.1. Experimental Setup

• Evaluation covers open-ended, multiple-choice, and long-video QA datasets, using top-1 test accuracy on the full test split unless otherwise stated.Source: §4.1 · p. 6

4.2. Zero-Shot Video QA Results

• 23% on iVQA, 25% on TGIF-QA, and 25% on EgoSchema; Table 3 reports ProViQ surpassing the supervised state-of-the-art on NeXT-QA.The authors report gains on open-ended and multiple-choice benchmarks.Source: §4.2 · p. 6, Table 3 · p. 6

4.3. Additional Capabilities

• Composing modules enables query-based multi-object tracking and basic editing, including tracking dancers across frames and removing clips that fail input criteria.Source: §4.3 · p. 7, Figure 5 · p. 7

4.4. Ablation Studies

• Video querying avoids single-frame errors and accounts for most gains on shorter datasets, while filtering and object filtering particularly help on longer datasets.Find helps with counting and color questions in ActivityNet; captioning boosts NeXT-QA on causal questions. More modules can slightly reduce individual-dataset performance or produce invalid programs, but broaden benchmark generalization.Source: §4.4 · pp. 7–8
• A single in-context example greatly improves performance over providing none; additional examples help with diminishing returns, and three to four worked well across benchmarks.Source: §4.4 · p. 8, Figure 6 · p. 8
• Error sources vary by dataset: lower-quality open-ended sets show annotation or module errors, while TVQA errors often arise from program generation selecting the wrong modality.On iVQA, the word-matching embedding model can misclassify a correct program output during post-processing.Source: §4.4 · p. 8, Figure 7 · p. 8

• ProViQ extends modular vision methods to zero-shot video question answering and improves over the state of the art using language-model temporal reasoning with code.Source: §5 · p. 8
• The conclusion describes grounded tracking and text-based video editing as additional capabilities requiring no modifications to the method.Source: §5 · p. 8
• The authors suggest procedural reasoning may improve performance on vision tasks more broadly and present ProViQ as a prospective video QA baseline.Source: §5 · p. 8

• The codebase is a heavily modified fork of ViperGPT, with overhauled infrastructure and more efficient video processing.Source: A. Implementation Details · p. 12
• Visual components include BLIP-2 for image QA, GroundingDino for object detection, LaViLa for video captioning, and ByteTrack for object tracking.Source: A. Implementation Details · p. 12
• Most datasets use 60 frames sampled uniformly over time, and inference can run on one Nvidia A100 GPU.Source: A. Implementation Details · p. 12

• TGIF-QA, MSVD-QA, and MSRVTT-QA are open-ended benchmarks generated from captions with some manual annotations; their questions use one-word or phrase answers.Source: §B.1 · p. 12
• iVQA uses instructional videos and multiple correct answers per question, while ActivityNet-QA covers videos up to three minutes and nine question categories.Source: §B.1 · p. 12
• TVQA combines visual and narrative questions with scene transcripts, EgoSchema tests long-term understanding in three-minute videos, and NeXT-QA targets causal and temporal reasoning.TVQA and NeXT-QA each provide five answer choices; EgoSchema has a held-out test split and no training data.Source: §B.1 · p. 12

B.2. Metrics

• Top-1 accuracy is retained as the fairest comparison metric, while conversational-evaluation works are omitted from the analysis.The authors report that LLM-based evaluation often gives incorrect answer-accuracy judgments.Source: §B.2 · p. 12
• Top-1 accuracy requires a single correct answer even when an open-ended question may have multiple valid answers.Source: §B.2 · p. 12
• LLM-based evaluation compares generated text with a ground-truth answer and prompts GPT-3.5 for binary correctness and subjective conversational scores.Source: §B.2 · p. 12

B.3. Per-dataset Results

• ActivityNet-QA results are broken down by question category in Table 5.Source: §B.3 · p. 12
• Table 6 reports per-category results for TGIF-QA and NeXT-QA.Source: §B.3 · p. 12
• Table 7 reports per-category results for MSVD and MSR-VTT QA.Source: §B.3 · p. 12

C. Additional Ablations

• The Word Matcher maps model outputs to the semantically nearest word or phrase in the output vocabulary, avoiding automatic errors for out-of-vocabulary predictions.FastText is used, though BERT or word2vec are possible alternatives.Source: C. Additional Ablations · p. 12
• For open-ended benchmarks, the output vocabulary uses the 1,000 most common training answers and can be further constrained by question category.The method can dynamically alter this vocabulary without training.Source: C. Additional Ablations · p. 12
• Vocabulary size matters most on lower-quality-label datasets, while its effect is minimal on TGIF and iVQA.Vocabulary restriction significantly boosts performance on poor-label, ambiguously phrased MSRVTT-QA.Source: Table 8 · p. 13, C. Additional Ablations · p. 13

D. Prompt

• The prompt excludes methods not applicable to a dataset and appends dataset-specific in-context examples alongside the API prompt and query.Source: D. Prompt · p. 13