PaperlayerView exact revision on arXiv

Source-linked AI summary

VQQA: An Agentic Approach for Video Evaluation and Quality Improvement

Yiwen Song, Tomas Pfister, Yale Song

arXiv:2603.12310v1cs.CVcs.AIcs.LGcs.MA

TL;DR

Video generation systems still struggle to align outputs with complex intent, while existing test-time optimization can be expensive or require white-box access. VQQA uses dynamic visual questions and VLM critiques as semantic gradients for black-box, closed-loop prompt refinement. It improves results across T2V and I2V settings, including absolute gains of +11.57% on T2V-CompBench and +8.43% on VBench2 over vanilla generation.

  • Problem

    Video generation outputs remain difficult to align with complex human intent, while existing test-time methods can be computationally intensive or require white-box access to model internals.

  • Method

    VQQA dynamically generates visual questions, uses VLM critiques as semantic gradients, and iteratively refines prompts through a black-box natural-language interface.

  • Results

    +11.57% on T2V-CompBench and +8.43% on VBench2 are absolute improvements over vanilla generation, with gains demonstrated across T2V and I2V tasks.

  • Takeaways & Limitations

    VQQA provides a scalable, task-agnostic approach for aligning visual generative models with complex human intent through interpretable iterative refinement.

  • Takeaways & Limitations

    Compared with methods that modify model internals, VQQA operates through discrete prompt-space optimization and does not use gradients through the generator.

Abstract

from arXiv · show

Despite rapid advancements in video generation models, aligning their outputs with complex user intent remains challenging. Existing test-time optimization methods are typically either computationally expensive or require white-box access to model internals. To address this, we present VQQA (Video Quality Question Answering), a unified, multi-agent framework generalizable across diverse input modalities and video generation tasks. By dynamically generating visual questions and using the resulting Vision-Language Model (VLM) critiques as semantic gradients, VQQA replaces traditional, passive evaluation metrics with human-interpretable, actionable feedback. This enables a highly efficient, closed-loop prompt optimization process via a black-box natural language interface. Extensive experiments demonstrate that VQQA effectively isolates and resolves visual artifacts, substantially improving generation quality in just a few refinement steps. Applicable to both text-to-video (T2V) and image-to-video (I2V) tasks, our method achieves absolute improvements of +11.57% on T2V-CompBench and +8.43% on VBench2 over vanilla generation, significantly outperforming state-of-the-art stochastic search and prompt optimization techniques.

1. Introduction

VQQA addresses difficult video-intent alignment and costly or inflexible evaluation by turning visual assessment into actionable, iterative prompt refinement. Its multi-agent, model-agnostic framework uses targeted questions, VLM critiques, candidate selection, and dynamic stopping to improve generations.

  • Motivation: Complex human intent remains difficult to align with video outputs, which can contain compositional errors, temporal inconsistencies, and physical hallucinations.Users consequently rely on tedious trial-and-error prompt engineering.
  • Motivation: Existing evaluation methods either miss complex compositional alignment, require computationally expensive model ensembles, or provide non-actionable feedback.These limitations make evaluation poorly suited to correcting individual generations.
  • Method: VQQA is a unified multi-agent framework that dynamically generates visual questions and converts VLM evaluations into actionable feedback for T2V and I2V tasks.This replaces static rubrics and passive benchmarking with adaptive evaluation and iterative prompt refinement.
  • Method: Global Selection compares candidates against the initial prompt, while dynamic stopping limits redundant inference and helps prevent semantic drift.The framework remains model-agnostic and requires no task-specific fine-tuning.
  • Results: VQQA substantially outperforms prompt optimization and sampling baselines across established benchmarks using both open-weights and proprietary models.The reported experiments cover T2V-CompBench, VBench2, and VBench-I2V.
  • Method: VQQA’s qualitative refinements use low-scoring question-answer pairs to localize visual flaws and construct prompts that mitigate those artifacts in subsequent generations.Figure 2 illustrates this closed-loop correction process.

2. Related Work

Prior video evaluation and optimization methods are often passive, open-loop, computationally intensive, or dependent on model internals. VQQA instead uses context-dependent VLM feedback to refine prompts through a natural-language interface.

  • Video Evaluation: Distribution-level metrics capture basic visual statistics but correlate poorly with instance-level human perception and provide little actionable guidance.This motivated later semantic and agentic evaluation approaches.
  • Video Evaluation: VLM-based evaluators improve semantic assessment, but existing approaches include frame-text scoring, affirmative-answer probabilities, opinion-score regression, and reasoning-based judging.These methods primarily evaluate quality rather than directly revising the generation.
  • Prompt Optimization: Prompt optimization methods such as APE, Promptist, Prompt-A-Video, and VPO generally optimize prompts using dataset-level priors rather than artifacts in the current generation.Video self-critique and revision therefore remain relatively underexplored.
  • Test-Time Scaling: Inference-time scaling for video commonly uses rejection sampling, trajectory search, candidate selection, or latent-space interventions.These strategies expand computation or modify the generation process directly.
  • Test-Time Scaling: Existing methods face a trade-off: gradient-based approaches require white-box model access, whereas VISTA requires large candidate pools and substantial computation.This limits compatibility with commercial APIs and efficient deployment.
  • VQQA: VQQA treats the prompt as the optimization variable and uses VLM-guided natural-language feedback as semantic gradients for precise error correction.This bypasses weight access and exhaustive sampling.

3. Methodology

VQQA formulates video evaluation as interpretable, text-based test-time optimization: a VLM produces scores and rationales that guide prompt refinement, while multi-agent evaluation and global selection preserve alignment with the original conditions.

  • Video Evaluation and Test-Time Optimization: VQQA models video generation as test-time prompt optimization, using an evaluation function to produce quantitative scores and qualitative rationales.The evaluation function is intended to be interpretable and to support downstream refinement.
  • Iterative Refinement: Because the target evaluator is typically unknown and non-differentiable, VQQA uses a proxy evaluator and treats its reasoning as a semantic gradient for identifying visual flaws.The VLM then uses the critique to rectify flaws in the subsequent generation.
  • Multi-Agent Architecture: The framework decomposes refinement into Question Generation, Question Answering, and Prompt Refinement agents that generate visual queries, evaluate videos, and update prompts.The questions cover video-prompt alignment, visual quality, and condition fidelity when additional conditions are provided.
  • Global Selection: A Global VLM Rater evaluates candidate videos against the original prompt and conditions, selecting the candidate with the highest global alignment score.This post-hoc comparison keeps local prompt exploration anchored to the user’s overarching intent.
  • Convergence Criterion: Refinement stops when a target quality threshold is reached or when the running maximum score stagnates within a patience window.The convergence criterion trades off inference cost against quality and halts when additional computation yields no improvement.

4. Experiments

VQQA is evaluated across text-to-video and image-to-video tasks against vanilla generation and optimization baselines, with results showing consistent quality gains, artifact detection, and efficient convergence. Ablations further indicate that separating localized refinement from global candidate selection helps preserve holistic prompt alignment.

  • Tasks and baselines: VQQA is evaluated on T2V and I2V tasks, using T2V-CompBench, VBench2, and VBench-I2V alongside established optimization and stochastic-search baselines.Comparisons include VPO, Best-of-N, VQAScore, VideoScore2, and VLM-Rating.
  • T2V-CompBench: 53.46%: VQQA with Gemini-3-Pro achieves the highest T2V-CompBench average, improving +11.57% over vanilla generation and +4.76% over VQAScore.The strongest category gains occur in consistent-attribute (+22.94%), spatial understanding (+14.31%), and numeracy (+13.85%).
  • VBench2: 50.41%: VQQA with Gemini-3-Pro achieves the highest VBench2 total score, improving +8.43% over vanilla generation and +3.46% over VQAScore.The GPT-4o variant achieves the second-highest total score at 48.18%.
  • VBench-I2V: +1.24%: VQQA improves over vanilla generation on VBench-I2V and exceeds the strongest Best-of-N baseline by +0.23%.VQQA with Gemini-3-Pro achieves the highest performance across all evaluated axes and averages 1.6 iterations to satisfy the stopping criterion.
  • Visual flaw identification: 11.9%: VQQA improves E2E-Recall over a zero-shot VLM baseline while both methods maintain near-perfect precision above 99%.The higher recall captures a wider range of visual artifacts for subsequent correction; marginal precision deficits are attributed to infrequent VLM hallucinations.
  • Convergence and efficiency: 3.80 to 4.22 iterations: with patience window k = 3, the stopping criterion converges across thresholds, while most performance gains occur within the first four rounds.The method uses early stopping after the running maximum Global Score saturates, reducing redundant inference steps.
  • Global selection: 1.02%: removing Global Selection lowers the overall average score, while Average-QA Selection underperforms global selection by 1.86%.The ablations support decoupling granular QA feedback for localized refinement from global scoring for final candidate selection.
  • Global-score ablation: 53.11% vs. 53.77%: incorporating the global score into iterative refinement degrades average generation quality compared with the standard configuration.Excluding the global score from Prompt Refinement improves attribute consistency, spatial understanding, and numeracy.

5. Conclusion

VQQA turns passive video evaluation into an active, closed-loop refinement process using dynamic visual questions and VLM critiques as semantic gradients. Evaluations across T2V and I2V show improvements over strong baselines with few iterations and broad model generalization.

  • VQQA transforms passive video evaluation into active, closed-loop prompt refinement through dynamically generated visual questions and VLM critiques.The critiques function as semantic gradients through a black-box natural-language interface.
  • VQQA generalizes across both T2V and I2V tasks and across open-weights and proprietary models.
  • VQQA improves video-generation performance over strong baselines while requiring very few iterations.

A. Limitations

VQQA’s effectiveness is bounded by its underlying foundation models and its sequential multi-agent inference process. These constraints limit correction of fundamental model deficiencies and increase latency relative to parallelizable methods.

  • VQQA cannot rectify fundamental architectural flaws, synthesize out-of-distribution concepts, or fully eliminate noisy semantic gradients from VLM hallucinations.Its black-box optimization depends on VLM reasoning and base video-model generation.
  • Sequential generation and querying give VQQA higher inference latency than parallelizable methods such as Best-of-N.The paper states that rapid convergence and improved alignment justify this sequential compute cost.

B. Inference Cost

VQQA uses multiple VLM calls per optimization round but usually converges quickly. On T2V-CompBench, its expected call count is comparable to Best-of-5.

  • N denotes sampled candidates in Best-of-N, while T denotes VQQA optimization iterations.
  • VQQA requires (4 + k)T generation-phase VLM calls, where k is the number of I2V reference images.Question Generation uses 2 + k calls, while Question Answering and Prompt Refinement use one call each.
  • VQQA’s theoretical maximum is (5 + k)T + 1 VLM calls, including Global Rater evaluation of generated candidates.
  • T=1.245 average optimization rounds on T2V-CompBench yield approximately 7.23 expected VLM calls.This is comparable to a standard Best-of-5 baseline and is enabled by dynamic stopping.

C.1.1. Vision-Language Models (VLMs)

The experiments use Gemini and OpenAI APIs alongside CogVideoX-5B and Veo 3.1 video-generation systems. Baseline evaluation includes VQAScore, VideoScore2, and Best-of-N selection.

  • Gemini and OpenAI models are accessed through Google Cloud Vertex AI and the official OpenAI API, respectively.
  • CogVideoX-5B is accessed through Hugging Face diffusers, while Veo 3.1 is accessed through the Google Cloud Vertex AI API.CogVideoX-5B uses 50 inference steps, 41 frames, and guidance scale 6.0; Veo 3.1 uses a 16:9 aspect ratio with default parameters.
  • VQAScore and VideoScore2 provide video-text alignment scoring signals for baseline evaluation.VQAScore uses llavaonevision-qwen2-7b-ov as its backbone; VideoScore2 is accessed through the Hugging Face API.
  • Best-of-N selects among sampled candidates using a scoring mechanism.

C.2. Video Generation Seeds

The experiments fix the initial generation seed for comparable models while randomizing later generations, with deterministic temperature settings for reproducibility.

  • The initial video-generation seed is fixed at 17 for CogVideoX-5B and CogVideoX5B-I2V, while Veo 3.1 does not accept seed inputs.This controls the starting generation for comparisons involving models that support explicit seeds.
  • For multi-sample benchmarks, initial sample seeds are deterministically assigned as 17 + 100k, with random seeds used in later iterations.Optimization is performed per sample rather than per prompt.
  • The VPO baseline uses a random seed because it requires only one video generation.
  • Generation temperature is set to 0.0 across question generation, question answering, video analysis, and prompt optimization.The stated purpose is deterministic, stable, and reproducible agent behavior.

C.4. Long-horizon runs

Long-horizon analysis shows that most semantic gains occur early, motivating four-round reporting, while visual comparisons use CogVideoX-5B against direct prompting and VPO.

  • Long-horizon convergence: Most semantic performance gains across T2V-CompBench categories occur within the first four optimization rounds.The analysis examines performance over 10 VQQA prompt-optimization iterations.
  • Long-horizon convergence: The study reports experimental results using four VQQA optimization rounds because later rounds provide limited additional semantic gains.
  • Visual comparisons: Side-by-side visual comparisons evaluate direct prompting, VPO, and VQQA with CogVideoX-5B as the fixed generator.CogVideoX-5B is used because VPO was specifically trained for that model.

D.2. VQQA Full Trajectory

A complete example shows VQQA diagnosing a static, incomplete blooming video and refining its prompt toward explicit motion and state-change descriptions.

  • Trajectory setup: The trajectory uses CogVideoX-5B as the generator and Gemini-3-Pro as the VQQA backbone.
  • Iteration 0: The initial video remains a closed bud and fails to generate the requested blooming action.The bloom-clarity score is 5, and the primary diagnosis is static subject or lack of motion.
  • Iteration 0: VQQA identifies incomplete transformation because the video never reaches the target full-flower state.The corresponding question asks how clearly a full flower is shown by the end of the video.
  • Prompt refinement: The refinement strategy replaces vague blooming language with explicit verbs, physical changes, stronger motion, and contrasting start and end states.The revised prompts emphasize petals unfurling, expanding, bursting open, and transforming from a bud into a flower.
  • Iterations 1–2: The trajectory achieves S*_t = 100 in two iterations by amplifying action descriptions to overcome the missing blooming motion.The example attributes the improvement to iterative prompt refinement.

E.0.2. Baseline Implementation

The baseline implementation uses VLM prompts that generate concrete visual-flaw questions and examples, emphasizing motion, physics, consistency, and artifact detection for non-expert raters.

  • Flaw generation: The baseline directly instructs a VLM to generate visual flaws, illustrated with color mismatches, floating objects, and unnatural joint articulation.
  • Prompt complexity: Question counts scale with prompt complexity, ranging from about five questions for simple prompts to up to ten for complex prompts.The examples cover cars, people, animals, objects, actions, and environmental interactions.
  • Question design: Question-generation prompts require clear, concise 0–100 questions tailored to the prompt’s specific subjects and actions.Questions are intended for non-expert human raters and must avoid generic subject references.
  • Evaluation priorities: The prescribed priority is defects and physics first, followed by object permanence, visual artifacts, and motion quality, with aesthetics lowest.Examples target sliding feet, chaotic limbs, melting or flickering objects, disappearance, and unrealistic interactions.
  • I2V evaluation: Condition-fidelity prompts evaluate whether generated videos preserve a reference image’s visual identity and render its subject naturally and aesthetically.
Loading 2603.12310v1…