PaperlayerView exact revision on arXiv

Source-linked AI summary

VISTA: A Test-Time Self-Improving Video Generation Agent

Do Xuan Long, Xingchen Wan, Hootan Nakhost, Chen-Yu Lee, Tomas Pfister, Sercan Ö. Arık

arXiv:2510.15831v1cs.CV

TL;DR

Text-to-video generation remains highly sensitive to prompt phrasing, while existing test-time optimization struggles with video’s multiple scenes, modalities, and contextual meaning. VISTA addresses this with iterative structured planning, pairwise selection, specialized critiques, and prompt rewriting, consistently improving video quality and alignment with user intent. Its gains reach up to 60% against state-of-the-art baselines, with human evaluators preferring VISTA outputs in 66.4% of comparisons.

  • Problem

    Text-to-video quality and user alignment remain sensitive to exact prompt phrasing, while existing test-time optimization methods struggle with video’s multi-scene, multimodal, and contextual complexity.

  • Method

    VISTA iteratively plans prompts into structured temporal scenes, selects candidates through pairwise tournaments, critiques visual, audio, and contextual fidelity, and rewrites prompts.

  • Results

    VISTA consistently improves video quality and alignment with user intent, improving state-of-the-art T2V models like Veo 3 by up to 60% under the authors’ metrics.

  • Takeaways & Limitations

    VISTA enables more reliable, user-aligned text-to-video generation and is potentially scalable to broader video synthesis applications.

  • Takeaways & Limitations

    Evaluation relies primarily on MLLMs and automated metrics, while comprehensive human evaluation is prohibitively expensive.

Abstract

from arXiv · show

Despite rapid advances in text-to-video synthesis, generated video quality remains critically dependent on precise user prompts. Existing test-time optimization methods, successful in other domains, struggle with the multi-faceted nature of video. In this work, we introduce VISTA (Video Iterative Self-improvemenT Agent), a novel multi-agent system that autonomously improves video generation through refining prompts in an iterative loop. VISTA first decomposes a user idea into a structured temporal plan. After generation, the best video is identified through a robust pairwise tournament. This winning video is then critiqued by a trio of specialized agents focusing on visual, audio, and contextual fidelity. Finally, a reasoning agent synthesizes this feedback to introspectively rewrite and enhance the prompt for the next generation cycle. Experiments on single- and multi-scene video generation scenarios show that while prior methods yield inconsistent gains, VISTA consistently improves video quality and alignment with user intent, achieving up to 60% pairwise win rate against state-of-the-art baselines. Human evaluators concur, preferring VISTA outputs in 66.4% of comparisons.

1. Introduction

VISTA addresses the difficulty of aligning text-to-video outputs with user goals by autonomously refining prompts through a multi-agent test-time framework. Experiments report improved performance over prior methods and user preferences.

  • VISTA transforms user input into temporally grounded, multi-scene, multi-aspect descriptions before generation.Its structured planning component is part of a four-component framework for prompt refinement.
  • The framework selects video-prompt candidates through pairwise tournaments and refines prompts using multi-dimensional multi-agent critiques.Its components target visual, audio, and contextual aspects of generated videos.
  • VISTA improves state-of-the-art T2V models like Veo 3 by up to 60% under the authors’ metrics.The study evaluates the framework on single-scene and internal multi-scene benchmarks.
  • VISTA uses configurable evaluation metrics to support autonomous, model-driven video evaluation and refinement.The framework is designed to improve generation without requiring users to repeatedly adjust prompts and filter outputs.
  • Human studies and extensive experiments show that VISTA consistently outperforms existing baselines and improves user preferences.

2. VISTA

VISTA is a modular, two-phase framework that iteratively improves text-to-video generation through structured planning, candidate selection, multidimensional critique, and prompt refinement. It decomposes evaluation across visual, audio, and contextual dimensions and uses pairwise tournaments to retain the strongest video-prompt pair.

  • 2. VISTA: VISTA produces an optimized video V* and refined prompt P* through Initialization and Self-Improvement phases.Initialization plans prompt variants and selects a champion pair; Self-Improvement critiques, refines, regenerates, and reselects until stopping.
  • 2.1. Initialization Phase: Structured planning decomposes a user prompt into timed, multi-scene candidates with configurable properties spanning context, visual, and audio dimensions.The framework uses a multimodal LLM to infer scene properties when needed and enforces configurable realism, relevancy, and creativity constraints.
  • 2.1. Initialization Phase: The system retains the residual original prompt alongside decomposed variants, allowing models that do not benefit from decomposition to remain in the candidate set.Default planning also grounds videos in real-world physics, avoids unnecessary invention, encourages beneficial ambient sound, and discourages excessive transitions for simple prompts.
  • 2.1. Initialization Phase: Pairwise tournament selection compares candidate videos across criteria including visual fidelity, commonsense, text-video alignment, audio-video alignment, and engagement.Binary tournaments reduce candidates through pairwise comparisons, with customizable penalties for specified T2V failures.
  • 2.2. Self-Improvement Phase: MultiDimensional Multi-Agent Critiques assign independent visual, audio, and context evaluation to triadic courts of Normal, Adversarial, and Meta Judges.The criteria include temporal consistency, audio-video alignment, semantic coherence, physical commonsense, engagement, and video format, with critiques and scores on a 1-10 scale.

3. Related Work

Related work includes test-time optimization, video-planning agents, motion-focused prompt methods, and multi-agent video systems, but these approaches address different subsets of the problem. VISTA combines black-box test-time prompt optimization with failure-sensitive, multimodal video critique.

  • 3.1. Text-To-Video Synthesis and Optimization: Existing methods refine video plans, learn motion-oriented token embeddings, or rewrite prompts with training-time target prompts rather than black-box test-time optimization.The cited approaches include online plan execution, MotionPrompt, and RAPO, while RAPO relies on target prompts during training.
  • 3.1. Text-To-Video Synthesis and Optimization: Multi-agent systems such as Mora and FilmAgent address scriptwriting and cinematography but omit test-time optimization.VISTA combines multi-agent systems with test-time optimization for video generation.
  • 3.2. Video Evaluation: Video evaluation lacks a single definitive ground truth and must account for multiple dimensions requiring complex holistic reasoning.Conventional visual metrics and audio-visual benchmarks provide complementary but rigid evaluation approaches.
  • 3.2. Video Evaluation: VISTA differs from related evaluators by emphasizing failure-sensitive visual and audio metrics, including for state-of-the-art text-to-video models.VideoScore and Evaluation Agent are described as not failure-focused and as overlooking audio dimensions.

4. Experiments

VISTA is evaluated on single- and multi-scene benchmarks against direct prompting and several prompt-optimization baselines using pairwise, conventional, and human metrics. Across these evaluations, VISTA generally improves video quality and scales more consistently with iterative test-time computation, though performance depends on its components and evaluation scope.

  • Automatic Evaluations: VISTA consistently outperforms baselines, reaching 45.9% and 46.3% win rates with Δ values of 32% and 35.1% across the evaluated settings.The cited comparison reports stronger scaling with test-time compute than enhanced baselines.
  • Automatic Evaluations: VISTA achieves 27.8–60.0% win rates in single-scene and 18.5–53.2% in multi-scene comparisons against baselines.Its largest margin over a baseline occurs despite that baseline performing best against Direct Prompting.
  • Automatic Evaluations: VISTA improves multiple visual, audio, and contextual dimensions, including Dynamic Quality of 89.87% versus 77.22% for the best baseline in single-scene evaluation.It also reduces audio noisiness by 0.1/5 and discontinuities by 0.11/5 across both datasets.
  • Human Evaluations: Human evaluators preferred VISTA over the best baseline in 66.4% of comparisons, while rating its self-improvement 3.78/5 versus 3.33/5 for VSR(++).Human ratings also improved Direct Prompting from 3.36 to 3.77 for visual quality and reached 3.47 versus 3.21 for audio quality.
  • Analyses: Removing any VISTA component reduces performance: single judges fail to generalize across scenes, PairwiseSelect destabilizes later iterations, and removing DTPA lowers performance ceilings.The ablation identifies complementary roles for structured planning, tournament selection, combined critiques, and reasoning-driven prompt revision.
  • Analyses: With more iterations, VISTA follows a stable upward trajectory while baselines show noisy or limited improvement, including little additional gain for VSR++ on MovieGenVideo.The multi-scene comparison uses Light VISTA with one sampled video per iteration and without Step 2.
  • Analyses: VISTA also improves generation quality with the less capable Veo 2 model, although its gains are smaller than with Veo 3.The authors attribute this difference to Veo 2 being less able to leverage the optimized details.

5. Conclusions

VISTA improves text-to-video generation at test time by jointly optimizing visual, audio, and contextual elements through prompt planning, critiques, and alignment-based selection.

  • VISTA jointly optimizes visual, audio, and contextual elements through explicit prompt planning, multi-agent critiques, and alignment-based video selection.
  • VISTA improves video quality in state-of-the-art models such as Veo 3 while preserving prompt intent and enhancing instruction-following.
  • The framework reduces physical, visual, and audio hallucinations and yields significant human preference gains.
  • VISTA is presented as potentially scalable toward more adaptive, human-aligned, and collaborative video generation.

Limitations

VISTA’s evaluation and adaptability remain limited by automated assessment, non-universal quality assumptions, and dependence on capable underlying models.

  • Evaluation relies primarily on MLLMs and automated metrics, which may introduce systematic biases or miss video qualities prioritized by humans.
  • VISTA’s default quality metrics may not generalize across cultural contexts, creative styles, or user preferences.
  • VISTA requires MLLMs and text-to-video models with strong instruction-following and reasoning capabilities to function effectively.

A.1. Evaluations on Conventional Video and Audio Generation Metrics

Conventional evaluation covers visual quality, audio quality, and text-video alignment for both single-scene and multi-scene generation.

  • Single-scene evaluations measure visual quality, audio quality, and text-video alignment using VBench, NISQA, and CLIP-Score.
  • Multi-scene evaluations use the same VBench, NISQA, and CLIP-Score metric families to assess visual, audio, and alignment quality.

A.2. Human Evaluation Results Details

The human-evaluation analysis reports annotator-based results alongside self-improvement, baseline win-rate, and visual-audio score measurements.

  • The evaluation reports VISTA’s win rate against the best baselines.
  • The analysis includes self-improvement scores and separate visual and audio scores, each measured on a five-point scale.
  • Human evaluation results are reported across annotators in Table 7.

A.3. Why Does VISTA Work? Case Studies

VISTA improves video generation by identifying nuanced visual, contextual, and physical flaws, then refining prompts to address them. Case studies show that these critiques extend beyond surface-level adjustments to improve transitions, audio, overlays, and motion plausibility.

  • Prompt Refinement: VISTA’s targeted prompt modifications improve scene transitions, audio, text-overlay placement, and end-of-video polish.The case study describes an original video with abrupt transitions, distracting audio, and less polished overlays, followed by refinements in each area.
  • Video Selection: VISTA filters out videos that are visually engaging but physically or auditorily nonsensical during winner selection.Human investigations report this filtering capability as a distinct benefit of the selection step.
  • Visual and Physical Fidelity: VISTA’s critiques identify subtle motion flaws, including incorrect spaceship direction, missing micro-dynamics, and unrealistic exhaust behavior.The negative judge contrasts visually smooth motion with physically implausible dynamics that reduce believability.
  • Contextual Fidelity: VISTA detects contextual and emotional inconsistencies, such as abrupt time-of-day shifts and static character expressions during a lantern release.The critique links the scene’s weak internal consistency and limited emotional interaction to shortcomings beyond basic action completion.
  • Critique Quality: VISTA’s multi-faceted critiques surface visual, contextual, and emotional flaws that conventional language-model critiques often miss but human judges notice.Table 8 summarizes critiques spanning visual fidelity, contextual consistency, and emotional expression.

A.4. Prompt Length Distribution among Methods

Prompt-length changes differ by scene complexity and optimization method. Single-scene methods generally lengthen prompts, whereas multi-scene methods show method-specific increases or decreases relative to the original.

  • Single-scene scenarios: All methods generally increase prompt length over the Original in single-scene scenarios, with Visual Self-Refine producing the longest prompts.The comparison is summarized alongside the average token distributions across iterations.
  • Multi-scene scenarios: In multi-scene scenarios, Rewrite produces shorter prompts than the Original, while VPO and Visual Self-Refine slightly lengthen them.The shorter Rewrite prompts follow Google Cloud guidance recommending fewer properties than the dataset prompts; the paper’s methods slightly shorten them.
  • Outcome comparison: Figures 8 and 9 compare VISTA with Direct Prompting using average win/tie/lose outcomes for single-scene and multi-scene videos.These figures provide outcome comparisons complementary to the prompt-length distributions.

A.6. Results with Qwen2.5-VL-32B-Instruct as the Evaluator

This evaluation examines VISTA against Direct Prompting in multi-scene videos using Qwen2.5-VL-32B-Instruct. The supplied materials identify the comparison and evaluator but do not report numerical rates.

  • Multi-scene evaluation: Figure 11 reports VISTA’s multi-scene win, tie, and loss rates against Direct Prompting, evaluated by Qwen2.5-VL-32B-Instruct.The supplied figure caption specifies the comparison scope and evaluator, without exposing the plotted values.
  • Evaluator context: Table 10 presents Veo 3 performance scored by different evaluators, providing the broader evaluation context for evaluator-based comparisons.The supplied table caption does not include individual scores or win rates.

A.7. Results with Gemini 2.5 Pro as the Evaluator

The Gemini 2.5 Pro evaluation contributes to a broader evaluator analysis of VISTA’s iterative improvements. Across evaluators, performance generally improves with more interactions, while Gemini results are more comparable to one another on multi-scene videos.

  • Evaluator results: Performance generally improves as the number of interactions increases across evaluator models.Table 10 reports this trend for methods’ win rates over Direct Prompting.
  • Evaluator results: Qwen-VL-32B-It achieves higher single-scene win rates than Gemini models, while its multi-scene results are more moderate and comparable to Gemini.The comparison is reported as an evaluator-dependent pattern rather than a single universal ranking.
  • Evaluator results: The evaluators collectively confirm the robustness of VISTA’s iterative improvements.This conclusion summarizes the cross-evaluator pattern reported for win rates over Direct Prompting.

D.2. VISTA’s Behaviors

VISTA improves prompt fidelity and instruction-following while reducing physical, visual, and audio inconsistencies in generated videos. Its iterative refinements preserve the original content while producing more realistic motions and smoother transitions.

  • Prompt Fidelity: VISTA improves prompt fidelity without inducing content drift, preserving the original video content while increasing quality.Human evaluation attributes this behavior to explicit constraints and text-video alignment.
  • Instruction Following: VISTA significantly improves instruction-following, correcting failures that Direct Prompting leaves unresolved.The comparison includes unrealistic gremlin motion and failed scene transitions.
  • Hallucination Reduction: VISTA reduces physical, visual, and audio hallucinations through constraint-guided selection and penalties for violations.These controls target abrupt object changes, implausible motions, and unsolicited audio or text.
  • Visual Comparisons: VISTA produces more realistic gremlin motions and smoother visual transitions than Direct Prompting with Veo 3.Direct Prompting fails on motion realism and coherent multi-scene transitions in the cited comparison.

D.5. Deep-Thinking Prompting Agent

The deep-thinking prompting analysis identifies nuanced visual, audio, transition, and aesthetic issues that detailed prompts still leave underspecified. It connects these omissions to default model behaviors and proposes targeted revisions for more polished outputs.

  • Case Analyses: The spaceship critique identifies static lighting, a static galaxy, uniform streaking stars, and abrupt transitions despite technically clean generation.The negative judge’s assessment is treated as more consequential than the positive assessment.
  • Case Analyses: The sky-lantern example exposes a major atmospheric omission: absent stars and an abrupt shift from late afternoon to artificial blue night.The scene also lacks emotional depth, limiting its intended resonance.
  • Deep-Thinking Critique: VISTA’s critiques surface nuanced flaws including static lighting, implausible transitions, and limited character depth that conventional critiques may miss.These issues are described as readily noticed by human judges.
  • Prompt Analysis: The analyzed prompt lacks specificity for text animation, audio refinement, transitions, and aesthetic controls, causing basic default implementations.Examples include abrupt overlays, wind noise, hard cuts, and visible camera reflections.
  • Targeted Revisions: The proposed revisions target visual effects, on-screen text animation, environmental audio, and scene transitions to improve production polish.The analysis recommends specifying animation, sound refinement, transition type, and reflection control.
Loading 2510.15831v1…