PaperlayerView exact revision on arXiv

Source-linked AI summary

Video-T1: Test-Time Scaling for Video Generation

Fangfu Liu, Hanyang Wang, Yimo Cai, Kaiyan Zhang, Xiaohang Zhan, Yueqi Duan

arXiv:2503.18942v2cs.CVcs.AI

TL;DR

Video-T1 addresses whether extra inference-time computation can improve video generation, where temporal coherence is challenging and further training-scale expansion is costly. It treats generation as search over video trajectories, with verifier feedback and heuristic methods including Tree-of-Frames. Experiments report improved video performance with more test-time compute, and ToF reduces scaling cost while achieving high-quality outputs.

  • Problem

    High-quality video generation must maintain temporal coherence and capture complex frame dynamics, while further scaling through training entails high costs and resource demands.

    Source: §1 · p. 2

  • Method

    Video-T1 searches video trajectories from Gaussian noise using verifier feedback and heuristics, including random linear search and autoregressive Tree-of-Frames branch expansion and pruning.

    Source: §1 · p. 2, §3.2 · p. 5

  • Results

    Test-time scaling improves performance across video-generation models, and ToF achieves comparable performance to random linear search at much lower computational cost.

    Source: §4.2 · pp. 7–8, §5 · p. 9

  • Takeaways & Limitations

    Scaling the search space can boost video performance across models, while ToF can significantly reduce scaling cost when producing high-quality outputs.

    Source: §5 · p. 9

    Improvements are less pronounced for hard-to-evaluate properties such as motion smoothness and temporal flickering.

    Source: §4.2 · p. 9

Abstract

from arXiv · show

With the scale capability of increasing training data, model size, and computational cost, video generation has achieved impressive results in digital creation, enabling users to express creativity across various domains. Recently, researchers in Large Language Models (LLMs) have expanded the scaling to test-time, which can significantly improve LLM performance by using more inference-time computation. Instead of scaling up video foundation models through expensive training costs, we explore the power of Test-Time Scaling (TTS) in video generation, aiming to answer the question: if a video generation model is allowed to use non-trivial amount of inference-time compute, how much can it improve generation quality given a challenging text prompt. In this work, we reinterpret the test-time scaling of video generation as a searching problem to sample better trajectories from Gaussian noise space to the target video distribution. Specifically, we build the search space with test-time verifiers to provide feedback and heuristic algorithms to guide searching process. Given a text prompt, we first explore an intuitive linear search strategy by increasing noise candidates at inference time. As full-step denoising all frames simultaneously requires heavy test-time computation costs, we further design a more efficient TTS method for video generation called Tree-of-Frames (ToF) that adaptively expands and prunes video branches in an autoregressive manner. Extensive experiments on text-conditioned video generation benchmarks demonstrate that increasing test-time compute consistently leads to significant improvements in the quality of videos. Project page: https://liuff19.github.io/Video-T1

1. Introduction

Video-T1 asks whether inference-time computation can improve video generation without costly retraining or model enlargement. It frames test-time scaling as search over video trajectories, using verifier feedback and heuristic algorithms, and reports quality and human-preference gains from increasing test-time compute.

1. Introduction

  • Video generation remains challenging because it must maintain temporal coherence and capture complex dynamics across frames, while scaling through training requires high costs and resources.The introduction contrasts this challenge with LLM research showing performance gains from additional inference-time computation.Source: §1 · p. 2
  • Video-T1 reframes test-time scaling as search over video trajectories originating from Gaussian noise, using verifiers to assess intermediate results and heuristics to guide the search.The goal is to find higher-quality, text-aligned videos without expensive retraining or model enlargement.Source: §1 · p. 2
  • Tree-of-Frames (ToF) adaptively expands and prunes video-frame branches autoregressively to balance computational cost and generation quality.It is introduced as a more efficient alternative because denoising all frames simultaneously is computationally intensive.Source: §1 · p. 2
  • Across text-conditioned video-generation benchmarks, increasing test-time compute improves video quality and human-preference alignment.The paper also reports that ToF can significantly reduce scaling cost while achieving high-quality results.Source: §1 · p. 2

2. Related Work

Related work covers test-time scaling in language and vision, alongside video-generation approaches that scale training or use autoregressive mechanisms. The included results figure describes consistent performance improvement as the test-time search space grows.

Related Work

  • LLM test-time scaling includes chain-of-thought prompting, reward models, self-consistency, best-of-N, Monte Carlo Tree Search, and reward-guided search.The cited discussion associates these approaches with improved reasoning and more contextually appropriate outputs.Source: §2 · p. 2
  • As the search-space sample count increases, model performance improves consistently; light bars show results without TTS and dark bars show improvement after TTS.The caption does not specify axes or numerical values.Source: Figure 2 · p. 3
  • Vision-language models use test-time scaling for visual question answering, while image-diffusion studies report generation-quality gains from verifier-guided selection of candidates.The discussion also describes training-scale video-generation work as improving visual fidelity and motion smoothness, at substantial computational cost, especially for long videos.Source: §2 · p. 3
  • Autoregressive video-generation approaches combine diffusion with sequential mechanisms to handle long and complex videos, with potential for longer, coherent generation and improved efficiency.Examples include NOVA’s sequential frame prediction and Pyramid-Flow’s multi-scale trajectory over compressed representations.Source: §2 · p. 3

3. Method

Video TTS is framed as searching from Gaussian noise for text-aligned, high-quality video trajectories, using verifier feedback to guide generation. The method progresses from random linear sampling to adaptive Tree-of-Frames search and combines multiple verifiers for candidate selection.

3.1. How to Scale Video Generation at Test Time

  • Video TTS must address spatial frame quality, temporal consistency, and the costly iterative denoising process of diffusion models.The challenge is especially relevant because videos require smooth transitions across frames.Source: §3.1 · pp. 3–4

3.2. Random Linear Search

  • Random linear search samples N Gaussian noises, fully denoises each into a video, then selects the highest-scoring candidate using test verifiers.Its exhaustive, independent paths cost O(TN) and provide no interaction between trees.Source: §3.2 · p. 5

3.3. Tree-of-Frames Search

  • Tree-of-Frames (ToF) grows video-frame branches autoregressively, using verifier-guided heuristic pruning to retain the most promising continuations.It organizes generation into image-level alignment, intermediate-frame motion and plausibility checks, and final video-quality assessment.Source: Figure 3 · p. 4, §3.3 · pp. 5–6
  • ToF’s three stages use different verifier prompts: core semantics for the first frame, motion continuity for intermediate frames, and overall text-video alignment at the end.Adaptive branching adds initial noise samples when switching stages to improve temporal coherence and diversity.Source: Figure 3 · p. 4, §3.3 · p. 6
  • ToF’s practical complexity can simplify to O(N + T) because branching occurs at selected prompt stages and b_t remains 1 for most timesteps.The passage contrasts this with random linear search’s quadratic complexity and describes dynamic branching as balancing exploration and convergence.Source: §3.3 · p. 6

3.4. Multi-Verifiers

  • A mixture of test verifiers ranks candidate videos with associated weights, and the candidate with the highest combined score is selected.The approach is intended to mitigate verifier biases.Source: §3.4 · p. 6

4. Experiment

The experiments evaluate test-time scaling across video models, search methods, verifiers, and VBench dimensions. Increasing inference compute improves results, while Tree-of-Frames (ToF) reaches comparable performance to random linear search at lower computational cost.

4.1. Experiment Setup

  • ToF search initializes roots, generates candidate frame continuations, scores them with a heuristic reward, and retains the top-scoring nodes at each step.The algorithm specifies tree depth, branching factors, and pruning sizes.Source: §4.1 · p. 7
  • The evaluation uses six video models spanning 0.6B–5B parameters, three learned test verifiers, and VBench’s 16 fine-grained dimensions.NFE is defined as denoising steps multiplied by the temporal length of latent embeddings.Source: §4.1 · p. 7

4.2. Analysis of Experimental Results

  • Across video models, increasing inference compute improves performance, with larger models such as CogVideoX-5B showing greater gains than lightweight models such as NOVA.Performance curves eventually approach a convergence limit.Source: §4.2 · pp. 7–8, Figure 4 · p. 8
  • ToF achieves performance comparable to random linear search at much lower computational cost.The comparison covers three autoregressive models; inference-time scaling costs are also reported in GFLOPs.Source: Figure 5 · p. 8, §4.2 · p. 8, Table 1 · p. 8
  • For common prompts such as Scene and Object, and assessable categories such as Imaging Quality, TTS produces significant improvements across models.The reported improvement comparison spans prompt complexities and VBench dimensions.Source: §4.2 · p. 9, Figure 6 · p. 9
  • Improvements are less pronounced for hard-to-evaluate properties such as Motion Smoothness and Temporal Flickering.The discussion links these challenges to precise motion-trajectory control and assessing consistency over time.Source: §4.2 · p. 9

5. Conclusion

Video-T1 frames video-generation test-time scaling as a search for optimal video trajectories, using verifier feedback and heuristic search. Experiments show that expanding the search space can improve performance across models, while ToF reduces scaling cost when producing high-quality outputs.

5. Conclusion

  • Video-T1 reframes test-time scaling as searching for optimal video trajectories using verifier feedback and heuristic algorithms.The algorithms include random linear search and the more efficient Tree-of-Frames (ToF).Source: §5 · p. 9
  • Scaling the search space boosts video performance across various video-generation models.Source: §5 · p. 9
  • ToF search significantly reduces scaling cost when achieving high-quality video outputs.The conclusion presents this as a more efficient search algorithm.Source: §5 · p. 9

A. More Implementation Details

Image-level alignment uses staged evaluation to stop weak generation paths early. Hierarchical prompting divides the input into scene, motion, and ending descriptions for stage-specific verification.

A.1. More Discussion of Image-level Alignment

  • A two-stage verifier first checks whether a partially generated frame is clear enough to evaluate, then whether it may lead to a high-quality final image.Unclear frames or paths judged unlikely to yield an optimal result are stopped before synthesis continues.Source: §A.1 · p. 13

A.2. More Discussion of Hierarchical Prompting

  • DeepSeek-R1-8B [8] decomposes each input into static-scene, action/motion, and expected-ending prompts for initial, intermediate, and final frames.Source: §A.2 · p. 13
  • At each video stage, the test verifier assesses the generated output against its corresponding hierarchical prompt.The prompts are returned as an ordered list of three.Source: §A.2 · p. 13

A.3. Detailed Complexity Analysis

Tree-of-Frames grows a forest of candidate video paths, evaluating branches and pruning them to retain promising continuations. In experiments, branching is limited to selected prompt stages, yielding an O(N + T) update complexity.

A.3. Detailed Complexity Analysis

  • Each surviving parent branches into candidate continuations, which are scored by heuristic reward and pruned to retain the top k_t nodes.The deepest leaves represent final frames, and their path represents the selected video sequence.Source: §A.3 · p. 13
  • The method uses a branching limit b and prunes branches below the average heuristic reward; in practice it sets k_0 = N and b = 2.Source: §A.3 · p. 13
  • With TTS, Pyramid-Flow scores close to or above HunyuanVideo’s 13B model in many dimensions.The figure’s gray dashed line marks HunyuanVideo’s score for each dimension.Source: Figure 7 · p. 14
  • O(N + T) is the simplified complexity when branching occurs only at selected prompt stages and the branching factor remains 1 for most timesteps.The experiments use this schedule to support a diverse and stable transition between stages.Source: §A.3 · pp. 13–14
  • Compared with linear search’s quadratic complexity, ToF converges at a geometric rate and achieves linear complexity while maintaining high sample diversity.Dynamically adjusting branching trades off early exploration against later convergence.Source: §A.3 · p. 14

B.1. More Quantitative Results

Random linear search experiments across models and verifiers show stable TTS improvements across VBench dimensions, while verifier accuracy varies by dimension.

B.1. More Quantitative Results

  • Random linear search experiments across models and verifiers indicate that TTS consistently improves performance across VBench dimensions.Source: §B.1 · p. 14
  • Verifier evaluation accuracy varies across VBench dimensions, supporting the use of multiple verifiers.Source: §B.1 · p. 14

B.2. More Qualitative Results

Additional examples show video quality and text alignment improving as TTS samples increase. With TTS, Pyramid-Flow (2B) approaches or surpasses HunyuanVideo (13B) in many dimensions.

B.2. More Qualitative Results

  • As NFE increases, visual results for Pyramid-Flow and CogVideoX-5B show improved video quality and text alignment.The examples use the same prompt input as NFE increases.Source: §B.2 · p. 14

B.3. Comparison with Large Models

  • Pyramid-Flow (2B) with TTS approaches or surpasses HunyuanVideo (13B) single in many dimensions.Figure 7 compares the small model with the large model.Source: §B.3 · p. 14
  • Figure 7's comparison concerns Pyramid-Flow (2B) with TTS and HunyuanVideo (13B) single.Source: §B.3 · p. 14

B.4. Failure Cases

The reported failure case shows that added inference computation does not always improve video quality, particularly when generating reasonable hand-movement details. Other supplementary captions describe verifier dimensions and qualitative examples of improvements in motion and spatial alignment.

B.4. Failure Cases

  • In Pyramid-Flow, added inference computation does not significantly improve some videos or generate reasonable details such as hand movements.The authors attribute this limit to model performance.Source: §B.4 · p. 14
  • The verifier-performance figures cover appearance style, aesthetic quality, and color.Source: Figure 10 · p. 16, Figure 11 · p. 16, Figure 12 · p. 16
  • Additional examples say TTS can alleviate issues with human or complex movements and enhance multi-object spatial perception.The captions describe results on Pyramid-Flow and CogVideoX-2B.Source: Figure 14 · p. 18, Figure 17 · p. 21
  • On NOVA, TTS improves alignment between video object relationships and the text prompt.Source: Figure 18 · p. 22

Related papers

Loading 2503.18942v2…