PaperlayerView exact revision on arXiv

Source-linked AI summary

Scaling Image and Video Generation via Test-Time Evolutionary Search

Haoran He, Jiajun Liang, Xintao Wang, Pengfei Wan, Di Zhang, Kun Gai, Ling Pan

arXiv:2505.17618v1cs.CVcs.AIcs.LG

TL;DR

Test-time scaling for image and video generation remains less understood than for language models, and existing vision approaches face limits in scope, exploration, or diversity. EvoSearch applies evolutionary selection and mutation along diffusion and flow denoising trajectories; the paper reports that it outperforms baselines across image and video tasks. The reported limitations of particle sampling include restricted exploration and diversity as search progresses.

  • Problem

    Test-time scaling for image and video diffusion and flow models is less understood, while existing approaches can be task-limited or struggle with exploration and diversity.

    Source: §1 · p. 2

  • Method

    EvoSearch uses selection and denoising-aware mutation to iteratively explore and refine candidates along diffusion and flow denoising trajectories.

    Source: §4.8 · p. 9

  • Results

    EvoSearch outperforms competitive baselines across image and video generation tasks, with SD2.1 surpassing GPT4o and Wan 1.3B achieving competitive performance with Wan 14B.

    Source: §1 · p. 3

  • Takeaways & Limitations

    The paper presents evolutionary search as a generalist test-time scaling framework for image and video generation across diffusion and flow models.

    Source: §1 · p. 3, §6 · p. 14

    Particle sampling is constrained by its initial candidate pool, limiting exploration and diversity as search progresses.

    Source: §4.2 · p. 5

Abstract

from arXiv · show

As the marginal cost of scaling computation (data and parameters) during model pre-training continues to increase substantially, test-time scaling (TTS) has emerged as a promising direction for improving generative model performance by allocating additional computation at inference time. While TTS has demonstrated significant success across multiple language tasks, there remains a notable gap in understanding the test-time scaling behaviors of image and video generative models (diffusion-based or flow-based models). Although recent works have initiated exploration into inference-time strategies for vision tasks, these approaches face critical limitations: being constrained to task-specific domains, exhibiting poor scalability, or falling into reward over-optimization that sacrifices sample diversity. In this paper, we propose \textbf{Evo}lutionary \textbf{Search} (EvoSearch), a novel, generalist, and efficient TTS method that effectively enhances the scalability of both image and video generation across diffusion and flow models, without requiring additional training or model expansion. EvoSearch reformulates test-time scaling for diffusion and flow models as an evolutionary search problem, leveraging principles from biological evolution to efficiently explore and refine the denoising trajectory. By incorporating carefully designed selection and mutation mechanisms tailored to the stochastic differential equation denoising process, EvoSearch iteratively generates higher-quality offspring while preserving population diversity. Through extensive evaluation across both diffusion and flow architectures for image and video generation tasks, we demonstrate that our method consistently outperforms existing approaches, achieves higher diversity, and shows strong generalizability to unseen evaluation metrics. Our project is available at the website https://tinnerhrhe.github.io/evosearch.

1 Introduction

As training-time scaling grows more costly, test-time scaling offers another way to improve generation, but image and video models pose search challenges that existing approaches do not resolve. EvoSearch uses evolutionary search across diffusion and flow models, and the paper reports gains over competitive baselines across image and video tasks.

1 Introduction

  • Training-time scaling faces rising computational costs and diminishing available internet data, while existing alignment approaches can over-optimize rewards, require weight access, or generalize poorly across models.Source: §1 · p. 2
  • Test-time scaling adds inference computation without additional training, but diffusion and flow models must search a high-dimensional denoising trajectory where best-of-N is inefficient and particle sampling can reduce exploration and diversity.Source: §1 · p. 2
  • EvoSearch selects high-reward parents while preserving diversity, then applies denoising-aware mutations to explore new states and iteratively improve sample quality.Source: §1 · p. 3
  • EvoSearch outperforms competitive baselines across image and video generation; SD2.1 surpasses GPT4o, while Wan 1.3B achieves competitive performance with Wan 14B.Source: §1 · p. 3

2 Preliminary

Diffusion models use stochastic denoising, whereas flow models follow a deterministic ODE that restricts search beyond the initial noise. The section also introduces evolutionary algorithms as population-based methods that use selection and mutation to explore solutions.

2.1 Diffusion Models and ODE-to-SDE Transformation of Flow Models.

  • Diffusion models perturb data toward noise using a timestep noise schedule, then use an SDE-based sampler to restore samples during inference.Source: §2.1 · p. 3
  • Flow-model ODE sampling is deterministic beyond the initial noise, so the paper transforms it into an equivalent SDE process to introduce stochasticity for test-time search.Source: §2.1 · pp. 3–4

2.2 Evolutionary Algorithms.

  • Evolutionary algorithms maintain candidate populations, evaluate fitness to select parents, and apply operators such as mutation to produce offspring over successive generations.Source: §2.2 · p. 4

3 Related Work

Prior alignment approaches include guidance and fine-tuning, while test-time search methods include best-of-N, particle sampling, and beam search. EvoSearch is presented as a broader alternative that actively generates offspring and applies across diffusion and flow models for image and video generation.

Alignment for Diffusion and Flow Models

  • Alignment methods use guidance or fine-tuning, but may require curated data and parameter tuning and can reduce diversity through mode-seeking and reward over-optimization.Source: §3 · p. 4

Test-Time Scaling in Vision

  • Best-of-N selects the highest-reward sample from generated batches, while particle sampling resamples candidates along the denoising trajectory using importance weights.Source: §3 · p. 4
  • Existing search methods depend heavily on initial candidates, whereas EvoSearch creates offspring iteratively and is described as applicable across diffusion and flow models for image and video generation.Source: §3 · p. 5

4 Proposed Method

EvoSearch treats reward-guided sampling as active evolutionary optimization along denoising trajectories. It schedules population evolution and uses selection and mutation to explore for higher-quality samples while preserving diversity.

4.1 Problem Formulation

  • Given a pretrained model and reward, the target distribution favors reward while limiting deviation from the pretrained distribution, but its normalization requires an intractable integral in high-dimensional spaces.Source: §4.1 · p. 5

4.2 Limitations of Existing Approaches

  • In the Gaussian-mixture alignment experiment, EvoSearch captures all target modes and achieves the highest reward (-0.74), while other search methods miss modes.Retraining methods also struggle to generalize to the unseen target distribution.Source: Figure 2 · p. 6, §4.2 · p. 6

4.3 Evolutionary Search

  • EvoSearch replaces passive filtering with active evolutionary optimization, evolving initial noise and intermediate states as the population advances along the denoising trajectory.The framework targets test-time scaling for image and video generation across diffusion and flow models.Source: §4.3 · p. 6

4.4 Evolution Schedule

  • EvoSearch schedules evolution at selected timesteps, first optimizing initial noise and then evolving intermediate states from cached states rather than repeatedly denoising from the start.Uniform timestep intervals reduce computational overhead.Source: §4.4 · pp. 6–7, Figure 3 · p. 7

4.5 Population Initialization.

  • The population-size schedule K specifies the initial Gaussian-noise population and the child population size at each evolution timestep.Initial noises serve as the first-generation parents.Source: §4.5 · p. 7

4.6 Fitness Evaluation.

  • At each evolution timestep, EvoSearch uses an off-the-shelf reward model to evaluate candidates on fully denoised samples, yielding high-fidelity reward signals.This avoids estimating clean samples from noisy states with lookahead estimators or Tweedie’s formula.Source: §4.6 · p. 7

4.7 Selection.

  • Tournament selection repeatedly samples b candidates, where b < k_ti, and chooses the best as a parent to propagate high-quality candidates while maintaining diversity.Source: §4.7 · p. 7

4.8 Mutation.

  • EvoSearch preserves elite parents and mutates the rest to explore nearby states, using mutations tailored to initial Gaussian noise versus intermediate denoising states.Initial-noise mutation balances preserving high-reward-region density with exploration; intermediate-state mutation uses the reverse-time SDE to preserve latent structure, and the framework encompasses best-of-N and particle sampling as special cases.Source: §4.8 · pp. 8–9

5 Experiments

The experiments evaluate EvoSearch on image and video generation across diffusion and flow models, reporting gains over baselines. They also examine inference-time scaling, generalization to unseen rewards, diversity, human preferences, and smaller-model performance.

5.1.1 Image Generation.

  • DrawBench’s 200 prompts across 11 categories are evaluated with ImageReward, HPSv2, Aesthetic score, and ClipScore, using ImageReward and ClipScore as search rewards.Experiments use SD2.1, a diffusion model with 865M parameters, and Flux.1-dev, a 12B rectified flow model; both use 50 denoising steps and guidance scale 5.5.Source: §5.1.1 · p. 9

5.1.2 Video Generation.

  • Video evaluation uses VideoReward for search and measures unseen-reward performance across 625 VBench and VBench2 prompts and human ratings on 200 VideoGen-Eval prompts.The evaluated models are HunyuanVideo and Wan; Wan uses the 1.3B variant and videos contain 33 frames.Source: §5.1.2 · pp. 9–10

5.1.3 Baselines.

  • The baselines are best-of-N, which selects the highest-reward sample from random initial noises, and particle sampling, which propagates high-reward candidates along denoising.The video comparisons use the same random seeds across methods.Source: §5.1.3 · pp. 10–11

5.2 Results Analysis

  • EvoSearch improves monotonically with inference-time function evaluations, while Flux.1-dev baselines plateau after about 1e4 NFEs.SD2.1 exceeds GPT4o below 5e3 NFEs in 30 seconds on one A800 GPU; with 5× inference compute, Wan 1.3B achieves competitive performance to Wan 14B.Source: §5.2 · pp. 11, 13, 14, Figure 5 · p. 11, Table 3 · p. 13
  • 32.8% and 14.1% higher VideoReward scores than best-of-N and particle sampling, respectively, on Wan 1.3B; EvoSearch also beats all baselines across image and video models.The image comparison spans SD2.1 and Flux.1-dev, while video results include Wan 1.3B and HunyuanVideo 13B.Source: Figure 5 · p. 11, §5.2 · p. 11, Figure 6 · p. 12
  • On unseen video metrics, EvoSearch shows more stable gains, including minimal degradation on Semantic and improved HunyuanVideo physics performance as both baselines degrade.The video evaluation covers nine unseen metrics across six categories; for an unseen Aesthetic metric not aligned with ClipScore, EvoSearch degrades less than particle sampling.Source: Table 1 · p. 12, §5.2 · pp. 12–13
  • Professional annotators give EvoSearch higher win rates than baseline methods across Visual Quality, Motion Quality, Text Alignment, and Overall Quality.Source: Figure 9 · p. 13, §5.2 · p. 13
  • At 100× inference-time compute, EvoSearch achieves the highest ImageReward and image diversity among methods tested on 10 DrawBench prompts.Diversity is measured by L2 distance between CLIP hidden features; qualitative examples show more varied backgrounds and poses.Source: Table 2 · p. 13, §5.2 · p. 13, Figure 10 · p. 13

6 Conclusion

EvoSearch applies evolutionary search to improve image and video generation across diffusion and flow models by exploring new denoising states. Its exploration depends on mutation rates, and Gaussian mutation leaves room for more informative strategies.

6 Conclusion

  • EvoSearch iteratively explores new states along the denoising trajectory to generate higher-quality samples across image and video tasks and diffusion and flow models.The framework uses specialized evolutionary mechanisms.Source: §6 · p. 14
  • Exploration depends on mutation rates β and σt: higher rates expand the search space, while lower rates can restrict it.The authors describe this as a trade-off.Source: §6 · p. 14
  • EvoSearch uses Gaussian noise to mutate selected parents; the authors identify more informative, prior-informed mutation strategies as a possible way to improve search efficiency.Source: §6 · p. 14
  • Interpreting denoising states and understanding shared structure between golden noises and intermediate denoising states remain open research directions.Source: §6 · p. 14

A.1.1 Implementation Details of EvoSearch

Implementation details describe flexible inference-time schedules and model-specific settings for EvoSearch, followed by the selected baseline implementations.

A.1.1 Implementation Details of EvoSearch

  • EvoSearch can vary its search timesteps and population size with the available inference-time compute; the implementation uses uniform timestep intervals and doubles the initial population size.The remaining generations use the same population size.Source: §A.1.1 · p. 22
  • For Stable Diffusion 2.1 and Flux.1-dev, the implementation uses guidance scale 5.5, resolution 512 × 512, and mutation rate β = 0.3.Stable Diffusion 2.1 uses DDIM; Flux.1-dev uses an SDE sampler.Source: §A.1.1 · p. 22
  • Wan and Hunyuan are configured for 33-frame videos at 832×480 and 544 × 960, respectively; Hunyuan uses 30 inference steps to save computation.Both use an SDE sampler for the ODE-to-SDE transformation described for video generation.Source: §A.1.1 · p. 22

A.1.2 Implementation Details of Baselines

  • Best of N selects the highest-quality sample from N candidates using a guidance reward shared with EvoSearch, while particle sampling uses Max-potential resampling with tuned hyperparameters.The particle-sampling implementation uses the framework proposed by.Source: §A.1.2 · pp. 22–23

A.2 Evaluation Metrics

The evaluation metrics cover preference, semantic alignment, and visual appeal for images, and several prompt-following and video-quality dimensions for videos.

A.2 Evaluation Metrics

  • Image evaluation uses ImageReward for human preference, CLIPScore for image-prompt semantic alignment, HPSv2 for perceptual preference, and Aesthetic for visual appeal.Source: §A.2 · p. 23
  • Video evaluation measures dynamic attributes, long-prompt semantics, human fidelity, composition, physics, and aesthetic quality.The metrics draw on specified VBench2.0 or VBench prompt categories, with video aesthetics assessed using the LAION aesthetic predictor [58].Source: §A.2 · p. 23

B.1 Ablation on Population Size Schedule

Under a fixed inference-time compute budget, population-size schedules perform similarly, while the initial population size is the most significant factor reported.

B.1 Ablation on Population Size Schedule

  • Different population-size schedules perform similarly with little reward difference at approximately 140 × 50 inference-time NFEs, but a larger initial population kstart benefits subsequent search initialization.The ablation uses Stable Diffusion 2.1 and reports DrawBench results in Fig. 11; a small kstart can affect performance substantially.Source: §B.1 · p. 23

B.1.1 Ablation on Evolution Schedule

The ablation finds population-size scheduling more influential than the evolution schedule, with larger initial populations and uniform evolution intervals associated with better performance. The appendix also reports generalization to unseen metrics and introduces qualitative image and video results.

B.1.1 Ablation on Evolution Schedule

  • Population-size scheduling matters more than the evolution schedule T; larger initial populations and uniform evolution intervals yield superior performance.Source: §B.1.1 · pp. 23–24
  • The evolution-schedule ablation adjusts population schedules to keep test-time computation budgets equal across settings.Source: Figure 12 · p. 24
  • EvoSearch generalizes to unseen metrics when ImageReward guides search on DrawBench with SD2.1 and Flux.1-dev.Source: Figure 13 · p. 25

B.2 Qualitative Results

  • The appendix presents qualitative results for both image and video generation.Source: §B.2 · p. 25

B.2.1 Results for Image Generation

Qualitative comparisons show that EvoSearch improves image-generation performance while using fewer computational resources.

B.2.1 Results for Image Generation

  • EvoSearch significantly enhances image-generation performance in qualitative comparisons with baselines.Source: §B.2.1 · p. 25
  • The image-generation improvements require lower computational resources.Source: §B.2.1 · p. 25

B.2.2 Results for Video Generation

For video generation, EvoSearch outperforms the reported baselines, and Wan 1.3B enhanced with EvoSearch outperforms Wan 14B at matched inference latency. The qualitative examples report prompt alignment, temporal consistency, and other video-generation improvements.

B.2.2 Results for Video Generation

NFEs

Loading 2505.17618v1…