Source-linked AI summary
Towards Accurate Generative Models of Video: A New Metric & Challenges
Thomas Unterthiner, Sjoerd van Steenkiste, Karol Kurach, Raphael Marinier, Marcin Michalski, Sylvain Gelly
TL;DR
Video generation needs evaluation that captures visual quality, temporal coherence, and sample diversity, alongside benchmarks that reflect challenging dynamics. This paper introduces Fréchet Video Distance and the StarCraft 2 Videos benchmark, finding that FVD agrees better with human judgment than SSIM and PSNR while SCV challenges remain open.
Problem
Video-generation research lacks metrics covering visual quality, temporal coherence, and diversity, as well as challenging benchmarks beyond simpler synthetic data.
Method
The paper introduces Fréchet Video Distance, which evaluates distributions over video-level features, and the StarCraft 2 Videos benchmark for testing long-term memory and relational reasoning.
Results
FVD consistently outperforms SSIM and PSNR in agreeing with human judgment of generated videos.
Takeaways & Limitations
FVD and SCV provide an objective measure and tailored benchmark for assessing progress in generative video models.
Takeaways & Limitations
SCV offers controllable content and challenging motion, but its visual fidelity cannot rival real-world datasets.
Abstract
from arXiv · showhide
Recent advances in deep generative models have lead to remarkable progress in synthesizing high quality images. Following their successful application in image processing and representation learning, an important next step is to consider videos. Learning generative models of video is a much harder task, requiring a model to capture the temporal dynamics of a scene, in addition to the visual presentation of objects. While recent attempts at formulating generative models of video have had some success, current progress is hampered by (1) the lack of qualitative metrics that consider visual quality, temporal coherence, and diversity of samples, and (2) the wide gap between purely synthetic video data sets and challenging real-world data sets in terms of complexity. To this extent we propose Fréchet Video Distance (FVD), a new metric for generative models of video, and StarCraft 2 Videos (SCV), a benchmark of game play from custom starcraft 2 scenarios that challenge the current capabilities of generative models of video. We contribute a large-scale human study, which confirms that FVD correlates well with qualitative human judgment of generated videos, and provide initial benchmark results on SCV.
1. Introduction
Video generation requires models to capture both visual content and temporal dynamics, but progress is limited by weak evaluation metrics and challenging data. The paper introduces FVD and SCV, validates FVD against human judgments, and benchmarks current models extensively.
- Motivation: Video generation must model temporal dynamics, including object interactions, in addition to visual presentation.The paper motivates applications including missing-frame prediction, instance segmentation, and relational reasoning.
- Challenges: Current video-generation models remain limited to a few seconds of video, making dynamics modeling and evaluation major challenges.Useful metrics should assess visual quality, temporal coherence, and sample diversity.
- Contributions: The paper introduces Fréchet Video Distance (FVD), whose video feature representation captures temporal coherence alongside frame quality.FVD builds on principles underlying Fréchet Inception Distance and differs from metrics such as PSNR and Structural Similarity.
- Contributions: FVD variants using different distance functions and embeddings respond to both temporal and frame-level perturbations.This sensitivity is demonstrated by adding noise to real videos.
- Contributions: A large-scale human study finds that FVD coincides well with qualitative human judgment of generated videos.This provides evidence that FVD reflects human assessments of video-generation quality.
- Contributions: The paper introduces StarCraft 2 Videos (SCV), four scalable datasets from handcrafted scenarios requiring relational reasoning and long-term memory.The datasets can vary in resolution and scene complexity, and the paper compares current models across BAIR, KTH, and SCV using FVD, examining 3000 models.
2. Fr´echet Video Distance
This section defines video-distribution distances by applying Fréchet distance in a learned feature space, introducing FVD as a general video metric and exploring an MMD-based Kernel Video Distance. The feature representation must capture both visual content and temporal coherence, while FVD is evaluated against PSNR and SSIM.
- Fréchet Video Distance: FVD measures the distance between real and generated video distributions using a learned feature embedding where Gaussian approximation makes Fréchet distance feasible.The embedding adapts the image-based FID approach to video evaluation.
- Fréchet Video Distance: The authors introduce FVD as a general metric for videos and conduct an extensive empirical study of its capabilities.A similar FID adaptation had previously been used to evaluate vid2vid.
- Fréchet Video Distance: The video feature representation must encode temporal coherence across frames in addition to visual presentation.ImageNet-trained representations may emphasize objects while suppressing other information content, and network layers capture different abstraction levels.
- Kernel Video Distance: Kernel Video Distance applies Maximum Mean Discrepancy to learned I3D features, comparing empirical distributions without assuming a particular distributional form.This alternative addresses potential error from estimating Gaussian distributions in the learned feature space and uses a polynomial kernel.
- Comparison Metrics: The experiments compare FVD and KVD with PSNR and SSIM, two metrics commonly used in the relevant literature.PSNR and SSIM can measure deviation from a ground-truth sequence, whereas FVD considers video distributions.
3. Starcraft 2 Videos
StarCraft 2 Videos (SCV) is a benchmark suite designed to advance video generation through interpretable unit tests and challenging tasks targeting real-world failure modes. It provides four configurable scenarios with virtually unlimited videos, while state-of-the-art models achieve only moderate results on the simplest scenario and struggle with the others’ temporal dynamics.
- Benchmark design: SCV introduces unit-test and limit-testing benchmark tasks that isolate challenges such as long-term memory and relational reasoning in video generation.The challenging tasks also emphasize typical failure modes and avenues for future research.
- Benchmark design: SCV contains four scenarios—MUtB, CMS, Brawl, and RTwM—and can generate a virtually unlimited number of videos at different resolutions.The benchmark uses the open-source StarCraft 2 Learning Environment and supplies datasets at 64 × 64 and 128 × 128 resolutions.
- Scenarios: MUtB tests stochastic unit appearance, color, destination, and animation while modeling temporal dynamics comparable to Moving MNIST or Stochastic Shapes.A randomly chosen unit moves from the map’s middle to its border.
- Scenarios: CMS requires learning unit–shard interactions and shortest-path movement as two units collect randomly placed mineral shards.The shard disappears when a unit reaches it.
- Scenarios: Brawl models complex interactions between varying armies, with outcomes determined by army composition and initial unit locations, while RTwM requires long-term tracking of units through pickup, transport, beacons, and dropoff.RTwM requires unloading the same number and types of units that were picked up.
- Benchmark results: With default parameters, state-of-the-art models achieve only moderate results on MUtB and fail to accurately capture the complex temporal dynamics and consistency of the other scenarios.SCV also releases code for generating datasets with custom scenario hyperparameters.
4. Experiments
Experiments show that FVD detects static and temporal video distortions, correlates well with human judgments of generated-video quality, and exposes substantial failures across challenging benchmark scenarios. FVD generally outperforms competing metrics, with differences of at least 50 FVD points typically corresponding to perceivable quality differences.
- Distortion sensitivity: FVD detected injected static and temporal distortions, while the image-based Inception baseline was generally inferior for temporal distortions.The logits layer of I3D pretrained on Kinetics-400 was among the best configurations by rank correlation with noise intensity.
- Human evaluation: FVD was the superior metric in human evaluations, inducing rankings that no other metric improved upon and distinguishing good models more reliably.KVD performed similarly but was slightly worse in most scenarios, whereas Avg. FID performed markedly worse except on spr. SSIM.
- Human evaluation: 50 FVD points or more typically corresponded to perceivable differences in generated-video quality.When FVD differences were smaller than 50, agreement with human raters was close to random but never worse, then increased rapidly beyond 50 points.
- SC2 benchmark: Benchmark models showed substantial FVD differences but similar failure modes across challenging scenarios, including poor modeling of units, event sequences, and long videos.On Brawl, models generated larger blurry blobs instead of all participating units, and neither model succeeded on RTwM.
- Metric correlations: SSIM and PSNR were strongly correlated across more than 20,000 models, with Pearson’s r=0.730 and Kendall’s τ=0.648.The study used models from Section 4.4 to cover a broad range of generated videos encountered during video-generation training.
5. Conclusion
The paper introduces Fréchet Video Distance (FVD) for evaluating video-generation models and presents the StarCraft 2 Videos (SCV) benchmark. A large-scale human study finds that FVD agrees with human judgment more consistently than SSIM and PSNR, while SCV exposes open challenges in long-term memory and relational reasoning.
- Contributions: FVD is introduced as a new evaluation metric for generative models of video.The paper describes it as an important step toward better evaluation of video-generation models.
- Evaluation: FVD consistently outperforms SSIM and PSNR in agreeing with human judgment of generated videos.This finding comes from a large-scale human study of videos generated by several recent generative models.
- Benchmark: The SCV benchmark provides data sets of different complexity that test long-term memory and relational reasoning, challenges that remain open for state-of-the-art video-generation models.Its visually simpler setting is intended to help researchers isolate, test, and analyze solutions to these specific problems.
A. Noise Study
The noise study evaluates FVD implementations and an FID-based baseline across HMDB, BAIR, and Kinetics-400, finding that Kinetics-400 I3D logits correlate well with noise intensity across noise types.
- A. Noise Study: The study uses HMDB, BAIR, and Kinetics-400, with 90% of available train and test samples used for comparison.Noise intensities are mapped to parameter values for the considered noise types.
- A. Noise Study: Kinetics-400 I3D logits correlate well with noise intensities across a variety of noise types.Figure 6 compares various FVD implementations and an FID-based baseline against sequences of noise intensities.
B. SCV Data Generation
SCV comprises custom StarCraft 2 gameplay videos generated from agent–environment interactions, with replay files rendered into datasets using scenario-specific termination and sampling procedures. Each dataset has fixed training, validation, and test splits, while environment-controlled randomness produces varied replay lengths.
- B. SCV Data Generation: Each SCV dataset contains 10 000 training, 2 000 validation, and 2 000 test gameplay videos from agents playing custom StarCraft 2 scenarios.Scenarios were created with the StarCraft 2 Editor, and agents were implemented in the SC2LE framework.
- B. SCV Data Generation: Agents are deterministic, while scenario environments control randomness and actions are provided within the agent–environment interaction.
- B. SCV Data Generation: Data generation first creates replay files by having agents play scenarios in SC2LE, encoding deterministic environment states that can be rendered at different resolutions.Sequences terminate when scenario-specific termination conditions are fulfilled.
- B. SCV Data Generation: Scenario termination occurs when a unit reaches the border in MUtB, after 2 in-game minutes in CMS, or upon victory or 2 in-game minutes in Brawl.
- B. SCV Data Generation: RTwM terminates when all units reach the final beacon, and videos use scenario-specific frame skipping: every 6th, 4th, or 8th frame, always skipping the first 2 frames.MUtB uses every 6th frame, CMS and Brawl every 4th, and RTwM every 8th; stochasticity produces different video lengths.
B.1. Scenario Parameters
SCV scenarios are implemented as editable SC2Map files that initialize maps and units, observe game state, and expose hyperparameters for adjusting complexity while preserving each core task. The benchmark includes varied unit-control, collection, combat, and transport scenarios with randomized or fixed configurations.
- Scenario implementation: Each scenario is an SC2Map file that initializes the map and units and monitors game state to test a termination condition.Scenario hyperparameters can change complexity while retaining the core task.
- Scenario configurations: Move Unit to Border randomly selects one unit from 6 types and assigns one of 4 colors before moving it toward a hidden border beacon.The unit spawns at the map center, and the beacon is placed along the outside border.
- Scenario configurations: Collect Mineral Shards uses two fixed Marine units to collect 20 randomly placed mineral shards by repeatedly targeting the nearest shard.The units spawn at random locations, and shards maintain a minimum distance from them.
- Scenario configurations: Brawl pits Zerg and Terran armies against each other, with 9 randomly positioned units per army selected from six race-specific unit types.Army spawn locations are vertical strokes along opposite map borders and are randomly assigned to the two armies.
- Scenario configurations: Road Trip with Medivac spawns one Medivac and 1-4 randomly selected, colored escort units, then places two beacons while avoiding the center.The Medivac transports the surrounding units according to a greedy policy.
C. Benchmark Hyperparameters
The benchmark uses Tensor2Tensor implementations with default parameters, while tuning selected learning-rate, VAE loss-trade-off, and model-specific settings through grid search or prescribed schedules.
- Models use Tensor2Tensor implementations and default parameters unless otherwise stated.
- Learning rates were selected from 10−3, 10−4, and 10−5, while VAE models tuned β values for the reconstruction-loss/KL-divergence trade-off.The searched β values were 10−6, 10−5, 10−6, and 10−3.
- SV2P followed the β annealing schedule from prior work, while SAVP additionally tuned GAN-loss and GAN-VAE-loss parameters.
D. Examples of Video Models on SCV
The section presents 128 × 128 SCV video examples, comparing original sequences with videos generated from their first two frames. Examples span four scenarios and four video models, with visual quality intended for comparison against FVD scores in Table 2.
- Example setup: Generated videos are conditioned on the first two frames of each 128 × 128 data-set sequence, enabling visual comparison with FVD scores in Table 2.The examples place the original sequence above the corresponding model-generated video.
- Move Unit to Border: Move Unit to Border examples compare original videos with CDNA, SV2P, SVP-FP, and SAVP.The figure uses 128 × 128 resolution and orders the original video first, followed by the four models.
- Collect Mineral Shards: Collect Mineral Shards examples compare original videos with CDNA, SV2P, SVP-FP, and SAVP.The figure is presented at 128 × 128 resolution.
- Brawl: Brawl examples compare original videos with CDNA, SV2P, SVP-FP, and SAVP at 128 × 128 resolution.The figure orders the original video first, followed by the four generated-video models.
- Road Trip with Medivac: Road Trip with Medivac examples compare original videos with CDNA, SV2P, SVP-FP, and SAVP at 128×128 resolution.The figure orders the original video first, followed by the four generated-video models.