Source-linked AI summary
How Far Can 5,500 Hours of Driving Take You? A Scaling Law Analysis of Video Diffusion Models
Victor Besnier, Anh-Quan Cao, Elias Ramzi, Spyros Gidaris, Tuan-Hung Vu, Andrei Bursuc, Eloi Zablocki, Matthieu Cord
TL;DR
Autonomous-driving video models must scale within expensive, privacy-constrained driving corpora rather than freely scraped web data. The paper fits scaling laws using from-scratch flow-matching models across model sizes and training exposures, then uses them to train a 9B model. Training exposure improves fixed models faster, larger models retain lower asymptotic loss, and the resulting 9B system reaches an open-source state of the art on nuScenes.
Problem
Driving video generation lacks controlled scaling-law evidence because driving data is costly and privacy-constrained, while diffusion and flow-matching objectives differ from language-model and autoregressive settings.
Method
The paper trains spatio-temporal latent video transformers from scratch on 5,500 hours of driving video and fits power laws across model size and training exposure.
Results
The 9B model reaches a validation loss of 0.0781, within 3.6% of an extrapolated prediction, and achieves the best image- and video-based perceptual metrics on nuScenes.
Takeaways & Limitations
Training longer is the fastest way to improve a fixed model, while compute-optimal budgets still favor larger models because capacity lowers asymptotic loss.
Takeaways & Limitations
The scaling analysis assumes a uniform learning rate, although the 9B model required learning-rate adjustment for stability; jointly modeling the frozen VAE remains future work.
Abstract
from arXiv · showhide
Video generation for autonomous driving cannot follow the web-scale route: driving data is expensive to collect, bound by privacy requirements, and cannot be scraped at will, so models must make the most of a fixed corpus. We present a systematic scaling-law study of video diffusion models trained from scratch on driving data: a family of models from 1M to 9B parameters, trained at different exposures on up to 5,500 hours of driving. Validation loss follows consistent power laws in both model size and training exposure, answering the questions that shape a training budget: whether compute is better spent on longer training or on a larger model, and whether more data is needed. Loss improves much faster with training exposure than with model size, making longer training the most effective way to improve a fixed model under limited compute. However, larger models continue to achieve lower asymptotic loss, so compute-optimal scaling still favors increasing model size when sufficient compute and data are available. Guided by these laws, we train a 9B-parameter model, to our knowledge the largest video diffusion model trained from scratch on driving data: it sets a new open-source state of the art for driving video generation, as measured on nuScenes. Our code and pretrained models are available at https://github.com/valeoai/VATIX. NATIX is separately releasing the underlying driving data in stages.
1 Introduction
The paper studies how to allocate compute when autonomous-driving video models must learn from a fixed, costly corpus rather than web-scale data. It fits scaling laws to determine whether larger models or longer training are more effective.
- Driving data is expensive to collect and constrained by privacy, leaving many labs with fixed datasets and limited GPU budgets.
- The central question is how far 5,500 hours of driving video can take a video diffusion model.
- The study compares spending compute on larger models versus more training steps in a data-constrained video-diffusion setting.
- Across more than 200 runs, training exposure improves fixed models faster than model size, while larger capacity continues lowering asymptotic loss.
- Repeating the fixed corpus behaves like fresh samples within fewer than five epochs, so the 5,500-hour dataset is not yet the bottleneck.
2 Related Work
Prior work largely adapts web-pretrained video models or studies scaling in other domains, leaving diffusion-based driving-video scaling laws unresolved. This paper instead trains from scratch on a controlled driving corpus to characterize compute, exposure, and model-size trade-offs.
- Many driving world models adapt general-purpose video-diffusion backbones pretrained on web-scale data using parameter-efficient tuning.
- Web-pretrained models prevent controlled analysis of how driving-specific data and compute affect performance.
- The paper trains diffusion models from scratch on 5,500 hours of driving video, enabling controlled scaling studies across model size, exposure, and compute.
- Scaling-law research established predictable power-law relationships and compute-optimal joint scaling in language modeling, with different trade-offs in data-constrained regimes.
- Scaling laws for diffusion-based driving-video generation remain missing because autoregressive results do not directly transfer to diffusion and flow matching.
3 Method and Experimental Setup
The method learns conditional flow matching with a spatio-temporal latent video transformer, then evaluates model and training scales on a large multi-camera driving dataset. Trajectory conditioning supports controllable generation.
- 3.1 Conditional Flow Matching (CFM): Conditional flow matching linearly interpolates noise and data while training a neural network to regress the target transport velocity.
- 3.1 Conditional Flow Matching (CFM): At inference, an ordinary differential equation is numerically integrated to transform noise into data.
- 3.2 Model architecture: The model is a Diffusion Transformer operating on Wan 2.1 VAE latents, with spatial and temporal attention, feedforward layers, and AdaLN conditioning.
- 3.2 Model architecture: The architecture is scaled from 1.6M to 9B parameters across 12 model sizes.
- 3.3 Training setup: Training uses v-prediction, a shifted log-normal noise schedule, warmup, and exponential moving averages.
- 3.3 Dataset: The dataset contains approximately 5,500 hours of footage split into 2.5-second, 9-Hz clips, producing around 6.3M distinct training clips.
4 Deriving Scaling Laws
The study fits power laws for model size, training exposure, data repetition, and compute, then uses them to identify effective training-budget allocations. Training exposure improves fixed models faster than capacity does, while larger models retain lower asymptotic loss and compute-optimal allocations shift toward them.
- Scaling-law framework: The shared law L(x) = L_0 + A x^{-α} models validation loss against model size, training exposure, or compute.Here, x is N, D, or C; L_0 is asymptotic loss, A sets the initial gap, and α controls the decay rate.
- Model scaling: 72 models spanning 11 sizes were evaluated at 10M samples, and model-size scaling closely matched measurements with RMSE 8.42 × 10^-4 and MAPE 0.56%.The fitted model-size law showed diminishing returns but no saturation through 1.1B parameters.
- Scaling-law extrapolation: The model-size asymptote was estimated at L(9B) ≈ 0.0746, suggesting further gains beyond the largest model used for fitting.The planned 9B run was positioned near the compute-optimal regime by combining model-size, exposure, and compute scaling laws.
- Training scaling: Training-exposure fits achieved MAPE below 1%, with α_D ≈ 0.74 on average across model sizes and most attainable gain estimated within 1.5 × 10^7 samples.The fitted exponent ranged from 0.66 to 0.84, and gains beyond that exposure were estimated below 0.004.
- Data restriction: A 100× reduction in unique footage from 5,500h to 55h changed loss from 0.0939 to 0.0980 at 1.25 × 10^7 samples, while 5.5h caused sharp degradation near 2,000 epochs.For the Base model and tested exposures, repetition below roughly 200 epochs was largely harmless; longer-trained larger models may need new data sooner.
- Compute scaling: Compute scaling accurately captured the lower envelope of validation loss, and optimal allocations shifted toward larger models as compute increased.The fitted asymptotic loss was L_0 ≈ 0.0522, although this estimate is cautious because the dataset is finite.
5 Scaling to 9B
Scaling laws guide a 9B-parameter model trained and evaluated for driving video generation. The study measures validation loss, perceptual quality, temporal consistency, and trajectory accuracy across model sizes and conditioning settings.
- Scaling to 9B: The 9B model is trained using a compute-optimal configuration predicted by scaling laws across model size, training exposure, and compute.The configuration corresponds to approximately 10^7 training samples.
- Validation Loss at 9B Scale: 0.0781 validation loss is achieved after 1.2 × 10^7 samples, the lowest among evaluated models and within 3.6% relative error of the extrapolated prediction.The prediction was extrapolated 8× beyond the largest fitted model, which had 1.1B parameters.
- Validation Loss at 9B Scale: The 9B model reaches the fitted 1.1B asymptote within 1.2 × 10^7 samples, while remaining far from its own asymptotic loss.Its post-hoc asymptotic loss is approximately 0.0748, within 1% of the a-priori compute-law prediction of 0.0753.
- Generation Quality: Evaluation covers visual fidelity, temporal consistency, and trajectory accuracy on balanced 2.5-second, 9-Hz generations conditioned on one frame.The test set contains 2,000 videos balanced across eleven ego-action classes.
- Generation Quality: Larger models improve evaluation metrics, with the strongest gains on video-based perceptual scores, while trajectory fine-tuning further improves perceptual metrics and reduces ADE.The comparison includes FID, FVD, and Average Displacement Error.
- Comparison with Prior Work: The approach achieves the best image- and video-based perceptual performance against prior driving world models after nuScenes fine-tuning.The authors attribute this to diverse large-scale driving pretraining and full-parameter fine-tuning rather than parameter-efficient adaptation.
6 Conclusion
The study finds that training exposure improves fixed models faster than model size, while larger capacity continues lowering asymptotic loss. Scaling remains effective on the fixed corpus, and a 9B model reaches strong extrapolated validation-loss and generation-quality results.
- 6 Conclusion: Across more than 200 runs, training-exposure scaling is steeper than model-size scaling, but compute-optimal budgets still favor larger models as capacity lowers asymptotic loss.The conclusion identifies longer training as the fastest improvement route for a fixed model.
- 6 Conclusion: Repeated use of the 5,500-hour corpus behaves like fresh samples within the studied regime, so the dataset is not yet the bottleneck.The authors hypothesize that flow-matching resamples noise and timestep values on each pass.
- 6 Conclusion: The scaling laws extrapolate to 9B: validation loss is within 3.6% of a prediction extended 8× beyond the largest fitted model.The 9B model is described as the largest open-source video diffusion model trained from scratch on driving data, to the authors’ knowledge.