Towards Precise Scaling Laws for Video Diffusion Transformers

1. Introduction

• Precise video-generation scaling laws are needed because video models are costly to train and video scaling research is scarce.The paper frames the task as predicting optimal model size, performance, batch size, and learning rate for a specified architecture and budget.Source: §1 · p. 1
• Video diffusion models are more sensitive to batch size and learning rate than the applied language-model scaling law accounts for, making its predictions less accurate.Fixed suboptimal settings can select larger models and produce higher validation loss beyond small compute budgets.Source: §1 · pp. 1–2
• 39.9% smaller model size than predictions from fixed suboptimal settings at Movie Gen's compute, with similar performance.The comparison uses the same computational resources as Movie Gen.Source: §1 · p. 2
• Optimal-hyperparameter fitting yields a more generalized validation-loss relationship across model sizes and training budgets, including predictions for non-optimal sizes.At fixed compute, validation loss remains relatively stable when model size is adjusted near its optimum.Source: §1 · p. 2
• The paper proposes scaling laws that predict optimal batch size and learning rate for a given model size and compute budget.The analysis finds that optimal hyperparameters depend primarily on model size and training-token count.Source: §1 · p. 2

2. Setup

• The notation defines validation loss L, parameter count N, context length nctx, training tokens T, and estimated non-embedding compute C.N excludes vocabulary and positional embeddings; nctx is defined from frame count and frame dimensions.Source: §2 · p. 2
• The study focuses on the stable training phase and uses a constant learning rate to simplify the analysis and control variables.The authors state that this scheduler yields effective predictions of optimal hyperparameters.Source: §2 · p. 3
• Validation loss on a held-out partition is the primary metric, measured every 200 training steps; external benchmarks are omitted.The authors cite difficulties consistently aligning multiple video benchmarks and use validation loss as a performance proxy.Source: §2 · p. 3

3. Predicting Optimal Hyperparameters

• Prior scaling and hyperparameter-transfer approaches do not fully account for video DiT sensitivity or model size, motivating a law that links hyperparameters to model and data size.The paper contrasts heuristic selection and transfer methods with a power-law approach for more precise hyperparameter prediction.Source: §3 · p. 3

3.1. Impact of model size on hyperparameters

• As model size and complexity increase, the analysis calls for a smaller learning rate and larger batch size, treating model size as an independent predictor.The paper relates this adjustment to the increasing Lipschitz constant and explicitly contrasts its approach with work that treats model size implicitly.Source: §3.1 · p. 4

3.2. Proposed Bopt(N, T) equation

• The optimal batch size depends primarily on model size and training tokens, balancing lower gradient noise per step against fewer update steps.Larger batches reduce gradient-estimate variance, while the fixed compute budget yields fewer total steps as batch size increases.Source: §3.2 · pp. 4–5

3.3. Proposed ηopt(N, T) equation

• The optimal learning rate balances per-step gains against the number of effective updates, with its scaling law depending on model size and training tokens.The paper derives the law under a fixed training-token budget.Source: §3.3 · p. 5

3.4. Results of optimal hyperparameters

• Across fitted configurations, larger models favor larger batches and lower learning rates; with more training tokens, batch size can rise while learning rate decreases moderately.Fits covered model sizes from 0.017B to 0.26B parameters and training-token budgets from 2B to 12B.Source: §3.4 · pp. 5–6
• Grid searches on a 1.07B model with 4B and 10B training tokens confirmed that the predicted learning rate and batch size minimized validation loss.Figure 4 marks the predicted settings and validation losses.Source: §3.4 · p. 6, Figure 4 · p. 6

4.1. Scaling law for optimal model size

• The model-size search fits a parabola to empirical losses at each compute budget, using its minimum to identify the empirical optimal size.Five experiments were conducted at each of five compute budgets.Source: §4.1 · p. 6, Figure 6 · p. 8

4.2. Scaling law for performance

• At fixed compute, the fitted loss relationship predicts validation loss for any model size, including sizes chosen for hardware or inference-cost constraints.The approach uses optimal hyperparameters and extends beyond fitting loss only at the optimal model size.Source: §4.2 · pp. 6–7, §4 · p. 6
• The section also frames loss behavior at infinite model size, token count, and compute, with the infinite-compute loss approaching training-data entropy.The asymptotic statements qualify the performance-scaling formulation.Source: §4.2 · p. 7
• For the performance fit, loss decreases with model size and training tokens; predictions for two tested settings deviate from experiments by 0.03% and 0.15%.Figure 5 presents fitted and extrapolated loss curves under optimal hyperparameters.Source: §4.2 · p. 7, Figure 5 · p. 7

4.3.1 More parameter-efficient model sizes

• For a compute budget of 5.85 × 10^20, a 719.3M-parameter, 14-layer model with batch size 832 and learning rate 1.6 × 10^-4 closely matches predicted loss.The predicted optimal size was 0.64B; discrete layer selection motivated the experimental model choice.Source: §4.3.1 · p. 7, Figure 5 · p. 7
• 39.9% fewer parameters achieve similar validation loss under the optimal-hyperparameter scaling law, compared with fixed suboptimal settings.At 1 × 10^10 TFlops, the reported savings arise when compute is abundant relative to data and model size is near optimal.Source: §4.3.1 · pp. 7–8

4.3.2 More precise validation loss predictions

• Under optimal hyperparameters, validation-loss fitting has 45.5% lower MSE than fitting with fixed suboptimal hyperparameters.MSE falls from 4.31 × 10^-7 to 2.35 × 10^-7.Source: §4.3.2 · p. 8
• Under optimal hyperparameters, predicted and empirical optimal-model-size scaling slopes differ by 3.57%, versus 30.26% with fixed suboptimal hyperparameters.The reported absolute errors in the exponent of compute are 0.0148 and 0.1581, respectively.Source: Figure 7 · p. 8, §4.3.2 · p. 9

Text-to-Video Generation Models

• Text-to-video systems evolved from GANs and VAEs to diffusion, DiT-based, and autoregressive models, with different approaches addressing temporal coherence and long-form generation.The passage associates DiT spatio-temporal attention with capturing complex video dynamics and frame coherence [4] [10] [16] [18].Source: §5 · p. 9

Scaling Laws

• Precise scaling relationships remain unclear in image generation, while scaling-law research in video generation is limited.The passage notes that video temporal complexity may yield scaling behavior different from other modalities.Source: §5 · p. 9
• A concurrent scaling-law study [33] is described as limited by smaller batch sizes and impractical hyperparameter configurations, reducing its applicability.Source: §5 · p. 9

6. Conclusion, Limitation and Future Work

• The study confirms that video DiT models exhibit scaling laws.The authors describe this as foundational support for the scalability of these models.Source: §6 · p. 9
• The proposed scaling laws predict optimal hyperparameters for any model size and compute budget.The conclusion identifies learning rate and batch size as the hyperparameters addressed.Source: §6 · p. 9
• The study provides precise predictions of optimal model size and validation loss under optimal hyperparameters.Source: §6 · p. 9
• The fits use a constant learning rate and validation loss, focus on low-resolution smaller models, and omit resolution and frame-rate effects.Learning-rate decay could yield different results; higher-resolution and larger models require new fitting.Source: §6 · p. 9

A.1. Models

• Experiments use Cross-DiT with a VAE, PixArt-XL-2-512x512 initialization, T5 text encoding, and 17-frame 256x256 inputs.Cross-DiT incorporates cross-attention to integrate text conditions.Source: §A.1 · p. 13

A.2. Datasets

• The experiments use Panda-70M, with 2,000 randomly selected samples reserved for validation.Source: §A.2 · p. 13

A.3. Main Results

• The section reports fitted optimal batch size and learning rate across model sizes and training-token counts.Table 4 is identified as the fitting results for Bopt and ηopt.Source: §A.3 · p. 13, Table 4 · p. 13
• αB = 2.1797 × 104 predicts tokens per batch, while αB = 17.0287 applies to samples with unchanged exponents.Source: §A.3 · p. 13
• The section fits validation loss for any model size and training-token count, and gives optimal model size and tokens for a fixed compute budget.Table 5 is identified as the fitting results for L(T, N).Source: §A.3 · p. 13, Table 5 · p. 13

B.1. Upper Bound of Stochastic Gradient

• The proof assumes the stochastic gradient is an unbiased estimate of G(θk) with bounded variance, then applies those assumptions in the upper-bound derivation.Source: §B.1 · p. 13

B.2. Convergence Rate of Mini-Batch SGD

• The convergence argument takes expectations over the filtration and averages across iterations; Figure 8 compares fitted and extrapolated loss curves under fixed suboptimal hyperparameters.The figure marks a projected 1.07B model trained on 10B tokens alongside experimental results.Source: Figure 8 · p. 14, §B.2 · p. 14

B.3. Stepwise Loss of Mini-Batch SGD

• The appendix approximates the gradient with a batch, whose variance decreases inversely with batch size, and uses the Hessian to approximate loss change before averaging expected updates.The resulting quantity is the expected loss change per update step.Source: §B.3 · p. 14

C. Experimental Conclusions with Fixed Suboptimal Hyperparameters

• The fixed-hyperparameter test uses B = 128 and η = 2.5313 × 10^-4, and tests a 1.07B model on 10B training tokens.The same hyperparameters as in the fitting experiment are used for the 1.07B-model test.Source: C. Experimental Conclusions with Fixed… · p. 14
• Figure 9 compares empirical and Equation (39)-predicted optimal model sizes across compute budgets under fixed suboptimal batch size and learning rate.The figure caption says the comparison confirms Equation (39)’s accuracy in predicting optimal model size.Source: Figure 9 · p. 15, C. Experimental Conclusions with Fixed… · p. 15
• Using optimal hyperparameters reduces fitted-result MSE from 4.31 × 10^-7 to 2.35 × 10^-7, a 45.5% improvement.The authors interpret this as more accurate capture of the model’s power-law performance.Source: C. Experimental Conclusions with Fixed… · p. 15
• Equation (37) differs from Equation (38) by an absolute error of 0.1581, or 30.26%, with a larger discrepancy than the optimal-hyperparameter fitting results.The authors attribute the discrepancy to poor fit of Equation (39), particularly at lower compute budgets.Source: C. Experimental Conclusions with Fixed… · p. 15

D. Parameters and Compute

• Cross-DiT compute estimates sum self-attention, cross-attention, and SwiGLU feed-forward operations; parameter count scales with layer count and model width.For consistency across model sizes, d/nlayer = 128 and the number of attention heads equals nlayer.Source: D. Parameters and Compute · pp. 15–16

• The image-generation experiments use the same setup as video generation but fix generated frames to one.The paper frames image generation as video generation with frames reduced to a single frame.Source: E. Precise Scaling Laws in Image… · p. 16
• The image-generation fitting procedure fits ηopt(N, T) and Bopt(N, T), then derives L(N, T) from the optimal hyperparameters.Tables 8 and 9 are identified as fitting results for the optimal hyperparameters and L(T, N), respectively.Source: E. Precise Scaling Laws in Image… · p. 16, Table 8 · p. 16, Table 9 · p. 16
• For image generation, the extrapolated 1.07B model trained on 2B tokens has predicted loss 0.6414 and actual loss 0.6340, an error of 1.167%.The authors state this demonstrates accurate prediction for image generation.Source: E. Precise Scaling Laws in Image… · p. 16