PaperlayerView exact revision on arXiv

Source-linked AI summary

History-Guided Video Diffusion

Kiwhan Song, Boyuan Chen, Max Simchowitz, Yilun Du, Russ Tedrake, Vincent Sitzmann

arXiv:2502.06764v2cs.LGcs.CV

TL;DR

Variable-length history is difficult to use for classifier-free guidance because existing video architectures impose fixed conditioning and binary history dropout performs poorly. The paper proposes DFoT, which uses independently noised frames to enable arbitrary history conditioning, and introduces History Guidance. The methods improve video quality, consistency, and motion dynamics, with history guidance across time and frequency producing an 862-frame rollout from a single image.

  • Problem

    Existing video diffusion architectures do not support flexible history conditioning, while CFG-style dropout of history subsets performs poorly.

  • Method

    DFoT trains a video diffusion transformer with independent noise levels per frame, enabling arbitrary history masking and History Guidance score composition.

  • Results

    History Guidance improves video quality, consistency, and motion dynamics, while HG-tf generates an 862-frame video from a single image.

  • Takeaways & Limitations

    DFoT and History Guidance support stable, exceptionally long autoregressive video generation and can be efficiently fine-tuned from existing models.

  • Takeaways & Limitations

    Framewise binary history dropout performs poorly, and video generation carries potential misuse risks requiring responsible use.

Abstract

from arXiv · show

Classifier-free guidance (CFG) is a key technique for improving conditional generation in diffusion models, enabling more accurate control while enhancing sample quality. It is natural to extend this technique to video diffusion, which generates video conditioned on a variable number of context frames, collectively referred to as history. However, we find two key challenges to guiding with variable-length history: architectures that only support fixed-size conditioning, and the empirical observation that CFG-style history dropout performs poorly. To address this, we propose the Diffusion Forcing Transformer (DFoT), a video diffusion architecture and theoretically grounded training objective that jointly enable conditioning on a flexible number of history frames. We then introduce History Guidance, a family of guidance methods uniquely enabled by DFoT. We show that its simplest form, vanilla history guidance, already significantly improves video generation quality and temporal consistency. A more advanced method, history guidance across time and frequency further enhances motion dynamics, enables compositional generalization to out-of-distribution history, and can stably roll out extremely long videos. Project website: https://boyuan.space/history-guidance

1 Introduction

The paper asks whether variable portions of video history can guide generation, then introduces DFoT and History Guidance to address architectural and training limitations. The methods improve video quality and consistency, including stable 862-frame rollout from a single image.

  • Variable-length history, frame subsets, and image-domain frequencies are proposed as guidance variables for video generation.
  • DFoT enables conditioning on any portion of input history by applying independent noise levels to each video frame.The framework supports selective history masking during sampling and is compatible with DiT and U-ViT architectures.
  • Vanilla History Guidance uses arbitrary-length history for CFG, while Temporal and Fractional History Guidance combine different history windows or noise levels.Fractional guidance acts as a low-pass filter on historical frames.
  • 862 frames were generated from a single image using history guidance across time and frequency with DFoT.The navigation video came from a test-set video unseen during training and exceeded the training-set maximum length.
  • The proposed methods improve quality, consistency, and motion dynamics while supporting exceptionally long autoregressive video generation.The paper reports performance beyond the de facto standard DiT diffusion baseline and on par with industry models trained with more compute.

2 Preliminaries and Related Work

The paper situates history guidance within diffusion modeling, classifier-free guidance, and independently noised sequence diffusion. Existing video models typically use fixed or limited frame conditioning, motivating a more flexible formulation.

  • Diffusion models learn to reverse a noise-adding process by estimating score functions for iterative denoising.The forward process transforms data into white noise across increasing noise levels.
  • Classifier-free guidance jointly trains conditional and unconditional diffusion models and combines their score estimates during sampling.The method is commonly used to improve conditional sample quality while trading off quality and diversity.
  • Video diffusion models commonly condition on text, a first frame, or a fixed set of frames.These conditioning schemes do not directly provide arbitrary history guidance.
  • The paper generalizes CFG to variable numbers of conditioning frames and composes scores from multiple conditioning contexts.This approach also removes reliance on binary-dropout training as the default mechanism for CFG.
  • Diffusion Forcing independently varies noise levels across sequence frames, and this work extends that idea to non-causal transformer architectures.CausVid previously scaled Diffusion Forcing to a causal transformer.

3 Challenges when Guiding with History

History guidance differs from conventional video conditioning because generated frames can become future history and the history set may vary arbitrarily. Existing architectures and framewise dropout create corresponding implementation and training challenges.

  • Video history is the set of observed frames used to condition generation, while the remaining frames are generated.The objective models p(x_G|x_H) for history indices H and generation indices G.
  • Generated frames can be reused as history for subsequent autoregressive generation, and history may be any subset of frames.Therefore, the model must estimate conditional and unconditional scores for arbitrary subsets.
  • Conventional DiT and U-Net designs constrain conditioning to a fixed-size vector through AdaLN injection or channel concatenation.Sequence encoders can support variable-length inputs but may be pretrained and parameter-separated from the diffusion model.
  • History Guidance composes scores conditioned on clean, fully masked, subset-masked, or partially masked histories.Noise serves as masking, allowing shorter or low-frequency history conditions.
  • Framewise binary dropout performs poorly for history guidance in the reported ablations.The paper hypothesizes inefficient token utilization because only generation-frame subsets contribute to the loss despite processing all frames.

4 The Diffusion Forcing Transformer

DFoT treats history and target frames as one sequence and trains on independently varying per-frame noise levels. This supports arbitrary history conditioning, theoretically corresponds to a reweighted ELBO, and underlies History Guidance.

  • Noise levels act as masks: clean frames retain full information, fully noisy frames are masked, and intermediate levels preserve partial information.This interpretation enables history to be represented through noise-free, partially masked, or fully masked frames.
  • DFoT unifies history and generated frames as inputs to a full-sequence transformer rather than separating history as conditioning.Variable-length history is obtained by varying noise levels within each sequence.
  • Each frame receives an independent noise level, and the model is trained to predict noise across the entire sequence.The training objective does not distinguish past from future frames and supports parallel training.
  • The DFoT training objective optimizes a reweighting of an Evidence Lower Bound on expected log-likelihoods.
  • DFoT outperforms generic diffusion baselines under the same architecture and is on par with industry models trained with more compute on Kinetics-600.The table also distinguishes single predefined, approximate arbitrary, and arbitrary history conditioning.
  • DFoT supports sampling with arbitrary history by feeding noisy generation frames together with clean history frames.Standard score-based samplers such as DDPM or DDIM can then be used.

5 History Guidance

History Guidance builds on DFoT to condition video generation on flexible history subsets, then composes scores across temporal and frequency axes to improve quality, consistency, dynamics, and robustness.

  • History Guidance is a family of history-conditioned video-generation techniques enabled by DFoT.
  • Vanilla History Guidance: Vanilla History Guidance applies CFG with an arbitrary history length by treating fully masked history as the unconditional condition.
  • Temporal History Guidance: Composing scores from different history subsequences reduces reliance on a single long, potentially out-of-distribution context.
  • Fractional History Guidance: Fractional History Guidance masks history fractionally so low-frequency consistency is guided while high-frequency details and fast motion remain less constrained.
  • History Guidance across Time and Frequency: Fractional History Guidance improves motion dynamics without sacrificing consistency, while DFoT with temporal guidance produces accurate samples under OOD history.

6 Experiments

Experiments evaluate DFoT and History Guidance across standard generation, flexible history, OOD histories, long contexts, imitation learning, and long rollouts. DFoT outperforms baselines while guidance improves quality, consistency, dynamics, robustness, and long-horizon generation.

  • DFoT Evaluation: DFoT outperforms standard diffusion and other baselines without guidance, while supporting flexible history conditioning without sacrificing task-specific performance.
  • DFoT Evaluation: Binary-dropout diffusion supports flexible history but suffers a significant performance drop and produces artifacts and inconsistent generations.
  • DFoT Evaluation: DFoT generates consistent, high-quality samples across future prediction, frame interpolation, and mixed-history tasks on RE10K.
  • DFoT Evaluation: Fine-tuning a full-sequence model into DFoT using 12.5% of the training cost yields performance comparable to DFoT trained from scratch.
  • History Guidance Evaluation: Vanilla guidance achieves its best FVD at ω=1.5, while stronger guidance improves quality and consistency but eventually reduces diversity.
  • History Guidance Evaluation: Fractional guidance lowers vanilla guidance’s best FVD from 181.6 to 170.4 and increases dynamics while maintaining quality.
  • New Abilities: Temporal guidance handles OOD histories, improves Minecraft FVD from 97.63 to 79.19, and supports long-horizon reactive imitation learning.
  • Long Video Generation: DFoT with History Guidance extends a single image into an 862-frame video through autoregressive rollout.

7 Conclusion

The paper concludes that DFoT’s flexible history conditioning enables History Guidance, which improves video quality, consistency, and motion while supporting OOD robustness and compositional generation.

  • DFoT and History Guidance enhance generation quality, consistency, and motion degree while enabling robustness to OOD histories and compositional video generation.

Impact Statement

The impact statement highlights potential benefits of longer, higher-quality video generation alongside risks from misuse and the need for responsible use.

  • The approach may support longer, higher-quality videos with potential applications in robotics and other fields.
  • The authors acknowledge misuse risks, including harmful or unethical content generation, and emphasize ethical considerations and responsible use.

A.1 Derivation of an ELBO

The section derives an ELBO for models and forward processes that factor along monotonic noising paths, then specializes the result to Gaussian diffusion. It also connects the framework to frequency attenuation and history-guidance score composition, while noting unresolved cross-path consistency.

  • Path-based ELBO: The derivation defines a path as a sequence from zero noise to full noise and uses paths with non-decreasing per-frame noise levels.Examples include autoregressive diffusion, full-sequence diffusion, and paths with skips or linearly increasing noise.
  • Path-based ELBO: Under the factoring property, the model and forward process factor along a path, yielding a general ELBO up to a constant independent of the model parameters.The proof applies the standard ELBO construction, Bayes’ rule, and Jensen’s inequality.
  • Gaussian diffusion: The general result specializes to Gaussian diffusion through standard likelihood and KL computations, with x-, ϵ-, and v-prediction formulations handled analogously.The presentation focuses on x-prediction and notes that the other parameterizations can be derived similarly.
  • Limitations: The learned model’s factorizations may be inconsistent across different paths, and enforcing explicit cross-path consistency remains open for future work.The forward process factorizes across the considered paths, but the corresponding model factorizations need not agree.
  • Frequency interpretation: Gaussian noising attenuates weaker frequency components more strongly as noise increases, without intrinsically preferring lower frequencies.For natural images, the weaker components often correspond to high frequencies, but this interpretation is domain-dependent.
  • History guidance: History Guidance across time and frequency combines diffusion scores conditioned on different history lengths and partially noisy history, with a probabilistic interpretation for the score addition.The method samples with linearly weighted diffusion scores across these conditioning choices.

C.1 Datasets

The evaluation uses four video and imitation-learning datasets spanning action recognition, camera-controlled walkthroughs, long-context navigation, and reactive fruit rearrangement. Experiments combine pixel and latent diffusion models with standard video-generation metrics.

  • Datasets: Kinetics-600 contains approximately 400K action videos across 600 classes and is evaluated at 128 × 128 resolution.It serves as a standard benchmark for history-conditioned video generation.
  • Datasets: RealEstate10K provides home walkthrough videos with camera-pose annotations and is used at 256 × 256 resolution.Camera poses support controlled stochasticity and 3D consistency evaluation.
  • Datasets: Minecraft contains 200K gameplay videos of 300 frames with forward, left, and right action labels, requiring long context for strong FVD.The dataset includes trajectories where players turn around and revisit areas.
  • Datasets: Fruit Swapping contains 300 expert demonstrations of a robot rearranging two fruits among three slots, requiring long-horizon memory and local reactivity.Adversarial perturbations make memorizing all waypoint combinations impractical.
  • Implementation and metrics: The study validates DFoT in both pixel and latent diffusion and reports FVD, VBench, and LPIPS for video quality, consistency, and perceptual similarity.Latent diffusion is used for Kinetics-600 and Minecraft, while LPIPS is reserved for highly deterministic tasks.

C.4 Details on Video Generation Benchmark (Section 6.2)

The benchmark evaluates history-conditioned prediction, long-rollout generation, OOD-history generalization, and robot imitation under matched experimental pipelines. DFoT is compared with literature baselines and history-guidance variants across these settings.

  • Video generation benchmark: Kinetics-600 prediction conditions on 5 history frames and predicts the next 11 frames using FVD-based evaluation.The rollout experiment instead generates 59 frames with sliding windows and evaluates 1,024 generated 64-frame videos.
  • Video generation benchmark: DFoT outperforms generic diffusion baselines and remains competitive with selected industry-level literature baselines trained with more computational resources.The comparison includes Video Diffusion, MAGVIT, MAGVIT-v2, W.A.L.T, and Rolling Diffusion.
  • OOD-history evaluation: RealEstate10K OOD histories are defined by camera rotations wider than the training distribution, with rotations above 140° classified as OOD.Models interpolate four frames from a four-frame history with varying rotation angles.
  • History guidance: Temporal history guidance composes scores from overlapping history subsequences, while the long-video experiment trains a 50-frame DFoT with up to 25-frame conditioning.The model autoregressively rolls out five 25-frame segments for 125 total frames.
  • Robot imitation: Robot imitation compares Markov, 2-frame, full-history, and composed-guidance models on a task requiring both long-term memory and local reactive behavior.The composed method uses 1-frame, 4-frame, and full-history models with weights 0.45, 0.45, and 0.2.
  • Long-video generation: Long navigation-video generation separates autoregressive rollout from interpolation and uses user-specified camera-motion inputs rather than ground-truth poses.During rollout, the model usually conditions on the last four frames to generate the next four.

D.1 Additional Results on Fine-tuning to DFoT

Fine-tuning a pretrained full-sequence model provides an efficient initialization for DFoT, although training-loss improvements do not immediately translate into better FVD. DFoT also substantially outperforms Binary-Dropout Diffusion when both use history guidance.

  • Fine-tuning: Fine-tuning from a pretrained full-sequence model reaches training-loss levels comparable to scratch training in approximately 40k iterations, yielding an approximately 10x speedup.The fine-tuned model achieves lower training loss than the scratch model after 80k iterations.
  • Fine-tuning: EMA is important for sample quality: at 80k iterations, FVD is 4.7 with EMA versus 7.3 without EMA.The fine-tuned model can have slightly higher FVD than scratch training despite lower training loss.
  • Binary-dropout ablation: At optimal guidance scale ω = 1.5, DFoT with vanilla history guidance achieves FVD 181.6 versus 196.0 for Binary-Dropout Diffusion.DFoT consistently outperforms BD across guidance scales except ω = 2.5.
  • Binary-dropout ablation: DFoT with fractional history guidance further reaches FVD 170.4, a method unavailable to Binary-Dropout Diffusion.The comparison supports DFoT as a stronger base model for both performance and guidance-method variety.
  • Additional results: Temporal guidance improves the long-horizon robot benchmark from FVD 97.625 for full-context conditional diffusion to 79.19.Lower FVD is better, and the metric is considered suitable here because action conditioning constrains variance.

D.4 Detailed Results on Long-horizon yet Reactive Imitation Learning (Section 6.4, Task 3)

The fruit-swapping task tests whether a policy can preserve the target plan while reacting to perturbed fruit locations. Short-context and naive full-history policies each fail for different reasons, whereas composed DFoT guidance achieves an 83% success rate.

  • Baseline behavior: Markov and 4-frame baselines cannot complete the task because they lack sufficient memory to maintain a consistent plan.The 4-frame model still reacts to perturbations locally but does not finish the rearrangement.
  • Task setup: The task requires remembering the initial fruit configuration while responding to within-slot perturbations that are combinatorially absent from the demonstrations.The setup uses two fruits, three slots, and adversarial human perturbations during execution.
  • Baseline behavior: The full-history model performs well without perturbations but often overfits to the demonstrated center trajectory when fruits are moved toward slot edges.It may close its gripper at the memorized center location after the fruit has moved elsewhere.
  • Composed guidance: DFoT composed guidance achieves an 83% success rate by combining long-context memory with locally reactive behavior.The reported interpretation is that it stitches subtrajectories while borrowing memory from the full-context model.

D.5 Additional Qualitative Results

Additional qualitative results show that DFoT and History Guidance improve consistency, quality, robustness to out-of-distribution history, and stability during extremely long video rollouts.

  • DFoT vs. Baselines: DFoT produces more consistent and higher-quality samples than all baselines in qualitative comparisons.These results are consistent with the quantitative findings in Table 1.
  • Empirical Flexibility of DFoT: DFoT successfully generates consistent samples from histories varying in both length and timestamps.The training objective supports flexible performance across diverse history-conditioned tasks.
  • Improving Video Generation via History Guidance: Increasing vanilla history-guidance scales yields higher frame quality and consistency across extrapolation and showcasing tasks.The qualitative behavior is reported on RealEstate10K and is consistent with results on Kinetics-600.
  • Robustness to Out-of-Distribution History: HG-t enables DFoT to remain robust to out-of-distribution history by composing in-distribution short history windows.Baselines show failures when history is out of distribution.
  • Long Rollout: DFoT with History Guidance stably generates extremely long navigation videos beyond the lengths represented in its training data.Additional samples cover multiple connected rooms or areas, whereas the training videos primarily cover a single room or small area.
Loading 2502.06764v2…