Source-linked AI summary
HiAR: Efficient Autoregressive Long Video Generation via Hierarchical Denoising
Kai Zou, Dian Zheng, Hongbo Liu, Tiankai Hang, Bin Liu, Nenghai Yu
TL;DR
Autoregressive video diffusion must preserve temporal continuity while limiting quality degradation from accumulated errors. HiAR uses matched-noise hierarchical denoising and forward-KL regularisation, achieving the best overall VBench score and lowest temporal drift among compared methods, with a reported ∼1.8× speedup in its 4-step setting.
Problem
Autoregressive video generation must maintain temporal continuity while preventing distribution drift caused by accumulated errors, but clean contexts can propagate prediction errors with high confidence.
Method
HiAR performs causal generation across all blocks at each denoising step using matched-noise contexts, and adds a forward-KL regulariser in bidirectional-attention mode to preserve motion diversity.
Results
HiAR achieves the best overall VBench result, including a Total score of 0.821, the lowest Drift score of 0.257, and ∼1.8× wall-clock speedup in the 4-step setting.
Takeaways & Limitations
Matched-noise hierarchical denoising mitigates long-horizon degradation while the forward-KL regulariser preserves motion diversity for causal inference.
Abstract
from arXiv · showhide
Autoregressive (AR) diffusion offers a promising framework for generating videos of theoretically infinite length. However, a major challenge is maintaining temporal continuity while preventing the progressive quality degradation caused by error accumulation. To ensure continuity, existing methods typically condition on highly denoised contexts; yet, this practice propagates prediction errors with high certainty, thereby exacerbating degradation. In this paper, we argue that a highly clean context is unnecessary. Drawing inspiration from bidirectional diffusion models, which denoise frames at a shared noise level while maintaining coherence, we propose that conditioning on context at the same noise level as the current block provides sufficient signal for temporal consistency while effectively mitigating error propagation. Building on this insight, we propose HiAR, a hierarchical denoising framework that reverses the conventional generation order: instead of completing each block sequentially, it performs causal generation across all blocks at every denoising step, so that each block is always conditioned on context at the same noise level. This hierarchy naturally admits pipelined parallel inference, yielding a 1.8 wall-clock speedup in our 4-step setting. We further observe that self-rollout distillation under this paradigm amplifies a low-motion shortcut inherent to the mode-seeking reverse-KL objective. To counteract this, we introduce a forward-KL regulariser in bidirectional-attention mode, which preserves motion diversity for causal inference without interfering with the distillation loss. On VBench (20s generation), HiAR achieves the best overall score and the lowest temporal drift among all compared methods.
1 Introduction
HiAR targets long-horizon video generation by matching context noise to the current denoising level, reducing error propagation while preserving continuity. Its hierarchical pipeline also supports pipelined inference and uses forward-KL regularisation to preserve motion diversity.
- Motivation: Long-horizon video generation must preserve temporal continuity while preventing distribution drift from accumulated errors.Reported drift includes oversaturation, over-sharpening, motion repetition, and semantic drift.
- Motivation: Conditioning on fully clean context propagates accumulated prediction errors with high confidence, worsening degradation over time.The context uses noise level tc = 0, which provides maximal signal-to-noise ratio.
- HiAR: HiAR performs causal generation across all blocks at each denoising step, conditioning blocks on matched-noise contexts instead of completing blocks sequentially.This reordering reduces inter-block error transmission and improves long-horizon stability.
- HiAR: 1.8× wall-clock speedup is achieved through pipelined parallel inference across denoising hierarchy levels.The reported speedup is measured in the authors’ implementation and 4-step setting.
- Training: Forward-KL regularisation in bidirectional-attention mode prevents low-motion shortcuts during self-rollout distillation and preserves dynamics.The regulariser is introduced because hierarchical training amplifies the low-motion shortcut of reverse-KL distillation.
- Evaluation: VBench evaluation, drift measurements, and ablations demonstrate long-horizon stability and the effectiveness of HiAR’s components.The evaluation includes 20-second generation and a dedicated drift metric.
2 Background
The background introduces flow matching and autoregressive video diffusion, then describes how causal generation extends beyond fixed temporal windows. It also explains train–test mismatch, exposure bias, and reverse-KL self-rollout distillation.
- 2.1 Diffusion Models and Flow Matching: Flow matching interpolates clean data with Gaussian noise across continuous time and learns a velocity field for reverse-time recovery.The interpolation uses x_t = (1 − σ_t)x_0 + σ_tϵ, with a shift parameter controlling noise-schedule curvature.
- 2.1 Diffusion Models and Flow Matching: A neural network predicts the velocity field, allowing clean data to be recovered by integrating the probability-flow ODE backward through discretised steps.The trajectory is discretised into S steps and updated with Euler integration.
- 2.2 Autoregressive Video Diffusion: Bidirectional diffusion operates on fixed temporal windows, whereas causal autoregressive generation streams frames and can support indefinite extension.Causal generation also supports real-time intervention and interactive control.
- 2.2 Autoregressive Video Diffusion: Autoregressive video generation partitions a latent sequence into successive blocks and denoises each block conditioned on preceding blocks.For block B_n, the context consists of previously generated blocks B_<n.
- 2.2 Autoregressive Video Diffusion: Teacher-forced training uses ground-truth context while inference uses model predictions, creating exposure bias and distribution drift.The drift appears as progressive oversaturation, motion repetition, and semantic drift.
- 2.2 Autoregressive Video Diffusion: Self-rollout training narrows the train–test gap by feeding a student-generated block into subsequent denoising, using reverse-KL DMD against a teacher distribution.The student uses a one-step output distribution while the teacher uses multi-step ODE integration.
3 Method
HiAR selects the noisiest context that still preserves temporal causality, then denoises all blocks causally at each step. Its training combines hierarchical self-rollout with forward-KL regularisation to preserve motion diversity while enabling pipelined inference.
- Context Noise Level and Error Propagation: The optimal context noise level is t∗_c = t_{j+1}, the output noise level of the current denoising step.This choice is the noisiest context that still fulfills temporal causality, attenuating inter-block bias without removing needed information.
- Context Noise Level and Error Propagation: Context noise controls a bias–information trade-off: increasing noise attenuates propagated error but also weakens useful conditioning signal.Prior autoregressive methods use t_c = 0, which preserves maximal signal but propagates prediction errors with high confidence.
- Hierarchical Denoising: Hierarchical Denoising generates causally across all blocks at every denoising step instead of fully denoising each block sequentially.At step j, block B_n uses preceding blocks at noise level t_{j+1} as context.
- Hierarchical Denoising: Blocks on the same anti-diagonal of the N×S grid are mutually independent, enabling pipelined parallel inference.Fused execution reduces the cost to N+2 passes per stage and yields an approximately 1.8× wall-clock speedup in the 4-step setting.
- Training with Forward-KL Regulation: Hierarchical self-rollout amplifies a low-motion shortcut from reverse-KL distillation, so HiAR adds a forward-KL regulariser to preserve motion diversity.The regulariser uses teacher trajectory checkpoints and is computed only in bidirectional-attention mode, leaving the causal DMD loss unmodified.
- Training with Forward-KL Regulation: The forward-KL regulariser is applied only during the first K of S denoising steps, leaving later high-frequency refinement unconstrained.The objective combines DMD loss and forward-KL loss as L = L_DMD + λ L_FKL, with λ > 0 balancing the terms.
4 Experiments
HiAR is evaluated against diverse video-generation baselines using 20-second VBench, drift, efficiency, qualitative, and ablation analyses. It achieves strong quality and stability while showing how context-noise and forward-KL design choices affect motion, drift, and train–test alignment.
- Quantitative Results: HiAR achieves the highest VBench Total score (0.821), best Quality score (0.846), and strong Semantic score (0.723) among the compared methods.Its Dynamic score is 0.686, close to the Wan2.1-1.3B teacher’s 0.690 and above Causal Forcing (0.672) and Self-Forcing (0.542).
- Quantitative Results: 0.257 Drift is lowest among distilled AR models, a 27.6% reduction relative to Self-Forcing’s 0.355 over 20 seconds.CausVid has the highest drift at 0.842, while Causal Forcing reaches 0.615.
- Inference Efficiency: 1.8× wall-clock speedup comes from pipelined parallelism, reaching 30 fps and 0.30 seconds per chunk versus 17 fps and 0.69 seconds for other distilled AR models.The comparison uses the same Wan2.1-1.3B backbone and 4-step denoising schedule.
- Qualitative Results: HiAR maintains stable colour fidelity, sharpness, and structural coherence across 20-second qualitative comparisons, whereas baselines develop colour oversaturation, hue drift, and detail loss.The comparison covers six prompts spanning scenery, objects, and human subjects; degradation is especially pronounced in human-centric content for the baselines.
- Context Noise Ablation: Matching context noise to the current input level minimizes drift at 0.184 but lowers Quality to 0.799 and Smooth to 0.978 by removing intra-step causality.The default output-noise context preserves smoothness near Self-Forcing (0.988 versus 0.991) while improving quality and reducing drift.
- Forward-KL Ablation: Applying forward-KL in causal mode lowers Dynamic to 0.625 versus 0.686 for the bidirectional-attention default, while increasing constrained steps yields only marginal dynamics gains and worsens quality and drift.A single constrained step is sufficient and optimal; removing forward-KL causes dynamics collapse to 0.445, and omitting retraining reduces Quality to 0.767.
5 Conclusion
HiAR reduces distribution drift in autoregressive long-video generation by conditioning blocks at matched noise levels rather than on fully clean contexts. This reordering also supports faster inference and, with forward-KL regularization, preserves motion diversity and improves 20-second generation quality.
- HiAR attenuates inter-block error propagation while preserving temporal causality by conditioning each block on context at a matched noise level.
- 1.8× wall-clock speedup is achieved through pipelined parallel inference in the 4-step setting.
- A forward-KL regulariser in bidirectional-attention mode counteracts reverse-KL distillation’s low-motion shortcut without interfering with the DMD objective.
- On 20-second VBench generation, HiAR achieves the best overall quality and the lowest temporal degradation among compared methods.