PaperlayerView exact revision on arXiv

Source-linked AI summary

Flex-Forcing: Towards a Unified Autoregressive and Bidirectional Video Diffusion Model

Xinyin Ma, Julius Berner, Chao Liu, Arash Vahdat, Weili Nie, Xinchao Wang

arXiv:2607.03509v1cs.CV

TL;DR

Existing video diffusion models typically trade global coherence for inference efficiency, lacking a unified way to support both bidirectional and autoregressive generation. Flex-Forcing uses flexible chunking across frames and denoising steps, achieving a stronger quality–efficiency trade-off and outperforming Self-Forcing across short- and long-video benchmarks.

  • Problem

    Existing methods lack an effective mechanism to jointly realize bidirectional coherence and autoregressive efficiency within one video-generation framework.

  • Method

    Flex-Forcing uses flexible temporal and denoising-step chunking, with bidirectional dependencies within chunks and autoregressive generation across chunks.

  • Results

    Flex-Forcing substantially outperforms Self-Forcing across short- and long-video benchmarks while achieving a stronger quality–efficiency trade-off than causal models and competitive performance with bidirectional models.

  • Takeaways & Limitations

    Bidirectional and autoregressive video generation can function as a controllable inference dimension within a single model, enabling adjustable quality–speed trade-offs.

  • Takeaways & Limitations

    Training–inference mismatch can still accumulate errors in long videos, and effectiveness may depend on bidirectional priors inherited from pre-training.

Abstract

from arXiv · show

Recent progress in large-scale generative models has substantially advanced video generation, yet existing methods remain constrained by a rigid inference paradigm. Bidirectional diffusion models excel at global coherence and visual fidelity but suffer from slow inference, while autoregressive models offer efficient and streaming generation at the cost of long-range consistency and exposure bias. We introduce Flex-Forcing, a unified training and inference framework that enables a video diffusion model to seamlessly operate under both bidirectional and autoregressive generation regimes. The core idea is a flexible chunking mechanism jointly defined over the temporal axis and denoising steps. This design allows the model to (1) perform flexible chunking according to different device budgets, (2) perform bidirectional inference across chunks for global structure planning, while generating frames autoregressively within each chunk for efficient and fine-grained synthesis, and (3) perform any-order, any-timestep autoregressive generation without the strict causal constraint. Extensive experiments on multiple video generation benchmarks demonstrate that Flex-Forcing achieves consistently better video quality, long-video stability than strong baselines with a rigid inference schedule, while offering faster inference.

1. Introduction

Flex-Forcing unifies bidirectional and autoregressive video generation in one model through flexible chunking across temporal frames and denoising steps. This design supports intermediate inference regimes, improves the quality–efficiency trade-off, and enables controllable any-order, any-step autoregressive editing.

  • Motivation: Bidirectional diffusion captures long-range dependencies but incurs high inference costs, whereas autoregressive generation is efficient but relies on causal conditioning.The two paradigms provide complementary strengths but impose different limitations for real-time and long-video generation.
  • Unified framework: Flex-Forcing unifies bidirectional and autoregressive video generation within a single model instead of treating them as mutually exclusive paradigms.The framework is designed to achieve coherent video generation and efficient long-video inference without increasing exposure bias.
  • Flexible Chunking: Flexible Chunking spans temporal frames and denoising steps, with autoregressive and bidirectional inference as its two extreme cases.Applying the formulation consistently during training and inference exposes the model to mixed causal and non-causal conditioning contexts.
  • Results: The flexible inference mode achieves a strong quality–efficiency trade-off and substantially outperforms Self-Forcing across short- and long-video benchmarks.It yields a markedly improved Pareto frontier, while outperforming causal models on the performance–efficiency trade-off and remaining competitive with bidirectional models.
  • Video editing: Any-order, any-step autoregressive editing enables controllable editing of videos.This extends the flexible inference paradigm beyond generation to downstream video-editing applications.

2. Related Work

Prior video-generation research includes bidirectional diffusion models that jointly denoise frames with full temporal context and autoregressive models that sequentially predict frames. Autoregressive research also includes causal-distillation approaches using diffusion forcing, causal attention masks, DMD, and self-rollout.

  • Bidirectional models for video generation: Bidirectional video diffusion models extend image diffusion to spatio-temporal generation by jointly denoising all frames with full temporal context.These models typically use bidirectional self-attention over the temporal axis.
  • Autoregressive models for video generation: Autoregressive video models sequentially predict frames, enabling real-time generation, prompt control, and long video generation.VideoGPT is identified as an example using a GPT-like model for autoregressive video generation.
  • Autoregressive models for video generation: Causal-distillation approaches initialize models with diffusion forcing and causal attention masks, then train with DMD and self-rollout.The cited methods combine these components in a causal-distillation training pipeline.

3. Flex-Forcing

Flex-Forcing unifies bidirectional and autoregressive video diffusion through flexible temporal chunking whose partitions can vary across denoising steps. Its training procedure supports mixed causal and non-causal dependencies and aligns key states across noise levels for stable flexible inference.

  • Framework: Flex-Forcing is a unified framework in which bidirectional and autoregressive inference coexist within one trained video diffusion model through flexible chunking.The chunking strategy is defined along temporal and denoising-step axes and is learned by post-training a bidirectional diffusion model.
  • Flexible chunking: Flexible chunks model dependencies bidirectionally within chunks while later chunks are generated autoregressively conditioned on earlier chunks.The frame partition can vary at each denoising timestep, adapting the factorization to the generation process.
  • Pyramid chunking: The pyramid strategy uses larger temporal chunks during noisy early denoising and progressively smaller chunks during later detail refinement.Nested partitions preserve existing boundaries while inserting additional ones as denoising proceeds.
  • Training: Training combines causality injection with non-causal future-chunk attention and stochastic chunking that exposes the model to causal, bidirectional, and intermediate dependencies.The two-stage pipeline uses ODE initialization followed by asymmetric DMD distillation with dynamically sampled chunk partitions.
  • Noise-level alignment: K-Projection maps clean-input key states into the current timestep’s noisy latent space, making attention representations noise-consistent across causal and non-causal contexts.The timestep-conditioned projection is applied on the fly while preserving cached KV tensors and their efficiency benefits.

4. Applications of Flex-Forcing

Flex-Forcing supports adaptive hybrid inference that combines autoregressive generation within chunks with bidirectional inference across chunks, while enabling any-timestep and any-order autoregressive editing. These applications improve the quality–efficiency trade-off, favor front-loaded chunk layouts, and preserve consistency during editing.

  • Hybrid and Flexible Inference: Flex-Forcing combines autoregressive and bidirectional inference with variable chunk sizes tailored to compute budgets and video lengths.This hybrid paradigm offers a more favorable trade-off between quality and efficiency.
  • Hybrid and Flexible Inference: Brute-force search over three-chunk partitions of 21 latent frames shows that uniform chunking is not optimal.Even frame partitions can perform markedly worse than asymmetric alternatives.
  • Hybrid and Flexible Inference: Chunk layout causes large performance differences beyond exposure bias, even when configurations use the same number of exposure rounds.Performance favors larger early chunks and smaller later chunks, which can occasionally surpass bidirectional inference.
  • Autoregressive Editing: Any-timestep editing restricts changes to low-level refinement timesteps while preserving high-level planning timesteps, improving consistency over baseline autoregressive editing.This decouples global structure formation from local detail modification, whereas Self-Forcing can propagate small local edits across frames.
  • Autoregressive Editing: Any-order editing attends to past and future tokens, allowing arbitrary temporal segments to be re-edited independently of their temporal order.After full-video generation, target chunks can be edited without regenerating the entire sequence, relaxing standard autoregressive causality.

5. Experiments

Experiments show that Flex-Forcing improves video quality, long-range coherence, and inference efficiency across short- and long-video settings. Hybrid chunking and K-projection further provide favorable quality–efficiency trade-offs and stable performance across inference configurations.

  • Performance on 5s videos: Flex-Forcing produces more coherent long-range motion and better alignment with the given instruction than Self-Forcing.Figure 7 compares the methods on generated 5-second videos at 0, 2.5, and 5 seconds.
  • Performance on 5s videos: Flex-Forcing consistently outperforms baselines in inference efficiency and video quality on 5-second videos.The evaluation uses VBench quality metrics and FPS for 81-frame generation on GB200.
  • Performance of hybrid chunking over timesteps: Hybrid chunking over denoising timesteps achieves a more favorable efficiency–quality trade-off and a better Pareto frontier than temporal-only chunking.The study compares chunking configurations [21], [14, 7], and [7, 7, 7] under identical conditions.
  • Performance on 30s videos: On 30-second videos, Flex-Forcing outperforms Infinity-RoPE across most evaluation metrics while achieving faster inference.The largest gains occur in dynamic degree, and improvements primarily stem from video quality rather than semantic alignment.
  • User preference study: Users prefer Flex-Forcing to Self-Forcing in both 5-second and 30-second settings, especially for visual quality.The preference advantage is reported as more significant for the visual-quality dimension.
  • Ablation study: impact of K-projection: K-projection consistently improves performance across inference configurations, with the largest gains near the fully bidirectional regime.Without K-projection, performance degrades as chunk size increases; with it, performance remains stable and improves smoothly.

6. Conclusion

Flex-Forcing treats bidirectional and autoregressive video generation as a controllable inference dimension rather than competing paradigms. Flexible chunking and noise-level bridging let the model adapt its causal structure at test time for globally coherent and efficient long-video generation.

  • Flex-Forcing reframes bidirectional and autoregressive generation as a smoothly adjustable inference dimension controlled by requirements.
  • Flexible chunking spans both temporal and denoising axes, bridging noise differences between causal and non-causal contexts.
  • Test-time causal-structure adaptation enables better globally coherent generation and efficient long-video synthesis.

Limitations

Flex-Forcing does not fully eliminate training–inference mismatch, and its effectiveness may depend on pretrained bidirectional encoding priors, limiting transferability to models lacking them.

  • Limitations: Training–inference mismatch remains unresolved and can accumulate errors during long-video generation.The method relaxes strict left-to-right training constraints but does not fully remove the mismatch.
  • Limitations: Effectiveness depends heavily on capabilities inherited from pre-training, especially bidirectional encoding priors.This dependence may limit transferability to models where such priors are weak or absent.

Impact Statement

Flex-Forcing unifies bidirectional and autoregressive video generation in one model, reducing the cost and environmental footprint of separate architectures. Its runtime flexibility supports resource-constrained high-fidelity long-form generation and enables localized editing.

  • Impact Statement: Flex-Forcing combines bidirectional and autoregressive paradigms in a single model, reducing the computational cost and environmental footprint of maintaining separate specialized architectures.The unified design replaces separate architecture maintenance with one model.
  • Impact Statement: Dynamic runtime trade-offs between speed and quality make high-fidelity, long-form video generation more accessible in resource-constrained environments.The framework is intended to broaden access to long-form generation when computational resources are limited.
  • Impact Statement: Order-agnostic generation provides precise, localized editing tools for creative and industrial applications.The passage identifies localized editing as an application of the model’s order-agnostic capabilities.

B. Qualitative results on flexible chunks on denoising timesteps.

This section illustrates flexible chunking across denoising timesteps and explains its execution dependencies. The first block runs bidirectionally immediately, while later blocks may wait for required KV caches from earlier frames.

  • Flexible chunking across denoising timesteps: The 0-th block is bidirectional and can execute immediately because it requires no prior KV cache.Its bidirectional operation makes it independent of any prior KV cache.
  • Flexible chunking across denoising timesteps: Block 4 must wait for Block 3 to finish because KV caches from earlier frames are unavailable, despite intermediate representations from Block 1.This demonstrates that flexible timestep chunking is constrained by cache availability across earlier frames.
  • Flexible chunking across denoising timesteps: Flexible chunking across denoising timesteps defines an execution order for the blocks.Figure 13 labels each block with the order in which it is executed.
  • Flexible chunking across denoising timesteps: Figure 15 provides visual comparisons between Flex-Forcing and Self-Forcing.The supplied passage identifies the comparison but does not state a specific qualitative outcome.

C. Case Study

The case study compares Self-Forcing with Flex-Forcing and examines how NFE=5 versus NFE=3 affects generation quality and efficiency.

  • C. Case Study: The study compares Self-Forcing and Flex-Forcing, then evaluates NFE=5 and NFE=3 to analyze denoising-step effects on generation quality and efficiency.These comparisons are presented in Figures 15 and 16.

D. Impact Statement

Flex-Forcing unifies bidirectional diffusion and autoregressive generation in one flexible video model, enabling inference-time trade-offs among quality, efficiency, and controllability while reducing the need for multiple specialized models.

  • Unified generation framework: Flex-Forcing unifies bidirectional diffusion and autoregressive generation, traditionally treated as mutually exclusive, within a single flexible model.This unified formulation advances the foundations of video generation.
  • Inference-time flexibility: The framework enables dynamic inference-time trade-offs among generation quality, efficiency, and controllability.These trade-offs allow the generation regime to be adapted without changing the underlying model.
  • Efficiency and sustainability: By reducing reliance on multiple specialized models, the framework lowers computational cost and environmental footprint.The impact follows from supporting both generation paradigms within one model.
Loading 2607.03509v1…