PaperlayerView exact revision on arXiv

Source-linked AI summary

STARFlow-V: End-to-End Video Generative Modeling with Normalizing Flows

Jiatao Gu, Ying Shen, Tianrong Chen, Laurent Dinh, Yuyang Wang, Miguel Angel Bautista, David Berthelot, Josh Susskind, Shuangfei Zhai

arXiv:2511.20462v2cs.CVcs.LG

TL;DR

Video generation remains dominated by diffusion despite the need for causal, long-horizon modeling and the potential advantages of normalizing flows. STARFlow-V applies autoregressive flows with a global–local latent design and causal denoising to video. It reports coherent, temporally stable generations at practical sampling speeds, while noting that unrealistic non-physical generations remain a limitation.

  • Problem

    Video generation needs causal, robust long-horizon synthesis, yet diffusion dominates and the practicality of normalizing flows for video remains unclear.

    Source: §1 · p. 3

  • Method

    STARFlow-V models video with autoregressive normalizing flows in a global–local spatiotemporal latent space, using flow-score matching and video-aware Jacobi updates.

    Source: §1 · p. 3, §3.1 · p. 5, §3.3 · p. 7

  • Results

    STARFlow-V reports visually coherent, temporally stable generations at practical sampling speeds relative to diffusion models, with consistent gains over autoregressive diffusion baselines at 480p/81f.

    Source: §1 · p. 3, §5 · p. 10

  • Takeaways & Limitations

    The results provide initial evidence that normalizing flows can support high-quality autoregressive video generation and potentially world models.

    Source: §1 · p. 3

    The authors report unrealistic, non-physical generations due to current model scale and available data.

    Source: §5 · p. 11

Abstract

from arXiv · show

Normalizing flows (NFs) are end-to-end likelihood-based generative models for continuous data, and have recently regained attention with encouraging progress on image generation. Yet in the video generation domain, where spatiotemporal complexity and computational cost are substantially higher, state-of-the-art systems almost exclusively rely on diffusion-based models. In this work, we revisit this design space by presenting STARFlow-V, a normalizing flow-based video generator with substantial benefits such as end-to-end learning, robust causal prediction, and native likelihood estimation. Building upon the recently proposed STARFlow, STARFlow-V operates in the spatiotemporal latent space with a global-local architecture which restricts causal dependencies to a global latent space while preserving rich local within-frame interactions. This eases error accumulation over time, a common pitfall of standard autoregressive diffusion model generation. Additionally, we propose flow-score matching, which equips the model with a light-weight causal denoiser to improve the video generation consistency in an autoregressive fashion. To improve the sampling efficiency, STARFlow-V employs a video-aware Jacobi iteration scheme that recasts inner updates as parallelizable iterations without breaking causality. Thanks to the invertible structure, the same model can natively support text-to-video, image-to-video as well as video-to-video generation tasks. Empirically, STARFlow-V achieves strong visual fidelity and temporal consistency with practical sampling throughput relative to diffusion-based baselines. These results present the first evidence, to our knowledge, that NFs are capable of high-quality autoregressive video generation, establishing them as a promising research direction for building world models. Code and generated samples are available at https://github.com/apple/ml-starflow.

1 Introduction

Video generation must combine perceptual quality with spatiotemporal structure, long-horizon robustness, and causal rollouts, but diffusion remains dominant despite challenges for streaming and autoregressive generation. STARFlow-V applies normalizing flows to video, and the reported results offer initial evidence that they can produce coherent, temporally stable videos at practical sampling speeds.

1 Introduction

  • Video generation requires high perceptual quality, rich spatiotemporal structure, long-horizon robustness, and causal rollouts for streaming and interactive use.These capabilities also matter for creative media and emerging world models for gaming, simulation, and embodied AI.Source: §1 · p. 1
  • Diffusion dominates text- and image-conditioned video synthesis, but parallel frame denoising is non-causal and causal variants can face train–test mismatch over long autoregressive horizons.Standard diffusion also requires iterative sampling, while its training simulates noise at different timesteps and frames.Source: §1 · pp. 1, 3
  • Video remains an open test for normalizing flows: despite recent image-generation progress, state-of-the-art video systems almost exclusively use diffusion.The substantially higher complexity and computational cost of video leave the practicality of normalizing flows unclear.Source: §1 · p. 3
  • STARFlow-V is a normalizing-flow video generator that combines end-to-end training with causal, likelihood-based modeling and a global–local latent architecture.Its compact global latent sequence carries long-range temporal context, while local latent blocks preserve within-frame structure; flow-score matching adds a lightweight causal denoiser, and video-aware Jacobi updates parallelize inner refinement.Source: §1 · p. 3
  • Across benchmarks, STARFlow-V produces visually coherent, temporally stable generations at practical sampling speeds relative to diffusion-based models.The authors present this as initial evidence that normalizing flows can support high-quality autoregressive video generation and potentially world models.Source: §1 · p. 3

2 Background

Video generation research faces a tension between diffusion models’ strong performance and the need for causal, long-horizon synthesis. Normalizing flows offer end-to-end likelihood-based modeling, but their video practicality remains less established than their image-generation progress.

2.1 Video Generative Models

  • Diffusion training corrupts frames at sampled noise levels and trains a denoiser, while inference iteratively denoises from Gaussian noise.The passage characterizes the training objective as closely related to a lower bound of log p(x1:N | C) and notes that each update supervises one noise level.Source: §2.1 · p. 3
  • Autoregressive diffusion models address causal synthesis, but exposure bias compounds train–test mismatch and can degrade long-horizon video quality.Self-Forcing seeks to mitigate this through sequential post-training, but is not readily applicable during raw-video pre-training.Source: §2.1 · p. 4

2.2 Autoregressive Normalizing Flows

  • Normalizing flows learn invertible mappings from continuous data to latents and support end-to-end maximum-likelihood training and sampling by inversion.The likelihood objective follows from the change-of-variables formula.Source: §2.2 · p. 4
  • Normalizing flows have historically been viewed as less competitive than diffusion models because of architectural rigidity and training instability.Source: §2.2 · p. 4
  • STARFlow stacks autoregressive flow blocks parameterized by causal Transformers, alternating left-to-right and right-to-left masks to capture bidirectional dependencies.The cited work reports that at least three blocks suffice for universal density modeling.Source: §2.2 · p. 4
  • Evidence for normalizing flows in video remains sparse, with VideoFlow constrained by limited capacity, low resolution, and domain-specific settings.The passage notes that video’s higher spatiotemporal dimensionality makes it more challenging for normalizing flows than image generation.Source: §2.2 · p. 4

3 STARFlow-V

STARFlow-V adapts normalizing flows to video with spatiotemporal architectural and algorithmic changes. Its design combines causal generation with local frame processing, flow-score matching, and faster inference.

3.1 Proposed Model

  • A global–local architecture confines shallow-flow processing to each frame while the deep autoregressive flow models intra-frame and inter-frame dependencies.The deep flow conditions on prior latents, which the authors say mitigates compounding data-space errors.Source: §3.1 · p. 5
  • The model trains end to end by exact maximum likelihood using a change-of-variables objective, without a per-step denoising schedule or surrogate loss.The authors say this simplifies optimization and reduces train–test mismatch.Source: §3.1 · p. 5

3.2 Revisiting Noise-Augmented Training

  • Flow-score matching trains a lightweight causal denoiser alongside the flow to refine noisy samples, addressing noisy, non-causal raw-score denoising.The cited discussion reports that decoder fine-tuning can lose temporal consistency with 3D causal VAEs, while raw score-based denoising can produce speckles and break causality.Source: §3.2 · p. 6

3.3 Fast Inference

  • Block-wise Jacobi iteration yields approximately 15× lower inference latency than standard autoregressive decoding while preserving visual fidelity.Blocks are processed sequentially, with parallel Jacobi updates within each block; the reported convergence is faster in practice for videos with strong global structure.Source: §3.3 · pp. 7–8

3.4 Versatility Across Tasks

  • Without changing the backbone, STARFlow-V supports image-to-video and video-to-video generation through flow-encoding observed frames and populating the KV cache.For longer generation, a sliding-window schedule rebuilds the cache from overlapping latents before continuing autoregressively.Source: §3.4 · p. 8

4 Experiments

STARFlow-V performs in the range of recent causal diffusion baselines on text-to-video benchmarks and produces coherent results across several video-generation tasks. Ablations examine denoiser quality and Jacobi-iteration runtime, including the effects of block size and video-aware initialization.

4.1 Experimental Setup

  • Training uses 70M text-video pairs, adds 400M text-image pairs for joint training, and finetunes on Señorita for video editing.Raw and synthetic captions are mixed at a 1:9 ratio; the editing dataset spans 18 subcategories.Source: §4.1 · pp. 8–9
  • Evaluation uses VBench across 16 dimensions and compares STARFlow-V with WAN-2.1 Causal, Self-Forcing, and NOVA.The video model has 7B parameters and is evaluated at 81 frames, with training resolution progressing from 384p to 480p.Source: §4.1 · p. 9

4.2 Quantitative Results

  • STARFlow-V reaches the performance range of recent causal diffusion baselines on VBench, though it does not match the strongest diffusion-based generators.Its scores substantially narrow the historical gap between normalizing flows and diffusion models for video.Source: §4.2 · p. 9

4.3 Qualitative Results

  • Without additional tuning, the same model produces temporally smooth, visually faithful text-to-video and image-to-video sequences, and handles varied video-editing tasks by changing the instruction.The paper attributes image conditioning to reusing the invertible decoder as an encoder.Source: §4.3 · p. 9
  • In the reported examples, STARFlow-V stays sharp and temporally consistent against autoregressive diffusion baselines, including when extended to 30 seconds beyond its 5-second training horizon.NOVA and Self-Forcing show blurring, color drift, and structural deformation in the extended-generation comparison.Source: §4.3 · p. 10

4.4 Ablation Study

  • Flow-score matching yields better video reconstruction under latent-space noise injection than decoder finetuning or score-based denoising.The comparison covers 1,000 random videos with large motions; the alternatives show frame-to-frame jitter and bright speckles, respectively.Source: §4.4 · p. 10, Figure 5 · p. 10
  • Runtime initially falls as Jacobi block size increases, then rises slightly when blocks become too large.Larger blocks improve intra-block parallelism but require more iterations to converge.Source: §4.4 · p. 10
  • Video-aware initialization reduces runtime across almost all block sizes, supporting a medium first-frame block and larger subsequent blocks.The suggested defaults are 64 for the first frame and 512 for later frames initialized from the previous frame’s converged state.Source: §4.4 · p. 10

5 Conclusion and Limitations

STARFlow-V reports gains over autoregressive diffusion baselines at 480p/81f and supports text-, image-, and video-to-video generation. The conclusion also identifies latency, data quality and scaling, and physical plausibility as limitations, with data curation and model efficiency as future directions.

5 Conclusion and Limitations

  • STARFlow-V shows consistent gains over autoregressive diffusion baselines at 480p/81f across text-to-video, image-to-video, and video-to-video tasks.The conclusion also reports long-horizon coherence, fine-grained controllability, and native likelihood estimation.Source: §5 · p. 10
  • Inference remains far from real time on commodity GPUs despite accelerated sampling.The authors propose more efficient sampling schedules and architectural optimizations to reduce latency.Source: §5 · pp. 10–11
  • Dataset noise and bias bound progress, and the authors observe no clean scaling law under current curation.They plan to revisit curation and active data selection, especially for large-motion and physically grounded scenarios.Source: §5 · pp. 10–11
  • At the current model scale and with available data, generations still include unrealistic, non-physical events.Examples include an octopus passing through a jar wall and a rock appearing beneath a goat as it lands.Source: §5 · p. 11
  • Future work includes distillation and pruning to produce compact student models that retain most of the full system's performance.The authors also identify physically grounded data selection as a direction for improving plausibility and scaling behavior.Source: §5 · p. 11

A.1 Derivation of STARFlow-V.

The derivation establishes the deep autoregressive Gaussian map as a normalizing flow and composes it with a shallow flow. It also describes flow-score matching, training, and sampling procedures.

A.1 Derivation of STARFlow-V

  • The deep block's triangular autoregressive map from u to z is bijective and continuously differentiable when σθ > 0.Its Jacobian is lower triangular, with diagonal entries ∂zi/∂ui = 1/σθ(u<i).Source: §A.1 · p. 15
  • Composing the deep map with the shallow block yields the data density, while the within-frame Gaussian model gains an additional shallow-block log-determinant term.The deep block models tokens in raster or block order, and the shallow-block contribution is Σn log |det JfS(xn)|.Source: §A.1 · p. 15
  • Flow-score matching fits a causal denoiser to approximate the single-step score-based denoising update while avoiding high-frequency artifacts and preserving streamability.The denoiser uses a block-causal mask with at most one-frame look-ahead.Source: §A.1 · p. 15
  • Training noise-augments each minibatch, applies shallow and deep forward maps, and combines the normalizing-flow negative log-likelihood with a flow-score-matching loss.The score target reuses the negative-log-likelihood backward pass and is detached before fitting the denoiser.Source: §A.1 · p. 16
  • Sampling draws a standard-normal latent, inverts the deep block in global autoregressive order, inverts the shallow block, and applies a one-step denoiser correction.The deep inverse computes each position from its conditional mean and scale.Source: §A.1 · p. 16

A.2 Training

The appendix points to a joint training procedure for the flow and its learnable denoiser, then describes conditions and block choices for Jacobi-style inference.

A.2 Training

  • Algorithm 1 trains both the flow and the learnable denoiser.The training procedure is identified as using noise augmentation and flow-score matching.Source: §A.2 · p. 16

A.3 Inference

  • Jacobi iteration converges rapidly when the deep map is sufficiently contractive in u, enabling parallelism within each block.Suggested blocks include spatial tiles per frame with no intra-tile dependencies or even/odd raster groups, preserving the training block-causal mask.Source: §A.3 · p. 16

B.1 Architecture Design

The architecture comparison varies the deep block's width while keeping the shallow stack and denoiser unchanged. The appendix also outlines a block-parallel inversion procedure.

B.1 Architecture Design

  • The Jacobi-style inversion algorithm partitions tokens into non-overlapping blocks and updates positions within a block in parallel.For later blocks, the shown initialization uses the corresponding positions from the past frame.Source: §B.1 · p. 17
  • The shallow stack uses alternating masked affine flows, and the denoiser is an 8-layer Transformer with a block-causal mask.The passage describes these as following the standard design.Source: §B.1 · p. 17
  • The 7B model is initialized from the 3B checkpoint and widens only fD from 3072 to 4096 channels; fS and sϕ remain identical.Table 2 is described as a minimal comparison in which only fD width differs.Source: Table 2 · p. 16, §B.1 · p. 17

B.2 Training Details

STARFlow-V is trained through progressively larger image-video stages and supports mixed-resolution inputs. The appendix also reports autoregressive baseline comparisons and video-to-video fine-tuning.

B.2 Training Details

  • Training uses approximately 20 million videos on 96 H100 GPUs, with gradient updates skipped after step 100 when the norm exceeds 1.The shared configuration specifies a batch size of 96 and AdamW optimization.Source: §B.2 · pp. 17–18
  • Progressive training moves from a 3B text-to-image initialization to 3B 384P/45-frame and 7B 384P/81-frame image-video joint training.The final stage uses 7B at 480P with 81 frames.Source: §B.2 · pp. 17–18
  • Mixed-resolution training preserves native frame aspect ratios by assigning videos to nine bins and adding fps and aspect-ratio tags to captions.The caption template includes the video's fps and original caption alongside its aspect ratio.Source: §B.2 · p. 18

B.3 Baseline Details

  • The autoregressive baselines are WAN 2.1-Causal FT and pure-AR NOVA, which predicts each latent frame with a diffusion head.WAN 2.1-Causal FT uses a WAN2.1-T2V-1.3B base fine-tuned with causal attention masking.Source: §B.3 · p. 18

C.1 Quantitative Comparison with Autoregressive Diffusion baselines

  • STARFlow-V substantially outperforms NOVA AR and WAN 2.1-Causal FT across all VBench dimensions.NOVA AR shows increasing blur and content collapse, while WAN 2.1-Causal FT shows temporal inconsistency and flickering.Source: Table 3 · p. 19, §C.1 · p. 19

C.2 Video-to-Video Generation

  • Video-to-video editing fine-tunes pretrained STARFlow-V on Señorita, pairing 33-frame input videos with 33-frame edited targets at 16 fps.The input and target videos are concatenated along the temporal dimension during fine-tuning.Source: §C.2 · p. 20

C.3 Additional Samples

The section points readers to additional STARFlow-V samples, including text-prompt generations extended across overlapping segments.

C.3 Additional Samples

  • Additional generation samples are provided in Figures 7 to 9.The cited samples include text-prompt generations extended with overlapping frames.Source: §C.3 · p. 20, Figure 9 · p. 21
  • The paper directs readers to its official codebase for further video-generation comparisons.The codebase is available at apple/ml-starflow.Source: §C.3 · p. 20
  • Extended text-prompt samples use 21 latent frames per segment with four latent frames of overlap at 480p and 16 fps.Figure 9 presents two videos generated with this overlapping-segment setup.Source: Figure 9 · p. 21

Related papers

Loading 2511.20462v2…