PaperlayerView exact revision on arXiv

Source-linked AI summary

Conditional Residual Prediction: Improving Autoregressive Video Diffusion without a Bidirectional Teacher

Bowen Zheng, Zhiguang Liu, Jiarong Ou, Rui Chen, Tianyang Hu

arXiv:2610.11479v1cs.CVcs.AIcs.LG

TL;DR

Causal video models suit streaming and long-video generation but often have lower quality than comparable bidirectional models, whose pretrained teachers are commonly used to improve them. The paper introduces Conditional Residual Prediction (CRP) to reduce reliance on history; in controlled experiments it nearly closes the VBench gap, and the authors train a scaled causal model without a bidirectional teacher.

  • Problem

    Causal video models often have lower visual quality than comparable bidirectional models, while many approaches to improving them rely on pretrained bidirectional teachers.

    Source: §1 · pp. 1–2

  • Method

    Conditional Residual Prediction first predicts without history and lets a history-conditioned branch add only a residual to that prediction.

    Source: §1 · p. 2, §3.2 · p. 6

  • Results

    CRP narrows the 6.14-point VBench Total gap to the bidirectional reference to 0.19 points, and the image VAE narrows it further to −0.20.

    Source: §6 · p. 11

  • Takeaways & Limitations

    The results support a simple recipe for native, scalable training of causal video models without relying on a bidirectional teacher.

    Source: §1 · p. 3

    Experiments evaluate only 5-second videos, so whether CRP suppresses error accumulation over longer autoregressive rollouts remains unverified.

    Source: §6 · p. 12

Abstract

from arXiv · show

Causal video diffusion models generate video autoregressively, which suits streaming, interactive, and long-video generation. Under standard training, however, they often yield lower generation quality than bidirectional models of the same size. Many existing approaches address this gap by initializing from or distilling a pretrained bidirectional teacher. We instead train a causal model from an image-model initialization, with no bidirectional video model at any stage. Because this path requires neither a large bidirectional teacher nor a complex distillation pipeline, it is simpler and more scalable. On this path, we find that a causal model trained on ground-truth history becomes strongly dependent on it, so that at inference errors in its own generated history propagate forward. We hypothesize that much of this dependence is unnecessary, because the current input already determines much of what the history provides. We propose Conditional Residual Prediction (CRP), a simple recipe for reducing a model's reliance on a condition: the model first predicts the target without the condition, and the condition may only add a residual on top of this prediction. Applied to history, CRP makes the model predict each chunk from the present as far as it can and use the past only for what the present cannot supply. In controlled experiments, CRP nearly closes the 6.14-point gap to a bidirectional model trained under the same setup. Scaling this recipe, we train Optica, a 2B-parameter causal video model that autoregressively generates 5-second 480p videos and reaches 82.78 on VBench with only about 15M training videos.

1. Introduction

Causal video generation is suited to streaming and long videos, but comparable causal models can have lower quality than bidirectional models, and many existing fixes depend on bidirectional teachers. The paper proposes reducing reliance on history with CRP and an image VAE, then reports a substantial narrowing of the VBench gap and a scaled causal model.

1. Introduction

  • Causal generation supports streaming and long-video settings, but comparable causal models often have lower visual quality and semantic fidelity than bidirectional models.Existing approaches have used bidirectional initialization or teachers for ODE initialization or score distillation, which the paper identifies as a scaling constraint.Source: §1 · pp. 1–2
  • CRP first predicts without history, then lets a history-conditioned branch add only a residual, reducing the model’s reliance on information already determined by the current input.The History-Free branch is kept free of history through detached features, while the History-Conditioned branch supplies the residual.Source: §1 · p. 2
  • CRP with x0-prediction narrows the VBench Total gap from 6.14 to 0.19 points; CRP itself accounts for 4.68 points.The image VAE further narrows the gap from 0.19 to −0.20.Source: §1 · p. 3
  • The image VAE addresses history dependence introduced by the video VAE’s causal temporal compression, while keeping the DiT cost unchanged.Independent frame encoding also removes the distinction between single-frame and four-frame latents.Source: Figure 1 · p. 2, §1 · p. 3
  • Optica is a 2B-parameter causal video model that reaches 82.78 on VBench without relying on a bidirectional teacher.The paper describes this as a recipe for native, scalable causal-model training.Source: §1 · p. 3

2. Background

The background contrasts bidirectional video diffusion, which jointly generates a window, with causal diffusion, which generates positions from earlier history. It then describes teacher-forced training, autoregressive inference, and prior approaches that use bidirectional models to improve causal generation.

2. Background

  • The paper denotes the video’s temporal latent sequence by x, with subscripts indicating temporal positions.It also uses y for the text condition and fθ for the DiT being trained.Source: §2 · p. 3

2.1. Bidirectional Video Diffusion Model

  • Bidirectional diffusion jointly denoises all latent positions in a fixed window, with temporal attention allowing each position to access every other position.Generation integrates the ODE over the window to produce all positions at once.Source: §2.1 · p. 3

2.2. Causal Video Diffusion Model

  • Causal diffusion factorizes video generation across temporal positions and uses a causal attention mask; this paper calls chunk sizes below two seconds causal and above two seconds bidirectional.Chunk-level causality lies between frame-level causal and bidirectional generation.Source: §2.2 · p. 3
  • Causal models train on clean ground-truth prefixes but infer using their own generated history, creating a training–inference gap associated with quality loss.The figure caption reports stable teacher-forced quality but declining autoregressive quality with chunk index.Source: §2.2 · p. 3, Figure 2 · p. 4

2.3. Initialization and Distillation from a Bidirectional Teacher

  • Prior methods improve causal models through bidirectional initialization or distillation, while later work makes the teacher causal to avoid future-information mismatch.The cited approaches include replacing ground-truth prefixes with rollouts and using causal models for ODE initialization or scoring.Source: §2.3 · p. 4

3. Method

The method addresses unnecessary reliance on history by separating information available from the current input from history-exclusive information, then constraining history to contribute a residual. It also replaces causal video latents with independently encoded frame latents to address additional VAE-induced history dependence.

3.1. The History Over-Reliance Hypothesis

  • On the causal baseline, autoregressive rollout quality falls to 0.592 from 0.824 in the first chunk, illustrating the mismatch between ground-truth-prefix training and generated history.The authors hypothesize that history supplies some information already determined by the current input, causing unnecessary history dependence.Source: §3.1 · p. 5

3.2. Conditional Residual Prediction

  • The controlled ablation shows CRP nearly closes the causal–bidirectional VBench gap, while the image VAE brings scores on par with the bidirectional baseline.The table describes successive configuration changes trained on 4M videos at 256p and 5 seconds.Source: Table 1 · p. 6
  • CRP uses a History-Free branch and a History-Conditioned residual branch, so history can add to—but not replace—the prediction from the current chunk and text.The same network computes both branches; the conditioned branch receives detached features, and the clean chunk is the target under 𝑥0-prediction.Source: §3.2 · p. 6

3.3. From Video VAE to Image VAE

  • Causal temporal compression in the video VAE introduces history dependence and latent imbalance; an image VAE encodes frames independently, removing error propagation to later frames and the chunk-0 drop.The image VAE uses the same Wan2.1 VAE in image mode; the authors report no loss of fidelity.Source: §3.3 · pp. 6–7
  • A lightweight temporal patch layer merges every four adjacent latents at the input and splits them at the output to keep DiT cost unchanged.The complete configuration is called “+ Image VAE.”Source: §3.3 · p. 7

4. Controlled Ablation Experiments

In controlled experiments, CRP brings the causal model close to the bidirectional baseline, and adding an image VAE matches it. Component ablations support reducing history over-reliance as the key to the improvement.

4.1. Setup

  • The controlled ablations use approximately 4M internal 256p video clips with a 5-second duration.The model initialization and architecture details are described in the setup passages.Source: §4.1 · p. 7

4.2. Bidirectional and Causal Baselines

  • 71.14 VBench Total for the causal baseline trails the bidirectional baseline’s 77.28, and the causal rollout rapidly develops distorted textures, incorrect structures, and artifacts.The rollout deterioration is also visible in Figure 3; CRP preserves scene structure, while the image VAE removes later-frame darkening and speckles.Source: §4.2 · p. 7, Figure 3 · p. 8

4.3. Causal Model with CRP

  • CRP raises VBench Total from 72.41 to 77.09, recovering 4.68 of the causal-to-bidirectional baseline gap of 6.14 points.It also reduces the autoregressive-versus-teacher-forcing quality gap from 0.138 ± 0.003 to 0.066 ± 0.003 and the corrupted-history quality loss from 0.089 ± 0.001 to 0.020 ± 0.001.Source: §4.3 · p. 8

4.4. Causal Model with Image VAE

  • 77.48 VBench Total with the image VAE matches the bidirectional baseline, and later-frame darkening and speckles disappear.The image VAE also reduces the reported generation-history diagnostics to 0.039 ± 0.002 and −0.004 ± 0.001, respectively.Source: §4.2 · p. 7, §4.4 · p. 8

4.5. Ablation of CRP Components

  • The controlled ablations support CRP’s output composition: without it, the model over-relies on history, and VBench Total falls to 72.63.Removing latent fusion lowers the score to 76.31, while removing the stop-gradient lowers it to 75.87.Source: §4.5 · pp. 8–9

5. Scaling Experiments

The scaling experiments train Optica from an image-model initialization without a bidirectional video teacher. Optica scores 82.78 VBench Total among methods without a bidirectional video model, while its Quality score remains below some comparisons.

Scaling Experiments

  • Optica uses a 2B-parameter model and approximately 15M internal video clips, with no initialization, ODE sample, or score from a bidirectional teacher.Training continues at 256p and 5 seconds before switching to 480p, followed by supervised fine-tuning; the scaled-up model uses chunks of four latent positions.Source: §5 · pp. 9–10

Results

  • Optica’s Semantic score of 81.83 is comparable to strong distilled causal models, while its Quality score is 1.0–4.0 points lower.Against the bidirectional SANA-Video 5-second checkpoint, Semantic scores are 81.83 versus 81.35, while Quality scores are 83.02 versus 84.35.Source: §5 · p. 10
  • In controlled experiments with identical data, model sizes, and evaluation protocols, the causal model matches the bidirectional reference; the authors identify training budget as the main source of remaining gaps.The authors note that the upstream training cost of pretrained video teachers is not reflected in distilled models’ parameter counts.Source: §5 · p. 10
  • 82.78 VBench Total is the highest among methods that do not rely on a bidirectional video model, exceeding Pyramid Flow, NOVA, and MAGI-1 by 1.06, 2.66, and 3.60 points.Optica scores 83.02 on Quality and 81.83 on Semantic; its Quality score remains below Pyramid Flow’s 84.74.Source: §5 · p. 10, Table 3 · p. 11

6. Conclusion

The conclusion argues that reducing history dependence can improve causal video generation without a bidirectional teacher, while preserving history-exclusive signals. It presents context compression as a possible application and notes limits in rollout duration, conditioning, and efficiency.

6. Conclusion

  • CRP narrows the causal-to-bidirectional VBench Total gap from 6.14 points to 0.19, and the image VAE narrows it further to −0.20.The 2B-parameter Optica uses no bidirectional-teacher initialization, sample, or score and achieves 82.78 on VBench.Source: §6 · p. 11
  • History remains necessary for motion, identity, and temporal consistency, and Optica may complement distillation by providing a natively trained causal teacher.The conclusion connects this to findings that few-step causal models benefit from distilling multi-step causal teachers [18].Source: §6 · p. 11
  • The decomposition suggests retaining history-exclusive information and discarding what the current position can infer, a criterion for designing context-compression algorithms.The paper frames this as an information-theoretic rather than heuristic target.Source: §6 · p. 12
  • Experiments and scaling runs cover only 5-second videos, leaving CRP’s effect on longer autoregressive rollouts unverified.Other stated boundaries include no image-to-video support, no demonstrated real-time efficiency, and no few-step Optica distillation.Source: §6 · p. 12

AI Use Statement

  • The authors report using LLMs for language polishing, coding assistance, formula review, and related-literature searches, but not for core method or research development.They state that they take responsibility for the paper’s final content.Source: AI Use Statement · p. 12

A. Extended Related Work

  • History-corruption methods trade off training–inference consistency or loss of useful history, whereas the paper’s runs use a clean prefix throughout.The passage contrasts noise augmentation and corruption/dropout with the paper’s clean-prefix setting.Source: A. Extended Related Work · p. 20
  • Prior causal-distillation methods retain rollout training pipelines, while Resampling Forcing [62] avoids a teacher but starts from pretrained Wan2.1-1.3B weights; CRP instead changes the prediction target at the cost of an extra forward pass.The cited rollout methods draw targets from a distillation pipeline [9] [11] [53] [55]; CRP is distinct from changing the conditioning signal.Source: A. Extended Related Work · p. 20
  • History Guidance controls history at sampling after training full-target predictions, while CRP adds a residual to a detached History-Free prediction.SCD [43] instead compresses history through a separate extractor.Source: A. Extended Related Work · p. 21

B. Implementation Details

The implementation uses flow matching, AdamW optimization, and a shared 50-step flow-adapted sampler, with resolution and text-guidance settings varying by experiment.

B. Implementation Details

  • Models use AdamW with β1=0.9, β2=0.999, ε=10−10, zero weight decay, gradient clipping at 0.5, and a constant learning rate after 500-step warmup.The learning rate is 3.5 × 10−5 except for the scaled model’s final SFT stage, which uses 1 × 10−5; training uses bf16 on H20 GPUs with global batch size 128 and no EMA.Source: B. Implementation Details · p. 21
  • Training uses flow matching with independently sampled noise and timesteps per latent position, sharing one timestep within chunks when chunk size exceeds one.Text conditions are dropped with probability 0.1 for classifier-free guidance.Source: B. Implementation Details · p. 21
  • All experiments sample with 50-step flow-adapted DPM-Solver++ and inference shift 7.0; controlled runs use text guidance 6.5, while scaling runs use 9.5.Controlled videos are generated at 256p and scaling videos at 480 × 832; the passage specifies 81 frames with the video VAE and 80 with the image VAE, both at 16 fps.Source: B. Implementation Details · p. 21

B.1. Architecture

The architecture adapts a SANA-1.5 DiT to video latents, moves temporal grouping into the transformer for the image VAE, and aligns the model’s input and output layers to the Wan VAE latent space.

B.1. Architecture

  • The DiT is warm-started from adapted SANA-1.5 weights and uses temporal convolutions in its MLP blocks, ReLU-feature attention with video RoPE, and softmax attention in every fourth block.The model retains 20 blocks at width 2240, with 20 attention heads of dimension 112.Source: §B.1 · p. 22

B.2. Temporal Patch Layer

  • With the image VAE, paired temporal projections group every four latent frames into one token group and restore them before output, leaving 20 temporal positions for transformer blocks.The two projections add 40M parameters, about 2% of the model.Source: §B.2 · p. 22

B.3. Adapting SANA-1.5 to the Wan VAE Latent Space

  • To adapt SANA-1.5 from DC-AE to Wan VAE latents, training first aligns input embeddings and then tunes only the input and output layers with image-only flow matching.The first stage trains 146K parameters; embedding cosine similarity exceeds 0.4 within 500 steps and rises to about 0.7, while the second stage uses on the order of 1M samples.Source: §B.3 · p. 22
  • The authors reimplemented the embedding-alignment procedure because DC-VideoGen code was unavailable at the time, and report that it sped subsequent video-training convergence.Source: §B.3 · p. 22

C. Training Data

The study trains on a filtered internal video corpus, evaluates VBench under specified but differing protocols, and measures rollout quality and sensitivity to corrupted histories over 1,000 clips.

C. Training Data

  • Training uses about 15M filtered internal clips, each 5–6 seconds long, with a uniformly sampled 5-second training window; the scaled model additionally receives SFT on 0.5M high-quality videos.The dataset is shot-segmented, deduplicated, concept-balanced, and filtered for aesthetic and technical quality, blur, and motion.Source: C. Training Data · p. 23

D.1. VBench

  • Optica’s scaled VBench evaluation uses augmented prompts for generation but original short prompts for scoring at 480 × 832, while controlled ablations use standard short prompts at 256p.Table 3 combines scores from public leaderboard, cited prior-paper, and original-paper sources; protocols differ across rows in prompt rewriting, resolution, and sampling.Source: §D.1 · p. 23

D.2. Per-Frame Quality and the History-Corruption Diagnostic

  • The diagnostic compares per-frame quality under teacher forcing and autoregressive rollout on 1,000 real clips, and tests five history corruptions while leaving the current chunk unchanged.Corruptions are brightness, contrast, saturation, elastic transform, and smooth grid warp; ΔQ is corrupted-history quality minus clean-history quality for the same clip and chunk.Source: §D.2 · pp. 23–24

D.3. VAE Decoder Perturbation Analysis

The analysis perturbs a latent frame in encoded video clips, decodes original and perturbed sequences, and measures pixel-space change across output frames. It also measures latent-position KL against a standard Gaussian prior.

D.3. VAE Decoder Perturbation Analysis

  • The experiment encodes 1,000 real video clips with the Wan2.1 VAE, then adds Gaussian perturbations to one latent frame.All other latent frames are left unchanged.Source: §D.3 · p. 24
  • Pixel-space RMS is computed for every output frame by comparing separately decoded original and perturbed latents.The RMS is taken over spatial dimensions and averaged over channels; Figure 5(a) reports the clip mean with clip-bootstrap 95% confidence intervals.Source: §D.3 · p. 24
  • At each latent position, the analysis estimates per-channel moments across clips and spatial positions, then averages the Gaussian-approximation KL to the standard prior across channels.This yields KL per latent dimension for Figure 5(b).Source: §D.3 · p. 24

E. History Dependence in the Video VAE

The video VAE's causal temporal convolutions couple latent frames, so errors in early latents persist in later decoded frames. The first latent also has different statistics from later latents under a moment-matched Gaussian approximation.

E. History Dependence in the Video VAE

  • The Wan2.1 VAE's causal temporal convolutions create dependencies between earlier and later frames, so perturbing an earlier frame affects later decoded positions.The architecture is common in video VAEs, and related temporally compressive representations are used in latent video generation.Source: E. History Dependence in the Video VAE · p. 24
  • An error in an early latent propagates through video-VAE decoding to later positions, reaching the last frame in a 5-second, 16-fps clip.Figure 5(a) shows a persistent tail; the VAE carries an incorrectly generated earlier signal into subsequent positions.Source: E. History Dependence in the Video VAE · pp. 24–25, Figure 5 · p. 25
  • The first latent frame has substantially larger KL to the standard Gaussian prior than later frames; from latent frame 3 onward, KL is lower.The comparison uses per-channel moment-matched Gaussian approximations.Source: E. History Dependence in the Video VAE · p. 24, Figure 5 · p. 25
  • The adopted image-VAE approach removes all VAE history dependence, including necessary dependence, leaving history modeling to the DiT and CRP.Retaining necessary VAE dependence while removing only unnecessary dependence is raised as a possible alternative for further exploration.Source: E. History Dependence in the Video VAE · p. 25

F. VAE Reconstruction: Image vs. Video Mode

Image-mode encoding improves paired reconstruction PSNR and prevents a perturbation to the latent for pixel frame 0 from changing later frames. The comparison uses the same VAE checkpoint in video and image modes.

F. VAE Reconstruction: Image vs. Video Mode

Loading 2610.11479v1…