PaperlayerView exact revision on arXiv

Source-linked AI summary

Looking Ahead: Modeling Future Evolution in Autoregressive Video Generation

Ruopeng Gao, Yukun Huang, Jiarui Hu, Shuai Wang, Wenbo Hu, Ying Shan, Limin Wang

arXiv:2610.29989v1cs.CV

TL;DR

Teacher-forced video models can rely on local continuity rather than learning how scenes evolve. Future Chunk Prediction adds training supervision from temporally separated video chunks, and controlled comparisons report improved physical plausibility alongside more structured representations.

  • Problem

    Under teacher forcing, temporal redundancy can let local visual continuity support current-chunk denoising, reducing pressure to model scene evolution.

    Source: §1 · p. 1, §4.1 · p. 4

  • Method

    Future Chunk Prediction jointly adds an objective that predicts temporally separated chunks from intermediate representations without intervening ground-truth visual context.

    Source: §1 · p. 2, §6 · p. 10

  • Results

    Controlled comparisons show improved physical plausibility across model scales and conditioning settings, while visualizations suggest a more structured representation space.

    Source: §1 · p. 2, §6 · p. 10

  • Takeaways & Limitations

    Future-chunk supervision from existing videos offers a route to improving physical plausibility without relying solely on increased training resources.

    Source: §1 · p. 2

    Differences in training data and settings limit attribution of cross-model performance gaps to methodological improvements.

    Source: §5.4 · p. 6

Abstract

from arXiv · show

Autoregressive video diffusion models generate videos chunk by chunk, yet producing plausible continuations does not guarantee that they capture how scenes evolve. Under teacher forcing, strong temporal redundancy may allow local visual continuity to serve as a shortcut for current-chunk denoising, reducing the pressure to model visual evolution. We introduce Future Chunk Prediction (FCP), an auxiliary objective jointly optimized with teacher-forced denoising that predicts temporally separated chunks without access to the intervening visual context. Drawing supervision directly from training videos, FCP requires no additional pretrained visual encoder and leaves autoregressive inference unchanged. Our method achieves superior physical plausibility over standard teacher-forced autoregressive training with a lower overall training compute budget, highlighting the effectiveness and efficiency of future-chunk supervision for modeling visual evolution. Intriguingly, our visualizations suggest that FCP fosters a more structured representation space, which may be valuable for building autoregressive world models.

1 INTRODUCTION

Teacher-forced denoising can exploit local continuity rather than model scene evolution. Future Chunk Prediction (FCP) adds future-chunk supervision, and the paper reports improved physical plausibility and more structured representations.

1 INTRODUCTION

  • Strong temporal redundancy lets teacher-forced models use neighboring chunks’ similar appearance and motion as a shortcut for current-chunk denoising, limiting pressure to capture scene dynamics.More temporally distant targets receive less direct support from local appearance and motion continuity; this supervision can come from the training video without external pretrained representations or semantic alignment.Source: §1 · p. 1
  • FCP jointly adds an objective that predicts a temporally separated chunk from an intermediate representation without exposing intervening ground-truth visual context.It is used during training only, leaving autoregressive inference unchanged.Source: §1 · p. 2
  • FCP improves physical plausibility over standard teacher-forced autoregressive training under matched data exposure and training budget.The evaluation focuses on physical plausibility as a concrete, challenging measure of visual evolution.Source: §1 · p. 2
  • Visualizations suggest FCP induces more pronounced spatiotemporal organization in intermediate representations, potentially benefiting autoregressive world modeling.The paper presents this as a possible benefit beyond generation quality.Source: §1 · p. 2

2 RELATED WORK

Prior work improves autoregressive rollout and manages historical context, while representation-alignment methods use external pretrained targets. This work complements those directions with future-chunk supervision drawn from training videos.

2.1 AUTOREGRESSIVE VIDEO DIFFUSION

  • Autoregressive video research addresses rollout quality through causal few-step generators, self-generated histories or extended rollouts, teacher-supervision compatibility, and context-management strategies.Context methods include active windows, cached states, global anchors, selective retrieval, and cache compression; FCP adds future-chunk supervision without prescribing rollout distillation or inference-time context management.Source: §2.1 · p. 2

2.2 REPRESENTATION ALIGNMENT FOR GENERATION

  • Representation-alignment methods use external pretrained encoders for feature, cross-frame, multi-feature, or relational supervision, whereas this approach derives supervision from future chunks in training videos.The paper presents this as a data-native alternative that does not require an additional pretrained vision model.Source: §2.2 · p. 2

3 PRELIMINARIES

The preliminaries define flow matching as learning a velocity field from noise toward video data, then describe chunk-wise autoregressive generation and its teacher-forced training objective.

3.1 FLOW MATCHING

  • Flow matching learns a time-dependent vector field connecting the data distribution to a simple noise distribution.A clean video sample, Gaussian noise, and a sampled time define an intermediate state along a linear probability path; the target velocity is ε − x, and inference follows the learned field from noise to video.Source: §3.1 · p. 3

3.2 AUTOREGRESSIVE VIDEO GENERATION

  • Autoregressive video generation factorizes a video into chunkwise conditional distributions, with each chunk conditioned on preceding chunks.The video is divided into ordered, non-overlapping equal-length chunks, and repeated chunk generation supports flexible-length rollout.Source: §3.2 · p. 3, §3.3 · p. 3

3.3 TEACHER-FORCED TRAINING

  • Teacher-forced training uses clean ground-truth preceding chunks as history while a flow-matching model denoises the target chunk.The model predicts the target chunk’s velocity while conditioned on that history and the text condition.Source: §3.3 · p. 3

4 METHOD

Future Chunk Prediction (FCP) supplements teacher-forced denoising with predictions of farther chunks from the same video, excluding intervening visual context. Its architecture conditions auxiliary predictors on generator representations, and training jointly updates both while leaving inference unchanged.

4.1 FUTURE CHUNK PREDICTION

  • FCP predicts temporally distant chunks without their intervening visual context, complementing next-chunk denoising to encourage representations that capture visual evolution.The target is less directly supported by immediately preceding context, while supervision remains intrinsic to the training video.Source: §4.1 · p. 4
  • The auxiliary predictor uses separate parameters and backpropagates through the generator’s intermediate hidden states, so future-prediction supervision regularizes the generator representations.Preliminary experiments found that sharing generator parameters between the two objectives degraded temporal coherence, possibly because they impose competing demands.Source: §4.1 · p. 4

4.2 FCP ARCHITECTURE

  • Parallel predictors condition directly on source representations, avoiding inter-offset information flow; the architecture also fuses features from multiple generator layers and omits text cross-attention.The parallel design removes the particular shortcut in which a noisy intermediate future input can inform a later predictor; omitting text cross-attention blocks a direct prompt-cue pathway and reduces predictor cost.Source: §4.2 · p. 5

4.3 TRAINING AND INFERENCE

  • Training jointly optimizes the generator and auxiliary predictors with flow matching across predefined offsets, using clean or noisy source representations and valid chunk pairs.Target noise and flow time are resampled for each prediction; clean and noisy source representations share a predictor for each offset, and weighted auxiliary losses backpropagate through the representations into the generator.Source: §4.3 · p. 6
  • At inference, FCP-specific modules are discarded, leaving the teacher-forcing baseline’s autoregressive generator and rollout procedure unchanged.Auxiliary-predictor visualizations suggest plausible future evolution despite limited visual fidelity; using them for acceleration or ensembling is left to future work.Source: §4.3 · p. 6

5 EXPERIMENTS

FCP is evaluated on physical plausibility across controlled and complementary benchmarks, with ablations examining its design and representation effects. It improves Physics-IQ scores over teacher forcing, while visualizations suggest more structured spatial and temporal representations.

5.1 IMPLEMENTATION DETAILS

  • Models start from official Wan checkpoints; the image-to-video setup fixes the initial frame as clean context and generates later chunks of three latent frames.Generators and three-block auxiliary predictors are jointly optimized. Evaluation centers on reference-based Physics-IQ Verified, with three complementary physics-related benchmarks; training uses general real-world videos and no simulator data.Source: §5.1 · p. 6, §5.2 · p. 6, §5.3 · p. 6

5.4 MAIN RESULTS

  • IQ rises from 27.6 to 30.9 for the 1.3B model, and from 30.4 to 33.8 (I2V) and 40.6 to 44.4 (V2V) for the 5B model versus teacher forcing.These controlled comparisons use matched data and experimental settings; qualitative examples also show more plausible domino toppling and a captured water-level rise with FCP.Source: §5.4 · pp. 7–8, Table 2 · p. 7, Figure 3 · p. 7
  • The 5B model also outperforms several larger bidirectional models and autoregressive pre-training results, including Wan2.2-14B, HunyuanVideo-1.5, Cosmos3-Nano, MAGI-1-4.5B, and Helios-Base.Table 1 provides broader context for these comparisons; the supplied passages do not give the corresponding scores.Source: §5.4 · pp. 7–8, Table 1 · p. 7
  • FCP generally outperforms teacher forcing on WorldModelBench, PAI-Bench-G, and PhyGround under matched settings, though VLM-based judgments may miss subtle physical changes.The evaluations use specified criteria, which may help explain modest gains on some metrics.Source: §5.4 · p. 8, Table 3 · p. 8

5.5 ABLATION STUDIES

  • Regularizing clean-context or noisy-target representations improves IQ over teacher forcing by 1.2 and 2.3 points, respectively, while combining them yields a 3.3-point gain.The ablations support treating clean historical context as predictive, not only as conditioning for the next chunk.Source: Table 4 · p. 8, §5.5 · p. 9
  • Parallel prediction improves IQ by 0.8 over chained prediction, and the chained variants give only marginal I2V gains and no V2V improvement in the controlled 5B comparison.Parallel branches also support nonconsecutive future offsets; the authors suggest higher-capacity models may more readily exploit leaked intermediate future information.Source: Table 2 · p. 7, §5.5 · p. 9
  • At equal IQ scores of 29 and 30, the default configuration uses 42.6% and 50.7% less estimated training compute than full-context FCP, respectively.It also reaches higher IQ than teacher forcing at comparable estimated compute; adding explicit text conditioning to future predictors reduces IQ by 1.1.Source: Figure 4 · p. 8, §5.5 · p. 9

5.6 REPRESENTATION SPACE ANALYSIS

  • Similarity maps suggest that FCP yields more selective spatial responses and clearer alignment of similar content as it moves across frames.These visualizations suggest more structured spatial features and meaningful temporal correspondence, potentially useful for future world models.Source: §5.6 · pp. 9–10, Figure 5 · p. 9, Figure 6 · p. 9

6 CONCLUSION

Future Chunk Prediction (FCP) adds temporally separated future-chunk prediction to teacher-forced denoising, using training-video supervision without an additional pretrained encoder. Controlled comparisons report improved physical plausibility and more structured representations, while inference remains unchanged.

6 CONCLUSION

  • Controlled comparisons show improved physical plausibility across model scales and conditioning settings.The auxiliary predictors are removed at inference, leaving the autoregressive generation process unchanged.Source: §6 · p. 10
  • FCP predicts temporally separated chunks from the same training video, jointly with standard teacher-forced denoising.It provides complementary supervision to the generator’s intermediate representations without requiring an additional pretrained encoder.Source: §6 · p. 10
  • Visualizations suggest that FCP induces a more structured representation space, which may be important for autoregressive world models.The paper presents future prediction as a promising training principle for autoregressive video generation.Source: §6 · p. 10

REPRODUCIBILITY STATEMENT

The reproducibility statement describes planned release materials and identifies a constraint on exact reproduction: the complete internally sourced training corpus cannot be released.

REPRODUCIBILITY STATEMENT

  • The complete internal training corpus cannot be released, so exact reproduction with identical data may not be possible.The authors state they will make every effort to support reproducibility.Source: REPRODUCIBILITY STATEMENT · p. 10
  • The authors say they will provide source links and selection criteria for the public portion of the training data.They also plan to provide representative examples of internally sourced data.Source: REPRODUCIBILITY STATEMENT · p. 10
  • The code is available at the stated GitHub repository, but model checkpoints are not included in the source package.Sections 4 and 5 describe the methods, training setup, datasets, and evaluation protocols; Appendix A contains further implementation details.Source: REPRODUCIBILITY STATEMENT · p. 10

AI USE STATEMENT

The appendix records the authors’ AI-use disclosures, model configurations, and training settings. It describes feature fusion and auxiliary branches, notes that source-layer selection strategies lack systematic comparison, and gives separate training configurations for the 1.3B and 5B models.

AI USE STATEMENT

  • Generative AI assisted with implementation, translation, manuscript drafting, literature work, and formatting, while authors reviewed the code and text and take responsibility for the final content.The authors state that AI was not used to generate datasets, develop the conceptual framework, formulate mathematical claims, or design or interpret experiments.Source: AI USE STATEMENT · p. 10

A.1 MODEL CONFIGURATIONS

  • Source features from four selected DiT blocks are concatenated at corresponding token positions and fused by a two-layer MLP with dimensions 4d →2d →d and GELU activation.Each future offset has an independently parameterized auxiliary branch; within an offset, parameters are shared between clean and noisy source pathways.Source: §A.1 · p. 15
  • The 5B model selects approximately evenly spaced source layers backward from the final block, excluding the earliest blocks whose features may not yet be sufficiently semantically informative.For the 1.3B model, source layers expanded from an early single-layer variant using layer 15; computational constraints have prevented systematic comparison of the selection strategies.Source: §A.1 · p. 15
  • For 5B auxiliary initialization, the predictor uses final backbone blocks, chosen to match the output head and simplify scale-up; preliminary experiments suggest modestly improved early-stage convergence.The 1.3B configuration instead uses source features from layer 15 and blocks 17–19 to initialize the auxiliary predictor.Source: §A.1 · p. 15

A.2 TRAINING CONFIGURATIONS

  • The 1.3B setup trains for 20,000 steps on eight GPUs using 85-frame, 768 × 432 clips at 16 FPS, with a global batch size of eight.It sets the teacher-forcing loss weight to 1 and each future-offset weight to 0.5; the 1.3B ablations follow these settings.Source: §A.2 · pp. 15–16
  • The first 5B training stage runs for 25,000 steps on 32 GPUs using 121-frame, 1024×576 clips at 24 FPS and a global batch size of 32.Other 5B training settings follow the 1.3B configuration unless otherwise specified.Source: §A.2 · p. 16

A.3 EVALUATION SETTINGS

The evaluation settings use EMA weights and remove FCP-specific modules at sampling. Physics-IQ Verified evaluates five-second I2V and V2V continuations, while other physics-related benchmarks evaluate I2V outputs with their official metrics.

A.3 EVALUATION SETTINGS

  • Sampling uses UniPC with 50 denoising steps per chunk, classifier-free guidance scale 3.0, and timestep shift 5.0; FCP-specific modules are removed.When inference exceeds training sequence length, the first three latent frames are retained as attention sinks during KV-cache eviction.Source: §A.3 · p. 16
  • Physics-IQ Verified evaluates five-second I2V and V2V continuations, excluding the conditioning video from evaluation.I2V uses the benchmark initial image; V2V uses the longest compatible suffix of the three-second conditioning video.Source: §A.3 · p. 16
  • Physics-IQ Verified uses all 198 benchmark samples, official metrics, and four independent runs with seeds 42–45, reporting the mean score.Unless otherwise specified, evaluation uses best-practice prompts and the model’s specified output frame rate.Source: §A.3 · p. 16
  • WorldModelBench, PAI-Bench-G, and PhyGround evaluate Wan2.2 5B teacher-forced and FCP-trained models on all samples in the I2V setting.Each uses official evaluation code and unmodified metrics, with scores averaged across seeds 42–45.Source: §A.3 · p. 16

A.4 EVALUATION OF PUBLIC MODELS

This material documents evaluation settings for public models and the protocols used to visualize representations and auxiliary predictions. It also notes that a fuller breakdown of component ablations is reported in Table 6.

A.4 EVALUATION OF PUBLIC MODELS

  • Public-model evaluations use each checkpoint’s stated inference settings, with benchmark-specific resolutions, frame counts, preprocessing, and sampling details.For Causal Forcing variants, all 198 Physics-IQ Verified samples are scored with official metrics across seeds 42–45, and the mean is reported.Source: §A.4 · pp. 16–17

A.5 REPRESENTATION VISUALIZATION PROTOCOL

  • Representation maps compare matched teacher-forced and FCP checkpoints using cosine similarity from selected query tokens across spatial or temporal tokens.The compared models receive the same clean video latents; their maps share a color map and value range within each comparison.Source: §A.5 · p. 17

A.6 AUXILIARY PREDICTION VISUALIZATION PROTOCOL

  • Auxiliary future chunks are sampled independently from noise using the final source-chunk hidden state, without intervening chunks or separate text input.Branches use 50 denoising steps, classifier-free guidance 3.0, and timestep shift 5.0; they are removed during standard inference.Source: §A.6 · p. 17

B.1 DETAILED COMPONENT ABLATIONS

  • Table 6 extends the condensed component ablations in Table 4 by reporting the complete breakdown of Physics-IQ Verified metrics, enabling a more comprehensive comparison of different component choices.Source: §B.1 · p. 17

B.2 FEATURE CONDITIONING

Feature-conditioning strategies differ in their reported IQ scores, while supplementary visualizations compare FCP and teacher-forced representations and display auxiliary future predictions. The visualizations describe qualitative patterns rather than exact future-frame reconstruction.

B.2 FEATURE CONDITIONING

  • IQ rises from 27.6 with concatenation and linear fusion to 30.9 with token-wise AdaLN, while concatenating source and future tokens within self-attention reaches 28.8.The first comparison is against teacher forcing; the authors hypothesize AdaLN’s layer-wise modulation strengthens conditioning and gradient propagation, while joint self-attention weakens fine-grained correspondence.Source: §B.2 · pp. 17–18, Table 7 · p. 18

B.3 REPRESENTATION SPACE VISUALIZATION

  • Across additional scenes, high-similarity regions remain aligned with corresponding visual content across frames.Figure 8’s examples supplement the temporal correspondence discussed in Section 5.6.Source: §B.3 · p. 18
  • FCP representations show more selective similarity around queried content, including localized responses around a small vehicle that are more diffuse under teacher forcing.The vehicle comparison suggests stronger spatiotemporal discrimination and semantic structure; Figure 7 adds further spatial examples.Source: §B.3 · p. 18, Figure 7 · p. 19

B.4 AUXILIARY FUTURE PREDICTION VISUALIZATIONS

  • Auxiliary predictions capture broad visual changes without intervening chunks and often retain ground-truth-consistent motion trends, though fidelity tends to decline at farther offsets.The examples suggest the source representation contains information useful for anticipating evolution, not exact future-frame reconstruction.Source: §B.4 · p. 18

C.1 COMPARISON WITH NEXT FORCING

Compared with Next Forcing, FCP uses only the source chunk for parallel future prediction and achieves comparable IQ scores with lower training compute. The discussion also describes untested inference-time possibilities and limits in model-scale and video-horizon coverage.

C.1 COMPARISON WITH NEXT FORCING

  • Next Forcing is framed around world-action modeling, while this work focuses on autoregressive video generation as a pathway toward generative world modeling.The papers were developed independently and concurrently, with FCP’s initial prototype predating public availability of Next Forcing.Source: §C.1 · pp. 19–20
  • At matched training steps, source-chunk-only FCP achieves comparable IQ scores to full-context FCP while requiring roughly 40–50% less training compute.Next Forcing conditions on full visible history, whereas FCP uses only the source chunk; the authors present this choice as more efficient and scalable.Source: §C.1 · p. 20
  • The chained variant retains a gain at 1.3B with three offsets but provides only a marginal I2V gain and no V2V gain at 5B; parallel prediction remains effective in both settings.The authors suggest intermediate-future information leakage may become more consequential at larger scales.Source: §C.1 · p. 20

C.2 POTENTIAL INFERENCE-TIME APPLICATIONS

  • Auxiliary predictors have not reached final-generation visual fidelity, and using or combining them during inference remains untested.Early high-noise use followed by main-generator refinement, or combining predictions, are proposed possibilities rather than evaluated approaches.Source: §C.2 · p. 21

C.3 FROM-SCRATCH PRETRAINING AND FURTHER SCALING

  • Experiments do not cover larger model scales or training on substantially longer videos, leaving broader scaling and longer-horizon evaluation for future work..Source: §C.3 · p. 21
  • All current models start from pretrained bidirectional checkpoints; the authors suspect this initialization could limit FCP’s benefits because bidirectional and autoregressive models may organize representations differently.Large-scale from-scratch pretraining with FCP is left for future investigation.Source: §C.3 · p. 21

Related papers

Loading 2610.29989v1…