PaperlayerView exact revision on arXiv

Source-linked AI summary

Stable Video Infinity: Infinite-Length Video Generation with Error Recycling

Wuyang Li, Wentao Pan, Po-Chien Luan, Yang Gao, Alexandre Alahi

arXiv:2510.09212v1cs.CV

TL;DR

Long-video generation struggles because DiTs train on clean trajectories but infer autoregressively from self-generated, error-prone outputs. SVI bridges this gap with Error-Recycling Fine-Tuning, which trains DiTs to correct recycled errors, and surpasses state-of-the-art methods across long, ultra-long, and conditional video generation.

  • Problem

    Long-video generation faces a training–test hypothesis gap: DiTs train with clean inputs and trajectories but test autoregressively with self-generated errors.

  • Method

    SVI injects, calculates, stores, and resamples self-generated errors as supervisory feedback through Error-Recycling Fine-Tuning.

  • Results

    Across three benchmarks, SVI surpasses state-of-the-art methods on long, ultra-long, and conditional video generation.

  • Takeaways & Limitations

    SVI breaks the video length limit by actively correcting self-generated errors while supporting long storylines and conditional generation.

  • Takeaways & Limitations

    The training formulation assumes error-free video latents, references, and trajectories, motivating the method’s error-recycling correction.

Abstract

from arXiv · show

We propose Stable Video Infinity (SVI) that is able to generate infinite-length videos with high temporal consistency, plausible scene transitions, and controllable streaming storylines. While existing long-video methods attempt to mitigate accumulated errors via handcrafted anti-drifting (e.g., modified noise scheduler, frame anchoring), they remain limited to single-prompt extrapolation, producing homogeneous scenes with repetitive motions. We identify that the fundamental challenge extends beyond error accumulation to a critical discrepancy between the training assumption (seeing clean data) and the test-time autoregressive reality (conditioning on self-generated, error-prone outputs). To bridge this hypothesis gap, SVI incorporates Error-Recycling Fine-Tuning, a new type of efficient training that recycles the Diffusion Transformer (DiT)'s self-generated errors into supervisory prompts, thereby encouraging DiT to actively identify and correct its own errors. This is achieved by injecting, collecting, and banking errors through closed-loop recycling, autoregressively learning from error-injected feedback. Specifically, we (i) inject historical errors made by DiT to intervene on clean inputs, simulating error-accumulated trajectories in flow matching; (ii) efficiently approximate predictions with one-step bidirectional integration and calculate errors with residuals; (iii) dynamically bank errors into replay memory across discretized timesteps, which are resampled for new input. SVI is able to scale videos from seconds to infinite durations with no additional inference cost, while remaining compatible with diverse conditions (e.g., audio, skeleton, and text streams). We evaluate SVI on three benchmarks, including consistent, creative, and conditional settings, thoroughly verifying its versatility and state-of-the-art role.

1 INTRODUCTION

Long-video generation is limited by accumulated autoregressive errors, while existing methods mainly alleviate rather than correct them and often produce short, homogeneous extrapolations. Stable Video Infinity addresses the training–test hypothesis gap by recycling self-generated errors during fine-tuning.

  • Video DiTs typically generate only about 5 seconds because autoregressive predictive errors compound, degrading fidelity, motion stability, and semantic controllability.
  • Existing approaches modify noise, anchor frames, or improve sampling, but primarily alleviate accumulated errors rather than correct them.
  • These methods generally remain constrained to roughly 10 seconds to 1 minute and extrapolate single-prompt clips with homogeneous scenes and repetitive motion.
  • Prompt-stream storylines are needed for plausible frequent scene changes and hour-scale online presentations, which current solutions do not satisfy.
  • The central gap is that training assumes error-free trajectories, whereas autoregressive testing conditions on predictive errors and error-included inputs.
  • SVI uses Error-Recycling Fine-Tuning to inject, calculate, store, and resample self-generated errors, enabling error correction with small-scale LoRA data, zero additional inference cost, and audio or skeleton controls.

2 RELATED WORK

Video generation research has progressed in short, controllable videos, but practical applications remain limited by short duration. Long-video methods mainly address error accumulation through scheduler changes, frame anchoring, or error-robust generation, whereas SVI recycles self-generated errors for correction.

  • Short-video systems support task-oriented controls such as audio and skeleton for applications including talking, dancing, navigation, and gaming.
  • Despite progress in short-video generation, limited duration remains an open challenge that restricts practical applications.
  • Long-video methods modify noise schedules, anchor clean frames, or use error-robust architectures and sampling to improve autoregressive consistency.
  • SVI differs by recycling errors and encouraging the Diffusion Transformer to correct errors it generated itself.

3 PRELIMINARIES AND MOTIVATION

The paper attributes long-video failure to a mismatch between error-free flow-matching training and error-included autoregressive inference. It proposes Error-Recycling Fine-Tuning to expose DiT to injected errors and train correction toward clean latents.

  • 3.1 ERROR-FREE HYPOTHESIS IN LONG VIDEO TRAINING: Flow matching trains a DiT to solve an ODE from noise and a clean reference image toward a video latent, assuming error-free intermediate states and trajectories.
  • 3.1 ERROR-FREE HYPOTHESIS IN LONG VIDEO TRAINING: At test time, numerical ODE integration generates video latents step by step from noise over discretized timesteps.
  • 3.2 ERROR-CORRUPTED INFERENCE IN LONG VIDEO GENERATION: Predictive error arises when inference uses an error-containing trajectory, shifting the predicted latent by E = ˆXvid − Xvid after step-wise errors accumulate.
  • 3.2 ERROR-CORRUPTED INFERENCE IN LONG VIDEO GENERATION: Cross-clip conditional error occurs when later clips use error-included frames instead of clean training references, producing out-of-distribution inputs and biased predictions.
  • 3.2 ERROR-CORRUPTED INFERENCE IN LONG VIDEO GENERATION: Predictive and conditional errors reinforce each other during autoregressive conditioning, rapidly causing catastrophic video degradation.
  • 3.3 BRIDGING THE TRAINING-TEST HYPOTHESIS GAP: Error-Recycling Fine-Tuning breaks the error-free hypothesis by using self-generated errors as supervisory prompts so DiT predicts a velocity toward clean latents.
  • 3.3 BRIDGING THE TRAINING-TEST HYPOTHESIS GAP: The closed loop injects errors, calculates and saves them, then dynamically banks and resamples them for new inputs during fine-tuning.

4 STABLE VIDEO INFINITY

Stable Video Infinity trains DiT to correct autoregressive errors by injecting, estimating, and recycling errors during flow-matching fine-tuning. Its pipeline uses bidirectional one-step error estimation, timestep-indexed replay memory, and optional multimodal controls.

  • Error Injection: Clean video, noise, and image latents are extracted, then historical errors are probabilistically injected to simulate inference-time degradation.The resulting noisy latent combines error-injected video and noise latents, while the image latent is supplied separately.
  • Control Injection and Velocity Prediction: SVI predicts velocity from error-injected latents with optional visual and embedding controls, including skeleton, text, and audio.Visual controls enter through token addition, whereas embedding controls use cross-attention in DiT blocks.
  • Prediction Approximation: One-step forward and backward integration approximates video and conditioned-noise predictions, avoiding the cost of solving full ODE trajectories.Residuals between approximated predictions and error-free or error-recycled targets provide bidirectional latent and noise errors.
  • Error Calculation: The method calculates errors for multiple injected-error cases, including clean-start, error-injected start-point, and degraded end-point trajectories.These cases model predictive shifts, cross-clip conditional errors, and accumulated errors requiring trajectory verification and correction.
  • Error Replay Memory: Calculated video and noise errors are dynamically stored in timestep-aligned replay banks and selectively resampled for later clean inputs.The banks align training and test timestep grids, use separate memories for video and noise errors, and cap each bank location at 500 errors.
  • Optimization and Evaluation: SVI is evaluated in generic, audio-conditioned talk, and skeleton-conditioned dance settings, while error-recycling tuning trains only LoRA parameters.The reported setup spans diverse video-generation settings and extends SVI to controlled applications.

5 EXPERIMENTS

SVI is evaluated on consistent, creative, and conditional long-video benchmarks, where it maintains stability over longer durations, supports storyline-driven scene transitions, and adapts to audio- and skeleton-guided generation.

  • Benchmark setup: The evaluation covers consistent, creative, and conditional video-generation settings, including audio-guided talking and skeleton-guided dancing.The consistent setting includes 50-second and 250-second videos, while the creative setting uses storyline-based prompt streams.
  • Quantitative comparison: 5.05% and 3.37% consistency gains and a 5.16% image-quality gain over FramePack are reported for SVI-Shot.These comparisons are reported for consistent video generation under the default setting.
  • Stability with length: 0.63% subject-consistency decrease is observed for SVI when extending to ultra-long videos, compared with larger drops for Wan 2.1 and FramePack.SVI maintains a satisfactory degree of dynamics while most methods experience substantial degradation.
  • Creative generation: SVI achieves the best consistency and quality in storyline-guided creative generation while maintaining satisfactory dynamics and frequent scene transitions.The reported superiority persists in ultra-long settings, where existing long-video methods fail to produce filming-level scene transitions.
  • Stability with length: SVI maintains robust consistency and quality as video length increases, unlike existing methods that exhibit decreasing trends.The authors present this stability as evidence that SVI can generate arbitrary-length videos by correcting accumulated errors.
  • Ablation study: Ablations show that removing reference-image errors causes a significant drop across metrics, while video-latent and noise errors provide auxiliary benefits.The results support intervening at the trajectory start to simulate error accumulation and using error recycling for robustness.

6 CONCLUSION

The conclusion identifies the training–test hypothesis gap as the core challenge in long-video generation and presents SVI as an error-correcting solution evaluated across multiple settings.

  • Conclusion: SVI bridges the training–test hypothesis gap by injecting, storing, and resampling historical errors while learning from autoregressive error feedback.The method predicts an error-recycled velocity using bidirectional one-step integration and replay memory.
  • Conclusion: Across three benchmarks, SVI surpasses state-of-the-art methods for long, ultra-long, and conditional video generation.The conclusion frames these results as breaking the time limit through active correction of self-generated errors.
  • Conclusion: SVI is designed to break the video time limit by actively correcting accumulated self-generated errors.The contribution summary characterizes this as extending video length from seconds to infinity.

A BENCHMARK SETUP

The benchmark setup uses an automated pipeline that converts high-level keywords and images into storyline-based prompt streams, which SVI then synthesizes into narrative videos.

  • Automatic prompt stream engine: The automatic pipeline requires only high-level subject keywords, retrieves relevant images, and uses an MLLM to generate storyline-aligned prompt streams.The resulting image–prompt pairs are supplied to SVI for narrative-driven video generation.
  • Automatic prompt stream engine: The workflow transforms keywords into structured image and prompt-sequence pairs, removing labor-intensive manual annotation and prompt engineering.Figure 8 presents this as an end-to-end automatic pipeline for infinite short-film generation.
  • Keyword-based image retrieval: Keyword-based image retrieval downloads diverse relevant images, while users may alternatively provide customized images.This is the first stage of the automated pipeline.
  • Automated prompt stream generation: Qwen2.5 generates a temporally coherent sequence of L distinct prompts describing plausible dynamic evolution from each input image.The paper calls this sequence a “prompt stream,” rather than a single static description.
  • Input preparation and video synthesis: SVI iteratively generates a video clip for each prompt, conditioning later clips on the last frames of previous generations.The normalized input image and generated prompt stream jointly specify the synthesis process.
  • Benchmark role: The pipeline enables scalable qualitative evaluation by producing diverse test cases that couple static content with dynamic textual guidance.It is used to benchmark the model’s ability to interpret and animate scenes in response to prompt streams.

A.2 BENCHMARK DATASETS

The benchmark datasets combine automatically harvested and user-sourced data to test long-video fidelity, creativity, controllability, and broader practical applications.

  • Dataset construction: Two-thirds of the dataset is harvested from the web automatically, while one-third comes from real users to balance scale, diversity, and authenticity.The automatic prompt-stream engine constructs test data for both creative and consistent generation.
  • Benchmark datasets: The generic benchmark contains 152 samples at 50 seconds and 14 samples at 250 seconds, with identical evaluation conditions across methods.The consistent track uses one stable text prompt paired with each input image, while the creative track uses multiple textual descriptions.
  • Benchmark scope: The benchmark suite jointly stresses fidelity and creativity through consistent and narrative prompt-stream settings, controlled ablations, and cross-method comparisons.It is intended to reflect practical long-video creation requirements beyond homogeneous scenes with repeated motions.
  • Filming and entertainment: SVI is presented as enabling end-to-end single-take filmmaking from high-level intent and brief descriptions, with controllable pacing and plausible transitions.The proposed system autonomously produces unlimited single-shot videos without human intervention.
  • Robotic world models: SVI is positioned as a potential tool for long-duration, controllable, semantically consistent robotic scenario synthesis, especially in navigation.The stated motivation is to address short horizons and limited diversity in existing robotic world models.
  • World generation and spatial AI: The authors connect long-term geometric and identity consistency with applications in world generation, gaming, spatial AI, and persistent-environment reasoning.The pipeline provides narrative-conditioned scene evolution while maintaining structural coherence.

B.2 BROADER METHODOLOGY IMPACT ON ACADEMIA

SVI frames error recycling as a general strategy for bridging the train–test hypothesis gap in autoregressive generation. The paper positions this principle as relevant beyond video, while distinguishing SVI through flexible control and adaptation properties.

  • Broader methodological impact: Error-recycling tuning exposes models to imperfect autoregressive rollouts and teaches recovery, aligning optimization with test-time generation.The paper identifies a mismatch between clean training inputs and error-containing generated context at inference.
  • Broader methodological impact: The train–test hypothesis gap extends beyond video to LLMs, MLLMs, and autoregressive image generation.The paper relates visual drift and compounding artifacts to hallucination in long generated contexts.
  • Broader methodological impact: Compared with concurrent approaches, SVI supports image-to-video applications, flexible scene transitions, unbounded length, and adaptation to arbitrary video generators.These properties are presented as advantages over the cited concurrent long-video methods.

B.4 LIMITATION AND FUTURE WORK

The paper identifies unresolved limitations involving style shifts, real-time streaming, identity consistency, and ethical misuse risks. Future work targets stronger robustness, interactive deployment, persistent identity handling, and safeguards.

  • Scaling Up: Style shifts between test-time images and training data can produce adjacent-clip color shifts.The authors attribute this likely to the model treating low-level style as an error and correcting it incorrectly.
  • Real-Time and Interactive Generation: The current Wan 2.1-based model generates frames in parallel rather than as a stream, challenging real-time deployment.The authors plan streaming generation and interactive controls such as live prompt updates and trajectory guidance.
  • ID Consistency: Without explicit long-term memory, SVI-Film can experience identity drift or swapping after a main character exits the frame.The authors propose persistent identity embeddings, cross-shot feature caching, and scene-aware anchors.
  • Ethical Concerns: Talking-video generation carries potential deepfake and fraud misuse risks, motivating compliance constraints and guardrails for future releases.The study uses openly available data and reports no human-subject, biometric, or biological data involvement.

C QUANTITATIVE EXPERIMENTS

The quantitative experiments probe whether self-generated errors are preferable to naive alternatives and whether error-recycling strength affects performance. They also document the experimental configuration and show that weaker correction is associated with worse metrics.

  • Metric robustness: Naive clip-copying and reference-image-copying designs can fool some consistency and quality metrics.The tested designs repeat clips or reference images rather than generating genuinely extended content.
  • Error design: Handcrafted color shifts, blur, and sharpness fail to help and substantially degrade image quality compared with self-generated errors.Combining handcrafted and self-generated errors causes further severe drops.
  • Error-recycling intensity: Reducing LoRA error-recycling intensity from α = 0.8 to 0.2 consistently decreases all metrics relative to α = 1.The result supports the reported role of active error correction, while the experiment varies test-time LoRA weight.

D IMPLEMENTATION DETAILS

The implementation uses lightweight LoRA fine-tuning with small public datasets and extends SVI across text-stream storytelling and multimodal controls. Qualitative results cover multiple creative and conditional video scenarios.

  • Model Implementation: The implementation is based on Wan2.1-I2V-14B-480P and tunes only LoRA adapters for flexible integration into private models.Models, source code, and benchmark datasets are reported as publicly available.
  • Training Data: SVI is fine-tuned with only 10 epochs on small-scale publicly available data, including 6K MixKit videos for creative and consistent generation.The paper also explores scaling with UltraVideo and uses a random subset for audio-guided talking.
  • Qualitative Applications: Text streams guide temporally coherent short films, while visual and embedding-based controls support long-range synthesis.The reported qualitative results span figures covering storytelling, dancing, talking faces, and other creative scenarios.
  • Qualitative Applications: Qualitative examples include cat, motorcycle, zoo, baby, airplane-landing, dancing, talking-face, and Tom-and-Jerry stories or clips.These figures document the range of showcased applications rather than quantitative comparisons.
Loading 2510.09212v1…