Source-linked AI summary
Frame Context Packing and Drift Prevention in Next-Frame-Prediction Video Diffusion Models
Lvmin Zhang, Shengqu Cai, Muyang Li, Gordon Wetzstein, Maneesh Agrawala
TL;DR
Next-frame video generation must balance remembering earlier content against preventing errors from accumulating during autoregressive prediction. The paper introduces FramePack, which compresses frame contexts by importance within a fixed context length and adds anti-drifting sampling and training methods. Experiments report support for thousands of frames, compatibility with pretrained video diffusion models, and improved responsiveness and training batch capacity, while noting tail-frame constraints and a modeling assumption favoring bidirectional prediction.
Problem
Forgetting and drifting are critical problems in next-frame video generation, and mitigating one can worsen the other through competing effects on memory and error propagation.
Method
FramePack progressively compresses input frames according to time proximity or feature similarity, while planned endpoints, adjusted sampling, and discrete history representation target drifting.
Results
FramePack can process thousands of frames, finetune existing pretrained video diffusion models, and improve responsiveness and training batch capacity.
Takeaways & Limitations
The approach provides a fixed-context memory structure combined with anti-drifting methods for scalable next-frame and bi-directional video generation.
Takeaways & Limitations
Extremely long inputs can push tail frames below the minimum unit size, and the paper offers several tail-processing options whose visual differences are negligible in tests.
Abstract
from arXiv · showhide
We present a neural network structure, FramePack, to train next-frame (or next-frame-section) prediction models for video generation. FramePack compresses input frame contexts with frame-wise importance so that more frames can be encoded within a fixed context length, with more important frames having longer contexts. The frame importance can be measured using time proximity, feature similarity, or hybrid metrics. The packing method allows for inference with thousands of frames and training with relatively large batch sizes. We also present drift prevention methods to address observation bias (error accumulation), including early-established endpoints, adjusted sampling orders, and discrete history representation. Ablation studies validate the effectiveness of the anti-drifting methods in both single-directional video streaming and bi-directional video generation. Finally, we show that existing video diffusion models can be finetuned with FramePack, and analyze the differences between different packing schedules.
1 Introduction
Next-frame video generation faces a trade-off between retaining long-range temporal information and preventing accumulated prediction errors. FramePack addresses this challenge by compressing frame contexts according to importance and combining the structure with anti-drifting training and sampling methods.
- Forgetting weakens memory of earlier content, whereas drifting degrades visual quality through error accumulation over time.
- Enhancing memory can increase error propagation, while interrupting error propagation can weaken temporal dependencies and worsen forgetting.
- Encoding more frames naively is computationally intractable because transformer attention has quadratic complexity and consecutive frames contain redundant visual features.
- FramePack compresses input frames by relative importance so transformer context length converges to a fixed upper bound, using temporal proximity or feature similarity.
- Anti-drifting methods break causal prediction chains with planned endpoints and bidirectional contexts, while history discretization reduces training–inference history disparity.
- Existing pretrained video diffusion models can be finetuned with FramePack, whose implementations process thousands of frames with 13B models on laptops using 6GB or 8GB GPU memory.
2 Related Work
Prior work extends video generation length through latent diffusion, noise rescheduling, temporal processing, planning, distributed generation, and memory mechanisms. These approaches address efficiency, continuity, or history reconstruction through different representations and conditioning strategies.
- Noise scheduling and history-frame augmentation mitigate drifting by reducing dependence on past frames, while related ablations examine added history noise and guidance-based schedules.
- Anchor-based approaches use reference images or video as planning elements for content planning and extendable generation.
- Latent-space compression methods improve video diffusion efficiency through variable token context, compressed latents, pyramidal processing, or multi-level causal attention.
- World-model memory mechanisms model long-term information with 3D geometry, domain-trained parameters, low-rank adaptation, or retrieval-based reconstruction.
- Long-video methods include latent diffusion, variable-length generation, temporal co-denoising, noise rescheduling, coarse-to-fine diffusion, distributed generation, and streaming generation.
3 Packing Frame Context
FramePack packs next-frame prediction contexts by assigning longer representations to more important frames and progressively compressing less important history. Importance can follow time proximity, feature similarity, or a hybrid schedule while keeping context bounded as history grows.
- Context setup: FramePack models next-frame-section prediction with S unknown output frames conditioned on T input frames, focusing on T much larger than S.Vanilla DiT context grows as L = Lf(T + S), creating context-length pressure for long histories.
- Time-proximity packing: Time-proximity packing ranks recent frames as more important and assigns each frame a context length through progressive compression.The length function is applied after VAE encoding and transformer patchifying.
- Packing schedules: The compression parameter λ > 1 controls frame-wise patchifying, with different kernel structures achieving equivalent compression rates.Examples include multiple 3D kernels for the same compression rate, plus temporal, duplicated, and symmetric packing variants.
- Bounded context: As T approaches infinity, total context length converges to (S + λ/(λ−1)), making the compression bottleneck invariant to input frame count.FramePack therefore maintains a fixed upper-bound context in the theoretical limit.
- Tail handling: Extremely long inputs can produce tail frames below the minimum unit size, handled by deletion, slight context growth, or global average pooling.The authors report relatively negligible visual differences among these three tail-processing options.
- Similarity-based packing: History frames can be reordered by feature similarity or a hybrid similarity-plus-time score, with hybrid sorting designed to transition smoothly as the generating window advances.The hybrid approach is presented for datasets requiring revisiting views or maintaining consistent identities.
4 Drift Prevention
The paper treats drifting as quality degradation that grows with video length and proposes sampling and history-representation methods to reduce error propagation. These methods include endpoint planning, inverted sampling, multiple endpoints, and discrete history.
- Drifting degrades visual quality as video length increases, motivating anti-drifting changes to sampling and history representations.
- Endpoint planning: Endpoint planning generates beginning and ending sections first, then fills the intervening gaps to reduce drifting in repetitive or limited-motion content.
- Inverted sampling: Inverted sampling is effective for image-to-video generation because later generations remain oriented toward the high-quality groundtruth input frame.
- Multiple endpoints: Multiple endpoints support more dynamic motions and complicated storytelling by planning several prompted sections before filling their gaps.
- History discretization: History discretization replaces continuous history frames with codebook-quantized representations during training to reduce the training-inference mode gap.
- History discretization: A suitable codebook size K can reduce error propagation while preserving plausible consistency between sections, whereas K = 1 removes memory and K →∞ retains drifting.
5 Experiments
Experiments evaluate FramePack configurations, sampling strategies, history discretization, and alternatives using global, drifting, and human-preference metrics. Inverted anti-drifting sampling leads drifting metrics, while discrete-history vanilla sampling remains competitive and preserves a larger dynamic range.
- Configuration notation: FramePack notation encodes frame counts, compression kernels, tail handling, skipped frames, sampling order, and history discretization.
- Compared configurations: The experiments compare vanilla, discrete-history, anti-drifting, and inverted anti-drifting sampling configurations implemented with HunyuanVideo and Wan.
- Evaluation setup: The evaluation uses 512 text-to-video prompts and 512 image-prompt pairs, with default durations of 30 seconds for long videos and 5 seconds for short videos.
- Metrics: The reported metrics cover clarity, aesthetics, motion, dynamic behavior, drifting, and human preferences measured with ELO-K32 scores.
- Ablation results: The inverted anti-drifting sampler achieves the best results in 4 of 7 metrics and leads all drifting metrics, but has a relatively small dynamic range.
- Ablation results: Vanilla sampling with discrete history achieves highly competitive human scores while maintaining a much larger dynamic range.
- History discretization: K = 128 provides strong drift reduction with relatively minimal training difficulties.
- Alternative architectures: In comparisons with relevant methods, inverted anti-drifting sampling leads drifting metrics, discrete-history vanilla sampling remains competitive, and human perception prefers the two proposed candidates.
6 Conclusion
FramePack addresses the forgetting-drifting dilemma through importance-based progressive frame compression, history discretization, and planned bi-directional sampling. The paper reports fixed-bounded context processing, compatibility with existing video diffusion models, and support for large frame counts and higher training batch sizes.
- FramePack progressively compresses input frames according to importance so context length converges to a fixed upper bound.
- The approach combines time-proximity-based and feature-similarity-based packing with history discretization and bi-directional context planning.
- Experiments suggest that FramePack can process many frames, improve model responsiveness, and allow higher training batch sizes.
- FramePack is compatible with existing video diffusion models and supports multiple compression variants for broader applications.