Source-linked AI summary
TinyHistory: Lightweight Video History Embeddings via Two-Stage Context Learning
Lvmin Zhang, Shengqu Cai, Muyang Li, Chong Zeng, Beijia Lu, Anyi Rao, Song Han, Gordon Wetzstein, Maneesh Agrawala
TL;DR
Autoregressive video generation needs history embeddings that preserve consistency and cover long histories under tight compute and memory budgets. TinyHistory learns lightweight embeddings through randomized-frame-query pretraining followed by repurposing in an autoregressive video diffusion model. The method achieves consistency comparable to heavier alternatives while reducing training overhead and extending encodable history length within a given memory budget.
Problem
Long video histories in autoregressive generation must balance content and identity consistency, uninterrupted coverage, query capability, visual fidelity, and memory cost, including in personal and offline workflows.
Source: §1 · pp. 1–2Method
Results
TinyHistory achieves content consistency, identity preservation, and user-preference scores comparable to heavier alternatives while reducing training overhead and extending encodable history length within a given memory budget.
Source: §1 · pp. 2–3Takeaways & Limitations
The training data distribution can shape which content the constrained history embedding prioritizes, such as scene appearance for game footage or faces and characters for cinematic data.
Source: §1 · p. 2
Abstract
from arXiv · showhide
History context is central to autoregressive video generation, driving consistency and storytelling for both commercial models and personal use cases. For example, personal users, offline workflows, and individual-scale finetuning need to encode longer video histories under tight compute and memory budgets. We observe that content and identity consistency is an essential requirement, and that complete, uninterrupted history coverage together with content query and interpretation capabilities is broadly desired. We present TinyHistory, a lightweight history embedding learned through two-stage context learning. In the first stage, we pretrain the encoder on large-scale video data with a randomized frame query objective; in the second stage, we repurpose the pretrained encoder within an autoregressive video diffusion model to learn content-level consistency. As a result, we show that the learned lightweight embeddings achieve consistency comparable (by VLM, VBench, ELO, etc) to heavier alternatives, while reducing training overhead and extending the encodable history length within a given memory budget. We conduct ablation studies to analyze the influence and trade-offs of each component.
1 Introduction
Autoregressive video generation needs history embeddings that preserve identity and content while covering long histories under tight compute and memory budgets. TinyHistory uses two-stage context learning and reports consistency comparable to heavier alternatives, with reduced training overhead and longer encodable histories at a given memory budget.
1 Introduction
- The approach addresses long-history demands affecting both commercial-scale systems and personal use cases, including offline workflows and individual-scale finetuning.The introduction reports quantitative evaluations, user studies, and ablations to analyze component trade-offs.Source: §1 · pp. 1, 3
- Existing approaches trade off history coverage, visual fidelity, and memory cost: windows lose distant history, compression loses detail, and efficient attention leaves the memory footprint unchanged.The introduction also notes detail loss from higher token-merging rates.Source: §1 · pp. 1–2
- Practical history embeddings must preserve content and identity, cover the full temporal extent without interruption, and support content query and interpretation.These demands address consistency, continuity, and task-specific prioritization within a constrained feature manifold.Source: §1 · p. 2
- TinyHistory pretrains an encoder with randomized frame queries, then repurposes it within an autoregressive video diffusion model to learn content-level consistency.The training distribution can shape which content the constrained feature manifold prioritizes, such as scene appearance or character consistency.Source: §1 · p. 2
- Comparable content consistency, identity preservation, and user preference scores are achieved against heavier alternatives while reducing training overhead and extending history length within a memory budget.The reported measures include VLM evaluation, VBench, and ELO.Source: §1 · pp. 2–3
2 Related Work
Related work addresses long video generation through diffusion length extension, history retrieval or compression, and more efficient attention and token processing. These approaches target longer contexts through different mechanisms, including persistent memory, compact representations, and compute-efficient designs.
Autoregressive video diffusion and long videos
- Long-video diffusion work extends generation horizons through noise or frequency rescheduling, variable-horizon conditioning, memory blending, flexible attention windows, distillation, and rollout-oriented training.Examples include Diffusion Forcing, HistoryGuidance [49], StreamingT2V [23], FAR [20], CausVid [68], StreamDiT, and Self-Forcing.Source: §2 · p. 3
Context learning and history representation
- History representation methods use retrieval or compression, including persistent view or spatial memory, learned context routing, packed frames, and recurrent or learned states.Examples include WorldMem, VMem, Mixture-of-Contexts, FramePack, and StateSpaceDiffuser.Source: §2 · pp. 3–4
Efficient video diffusion designs
- Efficient video diffusion designs improve throughput or reduce token and latent costs through sparse attention, learned routing, token merging, compact tokenizers, and compressed latent spaces.The cited designs range from FlashAttention and sparse patterns to SANA-Video.Source: §2 · p. 4
3 Method
TinyHistory maps video history to a lightweight embedding that conditions an autoregressive DiT generator. Its method combines an encoder designed to output in DiT hidden states, randomized-frame-query pretraining for temporal coverage, joint repurposing for content consistency, and incremental inference encoding.
3 Method
- TinyHistory learns an encoder that maps history H to embedding ϕ(H), which conditions an autoregressive DiT generator.The method uses a two-stage context-learning process: large-scale video pretraining followed by repurposing in an autoregressive video diffusion model.Source: §3 · p. 4, Figure 1 · p. 5
3.1 History Encoder Architecture
- The dual-branch encoder combines a downsampled-history path using the DiT’s VAE, patchifier, and first-layer projection with a lightweight original-resolution path, summing their features in DiT hidden states.This output bypasses the VAE channel bottleneck; an optional cross-attention refinement is also discussed.Source: §3.1 · p. 5
3.2 Stage I: Pretraining with Frame Query Coverage
- Stage I randomly samples frame indices as query targets, masking other history frames with noise so the DiT learns to query selected frames from the encoded history.Random positions encourage dense temporal coverage, while partial supervision avoids the overhead of training on all frames at once.Source: Figure 1 · p. 5, §3.2 · p. 6
3.3 Stage II: Repurposing for Content Consistency
- Stage II uses complete, noise-free histories and jointly optimizes the encoder with the LoRA-adapted DiT to produce the next segment and learn content-level consistency.The encoder remains trainable so its features can shift from frame-query targets toward content consistency, with priorities shaped by the training data distribution.Source: §3.3 · pp. 6–7
3.4 Autoregressive Inference
- At inference, newly generated segments can be encoded independently and concatenated to the existing embedding without recomputing the full history.The encoder’s nearly fully convolutional design reduces per-step encoding cost.Source: Figure 1 · p. 5, §3.4 · p. 7
4 Experiments
Experiments compare TinyHistory with seven history-handling paradigms, evaluate consistency through VLM, VBench, and preference measures, and test ablations and compatible extensions.
4.1 Experimental Details
- Stage I pretraining uses 8×H100 GPUs and Stage II LoRA training uses 1×H100 or A100; training covers about 5 million videos, with 1,300 prompts and 4,096 unseen test videos.Quantitative metrics are computed on videos generated from the test prompts.Source: §4.1 · pp. 7–8
4.2 Evaluation Metrics
- Evaluation combines gated VLM ratings of character, scene, object, and clothing consistency with VBench alignment, identity, and motion metrics, plus pairwise ELO preference.The VLM scores average yes/no attribute answers across gated-in videos; ELO uses Bradley-Terry ratings.Source: §4.2 · pp. 8–9
4.3 Comparison Baselines
- Seven history strategies span sliding windows, patchification, sparse keyframes, retrieval, image embeddings, feature compression, and external VAEs; comparisons use shared Wan 2.1 data and align context lengths where tunable.The baseline implementations use the same base model and training data for a controlled comparison.Source: §4.3 · p. 9
4.4 Results
- At 780 tokens/s, TinyHistory (2×2×2) leads Char, Obj, Semantic, FaceID, and ELO, and ranks second in Scene and Cloth.Table 1 reports the comparison on Wan 2.1 at 480p.Source: Table 1 · p. 10, §4.4 · p. 10
4.5 Ablation Studies
- Without Stage I, Char drops 80.19→52.64 and FaceID 70.71→58.38; freezing the pretrained encoder leaves Char at 61.42.Figure 4 contrasts preserved features with pretraining against identity and object inconsistency without it; frozen features require joint Stage II repurposing.Source: §4.5 · pp. 11–12, Figure 4 · p. 12
- Removing LR lowers Char to 67.73 versus 74.29 for removing HR; the dual-branch design reaches 80.19, while 2×2×2 matches 2×2×1 ELO (1332 vs 1342) at half the context cost.The rate comparison is 780 versus 1560 tokens/s; 4× spatial rate yields lower Char than 4× temporal rate (80.19 vs 90.24).Source: §4.5 · pp. 12–13, Table 2 · p. 12
4.6 Extensions and Compatibility
- Sliding-window overlap raises Scene by 10.3 points (72.21→82.51), while cross-attention raises Obj by 6.4 (74.38→80.75).The sliding window uses a 3-frame latent overlap; cross-attention adds a connection from the history encoder to DiT layers.Source: Table 3 · p. 13, §4.6 · p. 13, Figure 5 · p. 14
- The two encoders’ concatenated embeddings yield Char +8.1, Obj +9.0, and FaceID +2.7, with a longer context of 195+195 = 390 tokens/s total; KV caching speeds inference while lowering FaceID −7.1 and Scene −6.9.Because the history embedding is fixed once encoded, its key-value pairs can be cached across diffusion steps.Source: §4.6 · p. 13
5 Conclusion
TinyHistory combines two-stage context learning with a modular history-embedding design; experiments report strong evaluation results using shorter context than heavier alternatives.
5 Conclusion
- TinyHistory leads five of eight evaluation dimensions and ranks second in two others, using shorter context than heavier alternatives.The reported leading dimensions are Char, Obj, Semantic, FaceID, and ELO; Scene and Cloth rank second.Source: §5 · p. 14
- Stage I pretrains the encoder with randomized frame queries for dense history coverage, while Stage II jointly repurposes it with the DiT generator for content-level consistency.Source: §5 · p. 14
- Ablations confirm contributions from pretraining, joint encoder training, and dual-branch design, while encoding-rate trade-offs follow diminishing returns.Source: §5 · p. 14
- Sliding-window overlap, cross-attention, multiple encoders, and KV-caching can each be added independently.Source: §5 · p. 14