Source-linked AI summary
AdaState: Self-Evolving Anchors for Streaming Video Generation
Yusuf Dalva, Pinar Yanardag
TL;DR
Autoregressive video diffusion’s privileged first-frame anchor can preserve identity while suppressing motion and scene progression. AdaState replaces it with a denoised hidden state carried through the KV cache, and experiments report stronger dynamics and consistency across horizons.
Problem
A privileged first-frame cache anchor attracts disproportionate attention, locking scene composition and dampening motion, camera movement, and progression.
Method
AdaState jointly denoises a non-rendered adaptive state and video content, writing the state’s clean KV to position 0 as an evolving reference.
Results
AdaState preserves dynamics without losing consistency across within- and beyond-horizon evaluation, with the gap to baselines widening as rollout length increases.
Takeaways & Limitations
The method turns denoising and cached attention into a recurrent generation process without architectural change, auxiliary loss, or external memory.
Takeaways & Limitations
A single state frame is sufficient at the current 21-frame training horizon but may not capture longer or more complex scenes requiring structured external memory.
Abstract
from arXiv · showhide
Autoregressive video diffusion models generate streaming video by producing frames sequentially, conditioning each chunk on previously generated content. These models are structurally anchored to the first frame: its key-value representation occupies a privileged position in the attention cache and serves as the primary scene reference throughout generation. As the cleanest and most error-free position in the cache, this anchor draws disproportionate attention, suppressing video dynamics, and locking scene composition to the initial viewpoint even as the scene naturally evolves. The result is a temporally shallow video in which motion, camera movement, and scene progression are dampened in favor of static consistency. To address this, we replace the static anchor with an adaptive state, a hidden latent that the model denoises alongside content at every chunk but never renders. Rather than referencing a frozen first frame, the model generates its own scene anchor at each step by attending to both the previous state and the current content, producing a reference that evolves with the generated content. Unlike standard video generation, which encodes an absolute notion of time, our formulation treats time as relative: every generation step sees the same positional structure regardless of how far generation has progressed, and the state transition is identical at every chunk. Together, these properties introduce a recurrence into the generation process, where denoising serves as the transition function, and the KV cache serves as the carrier, requiring no external module. Experiments demonstrate that the adaptive state substantially improves video dynamics, enabling richer motion and natural scene progression within generated videos.
1 Introduction
Streaming video models over-attend to the first-frame anchor, preserving identity at the cost of motion and scene progression. AdaState replaces this static reference with an evolving hidden state that supports camera and environmental change.
- The first frame occupies a privileged cache position and attracts disproportionate attention, locking generation to its initial viewpoint.This anchor is the cleanest, most error-free cache position, so attention to it suppresses dynamics as scenes evolve.
- Existing static, averaged, and heuristic anchor updates preserve or refresh references without addressing the underlying attention bias.These approaches either retain a fixed anchor, converge toward a blurry mean, or bias continuation toward reproducing past frames.
- AdaState denoises a hidden adaptive state with content at every chunk and writes its clean representation to the cache anchor without rendering it.The state attends to current content and the previous state, producing a reference that evolves with the generated scene.
- Figure 2 shows AdaState maintaining identity while the camera moves and the environment evolves, unlike static references that freeze the scene.The comparison uses a 5-second generation on the same prompt.
2 Related Work
AdaState relates persistent hidden-state methods and latent-token approaches through an unobserved variable carried across steps. Its distinguishing recurrence uses the diffusion model’s denoising process and KV cache rather than a separate transition mechanism.
- Autoregressive video diffusion advances through sequential chunks using causal attention, KV caching, and distillation-based training.The related systems include Self-Forcing, Self-Forcing++, Causal Forcing, Rolling Forcing, and Reward Forcing.
- Persistent-state methods maintain hidden variables that are updated at each step and condition outputs without being directly observed.AdaState shares this structural property with RNNs, state-space models, test-time training, Titans, LaCT, and VideoSSM.
- Latent-reasoning methods use unobserved positions or hidden states for intermediate computation, including scratchpads, filler tokens, and continuous feedback.These approaches shape latent computation through task loss, distillation, or recurrent transformer structure.
- Figure 3 depicts the state as a non-rendered green stream whose clean KV recurs at the anchor while blue content produces continuous video.State previews expose the hidden representation in image space, while the bottom strip shows denoised video output from t=0 to t=12.
- AdaState’s recurrent hidden variable is updated through iterative denoising and carried by the KV cache across generation steps.Its transition is the diffusion model’s own refinement process rather than a learned gate, recurrence matrix, SSM convolution, or gradient update.
3 Method
AdaState jointly denoises content and a hidden state, then routes them differently: content becomes video and short-term cache context, while state becomes the next scene anchor. Horizon-weighted training emphasizes late frames where accumulated drift makes the state most consequential.
- 3 Method: The autoregressive backbone denoises each content chunk from noise while conditioning on clean key-value pairs from prior chunks in a sliding cache.Generation uses a K-step flow-matching schedule and bounded-cost KV caching.
- 3.1 Context Utilization in Self-Forcing: Attention analysis finds a bimodal pattern in which the position-zero anchor and freshest chunk-summary frame dominate, while roughly 70% of positions receive uniform mass.The anchor’s absolute share decreases with cache size, but its relative rank remains high.
- 3.2 Adaptive State: AdaState replaces the frozen position-zero reference with an adaptive state so the model’s dominant reference point evolves with the scene.The state and current content occupy a visible window alongside cached content and state entries.
- 3.2 Adaptive State: State and content start from independent Gaussian noise and are jointly denoised by the generator.The method uses the same generation mechanism for both streams rather than a separate state module.
- 3.2 Adaptive State: After denoising, content is decoded and cached for short-term context, whereas the state is not decoded and overwrites position 0 for the next chunk.The next state attends to this cached representation and current content, making the generator the transition function and the KV cache the carrier.
- 3.2 Adaptive State: The state absorbs current scene information through attention while content queries read it for context evicted from the sliding window.The recurrence uses joint denoising, cache update, and ordinary attention without a separate module or gating mechanism.
- 3.3 Horizon-Weighted Training: A linear frame-index ramp increases DMD loss weight toward later rollout frames, where accumulated drift is greatest.Gradient reaches the adaptive state through content attention rather than a separate state loss, and cross-chunk recurrence is detached at boundaries.
4 Experiments
Across 5- and 30-second evaluations, AdaState preserves video identity while enabling natural motion and scene progression, unlike baselines that either drift, collapse, or freeze. Ablations support a single concentrated state, horizon-specific weighting, and removal of the static anchor.
- 4.1 Qualitative Results: AdaState alone combines temporal stability with natural progression across 12-second portrait and 30-second coastal-drone rollouts.Baselines either accumulate artifacts or reproduce nearly identical compositions throughout generation.
- 4.2 Quantitative Results: At 30 seconds, AdaState occupies the high-consistency, high-dynamics region while most baselines stop moving or lose prompt alignment.Six of nine baselines fall below 0.5 dynamic degree, whereas AdaState posts the highest VBench total and text-video alignment at this horizon.
- 4.2 Quantitative Results: AdaState receives the highest user-study ratings for coherent progression and prompt following from 40 raters.The ranking places no-reference methods lowest, static and EMA references in the middle, and AdaState first.
- 4.3 Ablation Study: A single state slot in a tight content window produces stronger dynamics than multiple state slots in a wider window.The ablation attributes this to concentrating attention at one recurrent position rather than increasing absolute state capacity.
- 4.3 Ablation Study: α=2 maximizes dynamics and total score within the training horizon, while α=4 improves late-frame consistency and imaging quality for long-horizon generation.The two-regime strategy uses α=2 for within-horizon generation and α=4 for long-horizon generation.
- 4.3 Ablation Study: Retaining a static sink alongside the adaptive state suppresses dynamics, showing that removing the static anchor is necessary for the adaptive state to function.The model shortcuts through the clean static reference even when the evolving alternative remains available.
5 Limitations
The evaluated configuration heuristically uses one state frame, and more complex or longer scenes may require structured external memory.
- 5 Limitations: The state capacity is heuristically fixed at Fs=1, while longer rollouts or more complex scenes may require structured external memory.The paper leaves augmenting the adaptive state with such memory for future work.
6 Conclusion
AdaState replaces the static position-zero anchor with a hidden adaptive state that evolves across chunks and is trained to support motion. It requires no architectural change, auxiliary loss, or external memory, and improves dynamics across rollout lengths.
- 6 Conclusion: AdaState replaces the static anchor with a hidden latent denoised alongside content, propagated through the same cached position across chunks.The state is never rendered and evolves through horizon-weighted DMD so the anchor serves motion rather than suppressing it.
- 6 Conclusion: Across within- and beyond-horizon evaluations, AdaState improves dynamics and widens its gap over baselines as rollout length increases.A user study confirms the perceptual ranking for coherent progression and prompt adherence.
- 6 Conclusion: The construction uses no architectural change, auxiliary loss, or external memory while breaking the consistency-dynamics tradeoff.The adaptive state occupies the existing anchor slot and is carried by cached key-value representations.
A Implementation Details
AdaState training modifies cache management and loss weighting while retaining the Self-Forcing backbone. Each autoregressive rollout jointly denoises state and content, updates their cache locations separately, and evaluates the method through a randomized user study.
- Implementation Details: AdaState modifies only cache management and horizon loss weighting, leaving the backbone DiT architecture unchanged.The state shares the content-frame latent space and is processed by the same layers.
- Training algorithm: Each training iteration performs a full autoregressive rollout, jointly denoising state and content frames from independent Gaussian noise.Clean content key-value representations enter the sliding window, while clean state representations overwrite position 0.
- Hyperparameters: The only model-varying hyperparameter is horizon weight α: α=2 supports within-horizon evaluation, whereas α=4 benefits long-horizon generation.Training uses 1000 AdamW iterations on two H200 GPUs, with EMA beginning at iteration 200.
- User study: The user study compares AdaState with four baselines spanning no-anchor, static-anchor, and EMA-anchor categories.Forty raters each evaluate 20 randomized, identity-hidden videos covering 5-second and 30-second generations.
- User study: Videos receive 5-point Likert ratings for coherent progression and prompt adherence, with method identity hidden and presentation order randomized.The interface evaluates camera motion, subject action, scene development, and realization of the text prompt.
C Evaluation Details
The evaluation reports full VBench breakdowns at 5 and 30 seconds using prompt sets designed for general scenes and long-horizon moving-camera stress tests. Across dimensions, AdaState achieves the highest total score while preserving a balance between consistency and dynamics.
- Quantitative results: AdaState achieves the highest total VBench score at both 5 and 30 seconds without catastrophic degradation on any single dimension.Its lowest relative standing is on temporal flickering and subject consistency.
- Quantitative results: Static-anchor methods lead consistency and flickering metrics, whereas anchor-free methods favor dynamics, revealing a recurring tradeoff across dimensions.Dynamic degree is a binary per-clip metric, so its standard deviation is inherently high.
- Quantitative results: Tables 4 and 5 report ten VBench dimensions at 5 seconds and 30 seconds, with VisionReward reported only at 5 seconds.The tables mark best and second-best results using bold and underlining, respectively.
- Evaluation prompts: The 5-second evaluation uses MovieGenBench Extended prompts spanning static and dynamic scenes, subjects, and styles.The 30-second set uses aerial or drone-style moving-camera prompts over landscapes and cityscapes to stress long-horizon dynamics.
D Supplementary Video Results
Supplementary videos show AdaState maintaining coherent progression and natural motion within and beyond its training horizon. Ablations further indicate that a single tight state slot and horizon-specific weighting strengthen dynamics and long-horizon stability.
- 5-second generation: Within the 5-second training horizon, AdaState produces coherent scene progression with natural camera motion and subject action across diverse prompts.Examples include nature shots, character animation, fantasy scenes, and stylized content.
- 12-second generation: Beyond the 5-second training horizon, 12-second generations maintain temporal coherence and scene progression without drift or freezing.Examples include cinematic portraits, aerial views, and detailed close-ups.
- Ablation: state size and window: The final Fs=1, Wp+F=6 configuration produces the strongest dynamics compared with larger-state or wider-window alternatives.The comparison supports concentrating the state at a single recurrent position within a tight content window.
- Ablation: horizon weight α: At 5 seconds α=2 produces the richest dynamics, while at 30 seconds α=4 yields the most stable long-horizon quality.The results support using different horizon weights for short- and long-horizon regimes.
- 30-second comparisons: Across 30-second comparisons, AdaState alone maintains both dynamics and coherence while baselines either freeze scenes or accumulate artifacts.The comparisons span no-anchor, static-anchor, EMA-anchor, and AdaState methods at 120 frames.
E Attention Analysis Details
The attention analysis measures how cached-frame attention is distributed across layers, chunks, denoising steps, prompts, and cache depths. It separates current-chunk self-attention from cached past and excludes clean-input cache reruns from sampling-time records.
- Per-record frame mass: Attention mass is averaged over heads and a uniformly spaced query subsample of 32 queries for each layer, chunk, denoising step, and prompt record.Each record corresponds to a unique (layer, chunk, denoising step, prompt) tuple.
- Off-diagonal partition: The current chunk occupies the last 3 K-frames, while the off-diagonal cached past covers the preceding K-frames.The analysis partitions attention between current-chunk self-attention and cached positions.
- Off-diagonal partition: Renormalization treats the entire current chunk as self-attention, placing all cached positions on equal footing regardless of total self-attention mass.This avoids excluding only the diagonal element when computing the cached-position share.
- Aggregation: The analysis excludes t=0 cache reruns because clean inputs follow a different attention regime from sampling-time noisy queries.This exclusion keeps the analysis focused on sampling-time behavior.
- Analysis setting: The setting uses 5 prompts, 21 generated frames, 7 chunks, 3-frame chunks, 4 denoising steps, 30 layers, 12 heads, and 3,600 records.The records comprise 600 per cache depth after excluding cache reruns; anchor rank averages 2–3 across layers with standard deviation below 1.
F Limitations and Future Work
The paper identifies state capacity, cross-domain applicability, and generative-media risks as boundaries for AdaState. It also reports a lightweight fine-tuning setup and describes its training data and use of language models.
- State capacity: A single-frame state is sufficient at the current 21-frame training horizon but may lack capacity for longer rollouts with multiple actors or concurrent events.The paper proposes multi-slot states or explicit external memory as future directions.
- Broader applicability: AdaState’s broader applicability beyond video remains untested, including possible use in audio, 3D, and long-context language generation.The paper leaves cross-domain investigation for future work.
- Practical scope: The method requires 1000 fine-tuning iterations on two GPUs, supporting its use by academic and small-scale research groups.The stated accessibility claim concerns the fine-tuning framework.
- Potential risks: Improved realism and temporal coherence could lower barriers to disinformation or non-consensual deepfakes, so existing safeguards remain relevant.The paper recommends provenance tracking and watermarking alongside generation advances.
- Data and human subjects: Training uses public text prompts and no personal, sensitive, or copyrighted visual data; the user study used informed consent and anonymous participant identifiers.Participants were compensated above the platform’s recommended minimum.
- LLM usage: Language models assisted with engineering and evaluation tasks but not with designing AdaState, formulating its adaptive state, or analyzing results.The stated usage covers codebase setup, experiment scripting, and evaluation pipelines.