PaperlayerView exact revision on arXiv

Source-linked AI summary

In-Flight KV Cache with Clean Anchors for Faster Autoregressive Video Diffusion

Yikai Wang, Xiao Han, Mengmeng Xu, Juan Camilo Perez, Yiannis Douratsos, Sen He, Zijian Zhou, Fei Zhang, Zhaochong An, Juan-Manuel Perez-Rua, Chen Change Loy, Tao Xiang

arXiv:2609.32540v1cs.CVcs.AIcs.LGcs.MM

TL;DR

In few-step autoregressive video diffusion, additional forwards used only to prepare KV memory become an increasingly important bottleneck. FlashForward reuses each ordinary renderer forward’s same-stage KV and adds sparse clean anchor KV as complementary cross-chunk memory. For 16 FPS videos of 20 seconds or longer in up to four-GPU settings, FlashForward is 1.16–1.69× faster than HiAR and 1.42–2.92× faster than Self-Forcing.

  • Problem

    In few-step autoregressive video diffusion, additional forwards used only to prepare KV memory become an increasingly important bottleneck.

    Source: §1 · p. 1

  • Method

    FlashForward reuses each ordinary renderer forward’s same-stage KV and adds sparse clean anchor KV as complementary cross-chunk memory.

    Source: §1 · pp. 1–2

  • Results

    For 16 FPS videos of 20 seconds or longer in up to four-GPU settings, FlashForward is 1.16–1.69× faster than HiAR and 1.42–2.92× faster than Self-Forcing.

    Source: §1 · p. 3

  • Takeaways & Limitations

    For the 1.3B model at 480p, FlashForward scores 0.838 on VBench Total versus 0.805 for Self-Forcing and 0.821 for HiAR, with quality stable as duration increases to 65 seconds.

    Source: §1 · p. 3

    FlashForward targets long-sequence, few-step generation on multiple GPUs; warm-up reduces its gains for short videos, and one-GPU inference cannot use wavefront parallelism.

    Source: §4.2 · p. 9

Abstract

from arXiv · show

Few-step autoregressive video diffusion generates a long video by splitting the video into temporal chunks and generating chunk-by-chunk, each through a short sequence of denoising stages. To memorize chunks that are already generated, previous methods reconstruct a clean or less-noisy key--value (KV) cache by additional forwards to build the cache without advancing an output latent. However, every denoising forward itself already computes the in-flight KV of the current chunk. We introduce FlashForward, which directly reuses this cache to avoid the heavy cache-update-only model forwards. After the current chunk completes one denoising stage, its stage-specific cache is already available for the next chunk. Assigning one GPU to each stage therefore lets different chunks occupy different stages concurrently. This early availability has a quality cost: the resulting stage-matched history is noisy, causing appearance and motion drift among chunks. To complement it, FlashForward produces sparse auxiliary clean anchor latents before the corresponding region is generated so the generation trajectories can be stabilized by this two-sided conditioning. The two memories operate at different temporal scales: sparse clean anchor KV supplies coarse, long-range two-sided structural guidance, while dense stage-matched history preserves fine, recent evolution. With up to four GPUs, FlashForward runs $1.16$--$1.69\times$ faster than HiAR and $1.42$--$2.92\times$ faster than Self-Forcing for 16 FPS videos of 20 seconds or longer across 1.3B and 14B backbone scales at 480p and 720p. On VBench, for the 1.3B model at 480p, it achieves higher scores and remains stable at longer durations, demonstrating that FlashForward generates high-quality and temporally consistent videos across durations of 20s, 35s and 65s at a much faster generation speed.

1 Introduction

Few-step autoregressive video diffusion makes cache-only forwards an increasingly important latency bottleneck. FlashForward reuses in-flight KV and pairs it with sparse clean anchors; it reports faster generation and stable long-video quality.

1 Introduction

  • FlashForward reuses the current chunk’s in-flight KV instead of spending additional forwards solely to construct cache memory.Self-Forcing re-encodes completed clean chunks, while HiAR re-encodes context at every denoising stage.Source: §1 · pp. 1–2
  • Early stage-matched history is noisy and can propagate appearance and motion uncertainty, so FlashForward complements it with sparse clean anchors planned ahead.Clean anchors provide a two-sided memory alongside the promptly available history.Source: §1 · p. 2
  • The design assigns coarse, longer-range structure to sparse clean anchor KV and fine, recent appearance and motion changes to dense renderer history.Only sparse planner blocks require a clean-endpoint cache-extraction forward, and each local anchor window conditions multiple renderer chunks.Source: §1 · pp. 2–3
  • A shared generator backbone serves planner and renderer roles, trained through packed supervised fine-tuning and self-rollout distillation.The two-phase training establishes the planner–renderer graph from real videos, then adapts generation to generated context.Source: §1 · p. 3
  • Across backbone scales on 480p and 720p videos, for 16 FPS videos of at least 20 seconds in up to four-GPU settings, FlashForward is 1.16–1.69× faster than HiAR and 1.42–2.92× faster than Self-Forcing.For the 1.3B model at 480p, FlashForward scores 0.838 on VBench Total, versus 0.805 for Self-Forcing and 0.821 for HiAR.Source: §1 · p. 3

2 Related Work

Prior methods differ in what cross-chunk representation they publish and when it becomes available, shaping both cache-update overhead and chunk execution order. FlashForward combines same-stage renderer history with sparse clean anchors.

2 Related Work

  • FlashForward changes the representation passed between autoregressive units and, with it, their execution order.Video generators use varied diffusion or flow objectives and backbones, while few-step methods use implicit sampling, distillation, or consistency training.Source: §2 · p. 3
  • Self-Forcing publishes clean KV after a cache-only forward, N-C-Causal-rCM reuses the final denoising KV but remains serial, and HiAR pipelines chunks while re-encoding context at each stage.FlashForward instead publishes every ordinary renderer forward’s KV for later chunks at the same denoising stage, forming a wavefront without output-free re-encoding.Source: §2 · pp. 3–4
  • Sparse-planning and two-sided-infill methods use anchors for temporal coherence; FlashForward repurposes this conditioning as coarse-timescale state alongside fine-timescale renderer history.Its auxiliary anchor latents are conditioning-only, so the renderer still generates every final position.Source: §2 · p. 4
  • Noise schedules determine when chunks update, whereas the cross-chunk state rule determines which earlier representation conditions each update.HiAR sustains an anti-diagonal schedule through per-stage context re-encoding; FlashForward’s same-stage dependencies produce a wavefront without re-encoding output chunks.Source: §2 · p. 4

3 Stage-Matched History with Clean Two-Sided Anchors

FlashForward first plans sparse clean anchors, then renders dense chunks using both nearby anchor KV and stage-matched history. Its wavefront pipeline uses shared planner and renderer roles trained for generated contexts.

3 Stage-Matched History with Clean Two-Sided Anchors

  • For an 81-latent example, the planner creates nine anchors at positions 0, 10, …, 80, while the renderer generates 27 three-latent chunks using local anchor windows and in-flight memory.Each completed planner block receives one clean-KV extraction forward; the renderer’s next chunk can follow one stage behind as the current chunk traverses four stages.Source: §3 · p. 4, Figure 2 · p. 5
  • Interior renderer chunks read clean KV from anchors before and after their region, while boundary chunks use the available boundary-side anchors.In the empirical configuration, planner blocks and renderer chunks contain three latents, with anchor stride 10 and four denoising stages.Source: §3 · pp. 4–5

3.1 Stage-Matched Renderer History Enables Pipelined Rendering

  • A renderer forward reads same-stage history from earlier chunks and clean KV from nearby anchors, predicts a latent update, and publishes its own KV for later chunks.Renderer history retains up to five preceding chunks at the corresponding denoising stage.Source: §3.1 · p. 5
  • Noisy, dense stage-matched history carries recent appearance and motion, while sparse clean anchors provide stable coarse structure over a longer interval.A planner transition advances 30 latents versus 3 for a renderer transition, nearly a tenfold reduction in autoregressive depth; the coarse plan also improves long-horizon temporal coherence.Source: §3.1 · pp. 5–6
  • With one worker per denoising stage, same-stage dependencies form a wavefront that completes one renderer chunk per pipeline round after warm-up.The schedule avoids HiAR’s repeated stage-specific context re-encoding; dedicating one GPU to planning lets rendering begin after the first anchor block.Source: §3.1 · p. 6

3.2 Training a Shared Planner and Renderer

  • A shared backbone uses role-specific embeddings and LoRA adapters; supervised fine-tuning establishes both roles from real videos, then self-rollout distillation adapts them to generated context.The self-rollout phase uses a planner–renderer rollout and a distribution-matching objective, with gradients flowing from later chunks to earlier chunks and from renderer to planner.Source: §3.2 · pp. 6–7

4 Experiments

FlashForward is evaluated for video quality and denoising-path speed against autoregressive baselines, with analyses of its conditioning design and limitations. It is fastest from 20 seconds onward and maintains quality across the tested durations.

4.1 Fast and High-Quality Generation

  • From 20 s onward, using each method’s fastest setup with up to four GPUs, FlashForward is the fastest autoregressive schedule in every model–resolution setting, 1.16–1.69× faster than HiAR and 1.42–2.92× faster than Self-Forcing.Source: §4.1 · p. 8
  • FlashForward leads in VBench-Long Total and Quality at every tested duration, while semantic alignment remains competitive and behavior is stable across durations.On VBench-1.0, it has the highest Total and Quality scores and the highest Semantic score among distilled autoregressive methods.Source: §4.1 · pp. 8–9
  • Qualitative comparisons show FlashForward preserving temporal coherence and motion, while Self-Forcing accumulates color drift and HiAR exhibits other appearance changes.The comparison covers 20, 35, and 65 seconds; Figure 5 marks anchor frames in blue, and baselines have no anchors.Source: Figure 5 · p. 8, §4.1 · p. 9

4.2 Further Analysis

  • Time-rebased supervision improves initialization for distillation, and role specialization reduces seams between anchor and non-anchor positions.The training ablations are reported in the top of Table 3.Source: §4.2 · p. 9, Table 3 · p. 9
  • Stage-matched history alone flickers so heavily that VBench cannot score its temporal-flickering dimension, while combining it with clean history and future yields better quality and faster generation.Clean history and future alone mitigates Self-Forcing drift but retains high latency, so it is used as sparse conditioning rather than dense generation.Source: §4.2 · p. 9
  • FlashForward targets long-sequence, few-step generation on multiple GPUs; planner and pipeline warm-up reduce its gains for short videos, and one-GPU inference cannot exploit wavefront parallelism.Training only on real-world videos may limit performance on animation and other stylized domains; fixed anchor stride and conditioning window are also current design constraints.Source: §4.2 · p. 9

5 Conclusion

FlashForward treats cross-chunk memory as a state-availability problem, combining early stage-matched renderer history with sparse clean anchors. The resulting temporal-scale allocation enables inter-chunk stage pipelining, and the paper reports faster generation with better quality and temporal coherence in its evaluated settings.

5 Conclusion

  • Across 1.3B and 14B backbones at 480p and 720p, FlashForward is the fastest evaluated autoregressive schedule on four GPUs for outputs of 321 RGB frames or more.On four H100 GPUs with the 1.3B model at 480p, it generates videos faster than other methods with better quality and temporal coherence across several durations.Source: §5 · p. 10
  • The state-availability contract publishes ordinary renderer-forward KV for same-stage reuse, enabling an inter-chunk wavefront, while sparse clean anchor KV supplies a complementary coarse structural reference.The renderer uses clean anchor KV as conditioning and regenerates every final position.Source: §5 · p. 10

Appendix

  • The appendix covers wavefront execution and efficiency analysis, anchor-source and seam analysis, and training and implementation details.It also points to an interactive demo comparing generation quality and latency and illustrating the generation process and training phases.Source: Appendix · p. 14

A Generation Efficiency Analysis

The appendix describes batched multi-device wavefront generation, in which renderer workers handle different stages and chunks progress through the pipeline. Anchor spacing and planner rate are chosen so clean anchors remain available ahead of rendering.

A.1 Wavefront Execution and Anchor Spacing

  • In the four-stage algorithm, rank 0 samples noise and each higher rank receives the preceding rank’s latent from the prior round; the output latents are produced by the renderer on the last rank.The schedule keeps the most recent renderer-history chunks in the stage-specific cache bank.Source: §A.1 · p. 14
  • The batched S-device setup first runs the planner to produce the clean anchor KV bank, then assigns one renderer stage and cache bank to each device; after warm-up, the wavefront generates one chunk per round.The algorithm defines ranks as GPU workers and uses a synchronized renderer round in which each active rank executes at most one chunk-sized forward.Source: A Generation Efficiency Analysis · p. 14, §A.1 · p. 15
  • With planner and renderer block sizes both set to three and anchor stride 10, one anchor block takes five planner rounds, while each local three-anchor window guides six to seven renderer chunks.The stride matches anchor production to consumption with one planner device and four renderer devices; a smaller stride can bottleneck planning, while a larger stride adds no latency gain once planning stays ahead and leaves longer regions less constrained.Source: §A.1 · p. 15

A.2 Latency and Peak Memory Analysis

The latency and memory analysis times denoising-path execution under a defined protocol, compares planner allocations, and plots the break-even boundary between FlashForward and HiAR. The figure notes that short 21-latent cases can be poorly amortized.

A.2 Latency and Peak Memory Analysis

  • Latency and FPS measurements span the first planner or baseline denoising forward through completed latents, include schedule-specific computation and communication, and exclude text encoding and VAE decoding.Each cell averages 20 videos after cold-start exclusion and reports the slowest-rank elapsed time and highest peak allocation across ranks.Source: §A.2 · p. 15
  • The four-GPU default runs the planner on one device, while FlashForward-CP shards planner forwards across four devices; the vanilla allocation is generally faster at 480p and context parallelism at 720p.Planner-to-renderer weight-transfer time is included in four-device FlashForward latency.Source: §A.2 · p. 15
  • Figure 6 plots the forward-count break-even boundary between FlashForward and HiAR across 16 four-GPU context-parallel cells; its two non-winning outliers are both at 21 latents.Marker shape denotes system setting, color indicates the fastest of FlashForward, HiAR, and Self-Forcing, and mismatches between marker and region indicate cases where the pairwise boundary does not recover the three-way winner.Source: Figure 6 · p. 17

A.2.1 Analysis

FlashForward is fastest among the four-GPU schedules from 20 seconds onward, while its reported peak allocation stays close to HiAR’s. On one GPU, it remains faster than HiAR but is slightly slower than Self-Forcing.

Generation latency

  • At 65 seconds, FlashForward reduces latency by 1.16–1.68× relative to HiAR and is fastest among four-GPU schedules from 20 seconds onward.Across the model–resolution settings, its lead also holds at 35 and 65 seconds.Source: §A.2.1 · p. 17
  • On one GPU, FlashForward is 1.54–1.59× faster than HiAR for 20–65-second outputs, but 1.9–4.2% slower than Self-Forcing.Removing most HiAR cache-update-only forwards helps, while writing renderer-history KV at all four stages carries bookkeeping cost.Source: §A.2.1 · p. 17

Peak memory

  • FlashForward’s reported peak memory allocation remains within 0.3 GiB of HiAR’s in every cell.The autoregressive methods’ peak memory is nearly independent of video length because they use a fixed sliding window.Source: §A.2.1 · p. 17

A.2.2 Discussion: When Is the Pipeline Faster?

The analysis compares autoregressive schedules using logical forward counts and break-even conditions, then checks those predictions against measured latency. FlashForward’s advantage is strongest for longer, larger-model, and higher-resolution workloads, where fixed pipeline costs are better amortized.

Forward-count model

  • The schedules use 123 forwards for FlashForward, 134 for HiAR, and 212 for Self-Forcing at 81 latents.This case has 3 planner blocks, 27 renderer chunks, 27 baseline chunks, and 4 denoising stages; the four-device schedules pipeline stages for FlashForward and HiAR, while Self-Forcing remains serial.Source: §A.2.2 · pp. 17–18

When does the forward reduction become a wall-clock speedup?

  • Against HiAR, the required γplan falls from 0.83 at 21 latents to 0.52 at 261 latents; against Self-Forcing, γSF can rise from 2.27 to 3.21 before the ordering reverses.With γplan = 1, the analysis says FlashForward can beat Self-Forcing across a large interval.Source: §A.2.2 · p. 18
  • From 81 latents onward, FlashForward is fastest in every model–resolution setting, and the pairwise FlashForward–HiAR prediction is correct in 15 of 16 cells.Across all three schedules, it is fastest in 14 cells; the exceptions are the two 21-latent settings.Source: §A.2.2 · p. 18
  • As videos lengthen and forwards become more compute-intensive, fixed pipeline costs are better amortized and the break-even requirements become less restrictive.For small forwards with γplan < 1, the analysis recommends using one rank for the planner rather than context parallelism.Source: §A.2.2 · p. 18

From forward counts to wall-clock speedup

  • Forward-count ratios are upper-bound estimates, since context parallelism has collective overhead and the wavefront incurs fill, drain, and synchronization costs.Inter-stage latent transfer overlaps the next forward and is not the dominant communication term.Source: §A.2.2 · p. 18
  • At 81 latents, the forward counts imply a 1.72× advantage over HiAR, while measured speedup is 1.21–1.68× across four settings.At 14B and 720p, the measured 1.68× speedup approaches the raw-count estimate; fixed pipeline costs are more visible for short, low-compute workloads.Source: §A.2.2 · pp. 18–19

A.2.3 Streaming Generation with a Dedicated Planner

A dedicated planner can overlap later anchor generation with renderer work, reducing the planner’s exposed contribution to startup and enabling streaming. The experiments also examine why the renderer regenerates anchor positions rather than copying planner outputs into the final video.

A.2.3 Streaming Generation with a Dedicated Planner

  • The dedicated-planner schedule lets rendering begin after the first anchor block, reducing the anchor contribution to TTFF from a full planner rollout to one block.The four-device latency setup instead completes the sparse-anchor rollout before assigning those devices to the renderer wavefront.Source: §A.2.3 · p. 19
  • From 20 to 65 seconds, a dedicated H100 planner makes 1.3B generation 1.18–1.47× faster, while the 14B configuration is 1.15–1.36× faster than the four-GPU single-planner schedule.At 21 latents, there is only one anchor block and no later planner work to overlap; as sequence length increases, overlap benefit dominates.Source: §A.2.3 · p. 19
  • At nine planner blocks, renderer waiting remains about the cost of the first block, while the exposed fraction of planning falls to 9% at 480p and 6% at 720p.After startup, later anchor generation is hidden beneath the renderer wavefront.Source: §A.2.3 · p. 19

B Why the Renderer Regenerates the Anchor Positions

  • The splice comparison tests two mixed-source designs: keeping planner anchors in the final video with planner KV, or with renderer-side KV.FlashForward instead uses planner anchors only as conditioning and has the renderer generate every output position.Source: B Why the Renderer Regenerates the… · p. 19

Experiments.

Experiments compare anchor splicing with all-position rendering and analyze the resulting seams, output-level differences, and real-anchor probes. The reported findings point to a cross-role output mismatch that is not explained solely by planner generation error.

Experiments

  • For both splice variants, anchor-boundary color changes extend across static backgrounds as well as moving subjects, while FlashForward’s blurred differences stay dark except around large moving regions.The figure shows blurred, amplified per-channel RGB differences between anchor frames and their nearest non-anchor neighbors.Source: Experiments · p. 20, Figure 7 · p. 20
  • Four-step distillation widens the planner-anchor and neighboring-renderer luminance gap by more than an order of magnitude, even with a frozen backbone.Updating the shared backbone widens the gap several times further, with renderer luminance rising more than anchor luminance.Source: Experiments · p. 20
  • In the deployed four-step comparison, all-position rendering cuts the anchor seam by more than half relative to splicing and lowers the periodic total.The table reports normalized adjacent-frame luminance-jump statistics, with lower values indicating smaller seams.Source: Experiments · p. 20, Table 6 · p. 21
  • With real-video-encoded anchors, clips still show greater error at anchor boundaries than at mid-chunk controls; 47–49% of frame-level offset is chromatic.The results are consistent with an output-interface mismatch rather than an artifact explained solely by a bad planner sample or a single brightness offset.Source: Experiments · p. 21
  • Because the renderer generates every output latent, the final sequence avoids switching between planner and renderer outputs while retaining planner anchors as structural conditioning.The paper describes this as removing cross-role output switching as one possible source of periodic seams.Source: Experiments · p. 21

C Training and Implementation Details

Training first establishes the planner–renderer graph with real-video supervision, then distills a four-stage guidance-free student through self-rollouts. The setup combines sparse clean anchors with same-stage renderer history.

C.1 Phase 1: Time-Rebased SFT and Optimization

  • Packed SFT trains a planner over 9 anchors and a renderer over 81 positions in 27 chunks, with each renderer chunk reading up to five preceding chunks and selected clean anchors.The 81-position clip is packed with planner and renderer copies under an attention mask that isolates the sequences; clean-anchor reads are the only cross-role connection.Source: §C.1 · p. 22
  • 20-second real-video clips provide nine anchors across three blocks, explicitly supervising the planner’s autoregressive generation beyond the typical 5-second teacher horizon.The paper uses this duration to balance longer-horizon supervision against training efficiency.Source: §C.1 · p. 22
  • Time-rebased supervision samples four latent-time offsets per step, producing eight planner targets in 3 + 3 + 2 blocks from each crop.Only the first two blocks receive clean context copies because the final block has no later consumer.Source: §C.1 · p. 22
  • Phase 1 minimizes rectified-flow velocity error across planner, renderer, and time-rebased planner latents, using AdamW for 1,800 steps with global batch size 128.The backbone is initialized from Wan2.1-T2V-1.3B, with rank-256 role-specific LoRA adapters and role embeddings.Source: §C.1 · p. 23

C.2 Phase 2: Self-Rollout Guidance and Step Distillation

  • Phase 2 retains the planner–renderer layout and distills a four-stage guidance-free student from self-rollouts, sampling an exit stage and retaining gradients only there.Rollout generates planner blocks before renderer positions; a clean-endpoint cache-extraction forward builds anchor memory, whose renderer read remains connected at the student exit.Source: §C.2 · p. 23
  • At each renderer stage, packed training exposes each chunk only to itself, selected clean anchors, and at most five preceding chunks from that same stage.The packed forward preserves the same-stage dependencies used by sequential inference and permits gradients through KV edges.Source: §C.2 · p. 23
  • Teacher and critic scoring tiles the query into four targets because each score network accepts at most 21 latent-frame positions, while preserving the full-length student rollout.Later tiles receive a detached image head re-encoded from the last RGB frame of the decoded clean prefix; those head predictions are discarded after scoring.Source: §C.2 · p. 23
  • Phase 2 runs 600 AdamW steps at a 1:5 generator-to-critic ratio, with 100 generator updates and learning rates of 10^-5 and 10^-6, respectively.The real-score model is Wan2.1-T2V-14B; the critic is Wan2.1-T2V-1.3B with trainable rank-256 LoRA.Source: §C.2 · p. 23
Loading 2609.32540v1…