PaperlayerView exact revision on arXiv

Source-linked AI summary

RECAP-Forcing: Retaining Content Appearances for Long Video Generation

Haiyang Xu, Zheng Ding, Zhuowen Tu

arXiv:2608.26671v1cs.CV

TL;DR

Long autoregressive video generation must decide what to retain within a finite attention window, while existing memory methods prioritize recency. RECAP-Forcing instead retains newly appearing content through appearance-indexed memory, improving long-horizon dynamics while maintaining visual quality and consistency.

  • Problem

    Long-video generators must decide what to retain within a finite attention window, but existing memory is organized by temporal recency rather than appearance.

  • Method

    RECAP-Forcing combines an attention sink for the opening scene with an optical-flow novelty bank that retains newly appearing content during generation.

  • Results

    RECAP-Forcing improves long-horizon dynamics while maintaining visual quality and consistency, outperforming existing training-free memory methods.

  • Takeaways & Limitations

    Appearance novelty provides a useful principle for organizing memory in long-form video generation.

  • Takeaways & Limitations

    Optical-flow correspondence may repeatedly mark stochastic textures as novel, consuming memory without adding meaningful information.

Abstract

from arXiv · show

Long autoregressive video generation faces a fundamental memory challenge: with a finite attention window, a model must decide which information from an ever-expanding history to retain. Existing methods organize memory temporally, preserving recent frames while compressing or discarding older ones. We instead propose RECAP-Forcing, organizing memory by appearance novelty. A long video is not merely a sequence of frames, but an evolving cast of subjects, objects, and scenes whose identities must remain consistent over time. We organize memory by retaining the KV cache associated with newly appearing content--such as entering subjects, disoccluded regions, and newly introduced scenes--at the moment it first becomes visible, prioritizing novelty over recency. Memory should scale with the amount of newly introduced content, rather than with video length. This appearance-indexed memory makes long-range consistency an explicit property of the memory structure. Our framework unifies two mechanisms under this single principle. At the beginning of a video, when all visible content is novel, an attention sink preserves the initial scene. As the video evolves, an optical-flow-based novelty bank extends the same principle by selectively retaining newly revealed content. As a training-free inference method with no additional learnable parameters, RECAP-Forcing consistently improves visual quality and semantic fidelity across multiple strong baselines and outperforms existing memory methods.

1. Introduction

RECAP-Forcing reframes long-video memory from temporal recency to appearance novelty, retaining KV caches when subjects, objects, or regions first become visible. It combines a reinforced attention sink for the opening scene with an optical-flow-based novelty bank to preserve established content while admitting new content.

  • Problem: Time-indexed methods retain recent frames or compress older ones, progressively reducing detailed access to original visual evidence as video history grows [23] [65] [80].This creates the central memory tension: preserving the past competes with the finite attention budget required for continued generation.
  • Mechanisms: The reinforced attention sink anchors the opening scene, while the optical-flow-based novelty bank stores KV caches for entrances, disocclusions, and newly revealed regions.The memory unit is a patch at the moment it first appears, rather than an explicitly identified entity.
  • Mechanisms: Together, the sink and novelty bank alleviate the drift–freeze trade-off by preserving established subjects and objects while admitting newly introduced content throughout generation.The sink supports long-range consistency for the opening scene, while the novelty bank preserves content introduced later and maintains dynamics.
  • Appearance-indexed memory: RECAP-Forcing organizes autoregressive video memory by appearance novelty rather than temporal recency, recording content at novel appearance events and retaining sparse appearance anchors.Novel events include entrances, newly revealed regions, and new viewpoints of already-present subjects; memory grows with newly introduced content rather than video length.

2. Related Work

Prior work advances autoregressive video diffusion through longer rollouts, cache supervision, causal modeling, and bounded-window generation, while memory methods compress, prune, or retrieve context. RECAP-Forcing instead retains original mid-video KV states by appearance novelty in a query-agnostic memory bank.

  • Autoregressive Video Diffusion Models: Autoregressive video diffusion conditions generated frames or blocks on past context, with follow-ups extending horizons [13], supervising KV-cache information [92], and perturbing context for robustness [69].Related approaches also pursue causal distillation for quality and efficiency [36] [37] [86] [91], physical plausibility [85], flexible causal–bidirectional generation [43], and rolling or chunkwise bounded-window generation [38] [50] [51].
  • Autoregressive Video Diffusion Models: These autoregressive methods face a drift–stability trade-off: persistent early context reduces drift but can suppress motion.RECAP-Forcing is presented as complementary to these training-based approaches.
  • KV-Cache Memory for Video: Training-free video memory methods replay initial attention sinks [34] [73], compress evicted states [31], gate cache admission by relevance [45], or summarize frames by surprise [53].SlotMemory [39] learns object-centric semantic slots to route original KV tokens using prompt and visual relevance for retrieval and eviction.
  • Novelty-Indexed Memory: Unlike approaches organized by compression, pruning, semantic structure, or query-dependent relevance, RECAP-Forcing directly retains original mid-video KV cache by appearance novelty.Newly appearing content is stored in a query-agnostic, novelty-indexed memory bank.

3. Method

RECAP-Forcing replaces recency-based memory with appearance-indexed memory, retaining content when it first becomes visible. It implements this principle through a reinforced opening-frame attention sink and an optical-flow novelty bank for later newly revealed content.

  • 3.1: A bounded sliding KV cache forgets tokens outside its recent-frame window, so discarded early context allows small inconsistencies to accumulate into identity drift.Existing recency-based caches and compression methods preserve near-past frames or summarize the far past, sharing a timeline-centered memory assumption.
  • 3.2: RECAP-Forcing indexes memory by appearance novelty, recording sparse events such as subject entry and newly revealed regions instead of retaining the intervening timeline.This allocates fixed memory capacity by novelty rather than recency, targeting persistent identity across long videos.
  • 3.2.1: For the entirely novel opening frame, RECAP-Forcing reinforces the attention sink with a pre-softmax ln λ bias, multiplying sink keys’ unnormalized weights by λ while preserving normalized competition.The method uses λ = 5; on backbones without a sink, the plug-in instantiates one at inference.
  • 3.2.2: After the opening frame, RAFT optical flow identifies newly entering or disoccluded patches, whose per-layer key/value vectors are stored in a top-K novelty bank.The bank propagates claimed novelty so continuously traceable content fires once, while broken correspondences can create new appearance events; temporal RoPE is reset to zero for bank keys.

4. Experiments

Across VBench-Long, RECAP-Forcing improves long-video dynamics and overall quality across four generation-model baselines, while ablations show complementary roles for sink reinforcement and the novelty bank. Additional analyses identify conservative default settings, human evaluation protocol, and improved preservation of subjects, scenes, and styles over one-minute rollouts.

  • Main Results on VBench-Long: RECAP-Forcing improves VBench-Long performance across Self-Forcing [25], Infinite-Forcing, LongLive [70], and Helios [78], including Dynamic Degree gains from 27.5 to 58.1 and from 61.3 to 71.3 on the first two baselines.On Infinite-Forcing, Total increases from 78.3 to 79.7; the method also consistently improves performance over LongLive [70] and Helios [78].
  • Ablation of Components: The reinforced sink anchors early information and improves consistency and semantic quality, whereas the memory bank retains evolving content and substantially increases Dynamic Degree from 61.3 to 78.2.The sink can overly constrain later evolution, reducing Dynamic Degree to 41.6.
  • Strength: The default settings use bank capacity f = 1 and sink-reinforcement strength λ = 5 because larger banks offer marginal or negative Total-score changes, while λ = 5 raises Total from 78.8 to 79.7 and stronger values degrade it.At λ = 8, 10, and 12, Total falls to 79.2, 78.9, and 78.6, respectively.
  • Human Study: The human study uses 9 participants, 24 blind pairwise comparisons per participant, and 216 total judgments across Consistency, Motion, and Overall quality against four comparison systems.HARD TO SAY judgments are treated as ties with 0.5 credit to win rates.
  • Qualitative Results: On four one-minute prompts, RECAP-Forcing preserves subject identity, scene, and style throughout the video, while recency memory causes identity loss or duplication and scene-style drift.The method also stabilizes subjects introduced long after the opening frame while maintaining natural rather than frozen motion.

5. Conclusion and Limitations

RECAP-Forcing organizes finite video-generation memory by appearance novelty, combining an attention sink for the opening scene with an optical-flow novelty bank for newly appearing content. It improves long-horizon dynamics, visual quality, and consistency without training, while remaining limited by flow-based over-retention of stochastic textures and motivating richer novelty measures.

  • Conclusion: RECAP-Forcing is a training-free memory mechanism that organizes finite memory by appearance novelty, combining an attention sink with an optical-flow novelty bank.The attention sink preserves the opening scene, while the novelty bank retains newly appearing content in frozen causal backbones.
  • Conclusion: Applied to frozen causal backbones, RECAP-Forcing improves long-horizon dynamics while maintaining visual quality and consistency, outperforming existing training-free memory methods.
  • Future Directions: The appearance-novelty principle motivates richer novelty measures, including longer-range correspondence or higher-level semantic grouping, for more selective and persistent memory allocation.
  • Limitations: Optical-flow correspondence may repeatedly mark stochastic textures such as rain, spray, and rippling water as novel, consuming memory without meaningful new information.

Supplementary Material · A. Implementation Details

The supplementary implementation uses Self-Forcing as the primary baseline for causal autoregressive video diffusion rollouts. It specifies the video resolution, latent-block structure, sliding KV cache, attention window, and rollout length, while Table 6 lists RECAP-Forcing hyperparameters.

  • A. Implementation Details: Self-Forcing [25] is the primary baseline, built on Wan2.1-T2V-1.3B [57] as a DMD-distilled [74] [75] causal autoregressive video diffusion model.
  • A. Implementation Details: The model generates 832×480 videos block by block using three latent frames per block.
  • A. Implementation Details: Rollouts use a sliding KV cache to manage the model’s temporal context.
  • A. Implementation Details: Each latent frame is represented as a 30×52 grid containing 1,560 tokens.
  • A. Implementation Details: The attention window covers six latent frames: one persistent sink frame, two prior frames, and the three-frame current block.
  • A. Implementation Details: A 60-second video is produced through a rollout of 240 latent units.
  • A. Implementation Details: Table 6 reports the hyperparameters used by RECAP-Forcing.

B. Length Generalization … C.2. Identity Retention over Extended Horizons

RECAP-Forcing generalizes to longer videos by maintaining higher dynamics without sacrificing subject consistency, while its appearance memory preserves identity across minute- and five-minute horizons. Extended-horizon evaluations show that identity remains stable under continuous visibility and substantially outlasts recency-based memory.

  • B. Length Generalization: +10.1 Dynamic Degree points at 60 seconds, with RECAP-Forcing outperforming the baseline across all tested lengths while retaining comparable Subject Consistency.The length sweep covers videos from 5 to 60 seconds, with error bars reported over 5 seeds.
  • C. Appearance Memory and Identity Preservation: The appearance-memory study tests identity preservation during continuous visibility, extended rollouts, and controlled reappearance after absences longer than the local attention window.These settings directly assess whether stored appearance information remains useful over long horizons.
  • C.1. Long-Horizon Subject Identity: Long-horizon subject identity is measured on 20 prompts over 60 seconds using DINOv2 CLS similarity between early and final windows, on both full frames and detected subject crops.Each arm uses 5 seeds; subject detection achieves a 97% detection rate, and comparisons are pairwise per prompt and seed.
  • C.2. Identity Retention over Extended Horizons: Identity retention is computed as each frame’s DINOv2 CLS-token cosine similarity to frame 0, with identical prompts and frame times making the curve gap the meaningful comparison.Absolute similarity remains below 1 even for good videos because legitimate scene motion changes the global feature.
  • C.2. Identity Retention over Extended Horizons: 0.81 identity retention at both 1 and 5 minutes, while the recency baseline decays to 0.61 over five minutes.RECAP-Forcing stops decaying after the first minute, whereas the recency baseline continues to drift.
  • C.2. Identity Retention over Extended Horizons: The five-minute retention evaluation uses 20 prompts and 5 seeds, with midpoint and endpoint measurements defined consistently across the five-minute and 60-second settings.Table 8 compares RECAP-Forcing against recency-based memory over the corresponding horizons.

C.3. Controlled Reappearance · D. Memory Bank Analysis

The controlled reappearance benchmark tests whether persistent appearance memory preserves subject identity across a gap exceeding the attention window. The accompanying analysis frames this test as evidence for admitting historical states by appearance novelty rather than recency.

  • C.3. Controlled Reappearance: The experiment targets the demanding case where a subject disappears longer than the local attention window and later reappears with the same identity.This extends beyond testing appearance stability while the subject remains continuously visible.
  • C.3. Controlled Reappearance: Prompting alone cannot reliably create controlled reappearance because current backbones make mentioned entities appear immediately and essentially never leave.This motivates constructing the benchmark directly at inference time rather than through staged temporal prompts.
  • C.3. Controlled Reappearance: The controlled rollout uses four segments: an empty scene for 3 s, the subject for 6 s, an empty scene for 7.5 s, and subject re-conditioning for 7.5 s.The empty interval lasts five times the 1.5 s attention window.
  • C.3. Controlled Reappearance: At each boundary, the sliding window is cleared while the sink and bank persist, making the bank the only channel that can carry the subject’s appearance across the gap.Because the opening scene contains no subject, the subject never enters the sink.
  • C.3. Controlled Reappearance: A Qwen2.5-VL check verifies that the subject appears only in its designated segments, while identity is scored between the first-appearance and return windows using the dual metric from Sec. C.1.The subject must be detected in both subject segments and absent during the gap.
  • C.3. Controlled Reappearance: RECAP-Forcing improves both identity measures over the base model after a 7.5 s disappearance, with several hundred first-appearance states still resident when the subject returns.The base model often re-invents the subject with visibly different appearance, whereas RECAP-Forcing renders the return against stored first-appearance states.
  • D. Memory Bank Analysis: The memory-bank analysis asks which historical states should enter the bank and whether the resulting memory remains time-neutral rather than becoming a recency buffer.It follows the finding that persistent appearance memory improves long-horizon identity.

D.1. Admission Rule: What Should Enter the Bank? … F. Is the Gain Good Motion?

RECAP-Forcing benefits from admitting content by optical-flow appearance novelty rather than arbitrary, recent, or semantically diverse history. Its bank exhibits time-neutral retention while adding roughly constant inference overhead across video lengths.

  • D.1. Admission Rule: What Should Enter the Bank?: Flow Novelty selects candidates by optical-flow novelty, showing that appearance novelty is more effective than arbitrary, recent, or semantically diverse history.The admission rule determines what content enters the bank and therefore which historical information is retained.
  • D.1. Admission Rule: What Should Enter the Bank?: Random History and Recent History provide little benefit, while DINOv2 Diversity improves Total to 79.4 but leaves Dynamic Degree at 60.6.Random History and Recent History improve Dynamic Degree by only +1.1 and +0.7, and Random History lowers Total to 77.7 versus 78.3 without a bank.
  • D.2. Bank Retention Behavior: Across 220 rollouts, the final bank achieves 0.99 overlap with the top-K candidates ranked by frozen scores among all observed items.The small residual discrepancy is attributed to score ties.
  • D.2. Bank Retention Behavior: Eviction removes mostly below-cutoff admissions, while final bank entries have source-frame ages roughly uniform across the rollout with mean age near the video midpoint.This survival pattern indicates time-neutral retention determined by novelty rather than recency.
  • E. Inference Cost: Incremental bank refilling reproduces the naive path bit-identically while keeping per-block cost constant.The naive alternative would scale quadratically with duration and require hundreds of GB of cached history.
  • E. Inference Cost: RECAP-Forcing adds roughly 1.6× generation overhead that does not increase from 1- to 5-minute videos.On a single GH200 GPU, the method avoids quadratic bank refilling by storing only novel-token activations and incrementally refilling slots.

A Camera– Object Decomposition Analysis

The camera–object decomposition shows that RECAP-Forcing raises Dynamic Degree without injecting camera motion: it suppresses global drift while preserving most subject motion. Dynamic Degree can increase despite lower mean flow because near-frozen clips cross the genuine-motion threshold while runaway-drift outliers are reduced.

  • A Camera– Object Decomposition Analysis: Camera motion falls from 1.20→0.90 while object motion remains 0.71→0.66, raising the object-motion share from 37% to 42% on Self-Forcing.This shows the Dynamic Degree gain does not come from injecting additional global camera motion.
  • A Camera– Object Decomposition Analysis: The analysis uses RAFT optical flow, RANSAC homography fitting, and residual decomposition to separate camera and object motion across 128 prompts.Flow is computed on frame pairs sampled approximately every 0.5 seconds.
  • A Camera– Object Decomposition Analysis: Dynamic Degree rises as near-frozen clips cross its genuine-motion threshold while runaway-drift outliers decline, reducing total-flow deviation from ±2.9 to ±1.9 and drift-tail clips from 37 to 24.Dynamic Degree measures the fraction of videos exceeding a threshold, not mean flow magnitude.
Loading 2608.26671v1…