PaperlayerView exact revision on arXiv

Source-linked AI summary

Relax Forcing: Relaxed KV-Memory for Consistent Long Video Generation

Zengqun Zhao, Yanzuo Lu, Ziquan Liu, Jifei Song, Jiankang Deng, Ioannis Patras

arXiv:2603.21366v2cs.CV

TL;DR

Minute-scale autoregressive video generation remains challenged by accumulated temporal errors and by uncertainty over how historical KV memory should be organized. The paper introduces training-free Relax Forcing, which uses role-aware sparse memory and relaxed history selection, and reports improved motion dynamics, overall quality, and inference efficiency on VBench-Long.

  • Problem

    Minute-scale autoregressive generation remains difficult because accumulated errors cause drift, visual degradation, or constrained motion, while memory quantity and temporal placement have different effects.

  • Method

    Relax Forcing decomposes KV memory into Sink, Tail, and dynamically selected History components, selecting history aligned with Sink anchors while suppressing redundancy with Tail.

  • Results

    Relax Forcing improves long-horizon motion dynamics and overall generation quality on VBench-Long while reducing effective attention length and inference cost.

  • Takeaways & Limitations

    Structured temporal memory is an effective complementary direction for scalable long-video generation.

Abstract

from arXiv · show

Autoregressive video diffusion has recently emerged as a promising paradigm for long-video generation, enabling causal synthesis beyond the temporal limits of bidirectional models. Existing forcing-based training strategies reduce exposure bias by conditioning models on their own predictions during rollout, yet minute-scale generation remains challenging due to progressive temporal degradation and constrained motion evolution. In this work, we study the role of temporal KV memory during long-horizon autoregressive inference. Our analysis shows that simply retaining more historical frames does not consistently improve generation quality; instead, both the quantity and temporal placement of memory strongly affect motion dynamics. These findings suggest that temporal memory should be treated as structured context rather than a homogeneous chronological buffer. Motivated by this observation, we introduce Relax Forcing, a training-free memory mechanism for autoregressive video diffusion. Relax Forcing decomposes temporal context into three functional components: Sink frames that provide global stability, Tail frames that preserve short-term continuity, and dynamically selected History frames that supply mid-range motion structure. History frames are selected using a relaxation-based criterion that encourages alignment with global anchors while suppressing redundancy with recent context. This role-aware sparse memory design mitigates error accumulation during long-horizon rollout while preserving motion evolution and reducing attention overhead. Experiments on VBench-Long show that Relax Forcing improves long-video generation, achieving stronger motion dynamics and higher overall scores than existing autoregressive baselines. These results indicate that structured temporal memory is an effective and complementary direction for scalable long-video generation.

1 Introduction

Minute-scale autoregressive video generation remains difficult because accumulated errors cause drift, degradation, or constrained motion. The paper argues that temporal memory should be organized by functional roles and introduces Relax Forcing to select sparse, complementary context.

  • Motivation: Autoregressive rollout reduces training–inference mismatch but remains vulnerable to accumulated errors during minute-scale generation.These errors can produce gradual drift, visual degradation, and overly constrained motion dynamics.
  • Motivation: Increasing memory size alone does not consistently improve generation, while temporal placement strongly affects motion dynamics.Under a fixed budget, different history positions change motion dynamics while leaving image quality largely unchanged.
  • Method: Relax Forcing decomposes temporal context into Sink anchors, Tail continuity, and dynamically selected History for mid-range motion structure.History selection favors alignment with Sink and penalizes redundancy with Tail.
  • Method: The training-free relaxed memory mechanism retains complementary historical cues, reducing over-reliance on recent context and effective attention length.The stated design targets error accumulation during long-horizon rollout while preserving motion evolution.
  • Results: On VBench-Long, Relax Forcing yields a 66.8% relative improvement in Dynamic Degree and a 5.7% relative gain in overall score over Self Forcing.The approach also reduces effective attention length and improves inference efficiency.

2 Related Work

Autoregressive video diffusion enables causal sequential generation, but long-video quality depends on how stored temporal context is organized. Existing approaches include teacher-forcing-style training, diffusion-oriented training, queue-based conditioning, and asynchronous or masked autoregressive strategies.

  • Autoregressive Video Diffusion: Autoregressive video diffusion generates frames sequentially, aligning synthesis with the causal structure of temporal data.This direction addresses settings where future frames are unavailable.
  • Autoregressive Video Diffusion: Early methods use teacher-forcing-style next-frame prediction or noise-independent training schemes such as diffusion forcing and FAR.
  • Autoregressive Video Diffusion: Other methods extend temporal rollout through queue-based conditioning, FIFO-style history replacement, asynchronous generation, and masked autoregressive inference.

Methods

The method treats temporal KV memory as structured context rather than a homogeneous chronological buffer, combining fixed, recent, and dynamically selected frames for long-horizon autoregressive video generation.

  • Autoregressive Video Diffusion: Autoregressive video diffusion conditions each generated frame on historical context, but fixed-window rollout can compound residual errors and temporal biases across transitions.The setup uses sequential generation with KV caching and previously generated frames as context.
  • Temporal Memory Analysis: Different memory configurations expose distinct trade-offs: Sink preserves appearance, History enables motion variation, and Tail supports short-term continuity but can destabilize or collapse rollout when excessive.The configurations separately affect identity consistency, motion evolution, and long-term stability.
  • Temporal Memory Analysis: Increasing memory quantity alone does not consistently improve long-horizon generation because redundant temporal cues can weaken motion dynamics.Effectiveness depends on both the amount and temporal placement of retained context.
  • Temporal Memory Analysis: Removing Sink causes drift, removing History limits motion evolution, and removing Tail weakens continuity, whereas their balanced combination preserves stability and dynamic evolution.These ablations support treating temporal memory as heterogeneous rather than homogeneous.
  • Relaxed KV Memory: Relaxed KV Memory constructs Mi = [S;Hi;Ti], using fixed Sink anchors, dynamically selected History frames, and updated Tail frames instead of dense chronological memory.Sink provides long-term stability, Tail provides short-term continuity, and History supplies mid-range structure.
  • Relaxed KV Memory: History candidates are restricted to the later half of the mid-range region, scored by stability and redundancy, and selected with Top-K ranking.The relaxation score is r(h) = S(h)−λR(h), where λ controls the trade-off between global stability and local redundancy.
  • Relaxed KV Memory: The method uses hybrid positional indexing for non-contiguous memory, preserving absolute temporal positions for Tail while treating Sink and History relative to the Tail segment.This adjustment addresses the incompatibility between structured sparse memory and standard sliding-window RoPE resetting.

4 Experiments

Across VBench-Long and ablations, Relax Forcing improves long-horizon video quality and motion while reducing inference cost. Its gains come from structured sparse memory, which generalizes across autoregressive backbones and remains effective in challenging scenarios and human evaluation.

  • Comparisons to State of the Art: 80.87% overall score at 30 seconds and 80.88% at 60 seconds make Relax Forcing the top method in both VBench-Long regimes.At 30 seconds, it surpasses Deep Forcing by 0.93 percentage points.
  • Comparisons to State of the Art: Relax Forcing reaches 65.67% Dynamic Degree at 30 seconds and 66.49% at 60 seconds while maintaining competitive consistency and imaging quality.The results indicate stronger motion evolution without substantial degradation in visual fidelity.
  • Comparisons to State of the Art: Applying Relaxed KV Memory to CausVid without retraining raises its 30-second overall score from 77.69% to 79.85% and its 60-second score from 77.39% to 79.23%.Dynamic Degree also increases from 47.21% to 59.88% at 30 seconds and from 46.44% to 57.78% at 60 seconds.
  • Ablation Analysis: Two Sink frames, one selected History frame, and one Tail frame provide the best balance, whereas additional Sink, History, or Tail frames can constrain or redundantly cue motion.Increasing Sink frames from 0 to 2 raises overall score from 78.36% to 80.86%, while increasing Tail frames from 1 to 3 lowers Dynamic Degree from 65.62% to 55.86%.
  • Ablation Analysis: Relaxed KV Memory lowers FlashAttention latency from 444.2 ms to 168.1 ms per block, a 2.64× speedup, with History selection adding only 3.4 ms.Overall, self-attention time decreases 1.58×, diffusion generation accelerates 1.37×, and end-to-end speed improves 1.26×.
  • Human Evaluation: Human preference rates favor Relax Forcing over Self Forcing at 83.2%, Attention Sink at 74.3%, and Rolling Forcing at 52.5%.The narrower margin against Rolling Forcing indicates that training-based rollout improvements remain competitive in human perception.

5 Conclusion

Relax Forcing treats temporal memory as structured context rather than a uniformly expanding buffer. Its role-aware sparse design combines fixed and dynamically selected frames to improve long-horizon generation while reducing effective attention length and inference cost.

  • Method: Relaxed KV Memory decomposes historical context into fixed Sink, step-dependent Tail, and dynamically selected History components.The selection procedure ranks candidate History frames by alignment with Sink and redundancy with Tail.
  • Conclusion: Relax Forcing improves long-horizon motion dynamics and overall generation quality while reducing effective attention length and inference cost.The ablations additionally measure Drift, Repetition, and their combined Balance metric.
  • Method: The method uses absolute indices for Tail and relative indices for Sink and History to preserve temporal grounding and role ordering.This hybrid indexing supports global anchors and mid-range structure without collapsing them into short-term dynamics.
  • Experimental settings: The default Relaxed KV configuration uses 2 Sink frames, 1 Tail frame, 1 History frame, 4 candidate frames, and λ = 2.0.All methods are evaluated under the same inference settings for fair comparison.
  • Evaluation: VBench-Long evaluation covers 30-second and 60-second generation with subject, background, aesthetic, motion, and dynamic metrics.The evaluation uses 128 refined MovieGen prompts and five generated videos per prompt for robustness evaluation.

C Effect of History Frame Position

History-frame position affects motion-related performance more strongly than visual-quality metrics. Middle-to-later candidate positions provide stronger dynamic behavior, supporting selective mid-range memory over fixed chronological retention.

  • Effect of History Frame Position: Visual-quality metrics remain relatively stable across History positions, while motion-related metrics vary noticeably.The analysis fixes Sink=2 and Tail=1 when comparing History positions.
  • Effect of History Frame Position: Middle-to-later candidate positions tend to achieve stronger Dynamic Degree and better overall scores than earlier positions.This supports selecting informative mid-range History frames rather than relying on fixed chronological memory.
  • Effect of History Frame Position: The best reported memory configuration uses Sink=2, History=1, and Tail=1, achieving the highest overall score while maintaining strong motion dynamics.The results indicate that temporal memory should be balanced rather than simply expanded.
  • Candidate Pool Size: Performance remains highly stable for candidate pools of 2–4 frames, with overall score varying by only 0.02 points and Dynamic Degree staying above 65.3.This indicates that Relax Forcing does not require a precisely tuned candidate-pool size.
  • Candidate Pool Size: Increasing the candidate pool from 4 to 7 reduces Dynamic Degree from 65.62 to 63.44 as larger pools contain more similar temporal content.The redundant candidates provide less complementary motion information and may increase correlated historical cues.

D.2 Robustness to Redundancy Weight λ

The redundancy weight is not highly sensitive within a moderate range, but excessive penalties reduce motion dynamics. The selected default λ = 2.0 balances global anchoring, local continuity, and motion evolution.

  • Redundancy Weight λ: The redundancy weight λ controls how strongly History frames resembling recent Tail context are penalized.The relaxation score uses this penalty to regulate redundancy with recent context.
  • Redundancy Weight λ: Relax Forcing remains stable across a moderate range of λ values, with only minor variations in visual consistency, imaging quality, and overall score.The selection criterion therefore does not require precise tuning of the redundancy penalty.
  • Redundancy Weight λ: Excessively large λ decreases Dynamic Degree by rejecting History frames that share useful short-term information with Tail.A moderate penalty reduces over-reliance on recent context while retaining temporal cues needed for coherent motion progression.
  • Redundancy Weight λ: The experiments use λ = 2.0 as the default because it provides the best observed balance between global anchoring, local continuity, and motion evolution.The setting is applied across the reported experiments.

E Challenging-Prompt Robustness

Relax Forcing remains robust across abrupt scene changes, multi-subject generation, and high-speed or non-uniform motion. It achieves the highest overall score across all six challenging settings, with the largest gains again appearing in Dynamic Degree.

  • Challenging-Prompt Robustness: The robustness evaluation covers abrupt scene cuts and multi-shot editing, multi-subject generation, and high-speed or non-uniform motion.Each subset contains 30 MovieGen prompts evaluated at both 30 seconds and 60 seconds.
  • Challenging-Prompt Robustness: Relax Forcing achieves the highest overall score across all six challenging settings.The settings combine three prompt categories with 30-second and 60-second generation.
  • Challenging-Prompt Robustness: For abrupt scene changes, Average scores are 84.40 at 30 seconds and 83.96 at 60 seconds.The corresponding Dynamic Degree values are 88.73 and 86.47.
  • Challenging-Prompt Robustness: For multi-subject generation, Average scores are 83.08 at 30 seconds and 83.20 at 60 seconds.The corresponding Dynamic Degree values are 77.33 and 77.87.
  • Challenging-Prompt Robustness: For high-speed or non-uniform motion, Average scores are 83.13 at 30 seconds and 82.91 at 60 seconds.The corresponding Dynamic Degree values are 77.33 and 77.87 in the reported robustness results.

F Extended Efficiency Analysis

Relax Forcing combines structured memory with qualitative and perceptual evaluations to assess long-video quality and efficiency. Its sparse memory retains computational benefits over longer rollouts while balancing temporal consistency with motion evolution.

  • Efficiency Analysis: 2.45×, 2.53×, and 2.52× FlashAttention speedups are maintained at 30, 60, and 120 seconds, respectively.FlashAttention accounts for 27.9–29.8% of diffusion time with Relaxed KV Memory, versus approximately 53.5–54.6% for the dense 21-frame baseline.
  • User Preference Study: The user study evaluates visual quality, motion quality, and text–video alignment using randomized pairwise comparisons and percentage of pairwise wins.These criteria cover sharpness and realism, temporal smoothness and motion evolution, and consistency with the text prompt.
  • Qualitative Comparisons: Qualitative comparisons identify temporal drift, over-constrained motion, and balanced evolution as recurring differences among methods.Relax Forcing preserves global consistency while allowing sustained motion evolution, whereas insufficient anchoring can produce semantic drift and heavy recent or dense memory can produce repetitive motion.
Loading 2603.21366v2…