PaperlayerView exact revision on arXiv

Source-linked AI summary

LoViC: Efficient Long Video Generation with Context Compression

Jiaxiu Jiang, Wenbo Li, Jingjing Ren, Yuping Qiu, Yong Guo, Xiaogang Xu, Han Wu, Wangmeng Zuo

arXiv:2507.12952v1cs.CV

TL;DR

Long-form video generation remains difficult because DiTs face quadratic attention costs over dense video tokens and long contexts are needed for consistency. LoViC compresses preceding video-text segments with FlexFormer and generates videos segment by segment. It supports prediction, interpolation, retrodiction, and multi-shot generation, with reported advantages on continuity metrics across tasks.

  • Problem

    Long-form video generation is constrained by quadratic self-attention over dense spatiotemporal tokens, while high-complexity settings require long context to maintain consistency.

    Source: Introduction · p. 2, Diffusion Transformer · p. 4

  • Method

    LoViC conditions segment-wise DiT generation on preceding video-text segments compressed by FlexFormer, a flexible autoencoder with adaptive compression and position-aware encoding.

    Source: Introduction · p. 2

  • Results

    Across prediction, interpolation, and retrodiction, LoViC supports all three tasks and outperforms baselines on continuity metrics, while its non-reference metrics are comparable to 13B FramePack despite using around 2B parameters.

    Source: Main Results · p. 7

  • Takeaways & Limitations

    LoViC supports video continuation in any temporal direction and multi-shot generation, and has potential to generate significantly longer videos than vanilla DiT.

    Source: Conclusion · p. 8

Abstract

from arXiv · show

Despite recent advances in diffusion transformers (DiTs) for text-to-video generation, scaling to long-duration content remains challenging due to the quadratic complexity of self-attention. While prior efforts -- such as sparse attention and temporally autoregressive models -- offer partial relief, they often compromise temporal coherence or scalability. We introduce LoViC, a DiT-based framework trained on million-scale open-domain videos, designed to produce long, coherent videos through a segment-wise generation process. At the core of our approach is FlexFormer, an expressive autoencoder that jointly compresses video and text into unified latent representations. It supports variable-length inputs with linearly adjustable compression rates, enabled by a single query token design based on the Q-Former architecture. Additionally, by encoding temporal context through position-aware mechanisms, our model seamlessly supports prediction, retradiction, interpolation, and multi-shot generation within a unified paradigm. Extensive experiments across diverse tasks validate the effectiveness and versatility of our approach.

Introduction

Long-form video generation needs extended temporal context, but long-context DiT attention is computationally costly and existing approaches have not shown strong open-domain, high-variability performance. LoViC compresses video-text context and uses a unified segment-wise framework for long-form generation across multiple tasks.

Introduction

  • Open-domain, high-variability video generation demands longer context for semantic and temporal consistency, while extending DiTs naively is computationally prohibitive and ad-hoc strategies can compromise coherence.The required context length also varies with scene dynamics.Source: Introduction · p. 2
  • LoViC conditions each generation step on preceding segments and compresses arbitrary-length video-text context with FlexFormer before a DiT generates the next chunk.The framework was trained on a million-scale open-domain video corpus.Source: Introduction · p. 2
  • FlexFormer supports arbitrary input shapes with adaptive compression rates, using a single-query token mechanism and I-RoPE to preserve spatial relationships among video tokens.I-RoPE positional encoding is part of the proposed context encoder.Source: Introduction · p. 2
  • Relative positional encoding distinguishes contextual segments from the current segment, supporting single-shot extension and multi-shot generation with visually coherent results and realistic dynamics.The framework also treats prediction, retrodiction, and interpolation within one generation paradigm.Source: Introduction · p. 2

Related Work

Prior long-video work addresses DiT attention costs through segment-wise generation, auxiliary context modules, or more efficient attention architectures. Multi-shot methods also use dense captions, longer-video training, efficient attention, keyframe pipelines, and training-free techniques.

Related Work

  • Long-video methods split generation into portions, often autoregressively, or encode historical context with auxiliary modules; some also compress historical frames or append adjacent conditioning frames.These approaches aim to mitigate vanilla DiT's quadratic complexity.Source: Related Work · p. 3
  • Other long-video approaches replace self-attention with hybrid local and linear attention, overlapped-window attention, or sparse attention.The passage presents these as a second category of methods for addressing quadratic complexity.Source: Related Work · p. 3
  • Multi-shot generation methods use temporally dense captions, train vanilla DiTs on longer videos, adopt efficient attention, or generate keyframes before image-to-video synthesis.The passage describes these as distinct approaches to multi-shot video generation.Source: Related Work · p. 3
  • Training-free methods have also been proposed for generating long and multi-shot videos.Source: Related Work · p. 3

Method

LoViC compresses variable video-text context with FlexFormer and supplies the resulting context features to a DiT. Position-aware encoding supports different generation tasks and compression strategies.

Diffusion Transformer.

  • Quadratic self-attention over dense spatiotemporal tokens limits DiT scalability for long-form video, and M-RoPE performs suboptimally under some conditions.M-RoPE unifies text and image positions as special cases of 3D positional encoding.Source: Diffusion Transformer · p. 4

Context Compression with FlexFormer

  • FlexFormer adapts compression to arbitrarily sized and ordered video-text contexts by replicating a single learnable query token according to input length and compression strategy.It replaces Q-Former cross-attention with self-attention-based encoding.Source: Context Compression with FlexFormer · p. 4
  • I-RoPE assigns query tokens spatiotemporal positions by interpolating video-token indices, addressing blurry reconstructions observed with M-RoPE in this setting.The decoder uses a symmetric positional encoding scheme.Source: Context Compression with FlexFormer · p. 5
  • Time-scaled query density allocates lower compression ratios to frames near the current segment, improving temporal continuity while retaining computational efficiency.Another strategy distributes query tokens uniformly among video tokens.Source: Context Compression with FlexFormer · p. 5

Multi-Shot DiT

  • FlexFormer context features enter each DiT block through self-attention alongside input tokens, supporting prediction, interpolation, retrodiction, and multi-shot generation without added cross-attention layers.RoPE indices distinguish task positions, and multi-shot generation inserts a temporal gap between shots.Source: Multi-Shot DiT · p. 5

Experiments

LoViC supports prediction, interpolation, retrodiction, and multi-shot generation, with reported gains in video continuity and coherence. Ablations favor linear compression for DiT performance and identify benefits from FlexFormer’s single-query and I-RoPE design.

Experimental Setup

  • Experiments use 1.6 million downloaded Panda-70M clips and evaluate 100 test videos per prediction, interpolation, and retrodiction task, plus 100 multi-shot prompt sets.The clips are drawn from a 2-million-clip subset; the multi-shot evaluation reports VBench consistency, aesthetic-quality, and video-text-alignment metrics.Source: Experimental Setup · pp. 5–6

Main Results

  • Qualitatively, LoViC produces temporally stable prediction videos, clearer and smoother interpolation, and natural, coherent retrodiction.The comparisons note identity drift, background inconsistency, flickering, or brightness changes in some baselines.Source: Main Results · pp. 6–7
  • LoViC supports prediction, interpolation, and retrodiction, outperforming all baselines on continuity metrics and matching 13B FramePack on non-reference metrics with around 2B parameters.The reported continuity metrics are PSNR, LPIPS, Subject Consistency, and Background Consistency; non-reference metrics are Aesthetic Quality and Video-text Alignment.Source: Main Results · p. 7, Table 1 · p. 7
  • LoViC achieves the highest Subject Consistency and Background Consistency among multi-shot baselines, with slightly lower Aesthetic Quality and Video-text Alignment than base LTX-Video.The authors suggest lower training-data quality and prioritizing coherence with preceding segments over strict prompt adherence as possible factors.Source: Main Results · p. 7, Table 2 · p. 7

Ablation Study

  • FlexFormer’s single learnable query enables adaptive query length for videos of different lengths, while I-RoPE adds spatial correspondence between video and compressed tokens.Adding an MLP before the query tokens slightly reduces performance; the authors hypothesize I-RoPE alone is sufficient for spatial information.Source: Ablation Study · pp. 7–8, Table 3 · p. 7
  • Linear compression offers the most balanced DiT performance-efficiency trade-off, improving continuity despite lower autoencoder reconstruction performance than uniform compression.It assigns lower compression ratios to frames adjacent to the current frames; Table 4 also indicates that a lower overall compression ratio does not always improve generation.Source: Figure 8 · p. 7, Ablation Study · p. 8, Table 4 · p. 8

Conclusion

LoViC is an efficient, flexible architecture for long video generation that supports continuation in any temporal direction and multi-shot generation. The authors identify adaptive extraction and integration of relevant information from prior segments as future work.

Conclusion

  • LoViC supports video continuation in any temporal direction and multi-shot video generation, with greater scalability than vanilla DiT and potential for significantly longer videos.Future work will explore adaptive extraction and integration of relevant information from previously generated segments for more coherent, context-aware generation over extended spans.Source: Conclusion · p. 8

Related papers

Loading 2507.12952v1…