PaperlayerView exact revision on arXiv

Source-linked AI summary

SemanticGen: Video Generation in Semantic Space

Jianhong Bai, Xiaoshi Wu, Xintao Wang, Xiao Fu, Yuanxing Zhang, Qinghe Wang, Xiaoyu Shi, Menghan Xia, Zuozhu Liu, Haoji Hu, Pengfei Wan, Kun Gai

arXiv:2512.20619v3cs.CV

TL;DR

Video generation in VAE latent space can converge slowly and makes long-video diffusion costly to model. SemanticGen generates compact semantic features before conditioned VAE latents, and reports faster convergence and stronger results on short and long videos.

  • Problem

    VAE-latent video generation can converge slowly, while bidirectional attention over the many tokens in long videos is computationally challenging.

    Source: §1 · p. 2, §3.4 · p. 5

  • Method

    SemanticGen generates compact semantic video features for global planning, then generates conditioned VAE latents to produce the video.

    Source: §1 · p. 2, §5 · p. 8

  • Results

    SemanticGen reports faster convergence than VAE-latent generation and outperforms state-of-the-art approaches and strong baselines on short and long videos.

    Source: §1 · p. 2, §5 · p. 8, §4.3 · p. 8

  • Takeaways & Limitations

    The approach is reported to scale efficiently to long-video generation, though semantic features cannot fully preserve fine-grained texture details.

    Source: §5 · p. 8

    The paper validates SemanticGen with Qwen-2.5-VL and leaves systematic exploration of other semantic encoders for future work.

    Source: §3.2 · p. 4

Abstract

from arXiv · show

State-of-the-art video generative models typically learn the distribution of video latents in the VAE space and map them to pixels using a VAE decoder. While this approach can generate high-quality videos, it suffers from slow convergence and is computationally expensive when generating long videos. In this paper, we introduce SemanticGen, a novel solution to address these limitations by generating videos in the semantic space. Our main insight is that, due to the inherent redundancy in videos, the generation process should begin in a compact, high-level semantic space for global planning, followed by the addition of high-frequency details, rather than directly modeling a vast set of low-level video tokens using bi-directional attention. SemanticGen adopts a two-stage generation process. In the first stage, a diffusion model generates compact semantic video features, which define the global layout of the video. In the second stage, another diffusion model generates VAE latents conditioned on these semantic features to produce the final output. We observe that generation in the semantic space leads to faster convergence compared to the VAE latent space. Our method is also effective and computationally efficient when extended to long video generation. Extensive experiments demonstrate that SemanticGen produces high-quality videos and outperforms state-of-the-art approaches and strong baselines.

1. Introduction

The introduction identifies slow convergence and long-video scaling as limitations of VAE-latent generation. SemanticGen instead plans in compact semantic space before refining details in VAE latents.

1. Introduction

  • VAE-latent diffusion faces slow convergence and long-video scaling challenges, including over 0.5M tokens for a 60s, 480p, 24fps clip.Prior attempts to reduce complexity or generate autoregressively can suffer temporal drift or visual-quality degradation.Source: §1 · p. 2
  • SemanticGen first models compact semantic representations for global planning, then adds high-frequency details by generating VAE latents conditioned on them.The framework trains a VAE-latent denoiser conditioned on semantic representations, then a semantic representation generator; it uses semantic-space compression via a lightweight MLP.Source: §1 · p. 2
  • SemanticGen is presented as orthogonal to approaches that add semantic objectives to VAEs and is not tied to a specific VAE tokenizer.The introduction distinguishes those tokenizer-focused approaches from its framework.Source: §1 · p. 2
  • SemanticGen differs from TokensGen, which compresses VAE latents rather than modeling semantic features; the paper reports faster convergence in semantic space.The introduction describes semantic-space generation as fundamentally different from generation in compressed VAE space.Source: §1 · p. 2
  • SemanticGen is reported to outperform state-of-the-art approaches and strong baselines on short and long video generation.The introduction also reports ablation support for its design choices and advantages in convergence speed and long-video generation.Source: §1 · p. 2

2. Related Works

Related work covers diffusion, autoregressive, and hybrid video-generation paradigms, alongside approaches that use semantic representations in generative models.

2.1. Video Generative Models

  • Diffusion video models advanced from adding temporal layers to text-to-image models toward high-fidelity short-video generation, but full-sequence attention limits long-video scalability.Autoregressive methods generate frames or patches sequentially, while diffusion-autoregressive hybrids seek to combine the paradigms.Source: §2.1 · pp. 2–3
  • Diffusion-autoregressive hybrids address issues including training-inference gaps and error accumulation, but these approaches generally perform worse than diffusion-based methods.The paper positions its own diffusion-based paradigm as faster to converge and able to generalize to long-video generation.Source: §2.1 · p. 3

2.2. Semantic Representation for Generation

  • Semantic-generation research includes tokenizer methods that align VAE latents with semantic representations or add semantic objectives during VAE training.The cited approaches report faster convergence and stronger image-generation performance than raw VAE latents.Source: §2.2 · p. 3
  • Other methods optimize latent generators using semantic features, including staged semantic-to-image generation, representation alignment, and separate learning of semantics and high-frequency details.The paper describes its own approach as learning compressed semantic representations and mapping them to VAE latents.Source: §2.2 · p. 3

3. Method

SemanticGen first trains a VAE-latent denoiser conditioned on compressed semantic features, then learns their distribution for semantic-first video generation. For long videos, it uses full attention for compact semantic features and shifted-window attention when mapping them into VAE space.

3.1. Preliminary: Text-to-Video Base Model

  • The base model uses a VAE and Transformer diffusion model with Rectified Flow, training a vector field for a probability path and sampling by Euler discretization.The forward path runs between video latents and standard normal noise.Source: §3.1 · p. 3

3.2. Video Generation with Semantic Embeddings

  • Semantic encoders should capture temporal information, produce compact spatial and temporal representations, and support varied video lengths and resolutions.The paper uses Qwen-2.5-VL to validate SemanticGen and notes that the framework is not tied to a specific tokenizer.Source: §3.2 · p. 4
  • A learnable MLP compresses semantic features, predicts a Gaussian distribution’s mean and variance, and uses KL divergence to regularize the sampled embedding.The authors hypothesize that high feature dimensionality and the original semantic space’s sampling properties contribute to slower fitting.Source: §3.2 · pp. 4–5
  • The VAE-latent generator receives compressed semantic features through in-context conditioning, preserving reference layout and motion while allowing fine details to differ.The model concatenates noised VAE latents with the compressed semantic representation as input.Source: §3.2 · p. 5

3.3. Semantic Representation Generation

  • A second video diffusion model learns the compressed semantic-feature distribution, with the visual encoder and MLP frozen during its fine-tuning.The authors report faster convergence after MLP regularization and compared with modeling compressed VAE latents.Source: §3.3 · p. 5

3.4. Extension to Long Video Generation

  • 1/16 as many tokens are used for semantic representations as for VAE latents; long-video generation applies full attention to semantics and shifted-window attention in VAE space.Full attention models semantic features for consistency across scenes and characters, while shifted windows avoid quadratic growth with frame count.Source: §3.4 · p. 5, Figure 5 · p. 5

4. Experimental Results

SemanticGen is comparable to strong short-video baselines and performs better on long-video consistency and temporal stability. Ablations report improved quality with compressed semantic representations and faster convergence in semantic space than in VAE latent space.

4.1. Experiment Settings

  • Long-video training uses 60-second movie and TV segments, sampled at 24 fps for the VAE and 1.6 fps for the semantic encoder.Evaluation uses VBench and VBench-Long; long-video drift compares metric values between initial and final segments.Source: §4.1 · pp. 5–6

4.2. Comparison with State-of-the-Art Methods

  • SemanticGen is comparable to state-of-the-art and continued-training baselines on short video, while substantially outperforming long-video baselines on consistency and temporal stability.It also surpasses baselines on the long-video drift metric.Source: Table 2 · p. 7, Table 1 · p. 7, §4.2 · p. 7
  • Long-video comparisons show stronger consistency and fewer drifting issues, while short-video examples show prompt adherence and performance comparable to Base-CT.The paper reports inconsistent backgrounds and artifacts for Base-CT-Swin on long videos.Source: Figure 7 · p. 6, Figure 6 · p. 6, §4.2 · p. 7

4.3. More Analysis and Ablation Studies

  • Reducing semantic-representation dimensionality from 2048 to 64 or 8 channels improves visual quality, with fewer broken frames and artifacts at lower dimensions.The ablation uses an MLP for the 64- and 8-channel representations.Source: §4.3 · pp. 7–8, Table 3 · p. 8
  • After 10K training steps, semantic-space modeling produces reasonable videos, whereas compressed VAE-space modeling produces only coarse color patches.The comparison keeps the two-stage framework unchanged except for replacing the semantic encoder with a VAE encoder.Source: §4.3 · p. 8, Figure 9 · p. 8

5. Conclusion and Limitations

SemanticGen generates compact semantic features for global planning before producing conditioned VAE latents. The paper reports faster convergence and efficient extension to long videos, while noting limits in texture consistency and high-frequency information.

5. Conclusion and Limitations

  • SemanticGen first generates high-level semantic representations for global planning, then generates conditioned VAE latents to produce the video.The semantic features define the global layout, and the VAE latents add high-frequency details.Source: §5 · p. 8
  • The authors report faster convergence in semantic space than VAE latent space and efficient extension to long-video generation.Source: §5 · p. 8
  • Long-video texture consistency remains difficult, and low-fps semantic sampling loses high-frequency temporal information.Source: §5 · p. 8

Supplementary Material

The supplementary material describes the transformer-based latent diffusion model used as the base text-to-video generator. Its design combines a 3D VAE, spatiotemporal attention, and timestep-conditioned normalization.

A. Introduction of the Base Text-to-Video Generation Model

B.1. More Results of SemanticGen

SemanticGen’s additional results show high-quality videos that adhere to prompts, with long-form videos exhibiting long-term consistency and less drifting.

B.1. More Results of SemanticGen

  • The additional synthesized videos demonstrate high-quality outputs that adhere to the text prompts.The examples are presented in Figures 11 and 12.Source: §B.1 · p. 12
  • The examples demonstrate long videos with long-term consistency.Source: §B.1 · p. 12
  • SemanticGen significantly alleviates drifting issues in long videos.Source: §B.1 · p. 12

B.2. Comparison with Additional Baselines

SemanticGen’s qualitative long-video comparisons show better long-term consistency and fewer drifting issues than additional baselines. Comparisons with continued-training baselines use identical data and training steps.

B.2. Comparison with Additional Baselines

  • SemanticGen achieves better long-term consistency and significantly alleviates drifting issues in qualitative comparisons with additional long-video baselines.Baselines include MAGI-1 [57], SkyReels-V2 [10], Self-Forcing [30], LongLive [70], and HoloCine [45].Source: §B.2 · p. 12
  • Continued-training baselines use the same data and number of training steps, but standard diffusion loss without semantic modeling.These comparisons include Base-CT and Base-Swin-CT.Source: §B.2 · p. 12
  • Baselines can show severe color shifts or frame inconsistencies, while Base-CT-Swin shows inconsistent backgrounds and more artifacts.Source: §B.2 · p. 12

B.3. Failure Cases Visualization

SemanticGen’s semantic encoders can miss high-frequency temporal changes when sampled at low frame rates, and semantic features may not preserve fine details consistently in long videos.

B.3. Failure Cases Visualization

  • At 1.6 fps, the semantic encoder misses changes occurring within 1/24 of a second, such as lightning-driven shifts in the sky, so the generated video lacks flicker.Source: §B.3 · p. 12
  • For long-video generation, semantic features may not fully capture textures or small objects, which can therefore be inconsistently preserved.Source: §B.3 · p. 12
  • The authors anticipate that video understanding tokenizers combining high temporal compression with high sampling rates could further enhance SemanticGen.Source: §B.3 · p. 12

C. Future Work

Future work calls for broader evaluation of semantic encoders and more informative video representations, including encoders that combine high temporal compression with high-rate sampling.

C. Future Work

  • A systematic analysis could test whether generation performance varies across semantic encoders trained with different paradigms.The paper uses the vision tower of Qwen-2.5-VL [3] and identifies visual-text alignment and self-supervised learning as examples of training paradigms.Source: C. Future Work · p. 12
  • Future video semantic encoders should combine high temporal compression with high-rate sampling to better support modeling high-frequency temporal information.Source: C. Future Work · p. 12
  • The paper identifies more powerful pre-trained video semantic encoders as a possible route to better generation performance.Source: C. Future Work · p. 12

Related papers

Loading 2512.20619v3…