PaperlayerView exact revision on arXiv

Source-linked AI summary

LongCat-Video Technical Report

Meituan LongCat Team, Xunliang Cai, Qilong Huang, Zhuoliang Kang, Hongyu Li, Shijun Liang, Liya Ma, Siyu Ren, Xiaoming Wei, Rixu Xie, Tong Zhang

arXiv:2510.22200v2cs.CV

TL;DR

Video generation is a pathway toward world models, but efficient long-video inference remains important. LongCat-Video addresses this gap with a unified 13.6B-parameter model, Video-Continuation pretraining, efficient inference strategies, and multi-reward RLHF. It supports multiple generation tasks, produces 720p, 30fps videos within minutes, and achieves strong performance comparable to leading open-source and commercial systems.

  • Problem

    Efficient long-video generation is particularly important for using video generation to support world-model simulation and prediction.

  • Method

    LongCat-Video unifies Text-to-Video, Image-to-Video, and Video-Continuation, combining Video-Continuation pretraining, coarse-to-fine generation, block sparse attention, and multi-reward GRPO post-training.

  • Results

    LongCat-Video delivers strong performance across supported tasks, particularly long-video generation, with 720p, 30fps generation within minutes and performance comparable to leading open-source and commercial models.

  • Takeaways & Limitations

    The model provides a general-purpose video generation framework and marks the authors’ first step toward developing world models.

Abstract

from arXiv · show

Video generation is a critical pathway toward world models, with efficient long video inference as a key capability. Toward this end, we introduce LongCat-Video, a foundational video generation model with 13.6B parameters, delivering strong performance across multiple video generation tasks. It particularly excels in efficient and high-quality long video generation, representing our first step toward world models. Key features include: Unified architecture for multiple tasks: Built on the Diffusion Transformer (DiT) framework, LongCat-Video supports Text-to-Video, Image-to-Video, and Video-Continuation tasks with a single model; Long video generation: Pretraining on Video-Continuation tasks enables LongCat-Video to maintain high quality and temporal coherence in the generation of minutes-long videos; Efficient inference: LongCat-Video generates 720p, 30fps videos within minutes by employing a coarse-to-fine generation strategy along both the temporal and spatial axes. Block Sparse Attention further enhances efficiency, particularly at high resolutions; Strong performance with multi-reward RLHF: Multi-reward RLHF training enables LongCat-Video to achieve performance on par with the latest closed-source and leading open-source models. Code and model weights are publicly available to accelerate progress in the field.

1 Introduction

LongCat-Video is a 13.6B-parameter video generation model designed for unified multi-task support, efficient high-quality long-video generation, and strong overall performance. It targets video generation as a pathway toward world models, where efficient long-video generation is particularly important.

  • Motivation: Video generation is presented as a pathway toward world models because it can compress geometric, semantic, and physical knowledge for simulation and prediction.The report identifies efficient long-video generation as particularly essential for this direction.
  • Model overview: LongCat-Video is a 13.6B-parameter foundational model supporting Text-to-Video, Image-to-Video, and Video-Continuation tasks.A multi-task strategy distinguishes these tasks by using zero, one, or multiple conditioning frames, respectively.
  • Long video generation: Video-Continuation pretraining enables minutes-long videos without color drifting or quality degradation.The approach addresses error accumulation over time in long-video generation.
  • Efficient inference: Coarse-to-fine generation produces videos first at 480p, 15fps and then refines them to 720p, 30fps.High-resolution refinement uses an expert LoRA module and block sparse attention to improve efficiency.
  • Post-training and performance: Multi-reward RLHF with GRPO yields performance comparable to leading open-source models and the latest commercial solutions.Evaluations use human and model-based annotations on internal and public benchmarks.

2 Data

LongCat-Video’s data pipeline combines preprocessing, rich annotation, caption augmentation, and distribution analysis to construct flexible training datasets. It extracts consistent clips, records filtering attributes, enriches captions, and rebalances content categories when needed.

  • Data Preprocessing Stage: The preprocessing stage acquires diverse sources, removes duplicates, segments videos into clips, and crops black borders.These steps aim to preserve diversity, integrity, and content consistency in collected training videos.
  • Data Annotation Stage: The annotation stage stores video metadata and quality attributes in a library for targeted dataset construction.Recorded attributes include duration, resolution, frame rate, bitrate, aesthetics, blur, text coverage, watermarks, and motion.
  • Captioning workflow: Captions combine basic descriptions of video content with cinematography and visual-style attributes.Multiple models annotate appearance, temporal dynamics, camera movements, shot sizes, lens types, and visual style.
  • Caption augmentation: Caption augmentation translates captions between Chinese and English, generates concise summaries, and integrates randomly selected style elements.The resulting multiple textual styles are intended to improve robustness to diverse textual inputs.
  • Data Distribution: Text-embedding clustering categorizes clips by caption content and supports targeted supplementation or rebalancing of uneven categories.An LLM summarizes clusters and assigns tags, while data volume and distribution density are assessed for uniformity.

3 Method

LongCat-Video uses a unified DiT-based architecture for multiple video-generation tasks and combines specialized GRPO training with efficient high-resolution inference techniques. Its design separates condition and noisy frames, reuses cached condition features, and applies coarse-to-fine generation with block sparse attention.

  • Model Architecture: LongCat-Video uses a standard single-stream DiT with 3D self-attention, text cross-attention, SwiGLU FFNs, AdaLN-Zero modulation, RMSNorm, and 3D RoPE.The model specifications are summarized in Table 1.
  • Unified Model for Multiple Tasks: A unified framework supports Text-to-Video, Image-to-Video, and Video-Continuation by varying the number of conditioning frames.The tasks use zero, one, and multiple conditioning frames, respectively.
  • Unified Model for Multiple Tasks: Block causal attention prevents condition tokens from being influenced by noisy tokens and excludes them from cross-attention, enabling KV-cache reuse across denoising steps.Noisy queries attend to condition and noisy keys and values, while condition-token computations depend only on the input condition frames.
  • GRPO for Flow Matching Modeling: GRPO for video generation uses stochastic noise search, policy-loss reweighting, and multi-reward relative advantages to improve optimization stability and generation quality.The method is presented as a flow-matching training procedure with multiple reward models and group-normalized advantages.
  • Reward Models and Multi-Reward Training: Multiple reward signals provide balanced optimization across video-quality aspects and impose mutual constraints that reduce over-optimization of any single metric and reward hacking.The training procedure computes reward-specific group statistics and combines the resulting signals for policy optimization.
  • Efficient Video Generation: Coarse-to-fine generation and block sparse attention reduce high-resolution inference cost while preserving generation quality.The system distills the base model, generates lower-resolution video before refinement, and retains less than 10% of the original sparse-attention computational load with near-lossless quality.

4 Training

Training builds LongCat-Video through progressive pretraining, supervised fine-tuning, GRPO-based RLHF, and inference acceleration. The pipeline combines flow matching, staged training, distillation, refinement, and sparse attention.

  • Training pipeline: Training comprises base-model pretraining, supervised fine-tuning, RLHF with GRPO, and a final acceleration component.The overall procedure uses progressive pretraining and SFT before preference alignment and acceleration.
  • Base model training: Flow matching trains the network to predict velocity from noisy video latents and task conditions using mean squared error.The conditions include text prompts and conditional image or video latents.
  • Base model training: Progressive pretraining moves from low-resolution images to video training, enabling efficient learning of visual representations before motion dynamics.The stages begin with image training and transition to dedicated video training.
  • Supervised fine-tuning: SFT uses curated high-quality data filtered by aesthetic, video, and motion metrics, with density-aware sampling and specialized datasets.Samples are selected inversely proportional to caption-embedding density to balance categories.
  • RLHF training: GRPO post-training uses multiple video-quality rewards and trains only on Text-to-Video, while reported gains generalize to Image-to-Video and Video-Continuation.Task-specific rewards for long-video quality remain future work.
  • Acceleration: Distillation enables 16-step inference with quality comparable to inference using more than 50 steps, while refinement training activates 93.75% sparse attention after convergence.A refinement expert module and block sparse attention support efficient inference.

5 Evaluation

Evaluation uses internal human and automatic benchmarks for Text-to-Video and Image-to-Video, measuring alignment, visual quality, motion, and overall quality. LongCat-Video is competitive in Text-to-Video and visual quality, but trails competitors on several Image-to-Video dimensions.

  • Benchmark design: The internal benchmark contains 1,628 samples: 1,228 Text-to-Video cases and 400 Image-to-Video cases.Text-to-Video includes 500 human and 728 automatic assessments.
  • Text-to-Video evaluation: Text-to-Video evaluation measures text alignment, visual quality, motion quality, and overall quality using MOS and GSB protocols.MOS uses a 5-point absolute scale, while GSB provides relative comparisons.
  • Image-to-Video evaluation: Image-to-Video evaluation adds Image-Alignment, which measures preservation of reference-image subjects, backgrounds, relationships, and style.The benchmark uses diverse first-frame reference images spanning style, content, and quality.
  • Evaluation protocol: The automatic vision-language judge achieves correlations consistently exceeding 0.92 with human assessments across text alignment, visual quality, and motion quality.The judge is trained on high-quality human-annotated data.
  • Text-to-Video results: In Text-to-Video MOS, LongCat-Video surpasses PixVerse-V5 and Wan 2.2-T2V-A14B in overall quality while remaining nearly on par with Wan 2.2 in visual quality.Veo3 leads overall quality, supported by stronger text-alignment and motion scores.
  • Text-to-Video results: In Text-to-Video GSB, LongCat-Video is nearly tied with PixVerse-V5 overall at 242 vs. 246 and preferred over Wan2.2-T2V-A14B overall.The model shows a distinct visual-quality advantage over PixVerse-V5 and leads Wan2.2 in text alignment and motion quality.
  • Image-to-Video results: In Image-to-Video MOS, LongCat-Video leads Visual Quality at 3.27 but scores lower on Image-Alignment at 4.04 and Motion Quality at 3.59 than other models.Its Overall Quality is 3.17, below Seedance 1.0’s leading 3.35.

6 Conclusion and Future Work

LongCat-Video unifies three video-generation tasks in one 13B-parameter model and particularly emphasizes long-video generation. The authors position it as a first step toward world models and identify future work in physical knowledge, multimodal memory, and LLM or MLLM integration.

  • Conclusion: LongCat-Video is a 13B-parameter foundational model unifying Text-to-Video, Image-to-Video, and Video-Continuation.It is presented as a general-purpose model for diverse video content creation scenarios.
  • Conclusion: Pretraining on Video-Continuation enables the model’s reported strength in long video generation.The conclusion connects efficient long-video generation with expressing world knowledge through generated video.
  • Future work: Future directions include better physical-knowledge modeling, multimodal memory integration, and incorporating knowledge from LLMs and MLLMs.

7 Contributors and Acknowledgments

The report states that contributors are listed alphabetically by last name, with asterisks marking people who have left the team.

  • Contributors and acknowledgments: Contributor names are ordered alphabetically by last name, and an asterisk marks contributors who left the team.

A.1.1 GRPO Preliminaries

This section introduces GRPO optimization for generative flow models, beginning with grouped sampling from the current policy and defining policy, KL, and advantage components.

  • GRPO optimizes a generative flow model with a stated objective function.
  • Sampling Process: Samples are drawn from the current policy conditioned on the prompt and generated by discretizing the reverse-time stochastic differential equation.
  • Sampling Process: The sampling process organizes a group of G samples for policy optimization.
  • The objective compares individual rewards against group statistics through normalized advantage estimates.
  • KL Regularization: KL divergence regularization constrains deviation from the reference policy, with β controlling its strength.

A.1.2 The Gradient of the Policy and KL Loss

This section derives policy and KL-loss gradients for the flow-model setting, then introduces reweighting coefficients to address shared scaling issues before presenting a revised GRPO objective.

  • The policy gradient is derived from the Gaussian transition distribution used in the SDE sampling process.
  • The derivation simplifies the coefficient term associated with the stochastic timestep.
  • A reweighting coefficient is introduced for the policy loss, yielding a reweighted policy loss and modified gradient.
  • KL Loss: The KL-loss gradient exhibits the same scaling issues as the policy-loss gradient.
  • KL Loss: KL reweighting produces a reweighted KL loss, simplified gradient, and revised GRPO objective.

A.1.3 Fix the stochastic timestep in SDE sampling

This section fixes the stochastic timestep used in SDE sampling and describes the resulting multi-reward gradient formulation alongside block-sparse attention construction and implementation choices.

  • Stochastic Timestep: The simplified objective focuses on a critical stochastic timestep, sampled uniformly from the first T′ steps.Experiments set T′ = 6, while training uses 16 total sampling steps.
  • Multi-Reward Optimization: In flow-matching models, GRPO uses relative advantage estimates and noise to estimate the reward gradient with respect to the velocity field.
  • Multi-Reward Optimization: For multiple rewards, the total gradient is the weighted sum of individual reward gradients, making the effective relative advantage their weighted sum.
  • Block Sparse Attention: Block-sparse attention rearranges T × H × W video sequences into 3D blocks and selects top-r key blocks for each query block.
  • Block Sparse Attention: Pooled queries and keys produce block-level scores, which construct masks before masked attention and softmax compute the attention weights.
  • Context Parallelism: Context parallelism uses local block-selection masks and ring attention to overlap attention computation with key-value communication.
  • Implementation: The implementation uses Triton and Flash Attention foundations, with block size t = h = w = 4 and sparsity controlled by r.
  • Mask Construction: CDF-p gives better quality at high speedup in training-free settings, whereas top-r is adopted for trainable implementation because variable selection costs time.
Loading 2510.22200v2…