PaperlayerView exact revision on arXiv

Source-linked AI summary

LongCat-Video Technical Report

Meituan LongCat Team, Xunliang Cai, Qilong Huang, Zhuoliang Kang, Hongyu Li, Shijun Liang, Liya Ma, Siyu Ren, Xiaoming Wei, Rixu Xie, Tong Zhang

arXiv:2510.22200v2cs.CV

TL;DR

Efficient long-video generation is an important capability for world models. LongCat-Video unifies multiple video-generation tasks in one model, achieving comparable performance to leading open-source and commercial models while producing minutes-long videos.

  • Problem

    Efficient long-video generation is particularly important for video generation’s role as a pathway toward world models.

    Source: §1 · p. 4

  • Method

    LongCat-Video unifies Text-to-Video, Image-to-Video, and Video-Continuation in one framework and is pretrained on Video-Continuation tasks for long-video generation.

    Source: §1 · p. 4

  • Results

    LongCat-Video performs comparably to leading open-source and latest commercial models, and produces minutes-long videos without color drifting or quality degradation.

    Source: §1 · p. 4

  • Takeaways & Limitations

    LongCat-Video marks the authors’ first step toward developing world models.

    Source: §6 · p. 25

    GRPO training uses only Text-to-Video tasks; task-specific rewards for Image-to-Video and Video-Continuation remain future work.

    Source: §4.2 · p. 18

Abstract

from arXiv · show

Video generation is a critical pathway toward world models, with efficient long video inference as a key capability. Toward this end, we introduce LongCat-Video, a foundational video generation model with 13.6B parameters, delivering strong performance across multiple video generation tasks. It particularly excels in efficient and high-quality long video generation, representing our first step toward world models. Key features include: Unified architecture for multiple tasks: Built on the Diffusion Transformer (DiT) framework, LongCat-Video supports Text-to-Video, Image-to-Video, and Video-Continuation tasks with a single model; Long video generation: Pretraining on Video-Continuation tasks enables LongCat-Video to maintain high quality and temporal coherence in the generation of minutes-long videos; Efficient inference: LongCat-Video generates 720p, 30fps videos within minutes by employing a coarse-to-fine generation strategy along both the temporal and spatial axes. Block Sparse Attention further enhances efficiency, particularly at high resolutions; Strong performance with multi-reward RLHF: Multi-reward RLHF training enables LongCat-Video to achieve performance on par with the latest closed-source and leading open-source models. Code and model weights are publicly available to accelerate progress in the field.

1 Introduction

LongCat-Video is a 13.6B-parameter video generation model and a first step toward world models, with particular emphasis on efficient, high-quality long-video generation. It unifies text-to-video, image-to-video, and video-continuation tasks, and uses continuation pretraining, coarse-to-fine inference, and multi-reward RLHF to support its capabilities.

Contribution

  • LongCat-Video is a 13.6B-parameter general-purpose model that performs strongly across video-generation tasks, particularly efficient, high-quality long-video generation.The report describes it as the authors’ first step toward world models.Source: §1 · p. 4

Unified architecture for multiple tasks

  • One model supports text-to-video, image-to-video, and video-continuation generation through multi-task training and zero, one, or multiple conditioning frames.These conditioning-frame counts correspond respectively to text-to-video, image-to-video, and video continuation.Source: §1 · p. 4

Long video generation

  • Pretraining on video-continuation tasks enables minutes-long videos without color drifting or quality degradation.The report identifies accumulated generation error over time as a challenge for long-video generation.Source: §1 · p. 4

Efficient inference

  • A coarse-to-fine strategy generates video at 480p, 15fps before refining it to 720p, 30fps, while block sparse attention uses less than 10% of dense-attention computations.A high-resolution expert LoRA module leverages the base model’s knowledge during refinement.Source: §1 · p. 4

Strong performance with multi-reward RLHF

  • Multi-reward RLHF using GRPO [Guo et al., 2025] yields performance comparable to leading open-source models and the latest commercial solutions; code and model weights are released.The evaluations use internal and public benchmarks with human and model-based annotations.Source: §1 · p. 4

2 Data

LongCat-Video’s data pipeline preprocesses diverse raw videos into training clips, enriches them with metadata and captions, and analyzes caption-based content distributions to guide dataset construction.

2 Data

  • The curation pipeline has preprocessing and annotation stages: the first prepares diverse, intact clips, and the second adds metrics and attributes for downstream dataset assembly.The report introduces the pipeline and then presents the curated training-data distribution.Source: §2 · pp. 4–5

2.1.1 Data Preprocessing Stage

  • Raw videos are deduplicated, segmented into content-consistent clips, cropped to remove black borders, then compressed and packaged for training.Deduplication uses source video IDs and MD5 hashes; segmentation uses PySceneDetect [Castellano] and TransNetV2 [Souˇcek and Lokoˇc, 2020], while cropping uses FFMPEG [FFmpeg Developers, 2014].Source: §2.1.1 · p. 5

2.1.2 Data Annotation Stage

  • Clips receive metadata and captions spanning temporal actions, cinematography, visual style, and augmented multilingual descriptions for targeted filtering and diverse text conditioning.Metadata covers duration, resolution, frame rate, bitrate, aesthetic and blur scores, text coverage, watermarks, and optical-flow motion. Captions use fine-tuned LLaVA-Video [Zhang et al., 2024] and Tarsier2 [Yuan et al., 2025a] for temporal descriptions, Qwen2.5VL [Bai et al., 2025] for shot, lens, and style attributes, and Chinese-English translation, summaries, and attribute combinations for augmentation.Source: §2.1.2 · pp. 5–6, Figure 3 · p. 5

2.2 Data Distribution

  • Caption embeddings are clustered into content categories, whose volume and density inform targeted supplementation or rebalancing for different training phases.This enables data subsets to be allocated dynamically according to training objectives.Source: Figure 4 · p. 6, §2.2 · p. 7

3 Method

This section describes LongCat-Video’s method for text-, image-, and continuation-based video generation, including its DiT architecture, conditioning, and training approach. It also covers coarse-to-fine high-resolution inference, video refinement, and block-sparse attention to reduce computation while preserving near-lossless quality.

3.1 Model Architecture

  • A single DiT with single-stream transformer blocks supports Text-to-Video, Image-to-Video, and Video-Continuation by treating each as prediction of future frames from preceding conditions.Blocks use 3D self-attention, text cross-attention, SwiGLU FFNs, AdaLN-Zero, RMSNorm for QKNorm, and 3D RoPE; WAN2.1 VAE compression and patchifying yield 4 × 16 × 16 compression, and umT5 supports English and Chinese captions.Source: §3.2 · p. 7, §3.1 · p. 7

3.2 Unified Model for Multiple Tasks

  • Condition and noisy frames are concatenated with matching timestep partitions; zero-timestep, fixed condition frames are excluded from loss and cached for reuse across sampling steps.Block attention prevents noisy tokens from affecting condition tokens, while condition tokens do not participate in cross-attention computation.Source: §3.2 · pp. 7–8, Figure 5 · p. 8

3.3.1 GRPO for Flow Matching Modeling

  • Flow-matching GRPO fixes shared initial noise and injects SDE noise at one sampled critical timestep, while reweighting policy/KL losses and using maximum group standard deviation to stabilize optimization.The reweighting normalizes gradient magnitude across timesteps and step sizes; maximum group standard deviation scales down updates from groups with less reliable reward estimates.Source: §3.3.1 · pp. 10–11

3.3.2 Reward Models and Multi-Reward Training

  • Multi-reward GRPO combines visual-quality, motion-quality, and text-video-alignment signals, whose mutual constraints reduce over-optimization on a single metric and reward hacking.The reward models use HPSv3 for visual quality and alignment, and VideoAlign-based models for motion and alignment; grayscale input is used for motion assessment.Source: §3.3.2 · pp. 12–13, Figure 9 · p. 12

3.4.1 Coarse-to-Fine Generation

  • Coarse-to-fine generation first produces 480p, 15fps video, then upsamples and refines it to 720p, 30fps, improving image quality and high-frequency details.A LoRA-fine-tuned refinement expert reuses the base model; with tthresh = 0.5, refinement takes five sampling steps, and the approach can improve texture and correct local distortions.Source: §3.4.1 · pp. 13–14, Figure 10 · p. 14

3.4 Efficient Video Generation

  • 10.1× acceleration in 720p, 30fps generation is achieved versus native 720p, 15fps generation by combining coarse-to-fine generation and block sparse attention.The report also states that the combined strategies enable 720p, 30fps video generation within minutes.Source: §3.4 · p. 13, Table 2 · p. 13, §3.4.1 · p. 14

3.4.1 Coarse-to-Fine Generation

  • Refinement concatenates high-resolution condition frames with upsampled low-resolution video from the first stage, then denoises the input to produce high-resolution output.Noise is added to the VAE-encoded concatenation at level tthresh; the resulting video is Xsr.Source: §3.4.1 · p. 15

3.4.2 Block Sparse Attention

  • Block sparse attention retains less than 10% of the original computational load while achieving near-lossless generation quality and accelerating high-resolution training and inference.The method targets the quadratic growth of self-attention with token count.Source: §3.4.2 · p. 15
  • The 3D block-sparse method selects top-r key blocks by similarity to each query block, then computes standard attention over keys in those selected blocks.It partitions queries and keys into non-overlapping 3D blocks and scores blocks using their average values.Source: §3.4.2 · p. 16
  • After training, top-k block sparsity achieved lossless sparse-attention adaptation, so LongCat-Video adopted this pattern; the implementation also supports ring context parallelism and customizable masks.The 3D block-sparse attention implementation is open-sourced with forward and backward implementations.Source: §3.4.2 · pp. 15–16

4 Training

LongCat-Video training combines progressive pretraining and supervised fine-tuning, GRPO-based RLHF, and acceleration training through distillation and a refinement expert LoRA. Its training setup also uses task and resolution curricula, while RLHF improvements generalize from Text-to-Video to Image-to-Video and Video-Continuation.

4 Training

  • Training proceeds through base-model training, GRPO-based RLHF, and acceleration training, using LoRA for flexible stacking of enhancements.Base-model training includes progressive pretraining and supervised fine-tuning; acceleration training includes distillation and a refinement expert LoRA module.Source: §4 · p. 16

4.1 Base Model Training

  • The base model learns flow matching by predicting velocity toward the noise-free video latent and minimizing mean squared error against the ground-truth velocity.Training samples timesteps uniformly, applies logit-normal-like loss weighting for stability, and shifts timesteps toward higher noise for higher-resolution or longer videos.Source: §4.1 · pp. 16–17
  • Progressive pretraining moves from low-resolution images to video motion learning, then jointly trains T2I, T2V, I2V, and Video-Continuation across increasing resolutions.Video-Continuation conditioning frames receive independent per-frame noise to improve robustness to color drift; size buckets group samples by closest aspect ratio.Source: §4.1 · p. 17, Table 3 · p. 17
  • SFT uses curated high-quality data, balances categories by caption-embedding density, and adds specialized camera-motion and visual-style datasets to improve instruction following.Filtering considers aesthetic, video, and motion quality metrics.Source: §4.1 · p. 17, Table 4 · p. 17

4.2 RLHF Training

  • Text-to-Video-only GRPO training improves instruction-following, visual quality, and motion quality, with gains generalizing to Image-to-Video and Video-Continuation.Task-specific rewards, such as a long-video quality-degradation penalty for Video-Continuation, remain future work.Source: §4.2 · p. 18

4.3 Acceleration Training

  • CFG distillation uses CFG-Zero with default guidance strength 4.0; combined with consistency-model distillation, it enables 16-step inference with quality comparable to results from more than 50 steps.The model is distilled and a refinement expert module is trained to enable efficient inference.Source: §4.3 · p. 18

4.4 Training Infrastructure

  • Training infrastructure combines distributed parallelism, activation checkpointing, resolution buckets, and VAE-operation caching, achieving 33%–38% Model Flops Utilization.The reported infrastructure includes DeepSpeed-Zero2, Context Parallelism, and Ring Attention.Source: §4.4 · p. 18

5 Evaluation

LongCat-Video is evaluated on internal Text-to-Video and Image-to-Video benchmarks, VBench 2.0, and representative generation examples. The evaluations find competitive Text-to-Video performance, strong visual quality in Image-to-Video, and support for minutes-long continuation and instruction changes.

5.1 Internal Benchmarks

  • The internal benchmark covers 1,628 samples across Text-to-Video and Image-to-Video, using dimension-based human and automatic assessments and a weighted 2:1 average of human and automatic scores.Text-to-Video dimensions are text alignment, visual quality, motion quality, and overall quality; Image-to-Video adds image alignment. Human evaluation uses MOS and pairwise GSB, while the vision-language judge correlates above 0.92 with human assessments across dimensions.Source: §5.1 · p. 19
  • LongCat-Video’s Text-to-Video overall MOS exceeds PixVerse-V5 and Wan 2.2-T2V-A14B, while its visual-quality score is nearly on par with Wan 2.2.Veo3 leads overall, and LongCat-Video is described as balanced across quality dimensions.Source: Figure 14 · p. 20, §5.1 · p. 20
  • LongCat-Video is preferred over Wan2.2-T2V-A14B in overall GSB quality, while nearly tying PixVerse-V5 at 242 vs. 246 and showing a visual-quality advantage.It trails Veo3 in the preference study.Source: §5.1 · p. 20, Figure 15 · p. 21
  • 3.27 is LongCat-Video’s highest Image-to-Video MOS dimension, Visual Quality; its Overall Quality is 3.17, below Seedance 1.0’s 3.35.Its Image-Alignment score is 4.04 and Motion Quality is 3.59, below other models; Hailuo-02 and Wan2.2-I2V-A14B lead Image-Alignment at 4.18, and Hailuo-02 leads Motion Quality at 3.80.Source: Figure 16 · p. 21, §5.1 · p. 21

5.2 Public Benchmarks

  • On VBench 2.0, LongCat-Video’s total score is second only to Veo3 and Vidu Q1, and it leads all other methods in Commonsense.The Commonsense result indicates strength in motion rationality and physical laws.Source: Table 8 · p. 21, §5.2 · p. 21

5.3 Text-to-Video Examples

  • Figure 17 presents representative Text-to-Video generation results from LongCat-Video.Source: Figure 17 · p. 22

5.4 Image-to-Video Examples

  • Given the same initial image, LongCat-Video responds to instructions for different actions in its Image-to-Video examples.Source: Figure 18 · p. 23

5.5 Long-Video Generation Examples

  • LongCat-Video supports minutes-long video continuation without quality degradation and interactive generation with changing instructions for each clip.Source: Figure 19 · p. 24

6 Conclusion and Future Work

LongCat-Video is a unified 13B-parameter model for Text-to-Video, Image-to-Video, and Video-Continuation that performs strongly across these tasks, particularly in long video generation. The report frames efficient video generation as a step toward world models and identifies future research directions in physical knowledge, multimodal memory, and LLM/MLLM knowledge.

6 Conclusion and Future Work

  • LongCat-Video unifies Text-to-Video, Image-to-Video, and Video-Continuation in one 13B-parameter framework and performs strongly across all supported tasks, particularly long video generation.Pretraining on Video-Continuation enables its long video generation performance.Source: §6 · p. 25
  • The report presents LongCat-Video as a general-purpose model for diverse video content creation and its first step toward world models.Efficient long video generation addresses world models’ rendering problem by expressing world knowledge through generated video.Source: §6 · p. 25
  • Future directions include better physical knowledge modeling, multimodal memory integration in video generation, and incorporating knowledge from LLMs and MLLMs.Source: §6 · p. 25

A Appendix · A.1 Appendix-A · A.1.1 GRPO Preliminaries

The appendix introduces GRPO as an objective for optimizing a generative flow model. It outlines policy sampling, group-based advantage estimates, and KL regularization.

A.1.1 GRPO Preliminaries

  • GRPO optimizes the generative flow model by maximizing an objective function.Source: §A.1.1 · p. 28
  • A group of G samples is drawn from the current policy πθold conditioned on prompt c, with each sample generated by discretizing the reverse-time stochastic differential equation.Source: §A.1.1 · p. 28
  • The sampled group is used for policy optimization.Source: §A.1.1 · p. 28
  • The objective accounts for probability changes associated with a transition, whose probability follows a specified relation.Source: §A.1.1 · p. 28
  • The advantage term provides normalized estimates by comparing individual rewards against group statistics.Source: §A.1.1 · p. 28
  • KL regularization constrains deviation from a reference policy to stabilize training, with β controlling its strength in the flow-matching formulation.Source: §A.1.1 · p. 28

A.1.2 The Gradient of the Policy and KL Loss

The section derives the policy and KL gradients, identifies shared scaling issues, and introduces reweighting to modify both losses.

A.1.2 The Gradient of the Policy and KL Loss

  • The policy-loss gradient is derived from the Gaussian conditional score under the SDE reparameterization, treating x_t as constant when differentiating with respect to θ.The derivation simplifies the drift term before substituting into the score function.Source: §A.1.2 · p. 29
  • A reweighting coefficient is introduced for the policy loss, yielding a modified gradient.The derivation first simplifies coefficient and scaling terms before giving the final policy-gradient expression.Source: §A.1.2 · pp. 29–30
  • The KL-divergence gradient has the same scaling issues as the policy gradient, motivating KL reweighting and a revised GRPO objective.The reweighted KL loss yields a simplified gradient.Source: §A.1.2 · p. 30

A.1.3 Fix the stochastic timestep in SDE sampling

The objective is simplified to focus on a critical stochastic timestep, sampled uniformly from the first six steps during experiments.

A.1.3 Fix the stochastic timestep in SDE sampling

  • The objective is simplified to focus only on the critical stochastic timestep.This simplification is described for SDE sampling in Sec. 3.3.1.Source: §A.1.3 · p. 30
  • In experiments, the critical timestep is sampled uniformly from the first 6 steps, with 16 total training sampling steps.The sampling interval is t′ ∼ U(0, T′ − 1), and T′ = 6.Source: §A.1.3 · p. 31

A.1.4 Multi-reward GRPO Training … A.3 Appendix-C

The appendix derives multi-reward GRPO as a weighted combination of reward-specific advantages and describes block-sparse attention methods for efficient attention computation, including ring attention for context parallelism.

A.1.4 Multi-reward GRPO Training

  • Multi-reward GRPO uses the weighted sum of independently group-normalized, reward-specific relative advantages in its policy loss.The derivation expresses the total gradient as a weighted sum across reward functions.Source: §A.1.4 · p. 31
  • The GRPO experiment settings are presented in Table 9.Source: Table 9 · p. 32

A.2.1 Modeling of Block Sparse Attention

  • 3D block sparse attention rearranges video tokens into spatiotemporal blocks, pools their queries and keys, and selects each query block’s top-r key blocks.The selected neighbors define a binary mask for the attention computation.Source: §A.2.1 · p. 32

A.2.2 Modeling of Ring Block Sparse Attention for Context Parallelism

  • For context parallelism, each rank constructs a local block-selection mask by scoring pooled keys gathered across ranks and selecting top-r key blocks.The mask is based on pooled scores for each rank’s query blocks.Source: §A.2.2 · p. 33
  • Ring attention overlaps local pooled-score computation with key communication between adjacent ranks, then overlaps attention computation with key and value communication.Each rank computes its attention output using the online softmax algorithm.Source: §A.2.2 · p. 33

A.2.3 Implementation Details

  • CDF-p improves generation quality at high speedup ratios in training-free use, but top-r is used for trainable implementations to avoid variable key-selection counts.CDF-p selects keys until cumulative softmax scores reach a threshold; top-r selects a fixed number.Source: §A.2.3 · p. 34

Related papers

Loading 2510.22200v2…