Source-linked AI summary
Goku: Flow Based Video Generative Foundation Models
Shoufa Chen, Chongjian Ge, Yuqi Zhang, Yida Zhang, Fengda Zhu, Hao Yang, Hongxiang Hao, Hui Wu, Zhichao Lai, Yifei Hu, Ting-Che Lin, Shilong Zhang, Fu Li, Chuan Li, Xing Wang, Yanghua Peng, Peize Sun, Ping Luo, Yi Jiang, Zehuan Yuan, Bingyue Peng, Xiaobing Liu
TL;DR
Goku addresses the challenge of high-quality joint image-and-video generation with a rectified-flow Transformer family built on curated multimodal data, a shared latent architecture, and scalable training infrastructure. It achieves strong results across text-to-image and text-to-video benchmarks, including 0.76 on GenEval, 83.65 on DPG-Bench, and 84.85 on VBench.
Problem
The paper addresses the need for high-quality models that generate both images and videos within a unified framework.
Method
Goku combines rectified-flow Transformers, a 3D joint image-video VAE with shared latent representations, curated multimodal datasets, and infrastructure for large-scale training.
Results
Goku demonstrates strong performance across text-to-image and text-to-video benchmarks, including 0.76 on GenEval, 83.65 on DPG-Bench, and 84.85 on VBench.
Takeaways & Limitations
The results support Goku as a high-performing framework for joint image-and-video generation across research and commercial applications.
Abstract
from arXiv · showhide
This paper introduces Goku, a state-of-the-art family of joint image-and-video generation models leveraging rectified flow Transformers to achieve industry-leading performance. We detail the foundational elements enabling high-quality visual generation, including the data curation pipeline, model architecture design, flow formulation, and advanced infrastructure for efficient and robust large-scale training. The Goku models demonstrate superior performance in both qualitative and quantitative evaluations, setting new benchmarks across major tasks. Specifically, Goku achieves 0.76 on GenEval and 83.65 on DPG-Bench for text-to-image generation, and 84.85 on VBench for text-to-video tasks. We believe that this work provides valuable insights and practical advancements for the research community in developing joint image-and-video generation models.
1. Introduction
Goku is a family of rectified-flow Transformer models for joint image-and-video generation, supported by curated data, unified architecture, flow formulation, and scalable infrastructure. Across text-to-image and text-to-video benchmarks, it demonstrates strong performance, including a leading VBench score.
- Goku is a family of rectified-flow Transformer models designed for joint image and video generation.
- Its development centers on data curation, model architecture, flow formulation, and training-infrastructure optimization.
- The curated training data comprises approximately 36M video-text pairs and 160M image-text pairs.
- A 3D joint image-video VAE maps images and videos into a shared latent space, while full attention supports unified training across modalities.
- The training infrastructure uses parallelism, ByteCheckpoint, and MegaScale fault tolerance to support efficient and stable large-scale training.
- Goku-T2V achieves 84.85 on VBench, securing the top leaderboard position as of 2025-01-25.
2. Goku: Generative Flow Models for Visual Creation
Goku combines a joint image-video VAE, Transformer architecture, full attention, and rectified flow to model images and videos in a shared latent framework. Its multi-stage training strategy progressively develops text semantics, joint modality learning, and higher-resolution generation.
- Image-Video Joint VAE: Images and videos are encoded into a shared latent space by a 3D joint VAE, with images treated as the T=1 case.
- Image-Video Joint VAE: Goku uses 8 × 8 × 4 compression for video and 8 × 8 spatial compression for images.
- Goku Transformer: The Transformer combines self-attention, text cross-attention, feed-forward projection, and adaLN-Zero conditioning.
- Goku Transformer: Full attention models multi-dimensional interactions across image and video tokens, while Patch n’ Pack mixes varying lengths and aspect ratios in one minibatch.
- Rectified Flow: Rectified flow linearly interpolates between Gaussian noise and real data, then trains the model to predict velocity toward the data distribution.
- Multi-stage Training: The multi-stage strategy starts with text-to-image semantic pairing, adds joint image-video learning, and progressively increases resolution from 288 × 512 to 720 × 1280.
3. Infrastructure Optimization
Goku’s infrastructure targets long-context, large-scale training through three-dimensional parallelism, activation checkpointing, fault tolerance, and efficient checkpoint management. These mechanisms address memory, communication, cluster reliability, and state-recovery demands.
- Parallelism: Goku uses 3D parallelism across sequence, data, and model-parameter axes for sequences exceeding 220K tokens.
- Parallelism: Sequence parallelism reduces redundant computation and memory use by sharding inputs across the sequence dimension.
- Parallelism: FSDP partitions parameters, gradients, and optimizer states across data-parallel ranks while overlapping communication with computation.
- Activation Checkpointing: Fine-grained activation checkpointing balances computation and communication by maximizing their overlap.
- Fault Tolerance: MegaScale-derived fault tolerance provides diagnostics and recovery because larger GPU clusters have higher failure likelihood.
- Checkpointing: ByteCheckpoint enables parallel checkpoint I/O, distributed resharding, and switching across training scales and storage backends.
4. Data Curation Pipeline
Goku constructs large image-text and video-text datasets through collection, extraction, filtering, captioning, and semantic balancing. The pipeline combines quality thresholds, multimodal captioning, and category balancing to produce approximately 160M image-text and 36M video-text pairs.
- Data Overview: The curation pipeline has five stages: collection, video extraction and clipping, filtering, captioning, and data-distribution balancing.
- Data Overview: The final dataset contains approximately 160M image-text pairs and 36M video-text pairs from public and proprietary sources.
- Data Processing and Filtering: Video preprocessing standardizes encoding, duration, and frame rate before two-stage clipping creates consistent, meaningful clips.
- Data Processing and Filtering: Filtering uses visual aesthetics, OCR text coverage, motion, resolution, bitrate, and duration criteria to retain suitable video content.
- Captioning: InternVL2.0 generates image and keyframe captions, while Tarsier2 produces video-wide captions including camera-motion descriptions.
- Training Data Balancing: An internal classifier assigns videos to 9 primary classes and 86 subcategories for semantic distribution balancing.
5. Experiments
Goku is evaluated across text-to-image and text-to-video benchmarks, qualitative comparisons, image-to-video samples, and ablations of scaling and joint training. The results show strong alignment, video quality, and benefits from larger models and joint image-video training.
- Text-to-Image Results: 0.76 is Goku-T2I’s highest GenEval score with rewritten prompts, demonstrating strong alignment with detailed textual descriptions.The rewritten prompts preserve the original semantics while adding descriptive detail.
- Text-to-Image Results: Goku-T2I outperforms PixArt-α, SDXL, and DALL-E 2 on T2I-CompBench, with prompt rewriting improving all evaluated attributes.The benchmark evaluates alignment for color, shape, and texture attributes.
- Text-to-Image Results: 83.65 is Goku-T2I’s average DPG-Bench score, exceeding PixArt-α (71.11), DALL-E 3 (83.50), and EMU3 (80.60).DPG-Bench uses 1,000 detailed prompts to test dense prompt following.
- Text-to-Video Results: 217.24 is Goku’s FVD at 128×128 on zero-shot UCF-101, achieving state-of-the-art performance while consistently producing lower FVD and higher IS.Goku-2B generated 13,320 videos across three resolutions for evaluation.
- Text-to-Video Results: Goku-T2V achieves state-of-the-art overall performance on VBench across 16 video-generation dimensions.It is particularly strong in human action representation, dynamic degree, and multiple object generation, while remaining competitive in appearance style, quality, and semantic alignment.
- Image-to-Video Results: Goku-I2V maintains text alignment, visual quality, and temporal coherence when animating reference images after approximately 10k fine-tuning steps.The model is fine-tuned from T2V initialization using approximately 4.5M text-image-video triplets.
- Ablation Studies: Larger Goku-T2V models reduce distorted object structures, while joint image-video training produces more consistently photorealistic frames.The scaling comparison uses 2B and 8B models; the joint-training comparison starts from identical Goku-T2I 8B weights and equal fine-tuning steps.
6. Conclusion
Goku is presented as a joint image-and-video generation model designed for industry-standard performance. Its data curation and shared multimodal architecture support high-quality visual generation across modalities.
- Conclusion: Goku combines advanced data curation with an image-video joint VAE and rectified flow to integrate image and video generation.The framework uses a shared latent space and token interaction across modalities.
Appendix A. Benchmark Configurations
The appendix defines the benchmarks used to assess compositional image generation, dense prompt following, and video generation quality. Each benchmark specifies its evaluation focus and, for the image benchmarks, aggregation procedure.
- Image Benchmarks: T2I-CompBench evaluates compositional text-to-image alignment through color, shape, and texture binding scores.Each category uses 300 prompts and generates 10 images per prompt.
- Image Benchmarks: GenEval evaluates object-focused compositional properties including co-occurrence, position, count, and color.The final score averages tasks over 2,212 images from 553 prompts.
- Image Benchmarks: DPG-Bench uses longer, more detailed prompts to evaluate compositional generation in text-to-image models.The evaluation generates 4,260 images across 1,065 prompts and reports the average across tasks.
- Video Benchmarks: VBench evaluates video generative models through an Evaluation Dimension Suite containing precise dimensions, prompt suites, and generated videos.The structured suite supports detailed assessment across video-generation quality dimensions.
Appendix B.1. Goku-T2I Samples Visualization
The appendix presents randomly selected internet prompts and generated image samples illustrating Goku-T2I’s visual quality and text-image alignment. The examples emphasize detailed visual interpretation from complex natural-language descriptions.
- Samples Visualization: Goku-T2I generates images with rich details, including clear textures of leaves and berries.The samples are paired with randomly selected internet prompts.
Appendix B.2. Goku-T2V Samples Visualization
Goku-T2V generates four-second, 24-FPS, 720p videos in both landscape and portrait formats, including high-motion and realistic scenes.
- Goku-T2V produces four-second videos at 24 FPS and 720p resolution.
- The samples cover both landscape and portrait orientations.
- The examples include high-motion activities such as skiing and realistic environments such as forests.
Appendix B.3. Goku-T2V Comparisons with Prior Arts
Goku-T2V is compared with open-source and commercial text-to-video systems, demonstrating smooth motion and accurate low-angle camera views in the reported examples.
- Goku-T2V is compared against both open-source and commercial state-of-the-art video generation models.
- In one comparison, Goku generates smooth motion and correctly follows the specified low-angle shot.
- CogVideoX, Vidu, and Kling often produce incorrect objects or improper camera views in the cited comparison.
Appendix B.4. Goku-I2V Samples Visualization
Goku-I2V synthesizes coherent videos from diverse reference images while preserving motion and scene semantics, with examples spanning realistic, dynamic, and imaginative scenarios.
- Goku-I2V generates coherent and visually compelling videos from diverse reference images.
- The generated videos maintain consistency in motion and scene semantics.
- Water boxing examples show fluid splashes synchronized with the subject’s movements.
- A child riding a bike demonstrates smooth forward motion while preserving environmental consistency.
- Pirate ships battling on a swirling coffee cup illustrate photorealistic rendering and accurate motion trajectories in an imaginative scenario.
- The examples indicate generalization across varied inputs for video generation tasks requiring fidelity and adaptability.