PaperlayerView exact revision on arXiv

Source-linked AI summary

Lumina-T2X: Transforming Text into Any Modality, Resolution, and Duration via Flow-based Large Diffusion Transformers

Peng Gao, Le Zhuo, Dongyang Liu, Ruoyi Du, Xu Luo, Longtian Qiu, Yuhang Zhang, Chen Lin, Rongjie Huang, Shijie Geng, Renrui Zhang, Junlin Xi, Wenqi Shao, Zhengkai Jiang, Tianshuo Yang, Weicai Ye, He Tong, Jingwen He, Yu Qiao, Hongsheng Li

arXiv:2405.05945v3cs.CV

TL;DR

Diffusion models have shown strong image and video generation, but provide limited implementation guidance and are often designed for specific tasks. Lumina-T2X proposes a text-conditioned, flow-based transformer framework with unified tokenization for generation across modalities and spatial-temporal scales. The report describes broad generation capabilities and reports that Lumina-T2I uses 35% of PixArt-α’s training computational resources; multimodal models are currently trained separately.

  • Problem

    Diffusion models such as Sora and Stable Diffusion 3 [44] generate at arbitrary resolutions, but offer limited design guidance and are tailored to specific tasks, hindering replication and cross-modality adaptation.

    Source: §1 · p. 3

  • Method

    Lumina-T2X uses flow-based Flag-DiT models and unified one-dimensional token sequences with [nextline] and [nextframe] placeholders for text-conditioned generation across modalities and spatial-temporal scales.

    Source: §1 · p. 3, §2.2 · p. 7

  • Results

    Lumina-T2X reports generation across images, videos, multi-view 3D objects, and speech, while Lumina-T2I incurs only 35% of PixArt-α’s computational costs.

    Source: §1 · pp. 3–4

  • Takeaways & Limitations

    The framework supports resolution extrapolation, including Lumina-T2I generating 1792 × 1792 images after training at 1024 × 1024 pixels by adding [nextline] tokens.

    Source: §1 · p. 3

    Models for images, videos, multi-view 3D objects, and speech are trained separately; Lumina-T2V and Lumina-T2MV show preliminary temporal- or view-consistent generation but inferior sample quality to their counterparts.

    Source: §6 · p. 22

Abstract

from arXiv · show

Sora unveils the potential of scaling Diffusion Transformer for generating photorealistic images and videos at arbitrary resolutions, aspect ratios, and durations, yet it still lacks sufficient implementation details. In this technical report, we introduce the Lumina-T2X family - a series of Flow-based Large Diffusion Transformers (Flag-DiT) equipped with zero-initialized attention, as a unified framework designed to transform noise into images, videos, multi-view 3D objects, and audio clips conditioned on text instructions. By tokenizing the latent spatial-temporal space and incorporating learnable placeholders such as [nextline] and [nextframe] tokens, Lumina-T2X seamlessly unifies the representations of different modalities across various spatial-temporal resolutions. This unified approach enables training within a single framework for different modalities and allows for flexible generation of multimodal data at any resolution, aspect ratio, and length during inference. Advanced techniques like RoPE, RMSNorm, and flow matching enhance the stability, flexibility, and scalability of Flag-DiT, enabling models of Lumina-T2X to scale up to 7 billion parameters and extend the context window to 128K tokens. This is particularly beneficial for creating ultra-high-definition images with our Lumina-T2I model and long 720p videos with our Lumina-T2V model. Remarkably, Lumina-T2I, powered by a 5-billion-parameter Flag-DiT, requires only 35% of the training computational costs of a 600-million-parameter naive DiT. Our further comprehensive analysis underscores Lumina-T2X's preliminary capability in resolution extrapolation, high-resolution editing, generating consistent 3D views, and synthesizing videos with seamless transitions. We expect that the open-sourcing of Lumina-T2X will further foster creativity, transparency, and diversity in the generative AI community.

1 Introduction

Lumina-T2X addresses limited implementation guidance and task-specific designs in diffusion models with a text-conditioned framework for generating multiple modalities. Its Flag-DiT architecture and unified token representation support flexible generation, while the report also describes its training costs and planned open release.

1 Introduction

  • Sora and Stable Diffusion 3 generate at arbitrary resolutions, but limited implementation guidance and task-specific designs hinder replication and cross-modality adaptation.The report presents Lumina-T2X as a response to these gaps.Source: §1 · p. 3
  • Lumina-T2X uses Flag-DiT to transform text-conditioned noise into images, videos, multi-view 3D objects, and audio.The largest model combines a 7-billion-parameter Flag-DiT with the SPHINX text encoder, which handles 128K tokens.Source: §1 · p. 3

Flow-based Large Diffusion Transformers (Flag-DiT)

  • RoPE, RMSNorm, and KQ-Norm modifications support Flag-DiT’s stability and scalability, while flow matching builds continuous-time paths by linear interpolation between noise and data.The report says these modifications support scaling to 7 billion parameters and sequences of 128K tokens.Source: §1 · p. 3

Any Modalities, Resolution, and Duration within One Framework

  • Learnable [nextline] and [nextframe] tokens encode modalities and varying spatial-temporal dimensions as unified one-dimensional sequences.For example, Lumina-T2I trained at 1024 × 1024 pixels can generate images from 768 × 768 to 1792 × 1792 pixels by adding [nextline] tokens.Source: §1 · p. 3

Low Training Resources

  • Despite 8.3 times more parameters, Lumina-T2I incurs only 35% of PixArt-α’s computational costs.Table 1 compares Lumina-T2I with PixArt-α-0.6B; Lumina-T2I uses 14 million high-quality image-text pairs, while PixArt-α has 11 million additional high-quality natural image-text pairs.Source: Table 1 · p. 4, §1 · p. 4

1 Introduction

  • The report says it will release Lumina-T2X’s training and inference code and pretrained models to support future generative-AI research.Source: §1 · p. 4

2 Method

Lumina-T2X extends diffusion transformers into a flow-based framework for text-conditioned generation across modalities, while adapting Flag-DiT for stability, flexible sequence representations, and scale. The section describes its training pipeline and model variants, alongside training-free image-generation applications.

2.2 Architecture of Flag-DiT

  • Flag-DiT addresses DiT’s scaling and flexibility limits with stability modifications, relative positional encoding, and learnable tokens for variable-resolution and variable-duration sequences.RMSNorm and KQ-Norm support mixed-precision training; [nextline] and [nextframe] tokens help unify samples as one-dimensional sequences.Source: §2.2 · pp. 6–7
  • Lumina-T2X uses flow matching to interpolate between noise and data, learn a time-dependent velocity field, and generate samples by solving a Flow ODE.The framework adopts linear interpolation for simplicity; training regresses the target velocity with Conditional Flow Matching loss.Source: §2.2 · p. 7
  • RoPE, RMSNorm, and KQ-Norm support scaling Flag-DiT to 7B parameters and 128K-token sequences, including videos of up to 128 frames.The report also describes mixed-precision training and parallel strategies for scaling parameters and sequence length.Source: §2.2 · p. 8

2.3 The Overall Pipeline of Lumina-T2X

  • The pipeline encodes modalities as latent frame sequences, constructs tokenized noisy inputs and velocity targets, encodes prompts, and predicts with Flag-DiT.Images, video frames, and multiview images use SD 1.5 VAE encoding; speech spectrograms use identity mapping.Source: §2.3 · pp. 8–9

2.4 Lumina-T2X System

  • Lumina-T2X comprises image, video, multiview, and speech models, with configurations tailored to each modality and scenario.The report describes a 5B Flag-DiT image model and 2B Flag-DiT video and multiview models; progressive training moves from lower to higher resolutions and shorter to longer durations.Source: §2.4 · p. 9

2.5 Advanced Applications of Lumina-T2I

  • Lumina-T2I combines NTK-aware scaled RoPE, time shifting, and proportional attention for tuning-free generation at out-of-domain resolutions.The report says this approach can also improve generated-image quality.Source: §2.5 · p. 10
  • For style-consistent batches, Lumina-T2I shares reference-image keys and values before RoPE to avoid duplicated positional embeddings.The first image acts as the reference, supplying keys and values to other images in the batch.Source: §2.5 · p. 10
  • Lumina-T2I supports compositional generation by restricting each prompt’s cross-attention to its masked region, and editing by evolving noisy image latents under desired prompts.For style editing, channel-wise normalization addresses style leakage from the original image’s mean and variance.Source: §2.5 · pp. 10–11

3 Experiments

Experiments report ImageNet results for Flag-DiT and test Lumina-T2I and Lumina-T2V on resolution flexibility, editing, training behavior, and video generation. Lumina-T2MV and Lumina-T2Speech results are pointed to in appendices.

3.1 Validating Flag-DiT on ImageNet

  • Flag-DiT beats Large-DiT in FID and processes 40% more images per second than DiT, while training stably with mixed precision.The FID comparison also reports lower scores than SiT and suggests architecture modifications contribute to stability and performance.Source: §3.1 · pp. 11–12
  • Without CFG, Large-DiT-7B reduces FID from 8.60 to 6.09; at 512×512, Large-DiT-3B moves FID from 3.04 to 2.52 and IS from 240 to 303.Table 2 covers 256×256 and 512×512 label-conditional ImageNet generation.Source: §3.1 · p. 11, Table 2 · p. 12
  • LogNorm sampling accelerates convergence and improves FID; scaling Large-DiT from 600M to 7B speeds convergence, while scratch training beats ImageNet initialization on loss and speed.LogNorm emphasizes central time steps; the scaling comparison holds hyperparameters constant.Source: §3.1 · pp. 12–13

3.2 Results for Lumina-T2I

  • Extrapolation from 1K to 1.5K improves image quality and text alignment without training, but 2K generation can fail and domain mismatch may introduce artifacts.Lumina-T2I also generates at lower resolutions such as 512; Figure 7 shows resolution-extrapolation samples.Source: §3.2 · p. 15, Figure 7 · p. 15
  • Attention sharing yields style-consistent batches, while region-assigned prompts produce coherent, high-quality multi-concept images across resolutions.Figures 8 and 9 illustrate these style-consistent and compositional generation results.Source: §3.2 · p. 16, Figure 8 · p. 16, Figure 9 · p. 17
  • Training-free editing modifies global styles or adds subjects; a starting time near 0.2 balances instruction adherence with preserving structure, and normalization removes original style while retaining content.Figure 11 compares starting times and latent-feature normalization.Source: §3.2 · p. 17, Figure 10 · p. 18, Figure 11 · p. 18
  • The 5B Lumina-T2I model uses 35% of PixArt-α’s training cost—288 versus 828 A100 GPU days—despite having 8.3 times more parameters.The report attributes its simplified training pipeline to direct training on high-resolution synthetic domains.Source: §3.2 · pp. 17–18

3.3 Results for Lumina-T2V

  • Large-batch video training converges and 600M-to-5B scaling lowers loss at matched iterations; stage-two training produces temporally consistent, richer videos up to 128K tokens.First-stage short videos show scene transitions but are limited to 32K tokens; Figure 15 illustrates those samples.Source: §3.3 · p. 20, Figure 14 · p. 20, Figure 15 · p. 20

4 Related Work

The related-work discussion traces generative modeling from earlier approaches to diffusion models and scalable transformer backbones. It positions Lumina-T2X as an open implementation inspired by Sora’s image-and-video generation, whose detailed implementation had not been released.

4 Related Work

  • Figure 16 presents Lumina-T2V long-video samples described as temporally consistent and rich in scene dynamics.The caption identifies the samples as long-video generation examples.Source: Figure 16 · p. 21
  • Compared with GANs, VAEs [79] [147] [128] and flow-based models [40] [41] offer better training stability and interpretability but lower image quality; autoregressive models [149] [148] [34] perform exceptionally yet require more computation.Source: §4 · p. 21
  • Diffusion models became a de-facto approach to generative AI, while flow matching and vision-transformer backbones were presented as routes toward greater stability and scalability.The discussion also notes prior applications of diffusion to high-resolution, compositional, style-consistent, and controllable generation, as well as video, 3D-view, and audio generation.Source: §4 · p. 21
  • Sora demonstrated scaled Diffusion Transformers for joint image and video generation, but its detailed implementation had not been released, motivating Lumina-T2X.The report frames Lumina-T2X as an open-source effort to generate across modalities, resolutions, and durations.Source: §4 · p. 21

5 Conclusion

The conclusion presents Lumina-T2X as a text-conditioned framework spanning modalities and scales, built around Flag-DiT. It reports ImageNet validation and multimodal generation results, alongside training-free image-generation applications.

5 Conclusion

  • Lumina-T2X combines Flag-DiT with modality-specific tokens and flow matching to target scalable generation across text-conditioned modalities.The conclusion lists RoPE, RNSNorm, KQ-Norm, zero-initialized attention, [nextline] and [nextframe] tokens, and flow matching among its design choices.Source: §5 · p. 22
  • Flag-DiT shows superior performance and faster convergence on ImageNet, with results described as scaling with model parameters.The conclusion provides no benchmark values in this passage.Source: §5 · p. 22
  • The framework generates images and videos at arbitrary resolutions and supports training-free resolution extrapolation, high-resolution editing, and compositional generation.The conclusion also describes a unified recipe for text-to-image, video, multiview, and speech generation.Source: §5 · p. 22

6 Limitations and Future Work

The report identifies separate modality training and incomplete data coverage as limitations of the current Lumina-T2X models. It describes joint training and higher-quality real-world data as future directions.

6 Limitations and Future Work

  • Because modality data quantities are imbalanced and latent distributions differ, the current system trains image, video, multiview, and speech generation separately.The authors say video and multiview results are preliminary and lower quality than counterparts when not using pretrained 2D-image weights; they propose exploring joint training.Source: §6 · p. 22
  • Collected data inadequately covers the diverse data spectrum, limiting learning of real-world patterns and potentially reducing robustness in real-world scenarios.The authors identify higher-quality real-world data as a possible way to address this limitation.Source: §6 · p. 22

A.1 Unleashing the Full Potential of Lumina-T2X with Resolution Extrapolation

This section describes inference-time techniques for extending Lumina-T2I beyond its training resolution. Direct extrapolation is limited, while RoPE scaling, timestep adjustment, and attention rescaling are presented as ways to improve high-resolution generation.

A.1 Unleashing the Full Potential of Lumina-T2X with Resolution Extrapolation

  • Directly increasing sequence length and repositioning [nextline] permits inference above training resolution, but the model quickly collapses beyond a limited extrapolation range.The passage notes that position interpolation can improve context-length generalizability but still requires fine-tuning.Source: §A.1 · p. 34
  • ∼3× context length (1.8K images) is reached through NTK-aware scaled RoPE, which adjusts rotary frequencies for tuning-free generalization beyond the training context length.The method scales the rotary base so the lowest-frequency term matches position interpolation, transitioning from extrapolation at high frequencies to interpolation at low frequencies.Source: §A.1 · p. 34
  • Time shifting adjusts the flow-sampling schedule to match high-resolution and origin-resolution signal-to-noise ratios; the experiments adopt m = 6.0.The authors report that a larger shifting scale can improve generated-image quality; Figure 18 compares shifting values across resolutions.Source: §A.1 · pp. 34–35, Figure 18 · p. 36
  • Proportional attention scales terms in the attention softmax to restrict context aggregation and address repeated, incomplete, or disordered patterns during high-resolution extrapolation.The passage sets the scale using training and inference token counts, following an entropy-based setting in [75].Source: §A.1 · p. 35
  • Lumina-T2X explores tuning-free resolution extrapolation for flow-based transformers with RoPE, positioning token repositioning as an alternative to costly fine-tuning or architecture and pipeline modifications.The passage contrasts this approach with high-resolution adaptation and model-agnostic or CNN-specific extension methods.Source: §A.1 · p. 35

B Diverse Configurations

Lumina-T2X offers configurable model and text-encoder choices, with reported throughput advantages for Flag-DiT. It also supports multiple diffusion and VAE configurations while fixing several other settings.

B Diverse Configurations

  • Flag-DiT configurations span 600M to 7B parameters, and same-size Flag-DiT achieves around 50% faster throughput than the original DiT.Flag-DiT-5B reaches throughput comparable to DiT-XL at 1024 resolution.Source: B Diverse Configurations · p. 36
  • Text-encoder options include CLIP-L/G, LLaMa-7B, and SPHINX-13B, balancing GPU consumption with text-understanding capability.Source: B Diverse Configurations · p. 36
  • Lumina-T2X primarily uses flow matching but also supports DDPM; SD-1.5 VAE supports image and video features, whereas SDXL offers higher visual quality but not video generation.Source: B Diverse Configurations · p. 36
  • RoPE, image augmentation, and generation are fixed to 1-D, resize-then-crop, and parallel generation, respectively; the next version will explore these factors.Source: B Diverse Configurations · pp. 36–37

C.1 Influence of Time Shifting

Increasing the time-shifting factor improves visual quality across tested resolutions. Larger shifts make the main subject closer and brighter, with fewer artifacts.

C.1 Influence of Time Shifting

  • Larger time-shifting factors significantly improve visual quality across resolutions from 256 to 2048.Source: §C.1 · p. 37
  • Increasing the shifting factor from 1.0 to 10.0 makes the main subject closer and brighter, with fewer artifacts.Source: §C.1 · p. 37
  • The authors speculate that larger shifts spend more sampling steps early, helping compose global structure, while later steps can be skipped as denoising becomes easier.Source: §C.1 · p. 37

C.2 Results for Lumina-T2MV

Lumina-T2MV is trained on rendered Objaverse multi-view images and generates viewpoint-consistent views without camera-parameter input. Higher-resolution training improves fine-detail quality.

Dataset

  • Training uses about 40K LVIS Objaverse objects, each rendered as twelve views in a 3×4 grid at 256×256 or 512×512 pixels per view.The views use 30° elevation and sequential azimuth increments of 30°; the starting azimuth varies.Source: §C.2 · p. 37

Training

  • Training proceeds from twelve-view 256-pixel images to twelve-view 512-pixel images, using batch sizes 64 and 16 for 100K and 180K iterations, respectively.Both stages use 16 NVIDIA A100 GPUs with 80GB memory each, and provide merged views without camera parameters.Source: §C.2 · pp. 37–38

High-Resolution Multi-view Examples

  • Figure 20 presents qualitative results for high-resolution multi-view images generated by Lumina-T2MV.Source: Figure 20 · p. 38

Low-Resolution Multi-view Examples

  • Without camera parameters, Lumina-T2MV generates twelve 256×256 views with spatial consistency and reasonable viewpoint changes.Source: §C.2 · p. 38

High-Resolution Multi-view Examples

  • The 256×256 first-stage images limit fine-detail capture; training on 512×512 views substantially improves generated details while maintaining viewpoint accuracy.Source: §C.2 · p. 38

C.3 Results for Lumina-T2Speech

Lumina-T2Speech uses a Flag-DiT backbone with phoneme and pitch encoders, trained and evaluated on LJSpeech. Larger models improve reported objective and subjective speech results.

Basic Setups; Dataset

  • Lumina-T2Speech uses Flag-DiT with phoneme and pitch encoders; evaluation uses 13,100 LJSpeech clips at 22,050 Hz from one female speaker.The dataset totals about 24 hours of audio.Source: §C.3 · p. 38

Dataset; Training

  • Preprocessing converts text to phonemes, produces 80-bin mel-spectrograms, and extracts F0; HiFi-GAN V1 synthesizes waveforms from generated mel-spectrograms.Source: §C.3 · pp. 38–39

Training

  • Training runs for 200,000 steps on one NVIDIA 4090 GPU with batches of 64 sentences.The optimizer is Adam with β1 = 0.9, β2 = 0.98, and ϵ = 10^-9.Source: §C.3 · p. 39

Evaluation

  • Evaluation measures intelligibility with WER, speaker similarity with SIM, and human-rated quality with MOS and 95% confidence intervals.WER uses Whisper transcription; speaker similarity uses WavLM-TDNN.Source: §C.3 · p. 39

Results

  • Larger Flag-DiT models improve objective and subjective speech metrics, with the larger model also performing better in MOS testing.The authors state that model expansion enables finer-grained room-acoustic modeling; Table 5 compares Flag-DiT configurations.Source: §C.3 · p. 39, Table 5 · p. 39
Loading 2405.05945v3…