PaperlayerView exact revision on arXiv

Source-linked AI summary

Lumina-Next: Making Lumina-T2X Stronger and Faster with Next-DiT

Le Zhuo, Ruoyi Du, Han Xiao, Yangguang Li, Dongyang Liu, Rongjie Huang, Wenze Liu, Lirui Zhao, Fu-Yun Wang, Zhanyu Ma, Xu Luo, Zehan Wang, Kaipeng Zhang, Xiangyang Zhu, Si Liu, Xiangyu Yue, Dingning Liu, Wanli Ouyang, Ziwei Liu, Yu Qiao, Hongsheng Li, Peng Gao

arXiv:2406.18583v1cs.CVcs.LG

TL;DR

Lumina-T2X is limited by training instability, slow inference, and resolution-extrapolation artifacts. Lumina-Next addresses these issues with Next-DiT, improved RoPE, and faster flow-model sampling techniques, achieving stronger text-to-image generation and broad multimodal performance. The framework also supports zero-shot multilingual and resolution-extrapolated generation while extending to recognition, multi-view, audio, music, and point-cloud tasks.

  • Problem

    Lumina-T2X faces training instability, slow inference, and artifacts during resolution extrapolation, limiting its generative framework.

    Source: §1 · p. 2, §2.1 · p. 5, §2.2 · p. 7

  • Method

    Lumina-Next combines Next-DiT with 3D and Frequency- and Time-Aware Scaled RoPE, optimized flow-model time schedules, and Time-Aware Context Drop.

    Source: §1 · pp. 2–3, §2.3 · pp. 9–10

  • Results

    Lumina-Next delivers stronger and more efficient text-to-image generation, zero-shot multilingual and resolution-extrapolated generation, and strong performance across recognition, multi-view, audio, music, and point-cloud tasks.

    Source: §1 · p. 3, §3 · p. 10, §6 · p. 24

  • Takeaways & Limitations

    Lumina-Next functions as a unified, versatile, and efficient generative framework across image, video-related, audio, music, recognition, and point-cloud settings.

    Source: §3 · p. 10, §6 · p. 24

    NTK-Aware Scaled RoPE can still produce repetition during resolution extrapolation, indicating persistent spatial-awareness failures in some cases.

    Source: §2.2 · p. 7

Abstract

from arXiv · show

Lumina-T2X is a nascent family of Flow-based Large Diffusion Transformers that establishes a unified framework for transforming noise into various modalities, such as images and videos, conditioned on text instructions. Despite its promising capabilities, Lumina-T2X still encounters challenges including training instability, slow inference, and extrapolation artifacts. In this paper, we present Lumina-Next, an improved version of Lumina-T2X, showcasing stronger generation performance with increased training and inference efficiency. We begin with a comprehensive analysis of the Flag-DiT architecture and identify several suboptimal components, which we address by introducing the Next-DiT architecture with 3D RoPE and sandwich normalizations. To enable better resolution extrapolation, we thoroughly compare different context extrapolation methods applied to text-to-image generation with 3D RoPE, and propose Frequency- and Time-Aware Scaled RoPE tailored for diffusion transformers. Additionally, we introduced a sigmoid time discretization schedule to reduce sampling steps in solving the Flow ODE and the Context Drop method to merge redundant visual tokens for faster network evaluation, effectively boosting the overall sampling speed. Thanks to these improvements, Lumina-Next not only improves the quality and efficiency of basic text-to-image generation but also demonstrates superior resolution extrapolation capabilities and multilingual generation using decoder-based LLMs as the text encoder, all in a zero-shot manner. To further validate Lumina-Next as a versatile generative framework, we instantiate it on diverse tasks including visual recognition, multi-view, audio, music, and point cloud generation, showcasing strong performance across these domains. By releasing all codes and model weights, we aim to advance the development of next-generation generative AI capable of universal modeling.

1 Introduction

Lumina-T2X provides a flexible flow-based diffusion framework but faces weak image-text alignment, slow inference, and resolution-extrapolation artifacts. Lumina-Next addresses these limitations through Next-DiT, improved RoPE and sampling methods, and demonstrates stronger, more efficient multimodal generation.

Motivation

  • Lumina-T2X suffers from weak image-text alignment, slow inference, and extrapolation artifacts despite its flexible and training-efficient generative framework.These issues are attributed to inadequate training, insufficient training data, and an inappropriate context-extension strategy.Source: §1 · pp. 1–2

Architecture of Next-DiT

  • Next-DiT replaces 1D RoPE with 3D RoPE, removes redundant learned identifiers, adds sandwich normalization, and uses Grouped-Query Attention to improve stability and computational efficiency.The architecture controls activation magnitudes and is reported to converge faster than Flag-DiT and SiT on ImageNet-256.Source: §1 · pp. 2–3

Frequency- and Time-Aware Scaled RoPE

  • Frequency- and Time-Aware Scaled RoPE improves resolution extrapolation by reducing content repetition while targeting global consistency and local detail in high-resolution images.The proposal follows a comparison of Position Extrapolation, Position Interpolation, and NTK-Aware Scaled RoPE for text-to-image diffusion transformers.Source: §1 · p. 3

Optimized Time Schedule with Higher-Order Solvers

  • Optimized time schedules combined with higher-order ODE solvers enable high-quality text-to-image generation in only 5 to 10 sampling steps.The method is designed to reduce discretization error in flow models, where uniform Euler discretization is commonly used.Source: §1 · p. 3

Time-Aware Context Drop

  • Time-Aware Context Drop reduces redundant spatial context tokens by average-pooling keys and values, achieving a 2× inference-time speedup for 1K-resolution image generation while maintaining visual quality.When integrated with Flash Attention, it further improves inference speed for ultra-high-resolution image generation.Source: §1 · p. 3

Overall Framework

  • Lumina-Next’s 2B Next-DiT with Gemma-2B achieves better text-to-image generation than the 5B Flag-DiT with LLaMA-7B [74], while significantly reducing training and inference costs.It demonstrates strong zero-shot multilingual ability, tuning-free 2K generation, and extensions to high-resolution, multi-view, audio, music, and 16K point-cloud generation.Source: Figure 1 · p. 2, §1 · p. 3

2 Improving Lumina-T2X

Lumina-Next improves Lumina-T2X through Next-DiT architectural changes, specialized RoPE extrapolation, and faster flow-model sampling. These changes target training stability, resolution extrapolation, convergence, and inference efficiency.

2.1 Architecture of Next-DiT

  • 3D RoPE replaces flattened 1D positional encoding and learnable identifiers, providing unified spatial-temporal positions for images, videos, and multi-view inputs.For images with one frame, 3D RoPE naturally reduces to 2D RoPE and supports zero-shot resolution extrapolation.Source: §2.1 · pp. 4–5, Figure 2 · p. 4
  • Sandwich normalization controls activation magnitudes around attention and MLP layers, addressing instability and compounding sampling errors during scaling and resolution extrapolation.Grouped-query attention further reduces parameters, memory consumption, and inference cost; the 2B model uses 32 query heads and 8 shared key-value heads.Source: §2.1 · pp. 5–6, Figure 4 · p. 5
  • Next-DiT converges significantly faster than Flag-DiT and SiT on ImageNet-256, measured by FID and Inception Score.Long-skip connections were excluded because they caused training instability and significantly worsened performance.Source: Figure 6 · p. 5, §2.1 · p. 6

2.2 Improving NTK-Aware Scaled RoPE with Frequency- and Time-Awareness

  • Time-Aware Scaled RoPE shifts from position interpolation early in denoising toward NTK-Aware Scaled RoPE later, enabling globally consistent, high-detail resolution extrapolation.Frequency-Aware Scaled RoPE addresses repetitive content by treating RoPE dimensions according to wavelength rather than interpolating only the lowest-frequency component.Source: Figure 7 · p. 6, §2.2 · pp. 7–8

2.3 Improving Sampling Efficiency with Fewer and Faster Steps

  • Flow-model analysis finds larger discretization errors near pure noise and relatively larger errors near clean data, motivating nonuniform schedules instead of diffusion-oriented schedules.The adopted sigmoid schedule enlarges steps at the sampling boundaries, improves few-step image details, and uses µ = 0.6, α = 6, and β = 20 without extra computation.Source: Figure 5 · p. 5, Figure 8 · p. 7, §2.3 · pp. 8–9
  • Time-Aware Context Drop with 75% downsampling produces comparable or better 2K images while further improving inference speed, especially with Flash Attention.It averages spatial keys and values, applies stronger token dropping at t = 0, and gradually removes dropping by t = 1.Source: §2.3 · pp. 9–10, Figure 9 · p. 9

3 Lumina-Next by Composing Everything

Lumina-Next composes architectural, extrapolation, sampling, and text-encoding improvements into a stronger and faster generative framework. Experiments show flexible high-resolution generation, efficient few-step sampling, and zero-shot multilingual prompt understanding.

3 Lumina-Next by Composing Everything

  • Lumina-Next combines improved architecture, resolution extrapolation, sampling efficiency, and decoder-only LLM text encoding into a versatile framework across modalities.The framework is instantiated for text-to-image generation and further evaluated on recognition, multi-view images, music, audio, and point clouds.Source: §3 · p. 10

3.1 Flexible Resolution Extrapolation

  • At 2K resolution, Lumina-Next surpasses Lumina-T2X and avoids the repetitive or unreasonable content produced by competing methods beyond training resolution.Its direct extrapolation uses Next-DiT with frequency- and time-aware scaled RoPE rather than special multi-run inference procedures.Source: §3.1 · pp. 10–11
  • Lumina-Next’s 3D RoPE independently models temporal and spatial dimensions, while sandwich normalization reduces sensitivity to sequence length during resolution extrapolation.These architectural choices are identified as complements to Time-Aware Scaled RoPE in the reported 2K results.Source: §3.1 · p. 11
  • Even at 4× extrapolation and extreme 1024 × 4096 aspect ratios, Lumina-Next maintains content quality where MultiDiffusion and DemoFusion exhibit repetition or failures.The comparison covers panorama generation and arbitrary aspect-ratio extrapolation.Source: §3.1 · p. 11, Figure 11 · p. 12

3.2 Few-Step Text-to-Image Generation

  • With a midpoint second-order Runge-Kutta solver and sigmoid schedule, Lumina-Next achieves better conditional sampling than PixArt-α and SDXL at 10–20 NFEs.Higher-order solvers improve few-step quality but increase NFEs linearly, creating a speed-quality trade-off.Source: §3.2 · p. 12, Figure 12 · p. 12

3.3 Zero-Shot Multilingual Generation

  • Decoder-only LLM text encoders give Lumina models zero-shot multilingual prompt understanding despite training primarily on English-only corpora.Lumina-Next uses Gemma-2B to reduce GPU memory costs and increase training throughput.Source: §3.3 · pp. 12, 15
  • Lumina-Next interprets multilingual prompts with cultural nuances, handles emojis, and follows extremely long detailed prompts, including Chinese instructions.Figures 13, 14, and 16 provide the reported multilingual, emoji, and long-prompt examples.Source: Figure 13 · p. 13, Figure 14 · p. 13, Figure 16 · p. 14
  • Using Qwen-1.8B or InternLM-7B instead of Gemma-2B improves text-image alignment for diverse Chinese prompts and captures emotions in classic Chinese poems.The comparison links stronger multilingual abilities in the text encoder with more expressive generated results.Source: Figure 15 · p. 14, §3.3 · p. 15

4 Any Resolution Recognition with Lumina-Next

Lumina-Next adapts Next-DiT for image recognition across arbitrary resolutions and aspect ratios, using resolution-aware training and token handling. It achieves faster convergence and stronger ImageNet accuracy than DeiT-base while preserving flexible-resolution inference.

4.1 Pipelines

  • The recognition adaptation replaces time and label conditioning with a learnable attention gate, global average pooling, 2D RoPE, and sandwich normalization.Progressive fixed-resolution pre-training followed by arbitrary-resolution fine-tuning supports multi-resolution understanding.Source: §4.1 · p. 15
  • Dynamic partitioning preserves image resolution and aspect ratio, pads variable-length token sequences, and masks interactions between padding and regular image tokens.The scheme constrains patch count and aspect ratio while supporting efficient arbitrary-resolution fine-tuning.Source: §4.1 · p. 16, Figure 18 · p. 17
  • The recognition pipeline enables flexible inference on arbitrary resolutions and aspect ratios without fine-tuning on varied resolutions when using scaled RoPE and sandwich normalization.Dynamic padding preserves original image resolutions and aspect ratios during fine-tuning.Source: §4.1 · pp. 16–17

4.3 Experiments

  • 82.3% Top-1 accuracy at 300 epochs surpasses DeiT-base’s 81.6%, while 100 epochs reaches 81.6% and nearly matches DeiT-base after 300 epochs.Next-DiT uses a comparable parameter configuration on ImageNet-1K.Source: §4.3 · p. 17, Table 1 · p. 18
  • 84.2% accuracy is achieved by Next-DiT after arbitrary-resolution fine-tuning, while DeiT-base drops substantially under varied-resolution inference.The comparison uses DeiT-base pretrained at 224 × 224 with interpolated positional embeddings.Source: §4.3 · p. 18
  • Next-DiT generalizes better to larger image sizes than DeiT-base without fine-tuning, with fine-tuning especially improving performance at 1024 × 1024.The resolution comparison is shown in Figure 19.Source: §4.3 · p. 18, Figure 19 · p. 18

5 Generating Multi-Views, Music and Audio with Lumina-Next

Lumina-Next extends Next-DiT beyond image generation to multi-view images, audio, music, and point clouds. Across these applications, the framework uses modality-specific conditioning and resolution or density extrapolation strategies while reporting strong generation quality.

5.1 Image- and Text-conditional Multi-View Generation

  • Relative pose embeddings, optional VAE-encoded image conditioning, and progressive view-resolution training enable flexible-view multi-view generation.Training progresses from 256 × 256 with 4 views to 512 × 512 with 8 views, while inference can produce any number of views trained through random-view supervision.Source: §5.1 · pp. 18, 20, Table 2 · p. 20
  • MV-Next-DiT generates high-quality multi-view images with flexible azimuths, resolutions, conditioning modes, and view counts.It supports text-only or text-and-image conditioning and inference with 4, 6, or 8 views; comparisons with existing methods are reported in Table 3.Source: Figure 21 · p. 19, §5.1 · p. 20, Table 3 · p. 20

5.2 Text-Conditional Audio and Music Generation

  • The text-to-audio system achieves an FAD of 1.03, with MOS-Q 77.53 and MOS-F 76.52, indicating strong audio quality and text-audio faithfulness.The system combines an audio VAE, text encoder, flow transformer, and separately trained neural vocoder.Source: §5.2 · pp. 20, 22, Figure 22 · p. 21
  • Ablations show that dual text encoders, Audiocaps fine-tuning, and Next-DiT architecture improvements materially affect text-audio alignment and objective metrics.Dropping the CLAP encoder degrades alignment faithfulness, while omitting Audiocaps fine-tuning causes a significant objective-evaluation drop.Source: §5.2 · p. 22, Table 5 · p. 22
  • Text-to-music generation consistently surpasses diffusion-based methods and language models on FAD and KL metrics while showing strong text-music alignment.The comparison includes MusicGen, MusicLM, Mousai, Riffusion, MusicLDM, and AudioLDM 2.Source: §5.2 · p. 23, Table 6 · p. 23

5.3 Label- and Text-Conditional Point Cloud Generation

  • The point-cloud generator trains on 256 points but supports inference at densities up to 8192 points using Time-Aware Scaled Fourier Features.The method combines Time-Aware Scaled RoPE with Fourier Features so early denoising emphasizes global structure and later steps add shape details.Source: §5.3 · p. 23, Figure 23 · p. 24
  • The point-cloud generator achieves satisfactory performance against existing models, although generated density decreases as the sampled point count increases.The authors attribute this observed decrease possibly to smoother self-attention matrices at higher point counts.Source: Table 7 · p. 23, §5.3 · p. 24

6 Conclusion

Lumina-Next addresses Lumina-T2X limitations through Next-DiT, context extrapolation, and fast sampling techniques. It reports strong zero-shot image generation and broad extensions to recognition, multi-view, audio, music, and point-cloud generation.

6 Conclusion

  • Lumina-Next combines improved architecture, context extrapolation, and fast sampling to address Lumina-T2X’s limitations.The conclusion identifies these changes as supporting stronger generation capabilities and higher-resolution image generation.Source: §6 · p. 24
  • The framework reports high-quality images larger than its training resolution and multilingual text-to-image generation in a training-free manner.The conclusion states that these results outperform previous methods including SDXL and PixArt-α by a considerable margin.Source: §6 · p. 24
  • With minimal modifications, Lumina-Next extends to visual recognition, multiviews, audio, music, and point-cloud generation with superior results across these applications.The conclusion presents these modalities as evidence of the framework’s versatility.Source: §6 · p. 24
Loading 2406.18583v1…