PaperlayerView exact revision on arXiv

Source-linked AI summary

MiniMax-01: Scaling Foundation Models with Lightning Attention

MiniMax, Aonian Li, Bangwei Gong, Bo Yang, Boji Shan, Chang Liu, Cheng Zhu, Chunhao Zhang, Congchao Guo, Da Chen, Dong Li, Enwei Jiao, Gengxin Li, Guojun Zhang, Haohai Sun, Houze Dong, Jiadai Zhu, Jiaqi Zhuang, Jiayuan Song, Jin Zhu, Jingtao Han, Jingyang Li, Junbin Xie, Junhao Xu, Junjie Yan, Kaishun Zhang, Kecheng Xiao, Kexi Kang, Le Han, Leyang Wang, Lianfei Yu, Liheng Feng, Lin Zheng, Linbo Chai, Long Xing, Meizhi Ju, Mingyuan Chi, Mozhi Zhang, Peikai Huang, Pengcheng Niu, Pengfei Li, Pengyu Zhao, Qi Yang, Qidi Xu, Qiexiang Wang, Qin Wang, Qiuhui Li, Ruitao Leng, Shengmin Shi, Shuqi Yu, Sichen Li, Songquan Zhu, Tao Huang, Tianrun Liang, Weigao Sun, Weixuan Sun, Weiyu Cheng, Wenkai Li, Xiangjun Song, Xiao Su, Xiaodong Han, Xinjie Zhang, Xinzhu Hou, Xu Min, Xun Zou, Xuyang Shen, Yan Gong, Yingjie Zhu, Yipeng Zhou, Yiran Zhong, Yongyi Hu, Yuanxiang Fan, Yue Yu, Yufeng Yang, Yuhao Li, Yunan Huang, Yunji Li, Yunpeng Huang, Yunzhi Xu, Yuxin Mao, Zehan Li, Zekang Li, Zewei Tao, Zewen Ying, Zhaoyang Cong, Zhen Qin, Zhenhua Fan, Zhihang Yu, Zhuo Jiang, Zijia Wu

arXiv:2501.08313v1cs.CLcs.CV

TL;DR

Most models’ 32K–256K-token context windows fall short of practical needs, motivating a model that matches leading commercial performance while extending context by an order of magnitude. MiniMax-01 combines lightning attention with optimized hardware utilization and training recipes, and its models achieve top-tier benchmark performance with context windows up to 4 million tokens.

  • Problem

    Most models have 32K–256K-token context windows, while practical applications require substantially longer contexts; the report therefore targets leading-model performance with an order-of-magnitude longer context window.

  • Method

    The models integrate linear attention with optimized hardware utilization, engineering optimizations, and training recipes, including hybrid lightning attention and large-scale MoE computation strategies.

  • Results

    MiniMax-Text-01 and MiniMax-VL-01 demonstrate top-tier performance across standard and in-house benchmarks, particularly on long-context processing, with context windows up to 4 million tokens.

  • Takeaways & Limitations

    The results support scaling linear attention to large models and point toward more efficient systems with even longer context windows; the models and API are publicly available.

  • Takeaways & Limitations

    Long-context retrieval and reasoning evaluations remain limited in realistic applications, and the model retains a 1/8 component with vanilla softmax attention.

Abstract

from arXiv · show

We introduce MiniMax-01 series, including MiniMax-Text-01 and MiniMax-VL-01, which are comparable to top-tier models while offering superior capabilities in processing longer contexts. The core lies in lightning attention and its efficient scaling. To maximize computational capacity, we integrate it with Mixture of Experts (MoE), creating a model with 32 experts and 456 billion total parameters, of which 45.9 billion are activated for each token. We develop an optimized parallel strategy and highly efficient computation-communication overlap techniques for MoE and lightning attention. This approach enables us to conduct efficient training and inference on models with hundreds of billions of parameters across contexts spanning millions of tokens. The context window of MiniMax-Text-01 can reach up to 1 million tokens during training and extrapolate to 4 million tokens during inference at an affordable cost. Our vision-language model, MiniMax-VL-01 is built through continued training with 512 billion vision-language tokens. Experiments on both standard and in-house benchmarks show that our models match the performance of state-of-the-art models like GPT-4o and Claude-3.5-Sonnet while offering 20-32 times longer context window. We publicly release MiniMax-01 at https://github.com/MiniMax-AI.

1. Introduction

MiniMax-01 targets top-tier model performance with context windows far beyond the 32K–256K range common today. Its approach combines lightning attention, MoE scaling, optimized infrastructure, and staged training for million-token processing.

  • 32K–256K tokens is the current context-window range for most models, but practical uses such as books, software projects, and many-shot prompting demand more.
  • Transformer attention’s quadratic complexity makes further context extension increasingly difficult as computational demands outpace hardware capabilities.
  • MiniMax-01 uses a hybrid architecture centered on lightning attention, an I/O-aware linear-attention variant designed for efficient long-context processing.
  • 456 billion total parameters and 45.9 billion activated parameters per token balance model capacity with the constraint of processing more than 1 million tokens on one 8-GPU machine.
  • The redesigned training and inference framework supports lightning attention, softmax attention, MoE, hundreds of billions of parameters, and million-token contexts.
  • CUDA kernels for lightning-attention inference achieve over 75% Model Flops Utilization end-to-end on Nvidia H20 hardware.
  • MiniMax-Text-01 uses staged training to extend its context window to 1 million tokens, while MiniMax-VL-01 receives additional training with 512 billion vision-language tokens.
  • Both models reach performance comparable to closed-source top-tier systems on academic benchmarks, perform significantly better beyond 200k-token contexts, and rank top-tier on in-house evaluations.

2. Model Architecture

MiniMax-Text-01 combines lightning attention, softmax attention, and Mixture of Experts to improve efficiency and long-context capability. Experiments indicate that the hybrid-lightning design addresses pure linear attention’s retrieval weakness while achieving strong downstream performance and efficient scaling.

  • Architecture: MiniMax-Text-01 uses lightning attention and softmax attention in a structured hybrid architecture, with MoE feature mixers containing multiple feed-forward experts.A softmax-attention transformer block follows every seven linear-attention transormer blocks, for 80 layers total.
  • Architecture: 456 billion total parameters and 45.9 billion activated parameters per token define the selected model configuration.The architecture uses 32 experts per layer with top-2 routing.
  • Lightning Attention: O(N) time and space complexity is achieved by linear attention, while Lightning Attention’s tiling divides computation into intra-block and inter-block operations to avoid cumsum bottlenecks.The method uses left-product attention within blocks and right-product attention across blocks.
  • Efficiency: Lightning Attention maintains constant training speed as sequence length increases and is the only evaluated linear model to outperform FlashAttention2.Speed was measured using tokens processed per GPU per second on 3-billion-parameter models until single-node H800 out-of-memory limits.
  • Empirical Findings: The hybrid-lightning model outperforms other evaluated models across all reported metrics, particularly on the NIAH benchmark, while larger models generally perform better.The comparisons include hybrid variants and sliding-window baselines under equivalent-speed conditions.
  • Empirical Findings: The hybrid-lightning model matches or surpasses softmax attention on retrieval and extrapolation tasks, whereas pure linear attention falls short on retrieval.The authors connect this difference to lightning attention’s larger recurrent-state capacity, O(d^2/h), compared with softmax attention’s O(d).

3. Computation Optimization

The paper redesigns computation and communication for large-scale MoE and long-context lightning attention, addressing memory, communication, dependency, and variable-length batching challenges. These optimizations reduce MoE communication overhead, parallelize LASP computation, and improve inference efficiency.

  • 3.1. MoE Optimization: ETP manages expert weight partitioning while EDP encapsulates data parallelism for identical experts, enabling independent MoE parallel-strategy configuration.The design addresses low computational intensity with tensor-parallel expert partitioning and excessive parameter counts without it.
  • 3.1. MoE Optimization: 50% reduction in pure MoE communication overhead improves training efficiency after balancing expert storage and computational intensity.The optimization targets the trade-off among GPU memory, computation, and all-to-all communication.
  • 3.2.1. Varlen Ring Attention: Varlen Ring Attention applies ring attention to the entire packed sequence, avoiding excessive padding for batches containing unequal-length samples.The redesign avoids requiring each sequence length to be a multiple of 2 × sizeCP.
  • 3.2.2. Improved Linear Attention Sequence Parallelism: LASP+ removes computation dependencies between nodes, converting serial lightning-attention computation into parallel execution.The approach refines the computation-communication workflow to better exploit GPU parallelism.
  • 3.2.2. Improved Linear Attention Sequence Parallelism: Up to 1/N_pcn of the original LASP computation speed is achieved, with minimal AllGather overhead.N_pcn denotes the number of parallel computing nodes.
  • 3.3. Lightning Attention Inference Optimization: MFU exceeds 75% on H20 inference, while lightning attention contributes less than 12% of latency at sequence length 1,024,000 tokens.Under the same condition, softmax attention constitutes 95% of latency.

4. Pre-Training

MiniMax-Text-01’s pre-training combines curated data, statistical data experiments, repetition-aware mixture design, and staged context extension to 1 million tokens. The procedure also supports extrapolation to 4 million tokens in retrieval testing while preserving shorter-context performance.

  • Data construction: The pre-training methodology covers corpus construction, data experimentation, and in-depth analysis of training strategies for MiniMax-Text-01.The corpus emphasizes data quality, standardized formatting, and mixing strategies.
  • Data construction: The corpus combines academic literature, books, web content, and programming code, with quality improved through filtering, cleaning, deduplication, and reward-based assessment.A previous-generation MoE model provides reward labels for granular document-quality evaluation.
  • Data construction: 200K tokens is the resulting vocabulary size after byte-level BPE tokenization and multilingual up-sampling for improved compression efficiency.The tokenizer incorporates a pre-tokenizer methodology.
  • Data experiments: The data-experiment framework compares baseline and alternative data configurations as statistical hypothesis tests over weighted evaluation-metric distributions.The alternative hypothesis tests whether a new corpus improves mean performance relative to the baseline.
  • Data experiments: Low-quality data lose substantial performance after more than two epochs, whereas high-quality data remain effective for up to four epochs.The repetition-aware framework improves alignment with full-training results while reducing computational requirements.
  • Long-context extension: 1 million tokens is the expanded training context length, and the model processes up to 4 million tokens in vanilla Needle-In-A-Haystack retrieval despite that training limit.A three-stage procedure upsamples long-context data across length ranges while preserving short-context evaluation performance.

5. Post-training

Post-training evaluates MiniMax-Text-01 across academic, long-context, in-context learning, assistant, safety, and search-enabled scenarios. The model shows strong benchmark and user-oriented performance, while multi-level instruction following remains a documented limitation.

  • Evaluation overview: MiniMax-Text-01 achieves top performance across long-context retrieval, understanding, long in-context learning, knowledge requests, and real-user assistant scenarios.It also performs well on mathematics, reasoning, and coding tasks.
  • Core benchmarks: MiniMax-Text-01 surpasses all models on C-SimpleQA, ranks top-3 on MMLU, IFEval, and Arena-Hard, and exceeds GPT-4o, Claude-3.5-Sonnet, and Llama-3.1-405B on MATH pass@1.It also achieves comparable performance to instructed Qwen2.5-72B on HumanEval and scores 54.4 on GPQA Diamond.
  • Long-context evaluation: MiniMax-Text-01 shows less performance degradation than GPT, Claude, and Gemini variants as input lengths increase in English and Chinese retrieval evaluations.The comparison concerns robustness across a wide range of sequence lengths.
  • Long-context evaluation: At 128k, MiniMax-Text-01 surpasses all benchmark models on Ruler’s long-context reasoning tasks, with a particularly pronounced lead at 1M.It achieves state-of-the-art results on LongBench-V2 with chain-of-thought and remains effective without chain-of-thought.
  • Long in-context learning: The model surpasses all compared models on eng →kalam ChrF despite little Kalamang-related pre-training data, while remaining comparable on kalam →eng BLEURT.The no-context eng →kalam score is lowest, consistent with the limited relevant pre-training content.
  • Limitations: Insufficient training data for specific instruction types primarily limits performance on complex multilevel instruction-following requests.The authors plan to expand targeted high-quality training data to address these gaps.
  • User-oriented evaluation: Search-tool use improves out-of-domain Hailuo AI performance from 58% to 71.5% in human evaluations.Search scenarios cover approximately 30 ∼40% of user queries and target precision-demanding, domain-specific, or time-sensitive requests.

6. Vision-language Model

MiniMax-VL-01 extends MiniMax-Text-01 with visual understanding through a ViT, image adapter, and staged multimodal training. It performs competitively across standard and comprehensive benchmarks, but remains weaker on advanced mathematics and some long-context and planning tasks.

  • Architecture and training: MiniMax-VL-01 adds an image encoder and image adapter to MiniMax-Text-01, using proprietary data and multi-stage training for visual understanding.The encoder and adapter receive visual pre-training before full-pipeline fine-tuning.
  • Vision data: 694 million unique image-caption pairs train the vision encoder, including refined captions for 180 million images.Raw and refined captions are sampled with equal probability during training.
  • Vision-language data: The instruction dataset covers visual tasks including text extraction and object localization, with balanced sampling across QA prompt types.A separate analysis identifies roughly 50,000 unique tags and groups the most frequent 2,817 tags into 14 categories.
  • Architecture and training: The architecture uses a 303-million-parameter ViT, a randomly initialized two-layer MLP projector, and MiniMax-Text-01 as its language backbone.It follows the ViT-MLP-LLM paradigm.
  • Architecture and training: Directly using raw high-dimensional visual features avoids pooling-based information loss and improves adaptability to multi-scale inputs.Dynamic resolution resizes images from 336×336 to 2016×2016, partitions them into patches, and concatenates patch and thumbnail features.
  • Vision encoder: 80.55% zero-shot classification accuracy is achieved by ViT-L/14 at 336 × 336 resolution on ImageNet-1K.The encoder is trained at 224 × 224 and then fine-tuned at 336 × 336.
  • Benchmark results: MiniMax-VL-01 performs on par with GPT-4o on standard vision-language tasks, particularly visual question answering, but struggles on OlympiadBench’s advanced mathematical reasoning.On MMLongBench-Doc, it outperforms most counterparts except GPT-4o-11-20, while showing gaps on single-page and cross-page subsets.
  • Benchmark results: MiniMax-VL-01 surpasses existing open-source vision-language models on MEGA-Bench overall, while facing challenges in planning and metric-assessment tasks.The benchmark also reports strengths in knowledge and coding subtasks.

7. Conclusion and Future work

MiniMax-Text-01 and MiniMax-VL-01 achieve top-tier benchmark performance while supporting context windows up to 4 million tokens. The authors identify long-context evaluation, residual softmax attention, and advanced programming as areas for future work.

  • MiniMax-Text-01 and MiniMax-VL-01 demonstrate top-tier performance across standard benchmarks, particularly in long-context processing up to 4 million tokens.
  • The authors argue that state-of-the-art language models need not rely exclusively on traditional attention mechanisms.
  • The models’ reported efficiency and scalability are attributed to integrating linear attention with optimized hardware utilization and carefully designed training recipes.
  • Long-context retrieval and reasoning evaluations remain limited in realistic document-analysis settings.
  • The architecture retains one-eighth vanilla softmax attention, while advanced programming performance is constrained by limited pre-training coding data.

B.1. Learning A ‘New’ Language From Long Context

The long-context language example supplies grammar, vocabulary, and parallel sentences before requesting a Kalamang-to-English translation. MiniMax-Text-01 produces a semantically appropriate translation in this demonstration.

  • The prompt combines an instruction, grammar book, word list, and parallel sentences to support Kalamang-to-English translation.
  • The demonstration includes hundreds of parallel-sentence examples as translation guidance.
  • The reference translation states that Christians pray at church on Sunday.
  • Alternative responses include a semantically related worship formulation and an incorrect statement about being Christian on Sunday.
  • MiniMax-Text-01 can learn from the supplied grammar book, word list, and parallel sentences, whereas an under-performing model produces semantic errors.

B.2. Memorizing Long History Dialogues

The long-history dialogue benchmark tests whether a model can retrieve a specific earlier interaction from thousands of historical exchanges. The example shows successful retrieval of the first penguin poem, and the benchmark data are slated for release.

  • The benchmark asks the model to retrieve one historical user interaction from up to 1889 English or 2053 Chinese history interactions.
  • The prompt structure places irrelevant Haysack dialogues around a target request to write a penguin poem.
  • The target request is followed by a generated penguin poem beginning with Antarctic iceberg imagery.
  • When asked to repeat the first poem, the model returns the earlier poem verbatim in the example.
  • The authors state that the benchmark’s data and code will be released to the community.

B.3. Summarizing Long Papers with Figures

The long-paper figure task evaluates whether the system can summarize a supplied PDF and cite key figures. The provided material also describes Stream Aligner as an efficient sentence-level alignment method with measurable gains and stated limitations.

  • The long-paper task asks the system to summarize a PDF and include or analyze its key figures with citations.
  • Stream Aligner combines inference-time efficiency with additional-model performance to perform dynamic sentence-level correction.
  • Stream Aligner outperforms SFT and DPO on accuracy improvements and matches Aligner-70B using only 2B parameters.
  • Stream Aligner-2B applied to Llama2-70B-chat achieved a 41.2% increase in helpfulness and a 36.0% increase in harmlessness.
  • Stream Aligner-8B applied to Llama3-70B-Instruct achieved a 3.5% improvement in math ability.
  • The approach adds inference-time computational overhead, requires high-quality training data, and may be limited on extremely difficult out-of-distribution inputs.

B.4. Translating Long Papers

This material discusses brain-inspired AI and AGI, including multimodal integration, parallel processing, adaptability, and safety concerns. It also reviews neural and neuromorphic approaches related to these themes.

  • Multimodal systems integrate information from vision, audition, and touch to form a coherent perception of the world.
  • The discussion highlights parallel processing as important for handling multiple information streams simultaneously.
  • Related approaches: Related work includes CNNs, vision transformers, spiking neural networks, and neuromorphic chips such as IBM TrueNorth and Intel Loihi.
  • Brain-inspired AI is presented as drawing on human intelligence while potentially benefiting human intelligence in return.
  • Safety and ethical alignment are identified as major concerns for AGI adoption, including prevention of intentional and unintentional harm.

B.5. Lyrics Writing

The examples use lyrical and narrative writing to depict a traveler discovering a lost city, an adventurer uncovering a hidden world, and the tension between change and memory. Human evaluations emphasize imagery, emotional resonance, world-building, and narrative structure.

  • Lyrics Writing: The ballad depicts a traveler entering a lost city whose ruins, history, and mysteries create an atmospheric setting.
  • Lyrics Writing: The lyric evaluation praises vivid imagery, emotional depth, narrative structure, and chorus repetition as sources of engagement.
  • Story Writing: The story follows Elara from a village into Aetheria after she discovers a cryptic book containing maps to the hidden world.
  • Story Writing: The story evaluation highlights imaginative world-building, Elara’s development from curiosity to heroism, balanced pacing, and tension.
  • Verse Writing: The verse presents change as something to embrace while preserving memories and meaningful past experiences.

B.9. Search

The search example illustrates that tool-assisted analysis can correct a misleading premise about the 2022 major. The evaluator states that FaZe Clan, not NAVI, won the series 2-0.

  • The analysis describes search-enabled Hailuo AI as identifying the error in the user request.
  • The incorrect no-search response instead claims that NAVI won 2-0 and attributes the supposed victory to several performance factors.
  • The reported final was closely contested, with FaZe winning the first map 19-16 in overtime despite NAVI’s strong performance.
  • FaZe’s victory was described as its first Major win and the first by a completely international roster.
  • FaZe Clan defeated NAVI 2-0 in the 2022 major final, contradicting the user’s premise that NAVI won.

C. Vision In-house User Experience Benchmark Examples

The benchmark example asks which bodies have the most satellites and whether satellites are exclusive to planets. The supplied answer identifies Saturn and Jupiter separately and gives non-planet examples.

  • Saturn has the most satellites in the illustration, with 9 listed satellites.
  • Jupiter has the most large satellites, with 4: Io, Europa, Ganymede, and Callisto.
  • Satellites are not exclusive to planets; the illustration also includes asteroids and dwarf planets with satellites.

C.2. Daily Life

The daily-life scenarios cover highway-exit navigation and identifying outfits matched with long skirts. The responses provide concrete action guidance and row-wise counts.

  • Navigation: After 38 meters, enter the right exit ramp while preparing to leave the highway.The guidance also recommends maintaining the current lane, slowing appropriately, and signaling before entering the ramp.
  • Navigation: The navigation response emphasizes safety and focus during highway lane changes and ramp entry.
  • Outfit counting: The first row contains 3 outfits matched with long skirts, specifically columns 2, 5, and 6.
  • Outfit counting: The second row contains 2 outfits matched with long skirts, in columns 2 and 6.
  • Outfit counting: The third row contains 1 outfit matched with a long skirt, yielding row totals of 3, 2, and 1.

C.3. Digital Scenarios

The digital-scenario examples address process optimization and schedule prioritization. They identify the longest invoice-processing step and infer priorities from email content, timing, and importance.

  • Process optimization: 53.6 seconds is the average time for the longest step, entering invoice details.
  • Process optimization: Invoice-detail entry can be optimized through automated data entry, a simpler interface, batch processing, training, parallel work, and feedback.Suggested mechanisms include OCR, auto-fill, batch imports, operational guidance, task splitting, and issue reporting.
  • Schedule prioritization: The prioritization reasoning uses email sending time and content, the nature of personal arrangements, and explicit email priority.
  • Schedule prioritization: Katy’s potential priorities include arranging an apartment viewing, replying to birthday planning, confirming flight booking, and updating the team photo.

C.4. Special Imaging

The special-imaging example compares satellite images of the same area in August and September 2023. It shows reduced vegetation cover in September while leaving the specific causes for further analysis.

  • Image comparison: The left and right images are satellite views of the same area from August 2023 and September 2023, respectively.
  • Observed change: By September, many green areas had turned brown or become bare land, indicating a significant reduction or degradation of vegetation cover.
  • Interpretation: Possible causes include climate change, fires, droughts, agricultural development, or deforestation, but specific attribution requires additional data and background information.
  • Interpretation: The month-to-month comparison prompts attention to the causes of vegetation change and measures for ecological protection and restoration.

D. Evaluation Details for Selected Benchmarks

The selected-benchmark details describe multimodal evaluation configurations and scoring procedures. They cover MEGA-Bench inputs and model-specific image limits, while noting a fallback judge for some evaluations.

  • Evaluation configuration: For MMLongBench-Doc, extracted PDF images are concatenated into 5 images for open-source models and 10 for Claude-3.5-Sonnet-1022.Other commercial models and MiniMax-Text-01 use the default maximum image-page configuration described in the evaluation details.
  • MEGA-Bench: MEGA-Bench spans 7 input formats, 6 output formats, 10 skill types, and visual inputs including images and videos.Video inputs are sliced into multiple frames, and requests may contain multiple images alongside task descriptions and demonstrations.
  • MEGA-Bench: Table 14 presents the configuration of different models for MEGA-Bench.
Loading 2501.08313v1…