PaperlayerView exact revision on arXiv

Source-linked AI summary

Hunyuan-Large: An Open-Source MoE Model with 52 Billion Activated Parameters by Tencent

Xingwu Sun, Yanfeng Chen, Yiqing Huang, Ruobing Xie, Jiaqi Zhu, Kai Zhang, Shuaipeng Li, Zhen Yang, Jonny Han, Xiaobo Shu, Jiahao Bu, Zhongzhi Chen, Xuemeng Huang, Fengzong Lian, Saiyong Yang, Jianfeng Yan, Yuyuan Zeng, Xiaoqin Ren, Chao Yu, Lulu Wu, Yue Mao, Jun Xia, Tao Yang, Suncong Zheng, Kan Wu, Dian Jiao, Jinbao Xue, Xipeng Zhang, Decheng Wu, Kai Liu, Dengpeng Wu, Guanghui Xu, Shaohua Chen, Shuang Chen, Xiao Feng, Yigeng Hong, Junqiang Zheng, Chengcheng Xu, Zongwei Li, Xiong Kuang, Jianglu Hu, Yiqi Chen, Yuchi Deng, Guiyang Li, Ao Liu, Chenchen Zhang, Shihui Hu, Zilong Zhao, Zifan Wu, Yao Ding, Weichao Wang, Han Liu, Roberts Wang, Hao Fei, Peijie Yu, Ze Zhao, Xun Cao, Hai Wang, Fusheng Xiang, Mengyuan Huang, Zhiyuan Xiong, Bin Hu, Xuebin Hou, Lei Jiang, Jianqiang Ma, Jiajia Wu, Yaping Deng, Yi Shen, Qian Wang, Weijie Liu, Jie Liu, Meng Chen, Liang Dong, Weiwen Jia, Hu Chen, Feifei Liu, Rui Yuan, Huilin Xu, Zhenxiang Yan, Tengfei Cao, Zhichao Hu, Xinhua Feng, Dong Du, Tinghao Yu, Yangyu Tao, Feng Zhang, Jianchen Zhu, Chengzhong Xu, Xirui Li, Chong Zha, Wen Ouyang, Yinben Xia, Xiang Li, Zekun He, Rongpeng Chen, Jiawei Song, Ruibin Chen, Fan Jiang, Chongqing Zhao, Bo Wang, Hao Gong, Rong Gan, Winston Hu, Zhanhui Kang, Yong Yang, Yuhong Liu, Di Wang, Jie Jiang

arXiv:2411.02265v3cs.CLcs.AI

TL;DR

Most open-source models are dense, while relatively few use MoE at a large parameter scale. Hunyuan-Large combines a large Transformer-based MoE model with synthetic data and model and training strategies, and reports strong benchmark performance, including an advantage over LLama3.1-405B on MMLU.

  • Problem

    Most open-source models are dense, and the few MoE models have relatively small parameter scales, leaving a gap for large open-source MoE models.

    Source: §1 · pp. 1–2

  • Method

    Hunyuan-Large combines large-scale synthetic data with KV-cache compression, mixed expert routing, expert-specific learning-rate scaling, and MoE scaling-law exploration.

    Source: §1 · p. 2

  • Results

    Hunyuan-Large achieves the overall best performance among similar-scale open-source LLMs across diverse benchmarks, and its pre-trained model exceeds LLama3.1-405B on MMLU by 3.2%.

    Source: §1 · p. 2, §4.1.2 · p. 11

  • Takeaways & Limitations

    The authors release the pre-trained and post-trained models to the LLM community and present scaling-law analysis as guidance for future MoE development and optimization.

    Source: §1 · p. 2

Abstract

from arXiv · show

In this paper, we introduce Hunyuan-Large, which is currently the largest open-source Transformer-based mixture of experts model, with a total of 389 billion parameters and 52 billion activation parameters, capable of handling up to 256K tokens. We conduct a thorough evaluation of Hunyuan-Large's superior performance across various benchmarks including language understanding and generation, logical reasoning, mathematical problem-solving, coding, long-context, and aggregated tasks, where it outperforms LLama3.1-70B and exhibits comparable performance when compared to the significantly larger LLama3.1-405B model. Key practice of Hunyuan-Large include large-scale synthetic data that is orders larger than in previous literature, a mixed expert routing strategy, a key-value cache compression technique, and an expert-specific learning rate strategy. Additionally, we also investigate the scaling laws and learning rate schedule of mixture of experts models, providing valuable insights and guidances for future model development and optimization. The code and checkpoints of Hunyuan-Large are released to facilitate future innovations and applications. Codes: https://github.com/Tencent/Hunyuan-Large Models: https://huggingface.co/tencent/Tencent-Hunyuan-Large

1 Introduction

Open-source models can broaden technology and application development, but most are dense and the few MoE models are relatively small. Hunyuan-Large introduces a large Transformer-based MoE model and reports strong results across diverse benchmarks.

1 Introduction

  • The paper motivates open-sourcing as a way to spread technology and support flourishing application development.It situates this contribution alongside open models including LLama, Mistral, Qwen, and Deepseek.Source: §1 · p. 1
  • The model has 389 billion total parameters, 52 billion activated parameters, and supports up to 256K tokens.It uses a classical Transformer architecture with MoE, with pre-training for fundamental capabilities and post-training for instruction following, capability enhancement, and preference alignment.Source: §1 · p. 2
  • Hunyuan-Large combines nearly 1.5T tokens of high-quality synthetic data within 7T pre-training tokens with KV-cache compression, recycle routing, and expert-specific learning-rate scaling.The introduction also presents MoE scaling-law exploration as guidance for future model development and optimization.Source: §1 · p. 2
  • Hunyuan-Large reports the overall best performance among similar-scale open-source LLMs across English and Chinese benchmarks spanning commonsense, question answering, mathematics, coding, and aggregated tasks.The pre-trained and post-trained models are publicly released for the LLM community.Source: §1 · p. 2

2 Pre-Training

The pre-training account covers data and tokenizer choices, model structure, and training recipes, including a guide to MoE scaling-law explorations.

2 Pre-Training

  • Pre-training is organized around data and tokenizer, model structure, and pre-training recipes, including a guidebook of MoE scaling-law explorations.The section describes high-quality data, KV-cache compression, expert routing, and expert-specific learning-rate scaling as foundations of pre-training capability.Source: §2 · p. 2

2.1 Data and Tokenizer

Hunyuan-Large combines filtered synthetic instruction-response data with a tokenizer designed to balance compression and vocabulary coverage. Its synthetic-data workflow has four stages, and its 128K-token vocabulary includes additions for Chinese support.

2.1.1 Data Processing and Synthesis

  • Synthetic data supplements natural text in mathematics, coding, low-resource, and high-educational-value fields, with quality, diversity, and quantity as design requirements.The data is primarily Chinese and English; filtering addresses writing quality, educational value, and toxicity, and privacy-sensitive and harmful data are anonymized.Source: §2.1.1 · p. 3
  • The four-step synthesis process generates instructions, evolves them, generates responses, and filters the resulting pairs.Instruction seeds include web pages, question-answering data, code repositories, and books; specialized models produce responses, which are checked by a critique model and self-consistency filtering.Source: §2.1.1 · pp. 3–4, Figure 1 · p. 3

2.1.2 Tokenizer

  • The tokenizer uses 128K tokens, combining 100K tiktoken tokens with 28K tokens added for Chinese support, and improves compression versus the LLama3.1 tokenizer from 2.78 to 3.13 characters per token.The design balances compression for training and inference with a sufficiently large vocabulary for word embeddings.Source: §2.1.2 · p. 4

2.2 Model Structure

Hunyuan-Large combines shared and specialized experts with KV-cache compression and expert-specific learning rates. Its structure and routing address inference memory and expert-load differences through compression, token reassignment, and learning-rate scaling.

2.2 Model Structure

  • The section presents KV-cache compression, expert routing, and expert-specific learning-rate scaling as model-structure strategies for capability and training.The architecture uses shared and specialized experts, with only a small ratio of experts activated during training.Source: §2.2 · p. 4, §2.2.1 · p. 4

2.2.1 Overview of Hunyuan-Large

  • Hunyuan-Large has 389B total parameters and 52B activated parameters, with one shared expert and one specialized expert activated for each token.Its MoE structure uses shared and specialized experts; the architecture otherwise follows the classical MoE pattern of replacing Transformer FFNs with experts.Source: Table 1 · p. 4, §2.2.1 · p. 4

2.2.2 KV Cache Compression

  • Nearly 95% of KV cache is saved with GQA+CLA compared with original MHA, improving inference efficiency without much side effect on model performance.The model uses 8 groups of KV heads and shares the cache every 2 layers.Source: §2.2.2 · pp. 4–5, Table 2 · p. 5

2.2.3 Expert Routing Strategy

  • The mixed routing strategy sends every token to a shared expert and to its top-scoring specialized expert, selected from 16 specialized experts.When an expert is overloaded, recycle routing randomly reallocates its otherwise-discarded tokens to specialized experts still under capacity.Source: §2.2.3 · p. 5, Figure 2 · p. 6

2.2.4 Expert-Specific Learning Rate Scaling

  • The shared expert uses the optimal learning rate for batch size B, while specialized experts use a scaled rate with ratio approximately 0.31.The method treats specialized experts as having effective batch size B/16 because one of 16 specialized experts is activated per token; the passage relates the imbalance to different trained-token counts across experts.Source: §2.2.4 · pp. 5–6

2.3 Pre-Training Recipes

The pre-training recipes use MoE scaling-law analyses to guide model settings, then apply staged learning-rate scheduling and long-context pre-training. The section describes how these choices shape compute, training data, and sequence length.

2.3.1 MoE Scaling Law

  • The MoE scaling analysis estimates compute using activated parameters and accounts for sequence length and sparse activation, then uses critical batch size to estimate minimum compute.A series of models from 10M to 1B activated parameters, trained on 100B tokens, informs the isoFLOPs exploration.Source: §2.3.1 · pp. 6–7
  • Hunyuan-Large uses 52B activated parameters, close to the estimated optimum of approximately 58.1B, and selects approximately 7T pre-training tokens against an estimated optimum of 5.6T.The parameter choice follows a smooth quadratic fit; the data choice aims to use more training data within the stated cost-performance range.Source: §2.3.1 · p. 7, Figure 3 · p. 7, Figure 4 · p. 8

2.3.2 Learning Rate Scheduling

  • The learning-rate schedule has an initial warmup, a prolonged gradual-decay phase, and a short annealing phase.During the final 5% of pre-training tokens, the learning rate falls to one-tenth of its peak, with the highest-quality available dataset prioritized.Source: §2.3.2 · pp. 7–8

2.3.3 Long-Context Pre-Training

  • Long-context pre-training increases sequence length from 32K to 256K and scales the RoPE base frequency to 1 billion for the 256K stage.Each stage uses approximately 10B tokens, combining nearly 25% natural long-context data from books and code with nearly 75% normal-length data; the authors report retaining good normal-length task capabilities.Source: §2.3.3 · p. 8

3 Post-Training

Post-training aims to improve task-specific capabilities and align Hunyuan-Large with human preferences through supervised fine-tuning (SFT) and reinforcement learning from human feedback (RLHF). Its data pipeline collects, expands, balances, and filters instruction examples before SFT and DPO training.

3.1.1 Overview of SFT Data

  • SFT targets a broad set of capabilities using more than 1 million examples, while also emphasizing data security and alignment with human values.The listed capabilities include mathematics, coding, reasoning, question answering, agent behavior, text generation, and long-text tasks.Source: §3.1.1 · p. 9

3.1.2 Data Collection and Processing

  • The instruction pipeline extracts natural instructions, generalizes them into more diverse and complex examples, and labels examples to balance types across SFT data.The accumulated instruction set exceeds 10 million before the SFT process.Source: §3.1.2 · p. 9
  • SFT data quality control combines rule-based filtering, a critique model that assigns four-tier quality scores, and human annotation.The filters address issues such as truncation, duplication, garbled characters, and formatting errors; human review checks task-specific response patterns.Source: §3.1.2 · pp. 9–10

3.1.3 Training Details

  • SFT fine-tunes the model on high-quality data for three epochs, with attention dropout of 0.1 and hidden dropout of 0.2.The learning rate decays from 2e-5 to 2e-6; the authors report that Hunyuan-series MoE models benefit more than dense models from suitable dropout rates.Source: §3.1.3 · p. 10

3.2 Reinforcement Learning from Human Feedback

  • For RLHF, the paper uses single-stage DPO combining an offline preference dataset with policy-generated responses ranked by a reward model.An SFT loss on chosen responses is used to stabilize training, while exponential moving averages are intended to mitigate reward hacking and reduce alignment tax.Source: §3.2 · p. 10

4 Model Evaluations

The pre-trained model is evaluated on English and Chinese benchmarks spanning aggregated tasks, commonsense, question answering, reading comprehension, mathematics, and coding. The reported results show strong performance across these task groups.

4.1.1 Benchmarks and Experimental Settings

  • The evaluation covers aggregated benchmarks, commonsense, question answering and reading comprehension, mathematics, and coding in both English and Chinese.The named benchmarks include MMLU, MMLU-Pro, BBH, CMMLU, C-Eval, GSM8k, MATH, HumanEval, and MBPP.Source: §4.1.1 · p. 10

4.1.2 Model Performance of Pre-Training

  • On MMLU, Hunyuan-Large surpasses LLama3.1-405B by 3.2%, and the authors report the best overall performance among dense and MoE competitors with similar activated parameter sizes.The paper also reports leading results on GSM8K, MATH, CMATH, and Chinese aggregated tasks, with first-tier results on HumanEval and MBPP.Source: Table 3 · p. 11, §4.1.2 · p. 11

4.1.1 Benchmarks and Experimental Settings

  • The comparisons use commonly adopted benchmark settings and include dense and MoE pre-trained competitors such as LLama3.1-70B, Mixtral-8x22B, DeepSeek-V2, and LLama3.1-405B.The paper reports the best publicly reported or reproduced baseline results.Source: §4.1.1 · p. 11

4.2 Evaluations on Post-Trained Models

Hunyuan-Large-Instruct is evaluated against comparable-parameter instruction models on public benchmarks and additional instruction-following and preference tests. The authors report improvements on most task types and leading results on several benchmarks.

4.2.1 Benchmarks and Experimental Settings

  • The comparison uses public benchmarks for comprehension, question answering, commonsense reasoning, mathematics, coding, and aggregated tasks, against dense and MoE instruction models.Baselines include LLama3.1-405B-Instruct, LLama3.1-70B-Instruct, Mixtral-8x22B-Instruct, and DeepSeek-V2.5-Chat.Source: §4.2.1 · p. 12

4.2.2 Model Performance of Post-Training

  • Hunyuan-Large-Instruct outperforms LLama3.1-405B by 2.6% on MMLU and 3.6% on MATH, achieving these gains with only 52 billion activated parameters.The authors report consistent improvements on most task types compared with LLMs having similar activated parameters.Source: §4.2.2 · p. 12
  • Hunyuan-Large-Instruct has the best overall performance among the listed baselines on AlignBench, MT-Bench, IFEval strict-prompt, Arena-Hard, and AlpacaEval-2.0.These tests assess alignment with human intentions, expert preference, strict instruction following, real-world preference alignment, and instruction following.Source: §4.2.2 · pp. 12–13

4.3 Long-Context Capability Evaluations

The section evaluates Hunyuan-Large-Instruct on open-source long-context benchmarks and introduces PenguinScrolls, an in-house benchmark designed to address gaps in existing evaluations. The reported results favor Hunyuan-Large-Instruct across the tested long-context settings.

4.3.1 Open-Source Long-Context Benchmarks and Evaluation

  • Across RULER’s 64K–128K range, Hunyuan-Large-Instruct significantly outperforms the baseline with minimal performance degradation as context length increases.RULER covers retrieval, multi-hop reasoning, aggregation, and question answering across varying context lengths.Source: §4.3.1 · p. 13, Table 5 · p. 13
  • Hunyuan-Large-Instruct consistently outperforms LLama3.1-70B-Instruct across all LV-Eval length intervals.LV-Eval comprises 11 question-answering datasets with varying context lengths and confounding facts; the evaluation uses an LLM evaluator to address high false-negative rates from the original metrics.Source: §4.3.1 · p. 13, Table 5 · p. 13

4.3.2 In-House Evaluation with PenguinScrolls

  • PenguinScrolls spans natural long-form documents up to 128K tokens, fine-grained tasks, human-simulated multi-turn dialogue, and Chinese and English data.It was developed to address limited real-world content diversity and the lack of multilingual and multi-turn dialogue data in existing benchmarks.Source: §4.3.2 · p. 13
  • Hunyuan-Large-Instruct outperforms LLama3.1-70B-Instruct on all four PenguinScrolls tasks, and internal user studies find that benchmark improvements strongly correlate with actual user-experience improvements.The tasks are information extraction, information localization, qualitative analysis, and numerical reasoning.Source: §4.3.2 · pp. 13–14, Table 6 · p. 14

5 Conclusions and Future Work

The report concludes with Hunyuan-Large, a 389B-parameter Transformer-based MoE model with 52B activated parameters and support for contexts up to 256K tokens. Its evaluations show strong performance across diverse tasks, and the model weights are released to the community.

5 Conclusions and Future Work

  • Hunyuan-Large has 389B total parameters, 52B activated parameters, and supports up to 256K context length.The report describes it as the currently largest and best-performing Transformer-based MoE model.Source: §5 · p. 14
  • Evaluations across dozens of benchmarks cover language understanding, generation, reasoning, mathematics, coding, long context, and aggregated tasks.The report attributes favorable outcomes largely to high-quality training data with data synthesis, model structure, and pre- and post-training recipes.Source: §5 · p. 14
  • The report releases Hunyuan-Large model weights to facilitate community research and practical applications.It expresses the hope that the release will inspire research innovation and practical applications.Source: §5 · p. 14

Contributors and Acknowledgements

The authors acknowledge contributions from many Tencent teams and name the project’s central contributors.

Contributors and Acknowledgements

Related papers

Loading 2411.02265v3…