Source-linked AI summary
Hunyuan-A13B Technical Report
Tencent Hunyuan Team, Ao Liu, Botong Zhou, Can Xu, Chayse Zhou, ChenChen Zhang, Chengcheng Xu, Chenhao Wang, Decheng Wu, Dengpeng Wu, Dian Jiao, Dong Du, Dong Wang, Feng Zhang, Fengzong Lian, Guanghui Xu, Guanwei Zhang, Hai Wang, Haipeng Luo, Han Hu, Huilin Xu, Jiajia Wu, Jianchen Zhu, Jianfeng Yan, Jiaqi Zhu, Jihong Zhang, Jinbao Xue, Jun Xia, Junqiang Zheng, Kai Liu, Kai Zhang, Kai Zheng, Kejiao Li, Keyao Wang, Lan Jiang, Lixin Liu, Lulu Wu, Mengyuan Huang, Peijie Yu, Peiqi Wang, Qian Wang, Qianbiao Xiang, Qibin Liu, Qingfeng Sun, Richard Guo, Ruobing Xie, Saiyong Yang, Shaohua Chen, Shihui Hu, Shuai Li, Shuaipeng Li, Shuang Chen, Suncong Zheng, Tao Yang, Tian Zhang, Tinghao Yu, Weidong Han, Weijie Liu, Weijin Zhou, Weikang Wang, Wesleye Chen, Xiao Feng, Xiaoqin Ren, Xingwu Sun, Xiong Kuang, Xuemeng Huang, Xun Cao, Yanfeng Chen, Yang Du, Zhen Yang, Yangyu Tao, Yaping Deng, Yi Shen, Yigeng Hong, Yiqi Chen
TL;DR
Advanced LLMs can impose substantial inference latency and hardware costs, limiting accessibility. Hunyuan-A13B combines sparse MoE activation, extensive training, and dual-mode reasoning; the authors report competitive cross-task performance, including strong agent-task results, with high throughput.
Problem
Advanced LLM deployment demands substantial computational resources, inference latency, and hardware costs, limiting broad accessibility.
Source: §1 · p. 1Method
Hunyuan-A13B combines an 80B-parameter MoE that activates 13B per input with curated 20T-token pretraining, supervised fine-tuning, reinforcement learning, and dual-mode reasoning.
Source: §1 · pp. 1–2Results
Evaluations report competitive performance across reasoning, coding, and agent tasks, with agent-task gains over larger alternatives and high throughput.
Source: §1 · p. 2, §4.2.2 · p. 9Takeaways & Limitations
The authors present Hunyuan-A13B as an efficient, open-source alternative that lowers deployment costs while preserving advanced language-modeling capabilities.
Source: §1 · p. 1, §5 · p. 11
Abstract
from arXiv · showhide
We present Hunyuan-A13B, an open-source large language model based on a Mixture-of-Experts architecture. It contains 80 billion total parameters but activates only 13 billion during inference, balancing model capability, computational efficiency, and deployment cost. The model is pretrained on a rigorously filtered 20T-token corpus with enhanced STEM data curation, improving factual reliability and reasoning ability. High-quality supervised fine-tuning and large-scale reinforcement learning further enhance its overall performance. Hunyuan-A13B also introduces a dual-mode Chain-of-Thought framework that adapts reasoning depth to task complexity: fast thinking for routine queries and slow thinking for complex, multi-step problems. Evaluations show competitive performance across mathematics, science, programming, general language understanding, and agent tasks, often approaching that of much larger models. Its high inference throughput makes it suitable for latency-sensitive applications. We release Hunyuan-A13B to support open research and practical LLM deployment.
1 Introduction
Hunyuan-A13B targets the inference cost of advanced LLMs with a sparse MoE design, extensive training, and reasoning modes matched to task complexity. The authors report competitive cross-domain results and high throughput for resource-constrained use.
1 Introduction
- 80B total parameters with 13B activated per input underpin Hunyuan-A13B’s sparse design, which the authors say reduces latency and computational overhead versus dense models of similar scale.The paper presents this as a way to lower deployment costs while preserving language-modeling capability.Source: §1 · p. 1
- The model uses a 20T-token pretraining corpus with enhanced STEM curation, followed by long-CoT supervised fine-tuning and large-scale reinforcement learning.The authors describe the STEM data curation as intended to raise the upper bound for reasoning ability.Source: §1 · p. 1
- Dual-CoT offers concise short reasoning for simpler queries and detailed long reasoning for complex tasks, letting users select a mode for their application’s complexity and resource constraints.Source: §1 · p. 1
- Inference optimizations increase token throughput and support real-time, resource-constrained scenarios requiring fast predictions.Source: §1 · p. 2
- Pretraining includes fast annealing and a long-context stage that extends the context window to 256K.The model is first trained on more than 20T tokens.Source: §1 · p. 2
- Evaluations report competitive performance in mathematics, science, logic, coding, and agent tasks, with Hunyuan-A13B outperforming larger alternatives on agent-oriented tasks.The authors also report high throughput without compromising reasoning quality, accuracy, or generalization.Source: §1 · p. 2
2 Pre-Training
The pretraining section describes corpus construction, model architecture, and a staged training process. The approach combines curated data, a fine-grained MoE, and context-window expansion.
2 Pre-Training
- Pretraining is organized around data allocation, model structure, and sequential training stages.Source: §2 · p. 2, §2.3 · p. 3
2.1 Data for Pre-training
- The corpus pipeline filters, deduplicates, and labels data, with enhanced STEM acquisition yielding 250B tokens and refined knowledge labeling and difficulty grading.The STEM corpus is incorporated into Hunyuan-A13B’s training data.Source: §2.1 · p. 2
2.2 Model Architechture
- The fine-grained MoE uses one shared expert and 64 non-shared experts; experiments found one shared expert beneficial, while adding more brought marginal gains or fluctuations.Grouped-Query Attention is used to improve KV-cache memory efficiency.Source: §2.2 · pp. 2–3
2.3 Pre-training Stage
- Training proceeds from a 20T-token foundation stage to fast annealing, then two long-context phases expanding the window to 32K and 256K tokens.The foundation stage uses a fixed 4096-token context window, and annealing raises it to 8192 tokens.Source: §2.3 · p. 3
3 Post-training
Post-training combines reasoning-focused and all-scenarios fine-tuning with a dual-mode reasoning framework. Its supervised and reinforcement-learning stages use curated data and task-specific feedback.
3 Post-training
- Post-training comprises reasoning-oriented and all-scenarios fine-tuning phases that target complex reasoning and broader practical competence.Source: §3 · p. 3
3.1 Reasoning-oriented Fine-Tuning
- Reasoning-oriented fine-tuning uses curated chain-of-thought instruction data and correctness-based reinforcement feedback for mathematics, logic, code, and science.Source: §3.1 · p. 3
3.1.1 Reasoning-oriented SFT Stage
- Math, code, logic, and science reasoning examples are verified through automated checks, sandbox execution, or human review before inclusion in supervised fine-tuning data.Math solutions are iteratively assessed and refined; complex logical cases receive human verification, while complex science items undergo LLM-based checking and rejection sampling.Source: §3.1.1 · p. 4
3.1.2 Reasoning-oriented RL Stage
- GRPO reinforcement learning builds on the supervised fine-tuned foundation, sampling prompts where SFT performance is unstable; it uses 150K samples (10% overlapping SFT data, 90% novel) and excludes multiple-choice, true/false, and proof-based problems to ensure verifiable outcomes.RL training uses 24K-length contexts in Phase 1 and 32K-length contexts in Phase 2.Source: §3.1.2 · p. 4
3.2 All-Scenarios Fine-Tuning
- All-scenarios fine-tuning broadens competence across writing, knowledge QA, instruction following, and multi-turn conversation, with RL evaluating both correctness and stylistic quality.Source: §3.2 · p. 5
3.2.1 All-Scenarios SFT Stage
- All-scenarios SFT combines specialized reasoning data with general examples and curates language, creative, multilingual, complex-context, and agent interactions.Agent data uses synthetic multi-role conversations and sandbox, MCP, and synthetic tool responses; the training set includes 20,000 tool/action/response format combinations.Source: §3.2.1 · p. 5
3.2.2 All-Scenarios RL Stage
- All-scenarios RL unifies a flexible generative reward model with domain-specific pipelines across 16 sub-topics and over 30 scoring services.The generative reward model can compare open-ended answers with references and use ground truth for deterministic tasks; it can also assess reasoning traces or task constraints.Source: §3.2.2 · p. 6
3.3 Dual-Mode Chain-of-Thought
- Dual-Mode CoT adapts reasoning depth to task complexity: fast mode limits overhead, while slow mode uses reflection and backtracking for complex tasks; control tags select the mode, with slow thinking as default.Slow thinking produces longer chains and uses more tokens; fast-thinking examples omit detailed reasoning within the structured think block.Source: §3.3 · p. 7
4.1 Evaluations on Pre-Trained Model
The evaluation compares Hunyuan-A13B with representative dense and MoE models across public benchmark sets. The authors report broad benchmark advantages over two comparators despite smaller parameter budgets.
4.1.1 Benchmarks
- The evaluation covers general tasks, coding, mathematics, and STEM benchmarks, including language understanding, programming, math, and science test sets.The listed protocols include varied few-shot settings and chain-of-thought evaluation for some tasks.Source: §4.1.1 · p. 7
4.1.2 Evaluation Results
- 12 of 14 evaluation benchmarks favor Hunyuan-A13B over Hunyuan-Large, despite one-quarter the activated parameters and about one-fifth the total parameters.Against similarly sized Qwen2.5-72B, it scores higher on almost all benchmarks; against Qwen3-A22B, it wins 7 of 12 while showing comparable capabilities across multiple dimensions.Source: §4.1.2 · p. 7
4.2 Evaluations on Post-Trained Model
The evaluation compares Hunyuan-A13B with representative open- and closed-source models across reasoning, coding, language, instruction-following, and tool-use tasks. It frequently reaches top or near-top results, including leading scores in several reasoning and agent benchmarks.
4.2.1 Benchmarks
- The evaluation spans mathematics, science, coding, logical reasoning, instruction following, text generation, natural-language understanding, and agent tool-calling tasks.It uses public benchmarks and internal tests; ArtifactsBench targets frontend code generation, while LengthCtrl and InsCtrl test output-length control and complex instruction execution.Source: §4.2.1 · p. 8
- The benchmark suite uses MATH and AIME for mathematics, GPQA-Diamond and OlympiadBench for science, and LiveCodeBench, FullstackBench, McEval, and ArtifactsBench for coding.Logical reasoning uses BBH, ZebraLogic, and DROP; instruction following uses IFEval and SysBench.Source: §4.2.1 · p. 8
4.2.2 Evaluation Results
- Hunyuan-A13B frequently ranks at or near the top, leading on AIME2024, ZebraLogic, and BBH while showing strong science and instruction-following results and slightly lower overall coding performance.On FullstackBench and ArtifactsBench, its coding results are comparable to substantially larger models.Source: Table 3 · p. 9, Table 4 · p. 9, §4.2.2 · p. 9
- Hunyuan-A13B leads on BFCL-v3, τ-Bench, ComplexFuncBench, and C3-Bench, including under changing task combinations in C3-Bench.The authors interpret these results as evidence of strong tool-calling capability across varied application scenarios.Source: §4.2.2 · p. 9
4.3 Long-Context Evaluation
Hunyuan-A13B is evaluated on long-text tasks, general long-context understanding, retrieval-augmented generation, and context-length scaling. It performs competitively across the reported benchmarks, with stable results as contexts grow and a relative weakness on FRAMES against two open-source competitors.
4.3 Long-Context Evaluation
- The long-context evaluation covers real-world long-text tasks, broad long-context understanding, and RAG, alongside RULER QA across configurable context lengths.The paper describes RULER QA as closer to real-world scenarios and harder to shortcut than other tasks.Source: §4.3 · p. 10, §4.3.2 · pp. 10–11
4.3.1 Performance on Key Long-Context Benchmarks
- On PenguinScrolls, Hunyuan-A13B scores 87.7, close to Gemini 2.5 Pro (88.3) and above DeepSeek R1 (87.5) and Qwen3-A22B (87.1).The benchmark covers practical long-text tasks including information extraction, localization, qualitative analysis, and numerical reasoning.Source: §4.3.1 · p. 10, §4.3 · p. 10
- On LongBench-v2, Hunyuan-A13B scores 55.0, second to Gemini 2.5 Pro (60.9) and ahead of DeepSeek R1 (53.8) and Qwen3-A22B (48.4).The benchmark includes single- and multi-document QA, long in-context learning, dialogue history, code repositories, and structured data.Source: §4.3.1 · p. 10, §4.3 · p. 10
- On FRAMES, Hunyuan-A13B scores 81.1, above Gemini 2.5 Pro (80.1) but below DeepSeek R1 (85.7) and Qwen3-A22B (84.0).FRAMES evaluates end-to-end RAG factuality, retrieval accuracy, and reasoning with multi-hop questions.Source: §4.3.1 · p. 10, §4.3 · p. 10
4.3.2 Performance Scaling with Increasing Context Lengths
- At 64K–128K context, Hunyuan-A13B scores 73.9 on RULER QA, above DeepSeek R1 (65.6) and Qwen3-A22B (66.6), with long-range performance decay second only to Gemini 2.5 Pro.Its average RULER QA score is 76.7; Gemini 2.5 Pro leads across all context lengths.Source: Table 6 · p. 11, §4.3.2 · p. 11
4.4 Inference efficiency
The inference-efficiency section describes architectural and serving options for Hunyuan-A13B and identifies the precision setting used to report throughput.
4.4 Inference efficiency
- Hunyuan-A13B combines GQA and MoE architectures to enhance reasoning efficiency without sacrificing model performance.The report presents throughput under the A16W16C16 precision setting.Source: §4.4 · p. 11, Table 7 · p. 11
- The model supports vLLM, SGLang, and TensorRT-LLM for W16A16 inference, with Auto Prefix Caching and Chunk Prefill available at the service layer.The report describes these options as supporting high-performance inference and compatibility with common inference ecosystems.Source: §4.4 · p. 11
- Supported lossless quantization formats include Weight Only INT8, W8A8, and KV Cache FP8, alongside Tensor Parallel, Expert Parallel, and FusedMoE acceleration.Table 7 presents model throughput at A16W16C16 precision.Source: §4.4 · p. 11, Table 7 · p. 11
5 Conclusion
The conclusion presents Hunyuan-A13B as an open-source MoE model that combines a 20T-token pretraining dataset and post-training optimization with inference and agent-oriented enhancements. It reports comparable performance to much larger models across varied tasks and positions the model for resource-constrained deployment.
5 Conclusion
- Despite fewer active parameters, Hunyuan-A13B performs comparably to much larger LLMs across varied benchmark tasks.The model is based on an MoE architecture and uses a 20T-token pretraining dataset.Source: §5 · p. 11
- Structured supervised fine-tuning, reinforcement-learning-based optimization, and targeted architectural enhancements are used to improve capabilities, inference speed, and throughput.The conclusion also describes scenario-specific optimizations for agent-oriented tasks and complex decision-making.Source: §5 · p. 11
- The authors characterize Hunyuan-A13B as scalable and computationally efficient, suited to advanced reasoning and general-purpose deployment in resource-constrained settings.Source: §5 · p. 11