Source-linked AI summary
Baichuan 2: Open Large-scale Language Models
Aiyuan Yang, Bin Xiao, Bingning Wang, Borong Zhang, Ce Bian, Chao Yin, Chenxu Lv, Da Pan, Dian Wang, Dong Yan, Fan Yang, Fei Deng, Feng Wang, Feng Liu, Guangwei Ai, Guosheng Dong, Haizhou Zhao, Hang Xu, Haoze Sun, Hongda Zhang, Hui Liu, Jiaming Ji, Jian Xie, JunTao Dai, Kun Fang, Lei Su, Liang Song, Lifeng Liu, Liyun Ru, Luyao Ma, Mang Wang, Mickel Liu, MingAn Lin, Nuolan Nie, Peidong Guo, Ruiyang Sun, Tao Zhang, Tianpeng Li, Tianyu Li, Wei Cheng, Weipeng Chen, Xiangrong Zeng, Xiaochuan Wang, Xiaoxi Chen, Xin Men, Xin Yu, Xuehai Pan, Yanjun Shen, Yiding Wang, Yiyu Li, Youxin Jiang, Yuchen Gao, Yupeng Zhang, Zenan Zhou, Zhiying Wu
TL;DR
Existing LLMs are often closed-source or weak in languages beyond English, limiting transparent research and multilingual use. Baichuan 2 introduces multilingual 7B and 13B models trained on 2.6 trillion tokens, reports strong benchmark and domain performance, and releases models and training checkpoints. The models improve over Baichuan 1 and perform competitively with similar-sized open-source systems, while retaining documented safety, language, and knowledge limitations.
Problem
Many leading LLMs are closed-source, and many open-source models focus primarily on English, limiting access for research and non-English applications.
Method
Baichuan 2 introduces multilingual 7B and 13B models trained on 2.6 trillion tokens, alongside chat models, training-stage checkpoints, and training and alignment methods.
Results
Baichuan 2 improves over Baichuan 1 on general, mathematical, and coding benchmarks and outperforms other open-source models on medical and legal evaluations.
Takeaways & Limitations
The released models and checkpoints provide resources for multilingual LLM research, domain-specific optimization, safety work, and analysis of training dynamics.
Takeaways & Limitations
Baichuan 2 remains vulnerable to bias, toxicity, outdated or incorrect knowledge, misuse, over-cautious safety behavior, and limited coverage of non-Chinese cultures and languages.
Abstract
from arXiv · showhide
Large language models (LLMs) have demonstrated remarkable performance on a variety of natural language tasks based on just a few examples of natural language instructions, reducing the need for extensive feature engineering. However, most powerful LLMs are closed-source or limited in their capability for languages other than English. In this technical report, we present Baichuan 2, a series of large-scale multilingual language models containing 7 billion and 13 billion parameters, trained from scratch, on 2.6 trillion tokens. Baichuan 2 matches or outperforms other open-source models of similar size on public benchmarks like MMLU, CMMLU, GSM8K, and HumanEval. Furthermore, Baichuan 2 excels in vertical domains such as medicine and law. We will release all pre-training model checkpoints to benefit the research community in better understanding the training dynamics of Baichuan 2.
1 Introduction
Baichuan 2 addresses limited openness and non-English capability in current large language models by introducing multilingual 7B and 13B models with extensive training and released checkpoints. It reports stronger general, mathematical, coding, medical, and legal performance than Baichuan 1 and comparable open-source models.
- Motivation: Most leading LLMs are closed-source, while many open-source models primarily focus on English, limiting study, fine-tuning, and non-English applications.The introduction identifies openness and multilingual capability as central gaps in the field.
- Contribution: Baichuan 2 introduces separate 7 billion- and 13 billion-parameter multilingual models trained on 2.6 trillion tokens.The report presents Baichuan 2 as a large-scale multilingual model series.
- Results: Baichuan 2-7B achieves nearly 30% higher performance than Baichuan 1-7B on MMLU, CMMLU, and C-Eval.This comparison concerns general benchmarks reported in the introduction.
- Results: Baichuan 2 nearly doubles Baichuan 1 results on GSM8K and HumanEval and outperforms other open-source models on MedQA and JEC-QA.The reported evaluations cover mathematics, code, medicine, and law-related tasks.
- Release and transparency: The report releases chat models, training-stage checkpoints, methodological details, and unsuccessful experiments to support research, safety work, and understanding of training dynamics.Checkpoints span 200 billion tokens through the full 2.6 trillion tokens.
2 Pre-training
Baichuan 2’s pre-training combines broad data collection, large-scale quality processing, Transformer modifications, and elastic distributed infrastructure. The report also describes optimization techniques and scaling-law modeling for stable, efficient training.
- Overview: Baichuan 2’s pre-training procedure covers data sourcing, processing, architecture design, optimization, scaling analysis, and distributed training.The report introduces these components as the main elements of the foundation-model training procedure.
- Pre-training Data: Data is gathered from webpages, books, research papers, codebases, and other sources, then clustered, deduplicated, scored, and sampled for pre-training.The processing system supports LSH-like and dense-embedding features and operates at trillion-token scale.
- Architecture: The Transformer-based architecture expands the vocabulary from 64,000 to 125,696 tokens and uses different positional embeddings for the 7B and 13B models.Baichuan 2-7B uses RoPE, while Baichuan 2-13B uses ALiBi; preliminary experiments found no significant performance difference between the choices.
- Optimizations: NormHead normalizes output embeddings to stabilize training dynamics and reduce the influence of L2 distance on logits.The report links the method to unstable rare-token embedding norms and to semantic information being mainly encoded by embedding cosine similarity.
3 Alignment
Baichuan 2 alignment produces chat models through supervised fine-tuning and reinforcement learning from human feedback. The process uses categorized prompts, model-family responses, reward modeling, and PPO-based optimization.
- SFT and RLHF form the two main components of Baichuan 2’s alignment procedure for its 7B and 13B chat models.
- Human-labeled prompts are filtered through cross-validation to retain helpfulness and harmlessness annotations that meet quality standards.
- 6 primary, 30 secondary, and over 200 tertiary categories organize prompts to cover user needs and support reward-model generalization.
- Baichuan 2 model responses from different sizes and training stages are used for reward-model training because external model responses do not improve accuracy.
- The reward model’s accuracy increases with the score gap between two responses, and its performance is consistent with LLaMA 2.
- PPO updates actor and critic models while a fixed reference model supplies the KL penalty and a fixed reward model scores complete responses.
4 Safety
Baichuan 2 treats safety as a training-wide objective rather than limiting it to data cleansing or alignment. Its approach combines filtered and value-oriented pre-training data with red-teaming and safety-focused reinforcement learning.
- Safety improvements are pursued across all Baichuan 2 training stages by filtering harmful content and incorporating positive and negative knowledge.
- Pre-training data are filtered with rules and models to remove violence, pornography, racial discrimination, hate speech, and other harmful content.
- A Chinese-English dataset from hundreds of reputable websites covers positive value domains, and its sampling probability is increased.
- The red-teaming procedure uses 6 attack types, 100+ granular safety categories, and expert-guided annotation to initialize safe-alignment data.
- DPO addresses specific vulnerability issues, while a reward model combining helpful and harmless objectives supports PPO safety reinforcement.
5 Evaluations
Baichuan 2 is evaluated against similarly sized open-source models across general, domain-specific, mathematical, coding, multilingual, and safety tasks. It shows broad competitive or superior performance, while training checkpoints reveal continued improvement through 2.6 trillion tokens.
- Overall Performance: Baichuan 2 has a clear performance advantage over similar-sized open-source models, with especially large gains over Baichuan 1 on mathematics and coding.The report describes the overall comparison in Table 1 and highlights math and code as areas of significant improvement.
- Vertical Domain Evaluations: Baichuan 2-7B-Base outperforms comparable models in Chinese law and medicine, while improving by nearly 10 points over Baichuan 1-7B in Chinese law.It surpasses GPT-3.5 Turbo, ChatGLM 2-6B, and LLaMA 2-7B in Chinese law, ranking behind GPT-4.
- Vertical Domain Evaluations: Baichuan 2-13B-Base surpasses same-size alternatives in Chinese law and outperforms XVERSE-13B and LLaMA 2-13B in medicine.In Chinese law, it surpasses models other than GPT-4 and shows remarkable improvement over Baichuan 1-13B-Base.
- Math and Code: Baichuan 2-7B-Base surpasses LLaMA 2-7B in mathematics and ChatGLM 2-6B in coding, while Baichuan 2-13B-Base surpasses same-size models in mathematics and approaches GPT-3.5 Turbo.Both model sizes show significant improvement over their corresponding Baichuan 1 versions.
- Multilingual: Baichuan 2-7B-Base surpasses same-size models in all seven multilingual tasks, while Baichuan 2-13B-Base does so in four and reaches GPT-4 level on zh-en and zh-ja.The 13B model slightly surpasses GPT-4 on zh-en, although GPT-4 remains dominant overall in multilingual evaluation.
- Safety: Safety alignment significantly improves harmlessness without hurting helpfulness, and Baichuan 2 is on par with or better than other chat models in safety evaluations.The report also describes safety advantages over LLaMA 2 and introduces the Baichuan Harmless Evaluation Dataset covering seven major safety areas.
- Intermediate Checkpoints: Baichuan 2 improves consistently as training proceeds, with apparent room for further gains even after 2.6 trillion tokens.The report releases intermediate checkpoints and provides additional training-dynamics analyses for both 7B and 13B models.
6 Related Work
Related work frames Baichuan 2 within scaling-law research, the rise of proprietary and open-source LLMs, and methods for instruction alignment. It emphasizes that token quantity and data scaling remain important alongside parameter count.
- Scaling Laws: Scaling-law research linked model performance to increases in parameters and data size, shaping the development of increasingly large language models.Kaplan et al. provided a blueprint for large-scale pre-training by systematically analyzing these factors.
- Large Language Models: The growth of proprietary LLMs has been accompanied by specialized scientific models and few-shot or zero-shot capabilities across natural-language tasks.The related work cites GPT-3 and Galactica as examples of this broader development.
- Scaling Laws: Chinchilla showed that scaling model capacity with token count, rather than parameters alone, can improve sample efficiency.This motivates attention to training-data scale in addition to raw model size.
- Open-Source Models: Open-source alternatives have expanded, but many models with large parameter counts were trained on 500 billion tokens or fewer.The passage contrasts this limit with evidence that 7 billion parameter models can continue improving after trillions of tokens.
- Alignment Methods: Instruction-following chat models commonly fine-tune foundation models to align with human preferences, while DPO and RLAIF seek greater efficiency or effectiveness.These approaches complement the development of pretrained foundation models.
7 Limitations and Ethical Considerations
Baichuan 2 faces unresolved ethical and safety risks, including toxicity, misuse, outdated knowledge, and limited coverage of non-Chinese cultures and languages.
- Ethical and safety risks: Toxicity tends to increase with model size, despite mitigation efforts using benchmarks such as Toxigen.The models are also prone to biases, especially because much training data originates from the internet.
- Scope boundaries: Static model knowledge can be outdated or incorrect, creating challenges for applications requiring current information, including medicine and law.
- Scope boundaries: Baichuan 2 has limitations in languages beyond Chinese and English and may not capture biases relevant to non-Chinese cultures.
- Ethical and safety risks: Potential misuse includes generating harmful or misleading content, while safety measures may be over-cautious and reduce usability for some tasks.
B NormHead
NormHead normalizes output embeddings so logits depend on angular information rather than embedding magnitude, addressing instability associated with rare-token norms.
- Motivation and design: NormHead normalizes output embeddings before the output dot product, reducing the influence of L2 distance and emphasizing cosine-based semantic similarity.The motivation comes from KNN retrieval results showing cosine similarity produced semantically similar words, whereas L2 distance could produce meaningless neighbors.
- Ablation: A 7B ablation trained for 12k steps with otherwise identical settings found unstable early training without NormHead and more stable training with it.The comparison used the same hyperparameters and data as Baichuan 2-7B; the loss curves are shown in Figure 9.
C Training Dynamics
Training improves Baichuan 2 performance broadly, but benchmark progress differs by task: general benchmarks plateau after 2 trillion tokens while GSM8K continues improving.
- Training progression: Both Baichuan 2-7B and Baichuan 2-13B show substantial gains as training progresses across the evaluated benchmarks.Intermediate checkpoints were saved every 1,000 steps and evaluated on C-Eval, MMLU, CMMLU, JEC-QA, GSM8K, and HumanEval.
- Task-dependent dynamics: After 2 trillion tokens, improvements appear to plateau on MMLU and C-Eval, unlike the continued gains observed on GSM8K.The passage suggests that training FLOPs may correlate strongly with improvements in mathematical problem solving, which warrants further study.
D Baichuan Harmless Evaluation Dataset
The Baichuan Harmless Evaluation Dataset evaluates chat-model safety across seven categories using diverse sub-dimensions and 10k prompts per category.
- Dataset purpose: BHED is proposed to evaluate Baichuan chat models, with its principles and cases introduced in this section.
- Safety categories: The seven safety categories are bias and discrimination, insults and profanity, illegal or unethical content, physical health, mental health, financial privacy, and sensitive topics.
- Coverage design: Category diversity is expanded through sub-dimensions covering identities and regions, implicit and explicit abuse, legal domains, and other safety-related distinctions.
- Dataset scale: 10k prompts are collected for each safety category, with examples provided in Table 11.
E Details of MMLU and C-Eval
This section provides subject-level evaluation details for Baichuan 2-7B-Base and Baichuan 2-13B-Base on C-Eval and MMLU. It also identifies the tables containing these scores.
- Table 11 presents examples from the Baichuan Harmless Evaluation Dataset.
- Tables 12 and 13 report subject-level scores for Baichuan 2-7B-Base and Baichuan 2-13B-Base on C-Eval and MMLU.
- Table 12 contains the scores of each C-Eval subject for the two Baichuan 2 base models.
- Table 13 contains the scores of each MMLU subject for the two Baichuan 2 base models.
F Examples generated by Baichuan 2-13B-Chat
The examples show Baichuan 2-13B-Chat handling translation, factual question answering, reading comprehension, coding, and algebraic problem solving. Its responses include direct answers, explanations, and executable Python code.
- Baichuan 2-13B-Chat follows a translation instruction by prefixing each English translation with “Translate:”.
- Baichuan 2-13B-Chat computes the median of seven numbers as 6.2 and supplies Python code that sorts the array before selecting the middle value.
- Baichuan 2-13B-Chat solves the chicken-and-rabbit problem algebraically, obtaining 18 chickens and 8 rabbits.
- Baichuan 2-13B-Chat answers an astronomy question by identifying the claim that the Moon is a planet as false and explaining that it is Earth’s satellite.
- In a physics question, Baichuan 2-13B-Chat explains the relevant principles and identifies D as the correct answer, while its accompanying option text marks C as erroneous.
- Baichuan 2-13B-Chat evaluates a reading-comprehension exercise with two true statements and one false statement.