Source-linked AI summary
InternLM2 Technical Report
Zheng Cai, Maosong Cao, Haojiong Chen, Kai Chen, Keyu Chen, Xin Chen, Xun Chen, Zehui Chen, Zhi Chen, Pei Chu, Xiaoyi Dong, Haodong Duan, Qi Fan, Zhaoye Fei, Yang Gao, Jiaye Ge, Chenya Gu, Yuzhe Gu, Tao Gui, Aijia Guo, Qipeng Guo, Conghui He, Yingfan Hu, Ting Huang, Tao Jiang, Penglong Jiao, Zhenjiang Jin, Zhikai Lei, Jiaxing Li, Jingwen Li, Linyang Li, Shuaibin Li, Wei Li, Yining Li, Hongwei Liu, Jiangning Liu, Jiawei Hong, Kaiwen Liu, Kuikun Liu, Xiaoran Liu, Chengqi Lv, Haijun Lv, Kai Lv, Li Ma, Runyuan Ma, Zerun Ma, Wenchang Ning, Linke Ouyang, Jiantao Qiu, Yuan Qu, Fukai Shang, Yunfan Shao, Demin Song, Zifan Song, Zhihao Sui, Peng Sun, Yu Sun, Huanze Tang, Bin Wang, Guoteng Wang, Jiaqi Wang, Jiayu Wang, Rui Wang, Yudong Wang, Ziyi Wang, Xingjian Wei, Qizhen Weng, Fan Wu, Yingtong Xiong, Chao Xu, Ruiliang Xu, Hang Yan, Yirong Yan, Xiaogui Yang, Haochen Ye, Huaiyuan Ying, Jia Yu, Jing Yu, Yuhang Zang, Chuyu Zhang, Li Zhang, Pan Zhang, Peng Zhang, Ruijie Zhang, Shuo Zhang, Songyang Zhang, Wenjian Zhang, Wenwei Zhang, Xingcheng Zhang, Xinyue Zhang, Hui Zhao, Qian Zhao, Xiaomeng Zhao, Fengzhe Zhou, Zaida Zhou, Jingming Zhuo, Yicheng Zou, Xipeng Qiu, Yu Qiao, Dahua Lin
TL;DR
Open-source LLMs have struggled to match the capabilities of systems such as ChatGPT and GPT-4. InternLM2 addresses this through large-scale data preparation, long-context training, and SFT plus COOL RLHF, and reports strong performance across broad evaluations and long-context tests. The release includes multiple model sizes and training-stage checkpoints to support community analysis.
Problem
The open-source community has faced difficulty reproducing the capabilities demonstrated by proprietary LLMs such as ChatGPT and GPT-4.
Method
InternLM2 uses over 2T tokens of pre-training data, GQA, training up to 32k contexts, detailed data preparation, and SFT with Conditional Online RLHF.
Results
InternLM2 demonstrates strong performance in subjective and objective evaluations, including nearly perfect Needle-in-a-Haystack identification at 200k context.
Takeaways & Limitations
Releasing 1.8B, 7B, and 20B models plus checkpoints from different training phases enables community study of model evolution and training practices.
Abstract
from arXiv · showhide
The evolution of Large Language Models (LLMs) like ChatGPT and GPT-4 has sparked discussions on the advent of Artificial General Intelligence (AGI). However, replicating such advancements in open-source models has been challenging. This paper introduces InternLM2, an open-source LLM that outperforms its predecessors in comprehensive evaluations across 6 dimensions and 30 benchmarks, long-context modeling, and open-ended subjective evaluations through innovative pre-training and optimization techniques. The pre-training process of InternLM2 is meticulously detailed, highlighting the preparation of diverse data types including text, code, and long-context data. InternLM2 efficiently captures long-term dependencies, initially trained on 4k tokens before advancing to 32k tokens in pre-training and fine-tuning stages, exhibiting remarkable performance on the 200k ``Needle-in-a-Haystack" test. InternLM2 is further aligned using Supervised Fine-Tuning (SFT) and a novel Conditional Online Reinforcement Learning from Human Feedback (COOL RLHF) strategy that addresses conflicting human preferences and reward hacking. By releasing InternLM2 models in different training stages and model sizes, we provide the community with insights into the model's evolution.
1 Introduction
InternLM2 addresses the difficulty of matching proprietary LLM capabilities in open source through extensive data preparation, long-context training, and staged alignment. It combines SFT with COOL RLHF and releases models and checkpoints across sizes and training stages.
- InternLM2 targets the gap between proprietary systems such as ChatGPT or GPT-4 and open-source LLMs.
- InternLM2 trains first on 4k-context data and then on high-quality 32k-context data, continuing long-context construction through SFT and RLHF.
- The model is open-sourced in 1.8B, 7B, and 20B sizes, with checkpoints released before and after SFT and RLHF.
- InternLM2 nearly perfectly identifies all needles in the Needle-in-a-Haystack experiment with a 200k context window.
- The report details preparation of pre-training, domain-specific, SFT, and RLHF data to support analysis and future model training.
- The model uses COOL RLHF to reconcile conflicting human preferences and improve subjective dialogue evaluations.
2 Infrastructure
InternEvo is the training framework used across pretraining, SFT, and RLHF, combining distributed parallelism, sharding, communication-computation overlap, and long-sequence memory management. It is designed to scale training efficiently across GPUs and very long contexts.
- InternEvo combines data, tensor, sequence, and pipeline parallelism with ZeRO strategies to scale training across thousands of GPUs.
- 64% MFU is achieved on 8 GPUs, while 53% MFU is maintained on 1024 GPUs for InternLM-7B with a global batch size of 4 million tokens.
- Nearly 88% MFU is achieved when training InternLM-7B with a sequence length of 256,000 tokens, compared with about 65% for DeepSpeed-Ulysses and Megatron-LM.
- InternEvo reduces distributed-training overhead through adaptive sharding and by overlapping parameter communication with computation.
- Its execution simulator searches parallelization plans using training scale, sequence length, model size, and batch size, supporting contexts up to 1 million tokens during training.
- Grouped-Query Attention is selected for high-speed, low-memory inference with very long contexts.
3 Pre-train
InternLM2’s pre-training uses diverse, filtered text, code, and long-context data. The process combines source-specific quality control, iterative code annotation, and staged long-context filtering before enhancement training.
- Text Data: Pre-training data spans web pages, papers, patents, and books, which are standardized, categorized, and stored in JSON Lines format.
- Text Data: The text-data pipeline applies heuristic filtering, Locality-Sensitive Hashing deduplication, safety filtering, and source-specific quality filtering.These stages produce formatted, clean, deduplicated, safe, and high-quality pre-training data.
- Code Data: Code quality filtering combines rule-based and model-based scorers, using model-based scoring only where predictions align well with human evaluations across languages.The authors report that code style is unreliable as a quality metric and that more labeled data did not substantially improve scorer accuracy.
- Code Data: An iterative annotation workflow lets annotators verify high- and low-confidence scorer predictions, refine guidelines, and validate prior labels; three iterations finalized the scoring model.
- Long Context Data: Long-context data is filtered by length selection above 32K bytes, statistical anomaly detection, and perplexity differences between adjacent text segments.The selected long-context data is a subset of the standard pre-training corpus and is therefore learned at least twice.
- Long Context Data: Perplexity filtering removes samples where preceding context distracts from predicting subsequent text, targeting improperly joined or incoherent documents rather than low perplexity itself.
4 Alignment
InternLM2’s alignment combines SFT with RLHF, including COOL RLHF to reconcile conflicting preferences and reduce reward hacking. The approach preserves long-context capability and supports tool use through specialized formatting.
- Alignment uses supervised fine-tuning and reinforcement learning from human feedback to make models follow instructions and align with human values.
- 10 million screened instruction instances cover conversation, NLP, mathematics, code generation, and function calls in ChatML format.
- COOL RLHF uses a conditional reward model to reconcile diverse preferences and multi-round online RLHF to mitigate reward hacking.
- The conditional reward model uses system prompts to model multiple preference domains within one reward model instead of relying on multiple preference models.
- The reward-model objective combines focal ranking loss with a logarithmic barrier penalty, weighted by λ set to 0.02.
- A modified ChatML format adds an environment role and keywords for code interpretation and external plugins to support general tool calling.
5 Evaluation and Analysis
The evaluation analyzes InternLM2 across downstream tasks and alignment, covering six dimensions from comprehensive examinations to tool utilization. Results are reported separately for base and chat models across multiple benchmark families.
- Evaluation is organized into downstream tasks and alignment, with performance assessed across six dimensions including language, reasoning, coding, long-context modeling, and tool utilization.
- Comprehensive Examination: Comprehensive-examination evaluation includes MMLU, CMMLU, C-Eval, AGIEval, and GAOKAO-Bench with specified zero-shot or five-shot protocols.
- Comprehensive Examination: Tables 5 and 6 compare base and chat models on comprehensive examination, marking overall and similar-parameter-group leaders.
- Comprehensive Examination: InternLM2 performs well among similarly sized base models, with larger gains over InternLM2-Base on AGIEval and GAOKAO than on other datasets.
- Comprehensive Examination: Chat-model comparison indicates that COOL RLHF has little impact on comprehensive-examination performance for InternLM2-Chat-7B.
- Language & Knowledge: Language-and-knowledge evaluation reports base and chat results in Tables 7 and 8 and uses reading comprehension, translation, and question-answering datasets.
- Language & Knowledge: FLORES evaluates English-to-100-language and reverse translation directions using BLEU on 100 samples per translation pair with 8-shot prompting.
Evaluation Results
InternLM2 is evaluated on language understanding, knowledge application, reasoning, and mathematics using diverse benchmark types and prompting protocols. The reported language-and-knowledge results indicate a competitive edge among the evaluated models.
- InternLM2 demonstrates a competitive edge on language-understanding and knowledge-application tasks, supporting applications requiring robust comprehension and extensive knowledge.
- Reasoning: The benchmark suite includes WinoGrande, HellaSwag, BBH, GSM8K-Test, MATH, TheoremQA, and MathBench.
- Mathematics: MathBench contains 3709 progressively challenging bilingual theoretical and application-oriented questions tagged by fine-grained knowledge points.
- Reasoning: Table 9 reports the comparison of reasoning tasks.
- Evaluation Protocol: Multiple-choice reasoning tests primarily use zero-shot evaluation, while open-ended GSM8K, MATH, and MathBench tasks primarily use few-shot prompting.
- Reasoning: Reasoning evaluation compares InternLM2 base and chat models across commonsense, inference, and mathematical problem-solving tasks.
Evaluation Results
InternLM2 models show strong performance across reasoning, mathematics, and chat evaluations, with capability varying by parameter scale and model stage. Domain-enhanced knowledge and COOL RLHF are associated with improved task performance within the reported comparisons.
- InternLM2-7B outperforms Mistral-7B-v0.1 on WinoGrande by 9.4 points, scoring 84.7 versus 75.3.
- InternLM2-7B scores 70.8 on GSM8K, surpassing ChatGLM3-6B-Base by 10.1 percentage points.
- InternLM2-20B outperforms all tested base models in basic arithmetic and theorem proving at the 13∼20B scale.
- InternLM2-Chat leads on GSM8K, MATH, and TheoremQA at both the 7B and 20B scales.
Python Coding Tasks
The coding evaluation uses Python programming benchmarks spanning standard code generation tasks and entry-level programming problems. These datasets provide complementary tests of code-generation capability.
- HumanEval contains 164 carefully crafted Python programming tasks with functions and docstrings.
- MBPP contains 974 programming tasks solvable by entry-level programmers, including numeric manipulation and external-knowledge problems.
Multiple Programming Language Coding Tasks
InternLM2 is evaluated on multilingual coding, broad capability dimensions, long-context benchmarks, retrieval, and tool-assisted mathematical tasks. Reported results show leading code-generation performance, strong long-context modeling, and gains from tool use in most MathBench cases.
- Multiple Programming Language Coding Tasks: HumanEval-X extends HumanEval to C++, Java, JavaScript, Go, and Python, producing 820 problem-solution pairs.
- Multiple Programming Language Coding Tasks: The coding evaluation covers MBPP, HumanEval, MBPP-CN, and multilingual HumanEval-X.
- Multiple Programming Language Coding Tasks: InternLM2-Chat-20B surpasses the previous state-of-the-art by more than 10% on HumanEval, MBPP, and MBPP-CN.
- Multiple Programming Language Coding Tasks: InternLM2-Chat-20B surpasses InternLM2-Chat-7B on MBPP-CN but declines slightly on HumanEval-X, possibly reflecting Chinese-focused tuning.
- Capability-Specific Enhancement: Capability-specific enhancement training consistently improves InternLM2 relative to its counterpart across averaged evaluation capabilities.
- Capability Evaluation: InternLM2 evaluates coding, reasoning, question answering, and examination capabilities across the listed benchmark groups.
- Long-Context Understanding and Reasoning: InternLM2-Chat-20B-SFT performs best on L-Eval, while InternLM2-Chat-7B-SFT scores 48.1 on LongBench.
- Tool Utilization: Code-interpreter use substantially improves GSM8K, MATH, and MathBench performance in most reported cases, with especially notable gains on MATH.
5.3 Performance on Alignment
InternLM2 is evaluated for subjective alignment across English and Chinese benchmarks, instruction following, and reward-model design. The reported results generally place InternLM2 at or near the leading performance levels, with RLHF improving alignment over SFT in several comparisons.
- Subjective Alignment: InternLM2’s alignment performance reaches SOTA or near-SOTA results across multiple subjective benchmarks.
- English Subjective Evaluation: InternLM2-20B achieves a 21.8 win rate on AlpacaEval, the highest SOTA result among compared models.
- English Subjective Evaluation: InternLM2 achieves leading MTBench scores in both the 7B and 13∼20B parameter phases.
- Chinese Subjective Evaluation: InternLM2 records CompassArena win rates of 28.7 for 7B and 31.4 for 20B, while RLHF substantially improves performance over SFT.
- Chinese Subjective Evaluation: InternLM2 achieves AlignBench scores of 6.1 and 6.8 for its 7B and 20B versions, exceeding GPT-3.5’s 5.7.
- Instruction Following: InternLM2 ranks second and third in the 7B and 13-20B phases on IFEval, scoring 48.5 and 48.7 respectively.
- COOL RLHF Analysis: Conditional system prompts improve reward-model precision across several public datasets compared with omitting those prompts.
5.4 Discussion on Data Contamination
The report evaluates base-model contamination on GSM8K by comparing losses across training, test, and GPT-4-generated reference samples. Two loss differences diagnose potential leakage and training-split overfitting.
- Table 20 evaluates base-model contamination on GSM8K.
- The evaluation compares language-model loss on GSM8K training, test, and GPT-4-generated reference samples.The reference set is designed to mimic GSM8K.
- ∆1 = Ltest − Lref indicates potential test-data leakage, with lower values suggesting possible leakage.
- ∆2 = Ltest − Ltrain measures training-split overfitting, with higher values implying excessive overfitting.
6 Conclusion
InternLM2 is presented as an open-source family trained on over 2T tokens across multiple model sizes and extended to 32k contexts. The report also releases training-stage checkpoints and documents data and alignment practices, including Conditional Online RLHF for preference conflicts.
- InternLM2 is trained on over 2T of high-quality pre-training corpora in 1.8B, 7B, and 20B sizes.The model is intended for a variety of scenarios.
- InternLM2 supports long contexts through GQA, reduced inference costs, and additional training on contexts up to 32k.
- The report releases checkpoints from various training phases to facilitate future research on model development.
- The training description covers text, code, long-text, and alignment data, while Conditional Online RLHF addresses conflicting preferences during RLHF.
A.2 Prompts for Evaluation
This appendix collects prompts and system instructions used for evaluation and tool-use examples. It includes prompts for benchmark tasks, subjective evaluation, and external-tool interaction, alongside a function-call streaming-format figure.
- Figure 17 illustrates InternLM2-Chat’s streaming format for function calls, especially JSON for downstream applications.
- The appendix includes system prompts for assistants using IPythonInterpreter or other external tools.
- Figures 19–21 show prompts used in CIBench, MATH, and GSM8K evaluations.
- A separate system prompt describes selecting LLMs by response quality to construct a leaderboard of accurate and human-preferred answers.
- Figures 22–23 show prompts used in AlpacaEval and MTBench subjective evaluations.