InternLM2 Technical Report

1 Introduction

• COOL RLHF uses a conditional reward model to reconcile conflicting preferences and multi-round PPO to mitigate reward hacking, enhancing subjective dialogue evaluations.The report also preliminarily compares RLHF’s subjective and objective results.Source: §1 · pp. 3–4
• InternLM2 models in 1.8B, 7B, and 20B sizes perform well in subjective and objective evaluations, and the report releases pre- and post-SFT/RLHF stages.The stage releases are named InternLM2-Chat-{size}-SFT and InternLM2-Chat-{size}, respectively.Source: §1 · pp. 3–4
• At 200k context, InternLM2 nearly perfectly identifies all “needles” in the “Needle-in-a-Haystack” experiment.Training begins with 4k-context texts and continues with high-quality 32k texts; Group Query Attention reduces memory use for long-sequence inference, and positional encoding extrapolation supports the 200k-context test.Source: §1 · pp. 3–4
• InternLM2 details preparation of pre-training, domain-specific enhancement, SFT, and RLHF data, addressing an area that earlier LLM technical reports seldom covered.Its pre-training data preparation covers text, code, and long-context data.Source: §1 · pp. 3–4

2.1 InternEvo

• InternEvo combines data, tensor, sequence, and pipeline parallelism with ZeRO strategies, FlashAttention, and BF16 mixed-precision training.These techniques support scaling across thousands of GPUs while reducing memory footprint and improving hardware utilization.Source: §2.1 · p. 4
• At 1024 GPUs, InternEvo achieves 53% MFU versus DeepSpeed’s around 36% when training InternLM-7B.With 8 GPUs and a global batch size of 4 million tokens, InternEvo achieves 64% MFU. DeepSpeed’s 1024-GPU result uses ZeRO-1 and MiCS.Source: §2.1 · p. 4, Figure 1 · p. 5
• At 256,000 tokens, InternEvo achieves nearly 88% MFU on InternLM-7B, versus about 65% for DeepSpeed-Ulysses and Megatron-LM.The report says this training-performance improvement also holds for models with 30 billion or 70 billion parameters.Source: §2.1 · p. 4
• InternEvo uses adaptive Full-Replica, Full-Sharding, and Partial-Sharding to distribute model states and optimize sharding factors under GPU memory constraints.It overlaps parameter prefetch and gradient synchronization with backward computation, and overlaps optimizer-state parameter broadcasts with the next step’s forward computation.Source: §2.1 · p. 5
• InternEvo automatically searches parallelization plans and supports training contexts up to 1 million tokens.It analyzes memory and communication costs across four parallel dimensions and three sharding dimensions; a unified memory pool and defragmentation reduce fragmentation and out-of-memory risk.Source: §2.1 · pp. 5–6
• InternEvo’s fault-tolerant system diagnoses failures and recovers automatically, while asynchronous checkpointing limits lost training progress.It saves model states locally before uploading them to distributed storage, moves temporary checkpoints to cold storage, and can resume with a changed parallelization configuration.Source: §2.1 · p. 6
• InternEvo’s RLHF framework, built on InternEvo and Ray, coordinates four equally sized PPO models, two of which are trained.It supports multiple LLM execution engines and diverse algorithm designs.Source: §2.1 · p. 6

2.2 Model Structure

• InternLM2 follows LLaMA’s structural design, consolidates Wk, Wq, and Wv for over 5% pretraining acceleration, and uses GQA for fast, low-memory long-context inference.Interleaving each head’s matrices supports tensor-parallel size changes by splitting or concatenating along the last dimension.Source: §2.2 · pp. 6–7

3.1.1 Text Data

• Chinese and English web pages account for 86.46% of the pre-training data; text sources are standardized, filtered, deduplicated, safety-screened, and quality-filtered.The pipeline produces clean, safe, and high-quality data from sources including web pages, papers, patents, and books.Source: §3.1.1 · pp. 7–8

3.1.2 Code Data

• Code quality scoring combines heuristic and model-based filters, but uses model scores only for languages where predictions align well with human judgments.An iterative annotation process took three iterations; code files from each repository are also regrouped and dependency-sorted so the model can learn cross-file dependencies.Source: §3.1.2 · pp. 10–11

3.1.3 Long Context Data

• Long-context data filtering selects samples over 32K bytes, removes statistical anomalies, and uses perplexity differences to exclude distracting context.This data is a subset of the standard pre-training corpus and is therefore learned at least twice; thresholds are tailored to domains and languages rather than made universal.Source: §3.1.3 · p. 12

3.2 Pre-training Settings

• InternLM2’s vocabulary combines 60,004 selected cl100k tokens, 32,397 Chinese tokens, and 147 spare tokens.Training used AdamW with β1 = 0.9, β2 = 0.95, ϵ = 1e −8, and weight decay = 0.1, with cosine decay to 10% of the maximum learning rate.Source: §3.2.1 · p. 13, §3.2.2 · p. 13

3.3 Pre-training Phases

• 2.0T to 2.6T tokens trained InternLM2 models through three phases: up to 4K context, mixed 4K/32K context, then capability-specific enhancement data.About 90% of training steps used sequences up to 4096 tokens; the 32K phase accounted for about 9% of steps, retained 50% shorter-than-4096-token data, and increased the RoPE base from 50,000 to 1,000,000.Source: §3.3 · p. 13, §3.3.1 · p. 14, §3.3.2 · p. 14

3.3.3 Capability Specific Enhancement Training

• After enhancement training on 24 billion curated tokens, InternLM2 showed substantial improvements in coding, reasoning, question answering, and examinations.The enhancement corpus mixed retrieved and open-source data; the authors released checkpoints from before and after this phase as InternLM2-{size}-Base and InternLM2-{size}.Source: §3.3.3 · p. 14

4.1 Supervised Fine-Tuning

• SFT uses 10 million screened instruction instances spanning topics such as conversation, mathematics, code, and function calls, formatted as ChatML.The 7B and 20B models train for one epoch with AdamW and an initial learning rate of 4e-5.Source: §4.1 · p. 15

4.2 COOL Reinforcement Learning from Human Feedback

• COOL RLHF addresses conflicting human preferences and reward hacking with a conditional reward mechanism and multi-round online feedback.The reward model dynamically allocates attention across preferences according to conditions, while online updates help the model adapt to new feedback.Source: §4.2 · pp. 15–16

4.2.1 Conditional Reward Model

• The Conditional Reward Model uses preference-specific system prompts to model diverse preferences in one reward model.Its dataset includes up to 2.4 million binarized preference pairs. A focal ranking loss emphasizes difficult samples, and a logarithmic barrier confines reward scores to -5 to 5; the penalty weight λ defaults to 0.02.Source: §4.2.1 · pp. 16–17

4.2.2 Online RLHF

• Online RLHF pairs a fast path for targeted reward-hacking fixes with a slow path for improving reward-model reliability and robustness.The fast path uses 20–100 preference pairs per detected pattern; the slow path collects human preferences over responses from SFT and early- and late-stage PPO models, reusing accumulated preferences to reduce annotation delays. The implementation conducts three refinement rounds with thousands of fast-path patches and online preference data.Source: §4.2.2 · pp. 17–18

4.2.3 PPO Training Details

• PPO training uses actor, critic, reference, and reward models, iterates over about 200k queries in about 400 iterations, and selects checkpoints using validation sets.The actor and reference start from SFT weights; the critic starts from the reward model and receives 50 iterations of pre-training. Although this initialization has larger losses initially, after about 20 iterations it shows lower losses and leads to higher actor rewards than SFT initialization. A pre-train loss coefficient of 0.5 and pre-training data volume about 50% of PPO training are used to mitigate catastrophic forgetting.Source: §4.2.3 · pp. 18–20

4.3 Long-Context Finetuning

• Long-context data from books and GitHub repositories is retained during SFT and RLHF, and long-context code data improves both long-context and code capabilities.Code data is built from repositories referencing DS-1000 core libraries, with processed repository content concatenated to 32k tokens.Source: §4.3 · p. 20

4.4 Tool-Augmented LLMs

• Tool calling uses a modified ChatML format with an environment role and dedicated interpreter and plugin keywords, supporting varied tools and AI environments.For code-interpreter math problems, InternLM2-Chat uses reasoning interleaved with coding and iterative hard-example mining.Source: §4.4 · pp. 20–21

5.1 Overview

• Evaluations use OpenCompass unless specified otherwise and cover downstream tasks, alignment, and potential data-contamination effects on performance and reliability.The paper divides downstream tasks and alignment into subtasks to examine model strengths and weaknesses.Source: §5.1 · p. 21

5.2 Performance on Downstream Tasks

• Downstream-task assessment spans six dimensions: comprehensive examinations, language and knowledge, reasoning and mathematics, coding, long-context modeling, and tool utilization.The supplied overview introduces datasets, experimental setup, results, and comparisons with state-of-the-art methods.Source: §5.2 · p. 21

5.2.2 Language and Knowledge

• Language-and-knowledge evaluation includes open-domain QA and reading comprehension benchmarks, including Chinese C3 and the high-school subset of RACE.The QA benchmarks include TriviaQA and NaturalQuestions; the described protocols report 0-shot results.Source: §5.2 · p. 21, §5.2.2 · p. 24
• FLORES evaluation measures English-to-100-language and reverse translation on 100 samples per task pair using BLEU, with 8-shot results.FLORES is described as covering 101 languages.Source: §5.2 · p. 21, §5.2.2 · p. 24

5.2.1 Comprehensive Examination

• Comprehensive-examination benchmarks include 5-shot MMLU, CMMLU, and C-Eval, plus 0-shot AGIEval and objective-question GAOKAO-Bench.AGIEval draws on admission and qualification exams; GAOKAO-Bench covers China's college entrance exam from 2010 to 2022.Source: §5.2.1 · p. 21

Evaluation Results

• InternLM2 base models perform well among similar-sized models, with 7B and 20B showing significant increases over InternLM2-Base, especially on AGIEval and GAOKAO relative to other datasets.The paper attributes the exam gains to pre-training on general-domain and domain-enhanced corpora.Source: Table 5 · p. 22, Evaluation Results · pp. 22–23
• InternLM2 chat models excel among similar-sized models, while COOL RLHF has little impact on comprehensive-examination performance in the 7B SFT comparison.The comparison is between InternLM2-Chat-7B-SFT and InternLM2-Chat-7B.Source: Table 6 · p. 22, Evaluation Results · p. 23

Evaluation Results

• InternLM2 is described as having a competitive edge in language-understanding and knowledge-application tasks, with base and chat results reported in Tables 7 and 8.The passage attributes this edge to its rigorous, high-quality training corpus.Source: Evaluation Results · p. 24

5.2.3 Reasoning and Mathematics

• Multiple-choice reasoning and mathematics questions are evaluated predominantly zero-shot, while open-ended questions use few-shot prompting; base-model multiple-choice evaluation primarily uses perplexity.Source: §5.2.3 · p. 25
• Reasoning and mathematics performance is presented separately for Base and Chat models in Table 9.The passage introduces reasoning ability as relevant to complex problem-solving and decision-making.Source: §5.2.3 · p. 25, Table 9 · p. 25

Evaluation Results

• 84.7 on WinoGrande puts InternLM2-7B 9.4 points above Mistral-7B-v0.1 (75.3) and less than one point below InternLM2-20B (85.2).InternLM2-20B achieves the best overall performance among tested base models and averages a 10.4% increase over InternLM2-20B-Base across tested reasoning datasets.Source: Evaluation Results · p. 25
• InternLM2-Chat-20B exceeds GPT-3.5 on HellaSwag situational reasoning (↑15.6) and BBH comprehensive reasoning (↑26.4), while 7B models outperform most 13∼20B models on three test sets.RL and SFT models show similar effects on the advanced reasoning capabilities.Source: Evaluation Results · p. 25
• 70.8 on GSM8k puts InternLM2-7B 10.1 percentage points above ChatGLM3-6B-Base (60.7) and nearly twice InternLM2-7B-Base (36.0).InternLM2-7B also scores 10.5 on TheoremQA, above Qwen-14B-Base (10.4), and performs strongly in both languages on MathBench.Source: Evaluation Results · pp. 25–26, Table 10 · p. 26
• InternLM2-Chat leads on GSM8K, MATH, and TheoremQA at both 7B and 20B scales, with COOL RLHF-trained InternLM2-Chat-20B leading across all reported math metrics.InternLM2-Chat-20B also outperforms GPT-3.5 and Mixtral-8x7B-Instruct-v0.1 and performs strongly on bilingual MathBench.Source: Table 11 · p. 26, Evaluation Results · p. 27
• InternLM2-20B outperforms all tested base models in basic arithmetic and theorem proving, while Qwen-14B-Base excels in problem-solving and English and Chinese MathBench tests.At 7B, InternLM2-7B leads on the reported math assessments, though it is slightly behind on MATH.Source: Evaluation Results · p. 26

Python Coding Tasks

• HumanEval (Chen et al., 2021) contains 164 Python programming tasks, each pairing a function with a docstring, and is used to evaluate code generation.Its human-written code also supports assessment of models’ program generation and completion capabilities.Source: Python Coding Tasks · p. 27
• MBPP (Austin et al., 2021) comprises 974 entry-level programming tasks, and the study uses a hand-verified subset of the dataset.Tasks range from numeric manipulations to problems involving external knowledge, such as defining integer sequences.Source: Python Coding Tasks · p. 27

Multiple Programming Language Coding Tasks

• InternLM2-Chat-20B surpasses the previous state-of-the-art by more than 10% on HumanEval, MBPP, and MBPP-CN.It improves over InternLM2-Chat-7B on MBPP-CN but performs slightly worse on HumanEval-X, possibly because of fine-tuning for Chinese. HumanEval-X covers code generation and translation across C++, Java, JavaScript, Go, and Python.Source: Multiple Programming Language Coding… · p. 27

5.2.5 Performance Before and After Enhancement Training

• Capability-specific enhancement training consistently improves InternLM2 performance, and SFT models trained with enhancement perform better across capability dimensions.Figure 12 compares performance before and after enhancement training, with scores averaged across multiple evaluation sets; the report says later SFT results use base models before enhancement training.Source: §5.2.5 · pp. 27–29, Figure 12 · p. 29

5.2.6 Long-context Modeling

• InternLM2-Chat-20B-SFT performs best on L-Eval, while InternLM2-Chat-7B-SFT scores 48.1 on LongBench, slightly below ChatGLM3-6B’s 48.4.The 7B-SFT model outperforms other models of at most 7B across four of LongBench’s six subtask categories; parameter size does not lead to significant long-context performance differences for InternLM2.Source: §5.2.6 · pp. 29–30, Table 15 · p. 30

5.2.7 Tool Utilization

• Using an external code interpreter with ReAct substantially improves coding and mathematics results, with a notably large enhancement on MATH.On MathBench, the interpreter improves InternLM2 performance in most cases, with minor decreases attributed to incorrect interpreter use; improvements are noted in Knowledge for InternLM2-20B-Chat and Application for InternLM2-7B-Chat.Source: §5.2.7 · p. 31
• At identical model scale, InternLM2-Chat-7B is the top performer on T-Eval and CIBench, while InternLM2-Chat-20B is competitive on T-Eval and leads CIBench.The InternLM2 series also achieves impressive results in Chinese.Source: §5.2.7 · p. 31

5.3 Performance on Alignment

• InternLM2’s overall alignment-task performance reaches SOTA or near-SOTA across multiple benchmarks, with subjective outputs reported as consistent with human preferences.The section compares SFT and RLHF models; benchmark-specific results are detailed below.Source: §5.3 · pp. 32–33, Table 17 · p. 33

5.3.1 English Subjective Evaluation

• AlpacaEval gives InternLM2-20B a 21.8 win rate, highest among compared models; MTBench scores are 7.7 (7B) and 7.9 (20B), leading within both sizes.On AlpacaEval, the RLHF model also outperforms the SFT model in win rate.Source: §5.3.1 · p. 33, Table 17 · p. 33

5.3.2 Chinese Subjective Evaluation

• CompassArena win rates are highest for InternLM2 at 28.7 (7B) and 31.4 (20B), with category results showing Chinese creativity and language win rates rivaling GPT4-Turbo.The 7B–20B gap is described as relatively small, while RLHF significantly improves over SFT, suggesting gains beyond model size.Source: §5.3.2 · pp. 33–34, Figure 16 · p. 34
• On AlignBench (Liu et al., 2023a), InternLM2 scores 6.1 (7B) and 6.8 (20B), above GPT-3.5’s 5.7; both sizes improve significantly after RLHF.InternLM2 excels in question-answering and role-playing, while mathematics and reasoning remain areas for improvement.Source: Table 17 · p. 33, §5.3.2 · p. 34, Table 18 · p. 34

5.3.3 Instruct Following Evaluation

• On IFEval (Zhou et al., 2023), InternLM2 ranks second among 7B models at 48.5 and third in the 13–20B group at 48.7, remaining among similarly sized leaders.The reported average combines prompt-level strict and loose accuracy with instance-level strict and loose accuracy.Source: Table 17 · p. 33, §5.3.3 · pp. 34–35

5.3.4 Ablation Study of Conditional Reward Model

• The reward model achieves markedly higher precision with conditional system prompts than without them across public datasets, including helpfulness, harmlessness, summaries, math, and Reddit replies.The comparison includes an in-house 7B reward model trained with and without conditional prompts, alongside UltraRM-13B and QwenRM.Source: §5.3.4 · p. 35, Table 19 · p. 35

5.4 Discussion on Data Contamination

• Table 20 evaluates base models for GSM8K contamination using language-modeling loss on concatenated question-and-answer samples.The evaluation compares losses on the training split, test split, and a GPT-4-generated reference set designed to mimic GSM8K.Source: Table 20 · p. 35, §5.4 · pp. 35–36
• A lower ∆1 = Ltest − Lref indicates possible test-data leakage, while a higher ∆2 = Ltest − Ltrain indicates excessive training-split overfitting.Outliers for both ∆1 and ∆2 are highlighted in gray.Source: §5.4 · p. 36

6 Conclusion

• InternLM2, in 1.8B, 7B, and 20B sizes, was trained on over 2T of high-quality pre-training corpora and up to 32k contexts, using GQA to reduce inference costs.The report describes the model as performing exceptionally in subjective and objective evaluations.Source: §6 · p. 36
• The report details its training framework and pre-training and alignment data, and releases checkpoints from multiple training phases to support future research.The documented data includes text, code, and long-text pre-training data.Source: §6 · p. 36
• Conditional Online RLHF is proposed to harmonize different preferences encountered during the RLHF process.The report says this account may inform pre-training data preparation and more effective training of larger models.Source: §6 · p. 36

• Figure 17 illustrates InternLM2-Chat’s streaming format for function calls, with JSON formatting intended to ease downstream applications.Source: Figure 17 · p. 47
• The IPythonInterpreter prompt specifies Python code execution and requires tool calls to follow a prescribed format.Source: §A.2 · p. 49
• CIBench, MATH, and GSM8K prompts use a system instruction for an assistant with external tools and a prescribed tool-call format.Source: Figure 19 · p. 49, §A.2 · pp. 50–51, Figure 20 · p. 50
• AlpacaEval’s prompt asks an evaluator to select the best LLM responses by quality, aiming to create a leaderboard of accurate and human-preferred answers.Source: §A.2 · p. 52, Figure 22 · p. 52
• Figure 23 presents the prompt used in MTBench.Source: Figure 23 · p. 53