Qwen Technical Report

Model series

• QWEN comprises pretrained base models, human-aligned chat models using SFT and RLHF, and specialized models for coding and mathematics.The series includes models with varying parameter counts and is presented as a response to concerns about reproducibility, steerability, and accessibility.Source: §1 · p. 3

Base models

• Up to 3 trillion tokens of diverse text and code enable QWEN base models to consistently outperform substantially larger counterparts across downstream tasks.The training data span broad areas and include both texts and codes.Source: §1 · p. 4

Chat models

• RLHF-trained QWEN-CHAT models produce human-preferred responses and remain highly competitive, although they still trail GPT-4 on the reported benchmark.SFT models achieve superior benchmark performance, while reward models trained to mimic human preference support the RLHF process.Source: §1 · p. 4

Coding models

• CODE-QWEN models, pretrained on extensive code datasets and fine-tuned for code conversations, demonstrate high proficiency in code understanding and generation.The family includes CODE-QWEN-7B, CODE-QWEN-14B, and corresponding chat models, evaluated on HumanEval, MBPP, and HumanEvalPack.Source: §1 · p. 4

Mathematics models

• MATH-QWEN-7B-CHAT and MATH-QWEN-14B-CHAT outperform same-size open-source models by large margins and approach GPT-3.5 on GSM8K and MATH.These models are specifically designed for mathematical problem solving.Source: §1 · p. 4

Vision-language models

• QWEN-VL and QWEN-VL-CHAT outperform current open-source vision-language models while supporting bilingual text recognition, visual grounding, multi-image conversations, and storytelling.These models comprehend visual and language instructions in both Chinese and English; further details are attributed to Bai et al. (2023).Source: §1 · p. 4

2.1 DATA

• QWEN pretraining uses up to 3 trillion tokens from multilingual public web documents, encyclopedias, books, and code, with instruction data filtered against evaluation-set overlap.The data pipeline applies normalization, exact and fuzzy deduplication, rule- and model-based quality filtering, manual review, and selective up-sampling.Source: §2.1 · pp. 5–6

2.2 TOKENIZATION

• QWEN uses BPE tokenization with an approximately 152K vocabulary expanded for Chinese and other languages, achieving higher compression efficiency than competing tokenizers in most languages.Increasing vocabulary size did not reduce downstream pretrained-model performance in preliminary experiments.Source: §2.2 · p. 6

2.3 ARCHITECTURE

• QWEN modifies the LLaMA-style Transformer with untied embeddings, FP32 RoPE frequencies, attention-layer QKV biases, pre-normalization with RMSNorm, and SwiGLU activation.RMSNorm preserves equivalent performance while improving efficiency, and GLU-based activations generally outperform GeLU in initial experiments.Source: §2.3 · pp. 6–7

2.4 TRAINING

• QWEN is pretrained autoregressively with 2048-token contexts, Flash Attention, AdamW, cosine learning-rate decay, and BFloat16 mixed precision.The optimizer uses β1 = 0.9, β2 = 0.95, and ϵ = 10−8, with the learning rate decayed to 10% of its peak.Source: §2.4 · p. 7

2.6 EXPERIMENTAL RESULTS

• QWEN-14B outperforms previous 13B state-of-the-art open-source models on all reported benchmark datasets, while QWEN-14B also surpasses LLaMA2-70B on three tasks.QWEN-7B surpasses LLaMA2-13B and matches Baichuan2-13B comparably, while QWEN-1.8B remains competitive and sometimes outperforms larger models.Source: Table 2 · p. 8, §2.6 · p. 9
• Evaluation covers seven zero-shot and few-shot benchmarks spanning language understanding, knowledge, reasoning, mathematics, and coding for QWEN base models against recent open-source baselines.The benchmarks are MMLU, C-Eval, GSM8K, MATH, HumanEval, MBPP, and BBH, using the listed task-specific shot settings.Source: §2.6 · pp. 8–9

2.5 CONTEXT LENGTH EXTENSION

• Combining NTK-aware interpolation, LogN-Scaling, and layer-wise window assignment maintains model performance beyond 8192-token contexts.These methods are training-free inference techniques that adjust positional scaling and attention windows, with shorter windows for lower layers and longer windows for higher layers.Source: §2.5 · p. 8, §2.6 · p. 9

3.1 SUPERVISED FINETUNING

• SFT finetunes pretrained LLMs on diverse human-style chat conversations, including queries and responses, with safety annotations, ChatML-style formatting, next-token prediction, and loss masks on system and user inputs.Source: §3.1 · p. 9, §3.1.1 · p. 10, §3.1.2 · p. 10

3.2 REINFORCEMENT LEARNING FROM HUMAN FEEDBACK

• RLHF addresses SFT’s limited generalization and creativity and overfitting by training a reward model and optimizing the policy with PPO.The reward model is pretrained and finetuned on response-comparison data, with higher-quality annotations used during finetuning.Source: §3.2 · p. 10, §3.2.1 · p. 10

3.2.1 REWARD MODEL

• The reward model uses diverse, complexity-balanced prompts and responses from QWEN variants, while preference-model pretraining generalizes out of distribution and reward modeling improves on QWEN reward datasets.Table 4 reports pairwise accuracy for PMP and reward models on diverse human-preference benchmarks.Source: §3.2.1 · pp. 10–11, Table 4 · p. 11

3.2.2 REINFORCEMENT LEARNING

• PPO samples two responses per query, uses KL regularization and reward normalization, and benefits from pretrained gradients that mitigate alignment tax when pretrained data greatly exceeds PPO data.An overly large pretrained-gradient coefficient can impede alignment to the reward model, whereas an overly small coefficient has only marginal effect on alignment-tax reduction.Source: §3.2.2 · p. 11

3.3 AUTOMATIC AND HUMAN EVALUATION OF ALIGNED MODELS

• QWEN-14B-Chat outperforms all models except ChatGPT and LLAMA 2-CHAT-70B across MMLU, C-Eval, GSM8K, HumanEval, and BBH, with especially strong HumanEval performance.The comparison reports both zero-shot and few-shot settings in Table 5.Source: §3.3 · pp. 11–12, Table 5 · p. 12
• RLHF significantly outperforms SFT models in human preference evaluation, although the RLHF model remains behind GPT-4.Figure 4 reports win, tie, and loss rates against GPT-3.5 for Qwen-7B SFT, Qwen-14B SFT, Qwen-14B RLHF, and GPT-4 on 300 Chinese instructions.Source: §3.3 · p. 12, Figure 4 · p. 13

3.4 TOOL USE, CODE INTERPRETER, AND AGENT

• QWEN-7B-CHAT and QWEN-14B-CHAT significantly surpass similar-scale open-source alternatives on code-interpreter evaluations spanning math, visualization, and general-purpose tasks.The benchmark evaluates code executability and correctness, including multi-step Visualization-Hard tasks requiring sequential planning.Source: Table 7 · p. 14, Table 8 · p. 14, §3.4 · p. 16
• QWEN supports unseen-tool selection through ReAct prompting and performs well as a Hugging Face Agent, slightly behind GPT-4 while exceeding other open-source alternatives.Tool-use accuracy generally increases with model size, but gains plateau for selecting tools and supplying arguments; the preliminary benchmark may be too easy.Source: Table 6 · p. 13, §3.4 · pp. 15–16, Table 9 · p. 15

4 CODE-QWEN: SPECIALIZED MODEL FOR CODING

• The authors develop CODE-QWEN and CODE-QWEN-CHAT from QWEN base models, offering 14B and 7B parameter versions for coding tasks.The models target coding, debugging, and interpretation through continued pretraining and supervised fine-tuning.Source: §4 · p. 16

4.1 CODE PRETRAINING

• Starting from QWEN base language models, continued pretraining on around 90 billion code tokens trains CODE-QWEN models for contexts up to 8192 tokens.The learning rates are 6.0 × 10^-5 for CODE-QWEN-14B and 3.0 × 10^-5 for CODE-QWEN-7B.Source: §4.1 · p. 17

4.2 CODE SUPERVISED FINE-TUNING

• Multi-stage supervised fine-tuning yields the best performance among the tested methods for CODE-QWEN-CHAT.The 14B and 7B models use learning rates of 2.0 × 10^-6 and 1.0 × 10^-5, respectively, with cosine scheduling and 3% warm-up.Source: §4.2 · p. 17

4.3 EVALUATION

• CODE-QWEN and CODE-QWEN-CHAT outperform similarly sized baselines and rival larger Starcoder models, but trail state-of-the-art GPT-4 on pass@1.Evaluations cover Humaneval, MBPP, and HUMANEVALPACK; the authors note these tests do not fully capture model strengths and weaknesses.Source: §4.3 · p. 17

5 MATH-QWEN: SPECIALIZED MODEL FOR MATHEMATICS REASONING

• MATH-QWEN-CHAT is a human-aligned mathematics-specialized series built on QWEN pretrained language models, released in 7B and 14B parameter versions.The assistant models are designed for arithmetic and mathematics reasoning.Source: §5 · p. 17

5.1 TRAINING

• The models are obtained through supervised fine-tuning on an augmented mathematics-instruction dataset, using masked system and user inputs to accelerate convergence.Training uses sequence length 1024, AdamW, a peak learning rate of 2 × 10^-5, and 50,000 training steps.Source: §5.1 · pp. 17, 20

5.2 EVALUATION

• MATH-QWEN-14B-CHAT and MATH-QWEN-7B-CHAT deliver stronger math reasoning and arithmetic performance than comparable open-source and QWEN-CHAT models, with the 7B model exceeding Minerva-8B on MATH.The 14B model approaches Minerva-62B and GPT-3.5 on GSM8K and MATH, while outperforming them on arithmetic ability and Chinese math problems.Source: §5.2 · p. 20
• Evaluation covers GSM8K, MATH, Math401, and Math23K, comparing MATH-QWEN-CHAT with proprietary and open-source mathematics models.The benchmarks measure grade-school and competition mathematics, arithmetic ability, and Chinese grade-school mathematics.Source: §5.2 · p. 20

6.1 LARGE LANGUAGE MODELS

• Transformer-based pretraining drove transfer-learning success as LLMs scaled from 100 million to over 10 billion parameters, while GPT-3 demonstrated few-shot and zero-shot learning.ChatGPT and GPT-4 established LLMs as effective human-facing assistants, and LLaMA and LLAMA 2 accelerated open-source development.Source: §6.1 · p. 20

6.2 ALIGNMENT

• Alignment methods address repetition, hallucination, and preference deviations, but instruction-data collection remains difficult despite self-instruct and expanding open-source chat datasets and models.SFT and RLHF dominate chat-model training; LoRA and Q-LoRA reduce memory demands, while PPO, RRHF, DPO, and PRO offer alternative optimization approaches.Source: §6.2 · pp. 20–21

6.3 TOOL USE AND AGENTS

• LLMs can plan tool calls, use APIs, external memory, and search engines, motivating agent frameworks and systems involving specialized, embodied, multimodal, or interacting agents.ReAct structures tool selection, observation incorporation, and response generation through iterative thoughts and API feedback.Source: §6.3 · p. 21

6.4 LLM FOR CODING

• Large and code-trained LLMs improve code understanding and generation over general-purpose models across generation, completion, translation, debugging, refinement, and question answering.Specialized alignment methods have also been developed for coding models such as Code Llama-Instruct and StarCoder.Source: §6.4 · p. 21

6.5 LLM FOR MATHEMATICS

• Math reasoning improves with chain-of-thought, scratchpad, self-consistency, least-to-most prompting, math pretraining, and instruction fine-tuning, although accuracy remains inconsistent.Math-specific instruction fine-tuning is described as more cost-effective than math-specific pretraining, while prompt engineering and simple scaling are limited by time, trial-and-error, or inefficiency.Source: §6.5 · p. 22

Conclusion

• QWEN matches some proprietary models and competes with existing open-source models on comprehensive benchmarks and human evaluation.The reported series includes 14B, 7B, and 1.8B-parameter models.Source: §7 · p. 22
• The series combines pretraining on trillions of tokens with SFT and RLHF, and includes CODE-QWEN, CODE-QWEN-CHAT, and MATH-QWEN-CHAT for coding and mathematics.The specialized models use domain-specific data to excel in their respective fields.Source: §7 · p. 22
• Open access to QWEN is intended to foster community collaboration, innovation, new research, and applications advancing language-model development.The authors also describe QWEN as a major milestone in their development of large language models.Source: §7 · p. 22

A.1.1 DATA FORMAT FOR QWEN-CHAT

• SFT and RLHF require a purpose-designed conversational format, even though they use a training task similar to autoregressive next-token prediction.The format supports building a conversational AI assistant model.Source: §A.1.1 · p. 36
• The human-assistant format uses “\n\nhuman: ” and “\n\nassistant: ” prefixes, helping base models transfer to conversational patterns but risking ambiguity because the phrases are ordinary words.Anthropic (Bai et al., 2022b) is identified as an early example of this format.Source: §A.1.1 · p. 36
• Qwen instead adopts OpenAI’s ChatML format, whose special tokens “<im_start>” and “<im_end>” do not appear during pretraining and resolve this ambiguity.The format is presented as the data format used for Qwen-Chat.Source: §A.1.1 · p. 36

A.2 EVALUATION

• QWEN’s automatic evaluation compares the model series with baselines across OpenCompass examination, language, knowledge, understanding, and reasoning categories.For baselines, the higher value between reported results and leaderboard results is used.Source: §A.2.1 · p. 36

A.2.1 AUTOMATIC EVALUATION

• Examination performance is reported on MMLU, C-Eval, CMMLU, AGIEval, and Gaokao-Bench.MMLU, C-Eval, and CMMLU use five-shot evaluation, whereas AGIEval and Gaokao-Bench use zero-shot evaluation.Source: §A.2.1 · p. 36
• The examination benchmarks cover general language understanding, Chinese academic disciplines, human-centric exams, Chinese college-entrance questions, and grade-school science.ARC is divided into easy and challenge subsets, ARC-e and ARC-c.Source: §A.2.1 · p. 36
• Knowledge and understanding are evaluated on BoolQ, CommonsenseQA, NaturalQuestions, and LAMBADA, with results reported in Table 16.The evaluation uses zero-shot settings for BoolQ, NaturalQuestions, and LAMBADA, and eight-shot evaluation for CommonsenseQA.Source: §A.2.1 · p. 38, Table 16 · p. 39
• Natural-language reasoning is evaluated on HellaSwag, PIQA, SIQA, and OCNLI, with results reported in Table 17.All four datasets use zero-shot evaluation and assess commonsense, physical, social, or Chinese natural-language inference.Source: §A.2.1 · pp. 38–39, Table 17 · p. 39
• Mathematics and coding results are omitted from this repeated reasoning evaluation because detailed results are reported elsewhere.The reasoning subsection focuses on natural-language reasoning rather than those domains.Source: §A.2.1 · p. 38

• The evaluation dataset contains manually written or revised instructions drawn from CLiB6, C-Eval (Huang et al., 2023), FacTool (Chern et al., 2023b), and LeetCode7).Source: §A.2.2 · p. 40
• Each case presents all models’ responses and Elo ratings8 for comparison, alongside English translations because the evaluation data are Chinese.Source: §A.2.2 · p. 40

Knowledge

• Qwen correctly identifies the predecessor of Peking University and the world’s tallest existing wooden tower in the supplied knowledge examples.The responses identify the Imperial University of Peking, established in 1898, and the Yinxian Wooden Tower in Shanxi, including its Fogong Temple name and historical dates.Source: Knowledge · pp. 40–41

Language Understanding

• The models provide the pinyin for “吃葡萄不吐葡萄皮” and capture the son’s plans to adapt to university life, join activities, study seriously, and pursue honors.The dialogue responses also mention broadening horizons and social connections while laying a foundation for the future.Source: Language Understanding · pp. 42–43

Creative Writing

• The creative-writing responses generate supermarket jokes based on fruit-related misunderstandings, including confusion over sweet and red apples, banana descriptions, and fruit ownership.The examples resolve each misunderstanding through humorous dialogue between customers and supermarket staff.Source: Creative Writing · pp. 44–46
• In creative writing, the boy helps an old woman by offering what he believes is an excellent apple, which is revealed to be a tomato.The Chinese and English passages present the same humorous narrative sequence.Source: Creative Writing · p. 47

Mathematics

• Mathematics responses vary substantially: Qwen-7B-Chat (SFT) and GPT-4 give 60 centimeters, Qwen-14B-Chat (SFT) gives 300, Qwen-14B-Chat (RLHF) gives 100, and GPT-3.5 gives approximately 36.92.For the six-row arrangement problem, Qwen-14B-Chat (RLHF) correctly derives 42 people, while Qwen-7B-Chat (SFT) reports 7 and GPT-3.5 reports 11.Source: Mathematics · pp. 47–49

Code

• For regular-expression matching, the example uses Python’s re.match() with an anchored pattern intended to match 11-digit strings beginning with 139 and ending with 00.Source: Code · pp. 51, 52, 54

Responses

• Responses disagree about the binary-tree code: Qwen-7B-Chat (SFT) says it is correct, whereas Qwen-14B-Chat (SFT) says it returns node count instead of maximum depth because res is incremented every loop.The proposed fix is to increase res only when a new deepest level is found.Source: Responses · p. 55
• Responses correct binary-tree breadth-first search code by adding only nonempty child nodes, updating depth per level, and replacing an erroneous right-child reference to node.left with node.right.The corrections address both incorrect depth handling and a line-level child-pointer error.Source: Responses · pp. 56–58

A.3 ANALYSIS OF CODE INTERPRETER

• QWEN-CHAT uses a two-step code-interpreter plan to inspect CSV columns before plotting, whereas CODE LLAMA initially uses nonexistent columns and works reliably only when columns are supplied.The comparison demonstrates QWEN-CHAT’s advantages in tabular-data processing and complex tasks.Source: §A.3 · p. 58, Figure 5 · p. 59