PaperlayerView exact revision on arXiv

Source-linked AI summary

Qwen Technical Report

Jinze Bai, Shuai Bai, Yunfei Chu, Zeyu Cui, Kai Dang, Xiaodong Deng, Yang Fan, Wenbin Ge, Yu Han, Fei Huang, Binyuan Hui, Luo Ji, Mei Li, Junyang Lin, Runji Lin, Dayiheng Liu, Gao Liu, Chengqiang Lu, Keming Lu, Jianxin Ma, Rui Men, Xingzhang Ren, Xuancheng Ren, Chuanqi Tan, Sinan Tan, Jianhong Tu, Peng Wang, Shijie Wang, Wei Wang, Shengguang Wu, Benfeng Xu, Jin Xu, An Yang, Hao Yang, Jian Yang, Shusheng Yang, Yang Yao, Bowen Yu, Hongyi Yuan, Zheng Yuan, Jianwei Zhang, Xingxuan Zhang, Yichang Zhang, Zhenru Zhang, Chang Zhou, Jingren Zhou, Xiaohuan Zhou, Tianhang Zhu

arXiv:2309.16609v1cs.CL

TL;DR

Existing LLMs face concerns about reproducibility, steerability, and accessibility, so this report presents the Qwen series of base, aligned chat, coding, and mathematics models. Across benchmarks and human evaluation, Qwen is competitive with open-source models and matches some proprietary models, while RLHF chat models remain behind GPT-4 and math models approach GPT-3.5.

  • Problem

    LLMs remain limited by concerns about reproducibility, steerability, and accessibility to service providers.

  • Method

    The Qwen series combines base models trained on trillions of tokens with SFT- and RLHF-aligned chat models and specialized coding and mathematics models.

  • Results

    Qwen models are competitive with open-source models and match some proprietary models; RLHF chat models remain behind GPT-4, while Math-Qwen approaches GPT-3.5.

  • Takeaways & Limitations

    Qwen demonstrates that a single model series can support competitive general-purpose, conversational, coding, and mathematics performance across reported evaluations.

  • Takeaways & Limitations

    Traditional benchmark evaluations may not accurately measure aligned chat models, requiring more extensive and rigorous assessment.

Abstract

from arXiv · show

Large language models (LLMs) have revolutionized the field of artificial intelligence, enabling natural language processing tasks that were previously thought to be exclusive to humans. In this work, we introduce Qwen, the first installment of our large language model series. Qwen is a comprehensive language model series that encompasses distinct models with varying parameter counts. It includes Qwen, the base pretrained language models, and Qwen-Chat, the chat models finetuned with human alignment techniques. The base language models consistently demonstrate superior performance across a multitude of downstream tasks, and the chat models, particularly those trained using Reinforcement Learning from Human Feedback (RLHF), are highly competitive. The chat models possess advanced tool-use and planning capabilities for creating agent applications, showcasing impressive performance even when compared to bigger models on complex tasks like utilizing a code interpreter. Furthermore, we have developed coding-specialized models, Code-Qwen and Code-Qwen-Chat, as well as mathematics-focused models, Math-Qwen-Chat, which are built upon base language models. These models demonstrate significantly improved performance in comparison with open-source models, and slightly fall behind the proprietary models.

1 INTRODUCTION

Qwen is a comprehensive open-source LLM series spanning base pretrained, aligned chat, coding, mathematics, and vision-language models. Its models target broad downstream performance, human-preferred interaction, tool use, and specialized reasoning tasks.

  • Base language models: QWEN base models are trained on up to 3 trillion tokens and demonstrate superior performance across diverse downstream tasks, including against substantially larger models.The training data comprise diverse texts and code spanning a wide range of areas.
  • Chat models: QWEN-CHAT models are fine-tuned for task performance, conversation, tool use, agents, and safety, with RLHF variants producing responses preferred by humans.The report states that supervised fine-tuning models achieve superior benchmark performance and that reward models are used in RLHF.
  • Coding models: CODE-QWEN models are pretrained on extensive code datasets and fine-tuned for code generation, debugging, interpretation, and code-related conversations.The specialized series includes CODE-QWEN-7B, CODE-QWEN-14B, and corresponding chat models.
  • Mathematics models: MATH-QWEN-7B-CHAT and MATH-QWEN-14B-CHAT outperform same-size open-source models by large margins and approach GPT-3.5 on GSM8K and MATH.These models are specifically designed for mathematical problems.
  • Vision-language models: QWEN-VL and QWEN-VL-CHAT support visual-language instruction comprehension, text recognition, visual grounding in Chinese and English, multi-image conversations, and storytelling.They are reported to outperform current open-source vision-language models across various evaluation benchmarks.

2 PRETRAINING

QWEN’s pretraining uses a diverse multilingual dataset of up to 3 trillion tokens, a modified Transformer architecture, and efficiency-oriented tokenization and training techniques. Across downstream tasks, QWEN models perform strongly, with QWEN-14B surpassing larger models on some tasks and context extension preserving performance beyond 8192 tokens.

  • Data: QWEN’s pretraining dataset combines public web documents, encyclopedias, books, and code across languages, with substantial English and Chinese coverage.The data pipeline applies HTML text extraction, language identification, exact-match and fuzzy deduplication, quality filtering, and test-set overlap filtering before constructing a dataset of up to 3 trillion tokens.
  • Tokenization: QWEN uses BPE tokenization based on tiktoken’s cl100k_base vocabulary, augmented with commonly used Chinese characters to improve multilingual performance.QWEN achieves higher compression efficiency than competing tokenizers in most languages, potentially reducing serving costs through shorter token sequences.
  • Training and context extension: QWEN is trained autoregressively with 2048-token contexts, while Flash Attention improves computational efficiency and reduces memory usage.Training-free inference methods, including NTK-aware interpolation, extend the usable context length without modifying training.
  • Evaluation: QWEN models perform strongly across downstream tasks: QWEN-14B outperforms LLaMA2-70B on 3 tasks, while combined context-extension methods maintain performance beyond 8192 tokens.QWEN-7B surpasses LLaMA2-13B and matches Baichuan2-13B comparably, and QWEN-1.8B remains competitive on selected tasks.

3 ALIGNMENT

QWEN models are aligned through supervised finetuning and reinforcement learning from human feedback, using chat-style data, human conversation annotations, and comparison data for reward modeling. The resulting aligned models outperform comparable open-source systems across several evaluations, support tool-use and planning, and motivate more rigorous evaluation beyond traditional benchmarks.

  • Supervised finetuning: Supervised finetuning adapts pretrained LLMs to chat-style queries and responses to improve their ability to engage in natural conversation.SFT uses next-token prediction while masking system and user inputs.
  • Supervised finetuning: QWEN strengthens SFT data with human-style conversations annotated in multiple styles and uses ChatML to distinguish system, user, and assistant information.The approach targets helpfulness and natural-language generation across diverse tasks.
  • Reward modeling: Reward-model training uses comparison data containing two responses to each query and their corresponding human preferences, with test pairwise accuracy used as an evaluation metric.The reward-model process includes preference model pretraining followed by finetuning on comparison data.
  • Evaluation: QWEN-14B-Chat outperforms all models except ChatGPT and LLAMA 2-CHAT-70B across MMLU, C-Eval, GSM8K, HumanEval, and BBH, while remaining consistently better than similarly sized open-source models.HumanEval performance is significantly higher than that of other open-source models; the cited comparable models include LLaMA2, ChatGLM2, InternLM, and Baichuan2.
  • Limitations: Traditional benchmarks may not fully measure aligned chat models, so the authors call for new evaluation methods and more extensive, rigorous assessment.The limitation applies despite evidence that QWEN is competitive with open-source and proprietary models.
  • RLHF: RLHF models outperform SFT models by significant margins in win-rate evaluations against GPT-3.5, indicating stronger human preference for their responses.The passage describes bars reporting wins, ties, and losses against GPT-3.5.
  • Agents: QWEN supports agent applications through tool-use and planning, achieving competitive Hugging Face agent performance slightly behind GPT-4 and showing higher tool-relevance accuracy as model size increases.Tool selection and argument performance show little improvement beyond a certain model size, while the preliminary benchmark may require enhancement.

4 CODE-QWEN: SPECIALIZED MODEL FOR CODING

Code-Qwen specializes Qwen’s generalist base models for coding by continuing pretraining on code data while retaining text-and-code foundations and supporting contexts up to 8192 tokens. Code-Qwen and Code-Qwen-Chat outperform similarly sized open-source baselines and rival larger models, but remain behind GPT-4 and other state-of-the-art methods generally.

  • Motivation: Domain-specific code pretraining and fine-tuning are presented as effective for coding, debugging, and code interpretation.The specialized models build on the generalist pretraining and alignment foundation.
  • Training strategy: Code-Qwen continues pretraining Qwen base models on around 90 billion code tokens, using context lengths up to 8192 for lengthy coding scenarios.The base models are trained on combined text and code data before code-focused continuation pretraining.
  • Chat-model fine-tuning: Multi-stage supervised fine-tuning yields the best performance among the evaluated methods for Code-Qwen-Chat.The 14B and 7B models use learning rates of 2.0 × 10^-6 and 1.0 × 10^-5, respectively.
  • Evaluation: Code-Qwen and Code-Qwen-Chat significantly outperform similarly sized OCTOGEEX, InstructCodeT5+, and CodeGeeX2, while rivaling larger Starcoder models.The comparison uses pass@1 on Humaneval, MBPP, and the multilingual HUMANEVALPACK benchmark.
  • Evaluation: Code-Qwen and Code-Qwen-Chat show clear pass@1 advantages over some extremely large closed-source models but fall behind state-of-the-art methods such as GPT-4.The authors suggest scaling model and data size could narrow this gap, while noting current evaluations are insufficient for fully assessing relative performance.

5 MATH-QWEN: SPECIALIZED MODEL FOR MATHEMATICS REASONING

MATH-QWEN-CHAT is a mathematics-specialized, human-aligned model series built on QWEN pretrained models, released in 14B and 7B versions. It is trained with masked math supervised fine-tuning and evaluated against proprietary and open-source models across four mathematics benchmarks.

  • Model Overview: MATH-QWEN-CHAT comprises human-aligned assistant models specialized for arithmetic and mathematics, built on QWEN pretrained language models.The released variants are MATH-QWEN-14B-CHAT and MATH-QWEN-7B-CHAT.
  • Model Overview: The released MATH-QWEN-CHAT variants contain 14 billion and 7 billion parameters, respectively.They are named MATH-QWEN-14B-CHAT and MATH-QWEN-7B-CHAT.
  • Training: Math supervised fine-tuning uses sequence length 1024 on an augmented instructional dataset, with system and user inputs masked to accelerate convergence.Most training inputs are examination questions, and masking excludes their conditions and numbers from loss computation.
  • Training: Optimization uses AdamW with a peak learning rate of 2 × 10−5 and 50 000 training steps.These settings follow the supervised-fine-tuning hyperparameters except for the stated learning rate and training duration.
  • Evaluation: Evaluation covers GSM8K, MATH, Math401, and Math23K, comparing MATH-QWEN-CHAT with proprietary and open-source mathematics models.The comparisons include ChatGPT, Minerva, RFT, WizardMath, and GAIRMath-Abel.

6 RELATED WORK

Related work traces LLM progress from Transformer-based pretraining and scaling to few-shot learning, human-aligned assistants, open-source development, tool use, coding, and mathematical reasoning. Alignment research also addresses instruction-data scarcity through self-instruction and parameter-efficient tuning.

  • LLM foundations and scaling: Transformer-based pretraining enabled transfer learning as model sizes grew from 100 million to over 10 billion parameters.The cited progression spans the Transformer architecture, large-scale pretraining, and expanding model sizes.
  • LLM foundations and scaling: GPT-3 demonstrated few-shot and zero-shot learning through prompt engineering and in-context learning, followed by chain-of-thought prompting and further scaling studies.GPT-3 was described as 10 times larger than T5.
  • Assistants and alignment: ChatGPT and GPT-4 showed that LLMs can act as effective human-communicating assistants, stimulating research on human-value alignment and potential AGI.The passage characterizes these releases as historic moments in artificial intelligence.
  • Assistants and alignment: Alignment mitigates repetitive generation, hallucination, and preference deviation, while self-instruct and open-source chat datasets address the difficulty of collecting instruction data.Examples of open-source chat data include Alpaca, MOSS, Dolly, and Evol-Instruct.
  • Assistants and alignment: Effective chat-model training mainly uses SFT and RLHF, while LoRA and Q-LoRA provide parameter-efficient alternatives when memory limits hinder SFT research.SFT focuses on instruction following, whereas LoRA tunes only low-rank parameters.
  • Tools, coding, and mathematics: LLMs support tool invocation, code understanding and generation, and mathematical reasoning through approaches including ReAct, chain-of-thought, scratchpad, self-consistency, and least-to-most prompting.The cited work links tool use to in-context learning and mathematical improvements to structured prompting techniques.

7 CONCLUSION

The QWEN series comprises pretrained and fine-tuned large language models in 14B, 7B, and 1.8B parameter sizes, alongside specialized coding and mathematics models. Its open access is intended to foster collaboration, innovation, research, and applications in language modeling.

  • Model Series: QWEN includes 14B, 7B, and 1.8B-parameter models pretrained on trillions of tokens and fine-tuned with SFT and RLHF.The series also includes specialized models for coding and mathematics.
  • Model Series: Specialized models in the series include CODE-QWEN, CODE-QWEN-CHAT, and MATH-QWEN-CHAT.
  • Open Access: Open access to QWEN is intended to foster community collaboration and innovation by enabling researchers and developers to build upon the work.The authors also hope the public release inspires new research and applications that advance language modeling.

A APPENDIX · A.1 MORE TRAINING DETAILS · A.1.1 DATA FORMAT FOR QWEN-CHAT

Qwen-Chat uses a specialized conversational data format for supervised fine-tuning and RLHF rather than conventional autoregressive pretraining formatting. The model adopts ChatML, whose special tokens distinguish conversational segments without appearing in pretraining.

  • A.1.1 DATA FORMAT FOR QWEN-CHAT: Conversational assistant training requires a specially designed data format for SFT and RLHF, despite using a similar underlying training task to autoregressive pretraining.Common formats include human-assistant and ChatML formats.
  • A.1.1 DATA FORMAT FOR QWEN-CHAT: Qwen-Chat adopts the ChatML format proposed by OpenAI instead of the human-assistant format.The passage identifies both as common conversational data formats.
  • A.1.1 DATA FORMAT FOR QWEN-CHAT: ChatML uses special tokens, <im_start> and <im_end>, that do not appear during pretraining to resolve conversational formatting problems.The format example follows this description in the paper.

A.2 EVALUATION · A.2.1 AUTOMATIC EVALUATION

The QWEN series is evaluated comprehensively against baselines using OpenCompass categories spanning examination, knowledge and understanding, and natural-language reasoning. The evaluation specifies datasets and shot settings across Chinese, English, and general-purpose benchmarks, while omitting repeated mathematics and coding results.

  • A.2 EVALUATION: QWEN and baselines are compared across OpenCompass categories covering examination, language, knowledge, understanding, and reasoning.Results are organized into multiple tables according to the officially provided categories.
  • A.2.1 AUTOMATIC EVALUATION: Examination evaluation covers MMLU, C-Eval, CMMLU, AGIEval, Gaokao-Bench, ARC-e, and ARC-c with five-shot or zero-shot testing.MMLU, C-Eval, and CMMLU use five-shot results, whereas AGIEval, Gaokao-Bench, and ARC use zero-shot results.
  • A.2.1 AUTOMATIC EVALUATION: The examination benchmarks assess broad language understanding, Chinese subject knowledge, human-centric examinations, Chinese college-entrance questions, and grade-school science reasoning.ARC is divided into easy and challenge sets, referred to as ARC-e and ARC-c.
  • A.2.1 AUTOMATIC EVALUATION: Results are distributed across Table 13 for MMLU, Table 14 for C-Eval, Table 15 for the other examination datasets, Table 16 for knowledge and understanding, and Table 17 for reasoning.The examination tables include CMMLU, AGIEval, ARC-e, and ARC-c beyond MMLU and C-Eval.
  • A.2.1 AUTOMATIC EVALUATION: Chinese-oriented datasets expose weaker comparisons for models not optimized for Chinese, while LLAMA 2’s reported results are limited to its English AGIEval portion.For LLAMA 2, the evaluation uses results from OpenCompass because AGIEval contains both Chinese and English parts.
  • A.2.1 AUTOMATIC EVALUATION: Knowledge and understanding evaluation uses BoolQ, CommonsenseQA, NaturalQuestions, and LAMBADA, with zero-shot and eight-shot settings.BoolQ is tested zero-shot, while CommonsenseQA is tested eight-shot; results are reported in Table 16.
  • A.2.1 AUTOMATIC EVALUATION: Natural-language reasoning evaluation covers HellaSwag, PIQA, SIQA, and OCNLI, all reported with zero-shot results.These datasets target commonsense inference, physical knowledge, social commonsense intelligence, and Chinese natural-language inference.
  • A.2.1 AUTOMATIC EVALUATION: The reasoning section excludes repeated mathematics and coding evaluations because those results were presented in detail elsewhere.This section focuses specifically on natural-language reasoning.

A.2.2 HUMAN EVALUATION

The human evaluation uses a self-constructed dataset with manually written or revised instructions from public datasets, and compares model responses using Elo ratings with English translations provided for Chinese data.

  • Dataset construction: The self-constructed human-evaluation dataset contains manually written or manually revised instructions drawn from public datasets, including CLiB, C-Eval, FacTool, and LeetCode.The evaluation presents human-analysis cases based on these instructions.
  • Evaluation and comparison: The evaluation compares all models by presenting their responses and Elo ratings for each case, with English translations supplied because the data are Chinese.Translations accompany the Chinese human-evaluation data to support comparison.

Knowledge … A.3 ANALYSIS OF CODE INTERPRETER

The evaluated responses cover knowledge, language understanding, creative writing, mathematics, code, and code-interpreter comparison examples. They include factual answers, planned activities, jokes, mathematical solutions, regex implementations, and a tabular-data comparison.

  • Knowledge: Qwen chat responses correctly identify the Imperial University of Peking as Peking University’s predecessor and the Yinxian Wooden Tower as the tallest existing wooden tower.Both answers include supporting historical or location details, including 1898 for the university and Shanxi, China, for the tower.
  • Language Understanding: The language-understanding responses provide the pinyin for “吃葡萄不吐葡萄皮” and summarize the son’s plans for university life.The plans include adapting quickly, joining clubs and extracurricular activities, studying professional knowledge, pursuing good grades, and seeking scholarships or honors.
  • Creative Writing: The creative-writing responses generate supermarket fruit-misunderstanding jokes involving sweet versus red apples, talking bananas, or confusion between supermarket and personal fruit.The examples use misunderstandings between customers and shopkeepers or cashiers as the central comic mechanism.
  • A.3 ANALYSIS OF CODE INTERPRETER: The code-interpreter analysis presents a comparison case between CODE LLAMA and QWEN-CHAT, highlighting QWEN-CHAT’s advantages in tabular-data processing and complex tasks.The passage explicitly frames the example as a comparison between the two models.
Loading 2309.16609v1…