Source-linked AI summary
ChatGLM: A Family of Large Language Models from GLM-130B to GLM-4 All Tools
Team GLM, :, Aohan Zeng, Bin Xu, Bowen Wang, Chenhui Zhang, Da Yin, Dan Zhang, Diego Rojas, Guanyu Feng, Hanlin Zhao, Hanyu Lai, Hao Yu, Hongning Wang, Jiadai Sun, Jiajie Zhang, Jiale Cheng, Jiayi Gui, Jie Tang, Jing Zhang, Jingyu Sun, Juanzi Li, Lei Zhao, Lindong Wu, Lucen Zhong, Mingdao Liu, Minlie Huang, Peng Zhang, Qinkai Zheng, Rui Lu, Shuaiqi Duan, Shudan Zhang, Shulin Cao, Shuxun Yang, Weng Lam Tam, Wenyi Zhao, Xiao Liu, Xiao Xia, Xiaohan Zhang, Xiaotao Gu, Xin Lv, Xinghan Liu, Xinyi Liu, Xinyue Yang, Xixuan Song, Xunkai Zhang, Yifan An, Yifan Xu, Yilin Niu, Yuantao Yang, Yueyan Li, Yushi Bai, Yuxiao Dong, Zehan Qi, Zhaoyu Wang, Zhen Yang, Zhengxiao Du, Zhenyu Hou, Zihan Wang
TL;DR
The report addresses how to develop and align successive open large language models with strong general, Chinese, long-context, and agent capabilities. It presents the ChatGLM family and GLM-4 series, including a tool-using variant, and reports performance broadly competitive with leading models while identifying unresolved limitations in data-selection principles and mathematics.
Problem
The report examines how successive large language models can be pretrained and aligned to handle general language tasks, long contexts, Chinese usage, and complex tool-assisted tasks.
Method
The authors develop successive ChatGLM models using large-scale multilingual pretraining, supervised fine-tuning, human-feedback reinforcement learning, and additional alignment for autonomous tool selection.
Results
GLM-4 is broadly competitive with GPT-4-class, GPT-4-Turbo, Gemini, and Claude models across general benchmarks, instruction following, Chinese alignment, long-context tasks, and tool use.
Takeaways & Limitations
The ChatGLM family provides open models spanning compact deployments, long-context variants, and agents that can combine external tools and functions for complex tasks.
Takeaways & Limitations
The authors have not identified a fundamental principle for guiding data collection, cleaning, and selection, and report that the main performance gap with GPT-4 Turbo lies in mathematics.
Abstract
from arXiv · showhide
We introduce ChatGLM, an evolving family of large language models that we have been developing over time. This report primarily focuses on the GLM-4 language series, which includes GLM-4, GLM-4-Air, and GLM-4-9B. They represent our most capable models that are trained with all the insights and lessons gained from the preceding three generations of ChatGLM. To date, the GLM-4 models are pre-trained on ten trillions of tokens mostly in Chinese and English, along with a small set of corpus from 24 languages, and aligned primarily for Chinese and English usage. The high-quality alignment is achieved via a multi-stage post-training process, which involves supervised fine-tuning and learning from human feedback. Evaluations show that GLM-4 1) closely rivals or outperforms GPT-4 in terms of general metrics such as MMLU, GSM8K, MATH, BBH, GPQA, and HumanEval, 2) gets close to GPT-4-Turbo in instruction following as measured by IFEval, 3) matches GPT-4 Turbo (128K) and Claude 3 for long context tasks, and 4) outperforms GPT-4 in Chinese alignments as measured by AlignBench. The GLM-4 All Tools model is further aligned to understand user intent and autonomously decide when and which tool(s) touse -- including web browser, Python interpreter, text-to-image model, and user-defined functions -- to effectively complete complex tasks. In practical applications, it matches and even surpasses GPT-4 All Tools in tasks like accessing online information via web browsing and solving math problems using Python interpreter. Over the course, we have open-sourced a series of models, including ChatGLM-6B (three generations), GLM-4-9B (128K, 1M), GLM-4V-9B, WebGLM, and CodeGeeX, attracting over 10 million downloads on Hugging face in the year 2023 alone. The open models can be accessed through https://github.com/THUDM and https://huggingface.co/THUDM.
1 Introduction
ChatGLM evolves from open GLM and ChatGLM models into GLM-4 variants with expanded context, broad benchmark performance, and autonomous tool use. The report highlights progressive model development, open releases, and strong comparisons with leading proprietary systems.
- Model development: GLM-4 supports 128K context, while GLM-4-9B is available with 128K and 1M context-length variants.GLM-4-9B-Chat-1M supports about 2 million Chinese characters.
- GLM-4 capabilities: GLM-4 0520 performs closely comparably to GPT-4 0613 and Gemini 1.5 Pro across MMLU, GSM8K, MATH, BBH, GPQA, and HumanEval.On MMLU, it scores 83.3 versus 86.4 for GPT-4 0613 and 83.7 for Gemini 1.5 Pro.
- Evaluation: GLM-4 matches GPT-4-Turbo and Claude 3 Opus on LongBench-Chat, scoring 87.3 versus 87.2 and 87.7, respectively.It also approaches GPT-4-Turbo on instruction following and outperforms GPT-4 on Chinese alignment across eight AlignBench dimensions.
- All Tools: GLM-4 All Tools is aligned to understand user intent and autonomously select web browsing, Python, text-to-image, or user-defined functions.The model can use multiple tools sequentially for complex requests such as retrieving population data and calculating growth rates.
- Open development: The project released successive ChatGLM models and related open models including CodeGeeX, with open access through THUDM’s GitHub and Hugging Face repositories.The development history spans GLM-130B, ChatGLM, ChatGLM2/3, and GLM-4 All Tools.
2 ChatGLM Techniques
The ChatGLM techniques combine multilingual data processing, Transformer architecture exploration, long-context extension, and multi-stage alignment. GLM-4 All Tools adds planning and sequential external-tool use, while the authors note that no fundamental data-selection principle has yet been identified.
- Pre-Training Data: GLM-4 training uses multilingual documents processed through deduplication, filtering, and tokenization to improve data diversity and quality.Sources include webpages, Wikipedia, books, code, and research papers.
- Limitations: The authors have not identified a fundamental principle guiding data collection, cleaning, and selection despite finding data quality and diversity crucial.This unresolved issue is presented as a direction for future research.
- Architecture: ChatGLM explores Transformer design choices including bias removal, RMSNorm, SwiGLU, and two-dimensional RoPE to improve speed, performance, and positional handling.The reported choices reflect architectural experimentation across model generations.
- Architecture: Context length expands from 2K in ChatGLM to 32K in ChatGLM2/3 and 128K or 1M in GLM-4 through positional extension, continual training, and long-context alignment.The approach combines context extension with alignment for very long inputs.
- Alignment: GLM-4 alignment relies mainly on supervised fine-tuning and reinforcement learning from human feedback to improve intent understanding, instruction following, and dialogue.Later alignment data combines in-house annotation with third-party proprietary data under quality controls.
- GLM-4 All Tools: GLM-4 All Tools plans complex tasks and sequentially calls one or more external tools when independent completion is insufficient.The system uses intermediate tool feedback and results, and supports customized agents with APIs and external knowledge bases.
3 GLM-4 Capabilities
GLM-4 is evaluated across academic knowledge, mathematics, reasoning, commonsense, and coding benchmarks. Its overall base capacity approaches GPT-4-Turbo and Claude 3 Opus, while outperforming GPT-4 on most reported benchmarks.
- Six benchmarks cover knowledge, mathematics, reasoning, commonsense, and coding: MMLU, GSM8K, MATH, BBH, GPQA, and HumanEval.MMLU uses multiple-choice examinations; GSM8K and MATH evaluate mathematical problem solving; BBH tests challenging reasoning tasks; GPQA covers graduate-level science; HumanEval evaluates code correctness.
- HumanEval measures the correctness of synthesized functions through automatic test-case checking.
- GLM-4 approaches GPT-4-Turbo and Claude 3 Opus in overall base capacity across standard academic benchmarks.The evaluation spans knowledge, math, reasoning, commonsense, and coding.
- 96.3% of GPT-4’s accuracy is achieved by GLM-4 on MMLU, while GLM-4 outperforms GPT-4 on the other reported benchmarks.
3.2 Evaluation of Instruction Following
GLM-4 is evaluated on IFEval using verifiable instruction-following criteria in English and Chinese. It matches GPT-4 Turbo in loose mode and remains close in strict mode.
- IFEval contains 541 prompts derived from 25 instructions whose compliance can be verified using explicit criteria.The evaluation reports prompt-level and instruction-level accuracy in strict and loose modes for both languages.
- GLM-4 matches GPT-4 Turbo’s instruction-level accuracy in loose mode in both English and Chinese.
- 99.0% and 98.6% of GPT-4 Turbo’s instruction-level accuracy are achieved by GLM-4 in strict English and Chinese, respectively.
3.3 Evaluation of Alignment
GLM-4 demonstrates strong Chinese alignment on AlignBench, achieving the highest overall score among the compared baselines. Its main remaining gap with GPT-4 Turbo is in mathematics.
- GLM-4 achieves the highest overall AlignBench score among GPT-4 Turbo, Claude 3 Opus, and Gemini 1.5 Pro.
- GLM-4 significantly outperforms all other powerful models on Chinese logic reasoning and language understanding tasks.AlignBench evaluates Chinese alignment across eight categories using multidimensional, GPT-4-based scoring.
- The performance gap between GLM-4 and GPT-4 Turbo mostly lies in the Mathematics dimension.The authors report using techniques such as self-critique to improve mathematical reasoning.
3.4 Evaluation of Long Context Handling Abilities
GLM-4 is evaluated on LongBench-Chat across Chinese and English long-text tasks ranging from 10K to 100K tokens. It aligns with GPT-4 Turbo and Claude 3 Opus in English and surpasses them on Chinese prompts.
- LongBench-Chat covers context lengths from 10-100k across document question answering, summarization, coding, and other long-text scenarios.The benchmark is divided into Chinese and English portions for language-specific comparison.
- GLM-4 aligns with GPT-4 Turbo and Claude 3 Opus on English long-context prompts and outperforms the best of them on Chinese prompts.The reported metrics average repeated evaluations scored using GPT-4 with few-shot prompting.
- The evaluation repeats runs and reports averages to reduce score variation and obtain more reliable statistical conclusions.
3.5 Evaluation of Coding Abilities on Real-world User Prompts
The evaluation extends beyond introductory coding benchmarks to bilingual, real-world user prompts. GLM-4 performs closely to Claude 3 Opus, while remaining behind GPT-4 models on this practical coding benchmark.
- Results: GLM-4 achieves coding performance close to Claude 3 Opus on practical scenarios.
- Results: GLM-4 still trails GPT-4 models on NaturalCodeBench.The authors note potential for improvement through better training strategies and data.
- Benchmark: NaturalCodeBench evaluates bilingual coding performance using real user prompts in Python and Java.The benchmark is designed to reflect the complexity of practical coding tasks beyond introductory algorithms.
3.6 Evaluation of Function Call
The paper evaluates function calling through structured comparison, API execution, and relevance detection, and also tests agent behavior across practical environments. Performance varies by task: GLM-4 is strong on several agent settings, while execution-summary accuracy improves with model size.
- Function-call evaluation: The Berkeley Function Call Leaderboard evaluates function calling through AST comparison, API execution, and relevance detection.It contains 2k question-function-answer pairs.
- Function-call results: Execution-summary performance improves smoothly with model size, unlike overall function-call accuracy.GLM-4-9B-Chat can outperform GLM-4-Air on overall accuracy.
- Agent evaluation: AgentBench evaluates GLM-4 across seven practical environments, excluding Digital Card Game because it is too time-consuming.Overall scores use AgentBench’s original per-dataset weights.
- Agent results: GLM-4 outperforms GPT-4 Turbo and Claude 3 Opus overall on agent tasks, while GLM-4-Air performs comparably.The GLM-4 series performs especially well on Database, House-Holding, and Web Shopping tasks, but trails GPT-4 series on Operating System, Knowledge Graph, and Lateral Thinking Puzzles.
3.8 Evaluation of All Tools
GLM-4 All Tools is aligned to understand user intent, plan complex tasks, and autonomously invoke multiple tools. Its web and Python-tool performance is reported as similar to ChatGPT-4 in corresponding applications.
- Model design: GLM-4 All Tools autonomously understands user intent, plans step-by-step instructions, and calls multiple tools.Supported tools include a web browser, Python interpreter, text-to-image model, and user-defined functions.
- Results: GLM-4 All Tools achieves similar performance to ChatGPT-4 for browser-based information seeking and Python-based math solving.
4 Safety and Risks
The paper addresses safety through data filtering, safety-focused alignment, red-teaming, and evaluation across seven harm-related dimensions. GLM-4 is competitive overall and comparable to Claude 3 Opus, but remains behind GPT-4 models particularly on physical-health safety.
- Risk mitigation: GLM-4 safety work includes removing sensitive or blacklisted data, filtering risky alignment samples, and treating harmlessness as a preference criterion.
- Risk mitigation: Red-team harmful question-answer pairs are collected and improved through human annotations for further model alignment.
- Safety evaluation: SafetyBench evaluates seven dimensions, including ethics, illegal activities, mental health, offensiveness, physical health, privacy and property, and unfairness and bias.
- Results: GLM-4 achieves competitive safety performance on most dimensions and is comparable overall with Claude 3 Opus.
- Results: GLM-4 falls behind the GPT-4 family, especially on Physical Health safety.The paper attributes this dimension’s demand to robust common-sense knowledge about the physical world.
5 Conclusion
The report presents ChatGLM as an evolving family whose recent GLM-4 models advance complex-task performance and autonomous tool use, while open releases promote accessibility and safety.
- GLM-4 models demonstrate significant advances in understanding and executing complex tasks by autonomously employing external tools and functions.
- Recent GLM-4 models perform on par with, and sometimes surpass, GPT-4 Turbo, Claude 3 Opus, and Gemini 1.5 Pro, especially on Chinese-language tasks.
- The report covers ChatGLM generations from GLM-130B through GLM-4 All Tools.
- Open language, code, and vision models attracted over 10 million Hugging Face downloads in 2023.
- The authors frame open release of model weights and techniques as part of their commitment to accessibility and safety.