PaperlayerView exact revision on arXiv

Source-linked AI summary

GLM-130B: An Open Bilingual Pre-trained Model

Aohan Zeng, Xiao Liu, Zhengxiao Du, Zihan Wang, Hanyu Lai, Ming Ding, Zhuoyi Yang, Yifan Xu, Wendi Zheng, Xiao Xia, Weng Lam Tam, Zixuan Ma, Yufei Xue, Jidong Zhai, Wenguang Chen, Peng Zhang, Yuxiao Dong, Jie Tang

arXiv:2210.02414v2cs.CLcs.AIcs.LG

TL;DR

The paper addresses how to openly train a high-quality 100B-scale language model despite severe efficiency and stability challenges. It presents GLM-130B, a bilingual bidirectional model with blank infilling and engineering strategies for stable, affordable pretraining. GLM-130B surpasses GPT-3 across many benchmarks, outperforms ERNIE TITAN on Chinese tasks, and supports near-lossless INT4 inference.

  • Problem

    Open and reproducible high-quality 100B-scale language models remain limited, while training at this scale presents major stability, efficiency, and convergence challenges.

  • Method

    GLM-130B uses a bilingual bidirectional GLM architecture with autoregressive blank infilling, stability-focused training strategies, and post-training-free INT4 quantization.

  • Results

    GLM-130B surpasses GPT-3 across 112 tasks, outperforms ERNIE TITAN 3.0 across 12 Chinese tasks, and retains near-baseline performance after INT4 quantization.

  • Takeaways & Limitations

    The released model, code, logs, toolkits, and lessons learned support more accessible and open research on 100B-scale language models.

  • Takeaways & Limitations

    The paper notes that better architectures, data, and training FLOPS still require further investment, and its zero-shot evaluation follows a specific criterion because the setting lacks consensus.

Abstract

from arXiv · show

We introduce GLM-130B, a bilingual (English and Chinese) pre-trained language model with 130 billion parameters. It is an attempt to open-source a 100B-scale model at least as good as GPT-3 (davinci) and unveil how models of such a scale can be successfully pre-trained. Over the course of this effort, we face numerous unexpected technical and engineering challenges, particularly on loss spikes and divergence. In this paper, we introduce the training process of GLM-130B including its design choices, training strategies for both efficiency and stability, and engineering efforts. The resultant GLM-130B model offers significant outperformance over GPT-3 175B (davinci) on a wide range of popular English benchmarks while the performance advantage is not observed in OPT-175B and BLOOM-176B. It also consistently and significantly outperforms ERNIE TITAN 3.0 260B -- the largest Chinese language model -- across related benchmarks. Finally, we leverage a unique scaling property of GLM-130B to reach INT4 quantization without post training, with almost no performance loss, making it the first among 100B-scale models and more importantly, allowing its effective inference on 4$\times$RTX 3090 (24G) or 8$\times$RTX 2080 Ti (11G) GPUs, the most affordable GPUs required for using 100B-scale models. The GLM-130B model weights are publicly accessible and its code, training logs, related toolkit, and lessons learned are open-sourced at \url{https://github.com/THUDM/GLM-130B/}.

1 INTRODUCTION

GLM-130B targets an open, accurate 100B-scale bilingual model while addressing the technical challenges of training at this scale. Its design and engineering choices yield strong benchmark, ethical, and affordability outcomes.

  • Model and training: The project documents successful and failed design choices, training strategies, engineering efforts, and lessons learned for 100B-scale pretraining.Training stability is identified as decisive, with embedding gradient shrink significantly stabilizing GLM-130B.
  • Model and training: GLM-130B is a bilingual English-Chinese dense model with 130 billion parameters, pretrained on 400 billion tokens using the GLM algorithm.Training used 96 NVIDIA DGX-A100 GPU nodes between May 6 and July 3, 2022.
  • Evaluation: GLM-130B surpasses GPT-3 across 112 tasks and outperforms PaLM 540B in many cases, unlike OPT-175B and BLOOM-176B.It also outperforms ERNIE TITAN 3.0 260B across Chinese benchmarks and shows less bias and toxicity than 100B-scale counterparts.
  • Affordable inference: INT4 quantization causes negligible performance degradation, including -0.74% on LAMBADA and +0.05% on MMLU.This enables inference on 4×RTX 3090 or 8×RTX 2080 Ti GPUs.
  • Open science: The model checkpoints, code, training logs, toolkits, and lessons learned are open-sourced.

2 THE DESIGN CHOICES OF GLM-130B

GLM-130B combines a bidirectional GLM backbone, blank-infilling objectives, and multi-task instruction pretraining with architecture and systems choices tailored to stability and affordable training.

  • Architecture: GLM-130B replaces GPT-style decoder-only modeling with the bidirectional General Language Model backbone.Its autoregressive blank-infilling objective masks spans and recovers them autoregressively while allowing interactions through a sampled permutation.
  • Architecture: Bidirectional attention over unmasked contexts distinguishes GLM-130B from GPT-style unidirectional attention and supports both understanding and generation.The [MASK] and [gMASK] objectives respectively provide short-span and long-blank corruption modes.
  • Architecture evaluation: 80.2% zero-shot accuracy on LAMBADA is reported as a record high, surpassing GPT-3 and PaLM 540B.A unidirectional attention variant is comparable to GPT-3 and OPT-175B.
  • Stability-oriented design: DeepNorm-based Post-LN is selected after conventional Pre-LN, Post-LN, and Sandwich-LN practices failed to stabilize GLM-130B test runs.The accompanying initialization choice significantly benefits training stability.
  • Pretraining objective: The objective combines self-supervised blank infilling with multi-task instruction pretraining, using 95% and 5% of tokens respectively.MIP includes prompted language-understanding, generation, and information-extraction datasets while aiming to preserve general abilities.
  • Data and systems: Training uses balanced English and Chinese data, 3D parallelism, a global batch size of 4,224, and 4-way tensor plus 8-way pipeline parallelism.Reported utilization is 43.3% HFU and 32.5% MFU.

3 THE TRAINING STABILITY OF GLM-130B

Training GLM-130B is constrained by precision-related instability and loss spikes, so the paper combines stability-oriented normalization with embedding-gradient monitoring and shrinkage.

  • Instability challenge: Training stability is decisive for GLM-130B quality, while low-precision formats improve efficiency but risk overflow, underflow, and collapse.The number of processed tokens also substantially affects quality.
  • Instability challenge: Mixed-precision FP16 training produces frequent loss spikes that become more common over time.The spikes can recover spontaneously or precede divergence.
  • Stability strategies: DeepNorm-based Post-LN bounds the main-branch value scale in deeper layers, addressing one source of instability.
  • Gradient diagnosis: Embedding-layer gradient spikes precede training collapse by several steps and are much larger and more variable than gradients in other layers early in training.Gradient norm therefore serves as an informative indicator of collapse.
  • Gradient diagnosis: Setting the embedding-gradient shrink factor α to 0.1 removes most observed spikes with negligible latency.The strategy is implemented by scaling the embedding contribution while detaching the residual term.
  • Outcome: The final training run experienced only three late-stage loss-divergence cases, aside from numerous hardware failures.Further shrinking the embedding gradient helped stabilize the three unexpected spikes.

4 GLM-130B INFERENCE ON RTX 2080 TI

GLM-130B targets affordable 100B-scale inference by combining efficient implementation with INT4 quantization that preserves performance. Its weight distributions help explain why this quantization works for GLM but fails for some GPT-style models.

  • 130B parameters were chosen to run GLM-130B on a single A100 (40G×8) server instead of the higher-end hardware required by OPT-175B and BLOOM-176B.
  • 7-8.4× faster decoding than Huggingface BLOOM-176B was achieved on the same single A100 server using FasterTransformer.
  • INT4 quantization reduces required GPU memory to 70GB, enabling inference on 4 × RTX 3090 Ti (24G) or 8 × RTX 2080 Ti (11G).
  • Without post-training, INT4 quantization causes almost no performance degradation and preserves GLM-130B’s advantages over GPT-3 on common benchmarks.
  • Wider-distributed linear-layer weights require larger quantization bins and lose more precision, while GLM distributions are narrower than comparable GPT models.

5 THE RESULTS

GLM-130B is evaluated across English and Chinese language, reasoning, and understanding benchmarks under the paper’s stated zero-shot and contamination-filtering criteria. It reports strong results against GPT-3, PaLM, and ERNIE Titan, alongside weaker few-shot growth than GPT-3 on BIG-bench-lite.

  • 5.1 LANGUAGE MODELING: 80.2 zero-shot accuracy on LAMBADA sets a new record, attributed to GLM-130B’s bidirectional attention.
  • 5.1 LANGUAGE MODELING: GLM-130B performs best on 18 shared Pile test sets by weighted BPB against GPT-3 and Jurassic-1.
  • 5.1 LANGUAGE MODELING: 44.8 accuracy on 5-shot MMLU is reached after 400B training tokens, up from 43.9 after about 300B tokens and approaching GPT-3.
  • 5.3 BEYOND THE IMITATION GAME BENCHMARK (BIG-BENCH): GLM-130B outperforms GPT-3 175B and PaLM 540B zero-shot on BIG-bench-lite and maintains its GPT-3 advantage as shots increase.
  • 5.3 BEYOND THE IMITATION GAME BENCHMARK (BIG-BENCH): Few-shot growth on BIG-bench-lite is less significant for GLM-130B, increasing from 13.31 to 15.12 versus GPT-3’s increase from 4.35 to 13.18.
  • 5.4 CHINESE LANGUAGE UNDERSTANDING EVALUATION (CLUE): GLM-130B consistently outperforms ERNIE Titan 3.0 260B across 12 Chinese CLUE and FewCLUE tasks, with at least 260% higher performance on DRCD and CMRC2018.

6 RELATED WORK

Related work covers pre-training, transfer, and inference for large language models. The paper situates GLM-130B among efforts to make high-quality 100B-scale models publicly available while focusing less on transfer methods.

  • Pre-Training: Transformer language models exhibit scaling laws in which new abilities emerge as models grow from billions to 100B-scale parameters.
  • Pre-Training: Many 100B-scale English and Chinese models are not public or are accessible only through limited APIs, motivating open-source efforts such as GLM-130B, OPT, and BLOOM.
  • Transferring: The paper reviews parameter-efficient learning and prompt tuning but does not focus on these transfer approaches.
  • Inference: Inference-related work includes distillation, quantization, and pruning, with recent OPT and BLOOM efforts reaching 8-bit quantization.

7 CONCLUSION AND LESSONS

The paper presents GLM-130B as an open bilingual model whose architectural, training, stability, efficiency, and inference efforts support broad evaluation and lessons for future 100B-scale training.

  • GLM-130B contributes insights into architectures, pre-training objectives, training stability and efficiency, and affordable inference.
  • The model reports language performance on 112 tasks and ethical results on bias and toxicity benchmarks.
  • The authors condense successful and failed experiences into lessons for training 100B-scale language models.

ETHICS STATEMENT

The paper frames open LLM access as a way to improve transparency and enable research on potential harms, while evaluating bias and toxicity as an initial safety step.

  • The authors acknowledge that open LLMs could be used for harmful applications, including fraud, propaganda, and harassment.
  • Open LLM access can enable researchers to study and reduce potential harms such as fairness, bias, privacy, and truthfulness problems.
  • The authors evaluate GLM-130B on English benchmarks covering bias, hate speech detection, and toxic generation.
  • The evaluation suggests bilingual pre-training can mitigate bias and toxicity while retaining strong language performance relative to monolingual-English models.

REPRODUCIBILITY

The paper emphasizes openness and reproducibility by releasing model-related materials and organizing evaluation procedures for accessible reruns.

  • GLM-130B is open-sourced to promote openness and inclusivity in LLM research.
  • The authors disclose code, details, and the full pre-training process despite the high cost of reproducing it.
  • Complete training notes, TensorBoard logs, code, hyperparameters, cluster configuration, and corpus details are provided in the repository and paper.
  • Language and ethical evaluations are organized into one-command bash scripts with accompanying data-processing details.

A ETHICS: EVALUATION ON BIASES AND TOXICITY

The paper evaluates GLM-130B across bias, hate-speech, and toxic-generation benchmarks, reporting favorable comparisons while documenting training and evaluation considerations.

  • Bias measurement: GLM-130B shows fewer biases on almost all CrowS-Pairs stereotype categories except religion and nationality.
  • Bias measurement: GLM-130B outperforms GPT-3 Davinci and OPT-175B on all reported StereoSet metrics.
  • Hate speech detection: GLM-130B outperforms two other LLMs in four ETHOS hate-speech detection settings.
  • Toxic generation: The RealToxicPrompts evaluation generates 25 continuations for each of 10K sampled prompts using nucleus sampling with p = 0.9.
  • Toxic generation: GLM-130B has a lower toxicity rate than GPT-3 Davinci across all tested prompt-toxicity cases.
  • Training stability: Pre-LN and Sandwich-LN both collapse during GLM-130B training, showing that normalization choices remain constrained at 100B scale.

B.4 PIPELINE PARALLEL ANALYSIS

The paper analyzes pipeline bubbles and parallelism choices for efficient GLM-130B training, then reports implementation optimizations for faster inference.

  • Pipeline parallelism: Pipeline training uses forward, backward, and optimizer operations, while micro-batching reduces bubbles compared with naive sequential execution.
  • Pipeline parallelism: When m ≥ 4p, GPipe experiments report negligible pipeline-bubble time through forward recomputation and communication overlap.
  • Parallelism trade-offs: Increasing tensor parallelism reduces the bubble ratio, but excessive scaling lowers computational granularity and raises communication costs.
  • Inference engineering: A C++ FasterTransformer implementation optimizes kernels, GPU calls, cuBLAS selection, parameter layout, and FP16 throughput.
  • Inference engineering: 7.0 to 8.4 times faster decoding is achieved than the compared BLOOM-176B PyTorch implementation.

B.6 ACTIVATION OUTLIER ANALYSIS

GLM-130B’s activation outliers complicate activation quantization, even though its weights support effective INT4 quantization. Ablations also attribute improvements mainly to the GLM objective, with MIP helping selected task types.

  • Activation outliers: 30% of GLM-130B’s dimensions may contain activation outliers, unlike OPT-175B and BLOOM-176B, where few dimensions are outlying.The proposed higher-precision decomposition for outlying dimensions is therefore not applicable to GLM-130B.
  • Activation outliers: 50 to over 100 are the activation values reached by two outlying dimensions, while 99.98% of hidden-state dimensions remain below 6.The authors report that removing these outliers during quantization can significantly degrade performance.
  • Weight quantization: INT4 weight quantization is applied to linear layers, while embeddings, layer normalization, and biases remain unchanged.Two INT4 weights are compressed into one INT8 weight, and Absmax quantization is used to preserve performance.
  • Contribution attribution: 73% of the improvement on LAMBADA and 90% on MMLU are attributed to the GLM objective in an 8-dataset ablation.MIP contributes mainly on text-similarity and coreference tasks, while it can hurt performance on some other datasets.
  • Quantization scaling: GLM’s INT4 weight quantization scaling law is reported as unique relative to GPT-style BLOOM models.The paper presents this as a lesson learned from the model’s quantization behavior.

C.6 MMLU EVALUATION

The evaluation spans language understanding, generation, reasoning, knowledge, and commonsense tasks using zero- and few-shot prompting. GLM-130B generally performs strongly, while English factual knowledge remains a weaker area.

  • Natural language generation: GLM-130B achieves better performance than LaMDA and GPT-3 on all three evaluated natural-language-generation datasets.Bidirectional attention improves conditional generation and makes GLM-130B comparable to PaLM-540B in some cases.
  • Winograd-style tasks: GLM-130B performs best across evaluated models on Winogender but is marginally poorer than GPT-3 and PaLM on Winograd273.These Winograd-style tasks evaluate coreference resolution in ambiguous contexts.
  • Closed-book question answering: GLM-130B performs relatively poorly on Natural Questions but well on StrategyQA.The authors speculate that limited English-corpus exposure contributes to weaker factual memorization on Natural Questions.
  • Commonsense reasoning: GLM-130B performs best on both Commonsense QA and MC-TACO among the evaluated language models.The paper interprets these results as evidence of strong commonsense knowledge.
  • SuperGLUE and reasoning: Bidirectional GLM-130B has higher performance and lower variance than its unidirectional ablation across the SuperGLUE prompt-specific results.It exceeds 80% accuracy on CB, MultiRC, RTE, COPA, and BoolQ in some cases; chain-of-thought prompting also improves symbolic and commonsense reasoning.

E.2 MODEL TRAINING

The paper distributes work across training, evaluation, engineering, writing, and project leadership roles. Contributions cover both model development and the supporting evaluation and inference infrastructure.

  • Training and evaluation: Large-scale training and monitoring were assigned to Aohan Zeng and Xiao Liu.
  • Training and evaluation: Evaluation responsibilities included model validation, language modeling, MMLU, BIG-Bench, CLUE, FewCLUE, baseline, ethical, and framework evaluation.These roles were distributed across Aohan Zeng, Xiao Liu, Zhengxiao Du, Jifan Yu, Weng Lam Tam, Yifan Xu, and Zihan Wang.
  • Engineering: Engineering contributions included INT4 quantization, inference acceleration, low-resource inference, and demo/API development.The listed contributors include Aohan Zeng, Zihan Wang, Xiao Liu, Hanyu Lai, Gouyang Zeng, Xu Han, Weilin Zhao, Zhiyuan Liu, and others.
  • Writing and leadership: Xiao Liu, Yuxiao Dong, and Jie Tang wrote the main paper, while Xiao Liu, Aohan Zeng, and Zhengxiao Du wrote the appendix.
  • Writing and leadership: Jie Tang was the project leader, with Aohan Zeng and Xiao Liu identified as student leaders and multiple technical advisors listed.

F A BRIEF HISTORY OF GLM-130B

GLM-130B began as an effort to make highly accurate bilingual 100B-scale modeling more open and accessible, but the project encountered substantial resource, algorithmic, and engineering challenges. The team addressed these challenges through model and inference decisions that broadened access to both the model and its study.

  • The project sought an accurate bilingual model because GPT-3 was English-only and unavailable to most people.
  • The team identified three major obstacles: computational resources, a robust bilingual pre-training algorithm, and fast low-resource inference.
  • The team chose GLM and a 130-billion-parameter model because the design offered high practical performance and inference on a single A100 server.
  • Training at over 100B parameters exposed frequent hardware failures, exploding gradients, excessive memory use, and other difficulties beyond 10B-scale training.
  • The project eventually trained GLM-130B on 96 A100 servers and developed low-resource inference methods using swapping and quantization.
  • INT4 quantization reduced memory use to 25% of the uncompressed model while supporting inference on 4 × RTX 3090 Ti or 8 × RTX 2080 Ti.
Loading 2210.02414v2…