GLM-130B: An Open Bilingual Pre-trained Model

Model and training

• GLM-130B uses the GLM algorithm’s bidirectional attention and autoregressive blank-infilling objective, and embedding gradient shrink significantly stabilizes training.It is a dense bilingual model with 130 billion parameters, pre-trained on 400 billion tokens.Source: §1 · p. 2

Results

• Across 112 tasks, GLM-130B surpasses GPT-3 on many benchmarks and outperforms PaLM 540B in many cases (Chowdhery et al., 2022); it beats GPT-3 175B by +5.0% on zero-shot LAMBADA (Brown et al., 2020).The introduction notes that OPT-175B and BLOOM-176B do not show the same outperformance over GPT-3.Source: §1 · p. 2, Figure 1 · p. 2
• +24.26% on 7 zero-shot CLUE datasets and +12.75% on 5 zero-shot FewCLUE datasets versus ERNIE TITAN 3.0 260B (Wang et al., 2021).The paper identifies ERNIE TITAN 3.0 as the largest Chinese language model and describes GLM-130B as bilingual in English and Chinese.Source: §1 · p. 2

Ethical evaluation

• GLM-130B is associated with significantly less bias and generation toxicity than its 100B-scale counterparts.The ethical comparison is summarized in Figure 1.Source: §1 · p. 2, Figure 1 · p. 2

Affordable inference

• INT4 quantization without post-training causes only -0.74% on LAMBADA and +0.05% on MMLU, enabling inference on 4×RTX 3090 or 8×RTX 2080 Ti GPUs.The paper attributes the negligible performance degradation to a unique property of the GLM architecture.Source: §1 · p. 2

Open release

• The authors release GLM-130B’s model checkpoints, code, training logs, toolkits, and lessons learned to support broader 100B-scale research.The introduction motivates this openness by noting that GPT-3 and its training process had been intransparent to the public.Source: §1 · pp. 1, 3

2.1 GLM-130B’S ARCHITECTURE

• GLM autoregressively recovers masked text spans, using bidirectional attention over unmasked context and randomized span order to let corrupted spans interact.Training mixes [MASK] short blanks with [gMASK] long blanks at sentence ends with prefix context.Source: §2.1 · p. 3
• 80.2% zero-shot LAMBADA accuracy is reported for GLM-130B, outperforming GPT-3 and PaLM 540B; its unidirectional variant is comparable to GPT-3 and OPT-175B.The comparison is presented in Figure 2.Source: §2.1 · p. 3, Figure 2 · p. 3
• DeepNorm-initialized Post-LN improved training stability after Pre-LN, Post-LN, and Sandwich-LN alone failed to stabilize GLM-130B test runs.The approach also applies scaled Xavier normal initialization to selected projections and FFN weights, with biases initialized to zero.Source: §2.1 · p. 4

2.2 GLM-130B’S PRE-TRAINING SETUP

• 95% of training tokens use self-supervised blank infilling and 5% use multi-task instruction pre-training across understanding, generation, and information extraction tasks.The blank-infilling setup uses [MASK] on 30% of sequences and [gMASK] on the other 70%; the 5% MIP component uses 74 prompted datasets and is intended to avoid spoiling general abilities such as unconditional generation.Source: §2.2 · p. 4
• Training data combines 1.2T Pile English tokens, 1.0T WudaoCorpora Chinese tokens, and 250G additional Chinese web-crawled corpora in a balanced English-Chinese composition.Source: §2.2 · p. 4

2.3 PLATFORM-AWARE PARALLEL STRATEGIES AND MODEL CONFIGURATIONS

• GLM-130B was trained for 400 billion tokens over 60 days, roughly 200 billion each in Chinese and English, using fixed 2,048-token sequences.Training used a batch-size warmup to 4,224, AdamW, a cosine learning-rate decay, dropout 0.1, and gradient clipping at 1.0.Source: §2.3 · pp. 4–5
• A 3D parallel strategy combines data, tensor, and pipeline parallelism; with 4-way tensor and 8-way pipeline parallelism, reported utilization is 43.3% HFU and 32.5% MFU.The model was configured with 70 transformer layers to balance pipeline stages and fit a single 40G DGX-A100 node in FP16.Source: §2.3 · pp. 4–5

• Given computing constraints, low-precision formats improve efficiency but risk overflow and underflow that can collapse training, making stability a decisive factor in model quality.The paper also notes that quality is substantially affected by the number of tokens processed.Source: §3 · p. 5

Mixed-Precision

• With FP16 forwards and backwards and FP32 optimizer states and master weights, GLM-130B faces increasingly frequent loss spikes, sometimes culminating in NaN loss.The mixed-precision strategy reduces GPU memory use and improves training efficiency, but precision-related spikes can also recover on their own.Source: §3 · p. 5

• DeepNorm-based Post-LN bounds the transformer main-branch value scale, addressing the extreme values that can arise in deeper layers with Pre-LN.Source: §3 · p. 5

Mixed-Precision

• As models scale, attention scores can exceed FP16’s range; PB-Relax did not prevent GLM-130B disconvergence, while BF16 costs about 15% more runtime GPU memory and is unsupported on some platforms.Embedding norm with BF16 is another option, but can harm zero-shot learning and significantly penalize performance.Source: §3 · pp. 5–6

Embedding Layer Gradient Shrink (EGS)

• Gradient-norm spikes precede training collapse by a few steps and are often driven by abnormally large, fluctuating embedding-layer gradients early in training.The embedding gradient norm can be several magnitudes larger than those of other layers, and freezing the embedding layer is not an option in language models.Source: §3 · p. 6
• α = 0.1 wipes out most expected loss spikes with negligible latency, and the final GLM-130B run experiences only three late-stage divergence cases.Further shrinking the embedding gradient can help stabilize training after unexpected spikes.Source: Figure 4 · p. 5, §3 · p. 6

Inference implementation

• GLM-130B decoding with FasterTransformer is 7–8.4× faster than Huggingface’s PyTorch BLOOM-176B implementation on the same single A100 server.The 130B model size was selected to fit a single A100 (40G×8) server, unlike OPT-175B and BLOOM-176B, which require an A100 (80G×8) server.Source: §4 · p. 6

INT4 Quantization for RTX 3090s/2080s

• About 30% of GLM-130B activation dimensions are outliers, so it quantizes mostly linear-layer weights while retaining FP16 activations and dynamically converting weights to FP16 at runtime.This avoids the less efficient outlier-dimension decomposition approach; runtime conversion adds small computational overhead while reducing GPU memory for weights.Source: §4 · p. 6
• 70GB INT4 weights enable inference on 4 × RTX 3090 Ti or 8 × RTX 2080 Ti GPUs, with almost no post-training-free performance degradation and retained advantages over GPT-3 on common benchmarks.Compared with INT8, INT4 saves an additional half of the required GPU memory; no post-training is used.Source: Table 2 · p. 7, §4 · p. 7

GLM’s INT4 Weight Quantization Scaling Law

• Wider weight distributions require larger quantization bins and lose more precision; GLM’s narrower distributions help explain INT4’s failure for GPT-style BLOOM and the shrinking INT4–FP16 gap as GLM scales.The paper identifies attn-dense and w2 as wide-distributed matrices in BLOOM.Source: Figure 5 · p. 6, §4 · p. 7

5 THE RESULTS

• For English zero-shot evaluation, GLM-130B excludes fixed-label tasks and considers non-fixed-label tasks only when their domains transfer clearly from MIP; all Chinese datasets qualify through cross-lingual transfer.The paper frames this criterion around zero-shot learning involving unseen class labels.Source: §5 · p. 7
• Potentially contaminated datasets are withheld; LAMBADA and CLUE show minimal overlap under the 13-gram setting, while Pile, MMLU, and BIG-bench were held out or released after corpus crawling.The filtering follows prior practices and the paper’s stated dataset-selection criterion.Source: §5 · p. 7

5.1 LANGUAGE MODELING

• 80.2 zero-shot accuracy on LAMBADA sets a new record, while GLM-130B has the best weighted BPB average on 18 shared Pile test sets versus GPT-3 and Jurassic-1.LAMBADA evaluates last-word language-modeling capability; the Pile comparison uses weighted BPB averaged across 18 shared test sets.Source: §5.1 · p. 7, Table 3 · p. 7

5.2 MASSIVE MULTITASK LANGUAGE UNDERSTANDING (MMLU)

• 44.8 accuracy on MMLU after 400B tokens exceeds GLM-130B’s earlier 5-shot result’s proximity to GPT-3’s 43.9 after about 300B tokens.MMLU was released after the Pile corpus crawl and serves as a few-shot test bed.Source: §5.2 · p. 8, Figure 6 · p. 8

5.3 BEYOND THE IMITATION GAME BENCHMARK (BIG-BENCH)

• GLM-130B outperforms GPT-3 175B and PaLM 540B in zero-shot BIG-bench-lite, and maintains its advantage over GPT-3 as shots increase.The reported performance rises from 13.31 to 15.12 with more few-shot samples, compared with GPT-3’s 4.35 to 13.18. The authors suggest strong zero-shot performance and zero-shot-only MIP may explain the smaller few-shot gains, and propose training with varied in-context shots; they also conjecture PaLM’s stronger few-shot growth reflects its private training corpora and call for better architectures, data, and more training FLOPS.Source: §5.3 · p. 8, Table 4 · p. 8, Figure 7 · p. 8

5.4 CHINESE LANGUAGE UNDERSTANDING EVALUATION (CLUE)

• GLM-130B consistently outperforms ERNIE Titan 3.0 across 12 Chinese tasks and performs at least 260% better on DRCD and CMRC2018.The zero-shot comparison uses dev datasets; the authors suggest the pre-training objective may suit abstractive MRC tasks.Source: §5.4 · p. 9, Figure 8 · p. 9

Pre-Training

• GLM-130B joins EleutherAI, OPT-175B (Zhang et al., 2022), and BLOOM-176B (Scao et al., 2022) in efforts to provide high-quality open LLMs amid limited public access to many 100B-scale models.The passage links this openness to community development and describes scaling from smaller language models to 100B-scale GPT-3 (Brown et al., 2020).Source: §6 · p. 9

Transferring

• LLM evaluation has centered on prompting and in-context learning because of model size, while GLM-130B’s comprehensive evaluation of parameter-efficient learning and prompt tuning is deferred to future work.The passage names parameter-efficient learning (Houlsby et al., 2019) and prompt tuning, including P-tuning (Li & Liang (2021); Liu et al. (2021b); Lester et al. (2021); Liu et al. (2022)).Source: §6 · p. 9

Inference

• 4×RTX 3090 (24G) GPUs or 8×RTX 2080 Ti (11G) GPUs suffice for GLM-130B inference through INT4 weight quantization, extending prior 8-bit quantization work on OPT-175B and BLOOM-176B.The passage also identifies distillation, quantization, and pruning as related approaches to efficient inference.Source: §6 · p. 9

7 CONCLUSION AND LESSONS

• GLM-130B’s technical and engineering undertakings offer insights into LLM architectures, pre-training objectives, training stability and efficiency, and affordable inference.The paper condenses experiences of success and failure into lessons for training 100B-scale LLMs in Appendix B.10.Source: §7 · p. 9
• GLM-130B contributes language performance on 112 tasks and ethical results on bias and toxicity benchmarks.Source: §7 · p. 9
• GLM-130B is introduced as a bilingual pre-trained model intended to facilitate open and inclusive LLM research.Source: §7 · p. 9

ETHICS STATEMENT

• The authors acknowledge that the model could enable harmful applications, including fraud, propaganda, and harassment, but do not anticipate hazardous outputs toward vulnerable or historically disadvantaged groups.They state that all co-authors are aware of and honor the ICLR Code of Ethics.Source: ETHICS STATEMENT · p. 10

Open-Sourced LLMs for Ethical Risk Study

• The authors argue that open LLMs can broaden access and transparency, enabling researchers to examine model flaws and develop methods to reduce potential harms.They note that open models can expose parameters and internal states for specific inputs, unlike black-box APIs.Source: ETHICS STATEMENT · p. 10

Ethical Evaluation and Improvements

• The authors evaluate English benchmarks for bias, hate speech, and toxic generation, while acknowledging that these datasets have limitations.They describe the benchmarks as an initial step toward open quantitative LLM evaluation.Source: ETHICS STATEMENT · p. 10
• The evaluation suggests bilingual pre-training can significantly mitigate bias and toxicity while retaining strong language performance versus LLMs trained on monolingual English corpora (Brown et al., 2020; Zhang et al., 2022).The authors refer to Appendix A for more details.Source: ETHICS STATEMENT · p. 10

REPRODUCIBILITY

• GLM-130B is open-sourced, unlike mainstream closed-source LLMs, to promote openness and inclusivity in LLM research.Source: REPRODUCIBILITY · p. 10
• Despite the high cost of reproducing pre-training, the authors disclose its code, details, and full process.Source: REPRODUCIBILITY · p. 10
• Inference on popular GPUs such as RTX 3090 and 2080 Ti is intended to let academic researchers reproduce results offline, and free APIs let individuals test the model.Source: REPRODUCIBILITY · p. 10
• Pre-training resources include complete training notes, TensorBoard logs, and code in the repository.Source: REPRODUCIBILITY · p. 11
• Pre-training hyperparameters and cluster configuration are provided in Section 2.3 and Table 11, while corpus and instruction-pretraining details appear in Section 2.2 and Appendices C.1 and C.2.Source: REPRODUCIBILITY · p. 11
• One-command bash scripts organize language and ethical benchmark evaluations, with data-processing details supplied across the cited sections and appendices.The listed language benchmarks are LAMBADA, Pile, MMLU, BIG-bench, CLUE, and FewCLUE; ethical benchmarks are CrowS-Pairs, StereoSet, ETHOS, and RealToxicPrompts.Source: REPRODUCIBILITY · p. 11

A ETHICS: EVALUATION ON BIASES AND TOXICITY

• The authors require GLM-130B weight applicants to agree not to use it for harmful acts and frame bias and toxicity evaluation as an initial step toward understanding and reducing harms.They note that the benchmarks have limitations and say future work should address them.Source: A ETHICS: EVALUATION ON BIASES AND… · p. 21

A.1 BIAS MEASUREMENT: CROWS-PAIRS

• GLM-130B shows fewer biases than comparison models on almost all CrowS-Pairs stereotype categories except religion and nationality.The authors speculate that bilingual pre-training may reconcile biases on a benchmark built mainly around US stereotypes, while warning that Chinese biases lack adequate benchmarks.Source: Table 5 · p. 21, §A.1 · p. 21

A.2 BIAS MEASUREMENT: STEREOSET

• GLM-130B outperforms GPT-3 Davinci and OPT-175B on all StereoSet metrics: LMS, SS, and ICAT.The evaluation normalizes likelihood scores over tokens to calculate the metrics.Source: Table 6 · p. 22, §A.2 · p. 22

A.3 HATE SPEECH DETECTION: ETHOS

• On ETHOS hate-speech detection, GLM-130B outperforms two other LLMs across four different settings; the reported metric is F1, with higher scores better.The authors suggest that pre-training on diverse online forums and social media and MIP training may contribute to its detection and zero-/few-shot capabilities.Source: §A.3 · p. 22, Table 7 · p. 22

A.4 TOXIC GENEARATION: REALTOXICPROMPTS

• GLM-130B has lower continuation toxicity than GPT-3 Davinci across all evaluated prompt-toxicity cases.The evaluation samples 25 continuations for each of 10K prompts using nucleus sampling (p = 0.9), caps generation at 128 tokens, and scores the first complete sentence.Source: Figure 9 · p. 22, §A.4 · pp. 22–23

B TECHNICAL DETAILS; B.1 TOKENIZATION

• The technical appendix details training issues and solutions; its tokenizer uses 130,000 text tokens from a 150,000-token icetk vocabulary, with added prediction and separation tokens.The text vocabulary was trained on a 25GB bilingual English-Chinese corpus; the implementation excludes 20,000 image tokens and adds [MASK], [gMASK], <sop>, <eop>, and <eos>.Source: B TECHNICAL DETAILS · p. 23, §B.1 · pp. 23–24

B.2 LAYER NORMALIZATION

• Pre-LN crashed in GLM-130B’s early training, and Sandwich-LN also collapsed despite stabilizing a 4-billion-parameter text-to-image model.The authors also note that Pre-LN may harm performance after tuning compared with Post-LN.Source: §B.2 · p. 24

B.3 POSITIONAL ENCODING AND FEED-FORWARD NETWORK

• GLM-130B uses one-dimensional positional encoding with RoPE and a GeGLU feed-forward network; small-model ablations found RoPE and GeGLU improved performance.For short-span corruption, the second position dimension is discarded; for a long span at the end, positions become one-dimensional and generated tokens extend the final context position. GeGLU reduces feed-forward size to 8/3dH to preserve parameter count.Source: §B.3 · p. 25

B.4 PIPELINE PARALLEL ANALYSIS

• Gpipe splits data into micro-batches to reduce bubbles, while PipeDream-Flush interweaves forward and backward operations across stages to reduce activation memory (Huang et al., 2019; Narayanan et al., 2021).The passage contrasts these approaches with naive sequential pipeline execution, which has an unbearable amount of bubbles.Source: §B.4 · p. 25, Figure 11 · p. 26
• For p pipeline segments and m micro-batches, the default delivery strategy adds p − 1 forward and p − 1 backward bubbles, totaling (p − 1)(tf + tb).Ideal forward and backward time is m(tf + tb).Source: §B.4 · p. 25
• When m ≥ 4p, GPipe experiments show pipeline bubble time becomes negligible through forward recomputation that overlaps computation and communication.The authors conclude that pipeline bubbles do not seriously deplete training efficiency.Source: §B.4 · pp. 25–26
• With m = 176 and p = 8, GLM-130B’s optimal tensor parallelism was t = 4, yielding a 3.8% bubble share rather than scaling to t = 8.Increasing tensor parallelism reduces bubbles, but excessive scale lowers computational granularity and raises communication costs; the paper recommends scaling it slowly and not beyond the number of GPUs in one machine.Source: §B.4 · p. 26

B.5 INFERENCE ACCELERATION

• 7.0 to 8.4 times faster decoding was achieved by GLM-130B’s implementation than BLOOM-176B’s Huggingface Transformers implementation.The comparison used 16-bit precision with 8 × A100 (80G).Source: Table 9 · p. 27, §B.5 · p. 27
• The C++ FasterTransformer-based implementation accelerates inference by optimizing costly operations, fusing GPU kernels, selecting cuBLAS algorithms, pre-transposing parameters, and using half2 for FP16.The authors spent two months implementing GLM-130B in C++ because plain PyTorch can be intolerably slow for large language models.Source: §B.5 · p. 27
• The full FasterTransformer implementation is packaged as a plug-and-play Docker image, with work ongoing to adapt it to PyTorch by changing one line of code.The authors describe tolerable response speeds as important to popularizing LLMs.Source: §B.5 · p. 27

B.6 ACTIVATION OUTLIER ANALYSIS

• GLM-130B has outliers in 30% of dimensions, and omitting them during quantization can significantly degrade performance because they may encode memorized knowledge.While 99.98% of hidden-state dimensions stay below 6, two outlying dimensions can reach 50 or exceed 100; this also makes a prior higher-precision decomposition approach inapplicable.Source: §B.6 · pp. 27–28

B.8.1 QUANTIZATION RESULTS AT SCALES

• GLM maintains better LAMBADA performance than BLOOM at INT4 precision as model scale increases, while almost all models retain performance at INT8.The comparison covers GLM and BLOOM family models from 100M to 176B parameters.Source: §B.8.1 · p. 28, Table 10 · p. 29

B.8.2 WEIGHT DISTRIBUTION ANALYSIS

• GLM-130B’s w2 weight distribution is well-shaped with few outliers, unlike BLOOM-176B’s skewed w2 distribution, enabling GLM’s INT4 quantization with little performance loss.The analysis examines weight distributions in major linear layers.Source: §B.8.2 · p. 29, Figure 15 · p. 31

B.9 ABLATION ON CONTRIBUTION ATTRIBUTION

• On GLM-10B, the GLM objective accounts for 73% of LAMBADA improvement and 90% of MMLU improvement, while MIP chiefly helps text-similarity and coreference tasks.Across eight datasets, MIP may harm performance on WiC, ReCoRD, and Hellaswag; it mainly bridges gaps between self-supervised pre-training and text-similarity or coreference challenges.Source: Figure 14 · p. 29, §B.9 · p. 30

B.10 LESSONS LEARNED

• The lessons identify bidirectional attention as an alternative to GPT architectures, DeepNorm as a stabilizing Post-LN choice, and embedding-gradient shrink to 0.1 as a remedy for most numerical instability.The paper also notes that FP16 induces more instability but enables training and inference on diverse platforms.Source: §B.10 · p. 30
• GLM has an INT4 weight-quantization scaling law not observed in GPT-style BLOOM; the paper identifies more and better data, architectures and objectives, and sufficient training as future priorities.It also recommends configuring LLMs around the cluster and parallel strategy to use hardware effectively.Source: §B.10 · p. 30

C.1 MULTI-TASK INSTRUCTION PRE-TRAINING (MIP)

• MIP uses prompted instruction datasets for 5% of training tokens, including natural-language understanding and generation data and information-extraction data; DeepStruct tasks are reformulated for zero-shot generalization to new schemas.The authors leave systematic evaluation of GLM-130B’s information-extraction ability to future work.Source: §C.1 · p. 32, §C.2 · p. 32

C.2 DATA AND PROMPTS IN MIP FOR DEEPSTRUCT

• The appendix describes prompt formulations for dialogue-state tracking and slot filling, ACE05 event extraction, and relation extraction tasks expressed as sequence-to-sequence generation.The relation-extraction setup uses CoNLL04, NYT, and ACE2005, with raw text inputs and triple outputs.Source: §C.2.1 · p. 32, §C.2.2 · p. 33, §C.2.3 · p. 34

C.2.3 JOINT ENTITY AND RELATION EXTRACTION

• For joint entity and relation extraction, the paper supplements narrow relation schemas with KELM-aligned data and creates prompts for extraction and knowledge slot filling over Wikidata properties (Agarwal et al., 2021).The paper excludes other distantly supervised data such as T-REx because it can be extremely noisy.Source: §C.2.3 · pp. 35–36

C.2.4 NAMED ENTITY RECOGNITION

• Named entity recognition uses CoNLL03, OntoNotes 5.0, GENIA, and joint extraction datasets, with separate formulations for entity extraction and entity typing.The prompts extract entities by type or classify the type of a specified entity mention.Source: §C.2.4 · p. 36

C.2.5 RELATION CLASSIFICATION

• Based on TacRED, relation classification includes candidate-relation prediction, knowledge slot filling, and yes-or-no judgments about whether a proposed triple is correct.The section describes distant supervision as a common data-creation approach that aligns knowledge-base triples with text, while noting its assumption that alignments are correct under certain conditions.Source: §C.2.5 · p. 37

C.2.6 SEMANTIC ROLE LABELING

• Semantic role labeling uses CoNLL05, CoNLL12, and PropBank data to define semantic role labeling, semantic role filling, and predicate recognition tasks.These formulations ask the model to identify arguments for a marked verb, find arguments for a specified role, or identify the verb associated with a segment and role.Source: §C.2.6 · pp. 38–39

C.3 RESULT SOURCES FOR GPT-3, BLOOM-176B, AND OPT-175B

• GPT-3 comparison results mostly come from prior literature unless otherwise noted, while BLOOM-176B and OPT-175B results use published or repository sources; the authors could not independently evaluate OPT-175B because they lacked its checkpoint.The paper notes that it had not been granted the OPT-175B checkpoint despite several applications.Source: §C.3 · p. 39

C.4 PILE TEST-SET EVALUATION

• GLM-130B performs noticeably weaker than GPT-3 on phil_papers and pile_cc, likely due to its bilingual data and lack of more diverse, high-quality private corpora.The Pile evaluation reports BPB across 18 of 22 datasets to reduce vocabulary-related comparison mismatch, using [gMASK], a 1,024-token context, and bidirectional attention.Source: §C.4 · p. 39, Table 13 · p. 39

C.5 BIG-BENCH-LITE EVALUATION

• BIG-bench-lite evaluation covers 24 tasks, using option scoring for multiple-choice items and greedy decoding for generation tasks, with [MASK] and official few-shot prompts.The evaluation uses the official BIG-bench implementation to generate priming examples and calculate final scores (Srivastava et al., 2022).Source: §C.5 · p. 40

C.8 NATURAL LANGUAGE GENERATION

• GLM-130B outperforms LaMDA and GPT-3 (Davinci) on all three generation tasks, trails PaLM-540B slightly on data-to-text, and achieves higher ROUGE results on summarization.The evaluation uses one-shot prompts on WebNLG, E2E, and WikiLingua; bidirectional attention boosts performance over the unidirectional variant, which underperforms GPT-3 on all three datasets.Source: §C.8 · p. 41, Table 16 · p. 41

C.9 WINOGRAD-STYLE TASKS

• GLM-130B performs best on Winogender but marginally worse than GPT-3 and PaLM on Winograd273.The Winograd273 evaluation is zero-shot; the authors use a pronoun-to-[MASK] prompt they found better suited to GLM-130B.Source: §C.9 · p. 43, Table 17 · p. 43

C.10 CLOSED-BOOK QUESTION ANSWERING

• GLM-130B performs relatively poorly on Natural Questions but well on StrategyQA.The authors speculate that limited exposure to English corpora—roughly 200B English tokens—may leave factual knowledge insufficiently memorized, and suggest more training could improve performance.Source: §C.10 · pp. 43–44, Table 18 · p. 43

C.11 COMMONSENSE REASONING

• GLM-130B performs best on both Commonsense QA and MC-TACO among the evaluated language models.OPT is omitted from the comparison; these datasets were selected because PIQA, ARC, and OpenbookQA were included in MIP training.Source: Table 19 · p. 43, §C.11 · p. 44

C.12 FIXED LABEL DATASETS: A CASE STUDY IN NATURAL LANGUAGE INFERENCE

• GLM-130B’s “zero-shot” results on six NLI datasets may be much better because MIP exposed it to the task type.The datasets themselves were not incorporated in MIP, but other NLI datasets were, so this differs from the standard zero-shot setting.Source: §C.12 · p. 44, Table 20 · p. 44

C.13 SUPERGLUE

• On untuned SuperGLUE, bidirectional GLM-130B has less prompt-specific variance and higher performance than its unidirectional variant across tasks.Some tasks exceed 80% accuracy, but these results are for reference rather than relative comparison because seven of eight task training sets were included in MIP; full-parameter fine-tuning rapidly overfit, while P-Tuning v2 used 0.1%–3% of parameters.Source: §C.13 · pp. 44–45, Figure 17 · p. 45, Table 21 · p. 45

C.14 CHAIN-OF-THOUGHT PROMPTING

• Chain-of-thought prompting improves GLM-130B’s performance on symbolic and commonsense reasoning tasks.Evaluated tasks include Last letter concatenation, Coin Flip, Reverse List, Sports understanding, and Date understanding.Source: §C.14 · pp. 45–46, Figure 18 · p. 45

D SCALING AND EMERGENT ABILITIES IN GLM-130B

• GLM-130B shows logarithmic performance growth on some tasks and thresholded emergent gains on others as model size increases.The authors report these two scaling behaviors as similar to those observed in other large language models, while noting their causes remain unclear.Source: D SCALING AND EMERGENT ABILITIES IN… · pp. 46–47, Figure 19 · p. 46, Figure 20 · p. 47
• Table 15 reports detailed GLM-130B and BLOOM 176B results on MMLU, notes that prior literature reported no numerical GPT-3 175B accuracy, and says BLOOM was evaluated with Huggingface Transformers.The passage identifies MMLU as the benchmark (Hendrycks et al., 2021) and cites BLOOM (Scao et al., 2022), but supplies no result values.Source: Table 15 · p. 51

E CONTRIBUTIONS

• Conceived in December 2021, GLM-130B completed pre-training on July 3, 2022, while evaluation and applications continued; its progress involved multiple teams amid technical and engineering challenges.The collaborating groups were KEG, PACMAN, and THUNLP at Tsinghua University, as well as Zhipu.AI.Source: E CONTRIBUTIONS · p. 52

E.3 POST TRAINING

• Aohan Zeng and Zhengxiao Du handled the evaluation framework, while Aohan Zeng led language-modeling evaluation.Source: §E.3 · p. 52
• Aohan Zeng led MMLU and BIG-Bench evaluation; Xiao Liu and Aohan Zeng handled CLUE and FewCLUE evaluation.Source: §E.3 · p. 52
• Yifan Xu, Aohan Zeng, Xiao Liu, and Zihan Wang handled ethical evaluation; Xiao Liu, Jifan Yu, and Weng Lam Tam handled baseline evaluation.Source: §E.3 · p. 52
• Aohan Zeng, Zihan Wang, Xiao Liu, and Hanyu Lai worked on INT4 quantization, while Zihan Wang and Aohan Zeng handled inference acceleration.Source: §E.3 · p. 52
• Gouyang Zeng, Xu Han, Weilin Zhao, and Zhiyuan Liu handled low-resource inference.Source: §E.3 · p. 52
• Hanyu Lai, Jifan Yu, Xiaohan Zhang, Yufei Xue, Shan Wang, Jiecai Shan, Haohan Jiang, and Zhengang Guo worked on the demo and API.Source: §E.3 · p. 52
• Xiao Liu, Yuxiao Dong, and Jie Tang wrote the main paper; Xiao Liu, Aohan Zeng, and Zhengxiao Du wrote the Appendix.Source: §E.3 · p. 52

F A BRIEF HISTORY OF GLM-130B

• GLM-130B began as an effort to make a high-accuracy dense English-Chinese model publicly downloadable, choosing GLM at 130B to enable inference on one 8 × A100 server.The authors chose GLM (Du et al., 2022) for its performance in practice.Source: F A BRIEF HISTORY OF GLM-130B · p. 53
• Training faced resource shortages, scale-specific bugs, cross-platform operator and convergence issues, and engineering failures before launch on 96 A100 GPUs.After a GPU sponsorship, training began on May 6, 2022; the team adapted its code to Ascend 910, Hygon DCU, NVIDIA, and Sunway platforms.Source: F A BRIEF HISTORY OF GLM-130B · p. 53
• INT4 GLM-130B without post-training has negligible performance degradation, uses 25% of the GPU memory, and runs on 4 × RTX 3090 Ti or 8 × RTX 2080 Ti.The team planned further work to reduce resource requirements.Source: F A BRIEF HISTORY OF GLM-130B · pp. 53, 55

G BROADER IMPACT

• GLM-130B’s weights and code are open to anyone, unlike many large models that are privately owned or accessible only through paid inference APIs.The authors argue that its speed-up implementation and INT4 quantization also lower inference hardware requirements, broadening potential benefits for researchers, developers, small companies, and society.Source: G BROADER IMPACT · p. 55

G.1 IMPACT ON AI RESEARCH

• Open weights let researchers inspect model parameters and internal states, study LLM capabilities and flaws, and modify models to test improvement algorithms.INT4 quantization also enables inference on 4 × RTX 3090 or 8 × RTX 2080 Ti, GPUs that researchers can access through cloud services.Source: §G.1 · p. 55

G.2 IMPACT ON INDIVIDUAL DEVELOPERS AND SMALL COMPANIES

• Developers and small companies can deploy GLM-130B on accessible hardware, distill smaller task-specific models, and fine-tune on their own data to address domain-knowledge gaps.Open weights and code offer alternatives to paid inference APIs and prompt engineering for domain-specific applications (Sanh et al., 2019; Jiao et al., 2020).Source: §G.2 · p. 55

G.3 SOCIAL IMPACT

• The paper acknowledges risks from harmful synthetic-text uses and LLM flaws, while arguing open access enables transparency, research, and defenses.Risks include fraud, propaganda, and harassment (Weidinger et al., 2021; Sheng et al., 2021; Dev et al., 2021), as well as fairness, bias, privacy, and truthfulness issues (Zhang et al., 2021; Lin et al., 2022; Liang et al., 2021). The authors argue that access can support study of harmful uses and defenses such as synthetic-text detection (Gehrmann et al., 2019) and fake-news detection (Li et al., 2021).Source: §G.3 · pp. 55–56

H ENVIRONMENTAL IMPACT

• 257.01 metric tons of CO2 were emitted during GLM-130B’s 60-day training, roughly half GPT-3’s estimated 500-ton footprint.Training consumed 442.4 MWh; the paper attributes the comparison, probably, to efficient parallel strategies and NVIDIA hardware improvements. The emissions were roughly equivalent to the yearly emissions of 18 average Americans (Patterson et al., 2021).Source: H ENVIRONMENTAL IMPACT · p. 56