\section{Conclusion and Lessons}

We introduce \glm, a bilingual pre-trained language model that aims to facilitate open and inclusive LLM research.
\glm's technical and engineering undertakings generate insight into LLMs' architectures, pre-training objectives, training stability and efficiency, and affordable inference. Altogether, it contributes to the high quality of \glm in terms of both language performance on 112 tasks and ethical results on bias and toxicity benchmarks.
Our experiences of both success and failure are condensed into the following lessons learned for training 100B-scale LLMs:

\begin{insight}[\bf Bidirectional Architecture]
\rm The bidirectional-attention GLM is a strong  architecture alternative, in  addition to GPTs.
\end{insight}

\begin{insight}[\bf Platform-aware Configuration]
\rm Configure LLMs based on the cluster and parallel strategy used to squeeze hardware potential.
\end{insight}

\begin{insight}[\bf Improved Post-LN]
\rm Counter-stereotypically, DeepNorm, a type of Post-LN, is the option to stabilize \glm.
\end{insight}

\begin{insight}[\bf Training Stability Categorization]
\rm Unexpected training instability that LLMs suffer from arouses systematically and numerically.
\end{insight}

\begin{insight}[\bf Systematical Instability: FP16]
\rm Though FP16 induces more instability, it enables training and inference on diverse platforms.
\end{insight}

\begin{insight}[\bf Numerical Instability: Embedding Gradient Shrink]
\rm Shrinking embedding layer's gradient to its 0.1 can solve most numerical instability problems.
\end{insight}

\begin{insight}[\bf GLM's INT4 Quantization Scaling Law]
\rm GLM has a unique INT4 weight quantization scaling law unobserved in GPT-style BLOOM.
\end{insight}

\begin{insight}[\bf Future Direction]
\rm To create powerful LLMs, the main focus can be on 1) more and better data, 2) better architectures and pre-training objectives, and 3) more sufficient training.
\end{insight}

\clearpage

\hide{
We introduce \glm, an open bilingual pre-trained model that aims to promote transparency and inclusivity in LLM research.
Our engineering undertakings and unique insights into LLMs' architectures, objectives, and training process jointly contribute to its average strong language performance over 112 tasks and mitigated bias and toxicity benchmarks we evaluate.
We condense our experiences of both success and failure into the following valuable lessons to our community:

\begin{enumerate}[leftmargin=*,itemsep=0pt,parsep=0.2em,topsep=0.0em,partopsep=0.0em]
    \item Bidirectional-attention GLM can be stronger than unidirectional GPTs at large scale.
    \item Configure your LLMs based on your cluster and parallel strategy to squeeze hardware potential.
    \item Counter-stereotypically, DeepNorm, a type of Post-LN, is the LN option to stabilize \glm.
    \item Instabilities that LLMs suffer from in pre-training arouses systematically and numerically.
    \item FP16 is a challenging but rewarding decision for \glm: it induces more systematical instability, but is a must for enabling LLMs to train and inference on inclusive ranges of platforms.
    \item Shrinking embedding layer's gradient to its 0.1 can solve most numerical instability problems.
    \item GLM architecture presents a unique scaling law of INT4 weight quantization unseen in BLOOM.
    \item To create inclusive LLMs better than those of big companies', community's later focus should be on 1) more and better data, 2) better architectures and objectives, and 3) more sufficient training.
\end{enumerate}

}

\section{Introduction}

Large language models (LLMs), particularly those with over 100 billion (100B) parameters~\citep{brown2020language,thoppilan2022lamda,rae2021scaling,chowdhery2022palm,wang2021ernie}, have presented attractive
scaling laws~\citep{wei2022emergent}, where emergent zero-shot and few-shot capabilities suddenly arouse.
Among them, GPT-3~\citep{brown2020language} with 175B parameters pioneers the studies of 100B-scale LLMs by strikingly generating better performance with 32 labeled examples than the fully-supervised BERT-Large model on a variety of benchmarks.
However, both GPT-3 (and other 100B-scale ones)---the model itself---and how it can be trained, have been thus far not available to the public.
It is of critical value to train a high-quality LLM of such scale with both the model and training process shared with everyone.

We thus \textit{aim to pre-train an open and highly-accurate 100B-scale model} with ethical concerns in mind.
Over the course of our attempt, we come to realize that pre-training a dense LLM at such a scale raises numerous unexpected technical and engineering challenges compared to training 10B-scale models, in terms of pre-training efficiency, stability, and convergence.
Similar difficulties have also been concurrently observed in training OPT-175B~\citep{zhang2022opt} and BLOOM-176B~\citep{scao2022what}, further demonstrating the significance of GPT-3 as a pioneer study.

\input{tables/tldr_table}

In this work, we introduce the pre-training of a 100B-scale model---\glm, in terms of engineering efforts, model design choices, training strategies for efficiency and stability, and quantization for affordable inference.
As it has been widely realized that it is computationally unaffordable to empirically enumerate all possible designs for training 100B-scale LLMs, we present not only the successful part for training \glm but also many of the failed options and lessons learned.
Particularly, the training stability is {the} decisive factor in the success of training models of such a scale.
Different from practices such as manually adjusting learning rates in OPT-175B and using embedding norm in the sacrifice of performance in BLOOM-176B, we experiment with various options and find the strategy of embedding gradient shrink can significantly stabilize the training of \glm.

Specifically, \glm is a bilingual (English and Chinese) bidirectional dense model with 130 billion parameters, pre-trained over 400 billion tokens
on a cluster of 96 NVIDIA DGX-A100 (8$\times$40G) GPU nodes
between May 6 and July 3, 2022.
Instead of using the GPT-style architecture, we adopt the  General Language Model (GLM) algorithm~\citep{du2022glm} to leverage its bidirectional attention advantage and autoregressive blank infilling objective.
Table~\ref{tab:intro} summarizes the comparison between \glm, GPT-3 and another two open-source efforts---OPT-175B and BLOOM-176B, as well as PaLM 540B~\citep{chowdhery2022palm}---a 4$\times$ larger model---as a reference.

Altogether, the conceptual uniqueness and engineering efforts enable \glm to exhibit performance that surpasses the level of GPT-3 on a wide range of benchmarks (in total 112 tasks) and also outperforms PaLM 540B in many cases, while outperformance over GPT-3 has not been observed in OPT-175B and BLOOM-176B (Cf. Figure~\ref{fig:intro} (a)).
For zero-shot performance, \glm is better than GPT-3 175B (+5.0\%), OPT-175B (+6.5\%), and BLOOM-176B (+13.0\%) on LAMBADA~\citep{paperno2016lambada}, and achieves 3$\times$ better performance than GPT-3 on Big-bench-lite~\citep{srivastava2022beyond}.
For the 5-shot MMLU~\citep{hendrycks2021measuring} tasks, it is  better than GPT-3 175B (+0.9\%) and BLOOM-176B (+12.7\%).
As a bilingual LLM also in Chinese, it offers significantly better results than ERNIE TITAN 3.0 260B~\citep{wang2021ernie}---the largest Chinese LLM---on 7 zero-shot CLUE~\citep{xu2020clue} datasets (+24.26\%) and 5 zero-shot FewCLUE~\citep{xu2021fewclue} ones (+12.75\%).
Importantly, as summarized in Figure~\ref{fig:intro} (b), \glm as an open model is associated with \textit{significantly less bias and generation toxicity than its 100B-scale counterparts}.

Finally, we design \glm to empower as many people as possible to conduct 100B-scale LLM studies.
First, instead of using 175B+ parameters as OPT and BLOOM, the 130B size is decided because such a size supports inference on a single A100 (8$\times$40G) server.
Second, to further lower the GPU requirements, we quantize \glm into INT4 precision without quantization aware training  while OPT and BLOOM can only reach INT8.
Due to a unique property of the GLM architecture, \glm's INT4 quantization introduces negligible performance degradation, e.g., -0.74\% on LAMBADA and even +0.05\% on MMLU, making it still better than the uncompressed GPT-3.
This enables \glm's fast inference with performance guarantee on a server of 4$\times$RTX 3090 (24G) or 8$\times$RTX 2080 Ti (11G), \textit{the most ever affordable GPU required for using 100B-scale LLMs to date. }

We open-source the model checkpoints, code, training logs, related toolkits, and lessons learned.

\hide

\section{Introduction}
Large Language Models (LLMs), which refers to transformer-based~\citep{vaswani2017attention} statistical language models with an enormous quantity of parameters trained on web-scale text contents, have experienced a surge in the past few years.
Particularly, those who exceed 100 billion parameters, to which we use ``LLMs'' to refer to in a narrow sense~\citep{brown2020language,thoppilan2022lamda,rae2021scaling,chowdhery2022palm,wang2021ernie},  have presented attractive but mysterious scaling laws, where emergent zero-shot and few-shot capabilities suddenly arouse~\citep{wei2022emergent}.
Despite much controversy on whether merely scaling LLMs leads to general machine intelligence, now it is generally accepted that we shall look into LLMs.

Nevertheless, academic  researchers and individual developers have been suffered from limited access and poor usability of LLMs.
For accessibility, most LLMs are unavailable to public and a few of them~\citep{brown2020language,lieber2021jurassic} provide intransparent limited APIs with fees.
For usability, even if these LLMs are open-sourced, most individuals and academic institutions can hardly afford the outrageous cost for inference, let alone further tuning.
``LLM Monopoly'' does no good to LLM's development, as it requires the whole community's exertion, including big companies, academia, and individuals, to push LLM's boundary and realize its promised welfare to people.

In this work, we introduce \glm as an initial step to fulfill our commitment of ``LLM Inclusivity''.
It is a GLM~\citep{du2022glm} language model possessing 130 billion parameters and is pre-trained over both English and Chinese corpora for 400 billion tokens.
\glm's effort is in concurrence with recent OPT-175B~\citep{zhang2022opt} and BLOOM-176B~\citep{scao2022what} to open-source and popularize the research and use of LLMs.
However, compared to them \glm is uniquely focusing on offering our community a powerful and high-usability LLM, from the very beginning of its conception and design.
Specifically, our elaboration covers the four main aspects concerning an LLM's architecture, objective, training, and inference. For the first time,
\begin{itemize}[leftmargin=*,itemsep=0pt,parsep=0.2em,topsep=0.0em,partopsep=0.0em]
    \item \textbf{Architecture}: we adopt bidirectional GLM instead of unidirectional GPT in an LLM and demonstrate its superiority in downstream evaluation. Other necessary components are also identified.
    \item \textbf{Objective}: we include Multitask Instruction Pre-training (MIP), along with the autoregressive blank infilling objective of GLM, as the LLM's objective.
    \item \textbf{Training Strategy}: we unveil part of hidden reasons behind the extreme instability in LLM's pre-training, and propose effective strategies to stabilize its training without harming performance.
    \item \textbf{Inference}: we quantize \glm, a 130B LLM, into INT4 precision with little performance loss to allow its fast inference on even 4 $\times$ RTX 3090 Ti (24G) or 8 $\times$ RTX 2080 Ti (11G).
\end{itemize}

In the end, it turns out that our design and training enable \glm, a bilingual language model, to outperform similar-sized LLMs such as GPT-3 175B~\citep{brown2020language}, BLOOM-176B~\citep{scao2022what}, and ERNIE Titan 3.0 (260B)~\citep{wang2021ernie} over a wide range of zero-shot and few-shot benchmarks, including language modeling~\citep{paperno2016lambada,marcinkiewicz1994building}, multiple-choice question answering~\citep{hendrycks2021measuring}, and versatile reasoning challenges~\citep{srivastava2022beyond} and Chinese language tasks~\citep{xu2020clue,xu2021fewclue}.
It even outstrips PaLM-540B~\citep{chowdhery2022palm}, a 4$\times$ larger language model, on zero-shot language modeling dataset LAMBADA~\citep{paperno2016lambada} and zero-shot BIG-bench-lite~\citep{srivastava2022beyond} to set up new records.

To sum up, we introduce our following contributions regarding LLMs and \glm in this work
\begin{itemize}[leftmargin=*,itemsep=0pt,parsep=0.2em,topsep=0.0em,partopsep=0.0em]
    \item We demonstrate our long-term commitment towards ``LLM Inclusivity'' technically and economically. It cares about the participation of all community members in LLMs' research and application. We appeal to all individuals and organizations in this field for joint efforts in this commission.
    \item We introduce \glm, an open and inclusive bilingual LLM. It is fully optimized for high quality and popularization on its architecture, training and inference strategies. Our experiments on extensive zero-shot and few-shot benchmarks shows that \glm can be competent and even better than its counterparts such as GPT-3 175B, OPT-175B, and BLOOM 176B.
    \item We open-source model checkpoints, code, training logs, and toolkit related to \glm. Based on which people can easily reproduce all our evaluation, and do fast offline inference on their own machines with hardwares as popularized as 4 $\times$ RTX 3090 (24G) or 8 $\times$ RTX 2080 Ti (11G).
\end{itemize}

\subsection{Platform-Aware Parallel Strategies and Model Configurations} \label{sec:parallel_strategy}

\glm is trained on a cluster of 96 DGX-A100 GPU (8$\times$40G) servers with a 60-day access.
The goal is to pass through as many tokens as possible, as a recent study~\citep{hoffmann2022training} suggests that most existing LLMs are largely under-trained.

\vvpara{The 3D Parallel Strategy.}
The data parallelism~\citep{valiant1990bridging} and tensor model parallelism~\citep{shoeybi2019megatron} are the de facto practices for training billion-scale models~\citep{gpt-j,du2022glm}.
To further handle
the huge GPU memory requirement and the decrease in overall GPU utilization resulted from applying tensor parallel between nodes---as 40G rather than 80G A100s are used for training \glm, we combine the pipeline model parallelism with the other two strategies to form a 3D parallel strategy.

The pipeline parallelism divides the model into sequential stages for each parallel group, and to further minimize bubbles introduced by pipeline, we leverage the PipeDream-Flush~\citep{narayanan2021memory} implementation from DeepSpeed~\citep{rasley2020deepspeed} to train \glm with a relative big global batch size (4,224) to reduce time and GPU memory wasting.
Through both numerical and empirical examinations, we adopt 4-way tensor parallelism and 8-way pipeline parallelism, reaching 135 TFLOP/s per GPU (40G)
(Cf. Appendix~\ref{app:pipeline} for details).

\vvpara{\glm Configurations.}
We aim to enable our 100B-scale LLM to run a single DGX-A100 (40G) node in FP16 precision.
Based on the hidden state dimension of 12,288 we adopt from GPT-3, the resultant model size has to be no more than 130B parameters, thus \glm.
To maximize GPU utilization, we configure the model based on the platform and its corresponding parallel strategy.
To avoid insufficient memory utilization in the middle stages
due to the additional word embedding at both ends, we balance the pipeline partition by removing one layer from them, making 9$\times$8-2=70 transformer layers in \glm.

During the 60-day access to the cluster, we manage to train \glm for 400 billion tokens (roughly 200 billion each for Chinese and English) with a fixed sequence length of 2,048 per sample.
For the [gMASK] training objective, we use a context window of 2,048 tokens.
For the [MASK] and multi-task objectives, we use a context window of 512 and concatenate four samples together to cater the 2,048-sequence-length.
We warm-up the batch size from 192 to 4224 over the first 2.5\% samples.
We use AdamW~\citep{loshchilov2017decoupled} as our optimizer with $\beta_1$ and $\beta_2$ set to 0.9 and 0.95, and a weight decay value of 0.1.
We warm up the learning rate from $10^{-7}$ to $8\times 10^{-5}$ over the first 0.5\% samples, then decay it by a $10\times$ cosine schedule.
We use a dropout rate of 0.1 and clip gradients using a clipping value of 1.0 (Cf. Table~\ref{tab:config} for the full configurations).

\hide

\subsection{Parallel Strategies} \label{sec:parallel_strategy}
Training LLMs can be outrageously expensive.
It is said that GPT-3~\citep{brown2020language} was trained over 10,000 NVIDIA V100 GPUs for months at substantial expenses.
However, recent study~\citep{hoffmann2022training} shows that existing LLMs are largely under-trained; what matters in the first place to LLMs' performance is the number of trained tokens.
Thus for Inclusive LLM, an optimized framework that fully exploits computing potential is never overemphasized.

Here we introduce \glm's framework-level insights, including the 3D parallel strategy (Cf. Figure~\ref{fig:3d-parallelism} in Appendix), LLM configuration principles, and computation graph optimization.
They in all contribute \todo{} speeding up to pre-training, making a crucial improvement to \glm's quality.

\vvpara{3D Parallel Strategy.}
\aohan{I think this section should be downplayed as more details can be found in the appendix}
Parallel strategies, including data parallelism~\citep{valiant1990bridging} and tensor model parallelism~\citep{shoeybi2019megatron}, are de facto practices for training billion-scale language models.
However, when models continue to scale up to over 100B, the scaling up of a tensor parallelism group become insufficient.
On one hand, as the computation per layer is evenly distributed in a group, a single node's computational granularity would decrease when group sizes grow up, causing a performance drop of matrix multiplication operator \texttt{GEMM} and a decrease in overall utilization.
On the other hand, if group size exceeds certain threshold (e.g., number of GPUs per node), the \texttt{All-Reduce} operation becomes a bottleneck due to the expensive cross-node communication.

Therefore, in \glm we combine another parallel strategy---pipeline model parallelism---with two established practices to form the 3D parallel strategy.
The pipeline parallelism divides the model into sequential stages for each parallel group, and to minimize bubbles introduced by pipeline, we refer to Gpipe~\citep{huang2019gpipe} and PipeDream-Flush~\citep{narayanan2021memory} implementation from DeepSpeed~\citep{rasley2020deepspeed} to reduce time and GPU memory wasting (Cf. Appendix~\ref{app:pipeline}).

We analyze the bubble ratio in \glm's pre-training. Given the number of pipeline stages $p$, the number of micro-batches in a parallel group $m$, and the time for forward and backward per micro-batch $t_f$ and $t_b$.
In ideal case without pipeline, forward and backward take $t_{\mathrm{ideal}} = m(t_f + t_b)$.
But with pipeline, the splitting causes $p - 1$ forward and backward bubbles respectively for a total time of $t_{\mathrm{bubble}} = (p - 1)( t_f + t_b)$.
If there are $n$ GPUs in a parallel group, in practice we assign each GPU only with one tensor parallel split (group size $t$) from one pipeline parallel stage (altogether $p$ stages), which results in $n=t\times p$.
Then we have
\begin{equation}
    \begin{split}
        \text{bubble-ratio} = \frac {t_{\mathrm{bubble}}} {t_{\mathrm{ideal}} + t_{\mathrm{bubble}}} = \frac {p - 1} {m + p - 1} = \frac {n/t - 1} {m + n/t - 1}
    \end{split}
\end{equation}
\noindent where increasing tensor parallel splits $t$ and the number of micro-batches $m$ could reduce the bubble ratios.
But $t$ should not exceed the number of GPUs per node and a too large $m$ is known to harm language models' performance~\citep{you2020large}.
After testing on our 96 DGX-A100 nodes, we adopt $t=4$, $m=176$, and $p=8$, which leads to a bubble ratio of only 3.8\% and a utilization of 135 TFLOP/s per GPU in pre-training.

\vvpara{Principles for LLM Configuration.}
Compared to ordinary-sized language models, configurations of LLMs should be based on the computing platforms to exploit training efficiency.
In \glm, these peculiarly platform-dependent configurations include:
\begin{itemize}[leftmargin=*,itemsep=0pt,parsep=0.2em,topsep=0.0em,partopsep=0.0em]
    \item \textbf{Total size}: one of our commission for inclusivity is that \glm should be able to inference on a single DGX-A100 (40G) node in FP16 precision. Based on 12,288 hidden state dimension we adopt following~\citep{brown2020language}, it leads to not more than 130B parameters in our LLM and each parallel group contains not more than 9 layers.
    \item \textbf{Number of layers}: as the pipeline parallel stages in both ends in LLMs store additional word embeddings' parameters, to avoid insufficient memory utilization on the middle stages, they should contain one less layer than others. Given an $l$-layer \glm stage contains $k$ layers, we find the the most balanced partition being $l=kp - 2$, resulting in $9\times8 - 2=70$ transformer layers in \glm.
\end{itemize}

\yuxiao{how about move the configuration to sec 2.2?}

\begin{insight}
\rm Configure your LLMs based on the computing cluster and parallel strategy. Squeezing hardware potential is always a top priority for LLM pre-training out of big companies.
\end{insight}

\vvpara{Computation Graph Optimization.}
GPU operators, including compute-intensive and memory-intensive, manage the computation of modern neural networks in GPUs.
In Transformer, there is another time-consuming access-intensive element-wise operators in large quantities.
We may consider operator fusion for them in computational graphs to reduce memory access for acceleration.
Specifically, we use the JIT method provided by the PyTorch framework to implement operator fusion on the operator combinations of \texttt{bias+dropout+add}, \texttt{bias+gelu}, and the rotary position encoding (\cite{su2021roformer}, Cf. Section~\ref{sec:glm-arch}) used in \glm.
We also implement a customized CUDA kernel function from Megatron-LM for the \texttt{scale+mask+softmax} operation.
\yuxiao{possible to have a number to show off the optimization effort?, otherwise, we may have to remove this para or move it to appendix}

\hide{
\subsection{mixed precision training strategy}

Existing exascale pre-trained language models are often trained in BF16 half-precision format to save memory and speed up computation \cite{rae2021scaling, chowdhery2022palm}. BF16 format has the same representation range as single-precision, which can reduce the overflow up and down during the training of exascale language models. However, the efficient computation of BF16 format is only supported by a few high-end gas pedals (e.g., Nvidia's A100, 3090, etc.), and the parameters of models trained in BF16 format cannot be converted to FP16 format due to the difference in representation range, and regressing to single-precision for use will lead to a great waste of time and memory. Therefore, pre-trained language models in BF16 format greatly increase the threshold for researchers to use. Our work is devoted to exploring mixed precision training strategies using FP16 format in the training of very large scale language models.

The \ref{sec:mixed-precision-training} section of this paper introduces a number of mixed-precision strategies commonly used in training deep neural networks, including techniques such as preserving single-precision parameter copies and dynamic scaling of the loss function, which have successfully accelerated the training process for a range of models. However, in our experiments, we observed that the direct application of these strategies to the training of very large scale language models at the scale of tens and hundreds of billions in FP16 half-precision format still results in training non-convergence. It is not surprising that the pre-training model OPT \cite{zhang2022opt} with 100 billion parameters using FP16 precision also experienced multiple divergences during training, which implies that we need to propose targeted mixed-precision strategies for the training of very large scale language models.

Each attention head in the multi-headed attention layer of the Transformer model is computed as follows
\begin{equation}
    \operatorname{Attention}\left(\boldsymbol{Q}_i, \boldsymbol{K}_i, \boldsymbol{V}_i\right) = \operatorname{softmax}\left(\frac{\ boldsymbol{Q}_i\boldsymbol{K}_i^{\top}}{\sqrt{d}}\right) \boldsymbol{V}_i
\end{equation}

We observed that, especially in the case of large sizes of individual attention heads in large-scale language models, the elements in the fraction matrix $\boldsymbol{Q}_i\boldsymbol{K}_i^{\top}\left/\sqrt{d}\right.$ may be much larger than the input and overflow. Subsequent researchers have explored this problem, and Ming Ding et al. found that the fraction matrix inside each head in multihead attention has the property of small variance but large bias when training a multimodal model with billions of parameters, and targeted the PB-Relax method \cite{ding2021cogview}, which uses the Softmax function translation invariant property eliminates the bias in the fractional matrix, thus alleviating the overflow problem. Its core formula is as follows.
\vspace{0.5em}
\begin{equation}
   \operatorname{softmax}\left(\frac{\boldsymbol{Q}_i\boldsymbol{K}_i^{\top}}{\sqrt{d}}\right) =
   \operatorname{softmax}\left(\left(\frac{\boldsymbol{Q}_i\boldsymbol{K}_i^{\top}}{alpha\sqrt{d}} - \max\left(\frac{\boldsymbol{Q}_i\ boldsymbol{K}_i^{\top}}{\alpha\sqrt{d}}\right)\right)\times\alpha\right)
\end{equation}

Our experimental results show that the PB-Relax method can solve the training non-convergence problem for models with tens of billions of parameters, however, the training scattering phenomenon still occurs when the number of parameters is expanded by tens and hundreds of billions. Numerically, the PB-Relax method still cannot avoid the overflow phenomenon when the variance within the fraction matrix is relatively large, so we propose a hybrid precision-based calculation method of
\vspace{0.5em}
\begin{equation}
   \operatorname{softmax}\left(\frac{\boldsymbol{Q}_i\boldsymbol{K}_i^{\top}}{\sqrt{d}}\right) =
   \operatorname{FP16}\left(\operatorname{softmax}\left(\operatorname{FP32}\left(\frac{\boldsymbol{Q}_i\boldsymbol{K}_i^{\top}}{\ alpha\sqrt{d}}\right)\times\alpha\right)\right)
\end{equation}

In this method, the more computationally intensive $\boldsymbol{Q}_i\boldsymbol{K}_i^{\top}\left/\sqrt{d}\right.$ is still computed in half-precision to ensure efficiency, and $\alpha$ is pre-divided to avoid overflow. the Softmax operation is computed in single precision, and the The softmax operation is computed in single precision, and the single-precision fraction matrix is multiplied back to $\alpha$ before computation to ensure that the result remains unchanged. $\alpha$ is a data- and model-dependent hyperparameter, and in our experiments, taking $\alpha$ equal to the number of layers in the network avoids overflow. In terms of efficiency, the only cost of this mixed-precision strategy is the computation of the single-precision Softmax function, however, we find that matrix multiplication is the absolute dominant model computation in the pre-training of very large models. Compared to computing the Softmax function at half precision, the overall computational efficiency is only about 1\% loss. We were surprised to find that computing Softmax with half-precision and using the PB-Relax method to mitigate the overflow instead reduced the training efficiency by 8\%, which may be due to the reduction in efficiency caused by too intensive manipulation of the fractional matrix with large shapes in the multi-headed attention.
}
\section{The Design Choices of \glm}
\vspace{-2mm}
The architecture of a machine learning model defines its inductive bias.
However, it has been realized that it is computationally unaffordable to explore various architectural designs for LLMs.
We introduce and explain the unique design choices of \glm.

\vspace{-4mm}
\subsection{\glm's Architecture} \label{sec:glm-arch}
\vspace{-2mm}

\vvpara{GLM as  Backbone.}
Most recent 100B-scale LLMs, such as GPT-3, PaLM, OPT, and BLOOM, follow the traditional GPT-style~\citep{radford2019language} architecture of decoder-only autoregressive language modeling.
In \glm, we instead make an attempt to explore the potential of a bidirectional GLM---General Language Model~\citep{du2022glm}---as its backbone.

GLM is a transformer-based language model that leverages autoregressive blank infilling as its training objective.
Briefly, for a text sequence $\boldsymbol{x}=[x_1,\cdots,x_n]$, text spans $\{\boldsymbol{s}_1,\cdots,\boldsymbol{s}_m\}$ are sampled from it, each of which $\boldsymbol{s}_i$ denotes a span of consecutive tokens $[s_{i,1},\cdots,s_{i,l_i}]$ and is replaced (i.e., corrupted) with a single mask token to form $\boldsymbol{x}_{\text{corrupt}}$.
The model is asked to recover them autoregressively.
To allow interactions between corrupted spans, their visibility to each other is decided by a randomly sampled permutation on their order.
The pre-training objective is then defined as
\begin{equation}

    \mathcal{L

    = \max_\theta \mathbb{E}_{\boldsymbol{z}\sim Z_m}\left[\sum_{i=1}^m\log \prod_{j=1}^{l_i} p(s_{i,j}|\boldsymbol{x}_{\text{corrupt}},\boldsymbol{s}_{\boldsymbol{z}_{<i}},\boldsymbol{s}_{i,<j}) \right]
    \label{eqn:objective}
\end{equation}
\noindent where $Z_m$ denotes the set of the sequence's all permutations and $\boldsymbol{s}_{\boldsymbol{z}_{<i}}$ denotes $[\boldsymbol{s}_{z_1},\cdots,\boldsymbol{s}_{z_{i-1}}]$.

GLM's bidirectional attention over unmasked (i.e., uncorrupted) contexts distinguishes \glm from GPT-style LLMs in which the unidirectional attention is used.
To support both understanding and generation, it mixes two corruption objectives, each indicated by a special mask token:
\begin{itemize}[leftmargin=*,itemsep=0pt,parsep=0.2em,topsep=0.0em,partopsep=0.0em]
    \item \textbf{[MASK]}: short blanks in sentences whose lengths add up to a certain portion of the input.
    \item \textbf{[gMASK]}: random-length long blanks at the end of sentences with prefix contexts provided.
\end{itemize}

\begin{wrapfigure}{r}{6cm}
    \small
    \vspace{-6mm}
    \centering
    \includegraphics[width=1.0\linewidth]{figures/architecture.pdf}
    \vspace{-6mm}
    \caption{\glm and LLMs of similar scale on zero-shot LAMBADA language modeling.
    Details on GLM's bidirectional attention are provided in \citet{du2022glm}.}
    \label{fig:lambada}
    \vspace{-8mm}
\end{wrapfigure}

Conceptually, the blank infilling objective with bidirectional attention enables a more effective comprehension of contexts than GPT-style models:
when using [MASK], \glm behaves as BERT~\citep{devlin2019bert} and T5~\citep{raffel2020exploring};
when using [gMASK], \glm behaves similarly to PrefixLM~\citep{liu2018generating,dong2019unified}.

Empirically,
\glm
offers a record-high accuracy of 80.2\% on zero-shot LAMBADA  by outperforming both GPT-3 and PaLM 540B in Figure \ref{fig:lambada}.
By setting the attention mask, \glm's  unidirectional variant is comparable to GPT-3 and OPT-175B.
Our observations are in line with existing findings~\citep{liu2018generating,dong2019unified}.

\vvpara{Layer Normalization (LN,~\cite{ba2016layer}).}
Training instability is one major challenge for training LLMs~\citep{zhang2022opt,scao2022what,chowdhery2022palm} (Cf. Figure ~\ref{fig:collapse} in Appendix for  collapses in training several 100B-scale models).
A proper choice of LNs can help stabilize the training of LLMs.
We experiment with existing practices, e.g., Pre-LN~\citep{xiong2020layer}, Post-LN~\citep{ba2016layer},
Sandwich-LN~\citep{ding2021cogview}, which are unfortunately incapable of stabilizing our \glm test runs (Cf. Figure~\ref{fig:layernorm} (a) and Appendix~\ref{app:ln} for details).

Our search is later focused on Post-LN due to its favorable downstream results in preliminary experiments though it does not stabilize \glm.
Fortunately, one of the attempts on Post-LN initialized with the newly-proposed DeepNorm~\citep{wang2022deepnet} generates promising training stability.
Specifically, given the number of \glm's layers $N$,

    we adopt $\textrm{DeepNorm}(\boldsymbol{x}) = \textrm{LayerNorm}(\alpha\cdot \boldsymbol{x} + \textrm{Network}(\boldsymbol{x}))$, where $\alpha=(2N)^\frac{1}{2}$,
    and

   apply the Xavier normal initialization with the scaling factor of $(2N)^{-\frac{1}{2}}$ to \texttt{ffn}, \texttt{v\_proj} and \texttt{out\_proj}. Additionally, all bias terms are initialized to zero.
Figure~\ref{fig:layernorm} shows it significantly benefits the training stability of \glm.

\vvpara{Positional Encoding and FFNs.}
We empirically test different options for positional encoding (PE) and FFN improvements in terms of both training stability and downstream performance (Cf. Appendix \ref{app:pe-ffn} for details).
For PEs in \glm, we adopt Rotary Positional Encoding (RoPE, \cite{su2021roformer}) rather than  ALiBi~\citep{press2021train}.
To improve FFNs in Transformer, we pick GLU with the GeLU~\citep{hendrycks2016gaussian} activation as the replacement.

\vspace{-2mm}
\subsection{\glm's Pre-Training Setup} \label{sec:training_objective}
\vspace{-2mm}

Inspired by recent works~\citep{aribandi2022ext5,wei2022finetuned,sanh2022multitask}, the \glm pre-training objective includes not only the self-supervised GLM autoregressive blank infilling) but also multi-task learning for a small portion of tokens.
This is expected to help boost its downstream zero-shot performance.

\begin{figure}[t]
    \centering
    \includegraphics[width=.92\linewidth]{figures/layernorm.pdf}
    \vspace{-2mm}
    \caption{Trials on different LayerNorms for \glm training. It turns out that DeepNorm is the most stable one, as it has small gradient norm and does not spike in the early stage training.}
    \vspace{-4mm}
    \label{fig:layernorm}
\end{figure}

\vvpara{Self-Supervised Blank Infilling (95\% tokens).}
Recall that \glm uses both [MASK] and  [gMASK] for this task.
Specifically, [MASK] is used to mask consecutive spans in 30\% of training tokens for blank infilling. The lengths of spans follow a Poisson distribution ($\lambda=3$) and add up to 15\% of the input.
For the other 70\% tokens, the prefix of each sequence is kept as context and [gMASK] is used to mask the rest of it.
The masked length is sampled from the Uniform distribution.

The pre-training data includes 1.2T Pile~\citep{gao2020pile} English corpus, 1.0T Chinese WudaoCorpora~\citep{yuan2021wudaocorpora},
and 250G Chinese corpora (including online forums, encyclopedia, and QA) we crawl from the web, which form a balanced composition of English and Chinese contents.

\vvpara{Multi-Task Instruction Pre-Training (MIP, 5\% tokens).}
T5~\citep{raffel2020exploring} and ExT5~\citep{aribandi2022ext5} suggest that multi-task learning in pre-training can be more helpful than fine-tuning, we thus propose to include a variety of instruction prompted datasets including language understanding, generation, and information extraction in \glm's pre-training.

Compared to recent works~\citep{wei2022finetuned,sanh2022multitask} that leverage multi-task prompted fine-tuning to improve zero-shot task transfer, MIP only accounts for 5\% tokens and is set in the pre-training stage to prevent spoiling LLMs' other general ability, e.g., unconditional free generation.
Specifically, we include 74 prompted datasets from ~\citep{sanh2022multitask,wang2022deepstruct}, listed in Appendix~\ref{app:data} and Table~\ref{tab:mip}.
\glm users are suggested to {avoid evaluating its zero-shot and few-shot capabilities on these datasets} according to the criterion illustrated in Section~\ref{sec:results}.

\hide

\section{Algorithm Designs}
In this section, we introduce the algorithm designs concerning \glm's architectural thoughts and the pre-training objectives.
Compared to most existing LLMs such as GPT-3~\citep{brown2020language}, PaLM-540B~\citep{chowdhery2022palm}, OPT~\citep{zhang2022opt}, and BLOOM~\citep{scao2022what}, \glm is not a GPT architecture language model, but a bidirectional General Language Model (GLM, \cite{du2022glm}) trained to fill in the blanks.
The design introduces significant advantages, as well as emerged challenges.
In this section, we explain the reasons to adopt the GLM and the solutions for conquering the challenges in scaling it up to 130 billion parameters.

\subsection{\glm's Architecture} \label{sec:glm-arch}
Architectures define the inductive bias for machine learning models, which matters to their scaling efficiency~\citep{tay2022scaling}.
However, due to the outrageous computing costs of LLMs, it is unaffordable to exploit various architectural designs.
In \glm, based on real experiments we present several key insights on architectural designs for the goal of inclusive LLMs.

\vvpara{Backbone Architecture.}
In the past few years, the community has witnessed the surge of a series of different language model architectures~\citep{radford2019language,devlin2019bert,yang2019xlnet,raffel2020exploring}.
Nevertheless, existing LLMs mostly follow the traditional GPT-style~\citep{radford2019language} architecture of decoder-only autoregressive language modeling.
It is probably because LLMs can be extremely unstable in training and are thus too expensive to test new ideas once failed.
However, it is questionable that GPT-style architecture would be the best option for LLMs.

In \glm, we take an audacious step to adopt a new architecture---GLM~\citep{du2022glm}---for pre-training a 130-billion LLM.
In brief, GLM is transformer-based language model which leverages autoregressive blank infilling as its basic training objective.
For a text sequence $\boldsymbol{x}=[x_1,\cdots,x_n]$, text spans $\{\boldsymbol{s}_1,\cdots,\boldsymbol{s}_m\}$ are sampled from it, each of which denotes a span of consecutive tokens $[s_{i,1},\cdots,s_{i,l_i}]$.
They are replaced (i.e., corrupted) with a single mask token respectively to form $\boldsymbol{x}_{\text{corrupt}}$ and the model is asked to recover them autoregressively.
To allow interactions between corrupted spans, their visibility to each other are decided by a randomly sampled permutation on their order.
Then, we can define the pre-training objective for \glm as
\begin{equation}

    \mathcal{L}_{\text \glm}

    = \max_\theta \mathbb{E}_{\boldsymbol{z}\sim Z_m}\left[\sum_{i=1}^m\log \prod_{j=1}^{l_i} p(s_{i,j}|\boldsymbol{x}_{\text{corrupt}},\boldsymbol{s}_{\boldsymbol{z}_{<i}},\boldsymbol{s}_{i,<j}) \right]
    \label{eqn:objective}
\end{equation}
\noindent where $Z_m$ denotes the set of the sequence's all permutations and $\boldsymbol{s}_{\boldsymbol{z}_{<i}}$ denotes $[\boldsymbol{s}_{z_1},\cdots,\boldsymbol{s}_{z_{i-1}}]$.

\glm's bidirectional attention over unmasked (i.e., uncorrupted) contexts distinguishes itself from other GPT-style LLMs' unidirectional attention.
It mixes two corruption objectives to allow strong performance on both understanding and generation, each indicated with a special mask token:
\begin{itemize}[leftmargin=*,itemsep=0pt,parsep=0.2em,topsep=0.0em,partopsep=0.0em]
    \item \textbf{[MASK]}: short consecutive blanks in sentences whose lengths add up to 15\% of the input.
    \item \textbf{[gMASK]}: random-length long blanks at the end of sentences with prefix contexts provided.
\end{itemize}

\begin{wrapfigure}{r}{6cm}
    \small
    \vspace{-6mm}
    \centering
    \includegraphics[width=1.0\linewidth]{figures/architecture.pdf}
    \caption{\glm and other LLMs on zero-shot LAMBADA language modeling. More explanations on GLM's bidirectional attention are provided in \citet{du2022glm}.}
    \label{fig:lambada}
    \vspace{-6mm}
\end{wrapfigure}

The blanking infilling objective with bidirectional attention contributes to a more effective comprehension of the context information than GPT-style models: when using [MASK], \glm behaves as BERT~\citep{devlin2019bert} and T5~\citep{raffel2020exploring}; when using [gMASK], \glm behaves similarly to PrefixLM~\citep{liu2018generating,dong2019unified}.

As an example, we show its results on LAMBADA~\citep{paperno2016lambada} zero-shot language modeling.
We follow the evaluation method used in GPT-2~\citep{radford2019language} and GPT-3~\citep{brown2020language}.
We observe that \glm with bidirectional attention over context set up a new record of 80.2\% accuracy, even outperforming the 4$\times$ larger PaLM-540B~\citep{chowdhery2022palm}.
On the contrary, \glm's ablated unidirectional variant (by setting the attention mask) reports 75.3\% on LAMBADA, comparable to other similar-sized GPT-style models including GPT-3 and OPT-175B~\cite{zhang2022opt}.
Our observations also accord with findings in~\citep{liu2018generating,dong2019unified}.

\begin{insight}
\rm Bidirectional-attention GLM can be stronger than unidirectional GPT at large scale.
\end{insight}

\vvpara{Layer Normalization.}
Layer normalization (or LN, \cite{ba2016layer}) stabilizes the transformer training and has a significant impact on models' quality and downstream performance.
A proper choice of LNs becomes more critical for LLMs as models scale up, where the challenge of training instability significantly exaggerates.
There have been several successful LN variants including vanilla Post-LN~\citep{vaswani2017attention}, Pre-LN~\citep{xiong2020layer}, and Sandwich-LN~\citep{ding2021cogview}.
However, in our extensive experiments, we verify that none of their performance on \glm is satisfying (Cf. Appendix~\ref{app:ln} for our detailed study).

We commit to finding a Post-LN variant that can stabilize \glm's training, and finally targeting on DeepNorm~\citep{wang2022deepnet}, a new initialization approach for Post-LN.
It is reported to stabilize the training of a 1000-layer transformer.
Given the number of \glm's layers $N$

\begin{itemize}[leftmargin=*,itemsep=0pt,parsep=0.2em,topsep=0.0em,partopsep=0.0em]
    \item We adopt $\textrm{DeepNorm}(\boldsymbol{x}) = \textrm{LayerNorm}(\alpha\cdot \boldsymbol{x} + \textrm{Network}(\boldsymbol{x}))$, where $\alpha=(2N)^\frac{1}{2}$.
    \item We apply Xavier normal initialization with the scaling factor of $(2N)^{-\frac{1}{2}}$ to \texttt{ffn}, \texttt{v\_proj} and \texttt{out\_proj}. Additionally, all bias terms are initialized to zero.
\end{itemize}

\begin{wrapfigure}{r}{6cm}
    \small
    \vspace{-4mm}
    \centering
    \includegraphics[width=1.0\linewidth]{figures/layernorm.pdf}
    \vspace{-5mm}
    \caption{Different LNs' gradient norm along \glm's early training steps.}
    \label{fig:layernorm}
    \vspace{-6mm}
\end{wrapfigure}

Though \glm only possesses 70 layers, the strategy significantly benefits the training stability.
In Figure~\ref{fig:layernorm}, along the early pre-training steps of \glm, we plot gradient norm (i.e., the gradients' $l^2$-norm) of Sandwich-LN and Post-LN with DeepNorm initialization, which reflects how drastic a model is updated at a certain step.
We observe that \glm with Sandwich-LN has several magnitude larger gradient norms than that of Post-LN with DeepNorm, and it finally collapses at around 2.5k steps; while Post-LN with DeepNorm helps \glm to be stably trained.

\begin{insight}
\rm It is a counter-stereotype that the Post-LN initialized with DeepNorm would be the only feasible layer normalization to stabilize \glm's pre-training.
\end{insight}

\vvpara{Rotary Positional Encoding (RoPE, \cite{su2021roformer}).}
Vanilla transformer adopts absolute (or sinuous) PE, and is later evolved into relative PE~\citep{dai2019transformer}.
Two recent popular relative PE variants are RoPE adopted by PaLM-540B~\citep{chowdhery2022palm} and GPT-J~\cite{gpt-j}, and ALiBi~\citep{press2021train} adopted by BLOOM-176B~\citep{scao2022what}.
In our testing, we find RoPE to be a better option for \glm,
likely because its bidirectional attention makes ALiBi relatively inefficient for both engineering implementation and training convergence.

And in \glm, different from the two-dimensional positional encoding used in vanilla GLM, we turn back to conventional one-dimensional positional encoding.
At the time when we designed \glm, unfortunately we did not figure out how to implement two-dimensional RoPE\footnote{It was not until recently that we found a blog by RoPE's author on suggestions for two-dimensional RoPE.}.
As a substitute plan, in \glm we simply remove the second dimension used in original GLM as we find that the unidirectional attention mask sub-matrices for [MASK] generation indicate the token order as well.
For long generation with [gMASK] (which is always at the end of the context), generated tokens will just prolong the first-dimensional positional encoding from the last context token.
More details on \glm's positional encoding formulation are presented in Appendix~\ref{app:pe}.

\vvpara{Gated Linear Units (GLU, \cite{dauphin2017language}).}
Some recent efforts on transformer has been on improving the FFNs.
We have noticed the GLU adopted in PaLM-540B and its new variant Gated Attention Unit (GAU, \cite{hua2022transformer}), and test them in our experiments by pre-training GLM-base (110M) over a random 50G Chinese and English mixed corpus.
It turns out that both GLU and GAU can improve upon the vanilla FFN, while GLU can be better and more stable in training.
Consequently in \glm we adopt GLU with GeLU~\citep{hendrycks2016gaussian} activation
\begin{equation}
\operatorname{FFN}_{\mathrm{GeGLU}}\left(\boldsymbol{x}; \boldsymbol{W}_1, \boldsymbol{V}, \boldsymbol{W}_{2}\right)=\left(\mathrm{GeLU}(\boldsymbol{x} \boldsymbol{W}_1) \otimes \boldsymbol{x} \boldsymbol{V}\right) \boldsymbol{W}_{2}
\end{equation}

In order to keep the same parameter as the vanilla FFN, the feed-forward size $d_{\mathrm{ffn}}$ (which is usually $4 d_{\mathrm{H}}$, where $d_{\mathrm{H}}$ is the hidden dimension) is reduced to $\frac 8 3 d_{\mathrm{H}}$ as the $\boldsymbol{V}$ is additionally introduced.

\subsection{\glm's Pre-training Objectives and Data}
In \glm, instead of mere self-supervised autoregressive blank infilling, we refer to recent works~\citep{aribandi2022ext5,wei2022finetuned,sanh2022multitask} to include a small portion of multi-task learning in the pre-training to boost its downstream zero-shot learning capability.

\vvpara{Self-supervised Pre-training (95\% tokens).}
In the \glm's implementation, for 30\% training tokens, we use [MASK] to mask consecutive spans whose lengths conform to the Poisson distribution ($\lambda=3$) and they add up to 15\% of the input sequences.
For the other 70\% tokens, we provide the prefix of a sequence as context and use [gMASK] to mask the rest tokens for \glm to predict; the masked length is sampled from the Uniform distribution.
The setting endows \glm with strong capability on both natural language understanding (using [MASK] similar to BERT) and natural language generation (using [gMASK] similar to GPT) with proper prompts.

The pre-training data include 1.2T Pile~\citep{gao2020pile} English corpus, 1.0T Chinese WudaoCorpora~\citep{yuan2021wudaocorpora} \todo{to exclude or not?}, and around 250G several Chinese corpora we crawl from the web, which finally forms a balanced composition of English and Chinese corpora.

\vvpara{Multi-task Instruction Pre-training (MIP, 5\% tokens).}
Recent work such as FLAN~\citep{wei2022finetuned} and T0~\citep{sanh2022multitask} shows that LLMs fine-tuned on multi-task instruction datasets can be better at zero-shot learning.
Instead of doing it in fine-tuning, following conclusion in T5~\citep{raffel2020exploring} and ExT5~\citep{aribandi2022ext5} that multi-task learning in pre-training can be even helpful than fine-tuning, we propose to include a variety of instruction prompted datasets including language understanding, generation, and information extraction in \glm's pre-training.

We originally planed to only include training datasets' training splits of T0~\citep{sanh2022multitask} and DeepStruct~\citep{wang2022deepstruct}.
All prompts for T0 datasets are from PromptSource~\citep{bach2022promptsource} and prompts for DeepStruct datasets are newly created.
but by a mistake at first we include the training and evaluation datasets' training splits and exclude DeepStruct datasets.
We fixed the mistake at around 23k steps and shifted back to the correct version.
All 74 datasets, corresponding prompts used, and the data preprocess details in MIP are listed in Appendix \ref{app:data}.
We suggest that users should not evaluating \glm's zero-shot and few-shot capabilities on these datasets.

\section{The Training Stability of \glm}

The training stability is the decisive factor in \glm's quality, which is also largely impacted by the number of tokens it passes through~\citep{hoffmann2022training}.
Thus, given the computing usage constraint, there has to be a trade-off between efficiency and stability with regard to floating-point (FP) formats:
low-precision FP formats (e.g., 16-bit precision---FP16) improve computing efficiency but are prone to overflow and underflow errors, resulting in training collapses.

\vvpara{Mixed-Precision.}
We follow the common practice of a mixed-precision~\citep{micikevicius2018mixed} strategy (Apex O2), i.e., FP16 for forwards and backwards and FP32 for optimizer states and master weights, to reduce the GPU memory usage and improve training efficiency.
Similar to OPT-175B and BLOOM-176B (C.f. Figure~\ref{fig:collapse} in Appendix), the training of \glm faces frequent loss spikes resulted from this choice, which tends to become increasingly frequent as the training goes on.
The precision related spikes are often without clear reasons:
some recover on their own; others come with a portent of suddenly soaring gradient norm and eventually a spike or even NaN in loss.

 \begin{wrapfigure}{r}{4.8cm}
    \centering
    \vspace{-4mm}
    \includegraphics[width=\linewidth]{figures/shrink-small.pdf}
    \vspace{-7mm}
    \caption{EGS reduces gradient scale and variance to stabilize LLMs' pre-training.}
    \label{fig:shrink}
    \vspace{-8mm}
\end{wrapfigure}

OPT-175B attempted to fix by manually skipping data and adjusting hyper-parameters; BLOOM-176B did so via the embedding norm technique~\citep{dettmers20218}.
We spend months to empirically investigate the spikes and realize that a few issues emerge when transformers scale up:

First, the transformer main branch's value scale can be extremely large in deeper layers if using Pre-LN.
This is addressed in \glm by using DeepNorm based Post-LN (Cf.  Section ~\ref{sec:glm-arch}), which makes the value scale always bounded.

Second, the attention scores grow so large that they exceed FP16's range, as the model scales up.
There are a few options to overcome this issue in LLMs.
In CogView~\citep{ding2021cogview}, PB-Relax is proposed to remove bias terms and deduct extremum value in attention computation to avoid the problem, which unfortunately does not help avoid disconvergence in \glm.
In BLOOM-176B, the BF16 format is used instead of FP16, due to its wide range of values on NVIDIA Ampere GPUs (i.e., A100).
However, BF16 consumes $\sim$15\% more run-time GPU memory than FP16 in our experiments due to its conversion to FP32 in gradient accumulation, and more importantly it is not supported on other GPU platforms (e.g., NVIDIA Tesla V100), limiting the accessibility of produced LLMs.
Another option from BLOOM-176B is to apply embedding norm with BF16, but in sacrifice of a significant penalty on model performance~\citep{scao2022what}.

\vvpara{Embedding Layer Gradient Shrink (EGS).}
Our empirical search identifies that the gradient norm can serve as an informative indicator of training collapses.
Specifically, we find that
a training collapse usually lags behind a ``spike'' in gradient norm by a few training steps.
Such spikes are usually caused by the embedding layer's abnormal gradients, as we observe that its gradient norm is often several magnitude larger that those of other layers in \glm's early stage training (Cf. Figure~\ref{fig:shrink} (a)).
In addition, it tends to fluctuate dramatically in the early training.
The problem is handled in vision models~\citep{chen2021empirical} via freezing the patch projection layer.
Unfortunately, we cannot freeze the training of the embedding layer in language models.

Finally, we find the gradient shrink on the embedding layer can help overcome loss spikes and thus stabilize the \glm training.
This is first used in the multi-modal transformer CogView.
Specifically, let $\alpha$ be the shrinking factor, the strategy can be easily implemented via
  $  \mathsf{word\_embedding} = \mathsf{word\_embedding} * \alpha + \mathsf{word\_embedding.detach()} * (1 - \alpha)$.
Figure~\ref{fig:shrink} (b) suggests that empirically, setting $\alpha=0.1$ helps wipe out most spikes we would have met, with negligible speed loss.

In fact, the final \glm training run only experiences three late-stage loss divergence cases, though it fails numerous times due to hardware failures.
For the three unexpected spikes, it turns out further shrinking the embedding gradient can still help stabilize the \glm training.
See the training notes and Tensorboard logs in our code repository
for details.

\hide
\section{Training Strategy}

The root cause of  is that there is a trade-off between training efficiency and stability with regard to floating-point (FP) formats:
\begin{itemize}[leftmargin=*,itemsep=0pt,parsep=0.2em,topsep=0.0em,partopsep=0.0em]
\item \textbf{Efficiency}: low-precision FP formats (e.g., FP16) reduce memory and computation costs
\item \textbf{Stability}: low-precision FP formats are prone to overflow and underflow, resulting in collapses
\end{itemize}

\input{4_parallel}

\subsection{Stability: Systematic and Numerical Challenges}

 Modern computing devices such as NVIDIA A100s and TPUs support 16-bit precision computation with much higher throughput compared to 32-bit precision.
Therefore, it has become a de facto practice to train language models with mixed-precision~\citep{micikevicius2018mixed} to reduce GPU memory usage and improve computation efficiency without hurting performance.
However, as far as we have observed, LLMs are far more vulnerable to loss spikes in the mixed-precision setup than smaller ones, which is well-aligned with other LLMs' observations (C.f. Figure~\ref{fig:collapse} in Appendix) such as OPT-175B~\citep{zhang2022opt} and BLOOM-176B~\citep{scao2022what}.

Handling these unexpected spikes has been the most challenging part in \glm's training.
We empirically ascribe them into structural and numerical instability:

\begin{itemize}[leftmargin=*,itemsep=0pt,parsep=0.2em,topsep=0.0em,partopsep=0.0em]
\item \textbf{Systematic Instability}:
which tends to appear in the early stage of training.
Its appearance is often accompanied with a gradual but steady rise in gradient norm.
Skipping data or adjusting hyper-parameters do not eliminate, but only make it happen at other random neighboring steps.

\item \textbf{Numerical Instability}:
which instead tends to occur in the late stage of training and becomes increasingly frequent as the training goes on.
Its resultant spikes are often without clear reasons: some recover on their own; others come with a portent of suddenly soaring gradient norm and eventually a spike or even NaN in loss, as the model has been optimized by the abnormal gradients.
\end{itemize}

We refer to the first type of instability as \emph{systematic}, since regardless of its optimization configuration, a model just cannot converge ideally owing to its current setup (i.e., model architectures and mixed-precision training strategies).
To eliminate consequent disconvergence, a more stable model architecture and proper mixed-precision strategies are necessary.

The second type of instability is referred to as \emph{numerical}, since we believe it is likely to derive from some unknown numerical reasons in 16-bit precision training.
Its related spikes can be mitigated via re-loading the model from a checkpoint 100 to 200 iterations ahead with data skipping, or changing hyper-parameters such as the learning rate.
Such instability may not arise from bad data samples as is previously assumed, because changing the learning rate also allows the model to go through them.
Authors of~\citep{chowdhery2022palm} also report that no bad patterns have been observed from data causing training spikes in their examination.
Thus, numerical spikes are likely to occur only when a certain model goes through certain data, and can be avoided by applying some perturbation to the training process.

Due to the large cost of LLM training, conquering these spikes are crucial: it would be a tragedy if the training diverged on the half road and turned out unrecoverable.
We conduct plenitude of experiments spanning few months to find several effective strategies to stabilize \glm's training.

\begin{insight}
\rm It is both systematic and numerical instabilities that LLMs suffer from in training, which takes us months to identify and conquer from scratch in \glm.
\end{insight}

\vvpara{Mixed-precision strategy.}
Mixed-precision~\citep{micikevicius2018mixed} training with FP16 has become a default in mainstream frameworks such as DeepSpeed~\citep{rasley2020deepspeed}.
A common mixed-precision training strategy (Apex O2) is as follows: forward and backward are calculated in pure 16-bit precision (FP16), but the optimizer states and master weights are FP32 are used in the optimization.
However, such a strategy is prone to meet severe numerical instability in a conventional transformer architecture when a model scales up, due to its two main bottlenecks:
\begin{itemize}[leftmargin=*,itemsep=0pt,parsep=0.2em,topsep=0.0em,partopsep=0.0em]
    \item Under Pre-LN, the transformer main branch's value scale can be extremely large in deeper layers.
    \item As a model scales up, attention scores grow so large that they exceed FP16's representation scope.
\end{itemize}

For the first bottleneck, our use of DeepNorm (a variant of Post-LN) servers as LayerNorm in the main branch, so the value scale is always bounded.
For the second bottleneck, CogView~\citep{ding2021cogview} proposes PB-Relax to remove bias terms in attention's linear projection and deduct the largest attention score from each attention score matrix to avoid the problem.
Nevertheless, the technique does not eliminate disconvergence in \glm, probably due to its gigantic size.

As a remedy for fp16's narrow representation range, NVIDIA Ampere GPUs provide BF16 floating-point format (adopted by BLOOM 176B) to mitigate the problem.
Nevertheless, BF16's dissupport on other computing platforms (e.g., NVIDIA Tesla V100) has significantly hampered its wider applications.
In practice, we also notice that compared to FP16, BF16 requires around 15\% additional run-time GPU memory due to its necessary conversion to FP32 in gradient accumulation.

To support as many researchers and developers as possible, \glm thus chooses FP16 as its training floating-point format.
On the other hand, it means our model is faced with more stability challenges, as is also reported in another FP16 LLM OPT-175B.
Fortunately, after great struggle, we successfully get it through via a set new techniques derived from some following insights.

\begin{insight}
\rm FP16 is a challenging but rewarding decision: it suffers from more instability, but is a must for enabling LLMs to train and inference on inclusive ranges of platforms.
\end{insight}

\vvpara{Gradient Shrink on the Embedding Layer.}
Besides explicit precision issues caused by attention scores, there are implicit problems arouses by random noisy samples appearing along the training.
For example, in our observation these samples might consist bunches of semantically plausible repeating lines, which can be hardly cleaned in rule-guided preprocesses.
We find LLMs are so vulnerable to these noises, which likely results in unrecoverable unexpected training collapses.

\begin{figure}[t]
    \centering
    \includegraphics[width=\linewidth]{figures/shrink.pdf}
    \vspace{-6mm}
    \caption{Gradient shrink on embedding layer stabilizes \glm's training. (a) Preliminary experiments on GLM-base show that gradient norm for embedding layer can be much larger; (b) 40B-scale testing shows that embedding gradient shrink is necessary for LLM training.}
    \label{fig:shrink}
    \vspace{-6mm}
\end{figure}

For such instability, we notice that the gradient norm serves as an effective indicator, which accords with observations from vision transformers~\citep{chen2021empirical,ding2021cogview}: the happening of a training collapse usually lags behind a ``spike'' in gradient norm by few training steps.
Such spikes are usually caused by embedding layer's abnormal gradient, as we observe a remarkably larger gradient norm from it than other layers---often several magnitudes larger---in \glm's early stage training (Cf. Figure~\ref{fig:shrink} (a)).
It also tends to fluctuate dramatically in the early stage training of GLM compared to other layers.
The issue is handled in~\citep{chen2021empirical} via freezing the patch projection layer; however, a language model cannot freeze the training of its embedding layer.

Alternatively, we find the strategy of gradient shrink over embedding layer can overcome the problem in LLM's training, which is similarly applied in text-to-image transformer CogView~\citep{ding2021cogview}.
Let $\alpha$ be the shrinking factor, the strategy can be easily implemented via
\begin{equation}
    \mathsf{word\_embedding} = \mathsf{word\_embedding} * \alpha + \mathsf{word\_embedding.detach()} * (1 - \alpha)
\end{equation}
Specifically, we shrink to $\alpha=0.1$ and discover that it wipes out most spikes we would have met, with negligible speed loss (Cf. Figure~\ref{fig:shrink} (b)).
Actually, in our late stage training, we observe that further shrinking embedding gradient still works when unexpected spikes take place.

\begin{insight}
\rm Shrinking embedding layer's gradient to its 0.1 can solve most instability problems.
\end{insight}

We train \glm for 400 billion tokens (roughly 200 billion each for Chinese and English) with a fixed sequence length of 2,048 per sample.
For the [gMASK] training objective, we use a context window of 2,048 tokens.
For the [MASK] and multi-task training objectives, we use context windows of length 512 and concatenate four samples together to cater the 2,048-sequence-length.
We warm-up the batch size from 192 to 4224 over the first 2.5\% samples.
We use AdamW~\citep{loshchilov2017decoupled} as our optimizer with $\beta_1$ and $\beta_2$ set to 0.9 and 0.95, and a weight decay value of 0.1.
We warm-up the learning rate from $10^{-7}$ to $8\times 10^{-5}$ over the first 0.5\% samples, then decay it by a $10\times$ cosine schedule.
We use a dropout rate of 0.1 and clip gradients using a clipping value of 1.0.

\glm's pre-training lasts 60 days and it takes up 96 NVIDIA DGX-A100 (40G) nodes with 400G bandwidth IB network, which would cost equivalently 4.9 million dollars based on the GPU pricing on public cloud services in the same period.
On top of the basic training setup, during the long training we have experienced several crises and failures.
We successfully managed to recover the training via slight adjustments over some of configuration above (mostly about learning rate and shrinking factor), which are reported in our released training notes and Tensorboard logs\footnote{\url{https://github.com/THUDM/GLM-130B/tree/main/logs}}.

\section{\glm Inference on RTX 2080 Ti}

One of the major goals of \glm is to lower the hardware requirements for accessing 100B-scale LLMs without efficiency and effectiveness disadvantages.

As mentioned, the model size of 130B is determined for running the full \glm model on a single A100 (40G$\times$8) server, rather than the high-end A100 (80G$\times$8) machine required by OPT-175B and BLOOM-176B.
To accelerate \glm inference, we also leverage FasterTransformer~\citep{timonin2022accelerated}
to implement \glm in C++.
Compared to the PyTorch implementation of BLOOM-176B in Huggingface, \glm's decoding inference is 7-8.4$\times$ faster on the same single A100 server. (Cf. Appendix~\ref{app:inference_acceleration} for details).

\vvpara{INT4 Quantization for RTX 3090s/2080s.}
To further support popularized GPUs,
we attempt to compress \glm as much as possible while maintaining performance superiority, particularly via  quantization~\citep{zafrir2019q8bert,shen2020q,tao2022compression}, which introduces little task-agnostic performance drops for generative language models.

Typically, the practice is to quantize both model weights and activations to INT8.
However, our analysis in Appendix~\ref{app:activation_analysis} suggests that LLMs' activations may contain extreme outliers.
Concurrently, the emergent outliers in OPT-175B and BLOOM-176B are also discovered~\citep{dettmers2022llm}, which influence only about 0.1\% feature dimensions and are thus solved by matrix multiplication decomposition for the outlying dimensions.

Differently, there exist about 30\% outliers in GLM-130B's activations, making the technique above far less efficient.
Thus, we decide to focus on the quantization of model weights (i.e., mostly linear layers) while keeping the FP16 precision for activations.
We simply use post training absmax quantization, the weights are dynamically converted to FP16 precision at runtime, introducing a small computational overhead but greatly reducing the GPU memory usage for storing model weights.

Excitingly, we manage to reach the INT4 weight quantization for \glm while existing successes have thus far only come to the INT8 level.
Memory-wise, by comparing to INT8, the INT4 version helps additionally save half of the required GPU memory to 70GB, thus allowing \glm inference on 4 $\times$ RTX 3090 Ti (24G) or 8 $\times$ RTX 2080 Ti (11G).
Performance-wise, Table~\ref{tab:quantization} left indicates that without post-training at all, the INT4-version \glm experiences almost no performance degradation, thus maintaining the advantages over GPT-3 on common benchmarks.

\begin{wrapfigure}{r}{6.5cm}
    \small
    \vspace{-4mm}
    \centering

    \includegraphics[width=1.0\linewidth]{figures/quantization-scaling-law.png}
    \vspace{-4mm}
    \caption{(Left) \texttt{attn-dense} and \texttt{w2}'s weight distributions; (Right) \glm's INT4 weight quantization scaling law.}
    \label{fig:scaling_law}
    \vspace{-3mm}
\end{wrapfigure}

\vvpara{GLM's INT4 Weight Quantization Scaling Law.}
Figure~\ref{fig:scaling_law} right shows the performance trend as the increase of the model size, indicating the emergence of a scaling law for GLM's INT4 weight quantization performance.
We examine the underlying mechanism of this unique property observed for GLM.
We plot the weight value distributions in Figure~\ref{fig:scaling_law} left, which turns out to directly impact the quantization quality.
Specifically, a wider-distributed linear layer needs to be quantized with larger bins, leading to more precision loss.
Thus the wide-distributed \texttt{attn-dense} and \texttt{w2} matrices explain the INT4 quantization failure for GPT-style BLOOM.
Conversely, GLMs tend to have much narrower distributions than those of similar-sized GPTs, and the gap between INT4 and FP16 versions keeps further decreasing as the GLM model size scales up (Cf.  Figure~\ref{fig:quantization_appendix} in Appendix~\ref{app:quantization_analysis} for details).

\input{tables/quantization}

\hide

\section{Inference for Inclusivity}
In terms of LLM inclusivity, we think its scope should not be limited to accessibility, but also usability.
It is particularly a fact that, even with released LLM checkpoints such as OPT-175B~\citep{zhang2022opt} and BLOOM-176B~\citep{scao2022what}, individuals and academic researchers can hardly afford the cost for inference, which have been reported to depend on an 8 $\times$ A100 (80G) server.
Thus, a considerable amount of sweat we spend in \glm has been on supporting its high usability and fast inference speed on popularized GPUs, such as NVIDIA RTX 3090 Ti (24G).

Interestingly, such efforts actually deeply associate with our \textit{algorithm designs}, rather than mere engineering undertakings.
We will show that GLM series presents \textit{a unique scaling law to allow INT4 quantization}, while other existing public LLMs (OPT and BLOOM series) would fail.

\vvpara{Inference Acceleration.}
A model's plain PyTorch implementation is easy to read and run; but it can be intolerably slow for LLMs.
Based on NVIDIA's FasterTransformer\footnote{\url{https://github.com/NVIDIA/FasterTransformer}} we spend two months \aohan{no need to mention?} implementing \glm into C++ to speed up its inference (Cf. Appendix~\ref{app:inference_acceleration} for details).
Compared to a naive implementation (e.g., BLOOM-176B by Huggingface Transformers), \glm's decoding inference on the same-length sequences can be up to $\times$8.4 faster.

\vvpara{GLM's INT4 Weight Quantization Scaling Law.}
Notwithstanding \glm's accelerated inference speed, its demand for at least a DGX-A100 (40G) server can still discourage individual users.
In light of the situation, we endeavor to compress \glm to fit into 4 $\times$ RTX 3090 Ti (24G)'s capacity, particularly via the quantization~\citep{zafrir2019q8bert,shen2020q,tao2022compression}, which presents little task-agnostic performance drop for generative language models.

\input{tables/quantization}

\begin{wrapfigure}{r}{4.5cm}
    \small
    \vspace{-5mm}
    \centering
    \includegraphics[width=1.0\linewidth]{figures/quantization_scaling_law.pdf}
    \vspace{-4mm}
    \caption{\glm's INT4 weight quantization scaling law.}
    \label{fig:scaling_law}
    \vspace{-8mm}
\end{wrapfigure}

Typical methods quantize both model weights and activation to INT8
But in our analysis, we find LLMs' activations may contain extreme outliers (Cf. Appendix~\ref{app:activation_analysis}).
Concurrent to our efforts, \citep{dettmers2022llm} also discovers the emergent outliers in OPT-175B and BLOOM-176B, which influence only about 0.1\% feature dimensions and are thus solved by matrix multiplication decomposition for outlying dimensions.
However, we find substantial outliers (30\%) in GLM-130B's activations, making it far less efficient to apply due to the slow high-precision computation.

After a few weeks of trial, we finally decide to keep activations' FP16 precision and only consider the symmetric quantization of model weights (majorly consist of linear layers).
The quantized model is dynamically converted to FP16 precision at runtime, introducing a small computational overhead but greatly reducing GPU memory usage for storing model weights.

Surprisingly we notice \glm's unusual talent for INT4 weight quantization, which saves half of required GPU memory to 70GB compared to INT8 and thus allows \glm's inference on even 4 $\times$ RTX 3090 Ti (24G) or 8 $\times$ RTX 2080 Ti (11G).
More surprising, it requires no post-training at all and shows little quality degradation (Cf. Table~\ref{tab:quantization}).
However, such a nice property does not apply to BLOOM-176B and OPT series models (Cf. Figure~\ref{fig:scaling_law}).
To understand it deeper, we dive into LLMs' weight value distribution (Cf. Figure~\ref{fig:weight_dist},~\ref{fig:quantization_appendix}), which directly impact the quantization quality.
We find language models' skewed \texttt{w2} matrix is the major cause for INT4 quantization failure, especially for GPTs.
On the contrary, GLMs usually present more symmetrical \texttt{w2} value distributions than similar-sized GPTs, and it seems to keep improving as they scale up.
We summarize the discovery as \textbf{GLM's Scaling Law for INT4 Weight Quantization}, which demonstrates GLM's unique advantage to be quantized into INT4 precision as the architecture scales up.

\begin{figure}[t]
    \centering
    \includegraphics[width=\linewidth]{figures/quantization.pdf}
    \caption{Weight value distribution of \texttt{w2} from BLOOM-176B and \glm's first-8 layers' FFN (or GLU). A more skewed distribution with outliers is likely to suffer from more quantization loss.}
    \label{fig:weight_dist}
    \vspace{-5mm}
\end{figure}

\section{Inference for Inclusivity}
In terms of LLM inclusivity, we think its scope should not be limited to accessibility, but also usability.
It is particularly a fact that, even with released LLM checkpoints such as OPT-175B~\citep{zhang2022opt} and BLOOM-176B~\citep{scao2022what}, individuals and academic researchers can hardly afford the cost for inference, which have been reported to depend on an 8 $\times$ A100 (80G) server.
Thus, a considerable amount of sweat we spend in \glm has been on supporting its high usability and fast inference speed on popularized GPUs, such as NVIDIA RTX 3090 Ti (24G).

Interestingly, such efforts actually deeply associate with our \textit{algorithm designs}, rather than mere engineering undertakings.
We will show in the following part that \glm is the only LLM so far being able to be quantized into INT4 precision, while other existing public LLMs (OPT-175B, BLOOM-176B, and even those 10B-scale smaller language models) would fail.

\subsection{Inference Acceleration.}
\du{This section can be mostly moved to appendix, leaving only the resutls.}
A model's plain Pytorch implementation is easy to read and run; but it can be intolerably slow for LLMs.
Based on NVIDIA's FasterTransformer\footnote{\url{https://github.com/NVIDIA/FasterTransformer}} we spend two months implementing \glm into C++ to speed up inference, including the following main optimizations:
\begin{itemize}[leftmargin=*,itemsep=0pt,parsep=0.2em,topsep=0.0em,partopsep=0.0em]
    \item Optimize time-costing operations such as GeGLU, Layer Normalization and SoftMax.
    \item Reduce number of GPU kernel calls (e.g., fuse MultiheadAttention into one computation kernel).
    \item Specify the algorithm of the best performance when calling cuBLAS.
    \item Improve the computing efficiency by transposing the model parameters in advance.
    \item Use half2 in FP16 computation to double the half's access bandwidth and computing throughput.
\end{itemize}

We currently pack up the full FasterTransformer implementation for \glm into a plug-and-play docker image for users' convenience, and we are still working on adapting it to our Pytorch implementation
by only changing one line of code.
A comparison between our speeding up \glm implementation and the so far default available BLOOM-176B implementation in Huggingface Transformers\footnote{\url{https://huggingface.co/docs/transformers/model_doc/bloom}} is shown in Table~\ref{tab:fastertransformer}.
Our implementation for \glm can be 7.0 to 8.4 times faster than BLOOM-176B's Pytorch implementation.
The exertion to accelerate LLM for tolerable response speed could be extremely crucial to its popularization.

\input{tables/quantization}

\subsection{Compression via Quantization}
We have improved \glm's inference speed substantially, but its demand for at least a DGX-A100 (40G) server can still discourage individual developers and researchers.
In light of the situation, we endeavor to compress \glm to fit into 4 $\times$ RTX 3090 Ti (24G)'s capacity, particularly via the quantization~\citep{zafrir2019q8bert,shen2020q,tao2022compression}, which presents little task-agnostic performance drop for generative language models.

\vvpara{INT4 Weight Quantization.}
Typical methods quantize both model weights and activation to INT8, enabling the INT8 matrix multiplication kernel for efficiency.
However, we find substantial outliers in GLM-130B's activations, making it hard for precision reduction.
Concurrent to our efforts and observations in \glm, researchers~\citep{dettmers2022llm} also discover the issue of emergent outliers in LLMs.
In their in-depth analysis on OPT-175B and BLOOM-176B, they find the outliers make up only about 0.1\% of all feature dimensions.
Therefore, they propose to decompose the matrix multiplication for high-precision multiplication in outlying dimensions.

But our detailed analysis demonstrates that the outliers in GLM-130B can sometimes make up at most 30\% of the feature dimension, which is different from GPT-based LLMs.
Therefore, a mixed-precision decomposition for \texttt{matmul} can be far less efficient than a single FP16 \texttt{matmul}.
After a few weeks of trial, we finally decide to keep the FP16 precision of activation and only consider the quantization of model weights.
In this case, the quantized \glm is dynamically converted to FP16 precision at runtime, introducing a small computational overhead but greatly reducing GPU memory usage for storing model weights.
We quantize all linear layers, which take up most of the transformer parameters, via vector-wise symmetric quantization, and leave input/output embedding, layer normalization, and bias terms unchanged.

Surprisingly we note \glm's unusual talent for INT4 weight quantization.
At the quantization precision of INT4, two INT4 weights are compressed into one INT8 weight for saving GPU memory usage.
Thus approximately only 70GB GPU memory is required to serve INT4 \glm weights, which allow \glm's inference on even 4 $\times$ RTX 3090 Ti (24G) or 8 $\times$ RTX 2080 Ti (11G).
More surprising, it requires no post-training at all and shows almost no quality degradation.
On the contrary, it does not apply to BLOOM-176B in our experiments.

\vvpara{Empirical Understanding.}
Such nice property for \glm is fascinating but mysterious.
We try our best to understand it and empirically demonstrate that the INT4 weight quantization for \glm may be a compound result of multiple factors, rather than a single one.

In terms of whether a model can be properly quantized, the most fundamental reason lies in the skewness of its weight distribution.
Since existing methods on weight mostly rely on symmetrical quantization, if weight distributions in certain linear layers are too skewed, the quantization would lose much precision.
\begin{itemize}[leftmargin=*,itemsep=0pt,parsep=0.2em,topsep=0.0em,partopsep=0.0em]
    \item \textbf{GLM pre-training objective}:
    \item \textbf{Gated Linear Unit (GLU)}:
    \item \textbf{Scaling}:
\end{itemize}

\begin{insight}
\rm Backbone architecture matters in LLMs' post quantization. While GLM scaled up to \glm can be quantized to as low as INT4, other LLMs only support INT8.
\end{insight}

}

\section{The Results}
\label{sec:results}

We follow the common settings in LLMs such as GPT-3 and PaLM to evaluate \glm for English~\footnote{\small Results in OPT-175B's paper are reported as applications to access it have not been approved for months.}.
As a bilingual LLM with Chinese, \glm is also evaluated on Chinese benchmarks.

\vvpara{Discussion on the Scope of Zero-Shot Learning in \glm.}
Since \glm has been trained with MIP, here we clarify its scope of zero-shot evaluation.
In fact, ``zero-shot'' seems to have controversial interpretations without a consensus in the community.
We follow one of the influential related surveys~\citep{xian2018zero}, which says
\textit{``At test time, in zero-shot learning setting, the aim is to assign a test image to an unseen class label''}
where involving unseen class labels is a key.
Therefore, we derive our criterion to pick \glm's zero-shot (and few-shot) datasets as:
\begin{itemize}[leftmargin=*,itemsep=0pt,parsep=0.2em,topsep=0.0em,partopsep=0.0em]
    \item \textbf{English}: 1) For tasks with fixed labels (e.g., \textit{natural language inference}): no datasets in such tasks should be evaluated on; 2) For tasks without fixed labels (e.g., \textit{(multiple-choice) QA, topic classification}): only datasets with an obvious domain transfer from those in MIP should be considered.
    \item \textbf{Chinese}: All datasets can be evaluated as there exists a zero-shot cross-lingual transfer.
\end{itemize}

\vvpara{Filtering Test Datasets.}
Following prior practices~\citep{brown2020language,rae2021scaling} and our criterion mentioned above, we filter and refrain to report potentially contaminated datasets' evaluation results.
For LAMBADA and CLUE, we find minimal overlap under the 13-gram setting.
Pile, MMLU, and BIG-bench are either held-out or released later than the crawling of corpora.

\subsection{Language Modeling} \label{sec:language_modeling}

\vvpara{LAMBADA.}
LAMBADA~\citep{paperno2016lambada} is a dataset to test the last word language modeling capability.
The results previously shown in Figure~\ref{fig:lambada} suggest \glm achieves a zero-shot accuracy of 80.2 with its bidirectional attention, setting up a new record on LAMBADA.

\begin{wraptable}{r}{5.5cm}
	\centering
	\footnotesize
    \renewcommand\tabcolsep{2pt}
\renewcommand\arraystretch{1.2}
	\vspace{-4mm}
	\caption{\glm's average BPB on Pile evaluation (18 sub-datasets).}
	\vspace{-1mm}
	\scalebox{0.97}{
	\begin{tabular}{@{}lccc@{}}
    \toprule[1.2pt]
                       & Jurassic-1     & GPT-3          & \glm           \\ \midrule
    Avg. BPB      & 0.650          & 0.742          & \textbf{0.634} \\ \bottomrule[1.2pt]
    \end{tabular}
    }
    \vspace{-2mm}
\end{wraptable}

\vvpara{Pile.}
The Pile test-set~\citep{gao2020pile} includes a series of benchmarks for language modeling.
On average, \glm performs the best on its 18 shared test sets in terms of weighted BPB when compared to GPT-3 and Jurassic-1~\citep{lieber2021jurassic} whose results are directly adopted from the latter, demonstrating its strong language capability (Cf. Appendix~\ref{app:pile} for details).

\subsection{Massive Multitask Language Understanding (MMLU)} \label{sec:mmlu}

MMLU~\citep{hendrycks2021measuring} is a diverse benchmark including 57 multi-choice question answering tasks concerning human knowledge ranging from high-school-level to expert-level.
It is released after the crawling of Pile and serves as an ideal test-bed for LLMs' few-shot learning.
The GPT-3 result is adopted from MMLU and BLOOM-176B is tested by using the same prompts as \glm's
(Cf. Appendix~\ref{app:mmlu} and Table~\ref{tab:mmlu} for details).

\glm's few-shot (5-shot) performance on MMLU approaches GPT-3 (43.9) after viewing about 300B tokens in Figure~\ref{fig:mmlu}.
It continues moving up as the training proceeds, achieving an accuracy of 44.8 when the training has to end (i.e., viewing 400B tokens in total).
This aligns with the observation~\citep{hoffmann2022training} that most existing LLMs are far from adequately trained.

\subsection{Beyond the Imitation Game Benchmark (BIG-bench)} \label{sec:big-bench}

\begin{figure}[t]
\begin{minipage}{0.33\linewidth}
    \small
    \vspace{-5mm}
    \centering
    \includegraphics[width=1.0\linewidth]{figures/mmlu.pdf}
    \vspace{-4mm}
    \caption{\glm on MMLU (57 tasks) along training steps.}
    \label{fig:mmlu}
    \vspace{-6mm}
\end{minipage}
\hspace{.03\linewidth}%
\begin{minipage}{0.34\linewidth}
    \centering
	\includegraphics[width=0.99\columnwidth]{figures/big-bench-lite_1e10.pdf}
	\vspace{-5mm}
	\caption{BIG-bench-lite evaluation (24 tasks) across scales.}
	\label{fig:big-bench}
\end{minipage}
\hspace{.002\linewidth}%
\begin{minipage}{0.27\linewidth}
    \centering
    \renewcommand\tabcolsep{1pt}
    \renewcommand\arraystretch{1.2}
    \scalebox{0.9}{
    \footnotesize
    \begin{tabular}{@{}lccc@{}}
    \toprule[1.2pt]
               & 0-shot         & 1-shot         & 3-shot         \\ \midrule
    GPT-3 2.6B & 0.60           & 0.71           & 1.83           \\
    GPT-3 6.7B & -0.06          & 2.93           & 5.40           \\
    GPT-3 13B  & 1.77           & 5.43           & 7.95           \\
    GPT-3 175B & 4.35           & 11.34          & 13.18          \\ \midrule
    PaLM 540B  & 8.05           & \textbf{37.77} & -              \\ \midrule
    \glm       & \textbf{13.31} & 14.91          & \textbf{15.12} \\ \bottomrule[1.2pt]
    \end{tabular}}
    \vspace{1mm}
    \captionof{table}{Details on BIG-bench-lite (24 tasks).}
    \label{tab:big-bench}
\end{minipage}
\vspace{-2mm}
\end{figure}

BIG-bench~\citep{srivastava2022beyond} benchmarks challenging tasks concerning models' ability on reasoning, knowledge, and commonsense.
Given evaluating on its 150 tasks is time-consuming for LLMs, we report the BIG-bench-lite---an official 24-task sub-collection---for now.
Observed from Figure~\ref{fig:big-bench} and Table~\ref{tab:big-bench},
\glm outperforms GPT-3 175B and even PaLM 540B (4$\times$ larger) in zero-shot setting.
This is probably owing to \glm's bidirectional context attention and MIP, which has been proved to improve zero-shot results in unseen tasks~\citep{wei2022finetuned,sanh2022multitask}.
As the number of shots increases, \glm's performance keeps going up, maintaining its outperformance over GPT-3
(Cf. Appendix~\ref{app:big-bench} and Table~\ref{tab:big-bench-details} for details on each model and task).

\vvpara{Limitations and Discussions.}
In the experiments above, we observe that \glm's performance growth (13.31 to 15.12) with the increase of few-shot samples is not as significant as GPT-3's (4.35 to 13.18).
Here is out intuitive attempt to understand the phenomenon.

First, the  bidirectional nature of \glm could lead to strong zero-shot performance (as is indicated in zero-shot language modeling), thus getting closer to the few-shot ``upper-bound'' for models of similar scale (i.e., 100B-scale) than unidirectional LLMs.
Second, it may be also attributed to a deficit of existing MIP paradigms~\citep{wei2022finetuned,sanh2022multitask}, which only involve zero-shot prediction in the training and will be likely to bias \glm for stronger zero-shot learning but relatively weaker in-context few-shot performance.
To correct the bias, a potential solution we come up with would be to employ MIP with varied shots of in-context samples rather than only zero-shot samples if we ever got a chance to continue pre-training \glm.

Finally, despite almost the same GPT architecture as GPT-3, PaLM 540B's relative growth with few-shot in-context learning is substantially more significant than GPT-3's.
We conjecture this further acceleration in performance growth is a source of PaLM's high-quality and diverse private-collected training corpora.
By combining our experiences with~\citep{hoffmann2022training}'s insights, we come to realize that better architectures, better data, and more training FLOPS should be further invested.

\subsection{Chinese Language Understanding Evaluation (CLUE)} \label{sec:clue}

We evaluate \glm's Chinese zero-shot performance on established Chinese NLP benchmarks, CLUE~\citep{xu2020clue} and FewCLUE~\citep{xu2021fewclue}
Note that we do not include any Chinese downstream tasks in MIP.
To date, we have finished testing on part of the two benchmarks, including 7 CLUE and 5 FewCLUE datasets (Cf. Appendix~\ref{app:clue} for details).
We compare GLM-130B to the largest existing Chinese monolingual language model---the 260B ERNIE Titan 3.0~\citep{wang2021ernie}.
We follow its setting to report zero-shot results on dev datasets.
\glm consistently outperforms ERNIE Titan 3.0 across 12 tasks (Cf. Figure~\ref{fig:clue}).
Interestingly, \glm performs at least 260\% better than ERNIE on two abstractive MRC datasets (DRCD and CMRC2018), possibly due to \glm's pre-training objective that naturally resonates to abstractive MRC's form.

\begin{figure}[t]
    \centering
    \includegraphics[width=\linewidth]{figures/clue.pdf}
    \vspace{-6mm}
    \caption{\glm and ERNIE Titan 3.0 260B evaluated on zero-shot CLUE and FewCLUE.}
    \label{fig:clue}
    \vspace{-4mm}
\end{figure}

\hide

\section{Main Results}
\label{sec:results}
In this section, we introduce details in our experiments and evaluation settings.
Following other works on LLMs~\citep{brown2020language,rae2021scaling,chowdhery2022palm}, we report \glm's zero-shot and few-shot performance over a number of benchmarks, domains, and tasks.
Since \glm is a bilingual LLM proficient in both English and Chinese, we also evaluate it over popular Chinese language evaluation benchmarks in a zero-shot setting.

\subsection{Discussion: Scope of Zero-shot Learning in \glm} \label{sec:zero-shot-criterion}
As we are leveraging Multi-task Instruction Pre-Training (MIP), it is important to clarify our setting of "zero-shot".
However, there seems to be no officially recognized definition for zero-shot learning, and many different interpretations exist in the community.
To our best knowledge, we refer to the definition from one of the most influential zero-shot learning surveys~\citep{xian2018zero}, which says
\textit{``... At test time, in zero-shot learning setting, the aim is to assign a test image to an unseen class label, ...
''}
where whether the evaluated task involves unseen class labels is a key.
Therefore, for NLP tasks we derive our criterion to select \glm zero-shot and few-shot evaluation datasets:
\begin{itemize}[leftmargin=*,itemsep=0pt,parsep=0.2em,topsep=0.0em,partopsep=0.0em]
    \item \textbf{English}: 1) For tasks with fixed labels (e.g., \textit{natural language inference, paraphrase identification}): no datasets in such tasks should be evaluated on; 2) For tasks without fixed labels (e.g., \textit{question answering, topic classification, multiple-choice QA}): only datasets with an obvious domain transfer and different labels from those in MIP should be considered.
    \item \textbf{Chinese}: All datasets can be evaluated as there exists a zero-shot cross-lingual transfer.
\end{itemize}

\todo{Filtering test set}

\subsection{Language Modeling}
Language modeling is an intrinsic evaluation criterion for any language models.
Since memorization would substantially help language modeling, for \glm, we refrain to only report \glm's performance on LAMABADA~\citep{paperno2016lambada} and Pile test-set~\citep{gao2020pile}.

\vvpara{LAMBADA.}
Following other models~\citep{black2022gpt,gpt-j} trained on Pile corpus, we report results on LAMBADA, a dataset to test last word language modeling capability.
The results have been shown in Figure~\ref{fig:lambada}, in which \glm achieves a zero-shot accuracy of 80.2 with its superior bidirectional attention, setting up a new record on LAMBADA.

\begin{wraptable}{r}{6.5cm}
	\centering
	\footnotesize
	\vspace{-4mm}
    \renewcommand\tabcolsep{2pt}
	\caption{\glm and its similar-sized LLMs' average BPB on Pile (18 sub-datasets).}
	\vspace{-2mm}
	\scalebox{0.97}{
	\begin{tabular}{@{}lccc@{}}
    \toprule[1.2pt]
                       & Jurassic-1     & GPT-3          & \glm           \\ \midrule
    Weighted Avg. BPB      & 0.650          & 0.742          & \textbf{0.634} \\ \bottomrule[1.2pt]
    \end{tabular}
    }
    \vspace{-2mm}
\end{wraptable}

\vvpara{Pile Evaluation.}
We evaluate \glm's performance on Pile test-set~\citep{gao2020pile}, which includes a series of benchmarks for language modeling.
We compare \glm's BPB on Pile test-set with GPT-3 and Jurassic-1~\citep{lieber2021jurassic} based on results reported in~\citep{lieber2021jurassic}.
On average, \glm performs the best on these 18 shared test sets in these three LLMs, which proves its strong language capability. Detailed analysis and results are shown in Appendix~\ref{app:pile}.

\subsection{Massive Multitask Language Understanding (MMLU)} \label{sec:mmlu}

\begin{wrapfigure}{r}{5.3cm}
    \small
    \vspace{-4mm}
    \centering
    \includegraphics[width=1.0\linewidth]{figures/mmlu.pdf}
    \vspace{-4mm}
    \caption{\glm on MMLU (57 tasks) along its training trajectory.}
    \label{fig:mmlu}
    \vspace{-6mm}
\end{wrapfigure}

MMLU~\citep{hendrycks2021measuring} is a diverse benchmark including 57 multi-choice question answering tasks concerning human knowledge ranging from high-school-level to expert-level.
It serves as an ideal test-bed for LLMs' few-shot learning and is released after the crawling of Pile.

We plot \glm's few-shot (5-shot) performance along its training trajectory in Figure~\ref{fig:mmlu}.
It approaches GPT-3 comparable performance 43.9 after viewing about 300 billion tokens.
Its performance continues growing as the training proceeds, achieving 44.8 after viewing 400 billion tokens and does not seem to saturate when training terminates.
This aligns with the observation in~\citep{hoffmann2022training} that existing LLMs are still far from adequately trained.
GPT-3 results are from~\citep{hendrycks2021measuring} and BLOOM 176B results are tested by ourselves using the same prompts as \glm does.
More detailed results on each discipline of MMLU can be find in Appendix~\ref{app:mmlu} and Table~\ref{tab:mmlu}.

\subsection{Beyond the Imitation Game Benchmark (BIG-bench)}

\begin{figure}[t]
\begin{minipage}{0.72\linewidth}
    \centering
	\includegraphics[width=0.99\columnwidth]{figures/BIG-bench.pdf}
	\vspace{-2.2mm}
	\caption{BIG-bench-lite evaluation (24 tasks) across scales.}
	\label{fig:big-bench}
\end{minipage}
\begin{minipage}{0.27\linewidth}
\centering
\renewcommand\tabcolsep{1pt}
\renewcommand\arraystretch{1.4}
\scalebox{0.9}{
\footnotesize
\begin{tabular}{@{}lccc@{}}
\toprule[1.2pt]
           & 0-shot         & 1-shot         & 3-shot         \\ \midrule
GPT-3 2.6B & 0.60           & 0.71           & 1.83           \\
GPT-3 6.7B & -0.06          & 2.93           & 5.40           \\
GPT-3 13B  & 1.77           & 5.43           & 7.95           \\
GPT-3 175B & 4.35           & 11.34          & 13.18          \\ \midrule
PaLM 540B  & 8.05           & \textbf{37.77} & -              \\ \midrule
\glm       & \textbf{13.31} & 14.91          & \textbf{15.12} \\ \bottomrule[1.2pt]
\end{tabular}
}
\vspace{1mm}
\captionof{table}{Details on BIG-bench-lite (24 tasks).}
\label{tab:big-bench}
\end{minipage}
\vspace{-5mm}
\end{figure}

BIG-bench~\citep{srivastava2022beyond} is a novel benchmark with many challenging tasks concerning models' ability on reasoning, world knowledge, and commonsense.
Since the full BIG-bench covers 150 tasks, a complete evaluation can be too expensive and time-consuming for LLMs.
Thus the BIG-bench-lite, an officially released 26-task sub-collection, is considered in \glm at first.

Our results are shown in Figure~\ref{fig:big-bench} and Table~\ref{tab:big-bench}.
With 130 billion parameters, \glm outperforms larger-sized GPT-3 175B and even PaLM-540B in zero-shot setting.
This is probably owing to \glm's bidirectional attention over contexts and MIP training (which has been proved to encourage zero-shot learning in unseen tasks~\citep{wei2022finetuned,sanh2022multitask}).
As the number of shots increases, \glm's performance on BIG-bench-lite also grows and outperforms GPT-3.
More detailed results on each model for each task and evaluation can be find in Appendix~\ref{app:big-bench} and Table~\ref{tab:big-bench-details}.

\vvpara{Limitations and Discussion.}
We observe that \glm's growth when increasing few-shot samples is not as significant as GPT-3's.
There are some reasons that intuitively explain the phenomenon.
First, \glm's bidirectional nature could lead to stronger zero-shot performance (as is indicated in zero-shot language modeling) and approach significantly more to its few-shot upper-bound than unidirectional LLMs.
Second, it may be also attributed to a deficit of existing MIP paradigms~\citep{wei2022finetuned,sanh2022multitask}, which only involve zero-shot prediction in the training and will be likely to bias \glm for stronger zero-shot learning but relatively weaker in-context few-shot performance.
We think a potential remedy would be to employ MIP with varied shots of in-context samples rather than only zero-shot samples in our continual pre-training to correct the bias.

Finally, despite almost the same GPT architecture, PaLM-540B's growth with few-shot in-context learning is exceedingly substantial compared to GPT-3, which we suppose is because of its high-quality and diverse private collected corpora.
Combining observations from~\citep{hoffmann2022training}, we think it is high time that our community should invest more on better pre-training architecture, data, and more training FLOPS, rather than a mere larger size.

\subsection{Chinese Language Understanding Evaluation (CLUE)}

As GLM-130B is a bilingual language model, we also evaluate its zero-shot performance on established Chinese NLP benchmarks, CLUE~\citep{xu2020clue} and FewCLUE~\citep{xu2021fewclue}, which majorly address language understanding tasks such as text matching and sentimental analysis.
Note that we do not include any Chinese downstream tasks in MIP, so there is only possibly a zero-shot cross-lingual transfer that contributes to \glm's zero-shot Chinese performance.
We currently have finished the testing on part of the two benchmarks, including 7 CLUE and 5 FewCLUE datasets.

We compare GLM-130B to the largest existing Chinese monolingual language model ERNIE Titan 3.0~\citep{wang2021ernie}, which possesses 260B parameters.
Following its setting, we report zero-shot results on dev datasets.
As is shown in the Figure~\ref{fig:clue}, \glm generally outperforms ERNIE Titan 3.0.
Interestingly, \glm performs extraordinarily well on abstractive MRC datasets DRCD and CMRC2018.
We speculate that \glm's blank infilling objective naturally resonates to abstractive machine reading comprehension.
More details please refer to Appendix~\ref{app:clue}.

\begin{figure}[t]
    \centering
    \includegraphics[width=\linewidth]{figures/clue.pdf}
    \vspace{-6mm}
    \caption{\glm and ERNIE Titan 3.0 (260B) evaluated on zero-shot CLUE and FewCLUE.}
    \label{fig:clue}
    \vspace{-6mm}
\end{figure}

\section{Related Work}
In this section, we review related work to \glm on topics of pre-training, transferring, and inference of pre-trained LLMs~\citep{qiu2020pre,bommasani2021opportunities}, which
leverages self-supervised learning~\citep{liu2021self} over web-scale corpora and have deeply influenced the academia and society in recent years.

\vvpara{Pre-Training.}
Vanilla language modeling refers to decoder-only autoregressive models (e.g., GPT~\citep{radford2018improving}), but it also recognizes any forms of self-supervised objectives on texts.
Recently, transformer-based~\citep{vaswani2017attention} language models present a fascinating scaling law: new abilities~\citep{wei2022emergent} arouse as models scale up, from 1.5B~\citep{radford2019language}, 10B-scale language models~\citep{raffel2020exploring,shoeybi2019megatron,black2022gpt}, to 100B-scale GPT-3~\citep{brown2020language}.
Later, despite many emerged 100B-scale LLMs~\citep{lieber2021jurassic,thoppilan2022lamda,rae2021scaling,smith2022using,chowdhery2022palm,wu2021yuan,zeng2021pangu,wang2021ernie} in both English and Chinese, they are not available to public or only accessible via limited APIs.
The closeness of LLMs severely stymies its development.
\glm's efforts, along with recent ElutherAI, OPT-175B~\citep{zhang2022opt}, and BLOOM-176B~\citep{scao2022what}, aim to offer high-quality open-sourced LLMs to our community.
What additionally makes \glm distinct from others is that, \glm focuses on the inclusivity of LLMs to all researchers and developers.
From our decision of its size, the choice of its architecture, and the expenditure to allow its fast inference on popularized GPUs, \glm believes it is the inclusivity of all that be the key to realize LLMs' promised welfare to people.

\vvpara{Transferring.}
Though fine-tuning has been a \textit{de facto} way for transfer learning, the transfer learning for LLMs has been concentrated on prompting and in-context learning due to their tremendous sizes~\citep{brown2020language,liu2021pre}.
Following peer LLMs, in this work our evaluation is also under such setting.
Nevertheless, some recent attempts has been on parameter-efficient learning on language models
~\citep{houlsby2019parameter} and prompt tuning (i.e., P-tuning, ~\cite{li2021prefix,liu2021gpt,lester2021power,liu2022p}).
In this work, we do not focus on them and will leave their testing on \glm in future study.

\vvpara{Inference.}
Most public-accessible LLMs nowadays are providing their services via limited APIs
In this work, an important part of our endeavor has been on LLMs' efficient and fast inference.
Related work may include distillation~\citep{sanh2019distilbert,jiao2020tinybert,wang2020minilm}, quantization~\citep{zafrir2019q8bert,shen2020q,tao2022compression}, and pruning~\citep{michel2019sixteen,fan2019reducing}.
Very recent work~\citep{dettmers2022llm} shows that LLMs such as OPT-175B and BLOOM-176B can be quantized to 8 bit due to special distribution of outlier dimensions.

In this work, however, we demonstrate that GLM's scaling law for INT4 weight quantization, according to our novel insights for LLM architecture choices.
Such quantization, together with our engineering exertion on adapting \glm to FasterTransformer, allows \glm to be efficiently inferenced on as few as 4$\times$RTX 3090 (24G) GPUs or 8$\times$GTX 1080 Ti (11G) GPUs.
It realizes LLMs' economical availability to public and a phased accomplishment to our commitment of LLMs' inclusivity.
\section{Technical Details}

This section introduces additional details about the technical issues we have identified and solved throughout the \glm training.
Along with concurrent open-source LLM efforts, we believe those published details could serve as great cornerstones for future LLM training.

\subsection{Tokenization}
For the tokenization of the corpus, we implement a text tokenizer based on the package \textit{icetk} with several adjustments. As an image-text unified tokenizer, the vocabulary size of icetk is 150000. The first 20000 tokens are image tokens, and the rest are text tokens. The text tokenizer of icetk is formulated and trained by sentencepiece\footnote{\url{https://github.com/google/sentencepiece}}, on a 25GB bilingual corpus equally distributed with English and Chinese contents. We divide tokens recognized by the tokenizer into four categories. The common tokens are assigned from No.20000 to No.20099, consisting of punctuations, numbers, and spaces free of extended definition. No.20100 to No.83822 are English tokens, and No.83823 to No.145653 are Chinese tokens. Tokens after No.145653 are other special tokens, including concatenated punctuations, pieces from other languages, etc.

During our implementation, We ignored the first 20000 image tokens and utilized the latter 130000 intended for text tokenization. we disable the ignoring of linebreak to tokenize the linebreak mark
\texttt{\textbackslash n}
into No. 20004 token \texttt{<n>}. On the basis of inherent tokens, we add special tokens \texttt{[MASK]} and \texttt{[gMASK]} for model prediction. We also add special tokens \texttt{<sop>}, \texttt{<eop>}, \texttt{<eos>} for sentence and passage separation.

\subsection{Layer Normalization} \label{app:ln}
Here we briefly introduce the history of layer normalization in language modeling problems and how its variants perform in recent LLMs, including our experiments for them on \glm.

\vvpara{Post-LN~\citep{vaswani2017attention}.}
Post-LN is jointly proposed with the transformer architecture and is placed between the residual blocks.
It is then adopted by BERT~\citep{devlin2019bert} for bidirectional language model pre-training.
Nevertheless, Post-LN was later accused of transformers' slow and vulnerable converging~\citep{xiong2020layer}, and the Pre-LN emerged as a substitute.

\vvpara{Pre-LN~\citep{xiong2020layer}.}
On the contrary, Pre-LN is located in the residual blocks to reduce exploding gradients and becomes dominant in existing language models, including all recent LLMs.
However, OPT-175B~\citep{zhang2022opt}, BLOOM~\citep{scao2022what}, and text-to-image model CogView~\cite{ding2021cogview} later observe that Pre-LN is still unable to handle the vulnerable training when models scale up to 100B or meet multi-modal data.
This is also justified in \glm's preliminary experiments, where Pre-LN consistently crashes during the early training stage.

Another problem rooted in Pre-LN transformers is that it may harm the model performance after tuning compared to Post-LN.
This is observed in~\citep{he2021realformer}

\vvpara{Sandwich-LN~\citep{ding2021cogview}.}
As a remedy, on top of Pre-LN, CogView (later in Normformer~\citep{shleifer2021normformer}) develops Sandwich-LN, which appends extra normalization to the end of each residual branch.
Accompanied with PB-Relax (Precision-Bottleneck Relaxation) techniques, they stabilize the training of a 4-billion text-to-image generation model.
Despite its superiority over Pre-LN, sadly, Sandwich-LN is also proved to collapse in \glm training, let alone the potential consequent weaker tuning performance caused by its Pre-LN nature.

\subsection{Positional Encoding and Feed-forward Network} \label{app:pe-ffn}

\vvpara{Positional Encoding.} Vanilla transformer adopts absolute (or sinuous) position encoding and is later evolved into relative positional encoding~\citep{dai2019transformer}.
Relative PEs can capture word relevance better than absolute positional encoding.
Rotary Positional Embedding (RoPE) \citep{su2021roformer} is a relative position encoding implemented in the form of absolute position encoding, and its core idea is shown in the following equation.
\begin{equation}
(\boldsymbol{R}_m q)^{\top}(\boldsymbol{R}_n k) = q^{\top} \boldsymbol{R}_m^{\top}\boldsymbol{R}_n k = q^{\top} \boldsymbol{R}_{n-m} k
\end{equation}

The product of $q$ at position $m$ and $k$ at position $n$ is related to their distance $n - m$, which reflects the relativity of the position encoding. The definition of $\boldsymbol{R}$ in the above equation is

\begin{equation}
\boldsymbol{R}_{\theta, m}^{d}=\left(\begin{array}{ccccccc}
\cos m \theta_{1} & -\sin m \theta_{1} & 0 & 0 & \cdots & 0 & 0 \\
\sin m \theta_{1} & \cos m \theta_{1} & 0 & 0 & \cdots & 0 & 0 \\
0 & 0 & \cos m \theta_{2} & -\sin m \theta_{2} & \cdots & 0 & 0 \\\
0 & 0 & \sin m \theta_{2} & \cos m \theta_{2} & \cdots & 0 & 0 \\\
\vdots & \vdots & \vdots & \vdots & \ddots & \vdots & \vdots \\
0 & 0 & 0 & 0 & \cdots & \cos m \theta_{d / 2} & -\sin m \theta_{d / 2} \\
0 & 0 & 0 & 0 & \cdots & \sin m \theta_{d / 2} & \cos m \theta_{d / 2}
\end{array}\right)
\end{equation}

To allow its value to decay as the distance increases, we set $\theta$ to the value
\begin{equation}
\theta = \left\{\theta_i = 10000^{\frac{-2(i-1)} d},\quad i \in \left[1, 2, \cdots, \frac d 2\right]\right\}
\end{equation}
A two-dimensional absolute position encoding method is proposed in vanilla GLM for modeling intra- and inter-span position information.
In \glm, different from the two-dimensional positional encoding used in vanilla GLM, we turn back to conventional one-dimensional positional encoding.
However, we initially thought that we could not directly apply the two-dimensional form to RoPE\footnote{We later found the instructions to implement two-dimensional RoPE from its author blog \url{https://kexue.fm/archives/8397}. However, our training has proceeded for weeks.}.
As a substitute plan, in \glm, we remove the second dimension used in the original GLM as we find that the unidirectional attention mask sub-matrices for [MASK] generation also indicate the token order.
This observation results in our transforming \glm's positional encoding into a one-dimensional one according to the following strategies:

\begin{itemize}[leftmargin=*,itemsep=0pt,parsep=0.2em,topsep=0.0em,partopsep=0.0em]
    \item For sequences corrupted by short spans, we discard the second-dimensional position encoding.
    \item For sequences corrupted by a long span at the end, we change the positional ids to one-dimensional $0, 1, \cdots, s - 1$, and generated tokens will prolong the first-dimensional positional encoding from the last context token $s - 1$.
\end{itemize}

\vvpara{Feed-forward Network.}
Some recent efforts to improve transformer architecture have been on the FFN, including replacing it with GLU (adopted in PaLM). Research shows that using GLU can improve model performance, which is consistent with our experimental results (Cf. Table~\ref{tab:pe-ffn-ablation}). Specifically, we use GLU with the GeLU~\citep{hendrycks2016gaussian} activation. as
\begin{equation}
\operatorname{FFN}_{\mathrm{GeGLU}}\left(\boldsymbol{x}; \boldsymbol{W}_1, \boldsymbol{V}, \boldsymbol{W}_{2}\right)=\left(\mathrm{GeLU}(\boldsymbol{x} \boldsymbol{W}_1) \otimes \boldsymbol{x} \boldsymbol{V}\right) \boldsymbol{W}_{2}
\end{equation}

In order to keep the same parameter as the vanilla FFN, the feed-forward size $d_{\mathrm{ffn}}$ (which is usually $4 d_{\mathrm{H}}$, where $d_{\mathrm{H}}$ is the hidden dimension) is reduced to $\frac 8 3 d_{\mathrm{H}}$ as the $\boldsymbol{V}$ is additionally introduced.

\begin{wraptable}{r}{4.5cm}
	\centering
	\footnotesize
	\vspace{-4mm}
    \renewcommand\tabcolsep{2pt}
    \renewcommand\arraystretch{1.1}
	\caption{Ablation Study for PE and FFN on GLM\textsubscript{Base}}
	\vspace{-1mm}
	\scalebox{1.00}{
	\label{tab:pe-ffn-ablation}
	\begin{tabular}{@{}lc@{}}
    \toprule[1.2pt]
    Model           & Test PPL        \\ \midrule
    GLM\textsubscript{Base}        & 24.58 \\
    \quad + ALiBi         & 24.14 \\
    \quad + RoPE          & 22.95 \\
    \quad + RoPE + GeGLU  & \textbf{22.31} \\
    \bottomrule[1.2pt]
    \end{tabular}
    }

\end{wraptable}

\vvpara{Ablation Study on PE and FFN.} To validate our PE and FFN choices, we test them in our experiments by pre-training GLM\textsubscript{Base} (110M) over a random 50G Chinese and English mixed corpus. We compare absolute PE with two recent popular relative PE variants, RoPE~\citep{chowdhery2022palm} and ALiBi~\citep{press2021train}. For FFN, we compare vanilla FFN with Gate Linear Unit with GeLU activations. Results from Table \ref{tab:pe-ffn-ablation} show that both ALiBi and RoPE improve perplexity on the test set, and the improvement is more significant with RoPE while using GeGLU can further improve the model's performance.

\subsection{Pipeline Parallel Analysis} \label{app:pipeline}

\begin{figure}[t]
\centering
\subfigure[Naive pipeline implementation, which can be extremely inefficient.] {
 \label{fig:pipeline-naive}
\vspace{-4mm}
\includegraphics[width=0.92\linewidth]{figures/pipeline/pipeline.pdf}
}
\subfigure[GPipe~\citep{huang2019gpipe} implementation.] {
\label{fig:gpipe}
\includegraphics[width=0.92\linewidth]{figures/pipeline/pipeline-gpipe.pdf}
}
\subfigure[Pipedream~\citep{narayanan2021memory} implementation (used in \glm).] {
\label{fig:pipedream}
\vspace{-4mm}
\includegraphics[width=0.92\linewidth]{figures/pipeline/pipeline-pipedream.pdf}
}
\vspace{-4mm}
\caption{Different pipeline strategies and their conceptual comparison.}
\label{fig:pipeline}
\vspace{-5mm}
\end{figure}

In pipeline parallelism, each stage consists of three operations (Cf. Figure~\ref{fig:pipeline-naive}): forward (denoted as F), backward (marked as B), and optimizer step (denoted as U).
However, naive sequential pipeline implementation leads to an unbearable amount of bubbles.
The improved Gpipe~\citep{huang2019gpipe} (Cf. Figure~\ref{fig:gpipe}) strategy reduces bubbles drastically via splitting data into micro-batches; the more micro-batches there are, the more stages can compute simultaneously in an iteration.
The recent PipeDream-Flush~\citep{narayanan2021memory} (Cf. Figure~\ref{fig:pipedream}) additionally optimizes the GPU memory usage by interweaving forward and backward from different stages to reduce forward activation's memory occupation.

We analyze the bubble share in \glm's pre-training by assuming that the number of pipeline segments is $p$, the number of micro-batches is $m$, and the time for forward and backward per micro-batch are $t_f$ and $t_b$.
In ideal case, forward and backward take $t_{\mathrm{ideal}} = m(t_f + t_b)$.
But in practice, the default pipeline delivery strategy causes $p - 1$ forward propagation and $p - 1$ backward propagation bubbles, respectively, for a total time of $t_{\mathrm{bubble}} = (p - 1)( t_f + t_b)$, so that the bubble occupancy is
\begin{equation}
    \text{bubble-ratio} = \frac {t_{\mathrm{bubble}}} {t_{\mathrm{ideal}} + t_{\mathrm{bubble}}} = \frac {p - 1} {m + p - 1}
\end{equation}
For larger numbers of micro-batches, the bubble percentage will be reduced to an acceptable level. In particular, experiments in GPipe \cite{huang2019gpipe} show that when $m \ge 4p$, the total percentage of pipeline bubble time is reduced to a negligible level due to the forward recomputation technique in backpropagation that allows some overlap in computational communication, thus showing that the bubbles introduced in parallel by the pipeline model do not seriously deplete the training efficiency.

In general, to make full use of the hardware, it is common to place models into model parallel groups consisting of multiple nodes and try to use the full memory of each node. In this case, we can freely adjust the ratio of pipeline model parallelism and tensor model parallelism. Since data parallelism hardly affects the computation time, we assume that the scale of data parallelism is $d = 1$, the total number of nodes is $n$, the scale of tensor model parallelism is $t$, and the scale of pipeline model parallelism is $p$, and satisfies $n = t\times p$, the bubble share in this case is
\begin{equation}
    \text{bubble-ratio} = \frac {n/t - 1} {m + n/t - 1}
\end{equation}
From the above equation, we can see that increasing the size of tensor parallelism will further reduce the bubble ratio. However, the tensor parallelism scale cannot be increased indefinitely, which would reduce computational granularity and significantly increase the communication cost across a certain threshold. Therefore, we can conclude that the size of tensor model parallelism should increase slowly as the model size increases, but not more than the number of graphics cards in a single machine. In the training of \glm, the experiments show that the optimal tensor parallelism scale is $t = 4$ and does not scale up to $t = 8$ in the DGX-A100 system. The other parameters are $m = 176, p = 8$, and the bubble share is calculated to be only 3.8\%, sufficient to demonstrate pipeline model parallelism's efficiency.

\subsection{Inference Acceleration} \label{app:inference_acceleration}

A model's plain PyTorch implementation is easy to read and run, but it can be intolerably slow for LLMs.
Based on NVIDIA's FasterTransformer\footnote{\url{https://github.com/NVIDIA/FasterTransformer}} we spend two months implementing \glm into C++ to speed up inference, including the following main optimizations:
\begin{itemize}[leftmargin=*,itemsep=0pt,parsep=0.2em,topsep=0.0em,partopsep=0.0em]
    \item Optimize time-costing operations such as GeGLU, Layer Normalization, and SoftMax.
    \item Reduce GPU kernel calls (e.g., fuse MultiheadAttention into one computation kernel).
    \item Specify the algorithm of the best performance when calling cuBLAS.
    \item Improve the computing efficiency by transposing the model parameters in advance.
    \item Use half2 in FP16 computation to double the half's access bandwidth and computing throughput.
\end{itemize}

We currently pack up the full FasterTransformer implementation for \glm into a plug-and-play docker image for users' convenience, and we are still working on adapting it to our Pytorch implementation
by only changing one line of code.
A comparison between our speeding up \glm implementation and the so far default available BLOOM-176B implementation in Huggingface Transformers\footnote{\url{https://huggingface.co/docs/transformers/model_doc/bloom}} is shown in Table~\ref{tab:fastertransformer}.
Our implementation for \glm can be 7.0 to 8.4 times faster than BLOOM-176B's Pytorch implementation.
The exertion to accelerate LLM for tolerable response speed could be crucial to its popularization.

\input{tables/faster-transformer}

\subsection{Activation Outlier Analysis} \label{app:activation_analysis}

\begin{figure}[t]
    \centering
    \includegraphics[width=0.95\linewidth]{figures/activation_outliers_plot2d_combine.pdf}
    \caption{Distribution of outliers

    in \glm's activations. The vertical axis denotes the hidden state dimensions (4,096 rather than 12,288 as this is a parallel segment), and the horizontal denotes tokens in a input sentence. Using a 128$\times$128 2D histogram to get a better view of the distribution of outliers. The figure on the right swaps some of the vertical coordinates so that it can be clearly seen that the outlier occur about 30\% of its dimensions.
    }
    \label{fig:activation_outliers}
    \vspace{-6mm}
\end{figure}

As described in prior sections, \glm's weight can be quantized into INT4 to reduce parameter redundancy in the inference drastically.
However, we also find that \glm's activations (i.e., hidden states between layers) cannot be appropriately quantized, as they contain value outliers, as is also suggested in concurrent literature~\citep{dettmers2022llm}.
What is unique in \glm is that 30\% of its dimensions may present value outliers (Cf. Figure~\ref{fig:activation_outliers}), while other GPT-based LLMs (e.g., OPT-175B and BLOOM 176B) only have very few outlying dimensions~\citep{dettmers2022llm}.
Therefore, the solution to decompose matrix multiplication for higher-precision computation in outlying dimensions proposed in~\citep{dettmers2022llm} does not apply to \glm.

\begin{wrapfigure}{r}{4.2cm}
    \small
    \vspace{-6mm}
    \centering
    \includegraphics[width=1.0\linewidth]{figures/outlier_scale.png}
    \vspace{-6mm}
    \caption{\glm's activation outliers' absolute value scale.}
    \label{fig:outlier_scale}
    \vspace{-8mm}
\end{wrapfigure}

We study whether these outliers can be ignored in LLM quantization, and the answer is ``no''.
These values can be several orders of magnitude larger than typical activation values (Cf. Figure~\ref{fig:outlier_scale}).
While most values (accounts for 99.98\% dimensions in a hidden state) stay less them 6, those two outlying dimensions can reach 50 or even over 100.
They are speculated to be some important clues for \glm and potentially other LLMs to memorize some fixed world or language knowledge. Thus, removing or omitting them in quantization can lead to significant performance degradation.

\subsection{Weight Quantization} \label{app:quantization_analysis}

\subsubsection{Preliminaries}

\vvpara{Absmax Quantization.} is a symmetric quantization that a range of $[-\mathrm{absmax}(x), \mathrm{absmax}(x)]$ is mapped to $[-(2^b - 1), 2^b - 1]$ for $x$.
\begin{align}
    s_{x} &= \frac {\mathrm{absmax}(x)} {2^{b - 1} - 1} \\
    x_{q} &= \mathrm{round}(x / s_{x})
\end{align}
where $s_x$ is the scaling factor, $x_q$ is the quantization result, and $b$ is the bit width.
\vspace{-0.5em}

\vvpara{Zeropoint Quantization.} is an asymmetric quantization that a range of $[\min(x), \max(x)]$ is mapped to $[-(2^b - 1), 2^b - 1]$.
\begin{align}
    s_{x} &= \frac {\max(x) - \min(x)} {2^b - 2} \\
    z_{x} &= \mathrm{round}(\min(x) / s_{x}) + 2^{b - 1} - 1\\
    x_{q} &= \mathrm{round}(x / s_{x}) - z_{x}
\end{align}
where $z_x$ is the zero point.
\vspace{-0.5em}

\vvpara{Col/Row-wise Quantization.} Using a single scaling factor for the weight matrix often leads to more quantization errors because one single outlier decreases the quantization precision of all other elements. A common workaround is to group the weight matrix by rows or columns, with each group being quantized separately and having independent scaling factors.

\subsection{Quantization settings}

Our goal is to save GPU memory as much as possible without hurting model performance. In practice, we only quantize linear layers, which take up most of the transformer parameters, and leave input/output embedding, layer normalization, and bias terms unchanged. At the quantization precision of INT4, two INT4 weights are compressed into one INT8 weight to save GPU memory usage. Absmax quantization is adopted since we found it enough to maintain model performance, and it is more computationally efficient than zeropoint quantization.
During inference, only quantized weights are stored in GPU memory; the FP16 weights for linear layers are dequantized at runtime.

\subsubsection{Quantization Results at Scales}

\input{tables/weight_quantization}

GLM models at 110M to 10B scale are from GLM's original paper\citep{du2022glm}. Although the architecture of smaller scale GLMs is not the same as \glm, we believe that the training objective is the key factor for quantization. Table~\ref{tab:weight-quantization} shows the performance of GLM and BLOOM family models at different scales on the LAMBADA dataset with varying quantization methods. Almost all models maintain performance at INT8 precision. GLM maintains better performance than BLOOM at INT4 precision as it scales.

\subsubsection{Weight Distribution Analysis}

To achieve INT4 weight quantization, we analyze the weight value distribution of primary linear layers in \glm and a counterpart BLOOM-176B in a histogram (Cf. Figure~\ref{fig:quantization_appendix}).
The horizontal axis denotes the weight value, and the vertical axis indicates the number of weights of such value on the log scale.
As we can see, it is majorly the \texttt{w2} linear layers in BLOOM-176B that present skewed distributions, which would hinder the symmetrical quantization.
On the contrary, \glm's \texttt{w2} is well-shaped without many outliers and skewed distribution and thus paces the way for its INT4 quantization with little performance loss.

\clearpage

\begin{figure}[t]
    \centering

    \includegraphics[width=0.88\linewidth]{figures/quantization-appendix.png}
    \caption{Weight value distribution of linear layers in \glm (in orange, \texttt{attn-dense}, \texttt{attn-qkv}, \texttt{glu-w1}, \texttt{glu-w2}) and BLOOM-176B (in blue, \texttt{attn-dense}, \texttt{attn-qkv}, \texttt{ffn-w1}, \texttt{ffn-w2})'s first 28 transformer layers. Generally for \glm it is \texttt{attn-dense} and \texttt{w2} that may present narrow value distributions. \texttt{attn-qkv} and \texttt{w1} may also be a reason for enabling INT4 quantization in middle layers of \glm.}
    \label{fig:quantization_appendix}

\end{figure}
\section{Dataset and Evaluation Details} \label{app:data}
\subsection{Multi-task Instruction Pre-training (MIP)} \label{app:mip_description}
Following practices in~\citep{raffel2020exploring,wei2022finetuned,sanh2022multitask,aribandi2022ext5}, we include many prompted instruction datasets in \glm's MIP training, which accounts for 5\% of the training tokens.
All prompts for T0 datasets are from PromptSource~\citep{bach2022promptsource}, and prompts for DeepStruct datasets are newly created.
Their composition is shown in Table~\ref{tab:mip}, which makes up natural language understanding and generation datasets from T0~\citep{sanh2022multitask} and promptsource~\citep{bach2022promptsource}, and information extraction datasets from DeepStruct~\citep{wang2022deepstruct}.
In \glm's training, we calculate that approximately 36\% of the samples in each dataset have been seen.

T0 originally splits datasets for 1) multi-task prompted training and 2) zero-shot task transfer in two sections.
We initially planned to only include training sets of T0's multi-task prompted training section and DeepStruct~\citep{wang2022deepstruct}, but by mistake, we included both multi-task prompted training and zero-shot task transfer sections' datasets in MIP and excluded DeepStruct datasets.
We fixed the mistake at around 23k steps, and our model continued to train on the correct version.

\vvpara{Natural Language Understanding and Generation.}
We adopt datasets and corresponding prompts from promptsource~\citep{bach2022promptsource}.
For all prompted samples in each dataset, we set a truncation of maximal 10,0000 samples per dataset and combined them as the MIP dataset.
Details of the prompted samples and datasets are provided in promptsource's GitHub repository\footnote{\url{https://github.com/bigscience-workshop/promptsource}}.

\vvpara{Information Extraction.}
Based on the datasets from DeepStruct~\citep{wang2022deepstruct}, a multi-task language model pre-training approach for information extraction tasks, we create instructions and prompts for part of its datasets (as shown in Table~\ref{tab:mip}).
We reformulate information extraction tasks into instruction tuning formats to allow zero-shot generalization to a new extraction schema.
For all prompted samples in each dataset, we set a truncation of maximal 20,0000 samples per dataset as there are fewer information extraction datasets than common language understanding and generation ones.
For KELM~\citep{agarwal2021knowledge} and PropBank~\citep{kingsbury2002treebank} datasets, since their original size is gigantic, we sample 50,0000 samples for each of them from their prompted examples.

\subsection{Data and prompts in MIP for DeepStruct} \label{app:mip_dataset}
Prompts and instructions for all datasets in DeepStruct~\citep{wang2022deepstruct} are newly created by authors manually.
The introduction, task description, and full prompts for each dataset are attached in the following sections.
To allow template infilling, we write all prompts into Jinja\footnote{\url{https://github.com/pallets/jinja}} templates.
When a dataset sample is provided in our format, the Jinja engine will render it into a prompted sample with instructions.

A more systematic evaluation of \glm's information extraction ability is left for future work, as this work concentrates on the training and designing details of an LLM.

\subsubsection{Dialogue State Tracking}
We adopt Multiwoz 2.1~\citep{eric2020multiwoz} dialogue state tracking dataset.
The dataset is reformulated into two tasks, each with one prompt correspondingly:
\begin{itemize}[leftmargin=*,itemsep=0pt,parsep=0.2em,topsep=0.0em,partopsep=0.0em]
    \item \textbf{Dialogue state tracking}: which asks the model to extract information from dialogues given a list of certain slots, e.g., \texttt{taxi\_arrival\_time} and \texttt{destination}.
    \item \textbf{Slot filling}: which model should fill in one provided slot and identify situations without an answer.
\end{itemize}

\begin{tcolorbox}[left=0mm,right=0mm,top=0mm,bottom=0mm,boxsep=1mm,arc=0mm,boxrule=0pt, frame empty]
\textbf{(Dialogue State Tracking, Prompt 0)}
\small
\begin{lstlisting}
Read the dialogues between "[User]" and "[Agent]",

{{text}}

identify and extract the information related to the following categories (from top to down):

- {{allowed_relations | join("\n- ")}}

in the form of "( [User] ; Y ; Z )": ||| {{format_triple(relations, allowed_relations) | join(" ")}}
\end{lstlisting}
\end{tcolorbox}

\begin{tcolorbox}[left=0mm,right=0mm,top=0mm,bottom=0mm,boxsep=1mm,arc=0mm,boxrule=0pt, frame empty]
\textbf{(Slot Filling, Prompt 0)}
\small
\begin{lstlisting}
Given the following dialogue:

{{text}}

please answer the question: has "[User]" mentioned "{{allowed_relations[relation_idx].split(': ') | join("'s ")}}" ? If yes, please write down the answer from the dialogue; if not, please answer "not given".

Answer: |||
\end{lstlisting}
\end{tcolorbox}

\subsubsection{Event Extraction}
We adopt ACE05~\citep{walker2005ace} event extraction datasets following the setting in~\citep{wadden2019entity}.
The dataset is reformulated into two tasks with three prompts as follows:

\begin{itemize}[leftmargin=*,itemsep=0pt,parsep=0.2em,topsep=0.0em,partopsep=0.0em]
    \item \textbf{Event Argument Extraction}: given a trigger in text and a list of its argument roles, the model is asked to extract the arguments from the provided text.
    \item \textbf{Argument Identification}: given a trigger and a certain argument role, the model is asked to extract the argument if it exists in the provided text; otherwise, the model should generate nothing.
\end{itemize}

\begin{tcolorbox}[left=0mm,right=0mm,top=0mm,bottom=0mm,boxsep=1mm,arc=0mm,boxrule=0pt, frame empty]
\textbf{(Event Argument Extraction, Prompt 0)}
\small
\begin{lstlisting}
For the task of "Event Extraction", given a trigger one should extract its related arguments conditioned on a list of potential roles.

Given the following list of roles:

- {{shuffle(allowed_arguments[trigger['event_type']].values()) | join("\n- ")}}

extract related arguments of the trigger "{{trigger['text']}} ({{allowed_triggers[trigger['event_type']]}})" in the following sentence:

{{text}}

Extractions: ||| {{format_triple(relations, "") | join(" ")}}
\end{lstlisting}
\end{tcolorbox}

\begin{tcolorbox}[left=0mm,right=0mm,top=0mm,bottom=0mm,boxsep=1mm,arc=0mm,boxrule=0pt, frame empty]
\textbf{(Event Argument Extraction, Prompt 1)}
\small
\begin{lstlisting}
TEST

1. (Event Extraction) {{text}}

Please write down ALL event arguments related to the trigger "{{trigger['text']}} ({{allowed_triggers[trigger['event_type']]}})" marked with "[ ]", given the following categories:

- {{shuffle(allowed_arguments[trigger['event_type']].values()) | join("\n- ")}}

Answer: ||| {{format_triple(relations, "") | join(" ")}}
\end{lstlisting}
\end{tcolorbox}

\begin{tcolorbox}[left=0mm,right=0mm,top=0mm,bottom=0mm,boxsep=1mm,arc=0mm,boxrule=0pt, frame empty]
\textbf{(Argument Identification, Prompt 0)}
\small
\begin{lstlisting}
Let extract event related arguments!

In the following passage, an argument with the type "{{query_arg}}" is related to the event trigger "{{trigger['text']}} ({{allowed_triggers[trigger['event_type']]}})":

{{text}}

The argument should be (copy from the context if you find it; if not, do not generate): ||| {{filter_type(relations, query_arg) | join(" ")}}
\end{lstlisting}
\end{tcolorbox}

\subsubsection{Joint Entity and Relation Extraction}
Joint entity and relation extraction aims to recognize named entities in a piece of text and judge the relationships between them.
It is closely related to knowledge acquisition, where the ultimate target is to structure the unstructured web contents into knowledge triples (e.g., \texttt{(London, capital\_of, Britain)}).
The task can be formulated into a pipeline framework (a combination of named entity recognition and relation extraction) or end-to-end training.

In this work, we adopt three classical joint entity and relation extraction datasets: CoNLL04~\citep{roth-yih-2004-linear}, NYT~\citep{10.1007/978-3-642-15939-8_10}, and ACE2005~\citep{walker2005ace}.
In \glm, we follow~\citep{wang2022deepstruct} to formulate such challenges into sequence-to-sequence generation, where our inputs are raw texts and outputs are triples.
We only conduct relation-related tasks for these datasets here and leave the entity-related ones in the named entity recognition section.
\begin{itemize}[leftmargin=*,itemsep=0pt,parsep=0.2em,topsep=0.0em,partopsep=0.0em]
    \item \textbf{Relation Extraction}: here we extract knowledge triples consisting of ``head entity'', ``relation'', and ``tail entity'', given a list of relation candidates. For example, given the input ``\textit{In Kunming the 800-some faculty and student established the National Southwestern Associated University.}'', the model output could be \texttt{(National Southwestern Associated University, location of formation, Kunming)}.
    \item \textbf{Conditional Relation Extraction}: given a single relation candidate, judge if the input text contains the relation. If so, extract all related triples; if not, do not generate.
    \item \textbf{Knowledge Slot Filling}: assign a particular entity from text, and ask the model to extract all triples that take the entity as the head.
    \item \textbf{Relation Classification}: given two entities from texts, ask the model to judge the relation between them based on a list of candidate relations.
\end{itemize}

\begin{tcolorbox}[left=0mm,right=0mm,top=0mm,bottom=0mm,boxsep=1mm,arc=0mm,boxrule=0pt, frame empty]
\textbf{(Relation Extraction, Prompt 0)}
\small
\begin{lstlisting}
Can you figure out all triples regarding the relations of "{{shuffle(allowed_relations) | join('", "')}}" from the sentence? List them in the shape of "( X ; Y ; Z )":

{{text}} => ||| {{format_triple(relations, allowed_relations) | join(" ")}}
\end{lstlisting}
\end{tcolorbox}

\begin{tcolorbox}[left=0mm,right=0mm,top=0mm,bottom=0mm,boxsep=1mm,arc=0mm,boxrule=0pt, frame empty]
\textbf{(Conditional Relation Extraction, Prompt 0)}
\small
\begin{lstlisting}
Conditioned on the relation "{{allowed_relations[relation_idx]}}", what knowledge triples can be extracted from:

{{text}}

Please write them down here: ||| {{format_triple(relations, [allowed_relations[relation_idx]]) | join(" ")}}
\end{lstlisting}
\end{tcolorbox}

\begin{tcolorbox}[left=0mm,right=0mm,top=0mm,bottom=0mm,boxsep=1mm,arc=0mm,boxrule=0pt, frame empty]
\textbf{(Knowledge Slot Filling, Prompt 0)}
\small
\begin{lstlisting}

In the sentence

{{text}}

the X = "{{entities[entity_idx]}}" is an entity of the type "{{entity_types[entity_idx]}}". Extract all possible triples contains "{{entities[entity_idx]}}" in the form of ( X ; Y ; Z ), given the following candidate properties Y:

Answer: |||

\end{lstlisting}
\end{tcolorbox}

\begin{tcolorbox}[left=0mm,right=0mm,top=0mm,bottom=0mm,boxsep=1mm,arc=0mm,boxrule=0pt, frame empty]
\textbf{(Relation Classification, Prompt 0)}
\small
\begin{lstlisting}
QUIZ

1. Given the candidate relations:

- {{shuffle(allowed_relations) | join("\n- ")}}

what is the relation between "{{relations[triple_idx]['head'][0]}}" and "{{relations[triple_idx]['tail'][0]}}" in the following sentence?

{{text}}

Answer: ||| {{relations[triple_idx]['relation']}}
\end{lstlisting}
\end{tcolorbox}

Nevertheless, existing joint entity and relation extraction datasets have minimal relation schema.
For example, CoNLL04 only contains five different relations; the most diverse NYT dataset contains 24 Freebase predicates.
To allow the model to capture a diverse range of potential verbalized predicates, we extend the task with automatically generated knowledge-text aligned data from KELM~\citep{agarwal2021knowledge}.
Since they can be extremely noisy, we do not include other distantly supervised datasets (e.g., T-Rex~\citep{elsahar2018t}).

For KELM data, since it is based on the complete Wikidata schema (which contains too many relations to be enumerated), we create two KELM-specific prompts for the task of \textbf{Relation Extraction} and \textbf{Knowledge Slot Filling}:

\begin{tcolorbox}[left=0mm,right=0mm,top=0mm,bottom=0mm,boxsep=1mm,arc=0mm,boxrule=0pt, frame empty]
\textbf{(Relation Extraction, Prompt 1, KELM ONLY)}
\small
\begin{lstlisting}
{# kelm #}
Can you figure out all knowledge triples regarding whole Wikidata properties from the sentence? List them in the shape of "( X ; Y ; Z )":

{{text}} => ||| {{format_triple(relations, "") | join(" ")}}
\end{lstlisting}
\end{tcolorbox}

\begin{tcolorbox}[left=0mm,right=0mm,top=0mm,bottom=0mm,boxsep=1mm,arc=0mm,boxrule=0pt, frame empty]
\textbf{(Knowledge Slot Filling, Prompt 1, KELM ONLY)}
\small
\begin{lstlisting}
{# kelm #}
Given the entity "{{entities[entity_idx]}}" marked with "[" and "]" in the context:

{{text}}

please list all triples related to it (do not generate if there is no answer): |||
\end{lstlisting}
\end{tcolorbox}

\subsubsection{Named Entity Recognition}
Named entity recognition is a task that targets identifying named entities from a raw text corpus and assigning them with proper entity types.
For example, in the sentence ``\textit{In 1916 GM was reincorporated in Detroit as "General Motors Corporation".}'', \texttt{General Motors Corporation} could be of entity type \texttt{organization}.
We design two different types of tasks based on named entity recognition datasets CoNLL03~\citep{tjong-kim-sang-de-meulder-2003-introduction}, OntoNotes 5.0~\citep{pradhan-etal-2013-towards}, and GENIA~\citep{10.5555/1289189.1289260}. We also include named entity recognition sub-tasks from the joint entity and relation datasets.
\begin{itemize}[leftmargin=*,itemsep=0pt,parsep=0.2em,topsep=0.0em,partopsep=0.0em]
    \item \textbf{Named Entity Recognition}: given a certain list of possible entity types (e.g., \texttt{location}, \texttt{person}, \texttt{organization}), extract all related entities from the provided text content.
    \item \textbf{Entity Typing}: entity typing is one of the important derivative tasks from named entity recognition. It aims to classify the correct type of entity mention (without entity types) and is often appended to the entity mention extraction as post-processing.
\end{itemize}

\begin{tcolorbox}[left=0mm,right=0mm,top=0mm,bottom=0mm,boxsep=1mm,arc=0mm,boxrule=0pt, frame empty]
\textbf{(Named Entity Recognition, Prompt 0)}
\small
\begin{lstlisting}
Given the following list of entity types:

Z = {{shuffle(allowed_types) | join(", ")}}

please extract all mentioned entities from left to right in the sentence, in the form of "( X ; instance of ; Z )".

{{text}} => |||
\end{lstlisting}
\end{tcolorbox}

\begin{tcolorbox}[left=0mm,right=0mm,top=0mm,bottom=0mm,boxsep=1mm,arc=0mm,boxrule=0pt, frame empty]
\textbf{(Entity Typing, Prompt 0)}
\small
\begin{lstlisting}
Extract all entity mentioned in the sentence with entity type "{{allowed_types[type_idx]}}" in the form of "( X ; instance of ; {{allowed_types[type_idx]}} )"

{{text}} => |||
\end{lstlisting}
\end{tcolorbox}

\begin{tcolorbox}[left=0mm,right=0mm,top=0mm,bottom=0mm,boxsep=1mm,arc=0mm,boxrule=0pt, frame empty]
\textbf{(Entity Typing, Prompt 1)}
\small
\begin{lstlisting}
List all "{{allowed_types[type_idx]}}" entities appeared in the following passage, joined by " | ":

{{text}} => ||| {{filter_type(zip(entities, entity_types), allowed_types[type_idx]) | join(" | ")}}
\end{lstlisting}
\end{tcolorbox}

\begin{tcolorbox}[left=0mm,right=0mm,top=0mm,bottom=0mm,boxsep=1mm,arc=0mm,boxrule=0pt, frame empty]
\textbf{(Entity Typing, Prompt 2)}
\small
\begin{lstlisting}

Based on the list of potential entity types and ignore their order:

- {{shuffle(allowed_types) | join("\n- ")}}

the entity "{{entities[entity_idx]}}" marked with "[" and "]" in the following sentence:

{{text}}

belongs to ||| {{entity_types[entity_idx]}}

\end{lstlisting}
\end{tcolorbox}

\subsubsection{Relation Classification}
Relation classification is a fundamental task in information extraction, which identifies the relationships from a list of candidates between two given entities.
The problem is a long-standing one as it suffers from the outrageous cost of data labeling since manual labeling on knowledge-intensive tasks requires educated annotators that charge high.
A \textit{de facto} data creation method in relation extraction relies on distant supervision, which automatically aligns existing knowledge triples in knowledge bases to text contents. It assumes that such alignments are correct in certain conditions.
Here we only include TacRED~\citep{zhang2017tacred} dataset and create several different tasks based on it.
\begin{itemize}[leftmargin=*,itemsep=0pt,parsep=0.2em,topsep=0.0em,partopsep=0.0em]
    \item \textbf{Relation Classification}: the most traditional task formulation. Given two entities from the text, classify their relationship from a list of candidates. The form can answer the relation directly or in the form of a triple (similar to relation extraction).
    \item \textbf{Knowledge Slot Filling}: change the task into a given head entity and relation to identify whether the tail entity exists in the input text. If not, generate nothing.
    \item \textbf{Yes or No Question}: turn the problem into a task similar to natural language inference. For example, given the sentence ``\textit{The series focuses on the life of Carnie Wilson, daughter of Brian Wilson, founder of the Beach Boys.}'', the model will be asked to judge the correctness of a triple such as \texttt{Carnie Wilson, father, Brian Wilson} by answering ``yes'' or ``no''.
\end{itemize}

\begin{tcolorbox}[left=0mm,right=0mm,top=0mm,bottom=0mm,boxsep=1mm,arc=0mm,boxrule=0pt, frame empty]
\textbf{(Relation Classification, Prompt 0)}
\small
\begin{lstlisting}

Given the following categories of relations:

- {{shuffle(allowed_relations.values()) | join("\n- ")}}

predict the relation between "{{relations[0]['head']}}" and "{{relations[0]['tail']}}" in the following sentence:

{{text}}

The relation should be : ||| {{allowed_relations[relations[0]['relation']]}}

\end{lstlisting}
\end{tcolorbox}

\begin{tcolorbox}[left=0mm,right=0mm,top=0mm,bottom=0mm,boxsep=1mm,arc=0mm,boxrule=0pt, frame empty]
\textbf{(Relation Classification, Prompt 1)}
\small
\begin{lstlisting}
1. (Relation Extraction) Answer the relation between entities in the form of "( X ; Y ; Z )":

{{text}}

The relation between "{{relations[0]['head']}}" and "{{relations[0]['tail']}}" is: ||| ( {{relations[0]['head']}} ; {{allowed_relations[relations[0]['relation']]}} ; {{relations[0]['tail']}} )
\end{lstlisting}
\end{tcolorbox}

\begin{tcolorbox}[left=0mm,right=0mm,top=0mm,bottom=0mm,boxsep=1mm,arc=0mm,boxrule=0pt, frame empty]
\textbf{(Knowledge Slot Filling, Prompt 0)}
\small
\begin{lstlisting}
Based on the sentence provided below, infer the missing argument asked by the question:

{{text}}

Question: What/Who/Where is "{{relations[0]['head']}}" {{allowed_relations[relations[0]['relation']]}} ?

Answer: ||| {{relations[0]['tail']}}
\end{lstlisting}
\end{tcolorbox}

\subsubsection{Semantic Role Labeling}
Semantic role labeling is a long-standing information task that wants to identify the semantic arguments related to a given predicate in a sentence.
For example, in the sentence ``\textit{Grant was employed at IBM for 21 years where she held several executive positions.}'' and the predicate ``\texttt{employed}'' in it, semantic role labeling identifies the \texttt{Grant} as the subject and \texttt{IBM} as the second object.

We create two different tasks based on semantic role labelling datasets CoNLL05~\citep{carreras-marquez-2005-introduction}, CoNLL12~\citep{pradhan-etal-2013-towards}, and PropBank~\citep{kingsbury2002treebank}.
\begin{itemize}[leftmargin=*,itemsep=0pt,parsep=0.2em,topsep=0.0em,partopsep=0.0em]
    \item \textbf{Semantic Role Labeling}: the traditional task form, where a verb (i.e., predicate) is annotated in text, and the model is asked to generate related semantic roles.
    \item \textbf{Semantic Role Filling}: given a verb and a potential semantic role, the model is asked to judge whether the role exists in the sentence and generate it.
    \item \textbf{Predicate Recognition}: given a segment of a sentence and its corresponding semantic role, identify which verb it is related to.
\end{itemize}

\begin{tcolorbox}[left=0mm,right=0mm,top=0mm,bottom=0mm,boxsep=1mm,arc=0mm,boxrule=0pt, frame empty]
\textbf{(Semantic Role Labeling, Prompt 0)}
\small
\begin{lstlisting}
Provided with the target verb "{{verb}}" marked with "[" and "]" in the following sentence, find out its "{{allowed_types[type_idx]}}":

{{text}} => |||
\end{lstlisting}
\end{tcolorbox}

\begin{tcolorbox}[left=0mm,right=0mm,top=0mm,bottom=0mm,boxsep=1mm,arc=0mm,boxrule=0pt, frame empty]
\textbf{(Semantic Role Filling, Prompt 0)}
\small
\begin{lstlisting}
Given the following list of argument types:

Z = {{allowed_types | join(", ")}}

find out all arguments related to verb "{{verb}}" mentioned in the following sentence from left to right, in the form of "( X ; instance of ; Z )".

{{text}} => |||
\end{lstlisting}
\end{tcolorbox}

\begin{tcolorbox}[left=0mm,right=0mm,top=0mm,bottom=0mm,boxsep=1mm,arc=0mm,boxrule=0pt, frame empty]
\textbf{(Predicate Recognition, Prompt 0)}
\small
\begin{lstlisting}
FINAL EXAM

1. Based on the fact that "{{entities[entity_idx]}}" is a "{{entity_types[entity_idx]}}", which verb in the following sentence should it related to?

{{text}}

Answer: ||| {{verb}}
\end{lstlisting}
\end{tcolorbox}

\subsection{BIG-bench-lite Evaluation} \label{app:big-bench}
\begin{wrapfigure}{r}{5.5cm}
    \centering
    \vspace{-6mm}
    \includegraphics[width=\linewidth]{figures/big-bench-lite.pdf}
    \caption{A full scope of BIG-bench-lite (24 tasks) evaluation.}
    \label{fig:big-bench-lite-full}
    \vspace{-5mm}
\end{wrapfigure}
Recent works~\citep{wei2022chain,wang2022rationale} reveal that LLMs are capable of reasoning beyond conventional language tasks.
As a response, BIG-bench~\citep{srivastava2022beyond} is recently set up by crowdsourcing new types of tasks from global researchers to test LLMs' new abilities.
For economical consideration, we evaluate \glm on an official subset of the original 150-task BIG-bench, the BIG-bench-lite with 24 tasks.
These tasks can be categorized into two types: one is based on multiple-choice question answering with answer options, and another is a direct generation without options.
For the first category, we assess the probability of each option's complete content and pick the largest one as the answer; for the second one, we generate the response using greedy decoding.
All evaluations done in BIG-bench are based on [MASK] since answers here are usually short pieces of text.
All results on 24 BIG-bench-lite~\citep{srivastava2022beyond} datasets of three LLMs are shown in Table~\ref{tab:big-bench-details} and Figure~\ref{fig:big-bench-lite-full}.
We adopt the original prompts from BIG-bench and use the official implementation to generate priming examples for few-shot evaluation and to calculate the final scores.

\subsection{MMLU Evaluation} \label{app:mmlu}
All results on 57 MMLU~\citep{hendrycks2021measuring} datasets of \glm and BLOOM 176B are shown in Table~\ref{tab:mmlu}.
In Section~\ref{sec:mmlu}, we report weighted average accuracy (i.e., accuracy average per sample) of \glm, GPT-3 175B, and BLOOM 176B following the original benchmark setting~\citep{hendrycks2021measuring}.

Below is a prompted example with 1-shot priming. We predict the probability on \texttt{['A', 'B', 'C', 'D']} at the next token and take the one with the maximal probability as the answer.

\begin{tcolorbox}[left=0mm,right=0mm,top=0mm,bottom=0mm,boxsep=1mm,arc=0mm,boxrule=0pt, frame empty]
\textbf{(MMLU 1-shot Example)}
\small
\begin{lstlisting}
The following are multiple choice questions about philosophy.

According to d'Holbach, people always act according to _____.
(A) free choices (B) dictates of the soul (C) necessary natural laws (D) undetermined will
Answer: (C) necessary natural laws

Epicurus holds that philosophy is:
(A) not suitable for the young. (B) not suitable for the old. (C) important, but unpleasant. (D) none of the above.
Answer: (
\end{lstlisting}
\end{tcolorbox}

\vspace{5mm}
\subsection{Language Modeling Evaluation} \label{app:pile}

\vvpara{LAMBADA.}
We follow the evaluation setting in~\citep{radford2019language} to leverage a stop-word filter to select out a valid final word prediction with the largest score as our answer.
We use the beam search decoding strategy with beam size of 16 and limit the maximum generation length to 5.
Our predicted final word is a natural English word (i.e., may consist of multiple tokens) rather a single \glm token following the vanilla~\citep{paperno2016lambada} requirement.
Finally, we use string matching to judge the correctness.

\input{tables/Pile_test}

\vvpara{Pile.}
Pile evalution~\citep{gao2020pile} is a comprehensive language modeling benchmark that initially includes 22 different text datasets from diverse domains.
We report our results over a part of 18 datasets with previously reported baseline results~\citep{lieber2021jurassic}.
Unlike traditional language modeling benchmarks, Pile evaluation reports the BPB (bits-per-byte) perplexity to avoid the mismatch comparison between models with different vocabularies.
Because, in general, language models with a larger vocabulary will be favored in perplexity comparison if not restricted.
In the evaluation, we strictly follow the setting in~\citep{gao2020pile}, leveraging [gMASK] and a context-length of 1,024 with bidirectional attention, and the rest of 1024 tokens to calculate BPB in an autoregressive manner.
The weighted average BPB is calculated based on each shared dataset's ratio in Pile training-set~\citep{gao2020pile}.

The detailed metrics on the Pile test-set are reported in Table~\ref{tab:pile_full}. We observe that compared to GPT-3, \glm has a noticeably weaker performance on phil\_papers and pile\_cc, which is likely because of \glm's bilingual natural and lack of more diverse and high-quality private collected corpora.

\subsection{Chinese Language Understanding Evaluation} \label{app:clue}
Here we elaborate the prompts we use for CLUE~\citep{xu2020clue} and FewCLUE~\citep{xu2021fewclue} evaluation.
In Chinese datasets, prompting meets some challenges as Chinese texts are organized by single characters rather than words, leading to unequal length of verbalizers in many cases.
Albeit dataset-specific calibration~\citep{wang2021ernie,wu2021yuan} can help to mitigate the issue, the too-specified technique can be complicated in implementation.
Our evaluation in this paper adopts a more easy-to-solve method leveraging \glm's unique features.
As \glm is a bilingual LLM with English MIP, we adopt English prompts and verbalizers from similar tasks in~\citep{bach2022promptsource} for Chinese dataset evaluation and find such strategies to be quite effective.
In terms of evaluation metrics, except for DRCD and CMRC2018, two question answering datasets report EM, and other datasets report accuracy.

\input{tables/MIP_datasets}
\input{tables/big-bench-details}
\input{tables/MMLU}
\input{tables/configurations}

\clearpage
\section{Contributions} \label{app:contribution}

The GLM-130B project was conceived in Dec. 2021 with its pre-training part completed in July 3rd, 2022 and its evaluation and applications still ongoing.
Over the course, we have experienced various technical and engineering challenges (Cf. Appendix \ref{sec:appendix-history} and Figure~\ref{fig:timeline} for details).
It would not be possible to reach its current status if without the collaboration of multiple teams---the Knowledge Engineering Group (KEG), Parallel Architecture \& Compiler technology of Mobile, Accelerated, and Networked systems Group (PACMAN), and Natural Language Processing Group (THUNLP) at Tsinghua University,
as well as Zhipu.AI.
The detailed contributions are listed below.

\subsection{Preparation}

\begin{itemize}[leftmargin=*,itemsep=0pt,parsep=0.4em,topsep=0.0em,partopsep=0.0em]
\item \textbf{Model Implementation:} Aohan Zeng, Zhengxiao Du
\item \textbf{Self-Supervised Data Processing:} Ming Ding, Wendi Zheng
\item \textbf{Multitask Data Processing:} Xiao Liu, Xiao Xia
\item \textbf{Model Architecture:} Aohan Zeng, Xiao Liu, Zhengxiao Du, Hanyu Lai
\item \textbf{Training Stability:} Aohan Zeng, Xiao Liu, Ming Ding
\item \textbf{3D-Parallelism and Training Efficiency:} Aohan Zeng, Zixuan Ma, Jiaao He, Zhenbo Sun
\end{itemize}

\subsection{Model Training}

\begin{itemize}[leftmargin=*,itemsep=0pt,parsep=0.4em,topsep=0.0em,partopsep=0.0em]
\item \textbf{Large-Scale Training \& Monitoring:} Aohan Zeng, Xiao Liu
\item \textbf{Model Performance Validation:} Aohan Zeng
\end{itemize}

\subsection{Post Training}

\begin{itemize}[leftmargin=*,itemsep=0pt,parsep=0.4em,topsep=0.0em,partopsep=0.0em]
\item \textbf{Evaluation Framework:} Aohan Zeng, Zhengxiao Du
\item \textbf{Language Modeling Evaluation:} Aohan Zeng
\item \textbf{MMLU \& BIG-Bench Evaluation:} Aohan Zeng
\item \textbf{CLUE  \&  FewCLUE Evaluation:} Xiao Liu, Aohan Zeng
\item \textbf{Ethical Evaluation:} Yifan Xu, Aohan Zeng, Xiao Liu, Zihan Wang
\item \textbf{INT4 Quantization:} Aohan Zeng, Zihan Wang, Xiao Liu, Hanyu Lai
\item \textbf{Inference Acceleration:} Zihan Wang, Aohan Zeng
\item \textbf{Low-Resource Inference:} Gouyang Zeng, Xu Han, Weilin Zhao, Zhiyuan Liu
\item \textbf{Demo and API:} Hanyu Lai, Jifan Yu, Xiaohan Zhang, Yufei Xue, Shan Wang, Jiecai Shan, Haohan Jiang, Zhengang Guo
\item \textbf{Manuscript Writing:} Xiao Liu, Yuxiao Dong, and Jie Tang wrote the main paper, and Xiao Liu, Aohan Zeng, and Zhengxiao Du wrote the Appendix.
\end{itemize}

\subsection{Project Management}

\begin{itemize}[leftmargin=*,itemsep=0pt,parsep=0.4em,topsep=0.0em,partopsep=0.0em]
\item \textbf{Student Leaders:} Aohan Zeng, Xiao Liu
\item \textbf{Technical Advisors:} Yuxiao Dong, Jidong Zhai, Wenguang Chen, Peng Zhang, Jie Tang
\item \textbf{Project Leader:} Jie Tang
\end{itemize}

 \subsection{Computation Sponsor}
 \begin{itemize}[leftmargin=*,itemsep=0pt,parsep=0.4em,topsep=0.0em,partopsep=0.0em]
\item \textbf{GPU Sponsor:} Zhipu.AI

 \end{itemize}

\section{Broader Impact}
This paper introduces an open bilingual pre-trained language model with 130 billion parameters. Currently most pre-trained language models with over 100 billion parameters are privately owned by governments and large corporations~\citep{brown2020language,thoppilan2022lamda,rae2021scaling,chowdhery2022palm,wang2021ernie}. A few of them~\citep{brown2020language,lieber2021jurassic} provide limited inference APIs with fees. In contrast, the weights and code of \glm are open to anyone interested in LLMs. Moreover, we significantly lower the hardware requirements for inference by speed-up implementation and INT4 quantization. The paper can have a broader impact on the research community, individual developers, small companies, and society.

\subsection{Impact on AI Research}
Most research institutions cannot afford the substantial cost of pre-training large language models. As a result, most researchers, except employees of governments and large corporations, only have access to the limited inference APIs with fees. With the inference APIs, researchers can only analyze the outputs of models as black boxes, which limits the scope of potential work. With \glm, researchers can analyze the model parameters and internal states corresponding to specific inputs, leading to in-depth studies of LLMs' theory, capacity, and flaws. Researchers can also modify the model architecture and weights to validate the proposed algorithms to improve LLMs~\citep{Zhu2020Modifying,Cao2021Editing,Hase2021Belief,Mitchell2022Editing}.

With INT4 quantization, \glm can perform inference on popularized GPUs such as 4 $\times$ RTX 3090 or 8 $\times$ RTX 2080 Ti, which can be easily accessed from cloud service. As a result, researchers who cannot afford powerful data-center GPU servers like DGX-A100 can also utilize \glm.

\subsection{Impact on Individual Developers and Small Companies}
Individual developers and small companies who want to integrate LLMs into their business can only choose paid inference APIs. The increased cost can hinder their attempts. Instead, \glm can be deployed on popularized hardware that they own or can access via cloud service to reduce the cost. Furthermore, they can utilize distillation techniques~\citep{sanh2019distilbert,jiao2020tinybert} to obtain smaller models that preserve comparable performance on their specific tasks. While some developers may lack the ability to complete deployment and distillation on their own, we believe with \glm and more open LLMs in the future, the corresponding toolkits and service providers, will become more available.

We also note that most LLM applications are currently based on prompt engineering, partly due to the limitation of inference APIs. In downstream scenarios such as online customer service, the companies accumulate vast amounts of human-generated data that contain domain knowledge. With the open-source weights and code, developers can finetune \glm on their data to mitigate the gap in domain knowledge.

\section{Environmental Impact}
One of the major concerns about large language models is their huge energy usage and associated carbon emissions~\citep{Strubell2019Energy,Lacoste2019Carbon,Patterson2021Carbon,Bender2021Danger}. GPT-3 was estimated to use 500 tons of carbon emissions footprint (CO2eq)~\citep{Patterson2021Carbon}. We consumed a total of 442.4MWh of electricity over the 60-day course of training. Given the local power grid's 0.5810 kg/kWh carbon efficiency, the pre-training released 257.01 metric tons of CO$_2$. This is around half of GPT-3's carbon footprint, probably due to the efficient parallel strategies and NVIDIA's hardware improvements. The carbon emission is roughly the equivalent of the yearly emissions of 18 average Americans. However, we believe that with \glm released, more carbon emissions for reproducing 100B-scale LLMs can be saved.
\section{Environmental Impact}
One of the major concerns about large langauge models is their huge energy usage and associated carbon emissions~\cite{Strubell2019Energy,Lacoste2019Carbon,Patterson2021Carbon,Bender2021Danger}. GPT-3 was estimated to use 500 tons of carbon emissions footprint (CO2eq)~\cite{Patterson2021Carbon}. We consumed a total of 442.4MWh of electricity over the 60-day course of training. Given the 0.5810 kg/kWh carbon efficiency of local power grid, the pretraining released 257.01 metric tons of CO$_2$. This is around half of GPT-3's carbon footprint, probably due to the efficient parallel strategies and NVIDIA's hardware improvements. The carbon emission is roughly the equivalent of the yearly emissions of 18 average Americans. However, we believe that with \glm released, more carbon emissions for reproducing 100B-scale LLMs can be saved.
\section{A Brief History  of \glm}
\label{sec:appendix-history}

\begin{figure}[hbtp]
    \vspace{-10mm}
    \centering
    \includegraphics[width=\textwidth]{figures/glm130b-timeline-v2.pdf}
    \caption{The timeline of major issues that training \glm encountered and addressed, as of July 31st, 2022.}
    \label{fig:timeline}
\end{figure}

The \glm project\footnote{This section is largely extracted and updated from the blog introduction of \glm at  \url{http://keg.cs.tsinghua.edu.cn/glm-130b/} (Posted date: August 4, 2022).}
was conceived in Dec. 2021 in a brainstorming meeting at Tsinghua KEG.
We firmly believe that it is of value to pre-train a highly accurate language model, in particular for both Chinese and English.
Though GPT-3~\citep{brown2020language} is the pioneer for this effort, it is not available to most people in the world.
In addition, it supports English only.
We therefore decide to initialize the project \glm.
Please note that the WuDao 1.75T model we built last year is a sparse model with 480 mixture-of-experts (MoE), rather than a dense one as GPT-3.
Our goal then is to train a bilingual pre-trained dense model with high accuracy on downstream tasks, and to make it open to everyone in the world-anyone, anywhere can download it and use it on a single server with appropriate GPUs.

The ambitious project soon faced several important challenges:

\begin{itemize}[leftmargin=*,itemsep=0pt,parsep=0.2em,topsep=0.0em,partopsep=0.0em]
    \item \textbf{Lack of computational resources}: No organization is willing to sponsor such a big project and freely make it public.
    \item \textbf{Lack of a robust pre-training algorithm}: Despite GPT-3’s success on English corpus, it is unclear how to train a high-accurate bilingual model for both English and Chinese.
    \item \textbf{Lack of fast inference solutions}: Since the goal is to have the model public to everyone, we need to design fast inference solutions with low resource requirements to run the model.
\end{itemize}

For the pre-training algorithm, we finally chose GLM \citep{du2022glm} due to its high performance in practice.
We eventually decided to train a GLM model of 130 billion parameters after several rounds of discussions and exploration, because such a size makes it possible to run the inference on a single A100 (40G * 8) server.

Our first attempt at training the model was in January 2022, shortly after we received a small sponsor of GPUs for test running.
However, we soon realized that we had significantly underestimated the technical difficulties of pre-training a model at such a scale (>100B).
It seems that pre-training a highly accurate 100B-scale model is quite different from training a 10B-scale one. Due to frequent random hardware failures, model gradients exploding, unexpected excessive memory usage in the algorithm, debug for the 3D pipeline in the new Megatron and DeepSpeed frameworks, inability to recover from optimizer states, blocked TCP responses between processes, and many many unexpected ``bugs'', the project was delayed for many times.
The Tsinghua PACMAN team gave us a hand at this difficult time and together we successfully fixed most of the ``bugs''.

By March, we were still short on computational resources, but fortunately got a chance to try test runs on several other platforms, including Ascend 910, Hygon DCU, NVIDIA, and Sunway.
The immediate challenge was for us to adapt our training code to these different platforms, as the underlying operators are quite different.
Also, it introduced many new issues: the element-wise operators not supporting fast computation for large-dimension vectors, various issues that hindered convergence—the large gradient norms of input embeddings, native Post-LN, Pre-LN, and Sandwich-LN, dataloader state seeds, and computation precision choices in Softmax and Attention — as well as numerous mistakes we ourselves made.
With tremendous help from all of our generous partners, we finally succeeded in making our pre-training algorithms runnable across all the platforms—frankly, a surprising achievement for this project.
The timeline of \glm in Figure~\ref{fig:timeline} covers most of the issues we have encountered and addressed as of this writing.

On April 26th, we received a generous computing sponsorship from Zhipu.AI — an AI startup that aims to teach machines to think like humans.
After another week of testing, we finally kicked off the training of the \glm model on its 96 A100 (40G * 8) servers on May 6th.
Additionally, Zhipu.AI also sent a team to help evaluate the pre-trained model and build a demonstration website.

The training period spanned two months, during which we began developing a toolkit to allow \glm's inference in low-resource setting with swapping technique and quantization.
Though it is already the most accessible model of its scale,
together with our partner from Tsinghua NLP,
we have been exploring the limit of popularized hardware platforms, which would truly make the 100B-scale model accessible to as many people as possible.
To date, we managed to reach the INT4 weight quantization for \glm.
Importantly, the INT4 version of \glm without post training faces negligible performance degradation compared to its uncompressed original, while it consumes only 25\% of the GPU memory  required by the uncompressed version, thus supporting its effective inference on 4 $\times$ RTX 3090 Ti (24G) or 8 $\times$ RTX 2080 Ti (11G).
We will attempt to further reduce the resource requirements and keep the community updated on this important working item.
\section{Ethics: Evaluation on Biases and Toxicity} \label{app:ethics}
Albeit LLMs' strong abilities in language and beyond, which could bring substantial welfare to human beings, they can potentially produce toxic and illegal contents for evil use~\citep{weidinger2021ethical,Sheng2021SocietalBI,Dev2021HarmsOG,bommasani2021opportunities}.
In \glm, before granting model weight to applicants, in the model license, we demand them to agree that they will not use it for any deeds that may be harmful to society and human beings.

Additionally, from a technical perspective, we argue that we must also understand LLMs' toxic and biased behaviors and ultimately eliminate them.
This aligns with our commitment to ``LLM Inclusivity'', as it is necessary to include more people in the open-sourced LLM research to facilitate the process.
Moreover, suppose an LLM is good at identifying toxic and biased content. In that case, techniques such as self-diagnoses~\citep{schick2021self} can help to reduce the harmful generation in a self-consistent post-processing procedure.
Therefore, as an initial step, we evaluate \glm over various related benchmarks to shed light on the challenging topic.
Despite their limitations~\citep{blodgett2021stereotyping,jacobs2021measurement}, which should be addressed in future work, they still serve as a good start to arouse the community's awareness of the problem.

\subsection{Bias Measurement: CrowS-Pairs}

\input{tables/ethics/crows-pairs}
CrowS-Pairs~\citep{nangia2020crows}, or namely Crowdsourced Stereotype Pairs benchmark, is widely used for measuring biases for masked language models.
It collects 1508 examples with nine conventional biases and adopts a probing-based approach to compare the pseudo-log-likelihood of a pair of stereotypical and anti-stereotypical sentences.
Since \glm is pre-trained with autoregressive blanking infilling, CrowS-Pairs evaluation is directly applicable.
We compare the GPT-3 Davinci, and OPT-175B's results on CrowS-Pairs reported in~\citep{zhang2022opt} with \glm.

Our results are presented in Table~\ref{tab:crows-pairs}.
\glm shows fewer biases on almost all kinds of stereotypes except for religion and nationality.
We speculate that it is because \glm is a bilingual pre-trained LLM that learns semantics for certain content from both English and Chinese corpora.
Since CrowsS-Pairs' stereotypes mainly draw from the US Equal Employment Opportunities Commission's list\footnote{\url{https://www.eeoc.gov/prohibited-employment-policiespractices}},
the bias distributions in two different cultures and languages may be different and consequently reconcile social biases in \glm on a benchmark originally designed for English-language society.
We think this is an interesting finding, as multi-lingual pre-training may help LLMs to present less harmful biases for better fairness.
Finally, we also admit that \glm may, in turn, presents some special Chinese biases which currently lack testing benchmarks and require considerable future efforts to detect and prevent.

\subsection{Bias Measurement: StereoSet}
Another widely used bias and stereotype evaluation benchmark is StereoSet~\citep{nadeem2021stereoset}, which is also adopted in~\citep{lieber2021jurassic,artetxe2021efficient,zhang2022opt}.
To balance the evaluation between bias detecting and language modeling quality, StereoSet reports a series of metrics, including Language Modeling Scores (LMS), Stereotype Score (SS), and Idealized Context Association Test Score (ICAT) as an overall averaged metric.
For example, given the premise ``\textit{She is the twin's mother}'', StereoSet provides three candidate hypothesis: 1) ``\textit{the water is deep}'', 2) ``\textit{she is a lazy, unkind person}'', and 3) ``\textit{she is a kind, caring woman}''.
The first option servers as a distractor to test models' language capability and calculate LMS; the second and third statements are anti-stereotypical and stereotypical, respectively, and used for calculating SS.
A common practice here is to calibrate the likelihood of an option according to its length~\citep{lieber2021jurassic,zhang2022opt}, as the distractor term is particularly short.

Following~\citep{zhang2022opt}, we normalize scores over tokens rather than characters~\citep{lieber2021jurassic} to yield model predictions for calculating the metrics.
The results are shown in Table~\ref{tab:stereoset}.
As we observe, \glm exceedingly outperforms GPT-3 Davinci and OPT-175B on all metrics.
Such results align with our discoveries in language modeling experiments and CrowS-Pairs bias evaluation that \glm has a high quality in both language modeling and social fairness.

\input{tables/ethics/stereoset}

\subsection{Hate Speech Detection: ETHOS}
Social media corpus may contain hate speeches, and investigating to what extent LLMs know and can help to identify them is crucial.
We adopt the ETHOS dataset originally proposed in~\citep{mollas2020ethos} to detect sexism and racism speech on zero-shot or few-shot datasets created by~\citep{chiu2021detecting}.
GPT-3 Davinci (a public-accessible variant of GPT-3 175B) and OPT 175B are also tested on the benchmark (whose results are reported in~\citep{zhang2022opt}).
For binary classification including Zero-shot, One-shot, and Few-shot (binary) (which answers ``yes'' or ``no''), we report binary F1; for multiclass classification (which answers ``yes'', ``no'', or ``neither''), we report micro F1.
We adopt almost the same prompts as in~\citep{chiu2021detecting}, except aligning the Few-shot (binary) prompt to the form used in One-shot and adding the word ``\texttt{Classification}'' before the colon in the original Few-shot (multiclass) prompt.

\input{tables/ethics/ethos}

Results are shown in Table~\ref{tab:ethos}.
We find that \glm outperforms two other LLMs among four different settings.
On the one hand, \glm's pre-training over unsupervised diverse corpora from online forums and social media, including sections such as ``hackernews'', ``stackexchange'', and ``pile\_cc'' can endow our model with the background knowledge to identify those speeches.
On the other hand, the MIP training may also improve \glm's zero-shot and few-shot capabilities.

\subsection{Toxic Genearation: RealToxicPrompts}

\begin{wrapfigure}{r}{6cm}
    \small
    \vspace{-6mm}
    \centering
    \includegraphics[width=1.0\linewidth]{figures/rtp.pdf}
    \caption{RealToxicPrompts~\citep{gehman2020realtoxicityprompts} evaluation. Lower continuation toxicity probability is better.}
    \label{fig:rtp}
    \vspace{-6mm}
\end{wrapfigure}

Evaluating the generation toxicity by given prompts is essential to a model's safe deployment. We assess the toxic generation of \glm on the RealToxicPrompts~\citep{gehman2020realtoxicityprompts} dataset. Following its setting~\citep{zhang2022opt}, we use nucleus sampling ($p = 0.9$) to generate 25 continuations for each 10K random sampled prompts, limiting the maximum generated length to 128 tokens. Then we report the mean toxicity probabilities of 25 continuations evaluated by Perspective API\footnote{\url{https://www.perspectiveapi.com/}}. To make a fair comparison between different tokenization, we only report the toxicity score of the first complete sentence of a continuation as the score returned by the Perspective API seems to increase with sentence length~\citep{zhang2022opt}.

Results are shown in Figure~\ref{fig:rtp}. Generally, as the given prompt's toxicity increases, the continuation's toxicity probability increases accordingly in both models.
Compared to GPT-3 Davinci, \glm has a lower toxicity rate in all cases, indicating that \glm is less prone to generating harmful content.
We do not include results from~\citep{zhang2022opt}, as the API is updated and thus requires re-evaluation.

\part{Appendix}
\parttoc

\input{appendix/F_trajectory}
\input{appendix/S_ethics_and_environment}
\input{appendix/A_technical_details}
\input{appendix/B_dataset_details}
\input{appendix/D_broader_impact}
\input{appendix/C_contribution_details}

\end{document}

\section*{Acknowledgement}
This project is supported by the National Science Foundation for Distinguished Young Scholars (No. 61825602).
The GPU computation is sponsored by Zhipu.AI.
We thank all our collaborators and partners from
the Knowledge Engineering Group (KEG),
Parallel Architecture \& Compiler technology of Mobile, Accelerated, and Networked systems Group (PACMAN),
Natural Language Processing Group (THUNLP) at Tsinghua University,
and Zhipu.AI.

\section*{Ethical Statement}
We hereby acknowledge that all of the co-authors of this work are aware of the Code of Ethics and honor the code of conduct.
This work introduces an open-source Large Language Model (LLM), which could be used to generate synthetic text for harmful applications, such as telemarketing fraud, political propaganda, and personal harassment, as is discussed in~\citep{weidinger2021ethical,Sheng2021SocietalBI,Dev2021HarmsOG}.
We do not anticipate any hazardous outputs, especially towards vulnerable and historically disadvantaged groups of people, after using the model.

And to estimate and better collaborate with the community to prevent and ultimately eliminate the risks technically, we make the following crucial open efforts in this work:

\vvpara{Open-Sourced LLMs for Ethical Risk Studies.}
While some believe restricting access to LLMs can prevent such harmful applications, we argue that promoting the LLM inclusivity can lead to better defense against potential harms caused by LLMs.
There is no certain guarantee that organizations who can financially afford the cost of pre-training LLMs, such as big business corporations and governments, would never do harms with LLMs.
Without access to LLMs, we cannot even realize the potential role of LLMs in harms.

Thus, an open LLM can provide access and transparency to all researchers, and facilitate the research developments of reducing the potential harms of LLMs, such as algorithms to identify the synthetic text~\cite{gehrmann2019gltr}.
In addition, it is known that LLMs can suffer from problems in fairness, bias, privacy, and truthfulness~\cite{abs-2112-12938,lin2022truthfulqa,Liang2021SocialBias,Bender2021Danger}.
Thus, instead of providing APIs to black-box models, an open LLM can help reveal the model parameters and internal states corresponding to specific inputs.
In conclusion, an open LLM empowers us to conduct studies on LLMs' flaws in depth and to improve LLMs in terms of ethical concerns.

\vvpara{Ethical Evaluation and Improvements.}
We evaluate \glm on a wide range of ethical evaluation benchmarks, including bias measurement~\citep{nadeem2021stereoset,nangia2020crows}, hate speech detection~\citep{mollas2020ethos}, and toxic generation estimation~\citep{gehman2020realtoxicityprompts}.
Notwithstanding their deficiency~\citep{blodgett2021stereotyping,jacobs2021measurement}, these datasets serve as a meaningful and initial step towards an open quantitative evaluation of LLMs.

Our evaluation implies that the algorithm design choices of \glm, especially its bilingual pre-training, can significantly mitigate the biases and toxicity that an LLM may present while keeping its strong language performance compared to other LLMs~\citep{brown2020language,zhang2022opt} trained with monolingual English corpora (Cf. Appendix~\ref{app:ethics} for more details).

\section*{Reproducibility}
Different from most existing LLMs including GPT-3 175B\citep{brown2020language}, PaLM 540B~\citep{paperno2016lambada}, Gopher~\citep{rae2021scaling}, Chinchilla~\citep{hoffmann2022training}, LaMDA~\citep{thoppilan2022lamda}, FLAN~\citep{wei2022finetuned}, and many others, \glm is open-sourced and aims to promote openness and inclusivity in LLM research.

The reproducibility is ensured for \glm.
For the pre-training part, despite the expensive reproducibility cost it requires, we open source the code, training lessons, and the whole process of \glm's pre-training.
In addition, enabling \glm inference on popularized GPUs such as 3090/2080 Ti also aligns with the reproducibility undertaking, as it allows most researchers to reproduce \glm's results.
We are also providing free APIs for individual users to test \glm's performance.

\vvpara{Pre-Training.}
We provide the complete training notes, Tensorboard logs, and code for our pre-training in our repository (Cf. Abstract).
The pre-training hyper-parameters and cluster configuration are provided in Section~\ref{sec:parallel_strategy} and Table~\ref{tab:config}.
The training corpora composition and details for multi-task instruction pre-training are provided in Section~\ref{sec:training_objective} and Appendix~\ref{app:mip_description} and~\ref{app:mip_dataset}.

\vvpara{Evaluation.}
We organize all the evaluations, including language benchmarks (LAMBADA, Pile, MMLU, BIG-bench, CLUE, and FewCLUE) and ethical benchmarks (CrowS-Pairs, StereoSet, ETHOS, RealToxicPrompts), into one-command-to-run bash scripts in our code repository.
Data processing details for language modeling benchmarks are provided in Section~\ref{sec:language_modeling} and Appendix~\ref{app:pile}.
Details for MMLU are provided in Section~\ref{sec:mmlu} and Appendix~\ref{app:mmlu}, BIG-bench in Section~\ref{sec:big-bench} and Appendix~\ref{app:big-bench}, and CLUE and FewCLUE in ~\ref{sec:clue}.
For all ethical evaluations, please refer to Appendix~\ref{app:ethics} for details.