PaperlayerView exact revision on arXiv

Source-linked AI summary

Do Emergent Abilities Exist in Quantized Large Language Models: An Empirical Study

Peiyu Liu, Zikang Liu, Ze-Feng Gao, Dawei Gao, Wayne Xin Zhao, Yaliang Li, Bolin Ding, Ji-Rong Wen

arXiv:2307.08072v2cs.CLcs.AI

TL;DR

Low-bit quantization can reduce LLM memory use, but its effect on complex emergent abilities has not been deeply investigated. This study empirically evaluates those abilities across quantization levels and finds that 4-bit models retain them while 2-bit models degrade substantially.

  • Problem

    Prior quantized-LLM evaluations mainly analyze general performance, leaving limited investigation of complex-task abilities despite the need to understand performance at varied bit precisions.

    Source: §1 · p. 1

  • Method

    The authors empirically evaluate post-training quantization on in-context learning, chain-of-thought reasoning, and instruction following, and examine component sensitivity and fine-tuning as improvement approaches.

    Source: §1 · p. 2, §3.1 · p. 3, §6 · p. 10

  • Results

    4-bit and 8-bit quantization results are very similar to 16-bit performance across the tested abilities, while 2-bit results decline significantly toward near-random levels on several tests.

    Source: §3.2 · p. 4

  • Takeaways & Limitations

    Preserving crucial components, feature dimensions, and substructures, or applying parameter-efficient fine-tuning after quantization, can improve low-bit model performance with reduced computational resources.

    Source: §1 · p. 2, §6 · p. 10

    There is no consensus on whether emergent abilities exist, partly because empirical studies lack continuity in evaluation metrics and model sizes; the paper uses the term to emphasize LLMs’ superior performance.

    Source: §1 · p. 1

Abstract

from arXiv · show

Despite the superior performance, Large Language Models~(LLMs) require significant computational resources for deployment and use. To overcome this issue, quantization methods have been widely applied to reduce the memory footprint of LLMs as well as increasing the inference rate. However, a major challenge is that low-bit quantization methods often lead to performance degradation. It is important to understand how quantization impacts the capacity of LLMs. Different from previous studies focused on overall performance, this work aims to investigate the impact of quantization on \emph{emergent abilities}, which are important characteristics that distinguish LLMs from small language models. Specially, we examine the abilities of in-context learning, chain-of-thought reasoning, and instruction-following in quantized LLMs. Our empirical experiments show that these emergent abilities still exist in 4-bit quantization models, while 2-bit models encounter severe performance degradation on the test of these abilities. To improve the performance of low-bit models, we conduct two special experiments: (1) fine-gained impact analysis that studies which components (or substructures) are more sensitive to quantization, and (2) performance compensation through model fine-tuning. Our work derives a series of important findings to understand the impact of quantization on emergent abilities, and sheds lights on the possibilities of extremely low-bit quantization for LLMs.

1 Introduction

The paper asks whether post-training quantization preserves LLM emergent abilities and how low-bit performance can be improved. Across tested LLaMA sizes, 4-bit quantization retains these abilities better than 2-bit quantization, while component analysis and fine-tuning are explored as improvement paths.

1 Introduction

  • The study assesses in-context learning, chain-of-thought reasoning, and instruction following in LLaMA models to examine quantization’s effect on emergent abilities.The motivating question is whether these abilities persist when model size remains unchanged but precision is reduced.Source: §1 · p. 2
  • FFN plays a crucial role in retaining low-bit performance, while outlier dimensions affecting most Transformer layers concentrate mainly in FFN down projections.The authors identify these outlier dimensions as primarily responsible for quantization-performance decline.Source: §1 · p. 2
  • The authors propose fine-grained substructure quantization strategies to improve low-bit models, motivated by the identified component sensitivities.Their study also examines fine-tuning before and after quantization as a performance-enhancement approach.Source: §1 · p. 2
  • Parameter-efficient fine-tuning after quantization can tune a 2-bit LLaMA-65B model on one NVIDIA A100 and surpass a 16-bit LLaMA-13B model on zero-shot MMLU.The authors report this with significantly reduced computational resources.Source: §1 · p. 2

2 Background

The background introduces in-context learning, chain-of-thought reasoning, and instruction following as emergent abilities, then outlines post-training quantization and activation outliers. It also specifies the study’s selected abilities and the GPTQ method used in its experiments.

2 Background

  • Emergent abilities are described as special abilities associated with increasing model parameters and training corpus size; this study focuses on in-context learning, chain-of-thought reasoning, and instruction following.The paper notes that emergent abilities can be defined across different tasks or settings and selects these three because they are widely used for complex tasks.Source: §2 · pp. 2–3
  • In-context learning uses natural-language instructions and/or task demonstrations to guide test-example outputs without additional training or gradient updates..Source: §2 · p. 2
  • Chain-of-thought reasoning adds intermediate reasoning steps to demonstrations to elicit step-by-step problem solving.The paper describes it as a prompting strategy for tasks involving multiple reasoning steps.Source: §2 · p. 2
  • Instruction following is the ability to follow human instructions and complete a target task; the text notes that eliciting it often requires instruction tuning.Unlike in-context learning, it often includes no demonstrations.Source: §2 · pp. 2–3
  • Post-training quantization uses small calibration data to tune quantization parameters, and the experiments adopt GPTQ’s layerwise reconstruction loss to reduce the discrepancy between original and quantized weights.The passage says GPTQ supports lower-bit weight quantization and has promising 4-bit results on LLMs.Source: §2 · p. 3
  • Low-bit activation quantization is challenging because outlier dimensions have much higher feature-activation values than others and become more prominent as model scale increases..Source: §2 · p. 3

A.2 Case Study

This case study examines post-training quantization in LLaMA models across MMLU, GSM8K, and AutoEval, with results organized by model scale.

A.2 Case Study

Related papers

Loading 2307.08072v2…