PaperlayerView exact revision on arXiv

Source-linked AI summary

Memory-Efficient Fine-Tuning of Compressed Large Language Models via sub-4-bit Integer Quantization

Jeonghoon Kim, Jung Hyun Lee, Sungdong Kim, Joonsuk Park, Kang Min Yoo, Se Jung Kwon, Dongsoo Lee

arXiv:2305.14152v2cs.LGcs.AI

TL;DR

Large language model fine-tuning and deployment remain memory-intensive because PEFT does not shrink pretrained weights and quantization has mainly targeted deployment. PEQA addresses this gap by fine-tuning only quantization scales on quantized LLMs, and the paper reports restored or improved language-modeling, in-context-learning, and comprehension capabilities below 4-bit precision.

  • Problem

    PEFT reduces optimizer-state memory, but pretrained LLM weights remain large, while most quantization methods target deployment rather than fine-tuning.

  • Method

    PEQA fine-tunes only quantization scales while keeping the integer matrix frozen and shared across downstream tasks.

  • Results

    PEQA restored or improved language modeling, few-shot in-context learning, and comprehension relative to full-precision original performance even below 4-bit precision.

  • Takeaways & Limitations

    Quantized LLMs can retain comprehensive capabilities after PEQA while reducing model size and preserving low-bit inference structure.

  • Takeaways & Limitations

    The study focuses mainly on low-bit weight-only quantization in a linear asymmetric per-channel setting.

Abstract

from arXiv · show

Large language models (LLMs) face the challenges in fine-tuning and deployment due to their high memory demands and computational costs. While parameter-efficient fine-tuning (PEFT) methods aim to reduce the memory usage of the optimizer state during fine-tuning, the inherent size of pre-trained LLM weights continues to be a pressing concern. Even though quantization techniques are widely proposed to ease memory demands and accelerate LLM inference, most of these techniques are geared towards the deployment phase. To bridge this gap, this paper presents Parameter-Efficient and Quantization-aware Adaptation (PEQA) - a simple yet effective method that combines the advantages of PEFT with quantized LLMs. By updating solely the quantization scales, PEQA can be directly applied to quantized LLMs, ensuring seamless task transitions. Parallel to existing PEFT methods, PEQA significantly reduces the memory overhead associated with the optimizer state. Furthermore, it leverages the advantages of quantization to substantially reduce model sizes. Even after fine-tuning, the quantization structure of a PEQA-tuned LLM remains intact, allowing for accelerated inference on the deployment stage. We employ PEQA-tuning for task-specific adaptation on LLMs with up to 65 billion parameters. To assess the logical reasoning and language comprehension of PEQA-tuned LLMs, we fine-tune low-bit quantized LLMs using a instruction dataset. Our results show that even when LLMs are quantized to below 4-bit precision, their capabilities in language modeling, few-shot in-context learning, and comprehension can be resiliently restored to (or even improved over) their full-precision original performances with PEQA.

1 Introduction

Fine-tuning LLMs improves task alignment but remains constrained by the models’ memory demands. PEQA bridges PEFT and quantization by updating only quantization scales, preserving low-bit weights for efficient training and inference.

  • Existing PEFT methods reduce optimizer-state memory but leave the substantial memory demand of pretrained LLM weights unresolved.
  • Quantization reduces memory and accelerates inference, but QAT updates all parameters while PTQ primarily targets deployment and does not jointly solve training efficiency.
  • PEQA fine-tunes only quantization scales while keeping the integer matrix frozen, combining parameter-efficient adaptation with quantized LLMs.
  • PEQA reduces DRAM usage during training and deployment while enabling faster inference through fewer memory accesses.
  • Across instruction-tuning experiments, PEQA remained competitive with full-precision baselines for LLMs up to 65B despite reducing model size by 4 to 5 times.

2 Related Work

Related work covers LLM alignment, parameter-efficient fine-tuning, and neural-network quantization. Existing approaches reduce selected training costs or target deployment, motivating methods that combine PEFT with quantization for larger models.

  • Large Language Models and Alignment Learning: Instruction-tuning helps LLMs generalize instruction-following abilities across NLP tasks from natural-language task descriptions.
  • Parameter-Efficient Fine-Tuning: PEFT reduces the number of updated parameters and optimizer-state memory, but pretrained LLM weights still require substantial storage.
  • Parameter-Efficient Fine-Tuning: Earlier attempts to combine PEFT and quantization, including Quadapter and AlphaTuning, were primarily evaluated on models with 1.3B or fewer parameters.
  • Neural Network Quantization: QAT trains all network parameters to narrow the gap with full precision, making it impractical for LLMs, whereas PTQ occurs after training and cannot compress models during fine-tuning.
  • Figure 2 compares DRAM usage across tuning methods and perplexity against model size for LoRA and PEQA on Wikitext2.

3 Methodology

PEQA combines parameter-efficient fine-tuning with quantized LLMs by adapting quantization scales while retaining compressed integer weights. This reduces memory demands, supports task switching, and preserves quantization-based deployment benefits.

  • Benefits of PEQA Inherited from Bridging the Gap between PEFT and Quantization: Table 1 compares PEQA with alternative methods on LLaMA 65B across fine-tuning memory and time, deployment storage, inference acceleration, and task switching.The comparison includes PEFT, PEFT+PTQ, and PTQ+PEFT settings.
  • Parameter-Efficient and Quantization-aware Adaptation (PEQA): PEQA fine-tunes only quantization scales while freezing the integer quantization matrix, creating task-specific parameters for each downstream task.The frozen integer matrix is shared across tasks, while scale updates are adapted to each task.
  • Benefits of PEQA Inherited from Bridging the Gap between PEFT and Quantization: Table 2 evaluates 3-bit and 4-bit fine-tuned models on Wikitext2 using perplexity comparisons among QAT, PEFT+PTQ, and PEQA.The evaluated models are GPT-Neo 2.7B, GPT-J 6B, LLaMA 7B, and LLaMA 13B; lower perplexity is better.
  • Benefits of PEQA Inherited from Bridging the Gap between PEFT and Quantization: PEQA enables rapid task switching by swapping task-specific quantization scales while retaining shared quantized weights.Unlike PEFT followed by PTQ, PEQA preserves reversibility because it updates scales without applying non-reversible quantization operations after adaptation.
  • Benefits of PEQA Inherited from Bridging the Gap between PEFT and Quantization: Quantization reduces model size because PEQA stores frozen weights as a b-bit integer matrix rather than full-precision parameters.This extends PEFT’s optimizer-state savings to the model weights themselves.
  • Benefits of PEQA Inherited from Bridging the Gap between PEFT and Quantization: Quantized weights can accelerate token generation by reducing memory accesses and enabling kernels for quantized weights with half-precision activations.The method is designed to retain quantization-based inference acceleration after fine-tuning.

4 Experiments

Experiments evaluate PEQA’s quantization capability, task-specific adaptation, scalability, memory footprint, grouping strategy, and preservation of instruction-tuned abilities. Across these settings, PEQA remains competitive with full-precision or QAT baselines while retaining low-bit model advantages.

  • Comparing Quantization Capabilities: PEQA consistently outperforms LoRA+OPTQ across evaluated models at both 3-bit and 4-bit precision, while remaining competitive with QAT.QAT updates all parameters, whereas PEQA updates only quantization scales and freezes the integer matrix.
  • Task-specific Adaptation and Scalability: PEQA’s perplexity gradually approaches full-precision LoRA as model size grows, with only marginal degradation on Wikitext2 and PTB.The scalability study covers GPT-Neo, GPT-J, and LLaMA models up to 65B parameters.
  • Task-specific Adaptation and Scalability: Under restricted DRAM, PEQA enables larger LLaMA models than full-precision LoRA while preserving quantized deployment formats.The comparison concerns deployment-stage model operation under memory constraints.
  • Model Size and Number of Learnable Parameters: PEQA uses approximately 1.54 times fewer learnable parameters than LoRA QV4 for LLaMA models when scales are assigned per channel.Table 4 compares learnable parameters and model sizes for GPT-Neo, GPT-J, and LLaMA models using 3-bit or 4-bit channel-wise quantization.
  • Group-wise Quantization: Perplexity incrementally improves as PEQA incorporates more learnable parameters through smaller quantization groups.The grouping study evaluates PEQA-tuned LLaMA 7B and 13B models on Wikitext2.
  • Instruction-tuned Evaluation: LoRA- and PEQA-tuned LLaMAs maintain consistent common-sense reasoning trends, while PEQA instruction tuning restores MMLU performance toward full-precision levels.The evaluations use Alpaca instruction tuning, five common-sense reasoning tasks, and the MMLU benchmark.

5 Conclusion

The conclusion presents PEQA as a combination of parameter-efficient fine-tuning and quantization for aligning LLMs. It reports reduced fine-tuning DRAM use, faster deployment inference, and recovery of broad capabilities after low-bit quantization.

  • 5 Conclusion: PEQA combines parameter-efficient fine-tuning with quantization by updating quantization scales while retaining low-bit weights.The method is evaluated across datasets and LLMs, including models with substantial parameter-count reductions.
  • 5 Conclusion: PEQA reduces DRAM consumption during fine-tuning and accelerates deployment inference by keeping weights in low-bit quantized form.The conclusion attributes deployment efficiency to retaining the quantized weight format.
  • 5 Conclusion: Across task-specific adaptations, PEQA matches full-precision baselines and preserves or enhances comprehensive knowledge after instruction tuning.The reported recovery follows quantization-related compromises through updating only quantization scales.

B Experimental Settings of Section 4.1

Section 4.1 compares QAT, LoRA with PTQ, and PEQA using quantized LLM perplexity experiments. The settings use Wikitext2 with fixed sequence, batch, epoch, and optimizer configurations, alongside QAT-specific learning rates.

  • B Experimental Settings of Section 4.1: The experiment compares perplexity for QAT, LoRA with OPTQ, and PEQA on Wikitext2.The comparison uses quantized models and evaluates their fine-tuned language-modeling performance.
  • B Experimental Settings of Section 4.1: All Section 4.1 experiments use maximum sequence length 1024, batch size 128, and 15 training epochs.The passage also specifies the use of OPTQ for the PTQ condition.
  • B Experimental Settings of Section 4.1: QAT learning-rate choices are documented separately in Table 8.The table is identified as the source of QAT learning rates used in Table 2.

C Experimental settings of Table 3

Section 4.2 evaluates PEQA against LoRA and LoRA+OPTQ on Wikitext2 and PTB across models up to 65B parameters. The settings fix sequence, batch, epoch, and learning-rate configurations, with a shorter sequence for LLaMA 65B.

  • C Experimental settings of Table 3: The study compares PEQA, LoRA, and LoRA+OPTQ on Wikitext2 and PTB using models up to 65B parameters.LoRA uses the QV4 configuration and OPTQ supplies the PTQ baseline.
  • C Experimental settings of Table 3: Experiments use block size 1024 for models below 65B and 768 for LLaMA 65B because of memory limitations.Batch size and training epochs are set to 128 and 15, respectively.
  • C Experimental settings of Table 3: Learning rates for LoRA and PEQA on Wikitext2 and PTB are reported in Table 9.The table provides the learning-rate settings associated with the Table 3 experiments.

D The Perplexity of 3-bit and 4-bit PEQA on Wikitext2 Dataset

PEQA maintains competitive next-token prediction after 3-bit and 4-bit quantization, with 3-bit PEQA outperforming 3-bit post-training quantized LoRA on Wikitext2.

  • 3-bit PEQA achieves lower perplexity than 3-bit post-training quantized LoRA on Wikitext2.This supports continuity in model-size choices under DRAM constraints.
  • Figure 3 compares PEQA and LoRA+OPTQ perplexity against model size at 3-bit and 4-bit precision.

E OPT Models Adapted with PEQA and LoRA on Wikitext2 Dataset

On Wikitext2, PEQA and LoRA are compared across OPT models from 1.3B to 66B, with their perplexity gap narrowing as model size increases.

  • The perplexity gap between PEQA and LoRA becomes smaller as OPT model size increases.
  • Table 10 reports Wikitext2 perplexity for OPT models from 1.3B to 66B with weights quantized to 4-bit.Lower perplexity indicates better performance.
  • LoRA QV4 and QKVO16 configurations have little effect on Wikitext2 perplexity, so QV4 and QKVO16 are used in Sections 4.2 and 4.3.

G Experimental Settings of Multi-scale Performance

The multi-scale performance experiments use fixed sequence length, batch size, and epoch settings, with learning rates specified separately.

  • Experiments use maximum sequence length 1024, batch size 128, and 15 epochs.Learning rates are reported in Table 12.
  • Table 12 provides the learning rates used for the multi-scale performance experiments.

H Experimental Settings of Section 4.3

Section 4.3 evaluates instruction tuning and related quantization-scale choices using fixed experimental settings, Natural Instruction evaluation, AlphaTuning comparisons, and memory measurements.

  • The experiments use 4-bit quantization settings and learning-rate, epoch, and group-size configurations reported in Table 13, with AlphaTuning learning rates in Table 16.The authors did not exhaustively search learning rate or epoch hyperparameters because of limited time and resources.
  • LLaMA-1 experiments use truncated inputs, with tokenizer details specified for LLaMA-1 and LLaMA-2 model series.
  • Instruction-following evaluation uses 12 Natural Instruction subtasks with at most 200 instances per task, reporting zero-shot ROUGE-L for 4-bit PEQA and LoRA configurations.
  • PEQA uses one shared scale under uniform quantization, whereas AlphaTuning uses multiple scales but fine-tunes only one.
  • PEQA uses channel-wise trainable parameters and outperforms AlphaTuning by at least 0.7 PPL on Wikitext2 for GPT-Neo and OPT 1.3B.
  • PEQA results are also reported for LLaMA 7B and 13B on Wikitext2 with weights quantized to 4-bit.
  • On LLaMA-7B Wikitext2 training, PEQA peaks at 43GB versus 59GB for LoRA; for 65B, PEQA uses 33GB versus 130GB for full-precision LoRA.LoRA encounters out-of-memory errors at batch size 4, while PEQA continues training.
Loading 2305.14152v2…