PaperlayerView exact revision on arXiv

Source-linked AI summary

Alternating Multi-bit Quantization for Recurrent Neural Networks

Chen Xu, Jianqiang Yao, Zhouchen Lin, Wenwu Ou, Yuanbin Cao, Zhirong Wang, Hongbin Zha

arXiv:1802.00150v1cs.LGstat.ML

TL;DR

RNNs can be too large for portable devices and slow for latency-sensitive server workloads, while multi-bit quantization still faces a gap on challenging RNN tasks. The paper optimizes learned binary-code quantization with alternating minimization. On language models, it reports substantial memory savings and CPU inference acceleration at 2 and 3 bits, and also applies the method to image classification.

  • Problem

    RNN models can be too large for portable devices, and sequential inference creates latency; prior multi-bit RNN quantization still shows a full-precision gap on challenging large-input/output tasks.

    Source: §1 · pp. 1–2

  • Method

    The method formulates quantization of weights and activations into multiple binary codes {−1, +1} as an optimization problem and solves it by alternating minimization.

    Source: §1 · p. 2

  • Results

    2-bit quantization achieves ∼16× memory saving and ∼6× real inference acceleration on CPUs with reasonable accuracy loss; 3-bit quantization achieves ∼10.5× memory saving and ∼3× acceleration with almost no accuracy loss or better.

    Source: §1 · p. 2

  • Takeaways & Limitations

    The method also achieves plausible performance in image classification across RNNs and feedforward neural networks.

    Source: §1 · p. 2

    For CNN weight quantization, the refined approximation is still far from satisfactory in quantization accuracy.

    Source: §2 · p. 3

Abstract

from arXiv · show

Recurrent neural networks have achieved excellent performance in many applications. However, on portable devices with limited resources, the models are often too large to deploy. For applications on the server with large scale concurrent requests, the latency during inference can also be very critical for costly computing resources. In this work, we address these problems by quantizing the network, both weights and activations, into multiple binary codes {-1,+1}. We formulate the quantization as an optimization problem. Under the key observation that once the quantization coefficients are fixed the binary codes can be derived efficiently by binary search tree, alternating minimization is then applied. We test the quantization for two well-known RNNs, i.e., long short term memory (LSTM) and gated recurrent unit (GRU), on the language models. Compared with the full-precision counter part, by 2-bit quantization we can achieve ~16x memory saving and ~6x real inference acceleration on CPUs, with only a reasonable loss in the accuracy. By 3-bit quantization, we can achieve almost no loss in the accuracy or even surpass the original model, with ~10.5x memory saving and ~3x real inference acceleration. Both results beat the exiting quantization works with large margins. We extend our alternating quantization to image classification tasks. In both RNNs and feedforward neural networks, the method also achieves excellent performance.

1 INTRODUCTION

RNNs can be too large for portable devices and slow for latency-sensitive server workloads, while prior multi-bit quantization leaves a gap for challenging RNN tasks. This work learns multi-bit binary codes through an optimization and alternating-minimization approach, reporting substantial memory and CPU inference gains.

1 INTRODUCTION

  • RNNs’ high-dimensional inputs or outputs and deep recurrent networks can make models too large for portable devices, while sequential execution creates latency that limits concurrent server requests.Latency-sensitive server applications may therefore require more computing resources.Source: §1 · p. 1
  • The method quantizes all parameters into multiple binary codes {−1, +1}, rather than using only weight quantization.The work focuses on quantization among techniques that exploit network redundancy.Source: §1 · p. 2
  • Prior multi-bit RNN quantization remained challenging for large-input/output tasks, with noticeable gaps from full precision even at up to 4 bits.Existing work focused more on CNNs than RNNs.Source: §1 · p. 2
  • The quantization is formulated as an optimization problem, with learned binary codes that can be derived efficiently by a binary search tree once coefficients are fixed.This removes discrete unknowns that are difficult to handle.Source: §1 · p. 2
  • Alternating minimization separates binary codes from real coefficients so each subproblem can be solved efficiently when the other part is fixed.With proper initialization, two alternating cycles suffice for high-precision approximation, enabling online activation quantization.Source: §1 · p. 2
  • 2-bit quantization achieves ∼16× memory saving and ∼6× real inference acceleration on CPUs with reasonable accuracy loss; 3-bit quantization achieves ∼10.5× memory saving and ∼3× acceleration with almost no accuracy loss or better.The reported results beat existing quantization works with large margins.Source: §1 · p. 2

2 EXISTING MULTI-BIT QUANTIZATION METHODS

Existing multi-bit quantizers trade implementation simplicity against how well their partitions fit the data. The section contrasts uniform, balanced, greedy, and refined approaches with ternary and vector quantization alternatives.

2 EXISTING MULTI-BIT QUANTIZATION METHODS

  • For 2-bit quantization with α1 ≥ α2, the four codes are −α1−α2, −α1+α2, α1−α2, and α1+α2, separated by boundaries −α1, 0, and α1.Source: Figure 1 · p. 3
  • Uniform quantization is rule based and easy to implement; it can use cheap bit shift and count operations instead of costly multiplications and additions, but may be far from optimal for non-uniform data.Source: §2 · pp. 2–3
  • Balanced quantization equalizes data across intervals, but its affine mapping can remain suboptimal and evenly spaced partitions are not guaranteed to fit a given distribution.Source: §2 · p. 3
  • Greedy approximation sequentially minimizes quantization residue and is efficient, but may not reach high precision; refined greedy improves it yet remains unsatisfactory in CNN weight experiments.The refined result is reported as better than the original greedy approximation, but still far from satisfactory in quantization accuracy.Source: §2 · p. 3
  • Ternary quantization uses codes {−1, 0, +1}, is a 2-bit special case constrained by α1 = α2, and is described as lacking an efficient proposed algorithm.The cited approach empirically sets small-magnitude entries to zero and binarizes the remaining entries.Source: §2 · p. 3
  • Vector quantization learns a small codebook for weight reconstruction, reducing parameters by an order of magnitude with limited accuracy loss; product quantization could further compress multi-bit binary weights.Source: §2 · p. 4

3 OUR ALTERNATING MULTI-BIT QUANTIZATION

The method formulates multi-bit quantization around binary codes and real coefficients, using ordered-code search and alternating updates. With greedy initialization, two alternating cycles are reported as sufficient for high-precision approximation.

3 OUR ALTERNATING MULTI-BIT QUANTIZATION

  • For 2-bit quantization, the four codes are separated into nearest-code intervals by the midpoints −α1, 0, and α1, assuming α1 ≥ α2 ≥ 0.Source: §3 · p. 4
  • Recursively halving the ordered feasible codes lets the binary search tree find each optimal code in k comparisons rather than comparing against every boundary.The codes are ordered, and each comparison restricts the feasible set to one half.Source: §3 · pp. 4–5
  • Alternating minimization updates binary codes and real coefficients in separate subproblems; with greedy initialization, two cycles are reported sufficient for high-precision quantization.The coefficient update uses the least-squares solution.Source: §3 · p. 5
  • For w ∈ R^n and k-bit quantization, T alternating cycles require 2Tk^2n binary operations and 2(T+1)kn non-binary operations, including greedy initialization.Source: §3 · p. 5

4 APPLY ALTERNATING MULTI-BIT QUANTIZATION TO RNNS

The method quantizes RNN weights and activations into multiple binary codes, with row-wise weight quantization and online activation quantization. Training alternates full-precision weight updates with quantization and uses a straight-through estimate for gradients.

4 APPLY ALTERNATING MULTI-BIT QUANTIZATION TO RNNS

  • Weights are quantized row by row with scaling coefficients, while input-dependent hidden states are quantized online as activations.One-hot input tokens select a row of the quantized input matrix and therefore need no quantization.Source: §4 · p. 5
  • Reformulating binary matrix-vector multiplication enables parallel execution through binary matrix multiplication, with partitioned outputs summed afterward.The row-wise representation adds little computation while allowing more freedom in approximating weights.Source: Figure 3 · p. 6, §4 · p. 6
  • Against full-precision MKL, CPU inference reaches roughly 6× acceleration at (kh, kw) = (2, 2) and 3× at (3, 3).For an LSTM with hidden size 1024, the corresponding theoretical acceleration ratios are roughly 7.5× and 3.5×.Source: §4 · p. 6
  • Training retains full-precision weights to accumulate gradients, quantizes them each minibatch, uses straight-through gradients, and clips weights to [−1, 1].The same straight-through estimate is used for derivatives on the quantized hidden state.Source: §4 · pp. 6–7

5 EXPERIMENTS ON THE LANGUAGE MODELS

Alternating quantization performs strongly across LSTM and GRU language-model experiments, including comparisons with full-precision models and other quantizers. Results cover PTB, WikiText-2, Text8, and image classification.

5 EXPERIMENTS ON THE LANGUAGE MODELS

  • Directly quantizing pretrained weights without activation quantization or retraining, Alternating yields lower approximation error across bit widths and corresponding lower testing PPW.The comparison uses LSTM and GRU on PTB; rule-based Uniform and Balanced can perform substantially worse in this setting.Source: §5 · pp. 7–8
  • On PTB, Alternating matches Refined with one fewer weight or activation bit, while 3/3-bit LSTM outperforms the full-precision model.The authors suggest quantization-induced regularization as a possible explanation for the latter result.Source: Table 3 · p. 8, §5 · p. 8
  • On WikiText-2, Alternating attains compatible or lower PPW than Refined using one fewer quantization bit.The experiment uses a 512-unit hidden state and batch size 100.Source: Table 4 · p. 8, §5 · p. 8
  • On Text8, 2-bit Alternating LSTM exceeds Refined at 3 bits, while 3-bit GRU remains below full precision under shared hyperparameters.The authors note that dataset-specific hyperparameter tuning may close the GRU gap.Source: Table 5 · p. 8, §5 · p. 9
  • On image classification, Alternating achieves the lowest testing error among compared methods for both RNNs and feedforward networks.The results are deferred to Appendix B.Source: §5 · p. 9

6 CONCLUSIONS

The paper uses alternating quantization to reduce RNN memory and latency while retaining language-model accuracy. It reports competitive or better accuracy at 3 bits, substantial memory savings and CPU acceleration, and strong image-classification results.

6 CONCLUSIONS

  • Alternating quantization formulates approximation-error minimization as an optimization problem and separates parameters for efficient alternating updates.The method is applied to weights and activations in LSTM and GRU language models.Source: §6 · p. 9
  • At 2 bits, weights and activations yield about 16× memory reduction and 6× CPU acceleration with reasonable accuracy loss; at 3 bits, savings are about 10.5× and acceleration about 3× with compatible or better accuracy.The authors report that both settings outperform existing works by a large margin.Source: §6 · p. 9
  • In image classification, the method also achieves plausible performance in both RNNs and feedforward neural networks.Source: §6 · p. 9

A BINARY MATRIX VECTOR MULTIPLICATION IN CPUS

CPU tests compare binary matrix-vector multiplication with full-precision MKL and report substantial speedups at 2- and 3-bit quantization.

A BINARY MATRIX VECTOR MULTIPLICATION IN CPUS

  • Binary multiplication achieves roughly 6× acceleration with 2-bit quantization and 3× with 3-bit quantization versus full precision.The timing table includes the cost of running alternating quantization online.Source: Table 6 · p. 12, A BINARY MATRIX VECTOR MULTIPLICATION… · p. 12
  • The kernel uses entry-wise XNOR for products and bit counting for accumulation, tested with SIMD XNOR and popcount instructions.Tests use single-thread CPU execution and compare against optimized Intel MKL matrix-vector multiplication.Source: A BINARY MATRIX VECTOR MULTIPLICATION… · p. 12
  • Online alternating quantization accounts for a small portion of execution time, especially for the larger matrix-vector test; the authors also note possible GPU, ASIC, and FPGA extensions.The CPU measurements are described as a simple test.Source: A BINARY MATRIX VECTOR MULTIPLICATION… · p. 12

B IMAGE CLASSIFICATION

The section extends alternating quantization to image classification: it reports plausible performance for sequential MNIST with an LSTM and the lowest testing error among compared quantization methods for an MNIST MLP and CIFAR-10 CNN.

Sequential MNIST

  • On sequential MNIST, the LSTM quantizes inputs as well as weights and activations and still achieves plausible performance.Each image row is supplied sequentially over 28 time steps.Source: B IMAGE CLASSIFICATION · p. 12, Table 7 · p. 13

MLP on MNIST

CNN on CIFAR-10

Related papers

Loading 1802.00150v1…