PaperlayerView exact revision on arXiv

Source-linked AI summary

Memory-Efficient Integer-Only Training of Deep Neural Networks

Hazem Hesham Yousef Shalby, Fabrizio Pittorino, Manuel Roveri

arXiv:2610.16298v1cs.AI

TL;DR

Training methods that scale to standard computer-vision benchmarks still retain floating-point computation in at least one stage. This paper presents an integer-only training framework based on mantissa–exponent tensors and reports competitive accuracy with substantially reduced training memory. Its current optimizer is limited to SGD with momentum.

  • Problem

    Methods that scale to standard computer-vision benchmarks retain floating-point computation in at least one training stage, despite the cost of persistent floating-point state and the absence of floating-point support on some platforms.

    Source: §1 · pp. 1–2

  • Method

    The framework stores training tensors as INT8 mantissas with power-of-two exponents and uses integer operations throughout forward, backward, and SGD-with-momentum updates.

    Source: §1 · p. 2, §3.1 · p. 4

  • Results

    Across tested configurations, every persistent training tensor is INT8, training memory is reduced 4× versus FP32, and accuracy stays within a few percent of FP32.

    Source: §4 · p. 7, §5 · p. 9

  • Takeaways & Limitations

    The framework demonstrates integer-only training of modern CNNs and Vision Transformers on standard computer-vision benchmarks at roughly one-quarter the FP32 training-memory footprint.

    Source: §5 · p. 9

    The framework currently supports SGD with momentum rather than adaptive optimizers such as Adam or AdamW.

    Source: §4 · p. 9, §5 · p. 9

Abstract

from arXiv · show

Training neural networks in reduced precision can yield significant efficiency gains, yet all existing methods that scale to standard computer vision benchmarks retain floating-point in at least one stage, typically the weight update or optimizer state. We present a training algorithm that scales to modern deep neural networks, in which every tensor (weights, gradients, and activations) is stored as INT8, and every operation (forward pass, backward pass, weight update, optimizer update) is performed in integer arithmetic. Specifically, tensors are stored as signed integer mantissas paired with explicit power-of-two exponents, on which integer-only operations are performed using five custom composable building blocks (products, sums, divide-by-constant, nonlinear functions, and weight updates). This allows complete implementation of trainable integer layers, including convolutions, batch and layer normalization, softmax, and GELU. A per-group velocity mechanism with independent exponent tracking preserves gradient precision across neurons and channels with diverse magnitudes. As every tensor fits in INT8, training memory is roughly 4x smaller than the FP32 baseline. We validate our training algorithm on CNNs (ResNet-18/34/50/101, MobileNetV2) and Vision Transformers (ViT-Ti / B), demonstrating for the first time that training neural networks with INT8 tensors and integer arithmetic can achieve competitive accuracy on CIFAR-10, CIFAR-100 and ImageNet at a reduced memory budget.

1 Introduction

Lower precision can reduce the growing cost of training, but prior approaches that scale to standard vision benchmarks retain floating-point arithmetic in at least one critical stage. The paper proposes an integer-only framework designed to address this gap and preserve precision across differently scaled gradients.

1 Introduction

  • Prior approaches that scale to standard computer-vision benchmarks retain floating-point arithmetic in a critical component such as weight updates, optimizer state, or nonlinear functions.Earlier fully integer approaches either omit standard optimization mechanisms such as momentum and learning-rate schedules or remain restricted to small models and tasks.Source: §1 · pp. 1–2
  • Floating-point training state uses 12 bytes per parameter, constraining model and batch size, while some integer-only platforms lack floating-point support entirely.The paper identifies eliminating floating-point operations as a prerequisite for on-device learning in that setting.Source: §1 · p. 1
  • The framework represents tensors as signed integer mantissas with explicit power-of-two exponents and composes five primitives into integer-only layer operations.The primitives are products, sums, divide-by-constant, nonlinear functions, and weight updates.Source: §1 · p. 2
  • Per-group velocity tracks separate exponents across parameter groups to preserve gradient precision when magnitudes vary within a weight tensor.The mechanism supports integer-only SGD with momentum.Source: §1 · p. 2
  • The framework spans forward propagation, backpropagation, and parameter updates, and supports standard CNN and ViT layers including softmax, GELU, and batch and layer normalization.The authors describe it as the first such framework, to their knowledge, to scale to modern networks and standard computer-vision benchmarks.Source: §1 · p. 2

2 Related Work

Prior low-precision methods differ in how much of training remains floating-point and which model stages they support. The paper positions its work against methods that quantize parts or all of the training pipeline but retain floating-point components or have restricted scope.

2 Related Work

  • The related-work comparison is organized by how far each approach eliminates floating-point from training, with separate summaries of method support and per-tensor storage formats.Table 1's legend distinguishes fully integer/supported, floating-point/not supported, and partially integer cases.Source: §2 · p. 2, Table 1 · p. 3, Table 2 · p. 3

Low-precision training

  • Mixed-precision and quantization-aware methods reduce precision in parts of training but retain floating-point master weights, backward computation, or weight updates.FP16 and FP8 methods reduce the datapath; integer-arithmetic inference methods do not establish integer-only training.Source: §2 · p. 2

Integer forward and backward passes

  • Integer forward/backward methods quantize weights, activations, and gradients to varying degrees, but leave the optimizer in FP32.Some methods retain higher precision for final weight-gradient accumulation.Source: §2 · p. 2

Quasi-integer training

  • Quasi-integer methods vary in scope: some retain floating-point state or operations, while NITI uses integer forward, backward, and updates but lacks momentum and learning-rate scheduling and scales only to AlexNet.NITI's shared exponent can also cause small-gradient channels to lose effective integer range; the paper identifies per-group velocity as addressing that failure mode.Source: §2 · p. 2

Integer-only nonlinear functions

  • Integer-only nonlinear-function methods cited here address inference, whereas this paper uses lookup tables for its training framework's nonlinear functions.The paper says its tables require no significant memory overhead and no polynomial approximation or floating-point fallback.Source: §2 · pp. 2–3

3 Method

The method stores tensors as integer mantissa–exponent pairs and implements layer computations and SGD updates using integer operations. Per-group velocity and reactive weight rescaling address distinct scale-alignment problems in the optimizer.

3.1 Mantissa-exponent tensor representation

  • Each tensor is stored as a signed INT8 mantissa and shared exponent, representing the real-valued tensor as C · 2^e.Power-of-two scale changes can be handled by exact shifts of the exponent.Source: §3.1 · p. 4
  • Layer outputs accumulate in INT32 before compaction back to INT8, and right shifts during training use stochastic rounding.The paper reports 89.3% for deterministic rounding versus 94.7% with stochastic rounding on ResNet-18/CIFAR-10, a 5.4% gap.Source: §3.1 · p. 4

3.2 Layer building blocks

  • Products, sums, divide-by-constant, and nonlinear lookup tables provide integer building blocks for composing standard network layers.The described compositions include convolutions, linear layers, batch and layer normalization, ViT attention, and the output head.Source: §3.2 · p. 5

3.2.1 Products; 3.2.2 Sums

  • Products multiply mantissas in an INT32 accumulator and add exponents; sums align tensor exponents before accumulating and compacting the result.The sum procedure uses stochastic rounding when a right shift is needed to align tensors with a large exponent difference.Source: §3.2.1 · p. 5, §3.2.2 · p. 5

3.2.3 Divide-by-constant

  • Division by a power-of-two constant is absorbed into the exponent, while general integer divisors use a precomputed reciprocal, multiplication, and shift.The method describes this for average pooling and normalization statistics.Source: §3.2.3 · p. 5

3.2.4 Nonlinear lookup tables

  • Nonlinear functions use finite lookup tables initialized once and evaluated through indexed lookup; softmax and GELU have described integer forward and backward paths.Softmax uses an INT8 exponential lookup and integer reciprocal normalization, while GELU uses forward and derivative lookup tables.Source: §3.2.4 · pp. 5–6

3.3 Integer-only weight update with scale alignment; 3.3.1 Per-group velocity

  • A shared velocity exponent can cause large-gradient channels to dominate the dynamic range, so the method assigns separate exponents to velocity slices along output neurons or channels.Each gradient slice is compacted separately, allowing each velocity and gradient slice to use the INT8 range; the added storage is below 0.1% for a typical fully connected or convolution layer.Source: §3.3 · p. 6, §3.3.1 · p. 6

3.3.2 Update algorithm; 3.3.3 Reactive rescaling

  • The update applies momentum decay, aligns grouped gradients to velocity exponents, accumulates and compacts velocity, then aligns the update to the weight exponent before subtraction.After groupwise updates, reactive rescaling shifts the weight mantissa to address overflow or under-utilization; right-shift rescaling rounds while left shifts are exact.Source: §3.3.2 · p. 6, §3.3.3 · p. 7

4 Experiments

Experiments span CNNs and Vision Transformers across CIFAR-10, CIFAR-100, and ImageNet. The results compare accuracy and training memory, and probe lower-bit operation and per-group velocity.

Setup

  • Evaluations cover ResNets on three datasets, MobileNetV2 and ViT-Ti/B on CIFAR-10, using SGD with momentum and cosine learning-rate schedules.ViT-Ti is trained from scratch and ViT-B is fine-tuned from a pretrained checkpoint.Source: §4 · p. 7

CNN results

  • On ResNet-18, FP32 accuracy gaps are 0.6%, 0.5%, and 1.2% on CIFAR-10, CIFAR-100, and ImageNet; ResNet-50/ImageNet reaches 76.4% versus FP32’s 76.7%.The ResNet-50 result is within 1.2% of the best INT8 forward/backward result, 77.6%, while using 5.20 versus 20.82 GB.Source: §4 · p. 7
  • MobileNetV2/CIFAR-10 trails FP32 by 3.2%, with small, quantization-sensitive channel gradients and sequential requantization identified as contributing factors.Per-output-channel velocity contributes +2.2%, but the architecture remains challenging.Source: §4 · pp. 7–8

Main results

  • 4× lower training memory across configurations, while accuracy stays within a few percent of FP32 and within 0.1–0.7% of Ghaffari et al. [2022].Table 3 reports accuracy and memory across the benchmarks; Figure 2 plots their accuracy–memory tradeoffs.Source: §4 · p. 7, Figure 2 · p. 7, Table 3 · p. 8

Vision Transformers

  • ViT-Ti/CIFAR-10 reaches 88.0%, 2.9% below FP32, and ViT-B fine-tuning reaches 96.1%, 3.0% below FP32.The authors report these as the first fully integer-only Vision Transformer training or fine-tuning results.Source: §4 · p. 8
  • The framework currently supports SGD with momentum, not the Adam/AdamW optimizers typically used for Vision Transformers.The authors note that adaptive-optimizer moment updates would require integer square roots and divisions that complicate scale bookkeeping.Source: §4 · p. 9

Lower-bit regimes

  • At B=4, the method retains 70.5% accuracy on ResNet-18/CIFAR-10, while Ghaffari et al. [2022] diverges; its memory falls from 0.34 to 0.17 GB as bit-width decreases.The method retains INT8 velocity, while Ghaffari et al. [2022] retain FP32 master weights and INT16 velocity.Source: §4 · p. 9

Per-group velocity ablation

  • Disabling per-group velocity costs 0.9% on ResNet-18, 2.2% on MobileNetV2, and 2.7% on ViT-Ti.Table 5 reports the test-accuracy effect of the ablation.Source: Table 5 · p. 9, §4 · p. 9

5 Conclusion

The paper demonstrates end-to-end INT8 mantissa-exponent training with SGD and momentum, without floating-point fallback, across modern vision architectures. It reports substantially lower training memory and identifies adaptive optimizers and other extensions as future work.

5 Conclusion

  • Across CIFAR-10, CIFAR-100, and ImageNet, ResNets and MobileNetV2 train within a few percent of FP32 with roughly 4× smaller memory.The framework also trains ViT-Ti from scratch and fine-tunes ViT-B using integer-only operations.Source: §5 · p. 9
  • Per-output-channel velocity exponents are essential for maintaining integer optimizer state in the demonstrated training pipeline.The conclusion reports that every persistent tensor is INT8 and that the pipeline uses SGD with momentum.Source: §5 · p. 9
  • Adaptive optimizers, richer scaling schemes, operator fusion, and LLM training or LoRA remain proposed extensions.The paper identifies integer square roots, division, and exponent bookkeeping for bias correction as challenges for adaptive optimizers.Source: §5 · pp. 9–10

A Integer primitives, shifts, and safety caps

The appendix describes integer primitives for rescaling and stochastic rounding, plus accumulator safety caps that constrain shifts before reductions.

A Integer primitives, shifts, and safety caps

  • Saturation and signed shift-with-round are the two primitives from which the paper builds every rescale.Training uses stochastic rounding, inference uses a deterministic midpoint, and left shifts are exact.Source: A Integer primitives, shifts, and… · p. 12
  • Stochastic rounding samples an integer per tensor element; its PRNG adds well under 1% to per-layer arithmetic and is eliminated at inference.The text gives inner-product widths from 9 to 4608 and describes integer xorshift or PCG generation.Source: A Integer primitives, shifts, and… · p. 12
  • Accumulator safety caps bound compaction shifts to prevent overflow in K-wide inner products and M-way sums.If a cap is negative, the reduction must be tiled, operands pre-shifted, or a wider accumulator used; the reported INT32 caps stay positive for model layers.Source: A Integer primitives, shifts, and… · p. 12

B Concrete layer instantiations

Concrete layer operators are assembled by composing reusable integer primitives, with residual and pooling operations reusing existing operators.

B Concrete layer instantiations

B.1 Activations

The models use integer ReLU, ReLU6, and GELU operations; activation exponents are carried through unchanged, and GELU uses lookup tables.

B.1 Activations

  • ResNet-18, MobileNetV2, and ViT-Ti/B use ReLU, ReLU6, and GELU respectively; only GELU requires a lookup table.The activation operations leave exponent e unchanged.Source: §B.1 · p. 13, Table 7 · p. 13
  • ReLU6 clips using the per-exponent cap c = min(6 ≫ e, qmax), requiring no rounding for the deterministic threshold.When the cap reaches qmax, the clip reduces to ReLU.Source: §B.1 · p. 13
  • GELU uses paired INT8 lookup tables, with BG=6 keeping its derivative values within the INT8 range.The passage reports no measurable backward-accuracy change versus autograd from reducing BG from 8 to 6.Source: §B.1 · p. 13
  • For B < 8, one INT8 GELU table serves all bit widths by shifting inputs into INT8 and stochastically shifting results back.Source: §B.1 · p. 13

B.2 Batch and layer normalization

Batch and layer normalization share an integer pipeline, differing only in whether they reduce across the batch or feature dimension. The output head and ViT attention likewise compose integer operations for forward and backward computation.

B.2 Batch and layer normalization

  • Batch and layer normalization differ only in reduction axis and compute normalization and affine transforms with integer operations, including a square root and an INT32 divide per reduction row.A shared reciprocal table supports division by the odd part of the reduction size; running mean and variance remain integer mantissa-exponent pairs updated by an exponential moving average.Source: §B.2 · p. 14

B.3 Output head: softmax and cross-entropy

  • The output head produces unsigned INT8 softmax probabilities, a logging-only loss, and a loss gradient (p−onehot)/N, with other operations using integer primitives and shared INT8 tables.The cached probabilities are reused for the backward pass; the loss value is for reporting only.Source: §B.3 · pp. 14–15, Table 8 · p. 15

B.4 Vision Transformer block

  • ViT blocks use LayerNorm and integer GELU, with residual additions folded into the sum primitive.The block applies attention and then an MLP to normalized inputs, adding each result through the integer sum primitive.Source: §B.4 · p. 15
  • ViT attention forms Q, K, and V with integer projections, scales QKᵀ by a rational shift encoding 1/√dh, and applies row-wise integer softmax before computing pV.The forward pass caches p and V for backward; the reverse pass uses integer product, sum, and shift-round operations to compute projection gradients and merge input-gradient streams.Source: §B.4 · pp. 15–16

C Training memory computation

The appendix computes total training memory from persistent state and saved activations, using per-method storage widths and benchmark configuration inputs. It also distinguishes reduced-precision arithmetic from reduced-precision tensor storage in reported comparison methods.

C Training memory computation

  • The comparison counts stored tensors, not just arithmetic width: cited GPU-emulated INT8 methods carry inter-layer and saved activations as FP32 in the implementations producing reported accuracy.The appendix also notes FP8 inputs to GEMMs while outputs and saved activations remain FP16/BF16.Source: C Training memory computation · p. 16
  • Total training memory is Mtrain = bparam · P + bact · A · N, where the FP32 baseline uses 12 persistent-state bytes per parameter and 4 activation bytes per element.P is parameter count, A is saved-for-backward activations per sample, and N is batch size; Table 9 lists these configuration inputs.Source: C Training memory computation · p. 17, Table 9 · p. 17
  • The appendix says applying each method’s storage widths and configuration values to the formula reproduces Table 3’s training-memory entries.Persistent state is summed from weights, gradients, and velocity widths, while activation storage scales with batch size and input resolution.Source: C Training memory computation · pp. 16–17

D Training recipe and hyperparameters

The appendix records the implementation environment and training recipe, including boundary input quantization, activation quantization, and integer forms of optimization parameters. It also identifies the hyperparameter table used for the benchmark runs.

D Training recipe and hyperparameters

  • Inputs are normalized in float then quantized once to INT8 at the framework boundary, while activations are requantized between layers using a shift derived from their range.ViT-B clips activations at the 99.99th percentile before computing the shift; CNNs and ViT-Ti do not require this clipping at tested bit widths.Source: D Training recipe and hyperparameters · p. 17
  • In the reported setup, CNN biases are omitted because batch normalization absorbs them, whereas ViT-B retains biases in its Q/K/V and MLP projections.ViT-B biases are stored as integer mantissa-exponent pairs and aligned to the output exponent before addition.Source: D Training recipe and hyperparameters · p. 18
  • Momentum and learning rates are represented as integer numerator-and-shift pairs, while L2 regularization is added to the per-group gradient before momentum accumulation.The learning-rate schedule is precomputed per iteration, and weight decay is not applied as a separate weight-update step.Source: D Training recipe and hyperparameters · p. 18
  • All runs use SGD with momentum and cosine learning rate with linear warmup, and were performed on one NVIDIA A100-SXM4 GPU.The pipeline is implemented on PyTorch 2.10 with CUDA 12.8 and cuDNN 9.1.Source: D Training recipe and hyperparameters · p. 17, Table 10 · p. 18

E Additional experiments

The additional experiments attribute ViT-B’s pre-finetuning accuracy gap to linear projections rather than the integer normalization, activation, and softmax operators. They also show progressive representational divergence before fine-tuning and report stable INT8 training dynamics for ResNet-18 on CIFAR-10.

E Additional experiments

  • INT8 ResNet-18 training tracks the FP32 baseline on CIFAR-10 over 200 epochs, with step-learning-rate drops producing discrete transitions without reported instability or divergence.Figure 3 covers the 200-epoch run with step learning-rate changes at epochs 60, 120, and 180.Source: E Additional experiments · p. 18, Figure 3 · p. 19
  • 3.2% is the ViT-B pre-finetuning gap attributed to INT8 Linear projections; moving LayerNorm, GELU, and softmax into the integer pipeline adds no further cost overall.The reported operator changes are LayerNorm −0.1, GELU +0.1, and softmax +0.2.Source: E Additional experiments · p. 18, Table 11 · p. 19
  • Cosine similarity to the float model falls from 0.977 after patch embedding to below 0.01 after block 5 in the INT8 ViT without fine-tuning.The cited analysis locates the decline inside attention sublayers and attributes it to compounding quantization error in four Linear projections per block; fine-tuning closes the gap as weights adapt to the INT8 quantizer.Source: E Additional experiments · p. 19

Related papers

Loading 2610.16298v1…