PaperlayerView exact revision on arXiv

Source-linked AI summary

L1-Norm Batch Normalization for Efficient Training of Deep Neural Networks

Shuang Wu, Guoqi Li, Lei Deng, Liu Liu, Yuan Xie, Luping Shi

arXiv:1802.09769v1cs.LG

TL;DR

BN's square and root operations add computational cost and impede low-precision training. L1BN replaces them with absolute and signum operations; tests report comparable accuracy and convergence to L2BN, alongside FPGA operation-level gains.

  • Problem

    BN's nonlinear square and root operations impede low-bit-width quantization, while conventional L2-norm BN carries costly operations.

    Source: §V · p. 7

  • Method

    L1BN replaces each neuron's mini-batch L2-norm variance with an L1-norm variance and uses absolute and signum operations in backpropagation.

    Source: §I · p. 1, §V · p. 7

  • Results

    Across CNNs and GANs, L1BN achieves comparable accuracy and convergence speed to L2BN with higher computational efficiency; FPGA operations yield 1.5× speedup and 50% power saving, respectively.

    Source: §I · p. 2, §V · p. 7

  • Takeaways & Limitations

    By replacing square and root operations with absolute and signum operations, L1BN promises fully quantized neural networks with low-precision dataflow for efficient online training.

    Source: §V · p. 7

Abstract

from arXiv · show

Batch Normalization (BN) has been proven to be quite effective at accelerating and improving the training of deep neural networks (DNNs). However, BN brings additional computation, consumes more memory and generally slows down the training process by a large margin, which aggravates the training effort. Furthermore, the nonlinear square and root operations in BN also impede the low bit-width quantization techniques, which draws much attention in deep learning hardware community. In this work, we propose an L1-norm BN (L1BN) with only linear operations in both the forward and the backward propagations during training. L1BN is shown to be approximately equivalent to the original L2-norm BN (L2BN) by multiplying a scaling factor. Experiments on various convolutional neural networks (CNNs) and generative adversarial networks (GANs) reveal that L1BN maintains almost the same accuracies and convergence rates compared to L2BN but with higher computational efficiency. On FPGA platform, the proposed signum and absolute operations in L1BN can achieve 1.5$\times$ speedup and save 50\% power consumption, compared with the original costly square and root operations, respectively. This hardware-friendly normalization method not only surpasses L2BN in speed, but also simplify the hardware design of ASIC accelerators with higher energy efficiency. Last but not the least, L1BN promises a fully quantized training of DNNs, which is crucial to future adaptive terminal devices.

I. INTRODUCTION

Mini-batch distribution shifts can destabilize DNN activations and gradients, while batch normalization helps training but adds computational and quantization burdens. L1BN replaces L2-based variance with an L1-based measure and reports comparable model performance with greater computational efficiency.

I. INTRODUCTION

  • Mini-batch distribution shifts can propagate through layers until network outputs and gradients vanish or explode, slowing convergence and increasing tuning demands.The passage identifies this as internal covariate shift.Source: §I · p. 1
  • BN facilitates convergence and improves final accuracy, but its added computation burdens training and its square and root operations impede low-bit quantization.The overhead is especially salient for resource-limited devices and online learning.Source: §I · p. 1
  • L1BN replaces each neuron's mini-batch L2-norm variance with an L1-norm variance and uses signum and absolute operations in backpropagation.The paper states approximate equivalence to L2BN after applying a scaling factor.Source: §I · p. 1
  • L1BN achieves comparable accuracy and convergence speed to L2BN across CNN and GAN experiments, while FPGA operations provide 1.5× speedup and 50% power saving.The operation-level comparison is against the costly square and root operations, respectively.Source: §I · p. 2

II. PROBLEM FORMULATION

Mini-batch SGD estimates gradients from a subset of samples, but deep networks remain sensitive to initialization and learning-rate choices as layer inputs shift. Batch normalization addresses this by normalizing layer inputs across each training mini-batch.

II. PROBLEM FORMULATION

  • Mini-batch SGD estimates each iteration's gradient from m samples and updates parameters using their average sample gradients.The objective is expressed as a sum of differentiable sample losses.Source: §II · p. 2
  • Deep-network layer inputs depend on preceding parameters, so small changes can amplify and cause outputs or gradients to vanish or explode.This sensitivity makes training slower and requires careful learning-rate and initialization choices.Source: §II · p. 2
  • Batch normalization addresses internal covariate shift by normalizing layer inputs across each training mini-batch, allowing higher learning rates and less careful initialization.Batch-statistic randomness can also act as a regularizer and often remove the need for dropout.Source: §II · p. 2

A. Conventional L2BN

Conventional L2BN normalizes each feature using batch estimates of its mean and variance, then applies trainable scale and shift parameters. Its square and root operations add training cost and complicate low-bit quantization.

A. Conventional L2BN

  • BN normalizes each scalar feature independently to zero mean and unit variance, with expectations and variances computed over training samples.For a mini-batch, its mean and variance estimate those statistics, with a small positive epsilon used to avoid numerical error.Source: A. Conventional L2BN · p. 2
  • Trainable γ and β parameters scale and shift normalized values so the inserted transformation can represent the identity transform.Both parameters are trained alongside the original model parameters.Source: A. Conventional L2BN · p. 2
  • The paper derives a chain rule for backpropagation when batch normalization is included.Source: A. Conventional L2BN · p. 2
  • L2BN's variance estimate uses an L2-norm, requiring square and root operations that add training cost and complicate bit-width quantization.The table is labeled as comparing training samples per second with and without BN.Source: A. Conventional L2BN · p. 3, Table I · p. 3

B. Proposed L1BN

L1BN replaces the minibatch L2-norm variance estimate with an L1-norm estimate and derives a corresponding backward rule. The paper presents it as approximately equivalent to L2BN under a scaling relation while avoiding square and root operations.

B. Proposed L1BN

  • L1BN estimates minibatch scale with an L1 norm, which the paper motivates by its strong linear correlation with L2-norm variance.The paper states that equivalence to L2BN requires a scaling factor.Source: B. Proposed L1BN · p. 3
  • The derived backward rule lets L1BN avoid square and root operations in both forward computation and backpropagation.The paper says this significantly reduces computational cost compared with L2BN.Source: B. Proposed L1BN · p. 3
  • The equivalence discussion assumes zero-mean Gaussian inputs to all layers, and the paper says experiments validate the remark.The proof uses the half-normal distribution of the absolute value of a centered Gaussian variable.Source: B. Proposed L1BN · pp. 3–4

III. L1BN IN MLP AND CNN

L1BN applies to activation sets in MLPs and CNNs, with the L1 norm replacing the L2 norm in the scale calculation. Its scaling can be compensated explicitly or learned through the BN scale parameter.

III. L1BN IN MLP AND CNN

  • L1BN applies to activation sets in MLPs and CNNs, differing from L2BN in its use of the L1 norm to calculate σ.Other normalization processes remain the same as in L2BN.Source: §III · p. 4
  • Without a BN rescale parameter, multiply σL1 by the stated scaling factor to approximate σL2; with γ, let trainable γL1 learn the compensation.The paper describes these as two alternative strategies for compensating the standard-deviation difference.Source: §III · p. 4
  • For CNNs, normalization spans each feature map across batch elements and spatial locations, with learned γ and β per feature map.For MLPs, each neuron is normalized over the training-batch samples before activation.Source: §III · p. 4
  • Algorithm 1 presents the L1BN procedure.Source: §III · p. 4

A. L1BN on classification tasks

Across varied image-classification tasks, L1BN matches or slightly surpasses L2BN in many cases, with a small possible accuracy cost for MobileNet. Its early test-error instability can be reduced by changing the L1BN scale initialization, after which the ResNet-110 loss curves nearly overlap.

A. L1BN on classification tasks

  • Classification experiments varied model and task complexity, while keeping hyperparameters the same between L1BN and L2BN except for the batch-normalization norm.The tested settings include shallow Fashion-MNIST, deeper CIFAR models, and ILSVRC12 models.Source: A. L1BN on classification tasks · p. 4, §1.2 · pp. 4–5

1.2 Backward:

  • L1BN matches or slightly surpasses L2BN in many classification settings, though its additional regularization may slightly hurt MobileNet accuracy.The reported comparison covers test or validation error rates across multiple datasets and CNN architectures.Source: Table II · p. 5, §1.2 · p. 5
  • L1BN test error is somewhat more unstable early in training, but initializing γ at 0.8 instead of 1.0 makes the ResNet-110 loss curves nearly overlap.The plotted training losses include weight decay.Source: §1.2 · p. 5, Figure 1 · p. 5

B. L1BN on generative tasks

The paper evaluates L1BN in GAN image-generation settings, where training stability and generated-image quality are sensitive to numerical and hyperparameter differences. Across the reported CIFAR-10 and LSUN-Bedroom tests, L1BN and L2BN produce comparable outcomes.

B. L1BN on generative tasks

  • The GAN evaluation covers CIFAR-10 and LSUN-Bedroom; the CIFAR-10 implementations omit some training techniques and use different hyperparameters and network structures.The authors note that adding those techniques might further bridge the performance gap.Source: B. L1BN on generative tasks · p. 5
  • For CIFAR-10 GAN tests, the authors report that L1BN remains equivalent to L2BN in these hyperparameter-sensitive settings.The reported evaluation uses Inception scores, with both the average of the last 20 evaluations and the best score during training.Source: B. L1BN on generative tasks · p. 5, Table III · p. 6
  • On 128×128 LSUN-Bedroom, L1BN and L2BN generate samples with comparable detailed texture and no significant difference in artistic style.The figure shows L2BN samples on the left and L1BN samples on the right.Source: Figure 2 · p. 6, B. L1BN on generative tasks · p. 6

C. Layerwise and channelwise comparasion

The analysis compares layerwise and channelwise L2-to-L1 deviation ratios under matched layer inputs. Ratios approach approximately 1.25 in intermediate layers whose data are closer to Gaussian, consistent with the paper’s scaling argument.

C. Layerwise and channelwise comparasion

  • The comparison calculates both deviations within each layer while feeding L1BN and L2BN the same layer inputs after training and fixing parameters and moving averages.This controls for numerical differences accumulating across layers.Source: C. Layerwise and channelwise comparasion · p. 6
  • The observed ratio pattern is consistent with the paper’s Gaussian-input scaling factor of approximately 1.25 for L2BN relative to L1BN.The authors also suggest that L1-norm statistics introduce additional randomness that may further regularize large models.Source: C. Layerwise and channelwise comparasion · pp. 6–7
  • For intermediate layers 38–73, where data are relatively closer to Gaussian, σL2/σL1 is close to 1.25.The σL2 and σL1 histograms are similar apart from a phase shift on the log-scaled x-axis.Source: C. Layerwise and channelwise comparasion · pp. 6–7

D. Computational efficiency of L1BN

L1-norm operations are estimated to improve computational efficiency over L2-norm operations on FPGA. The comparison weights operation counts by FPGA time and power estimates, omitting root-operation cost in the model-level comparison.

D. Computational efficiency of L1BN

  • FPGA estimates use 32-bit floating point on a DE5-Net board, with time and power reduced by the null-operation reference.Altera OpenCL estimates performance, while PowerPlay Early Power Estimator estimates power.Source: D. Computational efficiency of L1BN · p. 7
  • Signum and absolute operations save registers and require no DSP blocks compared with square and root operations.Table IV presents computational overhead estimates for basic arithmetic operations on FPGA.Source: D. Computational efficiency of L1BN · p. 7, Table IV · p. 7
  • Approximately 1.5× speedup and 50% power saving are estimated for L1-norm operations in practice.The model-level comparison omits root-operation cost because it is performed only once per channel.Source: D. Computational efficiency of L1BN · p. 7
  • The CNN comparison weights operation counts from Equations 4, 8, 11, and 12 by FPGA time and power statistics.Root-operation cost is omitted because it occurs only once per channel.Source: D. Computational efficiency of L1BN · p. 7

V. DISCUSSIONS AND CONCLUSIONS

L1BN replaces L2-norm variance with L1-norm variance and substitutes hardware-friendly operations for square and root operations. The paper reports comparable CNN and GAN results, lower estimated hardware costs, and potential for low-precision online training.

V. DISCUSSIONS AND CONCLUSIONS

  • L1BN replaces L2-norm variance with L1-norm variance and uses absolute and signum operations instead of square and root operations.The paper derives the chain rule for differentiating L1BN during backpropagation.Source: §V · p. 7
  • Figure 4 estimates time and power consumption for L1-norm and L2-norm across different CNN models.The supplied caption gives the figure's scope but no individual model values.Source: Figure 4 · p. 7
  • L1BN reports comparable classification accuracy, generation quality, and convergence rates to L2BN, with 1.5× speedup and 50% energy saving in FPGA cost comparisons.The authors also report reduced register and DSP-block requirements.Source: §V · p. 7
  • L1BN is equivalent to L2BN up to a scaling factor when inputs follow a zero-mean Gaussian distribution.The paper describes this input distribution as commonly designed in conventional deep-learning works.Source: §V · p. 7
  • The authors present L1BN's lower resource overhead, speed, and energy use as relevant to online training on resource-limited edge devices.They also state that replacing square and root operations promises fully quantized neural networks with low-precision dataflow for online training.Source: §V · p. 7

Related papers

Loading 1802.09769v1…