PaperlayerView exact revision on arXiv

Source-linked AI summary

Deep neural networks are robust to weight binarization and other non-linear distortions

Paul Merolla, Rathinakumar Appuswamy, John Arthur, Steve K. Esser, Dharmendra Modha

arXiv:1606.01981v1cs.NEcs.CVcs.LG

TL;DR

The paper asks whether robustness associated with quantized-weight training extends to other weight distortions and training procedures. It examines alternative projections and clipping, and reports robustness across distortion families, with a stochastic projection rule achieving CIFAR-10 state-of-the-art results.

  • Problem

    Although quantized-weight training had been argued to be essential for low-precision representations, the paper asks whether weight projections can produce robustness beyond quantization and whether other training procedures can do so.

    Source: §1 · p. 1

  • Method

    The paper studies networks trained with varied weight projections or clipping and proposes a stochastic rule that projects weights to random intervals.

    Source: §1 · p. 2

  • Results

    Networks trained with certain weight distortions are robust to broader distortion families, and the stochastic rule reports CIFAR-10 state-of-the-art results of 7.64% in the non-binary case.

    Source: §1 · p. 2, §6 · p. 9

  • Takeaways & Limitations

    The reported robustness is not limited to quantized projections: other projections and even clipping can yield robust networks, though clipping can slightly reduce performance.

    Source: §1 · p. 2

    The proposed gradient-noise account is framed as a possible bridge to prior work, rather than as an established explanation.

    Source: §5 · p. 9

Abstract

from arXiv · show

Recent results show that deep neural networks achieve excellent performance even when, during training, weights are quantized and projected to a binary representation. Here, we show that this is just the tip of the iceberg: these same networks, during testing, also exhibit a remarkable robustness to distortions beyond quantization, including additive and multiplicative noise, and a class of non-linear projections where binarization is just a special case. To quantify this robustness, we show that one such network achieves 11% test error on CIFAR-10 even with 0.68 effective bits per weight. Furthermore, we find that a common training heuristic--namely, projecting quantized weights during backpropagation--can be altered (or even removed) and networks still achieve a base level of robustness during testing. Specifically, training with weight projections other than quantization also works, as does simply clipping the weights, both of which have never been reported before. We confirm our results for CIFAR-10 and ImageNet datasets. Finally, drawing from these ideas, we propose a stochastic projection rule that leads to a new state of the art network with 7.64% test error on CIFAR-10 using no data augmentation.

1 Introduction

Prior work established that networks can perform well with binary or ternary weights, commonly by projecting high-precision weights during training. This paper broadens the question to whether training-time weight distortions yield robustness to other distortions and to alternative training procedures.

1 Introduction

  • Reducing inference-time weight precision is a longstanding goal because it can lower computational and memory footprints, while prior work showed strong benchmark performance with binary or ternary weights.Prior approaches typically use high-precision weights for training and project them during forward and backward passes.Source: §1 · p. 1
  • Networks that perform well with binarized weights also perform well under additive noise, multiplicative noise, and nonlinear weight distortions.Binarization is presented as one case within a broader family of distortions.Source: §1 · p. 2
  • Training with projections other than quantization can yield robust networks, and standard backpropagation with weight clipping can yield base-level robustness with slightly reduced performance.The authors describe these as findings not previously reported.Source: §1 · p. 2
  • 8.25% in the binary case and 7.64% in the non-binary case are the reported CIFAR-10 state-of-the-art results for the proposed stochastic projection rule.The rule projects weights to random intervals rather than quantized values.Source: §1 · p. 2

2 Related work

Related work develops backprop-based methods for training with quantized weights and explores probabilistic or constrained approaches to discrete weights. This paper extends the projection-based line beyond quantization.

2 Related work

  • Backprop-based methods that quantize weights during propagation reported impressive CIFAR-10 results and were also developed for neuromorphic hardware [12] [10] [9].A similar rule was proposed for contrastive divergence.Source: §2 · p. 2
  • Other approaches formulate binary-weight approximation as constrained optimization, with promising ImageNet results [7].The authors expect those findings to be consistent with theirs but leave the comparison for future work.Source: §2 · p. 2
  • Probabilistic interpretations of discrete weights had not yet been extended to convnets or datasets larger than MNIST.The paper explores weight projections beyond quantization.Source: §2 · p. 2

3 DNNs with projected weights

The method applies scalar weight projections within a layer-indexed network, computes gradients with respect to projected weights, and updates the underlying weights. Training and testing projections are specified independently, and per-layer clipping can also be applied.

3 DNNs with projected weights

  • Each layer computes its next activation by convolving its input activation with projected weights and applying ReLU; biases are not considered.Fully connected layers are written as convolutions for notational simplicity.Source: §3 · p. 2
  • The projection functions are scalar mappings extended element-wise to weight tensors, and the Power projection generalizes Sign and None.Power(w_ki, 0) = Sign(w_ki), while Power(w_ki, 1) = None(w_ki); a per-layer factor normalizes weights to [−1, 1] for certain projections.Source: §3 · p. 2
  • Table 1 defines the weight projections, including the normal and uniform distributions used in their definitions.N(0, σ) denotes a normal distribution with mean 0 and standard deviation σ; U(a, b) denotes a uniform distribution over [a, b].Source: Table 1 · p. 3
  • Training projects weights before the forward pass, computes gradients with respect to projected weights, updates the underlying weights, and clips them after each update.The clipping boundary is per layer and is based on the initialized-weight standard deviation scaled by a global factor; with no projection and infinite clipping bounds, the algorithm reduces to standard backpropagation.Source: §3 · pp. 2–3
  • Training and testing projections are specified independently, and projections used at test time are called distortions.Testing may therefore use a projection different from the one used in training.Source: §3 · p. 3

4 Results

The experiments find that networks can retain performance under weight distortions beyond the projections used in training, including when projections are removed. The paper also introduces a stochastic projection that achieves state-of-the-art CIFAR-10 error without data augmentation.

4.1 CIFAR-10 experiments

  • The CIFAR-10 experiments train six networks with different projections and clipping settings, then test them under weight distortions; CIFAR-10 training uses no data augmentation.The networks use six convolutional and two fully connected layers and are trained for 500 epochs; Table 2 organizes test errors by trained network and test distortion.Source: §4.1 · pp. 3–4, Table 2 · p. 4

4.1.1 High precision or -1,+1? It doesn’t (much) matter

  • All tested networks have comparable Te-None and Te-Sign errors, although Tr-Sign-C’s weights differ across the two tests and its layer activations converge to similar patterns.The paper reports quantization errors throughout the layers, alongside similar propagated activity patterns.Source: §4.1.1 · p. 4

4.1.2 Robustness to weight distortions beyond -1,+1

  • Tr-Sign-C and Tr-Stoch-C are more resilient than NiN-ctrl to Gaussian noise, multiplicative noise, and Power distortions not used in their training.Tr-Stoch-C reaches 11% test error at σ = 0.55, corresponding to 0.68 effective bits per weight; under Power, lower β pushes Tr-Sign-C’s Conv2 weights toward a bimodal distribution approaching Sign(Wk).Source: §4.1.2 · pp. 4–5, Figure 2 · p. 6

4.1.3 Learning robust networks with and without weight projections

  • Training with stochastic Power projections using a new β for each minibatch yields robustness similar to Tr-Sign-C, while clipping without projections also provides robustness.Tr-None-C reaches 11.3% test error with binary weights despite never being trained with them; Tr-None-C is more robust than Tr-None-NC, while the authors describe Tr-None-NC’s robustness as a base level.Source: §4.1.3 · pp. 5–6

4.1.4 Stochastic multiplicative projection

  • Tr-StochM-C achieves 7.64% error for Te-None and 8.25% for Te-Sign, state of the art on CIFAR-10 without data augmentation, to the authors’ knowledge.With γ = 0.5 during training, the expected projected weight is no longer the same as wki.Source: §4.1.4 · pp. 6–7

4.2 Towards a robust AlexNet

  • For ImageNet AlexNet trained without projections, clipping produces similar errors across tests: 35% for Te-None, 34.7% for Te-Round, and 40.5% for Te-Sign.Without clipping, Tr-None-NC reaches 28.1% for Te-None, while Te-Round and Te-Sign errors are significantly worse; the authors say the CIFAR-10 finding did not hold for this network on ImageNet.Source: §4.2 · p. 7

4.2.1 ImageNet benchmarks

  • The ImageNet Tr-StochM3-C2 benchmark uses a 150K-iteration, 64-epoch training schedule and probes robustness with Te-Power against two recent binary-weight models.The learning rate drops at iterations 100K and 125K, while the global clip factor increases at 5K, 10K, and 15K to keep Te-None and Te-Round scores in sync while allowing lower error.Source: §4.2.1 · p. 7

5 Hypotheses on learning robust networks

The authors frame clipping as a regularizer and stochastic projections as a way to smooth the weight-space loss surface. They propose that these mechanisms can help explain robustness to weight distortions and suggest further directions for testing the ideas.

5 Hypotheses on learning robust networks

  • Figure 4 describes AlexNet trained via StochM3 as robust to power distortions; StochM3 drops the middle 50% of weights to zero with probability 0.5.Source: §5 · p. 7, Figure 4 · p. 8
  • A proximal update applies a loss-gradient step and then the regularizer’s proximal operator; with a per-layer clip value of 1, that operator clips weights elementwise to [−1, 1].Source: §5 · p. 8
  • In a convex setting, clipping constrains weights to the unit ℓ∞ ball; empirically, this regularization appears to reduce loss sensitivity to distortions in non-convex networks.When clip values are scaled, performance temporarily dips as the network adapts.Source: §5 · pp. 7–8
  • Stochastic projections make backpropagation sample a smoothed loss surface over projected weights, which may help explain the observed robustness.The authors describe this as additional gradient smoothing, even when the projection is partly deterministic.Source: §5 · p. 8
  • Future work could gradually cool the stochasticity of φθ and extend projections to depend on weight vectors, as in the filter-level ℓ1 projection of [7].The cited work [7] is described as being in a slightly different context.Source: §5 · p. 9
  • The authors suggest that clipping and deterministic or stochastic projections distort weight gradients like noise, potentially connecting their findings to explicit gradient-noise work [29].Source: §5 · p. 9

6 Conclusion and future work

The paper reports that training with certain weight distortions can yield robustness across a family of distortions, and proposes stochastic projection to random intervals. The rule reaches state-of-the-art CIFAR-10 performance for binary and non-binary networks; the authors also point to noisy-device weights as a possible application.

6 Conclusion and future work

  • Training with certain weight distortions can make networks robust not only to the training distortion but to an entire family of distortions.Source: §6 · p. 9
  • The proposed stochastic rule projects each weight to a random interval based on its current value; the authors hypothesize it optimizes a neighborhood of the weight vector.The rule achieves state-of-the-art CIFAR-10 performance for both binary and non-binary networks.Source: §6 · p. 9
  • A network achieves 89% on CIFAR-10 with 0.68 bits per weight, which the authors identify as potentially useful for noisy-device weights and neuromorphic computing.Source: §6 · p. 9
  • The authors hypothesize that neuron outputs in networks with binary activations are also robust to distortions, and speculate that other activation constraints could improve performance.Source: §6 · p. 9

Related papers

Loading 1606.01981v1…