Source-linked AI summary
Loss-aware Binarization of Deep Networks
Lu Hou, Quanming Yao, James T. Kwok
TL;DR
Deep networks are costly to store and compute, while existing binarization methods approximate weights without accounting for their effect on loss. The paper uses proximal Newton optimization with a diagonal Hessian to obtain loss-aware binary weights. Across feedforward and recurrent experiments, the method outperforms existing schemes and remains robust for wide and deep networks.
Problem
Existing binarization schemes use simple weight approximation and ignore binarization's effect on the loss, despite deep networks' space and time costs.
Method
The method directly optimizes binarized weights with proximal Newton and a diagonal Hessian, using an efficient closed-form proximal step and second-order information.
Results
The proposed algorithm outperforms existing binarization schemes, has comparable performance to the full-precision network, and is robust for wide and deep networks.
Takeaways & Limitations
Loss-aware curvature information provides a binarization method that remains robust across feedforward and recurrent networks, including deep recurrent settings.
Takeaways & Limitations
Convergence analysis for the convex proximal Newton algorithm does not apply because the loss and indicator function are nonconvex, and the cited nonconvex extension's assumptions do not hold.
Abstract
from arXiv · showhide
Deep neural network models, though very powerful and highly successful, are computationally expensive in terms of space and time. Recently, there have been a number of attempts on binarizing the network weights and activations. This greatly reduces the network size, and replaces the underlying multiplications to additions or even XNOR bit operations. However, existing binarization schemes are based on simple matrix approximation and ignore the effect of binarization on the loss. In this paper, we propose a proximal Newton algorithm with diagonal Hessian approximation that directly minimizes the loss w.r.t. the binarized weights. The underlying proximal step has an efficient closed-form solution, and the second-order information can be efficiently obtained from the second moments already computed by the Adam optimizer. Experiments on both feedforward and recurrent networks show that the proposed loss-aware binarization algorithm outperforms existing binarization schemes, and is also more robust for wide and deep networks.
1 INTRODUCTION
Deep networks deliver strong performance but incur substantial storage and computation costs, motivating compact approaches such as binarization. The paper proposes loss-aware binarization using proximal Newton optimization and reports robust improvements over existing schemes.
- Motivation: Deep networks require substantial storage and billions of high-precision operations, limiting deployment on embedded and portable devices.AlexNet, VGG-16, and ResNet-18 each require hundreds of megabytes for storage.
- Existing approaches: Binarization uses one bit per weight, reducing storage and replacing multiplications with cheaper operations.Prior work also binarizes activations and scales binary weights.
- Existing approaches: Existing binarization schemes include BinaryConnect, binarized activations, scaled binary weights, XNOR-Networks, and ternary quantization.
- Proposed approach: The paper considers binarization's effect on loss through a proximal Newton algorithm with a diagonal Hessian approximation.Its proximal step has a closed-form solution, and curvature information can be obtained from Adam's second moments.
- Results: Experiments on feedforward and recurrent networks show that the proposed method outperforms existing binarization algorithms and remains robust for wide and deep networks.BinaryConnect fails on deep recurrent networks because of exploding gradients, whereas the proposed method remains robust.
2 RELATED WORK
Related work covers weight and activation binarization, recurrent propagation with full-precision updates, and proximal Newton optimization for composite objectives. These methods motivate the paper's loss-aware treatment of binary weights.
- Weight binarization: During training, full-precision weights are updated with backpropagated gradients, while binarized weights are used for forward and backward propagation.
- Weight binarization: BinaryConnect binarizes individual weight elements, whereas BWN additionally learns a positive scaling parameter for the binary weights.
- Activation binarization: Binarized activations extend weight binarization by applying binary representations to layer activations during network computation.
- Proximal Newton: The proximal Newton algorithm addresses composite optimization with a smooth convex term and a possibly nonsmooth convex term using an approximate Hessian.Second-order information gives faster convergence than proximal gradient methods in the stated setting.
- Scope of comparison: A stochastic binarization scheme was proposed previously but is excluded here because it is much more computationally expensive than the compared deterministic formulation.
3 LOSS-AWARE BINARIZATION
The paper formulates weight binarization as loss-aware composite optimization and solves it with proximal Newton updates using diagonal curvature information. The resulting LAB procedure provides closed-form binary updates, retains full-precision weights, and extends to recurrent networks while addressing curvature and gradient-scaling issues.
- Motivation: Existing binarization methods approximate weights directly, whereas LAB incorporates binarization’s effect on the loss.LAB represents each layer’s binarized weight as ˆw_l = α_l b_l, with α_l > 0 and binary b_l.
- Optimization formulation: LAB formulates binarization as minimizing the loss subject to scaled binary-weight constraints.The feasible set is represented with an indicator function, converting the constrained problem into composite optimization.
- Proximal Newton method: Proximal Newton replaces the loss with a second-order expansion and uses diagonal Hessian information to account for heterogeneous curvature.The method uses an approximate Hessian at the current weights; Adam’s second moment supplies an efficient curvature estimator.
- Proximal Newton method: The proximal subproblem has a closed-form solution, and each iteration combines adaptive-rate gradient descent with projection to a binary solution.The algorithm maintains full-precision weights during training while producing binarized weights for network computation.
- Analysis: Under the stated curvature condition, the proximal Newton objective converges, although the loss and binary constraint indicator are nonconvex.The paper notes that existing convex convergence analysis does not apply and that the cited nonconvex extension assumes a decomposition unavailable here.
- Recurrent networks: For recurrent networks, LAB binarizes input-to-hidden and hidden-to-hidden weights and adaptively learns the hidden-state scaling parameter α_h to alleviate exploding gradients.The extension exploits the recurrent setting’s shared weights across time and addresses gradient growth associated with large binary recurrent matrices.
4 EXPERIMENTS
Experiments evaluate LAB against full-precision and established binarization schemes across feedforward, convolutional, and recurrent networks. LAB generally performs best and is less affected by increased model width or recurrent depth, while BinaryConnect can fail for long sequences.
- Experimental setup: Experiments compare full-precision networks with weight-binarized BinaryConnect, BWN, and LAB, plus weight-and-activation-binarized BNN, XNOR, and LAB2.The evaluation covers feedforward and recurrent neural networks without data augmentation or unsupervised pretraining.
- Feedforward networks: LAB achieves the lowest classification error on MNIST and SVHN, surpasses full precision on MNIST, and outperforms BinaryConnect and BWN there.On CIFAR-10, BinaryConnect is slightly better than LAB, although LAB remains better than the full-precision network.
- Feedforward networks: LAB2 outperforms BNN and XNOR among schemes that binarize both weights and activations.The comparison is reported for the feedforward experiments.
- Varying the number of filters in CNN: As CNN filter counts increase, degradation from binarization becomes less severe, and LAB has the best performance in the SVHN width experiment.The reported table measures test error rates and brackets the difference from the full-precision network.
- Recurrent networks: On LSTM language modeling, LAB outperforms other weight-binarization schemes and even full precision on Linux Kernel, while LAB2 outperforms BNN and XNOR.BWN and LAB scale the binary weight matrix; the paper attributes LAB’s advantage over BWN to its use of curvature information.
- Varying the number of time steps in LSTM: For LSTMs with TS ≥50, BinaryConnect fails while BWN and LAB perform better; deeper models also show stronger binarization degradation because weights are shared across time.At TS = 10, binarization schemes have performance similar to the full-precision network, whereas deeper settings expose larger differences.
5 CONCLUSION
The paper presents a loss-aware binarization algorithm based on proximal Newton optimization with a diagonal Hessian approximation. It reports efficient computation and improved robustness and performance relative to existing schemes.
- The proposed algorithm directly considers binarization’s effect on the loss.
- Binarized weights are obtained with a proximal Newton algorithm using a diagonal Hessian approximation.
- The proximal step has an efficient closed-form solution, while Hessian second-order information comes from the Adam optimizer.
- Experiments report that the method outperforms existing binarization schemes and has performance comparable to the original full-precision network.
- The method is reported to be robust for wide and deep networks.
A PROOF OF PROPOSITION 3.1
The proposition uses a sign-based rule to obtain binarized quantities across network layers. The displayed formulation indexes these quantities by time and layer.
- Binarization is defined through the sign function applied to w_t^l.
- The formulation indexes the binarized quantities across layers l = 1, 2, ..., L.
- The rule is expressed at time t, indicating a time-indexed sequence of layerwise binarization steps.
B PROOF OF THEOREM 3.1
Under the stated assumptions, the objective sequence converges because the loss is bounded below and the loss values decrease monotonically.
- The proof invokes Assumption A1 in deriving the stated result.
- The proof combines equations (9) and (10) to obtain the next inequality or conclusion.
- Assumption A2 ensures that the loss ℓ is bounded from below.
- The sequence of loss values ℓ(ŵ^t) decreases monotonically and therefore converges.
- The objective sequence F(ŵ^t, α^t) also converges.
C PROOF OF PROPOSITION 3.2
The proposition introduces the singular values of W in descending order, establishing notation for subsequent analysis.
- The matrix W has singular values denoted by λ1(W), λ2(W), through λm(W).
- The singular values are ordered from largest to smallest.
- The notation identifies m singular values for W.