PaperlayerView exact revision on arXiv

Source-linked AI summary

Extremely Low Bit Neural Network: Squeeze the Last Bit Out with ADMM

Cong Leng, Hao Li, Shenghuo Zhu, Rong Jin

arXiv:1707.09870v2cs.CV

TL;DR

Deep networks are difficult to deploy under memory or computational constraints because of their high storage and computation costs. The paper formulates extremely low-bit learning as a discrete nonconvex optimization problem, separates continuous and discrete variables with ADMM, and solves the resulting subproblems using extragradient and iterative quantization; experiments on image recognition and object detection report effectiveness against state-of-the-art approaches.

  • Problem

    High storage and computational costs limit deployment of deep learning models in scenarios with constrained memory or computation.

  • Method

    The method uses ADMM to decouple continuous network parameters from discrete constraints and solves the resulting proximal and projection subproblems with extragradient and iterative quantization.

  • Results

    The proposed method significantly outperforms state-of-the-art approaches across experiments on image classification and object detection.

  • Takeaways & Limitations

    Extremely low-bit neural networks can be learned through a unified ADMM-based optimization framework evaluated across image recognition and object detection.

  • Takeaways & Limitations

    GoogleNet’s extremely low-bit quantization suffers large degradation, suggesting that different network parts may require different bit widths, especially for 1×1 kernels.

Abstract

from arXiv · show

Although deep learning models are highly effective for various learning tasks, their high computational costs prohibit the deployment to scenarios where either memory or computational resources are limited. In this paper, we focus on compressing and accelerating deep models with network weights represented by very small numbers of bits, referred to as extremely low bit neural network. We model this problem as a discretely constrained optimization problem. Borrowing the idea from Alternating Direction Method of Multipliers (ADMM), we decouple the continuous parameters from the discrete constraints of network, and cast the original hard problem into several subproblems. We propose to solve these subproblems using extragradient and iterative quantization algorithms that lead to considerably faster convergency compared to conventional optimization methods. Extensive experiments on image recognition and object detection verify that the proposed algorithm is more effective than state-of-the-art approaches when coming to extremely low bit neural network.

1 Introduction

The paper targets the storage and computational barriers that limit deep-network deployment by formulating extremely low-bit weights as a discrete optimization problem. Its ADMM-based framework separates continuous weights from discrete constraints and uses specialized subproblem solvers, with experiments reporting advantages across vision tasks.

  • Motivation: Deep networks face substantial storage and computational costs, limiting deployment where memory or computation is constrained.VGG-16, for example, contains 528 MBytes of model parameters.
  • Problem formulation: The paper formulates extremely low-bit network learning as a discretely constrained nonconvex optimization problem related to mixed integer programs.The authors note that mixed integer programs are NP-hard.
  • ADMM framework: ADMM introduces auxiliary variables to decouple continuous weights from discrete constraints and decomposes learning into proximal, projection, and dual-update subproblems.The reformulation connects the continuous and discrete solutions through an augmented Lagrangian.
  • Optimization: Extragradient accelerates the proximal step, while iterative quantization solves the projection step.The paper reports that the resulting algorithm converges faster in practice than conventional optimization methods.
  • Evaluation: Experiments on image classification and object detection report that the proposed method significantly outperforms state-of-the-art approaches.The evaluation covers multiple convolutional neural network architectures and vision tasks.

2 Related Work

Prior work reduces deep-network cost through parameter factorization, weight discretization, and combined compression methods, while extremely low-bit training remains challenging on large-scale tasks. This paper instead formulates quantization as a discrete nonconvex problem and applies ADMM as an efficient heuristic framework.

  • Historical scope: Earlier low-bit quantization work established memory-efficiency and multiplication-free benefits but did not verify them in deep networks and large-scale datasets.The cited early networks were shallow.
  • Extremely low-bit training: BinaryConnect replaces network weights W with sign(W) during loss optimization and uses a hard-tanh approximation in backpropagation to avoid zero gradients.Its reported performance was good on simple datasets but degraded substantially on ImageNet.
  • Follow-up methods: BWN improves BinaryConnect with scale factors, while TWN extends the approach to ternary weights.Both methods modify the backward process to address the zero-gradient issue.
  • Paper’s positioning: The paper differs by jointly optimizing continuous and discrete variables within a unified ADMM framework rather than relying on gradient modifications for continuous weights.ADMM is presented as combining decomposability with favorable convergence properties and as a heuristic for nonconvex mixed programs.

3 The Proposed Method

The method formulates extremely low-bit neural-network training as discretely constrained nonconvex optimization and uses ADMM to separate continuous weights from discrete variables. Extragradient and iterative quantization efficiently solve the resulting subproblems.

  • 3.1 Objective function: The network weights are restricted to zero or powers of two, enabling floating-point multiplications to be replaced by bit shifts.The allowed constraint set includes signed powers of two and zero, with the number of bits determined by N.
  • 3.1 Objective function: Layer-specific scaling factors α_i expand the discrete constraint space while adding no convolutional computation.The scaling factor is applied after convolution using the unscaled discrete values.
  • 3.2 Decouple with ADMM: ADMM introduces an auxiliary discrete variable, decoupling continuous weights from discrete constraints through an equality constraint.The resulting formulation uses an augmented Lagrangian and treats the problem as nonconvex optimization with convex linear constraints.
  • 3.2 Decouple with ADMM: ADMM alternates proximal, projection, and dual-update subproblems while jointly optimizing in continuous and discrete spaces.The continuous and discrete solutions are connected during learning by ADMM.
  • 3.3.1 Proximal step: Extragradient replaces slow vanilla gradient descent in the proximal step with prediction and correction updates that account for problem geometry.The additional gradient step is described as improving convergence relative to standard gradient descent.
  • 3.3.2 Projection step: The projection step independently projects each layer's auxiliary variable onto its discrete constraint set, although the discrete nonconvex optimization remains nontrivial.Layerwise decoupling follows from the separability of the auxiliary variables in the projection subproblem.
  • 3.3.1 Proximal step: Iterative quantization alternates optimizing scaling factors α_i and discrete values Q_i, converging to a local minimum in fewer than five iterations in most cases.Each step decreases the loss, and the authors report fast practical convergence.

4 Experiments

The experiments evaluate the proposed algorithm on ImageNet for image classification and Pascal VOC for object detection.

  • 4 Experiments: The evaluation uses ImageNet for image classification and Pascal VOC for object detection.These two benchmarks are used to verify the effectiveness of the proposed algorithm.

4.1 Image Classification

ImageNet experiments evaluate extremely low-bit quantization across AlexNet, VGG-16, ResNet-18, ResNet-50, and GoogleNet. The method generally outperforms competing low-bit approaches, while compression difficulty varies by architecture.

  • Experimental setup: Experiments evaluate binary, ternary, one-bit shift, and two-bits shift quantization on ImageNet CNN architectures.Binary and ternary weights use one and two bits respectively; shift quantization uses {-2a, -a, 0, +a, +2a} or {-4a, ..., +4a}.
  • AlexNet and VGG-16: Only 3 bits achieve lossless compression on AlexNet and VGG-16 relative to full-precision references.For VGG-16, {-2, +2} and {-4, +4} quantized networks can even outperform the original full-precision network.
  • AlexNet and VGG-16: Binary and ternary quantization slightly outperform BWN and TWN on AlexNet and VGG-16, with ternary networks consistently better than binary networks.The authors attribute ternary efficiency partly to many zero-valued weights.
  • ResNet: 4 points in top-1 and 3.2 points in top-5 accuracy separate the proposed binary ResNet-18 network from BWN.The proposed ternary network exceeds TWN by 5.2 and 3.3 points in top-1 and top-5 accuracy, respectively.
  • ResNet: About 5 points of top-1 improvement over BWN is obtained on ResNet-50 with binary quantization.ResNet-18 and ResNet-50 remain harder to quantize than AlexNet and VGG-16, with about a 1-point top-1 gap for {-4, +4} networks versus full precision.
  • GoogleNet: GoogleNet shows a more than 2-point top-1 gap for {-4, +4} quantization and an 8-point loss for binary quantization versus full precision.The proposed method nevertheless outperforms BWN and TWN on GoogleNet.
  • Comparison with recent works: With only 3 bits, the method achieves ResNet-50 top-1 and top-5 accuracy reported for INQ at 5 bits.The cited accuracies are 73.2% top-1 and 91.2% top-5 for the 5-bit comparison.
  • INT8 quantized 1×1 kernel: INT8 quantization of GoogleNet’s 1×1 kernels considerably improves binary and ternary results over uniformly lower-bit quantization.The authors suggest different network parts should use different bit widths, while automatic bit-width selection remains future work.

4.2 Object Detection

The method is evaluated for object detection using quantized VGG-16 and Darknet models with SSD on Pascal VOC 2007. Four-bit quantization stays close to full precision, while INT8 treatment of 1×1 kernels improves Darknet results.

  • For {-4,+4} quantization, both models achieve mAP very close to their full-precision versions.On VGG16+SSD, the mAP loss is 0.002.
  • Darknet+SSD reaches 0.624 mAP when its 1×1 kernels are quantized as usual.A second Darknet version quantizing 1×1 kernels with INT8 improves mAP by 1.5 points.
  • Ternary quantization degrades Darknet+SSD more than VGG16+SSD because Darknet parameters are less redundant.
  • The results indicate that the proposed method is effective for object detection tasks.

5 Conclusion

The paper learns extremely low-bit neural networks by combining ADMM with extragradient and iterative quantization methods. Experiments on image recognition and object detection demonstrate the method’s effectiveness.

  • The method uses ADMM to decouple continuous network parameters from discrete constraints and decompose the problem into subproblems.Extragradient and iterative quantization algorithms solve these subproblems with faster practical convergence than conventional optimization methods.
  • Extensive experiments on convolutional networks for image recognition and object detection show the effectiveness of the proposed method.
Loading 1707.09870v2…