ProxQuant: Quantized Neural Networks via Proximal Operators

1 Introduction

• The straight-through method’s success is unclear because it uses gradients only at binary parameter values, which may not reveal function values or identify a minimizer.A simple convex Lipschitz example shows BinaryConnect can fail to find a minimizer because it lacks gradient information between those values.Source: §1 · p. 2
• The paper formulates quantization as regularized learning and trains with stochastic proximal gradient steps and a homotopy scheme.Compared with straight-through gradients, ProxQuant accesses gradients at non-quantized points, avoiding the toy failure case; its homotopy scheme prevents potential early-training overshoot.Source: §1 · pp. 2–3
• A unified regularization framework encourages binary, ternary, and multi-bit parameters by penalizing distance to quantized sets; the binary penalty is W-shaped and shrinks weights toward −1 or 1.Source: §1 · p. 2
• ProxQuant outperforms state-of-the-art binary quantization results and is comparable on ternary and multi-bit quantization across ResNet image classification and LSTM language modeling.The introduction also reports that ProxQuant converges to stationary points under mild smoothness assumptions, while lazy prox-gradient methods such as BinaryRelax can fail to converge in general.Source: §1 · p. 3
• ProxQuant converges to stationary points under mild smoothness assumptions, whereas lazy prox-gradient methods such as BinaryRelax can fail to converge in general.The introduction also says BinaryConnect requires a very stringent condition to converge to any fixed point, verified with a sign-change experiment.Source: §1 · p. 3

1.1 Prior work

• Prior work spans compression and quantized-network training; ProxQuant differs from earlier proximal or regularization approaches by using non-lazy, soft proximal descent.The paper reports theoretical and experimental advantages over lazy prox-gradient methods.Source: §1.1 · pp. 3–4

2 Preliminaries

• Quantized training is difficult because the parameter space is discrete: projected SGD may not move, nearest-neighbor search is costly, and quantizers such as sign are non-differentiable and piecewise constant.Source: §2 · p. 4

2.1 The straight-through gradient method

• The straight-through method passes gradients with respect to the quantized parameter back to the real parameter, enabling updates in Euclidean space but leaving its operation information-theoretically unclear.The method can fail even on simple convex Lipschitz functions.Source: §2.1 · p. 4

2.2 Straight-through gradient as lazy projection

• The straight-through gradient method is equivalent to dual averaging, or lazy projected SGD: it takes the gradient at the quantized vector and updates the original real vector.Source: §2.2 · p. 4

2.3 Projection as a limiting proximal operator

• A regularizer that is zero on the quantized set and positive outside it makes projection a limiting proximal operator: at λ = ∞, the prox is Euclidean projection onto that set.Source: §2.3 · pp. 4–5
• At finite λ, the prox operator softly moves parameters toward the quantized set, potentially avoiding early overshoot and allowing gradients to be queried throughout the space.Source: §2.3 · p. 5

3 Quantized net training via regularized learning

• ProxQuant adds a quantization-inducing regularizer to the loss and optimizes it with a non-lazy prox-gradient method; the prox step follows each stochastic gradient step.The prox step can be added to full-precision training and used with stochastic optimizers such as Adam.Source: §3 · p. 5

3.1 Regularization for model quantization

• Choosing the quantized set Q and an L1 or L2 penalty controls which parameters and bit structure are quantized, with harder or softer quantization encouraged respectively.The framework includes binary weights, ternary weights in {−α, 0, β}, and multi-bit quantization; the binary penalty is W-shaped and draws weights toward −1 or 1.Source: §3.1 · pp. 6–7
• The prox operator generalizes alternating minimization by trading off quantization against closeness to the input; as λ → +∞, it becomes projection onto the quantized set.The prox is computed by alternating minimization over the intermediate parameter and quantization variables.Source: §3.1 · p. 7

3.2 Homotopy method for regularization strength

• A linearly increasing λ_t = λ · t starts training near full-precision behavior and gradually moves parameters toward exact quantizedness.The regularization rate λ controls how aggressively parameters move toward the quantization constraint and can be tuned on validation loss.Source: §3.2 · p. 7

4.1 Image classification on CIFAR-10

• CIFAR-10 experiments use depth-20, -32, -44, and -56 ResNets with binary or ternary weights; binary comparisons use warm-started models trained for 300 epochs.BinaryConnect uses Adam with a decaying learning rate, while ProxQuant uses its binary or ternary regularizer with a linearly increasing regularization strength.Source: §4.1 · p. 8

4 Experiments

• Across ResNet image classification and LSTM language modeling, ProxQuant matches or often improves on straight-through results; binary models outperform BinaryConnect, while multi-bit results are comparable to state of the art.For binary ResNets, ProxQuant consistently beats BinaryConnect; for binary LSTMs, ProxQuant-Binary beats it by a large margin.Source: §4 · p. 7, Table 1 · p. 8, §4.1 · p. 8

4.2 Language modeling with LSTMs

• Penn Treebank experiments quantize the LSTM encoder, transition matrix, and decoder to 1, 2, or 3 bits, with each matrix row using its own codebook.The multi-bit method is compared with alternating minimization using straight-through gradients; the binary comparison is with BinaryConnect.Source: §4.2 · pp. 8–9

5.3 Convergence characterization for BinaryConnect

• BinaryConnect is the λ = ∞ limiting case of the prox-gradient setup, and its fixed-point condition may have no solution, preventing convergence from any initialization.The condition requires sign(∇L(s)[i]) = −s[i] wherever the gradient coordinate is nonzero; a CIFAR-10 sign-change experiment corroborates the failure to reach a fixed sign pattern.Source: §5 · p. 9, §5.3 · p. 11

5 Theoretical analysis

• Without the 0.3 scaling used for Xu et al.’s alternating quantizer, ProxQuant achieves better PPW than the unscaled ALT straight-through method.The scaled method’s reported PPW values without scaling are 116.7, 94.3, and 87.3 for 1, 2, and 3 bits, respectively.Source: §5 · p. 9

5.1 A convergence theorem for ProxQuant

• With a β-smooth loss, differentiable regularizer, and bounded-below composite objective, ProxQuant has a convergence guarantee using batch gradients and step size η = 1/(2β).The guarantee assumes constant regularization strength; the paper notes that the quantization regularizer can be replaced by a differentiable, point-wise arbitrarily close smoothing.Source: §5.1 · p. 10

5.2 Non-convergence of lazy prox-gradient

• Lazy prox-gradient can oscillate between two non-stationary points for some initialization, despite satisfying the assumptions under which ProxQuant converges.The paper gives a simple one-dimensional example using a smoothed W-shaped regularizer.Source: §5.2 · p. 10

6 Conclusion

• ProxQuant is proposed and experimentally evaluated as a training method for quantized networks.The paper describes it as a powerful alternative to the straight-through gradient method with theoretically better convergence properties.Source: §6 · p. 11
• The conclusion reports that ProxQuant has theoretically better convergence properties than the straight-through gradient method.Source: §6 · p. 11
• Future work includes alternative regularizers for ternary and multi-bit ProxQuant and experiments on larger tasks.Source: §6 · p. 11

A.1 Prox operators for binary nets

• The squared-L2 binary regularizer has a corresponding prox operator, given in vector form by θ⋆ = (θ + λsign(θ))/(1 + λ).Source: §A.1 · p. 14
• For binary quantization, the L1-version prox preserves the input sign and maps θ to (θ + λsign(θ))/(1 + λ).The coordinate-wise solution chooses the sign matching θ because it gives a smaller quadratic term.Source: §A.1 · pp. 13–14

A.2 Prox operator for ternary quantization

• For ternary quantization, the approximate alternating prox operator uses two rounds of computation and extends the TWN quantizer to allow different positive and negative levels.The paper reports that two rounds achieve good performance and are used in its experiments.Source: §A.2 · p. 14

B.1 Ternary quantization for CIFAR-10

• Ternarized ResNet training starts from pretrained full-precision models, hard-quantizes at epoch 400, and continues to epoch 600 to stabilize BatchNorm.Source: §B.1 · p. 14
• The best of four ProxQuant runs is slightly better than TTQ for ternarized ResNets on CIFAR-10, while the results are otherwise comparable.Table 3 reports top-1 classification error as mean(std) over four runs and also gives the best-of-four result for PQ-T.Source: §B.1 · p. 15, Table 3 · p. 15

C Sign change experiment

• SignChange is the proportion of differing signs between parameters, a rescaled Hamming distance measuring binarization closeness.The experiment tracks this metric along training and from initialization to the final binary model.Source: C Sign change experiment · p. 15
• ProxQuant has lower sign changes and higher performance than BinaryConnect across layers, warm starts, and repeated runs.Figure 3 compares the methods over four runs from the same full-precision ResNet-20 initialization.Source: Figure 3 · p. 15, C Sign change experiment · p. 16
• ProxQuant-Ternary and TTQ are not strictly comparable: ProxQuant uses better initializations, while TTQ trains its quantization levels.ProxQuant’s quantizer instead pre-computes its levels from the current full-precision parameter.Source: C Sign change experiment · p. 15

C.1 Raw data for sign change experiment

• Table 4 reports mean (std) and Table 5 raw values for ResNet-20 performance and whole-network sign changes across three initializations and four runs.Sign changes cover all quantized parameters in the network.Source: Table 4 · p. 16, Table 5 · p. 16

• Here, β-smoothness means the loss is differentiable and its gradient is β-Lipschitz.Source: §D.1 · p. 17
• The best iterate index can be monitored in practice through the proximity ||θt−θt−1||₂.Source: §D.1 · p. 17
• The proof gives an ergodic stationarity guarantee, bounding min_{0≤t≤T−1} ||θt+1−θt||² by 18β(Fλ(θ0)−F⋆)/T.The bound follows from the proximity guarantee under the smoothness assumption.Source: §D.1 · p. 17

D.2 Proof of Theorem 5.2

• The lazy prox-gradient counterexample uses a quadratic loss, a smoothed W-shaped regularizer, and a constant stepsize η ≤ 1/2.Source: §D.2 · p. 18
• In that example, lazy prox-gradient oscillates between two points that are not stationary, so its iterates do not approach stationarity in the ergodic sense.Source: §D.2 · p. 18

D.3 Proof of Theorem 5.3

• A BinaryConnect sign pattern is a fixed point when every nonzero gradient component at that sign points against the sign.The proof states the condition as sign(∇L(s)[i]) = −s[i] for each nonzero component.Source: §D.3 · p. 19