PaperlayerView exact revision on arXiv

Source-linked AI summary

Deep $k$-Means: Re-Training and Parameter Sharing with Harder Cluster Assignments for Compressing Deep Convolutions

Junru Wu, Yue Wang, Zhenyu Wu, Zhangyang Wang, Ashok Veeraraghavan, Yingyan Lin

arXiv:1806.09228v1cs.LGcs.CVstat.ML

TL;DR

Resource-constrained devices make it difficult to deploy increasingly complex CNNs, while reducing model size alone does not capture energy use. Deep k-Means retrains convolutional weights with a spectrally relaxed k-means regularizer, then compresses them through weight sharing. The authors report favorable energy efficiency and higher accuracy at the same compression ratio, with a limitation in directly verifying CAR’s total energy consumption.

  • Problem

    Edge devices have limited energy, computation, and storage, while CNN compression must address energy because weight and operation counts do not capture data-movement costs.

    Source: §1 · pp. 1–2

  • Method

    Deep k-Means encourages clustered convolutional weights during retraining with a spectrally relaxed k-means regularizer, then compresses them by storing cluster centers and assignment indexes.

    Source: §1.1 · p. 2, §2.1 · p. 3

  • Results

    The authors report that Deep k-Means consistently achieves higher accuracy at the same compression ratio as competing convolutional compression methods and shows favorable energy efficiency.

    Source: §1.1 · p. 2

  • Takeaways & Limitations

    Deep k-Means is presented as a convolution-focused compression pipeline with reported benefits in both compression performance and energy efficiency.

    Source: §1.1 · p. 2, §5 · p. 8

    The authors could not directly verify CAR’s total energy consumption with their metrics because CAR’s model parameters or pretrained model were unavailable.

    Source: §5 · p. 8

Abstract

from arXiv · show

The current trend of pushing CNNs deeper with convolutions has created a pressing demand to achieve higher compression gains on CNNs where convolutions dominate the computation and parameter amount (e.g., GoogLeNet, ResNet and Wide ResNet). Further, the high energy consumption of convolutions limits its deployment on mobile devices. To this end, we proposed a simple yet effective scheme for compressing convolutions though applying k-means clustering on the weights, compression is achieved through weight-sharing, by only recording $K$ cluster centers and weight assignment indexes. We then introduced a novel spectrally relaxed $k$-means regularization, which tends to make hard assignments of convolutional layer weights to $K$ learned cluster centers during re-training. We additionally propose an improved set of metrics to estimate energy consumption of CNN hardware implementations, whose estimation results are verified to be consistent with previously proposed energy estimation tool extrapolated from actual hardware measurements. We finally evaluated Deep $k$-Means across several CNN models in terms of both compression ratio and energy consumption reduction, observing promising results without incurring accuracy loss. The code is available at https://github.com/Sandbox3aster/Deep-K-Means

1. Introduction

The paper targets the gap between resource-constrained edge devices and CNNs whose parameters increasingly reside in convolutional layers, while compression must address energy as well as model size. It proposes retraining and weight sharing for convolutional layers and energy-aware metrics.

1. Introduction

  • Many recent CNNs shift parameters toward convolutional layers, yet few compression techniques target them; model size and operation counts also do not reliably predict energy use.Energy estimates depend heavily on memory data movement; SqueezeNet has 51.8× fewer weights than AlexNet but consumes 33% more energy.Source: §1 · p. 2

1.1. Contribution

  • Deep k-Means combines spectrally relaxed k-means regularization during retraining with weight sharing to compress convolutional layers.It stores cluster centers and weight-assignment indexes; the regularizer encourages harder assignments of weights to learned centers.Source: §1.1 · p. 2
  • The paper introduces energy-aware metrics that include data movement and reports estimates consistent with a tool extrapolated from hardware measurements.It also reshapes weights into row vectors for clustering, a formulation intended to favor energy-efficient hardware implementation.Source: §1.1 · p. 2

1.2. Related Work

  • Prior work includes pruning, scalar weight sharing, and clustering for fully connected layers, while comparatively few methods compress convolutional layers.Examples include Deep Compression, FreshNets [Chen et al., 2016a], and iterative filter pruning [Abbasi-Asl & Yu, 2017].Source: §1.2 · p. 2

2. Proposed Approach

Deep k-Means clusters convolutional-filter rows for parameter sharing and adds a spectrally relaxed k-means regularizer during retraining to encourage weights suited to that compression. The paper contrasts this harder-assignment approach with Bayesian soft weight sharing.

2.1. Parameter Sharing via Row-wise k-Means

  • For row-wise sharing, the method treats convolutional-filter rows as samples, clusters them into K groups, and stores cluster indexes and codebooks when K ≪ N.For multiple layers, it uses a uniform cluster rate for simplicity, except that the first layer often receives a higher rate.Source: §2.1 · p. 3
  • Row-wise k-means is motivated by data reuse under the row-stationary dataflow and by reduced complexity compared with alternative weight reshaping schemes.The alternatives are reported to perform similarly when their K values are chosen appropriately.Source: §2.1 · p. 3

2.2. k-Means Regularized Re-Training

  • Retraining adds a regularizer to the original energy objective so convolutional weights remain suited to k-means sharing, rather than clustering a static weight matrix.The regularizer uses an auxiliary orthogonal F; SGD updates W with gradient ∇E(W)+λW(I−FF^T), and F is updated using a truncated SVD.Source: §2.2 · p. 3
  • The interaction of F and W during retraining tends to produce weights near K unique vector values, encouraging approximately hard cluster assignments while optimizing inference accuracy.The pipeline starts from a pretrained model, applies the regularizer to convolutional layers, then performs row-wise k-means; the F update is computed once every five epochs in practice.Source: §2.2 · pp. 3–4

2.3. Comparison with Existing Work

  • Compared with Bayesian soft weight sharing, Deep k-Means derives harder assignments from the k-means objective, which the authors report yields more skewed weight distributions and better results.The approach adds a regularizer with little extra complexity; the authors report insensitivity to λ between 10−4 and 10−3 and no need for special learning-rate scheduling or postprocessing.Source: §2.3 · p. 4

3. Energy-Aware Metrics for CNN Energy Consumption Estimation

The section argues that compression ratio and operation counts do not reliably capture CNN energy use, because data movement also matters. It proposes computational and separate weight and activation representational costs to better characterize that cost.

3. Energy-Aware Metrics for CNN Energy Consumption Estimation

  • Compression ratio and operation reductions are not necessarily tied to energy efficiency, while CNN energy also depends on data movement and memory hierarchy.Energy is difficult to estimate directly from a model because data movement depends on the implementation’s memory hierarchy and dataflow.Source: §3 · p. 4
  • Computational cost estimates the resources needed for one decision using the number of 1-bit full adders required under ripple-carry-adder and Baugh-Wooley multiplier assumptions.The dot-product calculation depends on the fixed-point precisions assigned to weights and activations.Source: §3 · p. 4
  • Weight representational cost combines the bits needed to represent all weights with the number of times those weights are used in convolutions.The paper modifies representational cost to account for weight-use frequency and separates weight and activation costs.Source: §3 · pp. 4–5
  • Activation representational cost is defined analogously to weight representational cost, with activation-use frequency and activation indices describing its scope.The modified definition captures that removing a filter can reduce activation loads, reflected in Nx.Source: §3 · p. 5

4. Experiments

The experiments compare Deep k-Means with convolution-compression baselines using compression-ratio and energy-aware evaluations. On TT-conv-CNN, it shows minimal accuracy loss at CR=4 and a slight accuracy increase at CR=2.

4. Experiments

  • The experiments compare compression ratio and resulting accuracy loss, focusing on convolutional layers unless otherwise specified, and also evaluate energy-aware metrics against compression baselines.The authors validate their energy estimates against extrapolations from hardware measurements and describe the metrics as a way to examine design trade-offs.Source: §4 · p. 5

4.1.1. COMPARISON WITH ULTIMATE TENSORIZATION

  • Deep k-Means is evaluated on TT-conv-CNN, whose uncompressed model has 90.7% top-1 accuracy on CIFAR-10 and whose parameters are 99.54% convolutional.The model has six convolutional layers and one fully connected layer.Source: §4.1.1 · p. 5
  • On TT-conv-CNN, Deep k-Means at compression ratios (CR) 2 and 4 slightly increases accuracy after compression at CR=2 and incurs minimal accuracy loss even at CR=4.Source: §4.1.1 · p. 5

4.1.2. COMPARISON WITH FRESHNET

  • Table 2 presents compression results for FreshNet-CNN, a model introduced in the FreshNet comparison.The supplied passages identify the table’s model scope but do not state its result values.Source: Table 2 · p. 6

. 4.1.3. COMPARISON WITH GREEDY FILTER PRUNING

Deep k-Means outperforms CAR at high layer-wise compression, yields a smaller accuracy drop than soft weight-sharing at CR = 45, and is nearly lossless on GoogLeNet at CR ≤3. Its energy estimates strongly correlate with those from Yang et al.’s tool.

. 4.1.3. COMPARISON WITH GREEDY FILTER PRUNING

4.1.4. COMPARISON WITH SOFT WEIGHT SHARING

  • At CR = 45, Deep k-Means has a 1.63% top-1 accuracy drop versus 2.02% for soft weight-sharing, with accuracy decreasing smoothly at CR = 47 and 50.This comparison uses sparsity promotion for Deep k-Means.Source: Table 3 · p. 6, §4.1.4 · p. 7

4.1.5. EVALUATION WITH GOOGLENET ON IMAGENET

  • At CR ≤3, GoogLeNet compression is almost lossless and outperforms one-shot compression and low-rank regularization; performance deteriorates quickly when CR >4.Top-5 errors slightly increase after compression.Source: Table 4 · p. 7, §4.1.5 · p. 7

4.2.1. ENERGY-AWARE METRICS VERIFICATION

  • R2 was 0.9931 for AlexNet and 0.9675 for GoogLeNet_v1, indicating strong linear correlation with Yang et al.’s hardware-measurement-extrapolated tool.The proposed metrics are described as generally applicable to any CNN.Source: §4.2.1 · p. 7

4.2.2. COMPARISON WITH GREEDY FILTER PRUNING

  • Compared with CAR, Deep k-Means reduces weight representational cost by about 10% for Conv1 and 17.2% for Conv2 without accuracy loss, while CAR reduces activation cost more and computational savings depend on inputs.Deep k-Means’ computational cost reduction is either smaller than or equal to CAR’s, depending on the inputs.Source: Figure 2 · p. 8, §4.2.2 · p. 8

5. Conclusion and Discussions

The conclusion presents Deep k-Means as a convolution-compression pipeline and identifies adaptive layer rates and energy-aware regularization as future directions.

5. Conclusion and Discussions

  • Deep k-Means combines retraining with parameter sharing to compress convolutional layers, and the paper reports superiority over several competitive methods in compression ratio and energy efficiency.Future work will explore adaptive cluster rates across layers and energy-aware regularizations.Source: §5 · p. 8
  • Total energy for CAR could not be directly verified because its model parameters or a pretrained model were unavailable.Source: §5 · p. 8
  • The reported energy comparison excludes weight-assignment-index cost, which the authors consider negligible at achievable high compression ratios.Source: §5 · p. 8

Related papers

Loading 1806.09228v1…