Source-linked AI summary
Neuron merging via inverse-activation regression for post-training compression of sigmoid neural networks
Ao Kuniya, Jun Ohkubo
TL;DR
Pruning can discard useful information, motivating the question of what to retain when trained-network neurons are consolidated. The paper proposes cluster-based merging for sigmoid networks using inverse-activation least-squares reconstruction, and reports that weights help clustering while activation information helps reconstruction in the tested networks.
Problem
Structured pruning removes neurons that may still carry useful information, raising the question of what information to retain when trained-network neurons are consolidated.
Source: §I · p. 1Method
Results
Weight-based clustering with inverse-activation reconstruction maintains high accuracy across compression ratios, while activation information helps reconstruction and randomly generated inputs outperform pruning at tested high compression.
Source: §1 · p. 6, B. Training-data-free approach with… · p. 7, §V · p. 8Takeaways & Limitations
In the tested sigmoid networks, weight information is useful for clustering and activation information for representative-neuron reconstruction.
Source: §V · p. 8The inverse-space regression is a linear surrogate for activation-space reconstruction, not an exact minimizer of the original task's activation mean squared error.
Source: §2 · p. 5
Abstract
from arXiv · showhide
As neural networks continue to grow in scale, model compression is becoming increasingly important for efficient inference under limited computational resources. Structured pruning methods remove neurons or channels that are estimated to be less important, but the removed units may still contain useful information. From the viewpoint of coarse-graining a trained network, it is valuable to ask which information should be retained when multiple neuronal degrees of freedom are consolidated. In this paper, we discuss cluster-based merging methods for compression of trained neural networks. In addition to a data-free contribution-weighted averaging method, we propose neuron-merging methods in which neuron responses are mapped back to the pre-activation space via the inverse activation function, and the weights and biases of each representative neuron are estimated using the least-squares method. We also examine both a data-assisted strategy with actual training inputs and a data-free strategy using randomly generated inputs. The comparisons provide empirical evidence, in the tested sigmoid networks, that weight information is particularly useful for clustering whereas activation information is useful for representative-neuron reconstruction in the merging process.
I. INTRODUCTION
Compression matters when memory, computation, or inference latency is limited, but pruning can discard information carried by removed neurons. This paper proposes cluster-based merging methods for sigmoid networks to examine how weights and activation values contribute to neuron selection and reconstruction.
I. INTRODUCTION
- Compression is motivated by growing inference costs when memory, computation, or latency is constrained.The introduction also notes computational and energy costs associated with training and inference.Source: §I · p. 1
- The paper proposes cluster-based merging methods to test which weight and activation information is useful for neuron selection and reconstruction.Representative weights and biases are estimated by ridge-regularized least squares after mapping sigmoid activations to pre-activation space with the logit function.Source: §I · p. 1
- The proposed variations, including data-assisted and training-data-free methods, outperform structured pruning in the tested sigmoid networks.The paper also examines the respective roles of weight matrices and activation-function values in merging.Source: §I · p. 2
A. Structured pruning
- Structured pruning reduces matrix sizes and keeps dense operations, but removed neurons may carry useful information.Unstructured pruning instead removes individual connections, producing irregular sparse matrices that may not speed inference without specialized implementations.Source: §I · p. 1, A. Structured pruning · p. 2
B. Demonstration of structured pruning
On MNIST, structured-pruning accuracy drops rapidly beyond slight compression in the tested networks. The paper motivates merging as a way to consolidate removed neurons' contributions and proposes data-free and activation-informed options for clustering and reconstruction.
B. Demonstration of structured pruning
- The MNIST baseline uses three sigmoid hidden layers of width 1024, removes equal numbers of neurons per hidden layer, and uses no fine-tuning.Ten independently seeded networks are evaluated on the 10,000 MNIST test examples.Source: B. Demonstration of structured pruning · p. 2
- Accuracy falls rapidly beyond slight compression across the tested structured-pruning methods.The comparison plots test accuracy against hidden-layer node-retention ratio for random, ℓ1-, and ℓ2-norm weight-magnitude pruning; the latter two are data-free.Source: B. Demonstration of structured pruning · p. 2, Figure 2 · p. 3
C. Neuron merging with ReLU function
- Kim et al.'s merging method transfers removed-neuron contributions through a scaling matrix, but its derivation relies on ReLU-specific conditions.Moving the scaling matrix outside the activation requires nonnegativity and sparsity conditions for ReLU.Source: C. Neuron merging with ReLU function · p. 3
III. PROPOSED METHODS
- The proposed framework combines data-free and activation-based clustering with weight averaging or inverse-sigmoid least-squares reconstruction.The data-dependent procedures use input data but do not require output labels.Source: §III · p. 3
A. Problem formulation
The formulation replaces disjoint clusters of neurons with representative neurons in a smaller dense layer. It introduces data-free clustering based on neuron parameters and assumes an invertible hidden activation on the range of interest.
A. Problem formulation
- The formulation assumes the hidden activation function φ is invertible on the range of interest.Source: A. Problem formulation · p. 3
- Each layer is divided into Kℓ<dℓ disjoint neuron clusters, with each cluster replaced by one representative neuron.The resulting layer remains a standard dense layer.Source: §1 · p. 3
1. Data-free clustering approach (C-weight)
- C-weight forms each neuron's feature vector by concatenating its input weights, bias, and output weights.Source: §1 · p. 4
- C-weight standardizes feature coordinates and applies conventional k-means clustering.The stated implementation uses scikit-learn.Source: §1 · p. 4
- Output weights were empirically useful for detecting neuron similarities, alongside input weights.Source: §1 · p. 4
2. Data-dependent clustering approach (C-activation)
C-activation constructs neuron features from activation values on a portion of the training data, then clusters those features.
2. Data-dependent clustering approach (C-activation)
- C-activation uses activation-function values produced by feeding a portion of the training data into the trained network.Source: §2 · p. 4
- The training examples used to construct the feature vectors are selected uniformly at random.Source: §2 · p. 4
- After feature construction, C-activation standardizes the features and selects neurons for merging with conventional k-means clustering.Other procedures match those in C-weight.Source: §2 · p. 4
C. Weight merging
M-av is a data-free merging method that uses contribution-aware averaging rather than uniform averaging to set representative-neuron parameters.
C. Weight merging
- M-av assigns contribution scores to neurons and uses normalized coefficients to determine representative-neuron weight parameters.Source: §1 · p. 4
- The contribution-aware averaging approach was chosen because preliminary experiments found it yielded better results than uniform averaging.Source: §1 · p. 4
- The method is named M-av and is based on Eqs. (10), (11), and (12).Source: §1 · p. 4
2. Data-dependent merging approach (M-logit)
M-logit reconstructs representative incoming weights and bias from data by mapping cluster responses through the sigmoid inverse and fitting them in pre-activation space. Its regression objective is a surrogate for activation-space reconstruction, and outgoing weights are averaged separately.
2. Data-dependent merging approach (M-logit)
- The method uses the sigmoid activation function in its data-dependent approach.Source: §2 · p. 4
- With many input examples, the logit-transformed relation estimates the representative incoming weights and bias as a least-squares problem.Source: §2 · pp. 4–5
- The method defines logit as the inverse sigmoid function and applies it to a clipped target, since its argument must lie between 0 and 1 for numerical stability.Source: §2 · p. 5
- M-logit defines a target response for each input as a convex combination of the cluster activations.Source: §2 · p. 5
- The regression combines the data matrix of preceding-layer activations and bias terms with a matrix of transformed targets.Source: §2 · p. 5
- The least-squares regression minimizes error in transformed pre-activation space, not activation mean squared error for the original task.It is therefore a linear surrogate for direct activation-space reconstruction.Source: §2 · p. 5
- Outgoing weights are merged by simple averaging, without using activation-function values.The authors avoid contribution-aware outgoing-weight merging in M-logit to isolate the role of activation values.Source: §2 · p. 5
- The target expression is accurate when neurons in a cluster have similar responses.Source: §2 · p. 5
D. Comments on the merging algorithms
Merging proceeds layer by layer, with clustering and incoming- and outgoing-weight merging applied sequentially. The procedure incurs a one-time compression cost but adds no inference time relative to a network with the same compressed architecture.
D. Comments on the merging algorithms
- Clustering and incoming- and outgoing-weight merging are applied sequentially to each hidden layer, from the first layer through the final one.In data-dependent cases, activations are evaluated on the input dataset in the first layer, then on the same dataset fed through the merged network at each subsequent layer.Source: D. Comments on the merging algorithms · p. 5
- Data-dependent merging uses the same input dataset to evaluate activations layer by layer in the progressively merged network.These activation values are used for clustering and merging at each layer.Source: D. Comments on the merging algorithms · p. 5
- Compression requires a one-time cost for activation collection, clustering, and regression, but adds no inference time beyond the compressed dense layers and original activations.The inference graph contains only those compressed layers and original activation functions.Source: D. Comments on the merging algorithms · p. 5
A. Experiments on MNIST
On MNIST, neuron merging outperforms the cited structured-pruning comparison, with data-assisted M-logit merging after weight-based clustering achieving the highest mean accuracy. The results distinguish the information useful for selecting clusters from that useful for reconstructing representative neurons.
A. Experiments on MNIST
- The MNIST experiments use standardized, flattened 784-dimensional images and ten independently trained networks; the reported accuracy is summarized across those networks.The tested combinations pair C-weight or C-activation clustering with M-av or M-logit merging, with Ncomp set to 10,000 for C-activation and M-logit.Source: A. Experiments on MNIST · p. 5, §1 · p. 6
1. Accuracy evaluation
- Merging methods outperform the pruning comparison, and C-weight/M-logit(data) achieves the highest mean accuracy among the tested methods.The figure compares test accuracy across hidden-layer node-retention ratios; C-weight/M-logit(data) combines data-free clustering with data-dependent merging.Source: Figure 4 · p. 6, §1 · p. 6
- At small node-retention ratios, data-free C-weight clustering yields better accuracy than C-activation clustering using actual data.The authors interpret this as evidence that weight information, which aggregates information, is preferable to data-dependent activation values when selecting clusters.Source: §1 · p. 6
- For representative-neuron merging, data-dependent least-squares reconstruction is preferable to a simple linear combination.The authors attribute this to least squares generating sigmoid functions effectively when sufficient data are available.Source: §1 · p. 6
2. One-time compression time
Across the Fashion-MNIST results, the preferred merging method varies with compression level, while randomly generated inputs also support useful data-free merging. The reported compression-time comparison concerns a one-time cost, and the authors identify clustering—not activation evaluation or least-squares regression—as the main time burden in the stated comparison.
2. One-time compression time
- Merging takes longer than pruning, but the authors consider the difference not significant in practice because compression is performed once and the methods share node counts and dense-layer inference costs at a given retention ratio.The reported compression time includes clustering and merging; for C-activation and M-logit it also includes input processing and activation evaluation.Source: §2 · p. 6
- For C-weight/M-logit(data), k-means clustering takes more time than evaluating activations; the authors attribute this to the larger feature vectors used by C-activation, while least-squares regression is simple.The feature size in C-activation is determined by Ncomp and exceeds that of C-weight in the stated experimental settings.Source: §2 · p. 6
B. Training-data-free approach with randomly generated data
- With data-free MNIST procedures, activation values perform worse for clustering but improve performance when used for neuron merging.Figure 6 compares test accuracy across node-retention ratios.Source: Figure 6 · p. 7, B. Training-data-free approach with… · p. 7
- The data-free variant supplies randomly generated input vectors to trained networks to obtain activation values for C-activation clustering and M-logit merging.The random inputs are drawn from a standard normal distribution, with d0 denoting the input dimension.Source: B. Training-data-free approach with… · p. 7
- On Fashion-MNIST, C-weight/M-av performs well at moderate compression, while C-weight/M-logit is superior at high compression; randomly generated-data merging also achieves sufficient accuracy.The figure separates conventional, data-free, and data-dependent methods from training-data-free methods using random datasets, repeating C-weight/M-av as a reference.Source: Figure 7 · p. 7
- On MNIST, standard-normal random inputs yield a sufficiently well-conditioned design matrix for useful representatives despite the distribution mismatch; the authors also report worse performance than with actual data and emphasize the importance of data-distribution information.They attribute the poor clustering performance of C-activation to its lacking some information about the focused neurons.Source: B. Training-data-free approach with… · p. 7
C. Additional evaluation on Fashion-MNIST
On Fashion-MNIST, the results are less distinct but consistent with MNIST. The conclusion reports high accuracy across compression ratios for inverse-activation regression and distinguishes weight-based clustering from activation-based reconstruction.
C. Additional evaluation on Fashion-MNIST
- At moderate compression, data-free C-weight/M-av yields good results, while at high compression C-weight/M-logit performs better; randomly generated inputs also achieve sufficient accuracy.Fashion-MNIST results are less distinct than MNIST, but follow the same trend.Source: C. Additional evaluation on… · p. 8
V. CONCLUSION
- The framework reconstructs representative neurons by least squares in inverse-activation space; its sigmoid-network instance uses the logit function without fine-tuning.The evaluation uses the same architecture as MNIST and reports test accuracy because compression-time trends were similar.Source: C. Additional evaluation on… · p. 7, §V · p. 8
- Inverse-activation regression with weight-based clustering maintains high accuracy across compression ratios, while random inputs retain substantially higher accuracy than pruning at tested high-compression settings.The conclusion attributes clustering to weight information and representative-neuron reconstruction to activation information.Source: §V · p. 8