Source-linked AI summary
SETAREH Transform: A Sorting-Assisted Haar-Wavelet Sparsification Method for Approximate Inner-Product Computation
Seyed Kamaledin Setarehdan
TL;DR
Reducing model size and computation has received substantial attention, while the cost of repeatedly handling high-dimensional inputs for fixed-weight inner products remains a complementary concern. SETAREH applies shared permutations and Haar transforms before truncating transformed pairs; in the evaluated age-estimation task, 75 retained pairs yielded 0.13283% test CMRE. The practical scope depends on metadata storage and does not establish lower computation or end-to-end latency or energy.
Problem
Method
SETAREH applies the same permutation and Haar transforms to fixed coefficients and inputs, preserving the exact inner product before truncating transformed coefficient–input pairs.
Source: §2 · p. 5, §5 · p. 19Results
0.13283% test CMRE was obtained with 75 retained coefficients versus 4096 original coefficients on the evaluated regression task.
Source: §5 · p. 19Takeaways & Limitations
SETAREH demonstrates structured reduction of retained coefficient–input pairs for repeated inner products with fixed weights, without modifying the underlying network architecture or learned parameters.
Source: §5 · pp. 18–19The reported coefficient-only ratios exclude permutation storage, and the study does not establish reduced arithmetic computation, end-to-end latency, or energy consumption.
Source: §5 · pp. 19–20
Abstract
from arXiv · showhide
Modern deep learning systems increasingly rely on high-dimensional vector operations, in which large input and parameter vectors are repeatedly stored, transferred, and processed. Although substantial research has focused on reducing neural-network model size through pruning, quantization, knowledge distillation, and efficient architectures, the storage, memory-traffic, and communication costs associated with repeatedly presented high-dimensional input vectors remain important practical considerations, particularly in distributed and edge-computing environments. This paper proposes the SETAREH (Sorting-assisted Energy-concentrating Transform for Approximate Representation using Haar wavelets) transform, a Haar-wavelet-based sparsification approach for approximating the inner product between a fixed high-dimensional coefficient vector and successive input vectors. By applying the same sequence of permutations and Haar transforms to both vectors, SETAREH preserves the exact inner product before truncation. It then retains only a subset of the transformed coefficient-input pairs, reducing their representation cost while introducing a controllable approximation error. To evaluate SETAREH, we consider two complementary experimental settings: a synthetic Gaussian-vector baseline and a real apparent-age estimation task using a convolutional neural network on the APPA-REAL dataset. The results show that SETAREH can approximate the evaluated inner products using substantially fewer transformed coefficient-input pairs than the original vector dimension. The findings support its use as a potential structured data-representation method for repeated inner products with fixed weights, while the practical benefit depends on approximation accuracy and the cost of storing or transmitting permutation metadata.
1. Introduction
Neural-network efficiency work often targets model size, computation, or parameter and activation representations, while repeatedly supplied inputs to inner products pose a related data-reduction problem. SETAREH explores reducing those inputs by reordering and Haar-transforming a fixed coefficient vector, then approximating the inner product from a subset of transformed pairs.
1. Introduction
- Existing efficiency approaches primarily target model size, computational complexity, or the representation cost of parameters and activations.Related approaches include architectural design, pruning, quantization, knowledge distillation, and model compression.Source: §1 · pp. 2–3
- Repeated inner products with a fixed coefficient vector motivate reducing the input data stored, transferred, or processed for each operation.The setting includes fully connected layers and linear projections in convolutional and attention-based architectures.Source: §1 · p. 3
- SETAREH exploits inner-product permutation invariance by reordering a fixed coefficient vector and applying a Haar Discrete Wavelet Transform to compact its transform-domain representation.It approximates the inner product using a small subset of corresponding input samples rather than compressing the model itself.Source: §1 · p. 3
- SETAREH is presented as complementary data reduction with potential applications in memory-constrained processing and communication- or bandwidth-constrained neural-network systems.The paper investigates SETAREH-3 and SETAREH-5, which differ in their sorting and Haar-transform procedures.Source: §1 · p. 3
2. SETAREH Transform Framework
SETAREH applies shared permutations and Haar transforms to a fixed coefficient vector and its associated input, preserving their inner product until transformed pairs are truncated. It evaluates approximation error with conditioned relative-error statistics and separates offline coefficient preparation from online processing of successive inputs.
2. SETAREH Transform Framework
- The SETAREH objective is to approximate an inner product with substantially fewer than N coefficient–input pairs when the coefficient vector is fixed or changes infrequently.This targets repeated applications of the same linear operation to many input vectors.Source: §2 · p. 4
- Applying the same permutation and orthonormal Haar transform to both vectors preserves the exact inner product; approximation enters only when transformed pairs are truncated.Sorting places similar coefficient values together, reducing abrupt local variations and shaping the wavelet-coefficient distribution.Source: §2 · pp. 4–5
- SETAREH-3 and SETAREH-5 use different sequences of shared sorting and Haar-transform stages before retaining the first K transformed pairs.The variants are formulated from stage-specific permutations, and truncation yields approximate inner products.Source: §2 · pp. 4–5, §2.1 · p. 6
2.1 Approximate Inner-Product Computation and Performance Measures
- The study uses conditioned median relative error (CMRE), excluding the lowest 10% of trials ranked by exact-inner-product magnitude before aggregating the remaining errors.This conditioning limits domination by cases where the exact inner product is near zero and relative error can become disproportionately large.Source: §2.1 · p. 6
- Coefficient preparation can be done offline for a fixed vector, while each new input is transformed online using two permutations and one Haar transform for SETAREH-3 or three permutations and two transforms for SETAREH-5.The variants trade different online computational requirements against potentially different accuracy and retained-coefficient counts.Source: §2.1 · pp. 6–7
3. Experimental Methodology
The evaluation pairs a synthetic Gaussian-vector baseline with an apparent-age estimation task using VGG16 features and a learned regression vector. Its metrics track retained coefficient–input pairs and compare SETAREH variants at fixed or target approximation error.
3.1 Experimental Settings
- The synthetic experiment compares SETAREH-3 and SETAREH-5 across six dyadic dimensions from 1024 to 65536, using paired inputs and coefficients.Fifty coefficient vectors and 50 inputs per vector yield 2500 inner products.Source: §3.1 · pp. 7–8
- The real-data experiment uses a 4096-dimensional regression vector and fc2 features from pretrained VGG16 for APPA-REAL apparent-age estimation.Validation and test image sets are used for the feature vectors.Source: §3.1 · p. 8
3.2 Data Reduction and Compression Metrics
- The coefficient-only compression ratio compares original dimension N with retained pair count K, excluding permutation-information storage and therefore not representing total model-storage compression.Variant robustness is assessed through error-reduction factors, minimum K at a target CMRE, and the corresponding coefficient-only ratios.Source: §3.2 · p. 8
4. Results and Discussion
The supplied passage introduces the presentation of results for synthetic and real scenarios but reports no findings.
4. Results and Discussion
- Results are presented for two evaluation settings based on synthetic and real scenarios.The passage does not state the outcomes.Source: §4 · p. 9
4.1 The Results of the Synthetic Baseline
In the synthetic Gaussian baseline, approximation error generally falls as more coefficient–input pairs are retained, and SETAREH-5 shows a stronger advantage at moderate retention. Across independent coefficient vectors and prescribed error targets, it is more consistent and often reaches the same accuracy with fewer retained pairs than SETAREH-3.
4.1 The Results of the Synthetic Baseline
- Across the investigated dimensions, error decreased as K increased for both variants, while SETAREH-5 did not consistently outperform SETAREH-3 at K=10.At N=4096 and K=10, the conditioned median relative errors were 14.11% for SETAREH-3 and 14.80% for SETAREH-5; the paper treats this extreme-retention setting as a stress test.Source: §4.1 · pp. 9, 11, 12
4.1.1 Accuracy Improvement Achieved by SETAREH-5
- At N=4096 and K=100, SETAREH-5's conditioned median relative error was 0.0623% versus 1.59% for SETAREH-3, a 25.5× reduction.The comparison is part of the synthetic-baseline results.Source: §4.1 · pp. 9–10, §4.1.1 · p. 12
4.1.2 Robustness across Independent Coefficients Vectors
- Across independent coefficient vectors, SETAREH-5's win rate ranged from 98–100% at K=20 and was 100% at K≥30 for every tested dimension.At K=10, its win rate ranged from 22% to 44%, depending on vector dimension.Source: §4.1.2 · p. 12
4.1.3 Accuracy and Coefficient-Level Compression at Prescribed Error Levels
- For N=4096, reaching a conditioned median relative error below 5% required K=30 with SETAREH-3 and K=20 with SETAREH-5; for a 1% target, the respective K values were 200 and 30.At the 1% target, the reported coefficient-level compression ratios were 20.48 for SETAREH-3 and 136.5 for SETAREH-5, approximately 6.67 times higher for SETAREH-5.Source: §4.1.3 · p. 13
4.2. APPA-REAL Apparent-Age Regression: Model Construction and SETAREH Evaluation
The APPA-REAL evaluation applies SETAREH to a trained apparent-age Ridge model by converting its standardized-feature pipeline into an equivalent linear model on VGG16 fc2 features. Retaining a small coefficient subset closely approximates the model’s dot product, with CMRE measuring truncation error.
4.2.1. Feature Representation and Regression Model
- The 4096-dimensional VGG16 fc2 features feed a Ridge regression model trained on APPA-REAL images for apparent-age estimation.The dataset splits contain 4,113 training, 1,500 validation, and 1,978 test images; feature standardization uses training-set means and scales applied unchanged to validation and test sets.Source: §4.2 · p. 14, §4.2.1 · p. 14
- The exact Ridge model’s test MAE is 12.0018 years and test RMSE is 16.2052 years, establishing the prediction-error baseline for approximate-model comparisons.Its validation MAE is 8.9015 years and validation RMSE is 11.3421 years; these are prediction errors, distinct from SETAREH’s additional approximation deviations.Source: §4.2.1 · pp. 14–15
4.2.2. Conversion to an Equivalent Linear Model
- Combining standardization and Ridge regression yields an equivalent linear model on original fc2 features, isolating the approximated dot product while leaving the intercept unchanged.The transformed model’s predictions agree with the original pipeline to numerical precision.Source: §4.2.2 · p. 15
4.2.3. SETAREH Approximation Results
- SETAREH evaluates approximation error through CMRE against the original dot product; maximum absolute prediction deviation instead measures the difference between approximate and exact model predictions in years.The evaluation uses validation and test results.Source: §4.2.3 · p. 16
- At K = 200, retaining 4.88% of coefficients yields 0.0079891% test CMRE; K = 75 and K = 100 yield 0.13283% and 0.060019% test CMRE, respectively.The table summarizes SETAREH approximation results for the APPA-REAL Ridge regression model.Source: §4.2.3 · p. 16
4.3. Amortized Storage Benefit for Multiple Input Vectors
SETAREH’s storage advantage grows when the same fixed weight representation is reused across input vectors, because its permutation metadata is stored once. In the APPA-REAL configuration, the representation is larger for one input but smaller from the second onward.
4.3. Amortized Storage Benefit for Multiple Input Vectors
- SETAREH stores permutation information once and reuses it across inputs, unlike the dense representation, which stores each complete input vector.The comparison considers memory required for dense and SETAREH representations across different numbers of input vectors.Source: §4.3 · pp. 17–18
- The fixed permutation metadata’s contribution decreases inversely with the number of input vectors, so its relative storage cost shrinks as the weight representation is reused.For the APPA-REAL configuration, the three stored permutations require approximately 24 KiB, while K = 75 requires approximately 0.43945 KiB per additional input vector.Source: §4.3 · p. 17
- Storage savings are approximately 22.3% at M = 2, 82.3% at M = 10, 95.8% at M = 100, and 97.1% at M = 1000.At M = 1, SETAREH is larger than the dense representation because of the one-time permutation overhead.Source: §4.3 · p. 18
5. Conclusions
SETAREH uses permutation invariance and Haar transforms to sparsify repeated inner products with fixed coefficients, with approximation introduced by truncation. Experiments report close approximation with small retained subsets, while practical storage and computational benefits remain conditional.
Method
- SETAREH reorders fixed coefficients and applies Haar transforms so a small subset of transformed coefficient–input pairs can approximate inner products without changing network parameters.The method targets repeated use of a fixed or slowly changing coefficient vector.Source: §5 · p. 19
Variant comparison
- SETAREH-5 generally achieved lower CMRE than SETAREH-3 for K≥20, with a consistent advantage for K≥30 across tested dimensions.The paper selected SETAREH-5 as its main algorithm.Source: §5 · p. 19
Real-data evaluation
- 75 of 4,096 coefficients (1.83%) yielded 0.13283% test CMRE, while 200 (4.88%) yielded 0.0079891%.The evaluation used 4,096-dimensional APPA-REAL features and a fixed Ridge regression model.Source: §5 · p. 19
Scope and limitations
- Reported coefficient-only ratios exclude permutation storage, and the study demonstrates data reduction rather than arithmetic acceleration.Sorting and Haar transforms add operations; end-to-end latency and energy were not evaluated.Source: §5 · pp. 19–20
Storage analysis
- Storage was 52.7% higher for one input, then savings reached 22.3%, 82.3%, 95.8%, and 97.1% for 2, 10, 100, and 1,000 inputs.The analysis attributes the pattern to amortizing permutation metadata across inputs sharing fixed coefficients.Source: §5 · p. 20
Future work
- Applications beyond neural networks remain potential directions requiring dedicated validation, while metadata representation and hardware-aware cost studies remain future work.Suggested studies include end-to-end storage and communication, latency, memory traffic, and energy.Source: §5 · p. 20