PaperlayerView exact revision on arXiv

Source-linked AI summary

Weight Pair Encoding: Inducing a Smaller Grammar in Neural Network Weights

Irene Tallini, Daniele Solombrino, Alberto Cazzaniga, Emanuele Rodolà

arXiv:2609.31564v1cs.LG

TL;DR

Flat weight-compression schemes do not target recurring variable-length patterns nested within one another, and grammar size had not been used as an explicit training objective for network weights. WeightPE places lossy Re-Pair inside a straight-through estimator to train weights toward smaller grammars. On two CIFAR-10 ViTs, it yields substantially smaller Re-Pair grammars than int8 QAT with accuracy costs, and the reduction is also observed with other grammar compressors.

  • Problem

    Existing compact weight schemes use individual weights or fixed-size blocks and do not name nested variable-length recurring patterns; grammar size had not been an explicit training objective for network weights.

    Source: §1 · pp. 1–2

  • Method

    WeightPE flattens int8 weights into one string and uses lossy Re-Pair rewrites under a global L2 budget inside a straight-through estimator to train for smaller grammars.

    Source: §1 · p. 2, §3 · pp. 2–3

  • Results

    0.43× the int8 QAT Re-Pair grammar size on ViT-B/16 costs 1.9 accuracy points, while 0.37× on ViT-L/16 costs 1.1 points; SEQUITUR nearly matches Re-Pair rates and LZ78 reduces grammar size less.

    Source: §5 · p. 4

  • Takeaways & Limitations

    The grammar-size reduction is also observed with SEQUITUR and LZ78, compressors not used in the WeightPE perturbation during finetuning.

    Source: §5 · p. 4, §6 · p. 4

    Behavior across other model families, datasets, and quantization levels remains untested, and operator cost ranges from negligible to around 5× depending on parameters.

    Source: §6 · p. 4

Abstract

from arXiv · show

We show that neural network weights can be explicilty fintuned to admit a smaller grammar. Weight Pair Encoding (WeightPE) does so by placing a lossy Re-Pair compressor inside a straight-through estimator. The int8 weights of the network are flattened into one string, and near-matching Re-Pair patterns are made exactly equal within a global L2 budget. The network computes with the rewritten weights and trains through them with a straight-through estimator. Unlike a flat codebook of fixed-size entries, a grammar offers variable-length patterns and reuses them hierarchically inside larger ones. On the MLP weights of ViT-B/16 and ViT-L/16 finetuned on CIFAR-10, WeightPE produces a Re-Pair grammar 0.43x and 0.38x the size of the one produced by an equivalent int8 QAT run, at a cost of 1.9 and 1.1 accuracy points. The trend extends to different grammar compressors (LZ78, SEQUITUR), over which the networks has not be finetuned against. To our knowledge, this is the first time grammar size has been used as an explicit training objective for network weights.

1 Introduction and Related Works

The introduction motivates grammar structure as a way to represent hierarchical repetitions and identifies a gap in weight compression: prior flat representations do not target variable-length nested patterns during training. WeightPE makes grammar size an explicit training objective and reports smaller Re-Pair grammars on two CIFAR-10 ViTs.

1 Introduction and Related Works

  • Grammar structure exposes repetitions hierarchically and supports random access proportional to compressed size and direct computation over the compressed representation.The smallest grammar is an expressive, computable approximation of Kolmogorov complexity, although finding it is NP-hard and heuristics have worst-case guarantees.Source: §1 · p. 1
  • Existing compact weight representations use flat codebooks, clustering, or objective-based schemes, whose atoms are individual weights or fixed-size blocks rather than nested variable-length substrings.The authors state that, to their knowledge, grammar size had not previously been an explicit training objective for network weights.Source: §1 · pp. 1–2
  • WeightPE trains network weights to admit a smaller grammar, producing Re-Pair grammars 0.43× and 0.38× the equivalent int8 QAT size on ViT-B/16 and ViT-L/16, at accuracy costs of 1.9 and 1.1 points.The reported results are for MLP weights on models finetuned on CIFAR-10.Source: §1 · p. 2

2 Background

The background describes Re-Pair grammar construction and straight-through estimation, the gradient mechanism used to train through a transformed weight matrix.

2 Background

  • Re-Pair repeatedly replaces the most frequent adjacent pair with a non-terminal until no pair appears twice; its grammar size is |s| + 2|G| symbols.The paper reports this size measure for a rule set G over residual sequence s.Source: §2 · p. 2
  • Straight-through estimation computes the forward pass using transformed weights while treating the transformation displacement as constant with respect to the original weights.This lets the gradient reach the original weights unchanged, as if the transformation were the identity.Source: §2 · p. 2

3 Method

WeightPE applies lossy Re-Pair to quantized, serialized network weights, rewriting near-matching patterns under a global L2 distortion budget before building shared grammar rules. The perturbed weights are trained through a straight-through estimator.

3 Method

  • WeightPE applies a lossy Re-Pair perturbation to weights and trains through the resulting transformed weights using a straight-through estimator.The operator quantizes, serializes the network into one string, and compresses it; the formulation is noted as a rate–distortion problem.Source: §3 · p. 2
  • Each int8-quantized weight matrix is flattened row-major and concatenated into one string, allowing patterns from one block to be reused in another while preventing rules from crossing rows.The row-wise quantization is cited to Krishnamoorthi (2018).Source: §3 · p. 3
  • REWRITE maps adjacent-pair occurrences onto frequent cluster leaders under the global L2 distortion budget, then lossless MERGE turns the newly identical repeated pairs into shared rules.Rewriting proceeds cheapest-first over a non-overlapping set; once the budget is spent, subsequent rounds use plain Re-Pair.Source: §3 · p. 3
  • The perturbed string is expanded back into weights and trained through the estimator, with the perturbation operator applied every ten optimization steps.Source: §3 · p. 3

4 Experiments

Experiments finetune ViT-B/16 and ViT-L/16 on CIFAR-10, applying WeightPE to MLP projections and evaluating test accuracy alongside grammar sizes relative to an int8 QAT baseline.

4 Experiments

  • Experiments finetune ViT-B/16 and ViT-L/16 on CIFAR-10, restricting perturbation to the two MLP projections in each block, about two thirds of the parameters.Attention projections are quantized but not perturbed; patch embedding, normalization, and classification layers remain full precision.Source: §4 · p. 3
  • Top-1 accuracy on the CIFAR-10 test set is the downstream metric, and grammar sizes are normalized against the corresponding τ=0 int8 QAT baseline under identical settings.Both measures consider only the MLP weights included in the serialized string.Source: §4 · p. 4
  • The smallest grammar cannot be computed directly, so the experiments report upper bounds from Re-Pair, SEQUITUR, and LZ78, including compressors not used in the perturbation.Source: §4 · p. 4

5 Results

WeightPE reduces grammar size relative to int8 QAT on both ViT models, with an accuracy trade-off. The reduction also appears under compressors not used during finetuning, and ViT-L/16 has a more favorable trade-off than ViT-B/16.

Scaling model size

  • 0.43× the QAT grammar size on ViT-B/16 costs 1.9 accuracy points, while 0.37× on ViT-L/16 costs 1.1 points.At a comparable grammar size of approximately 0.38× the int8 QAT baseline, ViT-L/16 drops 0.3 percentage points in test accuracy versus 1.9 points for ViT-B/16.Source: §5 · p. 4

Accuracy vs. grammar size tradeoff

  • On ViT-B/16, grammar size falls to 0.49× the int8 QAT baseline with a 0.7-point accuracy loss; reducing it to 0.43× yields 97.3% test accuracy.The number of cluster leaders matters when the distortion budget is large enough not to be spent entirely on short sequences.Source: §5 · p. 4

Generalization across compressors

  • SEQUITUR matches Re-Pair grammar rates almost perfectly, while LZ78 reduces grammar size the least, despite neither compressor being used during finetuning.Both compressor scores are normalized against the baseline int8 QAT run.Source: §5 · p. 4

6 Conclusion

WeightPE makes network-weight grammar size an explicit training objective using a lossy Re-Pair perturbation within a global L2 budget and a straight-through estimator. On two CIFAR-10-finetuned ViT backbones, it yields grammars roughly 2.5× smaller than parallel int8 QAT at a small accuracy cost; broader scope and efficiency remain open.

Conclusion

  • WeightPE makes grammar size an explicit training objective by placing a lossy Re-Pair perturbation, bounded by a global L2 budget, inside a straight-through estimator.On two ViT backbones finetuned on CIFAR-10, deployed weights admit a grammar roughly 2.5× smaller than parallel int8 QAT at a small accuracy cost, with reductions also visible to compressors not used in the objective.Source: §6 · p. 4

Limitations

  • The evaluation covers two ViT backbones on CIFAR-10; behavior across other model families, datasets, and quantization levels remains untested.The experiments use row-major flattening, although serialization axis is a free method choice that exposes different patterns.Source: §6 · p. 4
  • The operator is not yet computationally optimized, with its cost negligible for some parameters and around 5× for others.Source: §6 · p. 4

Future work

  • Open questions include whether small-grammar structure transfers, whether shared rules map to interpretable computation, and how symbol-based sizes compare with standard compressors counted in bits.The paper also asks whether weights can be patched toward a smaller grammar without finetuning and whether grammars help models that consume weights as data.Source: §6 · pp. 4–5

A The operator in detail

The operator combines lossy rewrites under a distortion budget with parallel lossless merges. Its rounds select non-overlapping occurrences to reduce sequential work, while producing a grammar distinct from sequential greedy Re-Pair.

The operator in detail

  • Parallel rounds collapse independent sets of non-overlapping occurrences on the GPU, reducing sequential rounds to O(log N) rather than O(N); the resulting grammar differs from sequential greedy Re-Pair.Sequential Re-Pair is used only for evaluation, and the weight string stays on the GPU.Source: A The operator in detail · p. 8
  • In the twelve-code example, two rewrites cost one each, creating four non-adjacent copies of a pair that MERGE turns into one rule; the grammar shrinks from 12 to 10 symbols.A third candidate costs four, exceeding the budget of five after the two committed rewrites, so it waits for another round.Source: A The operator in detail · p. 8
  • Each rewrite round uses a round-start snapshot and chooses a different leader by incremental L2 cost, then commits the longest affordable prefix of non-overlapping occurrences and updates reconstruction.For an occurrence, the cost is the squared distance to the chosen leader minus the span's existing reconstruction cost.Source: A The operator in detail · p. 9
  • Leader selection considers at most 8T frequent candidate pairs and uses ρ as the leader de-duplication radius per element; T is the number of clusters.The lossy step tracks B, the distortion still affordable.Source: A The operator in detail · p. 9

B Experimental setup

The experiments finetune pretrained ViT-B/16 and ViT-L/16 on CIFAR-10, applying WeightPE to the transformer-block MLP projections under a shared training protocol and a sweep of distortion budgets and cluster counts.

B Experimental setup

  • ViT-B/16 and ViT-L/1 checkpoints pretrained on ImageNet-21k with the AugReg recipe were finetuned on CIFAR-10 with a freshly initialized 10-way head.Input transforms followed each checkpoint’s pretraining configuration, and validation used a dedicated seed independent of the run seed.Source: B Experimental setup · p. 9
  • Runs used six epochs, effective batch size 128, AdamW at learning rate 10^-5, weight decay 0.1, cosine scheduling, 500 warm-up steps, gradient clipping at 1.0, and no label smoothing.Runs within each sweep shared hyperparameters and differed only in τ and cluster count T.Source: B Experimental setup · p. 9
  • WeightPE perturbs the two MLP projections in each transformer block, while attention projections are quantized but left unperturbed and other listed layers remain full precision.The grammar is shared across perturbed MLP weights; row-major serialization confines rules to rows.Source: B Experimental setup · p. 9
  • The lossless MERGE step selects non-overlapping repeated adjacent pairs within rows and creates one grammar rule per distinct selected pair.No additional frequency threshold is imposed on this step.Source: B Experimental setup · p. 10
  • The sweep varied τfrac over 0.15–0.30 for ViT-B/16 and 0.15–0.35 for ViT-L/16, with T values of 32, 64, 128, and 256.The perturbation was refreshed every ten training updates; the global distortion budget is measured in quantized-code space.Source: B Experimental setup · p. 10

C Run environment

ViT-B/16 experiments ran on an NVIDIA A100-40GB GPU, while ViT-L/16 experiments ran on an NVIDIA H100-80GB GPU.

C Run environment

  • ViT-B/16 experiments ran on an NVIDIA A100-40GB GPU, and ViT-L/16 experiments ran on an NVIDIA H100-80GB GPU.Source: C Run environment · p. 10

D Run timings

Run wall-clock times are reported relative to each model’s int8 QAT baseline. Because measurements came from a shared cluster with uncontrolled contention, they indicate end-to-end overhead in this setting, not controlled operator microbenchmarks.

D Run timings

  • Figure 3 reports every run’s observed wall-clock time relative to the corresponding τfrac=0 int8 QAT baseline.Source: D Run timings · p. 10
  • Shared-cluster CPU, I/O, and node contention varied during measurement, so the timings indicate end-to-end overhead in this setting rather than controlled WeightPE-operator performance.Source: D Run timings · p. 10

Related papers

Loading 2609.31564v1…