PaperlayerView exact revision on arXiv

Source-linked AI summary

LEAN: Loss-Engineered Modulation of the Model Weight-Sensitivity Distribution for Efficient, Error-Tolerant Neural Networks

Habibur Rahaman, Swarup Bhunia

arXiv:2610.28872v1cs.LGcs.CR

TL;DR

Stored weights differ sharply in flip sensitivity, but existing schemes use the distribution left by training, affecting attack tolerance, protection, and storage costs. LEAN regularizes the flip-step and gradient factors to reshape that distribution. Across nine vision models it reduces the sensitive tail 5 to 13 times within 0.5 accuracy points, alongside reported protection and low-bit storage gains.

  • Problem

    Existing methods do not optimize the highly uneven weight-sensitivity distribution, although its tail sets bit-flip tolerance, selective-code coverage, and full-width storage needs.

    Source: §I · pp. 1–2

  • Method

    LEAN adds a composite regularizer on flip step and gradient, with λ controlling the amount of sensitivity-distribution reshaping.

    Source: §I · p. 2

  • Results

    Across nine vision models, LEAN shrinks the sensitive tail 5 to 13 times within 0.5 accuracy points, while three language models show 1.7-to-5.3-times reductions.

    Source: §I · p. 2

  • Takeaways & Limitations

    The reshaped distribution accompanies lower parity for a fixed protection guarantee and enables sensitivity-ordered low-bit storage, including a 3.29-bit ImageNet ResNet-50 payload.

    Source: §I · p. 2, §VII · p. 16

    LEAN needs training data and about 1.5 times the cost of a plain fine-tune, and its protection results rely on interleaving assumptions.

    Source: §V · p. 15

Abstract

from arXiv · show

The weights of a trained network do not have uniform sensitivity to model accuracy: the 1% most sensitive weights carry about a fifth of the damage a bit flip can do. This distribution decides the bit-flip tolerance of a model, the fraction of weights a code must cover and the bits each weight needs; existing schemes work with the distribution training leaves. We show that we can systematically modulate or alter the weight sensitivity distribution of a trained model, thereby shifting weights from one sensitivity bin to another. The sensitivity of a stored weight is the product of its gradient and the step a flip takes in the storage format, and LEAN adds a composite regularizer on both factors, a range term that bounds the flip step and a tail term that lowers the gradient of the most sensitive weights, with one coefficient {\lambda} setting how far the distribution moves. Ten epochs with the term, as a fine-tune or as the last epochs of training, shrink the sensitive tail 5 to 13 times on nine vision models (ResNet-18, ResNet-50, VGG-16 on CIFAR-10, CIFAR-100, ImageNet) within 0.5 points of accuracy; real flips confirm that the reshaped weights do less damage, a plain gradient-norm penalty does not reproduce the effect, and a preliminary extension to three small language models shows the same tail shaping. With a BCH code over 1% of the weights the model then survives 300 progressive bit-search flips in every training seed and 500 in two of three, where three of four published defenses with the same code fail by 300 and WCAT falls to 75% at 500. The reshaped distribution also cuts the parity for a fixed guarantee 3.7 times and makes a sensitivity-ordered 8/6/4/3-bit format affordable, which stores ImageNet ResNet-50 at a 3.29-bit weight payload, 2.4 times smaller than INT8, above the accuracy of LSQ at 3 bits.

I. INTRODUCTION

Weight sensitivity is highly uneven, and its tail shapes the costs of bit-flip tolerance, protection, and storage. LEAN reshapes that distribution during training, reducing the sensitive tail and improving protection and storage outcomes with small accuracy changes.

I. INTRODUCTION

  • The top 1% of ResNet-18 CIFAR-10 weights account for 19.5% of first-order sign-flip loss change, while the next 8% account for most of the remainder.The distribution's tail sets the attack surface, the amount selective codes must cover, and which weights require full-width storage.Source: §I · pp. 1–2
  • LEAN regularizes both factors of flip damage—the storage-format step and the weight gradient—so a single coefficient λ controls how far the sensitivity distribution moves.The range term bounds flip steps, while the tail term lowers gradients among the most sensitive weights.Source: §I · p. 2
  • On nine vision model-dataset configurations, LEAN shrinks the sensitive tail 5 to 13 times within 0.5 accuracy points; three language models show 1.7-to-5.3-times reductions.The effect is reported after one fine-tune and increases monotonically with λ.Source: §I · p. 2
  • A sensitivity-ordered format stores ImageNet ResNet-50 at 3.29 bits per weight, 2.4 times less payload than INT8, and reaches 77.2% accuracy versus 74.4% for LSQ at 3 bits.At exactly 4 bits per weight, it is 1.2 points above LSQ; the reported 3.29-bit payload also has an estimated 59% less weight-fetch energy.Source: §I · p. 2
  • With a 1% code, LEAN survives 300 progressive bit-search flips in every run and retains 94% accuracy after 500 flips in two of three seeds; parity for a fixed guarantee drops 3.7 times.The normal model falls to chance by 200 flips.Source: Figure 1 · p. 1, §I · p. 2

II. BACKGROUND

Weight sensitivity determines which stored-weight flips are most damaging and how much selective protection or precision is needed. The paper motivates reshaping this distribution because existing approaches largely accept the distribution training leaves behind.

II. BACKGROUND

  • Progressive bit search ranks candidate bits by loss gradient, tests flips among top candidates, and repeatedly commits the worst; a few flips can take ImageNet ResNets to chance.The attacker is white-box and sees weights, gradients, and any code.Source: §II · p. 3
  • In ResNet-18, the top 1% of weights account for about 20% of first-order sign-flip damage, and sensitivity spans a factor of 10,000 or more from median to most sensitive.A short tail means greater tolerance to adaptive flips and lower selective-code and mixed-precision budgets.Source: §II · p. 3
  • A sign-bit flip in per-tensor INT8 changes the stored integer by 128, so every weight in a layer takes the same large step and damage ranking depends on gradients.This links the gradient ranking used by progressive bit search to the ranking defining the sensitive set.Source: §II · p. 3
  • LEAN changes the distribution intentionally and by a chosen amount, unlike robust-training methods that change it as a side effect.Storage baselines include LSQ, which learns quantization step size, and KURE, which regularizes weight-distribution kurtosis.Source: §II · p. 3
  • A selective code covers the HIGH weights, encoding their four most significant bit planes with BCH(127,43), which corrects 12 errors per 127-bit codeword.The evaluation treats covered-bit flips as corrected and assumes interleaving prevents an attacker from accumulating more than 12 errors in one codeword before correction; the code itself is not simulated.Source: §II · p. 3

III. METHOD

LEAN measures stored-weight flip sensitivity, then regularizes both the flip step and the gradient of the most sensitive weights. Its ten-epoch training stage uses fixed baseline thresholds, while a separate quantization-aware stage supports sensitivity-ordered low-bit storage.

III. METHOD

  • For INT8, sign-bit damage is first-order loss change from a weight's gradient times its layer's flip step; baseline-derived HIGH and MID thresholds define the bins used to measure reshaping.The thresholds correspond to the baseline top 1% and top 9%, yielding 1% HIGH, 8% MID, and 91% LOW.Source: §III · pp. 3–4
  • LEAN minimizes cross-entropy plus λ times a composite penalty: a range term bounds layer flip steps, while a tail term lowers gradients among the current top 1% by damage.The range coefficient β is 100 in every experiment; the tail penalty is differentiated through the gradient by double backpropagation every fourth step, scaled by four, and the fine-tune costs about 1.5 times a plain one.Source: §III · p. 4
  • The INT8 pipeline measures the baseline, fixes HIGH and MID thresholds, trains for ten epochs, remeasures against those thresholds, and packs a fixed-budget or threshold-selected code.The tail is reselected and its penalty applied through double backpropagation every fourth step.Source: §III · p. 4
  • A tail-penalty gradient step locally decreases tail sensitivity when the tail set, gradient signs, and flip-step matrix are held fixed; the analysis makes no claim beyond that step.The tail term moves weights along Hessian rows, and the selected tail can change between steps.Source: §III · p. 4
  • The tail term lowers gradients that gradient-guided attackers read, but does not change flip steps; real-flip tests assess whether this also reduces damage.A detached proxy such as |wi| or frozen |gi| lacks the Hessian term.Source: §III · p. 4
  • The separate low-bit deployment fine-tunes a sensitivity-ordered 8/6/4/3-bit format from the LEAN INT8 checkpoint, ranking tiers by deployed-format damage.Tier scales use the 0.99 quantile rather than the maximum, reducing 3-bit sign-flip steps by 2 to 4 times; reported payload excludes scales and the tier map.Source: §III · p. 4

IV. RESULTS

LEAN reshapes the sensitivity tail across nine vision models while keeping clean-accuracy changes small. The results connect that reshaping to real-flip damage, protection costs, and lower-bit storage, while identifying limits on tail reduction and low-bit resilience.

Distribution reshaping

  • 5 to 13 times fewer weights remain above the baseline HIGH threshold across nine models, while the top 1% damage share falls 1.4 to 2.1 times and clean-accuracy changes are at most 0.5 points.The reported models use ten-epoch LEAN fine-tunes; the distributions are measured on held-out batches.Source: §IV · pp. 5–6, Table I · p. 6

λ and the reduction floor

  • LEAN’s HIGH share falls monotonically with λ and saturates near 0.1%; accuracy stays within 0.5 points up to λ = 40, then drops at λ = 100.The reduction floor reflects per-layer outlier ratios and loss curvature; layers without outliers offer little range-term reduction.Source: §IV · p. 6, Figure 7 · p. 9

Training placement

  • Holding the learning rate at 0.002 for the last ten training epochs reaches HIGH 0.04 to 0.14% at fine-tune accuracy, while the default decaying schedule reaches 0.25 to 0.50%.The default schedule is still reducing the penalty at epoch 200; the rate-held cosine schedule also yields 93% accuracy at 500 flips in the reported case.Source: §IV · pp. 6–7, Figure 9 · p. 11

Ablation and real-flip check

  • Real sign flips confirm lower damage: worst-100 loss increase falls from 0.031 to 0.009, while the range term does most of the INT8 reshaping and the tail term adds further reduction.The tail term alone can shrink the proxy HIGH bin without reducing real damage; a plain gradient-norm penalty leaves HIGH at 0.64% at its accuracy-preserving setting.Source: §IV · pp. 7–9, Table IV · p. 12

Bit-flip protection

  • With the same 1% BCH code, LEAN at λ = 40 holds 94% accuracy through 500 progressive bit-search flips in two of three seeds; WCAT falls to 75% at 500.The normal model and three of four compared defenses break by 300 flips; the code alone is insufficient unless the weights just below its cutoff are less damaging.Source: §IV · pp. 9, 11, Table V · p. 12

Protection cost

  • 3.7 times less parity meets the reported fixed guarantee with LEAN: 4.8% parity at 5.3% weight coverage, versus 18% parity at 20% coverage for the normal model.This is a separate fixed-guarantee test using λ = 10; λ = 40 misses the zero-drop criterion by 0.16 points at the same coverage.Source: §IV · p. 12, Table VII · p. 13

Precision and storage

  • At 3.29 bits per weight, tiered LEAN reaches 77.2% on ImageNet ResNet-50 versus 74.4% for LSQ at 3 bits; the weight payload is 10.5 MB versus 25.6 MB for INT8.At exactly matched 4.00 bits per weight, tiered LEAN is 1.2 points above LSQ and 3.7 above KURE; estimated weight-fetch energy falls 59%, excluding decoder and metadata activity.Source: §IV · pp. 12–13, Table VIII · p. 13

Low-bit model resilience

  • At 3 bits per channel, the 1% code cannot protect the low-bit model; with scale, ranking, and interleaved-code changes, CIFAR models withstand 100 to 500 flips within 2 to 7 points of clean at 3.63 bits per weight.ImageNet results vary by model at 4.09 bits per weight: VGG-16 is unchanged after 500 flips, while ResNet-18 and ResNet-50 lose 14 and 13 points.Source: §IV · p. 13

V. APPLICATIONS

LEAN yields separate protection and storage deployment modes, while a preliminary language-model extension shows tail shaping with model-size-dependent trade-offs. The section also notes training costs and controls for distribution shaping that remain unexplored.

a) Accelerators and deployment

  • LEAN reduces the number of protected HIGH weights 5 to 13 times without changing the memory system; hardware sees a shorter list of protected addresses.Source: §V · p. 13
  • The LEAN INT8 checkpoint with a 1% code survives 300 flips in every seed and 500 in two of three, with parity embedded in LOW weights at no added memory.Source: §V · p. 14
  • The tiered 3.29-bit format stores ImageNet ResNet-50 at 2.4 times less weight payload than INT8 and is estimated to cut fetch energy by 59%.Source: §V · p. 14, Figure 10 · p. 14, Table XI · p. 14

b) Effect on language models

  • Across three language models, the HIGH set shrinks 1.7 to 5.3 times, but coded-attack outcomes depend on model size and λ.Source: Table XII · p. 14, §V · p. 15, Table XIII · p. 15
  • On Qwen2-1.5B, λ = 3 was already too strong: both models held 20 flips with the code but were gone at 50; LEAN was slightly worse at 20 (16.3 against 11.2), so over-strong λ buys nothing.Source: §V · p. 15

c) Limitations

  • LEAN needs training data and about 1.5 times the cost of a plain fine-tune; per-bin targeting and other shaping controls remain unexplored.Source: §V · pp. 15–16

VI. RELATED WORK

LEAN is positioned against methods that target flip robustness, gradient norms, selective protection, and low-bit quantization by reshaping the sensitivity distribution itself. Its reported distinction is to reduce the sensitive set that protection and precision schemes must handle.

a) Robust training against bit flips

  • LEAN differs from flip-robust training methods –[11] by optimizing the size of the set a selective code must cover, and under the same code WCAT reaches 75% at 500 flips while LEAN reaches 94% in two of three seeds.The cited methods are at chance within 50 adaptive flips without a code; real-flip results show what LEAN’s tail term adds beyond range control.Source: §VI · p. 16

b) Gradient-norm regularization

  • LEAN’s top-k, storage-step-weighted gradient penalty differs from generic gradient regularization,; a plain gradient-norm fine-tune leaves a 0.64% tail and breaks at 200 flips.The comparison uses the largest coefficient the accuracy tolerates.Source: §VI · p. 16

c) Selective and in-place protection

  • LEAN makes the set protected by selective or embedded codes, smaller, and the paper notes that this can also lower checksum cost.The chosen correcting code needs no response path, while detection methods catch flips at run time.Source: §VI · p. 16

d) Low-bit quantization

  • LEAN’s sensitivity-ordered precision tiers differ from quantization and mixed-precision methods,, –[29] because λ shrinks the high-precision set before bit widths are assigned.At uniform bit width, LEAN matches LSQ; the reported gain comes from ordering weights by stored-format sensitivity.Source: §VI · p. 16

e) Attacks on large language models

  • LEAN’s sensitivity shaping also targets the structure exploited by transformer bit-flip attacks [21],, with a first adaptation reported in Section V.The related-work passage describes these attacks as requiring a few dozen flips.Source: §VI · p. 16

VII. CONCLUSION

LEAN reshapes the nonuniform weight-sensitivity distribution, reducing the share of weights in the highest-sensitivity bins. The reported shaping improves bit-flip protection and supports lower-cost protection and sensitivity-guided storage, while targeted attacks on ImageNet ResNets remain outside the current objective.

VII. CONCLUSION

  • LEAN shifts weights from the highest-sensitivity bins to lower ones, addressing the fact that existing methods do not optimize the sensitivity-distribution shape.The distribution shape sets bit-flip tolerance, code coverage, and per-weight storage width.Source: §VII · p. 16
  • A 1% code holds ResNet-18 through 300–500 adaptive flips, while three of four published defenses fail by 300 and WCAT reaches 75% at 500; fixed-guarantee parity drops 3.7 times.The 3.29-bit sensitivity-ordered format stores ImageNet ResNet-50 at 2.4 times less weight payload than INT8 and above LSQ accuracy at 3 bits.Source: §VII · pp. 16–17
  • The current objective does not reach targeted attacks on ImageNet ResNets that move through weights unranked by mixed-data sensitivity; class-aware sensitivity is identified as a next step.The conclusion also identifies per-bin targets or changing the balance of the regularizer terms as possible controls beyond λ.Source: §VII · p. 17
Loading 2610.28872v1…