Source-linked AI summary
Activation-Weighted Seeded Residual Coding for Low-Bit LLM Weight Repair
Zehao Liu, Chuangchuang Fang, Yang Ren
TL;DR
Low-bit quantization reduces storage but leaves residual errors that degrade language-model quality. AWSRC repairs those residuals with deterministic seed-generated bases, activation-weighted fitting, and a compact sidecar, recovering substantial matched gaps to BF16 while improving selected low-bit backbones and byte-matched codec comparisons.
Problem
Low-bit weight quantization saves storage but leaves residual errors, motivating compact repair without storing a dense high-precision residual.
Method
AWSRC keeps the parent reconstruction fixed and encodes residual tiles with deterministic seed-generated bases, activation-weighted fitting, and byte-normalized additive records.
Results
AWSRC recovers 88.2%, 78.9%, and 71.3% of the matched PPL, KL, and accuracy gaps to BF16 at +0.162 scope-bpw, while improving matched low-bit backbones and leading PPL and mean accuracy among byte-matched codecs.
Takeaways & Limitations
Matched parent-repair evaluation shows AWSRC’s quality-per-byte advantage within the tested codec comparisons, while repair gains depend on the chosen backbone.
Takeaways & Limitations
The reported evaluation concerns compressed quality rather than runtime acceleration, reconstructs dense BF16 weights, and primarily repairs MLP projections rather than all-linear scopes.
Abstract
from arXiv · showhide
Low-bit weight quantization saves storage but leaves errors that degrade language-model quality. We introduce Activation-Weighted Seeded Residual Coding (AWSRC), a compact repair codec for an existing quantization backbone. Given a reconstructed weight $W_0$, AWSRC encodes the residual $W-W_0$ using deterministic seed-generated bases. The sidecar stores seed selectors, low-bit coefficients, and scales rather than an explicit codebook. Activation statistics prioritize errors that affect layer outputs. On Qwen2.5-3B-Instruct, adding 0.162 scope-bits/weight to an INT4 RTN backbone closes 88.2%, 78.9%, and 71.3% of the matched PPL, KL, and accuracy gaps to BF16. Repairing a matched strong low-bit backbone also improves all measured quality metrics. With a matched 49.25 MB sidecar, about 0.8% of the BF16 model-weight payload, AWSRC gives the best perplexity and mean task accuracy among sparse, low-rank, and vector-quantized codecs.
1. INTRODUCTION
AWSRC treats quantization repair as a separately budgeted residual-coding problem, retaining a fixed parent reconstruction while targeting errors that matter for layer outputs. It uses deterministic seeded bases and activation-weighted, byte-aware records to provide removable, backbone-compatible repairs.
- Motivation: Quantization leaves a residual R = W −W0 whose exact storage would be as costly as the original weight.The challenge is to compress useful residual information without storing a dense high-precision copy.
- Design challenges: Residuals are not guaranteed to be sparse or well approximated at small rank, limiting straightforward sparse and low-rank repairs.Alternative codecs also incur coordinate, factor, assignment, or codebook storage costs.
- Design challenges: Activation-weighted fitting prioritizes residual errors according to their effect on layer outputs rather than treating all coordinates equally.The codec targets repair gain per stored byte using the model’s input distribution.
- AWSRC: AWSRC stores seed selectors, quantized coefficients, scales, and tile indices, while regenerating bases deterministically instead of serializing a codebook.Decoding requires no search, and omitted tiles retain the exact parent reconstruction.
- AWSRC: Each additive repair stream can be removed without changing the backbone format, and complete-record prefixes reconstruct against the same fixed W0.This supports independently budgeted and progressive repair.
- Results: At +0.162 scope-bpw, AWSRC recovers 88.2%, 78.9%, and 71.3% of the matched PPL, KL, and accuracy gaps to BF16.The evaluation reports these values for PPL, KL, and accuracy, respectively.
2. RELATED WORK
Related work reduces quantization error through calibrated backbones, sparse or low-rank corrections, vector quantization, and seed-based reconstruction. AWSRC is positioned as complementary residual compensation that avoids explicit codebooks while attaching to an existing quantizer.
- Quantization backbones: GPTQ uses second-order information for sequential rounding, while AWQ rescales activation-salient channels.QAM-W combines activation-aware scaling, Hadamard rotation, and learned two-dimensional codes as another strong backbone.
- Residual correction: Sparse correction retains sensitive outliers with coordinates, whereas LQER and QERA use dense low-rank factors.These approaches represent different storage and approximation choices for quantization error.
- Codebook and seed methods: AQLM and GPTVQ learn vectors and assignments, while SeedLM regenerates complete weight blocks from shared seeds without an explicit codebook.AWSRC similarly uses seeded reconstruction but applies it to residual repair.
3. AWSRC METHOD
AWSRC partitions residuals into tiles, represents them with deterministic signed and permuted Hadamard bases, and fits activation-weighted low-bit corrections. It allocates records by byte-normalized gain, supports progressive decoding, and reports storage using explicit serialized-rate accounting.
- Pipeline: AWSRC adds a seeded, activation-weighted residual stream to a fixed quantized backbone, with optional progressive ordering.The repair operates on R = W −W0 rather than replacing the parent quantizer.
- Pipeline: For any quantizer Q, AWSRC starts from W0 = Q(W), computes R = W −W0, and reconstructs corrected weights as W0 + bR.The parent reconstruction remains the reference for every repair stream.
- Seeded representation: Target matrices are split into row-wise tiles, and each seed deterministically generates a basis so no basis matrix is serialized.The generator selects signed, permuted Hadamard columns; stored records need only a seed selector.
- Seeded representation: AWSRC stores only a ⌈log2 S⌉-bit selector per retained tile plus fixed generator metadata, while increasing S mainly expands offline search.The basis diversity, coefficient precision, and subspace dimension still affect rate and computation.
- Activation-weighted fitting: Activation statistics define a diagonal weighting that estimates expected output perturbation and selects seeds after coefficient quantization.The activation diagonal is shared across output rows of a linear module and is computed from calibration inputs without storing full covariance.
- Allocation and progressive prefixes: Records are ranked by reconstruction gain per complete serialized byte, with non-positive records discarded and progressive prefixes retaining complete additive corrections.The progressive variant globally sorts eligible records, so every prefix decodes against the same W0.
- Serialized rate: Measured storage includes sidecar header and payload accounting, while scope-bpw covers repaired matrices and full-effective-bpw includes BF16 parameters outside the target scope.The Qwen2.5 main artifact uses 7-byte records, a 904-byte header, and a 49.25 MB sidecar.
4. EXPERIMENTS
The experiments evaluate AWSRC on Qwen2.5-3B-Instruct using matched quality metrics, byte-matched residual codecs, and paired repairs across quantization backbones and model families. Results show strong recovery toward BF16 at low added storage, while improvements depend on the parent backbone.
- Main results: At 4.23 scope-bpw, RTN-SDQ repair lowers PPL from 9.62 to 7.04 and KL from 0.35 to 0.08, while accuracy rises from 0.65 to 0.69.QAM-W repairs lower PPL and KL at all three seed-0 rates, with 4-bit repeats confirming both in 3/3 seeds and accuracy improving in 2/3.
- Backbone dependence: AWSRC consistently repairs RTN and clean-room QAM-W, but GPTQ improves KL and accuracy while slightly worsening PPL, and AWQ changes are negligible.GPTQ/AWQ are reference backbones with different scopes, so the comparisons are matched repairs rather than cross-row iso-bpw rankings.
- Byte-matched residual codecs: AWSRC has the best PPL and mean accuracy among residual codecs at identical serialized bytes, while learned VQ has the best unrounded KL.The comparison uses exactly matched sidecar bytes; the shared RTN-SDQ evaluation uses 49,245,876-byte sidecars.
- Main results: 88.2%, 78.9%, and 71.3% of the matched PPL, KL, and accuracy gaps to BF16 are recovered in the complete 11-task run.Scope-bpw increases from 4.068 to 4.230, and every task improves.
- Generalization: Paired gains extend across four model families, and the 11-task result is stable across three seeds.Cross-model rows use the same model, scope, and protocol rather than forming a cross-model ranking.
5. CONCLUSION
AWSRC repairs quantization residuals with seed-generated bases and activation-weighted fitting, improving compressed quality across backbones and model families. Its evaluation emphasizes matched parents and fully accounted sidecar bytes, while leaving packed-runtime performance for future work.
- AWSRC repairs quantization residuals with seed-generated bases and activation-weighted fitting.
- At +0.162 scope-bpw, AWSRC recovers 88.2%, 78.9%, and 71.3% of the matched PPL, KL, and accuracy gaps to BF16.
- AWSRC improves a stronger low-bit backbone, transfers across four model families, and achieves the best PPL and accuracy among four byte-matched codecs.
- Mixed GPTQ/AWQ results show that repair outcomes depend on the parent quantization backbone.
- The reported results concern compressed quality rather than runtime, with dense BF16 reconstruction and primary repair of MLP projections unlike the all-linear references.Future work will measure packed decoding, memory traffic, latency, loading cost, and serverless LLM cold-start time.
- Fixing W0 and accounting for every sidecar byte enables gain-per-byte evaluation against a matched parent rather than standalone cross-row ranking.
7. COMPLIANCE WITH ETHICAL STANDARDS
The study used public model checkpoints and benchmark datasets without human or animal subjects, so no ethical approval was required. The authors declare no conflicts of interest.
- The study used publicly available model checkpoints and benchmark datasets and involved no human or animal subjects.Therefore, no ethical approval was required.
- The authors declare no conflicts of interest.