EntroPack: Fast and Accurate Entropy-Coded Weight Compression at Arbitrary Bitrates

1 INTRODUCTION

• Fixed-width formats can leave deployment budgets between available storage sizes, while entropy-coded size depends on quantized-value distributions and coding overhead.Selecting a target rate without expensive repeated full-tensor encoding and reconstructing weights efficiently are the stated practical challenges.Source: §1 · p. 1
• EntroPack row-normalizes blocks of eight weights, quantizes them on the E8 lattice, and models lattice coordinates with conditional probabilities.Sampled-row storage estimates guide the quantization scale without repeated full-stream encoding; row-scale fitting and optional per-row rate–distortion refinement further improve reconstruction.Source: §1 · pp. 1–2
• Independent rANS tiles enable parallel decoding, while fused GPU decoding, lattice reconstruction, and row rescaling avoid intermediate symbol tensors and reconstruction codebooks.The decoder can output floating-point or integer weights, including FP8 and INT8.Source: §1 · p. 2
• At 4 bpp, EntroPack converts Z-Image-Turbo linear weights in 3.5 s and achieves lower weight error than tested fixed-width formats at comparable storage.The contribution summary also reports 7.7% additional denoising-step time relative to the uncompressed model.Source: §1 · p. 2

Entropy-coded quantization

• Entropy-constrained quantization considers coding rate and distortion; weight-compression methods include Huffman coding, context-adaptive arithmetic coding, and entropy reduction before coding.HRTN selects scalar scales using Huffman code length, while HyperQuant uses a precomputed rate–signal-to-noise relationship.Source: §2 · pp. 2–3

Structured weight quantization

• Weight quantizers vary in calibration use and representation: GPTQ and AWQ use activation information, NF4 is a nonuniform scalar format, and QuIP# uses an E8-based codebook.Other structured approaches include additive codebooks, grouped and Leech lattices, and trellis quantization.Source: §2 · p. 3

Deployment and GPU compression

• Deployment systems provide quantized storage and execution paths, while lossless and lossy compressors target numerical redundancy or precision- and error-controlled array compression.DietGPU provides GPU rANS primitives for parallel entropy coding.Source: §2 · p. 3

3.1 OBJECTIVE AND PIPELINE

• The pipeline selects a scale from sampled rows, quantizes the full tensor, fits row scales, optionally applies RDO, and rANS-encodes the fields into tiles.Decoding reverses these operations to produce numerical weights in the requested dtype.Source: §3.1 · p. 3

3.2 ROW-NORMALIZED LATTICE REPRESENTATION

• Row normalization and a shared scale map blocks of eight weights to E8 lattice points, whose invertible integer fields identify each point.The scale controls normalized-coordinate quantization, while the row magnitude restores the original scale.Source: §3.2 · pp. 3–4

3.3 COSET-CONDITIONED PROBABILITY MODEL

• A coset-conditioned probability model uses 17 categorical distributions and treats coordinate fields as conditionally independent given the coset.With exact empirical probabilities, conditioning reduces ideal coding cost by the sum of mutual informations between each coordinate field and the coset; stored-rate gains also depend on frequency quantization and table storage.Source: §3.3 · p. 5

3.4 TARGET-RATE SCALE SELECTION

• Scale selection estimates stored size from sampled complete rows and field counts, accounting for payload and reconstruction metadata without encoding candidate full streams.A geometric scale search selects the full-tensor quantization scale; probability precision is increased if the full-tensor field alphabets exceed representational capacity.Source: §3.4 · pp. 5–6

3.5 ROW-SCALE REFIT AND RATE–DISTORTION OPTIMIZATION

• After lattice points are fixed, least-squares row-scale refitting reduces row error without changing coded fields; optional RDO allocates candidate resolutions across rows under a shared storage budget.RDO alternates row-candidate selection with probability-model updates.Source: §3.5 · pp. 6–7

3.6 RANS ENCODING AND THE STORED REPRESENTATION

• The final fields use 17 shared frequency tables and independent rANS tiles, with each tile assigned to a GPU warp for parallel decoding.The stored representation includes payloads, offsets, terminal states, tables, row scales, and tensor layout; larger tiles amortize state and offset storage but expose less decoding parallelism.Source: §3.6 · p. 7

3.7 GPU DECODING AND WEIGHT RECONSTRUCTION

• GPU decoding recovers the coset and coordinate fields, applies the algebraic inverse, rescales each row, and converts to the requested dtype.These steps fuse symbol recovery, lattice reconstruction, and row rescaling without an intermediate symbol tensor or reconstruction codebook.Source: §3.7 · p. 7

4.1 EXPERIMENTAL SETUP

• Experiments compress linear weights from Z-Image-Turbo, MiniMax-H3, and Qwen3.8-27B, measuring storage, reconstruction error, model quality, and runtime on a single NVIDIA H20.The baselines include fixed-width formats and released HyperQuant, HRTN, and SDNQ implementations; unsupported weights remain at BF16 and count toward reported storage.Source: §4.1 · p. 8
• Stored rate includes side information and is calculated over evaluated weight elements, excluding biases; diffusion outputs are compared with BF16, while language-model quality uses perplexity.Inference timing follows each method’s execution path.Source: §4.1 · p. 8

4.2 RATE–DISTORTION AND TARGET-RATE ACCURACY

• At 4.02 bpp, EntroPack’s Z-Image-Turbo weights have 7.18% error, versus NF4’s 9.41% at 4.50 bpp; across targets, mean and maximum absolute targeting errors are 0.80% and 2.13%.HyperQuant and HRTN report 7.31% and 7.38% error at 4.15 and 4.06 bpp, respectively.Source: §4.2 · p. 8

4.3 DIFFUSION MODEL RESULTS

• At 4.02 bpp, Z-Image-Turbo outputs reach 19.62 dB PSNR and 0.80 SSIM; at 8.06 bpp, weight and output errors are 0.54% and 3.92%, with 29.14 dB PSNR and 0.95 SSIM.The table compares compressed-weight outputs with the BF16 reference; relative L2 measures first-step denoising output error.Source: §4.3 · p. 8, Table 1 · p. 9
• At a 4 bpp target, Z-Image-Turbo conversion takes 3.5 s versus 33.2 s for HRTN and 34.2 s for HyperQuant; denoising steps take 544.7–555.6 ms, 7.7–9.9% above BF16’s 505.7 ms.These inference times are lower than those of the tested torchao floating-point configurations, HRTN, and HyperQuant.Source: §4.3 · p. 8

4.4 LANGUAGE MODEL RESULTS

• On Qwen3.8-27B, EntroPack’s perplexity is 6.06 at 4.03 bpp, versus BF16’s 5.98, NF4’s 6.08 at 4.50 bpp, and HyperQuant’s 6.04 at 4.09 bpp.At 7.10 and 8.08 bpp, perplexity matches BF16 to the reported precision.Source: §4.4 · pp. 8–9, Table 2 · p. 9
• For a 32k-token Qwen3.8-27B prefill, EntroPack takes 18,709 ms versus BF16’s 18,639 ms; in the specified autoregressive workload, its 4 bpp configuration takes 272.8 ms versus 155.1 ms.At a 4 bpp target, conversion takes 5.8 s versus 109.1 s for HRTN and 149.6 s for HyperQuant.Source: §4.4 · p. 9

• EntroPack combines lattice quantization and conditional entropy modeling with sampled storage estimates to select rates without repeated full-stream encoding.Independent rANS tiles support fused GPU symbol decoding and weight reconstruction without intermediate symbol tensors or reconstruction codebooks.Source: §5 · p. 9
• The representation separates storage bitrate from numerical precision and supports floating-point and integer compute formats, including FP8- or INT8-quantized weights.Online decoding time grows with weight count.Source: §5 · p. 9
• In diffusion denoising and Transformer prefill, experiments show accurate bitrate control, fast encoding, lower weight error than tested fixed-width formats at comparable storage, and modest inference overhead.Source: §5 · p. 10

AI USE STATEMENT

• Generative AI tools assisted with parts of algorithm implementation and runtime optimization, literature retrieval, review of theoretical claims, and manuscript editing; the authors state they are responsible for the final implementation, results, and content.Source: AI USE STATEMENT · p. 10

APPENDIX

• The appendices cover evaluation protocols, video and audio results, design ablations, codec measurements, low-precision inference, per-layer storage allocation, search settings, RDO, GPU implementation, and limitations and future directions.Source: APPENDIX · p. 13, A SUPPLEMENTARY EXPERIMENTS · p. 13

A.1 EVALUATION DETAILS

• Evaluations use shared weight populations and quality evaluators, while runtime follows each method’s execution path, including required reconstruction.Z-Image-Turbo covers 276 linear-weight tensors, MiniMax-H3 covers 259, and Qwen3.8-27B covers 518 quantizable two-dimensional tensors; the specified token embedding and output projection are excluded for Qwen.Source: §A.1 · p. 13
• For Z-Image-Turbo, image metrics use six matched prompt-and-seed pairs, and PSNR and SSIM are reported as mean ± standard deviation; video and audio results use three seeds.Qwen3.8-27B perplexity uses WikiText-2 raw test text, 1,024-token target windows, and a stride of 512 tokens.Source: §A.1 · p. 13
• Diffusion timing reports GPU time per denoising step and includes required reconstruction, while language-model timing covers 32k-token prefill and a specified autoregressive workload.Conversion time sums synchronized per-layer quantization and compression measurements after warm-up.Source: §A.1 · pp. 13–14

A.2 ADDITIONAL DIFFUSION RESULTS: MINIMAX-H3

• At 4.03 bpp, EntroPack stores weights with 7.11% error; at 7 and 8 bpp, weight errors are 0.97% and 0.53%, with corresponding audio log-mel errors of 14.48% and 11.39%.The table reports weight compression, video and audio fidelity, and runtime; fidelity metrics are mean ± standard deviation over three seeds.Source: §A.2 · p. 14, Table 3 · p. 14
• EntroPack adds 0.5–0.8% to BF16 step time across the reported targets; model conversion takes 4.9–14.2 s, between tested fixed-width methods and HRTN and HyperQuant.Longer denoising steps make reconstruction a smaller fraction of runtime than on Z-Image-Turbo.Source: §A.2 · p. 14

A.3 ENTROPY-MODEL ABLATION

• The ablation compares coset-conditioned tables and parity-reduced fields against variants that remove one feature while preserving the lattice points and reconstruction scales.Coset conditioning models differences between the two cosets; parity reduction codes m instead of the eighth integer coordinate z8.Source: §A.3 · pp. 14–15
• Parity reduction saves 0.10–0.13 bpp across targets, while coset conditioning reduces stored rate from 2.07 to 2.02 bpp at the 2 bpp target.The coding configurations retain identical reconstructed weights; the largest coset-conditioning saving occurs at 2 bpp.Source: §A.3 · p. 15, Table 4 · p. 15

A.4 OPTIONAL PER-ROW RATE–DISTORTION REFINEMENT

• At five candidates per row, one allocation sweep lowers weight error from 14.35% to about 14.11%; eight sweeps take 26.3 s.Most of the observed improvement occurs in the first sweep, with little change thereafter.Source: §A.4 · p. 15, Figure 3 · p. 15
• With one sweep, increasing candidates from K = 1 to K = 20 raises encoding time from 4.2 s to 17.5 s; K = 5 gives 14.11% error and K = 20 gives 14.01%.Candidate scale ratios are nested within the fixed empirical range [0.70, 1.45] relative to the baseline scale.Source: §A.4 · p. 15, Figure 4 · p. 16

A.5 RECONSTRUCTION COST AND TILE OVERHEAD

• Smaller tiles decode faster but increase stored rate because more terminal states and offsets are needed; throughput counts reconstructed BF16 bytes per second.The reported stored rate includes metadata.Source: §A.5 · p. 16, Figure 5 · p. 16
• Reconstruction overhead falls from 132% at 256 tokens to 9.4% at 4096 tokens and 2.6% at 16,384 tokens.The comparison is against resident, uncompressed BF16 weights; larger matrix products amortize reconstruction over more computation.Source: §A.5 · p. 16, Figure 5 · p. 16
• Tile metadata occupies 1.80–3.60% of compressed storage across the displayed targets and is included in every reported bitrate.The per-tile metadata consists of terminal rANS states and a payload offset.Source: §A.5 · p. 16, Table 5 · p. 17

A.6 NUMERICAL DTYPE MEASUREMENTS

• Across seven dtype versions of the reference layer, stored rates remain close to targets, with encoding taking 11.0–36.3 ms and decoding 0.23–0.41 ms.Each tensor is reconstructed in its input dtype; errors are measured against that input after initial format conversion.Source: §A.6 · p. 17, Table 6 · p. 17

B ADDITIONAL APPLICATIONS

• EntroPack evaluates compressed FP8 and INT8 computation and reallocating a fixed storage budget among layers as additional uses.The evaluation covers retaining low-precision matrix products and assigning different rates to individual layers.Source: B ADDITIONAL APPLICATIONS · p. 17

B.1 FP8 AND INT8 COMPUTATION WITH COMPRESSED STORAGE

• At the 4 bpp target, compressed FP8 and INT8 store 4.02 bpp versus 8.01 bpp uncoded, with denoising steps of 392 and 393 ms versus 506 ms for BF16.Uncoded FP8 takes 343 ms; reported compressed-route times include weight reconstruction and matrix multiplication.Source: §B.1 · p. 17, Table 7 · p. 18
• At the 6 bpp target, direct BF16 compression has 1.89% weight error, versus 3.11% for the FP8 route and 2.37% for the INT8 route.FP8 and INT8 weight errors include initial format conversion and compression, using original BF16 weights as reference.Source: §B.1 · p. 17, Table 7 · p. 18

B.2 AGGRESSIVE NETWORK-WIDE WEIGHT COMPRESSION

• At approximately 2.5 stored bpp, mixed allocation raises mean SSIM from 0.64 to 0.67 relative to uniform allocation.The mixed allocation assigns 3 bpp to modulation, embedding, and output layers and approximately 2.47 bpp to the remaining layers; the example generation shows more consistent illumination.Source: §B.2 · p. 18, Table 8 · p. 18, Figure 6 · p. 18

C.1 SCALE-SEARCH SETTINGS

• Scale selection uses twelve geometric-bisection steps on [10^-3, 8] and a nominal sample budget of 2^18 eight-dimensional vectors, keeping sampled rows fixed across candidate scales.Rows are sampled with a shape-dependent seed.Source: §C.1 · p. 19
• Probability-table precision b starts at 11 bits for targets up to 7 bpp and 12 bits above that; if alphabets exceed capacity, precision increases up to 15 bits before the scale is coarsened.Precision determines how many distinct symbols can receive positive frequencies.Source: §C.1 · p. 19

C.2 RDO ALLOCATION ALGORITHM

• Per-row refinement precomputes candidate fields, row scales, and distortions, then alternates shared-model updates with row-candidate selection under the baseline’s estimated storage budget.A common rate-penalty search selects candidates; the algorithm retains the baseline if no budget adjustment is needed.Source: §C.2 · p. 19

C.3 TILED RANS IMPLEMENTATION

• Each independent tile stores 32 terminal rANS states and a 32-bit payload offset, totaling 132 bytes of tile metadata.Tile capacity determines the number of eight-dimensional vectors held; the tensor also stores a final boundary offset and FP32 row scales.Source: §C.3 · p. 19
• The tiled decoder uses 32-bit rANS states, 16-bit payload words, and lookup tables stored in shared or packed global memory depending on alphabet size and device resources.Supported probability precisions b ∈ {9, …, 15} allow at most one payload word per symbol.Source: §C.3 · pp. 19–20

D LIMITATIONS AND FUTURE WORK

• Scale search, probability modeling, and entropy coding make model conversion more costly than a single fixed-width quantization pass.Reducing repeated data passes could lower this cost.Source: D LIMITATIONS AND FUTURE WORK · p. 20
• Because the decoder materializes dense weights before each matrix product, reconstruction is harder to amortize in small workloads.Fusing decoding and reconstruction with matrix multiplication could avoid the intermediate tensor and reduce memory traffic.Source: D LIMITATIONS AND FUTURE WORK · p. 20
• Extending EntroPack to activations and key–value caches would require low-latency probability modeling and rate selection that track changing distributions.These tensors are produced during inference and would require online compression.Source: D LIMITATIONS AND FUTURE WORK · p. 20