Source-linked AI summary
Lossless Tensor Compression as Program Synthesis
Jieke Shi, Junda He, Wenjia Jiang, Weifeng Sun, Shidong Pan, Zhensu Sun, Chengran Yang, Peixin Zhang, Yifan Jia, Zhou Yang, Thong Hoang, Xiwei Xu, Zhenchang Xing, David Lo
TL;DR
Growing model checkpoints make archival, transfer, and deployment costly, while existing compressors overlook tensor structure or use fixed pipelines. Brevis instead synthesizes compact, reversible DSL programs for bit-exact tensor reconstruction, reducing 2.13 TB to 1.41 TB—a 33.93% storage reduction—across 10 public checkpoints.
Problem
Existing compressors either overlook tensor structure or rely on fixed, format-specific pipelines, motivating tensor-specific compression that can represent diverse structures compactly and bit-exactly.
Method
Brevis synthesizes self-contained programs in a typed reversible DSL, using a checkpoint-specific production prior to guide bounded A* search toward compact bit-exact representations.
Results
33.93% storage reduction: Brevis reduces 2.130 TB to 1.407 TB and produces the smallest archive on every evaluated checkpoint.
Takeaways & Limitations
Brevis provides a lossless tensor-compression approach that outperforms the evaluated general-purpose and tensor-specific compressors across the 10 public checkpoints.
Takeaways & Limitations
The evaluation covers public model checkpoints with limited and partly confounded domain and format diversity, and the implementation synthesizes each tensor independently.
Abstract
from arXiv · showhide
Model checkpoints are growing in both number and size, which makes archival, transfer, and deployment increasingly costly. General-purpose compressors can reduce storage requirements but ignore tensor structure, whereas existing tensor-specific compressors rely on fixed and format-specific pipelines. We present Brevis, which formulates lossless tensor compression as program synthesis. We design a typed domain-specific language (DSL) that captures recurring tensor structures, such as repeated regions and floating-point fields, through a set of reversible operators. Given a tensor, Brevis synthesizes a self-contained DSL program that reconstructs it bit-exactly. A checkpoint-specific production prior, learned from a small representative sample of tensors, guides a bounded A* search to synthesize compact programs, which can later be executed directly for bit-exact decompression. On 10 public checkpoints spanning language, audio, and image generation models, Brevis reduces 2.13 TB of checkpoint data to 1.41 TB, a 33.93% storage reduction. It produces archives up to 30.87% smaller than those of four general-purpose compressors, including zstd and gzip, and smaller archives than the tensor-specific compressors ZipNN and DFloat11. Under a practical concurrency configuration, Brevis achieves 3.60 GB/s compression and 6.61 GB/s decompression while preserving every source byte.
1 Introduction
Brevis formulates bit-exact tensor checkpoint compression as program synthesis, using self-contained DSL programs that reconstruct tensors exactly. A checkpoint-specific production prior guides bounded A* search toward compact programs that capture tensor-specific structure.
- Method: Brevis represents each tensor with a compact, self-contained DSL program that reconstructs the original tensor bit-exactly.The formulation allows different tensors to use different programs, capturing tensor-specific structures.
- Method: A checkpoint-specific production prior learned from a small representative tensor sample guides bounded A* search through the large program space.The search targets compact programs within a practical budget, including the overhead of instructions, parameters, and literals.
- Method: The typed DSL uses reversible operators for repeated values, subsequences, element relations, and floating-point fields, with exact literals as a fallback.These operators are designed to capture recurring tensor structures while retaining exact reconstruction.
- Results: 30.87% smaller archives are achieved than four general-purpose compressors, while Brevis also produces archives up to 2.90% smaller than ZipNN and DFloat11.Brevis produces the smallest archive on all 10 checkpoints against six baselines and saves 10.53 GB relative to the tensor-specific baselines.
2 Background and Related Work
Prior work spans general-purpose, float-aware, model-specific, and lossy tensor compressors, while program-synthesis systems guide search through grammars and learned production priors. Brevis differs by formulating tensor compression itself as program synthesis, using a DSL and exact reconstruction as its objective.
- Compression Methods: General-purpose codecs treat tensors as generic byte streams, while float-aware preprocessing exploits numerical structure through Bitshuffle, TDT, and ALP.The cited general-purpose codecs include gzip and Zstandard.
- Compression Methods: Model-specific compressors exploit learned-weight statistics, including floating-point fields, exponent distributions, model-specific layouts, and inference-oriented entropy coding.Examples include ZipNN, DFloat11, ECF8, ZipMoE, Huff-LLM, tile-aligned ANS, ZipServ, and ENEC.
- Compression Methods: Lossy methods modify weights or target near-lossless inference, whereas Brevis focuses on bit-exact checkpoint archival.The passage identifies quantization, pruning, LLM.265, and NeuZip as examples of approaches targeting inference efficiency rather than exact reconstruction.
- Program Synthesis: Program-synthesis research constrains search with grammars or learned production priors, while related compression systems synthesize floating-point algorithms or transformation pipelines.Examples include syntax-guided synthesis, PHOG, Euphony, TF-Coder, the KoLMogorov Test, and OpenZL.
- Brevis: Brevis formulates tensor compression itself as program synthesis: its DSL defines the search space and exact reconstruction determines validity.Figure 2 describes short reversible programs, target-guided A* search over a typed grammar, and ranking by complete serialized size L(P), including literal payload and codec tables.
3 Method
Brevis formulates lossless tensor compression as synthesis of typed, self-contained DSL programs that reconstruct tensor bits exactly. A checkpoint-specific production prior guides bounded target-directed A* search, which selects the smallest exact serialized program found within finite resource limits.
- Typed program representation: Brevis represents each tensor as a typed, self-contained program whose execution reconstructs the original bits exactly.The representation preserves signed zeros, NaN payloads, and every other bit pattern.
- Production prior: A checkpoint-specific production prior, learned from a deterministic stratified sample, changes exploration order without making any valid production unreachable.The prior uses contexts including hole type, parent operator, child position, depth, dtype, size bucket, zero fraction, distinct-value ratio, repetition ratio, and adjacent-difference entropy.
- Typed DSL operators: Lit, Const, Concat, Repeat, Map, Scan, and Merge capture literals, constants, regions, repetitions, transformations, correlations, and packed fields.Lit provides a universal fallback, while Map includes bijections such as XOR, modular addition, ZigZag, Gray coding, bit reversal, and rotation.
- Target-directed synthesis: Target-directed inverse expansion decomposes the target stream into child streams, preserving correctness by construction instead of repeatedly executing complete candidates.Operator-specific inverses handle exact repetitions, bijections, adjacent-word updates, field or plane splitting, target-derived concatenation boundaries, and constant streams.
- Bounded A* search: Bounded A* search combines production costs with an admissible completion estimate and prunes states whose byte lower bound cannot beat the current best program.The initial incumbent is a literal program; complete candidates are compared using exact serialized size, and finite node, depth, arity, memory, and expansion limits bound the search.
4 Evaluation
Brevis is evaluated on 10 public checkpoints spanning language, audio, and image generation models, with exact reconstruction verified across the comparison matrix. It achieves substantial storage savings versus general-purpose and tensor-specific baselines while maintaining high throughput and motivating a one-expansion search budget.
- Corpus-wide results: 33.93% storage reduction compresses 2.130 TB to 1.407 TB, saving 722.79 GB across 10 checkpoints.Brevis produces the smallest archive for every checkpoint and available comparison.
- Baseline comparisons: 30.87% mean per-checkpoint savings over Snappy and 12.94% over zstd span Brevis’s advantages across all 10 checkpoints.Brevis is smaller than the four general-purpose compressors on every checkpoint and also outperforms ZipNN in the complete-corpus comparisons.
- Baseline comparisons: 2.90% smaller than DFloat11, Brevis occupies 10.579 GB versus 10.896 GB on Llama-3.1-8B.This comparison is limited to DFloat11’s only publicly available, independently validated native result because the formats and remaining checkpoint coverage are not comparable.
- Throughput: 6.61 GB/s decompression and 3.60 GB/s compression yield a 1.522 compression ratio on Llama-3.1-70B.Compared with ZipNN, Brevis compresses 7.2% faster and produces a 0.82% smaller archive, while other baselines trade off throughput or storage reduction.
- Search-budget analysis: 9.04 seconds at budget 1 captures most achievable compression on a 4.98 GB shard, whereas budgets 32 and 256 take 135.04 and 1,399.89 seconds.The larger budgets save an additional 3.13 MB and 5.48 MB, motivating one expansion in the main evaluation.
5 Limitations and Conclusion
Brevis frames lossless tensor compression as synthesis of typed, self-contained programs that reconstruct tensors bit-exactly, achieving substantial storage savings across public checkpoints. Its evaluation remains limited in workload, domain, and format diversity, while cross-tensor synthesis and accelerator-aware decoding are future work.
- Conclusion: Brevis synthesizes typed, self-contained programs using reversible operators that expose value, sequence, field, and plane structure.Literal codecs encode resulting streams, while target-directed expansion preserves bit exactness by construction.
- Conclusion: 33.93% storage reduction: Brevis reduces 2.130 TB of checkpoint data to 1.41 TB across 10 public checkpoints.The checkpoints span language, audio, and image generation models.
- Conclusion: 30.87% smaller archives: Brevis outperforms four general-purpose compressors at the highest reported reduction.A checkpoint-specific production prior guides bounded A* search, and exact serialized size determines the selected program.
- Limitations: The evaluation covers public model checkpoints rather than all tensor workloads, with limited and partly confounded domain and format diversity.The current implementation synthesizes each tensor independently.
- Limitations: Cross-tensor synthesis, broader corpora, and accelerator-aware decoding remain future work.These directions address limitations of the current per-tensor synthesis implementation.
- Artifact Availability: The anonymous implementation, datasets, and evaluation scripts are included in the project’s GitHub repository.Repository: https://github.com/jiekeshi/Brevis.