Source-linked AI summary
The Case for Synthesized Data Compression
Ivan Iachnyk, Mihail Stoian, Luisa Neubauer, Ryan Marcus, Andreas Kipf
TL;DR
General-purpose compression algorithms serve many data types, but may not be optimal for specific datasets, while correctness remains challenging for synthesized database systems. The paper synthesizes data-specific codecs that can be checked against fixed inputs, and reports that they extend the compression-factor–decompression-speed Pareto front across five datasets.
Problem
General-purpose compression techniques are designed to serve many kinds of data, while synthesized database systems remain difficult to verify, particularly for inputs unlike those seen during synthesis.
Source: §1 · pp. 1–2Method
The paper uses coding agents to synthesize codecs for specific data and checks correctness by verifying byte-for-byte reconstruction of the fixed input.
Source: §1 · p. 2Results
Synthesized codecs extend the compression-factor–decompression-speed Pareto front across five datasets; on SQLStorm, they exceed bzip2-9's compression factor by 49% while decoding 34% faster.
Source: §1 · p. 3Takeaways & Limitations
Validated synthesized codecs can be embedded with their data in modern containers and run independently of the model.
Source: §1 · p. 2, §6 · p. 10The results characterize current agents under this protocol; the model, performance targets, and available tools influence what agents discover, leaving further headroom open.
Source: §6 · p. 10
Abstract
from arXiv · showhide
Compression is a fundamental tool in data management. Today's compression schemes are generally designed by hand to apply to many different types of data (e.g., run length encoding or arithmetic encoding), leading to a zoo of general-purpose algorithms which work well for many datasets, but may not be optimal for any specific dataset. Inspired by recent advancements in agentic coding, we propose synthesizing custom-tailored data compression algorithms on a per-dataset basis. Unlike prior work using agents to synthesize database components, a data-specific compression algorithm can be fully verified at construction time, alleviating most correctness concerns. Modern file formats, like AnyBlox and F3, allow including the synthesized code alongside the data itself. Experimentally, we show that our synthesis agent can expand the compression-factor/decompression-speed Pareto front, outperforming general-purpose methods by nearly $2\times$ in compression factor.
1 Introduction
The paper proposes synthesizing compression codecs for specific data rather than relying only on a zoo of general-purpose algorithms. It argues that codecs for fixed data can be checked byte for byte and reports synthesized codecs extending the compression-speed Pareto front across five datasets.
1 Introduction
- The paper proposes LLM-powered agents to synthesize custom compression codecs for specific pieces of data, as an alternative to relying on a diverse zoo of general-purpose algorithms.The motivation describes general-purpose techniques as intended to serve many kinds of data, without necessarily being optimal for any one dataset.Source: §1 · pp. 1–2
- For OpenStack logs, the synthesized codec stores constant characters in templates and packs changing timestamp and instance-ID digits into four-bit values for AVX-512 reconstruction.It outperforms LZ4 in decompression speed (13 GB/s vs. 8 GB/s) and compression ratio (11× vs. 9×).Source: §1 · p. 2
- The OpenStack codec is not presented as a novel compression scheme; the contribution is generating a custom-tailored codec automatically.The passage notes that a human expert could have built the codec without much trouble.Source: §1 · p. 2
- Correctness remains difficult to demonstrate for synthesized database systems, especially when inputs differ from those seen during synthesis; rigorous testing raises confidence but cannot prove correctness outright.The passage contrasts testing with formal methods, which require substantial specification and proof-strategy effort.Source: §1 · p. 2
- For a fixed data block, byte-for-byte reconstruction checks can essentially prove a synthesized codec correct, and modern containers can embed the codec with its data.The paper notes that this requires purely functional code and assumes a non-adversarial agent.Source: §1 · p. 2
- Synthesized codecs extend the compression-factor–decompression-speed Pareto front across five datasets; on SQLStorm, they exceed bzip2-9's compression factor by 49% while decoding 34% faster.On DBText, a synthesized codec achieves a 94% higher compression factor than FSST while processing selected rows 1.47–1.64× as fast.Source: §1 · p. 3
2 Synthesizing Compression Codecs
The synthesizer takes textual data and a performance goal, then iteratively evaluates proposed codecs for compression ratio, decompression speed, and correctness. Its prompts steer the agent toward data-specific optimization, and accepted codecs can be stored with compressed data in modern containers.
2 Synthesizing Compression Codecs
- The synthesizer takes textual input data and an executable performance goal, which may specify an absolute objective or a relative baseline target.The current prototype is limited to textual input.Source: §2 · p. 3
- A coding agent proposes codecs that the test environment measures for compression ratio, decompression speed, and correctness; the best-seen codec is exhaustively checked for byte-for-byte reconstruction before acceptance.Failed validation lets the agent revise and repeat; a passing codec and its compressed data can be copied into a container.Source: §2 · p. 3
- Prompts instruct the agent to overfit to the input data and provide unnamed reference performance values, giving targets to beat without steering it toward a particular algorithm.Without clear instructions, agents often tried to design new general-purpose compression algorithms.Source: §2 · p. 4
- Modern containers can embed decompression codecs, allowing synthesized codecs to be stored alongside compressed data; the agent therefore balances codec complexity against compression ratio.A compact domain-specific language for decompressors is noted as future work.Source: §2 · p. 4
3 Experiments
The experiments evaluate synthesized codecs against traditional baselines for block compression and DBText random access, then examine how reference values, external dependencies, and model choice affect outcomes.
3 Experiments
- Evaluation spans five datasets and four performance objectives, comparing synthesized codecs with traditional codecs using shared benchmarking and byte-exact correctness checks.The objectives target different trade-offs between compression factor and decompression speed.Source: §3 · p. 4
3.1 (RQ1) Can We Synthesize Competitive Block-Based Compression Codecs?
- Across four structured datasets, synthesized codecs extend the Pareto front: A1 decodes 22–61% faster than the fastest traditional baseline, while A2 exceeds the most compact baseline’s compression factor by 8–87% and still decodes faster.Figure 3 presents the block-decompression Pareto fronts for the traditional front-runners and standalone Astra codecs A1–A4.Source: Figure 3 · p. 5, §3.1 · pp. 5–6
- On DBText’s 23 heterogeneous columns, standalone A1 decodes about 30% faster than OnPair+ with slightly better compression, while A2 improves bzip2-9’s compression factor by 7% and decodes 8.4× as fast.Zstandard at high compression levels remains competitive between these endpoints.Source: §3.1 · p. 6
3.2 (RQ2) Can We Synthesize Codecs with Fast Random Access?
- For DBText random access, Astra A1 processes selected rows 1.67–1.87× as fast as FSST and outperforms OnPair+ at every tested selectivity, using 16.05 MB versus 24.04 MB for FSST and 27.71 MB for OnPair+.Figure 4(c) reports Astra with external dependencies and reference values.Source: §3.2 · p. 6, Figure 4 · p. 6
- A4 more than halves FSST’s footprint while matching or exceeding its throughput; A2 has the smallest footprint at 10.25 MB but processes only 17–23 M rows/s.Source: §3.2 · pp. 6–7
3.3 (RQ3) Ablation Studies
- Providing reference values gives the synthesizer a target for deciding when to stop or abandon an approach.Source: §3.3 · p. 7
- External dependencies help Astra find more compact representations; GLM produced four byte-exact codecs only with dependencies and reference values, but every evaluated GLM codec was surpassed by Astra on compression factor and decompression speed.The most compact synthesized codec combines front coding, PPMd, and Brotli.Source: §3.3 · p. 7
- At September 2026 API prices, 28 Astra runs cost an estimated $20.03–$98.50 (median $43.15), while four GLM runs cost $5.58–$15.85; decoding requires no model calls, and the authors expect model costs to drop.Source: §3.3 · pp. 7–8
4 Analysis of Synthesized Code
The section illustrates how synthesized codecs exploit structure in system logs, Python indentation, and database strings.
4.1 System Logs
- 87% higher compression factor than Brotli-11 and approximately 1.8× as fast decoding at the most compact endpoint.For system logs, dedicated decoding functions encode common field layouts; the codec reconstructs fixed text, timestamps, numeric values, and UUIDs directly.Source: §4.1 · p. 8
4.2 Python Code
- 8% higher compression factor than XZ-9 while decoding 76% faster.For Python, Astra uses three clues to classify indentation changes into 729 contexts; the decoder reconstructs indentation from the same clues without storing a context ID per line.Source: §4.2 · p. 8
4.3 Database String Columns
- Astra reuses one phrase encoder and decoder across nine DBText columns while giving each column its own learned dictionary.The columns include multilingual text, literary works, identifiers, and coordinates; unused phrases or grammar rules are removed, and A4 merges duplicate dictionary entries and compresses the dictionary.Source: §4.3 · p. 9
- DBText UUID records store a count of steps above a shared base and only the suffix bits that vary.Constant middle fields and some suffix bits become shared metadata; fixed-width records let the decoder locate a requested row directly.Source: §4.3 · p. 9
5 Related Work
Related work spans conventional string compressors, model-based text coding, and agents that develop compression algorithms.
Related Work
- FSST uses a shared static symbol table for independently decoded strings, while OnPair merges frequent adjacent substrings and OnPair+ optimizes compression factor and decompression throughput.The cited works are, [GV25], and, respectively.Source: §5 · p. 10
- LLMZip and FineZip encode text compactly with model predictions, but reconstruction requires model inference and adds computational costs.CandleZip uses agent-guiding hints and accounts for the information needed to replay them during decoding.Source: §5 · p. 10
- MinEvolve uses a language model to propose changes to an LZ77-style compressor, whereas this study has agents develop complete native codecs for fixed datasets.MinEvolve rejects candidates that fail reconstruction checks; validated codecs in this study no longer require the model for encoding or decoding.Source: §5 · p. 10
6 Conclusions & Future Work
Frontier models synthesize codecs that extend the compression-factor–decompression-speed Pareto front across five workloads, with byte-for-byte validation for fixed inputs. The authors identify synthesis-process choices as influential and leave the potential for further gains open.
Conclusions & Future Work
- Synthesized codecs extend the compression factor–decompression speed Pareto front across five workloads and can be checked byte for byte on fixed inputs.After validation, the codecs run independently of the model.Source: §6 · p. 10
- The model, performance targets, and available tools influence what agents discover, leaving the extent of further gains open.The authors propose exploring this headroom with prompt optimization, controlled ablations, and richer tools for data inspection, profiling, and experimental feedback.Source: §6 · p. 10
- The authors suggest exploring other DBMS components when their results can be exhaustively verified within a clearly defined scope.Source: §6 · p. 10