PaperlayerView exact revision on arXiv

Source-linked AI summary

Nacrith: Neural Lossless Compression via Ensemble Context Modeling and High-Precision CDF Coding

Roberto Tacconelli

arXiv:2602.19626v2cs.ITcs.CL

TL;DR

LLM-based compression can improve prediction but prior systems rely on massive models, fine-tuning, or coarse CDF quantization. Nacrith combines a small transformer with adaptive token predictors, high-precision arithmetic coding, and practical acceleration and binary-format support. It achieves the best evaluated natural-language compression results, including 0.9389 bpb on enwik8 and 0.918 bpb on alice29.txt, while also supporting arbitrary binary files.

  • Problem

    Prior LLM-based compressors use massive models, require fine-tuning, or sacrifice expressiveness through rank-based encoding and coarse CDF quantization.

  • Method

    Nacrith combines a pre-trained transformer with online token-level N-gram and adaptive bias predictors, high-precision CDF coding, and a hybrid format for text and non-text data.

  • Results

    Nacrith achieves the best evaluated enwik8 result at 0.9389 bpb and reaches 0.918 bpb on alice29.txt, outperforming the reported prior systems on both benchmarks.

  • Takeaways & Limitations

    Nacrith combines adaptive ensemble modeling with a pre-trained transformer and extends LLM-based compression to arbitrary binary files.

  • Takeaways & Limitations

    Compression runs at ∼21 tokens/s on a GTX 1050 Ti, and broader OOD evaluation is needed because several benchmarks may overlap with training data.

Abstract

from arXiv · show

We present Nacrith, a lossless compression system that combines a 135M-parameter transformer language model (SmolLM2-135M) with an ensemble of lightweight online predictors and a 32-bit arithmetic coder, achieving the best compression results among the systems evaluated in this study on natural language text. Beyond the base LLM-plus-arithmetic-coding paradigm, Nacrith introduces several contributions: (1) a CDF precision upgrade from 2^16 to 2^24 that eliminates ~75% of quantization overhead caused by minimum-probability floors in large vocabularies; (2) a token-level N-gram model for fast local predictions; (3) an adaptive log-space bias head correcting per-document LLM errors via online gradient descent; (4) confidence-based LLM skip for accelerating highly predictable tokens; (5) a hybrid binary format (NC06) extending neural compression to arbitrary binary files--to our knowledge a first among LLM-based compressors; (6) a llama cpp inference backend achieving ~7x faster single-token decode than PyTorch; (7) parallel multi-GPU compression across up to 8 workers; and (8) native KV cache sliding window reducing per-slide cost by ~37x. The system requires only ~500 MB of GGUF weights and ~1.2 GB VRAM per worker, running on consumer GPUs. On alice29 (Canterbury Corpus, 152 KB), Nacrith achieves 0.918 bits per byte (bpb)--outperforming gzip by 3.1x, bzip2 by 2.5x, CMIX v21 by 44%, and ts_zip by 20%, while compressing below the 0th-, 1st-, and 2nd-order byte-level Shannon entropy bounds. On enwik8 (100 MB), Nacrith achieves 0.9389 bpb (11.74%), surpassing ts_zip (~1.11 bpb) by 15% and FineZip (1.024 bpb) by 8% despite using a 60x smaller model with no fine-tuning. An out-of-distribution (OOD) evaluation on a document published after the model's training cutoff confirms these gains are not memorization artifacts, achieving 0.723 bpb on unseen text.

1. Introduction

Nacrith combines a transformer language model with online prediction and arithmetic coding to address practical limitations of neural compression. Its contributions target coding precision, document adaptation, and inference efficiency.

  • Contributions: Nacrith combines a transformer language model with arithmetic coding and lightweight online predictors for lossless compression.The system adds token-level N-grams, adaptive mixing, and an adaptive bias head to improve document-specific prediction.
  • Contributions: 75% of the CDF range is consumed by minimum-probability floors when V = 49,152 tokens and CDFtotal = 2^16.The upgrade to 2^24 reduces floor overhead from approximately 2 bits/token to approximately 0.004 bits/token.
  • Contributions: The token-level N-gram model adapts online to local document patterns, while an adaptive context mixer weights predictors according to current-document performance.The mixer blends LLM and N-gram predictions using online updates.
  • Contributions: The adaptive log-space bias head corrects systematic per-document LLM over- or under-prediction using online SGD.This provides a second document-adaptation mechanism beyond context mixing.
  • Contributions: The NC06 hybrid binary format applies neural compression to text-like regions and traditional codecs to opaque binary data.The format extends the system beyond text files.
  • Contributions: llama.cpp achieves approximately 7× faster single-token decoding than PyTorch, while multi-GPU workers and KV-cache sliding support faster large-input compression.The implementation also supports up to eight concurrent workers and retains HuggingFace tokenization for correctness.

2. Related Work

Prior compressors combine statistical adaptation, context mixing, or neural language models, but face scale, resource, or generality trade-offs. Nacrith positions itself as a smaller pre-trained-LLM system with strong results on both large and small text benchmarks and binary support.

  • Classical and adaptive compressors: Dictionary and entropy coders exploit local repetition or frequency structure, while PPM and PAQ extend adaptive context modeling with arithmetic coding.These methods establish the statistical-compression foundation against which neural systems are compared.
  • Classical and adaptive compressors: CMIX reaches approximately 1.17 bpb on enwik8 but requires 16–64 GB RAM and compresses alice29.txt at 1.63 bpb.Its performance degrades on smaller files because the adaptive context ensemble has less data to warm up.
  • Neural compression: NNCP achieves approximately 1.19 bpb on enwik8, but online model-training overhead raises alice29.txt output to approximately 3.96 bpb.The small-file penalty makes its scale dependence explicit.
  • Nacrith: Nacrith mixes a pre-trained LLM and token-level N-gram predictor with online weight adaptation.This combines a transformer prior with document-adaptive context modeling.
  • Nacrith: Nacrith achieves 0.918 bpb on alice29.txt and is described as the only LLM-based compressor supporting arbitrary binary files.The result outperforms ts_zip and CMIX on the small benchmark.

3. Method

Nacrith tokenizes input, predicts each token with an ensemble, converts probabilities to a high-precision CDF, and arithmetic-encodes the result. It extends this pipeline to hybrid binary files and parallel chunked compression.

  • Overview: Nacrith blends ensemble token probabilities and feeds them to an arithmetic coder, allowing deterministic lossless reconstruction.The decoder reproduces identical predictions and recovers each token.
  • Overview: NC06 segments non-text input into text-like and binary regions, applying neural compression to text and traditional codecs to binary chunks.Large inputs can be split into chunks and compressed in parallel across GPU workers.
  • Compression pipeline: The pipeline initializes an LLM KV cache, N-gram model, mixer, and adaptive head before iteratively predicting, encoding, and updating after each token.The algorithm returns the finished arithmetic-coded stream.
  • Compression pipeline: The N-gram predictor can skip LLM inference when its entropy is below a threshold; otherwise the LLM prediction is adjusted before CDF conversion.This confidence-based branch targets highly predictable tokens.
  • Neural probability model: SmolLM2-135M uses 135 million parameters, a 49,152-token BPE vocabulary, and FP32 inference for deterministic probability distributions.Determinism is required for lossless reconstruction across hardware.
  • Inference backend: llama.cpp performs GPU inference with approximately 7× faster single-token incremental decoding than PyTorch, while HuggingFace handles tokenization and detokenization.The dual tokenizer arrangement avoids dropped content from 47 whitespace and repeat tokens.
  • Inference backend: The system loads approximately 500 MB of GGUF weights, transfers the vocabulary logits to CPU, and optionally temperature-scales them before softmax.At τ = 1.0, temperature scaling is a no-op.
  • Inference backend: If llama.cpp is unavailable, Nacrith falls back to PyTorch with CUDA Graphs on GPU or a dynamic KV cache on CPU.This provides an alternative execution path for portability.

V · MIN_PROB

The token vocabulary makes 2^16 CDF coding inefficient because minimum-probability floors consume most of the available range. Nacrith addresses this bottleneck with 2^24 CDF precision while combining online N-gram adaptation and memory-efficient implementation.

  • CDF precision: 2^16 CDF coding leaves only approximately 25% of the range for actual probability information after minimum-probability allocation.The resulting quantization error degrades arithmetic coding for peaked distributions.
  • CDF precision: The CDF-24 upgrade changes the total range to 2^24, reducing minimum-probability floor overhead to approximately 0.004 bits/token.The remaining bins are allocated proportionally to token probabilities, with residual counts assigned to the maximum-probability token.
  • CDF precision: The 32-bit arithmetic coder remains safe because the minimum narrowed symbol width is at least 128 after renormalization.This follows from R·MIN_PROB/T ≥ 2^31/2^24 = 128.
  • N-gram adaptation: The interpolated token-level N-gram model uses orders 1–4 and updates online to capture document-specific vocabulary and phrasing.It complements the LLM with local statistical regularities.
  • Implementation: Hash-based context keys and capped continuation dictionaries reduce N-gram memory from approximately 3.6 GB to approximately 128 MB per worker.The reported reduction is 28× and makes multiworker operation feasible on consumer GPUs.

3.6 Adaptive Context Mixer

Nacrith combines predictions from an LLM and lightweight online models, adapting their weights and correcting document-specific LLM errors during compression.

  • Linear mixing preserves the dominant model’s confidence for arithmetic coding.With p_llm(t) = 0.90 and w_llm = 0.85, the mixed probability is ≥0.765.
  • Exponential weights updates increase the weights of models that consistently predict observed tokens well.Weights are renormalized after each observed token.
  • Initial weights are LLM-dominant, with w_llm = 0.85 and the remaining 0.15 split equally among secondary models.
  • A 100-token warmup uses the LLM alone while secondary models accumulate data for reliable distributions.
  • The adaptive bias head updates LLM log-probabilities by SGD after each observed token to correct document-specific over- and under-prediction.It uses α = 0.001, preserves lossless symmetry through identical updates, and relies on float64 for bit-exact reproducibility under matched configurations.

3.8 Confidence-Based LLM Skip

Nacrith skips expensive LLM inference when the N-gram model is sufficiently confident, improving both throughput and compression on highly predictable text while using efficient KV-cache sliding.

  • When H(p_ng) < τ with τ = 1.5 bits, Nacrith skips the LLM forward pass and uses the N-gram prediction directly.The threshold was calibrated empirically on a held-out sample because the LLM adds little accuracy for highly predictable tokens.
  • 30–70% skip rates on highly compressible text substantially reduce GPU load while improving compression quality.The ablation identifies confidence-based skipping as the primary channel through which the N-gram model contributes to compression.
  • Native KV-cache manipulation reduces per-slide cost by ∼37×, from 693 ms to 19 ms.The implementation removes and shifts cached positions, then re-evaluates only the final token.
  • Sliding-window overhead becomes approximately 1 + 1/C ≈1.002× instead of the 4× overhead of full cache rebuilds.

3.11 Hybrid Binary Compression (NC06)

NC06 extends Nacrith to arbitrary binary files by separating text-like and opaque data, compressing each region with an appropriate codec and preserving reproducible ensemble settings.

  • NC06 classifies input into alternating text and binary chunks using printable-byte rules and short-run, gap, and adjacency heuristics.Short text runs below 64 bytes are demoted, binary gaps up to 8 bytes may be bridged, and small adjacent binary chunks are absorbed.
  • Binary chunks are concatenated and compressed with LZMA, gzip, or raw storage, while text chunks use the full neural ensemble.LZMA is used for blobs ≥4 KB, gzip for smaller blobs when beneficial, and raw storage otherwise.
  • NC06 stores flags, temperature, an entry table, binary data, and parallelized text streams so decompression can reproduce the ensemble configuration.Its extended header includes the NC06 magic, a version byte, and a structured table for alternating text and binary chunks.
  • Parallel GPU workers compress text chunks independently, each owning its own model, secondary predictors, mixer, and adaptive head.Independent state eliminates shared-state synchronization overhead.
  • NC05 and NC06 store per-chunk metadata including token count, bit count, and stream length.NC05 additionally records feature flags, temperature, and chunk count; NC06 extends the header for hybrid streams.

4. Experimental Setup

The evaluation measures Nacrith against traditional and neural compressors on Canterbury text and additional English prose using an accessible consumer-GPU setup.

  • Nacrith is evaluated on an NVIDIA GeForce GTX 1050 Ti with 4 GB VRAM to demonstrate practical accessibility.The system uses approximately 1.2 GB VRAM per worker, ∼500 MB GGUF F32 weights, and supports up to 3 concurrent workers on this GPU.
  • The comparison includes gzip, xz, bzip2, Brotli, Zstandard, CMIX v21, and ts_zip.Traditional compressors are tested at maximum settings, while CMIX v21 and ts_zip results come from published results.
  • alice29.txt is the primary benchmark: a 152,089-byte Canterbury Corpus excerpt widely used in compression research.
  • Additional evaluation uses asyoulik.txt and three custom English prose samples of 3 KB, 50 KB, and 100 KB.

5. Results

Nacrith achieves strong lossless compression across short literary text, large Wikipedia text, and post-cutoff government prose, with results shaped by dataset scale and text predictability. Ablations identify CDF-24 and confidence-based N-gram skipping as the largest contributors, while the adaptive head provides a smaller consistent gain.

  • Compression Results: 0.918 bpb on alice29.txt outperforms gzip by 3.1×, bzip2 by 2.5×, CMIX v21 by 44%, and ts_zip by 20%.All results are fully lossless.
  • Compression Results: 0.63–0.76 bpb on modern English prose approaches Chinchilla 70B’s 0.664 bpb on enwik9, while Shakespeare text reaches 1.30 bpb.The direct comparison is confounded by dataset size and training-data differences; archaic vocabulary is less predictable for a modern-English-trained model.
  • Shannon Entropy Analysis: 0.918 bpb on alice29.txt falls below the H0=4.57, H1=3.42, and H2=2.49 byte-level Shannon entropy bounds.These bounds capture only short-range byte correlations and are reference points rather than fundamental source-compressibility limits.
  • Cross-System Comparison: 0.9389 bpb on enwik8 surpasses FineZip’s 1.024 bpb by 8% despite using a 60× smaller model without fine-tuning.Classical compressors range from 1.989 bpb for xz -9 to 2.916 bpb for gzip -9.
  • Out-of-Distribution Evaluation: 0.723 bpb on post-cutoff government prose outperforms ts_zip by 25% and CMIX by 37%, while compressing 26% smaller than controlled FineZip.The controlled comparison uses the same SmolLM2-135M model and isolates architectural contributions.
  • Performance: 20–30 tok/s is the single-worker steady-state throughput, increasing to 60–90 tok/s with three parallel workers on a GTX 1050 Ti.The reported throughput settles as the KV cache fills to its 2,048-token steady state.
  • Ablation Study: −0.52 bpb from CDF-24 and −0.39 bpb from confidence-based N-gram skipping are the largest ablation improvements, compared with −0.015 bpb from the adaptive head.The adaptive head’s gain is small but consistent and synergizes with skipping.

6. Discussion

Nacrith’s strongest gains arise from high-precision CDF coding and confidence-based N-gram skipping, while its practical scope is balanced by speed, model-overhead, contamination, context-window, and language limitations.

  • Ablation and design implications: CDF-24 eliminates nearly all minimum-probability-floor overhead in the 49,152-token vocabulary, making precision the dominant coding improvement.The upgrade reduces floor overhead from approximately 2 bits/token to approximately 0.004 bits/token.
  • Ablation and design implications: Confidence-based N-gram skipping contributes −0.39 bpb, or −30%, by bypassing the LLM on tokens with entropy below 1.5 bits.The context mixer instead converges toward approximately 100% LLM weight on nonconfident tokens.
  • Comparison with prior systems: Nacrith achieves 0.918 bpb on alice29.txt versus 1.14 bpb for ts_zip and 0.9389 bpb on enwik8 versus approximately 1.11 bpb for ts_zip.The paper attributes the gap likely to ensemble contributions combined with CDF-24, while noting direct Chinchilla comparison is confounded by dataset size and training-data differences.
  • Comparison with prior systems: Nacrith outperforms FineZip at 0.918 versus 1.024 bpb on alice29.txt and 0.9389 versus 1.024 bpb on enwik8, using a 60× smaller model without fine-tuning.The comparison supports the importance of precise CDF quantization and ensemble context mixing beyond raw parameter count.
  • Limitations and future directions: At approximately 21 tokens/s on a GTX 1050 Ti, Nacrith is suitable for archival applications, while its approximately 500 MB model must be available at both endpoints.The model overhead is amortized over many files or large corpora.
  • Limitations and future directions: Nacrith loses dependencies beyond 2,048 tokens and may compress less efficiently near window boundaries; English-focused training also limits other-language performance.The paper proposes larger models and context windows, quantization, and ANS coding as future improvements.

7. Conclusion

Nacrith combines a small pretrained language model with online ensemble predictors and high-precision arithmetic coding. It achieves strong lossless compression on text, extends neural compression to arbitrary binary files, and operates with consumer-hardware resource requirements.

  • Key findings: CDF-24 eliminates approximately 75% of wasted CDF range caused by minimum-probability floors in Nacrith’s 49,152-token vocabulary.The paper identifies this as a previously uncharacterized bottleneck in LLM-based compression.
  • Key findings: 0.918 bpb on alice29.txt and 0.9389 bpb on enwik8 outperform the reported CMIX, ts_zip, and FineZip baselines in the authors’ experiments.Nacrith uses 135M parameters and approximately 500 MB of GGUF weights; the enwik8 comparison uses a 60× smaller model without fine-tuning.
  • Broader applicability: NC06 extends neural compression to arbitrary binary files by combining neural compression for text-like regions with traditional codecs for opaque data.The authors describe it as the first LLM-based binary compressor to their knowledge.
  • Broader applicability: The results support practical access to LLM-based compression on consumer hardware while remaining fully lossless.The conclusion connects this result to the compression–prediction equivalence identified by Shannon and formalized for LLMs.
  • Availability: Nacrith is open-source and available through the project repository.The paper provides the repository URL for code availability.
Loading 2602.19626v2…