DeepZip: Lossless Data Compression using Recurrent Neural Networks

Related Work

• LZ77 asymptotically reaches the optimal compression rate for stationary sources without source statistics; Gzip combines it with Huffman coding for pointers and other streams.LZ77 searches earlier text for matching substrings and stores pointers to them.Source: Related Work · p. 2
• Arithmetic coding encodes streams given a probability model; CTW and PPM mix multiple models to generate predictions.LZMA combines LZ77 with arithmetic coding.Source: Related Work · p. 2
• Earlier character-level RNN compression was competitive on text but less so for complex, longer-memory sources; later CMIX mixed over 2,000 models using an LSTM context mixer.A preceding framework used an RNN as a context mixer for multiple compressors; CMIX still requires source-dependent context compressors, unlike DeepZip.Source: Related Work · p. 2

Methods

• DeepZip is presented as a framework for losslessly compressing a stream over an alphabet, with specific models described for the experiments.Source: Methods · p. 2

Framework Overview

• The predictor estimates each symbol’s conditional probability from the previous K symbols and passes that distribution to the arithmetic coder.K is a hyperparameter.Source: Framework Overview · p. 2
• The arithmetic coder acts as a reversible finite-state machine, encoding the estimated distribution into a state whose bits form the compressed sequence.Source: Framework Overview · p. 3

Encoding-Decoding Mechanism

• The predictor is trained on the sequence and its weights are stored; arithmetic coding uses its symbol probabilities, with a shared uniform prior for the initial K symbols.The decoder uses the same prior and stored weights to reproduce the predictor’s probabilities for subsequent symbols.Source: Encoding-Decoding Mechanism · p. 3, Figure 1 · p. 4
• Correct reconstruction requires a causal predictor using only past symbols and perfectly symmetric probability estimates at the encoder and decoder.The decoder mirrors the encoder: it decodes the initial K symbols with the uniform prior and subsequent symbols with predictions from the stored model weights.Source: Encoding-Decoding Mechanism · p. 3
• The predictor designs explored include fully connected networks and recurrent models; a dense model maps the previous K symbols to a softmax distribution over the next character.The output distribution has one probability for each symbol in the sequence alphabet.Source: Encoding-Decoding Mechanism · p. 3
• The single-output recurrent predictor feeds the past K symbols to bidirectional GRU cells and applies softmax to the final hidden state.The experiments use the bidirectional GRU variant, denoted biGRU.Source: Encoding-Decoding Mechanism · p. 4, Figure 2 · p. 4
• To address vanishing gradients and weak dependence on distant symbols, the concatenated LSTM framework combines outputs from K LSTM cells through a dense layer and softmax.Its input comprises the past K symbols, with explicit dependence on all of them through a fully connected layer on top of the LSTM embeddings.Source: Encoding-Decoding Mechanism · pp. 4–5, Figure 2 · p. 4

Training

• With K=64, overlapping 65-symbol windows map 64-symbol contexts to next-symbol targets; Adam trains batches of 1024 for up to 10 shuffled epochs with early stopping and best-loss checkpointing.LSTM and GRU models use cuDNN acceleration, reducing training time by approximately 7×.Source: Training · p. 5
• The authors omit cross-validation and attempt to overfit because the model weights are stored in the compressed representation and used only to predict the training data.Source: Training · p. 5
• Arithmetic coding encodes probability-conditioned symbols into bits; DeepZip parallelizes coding by splitting sequences into B=1000 segments, decoding them independently, and concatenating the results.Segment probabilities are computed in parallel during encoding, while decoding is performed symmetrically across segments.Source: Training · pp. 5–6
• Equation 5 shows average arithmetic-coded bits per symbol is very close to the Equation 4 loss, making categorical cross entropy the optimal training loss in DeepZip.The predictor’s joint probability distribution is used to connect the chain-rule expression for the sequence loss to arithmetic coding.Source: Training · pp. 5–6

Experiments

• DeepZip’s FC, biGRU, and LSTM-multi predictors are benchmarked against Gzip, BSC, GeCo for genomic data, and ZPAQ for text.BSC is described as a BWT-based compressor that improves on Gzip while remaining computationally efficient.Source: Experiments · p. 6
• Real-data tests cover Human chr1, C. Elegans chr1 and whole genome, PhiX quality values, and the 100MB text8 ASCII dataset.The C. Elegans whole genome concatenates six chromosomes; PhiX quality values have four possible symbols and are highly compressible because most are the best-quality value.Source: Experiments · pp. 6–7
• Human chr1 is a challenging compression source because genome sequences contain repeated regions that state-of-the-art compressors have been unable to capture.The sequence alphabet is {A, C, G, T, N}, with N denoting an unknown nucleotide.Source: Experiments · p. 6
• Synthetic sources with known entropy rates let the authors gauge compressor performance against entropy rate as a lower bound on compression ratio, using IID Bern(0.1) binary data and Markov-k sources with k = {20, 30, 40, 50}.Markov-k sources are closely related to the lagged Fibonacci pseudorandom generator and are difficult for most traditional compressors, even though their entropy rate is 0.Source: Experiments · p. 7
• The synthetic HMM uses a hidden Markov-k process with added IID Bern(0.1) noise and tests k = {20, 30, 40}.Its hidden process is H = Xn−1 + Xn−k−1 (mod M).Source: Experiments · p. 7

• DeepZip outperforms Gzip by about 20% on text and genomic data, is usually comparable to BSC, and biGRU generally performs best.DeepZip compresses the C. Elegans genome slightly better than BSC.Source: Table 1 · p. 8, Results and discussion · p. 8
• Model weights contribute significantly to compressed size, especially for short sequences; DeepZip stores them as uncompressed 32-bit floats.Attempts to use 16-bit floats and TensorFlow Lite faced stability and compatibility issues; the authors suggest techniques similar to [23] could reduce model size without losing compression performance, and models may be shared across similar sequences.Source: Table 2 · p. 8, Results and discussion · p. 8
• GeCo compresses human and C. Elegans genomes 5–10% smaller than DeepZip, while ZPAQ reaches 17.5MB on text8, 25% below DeepZip.The paper attributes these results to specialized compressors’ handcrafted, dataset-optimized mechanisms and ability to account for non-stationarity; DeepZip instead performs reasonably well across varied datasets.Source: Results and discussion · p. 8
• On synthetic sequences with long-term dependencies, DeepZip exploits structure that traditional compressors miss, though model storage and arithmetic-coding overhead leave a gap from entropy.XOR40 performance can depend strongly on training parameters because its pseudo-random source makes suitable optimization minima difficult to find.Source: Table 3 · p. 9, Results and discussion · p. 9
• For typical 10MB datasets, each training epoch takes 1–2 hours and encoding or decoding takes approximately 5–10 minutes.The reported setup used a 12GB NVIDIA TITAN X GPU and batch size 1024; datasets were typically trained for 3–4 epochs, while encoding and decoding each require a single forward pass through the neural network model.Source: Results and discussion · p. 9

Conclusion

• DeepZip improves over Gzip on varied real datasets and achieves near-optimal compression on synthetic datasets.The paper proposes a neural-network prediction-based framework for lossless sequential-data compression.Source: Conclusion · p. 9
• Future work proposes attention models to improve prediction and compression, and fine-tuning model weights during compression and decompression to adapt to non-stationary sources.The aim is to adapt quickly as source statistics change.Source: Conclusion · p. 9
• The authors suggest that compression experiments, viewed through information theory, could help improve understanding of neural-network models.Source: Conclusion · pp. 9–10