PaperlayerView exact revision on arXiv

Source-linked AI summary

Significantly Improving Lossy Compression for Scientific Data Sets Based on Multidimensional Prediction and Error-Controlled Quantization

Dingwen Tao, Sheng Di, Zizhong Chen, Franck Cappello

arXiv:1706.03791v1cs.IT

TL;DR

Large-scale scientific simulations produce data volumes that strain storage, transmission, and analysis, creating a need for lossy compression with controlled errors. The paper develops a compressor combining multidimensional prediction with adaptive error-controlled quantization and variable-length encoding. Across production scientific data sets, it reports higher compression factors and lower errors than competing compressors on average.

  • Problem

    Large-scale scientific applications generate data volumes that challenge storage, transmission, and analysis, while compression must preserve user-specified error bounds.

  • Method

    The method combines multidimensional prediction, adaptive error-controlled quantization, and variable-length encoding for scientific data.

  • Results

    The compressor's average compression factor is more than 2x that of the second-best compressor, while its average compression error is reduced by more than 3.8x on ATM, APS, and hurricane data sets.

  • Takeaways & Limitations

    SZ-1.4 provides an implemented and released approach for evaluating error-controlled lossy compression on production scientific data sets.

Abstract

from arXiv · show

Today's HPC applications are producing extremely large amounts of data, such that data storage and analysis are becoming more challenging for scientific research. In this work, we design a new error-controlled lossy compression algorithm for large-scale scientific data. Our key contribution is significantly improving the prediction hitting rate (or prediction accuracy) for each data point based on its nearby data values along multiple dimensions. We derive a series of multilayer prediction formulas and their unified formula in the context of data compression. One serious challenge is that the data prediction has to be performed based on the preceding decompressed values during the compression in order to guarantee the error bounds, which may degrade the prediction accuracy in turn. We explore the best layer for the prediction by considering the impact of compression errors on the prediction accuracy. Moreover, we propose an adaptive error-controlled quantization encoder, which can further improve the prediction hitting rate considerably. The data size can be reduced significantly after performing the variable-length encoding because of the uneven distribution produced by our quantization encoder. We evaluate the new compressor on production scientific data sets and compare it with many other state-of-the-art compressors: GZIP, FPZIP, ZFP, SZ-1.1, and ISABELA. Experiments show that our compressor is the best in class, especially with regard to compression factors (or bit-rates) and compression errors (including RMSE, NRMSE, and PSNR). Our solution is better than the second-best solution by more than a 2x increase in the compression factor and 3.8x reduction in the normalized root mean squared error on average, with reasonable error bounds and user-desired bit-rates.

I. INTRODUCTION

Scientific simulations generate data volumes that challenge storage, transmission, and analysis, while existing compression methods face trade-offs between compression rate, data irregularity, and strict error control. The paper proposes an error-controlled lossy compressor and evaluates it across scientific workloads using multiple accuracy, size, and speed metrics.

  • Motivation: 20 PB can be generated by a single 1 trillion-particle HACC simulation, while CMIP6 raw-data requirements are estimated to exceed 10 PB.These volumes create storage constraints for large-scale scientific applications.
  • Motivation: Lossless compression generally reaches only up to 2:1, motivating lossy compression with user-set error controls for large-scale scientific experiments.The paper frames controlled loss as necessary to meet both data-accuracy and execution-scale demands.
  • Research challenge: Existing predictors depend on local smoothness, but sharp or spiky changes can reduce prediction accuracy and degrade compression quality.Other methods mitigate smoothness dependence but may lack strict user-controlled error bounds, while ZFP may not respect bounds over huge value ranges.
  • Contributions: The proposed compressor combines multidimensional prediction, adaptive error-controlled quantization, and variable-length encoding to handle irregular data while respecting user-set error bounds.The work also implements SZ-1.4 and evaluates it on production data across multiple scientific domains.
  • Evaluation: Compression quality is assessed through pointwise and average errors, correlation, compression factor or bit-rate, and compression/decompression speed.The evaluation includes RMSE, NRMSE, PSNR, Pearson correlation, storage cost, and throughput-related measures.
  • Error control: The compressor guarantees absolute and/or value-range-based relative errors within user-specified bounds for every data point.The bounds are expressed as |eabsi| < ebabs and/or |ereli| < ebrel for 1 ≤ i ≤ N.

III. PREDICTION MODEL BASED ON MUTIDIMENSIONAL SCIENTIFIC DATA SETS

The compression pipeline first predicts each multidimensional data value, then applies adaptive error-controlled quantization and finally variable-length encoding to the resulting codes.

  • Prediction model: The pipeline predicts every data value with a multilayer model before quantization and encoding.The method also analyzes how to choose the best prediction layer.
  • Prediction model: Adaptive quantization uses a variable number of intervals, and variable-length encoding exploits the resulting uneven code distribution.This section presents the prediction stage before the paper's quantization and encoding stages.

A. Prediction Model for Multidimensional Scientific Data Sets

The paper develops a multilayer prediction model that uses processed neighboring values across dimensions to predict each data point. It derives explicit two-dimensional and generic multidimensional formulas, defining the layer parameter n.

  • The model processes data from lower to higher dimensions and predicts each point using previously processed neighboring values organized into layers.For a current point (i0, j0), processed points have i < i0 or i = i0 and j < j0.
  • An n-layer data subset contains processed points within layers one through n, including n(n+2) symmetric points for two-dimensional prediction.The subset is defined by offsets satisfying 0 ≤ k1 + k2 ≤ 2n − 1.
  • The two-dimensional prediction surface uses maximum order 2n − 1 and is determined from n(2n + 1) data points.The corresponding linear system provides an explicit solution for the prediction surface.
  • The predicted value at the current coordinate is expressed as a linear combination of values in the n-layer data subset.The paper uses the prediction surface value f(i0, j0) to predict V(i0, j0).
  • The paper derives a generic multilayer prediction formula for data sets of arbitrary dimensional size d, where n specifies the prediction layer.The Lorenzo predictor is a special case when n = 1.

B. In-Depth Analysis of the Best Layer for Multilayer Prediction Model

The paper analyzes how to choose the prediction layer by balancing additional nearby information against errors introduced when predictions use preceding decompressed values. It defines prediction hitting rate and shows that the best layer depends on the data and reconstruction setting.

  • Choosing the best n is critical because prediction quality depends on the number of layers used in the multilayer model.The paper seeks the best n for the generic prediction formula.
  • Increasing layers can add useful multidimensional information, but more distant data may be less correlated and preceding decompressed values can degrade prediction accuracy.The analysis connects layer count with compression factor, compression error, and compression/decompression speed.
  • Prediction hitting rate is the proportion of data points whose prediction difference from the original value does not exceed the error bound.It is defined as RP H = NP H / N.
  • For ATM data, 2-layer prediction is more accurate than other layers when predictions use original data values.The table separately reports hitting rates obtained with original and preceding decompressed values.
  • The best prediction layer is data-dependent, so users can set the layer value; the compressor defaults to n = 1.The paper notes that different scientific data sets may have different best layers.

IV. AEQVE: ADAPTIVE ERROR-CONTROLLED QUANTIZATION AND VARIABLE-LENGTH ENCODING

AEQVE combines adaptive error-controlled quantization with variable-length encoding to further optimize compression quality.

  • AEQVE adaptively controls quantization and then applies variable-length encoding to reduce the size of quantization codes.The model includes adaptive interval selection and exploits the uneven code distribution produced by quantization.

A. Error-Controlled Quantization

The error-controlled quantizer expands a multilayer prediction into uniformly spaced, nonoverlapping intervals whose representatives remain within the user-set error bound. Its uneven code distribution is then compressed with variable-length encoding.

  • The quantizer first computes a multilayer prediction, then expands it into 2^m − 2 second-phase predicted values by linearly scaling the error bound.Adjacent predicted values are separated by twice the error bound.
  • Each predicted value forms an interval of length twice the error bound, ensuring that the intervals do not overlap.The construction expands each predicted value by one error bound in both directions.
  • Values inside an interval are predictable and represented by that interval’s predicted value, guaranteeing an error below the error bound.Values outside all intervals are marked unpredictable.
  • With 2^m − 1 predictable intervals plus one unpredictable code, quantization uses m bits to represent all 2^m codes.The unpredictable data use code 0, while interval codes represent predictable data.
  • Unlike vector quantization, the proposed error-controlled quantization uses fixed-length intervals and controls compression error for every predictable data point.Vector quantization uses nonuniform intervals whose longer intervals can exceed twice the error bound.
  • For ATM data, 255 intervals with m = 8 produce an uneven code distribution whose nonuniformity depends on previous prediction accuracy.Variable-length encoding, including Huffman coding, exploits this distribution by assigning fewer bits to common symbols.

B. Adaptive Scheme for Number of Quantization Intervals

The method adaptively selects quantization intervals to maintain prediction hitting rates while reducing storage overhead through quantization and variable-length encoding.

  • Adaptive interval selection: The compressor chooses quantization intervals by balancing prediction hitting rate against the higher overhead of storing unpredictable data.Fewer intervals reduce code storage, but insufficient intervals can lower prediction hitting rates under stringent error bounds.
  • Adaptive interval selection: Prediction hitting rates can collapse abruptly as error bounds decrease, even with many quantization intervals.With 511 intervals, the rate drops from 97.1% to 41.4% at ebrel = 10^-6.
  • Adaptive interval selection: For ebrel = 10^-4 and 10^-5, the method recommends 511 intervals for ATM data and 15 and 63 intervals, respectively, for hurricane data.These recommendations reflect dataset-dependent prediction behavior shown in Figure 4.
  • Adaptive interval selection: Users specify m to obtain 2^m − 1 quantization intervals, then increase or decrease the interval count when the hitting rate falls below the target threshold.A tuned interval count can provide optimized compression factors when accuracy requirements remain stable.
  • Compression pipeline: The complete algorithm predicts each point, quantizes its prediction error, stores unpredictable values with binary-representation analysis, and applies variable-length encoding to quantization codes.It records each decompressed value for future prediction and precomputes prediction coefficients once.

V. EMPIRICAL PERFORMANCE EVALUATION

The empirical evaluation tests SZ-1.4 on climate, X-ray, and hurricane simulation data and compares it with established lossless and lossy compressors.

  • Data sets and baselines: SZ-1.4 is evaluated on single-precision 2D ATM, 2D APS, and 3D hurricane scientific data sets.The data come from climate simulations, X-ray research, and hurricane simulation, respectively.
  • Data sets and baselines: The comparison includes lossless compressors GZIP and FPZIP and lossy compressors ZFP, SZ-1.1, and ISABELA.The evaluation examines compression quality across these production scientific data sets.

A. Compression Factor

SZ-1.4 achieves the highest compression factors among the compared methods under the evaluated value-range-based relative error bounds, while ZFP may violate specified absolute error bounds for huge value ranges.

  • Compression-factor comparison: SZ-1.4 has the best compression factor among GZIP, FPZIP, ZFP, SZ-1.1, and ISABELA within relative error bounds of 10^-3 to 10^-6.The comparison uses absolute bounds computed from each relative bound and the global data value range.
  • Compression-factor comparison: For ATM at ebrel = 10^-4, SZ-1.4 averages 6.3× compression versus ZFP’s 3.0× and SZ-1.1’s 3.8×.It also exceeds ISABELA, FPZIP, and GZIP under the same condition.
  • Compression-factor comparison: For APS at ebrel = 10^-4, SZ-1.4 averages 5.2× compression versus ZFP’s 2.9× and SZ-1.1’s 3.0×.The reported average also exceeds the three other comparison methods.
  • Compression-factor comparison: For hurricane data at ebrel = 10^-4, SZ-1.4 averages 21.3× compression.The passage reports this as the highest compression factor in the comparison, though the remaining baseline values are truncated.
  • Error-bound behavior: ZFP can exceed the user-set absolute error bound when the data value range is huge because of fixed-point alignment.For ATM variable CDNUMC, ebabs = 10^-7 produces a compression error of 0.123668 at a data value of 6.936168.

B. Rate-Distortion

The rate-distortion evaluation compares compressors at matched bit-rates and error conditions, showing that SZ-1.4 generally provides the strongest reconstruction quality, especially on ATM, APS, and higher-rate hurricane data.

  • Evaluation setup: Rate-distortion curves compare reconstruction quality at matched bit-rates using PSNR, while fixed-error comparisons examine compression factors under comparable maximum errors.Higher bit-rates generally correspond to higher reconstructed-data quality.
  • 2D data sets: SZ-1.4 has the best rate-distortion curve on the 2D ATM and APS data sets.The paper reports substantially higher PSNR than the competing methods at representative rates.
  • 2D data sets: At 8 bits/value on ATM, SZ-1.4 reaches about 103 dB PSNR, 14 dB above ZFP’s 89 dB and representing more than 5× higher accuracy.The reported accuracy is also more than 7× SZ-1.1’s and 103× ISABELA’s.
  • 2D data sets: At 8 bits/value on APS, SZ-1.4 reaches about 96 dB PSNR, 9 dB above ZFP’s 87 dB and representing 2.8× higher accuracy.The paper also reports 8× and 790× accuracy relative to SZ-1.1 and ISABELA, respectively.
  • 3D data sets: On 3D hurricane data, SZ-1.4 is close to ZFP at 2 bits/value but has higher PSNR at higher bit-rates.At 8 bits/value, SZ-1.4 reaches about 182 dB, 11 dB above ZFP’s 171 dB and 47 dB above SZ-1.1’s 135 dB.
  • Evaluation scope: The reported rate-distortion results are limited to rates below 16 bits/value, corresponding to compression factors above 2.The paper motivates this range because lossless compressors can provide compression factors up to 2.

C. Pearson Correlation

The evaluation examines SZ-1.4's correlation, speed, error autocorrelation, scalability, and I/O performance against established lossy compressors. Results show strong speedups over SZ-1.1 and ISABELA, near-linear parallel scaling, and reduced total I/O time, while relative speed and error-correlation trade-offs remain dataset-dependent.

  • C. Pearson Correlation: SZ-1.4 is evaluated against ZFP and SZ-1.1 using Pearson correlation coefficients under matched maximum compression-error bounds.The comparison uses ZFP's maximum compression error as the input error bound for all three compressors.
  • D. Speed: 2.2x and 32x: SZ-1.4 is faster on average than SZ-1.1 and ISABELA, respectively, for the 2D ATM and APS data sets.For 3D hurricane data sets, SZ-1.4 is instead 2.4x faster than SZ-1.1 and 62x faster than ISABELA.
  • D. Speed: 50% and 48%: SZ-1.4 is slower on average than ZFP for compression and decompression, respectively.The authors attribute this to prioritizing high compression factors rather than optimizing performance.
  • E. Autocorrelation of Compression Error: 4×10^-3 versus 0.25: SZ-1.4 has lower maximum error autocorrelation than ZFP for FREQSH, but 0.5 versus 0.23 for SNOWHLND.Across APS and hurricane data, SZ-1.4 generally has lower autocorrelation on relatively low-compression-factor data, while ZFP is lower on relatively high-compression-factor data.
  • VI. DISCUSSION: Nearly 100% parallel efficiency from 1 to 128 processes demonstrates linear compression and decompression speedup; efficiency declines to about 90% beyond 128 processes.The reported degradation is attributed to node-internal limitations.
  • VI. DISCUSSION: At 32 or more processors, compressing and writing reduced data takes less total time than writing and reading the initial ATM data.The evaluation normalizes compression, decompression, and I/O times to compare total data-movement cost.

VII. RELATED WORK

Scientific compression methods include lossless and lossy approaches, with lossy methods addressing the limited compression factors of lossless compression. Existing lossy techniques differ in error control, prediction assumptions, and compression-factor limitations, motivating the paper's multidimensional, error-controlled design.

  • VII. RELATED WORK: Scientific data compression algorithms are categorized as lossless or lossy compression.
  • VII. RELATED WORK: Up to 2:1: lossless compressors such as GZIP, LZ77, and FPZIP generally provide limited compression factors.
  • VII. RELATED WORK: ISABELA uses B-spline interpolation after sorting data series but stores original indices, reducing its compression factor for large data sets.
  • VII. RELATED WORK: NUMARCK and SSEM use vector quantization but cannot guarantee bounded compression error and have compression-factor limitations.
  • VII. RELATED WORK: The paper proposes a generic multidimensional prediction model and an adaptive error-controlled quantization and variable-length encoding model for irregular data.
  • VII. RELATED WORK: More than 2x: the reported average compression factor exceeds the second-best compressor, while average compression error is reduced by more than 3.8x.These results are reported for ATM, APS, and hurricane data sets with reasonable error bounds and user-desired bit-rates.
Loading 1706.03791v1…