Source-linked AI summary
SZ3: A Modular Framework for Composing Prediction-Based Error-Bounded Lossy Compressors
Xin Liang, Kai Zhao, Sheng Di, Sihuan Li, Robert Underwood, Ali M. Gok, Jiannan Tian, Junjing Deng, Jon C. Calhoun, Dingwen Tao, Zizhong Chen, Franck Cappello
TL;DR
Scientific simulations and instruments produce data volumes that strain storage and I/O, while suitable error-bounded compression varies with datasets and error requirements. SZ3 addresses this with modular, composable prediction-based pipelines and adaptive predictor selection. Customized pipelines for real-world datasets improve compression ratios by up to 20% at the same data distortion compared with state-of-the-art compressors.
Problem
Scientific data volumes challenge storage and I/O, while existing compressors may not provide the best quality and performance for every dataset or error-bound requirement.
Method
SZ3 modularizes preprocessing, prediction, quantization, encoding, and lossless compression so users can compose customized pipelines and select predictors adaptively.
Results
Up to 20% improvement in compression ratios is achieved at the same data distortion compared with other state-of-the-art compressors.
Takeaways & Limitations
Customized SZ3 pipelines support efficient error-bounded compression for diverse data characteristics and user requirements, including GAMESS and APS datasets.
Takeaways & Limitations
SZ2’s lack of software architecture makes modification, extension, and correctness validation difficult and time-consuming.
Abstract
from arXiv · showhide
Today's scientific simulations require a significant reduction of data volume because of extremely large amounts of data they produce and the limited I/O bandwidth and storage space. Error-bounded lossy compressor has been considered one of the most effective solutions to the above problem. In practice, however, the best-fit compression method often needs to be customized/optimized in particular because of diverse characteristics in different datasets and various user requirements on the compression quality and performance. In this paper, we develop a novel modular, composable compression framework (namely SZ3), which involves three significant contributions. (1) SZ3 features a modular abstraction for the prediction-based compression framework such that the new compression modules can be plugged in easily. (2) SZ3 supports multialgorithm predictors and can automatically select the best-fit predictor for each data block based on the designed error estimation criterion. (3) SZ3 allows users to easily compose different compression pipelines on demand, such that both compression quality and performance can be significantly improved for their specific datasets and requirements. (4) In addition, we evaluate several lossy compressors composed from SZ3 using the real-world datasets. Specifically, we leverage SZ3 to improve the compression quality and performance for different use-cases, including GAMESS quantum chemistry dataset and Advanced Photon Source (APS) instrument dataset. Experiments show that our customized compression pipelines lead to up to 20% improvement in compression ratios under the same data distortion compared with the state-of-the-art approaches.
1 INTRODUCTION
SZ3 addresses the need to reduce rapidly growing scientific data while preserving user-specified error bounds and adapting compression to dataset characteristics. It provides modular composition and customized pipelines, with reported gains for GAMESS and APS data.
- Motivation: Scientific simulations and instruments generate data volumes that challenge storage and I/O bandwidth while scientists still require controlled data quality.A 1 km×1 km climate simulation can generate 260 TB of floating-point data every 16 seconds.
- Motivation: Existing prediction-based compressors are not universally optimal because compression efficiency depends on dataset characteristics and required error bounds.SZ-1.4 performs well at low error bounds, whereas regression- or interpolation-based predictors can be more efficient at high bounds.
- SZ3 framework: SZ3 abstracts preprocessing, prediction, quantization, encoding, and lossless compression into independently customizable modules for composing compressors on demand.The framework also supports linearization for unstructured-grid data and features such as relative error bounds, feature preservation, and module bypass.
- Applications: SZ3-based GAMESS compression substitutes a specialized quantizer and adds a lossless compression stage to improve performance over current state-of-the-art methods.The contribution emphasizes achieving this improvement with minimal effort.
- Applications: SZ3-based APS compression uses an adaptive pipeline and achieves the best rate-distortion under any bit rate.The pipeline is designed for APS instrument data and incorporates existing SZ3 modules.
- Evaluation: The paper compares SZ3 with leading prediction-based compressors and evaluates integrated pipelines on diverse scientific datasets.The evaluation characterizes performance and efficiency across multiple domains.
2 RELATED WORK
Related work establishes error-bounded lossy compression as a way to reduce scientific data with quantifiable distortion, while identifying SZ3’s finer-grained modular customization as its distinguishing contribution.
- Motivation: Scientific data growth makes storage, transmission, and analysis difficult because I/O systems remain a bottleneck relative to computing resources.Data compression is presented as a direct way to mitigate this bottleneck.
- Compression approaches: Lossless compressors often achieve low compression ratios on floating-point scientific data because mantissas are nearly randomly distributed.Prior work reports compression ratios of only 2 when state-of-the-art lossless compressors are directly applied to most such datasets.
- Compression approaches: Unbounded-distortion lossy compressors trade data quality for high compression ratios but can exceed users’ distortion expectations and risk misleading downstream analyses.The passage specifically mentions unexpected post hoc analytics and false discoveries.
- Compression approaches: Error-bounded lossy compression reduces scientific data size while guaranteeing quantifiable error bounds through prediction-based or transform-based models.ZFP is identified as a well-known transform-based example.
- SZ and extensions: SZ is a leading prediction-based compressor with a pipeline comprising prediction, quantization, Huffman encoding, and lossless compression.Prior extensions add preprocessing for relative error bounds and element-wise error control.
- SZ3 novelty: Unlike frameworks focused on selecting existing compressors, SZ3 abstracts prediction-based stages individually to enable fine-grained creation of customized compressors.The paper presents this as a generic framework for adapting compressors to actual user needs.
3 SZ3: A MODULAR COMPRESSION FRAMEWORK
SZ3 implements prediction-based compression as a modular pipeline whose stages can be independently selected, extended, and composed. Its modules cover preprocessing, prediction, quantization, encoding, and lossless compression.
- Framework design: SZ3 enables customization of prediction-based compression pipelines with minimal overhead through modular design.A compressor is formed by selecting module instances for the pipeline stages.
- Framework design: The framework defines five modules: preprocessor, predictor, quantizer, encoder, and lossless compressor.Developers can implement and plug in their own module instances.
- Preprocessor: Preprocessors transform input data and compression configuration before compression, supporting relative error bounds and data-layout changes for improved compression.The logarithmic transform converts point-wise relative-error compression into absolute-error compression; layout changes can treat 3D data as 2D or 1D.
- Predictor: Predictors decorrelate data using patterns such as Lorenzo or regression prediction, while composite predictors select the better predictor through blockwise error estimation.The composite design supports multiple prediction algorithms within one predictor instance.
- Quantizer: Quantizers map prediction errors to a smaller countable set while respecting error bounds and determine how decompressed errors are controlled.Linear-scaling, log-scale, and element-wise quantizers provide different error-distribution and granularity choices.
- Encoding and lossless compression: Encoders compress quantized integer indices, and attached lossless compressors further reduce encoded binary output through libraries such as ZSTD, GZIP, and BLOSC.SZ3’s interfaces allow additional lossless routines to be integrated as standalone modules.
4 DEVELOPING AN EFFICIENT COMPRESSOR FOR GAMESS DATA USING SZ3
The section develops SZ3-Pastri for GAMESS quantum-chemistry data by characterizing prediction behavior and customizing quantization and lossless compression. Evaluation under the domain error bound shows improved compression ratios, with a speed trade-off.
- Motivation: GAMESS simulations use ERI data whose recalculation is expensive, motivating in-memory compression and decompression between simulation iterations.SZ-Pastri avoids the ERI recalculation cost and can improve overall simulation performance.
- Data characterization: The pattern-based predictor produces quantization integers centered at zero, indicating high prediction accuracy.The section characterizes quantization integers because they strongly affect final compression ratios.
- Pipeline customization: SZ3-Pastri replaces direct truncation with an Unpred-aware Quantizer that stores unpredictable-data integers by bitplane.The pipeline also adds a lossless compression stage to exploit correlations in the bitplane representation.
- Evaluation: SZ3-Pastri is evaluated against SZ-Pastri and its lossless variation on three representative GAMESS fields.Experiments were conducted on the Bebop supercomputer at Argonne National Laboratory.
- Evaluation results: 40% and 20% compression-ratio improvements are reported for the ff|ff dataset over SZ-Pastri and its lossless variation, respectively.The rate-distortion comparison covers nearly all bit rates in Figure 4.
- Evaluation results: At absolute error bound 1E-10, SZ3-Pastri improves compression ratios but has lower performance than the comparison approaches.The reported degradation is attributed to embedded encoding of unpredictable data and final lossless compression.
5 COMPOSING AN EFFICIENT COMPRESSOR FOR APS DATA USING SZ3
The APS compressor is customized around the data’s temporal stack structure and variable spatial correlation. SZ3-APS adaptively selects pipelines by error bound and improves rate-distortion performance, including 18% and 12% compression-ratio gains on two datasets.
- APS data: APS ptychography produces large 3D diffraction datasets, motivating compression with high ratios.The datasets contain 19,500×514×1,030 and 16,800×514×1,030 arrays, respectively.
- Data characterization and pipeline customization: The APS data is a temporal stack of 2D images with relatively low spatial correlation, affecting predictor choice.Multidimensional Lorenzo prediction can introduce higher noise when more decompressed values are used.
- Evaluation results: SZ3-APS adaptively selects compression pipelines by error bound and achieves comparable high-error-bound performance to SZ-2.1’s 3D compressor.The evaluation compares SZ3-APS with generic SZ-2.1 compressors for 1D, 3D, and transposed 1D data.
6 SUSTAINABILITY, QUALITY, AND PERFORMANCE INVESTIGATION OF SZ3
SZ3 uses modular software abstractions and composable pipelines to improve sustainability while supporting different compression quality and performance requirements. Evaluations identify distinct use cases for its built-in pipelines across scientific datasets.
- Sustainability: SZ2’s 120-plus-function codebase lacks reuse, making extensions time-consuming and correctness validation difficult.Separate implementations for data types and dimensions increase maintenance and testing effort.
- Sustainability: SZ3 improves sustainability through compile-time polymorphism, datatype abstraction, and multidimensional iteration.These technologies support code reuse, generic data types and dimensions, and simpler multidimensional data access.
- Pipeline integration: SZ3 composes pipelines from preprocessing, prediction, quantization, encoding, and lossless-compression modules.Examples include SZ3-LR, SZ3-Truncation, and SZ3-Interp, each targeting different quality or speed requirements.
- Quality evaluation: 56% higher compression ratio is achieved by SZ3-Interp than SZ3-LR on Miranda at the same PSNR of 90, with ratios of 47 and 30, respectively.SZ3-Interp performs better on most datasets at lower bit rates, while SZ3-LR remains preferable for high accuracy on Scale and Hurricane.
- Performance evaluation: 1 GB/s compression throughput makes SZ3-Truncation 4X faster than the second-best compressor, while SZ3-Interp exceeds 100 MB/s in all cases.SZ3-LR-s has performance comparable with SZ2.1 across datasets.
- Pipeline selection: SZ3-Truncation suits strict time limits, SZ3-Interp suits high compression ratios with relaxed time constraints, and SZ3-LR balances quality and speed.The recommended pipeline depends on whether speed, compression ratio, or a balance of both is the priority.
7 CONCLUSION AND FUTURE WORK
The paper concludes that SZ3 enables adaptive, extensible, on-demand construction of error-bounded lossy compressors. Customized compressors for two application datasets improve compression ratios by 20% at the same data distortion as state-of-the-art approaches.
- Conclusion: SZ3 enables users to customize on-demand error-bounded lossy compressors in an adaptive and extensible framework.The framework is intended to reduce compressor development effort while accommodating dataset characteristics and user requirements.
- Conclusion: 20% higher compression ratios are achieved for two real-world application datasets at the same data distortion as state-of-the-art compressors.Future work includes adding more framework instances and support for GPUs and FPGAs.
APPENDIX A
The appendix introduces representative SZ3 interfaces and functions, including template parameters for data type, dimensionality, and quantized data type.
- Appendix overview: SZ3’s appendix demonstrates representative interfaces and functions using T, N, and X as templates for data type, dimensionality, and quantized data type.These templates express the generic parameters used by the framework interfaces.
A.1 Snippet of Preprocess Interface
The preprocess interface defines preprocessing and postprocessing operations over data and a configuration object parameterized by data type and dimensionality.
- Preprocess interface: The preprocess interface provides preprocess and postprocess operations on data using SZ3 configuration parameters.The interface is templated by the data type T and dimensionality N.
A.2 Snippet of Predictor Interface
The predictor interface defines the operations required for prediction-based compression, including prediction, error estimation, and state serialization.
- The predictor interface exposes methods to predict values, estimate prediction errors, and save or load predictor state.
A.3 Snippet of Quantizer Interface
The quantizer interface converts data and predictions into quantized values and reconstructs data from predictions and quantized values.
- The quantizer interface quantizes data using a prediction and recovers data from the prediction and quantized value.
A.4 Snippet of Encoder Interface
The compression pipeline processes multidimensional data in blocks, predicts and quantizes each element, then encodes and losslessly compresses the resulting data through interchangeable components.
- The encoder interface encodes binned values into bytes and decodes bytes back into values.
- The lossless interface compresses and decompresses byte data while tracking input and output sizes.
- The SZ compressor composes preprocessing, prediction, quantization, encoding, and lossless-compression components as configurable template parameters.
- The prediction-and-quantization routine returns the collected quantization results.
- The pipeline iterates over multidimensional data blocks before processing their elements.
- Each element is predicted, quantized against its prediction, and appended to the quantization results.