PaperlayerView exact revision on arXiv

Source-linked AI summary

Improving Progressive Compression with Adaptive Interpolation and Coefficient Decomposition

Wenbo Li, Xuan Wu, Qian Gong, Pu Jiao, Jieyang Chen, Qing Liu, Norbert Podhorszki, Scott Klasky, Xin Liang

arXiv:2609.04573v1cs.DC

TL;DR

Exascale data generation and unknown downstream tolerances motivate progressive compression methods that can retrieve data at different quality targets without fixing one tolerance in advance. This paper introduces adaptive interpolation, coefficient decomposition, and automatic workflow selection, reporting up to 42.3% and 92.5% compression-ratio improvements under matched error tolerance and PSNR, respectively.

  • Problem

    Existing lossy compressors require one tolerance at compression time, while datasets may serve analyses with varying precision requirements.

  • Method

    The framework adaptively selects interpolation and encoding strategies, exploits spatial correlations among decomposed coefficients, and tunes the progressive workflow online.

  • Results

    42.3% and 92.5% compression-ratio improvements are achieved under the same error tolerance and PSNR, respectively, with up to 1.26× faster 512 GB transfers.

  • Takeaways & Limitations

    The method retrieves less data while achieving the best visualization quality and improves progressive retrieval efficiency across diverse targets.

Abstract

from arXiv · show

Exascale simulations generate data far faster than it can be stored or analyzed, making efficient data reduction essential. Error-controlled lossy compression offers high compression ratios under user-specified error bounds, but the target tolerance must be fixed at compression time. Progressive compression relaxes this restriction, yet existing methods still rely on fixed refactoring strategies and do not fully exploit correlations among decomposed coefficients, limiting the efficiency of progressive retrieval. In this work, we present an adaptive progressive compression framework that improves retrieval efficiency for two common targets, namely error-bound and peak Signal-to-Noise ratios. Our contributions are fourfold. (1) We propose to leverage two complementary interpolation schemes for adaptive progressive compression toward different targets, and we optimize them to achieve high efficiency. (2) We propose coefficient decomposition, a novel method that exploits the commonly overlooked spatial correlations among decorrelated data, which further improves the efficiency. (3) We develop the adaptive progressive compression workflow with automatic selection of the best-fit refactoring pipeline and tailored optimizations. (4) We evaluate the proposed framework on five real-world scientific datasets against three state-of-the-art progressive compressors. Experimental results demonstrate that the proposed framework improves the compression ratio by up to $42.3\%$ under the same requested error tolerance and up to $92.5\%$ at the same PSNR, compared with the best-performing existing methods. When transferring $512$ GB of scientific data to remote sites, the framework delivers up to $1.26\times$ speedup in the end-to-end data transfer performance. Furthermore, our method achieves the highest visualization quality while retrieving the least amount of data from storage.

I. INTRODUCTION

Exascale data generation is outpacing storage and analysis capacity, while fixed-error lossy compression can waste compression potential when future tolerances are unknown. The paper addresses this with adaptive interpolation, coefficient decomposition, and an automatically selected progressive workflow.

  • Motivation: Exascale simulations can generate data faster than modern storage systems can store, transfer, or analyze.A turbulence simulation could produce about 0.5 PB per snapshot and up to ∼38 TB/s if every time step were retained, versus 1–2 TB/s file-system bandwidth.
  • Motivation: Fixed-error lossy compressors require a tolerance before storage, forcing conservative choices when downstream precision needs are unknown.Information lost during compression cannot be recovered, so conservative bounds reduce compression ratios.
  • Motivation: Progressive compression enables incremental reconstruction, but existing scientific methods retain efficiency limitations from fixed decomposition and retrieval strategies.PMGARD is limited by multilinear decomposition and greedy retrieval, while the cited discussion identifies room for improvement.
  • Contributions: The framework optimizes two complementary interpolation schemes to support adaptive progressive compression toward diverse targets.The interpolation schemes are designed to provide flexibility for achieving high efficiency.
  • Contributions: Coefficient decomposition exploits spatial correlations within decorrelated coefficients and improves retrieval efficiency with negligible metadata overhead.The method targets correlations that are commonly overlooked after interpolation-based decorrelation.
  • Contributions: A fully adaptive workflow selects a best-fit refactoring pipeline online and applies tailored optimizations to mitigate overhead.The framework uses sampling-based selection during execution.
  • Evaluation: 42.3% and 92.5% improvements in compression ratio are reported under the same error tolerance and PSNR, respectively, with 1.26× faster 512 GB transfers.Experiments cover five real-world datasets and also report the highest visualization quality with the least retrieved data size.

A. Error-controlled Lossy Compressors

Error-controlled compressors improve reduction of scientific data but tailor compressed representations to a single tolerance, whereas progressive compression organizes information for retrieval at varying accuracies. Existing progressive methods still have trade-offs that motivate adaptive refactoring and coefficient decomposition.

  • Error-controlled compression: Error-controlled compressors exploit scientific-data structure to achieve higher compression than lossless methods while bounding reconstruction error.Prediction-based and transform-based families decorrelate data before entropy coding through different representations.
  • Error-controlled compression: Single-tolerance compression is inefficient when one dataset serves analyses with different precision requirements.Users may select a conservatively tight bound for the most demanding analysis, lowering compression ratios and storage or transfer benefits.
  • Progressive compression: Progressive compression divides representations into ordered components so data can be reconstructed incrementally to a desired accuracy.This write-once, retrieve-progressively model suits datasets consumed by analyses with different fidelity requirements.
  • Progressive compression: Progressive image coding introduced the organization of encoded information into stages whose quality improves as more bits arrive.JPEG and JPEG2000 provide the historical basis for progressive transmission and embedded coding concepts.
  • Existing methods: PMGARD provides progressive scientific retrieval with strict L2 and L∞ error guarantees but uses linear interpolation, limiting correlation capture relative to higher-order schemes.Its multilevel coefficients are organized into bitplane streams decoded in stages.
  • Existing methods: IPComp combines interpolation-based decomposition with dynamic-programming retrieval, but preset pre-decorrelation quantization reduces achievable precision.The supplied passage identifies this as one of its remaining limitations.
  • Motivation for this work: These limitations motivate adaptive interpolation and encoding, coefficient decomposition, and a progressive pipeline with tailored optimizations.The proposed design targets spatial correlations within decomposed data and systematic adaptive refactoring.

III. OVERVIEW

The paper formulates progressive retrieval as refactoring data into ordered segments and selecting enough segments to meet either an error-bound or PSNR target. These targets address different notions of reconstruction quality and are optimized separately.

  • Progressive retrieval formulation: An error-controlled progressive method refactors data X into k progressive segments and selects n_τ fragments during retrieval.The selected fragments reconstruct X′ while satisfying the requested error bound τ.
  • Error-bound mode: The error-bound target requires reconstructed values to satisfy a maximum-error constraint of τ.This target supports applications requiring guaranteed absolute error control.
  • Progressive retrieval formulation: The retrieved size under τ is defined as the sum of the sizes of the n_τ retrieved segments.This quantity measures the data volume needed to meet the requested error bound.
  • PSNR mode: PSNR measures average reconstruction error across the domain and is preferred for applications concerned with global error distribution.The paper gives fusion energy science and climate studies as examples of such applications.
  • Separate targets: Error-bound and PSNR targets reflect reconstruction quality differently, so the paper formulates them as separate retrieval objectives.The designs are referred to as error-bound mode and PSNR mode, respectively.

B. Overview

The framework combines adaptive interpolation and adjusted retrieval components to support error-bound and PSNR targets. It uses two complementary multilevel interpolation schemes and selects between them according to the target.

  • Framework overview: The framework integrates adaptive interpolation, encoding, and retrieval adjustments into data refactoring and progressive retrieval pipelines.Blue boxes denote proposed components, cyan boxes optimized components, and gray boxes adjusted components in the overview.
  • Interpolation schemes: Both schemes use bottom-up multilevel decomposition, beginning at level 0 with all data points and reducing resolution by half at each level.Each successive level doubles the stride along each dimension.
  • Interpolation schemes: Per-level interpolation uses higher-level data points to interpolate remaining points within the current level, limiting error propagation.Its linear 3D example averages 2, 4, or 8 upper-level points for edge, face, or cube centers.
  • Interpolation schemes: Per-region interpolation applies the same interpolation formula to data points along each dimension while using different data, grouping points interpolated along one dimension into regions.The scheme is associated with the per-region decomposition pipeline illustrated in Figure 2.
  • Adaptive selection: Per-level interpolation is preferred for error-bound mode, whereas per-region interpolation is favorable for PSNR mode in most cases.The framework therefore uses a tuner to select the interpolation scheme online according to the target.

V. COEFFICIENT DECOMPOSITION

Coefficient decomposition exploits spatial correlations remaining in interpolated residual coefficients to improve progressive compression, while requiring retrieval adjustments to preserve error control. The method applies primarily at the finest level and adapts storage mappings for decomposed regions.

  • Motivation: Coefficients are residuals between original values and their interpolated counterparts, and their distribution determines progressive-compression efficiency.Near-zero coefficients create more zeros in significant bitplanes and thereby increase compressibility.
  • Decomposition method: Coefficient slices from the S3D CH4 field show high spatial correlation, and 2D linear CoeffDecom transforms the original coefficients into decomposed coefficients.Figure 4 compares the original and decomposed slices at the finest level in Region 3.
  • Decomposition method: The proposed coefficient decomposition applies 2D per-level multilinear interpolation to slices of 3D coefficients within each region.The 2D operation targets correlations mainly located in the hyperplane orthogonal to the interpolation direction.
  • Decomposition scope: Coefficient decomposition is used at the finest level because higher-level coefficients occupy less data and have weaker spatial correlation.For 3D data, higher levels can account for less than 1/8 of the data.
  • Retrieval adjustments: Decomposition reduces coefficient entropy but introduces interleaved regions and added levels that require retrieval adjustments for error-controlled reconstruction.The retrieval design must account for these changes when coupled with either interpolation scheme.
  • Retrieval adjustments: For per-level interpolation, the method combines within-region greedy ordering with max ordering to merge bitplanes across regions into unified error-bound steps.The merged ordering uses cumulative errors and reports the maximum error after each step.
  • Retrieval adjustments: For per-region interpolation, each decomposed region is split into three regions corresponding to decomposed coefficient levels before existing retrieval algorithms are applied.This preserves the region-level granularity used by prior retrieval methods.

VI. IMPLEMENTATION AND OPTIMIZATION

The implementation uses target-driven online tuning to select the refactoring configuration and applies tailored optimizations. A 1% sample is evaluated across multiple tolerances, with sampled decisions compared against full-data decisions.

  • Implementation: The implementation begins with target-driven tuning for automatic component selection, followed by the progressive-compression algorithm and tailored performance optimizations.The tuning determines the configuration of the data-refactoring pipeline online.
  • Target-driven tuning: Sampled tuning reproduces full-data decisions for 85.0% of interpolation-scheme selections and 88.1% of coefficient-decomposition selections.The comparison spans all fields and datasets in a single-machine experiment.
  • Target-driven tuning: The framework adopts 1% sampling for online tuning after comparing sampled and full-data configuration votes.Table I summarizes the evaluated and tuned configurations for 3D data.

B. Algorithm

The algorithm refactors data through multilevel interpolation and coefficient decomposition, then retrieves selected bitplanes from coarse to fine to reconstruct data under a requested tolerance.

  • Refactoring: The refactoring algorithm interpolates at the finest level, interleaves residual data into regions, and applies coefficient decomposition before bitplane encoding.Later levels are processed with interpolation and bitplane encoding, while metadata maps segments to error bounds.
  • Refactoring: Coefficient decomposition processes coefficients across three levels, encodes the resulting streams, and merges bitplanes using a max ordering strategy.The merged streams include metadata used to interpret retrieval requirements.
  • Retrieval: Retrieval begins at the coarsest level, determines required bitplanes from the requested error bound and metadata, and progressively reconstructs finer levels.Decoded coefficients are repositioned and added to interpolated reconstructions at each level.
  • Depth selection: Offline studies fix the decomposition depth at three levels because no single depth is cheapest across all error bounds and targets.Three-level decomposition is cheapest at 11 of 16 error bounds in error-bound mode and 13 in PSNR mode, while losing by at most 0.83%.

C. Performance optimization

The framework optimizes interpolation and generic bitplane encoding to improve throughput while retaining its progressive compression workflow and evaluating it across scientific datasets.

  • Optimization goals: Performance optimization targets refactoring and reconstruction throughput because these operations can lie on the critical path of data transfer.The implementation specifically addresses the cost of interpolation and encoding.
  • Interpolation: Fastest-direction interpolation improves cache efficiency by performing interpolation along the fastest-varying data direction.The optimization exploits an equivalence between per-level interpolation and sequential interpolation of regions.
  • Bitplane encoding: Generic bitplane encoding is split into stages so sign-bit information can eliminate an unnecessary branch while preserving the required encoding behavior.The optimization addresses the relative slowness of generic encoding in error-bound mode.
  • Evaluation: 2× performance improvement is achieved for both interpolation and generic bitplane encoding over existing implementations on all 9 valid S3D fields.The validation uses the S3D dataset and compares each operation before and after optimization.
  • Evaluation: The evaluation compares ProAICD with PMGARD, SZ3-R, and IPComp across retrieval efficiency, reconstruction quality, throughput, and two scientific use cases.Experiments use five real-world datasets spanning climate, hydrodynamics, weather, combustion, and turbulence simulations.

2) Platform:

The study evaluates progressive retrieval with rate-distortion metrics on a cluster environment, using separate error-bound and PSNR targets and an ablation study on S3D.

  • Platform: Runtime experiments run three times on the Morgan Compute Cluster, which uses 100 Gbps InfiniBand HDR and nodes with dual 64-core AMD EPYC processors.Each node has 256 GB of memory per processor.
  • Quality assessment: Rate-distortion graphs relate retrieved-data bitrate to distortion, with metadata included in retrieved size for every compressor.Bitrate is computed as Sτ × 8/n, where Sτ is retrieved size and n is the total number of data points.
  • Quality assessment: Error-bound and PSNR modes use separate rate-distortion plots, where lower-left curves are preferred for error bounds and upper-left curves for PSNR.The two plots reflect the framework’s distinct optimization targets.
  • Aggregation: Aggregated dataset bitrate is averaged across fields, while aggregated PSNR uses normalized root mean squared error across fields.The normalization uses each field’s RMSE and value range.
  • Ablation study: In S3D error-bound ablations, adaptive interpolation improves over PMGARD across all nine fields, while coefficient decomposition further widens the gap or preserves efficiency.The largest gains occur for CH4, CO, CO2, H2O, O2, and Temperature.
  • Ablation study: In PSNR ablations, adaptive interpolation improves efficiency over IPComp and coefficient decomposition further improves data quality across all nine fields.Adaptive interpolation explores the best-fit interpolation order rather than using IPComp’s fixed order.

C. Comparison with State of the Arts

Against three progressive compressors, ProAICD improves compression ratios in both error-bound and PSNR modes across evaluated scientific datasets, with gains reported against the best existing method.

  • Error-bound mode: Up to 42.3% compression-ratio improvement is achieved in error-bound mode relative to the best-performing existing method.The reported dataset-specific improvements are 23.9%, 29.7%, 29.3%, and 42.3%.
  • Error-bound mode: ProAICD outperforms PMGARD and IPComp across all error bounds and datasets, while SZ3-R is lower at some loose error bounds on CESM and SCALE.The passage attributes those points to SZ3-R’s exact target error bounds at those operating points.
  • PSNR mode: Up to approximately 92.5% compression-ratio improvement is achieved in PSNR mode compared with the best-performing existing methods.The reported improvements are approximately 92.5%, 51.1%, 75.6%, and 91.3%.

2) Performance:

The framework achieves strong bitrate and transfer performance, though its error-bound mode incurs some time overhead and reconstruction costs rise as tolerances tighten. Its transfer advantage comes from combining small retrieved sizes with fast reconstruction.

  • Reconstruction: Reconstruction time increases as tolerance tightens because more data must be retrieved and decompressed.For SZ3-R, the increase is sharpest because fine requests decode every preceding residual snapshot.
  • Bit-rate: At τ = 10^-6, the proposed approach achieves the lowest bitrate on all four datasets, while SZ3-R becomes the most expensive.On CESM, SZ3-R reaches BR = 9.12 versus BR = 6.42 for the proposed approach.
  • Bit-rate: The error-bound mode achieves the lowest bitrate on most datasets, while the PSNR mode maintains competitive speed and lower bitrate than all three baselines.The error-bound mode trades slightly longer refactoring and reconstruction time for bitrate advantages.
  • End-to-end transfer: In the JHTDB transfer experiment, the proposed method achieves the shortest end-to-end transfer time with 256, 512, and 1024 cores.It combines the second-smallest retrieved size with the second-shortest retrieval time.

E. Use case for PSNR mode: data visualization

For CESM temperature visualization, the method produces the most faithful reconstruction at a low bitrate and retrieves less data while maintaining better visual quality. This supports interactive exploration of large-scale scientific datasets.

  • Visualization quality: At a bitrate of 0.37, the proposed method produces the most faithful reconstruction in the CESM temperature visualization.The comparison uses a displayed slice and reports both global and ROI PSNR for each method.
  • Visualization quality: The method retrieves substantially less data from disk while yielding better visual quality.The authors identify this property as valuable for interactive exploration of large-scale scientific datasets.
  • Method: The framework is designed to improve retrieval efficiency and reconstruction quality through adaptive interpolation, encoding, and coefficient decomposition.Coefficient decomposition exploits correlations among decorrelated coefficients.
Loading 2609.04573v1…