PaperlayerView exact revision on arXiv

Source-linked AI summary

Tools-CC-Bench: a Benchmark Suite for Collective Communication with Compression in HPC and AI Workloads

Haozhe Fan, Wei Wang, Xingchen Liu, Man Liu, Xingjian Tian, Haoquan Long, Zedong Liu, Daran Sun, Jinwu Yang, Bo Yang, Jie Liu, Yonggang Che, Hairui Zhao, Guangming Tan, Dingwen Tao

arXiv:2609.08739v2cs.DC

TL;DR

Existing benchmarks provide limited support for evaluating communication compression across backends, realistic datasets, application-specific accuracy, and computation overlap. CC-Bench combines configurable workload-oriented evaluation with software and hardware profiling; its results show that compression trade-offs depend on workload characteristics.

  • Problem

    Existing benchmarks have limited support for diverse communication libraries, datasets, application-specific accuracy metrics, and profiling compression under computation overlap.

    Source: §1 · pp. 1–2

  • Method

    CC-Bench combines application-oriented configuration with function-level profiling and hardware monitoring to evaluate compression under realistic workload conditions.

    Source: §1 · p. 2, §3.1 · p. 4, §3.2 · p. 4

  • Results

    Compression trade-offs vary with workload characteristics: low-entropy HPC data tend to be compression-ratio-driven, while high-entropy training workloads are often accuracy-driven.

    Source: §5.4 · p. 10

  • Takeaways & Limitations

    Compression should be evaluated for full backend–compressor–data combinations rather than treated as a universal replacement.

    Source: §5.4 · p. 10

    Existing benchmarks provide coarse-grained latency, limited hardware and phase profiling, and insufficient diverse real-world datasets.

    Source: §2.4 · p. 4

Abstract

from arXiv · show

Distributed HPC and LLM workloads increasingly require efficient communication for scalability, yet growing data movement has become a major performance bottleneck. Communication compression can reduce this overhead and complement execution-level optimizations, but its benefits remain difficult to assess because existing benchmarks lack support for diverse backends, realistic datasets, application-specific accuracy metrics, and overlap-induced resource contention. We present CC-Bench, a lightweight, extensible, and application-oriented benchmark suite for evaluating communication compression under realistic execution conditions. CC-Bench uses declarative application-environment modeling to decouple profiling logic from communication libraries, datasets, and fidelity metrics, enabling portable cross-library evaluation. It further combines function-level interception and hardware counter monitoring to characterize per-phase latency, hardware utilization, numerical fidelity, and computation interference. With representative datasets from HPC and LLM workloads, CC-Bench evaluates three compression-enabled communication libraries on CPU and GPU clusters, revealing accuracy-performance trade-offs and bottlenecks to guide practical deployment and optimization.

1. Introduction

Communication is central to scaling HPC and LLM workloads, but growing communication demand makes it a major performance bottleneck. CC-Bench addresses gaps in evaluating compression with a portable, application-oriented benchmark and profiling suite.

1. Introduction

  • Communication overhead has become a major bottleneck for workload scalability and end-to-end performance as communication demand grows with workload scale.Distributed systems support workloads ranging from large molecular dynamics simulations to large AI models.Source: §1 · p. 1
  • Existing benchmarks provide limited support for diverse libraries, realistic datasets, application-specific accuracy metrics, and profiling overlap-induced contention.Compression ratio and fidelity depend on data, while isolated latency and throughput measurements miss interference and end-to-end bottlenecks.Source: §1 · pp. 1–2
  • CC-Bench provides a lightweight, customizable benchmark for evaluating communication compression across diverse backends and realistic workloads.Its application-oriented configuration and runtime profiling support workload emulation and cross-domain evaluation.Source: §1 · p. 2
  • CC-Bench decouples profiling logic from communication libraries, datasets, and error metrics through unified abstractions and declarative configuration.It also supports user-defined components to simplify cross-application evaluation.Source: §1 · p. 2
  • CC-Bench combines function-level interception and hardware-counter monitoring to characterize phase latency, hardware utilization, contention, and numerical fidelity.The suite includes representative scientific-computing and LLM datasets and evaluates three compression-supported libraries on CPU and GPU clusters.Source: §1 · p. 2

2. Background

HPC and AI workloads use distinct communication patterns and libraries, and compression effectiveness depends on data characteristics. Existing benchmarks leave gaps in extensibility, performance analysis, and realistic datasets.

2.1. Communication for HPC and AI Workloads

  • HPC and AI workloads differ in communication scale and frequency, from coarse-grained scientific transfers to repeated AI exchanges.Scientific workloads use CPU collectives and increasingly heterogeneous CPU–GPU paths; AI strategies use distinct patterns, including frequent smaller pipeline transfers and less frequent large gradient exchanges.Source: §2.1 · pp. 2–3

2.2. Collective Communication Libraries

  • MPI and CUDA-aware MPI serve scientific computing, while GPU-oriented AI workloads commonly use NCCL or RCCL collectives.MPI supports collectives across interconnects and CUDA-aware systems; NCCL and RCCL target NVIDIA and AMD GPU ecosystems, respectively.Source: §2.2 · p. 3

2.3. Communication Compression

  • Lossless compression preserves numerical fidelity but typically offers limited ratios and throughput, while lossy methods trade precision for higher ratios and throughput.Compression performance and fidelity depend on data distribution; libraries integrate compressors into collective communication for scientific and AI workloads.Source: §2.3 · p. 3

2.4. Limitations of Existing Benchmarks

  • Existing benchmarks offer limited flexibility in switching or integrating libraries, datasets, and error metrics, and many lack fine-grained profiling and hardware-utilization data.They also largely overlook diverse real-world datasets, leaving users to curate application-specific data for realistic evaluation.Source: §2.4 · pp. 3–4

3. Methodology of CC-Bench

CC-Bench is designed to evaluate compression across workload-specific stacks and conditions, then diagnose performance and fidelity. Its goals are implemented through configurable components and software- and hardware-level profiling.

3.1. Benchmark Goals

  • CC-Bench evaluates compressor–backend trade-offs, decomposes collectives into phases, monitors hardware utilization, and conditions tests on real data and computation interference.Together, these dimensions span stack comparison, bottleneck localization, contention diagnosis, and realistic workload conditions.Source: §3.1 · p. 4

3.2. Design Principles

  • A unified abstraction layer makes communication, compression, and profiling independently swappable, while configurable datasets and LD_PRELOAD injection support broad sweeps without recompilation.Users can integrate compressors with communication backends through uniform interfaces.Source: §3.2 · p. 4
  • Wrapper-based profiling separates software-phase costs, while daemons collect CPU, GPU, memory, and NIC utilization to diagnose bottlenecks and contention.The architecture describes swappable preload layers and daemons as the runtime for test binaries.Source: §3.2 · p. 4, Figure 2 · p. 5
  • A pluggable metric framework and representative scientific and LLM datasets support domain-specific accuracy assessment, including PSNR, SSIM, and error quantiles.The design is intended to characterize how data modality and accuracy requirements reshape compression-quality trade-offs.Source: §3.2 · p. 4

4. Design of CC-Bench

CC-Bench combines application-oriented configuration with full-stack profiling to evaluate compressed communication under configurable, realistic execution conditions. Its modular workflow supports diverse backends, datasets, metrics, and workload interference.

4.1. Overview

  • CC-Bench pairs configurable application emulation with full-stack profiling, and its five-stage workflow runs from JSONC configuration through analysis.The architecture extends to additional nodes by editing the host list.Source: §4.1 · p. 5, Figure 3 · p. 6

4.2. Application-Oriented Configuration Module

  • User wrappers preserve collective semantics across nonstandard backends, permit compressor substitution, and dynamically load custom deviation metrics and duty-cycle stressors.Custom metric functions are compiled into shared libraries and loaded at runtime without modifying core code; stress daemons approximate intermittent contention without full application stacks.Source: §4.2 · p. 5

4.3. Full-stack Profiling Module

  • Profiling traces per-call timestamps, message sizes, and compression ratios; counters and analysis expose phase latency, overlap efficiency, idle time, and hardware utilization.Pingpong tests support interpolation of non-blocking communication overhead, while concurrent compute and communication tests measure interference against serial execution.Source: §4.3 · pp. 5–6

4.4. Benchmark Coverage

  • Coverage spans diverse collectives, CPU/GPU compression paths and backends, scientific and LLM data, and performance, fidelity, and profiling measures.Supported workloads include scientific datasets, LLM KV-cache traces, Llama3 8B values, weights and gradients, and synthetic distributions.Source: §4.4 · p. 6

4.5. Benchmark Usage

  • Users select environment, library, compressor, dataset, message sizes, background load, and metrics through JSONC, then compile extensions and run scripts for latency, throughput, traces, and hardware logs.Post-processing reports phase breakdowns and overlap efficiency; hardware-utilization heatmaps offer a complementary bottleneck view when function-level interception is unavailable.Source: §4.5 · pp. 6–7

5. Evaluation

CC-Bench evaluates compression-enabled collective communication across CPU scientific and GPU LLM workloads, measuring fidelity, latency, phase costs, and contention. Results show workload- and message-size-dependent trade-offs, with codec work and shared-GPU load shaping observed performance.

5.1. Basic Benchmark Setup

  • Using AllReduce, CC-Bench compares uncompressed MPI/NCCL baselines with compression-enabled stacks on CPU and GPU clusters, using scientific fields and LLM tensors and traces.Source: §5 · p. 7, §5.1 · p. 7

5.2. Quality Evaluation

  • Fidelity assessment is workload-specific: CPU scientific tests use PSNR, while GPU Llama tensors and KV-cache traces use cosine similarity and relative error.Source: §5.2 · p. 7
  • CPU fidelity varies by dataset; on GPU, SDP4Bit has lower MAE than TAH-Quant, while lossless UCCL-DietGPU preserves zero MAE but is usually slower.Source: Table 2 · p. 7, §5.2 · pp. 7–8, §5.3.1 · p. 9

5.3.1. Cross-combination Performance.

5.3.2. Performance

  • Figure 9’s CPU breakdown finds 92.0% of communication concurrent with codecs; decompression fires more often (39,347 vs. 19,712 calls) despite comparable phase totals.Source: Figure 9 · p. 9, §5.3.2 · p. 9

5.3.3. Performance Slowdown.

  • Under injected GPU load, COCCL+SDP4Bit KV-cache slowdown generally rises with load but varies by message size, so isolated latency can overstate practical speedup.Source: Figure 10 · p. 10, §5.3.3 · p. 10

5.4. Insight Discussion

  • Compression is a workload-conditioned optimization: low-entropy HPC data tend to be compression-ratio-driven, while high-entropy training workloads are often accuracy-driven.Source: §5.4 · p. 10
  • CC-Bench profiles from module to function granularity and injects configurable GPU background load to examine bottlenecks and shared-resource interference.Source: §5.4 · p. 10

6. Conclusion and Future Work

CC-Bench combines flexible configuration with full-stack profiling to evaluate compression-enhanced collective communication. The evaluation finds that latency, throughput, and accuracy must be considered jointly, as overlap and interference shift bottlenecks and isolated microbenchmark conclusions may not hold under realistic conditions.

6. Conclusion and Future Work

  • Latency, throughput, and accuracy must be considered jointly, because bottlenecks shift with overlap and interference and isolated microbenchmark conclusions may not hold in realistic conditions.Source: §6 · p. 10
  • CC-Bench pairs a flexible configuration workflow for environments, libraries, compressors, metrics, and datasets with profiling that captures software phases and hardware utilization.Source: §6 · p. 10
  • Future work targets error propagation across iterative compression-decompression, plugins for compressors and hardware backends, and lightweight proxies for replaying collective traces.The planned plugin framework also includes I/O and power monitoring daemons.Source: §6 · p. 10

A.1. Abstract

The artifact provides CC-Bench source code and instructions for regenerating the experiments described in section 5.

A.1. Abstract

  • The artifact contains the full source code of CC-Bench.Source: §A.1 · p. 13
  • The source code is available at CCBench_IISWC_AE.Source: §A.1 · p. 13
  • The appendix gives instructions to regenerate the experiments described in section 5.Source: §A.1 · p. 13

A.2. Artifact check-list (meta-information) •

The checklist describes CC-Bench's software, data, hardware, metrics, and experiment workflow for lossy compression in collective communication.

A.2. Artifact check-list (meta-information) •

  • The checklist names Hurricane ISABEL and Llama 3 8B gradients as datasets and lists Open MPI, NCCL, CUDA, and Linux in the runtime environment.Source: §A.2 · p. 13
  • The listed hardware includes two-node CPU and GPU setups, with 10G InfiniBand for CPU and 200GbE plus InfiniBand for GPU.Source: §A.2 · p. 13
  • Metrics cover compression rate, error and similarity measures, latency, bandwidth, and slowdown, while experiments address quality trade-offs, interference, and overhead.Source: §A.2 · p. 13
  • The checklist estimates about 25GB of disk space and approximately 15 minutes each to prepare the workflow and complete experiments.Source: §A.2 · p. 13

A.3. Description

The artifact maps CC-Bench's three contributions to supporting experiments: configuration, full-stack profiling, and representative datasets.

A.3. Description

  • The application-oriented configuration module decouples profiling logic from communication libraries, datasets, and error metrics.The table maps this contribution to experiments E1 and E2.Source: §A.3 · p. 13
  • The full-stack profiling module combines function-level interception with hardware counter monitoring.The table maps this contribution to experiments E3 and E4.Source: §A.3 · p. 13
  • Built-in representative datasets span scientific computing and LLM training and inference.The table maps this contribution to experiments E1 and E2.Source: §A.3 · p. 13

A.3.1. How to access.

The artifact has a repository location and a frozen Zenodo archive; the section also directs readers to the hardware details in A.2.

A.3.1. How to access.

  • The artifact is available in the CCBench_IISWC_AE repository, with a frozen version archived on Zenodo.The passage gives the repository URL and DOI.Source: §A.3.1 · p. 13
  • Hardware details are in section A.2.Source: §A.3.2 · p. 13

A.3.2. Hardware dependencies.

The appendix lists software and dataset dependencies, installation and experiment workflows, expected outputs, and configuration options for customization.

A.3.3. Software dependencies. •

  • Dependencies include Ubuntu 22.04+ or CentOS 7+, Open MPI 4.1.4, CUDA 12.8 with NCCL 2.27, and listed compression and communication libraries.The appendix lists SZx and SDP4bit, plus CoCCL and ZCCL; it recommends locally revised communication libraries because of bugs in the originals.Source: §A.3.3 · p. 14

A.3.4. Data sets. •

  • The listed datasets are Hurricane and gradients captured from Llama 3 8B runs.Source: §A.3.4 · p. 14

A.4. Installation

  • Installation involves cloning the repository, datasets, and libraries, setting up the toolchain, choosing MPI or NCCL, and building the benchmark.The appendix gives module-based setup with manual environment-variable export as an alternative when modules are unavailable.Source: §A.4 · p. 14

A.5. Experiment workflow

  • The workflow builds and runs E1 and E2 on GPU, runs E3 separately, and provides CPU instructions for E1, E2, and E4.GPU E4 instructions include analyzing function traces; CPU instructions include preparing the pingpong baseline.Source: §A.5 · p. 14, §1 · p. 14

A.6. Evaluation and expected results

  • In the reported example, the output is 96.99 GB/s and marked Correct: NO, with cos_sim 9.813335e-01 and mae 3.090251e-04.Source: §A.6 · p. 14
  • At 0% GPU stress, expected output reports 3739.50 us, 143.59 GB/s, and 1.00x versus baseline; at 10%, it reports 15485.43 us, 34.69 GB/s, and 0.24x.The output is organized by per-GPU stress level and includes avg_us, avg_GBps, and vs-baseline columns.Source: §A.6 · p. 14
  • The E3 example reports total wall-clock time of 20812851.140 us, with communication at 33.90% and compression at 20.53%.The output also lists overlap (comm+comp) as 4272003.499 us (20.53%).Source: §A.6 · p. 14

A.7. Experiment customization

  • Users can customize CC-Bench through eight independent JSONC configuration files and by implementing backends, compressors, profiling kernels, and deviation metrics.Source: §A.7 · p. 14
Loading 2609.08739v2…