Source-linked AI summary
Adaptation Fidelity of SPEC CPU2026
Doa'a Al-Otoom, Mahesh Madhav
TL;DR
The paper addresses the lack of quantitative evidence about how adapting real applications into standardized SPEC benchmarks changes fidelity. It compares upstream and adapted binaries under single-copy and 192-copy workloads, finding close correspondence for most benchmarks but substantial outlier differences, especially under multi-copy load. The results validate the fidelity gap as a measurable consequence of portability, determinism, and CPU-focused adaptation rather than evidence of a flaw.
Problem
Quantitative evidence about fidelity loss caused by adapting upstream applications into standardized benchmarks has been lacking.
Method
The study compares upstream and adapted SPEC CPU 2026 binaries using identical official workloads in single-copy and 192-copy execution modes.
Results
Most benchmarks correlate closely with their upstream applications in single-copy runs, while outliers and several multi-copy cases reveal substantial adaptation effects, particularly from I/O reduction.
Takeaways & Limitations
The fidelity gap is a quantifiable consequence of enforcing portability, determinism, and CPU-bound measurement, providing data-driven validation of SPEC’s methodology.
Abstract
from arXiv · showhide
Standardized benchmarks are often criticized for not being "real workloads," but this critique is rarely backed by data. This paper provides the first systematic, quantitative analysis of the "fidelity gap" between the SPEC CPU2026 suite and its original, upstream open-source counterparts. We compile both the SPEC benchmarks and their upstream applications and execute them with official input workloads under two scenarios: a single-copy latency run and a 192-copy throughput run. Our findings show that most benchmarks exhibit high fidelity in single-copy runs, while a few outliers reveal the impact of SPEC's adaptation process. The multi-copy results further highlight the necessity of this adaptation: several benchmarks become significantly more efficient than their upstream versions under heavy load, underscoring the importance of I/O reduction. This work offers data-driven validation of SPEC's methodology, showing that the fidelity gap is not a flaw but a quantifiable consequence of enforcing portability, determinism, and CPU-centric measurement.
I. INTRODUCTION
SPEC CPU benchmarks must balance realism with portability, determinism, and reproducibility. This paper quantitatively investigates how adaptation changes fidelity to upstream applications.
- Benchmark adaptations remove randomness, I/O, and platform-specific code to enable fair CPU comparisons, raising questions about sacrificed realism.
- Critiques calling benchmarks synthetic have rarely been tested with quantitative evidence about adaptation-induced fidelity loss.
- The study compares all 26 SPECrate benchmarks with upstream open-source counterparts using identical official workloads to quantify adaptation deltas.
- Analyzing these deltas distinguishes high-fidelity proxies from significantly perturbed benchmarks and links divergence to changes such as removing intrinsics or I/O.
- The paper frames fidelity differences as a data-driven basis for understanding where standardized benchmarks reflect applications and where standardization causes divergence.
- SPEC warns that its benchmarks may perform differently from similarly named applications and should only be compared with other SPEC CPU 2026 results.
II. METHODOLOGY
The methodology builds upstream and adapted binaries from a common source baseline and runs them under matched harness conditions. It evaluates fidelity under both single-copy and 192-copy workloads while controlling architecture and execution variables.
- The experiment builds two binary sets from a common source baseline, executes them in an identical environment, and verifies the results.
- Build and Execution: Upstream binaries use SPEC’s redistributable source snapshot, while adapted binaries are built through the SPEC harness after portability and determinism hardening.
- Build and Execution: The comparison replaces the adapted binary call with the upstream binary while preserving workloads, arguments, and environment variables through specinvoke.
- Build and Execution: A few experiments deviated from this replacement procedure, with the exceptions documented in the benchmark-notes section.
- Runs on AArch64 stress-test portability because many upstream applications target x86 and system-call overhead can be higher on AArch64.
- Single-copy measures one official refrate invocation, whereas multi-copy runs 192 simultaneous copies to expose I/O and shared-resource effects under load.
B. Verification
The experiment verifies that upstream executions perform equivalent work by applying SPEC’s official output-comparison process after each run.
- After every upstream run, specinvoke compare.cmd checks outputs against golden reference files within each benchmark’s specified tolerances.
C. Platform
The study measures performance on one specified AArch64 AmpereOne platform. Results are reproducible there, but other platforms may produce different or even opposite outcomes.
- All measurements use an AArch64 platform based on the AmpereOne SoC with the configuration listed in Table I.
- The reported results are reproducible on this platform, while other platforms may behave differently or report opposite performance results.
III. RESULTS
Results compare official SPEC binaries with upstream sources across single-copy and full-load settings, focusing on 23 benchmarks after excluding three without suitable comparisons.
- Experimental Scope: 23 benchmarks were analyzed after excluding 772.marian, 749.fotonik3d, and 782.lbm.772.marian was heavily modified and did not build on AArch64; the other two lacked public reference workloads or redistributable sources.
- Result Presentation: Table II summarizes the performance comparisons from Figures 1 and 2, marking cases where SPEC is faster or slower.Green denotes SPEC-faster results, while red denotes SPEC-slower results.
- Measurement Settings: Single-copy performance was measured for one invocation using the official SPECrate workload size.The results were plotted as speedup-sorted S-curves and summarized in Table II.
- Measurement Settings: The results compare SPEC CPU2026 benchmarks against upstream baselines using single-threaded performance on AmpereOne.Values above 1.0 represent speedup relative to the original application.
A. Single-Copy Results
Single-copy comparisons across 23 benchmarks form three groups: SPEC-faster cases, near-equivalent cases, and two upstream-faster outliers tied to adaptation choices.
- Performance Groups: Ratios of 1.08×–1.44× identify benchmarks running notably faster under SPEC, primarily after removing I/O and system-level bottlenecks.These ratios compare SPEC performance with the upstream baseline.
- Performance Groups: Ratios near 1.00× fall within run-to-run variation, indicating close single-copy performance between SPEC and upstream builds.This is the central cluster in the three-group distribution.
- Outliers: 706.stockfish and 731.astcenc are the upstream-faster outliers, with deltas attributed to removing non-portable assembly or ISA intrinsics.For these benchmarks, the upstream build outperformed the SPEC binary.
- Adaptation Effects: SPEC also suppresses threading and mutex overhead, replaces non-standard extensions with ISO-standard code, and removes randomness that could contend for the system RNG.These modifications support portability, reproducibility, and focus on computational workload.
B. Multi-Copy Results
Under 192-copy full-system load, several adapted SPEC benchmarks become much more efficient than upstream applications because upstream I/O bottlenecks intensify under multitasking.
- Heavy-Load Outliers: 707.ntest, 735.gem5, and 723.llvm become far more efficient than their upstream applications in the multi-copy comparison.The passage attributes this difference to severe I/O bottlenecks in the original code under heavy multitasking load.
- Heavy-Load Outliers: 706.stockfish improves from 0.62× in single-copy to 0.75× in multi-copy, while remaining slower than upstream.The result suggests scaled-execution bottlenecks are consistent across both binaries.
- Resource Contention: 766.femflow and 767.nest remain positive outliers, but their SPEC-over-upstream deltas shrink under multi-copy execution.Possible contributors include cache pressure, memory bandwidth, and CPU scheduling overhead.
C. Other Factors
System time reduction and build-toolchain differences explain some observed deltas, especially for GCC and Cppcheck.
- System Time: 8.1 of GCC’s 14 percentage-point single-copy delta is attributed to intentionally reduced system time.SPEC targets user-level compute performance, whereas the comparison uses upstream total execution time.
- System Time: All other upstream applications showed system time below 1%, limiting this explanation outside GCC.The passage contrasts GCC with the remaining applications.
- Build Tooling: Cppcheck runtimes differed despite identical -O3 flags, suggesting toolchain, configuration, or Makefile differences can affect deltas.The passage identifies build tooling as another possible source of variation.
IV. BENCHMARK NOTES
Benchmark-specific adaptations produce measurable performance differences, with portability changes sometimes slowing SPEC and I/O or bottleneck reductions sometimes accelerating it, especially at high copy counts.
- Outliers: 38%: upstream Stockfish 15 outperformed the SPEC binary, as processor-specific intrinsics removed for portability reduced NNUE::evaluate(...) time by 77% upstream.The performance gap is concentrated in the NNUE evaluation routine.
- Outliers: 100×: ntest reduced write syscalls by flushing once per board instead of after every character, producing a large multi-copy speedup.The original I/O issue manifests at scale because typical users run one board rather than thousands.
- Outliers: SQLite’s -testset fp delta concentrated in software floating-point emulation, with __multf3 consuming 43% of runtime, followed by __divtf3 at 7.6% and __sfp_handle_exceptions at 7.0%.The developer attributed the gap to changing long double to double for portability and fairness.
- Outliers: 36%: LLVM/Clang improved at high copy count after SPEC replaced full-text disk output with an in-memory sha512sum while preserving equivalent work.
- Outliers: 35%: ASTCENC slowed after SPEC removed assembly-language intrinsics for portability.
- Outliers: SPEC suppressed gem5’s unnecessary large configuration-file I/O, explaining the upstream binary’s slowdown at high copy counts.
- Outliers: 18%: Femflow gained performance in single-copy mode after SPEC removed unnecessary mutexes and locks from the single-threaded benchmark.
V. CONCLUSION
The study quantitatively evaluates how SPEC CPU2026 adaptations change fidelity relative to upstream applications. Most adapted benchmarks remain closely correlated, while outliers expose principled trade-offs that support SPEC’s warning against direct comparisons.
- Most adapted SPEC benchmarks correlate closely with their upstream applications, especially in single-copy runs, while preserving portability and determinism.
- SPEC’s significant slowdowns typically result from removing non-portable hand-tuned assembly or ISA intrinsics, whereas speedups typically result from reducing I/O, RNG, and threading bottlenecks.
- Multi-copy execution amplifies I/O and other bottlenecks, making SPEC’s warning against cross-comparisons particularly justified under parallel load.
- The fidelity gap is a quantifiable consequence of enforcing portability, determinism, and CPU-bound focus rather than evidence of a benchmark flaw.
- Benchmark-by-benchmark measurement gives users context for interpreting SPEC CPU2026 results and offers committees an empirical basis for judging representativeness.