Source-linked AI summary
Rolling Round-Robin Rate: Standard Heterogeneous Throughput for SPEC CPU
Mahesh Madhav, Christoph Müllner, Jiangning Liu, Philipp Tomsich, Jeff Baxter
TL;DR
Heterogeneous CPU workloads are common, but researchers lack a widely adopted method for constructing reproducible full-application mixes. The paper introduces RRR’s rolling schedule and proposes a SPEC-like throughput score with variability diagnostics, while noting that its equal exposure is count-based rather than time-weighted.
Problem
Researchers lack a widely adopted, standard construction method for full-application heterogeneous CPU rate workloads, despite the need to study unlike-program interactions.
Source: §I · p. 1, §II · p. 1Method
RRR runs each copy through the same selected benchmark roster from a different starting position, and the proposed score averages per-benchmark copy ratios and per-copy suite geomeans.
Source: §III · p. 2, §IV · p. 3, §VII · p. 4Results
The proposed reporting model combines benchmark and suite throughput scores with coefficient-of-variation diagnostics that expose benchmark sensitivity to co-runners, copy-level balance, and relative spread.
Source: §I · p. 1, §VII · p. 4Takeaways & Limitations
RRR offers a common heterogeneous CPU experiment for studying noisy neighbors, OS scheduling, heterogeneous systems, resource partitioning, and benchmark methodology.
Source: §VII · p. 4RRR provides count-based equal exposure, not a time-weighted or perfectly balanced pairwise co-runner matrix.
Source: §V · p. 4
Abstract
from arXiv · showhide
SPEC CPU has long provided a common foundation for comparing processor, compiler, memory-system, and platform performance. Its multi-copy SPECrate mode measures homogeneous throughput by running many copies of the same benchmark at once. That mode remains valuable, but modern cloud and server systems commonly run heterogeneous collections of jobs whose interactions are shaped by shared caches, memory bandwidth, power management, operating-system scheduling, and noisy neighbors. SPEC CPU 2026 introduces Rolling Round-Robin Rate (RRR), an exhibition run style that uses the existing rate suites to generate deterministic heterogeneous multiprogrammed workloads. This paper describes RRR as a benchmark methodology and proposes a SPEC-like scoring model for future RRR reporting: compute per-benchmark average throughput and coefficient of variation from per-copy ratios, compute a SPEC-style geometric mean for each copy across the suite, and average that population of copy geomeans to form a suite score with its own coefficient of variation. RRR therefore preserves the familiar SPEC throughput tradition while exposing richer information about variability, interference, and heterogeneous-system behavior.
I. INTRODUCTION
SPEC CPU provides reproducible throughput comparisons, but homogeneous SPECrate does not answer how unlike programs interact under shared-system constraints. The paper introduces RRR as a deterministic heterogeneous run style and proposes a SPEC-like score with variability diagnostics.
I. INTRODUCTION
- SPEC CPU supports reproducible comparisons through portable source benchmarks, defined run rules, reference workloads, validation, and standardized metrics.Its harness turns complex software into a common measurement experiment.Source: §I · p. 1
- SPECrate measures homogeneous throughput by running multiple copies of one benchmark at a time, a useful stress case with a geometric-mean suite score.Synchronized copies can expose shared-cache, memory-bandwidth, thermal, or scheduler limits.Source: §I · p. 1
- Modern servers and cloud systems run unlike jobs concurrently, and their behavior can depend on interference among programs and shared-resource policies.The passage names cache partitioning, SMT pairing, power limits, frequency governors, and OS scheduling as examples of policies that may behave differently under heterogeneous load.Source: §I · p. 1
- Each RRR copy runs the same selected benchmark roster from a different starting point, creating a deterministic heterogeneous workload in which every benchmark receives equal exposure across copies.The concurrent mix changes over time.Source: §I · p. 1
- RRR is proposed as a standard methodology for heterogeneous SPEC CPU runs, with a score that averages per-copy suite geomeans and reports coefficient-of-variation diagnostics at benchmark and suite levels.The paper identifies noisy-neighbor studies, OS scheduling, heterogeneous-core evaluation, resource partitioning, and reproducibility analysis as uses for these diagnostics.Source: §I · p. 1
II. WHY HETEROGENEOUS RATE NEEDS A STANDARD
Prior work establishes the need for heterogeneous workload experiments, but workload construction remains unconverged because program phases and runtimes shape contention. The section surveys workload-selection approaches, shared-resource studies, and alternative metrics to motivate a reproducible standard experiment.
II. WHY HETEROGENEOUS RATE NEEDS A STANDARD
- The figure contrasts homogeneous refrate schedules, where copies run the same selected work, with RRR schedules that roll through the benchmark roster on 48 copies.The “inc” parameter customizes how the scheduler advances through the roster.Source: Figure 1 · p. 2
- Different multiprogram benchmark definitions built from the same programs can produce drastically different conclusions, while phase alignment changes shared-resource contention.A good method should control sample imbalance without erasing the schedule imbalance it intends to measure.Source: §II · pp. 1–2
- Architecture studies select mixes through sampling, stratification, principal-component or cluster analysis, representative program-input pairs, and suite subsetting.These studies show workload selection is a methodological choice with consequences.Source: §II · p. 2
- Custom benchmark mixes support studies of cache partitioning, SMT pairing, resource scheduling, colocation fairness, prefetching, noisy-neighbor isolation, OS scheduling, and runtime systems.The recurring challenge is making workloads reproducible and comparable across studies.Source: §II · p. 2
- SPECcast and BenchCast reduce simulation cost with synchronized representative slices, whereas RRR runs complete SPEC rate benchmarks through the standard harness.RRR retains full-application interactions, including cross-effects that selected-region execution may change.Source: §II · p. 2
- RRR separates the standard experiment and a natural throughput score from unresolved fairness-metric choices, while exposing distributions for fairness, variability, and isolation studies.The passage lists system throughput, weighted speedup, harmonic means, normalized turnaround time, and fairness indices as possible metrics.Source: §II · p. 2
III. ROLLING ROUND-ROBIN RATE
RRR runs each selected benchmark on every copy, with copies starting at different roster positions and advancing through a deterministic schedule. Its design controls workload construction while preserving full-program execution and equal count-based exposure.
III. ROLLING ROUND-ROBIN RATE
- Each copy runs every selected benchmark once in roster order, starting at a different cyclic position, and verification waits until all N × M benchmarks finish without intermediate barriers.The roster contains N benchmarks and the run uses M copies.Source: §III · p. 2
- The -rrrrate_inc option controls roster advancement; with six benchmarks and inc=2, copy 0 runs (a, c, e, b, d, f) while copy 1 runs (b, d, f, a, c, e).The special case inc=0 keeps each copy on its initial benchmark and is useful for quick validation, not the heterogeneous schedule of interest.Source: §III · pp. 2–3
- RRR is deterministic and gives every benchmark one run on every copy, so published settings can reconstruct the schedule and benchmark representation does not depend on a random mix or arbitrary pairing.SPEC CPU benchmarks are selected and hardened to reduce unrelated nondeterminism.Source: §III · p. 3
- RRR preserves full-program interactions by running standard validated workloads rather than kernel-extracted or ROI-only schemes.This keeps phase changes and long-range behavior in the experiment.Source: §III · p. 3
IV. A SPEC-LIKE RRR SCORE
The proposed RRR score extends SPEC-style throughput reporting with benchmark- and suite-level ratios and variability measures. It aggregates per-copy benchmark ratios and per-copy suite geomeans separately, retaining information about heterogeneous-load variation.
IV. A SPEC-LIKE RRR SCORE
- RRR proposes benchmark-level mean throughput ratios and coefficients of variation from per-copy ratios, with the current average-time outputs retained as companion diagnostics.The paper identifies ratio_avg and ratio_cv as the natural throughput metrics for scoring.Source: §IV · p. 3
- The proposed ratio-based quantities are additions to current RRR reporting, whose formatted outputs emphasize average time and time CV.The paper presents ratio_avg, ratio_cv, basemean_avg, and basemean_cv as proposed extensions.Source: §IV · p. 3
- The table caption proposes the RRR-to-rate ratio shift as an in-band memory-boundedness index: bandwidth-bound benchmarks should rise, while compute-bound benchmarks should be nearly indifferent.The caption also identifies ratio_cv as a complementary mix-sensitivity signal.Source: Table I · p. 3
- The proposed SPECrrrrate averages each copy’s suite geomean rather than taking the geomean of benchmark scores.Each copy’s geomean is computed across the suite’s normalized rate ratios.Source: §IV · p. 3
- Benchmark-level ratio_cv and suite-level basemean_cv report relative spread across, respectively, copy ratios and copy-level suite geomeans.Together with the corresponding averages, these measures describe throughput and its spread at two levels.Source: §IV · pp. 3–4
V. LIMITATIONS
RRR’s rolling schedule has three important boundaries: load may lighten as shorter jobs finish, mixing depends on the increment, and equal exposure counts runs rather than time.
V. LIMITATIONS
- Shorter instances can become late-comers under lighter system load; the schedule spreads this effect across benchmarks exactly only when benchmark and copy counts align cleanly.The run therefore does not maintain a perfectly constant heterogeneous load throughout.Source: §V · p. 4
- The roster rotation visits every benchmark before repeating only when gcd(k,N)=1; increments sharing a factor with N repeat alignments sooner and reduce co-runner diversity.Relatively prime increments provide the most diverse roster rotation.Source: §V · p. 4
- A benchmark’s overlap with phases and peers depends on runtime: longer-running benchmarks overlap with more than shorter-running ones.Equal exposure is count-based, so RRR is a standardized heterogeneous throughput experiment rather than a perfectly balanced pairwise co-runner matrix.Source: §V · p. 4
VI. USES FOR INDUSTRY AND ACADEMIA
RRR offers a repeatable mixed-load throughput experiment for industry and a common baseline for academic studies. Its uses include analyzing variability and targeted workload mixes, while its scores answer a different question from existing SPEC modes.
VI. USES FOR INDUSTRY AND ACADEMIA
- The same RRR run supports average-throughput comparison, per-benchmark sensitivity analysis, and variability analysis for noisy-neighbor behavior.It provides a repeatable mixed-load throughput experiment for CPU buyers, vendors, cloud providers, operating-system teams, compiler teams, and silicon designers.Source: §VI · p. 4
- RRR offers academia a common baseline for studies that currently build custom workload mixes, including scheduler, heterogeneous-core, and resource-partitioning research.Compiler and architecture studies can examine whether an optimization improves the average while increasing relative spread, or improves stability without changing the headline score.Source: §VI · p. 4
- RRR has been used as a full-suite heterogeneous reference and as a composition primitive for targeted proxy workloads that bracket selected microarchitectural pressure points to mimic DCPerf.The cited study distinguishes this use from a full quality-of-service benchmark for datacenter co-location.Source: §VI · p. 4
- SPECspeed, SPECrate, and SPECrrrrate answer latency-oriented single-workload, homogeneous-throughput, and heterogeneous-throughput questions, respectively, so their headline scores are not interchangeable.Taken together, the modes give a more complete view of CPU performance in the many-core era.Source: §VI · p. 4
VII. CONCLUSION
The conclusion presents RRR as a standardized heterogeneous extension to SPEC CPU throughput measurement. Its proposed reporting model retains reference-machine ratios while adding suite scoring and variability diagnostics.
VII. CONCLUSION
- Every copy runs every selected benchmark, rolling through the roster from a different starting point to control workload construction while preserving full-application execution.This supports measurements conducted similarly across systems and studies.Source: §VII · p. 4
- Averaging normalized throughput per benchmark and then averaging per-copy suite geomeans gives a practical RRR score grounded in ratios against a reference machine.The per-copy geomean is computed across the suite using a SPEC-style geometric mean.Source: §VII · p. 4
- The coefficient of variation of copy geomeans exposes benchmark sensitivity to co-runners, copy-level balance, and relative spread around the final score.The paper expects these diagnostics to be useful for studying noisy neighbors, scheduling, heterogeneous systems, resource partitioning, and benchmark methodology.Source: §VII · p. 4