BenchmarkAnything: Agent-Driven Construction of Simulator-Ready Microarchitecture Benchmarks

I. INTRODUCTION

• Fewer than 100 cases across the combined SPEC family raise the question of whether design decisions from such a small set represent pervasive real-world workloads.Manual selection is subjective, labor intensive, difficult to scale, and poorly suited to rapidly changing software and varied deployment needs.Source: §I · pp. 1–2
• BenchmarkAnything uses LLM-driven agents to collect open-source applications, resolve dependencies, compile executables, and statistically distill a representative suite from measured microarchitectural features.The stated goals include reducing human selection bias and improving scalability and transfer across use cases.Source: §I · p. 2
• openbench-universe contains 272 benchmarks across 14 domains, while openbench-core is a distilled subset designed for efficient simulation.The benchmark family bridges industry-standard tasks and newer software deployments.Source: §I · p. 2
• openbench achieves higher-fidelity performance assessments than legacy alternatives, capturing contemporary memory-hierarchy pressure and instruction footprints that expose bottlenecks legacy suites often miss.The authors position openbench as a complement to SPEC, which remains a stable reference.Source: §I · p. 2
• The authors report that principled core-subset selection broadens workload diversity and coverage while preserving simulation efficiency.They present openbench as a more realistic and timely target for next-generation processor design.Source: §I · p. 2

A. Benchmark Importance for CPU Design

• Benchmark suites provide a repeatable basis for comparing designs, quantifying bottlenecks, and reasoning about performance, power, and area tradeoffs.Architects use benchmark and input pairs to approximate deployment workloads in pre-silicon design, post-silicon evaluation, and compiler and runtime tuning.Source: A. Benchmark Importance for CPU Design · p. 2

B. Existing CPU Benchmark and Manual Benchmark Curation

• SPEC CPU suites have been central to CPU evaluation for more than three decades, but their application selection, build procedures, dependencies, and inputs require substantial manual curation.SPEC CPU2017 extends CPU2006 with more complex integer, floating-point, and multithreaded workloads.Source: B. Existing CPU Benchmark and Manual… · p. 3
• SPEC CPU2017 workloads occupy a relatively narrow behavior region and underrepresent memory, instruction-footprint, and other characteristics of modern data-intensive applications.Manual curation also scales poorly: SPEC CPU2017 took more than a decade to create, and adding workload variation requires substantial effort and application knowledge.Source: B. Existing CPU Benchmark and Manual… · p. 3

C. LLM Agents in Software Engineering

• LLM-agent systems combine models with perception, memory, and action for tool use and iterative refinement, while multi-agent systems coordinate specialized roles.The paper distinguishes its goal from repository-to-CLI systems: it transforms software into benchmarks.Source: C. LLM Agents in Software Engineering · p. 3

III. BENCHMARKANYTHING: AGENT-DRIVEN FRAMEWORK

• BenchmarkAnything retrieves, compiles, verifies, and standardizes repository applications through a four-phase agent workflow; 93% of projects retrieved in Phase 1 become standardized benchmarks in the current implementation.The agents use MiniMax-M2.5 as their single backbone model.Source: §III · p. 3, Figure 2 · p. 4

A. Phase 1: Benchmark Searching

• A scheduling agent classifies user requirements into domain-driven, adoption-driven, and capability-driven types, then dispatches specialized subagents to search repositories in parallel.This decomposition aims to cover both prominent applications and the ecosystem's long tail.Source: A. Phase 1: Benchmark Searching · pp. 3–4
• Review agents screen candidates for adoption, CPU executability, input availability, and simulator compatibility, rejecting GPU-only or non-application repositories unless a CPU implementation is targeted.Candidates are consolidated into a manifest, deduplicated, and expanded through further searching if the valid pool falls below its target.Source: A. Phase 1: Benchmark Searching · p. 4

B. Phase 2: Autonomous Build and Workload Synthesis

• Podman isolates each candidate's build and dependency installation, avoiding pollution of the shared host environment.The container allows dependency resolution without requiring superuser privileges on the host.Source: B. Phase 2: Autonomous Build and… · p. 4
• Compilation enforces a statically linked x86-64 executable because the simulated execution environment lacks dynamic-linker support.The main agent also disables unsupported vector extensions such as AVX-512 and enforces single-threaded execution.Source: B. Phase 2: Autonomous Build and… · p. 5
• Input datasets follow a provenance hierarchy and are sized to cover a 1B-instruction warmup plus a 100M-instruction region of interest.Official or repository-provided inputs are preferred, with documented public datasets as a fallback.Source: B. Phase 2: Autonomous Build and… · p. 5
• The Ralph loop sends compilation or runtime failures through repair-agent diagnosis and targeted feedback until all acceptance criteria hold or the iteration budget is exhausted.Exit requires a static ELF, successful standardized-script execution, and enough native instructions for the simulation window, later confirmed under gem5.Source: B. Phase 2: Autonomous Build and… · p. 5

C. Phase 3: Tier-2 Host Environment Verification

• Simulation failures from unsupported system calls or instructions trigger targeted artifact repair, then stricter recompilation if needed.Tier-2 host verification checks compatibility with gem5 O3CPU after native-executable criteria have been met.Source: C. Phase 3: Tier-2 Host Environment… · p. 5

D. Phase 4: Standardized Artifact Collection

• The framework packages validated static binaries, execution wrappers, and required inputs in a uniform topology, enabling plug-and-play deployment.Intermediate build files are discarded.Source: D. Phase 4: Standardized Artifact… · p. 5
• Accepted benchmarks can be replayed without invoking a model because their source revision, build settings, inputs, execution command, checks, and gem5 invocation are frozen.This keeps published packages fixed despite LLM nondeterminism or changes in later discovery runs.Source: D. Phase 4: Standardized Artifact… · p. 5

E. The Ralph Loop: Iterative Reflection and Repair

• The Ralph loop's acceptance gate requires static linking, crash-free execution, and completion of the region of interest before an iteration can exit.The gate is a formal verification criterion, not a soft heuristic.Source: E. The Ralph Loop: Iterative… · p. 5
• Build failures drive iterative refinement because repositories have distinct build systems, dependencies, and environmental assumptions.Automated repairs include substituting static-friendly libraries, stubbing optional features, and pruning irreducibly dynamic dependencies; applications that remain non-static are deferred.Source: E. The Ralph Loop: Iterative… · p. 5
• The repair cycle mirrors human debugging but operates at machine speed and scales to hundreds of repositories in parallel.The paper identifies the Ralph loop as BenchmarkAnything's most critical mechanism for operating at scale without human intervention.Source: E. The Ralph Loop: Iterative… · p. 5

A. Experimental Setup and Profiling Space

• 30 configurations vary core type and width, reorder-buffer and queue depths, cache sizes, branch predictors, prefetchers, and replacement policies.Infeasible combinations are pruned to retain architecturally realistic designs.Source: A. Experimental Setup and Profiling… · p. 6
• Profiling uses gem5 syscall emulation under x86, fast-forwarding 1B instructions before collecting statistics over a 100M-instruction ROI.Warmup runs on a simple CPU with warm caches; the ROI runs on the detailed out-of-order core.Source: A. Experimental Setup and Profiling… · p. 6
• The 30-point sweep samples balanced, realizable designs across principal architectural axes rather than densely varying a single structure.The instruction counts for warmup and ROI are user-adjustable and follow common simulation practice.Source: A. Experimental Setup and Profiling… · p. 6

B. The Scale, Diversity, and Coverage of the openbench-universe

• The openbench-universe is a BenchmarkAnything-collected workload pool of 272 standardized applications from diverse open-source projects, grouped into 14 application domains; the study focuses on 272 for tractability.Source: B. The Scale, Diversity, and Coverage… · p. 6, Table II · p. 7
• openbench-universe covers modern domains and languages beyond SPEC2017's 23 benchmarks, which are written in C, C++, and Fortran.Examples of additional domains include machine learning, graph processing, and cryptography; openbench also includes Rust, Go, and other modern systems languages.Source: C24 OoO 2 128 64 48/36 64kB/8-way… · p. 6

C24 OoO 2 128 64 48/36 64kB/8-way 1MB/16-way TAGE SPP DRRIP

• Ten behavioral clusters span distinct hardware bottlenecks, from low-IPC Memory Bound workloads to high-throughput High IPC (I-Cache Bound) workloads.The clusters derive from multidimensional profiles, and workloads with similar mean IPC can diverge in cache misses or configuration sensitivity.Source: C24 OoO 2 128 64 48/36 64kB/8-way… · pp. 6–7
• Core-8's convex hull covers 2.12× the SPEC2017 hull area, while Core-23 reaches 2.13× and further improves fidelity to the full universe.Core-23 is a nested superset of Core-8.Source: Figure 4 · p. 8
• C15 significantly outperforms C24 on openbench-universe, although the two configurations achieve parity under SPEC2017.C15 is a wider core with a leaner memory subsystem; the result illustrates that benchmark choice can flip the apparent ranking, not that C15 is universally better.Source: C24 OoO 2 128 64 48/36 64kB/8-way… · pp. 7–8
• Most benchmark construction costs fall below 0.05M tokens, with a mean of 0.03M and median of 0.021M.The paper presents this distribution as evidence of BenchmarkAnything's low construction cost and scalability.Source: C24 OoO 2 128 64 48/36 64kB/8-way… · p. 8

Subset selection

• PCA-based k-medoids selects k = 8 medoids using Pearson distance between IPC profiles, with a soft tie-break favoring widely adopted projects.Pearson distance groups benchmarks with similar configuration-response profiles despite different absolute IPC.Source: C. Experiment-based Selection from… · p. 8, Figure 7 · p. 9

Core-8 composition

• Core-8’s eight medoids span eight application domains, from memory-latency and hash-intensive workloads to bioinformatics and physics.Table III lists the retained medoids and their cluster justifications.Source: C. Experiment-based Selection from… · p. 8, Table III · p. 10

Construction cost

• Most openbench-universe benchmarks use less than 0.05M tokens; mean usage is 0.03M and median usage is 0.021M under MiniMax-M2.5.Source: Figure 6 · p. 9

Performance fidelity

• Core-8 has 3.14% mean absolute, 6.54% p90, and 9.10% maximum error; Core-23 reduces these to 0.83%, 2.06%, and 2.31%, respectively.Core-23 is formed by retaining Core-8 and adding 15 benchmarks.Source: C. Experiment-based Selection from… · p. 9

Behavioral coverage

• Core-8 covers 2.12× SPEC2017’s convex-hull area, while nested Core-23 reaches 2.13×.Core-23’s hull contains the Core-8 hull.Source: C. Experiment-based Selection from… · pp. 9–10

Ranking fidelity

• ρ = 0.989: Core-8 configuration rankings closely mirror those from the full 272-workload universe.Both rankings use geometric-mean IPC.Source: C. Experiment-based Selection from… · p. 8, Figure 8 · p. 10

Core-subset fidelity

• Core-8 reports mean absolute/p90/maximum errors of 3.14%/6.54%/9.10%, while Core-23 reports 0.83%/2.06%/2.31%.Source: Figure 9 · p. 10

Interpretation

• The authors suggest diminished openbench gains indicate SPEC-focused optimizations may generalize less reliably, amid a feedback loop of designing and validating on SPEC.Source: §V · p. 10

Microarchitectural coverage

• 470 versus 141 D-cache MPKI tails, a 95% L2 miss-rate tail, and 53 versus 14 branch-misprediction MPKI tails show openbench extends SPEC2017’s event envelope.The L2 comparison also has 39 openbench workloads above the SPEC2017 maximum.Source: §V · p. 10, Figure 10 · p. 11

Configuration responses

• 47 of 272 openbench workloads exceed SPEC2017’s 2.5 maximum for C24-to-C1 speedup, while prefetcher and replacement returns are near zero at the openbench median.The suites’ median width speedups both lie near 2.0.Source: §V · p. 10, Figure 10 · p. 11

Implications

• Broader, continuously refreshed workloads should complement legacy suites, and agent frameworks make large-scale acquisition, standardization, and distillation practical.Source: §V · p. 11

Evaluation boundary

• Syscall-emulation abstracts kernel execution, page faults, context switches, and complex I/O, limiting visibility into OS- or network-dependent bottlenecks.Source: §V · p. 11

Artifact scope

• The current artifact targets single-threaded, user-level, statically linked, AVX-free x86 binaries; retargeting requires changing declarative constraints, not redesigning the agent flow.Source: §V · p. 11

Future work

• Future work targets full-system simulation, synthesized multithreaded applications, and heterogeneous targets including GPUs and domain-specific accelerators.Source: §V · p. 11

• BenchmarkAnything collects 272 applications from open-source platforms into openbench-universe, then distills them into the efficient representative openbench-core subset.Source: §VI · p. 11
• Up to 2.12× the behavioral-space coverage of SPEC CPU2017, with Core-8 preserving strong ranking and performance fidelity to the full universe.Source: §VI · p. 11
• openbench reduces human selection bias and exposes critical execution bottlenecks missed by legacy suites, offering a target for more robust future architectures.Source: §VI · p. 11