Source-linked AI summary
SoK: ARCUS: On the Efficiency and Efficacy of Hardware Fuzzing
Alenkruth Krishnan Murali, Raghul Saravanan, Sai Manoj P D, Ashish Venkat
TL;DR
Hardware fuzzing must address verification challenges created by increasingly complex processors, but its goals and trade-offs differ across ISA, microarchitecture, and RTL layers. ARCUS systematically analyzes these approaches using a cross-layer taxonomy and common design dimensions, finding that current techniques remain constrained by incomplete reference models and siloed testing. The paper therefore highlights intelligent generation, scalable or layered oracles, standardized evaluation, and cross-layer testing as research directions.
Problem
Increasing processor complexity challenges scalable hardware verification, while existing fuzzing approaches differ across abstraction layers and retain important unmet needs.
Method
ARCUS systematically classifies hardware fuzzers by abstraction layer and methodology, then compares them across dimensions such as inputs, algorithms, feedback, coverage, and GRMs.
Results
Contemporary hardware fuzzing has moved beyond brute-force random processes but remains constrained by incomplete reference models and siloed testing methodologies.
Takeaways & Limitations
The analysis points to intelligent instruction generation, scalable and layered hardware oracles, standardized metrics and benchmarks, and cross-layer testing for more comprehensive verification.
Takeaways & Limitations
Comparative RTL evaluation is hampered by unavailable baselines and nonstandardized metrics, which produce discrepancies and hinder interpretation across studies.
Abstract
from arXiv · showhide
This work presents a comprehensive analysis of contemporary hardware fuzzing techniques applied across three major abstraction layers: Instruction Set Architecture (ISA), microarchitecture, and Register-Transfer Level (RTL). Our study examines key factors including input stimulus quality, mutation strategies, feedback mechanisms, target platforms, reference models, and achieved coverage. We find challenges, goals, and design trade-offs vary significantly across abstraction layers. We further identify several unmet needs in current hardware fuzzing practices, such as intelligent input generation, reliable and scalable golden reference models, expressive feedback channels, and cross-layer integration. Building on these insights, we outline future research directions, including hybrid fuzzing frameworks, AI-assisted test generation, scalable reference models, standardized evaluation metrics and benchmarks, and human-in-the-loop automation for guided exploration and analysis. Together, they aim to unlock efficient, reliable, and comprehensive hardware verification solutions.
1. Introduction
ARCUS provides a systematic analysis of hardware fuzzers across ISA, microarchitecture, and RTL abstraction layers. It introduces a taxonomy and shared analysis dimensions to compare diverse approaches, identifies shortcomings, and outlines future research opportunities.
- Hardware verification struggles to scale with processor complexity, while hardware fuzzing can explore unseen paths and uncover bugs.
- ARCUS analyzes contemporary hardware fuzzers across ISA, µArch, and RTL layers, covering diverse targets and verification goals.
- The two-level taxonomy classifies fuzzers by abstraction layer and by the fuzzing methodology used within each layer.
- The paper outlines future opportunities involving hybrid cross-layer pipelines and AI-driven input generation and mutation to improve precision, scalability, adaptability, and efficiency.
- ARCUS standardizes comparisons using dimensions including input stimulus, fuzzing algorithm, coverage feedback, and Golden Reference Models.
- Existing approaches have limited feedback, weak input generation, reliance on GRMs, and poor integration across abstraction layers.
2. ISA-Level Fuzzing
ISA-level fuzzing targets black-box, post-silicon processors and related software models to uncover deviations from documented or cross-implementation ISA behavior. Its effectiveness depends on input-generation structure, reference-model reliability, feedback, and the cost of manual diagnosis.
- Input stimulus: Structured random generation narrows the search space while preserving diversity, enabling effective discovery of undocumented instructions, disassembler bugs, and decoder bugs.SkipScan’s rule-driven mutations searched fewer instructions than Sandsifter and UISFuzz while identifying the same number of undocumented instructions as Sandsifter.
- Feedback and analysis: Fault analysis distinguishes undocumented instructions from privilege-restricted instructions, but misses bugs without observable exceptions and can be imprecise under hypervisors or emulators.Fault codes help separate undefined instructions from faults caused by executing privileged instructions in a lower privilege mode.
- Reference models: Differential fuzzing can isolate discrepancies across hardware and software references, but faulty or incomplete GRMs may create false positives or conceal real bugs.Combined GRM and N-oracle approaches improve robustness while increasing harness complexity and often requiring manual filtering of undefined or implementation-specific behavior.
- Reference models: Reference choice affects throughput: software models may be slower than hardware references, while combined GRM pipelines can check faster models before advancing to slower ones.Virtualized CPU emulation is faster than complete board emulation, and hardware execution speed depends on the platform’s capabilities.
- Manual analysis: Manual analysis can require additional effort when faulting instructions are not stored for later investigation.A filtering mechanism such as UISFuzz’s recheck can reduce this burden.
- Targets and bug classes: ISA fuzzers target black-box CPUs and identify undocumented instructions, decoder and disassembler bugs, and semantic deviations across hardware and software implementations.They compare target behavior with documentation, emulators, disassemblers, or other reference models.
3. Microarchitecture-Level Fuzzing
Microarchitectural fuzzing targets unspecified internal processor behavior using black-box CPUs, specialized inputs, observation channels, and relational models. Its effectiveness depends on input quality, custom observation models, and scalable automated analysis, while reference-model and attack-template limitations constrain coverage and diagnosis.
- Scope and objectives: Microarchitectural fuzzing exploits the gap between documented ISA behavior and implemented microarchitecture to uncover vulnerabilities in black-box, post-silicon CPUs.Targets include speculative execution, resource contention, and undocumented hardware behavior.
- Scope and objectives: µArch fuzzers use publicly available microarchitectural information to guide input generation between rounds, despite treating targets as black boxes.Generated stimuli are executed on targets and, when available, on golden reference models.
- Input generation: Structured, constrained-random, and mutation-based generation are common because long instruction sequences make exhaustive manual search infeasible.Random methods generate valid instructions, arrange them into basic blocks, and avoid faults such as out-of-bounds memory accesses.
- Input generation: Pattern-coverage-guided multiphase generation can identify bugs quicker, while simple randomization and exclusion strategies effective for ISA fuzzing are ineffective for µArch fuzzing.The paper characterizes test-generation complexity as scaling with bug complexity.
- Fuzzing methodologies: Observation-independent fuzzing avoids golden reference models but requires large test corpora and provides limited precision for bug localization and classification.Without a baseline, distinguishing expected from erroneous behavior is difficult when behavioral differences are subtle.
- Fuzzing methodologies: Model-based relational testing checks behavioral rules without a full hardware model, but models may over-approximate or under-approximate behavior and require manual construction for new side channels.Existing examples model cache side channels; adapting them to TLB or prefetcher tables requires a new model.
- Fuzzing methodologies: Directed fuzzers such as ABSynthe refine target-specific spy programs with runtime contention measurements and Gaussian Naïve Bayes classification without relying on a GRM.ABSynthe uses a leakage map and all possible instruction combinations to synthesize contention side channels.
- Findings: The specificity and complexity of targeted bugs require custom observation models and scalable automated analysis capabilities.The surveyed approaches identify attacks and variants including Medusa, SISCloak, Ghostwrite, and Osiris timing channels.
4. RTL-Level Fuzzing
RTL fuzzing operates pre-silicon on observable, controllable designs, using generated and mutated stimuli, coverage feedback, and often golden reference models to expose functional bugs. Approaches trade off exploration efficiency, scalability, coverage precision, instrumentation overhead, and reproducibility.
- Targets and characteristics: RTL fuzzing targets CPU cores and IP peripherals where source availability enables direct access to design internals.RTL models form the foundation for silicon tape-out and support fine-grained coverage extraction in simulation.
- Input stimulus: Instruction sequences commonly stimulate CPU designs, while bit streams provide lower-level inputs for IPs without instruction interfaces.
- Seed generation: Static random generation and FIFO scheduling make effective seed discovery resource-intensive and struggle to reach deeply buried bugs without guidance.The large ISA-exposed instruction space makes sufficient design-space coverage difficult to achieve with naïve methods.
- Seed generation: Dynamic and hybrid seed strategies prioritize high-yield mutations or coverage-improving seeds, and identify the same seven CVA6 vulnerabilities with fewer tests than TheHuzz on average.HypFuzz, PSOFuzz, and MABFuzz are reported as quicker because they prioritize interesting seeds.
- Evaluation and reproducibility: Comparisons are hindered by unavailable baselines, heterogeneous metrics, and missing contextual coverage breakdowns or experimental conditions.The paper recommends standardized metrics, measurement protocols, reporting, open tools, benchmarks, and datasets.
- Coverage metrics: 7% versus 71% overhead: DiFuzzRTL’s register coverage improves scalability relative to TheHuzz’s fine-grained instrumentation.ProcessorFuzz avoids RTL-simulator instrumentation with CSR-transition coverage, but register-focused approaches can miss bugs outside multiplexer select signals.
- Coverage metrics: Formal and ML-based coverage methods expose trade-offs between reaching hard-to-cover regions, scalability, accuracy, human bias, and false positives.The analysis calls for hybrid and adaptive metrics balancing accuracy, scalability, and overhead.
5. Cross-cutting Insights and Opportunities
Across abstraction layers, hardware fuzzing increasingly uses structured, targeted exploration but remains constrained by heterogeneous objectives, limited algorithmic diversity, incomplete reference models, manual effort, and inconsistent evaluation.
- Opportunities: These recurring limitations motivate research directions addressing input generation, reference models, feedback, standardization, and cross-layer integration.The paper presents the limitations as opportunities for a forward-looking research agenda.
- Structured Input Space Exploration: Structured input generation consistently outperforms unconstrained randomization across ISA, microarchitectural, and RTL fuzzing classes.Examples include SkipScan’s byte-level mutations, Revizor’s contracts, and hybrid or dynamic RTL seed generation.
- Heterogeneous Metrics and Closed-source Fuzzers: Evaluation metrics differ by abstraction layer and often omit coverage types, test counts, or experimental context, limiting quantitative comparison.Closed-source fuzzers also produce discrepancies between independently reproduced baselines.
- Attack-Driven µArchitectural Fuzzing: ISA fuzzing is specification-driven, RTL fuzzing is coverage-driven, and microarchitectural fuzzing is predominantly attack-driven.These objectives lead each layer toward different exploration and evaluation choices.
- Conservation of Manual Effort: Automation reduces manual test writing but shifts effort toward oracle construction, configuration, coverage extraction, and result triage.The paper characterizes this redistribution as conservation of manual effort rather than elimination of human work.
- Limited Algorithmic Diversity: Hardware fuzzing research shows limited algorithmic diversity, with many RTL methods refining coverage-guided fuzzing, genetic algorithms, or reinforcement learning.The paper identifies constraint solving, program synthesis, and other search paradigms as opportunities for diversification.
- Trusted Golden Reference Models: Reliable GRMs remain difficult for complex and proprietary designs because differential testing detects inconsistencies but not shared implementation bugs.Microarchitectural behaviors often lack complete specifications, while manual and HDL-derived models introduce additional difficulties.
6. Future Directions
The proposed research agenda targets intelligent generation, scalable oracles, richer feedback, coordinated cross-layer fuzzing, standardized benchmarks, and human-guided analysis. These directions address current limits in depth, scalability, comparability, and triage.
- Intelligent Input Generation: LLMs and AI agents could generate semantically rich instruction sequences and guide fuzzing toward edge cases and deeper state-space regions.The proposed agents would observe progress and reward inputs that expand coverage.
- Reliable and Scalable Reference Models: Layered, differential, and learned reference models are proposed as alternatives to impractical cycle-accurate GRMs for modern processors.Layered models combine lightweight architectural simulators with focused models for components such as caches, branch predictors, and TLBs.
- Richer Feedback: Microarchitectural side effects such as performance counters, cache misses, and branch mispredictions could supplement coarse oracles as feedback signals.The agenda also identifies side-channel-guided fuzzing as an alternative feedback direction.
- Cross-Layer (Hybrid) Fuzzing: Cross-layer fuzzing would coordinate stimuli, coverage, and oracles so workloads and anomalies can propagate between ISA, µArch, and RTL levels.Proposed metrics span instruction coverage, microarchitectural event space, and reached RTL state.
- Benchmarks and Standardization: A FuzzBench-like ecosystem using open CPU designs, controlled fault injection, and a common harness could standardize transparent comparisons.The harness would report coverage, bug-detection latency, test counts, and resource utilization in a standardized format.
- Human-in-the-Loop Automation: Human-in-the-loop frameworks could reduce redundant traces while letting experts redirect exploration, cluster failures, and refine candidate tests.Visualization and automated root-cause grouping are proposed to support triage and strategic guidance.
7. Conclusion
ARCUS systematizes hardware fuzzing across abstraction levels using a taxonomy and design dimensions, then identifies open challenges and practical guidance for selecting fuzzers. It concludes that intelligent inputs, scalable oracles, standardized evaluation, and cross-layer testing are needed for more systematic verification.
- Systematization: ARCUS classifies contemporary hardware fuzzers by ISA, µArch, and RTL abstraction layers, then characterizes them across common design dimensions.The paper also provides a flowchart used with Table 2 to choose among fuzzers.
- Conclusion: Current approaches have become more targeted than brute-force random fuzzing but remain limited by incomplete reference models and siloed testing.The conclusion frames intelligent generation, layered oracles, standardized metrics and benchmarks, and cross-layer testing as enabling directions.
Ethical Considerations
The work anticipates no potential harm from publication because it systematizes publicly available hardware-fuzzing research.
- The authors foresee no potential harm to stakeholders from publishing this work.They characterize the contribution as a systematization of the state of the art in hardware fuzzing.
Open Science Contribution
The study surveys peer-reviewed or publicly archived academic hardware-fuzzing works using quantitative data derived directly from those papers.
- The survey covers academic works accepted through peer review or publicly archived.The references section cites all surveyed works.
- The analysis presents quantitative data derived directly from the surveyed papers without running new experiments.
Appendix A. SW Fuzzing to HW Fuzzing
Hardware fuzzing retains software fuzzing’s generate, evaluate, and feedback loop, but adapts each stage to hardware targets, bug classes, observability, and reference-model requirements.
- Core workflow: Hardware fuzzing follows software fuzzing’s three-step pattern: input generation, evaluation, and feedback-driven refinement.The adaptations depend on target type, targeted bug class, and hardware-specific considerations.
- Input generation: Inputs may be generated through model-less, model-based, or mutation-based techniques, with formats constrained by explicit hardware specifications.Specification deviations can produce undefined behavior such as processor freezes or resets.
- Pre-processing/Instrumentation: RTL fuzzing can support instrumentation through custom coverage points, information-flow tracking, memory hashing, and hardware error handlers.The passage contrasts this feasibility with software-style instrumentation and limits it to RTL in hardware fuzzing.
- Evaluation: Hardware evaluation differs from software evaluation because bare-metal out-of-bounds accesses can remain undetected within the processor’s data-memory region.The passage states that accesses to kernel pages or I/O memory regions trigger exceptions.
- Reference models: Hardware fuzzers rely heavily on Golden Reference Models because buggy processors may raise exceptions on valid instructions.A GRM may be an ISA simulator, emulator, disassembler, another CPU, or an abstract CPU or submodule model.
- Feedback: Reliable feedback is difficult for black-box hardware targets with limited observability, whereas grey-box and white-box fuzzers can use coverage feedback.Feedback-enabled hardware fuzzers typically adopt AFL mutation strategies to guide input generation.
Appendix B. AFL Mutations used in Hardware Fuzzing
Hardware fuzzers predominantly use AFL mutation operators to generate new inputs from seed programs across bit sequences and processor instruction sequences.
- Most hardware fuzzers use American Fuzzy Lop mutation operators to generate new inputs from seed programs.The operators apply to bit-sequence seeds and RISC-V or x86 instruction sequences.
Appendix C. Comparison of Coverage Metrics
This appendix compares common coverage metrics, their use cases, and their trade-offs, noting that commercial EDA tools typically combine several metrics at substantial instrumentation cost.
- Commercial EDA tools typically combine Branch, FSM, Expression, and Line coverage metrics.These metrics can also be configured independently.
- Combining these coverage metrics incurs substantial instrumentation overhead.
- Table 8 organizes coverage metrics by definition, use case, and associated trade-offs.
Appendix D. Selecting a Hardware Fuzzer
Selecting a hardware fuzzer depends on the target type, bug class, and preferred methodology. The flowchart narrows candidate fuzzers using these choices, with Table 2 and the paper’s detailed analysis providing further differentiation.
- Fuzzer selection considers whether the target is black-box or white-box, the bug class, and the preferred fuzzing methodology.Examples include functional bugs, side channels, undisclosed instructions, tracing, differential fuzzing, directed fuzzing, and model-based relational testing.
- The selection flowchart presents a multi-step process based on target type and chosen methodology.
- For white-box differential fuzzing with ML-based mutation, the selection yields six options.Each option is annotated with input type, target type, and fuzzing algorithm.
- For IP-core verification, the available choices narrow to HFL and RLFuzz.
- Table 2 and the paper’s detailed analysis provide further differentiation among the remaining options.