SoK: You Find What You Seek: Rethinking Oracles, Guidance, and Input Generation in Hardware Fuzzing

• G Abarajithan and Andres Meza are listed with UC San Diego.Source: G Abarajithan UC San Diego · p. 1

1 Introduction

• Hardware fuzzing must account for parallel execution and bugs that rarely crash RTL simulation, so software oracles, guidance, and input generation do not always transfer.The paper notes that useful inputs also depend on whether the target is an IP, CPU, or SoC.Source: §1 · p. 1
• The SoK analyzes 52 hardware fuzzers across RTL/IP, CPU, NoC, and SoC targets to assess verification complementarity, campaign reach and recognition, and deployment evidence.Source: §1 · p. 1
• The framework treats hardware fuzzing as bounded search, separating failure recognition by the oracle from input reachability and guidance-based prioritization under a finite budget.Source: §1 · p. 1
• The analysis identifies a mismatch between commonly used feedback and the functional and security behaviors encoded in verification plans and threat models.Source: §1 · p. 2
• The paper compares evaluation and deployment requirements, including consistent metrics, versioned DUTs, documented bug corpora, shared infrastructure, component ablations, and open artifacts.Source: §1 · p. 2

2 The Framework: You Find What You Seek

• Fuzzing is guided verification: observations from an execution influence how inputs are generated for future executions toward a stated verification objective.The same DUT may be tested against different objectives, which drive the campaign's other choices.Source: §2 · p. 2

2.1 Core Components of a Fuzzer

• The oracle determines which failures can be recognized by checking whether observed behavior violates the verification objective.Some checks use externally visible outputs, while others inspect internal state or multi-cycle event sequences.Source: §2.1 · p. 2
• Guidance turns execution observations into coverage or a score that prioritizes tests, but its lossy feedback must preserve distinctions relevant to the objective.Feedback must remain inexpensive enough to collect throughout the campaign.Source: §2.1 · p. 2
• Input representation, seeds, mutation, and harness determine which tests can be expressed and practically reached while preserving required timing and protocol interactions.A generator cannot reach a triggering action that the input language or harness cannot express.Source: §2.1 · p. 2

2.2 Target Abstraction and Budget

• Target abstraction constrains visibility, controllability, and execution cost, while a finite budget forces tradeoffs among feedback expense, test generation, and campaign resources.Time, compute, memory, and human effort can all contribute to the budget.Source: §2.2 · pp. 2–3

3 Background

• The framework is applied to software fuzzing, a generalized hardware-fuzzing loop, CRV, and property-driven security verification.Source: §3 · p. 3
• Hardware verification includes static methods and dynamic execution of concrete tests; CRV generates inputs randomly within constraints, while fuzzing uses prior execution observations to guide later inputs.The paper focuses on dynamic campaigns and includes static analysis when it guides hardware fuzzing.Source: §3 · p. 3

3.1 From Software to Hardware Fuzzing

• Software-fuzzing techniques do not transfer directly because hardware bugs may span signals and cycles and generally do not crash the simulator.Hardware adaptation affects the harness, guidance, and explicit design- or security-specific oracle checks.Source: §3.1 · p. 3

3.2 Anatomy of a Hardware Fuzzer

• A hardware-fuzzer loop selects retained inputs, generates and executes candidates, measures progress, saves failures, and retains interesting inputs for further exploration.The harness maps candidates to executable stimulus; feedback guides exploration, while the oracle independently checks for failures.Source: §3.2 · p. 3

3.3 CRV vs. Hardware Fuzzing

• Hardware security verification combines functional plans with threat-model requirements because functional bugs can become vulnerabilities, while functional correctness alone does not establish adversarial resistance.Source: §3.3 · p. 4
• CRV uses constrained, structured requests and functional coverage to measure completeness, with a reference model, scoreboard, and assertions detecting failures.Engineers inspect coverage gaps and revise campaigns; fuzzing instead uses earlier executions to guide later inputs.Source: §3.3 · p. 4

4.1 Oracle Mechanisms and Blind Spots

• Architectural reference models may miss microarchitectural or information-flow violations that do not appear in architectural state.Specialized models, execution comparisons, and explicit properties can represent behavior beyond architectural state.Source: §4.1 · p. 5
• The survey groups oracles into architectural reference models, specialized models, differential or relational comparisons, and explicitly specified properties and verification infrastructure.These families can be combined in one oracle.Source: §4.1 · p. 5
• Assertions and protocol monitors check rules, timeouts detect hangs, and scoreboards, reference models, sanitizers, and hyperproperties check other behaviors.Hyperproperties can express confidentiality, integrity, and noninterference requirements.Source: §4.1 · p. 5
• Oracle scope should match the target: an IP may use local checks, a CPU architectural differential testing, and an SoC properties spanning firmware and system components.A local assertion does not establish an SoC-wide security invariant.Source: §4.1 · p. 5

4.2 Failure Validation and Triage

• An oracle report identifies a potential failure, not necessarily a confirmed bug or root cause; reports must be reproduced, minimized, grouped, and validated.Continued campaigns may produce many reports, including duplicate manifestations of one bug.Source: §4.2 · p. 5
• Validated failures can be prioritized by functionality, security, safety, repair scope, frequency, reproducibility, and analysis effort.Automated input reduction can preserve bug-triggering state while making long programs easier to analyze.Source: §4.2 · p. 5

5 Guidance

• Coverage is a lossy proxy, not a verification result; NoCFuzzer found CRV closed an easy router covergroup faster, while fuzzing reached harder code and functional targets within the budget.Guidance helps only when reduced search effort offsets feedback cost.Source: §5 · p. 6

5.1 Source and Types of Feedback

• Implementation-derived coverage is common and needs less target-specific effort than functional or security feedback, but new logic activity does not establish meaningful progress.Four mux select pins can reach 100% mux-toggle coverage in one short trace despite up to 24×10 ≈1 trillion traces across ten cycles; the signal preserves neither event order nor protocol context.Source: §5.1 · p. 6
• Fuzzers use coverage over observed behavior or numeric scores such as target distance, security cost, asset influence, and contention interval; both remain lossy.Coverage and scores can be combined, with coverage filtering candidates before a more expensive score is computed.Source: §5.1 · p. 7

5.2 Target Selection

• Directed guidance requires a verification goal, a selected target signal or region, and an input generator that can activate it.Security targets generally originate in a threat model or property; automatically selected structural targets remain structural unless connected to one.Source: §5.2 · p. 7

5.3 Alignment Across the Guidance Chain

• Guidance forms a chain from objective to target behavior, observable values, feedback, and search policy, with each mapping able to discard information.Structural feedback assumes new implementation activity correlates with the objective, while functional coverpoints and security-derived scores begin from verification behavior or properties.Source: §5.3 · pp. 7–8
• Feedback is not evidence that a design is bug-free; guidance should be designed backward from the verification objective through target, observation, feedback, and search policy.Source: §5.3 · p. 8

6 Input Generation

• Input generation determines which tests a fuzzer can produce and includes candidate representation, seeds, generation operations, effort allocation, and the DUT harness.A bug whose triggering input is never generated remains undetected.Source: §6 · p. 8

6.1 Input Grammar and Harness

• The grammar defines how the DUT interprets an input, and the harness applies it; the generator must balance validity with flexibility.Without concurrent crossbar requests, a transaction generator cannot expose contention failures; without interrupts, a CPU generator cannot expose an interrupt-dependent bug.Source: §6.1 · p. 8
• Harnesses vary by target: IP inputs may map to pins or transaction fields, CPUs need programs and execution state, and SoCs or NoCs require coordinated, longer executions.Firmware alone may not express malicious NoC transactions or memory faults, so the grammar must include threat-model entry points.Source: §6.1 · p. 8

6.2 Candidate Generation Mechanisms

• Surveyed fuzzers mutate parents, construct inputs without parents, solve backward from targets, or combine these mechanisms; a complete generator may not transfer across differing grammars.Source: §6.2 · p. 8

6.2.1 Mutation and Construction

• Mutation changes parent inputs, while parent-free construction can randomize vectors or packet fields and later use solver constraints when coverage stagnates.Constructed or solved inputs can subsequently become mutation parents.Source: §6.2.1 · p. 8

6.2.2 Construction using Language Models

• Language-model CPU fuzzers construct programs and refine generation using execution feedback, but learned program grammars do not derive tests from a threat model.GenHuzz resets its model when coverage stalls, while GoldenFuzz retains older blocks in recency-weighted memory to address overfitting toward rewarded patterns.Source: §6.2.2 · pp. 8–9

6.2.3 Solving from Target to Input

• After target selection, mutation offers cheap but indirect activation, whereas target-to-input solving is more direct but can be resource-intensive.Selective solver assistance may justify its overhead for slow RTL simulation; evaluations should report solving and execution costs separately.Source: §6.2.3 · p. 9

6.3 Search Effort: Seeds and Scheduling

• Seeds, parent selection, mutation energy, and operator scheduling shape reachability; fuzzer performance is highly sensitive to initial seeds.Evaluations should ablate these contributions and report verification artifacts, cost, and human effort.Source: §6.3 · p. 9

7.1 Comparability between Fuzzers

• Comparisons should align the verification objective, target abstraction, DUT revision, oracle, and budget before attributing differences to guidance or input generation.Different oracles can recognize different failure sets, so their specifications, observations, and blind spots should be reported.Source: §7 · p. 10, §7.1 · p. 10
• Change one guidance or input-generation choice at a time, control corpus and seeds, and report repeated-trial distributions and uncertainty.Fuzzer-generated tests should be replayed with the same independent coverage tool and configuration when coverage closure is the objective.Source: §7.1 · p. 10

7.2 Comparison with CRV

• Coverage-maximizing fuzzers should be compared directly with modern CRV, not only uniform-random generation or disabled guidance.Give both approaches equal time and compute, then compare coverage and bugs found; report the targets and conditions under which either performs better.Source: §7.2 · p. 10
• Replay fuzzer inputs through the CRV environment to measure test quality with its functional coverage model and oracle.Known-bug comparisons should use the same faulty revision or injected bugs and report how many are found and detection-time distributions.Source: §7.2 · pp. 10–11

7.3 Proposed Evaluation Practices

• Report DUT commit and configuration, objective, abstraction, oracle, corpus, budgets, tools, measurement conditions, repeated-trial distributions, coverage, time to failure, resources, and user effort.Claims of generality require varied representative objectives and target abstractions.Source: §7.3 · p. 11
• Benchmark families should package versioned DUTs, harnesses, corpora, oracles, coverage models, faulty and fixed revisions, injected bugs, and replay scripts.Curated assertions, reference models, covergroups, and known bugs are the practical alternative while automatic property generation remains an open problem; building a suite requires substantial community effort, and no initial suite will cover every objective.Source: §7.3 · pp. 11–12

8.1 Required Artifacts

• Required design representations range from high-level hardware descriptions and RTL to synthesized netlists, standard-cell libraries, ATPG configurations, or complete UVM environments.Harnesses and seed corpora also vary by abstraction: processor fuzzers use program-aware inputs, while SoC, interconnect, and RTL-IP fuzzers rely on target-specific models and harnesses.Source: §8.1 · p. 12
• Directed adversarial testing often requires manual encoding of threat models and security specifications, while newer ML-based approaches may also need datasets, state representations, rewards, and configuration.The domain-specific information required by a methodology increasingly determines user effort.Source: §8.1 · p. 12

8.2 Infrastructure and Design Visibility

• Many surveyed CPU, SoC, and NoC fuzzers depend on proprietary verification tools; open-source simulators often still need custom co-simulation, and academic tools commonly require adaptation to custom environments.Some methods also need specialized hardware or substantial compute beyond conventional UVM infrastructure; grey- and white-box visibility use internal design information, whereas black-box approaches provide less internal observability.Source: §8.2 · pp. 12–13
• Reproducing a campaign may require the exact DUT and configuration, input artifacts, tool versions or licenses, verification infrastructure, and specialized hardware.Unavailable components can make independent reproduction difficult even when the design is public; the paper recommends stable interfaces between reusable fuzzer engines and target-specific verification assets, with those assets released.Source: §8.2 · p. 13

9 Summary and Conclusion

• The framework treats hardware verification as bounded search shaped by the objective, budget, target abstraction, oracle, guidance, and input generator.It was developed through analysis of 52 hardware fuzzers.Source: §9 · p. 13
• A campaign finds only failures its oracle can detect, guidance can prioritize, and input generator can produce within its budget.Improving one component cannot remove limits imposed by the others; quality guidance must preserve distinctions relevant to the verification objective.Source: §9 · p. 13
• Benchmarks should span IP, CPU, NoC, and SoC designs and include versioned DUTs, verification collateral, bugs, replay scripts, metrics, budgets, and procedures.Coverage-maximizing fuzzers should be compared with CRV in the same environment, while adversarial campaigns should be compared against alternatives with a comparable objective, threat model, and budget.Source: §9 · p. 13
• Mainstream adoption requires reusable tools integrated through stable interfaces and reproducible campaigns reporting effectiveness, cost, and required user effort.Tools should support continuous fuzzing, failure triage, and reliable replay across platforms.Source: §9 · p. 13

A Open Science

• The paper is a Systematization of Knowledge based on an analysis of publicly available literature.The analyzed works are identified in the bibliography.Source: A Open Science · p. 14
• The paper reports no new experiments or empirical measurements requiring source code, datasets, binaries, or other experimental artifacts.Source: A Open Science · p. 14
• The works analyzed in the paper are identified in the bibliography.Source: A Open Science · p. 14