PaperlayerView exact revision on arXiv

Source-linked AI summary

Lessons from the Hardware Hacking Competitions: Verification Techniques, Findings, and Insights

Sudipta Paria, Aritra Dasgupta, Raghul Saravanan, Jayanth Thangellamudi, Sai Manoj P D, Swarup Bhunia

arXiv:2608.22202v1cs.CRcs.ET

TL;DR

The paper addresses limited systematic guidance for finding and validating subtle pre-silicon SoC vulnerabilities. It combines simulation, formal verification, lint, LLM-assisted analysis, fuzzing, and exploit development in a cross-validated workflow. The study finds that these techniques provide complementary capabilities, while competitions and their artifacts support security education and reproducible evaluation of security-aware EDA tools.

  • Problem

    Existing verification methods provide limited systematic guidance for identifying and validating subtle security vulnerabilities in complex SoCs.

  • Method

    The paper applies a multi-strategy workflow combining simulation, formal verification, structural analysis, LLM-assisted reasoning, fuzzing, exploit development, and independent candidate validation.

  • Results

    The techniques provide complementary vulnerability-detection capabilities, with findings requiring explicit security intent, systematic failure analysis, and independent validation.

  • Takeaways & Limitations

    Competition designs and known vulnerabilities can serve as benchmarks for evaluating security-aware EDA tools and emerging automated analysis techniques.

Abstract

from arXiv · show

Hardware hacking competitions have emerged as practical platforms for evaluating security weaknesses in complex System-on-Chip (SoC) designs while promoting security-aware verification and tool development. This paper presents a systematic study of SoC security verification through open-box hardware hacking competitions, focusing on practical vulnerability analysis strategies, observed findings, and lessons for security-aware verification. We present a multi-strategy vulnerability analysis methodology, combining simulation-based verification, formal verification, lint analysis, Large Language Model (LLM)-assisted bug detection, and coverage-guided hybrid fuzzing. Representative vulnerability findings are analyzed to illustrate how different techniques expose complementary classes of security flaws, and we derive practical lessons for pre-silicon security verification. Finally, we discuss how competition benchmarks can support the reproducible evaluation of emerging hardware security techniques and guide future security-aware EDA research.

I. INTRODUCTION

Hardware design flaws can undermine software isolation, while conventional verification often checks intended behavior without systematically detecting malicious behavior. The paper addresses this gap through open-box competitions, practical workflows, representative findings, and lessons for complementary security verification.

  • Motivation: Hardware vulnerabilities can undermine software-level isolation, motivating security verification beyond traditional functional correctness.Examples include microarchitectural and physical exploits such as Meltdown, Spectre, and power or fault injection attacks.
  • Motivation: Existing methods can miss subtle corner-case vulnerabilities because they depend on relevant security properties, assertions, test vectors, and automated flows.This challenge becomes more pronounced in complex or large-scale designs.
  • Competition Setting: Open-box hardware hacking competitions expose designs to diverse adversarial strategies under defined threat models and time limits.They can reveal previously unknown bugs and specialized verification approaches that traditional methods may not surface.
  • Contributions: The paper formalizes systematic methods for finding, validating, and mitigating subtle pre-silicon vulnerabilities in large SoCs.It connects adversarial analysis with hardware verification and automated security-verification tool development.
  • Contributions: The proposed contributions include a practical SoC-analysis workflow, representative vulnerabilities across multiple IPs, technical verification lessons, and broader competition impacts.The impacts span security education, benchmark development, tool evaluation, and hardware-security research.
  • Paper Scope: The paper organizes its discussion around SoC assurance, competition background, analysis methodologies, lessons learned, community impact, and future directions.

A. Security Assurance for SoCs

SoC security assurance must account for adversarial behavior across integrated IP blocks, not only specified functionality. Hardware competitions provide complementary closed-box and open-box settings for studying deployed-system attacks and pre-silicon weakness detection.

  • Security Assurance: Complex SoCs combine third-party IP and in-house cores, so one access-control flaw or hardware Trojan can compromise system-wide security.Security verification therefore considers adversarial inputs, privilege violations, invalid states, and cross-component interactions.
  • Competition Models: Hardware-security competitions study vulnerabilities in processors, SoCs, firmware, and physical devices through hands-on analysis, exploitation, or mitigation tasks.
  • Closed-Box Competitions: Closed-box competitions restrict implementation access and emphasize attacks against deployed hardware through interfaces, measurements, fault injection, side channels, or reverse engineering.
  • Open-Box Competitions: The designer-centered objective extends beyond observing failures to locating root causes, determining security impact, and proposing design-level corrections.
  • Open-Box Competitions: Open-box competitions support pre-silicon detection and correction of weaknesses, complementing closed-box analysis of implemented systems under physical constraints.The distinction concerns target access, techniques, attacker perspective, validation evidence, and related features.

C. Comparing popular Hardware Security Competitions

Open-box hardware competitions differ in target abstraction and security objectives, ranging from logic-locking and cryptographic attacks to broader RTL-based SoC vulnerability analysis. Their shared artifacts also support training, benchmark evaluation, and security-aware EDA research.

  • Competition Landscape: Community competitions provide a shared platform for evaluating attacks, defenses, and analysis tools under common assumptions.Their main differences concern target abstraction and security objectives.
  • Competition Landscape: Logic-locking and obfuscation contests evaluate key recovery, functional reconstruction, and structural attacks against protected designs.
  • Competition Landscape: WhibOx uses an open-box model for cryptographic implementations, while HackTheSilicon applies complete RTL-based SoCs to a broader range of security flaws.
  • Research and Training: Competition artifacts support practical training in RTL analysis, simulation, formal verification, security-property development, exploit construction, and vulnerability reporting.
  • Research and Training: Shared vulnerable designs, known bugs, specifications, and verification infrastructure enable reproducible evaluation of vulnerability detection, false positives, and complementary verification strategies.
  • Research and Training: The paper applies multiple security-analysis strategies to a buggy OpenTitan SoC using competition infrastructure, including RTL, tests, properties, and EDA tools.

A. Analysis Workflow

The workflow establishes a baseline, generates vulnerability candidates with multiple analysis strategies, filters and cross-validates them, and documents confirmed bugs with CWE mappings, impact, and mitigations. Simulation and formal methods expose complementary security weaknesses, but observed failures require security-requirement-based validation.

  • A. Analysis Workflow: The four-phase workflow builds the environment and baseline, generates candidates, removes errors and duplicates, then reproduces and documents confirmed vulnerabilities.Confirmed findings receive a hardware CWE mapping, security impact, and possible mitigation.
  • A. Analysis Workflow: Cross-validation links methods by turning regression failures, lint warnings, or LLM-generated properties into independently checked evidence.This process reduces false positives and strengthens support for reported vulnerabilities.
  • B. Simulation-Based Security Verification: Simulation uses DVSim, VCS, regression suites, and automation to identify failing tests, unexpected outputs, assertion failures, timeouts, and abnormal state transitions.
  • B. Simulation-Based Security Verification: A regression failure becomes a vulnerability only when RTL analysis shows violation of a defined security requirement.Failures may instead result from incorrect constraints, unsupported configurations, or testbench errors.
  • B. Simulation-Based Security Verification: Simulation exposed incomplete secret wiping, incorrect FSM behavior, improperly enabled debug functionality, and delayed error handling.These findings came from smoke, stress, and security-countermeasure tests followed by RTL-level analysis.
  • C. Formal Property and Security Verification: FPV checks assertions for functional and security requirements, whereas FSV analyzes unintended information flow from sensitive sources to unauthorized destinations.
  • C. Formal Property and Security Verification: Formal analysis found incorrect security-control dependencies, register-access behavior, and an unauthorized path from AES registers to a software-visible destination.These findings illustrate complementary detection of control and confidentiality weaknesses.

D. RTL Lint and Syntax Checking

RTL lint provides fast structural coverage for security-relevant design errors, but warnings require contextual review and validation because structural analysis alone cannot establish exploitability. LLM-assisted analysis addresses a different challenge by narrowing security-focused RTL review and validating candidate findings with conventional tools.

  • D. RTL Lint and Syntax Checking: Lint analysis flags implementation patterns such as incomplete assignments, latch inference, width mismatches, uninitialized registers, and multiple signal drivers.These patterns can contribute to security vulnerabilities but are not automatically security findings.
  • D. RTL Lint and Syntax Checking: Security-relevant lint findings are prioritized in key registers, privilege checks, error logic, access-control signals, state machines, and externally visible outputs.Findings are confirmed through RTL inspection and, where possible, simulation or formal verification.
  • D. RTL Lint and Syntax Checking: Lint offers fast multi-module coverage but lacks the system-level security context needed to determine whether suspicious signal relationships are exploitable.Representative issues include incorrect error-signal usage, secret-wipe control, and missing integrity-related FSM logic.
  • D. RTL Lint and Syntax Checking: LLM-assisted analysis narrows review to relevant weakness classes and validates candidate bugs and security properties with conventional EDA tools.Direct application to large SoCs remains challenging because of design complexity, context limitations, and potentially unsupported outputs.

1) Security-Aware Design Decomposition:

The LLM-assisted workflow decomposes SoC designs around security-sensitive assets, uses relevant CWE classes to guide RTL inspection, and generates targeted properties for conventional verification. Representative findings include disabled masking, incorrect secret-wipe control, and secret-key propagation to software-visible paths.

  • 1) Security-Aware Design Decomposition:: Analysis is restricted to security-relevant modules containing assets or controls such as cryptographic keys, privilege signals, lifecycle states, debug authorization, and access controls.The decomposition follows the design hierarchy and also considers integrity checks, masking, reset controls, and error-handling signals.
  • 1) Security-Aware Design Decomposition:: Module functionality narrows the candidate CWE classes before the LLM performs targeted RTL analysis for corresponding weakness patterns.Examples include secret-data protection in cryptographic blocks and state-transition or privilege-control weaknesses in lifecycle controllers.
  • 1) Security-Aware Design Decomposition:: The workflow generates a security property after identifying a candidate weakness, targeting a specific hypothesis rather than broadly generating properties for an entire module.Generated properties are translated into equivalent SVAs, although LLM-generated assertions may contain incorrect clocking or reset behavior.
  • 1) Security-Aware Design Decomposition:: Representative LLM-assisted findings involved disabled cryptographic masking, incorrect secret-wipe control, and secret-key propagation to a software-visible register path.These findings supported targeted properties that were subsequently validated using simulation or formal verification.

F. Hybrid Fuzzing

Hybrid fuzzing combines coverage feedback with LLM-generated verification artifacts targeted at unexplored or difficult-to-activate hardware regions. Validated assertions, directed scenarios, and input constraints are incorporated into the fuzzing loop to guide subsequent exploration.

  • F. Hybrid Fuzzing: Coverage-guided fuzzing mutates input sequences and records coverage to prioritize new states, control paths, and functional regions.The objective is to expose functional errors or security-critical vulnerabilities that conventional verification may miss.
  • F. Hybrid Fuzzing: The hybrid approach first analyzes coverage data to identify regions that remain unexplored or are difficult to activate, rather than relying primarily on mutation.These uncovered points are supplied as context to the LLM.
  • F. Hybrid Fuzzing: The LLM generates SVAs, directed test scenarios, or input constraints for uncovered regions, and verification tools validate the artifacts before integration.The validated artifacts guide input generation toward previously uncovered behavior in the fuzzing framework.

G. Custom Exploit Development

Custom exploit development uses reported bug descriptions, project hierarchy, and relevant RTL and verification files to generate targeted UVM changes and checks. Simulation outputs then confirm whether the reported vulnerabilities reproduce under their triggering conditions.

  • G. Custom Exploit Development: The exploit workflow gives the LLM the bug description and OpenTitan project hierarchy to locate affected RTL and verification components.The description includes the vulnerable location, triggering conditions, expected behavior, security impact, and proposed mitigation.
  • G. Custom Exploit Development: Generated changes extend UVM sequences, add custom sequences, introduce SVAs or functional checks, and integrate with the existing verification environment.The goal is to observe the exploit outcome through simulation-side monitoring and checks.
  • G. Custom Exploit Development: UART simulation reproduced invalid lsio_trigger_o assertions when FIFO occupancies violated the specified watermark conditions.Assertion failures and accompanying FIFO-level messages identified the triggering conditions.
  • G. Custom Exploit Development: HMAC simulation reproduced the WIPE_SECRET vulnerability, with a malformed transaction triggering HmacErr and producing reported digest values.The sequence configured the module, issued the malformed write, and checked digest registers and interrupt status.

H. Bug Classes and Distribution

The observed vulnerabilities were organized into seven primary-root-cause security classes, with data sanitization failures and sensitive data exposure forming the largest fractions. Their diversity shows why complementary verification techniques are needed to expose different security-failure types.

  • Bug taxonomy: Seven security classes group the reported vulnerabilities by primary RTL root cause and security impact, without replacing individual CWE mappings.Each distinct bug was assigned once to avoid double counting across overlapping security effects.
  • Distribution: Data sanitization failures and sensitive data exposure represent the largest fractions, followed by FSM/control-flow issues and incorrect error handling.Access-control violations, integrity weaknesses, and data-path/signal-control errors account for the remaining findings.
  • Finding diversity: The findings span confidentiality leakage, improper secret clearing, lifecycle and privilege violations, faulty state transitions, weakened countermeasures, and incorrect RTL signal dependencies.This range illustrates the diversity of security failures encountered across the competitions.
  • Verification implications: Different bug classes are exposed more effectively by different forms of simulation, formal analysis, linting, and related verification techniques.The taxonomy therefore supports using complementary methods rather than relying on a single analysis approach.
  • Security intent: Security interpretation must account for protected assets, trust boundaries, privilege rules, and expected security behavior rather than treating every functional failure as a vulnerability.Functionally valid logic can still violate security requirements under unauthorized, error, or exceptional conditions.

3) Verification Infrastructure Requires Security-Aware Guidance:

Existing verification infrastructure is a useful foundation, but SoC security analysis requires security-directed guidance, independent validation, cross-layer reasoning, and automation. Functional regressions and isolated-module analysis do not adequately cover unauthorized behavior, hardware-software interactions, or security-critical states.

  • Security-aware guidance: Functional regressions mainly exercise expected behavior and may miss unauthorized access, malicious transitions, and abnormal error conditions.Existing assertions, testbenches, and lint flows should be augmented with security-directed stimulus and security-relevant coverage metrics.
  • Independent validation: Lint warnings, LLM-generated hypotheses, failed assertions, and fuzzing anomalies are vulnerability candidates rather than confirmed bugs.Candidates should be traced to RTL root causes and reproduced through simulation, formal counterexamples, information-flow paths, or custom exploits.
  • Independent validation: LLM-generated properties can be syntactically valid but semantically incorrect, so simulation or formal tools should provide the acceptance criterion.The paper identifies closed-loop AI-assisted verification as a direction for combining LLM candidate generation with EDA-based validation.
  • Cross-layer analysis: Many vulnerabilities depend on register transactions, firmware operations, privilege states, or gate-level translation and cannot be detected in isolated RTL modules.Scalable verification therefore needs cross-module analysis across abstraction levels.
  • Automation: Automation of test execution, log collection, property checking, and result triage reduces repetitive effort for large SoCs with many IPs and security-critical interfaces.A coordinated multi-method agentic LLM-driven workflow is described as providing better coverage than sequential single-technique analysis.
  • Community setting: Open-box competitions bring hardware design, verification, firmware, and security participants together under common threat models and evaluation criteria.This shared setting supports cross-domain collaboration and practical security-analysis workflows.

1) Security Training and Community Development:

Hardware hacking competitions support security training, cross-domain collaboration, benchmark reuse, and evaluation of security-aware EDA methods. Their lessons point toward integrated verification, security-oriented coverage, and benchmarks that reflect increasingly heterogeneous SoC systems.

  • 1) Security Training and Community Development:: Competitions provide hands-on experience in threat modeling, RTL analysis, simulation, formal verification, exploit development, and vulnerability reporting.Interaction among participants from different technical backgrounds also supports knowledge transfer across hardware design, verification, and security.
  • Benchmark development: Competition artifacts provide reusable benchmarks containing vulnerable RTL designs, known bug sets, security properties, test environments, and exploit cases.Hack@DAC benchmarks have been reused to evaluate static analysis, information-flow tracking, symbolic and concolic execution, formal verification, and LLM-assisted bug detection.
  • Benchmark development: The diversity of competition bugs enables security-aware EDA tools to be evaluated for detection capability, scalability, false-positive rate, and verification effort.These evaluations can reveal gaps in conventional design-automation flows and motivate tighter security-analysis integration.
  • 4) Open Challenges and Research Directions:: Scalable SoC assurance remains challenged by extracting security intent, capturing cross-IP and hardware-software interactions, and measuring adversarial exploration beyond functional coverage.Security-oriented coverage metrics are needed to measure security-critical assets, privilege transitions, and adversarial scenarios.
  • 4) Open Challenges and Research Directions:: Future work should integrate simulation, formal verification, static analysis, and fuzzing so that results from one technique guide another.LLM-assisted analysis should remain grounded in RTL semantics and validated through EDA tools, while benchmarks should expand to heterogeneous SoCs, AI accelerators, chiplets, and cross-abstraction vulnerabilities.
  • VI. CONCLUSION: The paper concludes that explicit security intent, systematic failure analysis, and independent validation are required for reliable vulnerability discovery.Released designs and known vulnerabilities also support reproducible evaluation of security-aware EDA tools and automated analysis techniques.
Loading 2608.22202v1…