Fixing Hardware Security Bugs with Large Language Models

1 INTRODUCTION

• Few tools address automated hardware-bug repair, and security flaws are especially consequential because fabricated chips cannot be patched.The paper notes that existing hardware repair work focused on functional bugs, leaving security-bug repair under-addressed.Source: §1 · p. 1
• The authors investigate whether prompted LLMs can generate repairs for hardware security bugs across different designs and styles.They contrast this approach with repair techniques based on mutation, oracle checks, or templates.Source: §1 · p. 1
• The paper curates a benchmark of hardware security bugs and corresponding designs.The benchmark is open-sourced.Source: §1 · p. 1
• The authors provide an automated framework to generate and evaluate repairs, including an end-to-end solution for certain bugs using prior static-analysis scanners.They also explore different LLMs and parameters to identify effective settings.Source: §1 · p. 2

2.1 Code Repair

• Repair approaches include mutation and templates validated by tests, learned or example-based methods, and CirFix, which localizes and repairs RTL bugs using an oracle.This paper focuses on repair and does not require an oracle.Source: §2.1 · p. 2

2.2 Bugs in Register Transfer Level design

• RTL describes hardware behavior, while hardware security bugs arise when circuit objectives such as confidentiality or integrity are unmet.Confidentiality concerns unauthorized exposure; integrity concerns unauthorized modification.Source: §2.2 · p. 2

2.3 Static Analysis

• Static analysis extracts syntax and lexical structure, often as an AST, and can support early bug detection before fabrication makes patches impossible.The paper notes that fixing issues becomes more costly after RTL is synthesized and fabricated.Source: §2.3 · p. 2

2.4 Common Weakness Enumerations

• CWEs provide a shared taxonomy for discussing and comparing vulnerabilities; cited examples include debug-mode lock overrides, uninitialized security registers, late access checks, and improper FSMs.The paper identifies CWEs that best describe the bugs in its designs.Source: §2.4 · p. 3
• CWE 1276 concerns a child block incorrectly connected to its parent in a way that can affect security attributes while preserving functionality.The passage notes that this can violate the child block's data integrity.Source: §2.4 · p. 3

2.5 Prompt Engineering

• Prompt engineering matters for LLM performance, and prior examples report gains from prompt changes in coding and text-to-image tasks.The section uses these examples to motivate prompt engineering for code repair.Source: §2.5 · p. 3

3 DESIGNS AND BUGS

• Ten hardware security bugs come from MITRE CWE examples, OpenTitan, and Hack@DAC-21 designs.MITRE examples use simple designs representing CWEs; OpenTitan bugs tweak security measures, and Hack@DAC-21 examples come from SoC competition designs.Source: §3 · p. 3, §3.1 · p. 3, §3.2 · p. 3

3.1 MITRE’s CWEs

• The MITRE examples cover a debug override of a register lock, a missing reset assignment, an access check occurring after a blocking write, and a peripheral privilege signal grounded to zero.The latter two can permit modification despite an incorrect user ID and incorrect privilege escalation of input data, respectively.Source: §3.1.1 · p. 3, §3.1.2 · p. 3, §3.1.3 · p. 3

3.2 Google’s OpenTitan

• OpenTitan bugs alter RTL security measures in different modules of a project designed to provide a silicon root of trust.Source: §3.2 · p. 3
• OpenTitan’s ROM controller incorrectly asserts an alert when start_i is high outside the Waiting state.The design interfaces the ROM and system bus and descrambles ROM contents for memory requests.Source: §3.2.1 · p. 3, §3.2 · p. 3

3.2.2 OTP Control.

• The OTP controller fails to raise an error when the life-cycle escalation signal is asserted, despite the targeted measure requiring an error-state transition.The controller supports transitions among Silicon Creator, Silicon Owner, Application Provider, and End User.Source: §3.2.2 · pp. 3–4
• The OpenTitan KMAC interface incorrectly asserts its done signal in transmission state StTx, outside the accepted done-state window.Source: §3.2.3 · p. 4

3.3 Hack@DAC-21

• A Hack@DAC-21 control-and-status-register design lets debug override interrupt signals when the core should stall.Source: §3.3.1 · p. 4, §3.3 · p. 4

3.3.2 DMA.

• The Hack@DAC-21 DMA’s PMP register lacks a reset value, while its AES interface has too few case branches and no default statement.The DMA uses memory addresses and PMP configuration to govern read or write operations.Source: §3.3.2 · pp. 4–5, §3.3.3 · p. 5

4.1.1 Repair Generator.

• For MITRE and OpenTitan, the bug location is assumed known; for Hack@DAC-21, a detector supplies the location and relevant CWE.Source: §4.1.1 · p. 5
• The prompt combines surrounding code, the buggy code in comments, and bug and fix instructions; instruction detail varies across experiments.The LLM receives this prompt and generates repairs, which may be correct or incorrect.Source: §4.1.1 · p. 5

4.1 LLM-based Repair Evaluation Framework

• The framework evaluates LLM-generated repairs by checking both functionality and security; a repair is successful only if it passes both.Source: §4.1.2 · p. 5, §4.1 · p. 5, §4 · p. 5

4.1.2 Evaluator.

• Functional testbenches cover all intended functionality for MITRE designs but only bug-related partial functionality for OpenTitan and Hack@DAC designs.OpenTitan and Hack@DAC simulation also requires assembling the Device Under Test from instantiated and relevant files.Source: §4.1.2 · p. 6
• Security checks use custom testbenches for MITRE and OpenTitan and CWEAT for Hack@DAC; OpenTitan checks cover specific introduced-bug functionality rather than the full SoC.Complete verification is difficult because it requires software-driven simulation of the entire SoC; the OpenTitan team’s method remains a work in progress.Source: §4.1.2 · p. 6
• Functional and security verification can overlap, and for Bug 3 the functional check is a subset of security evaluation.For CWE 1271, security checks require both a reset value for the sensitive register and the correct lock mechanism, which also supports functionality.Source: §4.1.2 · p. 6

• The demonstrated end-to-end Verilog framework connects bug detection, repair generation, and evaluation for some CWEs.Source: §4.2 · p. 6
• The Hack@DAC-21 detector traverses Verific-parser ASTs using keywords and patterns to flag potential CWEs 1234, 1271, and 1245.The authors selected three SoC instances, one per CWE, and reuse the detector for security evaluation; a repeated location and CWE means a repair is not successful.Source: §4.2 · p. 6
• The envisioned workflow lets RTL designers use CWEAT to flag weaknesses, obtain LLM repairs, and receive secure repairs as suggestions during early design.Source: §4.2 · p. 6

4.3 Experimental Parameters

• The experiments vary instructions, temperature, and model while holding top_p, completion count, and maximum tokens constant.Source: §4.3 · p. 6

4.3.1 Instruction Variation .

• Five instruction variants range from no assistance to bug descriptions, natural-language or pseudocode-like fix guidance, and a generalized bug-and-fix example.Each variant has a bug instruction and a fix instruction; the example is added in variation e.Source: §4.3.1 · p. 7, Table 3 · p. 7

4.3.2 Temperature (t).

• Temperature tests use t values of 0.1, 0.3, 0.5, 0.7, and 0.9; higher values mean more risk-taking and creative completions.Source: §4.3.2 · p. 7

4.3.3 Models.

• Four evaluated models comprise three OpenAI Codex variants and CodeGen-16B-multi, called CodeGen in the paper.The Codex variants are code-davinci-001, code-davinci-002, and code-cushman-001.Source: §4.3.3 · p. 7
• Prompts include 25–50 lines of preceding code, extending to the start of the containing block when more than 25 lines precede the bug.Source: §4.3.4 · p. 8

4.3.5 Stop keywords.

• The default stop keyword is endmodule, with additional keywords used for particular cases.Source: §4.3.5 · p. 8, Table 4 · p. 8

5 EXPERIMENTAL RESULTS

• Twenty responses per bug, temperature, and instruction combination were tested for functional and security correctness; Figures 7–8 present repair counts as heatmaps.Figure 7’s maximum per cell is 20, and Figure 8’s maximum per cell is 200.Source: §5 · p. 9, Figure 7 · p. 10, Figure 8 · p. 11

5.1 RQ1: Can LLMs fix hardware security bugs?

• Across 20,000 requested repairs, 6,376 were correct (31.9%); three Codex models repaired every dataset bug at least once, while CodeGen repaired 7 of 10.The best-observed success rates were 71% for code-davinci-001, 70% for code-davinci-002, 58% for code-cushman-001, and 12% for CodeGen.Source: §5.1 · p. 9
• With per-bug best-observed settings, Bug 7 reached an 85% success rate and every other bug reached 100%.Table 5 lists the selected model, instruction variation, and temperature for each bug.Source: §5.1 · p. 9, Table 5 · p. 9

5.2 RQ2: What bugs are amenable to repair?

• Bugs 3 and 4 were most amenable, with success rates above 50%; Bugs 2, 6, 7, and 9 were hardest, each below 25%.The authors link Bugs 3 and 4 to clear signal names, while difficult fixes included adding reset behavior and removing a line without replacement.Source: §5.2 · p. 9, Figure 9 · p. 11

5.3 RQ3: How should prompts be engineered to assist the repair of hardware bugs?

• More detailed prompts generally helped all models except CodeGen; variation d performed best among OpenAI models, while code-davinci-002 and CodeGen did best with variation e.Variations c–e include a fix instruction after the buggy code; variation d uses pseudo-code-style fix instructions.Source: §5.3 · p. 9, Figure 10 · p. 11

5.4 RQ4: Does the temperature matter?

• OpenAI models performed best at temperature 0.1, while CodeGen performed best at 0.3; lower temperatures also produced less varied responses.The authors report that the less creative responses were more likely to be correct repairs.Source: §5.4 · p. 9, Figure 10 · p. 11

5.5 RQ5: Are some LLMs better than others?

• code-davinci-002 led with 2,371 correct repairs out of 5,000 (47.4%); CodeGen had the lowest success rate at 6.68%.The reported rates were 40.4% for code-davinci-001 and 33.1% for code-cushman-001; the authors attribute CodeGen’s gap to its smaller size.Source: §5.5 · pp. 9, 11

5.6 Comparison with CirFix

• Variation a yielded 17.5 correct repairs versus CirFix’s 16; variation b fixed 20 of 32 benchmarks, and both variations together repaired 23.5.The LLM comparison generated one repair per attempt and manually evaluated it for correctness.Source: §5.6 · p. 11

6 DISCUSSION AND LIMITATIONS

• Instruction variations are subjective, and the authors’ experience shapes Fix instructions; more varieties may be needed to standardize their effects.Instructions are also difficult to generalize across bugs because instance-specific information can improve the probability of a successful fix.Source: §6 · pp. 11–12
• Table 6 compares repairs on CirFix benchmarks using two instruction variations and marks successful repairs with “y.”Its caption also defines “- | y” and “1/2” notation; it does not provide the table’s cell outcomes.Source: Table 6 · p. 12
• Security evaluation is limited to the particular bug that makes a design insecure, while functional evaluation is not exhaustive for larger designs.The authors used custom testbenches for code a repair could impact because exhaustive testing of complex SoCs would take substantial time.Source: §6 · p. 12
• End-token choices are subjective: stopping too early can truncate a repair, while not stopping appropriately can add code that causes syntax errors or affects functionality.A post-processing script adjusts begin/end counts before the repair replaces the buggy code.Source: §6 · p. 12
• Twenty repair responses per request are generated in under a minute, though API token limits raised total generation time to about 20 minutes per LLM.The authors describe the designer as choosing the best suggestion from the quickly generated options.Source: §6 · p. 12
• More examples of hardware security bugs are needed before repair techniques can support more conclusive claims.The authors describe their work as a step toward building such a benchmark corpus.Source: §6 · p. 12

7 CONCLUSIONS AND FUTURE WORK

• All ten corpus bugs had at least one successful repair, and nine had 100% correct responses with the best parameter set.The authors report that repairs were harder when they spanned more than one line or required removing a buggy line.Source: §7 · pp. 12–13
• Detailed instructions tended to improve success rates, variation d with pseudo-code-like Fix instructions performed best, and lower temperatures and larger LLMs performed better.The authors also report better performance on function-related Verilog bugs than CirFix’s program repair mechanism.Source: §7 · p. 13
• Future work includes hybrid detection and repair, HDL fine-tuning, broader parameter sweeps for functional bugs, and a consolidated database of security-related hardware bugs.The proposed hybrid approach combines detection tools with LLMs and oracle-guided modifying algorithms for repair.Source: §7 · p. 13

Compute environment

• Experiments ran on an Intel Core i5-10400T CPU at 2 GHz with 12 threads, 16 GB RAM, and Ubuntu 20.04.5 LTS.Source: Compute environment · p. 14

Open source details

• Verific libraries were used under an academic license, and the authors direct readers to Verific for access.Source: Open source details · p. 14
• For CWEAT, CirFix, and the Hack@DAC SoC, the authors direct readers to the respective authors or organizers for access or assistance.The CirFix benchmarks and results came from the authors’ open-source GitHub repository.Source: Open source details · p. 14