Source-linked AI summary
An Executable Benchmark for LLM-Based HLS Repair:Design Complexity and Repair Underconstraint
Maisha Mastora, Dean Sullivan
TL;DR
HLS logic repair has limited systematic evidence, and prior work does not explain why repairs fail or which design properties drive difficulty. This paper evaluates LLM repair on an executable benchmark with suite-specific functional oracles. Repair rates vary sharply with design context and scale, while solution multiplicity predicts repair failure across models.
Problem
Prior systematic HLS logic-repair evidence was limited, with no analysis of why repair fails or which design-instance properties drive success or failure.
Source: §I · p. 1Method
The paper builds a validated executable HLS repair benchmark with suite-specific functional oracles and evaluates repair patches by whether they restore oracle-passing behavior.
Source: A. HLS Benchmark Construction · p. 3, §VI · p. 7Results
Repair rates range from 6–45% on CHStone’s large cryptographic kernels to 81–93% on MachSuite’s compact algorithmic kernels despite identical bug-type distributions.
Source: §I · p. 2Takeaways & Limitations
Solution multiplicity provides an LLM-native signal of repair difficulty, and complex cryptographic designs remain substantially unrepaired while frontier models largely solve simple kernels.
Source: §VI · p. 7, §VII · p. 8The evaluation measures C-level functional repair only; whether patches remain correct after HLS synthesis is outside the benchmark’s scope.
Source: §VI · p. 8
Abstract
from arXiv · showhide
Automated repair of High-Level Synthesis (HLS) designs using large language models (LLMs) is an emerging but underexplored problem. While LLM-based repair shows strong results on register-transfer level (RTL) Verilog, the only prior systematic study of HLS logic repair reports just 10.5% correction accuracy for GPT-4, with no analysis of why repair fails or what drives difficulty. We present the first comprehensive evaluation of LLM-based HLS repair across four models (GPT-4o, GPT-4o-mini, GPT-5.4, and Claude Opus 4.6) on 125 benchmark instances spanning eight logic bug types across three open-source HLS suites (CHStone, MachSuite, Polybench). We construct the first executable APR-style HLS repair benchmark with suite-specific functional oracles, enabling pass@k evaluation rather than the string-match approximations used in prior work. Design context and scale, rather than bug type alone, dominate repair difficulty: repair rates range from 6-45% on complex cryptographic kernels (CHStone) to 81-93% on compact algorithmic kernels (MachSuite) despite identical bug type distributions. We introduce solution multiplicity, the fraction of distinct patches generated across repeated repair attempts, as an empirical measure of repair underconstraint, and show it strongly predicts repair failure across all four models (Spearman rho = -0.393, p = 6.61 x 10^-20). Frontier models reduce underconstrained instances from 41-46 (GPT-4o, GPT-4o-mini) to just 3 (Claude Opus 4.6), with pass@1 improving from 42-53% to 73-76%. SHFT bugs remain consistently hard across all models, and semantic bug classes such as buffer indexing become reliably repairable only at frontier scale. These findings show that future APR benchmarks must include complex, executable, and weakly identifiable designs that remain challenging after frontier LLM repair.
I. INTRODUCTION
HLS repair remains poorly understood despite stronger LLM results on RTL, and prior work did not explain why HLS repairs fail or which design properties drive difficulty. This paper builds an executable benchmark and finds that design context and scale, alongside repair underconstraint, shape outcomes.
I. INTRODUCTION
- Prior HLS repair evidence was limited to GPT-4’s 10.5% correction accuracy, without analysis of why repair is difficult or what drives success and failure.The paper also asks whether existing HLS benchmark families remain difficult enough for next-generation repair engines.Source: §I · p. 1
- The benchmark contains 125 validated instances across eight C-logic bug types and three HLS suites, with functional oracles enabling pass@k rather than string-match scoring.The prior study [6] compared patches with ground-truth snippets, which cannot distinguish plausible from functionally correct repairs.Source: §I · p. 1
- Repair rates range from 6–45% on CHStone’s large cryptographic kernels to 81–93% on MachSuite’s compact algorithmic kernels despite identical bug-type distributions.The pattern holds across all four evaluated models.Source: §I · p. 2
- Solution multiplicity—the fraction of distinct patches across repeated attempts—strongly predicts repair failure across all four models (Spearman ρ = −0.393, p = 6.61 × 10−20).Underconstrained instances fall from 41–46 for GPT-4o and GPT-4o-mini to 3 for Claude Opus 4.6.Source: §I · p. 2
II. BACKGROUND AND RELATED WORK
LLM-based repair has developed substantially at the RTL level, but systematic HLS logic-repair evaluation remains limited. Prior HLS work uses string matching and does not investigate structural drivers of repair difficulty.
A. Automated RTL Repair
- RTL repair research spans search-based, symbolic, and LLM methods, with reported LLM gains from localization and semantic partitioning.LiK [4] reports pass@1 rising from 40.39% to 58.92%, while ARSP [3] reports pass@1 of 77.92% and pass@5 of 83.88%.Source: A. Automated RTL Repair · p. 2
B. LLM-Based HLS Debugging and Repair
- No prior work had systematically evaluated LLM repair of HLS logic bugs with functional oracles, and existing hardware repair benchmarks contained no HLS instances [13].Earlier HLS assistance focused mainly on code generation or synthesizability, while HLSDebugger [6] evaluated logic repair by string matching and did not study structural difficulty.Source: A. Automated RTL Repair · p. 2, B. LLM-Based HLS Debugging and Repair · p. 2
III. MOTIVATION
The paper frames HLS repair difficulty partly as an identifiability problem: test observations may fail to uniquely constrain a correct patch. It links underconstraint to oracle coverage, algorithmic opacity, and large source contexts.
A. Repair Identifiability and Underconstraint
- Solution multiplicity is introduced as an empirical measure of identifiability, computed from LLM patch diversity without formal specification analysis or ground-truth models.Underconstraint means multiple structurally distinct patches satisfy the available oracle; an identifiable instance has observations that uniquely constrain the correct patch.Source: A. Repair Identifiability and… · p. 2, B. Repair Identifiability in HLS · p. 3
B. Repair Identifiability in HLS
- Seventy-five candidate instances were discarded because single-vector oracles could not distinguish buggy from correct behavior, including AES tests that omit 192- or 256-bit key paths.The authors characterize single-vector coverage as a structural source of underconstraint in HLS benchmark suites.Source: B. Repair Identifiability in HLS · p. 3
- Complex HLS algorithms leave many syntactically valid edits that satisfy the oracle but implement incorrect hardware behavior, widening the plausible-but-incorrect patch space.HLS source omits explicit timing, wire assignments, and signal topology, so hardware intent must be inferred from C source.Source: B. Repair Identifiability in HLS · p. 3
- CHStone’s ∼40 KB entry-point context achieves 6–45% pass@1 versus 81–93% for MachSuite’s ∼3 KB context despite identical bug-type distributions.Entry-point compilation aggregates helper modules into the context, while the bug site may occupy only a few lines.Source: B. Repair Identifiability in HLS · p. 3
- Large entry-point programs, difficult-to-infer cryptographic kernels, and weak single-vector harnesses compound in CHStone’s hardest instances.Solution multiplicity captures their combined effect through patch diversity across repeated attempts.Source: B. Repair Identifiability in HLS · p. 3
IV. EXPERIMENTAL SETUP
The experimental setup pairs Chrysalis-HLS bug specifications with three open-source HLS suites and validates candidate correct and buggy programs against executable oracles. The resulting benchmark contains 125 instances across eight bug types.
A. HLS Benchmark Construction
- The benchmark construction yields 125 instances across eight C-logic bug types from CHStone, MachSuite, and Polybench, using single-bug Chrysalis-HLS specifications.The workflow pairs suites with oracles, validates bug injection, and carries validated instances into zero-shot LLM repair and evaluation.Source: A. HLS Benchmark Construction · p. 3, §IV · p. 3, Figure 1 · p. 4
- Each retained instance has a correct version that passes its oracle and an injected buggy version that fails it, making a passing repair a functional correction under that suite’s test.Candidates failing either side of this two-sided validation are discarded.Source: A. HLS Benchmark Construction · p. 3
- CHStone uses self-checking programs, MachSuite uses harness reference data, and Polybench uses differential output against a saved reference.The final suite has 41 CHStone, 40 MachSuite, and 44 Polybench instances; BUF and INIT are underrepresented because single test vectors often miss their bug sites.Source: A. HLS Benchmark Construction · p. 3
B. LLM Repair Pipeline
The repair pipeline uses a uniform zero-shot prompt across four models, with five independent attempts per instance. Metrics capture repair success, patch diversity, and failure modes.
B. LLM Repair Pipeline
- Four models receive the same minimal zero-shot prompt and full buggy source, with five independent repair attempts per instance.The generation-4 models are GPT-4o and GPT-4o-mini; GPT-5.4 and Claude Opus 4.6 represent the generation-5 frontier.Source: B. LLM Repair Pipeline · p. 4
C. Metrics
- pass@1 is the primary metric, while pass@5 gives an upper bound on success with the five-run sample.The estimator uses n = 5 total runs and c passing runs.Source: C. Metrics · p. 4
- Solution multiplicity is the fraction of distinct patches across five runs; values below 0.5 indicate convergence on a small set of repairs.Multiplicity depends on the model and decoding, so it measures repair behavior rather than an intrinsic instance property.Source: C. Metrics · pp. 4–5
- Non-passing patches are classified as compile failures, test failures, or timeouts for non-terminating INF-type bugs.Source: C. Metrics · p. 5
A. Overall Repair Performance
Repair success rises across model generations, but design scale strongly conditions difficulty: suites with identical bug-type distributions have markedly different repair rates. Within CHStone, larger target files are also harder to repair.
A. Overall Repair Performance
- Pass@1 rises from 42.4% for GPT-4o-mini to 75.7% for Opus 4.6 across all 125 instances, with GPT-4o at 53.3% and GPT-5.4 at 73.4%.The weaker models also generate more diverse patches, while Opus 4.6 often converges to the same patch.Source: A. Overall Repair Performance · p. 5
B. Design Complexity Dominates Repair Difficulty
- CHStone pass@1 ranges from 13.7–44.9%, versus 48.5–92.5% for MachSuite and 56.8–90.9% for Polybench, despite identical bug-type distributions.The MachSuite–CHStone gap ranges from 27.5 percentage points for Opus 4.6 to 60.5 points for GPT-4o; suite complexity dominates bug type as a difficulty predictor.Source: B. Design Complexity Dominates Repair… · p. 5
- Within CHStone, target-file LOC correlates negatively with pass@1 (ρ = −0.389, p = 0.012); MachSuite and Polybench trends are weaker and non-significant.The narrower file-size ranges in those suites may account for the non-significant correlations.Source: B. Design Complexity Dominates Repair… · p. 5, Figure 2 · p. 5
C. Solution Multiplicity and Underconstraint
Solution multiplicity is strongly associated with repair failure and distinguishes diverse unsuccessful guesses from repeated incorrect patches. Bug-category difficulty also varies: SHFT remains hard, while buffer indexing improves at frontier scale.
C. Solution Multiplicity and Underconstraint
- Across all 125 instances, solution multiplicity is negatively correlated with pass@1 (Spearman ρ = −0.393, p = 6.61 × 10−20).Instances with a different patch on every run are overwhelmingly failures, while convergence on one or two patches is associated with passes.Source: C. Solution Multiplicity and… · p. 5
- Underconstrained instances are defined by multiplicity ≥0.8 and pass@5 = 0; frontier models reduce their count to 3, while older models have 35–46.The remaining Opus 4.6 cases involve large cryptographic files or complex loop structures with oracles covering only some code paths.Source: C. Solution Multiplicity and… · p. 5, Table IV · p. 6
- Multiplicity distinguishes confident-but-incorrect convergence on BUF from high-diversity underconstraint on SHFT.For GPT-4o, BUF has multiplicity 0.800 and 0.0% pass@1, while SHFT has multiplicity 1.000 and 5.0% pass@1.Source: C. Solution Multiplicity and… · pp. 5–6
D. Bug-Category Analysis
- Model scaling improves most bug types, but SHFT remains the hardest and buffer-indexing bugs improve only with frontier models.INF is consistently easiest; SHFT reaches only 62.5% for Opus 4.6, while GPT-4o repairs none of the three BUF instances and generation-5 models repair most.Source: D. Bug-Category Analysis · p. 6, Figure 3 · p. 6
- Repair rates vary across suites for the same bug type; GPT-4o shows a 3.5% USE gain but 0.0% BUF gain at any k, highlighting context-dependent differences.The correct shift value can occasionally be sampled but is not reliably produced.Source: Figure 4 · p. 6, D. Bug-Category Analysis · p. 6
E. Structural Metrics and Solution Multiplicity
Target-file size is a stronger repair-difficulty predictor than cyclomatic complexity, while solution multiplicity captures a complementary dimension. The syntactic-versus-semantic distinction narrows sharply for frontier models.
E. Structural Metrics and Solution Multiplicity
- LOC is the strongest predictor of repair difficulty (ρ = −0.564, p < 0.001), while CCN has a non-significant positive correlation (ρ = +0.174, p = 0.052).CCN is near-uniform across the three suites, so CHStone files are large but not structurally more branchy.Source: E. Structural Metrics and Solution… · p. 6
- Solution multiplicity correlates with repair difficulty (ρ = −0.393, p < 0.001), capturing a dimension that CCN does not.LOC and multiplicity capture complementary dimensions: structural scale and semantic underconstraint.Source: E. Structural Metrics and Solution… · p. 6
- The syntactic–semantic pass@1 gap narrows from 0.47 vs. 0.34 for GPT-4o-mini and 0.61 vs. 0.37 for GPT-4o to 0.73 vs. 0.69 for GPT-5.4 and 0.75 vs. 0.72 for Opus 4.6.The difference is not statistically significant (Mann-Whitney U, p = 0.159).Source: E. Structural Metrics and Solution… · pp. 6–7
VI. ANALYSIS AND DISCUSSION
Repair difficulty varies sharply with design context, and compact HLS suites can be saturated while larger CHStone designs remain challenging. The discussion argues for executable, discriminating benchmarks and identifies oracle coverage, repair strategy, and synthesis validation as important boundaries.
Benchmark saturation and the need for harder APR tasks
- GPT-4o repair rates span 13.7–25.9% on CHStone and 74–92.5% on MachSuite, while frontier models saturate MachSuite and Polybench above 90%.A single HLS suite can therefore give a misleading estimate of repair difficulty; the discussion recommends retaining complex designs that remain hard for frontier models.Source: §VI · p. 7
Construction discipline and evaluation criterion
- The benchmark filters candidates for oracle-discriminating tests and scores C-level patches against executable suite oracles, grounding pass@k in functional behavior.Candidates are retained only when correct programs pass and injected buggy programs fail their oracle.Source: §VI · p. 7
Solution multiplicity as an LLM-native metric
- LOC and mean solution multiplicity are strongly negatively correlated with mean empirical pass@1, whereas cyclomatic complexity is not a significant negative predictor.Solution multiplicity measures distinct patches across attempts; high multiplicity with no passing runs flags possible oracle-coverage gaps or context-length problems.Source: Figure 5 · p. 7, §VI · p. 7
Failure mode analysis
- Compile failures remain at 11–20% across suites and bug types, while plausible-but-incorrect test failures reach 54.1% on CHStone.The authors connect CHStone’s test failures to its single hardcoded vector, which accepts a wide range of wrong patches.Source: §VI · p. 7
Frontier model saturation
- GPT-5.4 and Opus 4.6 tie at pass@5 (78.4%), but CHStone remains unsaturated at 51.2% pass@5 for Opus 4.6.The discussion identifies CHStone as the relevant hard benchmark for future work.Source: §VI · p. 7
Reproducibility and implications for repair tools
- The harness lets LLM-only, symbolic, localization-guided, and hybrid repair systems share an executable evaluation, while pointing to localization, execution feedback, and stronger oracles as needed capabilities.Future work proposes applying localization-guided or iterative repair to CHStone, strengthening its tests, and adding proprietary HLS toolchains for synthesis-aware oracles.Source: §VI · pp. 7–8
Threats and scope
- Public benchmark training exposure remains a possible memorization confound, and the evaluation covers zero-shot C-level repair rather than post-synthesis correctness.The authors also identify oracle sensitivity: bugs in code paths not exercised by single-vector tests are invisible.Source: §VI · pp. 7–8
VII. CONCLUSION
Across 125 executable HLS repair instances, design context and scale dominate difficulty alongside solution multiplicity’s strong prediction of repair failure. Frontier models repair simple kernels more successfully, but complex cryptographic designs remain substantially unrepaired.
VII. CONCLUSION
- Solution multiplicity strongly predicts repair failure and provides an LLM-native measure of oracle underconstraint.Frontier models largely solve simple HLS kernels but leave complex cryptographic designs substantially unrepaired.Source: §VII · p. 8
- SHFT bugs remain the hardest across all models, while semantic bug classes such as buffer indexing become reliably repairable only at frontier scale.Source: §VII · p. 8
- Meaningful HLS repair benchmarks must sample across design complexity, and hard instances motivate oracle-guided repair strategies such as fault localization and iterative feedback.Source: §VII · p. 8