PaperlayerView exact revision on arXiv

Source-linked AI summary

Benchmarking Agentic HLS Design Tasks With HLS-Eval

Stefan Abi-Karam, Callie Hao

arXiv:2609.09526v1cs.AR

TL;DR

High-performance HLS design requires substantial expertise, and existing benchmarks do not comprehensively capture realistic agentic HLS tasks or efficiency. The paper extends HLS-Eval with an agentic harness that gives models shell tools and compilers for design work and self-verification. In the initial evaluation, gpt-oss-120b saturates pass@10 on the simple generation benchmark, motivating harder tasks and further trace analysis.

  • Problem

    High-performance HLS design remains expertise-intensive, while existing benchmarks do not capture realistic task complexity or measure agent efficiency against design performance.

    Source: §I · p. 1

  • Method

    The paper extends HLS-Eval with a mini-swe-agent flow that provides agents shell tools and C/C++ compilers for HLS design and compiler-based self-verification.

    Source: §I · p. 1, §II · p. 1, A. HLS Design Generation Task · p. 2

  • Results

    gpt-oss-120b saturates pass@10 and exceeds 90% pass@1 at every HLS design validation stage.

    Source: A. HLS Design Generation Task · p. 2

  • Takeaways & Limitations

    Saturation on the simple generation benchmark motivates more complex cases resembling full end-to-end domain accelerator development.

    Source: A. HLS Design Generation Task · p. 2

    The causes of trace dynamics and their relationship to model scale and task complexity remain unclear.

    Source: C. Agent Trajectories · p. 3

Abstract

from arXiv · show

Large language models (LLMs) and AI agents are increasingly explored for hardware design, including high-level digital design. While most work targets code generation and editing for hardware description languages (HDLs), our prior work introduced HLS-Eval, an open-source benchmark for evaluating LLMs on high-level synthesis (HLS) design tasks. Those evaluations, however, focused on zero-shot generation and editing, leaving open how agents achieve HLS design tasks. We therefore extend HLS-Eval with an agentic evaluation flow built on the open-source mini-swe-agent framework. The flow lets HLS design agents use file-editing tools, invoke a C++ compiler for self-verification, and iteratively refine designs during inference, while logging agent traces for analysis of cost, token usage, and iteration count. We present initial results on the existing HLS-Eval benchmarks. In our initial evaluation, we find open-source LLMs paired with an agentic harness solve every simple HLS code generation task in our evaluation, underscoring the need to expand benchmark difficulty as model capabilities advance. Analyzing traces from passing and failing runs, we show how model size, token usage, and trajectory length relate to design pass rates. These results establish a foundation for agentic HLS design and motivate harder benchmarks and new agentic tooling as model capabilities progress.

I. INTRODUCTION

High-performance HLS design remains expertise-intensive, while agentic HLS lacks comprehensive benchmarks for realistic task complexity and efficiency. The paper extends HLS-Eval with an agentic flow and evaluates HLS kernel generation.

I. INTRODUCTION

  • HLS supports accelerator development from C, C++, or Python descriptions,,,, but high-performance designs still require substantial hardware expertise.Designers must reason about directives, dataflow, parameterized design spaces, and the effects of source choices on latency and resource use.Source: §I · p. 1
  • Existing benchmarks do not capture realistic HLS task complexity or measure agent efficiency, such as cost and runtime, against achieved design performance.The authors identify this benchmarking gap as a key reason progress in agentic HLS automation has lagged behind LLM-driven RTL design.Source: §I · p. 1
  • The paper extends HLS-Eval with a standardized agentic evaluation flow and evaluates natural-language and specification-based HLS kernel generation using pass rates, inference scaling, and agent traces.The authors describe these results as a starting point for integrating and benchmarking the ideas in a larger agentic HLS design flow.Source: §I · p. 1

II. METHODOLOGY

The evaluation flow adds a mini-swe-agent harness to HLS-Eval, gives agents a Linux container with shell tools and compilers, and checks their final outputs.

II. METHODOLOGY

  • The flow uses mini-swe-agent, a simplified SWE-agent [28] harness, with a Bash shell in a Linux Docker container and standard file utilities, C/C++ compilers, and scripting tools.This provides a minimal tool baseline for the agent.Source: §II · p. 1
  • Agents receive an initial prompt and mounted design files; after completion or a step or cost limit, the output is checked for a final HLS implementation and unchanged testbench and header files.The file-integrity check prevents agents from altering the provided tests or headers.Source: §II · p. 1
  • The flow subclasses HLS-Eval’s Evaluation base class, inheriting parallelization, automated HLS tool calls, and model and task parameterization; integration and new results took less than one PhD-student hour.Implementation details are available in the HLS-Eval open-source repository.Source: §II · p. 1

III. RESULTS

The initial evaluation uses existing HLS-Eval designs and compares two open-source model sizes within the proposed agentic workflow.

III. RESULTS

  • The evaluation covers 85 designs from CHStone, MachSuite, Polybench, Rosetta, and C2HLSC.The authors selected gpt-oss-20b and gpt-oss-120b as strong, low-cost open-source baselines with two model-size variations.Source: §III · pp. 1–2
  • For the initial evaluation, the study uses strong, low-cost, open-source baselines with two model-size variations for comparison.The figure caption presents an end-to-end agentic HLS workflow for rapid prototyping of high-performance domain accelerators, emphasizing HLS-Eval extension for benchmarking agentic systems for HLS design.Source: §III · p. 2, Figure 1 · p. 2

A. HLS Design Generation Task

On simple HLS generation tasks, gpt-oss-120b saturates pass@10 and exceeds 90% pass@1 at every validation stage. This result motivates more complex benchmarks.

A. HLS Design Generation Task

  • The evaluation reports pass@1 and pass@10 across parsing, compilation, functional testbench checks, and successful HLS synthesis, using N=10 samples per case.These stages measure whether generated designs are parseable, compilable, functionally correct, and synthesizable.Source: A. HLS Design Generation Task · p. 2
  • gpt-oss-120b saturates pass@10 and exceeds 90% pass@1 at every HLS design validation stage.The authors attribute this capability to compiler-based syntactic and functional self-verification during inference.Source: A. HLS Design Generation Task · p. 2
  • Saturated results provide no further informative feedback about model capabilities, motivating more complex cases resembling full end-to-end domain accelerator development.Source: A. HLS Design Generation Task · p. 2

B. Verifiers and Inference Scaling

Verifiers can help agents check designs during inference, while sampling more rollouts improves pass rates. In these evaluations, inference-scaling gains were larger for the smaller model, and the tested gpt-oss-120b model reached 100% pass rates across stages at k = 10.

B. Verifiers and Inference Scaling

  • C++ compilers, co-simulation, HLS synthesis, DSE, and downstream implementation tools can verify designs and support inference scaling.Sampling more agent rollouts can improve pass rates because only one rollout needs to pass every verification stage.Source: B. Verifiers and Inference Scaling · pp. 2–3
  • 100% pass rate at every HLS design-flow stage was reached by gpt-oss-120b with k = 10.The figure reports pass rates at each stage of the flow.Source: Figure 2 · p. 2
  • At k = 10, synthesis pass rate increased by +60% for gpt-oss-20b and +11% for gpt-oss-120b.The gains were measured against k = 1 samples.Source: Figure 3 · p. 2, B. Verifiers and Inference Scaling · p. 3
  • The reported sampling strategy is only one form of inference scaling; future work will examine evolutionary search and balancing gains against inference cost and tool runtime.Source: B. Verifiers and Inference Scaling · p. 3

C. Agent Trajectories

The initial trace analysis found that token use and trajectory length relate to outcomes differently across the two evaluated models. The causes of these dynamics and their relationship to model scale and task complexity remain unclear.

C. Agent Trajectories

  • For gpt-oss-20b, passing traces used more tokens and steps, while for gpt-oss-120b, failing traces used more.The analysis groups traces by whether generated designs passed testbench and synthesis.Source: C. Agent Trajectories · p. 3
  • The analysis presents an initial view of inference traces on HLS design tasks.It compares token counts and trajectory steps by outcome.Source: C. Agent Trajectories · p. 3
  • The causes of these dynamics and their relationship to model scale and task complexity remain unclear, motivating deeper investigation.Future analysis will include tool-use frequency and correlations between tool runtimes and final design performance.Source: C. Agent Trajectories · p. 3

IV. CONCLUSION

The conclusion presents an initial effort to extend HLS-Eval toward end-to-end agentic HLS design and reports preliminary agentic evaluation results.

  • The work extends HLS-Eval toward end-to-end agentic design of domain-specific accelerators using HLS.Source: §IV · p. 3
  • It reports preliminary results on open-ended agentic HLS design evaluation and explores inference scaling and trace analysis.Source: §IV · p. 3
  • The authors aim to combine LLM-driven agents, tool automation, and standardized benchmarking in reproducible open-source frameworks for domain acceleration.They describe this as supporting scalable, intelligent, and accessible domain acceleration for scientists, engineers, and researchers.Source: §IV · p. 3
Loading 2609.09526v1…