Source-linked AI summary
HLSFactory-Agent: Large-Scale Agentic HLS Dataset Construction from Academic and Open-Source Projects
Kaushik Chandana, Jay Imperatori, Tanmay Shukla, Justin Zhou, Stefan Abi-Karam, Callie Hao
TL;DR
Large, diverse HLS datasets are needed for QoR modeling and LLM-based hardware-design research, but curation still requires locating repositories and extracting standalone designs manually. HLSFactory-Agent automates extraction in Docker and pairs it with publication-indexing scripts; an initial evaluation produced 271 candidates across 26 repositories, including 130 that passed Vitis HLS synthesis.
Problem
Existing HLS datasets and benchmarks are limited in diversity and size, while curation still requires manual discovery and extraction of designs from academic and open-source codebases.
Method
HLSFactory-Agent uses an LLM agent in Docker to extract standalone synthesizable designs, while scripts scrape, merge, and deduplicate HLS-related publications for repository discovery.
Results
The initial evaluation extracts 271 candidate designs across 26 repositories, of which 130 pass Vitis HLS synthesis and 141 fail.
Takeaways & Limitations
The system provides an automated path for expanding HLSFactory-compatible design resources from academic and open-source repositories.
Abstract
from arXiv · showhide
Building large, diverse datasets of high-level synthesis (HLS) designs beyond common community benchmarks remains an open challenge. This challenge is made urgent by the rise of deep learning and LLMs for hardware design, which demand such datasets to train QoR models and benchmark LLMs on HLS tasks. Despite ongoing efforts to broaden sources, dataset curation still depends on manual work: locating HLS designs across academic publications and open source, then extracting standalone designs from larger codebases. The process is error-prone and demands expert knowledge, iterative testing, and substantial per-repository engineering. To address this, we present HLSFactory-Agent, an LLM agent that automates large-scale HLS dataset curation by extracting standalone designs from larger codebases. HLSFactory-Agent runs the open-source Pi agent framework inside Docker containers to build and evaluate each extracted design. This turnkey automation allows users to pass a GitHub link or code directory to HLSFactory-Agent and receive a folder of extracted HLS designs ready to be integrated into the HLSFactory dataset framework. Additionally, we provide open-source scripts to scrape and index papers from computer architecture, EDA, and FPGA conferences that possibly implement or use HLS designs, allowing for faster human discovery and curation of HLS designs for HLSFactory-Agent. We report initial results from running HLSFactory-Agent across a small subset of our indexed repositories, demonstrating successful extraction of synthesizable designs from structured codebases. We open source HLSFactory-Agent and indexing scripts at https://github.com/sharc-lab/hlsfactory-agent.
I. INTRODUCTION
HLS datasets are increasingly important for QoR modeling, design-space exploration, and LLM-based HLS research, but existing benchmarks remain limited in diversity and size. HLSFactory-Agent addresses this curation challenge by extracting standalone synthesizable designs and supporting repository discovery.
- Existing HLS benchmarks remain limited in diversity and size, despite growing needs for QoR models and LLM-based HLS evaluation.
- HLSFactory-Agent automates dataset curation by extracting standalone, synthesizable designs from unstructured academic and open-source repositories.
- Open-source indexing scripts scrape HLS-related publications to accelerate discovery of candidate repositories for automated extraction.
A. Design Curation from Academic Publications and Open-Source Code Repositories
The curation workflow searches major architecture, FPGA, and EDA publication venues for HLS-related work, then merges and deduplicates records into candidate-paper files. It produces a broad discovery pool that still requires human review before extraction.
- A. Design Curation from Academic Publications and Open-Source Code Repositories: The workflow searches computer architecture, FPGA, and EDA venues for publications containing HLS-related keywords.
- A. Design Curation from Academic Publications and Open-Source Code Repositories: Scripts merge and deduplicate records using DOI extraction or normalized title-and-year matching, producing CSV/Excel files with paper metadata and links.
- A. Design Curation from Academic Publications and Open-Source Code Repositories: 2,517 candidate publications from 1984–2026 are identified, but human review remains necessary because HLS use, artifact availability, and code redistributability vary.
- A. Design Curation from Academic Publications and Open-Source Code Repositories: The workflow targets public repositories associated with reviewed papers so HLSFactory-Agent can perform subsequent design extraction.
B. Agentic Design Extraction from Codebases
HLSFactory-Agent runs an LLM extraction process in an isolated Docker environment, where it identifies designs, packages their dependencies, adapts code for Vitis HLS, and checks compilability. The evaluation visual distinguishes passing and failing extracted candidates by repository.
- B. Agentic Design Extraction from Codebases: Docker isolation and a minimal toolset provide file operations, Bash, Clang compilation, and reproducible agent sessions for self-testing extracted designs.
- B. Agentic Design Extraction from Codebases: The agent analyzes repositories, creates one directory per design, copies dependencies and test data, adapts non-Vitis constructs, and generates synthesis scripts.
- B. Agentic Design Extraction from Codebases: Figure 1 reports 271 extracted candidates across 26 repositories, with 130 passing and 141 failing Vitis HLS synthesis checks.
III. INITIAL RESULTS
Across 26 repositories, the initial evaluation extracts 271 candidate designs, but strict Vitis HLS validation passes 130 and fails 141. Extraction effort also varies with repository structure, motivating cost- and time-aware evaluation.
- III. INITIAL RESULTS: Validation counts designs as passing only when generated synth.tcl completes Vitis HLS synthesis; Tcl errors, missing headers, target issues, unsupported constructs, and timeouts fail.
- III. INITIAL RESULTS: 271 candidate designs are extracted across 26 repositories, with 130 passing and 141 failing strict Vitis HLS validation.Five repositories produce no extracted designs, while others yield candidates that are not converted into valid standalone designs.
- III. INITIAL RESULTS: More extracted designs generally require more inference effort, although regular repository structure can produce many designs cheaply.
- III. INITIAL RESULTS: The results motivate evaluating extraction by designs per dollar and designs per minute, rather than dataset size alone.
IV. ONGOING / FUTURE WORK
Future work targets broader automation, stronger verification, and integration of newly extracted designs into community benchmarks. Extraction cost should also be evaluated relative to design yield and runtime.
- IV. ONGOING / FUTURE WORK: The authors plan to automate paper review and repository discovery with LLMs instead of relying on human review.
- IV. ONGOING / FUTURE WORK: They aim to run Vitis HLS Synthesis inside Docker containers to provide additional verification feedback and catch more failing cases.
- IV. ONGOING / FUTURE WORK: Extraction evaluation should consider designs per dollar and designs per minute, not only total dataset size.