PaperlayerView exact revision on arXiv

Source-linked AI summary

Benchmarking AI Agents for Hardware Design Automation via MCP Tool Calling

Leonardo Liparulo, Francesco Pierri

arXiv:2608.26199v1cs.AIcs.SE

TL;DR

The paper asks whether locally deployed open-source models can reliably automate expert-defined, dependency-ordered hardware design workflows under confidentiality constraints. It builds an MCP replica and benchmark to evaluate models and agent configurations, finding near-complete expected-call coverage for strong models and configuration-dependent reliability. The findings yield deployment practices for local agents in stateful hardware environments.

  • Problem

    The paper examines whether local LLM agents can reliably automate expert-defined hardware design workflows when confidentiality constraints limit hosted proprietary APIs.

  • Method

    The study builds an MCP server replicating a proprietary tool’s state and dependency logic, then benchmarks seven locally deployed open-source models across tasks and agent configurations.

  • Results

    Strong open-source models achieve near-perfect expected-call coverage under their best configurations, including multi-step sessions with implicit state and colocated GDB and Git servers.

  • Takeaways & Limitations

    Recommended practices include comprehensive tool descriptions, workload-representative benchmarking, target-model prompt testing, context management, and selective multi-agent decomposition.

  • Takeaways & Limitations

    The context-management study tests only per-task versus cumulative history, while the server and benchmark are faithful replicas rather than production APIs and cannot be released.

Abstract

from arXiv · show

We ask whether AI agents powered by locally deployed large language models can reliably automate expert-defined hardware design workflows in an industry-realistic tool-calling setting. In these environments, engineers issue repetitive, dependency-ordered operations---such as creating components, adding ports, and wiring connections---through specialised tools. Confidentiality constraints on component specifications and naming conventions often preclude hosted proprietary APIs, motivating the use of locally deployed models. To study this setting, we build a Model Context Protocol (MCP) server that reproduces the state and dependency logic of a proprietary hardware design tool used in embedded system development and construct a benchmark covering single-operation edits, multi-step dependency chains, invalid requests, misspelled prompts, and multi-server tool contexts. We evaluate seven open-source models comparing pipeline choices including system prompts, tool-description detail, context scope, and single-agent versus multi-agent architectures. Results show that strong models can achieve near-complete expected-call coverage on the benchmarked workflows, but reliability depends strongly on both task structure and agent configuration. Comprehensive tool descriptions consistently reduce failures, few-shot prompting can cause severe inaction for some models, cumulative context harms constrained models, and multi-agent decomposition helps weak workers or long sessions at the cost of additional calls. These findings provide practical guidance for deploying local LLM agents in stateful hardware design environments.

1 Introduction

Hardware design involves repetitive, dependency-ordered edits to persistent component models, making it a promising setting for local tool-calling agents. This paper addresses the reliability of such agents by introducing an MCP replica, benchmark, and systematic evaluation.

  • Hardware design workflows: Engineers use specialised tools to build structured component models containing components, ports, parameters, subcomponents, and connections.
  • Hardware design workflows: Dependency ordering requires each referenced element to be created before later operations use it.For example, ports must precede connections, and subcomponents must precede references to their ports.
  • Motivation: Repetitive template-like edits, including batches of ports and regular connection patterns, are attractive targets for automation.Automation preserves the structured operations and dependency constraints enforced by the design tool.
  • Deployment constraints: Local models address confidentiality constraints around component specifications, port names, and internal naming conventions, although they may be less capable than hosted frontier models.
  • Study design: The study implements a 14-tool MCP server replica and evaluates seven locally deployed open-source models across an expert-informed benchmark and pipeline configurations.The benchmark covers independent tasks, dependency chains, cross-task sessions, error handling, and multi-server contexts.

2 Related Work

Prior work benchmarks stateful tool-calling agents and applies LLMs to hardware generation or orchestration, but does not evaluate MCP agents operating persistent hardware design applications. This paper therefore combines hardware workflows with call-level and configuration-focused evaluation.

  • Stateful agent benchmarks: Application-operating agents are evaluated in retail, CRM, enterprise, and multi-application settings requiring dependency-ordered actions over shared mutable state.
  • Evaluation: Call-level metrics are needed because task-level success can hide omitted or extraneous operations in dependency-sensitive workflows.MCP-RADAR and MCP-Bench use call-level scoring for MCP-based tool-calling systems.
  • Hardware applications: Earlier hardware applications of LLMs focused on RTL generation and EDA-flow orchestration rather than structured calls to persistent design applications.
  • Research gap: Prior work did not evaluate MCP-style hardware agents across persistent cross-task state, invalid requests, or multi-server tool contexts.
  • Agent configuration: Agent configuration factors—including architecture, prompting, tool descriptions, and context length—can substantially affect tool-use performance.

3 Experimental Setup

The study evaluates locally deployed open-source agents in an MCP-based hardware-design environment, using a benchmark of dependency-aware tasks and call-level metrics. It varies agent architectures, prompting, context scope, and tool descriptions while testing clean and multi-server settings.

  • MCP Server: The MCP server reproduces a proprietary hardware design tool’s state, data model, and dependency constraints through 14 callable tools.Later calls may depend on earlier ones.
  • Agent Configuration: The evaluation compares single-agent ReAct with multi-agent Plan-and-Act architectures and varies system prompts, tool descriptions, and history scope.Plan-and-Act uses a planner, validator, and independent ReAct workers; history scope is either run-wide or task-local.
  • Benchmark: The expert-informed benchmark contains six core suites and two multi-server suites covering independent tasks, dependencies, history, errors, cross-task state, and noisy contexts.Easy, Medium, and Hard isolate task length, while History requires recovering identifiers from prior tasks.
  • Evaluation Metrics: Call-level metrics jointly measure expected-call completion, successful extraneous calls, rejected calls, and correct abstention on no-call tasks.ECC excludes tasks with empty expected call sets; EVCR is defined as 0 when no successful calls occur, and TFR as 0 when no calls occur.
  • Evaluation Protocol: Each task is submitted sequentially, agent mutations are removed afterward, and the server is restored to a canonical state before the next task.Replay preserves intended dependencies for History and Cross while preventing one error from blocking later tasks.

4 Results

Results show that local agents can perform structured hardware-design workflows, but performance varies sharply with task structure, configuration, and tool context. Best-observed results are configuration-dependent, while errors and multi-server contexts expose distinct reliability weaknesses.

  • Overall performance: Best-observed ReAct performance is reported per model across the six core task suites, rather than as out-of-the-box performance.Table 2 identifies each model’s ECC-maximising configuration and reports the resulting EVCR and TFR.
  • Task structure: Task structure strongly affects reliability: Llama 3.1 8B reaches 0.753 ECC on independent suites but 0.139 on state-carrying suites, an 82% drop.The state-carrying suites are History and Cross, whereas the independent suites are Easy, Medium, and Hard.
  • Configuration dependence: Configuration quality is workload-dependent: Llama 3.1 8B improves from 0.113 global-optimum ECC to 0.538 with the History-specific optimum.For Gemma 4 E4B, a poor configuration lowers ECC from 0.811 to 0.168 on the same tasks.
  • Failure modes: Errors and noisy tool contexts reveal failures beyond coverage: no-call tasks require NCA, EVCR, and TFR, while extra servers raise EVCR without materially changing ECC.The noisy-context result suggests additional tools mainly increase extraneous routing errors rather than reducing coverage.
  • Agent configuration: Few-shot prompting can cause severe inaction, and cumulative history can harm constrained models, while temperature checks preserve model rankings and produce small ECC shifts.For Llama 3.1 8B, cumulative history cuts ECC from 0.667 to 0.192 on independent tasks; across temperatures, Gemma 4 26B ranges from 0.881 to 0.896.
  • Agent configuration: Comprehensive tool descriptions are the most consistent configuration benefit, maximizing ECC in 35 of 42 model–task-suite combinations.Minimal descriptions raise TFR for every model, roughly doubling it for most, while comprehensive descriptions expose parameter semantics, constraints, and failure conditions.

5 Conclusion

The benchmark shows that local open-source agents can achieve near-perfect expected-call coverage, but reaching that performance depends on model choice and system configuration. The authors recommend workload-specific testing, comprehensive tool descriptions, context management, and selective multi-agent decomposition.

  • Near-perfect expected-call coverage is achievable by the strongest open-source models under their best configurations, including multi-step sessions and colocated GDB and Git servers.
  • Model choice sets the achievable performance range, while system configuration determines whether that performance is reached in practice.
  • Deployment should combine target-model prompt testing, comprehensive tool descriptions, representative workload benchmarks, constrained-model context management, and selective multi-agent decomposition.
  • The findings indicate that local LLM agents are practical for structured, stateful hardware component-editing workflows when the tool-calling pipeline is appropriately configured.

6 Limitations

The paper’s limitations concern context-management coverage, metric severity, latency measurement, and evaluation against a faithful replica rather than production APIs. These boundaries limit what can be concluded about longer sessions, latency-sensitive deployment, severity-weighted errors, and production behavior.

  • Context-management experiments tested only per-task versus cumulative history, leaving variable memory windows and longer session effects unmeasured.
  • EVCR weights an unnecessary parameter and an incorrect subcomponent connection equally, despite their different impacts on design correctness.
  • Latency was not systematically measured, so the trade-off between smaller models, inference time, and coverage remains uncaptured.
  • The MCP server mirrors the proprietary tool but does not call production APIs, so some gap with production behavior should be expected.

A Agent Architecture Diagrams

The diagrams describe two tool-calling architectures: ReAct iteratively executes one tool call at a time, while Plan-and-Act validates a decomposed plan and delegates steps to independent workers.

  • ReAct: ReAct loops between LLM responses and host-executed tool calls until no call is produced or the turn limit is reached.
  • Plan-and-Act: Plan-and-Act uses a planner, one-plan validator with replanning, independent ReAct workers, and a final synthesis call.

B.1 Easy

The benchmark’s representative tasks cover component creation, port and parameter edits, saving, dependency-ordered multi-step workflows, and session-based state recovery. Easy through Hard tasks use increasingly involved component-editing sequences, while History tasks omit identifiers that must be recovered from earlier turns.

  • Easy: Easy examples create VLNV-identified components and add signal, transactional, or user-parameter elements with specified attributes.
  • Medium: Medium examples combine component creation with port additions, memory implementations, parameter values, and saving.
  • Hard: Hard examples require longer ordered sequences involving multiple ports, implementations, multiplicities, and final component saves.
  • History: History tasks run in session order, and later requests omit identifiers that agents must recover from earlier turns before editing the correct component.

B.5 Errors — misspelled vs. clean

The Errors prompts preserve the intended operations of their Hard counterparts while adding heavy spelling noise. Each clean–misspelled pair therefore shares identical expected tool calls, isolating prompt-noise effects.

  • Paired prompt design: Each misspelled Errors prompt mirrors a Hard prompt with heavy orthographic noise and identical intent.The paired prompts use the same expected calls, making prompt noise the isolated difference.
  • CPU task: The clean CPU task creates riscv tile, adds m axi and clk, and saves the component.The expected sequence is create component, add transactional port, add signal port, then save.
  • Misspelled variants: The misspelled CPU, memory-controller, and interconnect prompts retain the corresponding operations and parameters despite orthographic corruption.Their expected calls match the clean versions for each task.
  • Memory-controller task: The clean memory-controller task creates sram ctrl, adds s axi, rst n, and clk, then saves it.The transactional port uses a memory implementation named mem with 30 bytes.
  • Interconnect task: The clean interconnect task creates axi interconnect, adds m0 port, s0 port, and s1 port, then saves the component.The target ports use user defined implementation.

B.6 Errors — no-call

The no-call Errors cases test whether agents reject incomplete requests instead of guessing missing state or arguments. Valid multi-component and multi-server cases specify concrete operations and routing targets.

  • No-call policy: Incomplete requests require no tool call and an explanation when the missing information prevents a valid operation.The benchmark explicitly defines this as the correct behaviour.
  • Missing arguments: A component-creation request without version is rejected because version is required and cannot be inferred.The vendor, library, and name alone are insufficient.
  • Missing arguments: Signal-port creation is rejected when direction and width are missing and the target component is absent from state.The request cannot be completed from the supplied information.
  • Missing arguments: A memory-backed transactional port is rejected when its memory name and size are unspecified.Both fields are required for the memory implementation.
  • Dependency chains: Cross-component tasks require ordered creation, port additions, subcomponent instantiation, and connections.The benchmark includes four CPU shells, repeated signal ports, four initiator ports, and hierarchical connections.
  • Multi-server context: External Git and GDB tasks are interleaved with design tasks, requiring routing to the correct server.Routing mistakes are counted as EVCR rather than ECC.

C Stability Experiments

The stability experiments examine temperature, repeated-run variability, system prompts, and tool-description detail across demanding benchmark suites. Results show stable temperature rankings, prompt-sensitive collapses, and consistently higher tool-failure rates with minimal descriptions.

  • Temperature sweep: Rankings and absolute performance remain stable across temperatures 0, 0.5, and default settings for every model.Lowering temperature does not consistently reduce TFR.
  • Run stability: Gemma 4 E4B reaches σmax = 0.494 in the run-scope, few-shot, minimal-tools configuration.The same configuration produces the few-shot collapse, supporting a genuine effect rather than sampling artefact.
  • Run stability: Llama’s worst configuration has ECC standard deviation 0.109 because 9 of 10 runs collapse to ECC ≈0.This differs from ordinary run-to-run noise.
  • System prompts: Gemma 4 31B falls from 0.956 to 0.571 and Gemma 4 E4B from 0.731 to 0.179 under few-shot prompting.EVCR and TFR fall together, indicating inaction rather than error.
  • Tool descriptions: Minimal tool descriptions raise TFR for every model, roughly doubling it in most cases, while ECC changes less uniformly.The comparison averages across suites, system prompts, and history scopes.

D.3 History-Scope Ablations

The history-scope ablation compares per-task and cumulative context while documenting prompt and tool-description alternatives. Llama 3.1 8B uniquely prefers per-task scope, whereas other models tolerate or benefit from cumulative history at this session length.

  • History scope: Llama 3.1 8B is the only model that prefers per-task history scope.All other models tolerate or benefit from cumulative history at the tested session length.
  • Tool descriptions: Comprehensive signal-port descriptions specify existence, uniqueness, valid directions, positive dimensions, defaults, and downstream availability.Minimal descriptions omit these operational constraints.
  • ReAct prompts: The ReAct prompts require one tool call per turn, complete execution of all requested operations, and no calls when required arguments remain missing.They also prohibit invented values and require clear explanations for invalid requests.
  • Connection semantics: Hierarchical denotes the parent component itself and is used only when a connection endpoint is a top-level parent port.It must not represent a subcomponent instance or connect two subcomponents.
  • Failure handling: When a connection fails, the prescribed response is to inspect the parent and report missing ports rather than modify children or retry guessed names.The example reports an invalid source port after retrieving parent details.
  • Planning: Plan-and-Act planners retrieve component details before steps whose instance or port names are not already known.The planner explicitly forbids guessing names.
Loading 2608.26199v1…