PaperlayerView exact revision on arXiv

Source-linked AI summary

Closed-loop evaluation of LLM agents for embedded software development

Jorge García-Carrasco, Sergio García-Carrasco, Alejandro Maté, Juan Trujillo

arXiv:2610.11447v1cs.SEcs.AIcs.ARcs.LG

TL;DR

Embedded-agent evaluation has largely emphasized static synthesis or offline correctness, despite firmware correctness depending on behavior under sensing, timing, and safety constraints. This paper presents a closed-loop benchmark in which agents implement and self-verify firmware from prose requirements across tasks and feedback scenarios. In realistic self-verification, gpt-5.4 leads the evaluated configurations, while qwen3.5-27B is the strongest observed local model.

  • Problem

    Embedded-agent evaluation remains limited and often emphasizes static synthesis or offline correctness, although firmware correctness depends on closed-loop behavior under sensing, timing, and safety constraints.

    Source: §1 · p. 2

  • Method

    The benchmark has agents translate prose engineering handoffs into implementation and visible self-verification steps, then iterate toward behaviorally correct firmware across five tasks and four feedback scenarios.

    Source: §1 · p. 2, §3.2 · p. 5, §3.4 · p. 9

  • Results

    In realistic_self_verify, gpt-5.4 passed 14/15 (93.3%) and qwen3.5-27B passed 9/15 (60.0%), the highest observed local-model count.

    Source: §6 · p. 19

  • Takeaways & Limitations

    The four feedback scenarios distinguish self-verification quality from first-pass implementation and dependence on richer external repair guidance.

    Source: §6 · p. 20

    The evaluation uses deterministic simulated plants on one ESP32/QEMU stack, a small control-centric task suite, and one agent/tool surface and local deployment regime.

    Source: §5 · p. 19

Abstract

from arXiv · show

Large language models (LLMs) are increasingly deployed as coding agents that edit files, run builds and tests, inspect execution results, and repair software iteratively. Embedded firmware is a demanding target because correctness depends on closed-loop behavior under sensing, timing, and safety constraints, not only on static source quality. Yet embedded-agent evaluation remains limited and often emphasizes one-shot synthesis or offline correctness. We present a benchmark for closed-loop evaluation of embedded coding agents. Each task provides a plain-text engineering description, constrained workspace, and visible build-and-runtime surface. The agent must translate requirements into implementation and self-verification steps, then iterate until the required device behavior is achieved. The suite contains five embedded-control tasks and four feedback scenarios: one-shot generation, realistic self-verification, CI-style red/green feedback, and oracle-style detailed feedback. The implementation targets simulated ESP32 firmware for reproducibility. We evaluate seven GPT-family and Qwen-family configurations across five tasks and four scenarios, with three repetitions per condition for 420 runs. gpt-5.4 has the highest pass rate among evaluated configurations but does not saturate the benchmark; qwen3.5-27B is the strongest observed local model; and smaller local models degrade sharply in pass rate and search efficiency. These results suggest that capable local embedded coding agents are emerging.

1 Introduction

Embedded firmware evaluation must capture behavior in closed loop with sensors, actuators, timing, and safety constraints, not just plausible source code. The paper introduces a prose-first benchmark for agents to implement, verify, and repair firmware, and evaluates it across tasks, feedback scenarios, and models.

1 Introduction

  • Embedded firmware correctness depends on closed-loop behavior under sensing, timing, and safety constraints, so static source quality alone is insufficient.A controller can appear reasonable in code review yet fail with asynchronous inputs, hysteresis, timeouts, or sequence control.Source: §1 · p. 2
  • Existing embedded-agent evaluation emphasizes static synthesis, component checks, or offline correctness more than the full loop from prose requirements through testing, execution, feedback, and repair.The paper asks whether agents can converge to required device behavior by iteratively writing tests and firmware and executing them in a target environment.Source: §1 · p. 2
  • The benchmark evaluates prose-first, behavior-level embedded engineering through contract-aligned task packets, visible self-verification, five tasks, and four feedback scenarios.Its reference instantiation uses ESP32 firmware in QEMU with deterministic plant simulators for safe, reproducible evaluation.Source: §1 · pp. 2–3
  • Across 420 runs, gpt-5.4 had the highest observed pass rate without saturating the suite, while qwen3.5-27B was the strongest observed local model.The evaluation spans five tasks, four feedback scenarios, and seven models; smaller local models degrade in accuracy and efficiency.Source: §1 · p. 3

2 Related Work

Prior software-engineering benchmarks emphasize iterative repository work, while embedded and low-level evaluations often focus on static synthesis or narrower checks. This paper adopts simulation as a reproducible staging ground for studying the full embedded-agent workflow.

2 Related Work

  • Repository-oriented evaluations established the value of testing whether agents use failure signals and iterative repair, but embedded firmware adds behavior under temporal and safety constraints.The cited perspective treats practical capability as productive feedback use, not code synthesis alone [6] [7] [10].Source: §2 · pp. 3–4
  • Embedded and hardware-adjacent work has broadened empirical coverage, yet much still centers on static synthesis, component checks, offline correctness, or narrow reasoning tasks.Examples include embedded-system benchmarks [8], Verilog-focused work [18], and adjacent edge-device and on-device studies.Source: §2 · p. 4
  • Simulation supports reproducible, controlled closed-loop comparisons without immediate physical deployment, while leaving real-device validation for future hardware-in-the-loop and machine studies.The paper uses simulation as a methodological staging ground rather than a replacement for real-device validation.Source: §2 · p. 4

3 Benchmark Design

The benchmark tests whether agents can turn prose engineering handoffs into behaviorally correct firmware through visible, iterative verification in a controlled execution environment. Its design combines a constrained workflow, contract-aligned tasks, progressively harder control problems, and feedback scenarios that distinguish self-verification from externally guided repair.

3.1 Benchmark objective

  • The benchmark evaluates agents from a prose engineering handoff through implementation, testing, builds, runtime checks, and iteration toward behavior that satisfies nominal and safety requirements.The objective is closed-loop engineering performance rather than final-source quality alone.Source: §3.1 · p. 5

3.2 Benchmark workflow and contract

  • In realistic_self_verify, agents use visible local tooling in a constrained workspace and decide when to submit; the hidden evaluator provides no repair feedback before submission.The hidden evaluator runs only after submission, which is the main workflow target; Figure 1 summarizes this sequence.Source: §3.2 · p. 5, Figure 1 · p. 6

3.3 Design principles and construction process

  • The design aligns hidden checks with public requirements and makes failures attributable to control behavior, while providing visible self-verification affordances and aiming for stack portability.A fixed construction pipeline derives the public handoff and hidden evaluator from a shared engineering control plan; hidden tests cover nominal, boundary, and safety-critical cases.Source: §3.3 · p. 7

3.4 Task suite

  • The five-task ladder progresses from single-sensor threshold control to plant lag, asynchronous interlocks, coupled actuators, and timed sequence control with disturbance recovery.The final filter_tank_sequence task requires three valid clear samples, a settling delay, and recovery by restarting filtering after disturbances.Source: §3.4 · p. 9

3.5 Evaluation scenarios

  • Four scenarios separate first-pass coding, autonomous self-verification, minimal CI-style feedback, and detailed oracle feedback.The contrast between one-shot and realistic self-verification isolates whether agents can use visible build, test, and runtime tooling before submission.Source: §3.5 · pp. 9–10

3.6 Implementation details

  • The reference implementation runs ESP32 firmware in QEMU with task-specific simulators and UART-virtualized I/O, while hidden evaluation grades externally visible behavior.Deterministic plant models support reproducible runs; hidden checks distinguish host-test, integration, pass, and non-submission outcomes.Source: §3.6 · pp. 10–11
  • For tank_fill_drain, agents must verify the mid-band latch and timeout behavior, since plausible code and incomplete local tests can produce false greens.The hidden checks include preserving the safe ON state in the mid-band and forcing pump-off after the bounded timeout window.Source: Illustrative example · p. 11

4 Evaluation

The evaluation tests seven hosted and local models across five embedded-control tasks and four feedback scenarios, with realistic self-verification as the primary condition. Results distinguish model performance, feedback sensitivity, and local-model resource tradeoffs.

4 Evaluation

  • The study evaluates seven models across five tasks and four feedback scenarios, repeating each three times for 420 runs; all evaluations use the same pi harness.Close model orderings are treated as descriptive trends because each per-model, per-scenario analysis has 15 outcomes from five tasks and three repetitions.Source: §4 · pp. 11–12

4.1 Primary Results: Realistic Self-Verification

  • 14/15 (93.3%) passes gives gpt-5.4 the highest primary-condition result; qwen3.5-27B leads local models at 9/15 (60.0%).The suite remains unsaturated: gpt-5.4 misses one run. filter_tank_sequence is the strongest task separator, while qwen3.5-2B never passes and its modal outcome is no submission.Source: Table 3 · p. 13, §4.1 · pp. 13–15, Figure 2 · p. 14

4.2 Effect of Feedback Visibility

  • qwen3.5-27B reaches 12/15 (80.0%) under oracle feedback versus 9/15 (60.0%) in realistic self-verification, while weaker local models are more feedback-sensitive.The 35B-A3B model rises from 4/15 in one-shot and realistic self-verification to 8/15 with CI-style feedback and 10/15 with oracle feedback; 2B remains at zero across all 60 runs.Source: Table 4 · p. 15, §4.2 · p. 16

4.3 Local Models on Shared Hardware

  • 38/60 (63.3%) makes qwen3.5-27B the most accurate local model; 35B-A3B submits earlier but reaches 26/60 (43.3%).Below the useful local range, accuracy and search efficiency degrade: 4B reaches 5/60 (8.3%), while 2B produces no passes and averages 54.95M tokens across 60 runs.Source: Table 5 · p. 16, §4.3 · pp. 16–17

5 Discussion

The discussion treats realistic self-verification as the clearest measure of autonomous embedded-agent capability, with feedback scenarios diagnosing different skills and local-model results informing deployment trade-offs. It also emphasizes that the benchmark’s findings are bounded by its simulated, control-centric setting.

5 Discussion

  • Under realistic_self_verify, gpt-5.4 leads at 14/15 (93.3%), versus qwen3.5-27B at 9/15 (60.0%); local variants drop to 4/15, 2/15, 2/15, and 0/15.The 4/15, 2/15, 2/15, and 0/15 results correspond to the 35B, 9B, 4B, and 2B local variants, respectively; qwen3.5-2B usually failed to submit. The ordering reveals a capability gradient: qwen3.5-27B is beyond a toy baseline but not at frontier parity.Source: §5 · p. 18
  • Feedback scenarios diagnose different capabilities: gpt-5.4 leads in every scenario, while the 35B-A3B model improves sharply with richer hidden feedback.gpt-5.4-mini improves most under CI-style red/green feedback; qwen3.5-27B is strongest in oracle_full and remains competitive in realistic_self_verify. Across all four scenarios, the benchmark separates implementation, self-verification, and feedback dependence.Source: §5 · p. 18
  • On one RTX 4090 with llama.cpp, the 27B dense model is the best local operating point at 38/60 runs; the faster-to-submit 35B-A3B model is not better.The 9B model remains a meaningful but weaker agent, while the 4B model spends large interaction budgets for relatively few successes and the 2B model falls into long no-submission loops. This same-hardware comparison addresses local deployment choices under a fixed on-premises budget.Source: §5 · pp. 18–19
  • filter_tank_sequence is the suite’s main separator and remains difficult for strong models, whereas tank_fill_drain serves as a baseline that weaker models can sometimes solve.These task-level patterns are presented as benchmark-design lessons from the pooled results.Source: §5 · p. 19
  • The results are bounded by deterministic simulated plants on one ESP32/QEMU stack and a small, control-centric task suite, rather than broader embedded workloads.The local study also uses a common 4-bit GGUF regime; higher-precision effects, especially around 9B, remain a separate question for future work.Source: §5 · p. 19

6 Conclusions and Future Work

The benchmark frames embedded-agent evaluation around behavior-level verification and closed-loop repair, with robust autonomous self-verification remaining a key gap. Results distinguish models and feedback strategies, while future work targets hardware-facing evaluation and broader local-model studies.

6 Conclusions and Future Work

  • 14/15 (93.3%) for gpt-5.4 and 9/15 (60.0%) for qwen3.5-27B in realistic_self_verify; gpt-5.4 leads, and qwen3.5-27B is the strongest local model.The primary condition requires agents to construct and trust their own local evidence.Source: §6 · p. 19
  • Feedback visibility distinguishes first-pass synthesis, self-verification, and externally guided repair instead of collapsing them into one aggregate score.Models differ in how they construct self-verification loops and depend on richer feedback to repair failures.Source: §6 · p. 20
  • Under a common local deployment regime on shared hardware, dense qwen3.5-27B is stronger overall than qwen3.5-35B-A3B, which reaches first submission faster.The comparison is framed as a trade-off between throughput and effective capability.Source: §6 · p. 20
  • Smaller local models degrade in pass rate and search efficiency, and the 2B checkpoint is not necessarily cheaper or more usable in closed-loop interaction.The deployment boundary is reliable convergence through verification and revision, rather than plausible code alone.Source: §6 · p. 20
  • The authors argue that embedded-agent evaluation merits a distinct benchmark, with robust autonomous self-verification—not only implementation quality—as a key gap below frontier performance.The benchmark is designed to expose whether agents generate local evidence, interpret it correctly, and stop only when the controller is ready.Source: §6 · p. 20
  • Future work should extend the methodology from deterministic simulation to hardware-in-the-loop and carefully controlled real-device studies.Real hardware can surface sensor jitter, communication latency, watchdog behavior, startup transients, peripheral interaction, and deployment packaging.Source: §6 · p. 20
  • Future work also calls for studying local-model operating points and extending tasks to noisier sensors, delayed signals, actuator saturation, recovery, and richer fault management.The paper identifies local models as promising for privacy-sensitive, on-premises, and industrial settings.Source: §6 · p. 20

CRediT Authorship Contribution Statement

The contribution statement assigns conceptualization, methodology, software, validation, supervision, administration, and writing roles across the four authors.

CRediT Authorship Contribution Statement

Data Availability

The benchmark repository and evaluation artifacts are publicly available, including materials for the official 420 runs.

Data Availability

  • The publicly available repository includes the benchmark, task packets, runner scripts, configuration files, and curated evaluation artifacts.Source: Data Availability · p. 21
  • The paper-facing artifact bundle is organized under artifacts/evaluations/paper_closed_loop_eval_420/.Source: Data Availability · p. 21
  • The bundle includes per-run prompts, session transcripts, runner logs, and summary files for all 420 official runs.Source: Data Availability · p. 21

Related papers

Loading 2610.11447v1…