PaperlayerView exact revision on arXiv

Source-linked AI summary

EdgeCraft: Automated Model Crafting for Edge IoT

Genglin Wang, Kaiwei Liu, Liekang Zeng, Wangsong Yin, Shangcheng Jin, Guoliang Xing, Zhenyu Yan

arXiv:2609.35167v1cs.LGcs.DC

TL;DR

Edge ML development requires navigating a fragmented, scenario-dependent design space while meeting task-quality, latency, and energy SLOs, and trustworthy target-device verification is costly. EdgeCraft combines an SLO-guided synthesis tree with progressive verification and shared services. Across 50 public tasks, it beats the task-specific Reference in best-observed quality on 40 and finds an SLO-feasible artifact on 45.

  • Problem

    Edge ML development must navigate coupled, scenario-dependent choices to meet diverse quality, latency, and energy SLOs, while trustworthy target-device verification is costly.

    Source: §1 · pp. 1–2

  • Method

    EdgeCraft guides candidate revisions with measured SLO gaps, progressively verifies them from low-cost checks to full target-device evaluation, and reuses verified compatibility rules.

    Source: §1 · p. 2

  • Results

    Across 50 public tasks, EdgeCraft exceeds the task-specific Reference in best-observed quality on 40 and finds an SLO-feasible artifact on 45, with the outcomes overlapping on 38.

    Source: §1 · p. 2

  • Takeaways & Limitations

    Competitive performance on the self-collected SEN dataset suggests generalizability to a real-world wearable IoT sensing task.

    Source: §1 · p. 3

    EdgeCraft does not guarantee a global optimum, and data collection, labeling, and task definition remain outside its automation pipeline.

    Source: §5 · p. 12

Abstract

from arXiv · show

Machine learning (ML) increasingly powers Internet of Things (IoT) applications at the edge. Yet producing a deployable edge ML artifact for a specific scenario requires navigating a huge search space spanning data representation, model design, training on domain-specific data, and runtime customization. This workflow is fragmented and difficult to scale across diverse edge applications. We present EdgeCraft, an LLM-driven system that turns high-level intent into deployable edge ML artifacts. Building such a system raises two challenges: (1) How can an LLM be guided to find high-quality solutions that meet dynamic SLOs for task quality, latency, and energy? (2) How can trustworthy target-device verification be obtained at low cost? EdgeCraft addresses these challenges with two designs. (1) A constraint-aware synthesis tree explores alternative candidates and uses measured SLO gaps to guide each improvement. (2) A multi-fidelity verifier progressively combines low-cost checks with full target-device verification to reduce verification cost while preserving reliable verification results. It also records verified failures for reuse, avoiding repeated device work. To support concurrency, EdgeCraft provides a multi-tenant runtime that runs cloud training and target-device verification in parallel while isolating requests. Across 50 public tasks, EdgeCraft exceeds the task-specific Reference in best-observed quality on 40 tasks and finds an SLO-feasible artifact on 45, with the two outcomes overlapping on 38 tasks. Moreover, EdgeCraft achieves competitive performance on our self-collected SEN dataset, suggesting its generalizability to real-world IoT sensing tasks.

1 Introduction

Edge ML development is fragmented and scenario-dependent, requiring developers to navigate coupled design choices and competing SLOs. EdgeCraft automates this workflow through SLO-guided synthesis and progressive verification, with broad results across public tasks.

1 Introduction

  • Edge ML solutions require scenario-specific choices across task quality, latency, and energy because data representation, model, training, and runtime decisions are coupled.Source: §1 · p. 1
  • MCaaS maps user intent, a dataset, target device, and SLOs to a deployable artifact within a limited search budget, reporting the best verified candidate if none meets the SLOs.Source: §1 · p. 2
  • The system coordinates cloud training and device verification across requests and reuses compatibility rules derived from reproduced device–runtime failures.Source: §1 · p. 2
  • EdgeCraft uses a constraint-aware synthesis tree to steer revisions by measured SLO gaps and a multi-fidelity verifier that reserves feasibility decisions for full evaluation.Source: §1 · p. 2
  • Across 50 public tasks, EdgeCraft exceeds the task-specific Reference in best-observed quality on 40 and finds SLO-feasible artifacts on 45, with the outcomes overlapping on 38.Source: §1 · p. 2
  • Verification time falls 25.7% while preserving the selected artifact and its task quality; EdgeCraft produces SLO-feasible artifacts for 2.4× as many tasks as agentic approaches.Source: §1 · p. 3

2 Background and Motivation

Edge ML solutions depend on their task, device, and SLOs, while existing automation covers only parts of the path to deployment. Preliminary studies motivate selective, verification-grounded search, and MCaaS formalizes the goal as finding a high-quality feasible artifact within a bounded budget.

2.1 Edge ML Development Characteristics

  • Edge ML development remains fragmented: AutoML typically stops before device deployment and measurement, while LLM program synthesis does not reliably close the loop through target-device verification.End-to-end automation is proposed to connect these stages and return verification results to subsequent revisions.Source: §2.1 · p. 3

2.2 Preliminary Study

  • Across seven dataset–device–SLO scenarios, five different designs achieve the highest SLO-feasible quality, and no design wins more than twice.For ResNet-18, p95 latency differs by 122× across four device–runtime combinations, while highest measured power is 2.4× the lowest.Source: §2.2 · p. 3
  • All 81 LLM-generated candidates produce valid code, but only 15 produce the requested artifact, 10 execute on-device, and none yields evaluator-validated output or an SLO-feasible candidate.Two candidates report measurements within numerical SLO thresholds, but their outputs fail functional validation.Source: §2.2 · p. 4
  • Under a 20-trial budget, the LLM Prior achieves 80% Hit@10, versus 33% for Random and 30% for Size-Proxy, while measured device latency determines feasibility.The LLM Prior ranks by expected task quality; the Oracle achieves 100% Hit@10.Source: §2.2 · p. 4

2.3 Problem Formulation

  • MCaaS maps user intent, dataset, target device, and SLOs to a deployable artifact within a bounded budget, seeking the highest-quality valid candidate that satisfies the specified constraints.The request specifies a task-quality objective and applicable physical SLOs; if none satisfies every SLO, the service returns the best candidate and remaining gaps.Source: §2.3 · pp. 4–5

3 EdgeCraft Design

EdgeCraft separates candidate proposal from verification authority, using a synthesis tree to guide revisions and a staged verifier to control verification cost. Its shared services overlap cloud training with device execution and reuse reproduced compatibility failures across tenants.

3.1 Overview

  • The LLM selects a live branch and proposes candidates, while P0–P2 verification results determine rejection or acceptance and feed subsequent tree expansions.Only P2 can accept a candidate; reliable P0 or P1 results can prune one.Source: §3.1 · p. 5

3.2 Constraint-Aware Synthesis Tree

  • Each tree branch tracks quality progress and signed SLO gaps, so revisions can target unmet constraints while retaining slack and verification history.A positive gap means a constraint is unmet; a negative gap is slack, which can leave room for a higher-quality child candidate.Source: §3.2 · p. 6

3.3 Multi-Fidelity Verifier

  • The verifier progresses from static P0 checks to target-device P1 measurements and full-training P2 verification; only P2 can accept an artifact.P1 may prune before full training only with a matching calibration context and when its conservative lower bound still exceeds an upper-bounded physical SLO; data-dependent quality proceeds to P2.Source: §3.3 · pp. 6–7

3.4 Cross-Tenant Shared Services

  • Cross-tenant services overlap cloud training and device execution, while reproduced device–runtime failures become compatibility rules that can prevent repeated failures and device work.Rules are reused only while their runtime or driver context remains current; a change leaves a rule stale pending reproduction.Source: §3.4 · pp. 7–8

4 Evaluation

Across 50 public tasks, EdgeCraft improves best-observed quality over the task-specific Reference on 40 and finds an SLO-feasible artifact on 45. Its evaluation also examines agentic and AutoML/NAS comparisons, design ablations, sensitivity, SEN generalizability, and capability boundaries.

4.1 Overall Performance

  • 40 of 50 tasks show better best-observed quality than the Reference, and 45 of 50 yield an SLO-feasible artifact.The SLOs set latency and, when applicable, energy limits at 0.8× the corresponding Reference measurements.Source: §4.1 · pp. 8–9, Figure 8 · p. 9
  • EdgeCraft reaches 45 SLO-feasible tasks versus 19 for MetaGPT-Verification, or 2.4× as many, and reaches 19 in 5.9 minutes versus 62.8 minutes.Among 16 tasks where both produce feasible artifacts, EdgeCraft has higher quality on 15 and ties on one.Source: §4.1 · p. 9, Figure 9 · p. 9
  • EdgeCraft achieves the highest task quality in 12 of 20 comparisons with supported AutoML/NAS frameworks.Each framework is evaluated on five datasets aligned with its native support; latency and energy SLO satisfaction are not part of this comparison.Source: §4.1 · pp. 9–10, Figure 10 · p. 9
  • At 8 requests/hour, the multi-tenant runtime reduces mean queueing delay by 24.9% and p95 completion time by 14.6% relative to Tenant RR.The comparison uses identical request arrivals and work across scheduling policies.Source: §4.1 · p. 10, Figure 11 · p. 10

4.2 Ablation Study

  • Constraint-Aware yields the largest feasible-quality gain on four of five tasks and matches or exceeds alternatives in SLO-feasible improvement rate on all five.Its expansions improve the feasible incumbent 10–30% of the time, versus 0–20% for the other search strategies.Source: §4.2 · p. 10, Figure 12 · p. 10
  • Multi-Fidelity reduces total verification time from 69.13 to 51.35 minutes (25.7%) and selects the same artifact with the same task quality in all four runtimes.The fast check is limited to graph-preserving candidates with completed calibration for the device, runtime, and execution setting; all 60 early rejections were confirmed by full verification.Source: Figure 13 · p. 10, §4.2 · p. 10

4.3 Sensitivity Analysis

  • From 4 to 24 candidates, best verified SLO-feasible quality improves on every task, averaging 11.3% and reaching 31.7%.At 0.4× and 0.3× SLO limits, 34 and 32 of 42 tasks remain feasible, retaining 79.1% and 73.1% of aggregate task-normalized quality.Source: §4.3 · pp. 10–11, Figure 15 · p. 11

4.4 Generalizability

  • On the SEN sample-level split, EdgeCraft achieves the highest mean AUROC at 0.8953 ± 0.0180; on subject-disjoint data, its 0.5683 ± 0.0092 AUROC nearly matches MiniRocket’s 0.5697 ± 0.0044.Selected artifacts satisfy the shared Raspberry Pi 5 limits of p95 latency ≤0.4 ms and energy ≤2.8 mJ per inference across all three runs.Source: §4.4 · p. 11, Table 3 · p. 11

5 Discussion

EdgeCraft targets workloads where compact models, data representations, or runtime optimizations can preserve quality under physical SLOs, but it does not guarantee a global optimum. The authors also note possible public-dataset pretraining exposure and include a self-collected SEN dataset unavailable during pretraining.

Scope and optimality

  • EdgeCraft returns the highest-quality artifact fully verified within its candidate budget, not a guaranteed global optimum.Data collection, labeling, and task definition remain outside its automation pipeline; its boundary arises when quality depends on a high-capacity backbone that cannot meet physical SLOs.Source: §5 · p. 12

Public-benchmark exposure

  • LLM pretraining exposure to public benchmarks cannot be ruled out, while the self-collected SEN dataset was unavailable during pretraining.The authors use diverse public tasks and SEN to reduce the influence of that exposure concern and assess performance beyond public benchmarks.Source: §5 · p. 12

6 Related Work

Related work includes managed low-code platforms, AutoML/HPO/NAS, coding agents, and LLM-driven IoT synthesis. EdgeCraft differs by searching complete edge ML programs and grounding decisions in deployment evidence from the assigned device.

Managed platforms and AutoML/NAS

  • Unlike managed ML platforms, which leave users to choose coupled modeling and deployment choices, EdgeCraft automates them from high-level intent and returns a complete ML artifact.AutoML and NAS use prescribed search spaces and common candidate interfaces, whereas EdgeCraft searches programs whose pipelines, models, training, export, and runtimes may differ.Source: §6 · p. 12

Coding agents and IoT synthesis

  • EdgeCraft grounds each branch in measured task quality, latency, energy, and runtime status on the assigned device, and completes a task only when deployment requirements are met.The related IoT synthesis systems are described as program-centric, while EdgeCraft focuses on automated edge-model development and deployment.Source: §6 · pp. 12–13

7 Conclusion

The conclusion presents EdgeCraft as verification-grounded edge ML synthesis combining constraint-aware LLM search, progressive target-device verification, reusable compatibility rules, and multi-tenant execution.

Conclusion

  • Across 50 tasks, EdgeCraft combines broad development capability and competitive quality with lower verification time and token use.Source: §7 · p. 13

A Public Benchmark

Table 4 covers edge-oriented on-device AI tasks and reports quality, latency, and energy for each frozen Reference.

A Public Benchmark

  • Table 4 benchmarks edge-oriented on-device AI tasks against frozen References.The table reports task quality (Q), p95 latency in ms (L), and energy per inference in mJ (E).Source: Table 4 · p. 18
  • The table reports task quality as Q and p95 latency in milliseconds as L.Source: Table 4 · p. 18
  • Energy constrains the search for 12 tasks with an application-level energy SLO; “–” marks tasks without one.Source: Table 4 · p. 18

B Benchmark and Evaluation Details

EdgeCraft is evaluated against task-matched frozen References under uniformly tighter physical SLOs. It records quality wins and SLO-feasible artifacts across all three Reference families, with measurements and evaluation protocols also reported for device metrics, held-out tests, SEN, and a linear-search baseline.

B Benchmark and Evaluation Details

  • Latency and active-energy SLOs are fixed at 0.8× each task’s Reference measurement, making the physical budgets 20% tighter than the baseline.The 0.8× targets are experimental stress targets, not universal application deadlines.Source: B Benchmark and Evaluation Details · p. 19
  • EdgeCraft achieves quality wins on 11/12 classical, 25/32 compact learned, and 4/6 specialized pretrained Reference tasks, and SLO feasibility on 10/12, 29/32, and 6/6, respectively.The results span the full Reference portfolio rather than concentrating in one model family.Source: B Benchmark and Evaluation Details · p. 19
  • Latency is reported as p95 over repeated post-warm-up invocations, while energy per inference uses mean active power, timed duration, and repetition count.Energy measurements use aligned hardware-power samples; total energy is the conservative primary metric and idle-subtracted dynamic energy is auxiliary.Source: B Benchmark and Evaluation Details · p. 19
  • Search uses validation feedback, freezes train/validation/test partitions before experiments, and reports final performance on held-out test data when available.Source: B Benchmark and Evaluation Details · p. 19
  • All six SEN artifacts satisfy both physical SLOs; mean p95 latency is 0.251/0.352 ms and mean energy is 1.052/1.581 mJ for sample-level/cross-subject settings.Each setting uses three independent syntheses, and validation selects the artifact evaluated on the held-out test partition.Source: B Benchmark and Evaluation Details · p. 19
  • MetaGPT-Verification retains MetaGPT’s native linear workflow while receiving the same target-device evidence and 24-candidate cap as EdgeCraft.Tree control and cross-request rule reuse are the variables introduced by EdgeCraft.Source: B Benchmark and Evaluation Details · p. 19

C Multi-Fidelity Verification and Failure Reuse

Multi-fidelity verification gates fast rejection on a matching prior full verification, while reproduced device–runtime failures become narrowly matched reusable rules. Workload or platform shifts reduce rule matches rather than broadening existing rules.

C Multi-Fidelity Verification and Failure Reuse

  • Fast measurement may reject only after full verification covers the same model graph, device, runtime, precision, input profile, and measurement setup; a new or changed setting requires full verification.All 60 early rejections agreed with full verification, with a one-sided exact 95% upper bound of 4.87% on disagreement.Source: C Multi-Fidelity Verification and… · p. 20
  • A failure rule requires a reproduced device–runtime incompatibility and skips candidates only when its failure and artifact-context fields match.The 50 rules cover four devices and three runtimes; 25 also match artifacts from a different dataset.Source: C Multi-Fidelity Verification and… · p. 20
  • Table 5 presents evaluation scope and aggregate outcomes for the two verification mechanisms.Source: Table 5 · p. 20
  • A workload, device, or runtime shift reduces existing-rule matches without broadening their scope; unmatched candidates run on-device and can establish rules for the new setting.Source: C Multi-Fidelity Verification and… · p. 20
Loading 2609.35167v1…