PaperlayerView exact revision on arXiv

Source-linked AI summary

AgentDP: Towards Automated Model Deployment for Heterogeneous Embedded AI Applications

Leming Shen, Qiang Yang, Xinyu Huang, Zijing Ma, Yuanqing Zheng

arXiv:2610.30760v1cs.AI

TL;DR

Open-source models can be difficult to optimize and adapt for heterogeneous embedded devices because of resource constraints, mismatched execution environments, and laborious tuning. AgentDP uses collaborative LLM agents and specialized modules to automate O&A. It reports up to 61% less O&A time than manual execution by experts, alongside improved task accuracy and resource consumption against baselines.

  • Problem

    Open-source models often face embedded-device resource constraints and incompatible execution environments, while their O&A requires extensive hyperparameter search and device-specific configuration.

    Source: §1 · pp. 1–2

  • Method

    AgentDP uses collaborative LLM agents to automate model optimization and adaptation, with modules for requirement standardization, operator-aware partitioning, and configuration reuse.

    Source: §1 · p. 2

  • Results

    AgentDP reduced O&A time by up to 61% versus manual execution by experts, while task accuracy exceeded 13% and resource consumption was below 14% on average against baselines.

    Source: §1 · p. 2

  • Takeaways & Limitations

    The authors conclude AgentDP can support practical deployment of open-source AI models across diverse and distributed embedded devices.

    Source: §9 · p. 12

Abstract

from arXiv · show

This paper presents AgentDP, which aims to deploy pre-trained AI models on heterogeneous embedded devices by fully automating model optimization and adaptation (O&A) with Large Language Model (LLM) agents. AgentDP not only significantly reduces human effort via automation but also enhances model inference efficacy by integrating LLMs with device-specific domain knowledge. To deliver a robust automation framework, AgentDP develops novel solutions to address three key technical challenges: 1) To help users elicit clear requirements, we propose a user requirement standardization module; 2) To mitigate the execution environment mismatch between AI models and heterogeneous embedded devices, we design an operator-aware model partition paradigm that enables different parts of the model to be efficiently executed on distinct inference engines. 3) To avoid re-searching optimization hyperparameters for new tasks, we propose a configuration reuse module that heuristically retrieves previous hyperparameter configurations for efficient reuse. Extensive experiments demonstrate that by fully automating the entire O&A process, AgentDP reduces entire model adaptation time by up to 80% compared with manual O&A while achieving higher task accuracy (more than 13% on average) than the baselines. Additionally, a comprehensive user study also exhibits the robust generalizability of AgentDP in adapting to different user requirements for heterogeneous embedded devices.

1 INTRODUCTION

AgentDP uses collaborative LLM agents to automate model optimization and adaptation for heterogeneous embedded devices. It addresses user-requirement ambiguity, model–engine mismatch, and repeated hyperparameter search, and reports faster adaptation with improved accuracy and resource consumption.

Motivation

  • Manual O&A is burdensome because developers must search large hyperparameter spaces, configure device-specific engines, and iteratively refine models.Open-source models may also exceed device resources or use formats unsupported by heterogeneous embedded environments.Source: §1 · pp. 1–2

Technical approach

  • AgentDP standardizes user requirements, partitions models across inference engines, and reuses configurations from previous models with similar resource profiles.The reuse approach models tuning as a finite state machine and retrieves configurations from a graph database.Source: §1 · p. 2

Evaluation overview

  • AgentDP reduced O&A time by up to 61% versus manual execution by experts, while task accuracy exceeded 13% and resource consumption was below 14% on average against baselines.In a user study, overall satisfaction was over 4.8 out of 5.Source: §1 · p. 2

System contribution

  • Collaborative LLM agents automate the O&A process from open-source model optimization through embedded application deployment.The system integrates device-specific domain knowledge to support this end-to-end automation.Source: §1 · p. 2

2 BACKGROUND & MOTIVATION

Embedded AI O&A is difficult because model optimization, device-specific execution environments, and repeated refinement demand expertise. AgentDP proposes collaborative, tool-enabled LLM agents to automate this process.

2.1 Challenges in Embedded AI

  • Embedded O&A requires expertise to select optimization hyperparameters and inference engines, while unsupported operators can force models into less-efficient CPU fallback.Resource priorities can vary by deployment, and no representative engine fully supports all operators in the Mask R-CNN example.Source: §2.1 · pp. 2–3, Figure 2 · p. 3

2.2 LLM Agents

  • LLM agents can use third-party tools such as web search and knowledge databases to automate complex workflows with enriched external knowledge.The section also describes multiple agents collaborating by decomposing complex tasks into subtasks.Source: §2.2 · p. 3

2.3 Agent Meets O&A

  • AgentDP orchestrates tool-enabled agents to automate O&A end to end and uses information retrieval and requirement elicitation to support reliability.Its stated efficiency scope runs from hyperparameter tuning and execution-environment configuration to model deployment.Source: §2.3 · p. 3

3 SYSTEM OVERVIEW

AgentDP coordinates four specialized agents to carry out model optimization and adaptation for embedded deployment. Users provide a model, target device, and real-time performance requirements; inference and evaluation occur on the target device.

Deployment setup

  • AgentDP uses cloud LLMs, runs on an edge server, and accesses the target device over a local network.Model inference and evaluation take place on the target device, and only inference results are transmitted to AgentDP.Source: §3 · pp. 3–4

User input

  • Users provide the model name, target device, and real-time performance requirements for deployment.An example requirement specifies RAM below 3GB, throughput of at least 5 images per second, and accuracy of at least 65%.Source: §3 · p. 4

Supporting tools

  • The system provides web search, vector and graph databases, a sandboxed Python executor, and a document loader to its agents.Agents share model and device profiles, standardized requirements, configurations, and evaluation results.Source: §3 · p. 4

Agent workflow

  • The four-agent workflow profiles the model and device, standardizes requirements, tunes hyperparameters, adapts models, and reuses prior configurations.The target device returns real-time performance results to HTA as feedback for another optimization round.Source: §3 · p. 4, Figure 4 · p. 4

4 SYSTEM DESIGN

AgentDP coordinates agents to standardize deployment requirements, tune models, adapt them across heterogeneous inference engines, and reuse configurations from similar prior models. Its design combines iterative user clarification, automated optimization and adaptation, and retrieval-based configuration reuse.

4.1 User Interface Agent (UIA)

  • UIA iteratively refines natural-language requirements by detecting ambiguity and asking follow-up questions, producing structured metric-and-value requirements.Requirements cover resource constraints and target-device performance metrics.Source: §4.1 · p. 4, §4.1.2 · p. 5

4.2 Hyperparameter Tuning Agent (HTA)

  • HTA identifies tunable hyperparameters, generates search ranges from model profiles and resource requirements, then searches for configurations satisfying resource constraints.It filters out architecture-changing hyperparameters and synthesizes scripts to tune and generate portable models.Source: §4.2 · pp. 5–6

4.3 Model Adaptation Agent (MAA)

  • OAMP partitions portable models across compatible inference engines, assigning operators by availability and latency while accounting for context-switching costs.Unsupported operators are assigned to a fallback environment such as the CPU.Source: §4.3.1 · p. 6, §4.3 · p. 6
  • CEIP runs dependent model partitions sequentially or concurrently and uses shared memory, DMA, and prefetching to reduce transfer and switching overhead.Partitions are loaded onto their assigned engines with consistent memory and data alignment.Source: §4.3.2 · p. 7

4.4 Configuration Reuse Agent (CRA)

  • CRA models tuning histories as FSMs, retrieves those with similar initial resource profiles, and uses them to narrow hyperparameter search spaces.It stores system profiles and hyperparameters separately; BFS and DFS over retrieved FSMs help identify search bounds.Source: §4.4.2 · pp. 7–8, §4.4 · p. 7
  • CRA reduces model optimization time by up to 74% through automated access to its graph database and analysis of FSMs.Source: §4.4.2 · p. 8

5 EXPERIMENT SETUP

AgentDP is implemented as a multi-agent system with tools and inference engines for model optimization and deployment. It is evaluated on three modalities, models, and embedded-device targets.

5.1 Implementation

  • AgentDP uses LangChain agents, search tools, vector and graph databases, a sandboxed Python executor, and a remote-control interface.GPT-4o serves as the backend LLM; the cross-engine inference pipeline includes ONNX Runtime, TensorRT, OpenVINO, TVM, and MLIR.Source: §5.1 · p. 8

5.2 Embedded AI Applications & Datasets

  • The evaluation covers ResNet-152 image recognition on Raspberry Pi, fine-tuned Whisper speech sentiment analysis on Jetson Nano, and DeepSeek-VL-1.3b on an embedded mini PC.The applications use ImageNet, TESS, and MMMU, respectively; Whisper and DeepSeek-VL include operators not fully supported by any single inference engine.Source: §5.2 · p. 8

6 EVALUATION

AgentDP is evaluated against original models, MapCoder, and human developers across three applications, with metrics covering resource use, task performance, speed, and O&A time. Results include application-specific performance and efficiency comparisons, ablations, and user-study ratings.

6.1 Metrics

  • Evaluation metrics include maximum RAM and GPU memory, model size, task accuracy, throughput or tokens per second, and end-to-end wall-clock O&A time.IR and SSA use accuracy, precision, recall, and F1; VLU uses MMMU micro-averaged accuracy.Source: §6.1 · pp. 8–9

6.3 Image Recognition Evaluation

  • 13.8% higher average accuracy and 14.5% lower average RAM use were reported for AgentDP in image recognition.AgentDP completed O&A in 3.3 hours, compared with about 6 hours spent on hyperparameter tuning by expert U1.Source: Figure 13 · p. 9, §6.3.2 · p. 9

6.2 Baselines

  • Baselines comprise original models, MapCoder as a programming-agent baseline, and 15 developers grouped by deployment and model-optimization experience.The human evaluation process received institutional ethics-board approval.Source: §6.2 · p. 9

6.4 Speech Sentiment Analysis Evaluation

  • Only AgentDP met both SSA resource and performance requirements; U1 throughput was 49% lower, while AgentDP completed O&A in 17 hours versus 60 and 80 hours for U2 and U3.AgentDP used operator-aware partitioning and engine switching; U2 and U3 fell back to CPU execution after operator mismatches.Source: Figure 14 · p. 10, §6.4.2 · p. 10

6.5 VLU Evaluation

  • Up to 80% lower O&A time than human baselines was reported for VLU, although AgentDP’s overall accuracy was slightly below expert U1.U1 fine-tuned the model using additional VLU datasets, including MMBench.Source: Figure 15 · p. 10, §6.5.2 · pp. 10–11

6.6 Ablation Study

  • Ablations show URS removal drops accuracy around 10% and lengthens tuning; CRA reduces optimization time by up to 74% versus searching from scratch as utilization rises; removing OAMP/CEIP drops DeepSeek-VL speed from 15.6 to 4.7 TPS.Without OAMP and CEIP, unsupported operators require CPU execution.Source: §6.6.1 · p. 11, §6.6.2 · p. 11, §6.6.3 · p. 11

6.7 User Study for Generalizability Evaluation

  • Experts and semi-experts rated AgentDP above 4.6 for overall model performance, code readability, and user satisfaction; both efficiency ratings rose with long-term use.Semi-experts scored optimization and deployment efficiency higher than experts, while optimization-efficiency ratings increased faster as use accumulated.Source: §6.7.2 · pp. 11–12, Figure 17 · p. 12, §6.7.3 · p. 12

7 DISCUSSION

The discussion identifies intermediate-result errors and the separation between hyperparameter tuning and device-side performance measurement as reliability and efficiency concerns. It outlines additional techniques for future integration.

7 DISCUSSION

  • Reflexion lets agents learn iteratively from failed attempts, while integrating ReAct is proposed as future work to improve error detection and recovery.The paper describes incorrect intermediate results as a source of propagated errors in multi-step reasoning.Source: §7 · p. 12
  • Because HTA measures resource use but not inference latency, it depends on MAA feedback across O&A rounds; real-time prediction models are proposed to narrow tuning searches.HTA presents users with applicable configurations that satisfy their requirements.Source: §7 · p. 12

8 RELATED WORK

Related work covers embedded-model deployment and LLM agents. AgentDP is positioned as automating both optimization and adaptation for heterogeneous embedded devices.

Model Deployment on Embedded Devices

  • Prior embedded deployment work optimizes for constrained compute, memory, and energy, but tuning remains trial-and-error and deployment frameworks are not tailored to heterogeneous environments.CPU fallback, new cross-platform frameworks, and containers each have limitations described for embedded deployment.Source: §8 · p. 12

LLM Agents

  • Unlike general-purpose programming agents such as MapCoder, AgentDP automates model optimization and adaptation while addressing model–engine mismatch for embedded devices.Prior LLM-agent work includes multi-agent collaboration for complex tasks.Source: §8 · p. 12

9 CONCLUSION

AgentDP is a multi-agent system for automated model optimization and adaptation on heterogeneous embedded devices. The paper reports improved efficiency and model performance, with user studies demonstrating generalizability across models, devices, and user requirements.

9 CONCLUSION

  • AgentDP uses four specialized agents to address three technical challenges in model optimization and adaptation for heterogeneous embedded AI applications.The paper describes the system as a multi-agent automatic model optimization and adaptation system.Source: §9 · p. 12
  • User studies demonstrate AgentDP’s high generalizability in adapting open-source models to heterogeneous embedded devices with distinct user requirements.Source: §9 · p. 12
  • AgentDP significantly enhances optimization and adaptation efficiency while improving model performance.Source: §9 · p. 12
  • The authors believe AgentDP can support practical deployment of open-source AI models across diverse, distributed embedded devices in everyday applications.Source: §9 · p. 12

Related papers

Loading 2610.30760v1…