Source-linked AI summary
From PyTorch to the NPU: LLM-Agent-Driven Model Conversion Across Heterogeneous Inference Runtimes
Jianhao Su, Zhanwei Wu, Chia-Heng Tu, ShengTing Huang
TL;DR
Heterogeneous inference runtimes impose differing toolchains and deployment constraints, while existing tools lack a uniform framework for the engineering workflow. The paper extends AIPC with runtime-specific skills and verification loops across four runtimes; experiments report deployment feasibility for regular vision models on most runtimes, alongside more frequent agent deviations for complex models.
Problem
Heterogeneous runtimes differ in toolchains and constraints, while existing toolchains lack a uniform framework for repetitive, error-prone deployment work.
Source: §1 · p. 1, §2.8 · p. 4Method
The paper extends AIPC to four runtimes using runtime-specific skills, auxiliary functions, staged verification, and an ONNXWrapper for layout adaptation.
Source: §1 · p. 1, §2.8 · p. 4, §6 · p. 13Results
Structurally regular vision models were successfully deployed on most runtimes, while complex multi-encoder models showed more agent deviations and general knowledge alone was insufficient for reliable handling.
Source: §1 · p. 2, §6 · p. 13Takeaways & Limitations
The reported feasibility is preliminary and bounded to single-model-to-single-runtime deployment, with ONNXWrapper removing manual layout adaptation from the agent’s repair burden.
Source: §6 · p. 13Experiments cover only Real-ESRGAN and YOLO-World, and case-by-case operator-repair observations do not establish cross-runtime best practices.
Source: §5.5 · pp. 12–13
Abstract
from arXiv · showhide
Edge AI model deployment is a multi-stage engineering process involving model conversion, operator compatibility handling, runtime integration, and precision verification. While prior work has demonstrated agent-based automation for Qualcomm AI Runtime, the broader edge inference runtime ecosystem, including Intel OpenVINO, Rockchip RKNN, NVIDIA TensorRT, and ONNX Runtime, presents distinct toolchains and optimization strategies. This paper extends AIPC (AI Porting Conversion, an LLM agent-driven methodology for AI model deployment automation previously demonstrated on Qualcomm AI Runtime) to multi-runtime scenarios, proposing an LLM agent-driven approach for automated single-model-to-single-runtime deployment across heterogeneous inference backends, such as Intel OpenVINO, Rockchip RKNN, NVIDIA TensorRT, and ONNX Runtime. We decompose the edge AI deployment into standardized, verifiable stages, and inject runtime-specific domain knowledge into the agent execution flow through agent skills, auxiliary scripts, and staged verification loops. Using representative vision models, we demonstrate that agent-based deployment can complete the conversion from a PyTorch model to its executable inference, targeting OpenVINO for x86/NPU, RKNN for RK3588, TensorRT for NVIDIA GPU, and ONNX Runtime for Qualcomm NPU with a focus on FP16 precision deployment feasibility verification. The contributions of this paper primarily lie in providing multi-runtime deployment engineering practice experience, toolchain mapping analysis, a layout-adaptation and inference-replacement layer that removes manual transpose insertion from the agent's repair burden, and an empirical characterization of agent deviation behavior under structured knowledge injection, rather than large-scale systematic benchmarking or cross-runtime operator repair strategy comparison.
1 INTRODUCTION
Heterogeneous runtimes impose distinct deployment constraints, leaving repetitive, error-prone engineering work that existing toolchains do not address through a uniform automation framework. This paper extends AIPC across four runtimes and examines whether structured agent knowledge and verification can support repeatable deployment and bounded repair.
Motivation
- Deployment spans ONNX conversion, operator substitution, shape resolution, graph repair, runtime compilation, I/O adaptation, and precision verification, requiring runtime-specific knowledge.Existing toolchains address format conversion individually but lack a uniform framework for these challenges across runtimes.Source: §1 · p. 1
Research question
- The paper asks whether agents supplied with domain knowledge can orchestrate multi-runtime deployment and perform bounded repair after conversion failures or runtime exceptions.The framing emphasizes repeatable, verifiable workflows rather than assuming LLMs understand every hardware detail.Source: §1 · p. 1
Contributions
- The paper extends AIPC from QAIRT to OpenVINO, RKNN, TensorRT, and ONNX Runtime QNN as a preliminary test of the “skill + verification loop” across toolchains.It also maps conversion modes, output formats, and toolchain characteristics across five runtimes, including QAIRT.Source: §1 · p. 1
- ONNXWrapper loads the corresponding hardware runtime model for precision comparison and adapts NCHW ↔ NHWC layouts at the ONNX level.This removes per-model manual transpose insertion and a class of layout-induced conversion failures.Source: §1 · pp. 1–2
- Agents deviated from prescribed workflows both with and without structured domain-knowledge injection, highlighting the need for carefully designed verification mechanisms.The paper reports a controlled comparison of deployments under the two conditions.Source: §1 · p. 2
2 BACKGROUND
The background describes heterogeneous hardware runtimes and the common deployment flow from a source model through conversion to verification. Runtime constraints vary, especially across precision support and NPU flexibility.
2 BACKGROUND
- The background section covers the common deployment pipeline, runtime toolchains, agent-driven deployment work, and cross-runtime challenges motivating the study..Source: §2 · p. 2
2.1 Hardware Inference Runtime Overview
- Native CPU, GPU, and NPU runtimes are optimized for hardware characteristics, while differing in toolchains, operator support, quantization, APIs, and precision support.NPU devices are described as particularly restrictive relative to CPUs and GPUs.Source: §2.1 · p. 2
- The typical pipeline checks the original model, exports it to ONNX, converts it with target-runtime tools, and verifies output shapes and precision before later optimization.Quantization may be built into conversion; optimization and performance analysis can follow once deployment stabilizes.Source: §2.1 · p. 2
2.2 Intel OpenVINO
OpenVINO and RKNN present distinct hardware and model constraints, including plugin-dependent operator support, device-specific dynamic-shape limits, and strict RKNN shape, layout, and quantization requirements.
2.2 Intel OpenVINO
- OpenVINO supports Intel CPUs, integrated GPUs, and NPUs, with operator and dynamic-shape support varying across device plugins.NPUs have the most limited operator set and may require workarounds for dynamic shapes; memory constraints can require partitioning or subgraph offloading.Source: §2.2 · p. 2
2.3 Rockchip RKNN
- RKNN operator support and precision depend on the chip, and limited operator coverage may require custom implementations or CPU fallback.Deployment must also account for strict tensor-shape, NHWC-layout, and quantization-parameter constraints.Source: §2.3 · p. 2
2.4 NVIDIA TensorRT
TensorRT offers ONNX-based and direct PyTorch deployment paths, with platform-specific engine and operator constraints. The background figure describes a general edge deployment flow with repair and verification paths when stages fail.
2.4 NVIDIA TensorRT
- TensorRT supports ONNX-to-TensorRT compilation and direct Torch-TensorRT compilation of PyTorch models into optimized subgraphs.The direct path can support mixed deployment with partial PyTorch-native execution.Source: §2.4 · p. 2
- TensorRT engines require recompilation for the target GPU architecture, and unsupported ONNX operators may require plugins or graph rewriting.Dynamic dimensions also require minimum, optimal, and maximum values in an Optimization Profile; inputs outside that range cannot execute.Source: §2.4 · pp. 2–3
- The deployment figure distinguishes forward-flow arrows from repair and verification paths exercised when a stage fails.These paths frame deployment as staged execution with recovery, rather than export alone.Source: Figure 1 · p. 3
2.5 ONNX Runtime QNN
ONNX Runtime supports hardware backends through Execution Providers, with QNN EP connecting ONNX models to Qualcomm Hexagon NPUs. The deployment uses Olive’s QNNPreprocess pass for QNN compatibility rather than its broader recipe pipeline.
2.5 ONNX Runtime QNN
- ONNX Runtime’s Execution Provider architecture supports multiple hardware backends, and QNN EP integrates it with QAIRT for Qualcomm Hexagon NPU inference.ONNX models can run on Qualcomm devices including smartphones, notebooks, and IoT platforms.Source: §2.5 · p. 3
- QNN EP operator coverage depends on QNN backend support, unsupported operators may fall back to CPU, and models require fixed input dimensions.Compatibility between QAIRT SDK and ONNX Runtime versions also requires careful management.Source: §2.5 · p. 3
- Olive provides declarative optimization recipes for FP16 conversion, static quantization, graph optimization, QNN Context Binary generation, and runtime tuning.Its recipe approach emphasizes reproducibility and version traceability.Source: §2.5 · p. 3
- Olive’s QNNPreprocess pass replaces or decomposes unsupported operators into supportable subgraphs, reducing the likelihood of CPU fallback on HTP.The pass is used instead of Olive’s recipe pipeline; later Olive passes can be added for calibration, graph optimization, or performance tuning.Source: §2.5 · p. 3
2.6 Qualcomm AI Runtime
QAIRT provides SNPE and QNN execution frameworks for Qualcomm inference. Their deployment constraints vary by target SoC, platform, toolchain, and input-shape requirements.
2.6 Qualcomm AI Runtime
- QAIRT is Qualcomm’s AI inference SDK, with SNPE using DLC for cross-backend execution and rapid integration, while QNN uses platform-specific Context Binary generation for production deployment.QNN is described as closer to the hardware execution layer.Source: §2.6 · p. 3
- QNN HTP does not support dynamic shapes, so all input dimensions must be fixed before deployment.ARM/Linux conversion typically requires development and compilation on x64/Linux hosts, while ARM64/Windows conversion requires x64 emulation.Source: §2.6 · p. 3
- Precision and operator support vary with the target Snapdragon SoC, and Context Binary generation depends closely on the toolchain and system environment.QAIRT SDK and ONNX Runtime QNN EP are independent deployment paths that share the QNN library but differ in pipeline orchestration and runtime integration.Source: §2.6 · p. 3
2.7 Related Work for AI Agent Deployment
Prior work includes agent-based optimization and deployment, but the paper identifies end-to-end conversion retargetable across hardware platforms as an uncovered area. AIPC addresses this through conversion workflows, operator handling, layout adaptation, and staged verification.
2.7 Related Work for AI Agent Deployment
- Huawei Ascend’s agent-skills repository uses a skill-plus-verification architecture for structured PyTorch-to-ONNX-to-.om conversion, and the paper describes this practice as supporting the pattern’s generality.The .om format is Ascend’s native inference format.Source: §2.7 · pp. 3–4
- Related agent-based hardware work mostly targets subsystem optimization, including operator and kernel performance, while Li et al. [6] describe a two-stage agent-skill system for AMD XDNA 2 NPU deployment of multiple LLMs.The described system distills human expertise into structured documentation.Source: §2.7 · p. 4
- The paper identifies no prior cited work that handles complete PyTorch-to-ONNX-to-target-format conversion with a deployment pipeline retargetable across hardware platforms.Ascend agent-skills is described as one of the few agent-driven end-to-end conversion efforts, but it covers one runtime.Source: §2.7 · p. 4
- AIPC systematizes conversion from PyTorch through ONNX to multiple target formats and explores a unified deployment methodology across heterogeneous runtimes.The approach includes operator compatibility handling, layout adaptation, and staged verification loops.Source: §2.7 · p. 4
- AIPC handles unsupported operators through substitution, decomposition, or CPU fallback strategies, and automatically adapts NCHW↔NHWC layouts at the ONNX level through ONNXWrapper.Source: §2.7 · p. 4
2.8 Cross-Runtime Deployment Challenges and Motivation
Heterogeneous runtimes impose distinct operator, layout, shape, and quantization constraints, making deployment repetitive and error-prone. The paper motivates runtime-specific agent knowledge and staged verification as a way to automate this work within constrained workflows.
2.8 Cross-Runtime Deployment Challenges and Motivation
- The described deployment challenge set spans runtimes and hardware with distinct toolchains and APIs.Source: §2.8 · p. 4
- Unsupported ONNX operators require runtime-specific substitution or graph rewriting because support and replacement strategies differ across runtimes.Source: §2.8 · p. 4
- NCHW-to-NHWC layout conflicts can cause conversion failures or silent performance degradation.Source: §2.8 · p. 4
- Dynamic model dimensions often conflict with edge runtimes’ preference for fixed-shape compilation, and resolving them frequently requires manual intervention.Resolution requires understanding model semantics and hardware constraints.Source: §2.8 · p. 4
- Quantization stability depends on calibration-data representativeness, preprocessing consistency, and environmental complexity such as coexisting toolchain versions.Source: §2.8 · p. 4
- The paper proposes encapsulating runtime knowledge in skill modules and using staged verification loops to automate common deployment challenges within constrained workflows.The motivation is that deployment remains substantial, repetitive, domain-knowledge-intensive, and error-prone despite individual toolchains addressing format conversion.Source: §2.8 · p. 4
3 AGENTIC DEPLOYMENT METHODOLOGY
AIPC uses an LLM agent with runtime-specific skills and staged verification to convert a PyTorch model into executable inference for one target runtime. Its workflow combines a common deployment pipeline with repair and inference-replacement mechanisms.
3 AGENTIC DEPLOYMENT METHODOLOGY
- An agent combines an LLM backend, tool-calling interface, and skill modules that encode runtime-specific deployment knowledge.Skills include toolchain commands, operator compatibility information, and common failure patterns.Source: §3 · p. 4
3.1 The “Skill + Verification Loop” Design Pattern
- AIPC divides deployment into six verifiable stages, from model preparation and ONNX export through runtime construction, precision alignment, optional optimization, and reporting.Optimization and quantization begin only after the first four stages are stable.Source: §3.1 · pp. 4–5
- The “skill + verification loop” wraps the pipeline with runtime-specific knowledge, deterministic auxiliary operations, stage-level pass/fail rules, and recoverable repair actions.Repairs can target PyTorch source code, ONNX graph nodes and attributes, or runtime compatibility, with fallback to earlier stages when needed.Source: §3.1 · p. 5
3.2 Agent Skills
- The methodology’s agent skills supply runtime-specific commands, compatibility knowledge, and failure fixes, while auxiliary scripts and verification rules support the staged workflow.Table 1 lists the skill modules required by the methodology.Source: §3.1 · p. 5, §3.2 · p. 5, Table 1 · p. 5
3.1 The “Skill + Verification Loop” Design Pattern
- ONNXWrapper replaces ONNX Runtime in inference scripts, loads the corresponding hardware-runtime model, adapts tensor layouts, and compares outputs without changing underlying code or regenerating target programs.The comparison uses ONNX Runtime reference outputs for precision checking.Source: §3.1 · p. 5
3.3 Single-Model Single-Runtime Orchestration
- For each independent model-to-runtime task, the agent identifies the target, exports ONNX, invokes its runtime converter, verifies and repairs results, and outputs files loadable by that runtime.Deployments do not use cross-runtime parallel processing or shared repairs.Source: §3.3 · p. 5
4 RUNTIME-SPECIFIC DEPLOYMENT PIPELINES
Section 4 adapts AIPC’s deployment workflow to runtime-specific toolchains, which differ in conversion, quantization, and layout requirements. ONNXWrapper handles layout adaptation without changing the original ONNX inference logic.
4 RUNTIME-SPECIFIC DEPLOYMENT PIPELINES
- AIPC skills are adapted and verified for one target runtime at a time, with failed verification returning to an earlier pipeline stage through bounded repair.The implementation adapts the QAIRT skill with AI assistance, then uses manual verification and experimental adjustment.Source: §4 · p. 5, Figure 2 · p. 6
- TensorRT and RKNN generate runtime-specific outputs, whereas OpenVINO and QAIRT can use shared intermediate formats; quantization may be inline or post-conversion.Device-agnostic formats do not guarantee full cross-device portability because operator support can differ.Source: §4 · p. 6
4.1 OpenVINO Deployment Pipeline
- OpenVINO converts models to IR and applies NNCF quantization, then verifies inference against PyTorch using cosine similarity and element-wise tolerance.The workflow also supports a combined conversion-and-quantization API and integration packages.Source: §4.1 · p. 6
4.2 RKNN Deployment Pipeline
- RKNN deployment simplifies the ONNX graph, then performs conversion and optional mixed quantization for RK3588 before on-device output verification.The agent manages per-layer quantization settings and calibration data.Source: §4.2 · pp. 6–7
4.3 TensorRT Deployment Pipeline
- The TensorRT path uses ONNX export with an explicit batch dimension, then configures conversion, precision, dynamic shapes, and calibration before comparing outputs with PyTorch.This study uses the ONNX-to-TensorRT path rather than Torch-TensorRT.Source: §4.3 · p. 7
4.4 ONNX Runtime QNN Deployment Pipeline
- ONNX Runtime’s QNN EP requires an independently quantized QDQ model and verifies inference against PyTorch while configuring the QNN backend and target device.The study focuses on QNN EP; it also requires checking opset support for the target QAIRT SDK version.Source: §4.4 · pp. 7–8
4.5 QAIRT Deployment Pipeline
- QAIRT’s QNN path converts ONNX through QNN Converter and can compile a Context Binary; its SNPE path converts ONNX to DLC for cross-backend execution on the same SoC.QAIRT supports built-in quantization during conversion or a two-stage AIMET quantization workflow.Source: §4.5 · p. 8
4.6 ONNXWrapper Inference Replacement and Layout Adaptation
- ONNXWrapper analyzes ONNX layouts, performs needed NCHW/NHWC adaptation with shape checks, and preconverts RKNN graphs to NHWC to eliminate layout-related failures.The wrapper sits at the inference-replacement boundary, and its RKNN integration reduces the agent’s repair burden.Source: §4.6 · pp. 8–9
5 CASE STUDIES
The case studies compare AIPC-guided deployment with direct conversion across four runtimes and two vision models, focusing on FP16 deployment feasibility rather than performance. They show both successful deployments and agent deviations, with outcomes shaped by model complexity, runtime-specific constraints, and verification.
5 CASE STUDIES
- The study compares AIPC skills and staged verification against direct conversion using general model knowledge, with the same agent model and target models in both groups.Both groups focus on deployment feasibility at NPU FP16 precision, not performance or throughput.Source: §5 · p. 9, §5.3 · p. 10
5.1 Experimental Setup
- The tests use Real-ESRGAN as a structurally regular model and YOLO-World as a more complex dual-encoder model, with reference outputs prepared from verified PyTorch runs.The model pair is intended to reveal how deployment behavior differs across model complexity.Source: §5.1 · p. 9
5.2 Experiment 1: AIPC Agentic Deployment
- With AIPC, deployments encountered runtime-specific failures and deviations, yet the agents completed OpenVINO and RK3588 deployment and, after correction, TensorRT FP16 deployment.For QNN, the agent first failed to trigger the provided Einsum repair procedure, then succeeded by replacing Einsum with an equivalent matrix-multiplication subgraph; it also autonomously repaired duplicate Cast outputs and graph ordering.Source: §5.2 · p. 10
5.3.1 Real-ESRGAN Super-Resolution.
- Without structured skills, TensorRT and OpenVINO produced usable Real-ESRGAN models, while QNN inference succeeded using the older onnxruntime-qnn 1.x package and fixed input dimensions.RKNN did not complete until the agent was explicitly prompted about NHWC layout; QAIRT conversion took approximately one hour.Source: §5.3.1 · p. 10
5.3.2 YOLO-World (Open-Vocabulary Detection).
- Without skills, OpenVINO and TensorRT passed for YOLO-World, while QNN initially reported success without using the NPU and completed HTP inference only after feedback and correction.A later attempt in a new conversation also completed QNN Context Binary generation and inference verification.Source: §5.3.2 · p. 11
5.4 Analysis
- The analysis links agent failures to deployment knowledge: explicit operator, layout, and toolchain guidance supports correct repair paths, while implicit knowledge is associated with workflow deviation and inefficient trial strategies.Across runtimes, the authors report that longer workflows and more complex YOLO-World deployments showed more frequent deviations, including despite skill injection.Source: §5.4 · p. 12
5.5 Observations and Limitations
- The experiments cover only two CV models and do not systematically compare cross-runtime operator-repair strategies or implement quantization and optimization testing.The authors also note that the small sample of 4–8 deployment tests per group yields preliminary observations without statistical significance.Source: §5.5 · pp. 12–13, §5.4 · p. 12
- Agent-model generalization remains untested at scale because the study primarily uses MIMO v2.5, with a different agent used only for Windows on ARM.The authors identify cross-model verification as needed to assess differences in skill compliance, failure recovery, and platform knowledge.Source: §5.5 · p. 13
6 CONCLUSION
The paper extends AIPC to four heterogeneous runtimes and finds preliminary feasibility for deployment of tested, structurally regular vision models on most runtimes. It also reports more frequent agent deviations on complex models and identifies limits in the scope of its validation.
6 CONCLUSION
- The extended AIPC methodology targets single-model-to-single-runtime deployment across OpenVINO, RKNN, TensorRT, and ONNX Runtime QNN, using structured knowledge for conversion, repair, and precision verification.The ONNXWrapper layer removes tensor layout adaptation from the agent’s manual repair burden; controlled comparisons attribute failure modes to knowledge availability rather than model capability alone.Source: §6 · p. 13
- Agents successfully deploy the tested structurally regular vision models on most runtimes, while general knowledge alone is insufficient for reliable handling of more complex structures.Examples include limited operator coverage in ONNX Runtime QNN and implicit NHWC layout constraints in RKNN.Source: §6 · p. 13
- Native hardware inference runtimes typically bring significant performance and power-consumption improvements over generic PyTorch execution, particularly on resource-constrained edge devices.The paper frames agent-orchestrated hardware adaptation as potentially reducing the cost and time of deployment work.Source: §6 · p. 13