PaperlayerView exact revision on arXiv

Source-linked AI summary

AIPC: Evidence-Guided Agent Workflows for PyTorch Project Deployment

Zhanwei Wu, Jianhao Su, Brian Zhang, ShengTing Huang, Weidong Feng, Gang Liu, Zhiguo Li, Wei Huang, Li Tan, Wanjun Li, Zejiong Teng, Changgeng Zhong, Guanping Wu, Han-Hsing Tu

arXiv:2610.34890v1cs.SEcs.AIcs.LG

TL;DR

Dedicated on-device NPUs require PyTorch projects to be converted into hardware-runtime artifacts, while toolchain feedback alone does not close the loop from diagnosis to repair. AIPC structures deployment around Config, Pass, and Gate; on Qualcomm QAIRT, its Skill-guided optimization cases report improved outcomes, including Whisper-Small accuracy-and-latency success in 6/7 runs versus 0/6 control runs.

Abstract

from arXiv · show

Deploying and optimizing a PyTorch inference project on a dedicated accelerator, especially an on-device NPU, requires more than using a framework device API as with CUDA. The project must be converted into hardware-runtime artifacts and adapted to the target toolchain and application interface. Differences in model formats, quantization flows, operator support, and toolchain interfaces create cross-layer problems spanning PyTorch/ONNX models, toolchains, runtimes, and device execution. LLM Agents can invoke tools to address such problems, but may drift because of insufficient domain knowledge, large configuration and repair spaces, or poor path selection. We propose AIPC (AI Porting & Conversion), an Agent method for deploying PyTorch inference projects. AIPC combines structured Skills with toolchain, runtime, and profiling evidence to drive cross-layer diagnosis and repair, organizing model adaptation, artifact building, on-target execution, accuracy repair, and performance optimization into one deployment flow. We evaluate AIPC on the on-device NPU path of Qualcomm QAIRT across vision, speech, LLM, and diffusion models. In two optimization cases, 6/7 Skill-arm Whisper-Small runs meet predefined accuracy and latency criteria versus 0/6 control runs, while YOLO-Worldv2 achieves a 2.791x end-to-end speedup versus 1.599x for the control. Under a pre-fixed deployment contract, Skill, tool permissions, and check conditions, some runs achieve zero in-run human intervention (0-HITL). We also examine Qwen3.8-27B as a locally deployable model for driving an Agent workflow.

I. INTRODUCTION

Dedicated-NPU deployment requires converting PyTorch projects into hardware-runtime artifacts across inconsistent model, toolchain, runtime, and device interfaces. Existing tools and agents leave gaps in cross-layer repair, automated verification, and bounded execution.

A. Motivation and Engineering Pain Points

  • Dedicated NPUs require hardware-runtime artifacts, while inconsistent interfaces scatter deployment knowledge across layers and increase expert maintenance costs as models and configurations grow.The work spans frameworks, graph representations, conversion tools, runtimes, and device backends.Source: A. Motivation and Engineering Pain… · p. 1
  • Conversion failures, semantic mismatches, and inefficient accelerator use are distinct deployment challenges requiring adaptation, reference-output checks, and hardware-aware optimization.Dynamic shapes or unsupported operators can block conversion; quantization drift can compromise correctness; fallback and data movement can reduce performance.Source: A. Motivation and Engineering Pain… · p. 1
  • Repeated cross-layer iteration limits deployment efficiency and scalability; AgentDeploy reports experts spent around 6 h on image-recognition tuning, versus about 60 h and 80 h for semi-experts and non-experts on speech-sentiment optimization and adaptation.Source: A. Motivation and Engineering Pain… · p. 1

B. Limitations of Existing Solutions

  • Existing solutions lack a closed loop from tool feedback through diagnosis and repair to rebuild and reverification, automated result verification, and a deterministic boundary for open-ended agent diagnosis.Without automated inference, reference alignment, and backend checks, silent failures can be difficult to detect systematically; agents may retry blindly or drift off task.Source: B. Limitations of Existing Solutions · p. 2

C. AIPC Method Overview and Contributions

AIPC extends agent-driven on-device deployment into a workflow for complete PyTorch inference projects, adding project adaptation, accuracy repair, and performance optimization. Its QAIRT proof of concept evaluates deployment, repair, and optimization questions on target NPUs.

C. AIPC Method Overview and Contributions

  • Toolchain, runtime, numerical, and profiling feedback guides cross-layer repair and rechecking of the model graph, project interface, or runtime configuration.The workflow includes the model, preprocessing and postprocessing, I/O contract, and target-device requirements, with project checks and run records confirming outcomes.Source: C. AIPC Method Overview and… · p. 2
  • The QAIRT proof of concept evaluates whether an Agent can deploy PyTorch projects, diagnose and repair execution failures, and optimize performance while preserving accuracy and NPU execution.Source: C. AIPC Method Overview and… · p. 2

A. On-Device Conversion and Compiler Frameworks

  • Dedicated hardware runtimes require graph export, runtime-artifact construction, and hardware-specific execution rather than framework device APIs, creating adaptation problems.Source: A. On-Device Conversion and Compiler… · p. 2
  • Conversion and optimization frameworks provide infrastructure, but typically do not turn device-specific conversion, runtime, numerical, or profiling feedback into repairs and rechecks.Vendor runtimes combine conversion, hardware-specific compilation, and target execution, but their artifacts and interfaces differ.Source: A. On-Device Conversion and Compiler… · p. 2

B. Qualcomm QAIRT Toolchain and Application Runtime

The QAIRT path combines Qualcomm’s heterogeneous-device inference stack with distinct model artifacts, APIs, and toolchains. Deployment also requires application integration, optional quantization or precompilation, and attention to hardware-specific constraints.

B. Qualcomm QAIRT Toolchain and Application Runtime

  • QAIRT supports heterogeneous Qualcomm SoCs with CPU, Adreno GPU, and Hexagon NPU, and can dispatch supported models to the HTP tensor-compute accelerator.Genie is a dedicated inference extension for on-device LLMs, vision-language multimodal models, and diffusion models.Source: B. Qualcomm QAIRT Toolchain and… · pp. 2–3
  • DLC, QNN model libraries, and Context Binaries are distinct artifacts, while SNPE, QNN, and QAIRT APIs use different typical artifact and toolchain combinations.QNN model libraries depend on compiler, ABI, operating system, and target architecture; Context Binaries are AOT-precompiled for a target backend or SoC.Source: B. Qualcomm QAIRT Toolchain and… · pp. 2–3
  • QAI AppBuilder provides target-side Python and C++ APIs for loading QAIRT artifacts, configuring a backend, and running inference; Model Builder handles conversion and deployment, and AIPC is its Skill-based workflow method.Source: B. Qualcomm QAIRT Toolchain and… · p. 3
  • The deployment chain can export PyTorch to ONNX, build conversion and runtime artifacts, optionally quantize and prepare a Context Binary, integrate the application interface, run inference, and optimize through profiling.Source: B. Qualcomm QAIRT Toolchain and… · p. 3
  • HTP VTCM capacity can affect compilation or off-chip access, and converter layout changes or unsupported operators, dynamic shapes, and complex control flow can disrupt deployment.QNN conversion may change ONNX I/O to NHWC, whereas QAIRT conversion usually preserves the source layout; AIMET supplements QAIRT quantization, and AI Hub supports cloud compilation and performance evaluation.Source: B. Qualcomm QAIRT Toolchain and… · p. 3

C. Agentic Deployment and LLM Agent Related Work

AIPC takes an existing PyTorch inference project as its unit of work, using a Skill-guided workflow to select tools and repairs from project state and runtime feedback. Its Model Builder adapts, builds, executes, and checks target-device artifacts through configurable stages.

C. Agentic Deployment and LLM Agent Related Work

  • Unlike related approaches focused on kernels, operators, or configuration search, AIPC organizes cross-model project adaptation, runtime model and code generation, on-target inference, accuracy repair, and optimization.Prior work includes AgentDeploy, Spatial-NPU Agent Skill [30], and EdgeFM [31].Source: C. Agentic Deployment and LLM Agent… · p. 3

III. AIPC SYSTEM FRAMEWORK

  • AIPC separates Model Builder baseline construction from Optimizer variant generation, while the Skill guides the Agent to read state, select tools, and propose repairs.Source: §III · p. 3
  • Config fixes project and target requirements, Pass defines a configurable stage, and Gate derives status from artifacts and run results to continue, repair, transfer, or stop.Engineers can skip or add steps, while the Agent selects tools and candidate repairs within the workflow.Source: §III · p. 3

A. Model Builder

  • The Model Builder runs ordered Passes with recorded state and artifact dependencies, preserving required inputs and Gates even when a project skips or adds steps.Its inputs are a PyTorch project, deployment Config, and Gate conditions; outputs are runtime artifacts, interface adaptation, and a deployment baseline.Source: A. Model Builder · p. 3
  • The QAIRT default workflow spans configuration, PyTorch/NPU adaptation, ONNX export, graph repair, conversion and runtime build, Context Binary preparation, on-target inference, accuracy reporting, and profiling.Source: A. Model Builder · p. 3
  • Before export, PyTorch/NPU adaptation can address dynamic shapes and control flow, externalize autoregressive cache, establish an exportable entry, or rewrite RoPE and attention for static tensor computation.Source: A. Model Builder · p. 4
  • Graph repair selects an equivalent rewrite at the PyTorch, export-entry, or ONNX layer according to preconditions, then checks reference outputs, conversion, and target execution.Application integration binds artifacts, preprocessing, invocation, and postprocessing to on-target code; specialized APIs may require an additional Pass.Source: A. Model Builder · p. 4
  • When a Gate fails, logs, output comparisons, and runtime feedback route repair to the PyTorch, ONNX, or runtime-interface layer, with reruns from the affected Pass and retained run state.Repair includes graph rewriting, operator replacement, graph partitioning, and I/O or layout repair.Source: A. Model Builder · p. 4

B. Optimizer

The Optimizer uses deployment artifacts as a baseline and combines profiling, runtime, numerical, and task evidence with graph-optimization knowledge to select candidates. Accuracy repair uses reference comparisons and controlled changes to address deviations while preserving acceptance requirements.

B. Optimizer

  • Candidates become the new optimization baseline only after rebuilding and passing the acceptance Gate; failures in accuracy, project I/O, or target-NPU execution are rejected.The knowledge substrate records rewrite patterns and preconditions, hardware-benefit hypotheses, historical outcomes, and selection rules.Source: B. Optimizer · p. 4

1) Accuracy Optimization:

  • Accuracy optimization compares FP reference and target-device outputs with numerical-error or task metrics to locate deviations, sensitive outputs, and task-accuracy degradation.For difficult-to-localize problems, it can analyze clipping, layer-wise deviations, and residual outliers.Source: 1) Accuracy Optimization · p. 4
  • A/B comparisons change one variable at a time, treating activation and weight calibration methods as independent variables.Candidates include calibration data and methods, quantization granularity, global or mixed precision, FP16 retention for sensitive layers or output heads, and equivalent graph rewrites.Source: 1) Accuracy Optimization · p. 4

2) Performance Optimization:

Performance optimization selects changes using profiling, VTCM, operator behavior, and data-movement evidence. The described workflow also supports continuous or interactive task execution and evaluates deployment against runtime, output, and delivery checks.

2) Performance Optimization:

  • Performance optimization locates bottlenecks through profiling, VTCM usage, operator behavior, and data movement, then considers layout, operator, attention, fusion, VTCM, or graph-partitioning changes.Examples include NCHW↔NHWC layout optimization, Einsum decomposition, MHA→SHA restructuring, and operator fusion.Source: 2) Performance Optimization · p. 4

C. Usage Modes

  • Users can run a confirmed workflow continuously or pause after each stage for confirmation, while single tasks can invoke a corresponding script directly.The resulting package can be handed to the App Builder Skill for a local WebUI or loaded through QAI AppBuilder Python/C++ APIs.Source: C. Usage Modes · p. 4

A. Deployment and Agent Workflow Evaluation

  • A target-device run is graded O only when the target path, declared numerical or task Gate, and critical delivery capabilities pass; ? indicates missing capability or evidence, and X indicates a substantive path, numerical, or contract error.The project begins from an already runnable PyTorch workspace; missing dependencies or an unrunnable baseline are unmet preconditions.Source: A. Deployment and Agent Workflow… · p. 5
  • 0-HITL means a run after setup is frozen has no human intervention that changes its decision or execution path; supplying task information, changing code or configuration, authorizing a new operation, or appending an instruction counts as HITL.Environment recovery and confirmation of the established configuration do not count as intervention.Source: A. Deployment and Agent Workflow… · p. 5
  • Deployment experiments use a QNN model-library plus Context Binary path, while model optimization uses DLC artifacts and the newer unified QAIRT workflow.The primary deployment path exposes local compilation-toolchain, I/O-layout, and backend-configuration issues.Source: A. Deployment and Agent Workflow… · p. 5

1) Purposively Covered Model Set:

The evaluation covers selected PyTorch projects across vision, speech, diffusion, and decoder-only LLMs, then examines QAIRT deployment and feedback-driven repair. Its findings distinguish removal of an initial deployment blocker from passing the later accuracy or application Gates required for delivery.

1) Purposively Covered Model Set:

  • The seven purposively selected projects span vision, speech, diffusion, and decoder-only LLMs, covering varied structures and I/O interfaces; the YOLO-Worldv2 tests keep CLIP frozen.Supplementary functional checks cover additional vision, speech, and multimodal projects, including PPOCR, RT-DETR, YOLO variants, Wav2Vec2, SmolVLM, and Qwen3.5-0.8B.Source: 1) Purposively Covered Model Set · p. 5

2) QAIRT Deployment and Failure-Recovery Case Studies:

  • The no-Skill comparison has access to the source project, QAIRT SDK, and official documentation, and Table I summarizes Skill-on/off deployment results, process costs, and observations.The supplied passages describe the table’s scope but do not provide its cell values.Source: 2) QAIRT Deployment and… · p. 5, Table I · p. 6
  • Successful conversion, target-process termination, or intermediate outputs do not establish deployment readiness; runtime, output-comparison, and project-deliverable requirements must also pass.Observed issues included missing expected images or audio, an uncovered KV-cache path, and an HTP U-Net loading failure in one SD1.5 no-Skill run.Source: 2) QAIRT Deployment and… · p. 5
  • Skill runs can take longer because they include additional checks, while no-Skill time is more variable and can rise with incorrect settings, retries, and repairs.The table’s observations also include path deviations and incomplete expected deliverables, so elapsed time alone does not establish readiness.Source: 2) QAIRT Deployment and… · p. 5
  • Feedback-driven repair removed specific blockers: MeloTTS temporal chunking removed a decoder VTCM limit and restored target HTP execution, while SD1.5 graph splitting removed the full U-Net weight-mapping limit.The MeloTTS case remained pending because its waveform Gate missed the threshold; later numerical, task, or application Gates independently determine delivery status.Source: 2) QAIRT Deployment and… · p. 5

3) Auxiliary Experiment: Feasibility of a Qwen3.8-27B-Driven Agent Workflow:

The auxiliary experiment tests whether Qwen3.8-27B can drive an AIPC deployment workflow and compares Skill-on/off settings under otherwise matched conditions. The reported cases include path ambiguity, contract-violating execution, and delivery or interface-generation failures.

3) Auxiliary Experiment: Feasibility of a Qwen3.8-27B-Driven Agent Workflow:

  • The auxiliary experiment examines Qwen3.8-27B driving an AIPC deployment workflow and reports end-to-end records for three models alongside a vanilla-agent comparison.The model is invoked through Open-Router.Source: 3) Auxiliary Experiment: Feasibility… · p. 6
  • The two settings differ only in whether the Agent receives the AIPC Skill, including its workflow prompts, with all other conditions held constant; each setting was planned for three independent runs.The current Whisper-tiny record contains three Skill runs and one no-Skill run.Source: 3) Auxiliary Experiment: Feasibility… · p. 6
  • Qwen3.8-27B sometimes misidentified the target through an x86-emulated Python environment and shifted to CPU-simulated or GPU execution, or switched from QNN to DLC after an error.The table caption identifies the comparison as executions of the same model under Skill-on/off settings.Source: 3) Auxiliary Experiment: Feasibility… · p. 6, Table III · p. 7
  • Unsupported ONNX messages and false alarms lengthened workflow time through extra diagnosis, graph surgery, and validation; observed failed runs also included contract-violating execution and missing delivery or interface generation.Examples include a YOLO-Worldv2 DLC/GPU run, a Whisper-tiny CPU decoder run, and a Real-ESRGAN run without Python inference code.Source: 3) Auxiliary Experiment: Feasibility… · p. 6

4) Summary:

The summary reports that AIPC’s Skill constrains path selection and checks, helping Agents complete delivery along project-required paths. The optimization evaluation compares Skill-guided runs with controls under fixed task specifications and held-out data.

4) Summary:

  • Without the Skill, Agents may take shortcuts such as DLC, fixed shapes, no KV-cache, or NPY-only output, leaving delivery incomplete or incorrect and requiring human realignment.These paths may shorten a local step without meeting project delivery expectations.Source: 4) Summary · p. 7
  • The Skill encodes project contracts, deployment steps, tool invocations, and Gates as executable constraints, including explicit KV-cache export and target-NPU tensor I/O for autoregressive models.Reusing cached history reduces per-step computation from O(n^2) over the full prefix to O(n).Source: 4) Summary · p. 7
  • For Qwen3.8-27B, Skill-guided runs more often stayed on project-required paths and completed delivery, while the model could produce executable artifacts but needed more Harness support for stable execution than GPT-5.6-Luna.Skill=N runs shifted to DLC/GPU or HTP/CPU mixed execution, or omitted Python inference code.Source: 4) Summary · p. 7
  • The tested projects were deployed as target-device inference paths, with the Skill organizing path selection and checks so more recorded runs completed delivery along project-required paths.The two groups of cases show that Skill constraints affect delivery.Source: 4) Summary · p. 7

B. QAIRT Accuracy and Performance Optimization

  • The optimization evaluation uses a single prompt per session, shared task specifications, tools, permissions, thresholds, SDK, and device configuration, and held-out evaluation data.Supplementary builds include a naive reference, single-factor ablations, and operator-level profiling.Source: B. QAIRT Accuracy and Performance… · p. 7

1) Case 1: YOLO-Worldv2:

YOLO-Worldv2 results show stronger Skill-arm latency savings, while both arms have one run that meets all accuracy criteria. The evaluation also finds that the successful recipe depends on several changes rather than a single isolated adjustment.

1) Case 1: YOLO-Worldv2:

  • Accuracy-series criteria require mAP@0.5:0.95 and mAP@0.5 each to reach at least 95% of FP32, alongside per-class recall, class-presence, and empty-detection checks.The performance series instead judges A3–A6 and confidence-threshold consistency against each arm’s own starting artifact.Source: 1) Case 1: YOLO-Worldv2 · p. 7
  • Only 1 run per arm passes all criteria, while 3/4 Skill runs versus 1/4 Agent-only runs meet the overall-accuracy criterion.The difference for overall accuracy is not significant in this small sample (p = 0.49).Source: 1) Case 1: YOLO-Worldv2 · p. 7
  • 2.791× speedup comes mainly from host-side work; the Skill arm saves 11.69 ms end to end versus 5.29 ms for Agent-only.Postprocessing accounts for 76% of the Skill-arm savings, preprocessing 13%, and the NPU 11%.Source: 1) Case 1: YOLO-Worldv2 · p. 7, Table V · p. 8
  • The passing recipe combines multiple changes: omitting split outputs yields 0.0% accuracy, while other tested missing-item builds reach 74.6%–90.2%.The best build reaches 96.2%, approaching the 96.4%–97.2% delivered by the Agent.Source: 1) Case 1: YOLO-Worldv2 · p. 8

2) Case 2: Whisper-Small:

For Whisper-Small, six of seven Skill runs met all criteria versus none of six No-Skill runs, with one quantization option explaining most of the latency gap. The discussion qualifies the cost and scope of these results and draws deployment lessons for other models.

2) Case 2: Whisper-Small:

  • 6/7 Skill runs met all Whisper-Small criteria versus 0/6 No-Skill runs, a result repeated on 300 additional utterances.Across 16 threshold combinations, the conclusion was unchanged in 12; only at a relaxed 1.5× encoder bound did one No-Skill run pass.Source: 2) Case 2: Whisper-Small · p. 8, Table VII · p. 9
  • Changing only activation quantization to unsignedsymmetric covered 91% of the naive-to-expert latency gap, from 63.54 to 33.12 ms.All seven Skill runs used this schema, while none of the six No-Skill runs did.Source: 2) Case 2: Whisper-Small · p. 8

3) Discussion: Cost and Limitations:

  • Qualifying Whisper artifacts cost 146 thousand output tokens and 79.5 minutes each in the Skill arm, while No-Skill runs produced none.The six No-Skill runs used 920 thousand output tokens and 490 minutes; the seven Skill runs used 875 thousand tokens and 477 minutes and yielded six artifacts.Source: 3) Discussion: Cost and Limitations · p. 8
  • Across both optimization cases, latency improved while meeting accuracy constraints and retaining NPU execution, through different mechanisms for Whisper and YOLO.Whisper's gain came almost entirely from one quantization option; YOLO's came mainly from the host CPU, with its NPU-side gain from NHWC layout. Host-side pre/postprocessing accounts for 89% of YOLO's 2.791× speedup.Source: 3) Discussion: Cost and Limitations · p. 9, A. Engineering Implications and Future… · p. 10
  • The evidence is limited to 1–4 runs per cell and one SoC and SDK version per model, and the experiments cannot separate Skill structure from model-specific recipes.The authors therefore caution against extrapolation and leave a workflow-only ablation to future work.Source: 3) Discussion: Cost and Limitations · p. 9

A. Engineering Implications and Future Directions

  • Deployment acceptance should use blocking task-level checks, abort-capable gates, audited scripts, and whole-pipeline measurement.Across two cases, five of 14 Skill-arm runs still delivered artifacts without an audit pass verdict; a calibration option was never tried in the right direction across 22 variants in a run later invalidated to fix the Skill.Source: A. Engineering Implications and Future… · p. 9
  • Composite diffusion deployment remains constrained by host memory and component size, with PixArt-Σ's T5 encoder exceeding 5 GB.One exploration path completed DiT deployment but kept T5 on CPU, while other attempts failed during repeated conversion.Source: A. Engineering Implications and Future… · p. 10
  • Current AIPC covers adaptation, quantization, and graph-level optimization of existing PyTorch projects; broader model coverage and optimization capabilities remain future work.Planned extensions include Mamba and MoE backbones, kernel/operator generation, QAT, pruning, and evaluation on more models.Source: A. Engineering Implications and Future… · p. 10

B. Conclusion

AIPC encodes deployment of a complete PyTorch inference project as a Config–Pass–Gate workflow, with the Agent selecting steps and tools according to workflow state. On Qualcomm QAIRT NPUs, the paper reports results across four model areas and two optimization cases, supporting runtime-feedback-driven cross-layer diagnosis, repair, and rechecking.

B. Conclusion

  • AIPC combines Config, Pass, and Gate to establish a target-device baseline and optimize variants under accuracy, project-interface, and backend constraints.The Agent selects Passes and tools according to workflow state.Source: B. Conclusion · p. 10
  • On Qualcomm QAIRT NPUs, Whisper-Small met all criteria in 6/7 Skill runs versus 0/6 controls, while YOLO-Worldv2 achieved 2.791× speedup mainly from the host side.The paper also reports target-backend execution, output comparison, fault diagnosis, and profiling for vision, speech, LLM, and diffusion tasks; some fixed-contract runs achieved 0-HITL.Source: B. Conclusion · p. 10
  • The reported results support runtime feedback as a basis for cross-layer diagnosis, repair, and rechecking of the graph, project interface, or runtime configuration.The authors describe AIPC as an implementable engineering path for “AI Deployment by AI.”Source: B. Conclusion · p. 10

Related papers

Loading 2610.34890v1…