Source-linked AI summary
AutoNeural: Co-Designing Vision-Language Models for NPU Inference
Wei Chen, Liangmin Wu, Yunhai Hu, Zhiyuan Li, Zhiyuan Cheng, Yicheng Qian, Lingyue Zhu, Zhipeng Hu, Luoyi Liang, Qiang Tang, Zhen Liu, Han Yang
TL;DR
GPU-centric VLMs are difficult to deploy efficiently on NPUs because attention-heavy architectures are brittle under quantization and impose substantial memory costs during generation. AutoNeural co-designs a MobileNet-based vision encoder with a hybrid Transformer-SSM language backbone for NPU-native inference. It reports lower quantization error and latency than ViT-Transformer baselines, with real-time automotive deployment results.
Problem
GPU-centric VLMs suffer high latency and quantization brittleness on NPUs, especially in attention-heavy vision encoders and autoregressive language generation.
Method
AutoNeural combines a MobileNet-style depthwise-convolutional vision encoder with a hybrid Transformer-SSM language backbone using efficient gated convolutions.
Results
Up to 7× lower quantization error and 14× lower end-to-end latency are reported versus ViT-Transformer baselines under the same NPU precision constraints.
Takeaways & Limitations
Real-world deployment on the Qualcomm SA8295P NPU demonstrates real-time responsiveness for automotive scenarios.
Takeaways & Limitations
The benchmark results are conducted under full precision conditions, so comprehensive in-vehicle validation of quantized performance remains future work.
Abstract
from arXiv · showhide
While Neural Processing Units (NPUs) offer high theoretical efficiency for edge AI, state-of-the-art Vision--Language Models (VLMs) tailored for GPUs often falter on these substrates. We attribute this hardware-model mismatch to two primary factors: the quantization brittleness of Vision Transformers (ViTs) and the I/O-bound nature of autoregressive attention mechanisms, which fail to utilize the high arithmetic throughput of NPUs. To bridge this gap, we propose AutoNeural, an NPU-native VLM architecture co-designed for integer-only inference. We replace the standard ViT encoder with a MobileNetV5-style backbone utilizing depthwise separable convolutions, which ensures bounded activation distributions for stable INT4/8/16 quantization. Complementing this, our language backbone integrates State-Space Model (SSM) principles with Transformer layers, employing efficient gated convolutions to achieve linear-time complexity. This hybrid design eliminates the heavy memory I/O overhead of Key-Value caching during generation. Our approach delivers substantial efficiency gains, reducing quantization error of vision encoder by up to 7x and end-to-end latency by 14x compared to conventional baselines. The AutoNeural also delivers 3x decoding speed and 4x longer context window than the baseline. We validate these improvements via a real-world automotive case study on the Qualcomm SA8295P SoC, demonstrating real-time performance for cockpit applications. Our results highlight that rethinking model topology specifically for NPU constraints is a prerequisite for robust multi-modal edge intelligence.
1 Introduction
AutoNeural addresses the latency and quantization weaknesses of GPU-centric VLMs on NPUs through co-designed vision and language topologies. Its architecture combines convolutional visual processing with a hybrid Transformer-SSM backbone and reports improved efficiency on automotive NPU hardware.
- Motivation: GPU-centric VLMs incur high latency and low-precision brittleness because attention-heavy vision and language components are poorly matched to edge NPU execution.Vision encoding and autoregressive generation affect both time-to-first-token and time-per-token.
- Our approach: AutoNeural replaces the standard ViT-LLM pairing with a MobileNet-style depthwise-convolutional encoder and a hybrid Transformer-SSM language backbone.The design targets bounded activations, linear-time sequence modeling, and reduced memory traffic.
- Our approach: The hybrid language backbone uses gated convolutions and Transformer layers to avoid explicit KV caching and reduce memory I/O during generation.The reported reduction in memory I/O reaches up to 60%.
- Key findings: Up to 7× lower quantization error and 14× lower end-to-end latency are reported versus ViT-Transformer baselines under the same NPU precision constraints.The results are associated with stable INT8/16 vision inference and reduced memory I/O during generation.
- Key findings: AutoNeural is evaluated through real hardware measurements on the Qualcomm SA8295P NPU for an in-car assistant scenario.The supplied passages describe real-time responsiveness and ablations of the encoder, SSM integration, and token budgeting.
2 Related Work
Related work spans automotive multimodal applications, NPU-oriented model adaptation, and efficient vision encoders. AutoNeural is positioned as addressing the deployment gap between capable VLMs and automotive edge constraints.
- Multimodal models for automotive deployment: Automotive multimodal systems support applications including driver assistance, in-car assistants, and autonomous driving, but production deployment requires hardware-aware co-design.Existing systems often compromise model capacity or resolution to satisfy latency budgets.
- Models on NPU architectures: NPU deployment research has explored quantization, pruning, operator fusion, and quantization-aware training for mobile and automotive platforms.The related work identifies NPUs from Qualcomm, MediaTek, and Apple as important on-device compute substrates.
- Efficient vision encoders for on-device deployment: Vision Transformers deliver strong performance but create on-device challenges through global attention and quantization brittleness.Alternative encoder designs such as MobileNet architectures have been explored for edge deployment.
3 Methodology
The methodology combines a MobileNet-based visual encoder, a lightweight connector, and a hybrid Transformer-SSM language backbone for NPU deployment. It also includes automotive data collection and on-device mixed-precision evaluation.
- Model architecture: The model uses a MobileNet-based vision encoder and a hybrid Transformer-SSM language backbone to reduce memory I/O during autoregressive generation.These are the two main architectural components of the proposed NPU-native VLM.
- Vision encoder: The vision encoder uses depthwise separable convolutions with bounded activations, consumes 768×768 inputs, and produces flattened visual tokens.Later low-resolution stages add sparse multi-query attention, while MSFA fuses two stage outputs at bounded cost.
- Vision–language connector: The connector is a two-layer GELU MLP without normalization, chosen to maintain NPU quantization robustness between visual features and language embeddings.It serves as the alignment bridge between the encoder output space and the language model embedding space.
- Language model backbone: The Liquid AI 1.2B backbone interleaves 10 gated-convolution layers with 6 Transformer layers, providing linear-time sequence modeling and compact states without explicit KV caching.The architecture contains 16 layers in a 5:3 convolution-to-Transformer ratio.
- Automotive dataset: The automotive dataset contains 0.2M cockpit-task samples collected from approximately 400 volunteers across demographic groups.The described tasks include vehicle security, identity recognition, and hazard detection.
- On-device evaluation: The quantized model is evaluated on the Qualcomm SA8295P NPU using W8A16 for vision and W4A16 for language, with measurements taken from actual device execution.Figure 2 reports the resulting performance profile rather than a PyTorch simulation.
4 Experiments
AutoNeural is evaluated against multimodal baselines through benchmark, ablation, on-device latency, and end-to-end deployment analyses. The experiments show competitive accuracy alongside substantial latency, throughput, context, and quantization advantages on Qualcomm SA8295P.
- Benchmark comparison: 60.75 average accuracy outperforms InternVL2-1B at 55.96 and approaches InternVL2-2B at 61.97 with 33% fewer parameters.Table 1 reports full-precision accuracy across five multimodal benchmarks for AutoNeural’s 1.47B-parameter model.
- Ablation studies: MobileNet-Qwen reaches 65.15 average accuracy versus InternViT-Qwen’s 63.08 while delivering 14× lower latency, showing the MobileNet encoder is not worse than the ViT encoder in this comparison.The ablation isolates the vision encoder while holding the Qwen language backbone constant.
- Ablation studies: AutoNeural’s hybrid language backbone trades a modest accuracy reduction for 2.9× higher decode throughput and a 4× larger context length on NPU hardware.The comparison is between MobileNet-Qwen at 65.15 average accuracy and AutoNeural at 60.75.
- Latency and quantization analysis: 14× lower latency at 512×512 comes from AutoNeural’s 101.7ms versus InternViT-300M’s 1415.0ms, while 768×768 runs in real time and the baseline exceeds NPU memory.At 256×256, AutoNeural also achieves 28.0ms versus 163.3ms, a 5.8× speedup.
- Latency and quantization analysis: The MobileNet encoder uses bounded activation footprints and predictable memory access patterns, enabling NPU operator fusion and efficient on-chip buffer utilization.These architectural properties are contrasted with the quadratic activation growth of global self-attention.
- Limitations and future work: Comprehensive in-vehicle evaluation, broader NPU validation, and improved quantization approaches remain future work because the benchmark results use full precision.The paper specifically calls for testing quantized performance across diverse driving scenarios and production automotive environments.
5 Conclusion
AutoNeural co-designs a MobileNet-based vision encoder with a hybrid Transformer-SSM language backbone for efficient NPU deployment. The resulting architecture reduces quantization error and latency while improving generation efficiency and supporting real-time automotive use.
- Conclusion: AutoNeural replaces the standard ViT-Transformer design with a MobileNet-based encoder and hybrid Transformer-SSM backbone to address quantization brittleness and memory I/O.The architecture targets efficient edge deployment by selecting operators aligned with NPU execution models.
- Conclusion: Up to 7× lower quantization error and 14× lower end-to-end latency are reported against ViT-Transformer baselines under the same NPU precision constraints.The MobileNet encoder provides stable INT8/16 inference, while the hybrid backbone reduces memory I/O by up to 60% during autoregressive generation.
- Conclusion: NPU-native operator selection enables robust, low-latency multimodal intelligence at the edge, with real-time automotive responsiveness validated on Qualcomm SA8295P.The paper identifies depthwise separable convolutions for vision and SSMs for language as the key hardware-aligned choices.