Source-linked AI summary
Research on Intra-Chip Fusion Deployment and Optimization of Embodied Intelligence Business Operator NPU
Yuchen Zhu, Longxiang Yin, Wanyu Wang, Jieke Lin, Guoqiang Zou, Zirui Cao, Yuling Yuan, Xiaolan Fan, Lifen Chen, Hao Zheng, Qizhang He, Hongyu Zhou, Chunhai Yu
TL;DR
Distributed perception-computation-control systems incur substantial transmission overhead, while existing optimization is insufficiently adapted to domestic heterogeneous hardware. The paper develops full-process operator optimization and single-MLU370 on-chip fusion on a Phytium-Cambricon platform. It reports 18.7 ms latency, 2.89× speedup over NVIDIA Jetson AGX Xavier, and 82.6% MLU utilization.
Problem
Full-process collaborative optimization and end-to-end millisecond-level verification remain insufficient for domestic heterogeneous perception-computation-control systems.
Method
The paper combines stage-specific operator optimization with single-card on-chip fusion and closed-loop pipeline execution on a Phytium-Cambricon platform.
Results
18.7 ms end-to-end single-frame latency and 2.89 times acceleration over NVIDIA Jetson AGX Xavier are achieved, with 82.6% MLU utilization.
Takeaways & Limitations
The integrated deployment provides a practical domestic platform reference for embodied intelligent computing services.
Takeaways & Limitations
End-to-end millisecond-level real-time verification remains a stated deficiency in existing research motivating the paper.
Abstract
from arXiv · showhide
Embodied intelligent computing integrates perception, computation and control. Traditional separate deployment of the three tasks leads to frequent data transmission, high latency and low hardware efficiency, failing to satisfy millisecond-level real-time requirements in dynamic scenarios. Besides, most operator optimization methods rely on foreign GPU platforms, while full-process collaborative optimization for domestic Phytium-Cambricon heterogeneous architectures is insufficient. This paper builds a domestic heterogeneous computing platform with Phytium FT-2000/4 processor and Cambricon MLU370 acceleration card, and proposes an NPU on-chip fusion deployment and full-process operator collaborative optimization strategy for perception, computation and control pipelines. Targeting embodied robot applications, modular optimization is conducted, including MLU hardware adaptation of motion blur correction operators for high-speed imaging, lightweight inference optimization of ViT models, and customized operator development for multi-DOF inverse kinematics solution. An on-chip data closed-loop and pipeline collaboration-based single-card solution is proposed to implement integrated execution of all perception-computation-control tasks on MLU370. Experimental results show that the proposed method achieves a full-process single-frame latency of 18.7 ms and a speedup of 2.89 compared with NVIDIA Jetson AGX Xavier, with 82.6% MLU utilization and comparable accuracy to mainstream platforms. This work offers a practical reference for domestic engineering applications of integrated embodied intelligent computing services.
1 Introduction
The paper targets low-latency, fully integrated perception-computation-control processing on a domestic Phytium-Cambricon platform. It combines full-process operator optimization with single-MLU370 on-chip fusion deployment and evaluates the resulting system experimentally.
- 30% to 50% of total process latency can come from inter-processor data transmission in distributed deployments, creating a major real-time bottleneck.
- Existing optimization methods largely rely on foreign hardware and do not fully adapt to domestic hardware architectures.
- Full-process collaborative optimization and integrated perception-computation-control deployment remain insufficient on Phytium-Cambricon heterogeneous platforms.
- The study builds a Phytium FT-2000/4 CPU and Cambricon MLU370 NPU platform for embodied intelligent robot scenarios.
- The proposed method performs stage-specific operator optimization and hardware adaptation for perception, computation, and control.
- Single-chip pipeline scheduling and operator compatibility adaptation execute the complete workflow within one MLU370 while reducing data-transmission overhead.
2 Related Work
Prior work reduces latency through shared memory, on-chip pipelines, heterogeneous scheduling, model optimization, and domestic hardware adaptation. However, full-link collaborative deployment with millisecond-level end-to-end verification on domestic heterogeneous platforms remains underdeveloped.
- Shared memory, on-chip pipelines, and heterogeneous scheduling are used to reduce data migration overhead in integrated deployments.
- GPU-based joint perception-inference pipelines reduce transmission latency, while GPU-FPGA frameworks accelerate control algorithms.
- Existing integrated solutions rely on foreign platforms and do not provide true single-NPU closed-loop perception-computation-control integration.
- NPU deployment research commonly uses CUDA, TensorRT, and ONNX Runtime, emphasizing operator fusion, inter-layer optimization, mixed precision, and dynamic-shape adaptation.
- Operator merging, constant folding, memory reuse, and INT8 quantization reduce intermediate tensors, model size, or latency while retaining acceptable accuracy loss.
- Lightweight vision-model methods include attention sparsification, dimension compression, distillation, pruning, quantization, and operator adaptation for edge NPUs.
- Existing lightweighting largely targets isolated vision tasks without logical links to downstream control algorithms.
- The main gaps are insufficient full-link operator design and pipeline scheduling, plus missing end-to-end millisecond-level verification in highly dynamic scenarios.
3 Principles of Operator Optimization Algorithms for Full-
The passage refers to process sensing, computing, control, and methods.
- The passage names process sensing, computing, control, and methods.
Hardware Adaptation
The perception stage addresses motion blur from high-speed rolling-shutter imaging through DeepDeblur-based correction and MLU hardware adaptation. Its design combines blur-kernel estimation, multi-scale feature fusion, attention enhancement, and physics-guided constraints.
- Rolling-shutter motion blur can distort visual features and control commands, while CPU serial correction exceeds 250 ms per frame.
- DeepDeblur is adapted to MLU hardware and optimized for motion-blur distortion correction in the perception pipeline.
- The model uses multi-scale feature fusion and a physics-guided GAN to jointly model blur-kernel estimation and clear-image generation.
- Motion trajectory prediction uses TCN or optical-flow networks to produce pixel-wise motion-vector fields from blurred imagery or temporal assumptions.
- The physics-constrained loss enforces consistency between predicted motion fields and blurred images, with B as the blurred image, I as the sharp image, and K as the predicted kernel.
- Dual attention combines channel reweighting through an SE block with deformable-convolution spatial attention for blurred boundary regions.
- Frequency-domain attention applies FFT and band-pass filtering to enhance high-frequency edge components.
Acceleration in Computing Processes
The study accelerates ViT-based computing through structural optimization, quantization, and operator fusion on the Phytium-Cambricon MLU370 platform. These methods target ViT’s computational complexity, memory consumption, and inference latency while maintaining accuracy.
- Motivation: ViT inference is computationally intensive because global attention and deep network structure increase computational complexity and memory consumption.This produces excessive latency on traditional CPUs for perception-computation-control applications.
- Structural optimization: Eight of twelve attention heads are retained based on importance analysis to reduce attention computation while limiting inference accuracy loss to 3%.The lightweight transformation screens attention heads according to their contribution to feature extraction.
- Structural optimization: Embedding dimension compression reduces image patch embeddings from 768 to 512, decreasing matrix-multiplication computational complexity.
- Operator fusion: Structural fusion combines layer normalization, multi-head attention, and residual connections to reduce intermediate-feature-map read/write overhead and improve computational parallelism.
- Hardware adaptation: MagicMind converts the PyTorch ViT model through ONNX parsing and model building into an MLU370-executable .mm model, supporting adapted core operators.
- Quantization: INT8 quantization and mixed INT8+FP16 precision are used to reduce model memory footprint and improve inference speed while addressing precision-sensitive attention modules.The process uses post-training quantization with calibration data from ImageNet validation images.
3.3 BANG C Core Operator Development and Optimization
The operator-development workflow uses MagicMind to convert and optimize ViT models for MLU370 execution. It combines post-training quantization with mixed precision to balance inference speed and accuracy.
- Quantization: Post-training quantization converts model weights and activations from FP32 to INT8 using 100 ImageNet validation images as calibration data.
- Quantization: Mixed INT8+FP16 quantization is applied to accuracy-sensitive attention calculations to balance inference speed and precision.
4 Implementation plan
The implementation combines MagicMind model deployment with a single-card on-chip fusion architecture for perception, computation, and control. Three-stage pipeline execution overlaps these tasks in MLU370 on-chip memory, reducing data-transfer overhead and end-to-end latency.
- Model deployment: MagicMind parses PyTorch models, uses a builder for optimization and generation, and serializes the result as a reusable model file.
- Model optimization: INT8 model quantization reduces memory footprint and improves inference speed, with Figure 3 comparing memory usage and inference time before and after quantization.
- Model optimization: Operator fusion combines consecutive convolution, activation, and normalization operators, while NHWC feature-map layout adapts memory access to MLU vector units.
- Integrated architecture: The architecture integrates computing-intensive perception, computation, and control tasks on a single MLU370 card through on-chip data closed-loop and pipeline collaboration.
- Pipeline scheduling: Three-stage pipeline execution overlaps preprocessing, ViT inference, and control inside MLU370, keeping data in on-chip memory and reducing end-to-end latency and PCIe bandwidth occupancy.
5 Introduction to the Experimental Environment
The experimental environment combines a Cambricon MLU370-S4 accelerator with a Phytium FT-2000/4 processor for integrated perception-computation-control evaluation. Experiments assess latency, accuracy, and hardware utilization across image restoration, ViT inference, and robot control tasks.
- Hardware environment: MLU370 is an edge AI accelerator designed for high-computing-power, low-latency, high-concurrency perception-computation-control services.It supports image preprocessing, vision-model inference, and control algorithm operators simultaneously.
- Hardware environment: The MLU370-S4 uses a half-height, half-length, single-slot design and connects with the Phytium FT-2000/4 through a high-speed PCIe interface.
- Hardware environment: MLU370 supports INT8, FP16, BF16, and FP32 mixed-precision computing, with peak INT8 computing power of 192 TOPS.
- Hardware environment: The Phytium FT-2000/4 uses an ARM v8.2 instruction set and a four-core FT663 high-performance core design.
- Evaluation tasks: The evaluation uses motion-blurred GOPRO images, ImageNet-21k and ILSVRC 2012 for ViT inference, and robot joint-angle and target-tracking measures for control.
- Evaluation metrics: Overall evaluation covers end-to-end latency, processing accuracy, and hardware utilization, including computing-unit utilization and PCIe transmission bandwidth.
6 Experimental Results and Analysis
Experiments evaluate perception, computation, control, and integrated deployment on the Phytium+MLU370 platform. The optimized system reduces stage-level and end-to-end latency while maintaining comparable accuracy and improving concurrent hardware utilization.
- Perception: 3.65 times acceleration reduces motion-blur restoration latency from approximately 250 ms to around 70 ms on MLU370.MagicMind optimization and MLU370 acceleration are applied to the DeepDeblur.mm model for high-speed vehicle images.
- Computation: 128.6 ms CPU ViT inference falls to 3.8 ms for the INT8 model on MLU370.The heterogeneous evaluation uses ImageNet-1K validation data to compare latency, accuracy, and utilization.
- Computation: 33 times acceleration is achieved for the optimized ViT model, while mixed precision reaches 4.5 ms with a 28-times acceleration ratio.Both optimized configurations meet the stated real-time requirements for perception-computation-control services.
- Computation: 75.8% Top-1 accuracy for INT8 incurs a 2.1% loss, while mixed precision reaches 77.2% with only a 0.7% loss.The reported losses remain within the paper's acceptable range.
- Integrated evaluation: 18.7 ms is the Phytium+MLU370 average single-frame latency, 2.89 times faster than Jetson AGX Xavier and 35.6% more efficient than Jetson Orin Nano.The MLU370 platform's data-transmission overhead ratio is 0.3%.
- Integrated evaluation: Phytium+MLU370 accuracy is broadly consistent with NVIDIA platforms, with a 0.5% to 1.0% ViT accuracy drop and no extra control-task accuracy loss.The reported ViT difference is mainly attributed to INT8 quantization, while BANG C operators preserve control numerical accuracy.
- Integrated evaluation: 82.6% MLU370 computing-unit utilization accompanies 31.6% PCIe bandwidth utilization during concurrent execution.The results associate on-chip fusion and pipeline scheduling with higher concurrent resource use and reduced cross-chip data-transfer overhead.
7 Conclusion
The conclusion presents a single-card Phytium-Cambricon deployment that combines module-specific optimization with on-chip data circulation and pipeline collaboration. It reports substantial latency and transmission reductions while retaining mainstream-platform accuracy.
- Module optimization: 3.65-fold acceleration is achieved for high-speed motion-blur restoration through MLU adaptation and MagicMind optimization.This optimization targets the perception module.
- Module optimization: 33-fold ViT inference acceleration is achieved with accuracy loss controlled within 2.1% through lightweight design, INT8 quantization, and operator fusion.These methods target the computation module.
- Module optimization: 15.28-fold acceleration is achieved for hundred-degree-of-freedom robot control calculation using customized BANG C operators.The control calculation is reduced to a lower single-frame latency.
- System architecture: A single-card integrated architecture places perception, computation, and control functions on one MLU370 accelerator card.The design uses on-chip data circulation and pipeline collaboration for high-real-time service flows.
- Overall results: Data-transmission overhead is reduced to 0.3% through on-chip data circulation in the single-card architecture.The conclusion frames the design as a replicable solution for localized engineering applications.
- Overall results: 18.7 ms end-to-end latency and 2.89-times acceleration over NVIDIA Jetson AGX Xavier are reported for the integrated system.MLU computing-unit utilization reaches 82.6%.