Source-linked AI summary
A lightweight C Code Generator for Heterogeneous TinyML Systems
Chen Liu, Bernhard Vogginger, Christian Mayr
TL;DR
Deploying models across diverse edge hardware remains complex, with efficiency and compatibility constraints in existing approaches. This paper presents a lightweight C generator with ahead-of-time optimizations; in a radar-DSP case study, it reports lower execution time and binary size than microTVM-based flows.
Problem
Deploying models across diverse edge devices is complex, and existing tools face C++ binary-size overhead, hardware dependence, and limited support for custom DSAs.
Source: §1 · pp. 1–2Method
A lightweight generator extracts model representations and produces backend-compatible C code, using weight tuning and bias fusion as ahead-of-time optimizations.
Source: §2.2 · p. 2, §4 · p. 5Results
Execution time and binary size fall to 13.8% and 63.6% of the baseline microTVM flow, respectively, and the generator is more than twice as fast as microTVM+UMA.
Source: §3.2 · p. 4Takeaways & Limitations
Adapting the generator to another backend requires preparing templates for its required operators.
Source: §4 · p. 5Weight tuning and bias fusion introduce truncation errors, though experiments report minimal impact on model accuracy.
Source: §2.2 · p. 3
Abstract
from arXiv · showhide
Over the past decade, the significant advancements in artificial intelligence have been closely tied to increasingly sophisticated training frameworks and steadily powerful computing hardware. However, the entire ecosystem remains fragmented. On one hand, models are represented with frameworks dependent formats, while on the other hand, new tailored hardware for accelerating novel operators has continued to emerge. To effectively diminish the transfer costs from training to inference, we propose a lightweight C code generator designed for bare-metal heterogeneous systems that incorporate domain-specific accelerators. Through a comprehensive case study on a RISC-V based radar DSP, we demonstrate that by employing an accelerator with ahead-of-time optimizations like weight tuning and bias fusion, our approach achieves a 2.3 times improvement in inference speed and a 20% reduction in binary size compared to the microTVM flow registered with the same accelerator, thus offering another compelling choice for resource-constrained edge devices. The code is open-sourced in Gitlab.
1 Introduction
Deploying models across diverse edge devices is complex, and runtime efficiency and deployment effort are not guaranteed. Existing approaches also face binary-size, hardware-dependence, and custom-accelerator support constraints; the proposed lightweight C generator targets these gaps.
1 Introduction
- Edge deployment across diverse hardware is complex, with runtime performance and human-resource efficiency not guaranteed.The paper frames this as a challenge in deploying lightweight models on edge devices.Source: §1 · p. 1
- Current embedded deployment tools can retain C++ overhead, while vendor toolkits may lock users to proprietary hardware.The introduction notes that C usage exceeds C++ usage by more than twofold in an embedded market study.Source: §1 · p. 1
- Existing deployment support is limited for custom domain-specific accelerators, and adding MLIR operators requires substantial infrastructure work.The paper also identifies floating-point rescaling overhead for quantized weights as an overlooked deployment cost.Source: §1 · p. 2
- The proposed generator accepts trained models, applies weight tuning and bias fusion, and emits editable C source with templates for customized accelerators.The authors present it as supporting deployment across diverse edge devices and both standard and specialized hardware.Source: §1 · p. 2
2 Implementation
The implementation uses a modular pipeline to optimize model operators and generate backend-compatible C source from model representations. Its frontend, ahead-of-time optimizations, and DSA adaptation prepare models for code generation while keeping the output customizable.
2 Implementation
- Pure C output targets broad embedded compatibility, while templates map operators to editable backend functions and leave final compilation to the backend toolchain.The design is intended to keep binaries streamlined and minimize potential compilation errors.Source: §2 · p. 2
2.1 Overview
- The pipeline parses a model into intermediate representations, optimizes operators, optionally adapts them for a DSA, then generates source code and stores weights and activations.The first two modules optimize the model; DSA optimization can be skipped when no accelerator is attached.Source: §2.1 · p. 2
2.2 Modules
- The frontend currently parses TFLite models, extracts layerwise weights, activations, quantization parameters, and operator attributes, and supports core CNN operators.Supported operators include conv2d, dense, flatten, and maxpool; users can extend the parser.Source: §2.2 · p. 2
- Weight tuning and bias fusion target CPU-side rescaling and bias addition in quantized inference, using ahead-of-time optimization for weighted-sum layers.The paper describes the process for fully connected and 2D convolutional layers; weight tuning can apply to quantized weighted-sum layers, while bias fusion applies when bias is enabled.Source: §2.2 · p. 2
- Bias fusion folds bias addition into the weighted sum by extending fully connected matrices or padding convolution inputs and kernels.These optimizations introduce truncation errors, but experiments report minimal impact on model accuracy.Source: §2.2 · p. 3
- DSA adaptation checks operator and feature-map limits, converts tensor or weight layouts when needed, verifies alignment, and uses ping-pong buffers to reduce allocated memory.Width padding may align values to boundaries such as 128 bits.Source: §2.2 · p. 3
3 Evaluation
The evaluation demonstrates a codegen-based RadarNet deployment on a custom radar DSP, benchmarks it against TVM-based flows, and discusses backend adaptation and optimization.
3 Evaluation
- The study deploys a CNN-based radar target classifier on a custom radar DSP with an accelerator for GEMM and convolution.Users provide input files, generate C source, and compile it with the backend’s default toolchain.Source: §3 · p. 4, §3.1 · p. 4
3.1 Case Study
- RadarNet has 9,704 parameters and achieves 92% accuracy on four radar-image classes after static int8 quantization.Ping-pong buffers alternate layer-output storage, with generated buffer sizes based on the maximum tensor size assigned to each buffer.Source: §3.1 · p. 4
- The custom radar DSP has 25 processing elements, and each of its 24 signal-processing cores combines RISC-V, 128 kB of memory, and an MLA with 64 MAC units.The MLA accelerates integer GEMM and convolution and supports bit-shifted outputs.Source: §3.1 · p. 4
3.2 Result
- 13.8% of microTVM’s execution time and 63.6% of its binary size are achieved by the generator, relative to the microTVM baseline.Table 2 reports single-inference time, compiled binary size, and manual intervention.Source: Table 2 · p. 4, §3.2 · p. 4
- More than double the speed of microTVM+UMA is achieved by the generator through compile-time dimension reordering and weight tuning for rescaling.Compile-time IR optimizations also reduce binary size, according to the authors.Source: §3.2 · p. 4
- The generator’s overall code size is more than twice UMA’s, but the incremental code size for registering new operators is comparable to TVM’s.The comparison reflects that the generator implements all RadarNet operators and additional IR optimizations, whereas UMA focuses on accelerator-mapped operators.Source: §3.2 · p. 5
3.3 Discussion
- The generator implements common CNN operators and adapts to new backends through operator templates, requiring a one-time effort to build a comprehensive set.The authors describe the workflow as resembling TVM’s while aiming for a turnkey solution with minimal external dependencies and a lightweight runtime.Source: §3.3 · p. 5
4 Conclusion
The paper presents a lightweight generator that translates TFLite model IRs into backend-compatible template-based C, demonstrated in radar target classification.
4 Conclusion
- The generator translates IRs extracted from TFLite models into template-based C compatible with a backend chip, and the radar-DSP case study reports the smallest executable size and lowest execution time among compared methods.Adapting to another backend requires preparing templates for the required operators.Source: §4 · p. 5