PaperlayerView exact revision on arXiv

Source-linked AI summary

On-Device Super-Resolution Imaging for a Low-Cost SPAD Array on a 640-KB-SRAM Microcontroller

Zhenya Zang, Istvan Gyongy, Mike Davies

arXiv:2610.09098v1eess.IV

TL;DR

Compact SPAD sensors provide low-resolution depth and intensity images, while limited MCU memory makes super-resolution deployment challenging. The paper develops a compact, firmware-integrated approach for jointly upscaling both modalities on an STM32H563ZI. Synthetic experiments improve over bicubic reconstruction, and real measurements produce denser maps, while INT8 quantization degrades absolute depth accuracy.

  • Problem

    Super-resolution on resource-constrained MCUs is challenging because high-resolution intermediate features consume SRAM, while the target platform also needs memory for firmware and image data.

    Source: §I · p. 1, §II · p. 3

  • Method

    The framework uses separate depth and intensity branches with lightweight backbones, a two-stage PixelShuffle reconstruction head, and simplified sensor firmware for MCU deployment.

    Source: §I · p. 2, A. Network Architecture · p. 3, §VII · p. 8

  • Results

    Synthetic experiments consistently improve over bicubic reconstruction, real measurements show substantially denser depth, intensity, and pointcloud maps, and INT8 quantization measurably degrades absolute depth accuracy.

    Source: §VII · p. 8

  • Takeaways & Limitations

    The framework enables joint depth-and-intensity super-resolution on an STM32H563ZI with only 640 KiB SRAM.

    Source: §VII · p. 8

    The work does not introduce a new architecture and uses ST Edge AI's predefined deployment interface, leaving lower-level kernel, DSP, and buffer-management optimizations open.

    Source: B. Design Constraint and Potential… · p. 8

Abstract

from arXiv · show

This work presents a super-resolution (SR) deep-learning (DL) architecture to simultaneously upscale low-resolution (LR) depth and intensity images acquired by a consumer-grade VL53L9CX time-of-flight (ToF) sensor, a compact single-photon avalanche diode (SPAD) ranging device, while targeting deployment on a resource-constrained microcontroller (MCU) with only 640 KB SRAM and 2 MB Flash. The VL53L9CX provides 54 $\times$ 42 depth and intensity measurements. The proposed framework performs $\times$4 spatial SR to reconstruct 216 $\times$ 168 depth and intensity images simultaneously. The network employs separate reconstruction branches for intensity and high-resolution (HR) depth. We train the network with efficient loss functions and investigate the performance by testing multiple compact, hardware-oriented backbones, including Spatially Adaptive Feature Modulation (SAFM), Swift Parameter-Free Attention Network (SPAN), and Residual Local Feature Network (RLFN). We evaluate the network on both synthetic and real measurements, with particular attention to constrained activation and memory storage. We export the selected network to ONNX, quantize it to INT8, generate C inference code for an STM32H563ZI Arm Cortex-M33 MCU, and integrate it with our simplified sensor's firmware. This work is the first to demonstrate a compact DL SR model on a low-cost bare-metal MCU for a consumer-grade LR SPAD sensor, integrated with customized MCU firmware to enable on-device SR inference.

I. INTRODUCTION

Compact SPAD sensors offer increasingly dense measurements, but memory limits still constrain camera-comparable resolution. This work targets joint depth-and-intensity super-resolution on a severely constrained MCU through compact modeling, evaluation, and firmware-aware deployment.

I. INTRODUCTION

  • Direct ToF sensing measures photon travel time, and SPAD-based systems use photon-arrival statistics to estimate depth.The introduction contrasts direct ToF with indirect ToF approaches that measure phase shift.Source: §I · p. 1
  • Limited on-chip memory constrains fine temporal resolution, long-range measurements, and camera-comparable image resolution in compact SPAD sensors.The sensor modules cost under USD 65 and have reached arrays of 54×42 ranging zones.Source: §I · p. 1
  • On an STM32H563ZI with 640 KB SRAM and 2 MB Flash, intermediate activations—not only model parameters—are a central memory constraint during upscaling.Firmware must also reserve memory for calibration and LR and HR image storage.Source: §I · p. 1
  • The work trains a compact model for joint depth-and-intensity super-resolution from VL53L9CX measurements, targeting processing on an STM32H563ZI evaluation board.The reported total system cost is approximately USD 120.Source: §I · p. 1
  • Accuracy and model complexity are evaluated on synthetic and real-world data, with backbone, weight-parameter, and loss-function studies.Multiple deep-learning backbones are trained and evaluated.Source: §I · p. 2
  • The framework uses a bandwidth-friendly architecture with simplified PixelShuffle upscaling and an automated quantization and MCU implementation pipeline integrated with simplified sensor firmware.The stated aim is compact integration and on-device super-resolution imaging.Source: §I · p. 2

II. RELATED WORK

Prior compact-SPAD super-resolution work spans camera fusion, offline models, and embedded inference on platforms with substantially more memory. MCU deployment of super-resolution remains difficult because high-resolution intermediate features consume substantial SRAM.

II. RELATED WORK

  • Camera-fusion SR studies use histograms and high-resolution RGB inputs and are trained and evaluated on GPUs; other platforms include a 16 GB embedded neural processing unit.A cited GAP9 MCU work instead uses SPAD depth measurements to generate pseudo-labels for on-device fine-tuning of an RGB-to-depth model.Source: §II · p. 2
  • Several compact-SPAD approaches use LR images from a single sensor, but reported examples either lack MCU integration or run with about 2 GB of memory and operating-system support.The review states that fully on-MCU, non-fusion SR for compact SPAD ToF sensors remains lacking.Source: §II · p. 2
  • SR deployment on a submegabyte MCU is difficult because dense high-resolution intermediate features create large activation tensors and consume substantial SRAM.The passage identifies simplified design, quantization, feature-map control, and activation-memory allocation as deployment needs.Source: §II · p. 3
  • MCU toolchains and co-designed runtimes support quantized operators and optimized kernels, but super-resolution additionally requires control of feature-map dimensions and activation-memory allocation.The related-work discussion names MCUNet, TensorFlow Lite Micro, CMSIS-NN, and STM32Cube.AI.Source: §II · pp. 2–3

III. METHOD

The method jointly reconstructs high-resolution depth and intensity through separate branches, with losses tailored to each modality and an upsampling design constrained by MCU memory. It compares lightweight backbones and finds feature-channel count more useful than block count for improving intensity PSNR.

A. Network Architecture

  • The network maps paired 42×54 LR depth and intensity inputs to 168×216 outputs at ×4, using separate branches to limit intensity-texture propagation into depth.Depth primarily represents scene geometry, while intensity reflects returned optical signal strength.Source: A. Network Architecture · p. 3
  • Depth loss combines Charbonnier, Sobel-gradient boundary matching, TV artifact suppression on planar regions, and SSIM structure penalties; gradient and TV weights are tuned jointly.The gradient term penalizes edge displacement, while TV suppresses high-frequency artifacts on planar surfaces.Source: A. Network Architecture · p. 3
  • Intensity loss uses Charbonnier, gradient, and structural terms but omits TV because reflectance texture is signal, not artifact.The gradient and structural terms are the intensity branch’s counterparts to those in the depth objective.Source: A. Network Architecture · p. 3
  • SAFM, RLFN, and SPAN provide alternative lightweight feature backbones, but feature maps, unsupported operators, and quantization boundaries may dominate MCU deployment cost.Their attention mechanisms differ, including parameter-free attention in SPAN.Source: A. Network Architecture · p. 3
  • Intensity PSNR improves steadily with feature channels, while block count changes neither metric beyond run-to-run spread; #B is fixed at 1 thereafter.Figure 2 sweeps #FC at #B = 3 and #B at #FC = 12 on Middlebury test scenes across three backbones.Source: Figure 2 · p. 4, A. Network Architecture · p. 4
  • Two-stage ×2 PixelShuffle reduces the largest paired activation footprint from 850 KB to 142 KB, 5.99× smaller and within the 640 KB SRAM budget.The conventional single-stage ×4 head generates 850 KB of INT8 features, exceeding available SRAM.Source: A. Network Architecture · pp. 3–4

B. Training Strategy

Training uses paired low- and high-resolution depth and intensity images, with augmentation and a shared optimization schedule. Modality-specific composite losses guide checkpoint selection.

B. Training Strategy

  • Training pairs stacked 54×42 depth and intensity inputs with 216×168 ground truth, using an 85%/15% train-validation split and horizontal and vertical flips.The paired data comprise LR and HR depth and intensity images.Source: B. Training Strategy · p. 4
  • All backbones train for 200 epochs with batch size 32 using AdamW, five-epoch linear warmup, cosine decay, and gradient clipping at norm 1.0.The initial learning rate is 1×10^-4 and weight decay is 1×10^-3.Source: B. Training Strategy · p. 4
  • Depth and intensity use separately weighted composite losses, and checkpoints are saved when both validation SSIM and PSNR improve.Depth combines Charbonnier, SSIM, TV, and Sobel-gradient terms; intensity omits TV.Source: B. Training Strategy · p. 4

C. Model Implementation and Firmware Customization

The implementation trims sensor preprocessing to the kernels needed for network inputs, reducing SRAM use and latency before joint depth-and-intensity inference. The quantized model is compiled to C and integrated with sensor firmware for STM32H563ZI deployment.

C. Model Implementation and Firmware Customization

  • The model is exported from PyTorch to ONNX for static memory planning, quantized to INT8, compiled into C sources, then integrated with sensor firmware and programmed onto the STM32H563ZI.The generated code stores weights in Flash and activations in compile-time-sized SRAM.Source: Figure 3 · p. 5, C. Model Implementation and Firmware… · p. 5
  • Retaining extraction, distance calibration, and radial-to-perpendicular conversion avoids approximately 67.7 KB of SRAM and reduces preprocessing time 5.4×, from 19.5 ms to 3.6 ms per frame.Acquisition with the reduced preprocessing pipeline takes approximately 57 ms per frame; joint depth-and-intensity inference takes 1425 ms.Source: C. Model Implementation and Firmware… · pp. 4–5
  • SPAN runs in 0.79 ms versus 1.72 ms for SAFM and 1.40 ms for RLFN, despite SAFM having the fewest parameters and MACs; RLFN needs the least RAM and SPAN the least Flash.SAFM's slicing, resizing, pooling, concatenation, element-wise operations, and layout transformations add memory-access and tensor-manipulation costs not reflected in MAC count.Source: C. Model Implementation and Firmware… · p. 5
  • UART transfer supports offline visualization at approximately 149 ms per frame, while LCD display over SPI achieves approximately 79 ms per HR frame.The prototype combines the MCU evaluation board, sensor board, and LCD; the heap reservation is reduced from 128 KB to 92 KB.Source: C. Model Implementation and Firmware… · p. 5, Figure 4 · p. 5

A. Dataset

The training data combine NYU Depth v2 and Hypersim, while the synthetic test set uses Middlebury scenes. A zone-based model synthesizes sensor-like LR inputs, using return-strength winner-take-all for depth and averaging for intensity.

A. Dataset

  • Training pairs combine 2,000 NYU Depth v2 image pairs from 16 scenes with 2,000 Hypersim pairs from 44 scenes; Middlebury's eight stereo scenes form the synthetic test set.The HR depth and intensity maps have ground truth, and LR inputs are synthesized to match the VL53L9's 54 × 42 zone geometry.Source: A. Dataset · p. 5
  • Per-zone depth is selected by winner-take-all on return strength w = I/d^2, while intensity is averaged over the HR pixels in each zone.This models the stronger return determining measured distance when two surfaces fall within one illumination cone.Source: A. Dataset · pp. 5–6
  • The Middlebury test tables report depth and intensity reconstruction from 54 × 42 to 216 × 168, averaged over eight scenes.Source: Table III · p. 6, Table IV · p. 6

A. Evaluation on Synthetic Data

On synthetic Middlebury scenes, the learned models improve depth reconstruction over bicubic interpolation, and SAFM leads the reported intensity metrics. Real captures show sharper depth and intensity structure, while the reported depth quantization changes are larger than the intensity changes.

A. Evaluation on Synthetic Data

  • Depth RMSE falls from 0.051 meter for bicubic interpolation to 0.042 meter for all three learned models; SAFM improves PSNR by 0.83 dB and E-PSNR by 1.07 dB over bicubic interpolation.The δ thresholds are nearly saturated across methods for the considered 0.4 m to 2.3 m range; all learned models have rounded GMSD 0.0014 versus 0.0017 for bicubic interpolation.Source: A. Evaluation on Synthetic Data · p. 6
  • In Art, thin rods and ring outlines appear continuous after SR, but Reindeer antlers merged in the LR input cannot be reliably reconstructed as separated geometry.Depth level-set curves expose boundary placement errors that the color map can hide; recovered book text, laundry window mullions, and Moebius plaid are more legible than with bicubic interpolation.Source: A. Evaluation on Synthetic Data · p. 6
  • INT8 changes SAFM intensity PSNR from 27.71 to 27.68 dB and SSIM from 0.8226 to 0.8213, while depth RMSE rises from 0.0423 m to 0.0470 m and AbsRel from 0.0093 to 0.0158.The PyTorch FP32 and ONNX FP32 models produce identical results; δ1 and δ2 remain unchanged after quantization.Source: A. Evaluation on Synthetic Data · p. 6

B. Evaluation on Real Data

  • In seven real captures, SR resolves the subject's outline, arm-torso gap, and raised hand; the HR point cloud separates the person from the wall behind.The intensity images also reveal facial features and clothing texture, although photometric levels remain unchanged.Source: B. Evaluation on Real Data · p. 6

VI. DISCUSSION

The discussion tests loss-weight sensitivity and shared-stem computation, finding small overall weight-sweep changes alongside a speed–accuracy trade-off from sharing.

A. Ablation Study

  • Across 12 one-factor-at-a-time runs, varying wTV, wSSIM,d, and wSSIM,i over an order of magnitude did not change performance; intensity PSNR spanned approximately 0.15 dB overall.Increasing w∇d from 0.2 to 0.5 worsened depth RMSE and AbsRel, while the sweep suggested w∇d = 0.05, wi = 3.0, and w∇i = 0.2 for further joint tuning.Source: A. Ablation Study · p. 7
  • Increasing wi to 3.0 improved several metrics in the one-factor sweep, while w∇d = 0.5 yielded the lowest intensity PSNR, SSIM, and E-PSNR; GMSD favored sharper reconstructions.The study varied one weight at a time while holding the other five at baseline values.Source: A. Ablation Study · p. 7
  • Depth RMSE rose from 0.042 m to 0.045 m for SAFM and RLFN, and to 0.048 m for SPAN, while sharing reduced host-GPU latency.Latency fell to 0.48 ms for SAFM, 0.31 ms for SPAN, and 0.41 ms for RLFN; intensity PSNR also decreased, with SPAN showing the largest accuracy degradation.Source: A. Ablation Study · p. 7, Table V · p. 7

B. Design Constraint and Potential Improvement

The paper identifies deployment-pipeline and ROI processing as possible efficiency improvements, while concluding that its SR system runs on a bare-metal MCU with constrained memory.

B. Design Constraint and Potential Improvement

  • Further efficiency may come from lower-level kernel, DSP, and buffer optimization or ROI-based SR, whose effectiveness requires assessing ROI-selection accuracy and computational savings.Uniform SR of large background regions incurs unnecessary computation and latency; ROI-based reconstruction is proposed to restrict processing to regions needing greater detail.Source: B. Design Constraint and Potential… · p. 8

VII. CONCLUSION

  • INT8 quantization preserved intensity reconstruction closely but measurably degraded absolute depth accuracy; real measurements yielded denser depth, intensity, and pointcloud outputs.The system performs SR on a bare-metal STM32H563ZI MCU with 640 KiB SRAM, without a GPU, embedded Linux processor, or dedicated neural processing unit.Source: §VII · p. 8
Loading 2610.09098v1…