Source-linked AI summary
2023 Low-Power Computer Vision Challenge (LPCVC) Summary
Leo Chen, Benjamin Boardley, Ping Hu, Yiru Wang, Yifan Pu, Xin Jin, Yongqiang Yao, Ruihao Gong, Bo Li, Gao Huang, Xianglong Liu, Zifu Wan, Xinwang Chen, Ning Liu, Ziyi Zhang, Dongping Liu, Ruijie Shan, Zhengping Che, Fachao Zhang, Xiaofeng Mou, Jian Tang, Maxim Chuprov, Ivan Malofeev, Alexander Goncharenko, Andrey Shcherbin, Arseny Yanchenko, Sergey Alyamkin, Xiao Hu, George K. Thiruvathukal, Yung Hsiang Lu
TL;DR
The 2023 LPCVC addresses semantic segmentation for post-disaster UAV imagery under embedded-device resource constraints. It evaluates solutions by balancing segmentation accuracy with inference time and highlights winning architectures and optimizations. The winning teams demonstrated distinct trade-offs among score, accuracy, and speed, while the competition remained bounded by its 14-class labeling scheme and supported runtime frameworks.
Problem
The challenge is to achieve accurate computer vision on embedded devices, where improving accuracy often increases model size and resource demands.
Method
The competition evaluates semantic segmentation solutions for post-disaster UAV imagery using accuracy and inference time, and analyzes the winning teams’ model designs and optimizations.
Results
Winning solutions used Yocto-Revival Network, TopFormer [24], and PIDNet, achieving different score, accuracy, and inference-time outcomes.
Takeaways & Limitations
The 2023 results show that low-power semantic segmentation can be pursued through varied architectures and optimization strategies tailored to the target device and runtime.
Takeaways & Limitations
The competition’s 14 permitted pixel classes do not fully reflect realistic disaster scenes, and acceptable frameworks were restricted by Jetson Nano support constraints.
Abstract
from arXiv · showhide
This article describes the 2023 IEEE Low-Power Computer Vision Challenge (LPCVC). Since 2015, LPCVC has been an international competition devoted to tackling the challenge of computer vision (CV) on edge devices. Most CV researchers focus on improving accuracy, at the expense of ever-growing sizes of machine models. LPCVC balances accuracy with resource requirements. Winners must achieve high accuracy with short execution time when their CV solutions run on an embedded device, such as Raspberry PI or Nvidia Jetson Nano. The vision problem for 2023 LPCVC is segmentation of images acquired by Unmanned Aerial Vehicles (UAVs, also called drones) after disasters. The 2023 LPCVC attracted 60 international teams that submitted 676 solutions during the submission window of one month. This article explains the setup of the competition and highlights the winners' methods that improve accuracy and shorten execution time.
2023 LPCVC
The 2023 LPCVC focuses on semantic segmentation of post-disaster UAV imagery, requiring models to balance accuracy with execution time on resource-limited embedded devices.
- 2023 LPCVC: The challenge requires semantic segmentation, categorizing every image pixel into one of 14 classes relevant to disaster scenes.The evaluation uses UAV-captured images, including a 600-image, 512 × 512 test set with hand-labeled ground truth.
- 2023 LPCVC: The competition promotes accurate yet efficient models because disaster-response segmentation requires reliable assessments, while drones have limited onboard computing resources.UAVs can support rapid scene assessment and navigation after disasters, but their lightweight design constrains available computation.
- 2023 LPCVC: The evaluation tracks accuracy and computational efficiency, reflecting the need for segmentation maps that correctly represent classes present in UAV disaster imagery.Figure 1 reports the highest score, accuracy, and lowest execution time for each competition day; scoring paused during an outage and the competition was extended by four days.
- REFERENCE SOLUTION: The reference solution, based on FANet, scored 50.11 accuracy with 200ms average inference time and established the qualification standard.Submissions were disqualified if they performed worse than the sample solution on either accuracy or time.
EVALUATION
The evaluation balances segmentation accuracy and computational efficiency on a power-limited Jetson Nano, using mDSC, mean inference time, and their ratio. An automated referee system runs submissions on 600 test images, computes these metrics, and returns the resulting score.
- Accuracy: mDSC averages per-class Dice scores across each image and then across the test set to produce the submitted model’s accuracy score.The class set is the union of classes appearing in the prediction map and ground truth; TP, FP, and FN quantify pixel-level outcomes.
- Accuracy: An extra predicted class can reduce mDSC from approximately 1.0 to 0.75 even when the three ground-truth classes are otherwise nearly perfectly predicted.This strict treatment penalizes false positives and false negatives because incorrect UAV segmentation could trigger or miss critical search-and-rescue actions.
- Efficiency: Mean inference time measures the time required to process model inputs and generate outputs, averaged over the test images.Inference time is defined per model operation, while the test set contains N images.
- Scoring: The competition score is the ratio of accuracy to inference time, explicitly capturing the trade-off between predictive quality and computational efficiency.The evaluation runs on an NVIDIA Jetson Nano 2GB Developer Kit in a 5W power-efficient mode.
- Referee System: The automated referee queues uploaded zip files, executes them on the Jetson Nano, evaluates 600 test images, computes mDSC and mean inference time, and sends scores to the web server.Submissions must produce the required 512x512 prediction maps and total inference-time output, or they are disqualified.
Winners’ Solutions
The winning teams combined lightweight segmentation architectures with quantization, pruning, distillation, re-parameterization, and TensorRT-oriented optimizations to balance accuracy and inference speed. Their solutions used different design choices, with ModelTC emphasizing efficiency, AidgetRock speed, and ENOT accuracy.
- Team ModelTC: 75.608 score, 51.2% mDSC, and 6.8 ms average inference time made ModelTC the first-place team with its UNet-based Yocto-Revival Network.ModelTC used dynamic network techniques and further improved efficiency through re-parameterization and batch-size optimization.
- Team AidgetRock: 55.4% mDSC, 15 ms, and a 36.9 score made AidgetRock the speed-award winner using a lightweight TopFormer-based model.The team applied pruning, knowledge distillation, feature pyramids, transformer attention, and ImageNet pre-training to balance efficiency and accuracy.
- Team AidgetRock: AidgetRock found that pruning caused less than a 1% Dice-accuracy drop while increasing speed by more than 10%, and knowledge distillation recovered 2% accuracy at best.They also used standard and Mixup/Mosaic augmentation while additional data reduced performance.
- Team AidgetRock: ImageNet pre-training of AidgetRock’s modified TopFormer backbone improved accuracy by more than 10% after the original weights failed to meet the cutoff.The optimized classification weights initialized the segmentation network.
- Team ENOT: 60.1% mDSC, 67 ms, and an 8.974 score made ENOT the accuracy-award winner with a PIDNet-based solution.ENOT replaced the final convolution with transposed convolution, removed softmax, used additional UAVid samples, and applied prediction-cleaning heuristics.
- Commonalities of the teams: TensorRT improved edge inference through precision, operator fusion, auto-tuning, dynamic tensor memory, and multi-thread execution, but required Nvidia GPU, Ubuntu, and CUDA.These optimizations target throughput, memory use, kernel selection, and parallel processing.
Future Competitions
Future LPCVC organizers should prepare for operational disruptions, communicate rules precisely, test systems early, and address constraints in labels and supported frameworks.
- Future Competitions: Precise rules and FAQs are necessary because misunderstanding the 512x512 input assumption caused one team to lose performance by forgoing model quantization.The competition used Slack for clarification and participant assistance, with more than 1000 messages exchanged.
- Future Competitions: A pre-competition system test can detect errors, verify compatibility with other solutions, and familiarize participants with the evaluation system.The 2023 competition held its system test two weeks before the start.
- Future Competitions: Limiting segmentation to 14 pixel-label classes does not reflect realistic disaster scenes and may explain accuracy losses when two winning teams added data.The restricted label space is identified as a competition issue rather than a property of disaster imagery.
- Future Competitions: Restricted frameworks on the Jetson Nano limit competition participation, motivating submission methods that mimic competitors’ testing environments.Organizers cannot support every library, framework, or GPU optimization technique.
- Future Competitions: A power outage kept the submission and evaluation systems offline for several days, requiring four days to be added to the month-long competition.Organizers should be prepared to adapt immediately and effectively to unpredictable circumstances.
CONCLUSION
The 2023 LPCVC evaluated low-power semantic-segmentation solutions and showcased diverse winning architectures. The organizers aim for the paper to inform future low-power computer-vision models and competition design.
- CONCLUSION: The 2023 LPCVC results featured three distinct winning models: ModelTC’s Yocto-Revival Network, AidgetRock’s TopFormer [24], and ENOT’s PIDnet.The paper reports and analyzes the challenge results for semantic segmentation.
- CONCLUSION: The paper aims to educate and inspire developers of future low-power computer-vision models and guide the design of future competitions.Its stated goal is to support further growth in the field.