Source-linked AI summary
AdaptAV: Continuous Adaption of Vision Models for Autonomous Vehicles Using Cloud-based Oracle
Yuheng Zhu, Dhruva Ungrupulithaya, Boluo Ge, Man-Ki Yoon
TL;DR
Autonomous vehicles use resource-efficient perception models that may perform poorly on novel scenarios, while conventional updates are slow and resource-intensive. AdaptAV continuously uploads compressed imagery to the cloud, uses a high-accuracy oracle to guide retraining, and redeploys the updated model to the vehicle. Repeated retraining reduces validation loss, selective sampling improves accuracy, and the system maintains millisecond-scale inference, though evaluation is limited to object detection.
Problem
Resource-constrained vehicles require fast, compact perception models, but these models may perform poorly on novel scenarios and conventional retraining updates are slow and resource-intensive.
Method
AdaptAV uploads compressed vehicle imagery to the cloud, where an accurate oracle guides retraining of the small on-vehicle model before redeployment.
Results
Repeated retraining decreases validation loss, selective sampling improves model accuracy over periodic sampling, and inference takes 4.7 ms per frame for one instance or 5.03 ms concurrently.
Takeaways & Limitations
AdaptAV continuously improves an autonomous vehicle’s vision model while using cloud resources for retraining and local inference for perception.
Takeaways & Limitations
The evaluation is limited to object detection because accurate oracle models and datasets are less readily available for other autonomous-vehicle tasks.
Abstract
from arXiv · showhide
Deploying vision perception models in autonomous vehicles requires that we prioritize inference speeds, resulting in a model with shallower architectures and lesser model parameters (i.e., more pruned). Such small models do not generalize well, which could result in poor performance when encountered with novel scenarios. We propose a system that overcomes this by continuously retraining the vision models on the cloud with data uploaded by vehicles. We leverage the abundant compute resources, including machine learning accelerators, of the cloud to run a highly-accurate oracle model that will guide the retraining process of the on-vehicle model. This newly trained model is transmitted to the vehicle over the network and is utilized by the vehicle for perceptions, leading to improved inference accuracy over time.
I. INTRODUCTION
Autonomous-vehicle perception models must balance real-time inference with accuracy, but resource-efficient models can fail on novel scenarios. AdaptAV addresses this gap with continuous cloud-assisted retraining and frame sampling.
- I. INTRODUCTION: Vehicle perception models prioritize fast inference under constrained compute and memory, which can reduce accuracy in novel scenarios.The paper contrasts larger Faster R-CNN detections with smaller SSD-MobileNet outputs that miss important objects.
- I. INTRODUCTION: Conventional improvement requires collecting additional driving data and later updating the vehicle model through service-center or OTA software updates.The process is described as resource- and time-intensive and may not address all shortcomings of the previous model.
- I. INTRODUCTION: AdaptAV continuously logs and compresses vehicle image streams, uploads them to the cloud, and uses a high-accuracy oracle to retrain the on-vehicle model.The retrained model is deployed back to the vehicle, forming a closed loop.
- I. INTRODUCTION: The framework proposes frame-sampling techniques intended to reduce overfitting during retraining.It also analyzes design parameters affecting inference quality and demonstrates practicality on a prototype vehicle connected to a public cloud.
II. RELATED WORK
Related approaches distribute inference or knowledge transfer between devices and cloud or edge servers, but they impose architectural, bandwidth, or deployment constraints. AdaptAV instead performs inference locally while using a cloud oracle for continuous improvement.
- II. RELATED WORK: One-stage detectors favor vehicle deployment because they provide faster inference, whereas two-stage models generally offer better accuracy at higher cost.The passage identifies YOLO and SSD as one-stage examples and R-CNN variants as two-stage examples.
- II. RELATED WORK: Early-exit approaches split model layers between devices and edge platforms, but they rely on consistent model architectures.This creates a different constraint from AdaptAV’s decoupled device and cloud models.
- II. RELATED WORK: Knowledge-distillation approaches keep inference on the device and use a cloud teacher, but sending vehicle data for inference increases network-bandwidth requirements.The passage distinguishes this bandwidth demand from AdaptAV’s approach.
- II. RELATED WORK: Ekya uses edge servers for continuous retraining and performs vision inference there, while AdaptAV performs inference locally on autonomous-vehicle devices.AdaptAV’s frame sampling also relies on oracle predictions rather than potentially inaccurate device-model outputs.
III. ADAPTAV: CONTINUOUS ADAPTION USING CLOUD-BASED ORACLE MODEL
AdaptAV uses a cloud-based, high-accuracy oracle model to retrain the on-vehicle model continuously, targeting improved robustness to novel scenarios.
- Model roles: Faster R-CNN serves as the high-accuracy oracle, while SSD-MobileNet is the real-time model deployed on the vehicle.The oracle is intended for accuracy rather than on-vehicle inference speed.
- Evaluation: Average precision summarizes the precision-recall curve across confidence thresholds, with higher AP indicating higher accuracy.
- Evaluation: The evaluation includes 3,968 test images from a separate Waymo Open Dataset subset.Both models were trained on 35,712 images.
- AdaptAV pipeline: The oracle model provides predictions treated as ground truth for retraining the on-vehicle device model.The retrained model is sent back to the vehicle in a closed-loop process.
- AdaptAV pipeline: AdaptAV continuously captures camera images, streams them to the cloud, runs oracle inference, retrains the device model, and returns the updated model.
B. Video Compression
AdaptAV reduces cloud-upload and retraining burdens by compressing camera streams and selectively sampling frames, using oracle predictions to retain informative changes while avoiding overfitting.
- Video Compression: Raw camera frames create a bandwidth bottleneck because autonomous-vehicle sensors produce high-resolution images at high rates.The paper gives an example requiring approximately 1.4 Gbps upload capacity.
- Video Compression: H.264 compression converts raw frames into a smaller video stream; the Waymo evaluation data shrinks from approximately 286 GB to 4.2 GB, saving more than 98%.FFmpeg with the libx264 backend performs the compression.
- Frame Sampling: Frame sampling limits retraining data because using all incoming images requires substantial compute and can cause overfitting.The baseline periodically samples one image within each sampling period.
- Frame Sampling: Density-based sampling selects the frame with the largest number of oracle-detected objects within each sampling period.If counts are equal, one frame is selected randomly.
- Frame Sampling: Event-based sampling selects frames when oracle predictions indicate changes such as a new object appearing or an existing object disappearing.The approach can incorporate object tracking information.
- Frame Sampling: Unlike device-model sampling, AdaptAV uses oracle predictions, reducing reliance on potentially inaccurate on-vehicle detections during retraining.The paper distinguishes sampling from filtering because filtering errors may drop frames for extended periods in safety-critical systems.
A. Setup
The prototype implementation uses an NVIDIA AGX Orin development kit on a small-footprint vehicle, with dedicated CPU, GPU, and memory resources.
- A. Setup: The prototype runs on an NVIDIA AGX Orin Development Kit integrated with a small-footprint vehicle.The vehicle is approximately 3 ft x 6 ft.
- A. Setup: The Orin platform includes a 12-core ARM Cortex-A78AE CPU, a 2048-core NVIDIA Ampere GPU, and 64 GB of memory.Each CPU core runs at 2.2 GHz.
- A. Setup: The software stack uses NVIDIA JetPack SDK v5.1.2, Linux for Tegra 35.4.1, CUDA v11.4, cuDNN v8.6.0, and TensorRT v5.1.2.
- A. Setup: In-lab evaluation uses a workstation with an Intel i9-13900F CPU, 64 GB of memory, and an NVIDIA GeForce RTX 4090 GPU.The workstation runs Ubuntu.
2) Datasets: •
The experiments use Waymo for initial model training, BDD100K to simulate streamed driving data for continuous retraining, and an Orin device to compare model inference times.
- Datasets: The Waymo Open Dataset subset contains 39,681 frames captured at 10 Hz and is used only for initial detection-model training.The frames have 1920x1280 resolution.
- Datasets: BDD100K provides a simulated video stream of 39,973 frames captured at 5 Hz for continuous SSD-MobileNet retraining.The images have 1280x720 resolution and come from 200 driving video records.
- Models: Faster R-CNN is used as the oracle model, with 41,092,136 parameters and a 161,916 KB model file.
- Models: SSD-MobileNet is deployed on the vehicle for real-time detection, with 3,206,976 parameters and a 27,641 KB model file.
- Models: 5 ms versus 130 ms: SSD-MobileNet is faster per inference on the Orin device than Faster R-CNN.The result supports using SSD-MobileNet on the vehicle and Faster R-CNN as the oracle.
B. Results
AdaptAV’s continuous retraining improves object-detection accuracy while sampling and retraining hyperparameters determine the trade-off between training speed, accuracy, and overfitting.
- The BDD100K evaluation uses a 90% training split and 10% validation split, with both oracle and device models pretrained on Waymo and unseen BDD100K data.
- The retraining process divides BDD100K into four equal windows and deploys the updated model after convergence on each window.Each window contains about 9000 images.
- A single training epoch sharply improves performance, while continuous retraining ultimately reaches nearly 60% AP.The unretrained model achieves only 0.6% AP on BDD100K.
- Higher sampling rates produce larger training sets and higher accuracy but require longer convergence times, whereas low sampling rates converge faster with lower accuracy.Retraining window size and sample rate must be selected according to cloud resources and camera sampling rate.
- Density-based and event-based sampling improve accuracy and prevent overfitting compared with periodic sampling, with event-based sampling performing best overall.Density-based sampling is preferable when computing power is limited because event-based sampling requires object tracking.
C. Usecase
A prototype shuttle demonstrates AdaptAV by uploading camera frames to the cloud, using cloud-generated labels to retrain its on-vehicle detector, and improving accuracy over repeated windows.
- The prototype runs on an NVIDIA Jetson Orin with a ZED2 stereo camera and captures approximately 14,000 frames at 15 frames per second.
- Compressed frames are sent individually to Google Cloud, where periodic sampling with a window size of five supports four retraining windows.Faster R-CNN generates the ground truth used to retrain the vehicle’s SSD-MobileNet model.
- The final validation loss decreases across retraining windows, indicating improved accuracy for the on-vehicle model.
- Cloud execution reduces Faster R-CNN inference time from 130 ms on Jetson Orin to 28 ms and cuts 10-epoch training time from 422 to 178 seconds.
D. Discussion •
The discussion addresses deployment overhead, model flexibility, and security while identifying oracle availability and task coverage as practical boundaries.
- TLS, storage encryption, and confidential VMs are used to protect vehicle vision data in transit, at rest, and during cloud computation.Confidential VMs use hardware-based memory encryption and isolate data from other tenants and the cloud provider.
- Running retrained and existing SSD-MobileNet models simultaneously adds less than 1 ms to per-frame inference time on Jetson Orin.A single instance takes 4.7 ms per frame, compared with 5.03 ms during concurrent execution.
- AdaptAV is model-agnostic, and more complex oracle architectures such as vision transformers could provide more accurate ground truth.
- Evaluation is limited to object detection because accurate oracle models and datasets are less readily available for tasks such as lane detection, steering prediction, and segmentation.
V. CONCLUSION
AdaptAV uses a cloud oracle and uploaded camera streams to continuously retrain a small vehicle model, while frame sampling improves accuracy and reduces overfitting risk.
- The cloud runs a highly accurate oracle that guides retraining of the vehicle’s smaller perception model using live camera data.
- Frame sampling techniques can reduce the chance of overfitting and improve prediction accuracy during continuous retraining.
- Collecting image streams from multiple vehicles is proposed as future work to improve robustness across diverse novel scenarios.