Source-linked AI summary
Ekya: Continuous Learning of Video Analytics Models on Edge Compute Servers
Romil Bhardwaj, Zhengxu Xia, Ganesh Ananthanarayanan, Junchen Jiang, Nikolaos Karianakis, Yuanchao Shu, Kevin Hsieh, Victor Bahl, Ion Stoica
TL;DR
Edge video analytics must continually retrain compressed models as live data drifts, while sharing limited GPU resources with inference. Ekya balances these competing tasks across models using scheduling and micro-profiling. It achieves 29% higher accuracy than a baseline scheduler, while the baseline needs 4× more GPU resources to match Ekya’s accuracy.
Problem
Compressed edge models face data drift, while jointly supporting inference and retraining requires balancing model accuracy against inference accuracy under limited resources.
Method
Ekya uses a resource scheduler and micro-profiler to prioritize promising retraining tasks and estimate their accuracy benefits and resource costs.
Results
29% higher accuracy is achieved than a baseline scheduler, while the baseline requires 4× more GPU resources to achieve Ekya’s accuracy.
Takeaways & Limitations
Ekya makes joint continuous retraining and inference practical on edge servers by balancing retraining gains against temporary inference losses.
Takeaways & Limitations
The scheduler techniques are evaluated for other optimization metrics only as future work, including max-min accuracy.
Abstract
from arXiv · showhide
Video analytics applications use edge compute servers for the analytics of the videos (for bandwidth and privacy). Compressed models that are deployed on the edge servers for inference suffer from data drift, where the live video data diverges from the training data. Continuous learning handles data drift by periodically retraining the models on new data. Our work addresses the challenge of jointly supporting inference and retraining tasks on edge servers, which requires navigating the fundamental tradeoff between the retrained model's accuracy and the inference accuracy. Our solution Ekya balances this tradeoff across multiple models and uses a micro-profiler to identify the models that will benefit the most by retraining. Ekya's accuracy gain compared to a baseline scheduler is 29% higher, and the baseline requires 4x more GPU resources to achieve the same accuracy as Ekya.
1 Introduction
Edge video analytics must continually adapt compressed models to data drift while sharing limited GPU resources between inference and retraining. Ekya addresses this multi-model scheduling problem by estimating retraining benefits and costs, then prioritizing useful configurations and tasks.
- Motivation: 22% accuracy loss can result from data drift when live video characteristics diverge from compressed models’ training data.Continuous retraining on recent samples helps models maintain accuracy as scenes change.
- Motivation: Continuous retraining improves models but competes with inference for GPU resources, creating a tradeoff between faster retraining and temporary inference degradation.Resource taken from inference can force frame sampling, lowering accuracy while retraining proceeds.
- Problem formulation: Each retraining window requires selecting models, allocating GPU resources, and choosing retraining and inference configurations while maintaining a minimum inference-accuracy constraint.The objective is to maximize inference accuracy averaged across the retraining window and videos.
- Problem formulation: Configuration choices create a large, changing decision space because hyperparameters affect both resource demand and accuracy, while video characteristics change across windows.Estimating each configuration’s performance directly would require actually retraining with it.
- Ekya approach: Ekya prioritizes retraining streams most affected by data drift and prunes the decision space using coarse GPU allocations and performance estimates from micro-profiling.The micro-profiler observes configurations on sampled data for a few epochs and estimates accuracy and resource demands.
2 Continuous training of models on edge compute
Edge servers support video analytics because they reduce network costs and preserve privacy, but their compressed models lose accuracy as video conditions drift. Continuous retraining on recent data restores accuracy while requiring careful configuration choices under constrained edge resources.
- Edge computing: Edge deployments process video on-premise to avoid expensive or unreliable cloud links and to preserve video privacy.Typical edge servers support many streams using customized analytics and models.
- Compressed models and drift: Compressed DNNs are used because large, accurate models are expensive and slow for resource-constrained edge servers.Edge servers may have weak GPUs, motivating specialized and compressed models.
- Compressed models and drift: Continuous learning periodically retrains edge models on newly accumulated video samples as class distributions and object appearances change over time.The Cityscapes evaluation divides data into fixed retraining windows to capture these changes.
- Accuracy benefits: Continuous retraining achieves up to 22% higher accuracy than the other evaluated training options in the Cityscapes example.Training only on earlier windows or other cities performs worse than using the most recent data continuously.
- Accuracy benefits: 13× slower execution makes ResNet101 less suited to edge deployment than compressed ResNet18, despite ResNet101’s higher accuracy on representative earlier-window data.Continuous retraining helps the lightweight ResNet18 maintain high accuracy under drift.
3 Scheduling retraining and inference jointly
Joint scheduling of retraining and inference must balance model accuracy, inference accuracy, GPU allocation, and configuration choices over each retraining window. Ekya uses accuracy-aware resource allocation and configuration selection to improve average inference accuracy while respecting an accuracy floor.
- Illustrative scheduling example: The scheduling example assumes known resource-accuracy profiles and uses 3 GPUs across two video streams and retraining windows of 120 seconds.In practice, obtaining these profiles is expensive, motivating Ekya’s profiling techniques.
- Uniform scheduling: The uniform scheduler’s inference accuracy drops from 65% to 49% for video A and from 50% to 37.5% for video B during Window 1.Retraining temporarily takes resources from inference, while the benefits of improved models arrive later in the window.
- Accuracy-optimized scheduling: 73% average inference accuracy is achieved by Ekya’s accuracy-optimized scheduler, compared with 56% for the uniform scheduler.The uniform scheduler evenly splits 3 GPUs, while the accuracy-optimized scheduler optimizes inference accuracy over two 120-second retraining windows.
- Configuration trade-offs: Ekya selects retraining configurations by weighing accuracy improvements against GPU cost, choosing cheaper options when they provide better resource efficiency.Retraining fewer layers or samples reduces GPU use but can lower model accuracy; resource usage can vary by up to 200× across configurations.
- Accuracy-optimized scheduling: Ekya prioritizes retraining tasks with higher expected accuracy improvement, leaving more time to benefit from the resulting models.This scheduling decision considers the multi-dimensional trade-offs among configurations, resource allocations, and inference accuracy.
4 Ekya: Solution Description
Ekya makes joint continuous-learning decisions tractable by pruning configuration and allocation choices, prioritizing promising retraining tasks, and estimating performance through lightweight micro-profiling. It optimizes average inference accuracy over each retraining window under shared edge-GPU constraints.
- Scope: Evaluating optimization metrics beyond the supported objective, including max-min accuracy, is left to future work.The scheduler techniques are stated to apply to other metrics, but those metrics are not evaluated here.
- Problem formulation: Ekya jointly chooses retraining configurations, inference configurations, and GPU allocations to maximize average inference accuracy across videos within a retraining window.The formulation accounts for GPU-time, instantaneous GPU capacity, and one configuration choice of each type per video.
- Problem formulation: The optimization is computationally more complex than multi-dimensional knapsack and multi-armed bandits because costs vary and rewards require scarce golden-model labels.Known accuracy values reduce the problem to a multi-dimensional binary knapsack, while uncertain rewards create a bandit-like challenge.
- Thief scheduler: Ekya’s thief scheduler decouples resource allocation from configuration selection and repeatedly keeps resource reallocations only when estimated window-average accuracy improves.It starts from a fair allocation, steals small resource quanta among jobs, and repeats until accuracy stops increasing.
- Thief scheduler: The scheduler prioritizes video streams most affected by data drift, avoids retraining models that do not improve the target metric, and prunes allocations to coarse GPU fractions.These choices reduce the search space while preserving the objective of improving aggregate inference accuracy.
- Micro-profiling: Ekya’s micro-profiler samples 5%−10% of retraining data, tests configurations for about five epochs, extrapolates longer-run accuracy, and prunes historically poor configurations.It estimates retraining accuracy and time using a small set of promising configurations rather than exhaustive profiling.
5 Ekya Implementation
Ekya is implemented with PyTorch and modular Ray actors for the scheduler, micro-profiler, and video jobs. It uses Nvidia MPS for practical GPU resource isolation while retaining models in GPU memory to reduce restart costs.
- System implementation: Ekya uses PyTorch for model execution and training, with Ray actors representing scheduler, micro-profiler, and training or inference modules.The modular design supports scaling across many video streams and resources.
- GPU placement: Ekya uses Nvidia MPS for GPU resource isolation, while actor-based processes keep DNN models in GPU memory and largely avoid restarting costs.MPS changes resource allocation by intercepting CUDA calls and rescheduling them, but normally requires process restarts.
6 Evaluation
Ekya is evaluated across accuracy, resource, capacity, component, microprofiling, and alternative-design comparisons. It outperforms static and uniform baselines while using adaptive allocation and configuration selection to balance inference and retraining.
- Overall results: 29% higher accuracy than static retraining baselines, with the baseline requiring 4× additional GPU resources to match Ekya.These are the evaluation’s headline comparisons.
- Accuracy and capacity: Ekya’s accuracy degrades more gracefully than uniform baselines as more video streams share fixed resources.The comparison covers increasing concurrent streams under fixed GPU provisioning.
- Accuracy and capacity: With 4 GPUs, Ekya achieves higher accuracy than the baselines at 16 GPUs, corresponding to 4× resource savings.Figure 7 stress-tests ten concurrent video streams under varying GPU provisioning.
- Component analysis: Both adaptive resource allocation and microprofiling-based configuration selection substantially contribute to Ekya’s accuracy gains, especially under resource stress.The factor analysis compares variants that remove either adaptation mechanism.
- Scheduler analysis: The thief scheduler trades a small runtime increase for improved accuracy when its Δ parameter is smaller.The added runtime remains a tiny fraction of the 200-second retraining window.
- Microprofiling: Microprofiling estimates are largely unbiased with a 5.8% median absolute error, and up to 20% prediction errors reduce accuracy by at most 3%.Estimates use 10% of Cityscapes retraining data and five training epochs; reference accuracies use all data.
- Alternative designs: Ekya achieves higher accuracy than cloud retraining without network costs and preserves video privacy, while cached pretrained models reach 0.72 versus Ekya’s 0.78 accuracy.Cloud alternatives require additional uplink and downlink capacity to match Ekya’s accuracy.
7 Related Work
Related work includes cloud-scale machine-learning training, collaborative decentralized training, hyperparameter-focused techniques, and continuous learning. Ekya differs by optimizing inference accuracy over retraining windows through resource allocation across inference and retraining tasks.
- ML training systems: Cloud ML training systems optimize objectives such as parallelism, efficiency, fairness, or job completion rather than Ekya’s edge-video objective.Collaborative training systems instead focus on decentralized data on mobile phones.
- Continuous learning: Ekya focuses on resource allocation and inference accuracy over the retraining window, where the highest-accuracy retrained model can be sub-optimal.This distinguishes Ekya from techniques focused on finding hyperparameters for maximum model accuracy.
8 Conclusion
Ekya addresses the tradeoff between retraining and inference in continuous learning for edge video analytics. Its scheduler prunes decisions and prioritizes promising tasks, while its estimator provides low-overhead accuracy estimates.
- Conclusion: Ekya maximizes inference accuracy across multiple retraining and inference tasks by balancing their resource tradeoff.The conclusion frames this as the central challenge of continuous learning on edge servers.
- Conclusion: 29% higher accuracy than a baseline scheduler, with the baseline needing 4× more GPU resources to reach Ekya’s accuracy.The evaluation used a diverse set of video streams.