Source-linked AI summary
NestDNN: Resource-Aware Multi-Tenant On-Device Deep Learning for Continuous Mobile Vision
Biyi Fang, Xiao Zeng, Mi Zhang
TL;DR
Mobile vision systems run multiple applications under changing and limited resources, while conventional compressed models provide fixed resource-accuracy trade-offs. NestDNN builds flexible multi-capacity models and dynamically schedules their trade-offs and allocations at runtime. Compared with the resource-agnostic status quo, it achieves up to 4.2% higher inference accuracy, 2.0× higher video frame processing rate, and 1.7× lower energy consumption.
Problem
Mobile vision systems concurrently run multiple applications under changing, limited resources, while compressed models use fixed resource-accuracy trade-offs after deployment.
Method
NestDNN combines nested parameter-sharing multi-capacity models with a resource-aware runtime scheduler that selects model capacities and resource allocations.
Results
Up to 4.2% increase in inference accuracy, 2.0× increase in video frame processing rate and 1.7× reduction on energy consumption were achieved over the resource-agnostic status quo approach.
Takeaways & Limitations
NestDNN enables resource-aware multi-tenant on-device deep learning by adapting resource-accuracy trade-offs to runtime resources and jointly maximizing concurrent application performance.
Takeaways & Limitations
The concurrent applications’ total memory footprint cannot exceed the mobile vision system’s maximum memory space, Smax.
Abstract
from arXiv · showhide
Mobile vision systems such as smartphones, drones, and augmented-reality headsets are revolutionizing our lives. These systems usually run multiple applications concurrently and their available resources at runtime are dynamic due to events such as starting new applications, closing existing applications, and application priority changes. In this paper, we present NestDNN, a framework that takes the dynamics of runtime resources into account to enable resource-aware multi-tenant on-device deep learning for mobile vision systems. NestDNN enables each deep learning model to offer flexible resource-accuracy trade-offs. At runtime, it dynamically selects the optimal resource-accuracy trade-off for each deep learning model to fit the model's resource demand to the system's available runtime resources. In doing so, NestDNN efficiently utilizes the limited resources in mobile vision systems to jointly maximize the performance of all the concurrently running applications. Our experiments show that compared to the resource-agnostic status quo approach, NestDNN achieves as much as 4.2% increase in inference accuracy, 2.0x increase in video frame processing rate and 1.7x reduction on energy consumption.
1 INTRODUCTION
Mobile vision systems must concurrently support multiple vision applications under changing runtime resources, while existing compressed models use fixed resource-accuracy trade-offs. NestDNN addresses this with flexible model capacities and runtime scheduling to improve joint application performance.
- Motivation: Concurrent mobile vision applications face changing runtime contexts and must balance different accuracy and latency needs.For example, vehicle counting prioritizes low latency, whereas license-plate reading prioritizes accuracy.
- Motivation: Fixed resource-accuracy trade-offs are predetermined during development and cannot adapt to the resources available after deployment.Model compression reduces resource demand at a modest accuracy loss, but its selected trade-off is fixed.
- NestDNN approach: NestDNN creates a compact multi-capacity model whose nested descendant models provide different resource-accuracy trade-offs while sharing parameters.Parameter sharing reduces model memory footprint and model switching overhead.
- NestDNN approach: A resource-aware scheduler selects each model’s trade-off and resource allocation to jointly maximize inference accuracy and minimize processing latency across applications.The scheduler uses cost functions encoding descendant-model accuracy and latency.
- Experimental results: 4.2% increase in inference accuracy, 2.0× increase in video frame processing rate and 1.7× reduction on energy consumption were achieved over the resource-agnostic counterpart.These are reported as maximum improvements in the experimental results.
2 NESTDNN OVERVIEW
NestDNN uses offline model preparation and profiling, followed by online runtime scheduling that adapts descendant-model selection to changing resources. Its architecture combines pruning, recovery, and continuous resource-aware allocation.
- Architecture: NestDNN’s architecture is divided into offline and online stages.The offline stage prepares models, while the online stage responds to runtime resource changes.
- Offline stage: The offline stage performs model pruning, model recovery, and model profiling.These phases respectively create nested capacities and characterize their runtime properties.
- Model pruning: TRR ranks filters by importance and prunes them iteratively until the pruned model cannot meet the user’s minimum accuracy goal.The resulting pruning roadmap records each pruned model and its filter-pruning record.
- Model recovery: Model recovery freezes existing filter parameters and grows filters back along the pruning roadmap in reverse order to generate the multi-capacity model.Recovery starts from the seed model.
- Model profiling: Profiling records each descendant model’s inference accuracy, memory footprint, and processing latency for the target mobile system.These profiles support online scheduling decisions.
- Online stage: The online scheduler monitors resource-changing events, then selects descendant models and allocates resources to jointly optimize concurrent applications.It consults the profiles of all concurrently running applications after detecting an event.
3.1 Filter based Model Pruning
NestDNN’s filter-pruning pipeline ranks and iteratively removes less important CNN filters to reduce model size and computation while preserving accuracy. Its TRR ranking identifies filters with less impact on accuracy than L1-norm ranking, and the resulting roadmap guides recovery.
- Filter pruning: CNN convolutional layers use 3D filters as feature extractors, making them the main computational target for pruning.Each filter generates one output feature map.
- Filter pruning: Pruning one filter removes associated parameters, output feature maps, and FLOPs in the current and subsequent convolutional layers.Thus, filter pruning reduces both model size and computational cost.
- Filter importance ranking: TRR ranks filters by the relative similarity of feature maps from same-class and different-class image triplets.Its score uses residuals between anchor-negative and anchor-positive feature-map distances.
- Filter importance ranking: 59.96% of conv13 filters can be pruned without changing accuracy, indicating that TRR identifies redundant filters.TRR also accounts for differing pruning sensitivity across convolutional layers during iterative pruning.
- Filter importance ranking: TRR achieves higher accuracy than L1-norm at almost every pruning percentage across all 13 VGG-16 convolutional-layer curves.At conv13, TRR reaches 89.72% and 87.40% accuracy at 50% and 90% pruning, versus 75.45% and 42.65% for L1-norm.
- Filter pruning roadmap: Iterative pruning retrains each pruned model until it cannot meet the user’s minimum accuracy goal, producing a roadmap whose smallest model is the seed model.The roadmap records each model’s filter-pruning history and guides model recovery.
3.2 Freeze-&-Grow based Model Recovery
NestDNN recovers a single multi-capacity model from independent pruned variants using freeze-&-grow, nesting multiple resource-accuracy trade-offs through parameter sharing. This compact structure also reduces memory use and switching overhead.
- Motivation and key idea: Independent pruned variants offer different resource-accuracy trade-offs but are impractical to keep simultaneously on resource-limited mobile systems.Retraining makes the variants use different parameters.
- Motivation and key idea: Freeze-&-grow generates one multi-capacity model that reproduces the variants’ trade-offs while using much less space than their accumulated model sizes.The method freezes existing filters and grows capacity by adding pruned filters back.
- Model recovery: Recovery starts from the seed model, iteratively freezes current filter parameters, reverses the pruning roadmap, and retrains each larger descendant model.Repeating the process nests successive capacities in the final model.
- Model recovery: Parameter sharing lets smaller descendant models nest inside larger ones without taking extra memory space.The final descendant contains the capacities of all preceding descendants.
- Superiority of the multi-capacity model: The multi-capacity model reduces memory footprint, provides optimized resource-accuracy trade-offs, and enables efficient model switching.These benefits eliminate installing many independent variants and reduce page-in/page-out overhead.
- Superiority of the multi-capacity model: During an upgrade, switching incurs zero page-out overhead and pages in only the extra filters required by the larger descendant model.The figure contrasts model upgrade and downgrade behavior.
3.3 Resource-Aware Scheduler
NestDNN’s resource-aware scheduler models application-specific accuracy and latency preferences, then allocates limited runtime resources across concurrent vision applications. It supports both cost-focused and fairness-oriented scheduling through approximate greedy optimization.
- Cost Function: The scheduler encodes each descendant model’s accuracy and latency into a cost function with a user-controlled accuracy-latency preference.The cost rewards higher accuracy and penalizes latency exceeding the application’s maximum target; α weights the latency penalty.
- Scheduling Schemes: MinTotalCost minimizes aggregate application cost, favoring lower-cost applications for additional runtime resources.The total memory footprint must remain within the system’s maximum available memory.
- Scheduling Schemes: MinMaxCost minimizes the highest application cost, allocating resources more fairly to balance concurrent application performance.The scheme constrains every application’s cost below a minimized bound k.
- Greedy Approximation: Because both scheduling formulations are computationally hard, NestDNN uses a greedy heuristic that iteratively allocates minimum resource units until resources are exhausted.MinTotalCost targets the smallest cost increase, whereas MinMaxCost serves the application with the highest current cost and selects its best descendant model.
- Cached Greedy Heuristic: Caching shortens greedy-scheduler runtime by resuming from a previously allocated resource state instead of recomputing from scratch.The paper illustrates caching an unfinished scheme after 70% of resources have been allocated.
4 EVALUATION
The evaluation tests NestDNN across six mobile vision applications spanning generic and class-specific recognition tasks, multiple datasets, and two representative DNN architectures.
- Datasets: The class-specific datasets contain over 50K road-sign images, over 14K face images, and over 158K scene images.These correspond to GTSRB, Adience-Gender, and Places-32, respectively.
- DNN Models: VGG-16 and ResNet-50 provide the two DNN architectures used to evaluate NestDNN’s generalization across model types.VGG-16 is described as straightforward to implement, while ResNet-50 is characterized by superior recognition accuracy.
- Application Mapping: The six applications pair datasets with architectures as VC, RI-50, RI-100, VS, VG, and RP, following each dataset’s training and testing protocol.VGG-16 is assigned to CIFAR-10, GTSRB, and Adience-Gender; ResNet-50 is assigned to ImageNet-50, ImageNet-100, and Places-32.
4.2 Performance of Multi-Capacity Model
NestDNN’s multi-capacity model improves accuracy at matched model sizes while reducing memory and model-switching overhead through shared parameters among descendant models.
- Experimental Setup: Five descendant models are generated per application to provide diverse resource-accuracy trade-offs under varying resource availability.The number and trade-offs of descendant models are configurable according to user preferences.
- Optimized Resource-Accuracy Trade-offs: Descendant models achieve 4.98% higher average accuracy than baseline models at every compared model size.The two smallest descendants average 6.68% higher accuracy, versus 3.72% for the two largest descendants.
- Memory Footprint: Parameter sharing reduces the multi-capacity model’s memory footprint relative to storing five independent descendant models.VC saves 241.5 MB, while running all six applications concurrently saves 587.4 MB.
- Model Switching: Multi-capacity models reduce model-switching memory overhead because switching pages in only a small portion of shared models.During downgrade, the multi-capacity model has zero page-in overhead.
- Model Switching: Switching multi-capacity models consumes less energy than switching independent models across all applications, with greater benefit at higher switching frequencies.The comparison measures 250, 500, 750, and 1,000 random switches on a Samsung Galaxy S8.
4.3 Performance of Resource-Aware Scheduler
The resource-aware scheduler is evaluated against a fixed resource-agnostic baseline under MinTotalCost and MinMaxCost. By varying α, NestDNN offers accuracy-frame-rate trade-offs unavailable to the baseline while also reducing energy consumption.
- Experimental Setup: NestDNN is compared with a resource-agnostic baseline using fixed resource-accuracy trade-offs on three Android smartphones.Results are reported from the Samsung Galaxy S8 after consistent results across the three devices.
- Inference Accuracy and Frame Rate: Adjusting α provides multiple inference-accuracy and frame-rate trade-offs, including regions where NestDNN exceeds the baseline on both metrics.The upper-right quadrant of Figure 10 denotes simultaneous gains in top-1 accuracy and frame rate.
- MinMaxCost: 4.2% average accuracy gain is achieved at the baseline’s average frame rate under MinMaxCost.At equal average top-1 accuracy gain, NestDNN achieves 1.9× frame-rate speedup; its knee achieves 1.5× speedup and 2.1% gain.
- Energy Consumption: NestDNN reduces average energy consumption by 1.7× under MinTotalCost and 1.5× under MinMaxCost.The comparison uses NestDNN at the knee against the baseline across different numbers of inferences.
5 DISCUSSION
NestDNN is presented as a general framework for resource-aware multi-tenant on-device learning, while acknowledging a computational-cost limitation in its current pruning approach.
- Impact on Mobile Vision Systems: NestDNN dynamically selects resource-accuracy trade-offs at runtime to maximize performance under resource constraints.
- Generality of NestDNN: The framework is designed to generalize beyond VGG Net and ResNet to other deep learning models and computing tasks.
- Limitation: TRR-based filter pruning outperforms L1-norm pruning but has substantially higher computational cost.
6 RELATED WORK
Prior work compresses deep neural networks for mobile deployment, but commonly produces fixed resource-accuracy trade-offs. NestDNN instead provides dynamic trade-offs through a multi-capacity model.
- Deep Neural Network Model Compression: Pruning is a prevalent compression method for reducing deep neural network resource demands on mobile systems.
- Deep Neural Network Model Compression: Parameter pruning can reduce model sizes, but existing compression methods generally produce fixed resource-accuracy trade-offs.
- Deep Neural Network Model Compression: NestDNN's multi-capacity model provides dynamic resource-accuracy trade-offs, resembling dynamic neural network approaches.
- Continuous Mobile Vision: Continuous mobile vision has motivated multiple efforts toward realizing persistent mobile vision systems.
7 CONCLUSION
The paper presents and evaluates NestDNN as a resource-aware framework for concurrent on-device deep learning in mobile vision systems. Across six applications, it outperforms a resource-agnostic status quo in accuracy, frame processing rate, and energy consumption.
- NestDNN dynamically selects resource-accuracy trade-offs and allocates resources across concurrently running deep learning models.
- Six mobile vision applications targeting important vision tasks were used to evaluate NestDNN.
- NestDNN outperforms the resource-agnostic status quo in inference accuracy, video frame processing rate, and energy consumption.
- The authors position NestDNN as a step toward realizing continuous mobile vision.