Source-linked AI summary
Not Just Privacy: Improving Performance of Private Deep Learning in Mobile Cloud
Ji Wang, Jianguo Zhang, Weidong Bao, Xiaomin Zhu, Bokai Cao, Philip S. Yu
TL;DR
Mobile devices need efficient deep learning, but their limited capacity and the privacy risks of sending data to the cloud create a deployment challenge. ARDEN partitions DNNs between mobile devices and the cloud, applies private perturbation locally, and uses noisy training to improve robustness; experiments report preserved privacy, improved inference performance, and lower mobile resource use.
Problem
Mobile devices have limited processing, storage, and memory capacity, while uploading data for cloud-based DNN services creates privacy risks.
Method
ARDEN partitions DNNs between mobile devices and the cloud, using local feature extraction and differential-private perturbation alongside cloud-side noisy training with raw and generated noisy samples.
Results
ARDEN improves robustness to layer-wise perturbation, mitigates accuracy loss from private perturbation, and reduces mobile time, memory, and energy by 60.10%, 92.07%, and 77.05%, respectively.
Takeaways & Limitations
The framework provides a cloud-based approach that combines privacy protection with improved inference performance and reduced mobile resource use.
Takeaways & Limitations
The evaluation does not compare ARDEN with advanced cloud-side DNNs such as DenseNet, because it focuses on noisy-training performance improvement.
Abstract
from arXiv · showhide
The increasing demand for on-device deep learning services calls for a highly efficient manner to deploy deep neural networks (DNNs) on mobile devices with limited capacity. The cloud-based solution is a promising approach to enabling deep learning applications on mobile devices where the large portions of a DNN are offloaded to the cloud. However, revealing data to the cloud leads to potential privacy risk. To benefit from the cloud data center without the privacy risk, we design, evaluate, and implement a cloud-based framework ARDEN which partitions the DNN across mobile devices and cloud data centers. A simple data transformation is performed on the mobile device, while the resource-hungry training and the complex inference rely on the cloud data center. To protect the sensitive information, a lightweight privacy-preserving mechanism consisting of arbitrary data nullification and random noise addition is introduced, which provides strong privacy guarantee. A rigorous privacy budget analysis is given. Nonetheless, the private perturbation to the original data inevitably has a negative impact on the performance of further inference on the cloud side. To mitigate this influence, we propose a noisy training method to enhance the cloud-side network robustness to perturbed data. Through the sophisticated design, ARDEN can not only preserve privacy but also improve the inference performance. To validate the proposed ARDEN, a series of experiments based on three image datasets and a real mobile application are conducted. The experimental results demonstrate the effectiveness of ARDEN. Finally, we implement ARDEN on a demo system to verify its practicality.
1 INTRODUCTION
Mobile devices need on-device deep learning, but large DNNs exceed their resource capacity and cloud offloading introduces privacy risks. ARDEN partitions the DNN across mobile and cloud sides, protects uploaded data, and uses noisy training to preserve inference performance.
- DNN advances have increased model complexity, with millions to hundreds of millions of parameters requiring substantial processing and storage resources.
- Mobile devices have limited on-chip memory, leaving resource-hungry DNN applications beyond their intrinsic capacity.
- Compressing DNNs can fit applications onto mobile devices but may cause performance degradation and uncontrolled energy consumption.
- Cloud partitioning places shallow DNN portions on mobile devices and offloads complex, large portions to cloud data centers after local input transformation.
- Cloud inference reduces mobile resource demands but exposes transmitted data to privacy risks and requires lightweight, rigorous protection without sacrificing classification accuracy.
- ARDEN combines mobile-cloud DNN partitioning, local differential-privacy noise, and noisy cloud training to improve inference robustness to perturbed data.
2 PRELIMINARY
This section introduces DNN computation and training, transfer learning, and differential privacy. It explains how layered representations support feature reuse and how randomized mechanisms provide formal privacy guarantees.
- 2.1 Deep Learning: Deep neural networks progressively extract complex representations from high-dimensional inputs through stacked layers of affine transformations and nonlinear activations.
- 2.1 Deep Learning: DNN training learns parameters that minimize a loss function, typically using mini-batch stochastic gradient descent and backpropagation.
- 2.1 Deep Learning: Transfer learning treats shallow layers of a pretrained DNN as general feature extractors while retraining the remaining layers for a target task or dataset.
- 2.2 Differential Privacy: Differential privacy aims to provide provable privacy guarantees for sensitive data through randomized mechanisms applied to adjacent inputs and outputs.
- 2.2 Differential Privacy: Adjacent inputs differ by one data item, and the privacy budget ε controls protection strength, with smaller ε indicating stronger privacy.
- 2.2 Differential Privacy: Noise calibrated to a function’s global sensitivity provides a general method for making deterministic functions ε-differentially private, while post-processing cannot increase privacy loss.
3 THE PROPOSED FRAMEWORK
Arden partitions DNN inference between mobile devices and the cloud, using lightweight private transformation locally and cloud-side training and inference. Its nullification, noise injection, privacy analysis, and noisy training are designed to preserve privacy while improving robustness to perturbed representations.
- 3.1 Overview: Arden divides the DNN into frozen local and fine-tuned cloud components, with mobile devices performing lightweight transformation while cloud centers handle training and complex inference.The local network derives from a pretrained DNN, whereas the cloud network is fine-tuned during training.
- 3.2 Differentially Private Transformation: Sensitive inputs are transformed into abstract features, then protected through nullification and random noise before transmission to the cloud.The transformation is performed on the local side, and the perturbed representation is sent for further inference.
- 3.2 Differentially Private Transformation: Arden combines user-specified or random item masking with layer-wise perturbation, addressing the utility problems of conservative sensitivity estimates and output-only noise injection.Nullification masks ⌈N · µ⌉ input items, while noise is injected during the transformation rather than only at the final output.
- 3.2 Differentially Private Transformation: Theorem 3.1 establishes that Arden’s Algorithm 1 is ε-differentially private for sensitive data processed by the local network.The privacy budget analysis generalizes the Laplacian mechanism; without nullification, the mechanism reduces to ε = 2σ.
- 3.3 Noisy Training: Noisy training mixes clean and generated noisy representations, then adds worst-case perturbations to improve cloud-side robustness against unknown inference noise.The mixed loss uses λ to control the clean-versus-noisy tradeoff, while min-max training seeks robustness to deviations from generated noisy representations.
4 EXPERIMENTAL EVALUATION
ARDEN is evaluated on image-classification benchmarks and the DeepMood mobile application through parameter studies, privacy analyses, framework comparisons, and overhead measurements. Results indicate robustness to perturbation, improved private inference, applicability across privacy budgets, and reduced mobile resource use.
- Experimental Setup: Experiments cover MNIST, SVHN, CIFAR-10, and the DeepMood application, including ALPH. and ACCEL. tests.The study also evaluates parameter effects, privacy budgets, and implementation overhead.
- Parameter Selection: η is set to 5 because smaller values insufficiently resist perturbation randomness, while larger values overemphasize the noisy-training loss.Small λ aggravates the same imbalance by increasing the weight of L3.
- Perturbation Robustness: Noisy training keeps accuracy relatively high under different layer-wise perturbations and limits performance changes to within 10% in most perturbation-strength settings.Accuracy generally peaks near the pre-assigned perturbation strength, but declines when perturbation becomes too large.
- Perturbation Robustness: At perturbation strength (5,10%), perturbed images can hardly be reconstructed, indicating that attackers cannot restore the original data from revealed perturbations even when the strength is public.Weaker perturbations allow some reconstruction, although reconstructed images remain quite different from the originals.
- Scope: The evaluation does not compare ARDEN with advanced DNNs such as DenseNet because the experiments target noisy-training performance improvement.The cloud-side DNN can be replaced by other advanced DNNs, and ARDEN can switch between models transparently.
- Privacy-Budget Analysis: ARDEN maintains high accuracy across a wide privacy-budget range, with accuracy decreasing by 14% and 12% for ALPH. and ACCEL. when ϵ < 1.The accuracy remains nearly unchanged until ϵ decreases to 1.9 for ALPH. and 2.4 for ACCEL.
- Android Implementation: ARDEN reduces time, memory, and energy by 60.10%, 92.07%, and 77.05%, respectively, compared with four fully on-device DNNs.Local transformation still raises CPU temperature by 9 degrees during consecutive SVHN processing.
5 RELATED WORK
Prior work addresses mobile deep learning deployment through DNN compression or mobile-cloud partitioning, while privacy-preserving inference uses encryption, hybrid architectures, flexible transformations, and differential privacy.
- DNN deployment on mobile devices is constrained by the mismatch between large network sizes and limited device capacity.Prior approaches include compression and partitioning DNNs across mobile devices and clouds.
- Privacy-preserving deep learning has been explored through encrypted data, hybrid mobile-cloud architectures, flexible local transformations, and differentially private training or inference.The cited approaches include BGV encryption, k-anonymity protection, variable feature transformations, and private distributed SGD.
- Differential privacy provides a provable privacy guarantee for deep learning applications.
6 CONCLUSIONS
Arden partitions DNN computation between mobile devices and clouds, protects uploaded data with differential privacy, and uses noisy training to preserve inference performance. Experiments and a demo system support its effectiveness, practicality, and reduced resource consumption.
- Arden partitions DNNs across mobile devices and clouds, leaving simple transformation and perturbation on the device while offloading heavy computation.
- Arden combines a flexible differentially private perturbation with rigorous privacy-budget analysis and noisy training for perturbed-data robustness.
- Experiments on three image datasets and a real mobile application show that Arden preserves privacy and improves inference performance.
- Arden reduces resource consumption by over 60% and is implemented on a demo mobile-cloud system to test practicality and overhead.
A PROOF OF THEOREM 3.1
The proof analyzes Laplace noise under bounded transformations and then incorporates item-wise nullification to derive the privacy guarantee of the combined local transformation.
- Theorem A.1 establishes differential privacy for a bounded function perturbed with scaled Laplace noise.
- Theorem A.2 shows that item-wise nullification of an ε-differentially private mechanism yields ε′ = ln[(1 −µ)e^ε + µ]-differential privacy.
- The proof handles adjacent inputs by separating whether their differing item is nullified, then combines the two cases using the nullification probability.
- The proof first analyzes the local transformation without nullification and represents it using the output-bounded neural network M_l.
- The local transformation without nullification is shown to be (2σ/Λ)-differentially private, where Λ is the infinity norm of the transformation gradient.
- The resulting overall transformation applies the privacy analysis to the nullified input A(x_s ⊙ I_n).