PaperlayerView exact revision on arXiv

Source-linked AI summary

Mind Your Weight(s): A Large-scale Study on Insufficient Machine Learning Model Protection in Mobile Apps

Zhichuang Sun, Ruimin Sun, Long Lu, Alan Mislove

arXiv:2002.07687v2cs.CRcs.LG

TL;DR

On-device ML puts valuable models on untrusted user devices, but the prevalence and robustness of model protections and the impacts of leakage have been insufficiently studied. This paper analyzes Android apps from US and Chinese markets using static and dynamic techniques. It finds widespread absent or weak protection and estimates substantial financial and security impacts from model leakage.

  • Problem

    On-device ML stores vendors’ valuable models on untrusted user devices, while the extent and robustness of model protection and the costs of leakage have been insufficiently studied.

    Source: §1 · p. 1

  • Method

    The authors analyzed 46,753 Android apps from US and Chinese markets, using static analysis to identify models and protection and dynamic analysis to extract decrypted models from memory.

    Source: §1 · p. 2

  • Results

    41% of identified ML apps had no model protection, and 66% of apps using protection still leaked models to an unsophisticated runtime attack.

    Source: §1 · p. 2

  • Takeaways & Limitations

    Model leakage can financially benefit attackers by millions of dollars and enable evasion of model-based authentication with access to users’ private information.

    Source: §10 · p. 16

    Obfuscation remains vulnerable to attackers skilled in binary decompilation, while encryption is vulnerable to dynamic analysis and runtime memory instrumentation.

    Source: §7.2 · p. 14

Abstract

from arXiv · show

On-device machine learning (ML) is quickly gaining popularity among mobile apps. It allows offline model inference while preserving user privacy. However, ML models, considered as core intellectual properties of model owners, are now stored on billions of untrusted devices and subject to potential thefts. Leaked models can cause both severe financial loss and security consequences. This paper presents the first empirical study of ML model protection on mobile devices. Our study aims to answer three open questions with quantitative evidence: How widely is model protection used in apps? How robust are existing model protection techniques? What impacts can (stolen) models incur? To that end, we built a simple app analysis pipeline and analyzed 46,753 popular apps collected from the US and Chinese app markets. We identified 1,468 ML apps spanning all popular app categories. We found that, alarmingly, 41% of ML apps do not protect their models at all, which can be trivially stolen from app packages. Even for those apps that use model protection or encryption, we were able to extract the models from 66% of them via unsophisticated dynamic analysis techniques. The extracted models are mostly commercial products and used for face recognition, liveness detection, ID/bank card recognition, and malware detection. We quantitatively estimated the potential financial and security impact of a leaked model, which can amount to millions of dollars for different stakeholders. Our study reveals that on-device models are currently at high risk of being leaked; attackers are highly motivated to steal such models. Drawn from our large-scale study, we report our insights into this emerging security problem and discuss the technical challenges, hoping to inspire future research on robust and practical model protection for mobile devices.

1 Introduction

On-device inference offers offline use and privacy benefits, but stores vendors’ valuable models on untrusted devices, exposing them to theft and abuse. This study measures protection and leakage across mobile apps and assesses potential financial and security consequences.

1 Introduction

  • On-device models contain vendors’ core intellectual property yet reside on fundamentally untrusted user devices, creating exposure to theft or abuse.The paper also links leakage to lost R&D investment, pricing advantage, and adversarial inputs that can bypass or confuse ML systems.Source: §1 · p. 1
  • The study asks how widely model protection is used, how robust it is, and what impacts stolen models can incur.It presents a large-scale empirical study of model protection and theft on mobile devices.Source: §1 · p. 1
  • Dynamic analysis extracted 18 unique protected models affecting 347 apps and spanning frameworks and uses from face tracking to malware detection.The extracted models covered liveness detection, OCR, ID and bank card recognition, and photo processing.Source: §1 · p. 2
  • Model leakage can cost millions of dollars, while model-based access-control bypass may damage vendors’ reputations or cause product failure.Identified impact sources include model R&D investment, competitive financial loss, and security consequences.Source: §1 · p. 2
  • 41% of 1,468 on-device ML apps leave models unprotected, while 66% of apps using protection still leak models to unsophisticated runtime attacks.The study analyzed 46,753 Android apps from US and Chinese markets.Source: §1 · p. 2

2 Background

On-device ML supports offline and real-time inference while reducing the need to send private data to servers. Models are delivered and protected in varied ways, but common obfuscation or encryption practices do not strictly prevent theft or reuse.

2 Background

  • On-device inference works without network access, supports real-time tasks, and seldom requires private user data to leave the device.Cloud-based inference requires network connections and sending raw user data to servers.Source: §2 · p. 2
  • Models are typically trained on servers and shipped in app packages, though they may also be downloaded after installation.Inference then runs on user devices using model files and ML frameworks or SDKs.Source: §2 · pp. 2–3
  • Developers may encrypt or obfuscate models or compile them into stripped binaries, but these techniques make reverse engineering harder rather than preventing theft or reuse.Source: §2 · p. 3
  • Mobile apps use frameworks with differing model file formats, while third-party providers sell offline ML SDKs that package models as black-box functionality.Providers such as Face++ and SenseTime offer capabilities including facial and voice recognition, liveness detection, image processing, and OCR.Source: §2 · p. 3

3 Analysis Overview

The paper studies model protection prevalence, robustness, and leakage impacts using a deliberately simple static-dynamic analysis pipeline. Its findings are framed as a conservative estimate of the model-leak problem and include financial and security consequences.

3 Analysis Overview

  • The study asks how widely apps protect on-device models, how robust those protections are, and what impacts stolen models can incur.The paper notes that these threats and protection options had not been sufficiently studied.Source: §3 · p. 3
  • The deliberately simple pipeline represents what a knowledgeable but not highly skilled attacker can achieve, so its results are a lower bound on leak severity.Source: §3 · p. 3
  • Static ModelXRay detects on-device ML and examines protection; for encrypted models, dynamic ModelXtractor extracts plaintext model representations from memory.The authors report that both tools are highly accurate for their use despite their simple techniques.Source: §3 · p. 3
  • Leakage can save attackers license fees and R&D costs, while vendors may lose pricing advantages and market share; security effects include access-control bypass and privacy breaches.Source: §3 · p. 3

4 Q1: How Widely Is Model Protection Used in Apps?

The study examines model protection across Android app markets and evaluates ModelXRay for detecting on-device ML models and encryption. It finds substantial gaps in protection, with plaintext models common and protection varying across markets and frameworks.

4.1 Android App Collection

  • The study collected 46,753 trending or recently updated Android apps from Google Play, Tencent My App, and 360 Mobile Assistant.It identified 1,468 on-device ML apps, with more ML apps in Chinese markets by both percentage and absolute count.Source: §4.1 · p. 3, §4.4 · pp. 5–6

4.2 Methodology of ModelXRay

  • ModelXRay extracts APK assets and libraries, identifies ML frameworks and model files, then uses file filters and entropy analysis to detect encrypted models.It uses a 7.99 entropy threshold and excludes compressed files using file types and magic numbers.Source: §4.2 · p. 4

4.3 Accuracy Evaluation of ModelXRay

  • 6.8% false negatives and 0% false positives occurred on 438 apps; a separate 100-app sample had a 3% false-positive rate for ML-app detection.Model-file true-positive rates were 91% for Chinese-market samples and 97% for Google Play samples; encrypted model files had entropy close to 8, versus 3.5–7.5 typically for unencrypted files.Source: §4.3 · pp. 4–5, Figure 3 · p. 5, Figure 4 · p. 5

4.4 Findings and Insights

  • On-device ML appears across app categories and functions, while Chinese-market apps show higher adoption: Google Play had 178 ML apps (1.40%), Tencent My App 159 (7.25%), and 360 Mobile Assistant 1,131 (3.55%).The study reports functions including OCR, face tracking, speech recognition, ID-card recognition, liveness detection, and face recognition.Source: §4.4 · pp. 5–6
  • Only 59% of ML apps protect their models; 602 apps include plaintext models that can be extracted from app packages or installation directories.At the model level, 4,254 of 6,522 models (77%) are unprotected.Source: §4.4 · pp. 6–7
  • Protection varies by market and framework: 26% of Google Play ML apps protect models, compared with 66% in 360 Mobile Assistant and 59% in Tencent My App.More than 79% of apps using SenseTime (Parrots) and 60% using Caffe protect models, versus around 20% for TensorFlow and NCNN.Source: §4.4 · pp. 6–7
  • Only 22% of protected models are unique; the most widely shared model appears in 81 apps, and 60 cases involve different app companies reusing model licenses.The study identifies possible illegal use in a license shared by 12 app companies.Source: §4.4 · p. 7
  • 797 ML apps (54%) use GPU acceleration, while 1,341 use remote models, 1,468 use on-device models, and 182 use both.Remote models are reported mainly for NLP and vision APIs; the study found no security-critical remote-model use cases and notes on-device use for security-critical and real-time scenarios.Source: §4.4 · pp. 7–8, Table 4 · p. 7, Table 5 · p. 8

5 Q2: How Robust Are Existing Model Protection Techniques?

ModelXtractor tests whether encrypted on-device models can be recovered from app memory using practical runtime instrumentation. It extracted models from 18 of 29 apps whose ML functions were triggered, affecting 347 apps through model reuse.

5.1 App Instrumentation

  • ModelXtractor defaults to capturing buffers just before deallocation, when models are decrypted, and uses heap, loading, decryption, or customized-deallocator strategies when needed.The default strategy is optimized to activate after ML libraries load and to target buffers above a configurable minimum size.Source: §5.1 · p. 9

5.2 Model Representation and Recognition

  • Recognition combines framework-specific content and encoding signatures with buffer-size matching when a model’s format is unknown.For Protobuf models, content signatures identify model buffers, while encoding signatures locate a model’s beginning.Source: §5.2 · pp. 9–10

5.3 Evaluation of ModelXtractor

  • ModelXtractor failed to extract 11 models despite triggered ML functions, and is intended as a practical attack rather than an exhaustive extraction method.The authors attribute failures to instrumentation-strategy limitations.Source: §5.3 · p. 10

5.4 Findings and Insights

  • 12 of 15 decrypted models were captured at deallocation, while extracted models spanned commercial providers, security-critical functions, and reusable apps.Eight of 15 listed apps had more than 10 million downloads; more than 81% of apps in the study contained directly usable models, and SDK licenses were sometimes shipped unprotected.Source: §5.4 · p. 11
  • 18 of 29 triggered apps yielded models (66%), affecting 347 apps through model reuse, showing file encryption often fails against an unsophisticated attack.The extraction statistics are grouped by ML framework in Table 6.Source: §5.4 · pp. 10–11, Table 6 · p. 11

5.5 Interesting Cases of Model Protection

  • Encrypted code and models, encrypted feature vectors, and repeated encryption still yielded recoverable models through memory capture or decryption-function instrumentation.In one case, a customized WebView ran encrypted JavaScript; in another, recovered feature vectors exposed the model’s input format, while a third app encrypted individual models inside an encrypted tar file.Source: §5.5 · pp. 11–12

5.6 Responsible Disclosure

  • Vendors reported limited practical protections, while one vendor began internal work with security partners to improve model security after disclosure.The study contacted 12 vendors and received responses from five.Source: §5.6 · p. 12

6 Q3: What Impacts can (Stolen) Models Incur?

The paper examines how leaked models can benefit attackers and harm vendors, including through financial competition and security-critical identity checks. It estimates attacker savings and describes risks to app users and service providers.

6.1.1 Financial Benefit for Attackers

  • Attackers can avoid offline licenses priced at $50,000 to $200,000 annually and an estimated $772,040 in model R&D costs.The license estimate applies to offline authorization, while the R&D figure is a conservative estimate based on the paper’s example.Source: §6.1.1 · p. 13

6.1.2 Financial Loss for Model Vendors

  • A SenseTime customer loss could cost at least $10,000, while pricing disadvantages may reduce vendors’ customers, market share, and revenue.The paper notes that SenseTime models were reused by an average of 21 apps per model in its study.Source: §6.1.2 · p. 13

6.2 Security Impact

  • Leaked security-critical models can help attackers bypass identity checks, including across 81 apps sharing one liveness-detection model.The paper also describes potential loan-application identity fraud and exposure of patients’ medical records.Source: §6.2 · p. 14

7 Countermeasures

Existing defenses make on-device models harder to access or reverse engineer, but each has limitations. The paper identifies secure hardware as a future direction while noting mobile resource and GPU constraints.

7.1 Current Model Protection

  • Encryption can protect models from direct APK access, yet the study's unsophisticated dynamic analysis extracted models despite encryption of vectors, models, or inference code.The authors also report encrypting code and model files, encrypting model files multiple times, and encrypting feature vectors without preventing plaintext extraction from memory.Source: §7.1 · p. 14

7.2 Limitations

  • Customized formats and frameworks increase the effort to identify and reuse models, but documentation leakage can expose them.The paper notes customized formats including MessagePack, pickle, and Thrift, as well as encrypted JavaScript in a customized WebView.Source: §7.1 · p. 14, §7.2 · p. 14
  • Obfuscation raises recovery effort, but devoted attackers with binary-decompilation knowledge can recover models using program slicing and partial execution.Even without that knowledge, attackers may reuse a model as a black box.Source: §7.2 · p. 14, §7.1 · p. 14

7.3 Future Works

  • Secure hardware is presented as a promising protection direction, with prior approaches using secure enclaves for model inference and related computation.The cited examples include TF-Trusted, Slalom, Privado, TensorScone, and Graviton; the paper notes that this research has focused on cloud-end security.Source: §7.3 · p. 14
  • Future mobile protections should consider hardware-backed inference, but limited computation resources make running inference entirely in a TEE impractical.The paper also calls for secure GPU acceleration given the wide adoption of GPUs on mobile devices.Source: §7.3 · pp. 14–15

8 Discussion

ModelXtractor automates instrumentation-script generation, but dynamic extraction still requires human interaction and verification. The paper connects model theft to white-box adversarial research and plagiarism detection, while qualifying the scope of its analysis and impact estimates.

Manual analysis effort

  • Dynamic extraction requires manual app navigation, registration when needed, and verification that dumped buffers contain valid models.Analysts identify model starts using encoding signatures, truncate buffers, and parse them with a decoder such as protobuf.Source: §8 · p. 15
  • Extraction effort varies with how easily an app's ML functionality can be triggered, so the study prioritized easier apps and budgeted two hours per app across 82 apps.Some AI camera models took less than an hour to extract, while some P2P loan apps required registration and could take hours.Source: §8 · p. 15

Research Insights

  • Because protected private models can be extracted, the authors call for more white-box adversarial-ML research for security-critical applications.They frame this work as improving the resiliency of models used in those applications.Source: §8 · p. 15
  • Model plagiarism detection is needed because attackers may reuse competitors' stolen models instead of training their own.The authors note that retraining from a stolen model can make the resulting model look very different, motivating forensic tools for illegal reuse analysis.Source: §8 · p. 15

Limitations

  • The extraction tools use straightforward keyword matching, and their encoding and content features cover documented formats of well-known ML SDKs.The authors say program slicing, partial execution, Android app de-obfuscation, and additional knowledge of special formats could extend extraction scale.Source: §8 · p. 15
  • The financial-loss analysis is subjective because R&D costs and company revenues are asymmetrically known.The authors say the estimate emphasizes that costs can be high and that a more comprehensive study would require stakeholders' real case data.Source: §8 · p. 15

9 Related Work

The paper situates on-device model extraction alongside Android program-analysis techniques, model-extraction attacks, adversarial ML, and proprietary-model protection. Its focus differs from prior API-based work by targeting inference on mobile devices with physical access to the device.

Android analysis

  • Android analysis work combines static and dynamic techniques, including slicing, partial execution, memory instrumentation, and runtime monitoring to recover values or locate relevant code.Examples include DroidTrace, Rasthofer's work, DeGuard, TIRO, DroidKex, ARTIST, and AndroidSlicer.Source: §9 · p. 15

Model extraction

  • Prior model-extraction research largely targets ML-as-a-service through prediction APIs, stolen hyperparameters, gradients, classifier exploration, or side channels.The paper describes its own focus as orthogonal: on-device inference with attackers assumed to have physical access to the running mobile device.Source: §9 · pp. 15–16

Adversarial machine learning

  • The paper connects model extraction to white-box adversarial ML, contrasting it with prior attacks that mostly assumed black-box access.Once models are white-box, the cited discussion says attackers can craft adversarial examples to deceive learning systems.Source: §9 · p. 16

Proprietary model protection

  • Watermarking has been used to detect illegitimate model use, while fingerprinting can attest model integrity; the paper links its findings to realistic mobile model plagiarism.The cited fingerprinting approach encodes a fingerprint in DNN weights to check whether models have been modified.Source: §9 · p. 16

10 Conclusion

The conclusion reports widespread inadequate protection and easy recovery of many encrypted models across a large Android-app study. It argues that model leakage can create financial and security harms and calls for robust protection research.

10 Conclusion

  • Across 46,753 Android apps from US and Chinese markets, the study found on-device ML in every app category and 41% of ML apps without model protection.Many apps that did protect models used weak mechanisms, including the same encrypted model across multiple apps.Source: §10 · p. 16
  • Even encrypted models could be recovered through the study's unsophisticated analysis.The conclusion describes attackers as technically able and financially motivated to steal models.Source: §10 · p. 16
  • Model leakage can financially benefit attackers by millions of dollars and enable evasion of model-based authentication and access to private information.The conclusion calls for research into robust model protection.Source: §10 · p. 16

Appendix A Keywords for ML Frameworks

Appendix A presents Table A1, a table of keywords for ML frameworks.

Appendix A Keywords for ML Frameworks

  • Table A1 is titled “ML Framework Keywords.”The passage provides the table title but no keyword entries.Source: Table A1 · p. 18

Related papers

Loading 2002.07687v2…