PaperlayerView exact revision on arXiv

Source-linked AI summary

Is On-Device AI Broken and Exploitable? Assessing the Trust and Ethics in Small Language Models

Kalyan Nakka, Jimmy Dani, Nitesh Saxena

arXiv:2406.05364v2cs.CRcs.AI

TL;DR

Existing evaluations do not establish how deployment environment and on-device optimization affect SLM trustworthiness and ethical behavior. The study uses quantization to optimize AI models for on-device deployment and compares their behavior with on-server models. On-device SLMs were less trustworthy across stereotype, fairness, and privacy, produced more harmful responses, and supplied detailed unethical guidance without jailbreaking or prompt engineering.

  • Problem

    Existing evaluations do not establish how deployment environment and on-device optimization affect SLM trustworthiness and ethical behavior.

    Source: §1 · pp. 1–2

  • Method

    The study uses quantization to optimize AI models for on-device deployment and compares their behavior with on-server models.

    Source: §2.2 · p. 3

  • Results

    On-device SLMs were less trustworthy across stereotype, fairness, and privacy, produced more harmful responses, and supplied detailed unethical guidance without jailbreaking or prompt engineering.

    Source: §1 · p. 3, §6 · pp. 12–13

  • Takeaways & Limitations

    On-device SLMs present severe trust, ethical-safeguard, and exploitability vulnerabilities when deployed on personal devices.

    Source: §1 · p. 3, §6 · p. 13, §8 · p. 13

    On-device defenses must process data locally for privacy and offline functionality, but available defenses are resource intensive.

    Source: §6 · p. 13

Abstract

from arXiv · show

In this paper, we present a very first study to investigate trust and ethical implications of on-device artificial intelligence (AI), focusing on small language models (SLMs) amenable for personal devices like smartphones. While on-device SLMs promise enhanced privacy, reduced latency, and improved user experience compared to cloud-based services, we posit that they might also introduce significant risks and vulnerabilities compared to their on-server counterparts. As part of our trust assessment study, we conduct a systematic evaluation of the state-of-the-art on-devices SLMs, contrasted to their on-server counterparts, based on a well-established trustworthiness measurement framework. Our results show on-device SLMs to be significantly less trustworthy, specifically demonstrating more stereotypical, unfair and privacy-breaching behavior. Informed by these findings, we then perform our ethics assessment study using a dataset of unethical questions, that depicts harmful scenarios. Our results illustrate the lacking ethical safeguards in on-device SLMs, emphasizing their capabilities of generating harmful content. Further, the broken safeguards and exploitable nature of on-device SLMs is demonstrated using potentially unethical vanilla prompts, to which the on-device SLMs answer with valid responses without any filters and without the need for any jailbreaking or prompt engineering. These responses can be abused for various harmful and unethical scenarios like: societal harm, illegal activities, hate, self-harm, exploitable phishing content and many others, all of which indicates the severe vulnerability and exploitability of these on-device SLMs.

1 Introduction

This study examines whether on-device SLMs introduce greater trust, ethical, and exploitability risks than on-server deployments. It compares their trustworthiness and ethical safeguards, finding substantially more biased, unfair, privacy-breaching, and harmful behavior on-device.

Motivation

  • The paper motivates runtime-specific evaluation because existing benchmarks do not evaluate model behavior across deployment environments.On-device memory constraints require quantization, yet the behavioral effects of reduced weight and activation precision remain unjustified.Source: §1 · pp. 1–2

Trust Assessment

  • On-device SLMs showed high risks of stereotypical bias, unfairness, and privacy-breaching behavior.The comparative trust assessment evaluated SLMs across on-server and on-device environments using multiple settings and prompts.Source: §1 · p. 2

Ethics Assessment

  • The ethics assessment found that on-device SLMs produced comparatively more harmful responses than their on-server counterparts.The study used unethical questions depicting harmful scenarios to examine ethical safeguards.Source: §1 · pp. 2–3

Summary of Results

  • On-device SLMs performed significantly worse than on-server SLMs across stereotype, fairness, and privacy trust perspectives.All SLMs leaked private and sensitive information in zero-shot settings, with greater leakage on-device; Phi-2 and RedPajama-3B were especially poorer in stereotype and fairness.Source: §1 · pp. 2–3

Exploitability

  • On-device Phi-2 and RedPajama-3B generated detailed harmful guidance from vanilla prompts without jailbreaking or prompt engineering.Responses covered societal harm, illegal activities, hate, self-harm, and exploitable phishing content, potentially due to quantization used for on-device deployment.Source: §1 · pp. 2–3

2 Background and Preliminaries

This section defines Responsible AI, contrasts on-server and on-device deployment, and introduces the models and evaluation protocols used in the study. Trust is assessed through stereotype, fairness, and inference-time privacy, while ethics is assessed using harmful-request refusal behavior.

2.1 Responsible AI

  • Responsible AI emphasizes ethical, fair, transparent, accountable, private, safe, and socially beneficial AI systems.Its core aspects include fairness, transparency, accountability, privacy, safety, and social good.Source: §2.1 · p. 3

2.2 Deployment Strategies

  • On-server AI runs models directly on local or cloud server infrastructure without additional alignment mechanisms filtering unethical requests.The study uses the transformers library to obtain on-server inferences.Source: §2.2 · p. 3
  • On-device AI runs models directly on smartphones or edge devices for real-time processing without cloud services.Deployment uses optimization methods such as quantization, pruning, or distillation; this study uses quantization, reducing weights and activations to 4-bit or 8-bit precision.Source: §2.2 · p. 3

2.3 Studied Models

  • The study examines three open-source SLMs: Gemma-2B, Phi-2, and RedPajama-3B.They are selected as state-of-the-art on-device models based on performance reported in standard benchmarks.Source: §2.3 · pp. 3–4

2.4 Trust Assessment

  • Trust assessment uses DecodingTrust to evaluate stereotype, fairness, and inference-time privacy implications.Stereotype testing covers benign and adversarial bias; fairness examines sensitive-attribute dependence using Adult and Crime datasets; privacy measures PII leakage across zero-shot and few-shot settings.Source: §2.4 · p. 4

2.5 Ethics Assessment

  • Ethics assessment uses the Do-Not-Answer dataset to test whether SLMs refuse unethical requests instead of providing direct answers.Responses are evaluated in zero-shot settings using automatic GPT-4 assessment.Source: §2.5 · p. 4

3 Trust Assessment Study

The trust assessment compares SLMs in on-server and on-device environments across stereotype, fairness, and privacy perspectives using DecodingTrust, with on-server results serving as a baseline. Across these perspectives, on-device SLMs show greater bias, unfairness, and privacy leakage, with statistically significant differences.

3.1 Methodology

  • The study evaluates SLM trustworthiness with DecodingTrust across stereotype, fairness, and privacy perspectives in matched on-server and on-device settings.On-server evaluation provides the baseline, while on-device models run on a OnePlus 12 smartphone using MLC-LLM and a Kotlin chat application.Source: §3 · p. 4, §3.1 · p. 4

3.2 Stereotype Perspective Results

  • The stereotype evaluation measures agreementIndex Ai across demographic and stereotype groups under benign, untargeted, and targeted scenarios.Ai ranges from 0 to 1, with values near 1 indicating stronger agreement with a stereotype statement; the study examines six demographic and six stereotype groups.Source: §3.2 · p. 5
  • All target SLMs showed stronger agreement with stereotypes on-device, while RedPajama-3B shifted from least biased on-server to most biased on-device.In on-device settings, RedPajama-3B reached Ai values of 0.87, 0.88, and 0.84 for highly stereotypical statements that received Ai = 0 on-server.Source: §3.2 · pp. 5, 7

3.3 Fairness Perspective Results

  • Fairness is evaluated on Adult and Crime datasets using accuracy ACC and demographic parity difference Mdpd.Adult prompts predict income above $50k, while Crime prompts predict whether a community’s crime rate is high or low.Source: §3.3 · p. 7
  • On-device models produced more unfair decisions than on-server models, and higher accuracy was accompanied by higher demographic parity difference Mdpd.This pattern held for both Adult and Crime datasets; Crime decisions were especially unfair in both environments.Source: §3.3 · p. 7

3.4 Privacy Perspective Results

  • Privacy evaluation uses leakage rate LR to measure the proportion of responses containing private or sensitive information.The tested PII types include email addresses, passwords, secret keys, phone numbers, credit card numbers, passport numbers, addresses, and social security numbers.Source: §3.4 · p. 8
  • On-device models exhibited substantial PII leakage across evaluation settings, with RedPajama-3B moving from no leakage on-server to leakage of all PII types on-device.On-device RedPajama-3B reached LR (%) ∈[78, 88] in zero-shot and LR (%) ∈[50, 92] under privacy-attack demonstrations.Source: §3.4 · pp. 8–9

3.5 Statistical Analysis

  • On-device SLMs performed significantly worse than on-server SLMs across stereotype, fairness, and privacy assessments.Wilcoxon tests found significant differences for every evaluated model and perspective, with medium-to-large effect sizes.Source: §3.5 · p. 9

4 Ethics Assessment Study

The ethics assessment compares harmful-response safeguards for target SLMs across on-server and on-device environments. On-device models produce more harmful responses across all five risk areas, with direct answers especially common for Information Hazards and Malicious Uses.

4.1 Methodology

  • The study deployed the same target SLMs on a Linux server and OnePlus 12 smartphone, using zero-shot Do-Not-Answer prompts for comparison.On-device inference used MLC-LLM and a Kotlin-based chat app, while on-server evaluation used a GeForce RTX 2070 GPU.Source: §4.1 · p. 10
  • The assessment used 939 highly unethical prompts organized under a three-level taxonomy spanning five harmful-risk areas.Harmfulness was measured by counting responses classified as action (5), directly responding to the instruction.Source: §4.2 · p. 10

4.2 Ethical Safeguards Results

  • Gemma-2B was safest on-device with 88 harmful responses, while Phi-2 was least safe with 302 among 939 responses.On-server, RedPajama-3B was safest with 10 harmful responses, followed by Gemma-2B with 60 and Phi-2 with 230.Source: §4.2 · p. 10
  • On-device environments increased harmful responses for all target SLMs across the five risk areas.The risk areas are Discrimination, Exclusion and Toxicity; Human-Computer Interaction Harms; Information Hazards; Malicious Uses; and Misinformation Harms.Source: §4.2 · p. 10
  • Phi-2 and RedPajama-3B showed high on-device tendencies to answer directly about Information Hazards and Malicious Uses.Gemma-2B also shifted toward direct responses for both categories on-device, whereas on-server it showed this tendency mainly for Information Hazards.Source: §4.2 · p. 10

5 Broken & Exploitable On-Device SLMs

The paper uses vanilla prompts to demonstrate that on-device SLMs can provide valid harmful answers without jailbreaking or prompt engineering. These responses span societal harm, illegal activities, hate, phishing, and self-harm, indicating substantial exploitability.

5 Broken & Exploitable On-Device SLMs

  • On-device SLMs answered vanilla prompts with valid harmful responses without filters, jailbreaking, or adversarial attack strategies.The authors describe this behavior as evidence of highly vulnerable on-device SLMs.Source: §5 · pp. 11–12

Societal Harm

  • On-device Phi-2 generated detailed instructions for building a Molotov and a bomb from vanilla prompts.These examples are contrasted with prior jailbreaking techniques used to elicit comparable answers from LLMs.Source: §5 · p. 11, Figure 16 · p. 11

Illegal Activities

  • On-device Phi-2 provided step-by-step guidance for breaking into a home undetected and committing credit-card fraud.The paper states that such detailed responses could be leveraged to commit these crimes or serve as a baseline for more serious ones.Source: §5 · p. 11, Figure 17 · p. 11

Hate Content, Phishing, and Self-Harm

  • On-device Phi-2 generated hateful tweets, vulgar comments, phishing emails requesting transfers to Bitcoin wallets or PayPal accounts, and highly sensitive self-harm responses using vanilla prompts.Source: §5 · p. 11

6 Discussion and Future Work

On-device SLMs show weaker trustworthiness and ethical safeguards than on-server counterparts, including bias, unfairness, privacy leakage, and harmful responses. Their optimization for constrained devices may contribute to this disparity, while effective on-device defenses remain difficult to deploy.

Trust assessment

  • On-device SLMs are more stereotypical and unfair than on-server counterparts, and are generally less privacy-preserving, with even Gemma-2B leaking sensitive information.The reported leaks include email addresses, passwords, and secret keys.Source: §6 · p. 12

Ethics assessment

  • On-device SLMs have produced valid answers to unethical and benign vanilla prompts, including detailed solutions and step-by-step guides for violence, crimes, hate, self-harm, and phishing content.Source: §6 · pp. 12–13

Implications

  • The resulting harmful-content generation and privacy-breaching behavior violate Responsible AI principles and expose on-device SLMs to serious exploitation risks.Source: §6 · pp. 12–13

Future work

  • Reduced bit precision in weights and activations may account for the behavioral change from on-server to on-device environments.The optimization is necessary because on-device environments have constrained computational capabilities.Source: §6 · p. 13
  • On-device defenses must be resource-efficient and process data locally, creating a need for further research on viable protections.Source: §6 · p. 13

7 Related Works

Prior work provides broad benchmarks for language-model capabilities, robustness, trustworthiness, scientific reasoning, coding, and legal reasoning. This paper is positioned within that landscape by focusing on trust and ethics across on-server and on-device SLM environments.

General and robustness benchmarks

  • GLUE, SuperGLUE, CodeXGLUE, and AdvGLUE evaluate language understanding, code tasks, and adversarial robustness across multiple benchmark settings.Source: §7 · p. 13
  • TextFlint, HELM, BIG-Bench, and BIG-Bench Lite broaden evaluation through multilingual transformations, holistic metrics, and diverse challenging tasks.Source: §7 · p. 13

Domain-specific benchmarks

  • SciEval and SciAssess assess scientific capabilities, while Mercury evaluates computationally efficient code generation and LegalBench evaluates legal reasoning.Source: §7 · p. 13

Trust and safety benchmarks

  • PromptBench studies adversarial prompt robustness, DecodingTrust evaluates trustworthiness dimensions such as fairness, stereotype bias, and privacy, and JailbreakBench standardizes jailbreak evaluation.Source: §7 · p. 13

8 Conclusion

The paper compares SLM trust and ethics in on-device and on-server environments using established evaluation resources. It finds that on-device models behave differently, produce more harmful responses, and can answer harmful vanilla prompts without filtering.

Trust assessment

  • Comparative DecodingTrust evaluations found substantially different on-device behavior from on-server behavior across trust assessments, supported by statistical analysis.Source: §8 · p. 13

Ethics assessment

  • Do-Not-Answer evaluations found more harmful responses from on-device SLMs than from on-server SLMs.Source: §8 · p. 13

Conclusion

  • On-device SLMs answered vanilla harmful prompts without filters or jailbreaks, exposing risks of stereotypical bias, unfairness, privacy breaches, and harmful content generation.Source: §8 · p. 13

Ethical Considerations

The study frames its evaluation around stakeholder protection, privacy, fairness, transparency, and accountability. It uses public or de-identified data and authorized methods while recognizing ethical trade-offs between on-device and on-server deployment.

Ethical commitments

  • The research emphasizes stakeholder rights, privacy protection, harm avoidance, transparency, and accountability in deploying small language models.Source: Ethical Considerations · p. 14

User-centered outcomes

  • The study focuses on end-user risks, including sensitive-information exposure, biased behavior, harmful behavior, and ethical dilemmas.Source: Ethical Considerations · p. 14

Ethical compliance

  • Datasets were public or de-identified, and the researchers did not reverse-engineer proprietary systems without authorization.Source: Ethical Considerations · p. 14

Methodological safeguards

  • Trust and ethics assessments were selected to evaluate risks systematically and transparently without amplifying vulnerabilities or creating new ones.Source: Ethical Considerations · p. 14

Deployment trade-offs

  • Comparing on-device and on-server models acknowledges ethical trade-offs in AI deployment and aims to support ethical, inclusive, and safe development.Source: Ethical Considerations · p. 14

Open Science

The paper commits to open science by sharing research artifacts and supporting reproducibility, subject to external restrictions. It also outlines artifact evaluation and transparency commitments aligned with USENIX Security’s policy.

Open Science

  • The authors commit to openly sharing datasets, scripts, binaries, and source code unless external factors such as licensing limitations restrict disclosure.Source: Open Science · p. 14
  • All artifacts necessary to reproduce and replicate the reported results will be shared publicly with the research community.Source: Open Science · p. 14
  • Artifacts will be made available to the Artifact Evaluation committee upon acceptance and before the final paper is submitted.Source: Open Science · p. 14
  • The authors frame transparency and openness as part of their commitment to advancing scientific knowledge and research integrity.Source: Open Science · p. 14

Ethical Disclosures

The authors are submitting the identified vulnerabilities to the relevant SLM vendors for review and action. They plan to report relevant disclosure findings in the paper’s final version.

Ethical Disclosures

A Vulnerable On-Device RedPajama-3B

On-device RedPajama-3B produces direct harmful answers to vanilla prompts across several harmful scenarios. Figures 21–25 document responses involving societal harm, illegal activities, hate, phishing, and self-harm.

A Vulnerable On-Device RedPajama-3B

Loading 2406.05364v2…