PaperlayerView exact revision on arXiv

Source-linked AI summary

TZ-LLM: Protecting On-Device Large Language Models with Arm TrustZone

Xunjie Wang, Jiacheng Shi, Zihan Zhao, Yang Yu, Zhichao Hua, Jinyu Gu

arXiv:2511.13717v1cs.CR

TL;DR

On-device LLMs can keep personal data on-device, but storing proprietary models on phones risks their theft. TZ-LLM protects inference with TrustZone, using pipelined restoration and a co-driver design for NPU time-sharing. Against a strawman TEE baseline, it reduces TTFT and improves decoding speed across evaluated models and benchmarks.

  • Problem

    On-device LLMs risk exposing proprietary model parameters stored on personal devices, while protecting them in a TEE raises secure-memory scaling and NPU time-sharing challenges.

    Source: §1 · p. 1, §2.3 · p. 3

  • Method

    TZ-LLM protects LLM inference in TrustZone using pipelined parameter restoration and TEE–REE NPU time-sharing with control-data separation.

    Source: §3.2 · p. 6, §9 · p. 14

  • Results

    Compared with the strawman TEE baseline, TZ-LLM reduces TTFT by 76.1%∼90.9% and increases decoding speed by 0.9%∼23.2%.

    Source: §1 · p. 2

  • Takeaways & Limitations

    TZ-LLM combines elastic secure-memory scaling and TEE–REE NPU time-sharing to protect on-device LLMs while meeting performance and memory-efficiency requirements.

    Source: §9 · p. 14

    Pipelined restoration may perform suboptimally on non-deterministic workloads, prefetching unused parameters in MoE or early-exit models.

    Source: §4.1 · p. 8

Abstract

from arXiv · show

Large Language Models (LLMs) deployed on mobile devices offer benefits like user privacy and reduced network latency, but introduce a significant security risk: the leakage of proprietary models to end users. To mitigate this risk, we propose a system design for protecting on-device LLMs using Arm Trusted Execution Environment (TEE), TrustZone. Our system addresses two primary challenges: (1) The dilemma between memory efficiency and fast inference (caching model parameters within TEE memory). (2) The lack of efficient and secure Neural Processing Unit (NPU) time-sharing between Rich Execution Environment (REE) and TEE. Our approach incorporates two key innovations. First, we employ pipelined restoration, leveraging the deterministic memory access patterns of LLM inference to prefetch parameters on demand, hiding memory allocation, I/O and decryption latency under computation time. Second, we introduce a co-driver design, creating a minimal data plane NPU driver in the TEE that collaborates with the full-fledged REE driver. This reduces the TEE TCB size and eliminates control plane reinitialization overhead during NPU world switches. We implemented our system on the emerging OpenHarmony OS and the llama.cpp inference framework, and evaluated it with various LLMs on an Arm Rockchip device. Compared to a strawman TEE baseline lacking our optimizations, our system reduces TTFT by up to 90.9% and increases decoding speed by up to 23.2%.

1 Introduction

TZ-LLM protects on-device LLMs in TrustZone while addressing secure-memory scaling and NPU time-sharing challenges. Its pipelined restoration and co-driver design reduce inference overhead compared with a strawman TEE baseline.

1 Introduction

  • On-device LLMs can use personal-data semantics without cloud transmission, but storing proprietary models on personal devices exposes plaintext parameters during inference.A malicious application or curious user may compromise the mobile OS to steal the models.Source: §1 · p. 1
  • Static secure-memory reservation wastes scarce mobile memory, while on-demand allocation, model loading, and decryption can add 11.6 seconds to Llama-3-8B TTFT.An 8-bit Llama-3-8B requires 8GB for parameters; commodity smartphones have at most 24GB of memory.Source: §1 · pp. 1–2
  • Pipelined restoration uses deterministic inference memory-access patterns to prefetch parameters and overlap allocation, I/O, and decryption with computation.It also prioritizes urgent pipeline tasks and partially caches early parameters when memory is available.Source: §1 · p. 2
  • The co-driver keeps a minimal data-plane NPU driver in the TEE and leaves control-plane operations to the REE driver, avoiding control-plane reinitialization on world switches.The TEE driver is about 1K LoC and launches secure NPU jobs with TrustZone configuration.Source: §1 · p. 2
  • Compared with the strawman TEE baseline, TZ-LLM reduces TTFT by 76.1%∼90.9% and increases decoding speed by 0.9%∼23.2%.The prototype uses OpenHarmony and llama.cpp on an Orange Pi with a Rockchip 3588 CPU/NPU.Source: §1 · p. 2
  • The paper identifies efficient secure-memory scaling and TEE–REE NPU time-sharing as challenges, and presents TrustZone protection of on-device LLMs as its first attempt.The authors characterize their evaluations as preliminary.Source: §1 · p. 2

2 Background and Motivation

The section motivates protecting proprietary models stored on mobile devices and explains why TrustZone-based inference must balance memory, performance, security, and compatibility. It reviews prior approaches and the trade-offs in elastic secure-memory protection.

2.1 Confidentiality of On-Device LLMs

  • Storing LLM parameters on mobile devices risks model theft by untrusted users, while on-device inference keeps personal data on-device and avoids cloud-network latency.Prior study reports that some applications leave models unprotected and others encrypt files while leaving plaintext parameters extractable from memory.Source: §2.1 · p. 2

2.2 Arm TrustZone

  • TrustZone separates secure TEE and non-secure REE execution and uses TZASC to protect contiguous physical memory from non-secure CPUs.TrustZone also controls device DMA permissions, while CMA can reserve and migrate pages to provide contiguous memory.Source: §2.2 · p. 3

2.3 Challenges of LLM Inference in TEE

  • TEE inference faces a memory-efficiency versus TTFT trade-off: static secure memory reserves scarce capacity, while dynamic scaling incurs cold-start costs totaling 11.6 seconds.A Rockchip NPU can improve Llama-3-8B prefill and decoding speed by 12.5× and 1.3×, respectively, but sharing it between worlds adds driver-switching challenges.Source: §2.3 · p. 3

2.4 Existing Approaches Studies

  • Prior TEE protections either keep models within accelerator-absent TEEs, offload parameters with security trade-offs, or enable accelerators with other memory and framework limitations.Some offloading approaches require model modification or extra copying and computation; StrongBox statically reserves secure memory and does not protect framework integrity in the REE.Source: §2.4.1 · p. 4

2.4.1 TEE-based Model Protection.

  • Existing TEE-based systems do not simultaneously meet performance, security, and compatibility requirements when dynamic secure-memory scaling or REE–TEE accelerator time-sharing is absent.S2PT can avoid contiguous allocation, but its 4KB mappings incur continuous REE overhead, averaging 2.0% on Geekbench and reaching 9.8%.Source: §2.4.1 · p. 4, §2.4.2 · p. 4

2.4.2 Elastic Secure Memory Protection.

  • S2PT-protected parameter caching imposes continuous REE overhead, whereas CMA allocation overhead is transient and can be hidden under high memory pressure.With high memory pressure, CMA allocation throughput is 1.9GB/s, similar to the platform’s 2GB/s sequential-read throughput; multithreading can raise it to 3.8GB/s.Source: §2.4.2 · pp. 4–5

3 Overview

TZ-LLM places LLM inference in a TEE TA and relies on REE components for client access and resource delegation. Its overview combines pipelined parameter restoration with secure NPU time-sharing, within a stated threat model.

3.1 Threat Model

  • The threat model covers theft of model parameters or intermediate results through memory, flash, malicious DMA, or compromised interfaces, but excludes physical memory-confidentiality attacks, side channels, cryptographic attacks, and denial of service.The TEE OS, TEE NPU driver, and LLM TA are trusted; REE components are outside the TCB.Source: §3.1 · p. 5

3.2 System Architecture

  • The LLM framework runs as a TEE trusted application invoked by an REE client, while the TZ driver delegates model loading, memory scaling, and NPU scheduling to REE services.Pipelined restoration prefetches parameters in the computation graph’s topological order; priority scheduling and partial caching reduce pipeline bubbles, with early parameters retained and memory released in reverse order.Source: §3.2 · pp. 5–6

4 Detailed Design

TZ-LLM overlaps parameter restoration with prefill and schedules work to reduce pipeline bubbles, while its secure-memory layout supports contiguous scaling. A co-driver enables NPU time-sharing between TEE and REE while keeping the TEE-side driver small.

4.1 Pipelined Parameter Restoration

  • Pipelined restoration overlaps memory allocation, parameter loading, and decryption with prefill computation to accelerate cold starts.The computation graph is extended with restoration operators before prefill operators that use the parameters being restored.Source: §4.1 · p. 6
  • Because critical paths vary by model, prompt, and hardware, TZ-LLM greedily schedules ready CPU computation or restoration for the earliest computation operator.This policy targets CPU-busy and computation-stall cases, and its measured performance is close to optimal.Source: §4.1 · p. 7
  • Preemption lets ready CPU computation interrupt allocation or decryption, while partial caching avoids initial and later bubbles by retaining parameters across inferences.Cached parameters can be released lazily in reverse topological order according to REE memory pressure.Source: §4.1 · pp. 7–8

4.2 Pipeline-Aware Secure Memory Management

  • Parameters occupy a separate TZASC region and are progressively allocated then released in reverse order, keeping the region contiguous.A second region holds the KV cache, activations, and other data, with their allocation patterns arranged to preserve contiguity.Source: §4.2 · p. 8
  • Separating memory allocation from protection lets the REE load encrypted parameters directly into allocated memory, avoiding an I/O bounce buffer and extra copying.The TEE OS exposes extend-and-shrink interfaces to scale TZASC regions and clear memory before release.Source: §4.2 · p. 8

4.3 TEE-REE NPU Time-Sharing

  • The co-driver keeps the full NPU control plane in the REE and integrates only the minimal job-launch data plane into the TEE.The REE schedules secure and non-secure jobs through a unified queue and shadow jobs; the TEE verifies job state and sequence before launch.Source: §4.3 · pp. 8–9
  • The TEE driver switches NPU access modes to isolate secure jobs, and running it in user mode limits its privileges and additional TCB.The TEE restricts the driver's MMIO access and the NPU's access to secure job execution contexts.Source: §4.3 · pp. 9–10

5 Implementation

TZ-LLM is prototyped on OpenHarmony with llama.cpp and the Rockchip NPU driver. The implementation adds limited code to the TEE and REE, and the design is also applicable to other Arm platforms.

5 Implementation

  • The prototype uses OpenHarmony v4.1 with Linux v5.10, llama.cpp, and Rockchip NPU driver v0.9.8.The LLM trusted application is built on llama.cpp.Source: §5 · p. 10
  • The implementation adds 62 LoC and 50 LoC to the TEE OS, 1.2K LoC and 1K LoC to llama.cpp, and 364 LoC to the REE OS.The TEE additions support CMA mapping and dynamic TZASC/TZPC configuration; REE additions support shadow-job scheduling and CMA allocation/deallocation.Source: §5 · p. 10
  • Although the current implementation targets Rockchip, the design is applicable to other Arm platforms, and the authors identify a small data plane driver in Qualcomm's open-source Linux NPU driver.Source: §5 · p. 10

6 Security Analysis

TZ-LLM protects model parameters against REE and other-TA access, untrusted-device DMA, and forged or reordered REE interactions. The analysis also identifies side-channel and physical-attack boundaries.

6 Security Analysis

  • TZASC blocks REE access to plaintext secure-memory parameters, TEE address-space isolation blocks other TAs, and encrypted flash files protect stored model contents.The model key is encrypted with a hardware-protected TEE key and is accessible only to the LLM TA through the TEE OS.Source: §6 · p. 10
  • Before granting the NPU secure-memory access, the TEE isolates its MMIO interface and waits for any ongoing non-secure job to complete, preventing malicious DMA targeting.TZASC also rejects access to parameter regions from untrusted devices.Source: §6 · p. 10
  • TZ-LLM validates returned memory contiguity, secure-job authorization and order, model-loading checksums, and TA-thread synchronization to counter Iago attacks.Source: §6 · p. 10
  • Parameter tensor sizes and secure NPU execution times are exposed to the REE and may reveal model structure, while offline DRAM attacks stem from TrustZone hardware limitations.Existing side-channel attacks and physical attacks through offline DRAM analysis are outside the paper's scope.Source: §6 · pp. 10–11

7 Performance Evaluation

The evaluation measures end-to-end performance, pipelined restoration, NPU time-sharing, and CMA interference across on-device LLMs and benchmarks. TZ-LLM improves TTFT against the strawman while adding overhead relative to REE baselines and during shared-NPU use.

7.1.1 Prefill Performance.

  • Across prompt lengths, TZ-LLM reduced TTFT 77.1%∼91.1% versus the strawman, but incurred up to 8.5× overhead versus REE-LLM-Memory.Versus REE-LLM-Flash, overhead was 2.5%∼22.3%, 22.2%∼55.3%, and 10.2%∼15.2% at 32-, 128-, and 512-token prompts; medium prompts allowed less overlap of allocation and decryption with computation and I/O.Source: §7.1.1 · pp. 11–12
  • Across benchmarks, geometric-mean TTFT reduction was 76.1%∼90.9% versus the strawman, with 5.2%∼28.3% slowdown versus REE-LLM-Flash.Figure 10 reports average TTFT on UltraChat, PersonaChat, and DroidTask.Source: §7.1.1 · p. 12, Figure 10 · p. 12

7.1.2 Decoding Performance.

  • Decoding speed improved 0.9%∼23.2% versus the strawman but slowed 1.3%∼4.9% versus REE-LLM, as single-token iterations underuse NPU compute.Figure 11 reports token-generation speed by model; its percentages show TZ-LLM's relative performance change against each baseline.Source: §7.1.2 · p. 12, Figure 11 · p. 12

7.2.1 Scheduling Policy Effectiveness.

  • With memory stress enabled, TZ-LLM's scheduling policy incurred 0.01%∼9.9% overhead over the theoretical TTFT lower bound; without stress, overhead was 10.4%.The lower bound is the maximum latency among the three potential critical paths; Figure 12 compares path latencies and TTFT with 20% of parameters cached.Source: Figure 12 · p. 12, §7.2.1 · p. 13

7.2.2 Effect of Preemptive Scheduling.

  • Pipelining without preemption reduced TTFT by up to 31.7%; preempting allocation and decryption operators further reduced it by up to 16.2%.Preemption eliminates pipeline bubbles caused by misaligned operator execution times.Source: §7.2.2 · p. 13, Figure 13 · p. 13

7.2.3 Effect of Partial Parameter Caching.

  • As caching increases, TTFT falls approximately linearly until a model- and prompt-dependent threshold, after which additional caching helps less.The threshold depends primarily on NPU computation time and can be identified through profiling; Figure 14 normalizes TTFT to the 0% cache setup.Source: §7.2.3 · p. 13, Figure 14 · p. 13

7.3 NPU Time-Sharing Performance

  • Concurrent NPU sharing adds up to 3.8% slowdown for NN applications and 3.0% for LLMs versus REE sharing; time-sharing overhead is 1.6%∼2.7% of TTFT and 2.3%∼5.7% of decoding time.Figure 15 plots NN-application and LLM throughputs under exclusive and concurrent NPU use.Source: Figure 15 · p. 13, §7.3 · pp. 13–14

7.4 Interference Between CMA and REE Applications

  • During prefill, Geekbench scores degraded by up to 6.7% versus REE-LLM-Memory and 5.8% versus REE-LLM-Flash; CMA overhead was negligible during decoding or when inference was idle.Figure 16 reports Geekbench scores during Llama-3-8B prefill with a 512-token prompt.Source: §7.4 · p. 14, Figure 16 · p. 14

8 Other Related Work

Related work addresses parameter offloading, model confidentiality, and accelerator protection through approaches with distinct device availability and performance trade-offs.

On-device LLM inference

  • Prior parameter-offloading techniques swap LLM parameters between memory and flash, but combining them with TZ-LLM is future work; TZ-LLM currently keeps parameters in memory during decoding.Other work uses quantization, distillation, or pruning to reduce parameter sizes, while dynamic secure memory scaling remains necessary for memory efficiency.Source: §8 · p. 14

Model confidentiality protection

  • Homomorphic encryption and multi-party computation protect model confidentiality but incur significant performance degradation, particularly on resource-constrained mobile devices..Source: §8 · p. 14

TEEs with accelerators

  • CVM-based systems are not currently available on mobile devices; other accelerator-protection approaches include StrongBox and sNPU, which extends the TCB or requires hardware modifications, respectively.TZ-LLM partitions the NPU driver into control and data planes and supports NPU use in the TEE with minimal security impact.Source: §8 · p. 14

9 Conclusion

TZ-LLM protects on-device LLMs with Arm TrustZone while targeting performance, memory efficiency, and security through elastic secure-memory scaling and TEE–REE NPU time-sharing.

9 Conclusion

  • TZ-LLM uses elastic secure-memory scaling with pipelined restoration to address LLM performance, memory efficiency, and security requirements.The system also uses TEE–REE NPU time-sharing with control-data separation.Source: §9 · p. 14
  • TEE–REE NPU time-sharing with control-data separation is the system’s second design element.Source: §9 · p. 14
  • The paper presents TZ-LLM as a system for protecting on-device LLMs with Arm TrustZone.Source: §9 · p. 14

A.1 Abstract

The artifact includes the prototype source code and scripts for conducting the paper’s experiments.

A.1 Abstract

  • The artifact contains source code for the prototype system.Source: §A.1 · p. 18
  • The artifact includes scripts for conducting the experiments presented in the paper.Source: §A.1 · p. 18
  • The artifact packages both the prototype source code and experiment scripts.Source: §A.1 · p. 18

A.2 Description & Requirements

The artifact is available through a DOI and requires an Orange Pi 5 Plus board plus a Linux build machine with specified resources.

A.2 Description & Requirements

  • The artifact is available at DOI 10.5281/zenodo.17213486.Source: §A.2 · p. 18
  • Artifact hardware requirements include an Orange Pi 5 Plus board with an RK3588 CPU/NPU.Source: §A.2.1 · p. 18
  • Building and communicating with the board requires a standalone machine, USB-to-USB cable, at least 8 GB memory, and 100 GB free disk space.Source: §A.2.1 · p. 18

A.2.1 Hardware Dependencies.

The appendix specifies artifact setup requirements and experiment scripts, and reports TTFT, decoding-speed, and caching claims.

A.2.1 Hardware Dependencies.

  • The setup machine should run Linux, with Ubuntu 22.04.3 tested, and have OpenHarmony Device Connector and Python3 installed.Source: §A.2.2 · p. 18

A.2.2 Software Dependencies.

  • The listed software dependencies also include matplotlib for plotting and analysis, and Docker for containerized builds.Source: §A.2.2 · p. 18

A.3 Setup

  • Setup requires downloading the source code and consulting README.md for details.Source: §A.3 · p. 18

A.4.1 Major Claims.

  • 76.1%∼90.9% lower TTFT than Strawman and 5.2%∼28.3% overhead versus REE-LLM-Memory are reported for TZ-LLM.The claim is demonstrated by experiment E1, with results reported in Figure 10.Source: §A.4.1 · p. 18
  • 0.9%∼23.2% higher decoding speed than Strawman and 1.3%∼4.9% overhead versus REE-LLM are reported for TZ-LLM.The claim is demonstrated by experiment E2, with results reported in Figure 11.Source: §A.4.1 · p. 18
  • Partial parameter caching reduces TTFT approximately linearly as more parameters are cached, up to a threshold.Experiment E3 reports this effect across cache proportions and prompt lengths in Figure 14.Source: §A.4.1 · p. 18

A.4.2 Experiments.

  • Experiments E1, E2, and E3 evaluate TTFT across benchmarks, decoding speed across models, and caching effects across cache proportions and prompt lengths.The appendix provides scripts for each experiment and estimated compute times of approximately 60, 20, and 60 minutes, respectively.Source: §A.4.2 · p. 18

Related papers

Loading 2511.13717v1…