mzCache: On-Device LLM Memory Management under Multitasking

Hongseung Yu; Minsung Kim; Jongseok Park; Kyunghan Lee

Source-linked AI summary

mzCache: On-Device LLM Memory Management under Multitasking

Hongseung Yu, Minsung Kim, Jongseok Park, Kyunghan Lee

arXiv:2609.01338v1Sep 1, 2026cs.OScs.DCcs.LG

TL;DR

Mobile multitasking can evict large LLM weights and KV caches, making restoration or KV-cache recomputation a major responsiveness problem. mzCache uses restoration-oriented memory management with shared buffers, hybrid swap, and ordered eviction to overlap GPU inference with CPU restoration. On commercial smartphones, it achieves 2.1–5.5× faster TTFT than storage-backed partial offload and works under real-world app pressure.

Problem
Dynamic multitasking memory pressure evicts LLM weights and KV caches, while OS restoration and KV-cache recomputation substantially delay the next response.
Method
mzCache partitions LLM memory into fine-grained shared buffers and combines hybrid swap with backward-out eviction to overlap GPU inference and CPU-side restoration.
Results
2.1–5.5× faster TTFT than storage-backed partial offload was achieved on commercial smartphones under real-world multitasking pressure.
Takeaways & Limitations
Dedicated LLM-aware memory management is essential for responsive on-device inference in multitasking environments.
Takeaways & Limitations
OS memory management can terminate the LLM process under sustained pressure, losing the entire KV cache and requiring a cold start.

Abstract

from arXiv · show

On-device mobile Large Language Model (LLM) inference is gaining significant attention. However, mobile devices operate in highly dynamic multitasking environments where users frequently switch between applications. This creates memory pressure, forcing LLM memory (model weights and KV cache) to be evicted by the operating system. When a new inference request arrives, the inference system must restore the evicted memory through slow storage reads or recompute the entire KV cache, severely degrading responsiveness. To address this, we present mzCache, an on-device LLM inference system with specialized memory management for multitasking environments. Under unpredictable memory pressure, mzCache elastically evicts LLM memory and leverages the unified memory of mobile SoCs to enable zero-wait inference on the GPU with concurrent CPU-side restoration. mzCache realizes this through restoration-oriented memory management: LLM memory is partitioned into fine-grained shared buffers to enable partial eviction and restoration with concurrent cross-processor access, while hybrid swap and backward-out eviction policies ensure low-latency restoration from any eviction state. Implemented on llama.cpp and deployed as an Android application, mzCache achieves 2.1-5.5$\times$ reduction in Time-to-First-Token compared to storage-backed partial offload and demonstrates its effectiveness in real multitasking scenarios.

1 INTRODUCTION

Mobile LLMs preserve interaction context in KV caches, but multitasking memory pressure can evict LLM memory and sharply delay responses. mzCache addresses this with restoration-oriented memory management and achieves faster TTFT than storage-backed partial offload.

Motivation: KV caches preserve prior interaction context, but can grow to around 4 GB, bringing total LLM memory to 5–7 GB with model weights.This footprint consumes nearly half of a typical 12 GB smartphone’s RAM.
Motivation: As users switch to other applications, OS eviction forces LLM memory restoration before inference, increasing TTFT from 0.8 to 16 seconds.Storage reads and on-demand paging block inference during restoration.
Motivation: Existing OS memory management is unsuitable for LLM workloads, motivating LLM- and mobile-aware handling of external memory pressure.The paper identifies severe TTFT and user-experience degradation under multitasking.
mzCache: mzCache elastically reduces its memory footprint under unpredictable pressure while maintaining readiness for rapid restoration throughout eviction.Its restoration-oriented design targets responsive inference from any eviction state.
mzCache: Fine-grained shared buffers partition weights and KV cache, enabling partial eviction and concurrent CPU restoration with GPU inference on unified memory.Hybrid restoration paths and backward-out eviction preserve early layers so inference can begin while later data is restored.
Evaluation: 2.1–5.5× faster TTFT than partial offload demonstrates mzCache’s effectiveness under realistic multitasking pressure on commercial smartphones.The system was implemented on llama.cpp and evaluated across mobile devices and LLM models.

2 BACKGROUND

Mobile LLM inference relies on KV caches to avoid recomputation, but growing context and unpredictable multitasking pressure increase both memory use and TTFT. General paging and compression mechanisms provide limited relief.

LLM Characteristics: LLM inference has prefill and decode stages: prefill processes the input and builds the KV cache, while decode generates tokens using accumulated cache entries.Prefill TTFT grows quadratically with input length, whereas decode avoids recomputing prior attention through the KV cache.
LLM Characteristics: KV caches preserve private interaction context across requests, supporting personalized responses from message history, email threads, and chat logs.The model references stored key–value pairs during later inference.
LLM Characteristics: For models under 3 billion parameters, KV caches can reach 4 GB and total memory can reach 5–7 GB with model weights.Long contexts and accumulated interactions drive this footprint.
LLM Characteristics: Larger accumulated KV caches increase prefill computation because new inputs attend over the entire stored cache, raising TTFT.The scaling depends on both new input length and accumulated KV-cache size.
Multitasking Memory Pressure: 0.2–9.3% space savings under lz4 and zstd show that general-purpose zRAM compression compresses KV caches poorly.The reported setting is an 8k-token WikiText context.
Multitasking Memory Pressure: Users switch between applications frequently, creating unpredictable memory pressure that forces the OS to reclaim memory from inactive processes.The passage reports switching between applications over 100 times per day.
Multitasking Memory Pressure: General paging restores evicted data through storage or swap space, whose high read latency can severely degrade responsiveness.Android uses zRAM to provide swap space, but the general mechanism remains separate from LLM-specific memory structure.

3 MOTIVATION

Mobile OS memory management handles neither pageable nor page-locked LLM memory well, causing severe restoration latency under multitasking pressure. mzCache targets these constraints with fine-grained unified-memory management, dual restore paths, and concurrent CPU–GPU operation.

Limitations of OS Memory Management for LLMs: LLM memory in mobile multitasking is difficult to manage because OS policies treat weights and KV cache differently across pageable and page-locked regions.Pageable weights may be dropped, KV cache may be swapped, and page-locked memory cannot be evicted except through process termination.
Limitations of OS Memory Management for LLMs: 0.2% to 9.3% space savings: general-purpose zRAM compression provides little relief for KV cache across different LLM models.Poor compression leaves substantial KV cache resident in zRAM, while continued pressure can trigger process termination and loss of context.
Limitations of OS Memory Management for LLMs: 18–20 times with flash storage swap and 5–6 times with zRAM: full eviction increases TTFT relative to the all-in-memory case.On-demand paging continues throughout prefill because the first token requires all weights and the entire KV cache.
Design Constraints: 10–13% of TTFT: reallocation contributes a non-trivial restoration cost when evicted regions must be recreated before data loading.The overhead includes page allocation, zeroing, and IOMMU/SMMU page-table updates, and recurs at every restoration.
Opportunities: mzCache partitions weights and KV cache into fine-grained shared buffers in unified memory, enabling partial eviction and concurrent CPU restoration with GPU inference.It reinterprets compressed memory and storage as dual restore paths, while shared physical memory lets processors coordinate without redundant copies.
Opportunities: Shared memory bandwidth is treated as an enabler because no single processor fully utilizes it and storage reads occupy only a small fraction.Using CPU, GPU, and storage together puts otherwise spare bandwidth to work during restoration, although the processors and paths share the subsystem.

4 SYSTEM DESIGN

mzCache manages mobile LLM memory under dynamic pressure through fine-grained shared buffers, concurrent GPU inference and CPU restoration, hybrid swap, and staged eviction policies.

Memory management: mzCache partitions weights and KV cache into fine-grained shared buffers, enabling partial eviction and concurrent CPU restoration with GPU inference.The shared buffers reside in unified memory and avoid redundant processor-to-processor data transfers.
Memory management: During inference, the GPU begins prefill using resident data while the CPU restores evicted weights and KV chunks from storage and in-memory swap.Restoration writes directly into shared buffers as computation proceeds.
Hybrid swap: Hybrid swap distributes evicted KV cache between in-memory decompression and storage reads because their throughputs differ and context lengths vary.The policy uses KV chunks as a balancing lever so both restore paths remain utilized.
Eviction policy: mzCache uses four eviction stages—KVonly, KVandW, Wonly, and CompKV—to adapt eviction as KV and weight memory become available or exhausted.The stages progressively move KV chunks and weights across the two restore paths, including compressed KV eviction to storage.
Eviction policy: Backward-out eviction preserves early layers, allowing restoration and prefill to proceed concurrently despite sequential layer dependencies.Alternative forward-out and random-out policies are contrasted with backward-out in the evaluation.

5 IMPLEMENTATION

mzCache is implemented on llama.cpp with modular KV-cache compression and interfaces for externally triggered eviction and restoration on Android mobile GPUs.

Implementation: mzCache is implemented with 6k lines of C/C++ on top of llama.cpp and supports mobile GPU backends.The implementation uses 8-bit quantization by default while allowing compression algorithms to be replaced.
Interfaces: The system exposes evict(size) and restore_generate(input_tokens) interfaces for memory release and inference-time restoration.GPU prefill can begin as soon as restored layers become available through C++ synchronization primitives.

6 EVALUATION

The evaluation measures mzCache against OS Paging and Partial Offload across devices, models, memory levels, optimization components, energy use, compression choices, and multitasking deployment.

Overall performance: 2.1–5.5× speedup is achieved by mzCache over Partial Offload at 0%, 25%, 50%, and 75% remaining memory levels.The speedup is consistent across KV-cache sizes, model types, and devices.
Overall performance: OS Paging performs worse than Partial Offload in most cases, with restoration latency dominated by on-demand paging and substantial variance on OnePlus 12.Full OS eviction still retains substantial memory because of poor KV-cache compression, whereas mzCache can reach 0% remaining memory.
Core techniques: 1.37× allocation speedup and 2.15× reload speedup are reported for mzCache without overlapping execution in the representative breakdown.The improvements are attributed to parallelized allocation and zero-copy transfers.
Core techniques: Parallel restoration significantly outperforms sequential restoration because hybrid swap balances the in-memory decompression and storage-read paths.The sequential paths exhibit similar latencies, indicating effective load balancing.
Power and energy: 19.2 W peak power is measured for mzCache versus 14.6 W for Partial Offload, while mzCache consumes less total energy.The higher peak reflects parallel storage, CPU, and GPU utilization; the lower energy follows from faster completion.
Compression: 8-bit quantization yields lower TTFT at 50% remaining memory, whereas CacheGen performs better at 0% because of its higher compression ratio.The comparison also reports space savings and F1 score relative to FP16.
Real-world deployment: mzCache (25 pp) survives all ten multitasking rounds and restores context without cold starts, while mzCache (15 pp) is terminated in four of ten rounds.OS Paging is terminated by the LMK because poorly compressed KV cache continues to occupy substantial memory.

7 DISCUSSION

mzCache separates deployment-specific pressure detection and eviction sizing from its core mechanism and discusses portability, multi-context extension, and thermal considerations.

Deployment: mzCache can use application callbacks or system-level kernel signals for pressure detection, while deployment settings determine the eviction amount.The core mechanism remains unchanged across third-party application and system-level service deployments.
Hardware portability: CPU decompression and GPU prefill are not fixed roles, allowing adaptation to configurations using an NPU or GPU-based decompression.The design is described as applicable to unified-memory SoCs beyond the implemented Qualcomm platform.
Scope: The current implementation supports a single LLM context, while multiple coexisting LLM applications require additional coordination policies and data structures.Multi-context support is left for future work.

8 RELATED WORK

Prior mobile LLM systems optimize runtime inference or controlled data offloading, but mzCache addresses externally imposed, unpredictable eviction during idle periods between inferences.

Offloading-based LLM inference: Existing offloading systems move LLM data between GPU, CPU, or disk while controlling what to offload and when.Other systems manage stored KV-cache contexts across memory tiers to accelerate restoration upon reuse.
Mobile LLM inference: Mobile LLM optimizations primarily accelerate runtime execution through heterogeneous processors, compiler stacks, or storage-based weight offloading.These approaches do not manage memory during idle periods between inferences, when preserved KV cache creates a multitasking challenge.
Distinguishing setting: mzCache targets externally imposed eviction under unpredictable memory pressure, requiring adaptation to unknown eviction amounts at runtime.This setting differs from systems that control offloading with full knowledge of available memory.
Contribution: mzCache is the first system described here to manage preserved KV-cache memory during idle periods in mobile multitasking environments.Its design is inspired partly by mobile operating-system systems combining compressed and storage-backed swap, although general-purpose compression is insufficient for this setting.

9 CONCLUSION

mzCache elastically evicts and rapidly restores LLM memory under external pressure. On commercial smartphones, it improves TTFT over storage-backed partial offload and preserves user context during real-world app usage.

Results: mzCache achieves 2.1–5.5× faster TTFT than storage-backed partial offload on commercial smartphones.The evaluation compares Time-to-First-Token under mobile multitasking memory pressure.
System: mzCache elastically evicts and rapidly restores LLM memory under external memory pressure.The system is evaluated as an on-device LLM inference system.
Results: mzCache preserves the user’s context during pressure from real-world app usage.This result demonstrates effectiveness in real multitasking scenarios.
Conclusion: The results support dedicated memory management rather than reliance on OS-based mechanisms for responsive on-device LLM inference in multitasking environments.This conclusion is stated within the evaluated mobile multitasking setting.