MobileAIBench: Benchmarking LLMs and LMMs for On-Device Use Cases

Motivation

• Mobile hardware limits model size and resource availability, while quantization reduces weight precision and resource footprint with seemingly little performance penalty.Parameters drive computation and consumption of memory, CPU, and GPU resources.Source: §1 · pp. 1–2

Research gaps

• Quantization’s effects on task performance, trust and safety, and real-device behavior remain insufficiently measured, alongside limited systematic testing tools.The gaps span LLM tasks, LMM tasks, and deployment on actual mobile devices.Source: §1 · p. 2

Approach

• MobileAIBench evaluates open-source LLMs and LMMs with up to 7B parameters across 16-bit to 3-bit quantization, representative tasks, and iPhone 14 latency and utilization metrics.Measured metrics include time-to-first-token, CPU usage, and RAM usage.Source: §1 · p. 2

Contributions

• The framework combines a desktop/server evaluation library with an open-source iOS app for quantized-model latency and CPU/RAM measurements.This two-part design supports benchmark evaluation and direct mobile deployment assessment.Source: §1 · p. 2
• MobileAIBench provides reproducible benchmarking across quantization levels and tasks, including open-source on-device testing for assessing mobile deployment viability.The framework is positioned for both researchers evaluating small models and practitioners assessing deployment feasibility.Source: §1 · p. 2

2 Related Work

• Established benchmarks cover knowledge, reasoning, open-ended dialogue, trust and safety, instruction following, formatting, and multimodal abilities.Examples include MMLU [17], AlpacaEval [12], MT-Bench [50], TruthfulQA [25], TrustLLM [36], Safetybench [48], and MME.Source: §2 · p. 2
• These benchmarks generally do not evaluate quantized model versions or determine how deployment constraints affect task performance.MobileAIBench addresses this gap by focusing on real mobile-device deployment.Source: §2 · pp. 2–3
• MobileAIBench emphasizes model quantization, inference speed, and required deployment resources in addition to task-specific model performance.Its focus is deployment utilization on real mobile devices.Source: §2 · p. 3
• Prior mobile-model work studies quantization, architectures, vision-language models, agents, and small-model evaluations, but lacks MobileAIBench’s comprehensive combination of data, models, and metrics.The standardized benchmark is intended to help developers consider multiple factors in mobile-ready model development.Source: §2 · p. 3

3 MobileAIBench Framework

• MobileAIBench is a two-part framework: a desktop/server benchmark pipeline and an open-source iOS app for latency and mobile hardware utilization.The framework extends task evaluation from desktop performance to actual mobile-device deployment conditions.Source: §3 · p. 3, §3.3 · pp. 4–5

3.1.1 Standard NLP Tasks

• The benchmark covers standard NLP tasks including question answering, summarization, text-to-SQL, multitask knowledge, mathematics, and open-ended dialogue.Datasets include Databricks-dolly, HotpotQA [44], CNN/Daily Mail [19] [29], XSum [30], Sql-Create-Context, MMLU, GSM8K [10], AlpacaEval, and MT-Bench.Source: §3.1.1 · pp. 3–4

3.1.2 Multimodal Tasks

• Multimodal evaluation uses five visual question-answering datasets spanning direct answers and multiple-choice reasoning, with 1,000 randomly selected samples per dataset.The datasets are VQA-v2 [15], VizWiz [16], GQA [20], TextVQA [34], and ScienceQA [28].Source: §3.1.2 · p. 4

3.1.3 Trust and Safety

• Trust-and-safety evaluation spans truthfulness, safety, robustness, fairness, privacy, and ethics using dedicated datasets and prompt sets.The suite includes TruthfulQA [25], Do-Not-Answer [42], Adversarial Instruction [36], BBQ [32], Priv-Lk, and Social Chemistry 101 [13].Source: §3.1.3 · p. 4

3.2 Part 1: Evaluation Pipeline for Desktop and Cloud

• The desktop pipeline preprocesses task data, hydrates prompts, generates model predictions, and applies task-specific or generic metrics in a plug-and-play design.The design supports adding tasks, metrics, and custom leaderboards.Source: §3.2 · p. 4

3.3 Part 2: Mobile App

• The iOS app measures model efficiency through TTFT, ITPS, OET, and total time, plus device utilization through CPU, RAM, and battery-drain rate.Efficiency depends on model structure, quantization level, and prompt template.Source: §3.3 · p. 5

3.4 Model Libraries Supported

• Hugging Face inference tests pretrained models on predefined tasks, whereas Llama.cpp supports quantization for smaller models and real-device deployment.The two libraries accommodate a broad range of LLMs and LMMs.Source: §3.4 · p. 5

4 Benchmarks and Analysis

• MobileAIBench combines desktop benchmark evaluation with an iOS app that measures latency and hardware utilization for mobile deployment studies.Effectiveness and trust-and-safety experiments were run on desktops, while efficiency and resource utilization were measured on an iPhone 14.Source: §4 · p. 5

4.1.1 Standard NLP Tasks

• Quantized models generally retain task performance, with minimal impact reported for standard NLP and trust-and-safety tasks, although sensitivity varies by model and task.No single model consistently leads across all tasks, while larger LLMs generally outperform medium-sized models.Source: §4.1.1 · p. 6, §4.2 · p. 7

4.1.2 Multi-modality Tasks

• LMM accuracy remains consistent across quantization levels until 3-bit quantization, when most models experience a significant drop; Moondream2 remains robust even at 3 bits.At 16-bit precision, Llava-v1.5-7B and BakLLava generally outperform others, while the smaller Moondream2 is highly comparable except on SQA.Source: §4.1.2 · p. 6
• Accuracy generally increases with disk usage, making top-left models in the accuracy–size plot the best overall trade-offs.Disk usage is related to the number of model parameters.Source: §4.1.2 · pp. 6–7

4.3 Efficiency and Utilization Evaluation

• On an iPhone 14, smaller LLMs decode faster, but total response time also depends on output length; 4-bit TinyLlama 1B still uses more than half of the device’s 6 GiB RAM.CPU utilization is not related to model size, while battery drain increases with model size and output-token count.Source: §4.3 · p. 9
• On-device LMM inference is substantially more computationally intensive than NLP inference, with average TTFT above 60 seconds and ITPS below 5 for both evaluated tasks.The results indicate that current LMMs may not yet be suitable for mobile deployment, although newer devices could improve this outcome.Source: §4.3 · p. 9

• MobileAIBench covers text and multimodal tasks alongside trust and safety, evaluating effectiveness, efficiency, and utilization.Source: §5 · p. 9
• Quantization reduces model size while preserving performance, but models and tasks vary in sensitivity to quantization levels.Source: §5 · p. 9
• Even 1B models require significant CPU and RAM on mobile devices, indicating a need for more compact and optimized models.Source: §5 · p. 9
• The framework excludes model pruning and lacks comprehensive cross-hardware comparisons, while support for additional mobile device types remains under development.Source: §5 · p. 9

• Twenty datasets cover NLP, multimodality, and trust-and-safety tasks for mobile use cases.Source: §A.1 · p. 14
• Most original datasets are down-sampled to approximately 1,000 samples to support runnable mobile-device experiments.Source: §A.1 · p. 14
• The datasets are summarized in Table 6, providing the framework’s integrated evaluation collection.Source: Table 6 · p. 14, §A.1 · p. 14

LLM Model Selection

• LLM coverage spans large models above 6B parameters and medium models from 1B to 6B, including Llama2, Mixtral-7B, Gemma-7B, Phi-2, TinyLlama-1.1B, Gemma-2B, and StableLM-3B.Source: §A.2 · p. 14

LMM Model Selection

• LMM coverage includes large Llava-v1.5-7B and BakLLava-7B models plus medium Llava-phi-2, Mobile-VLM, and Moondream2 models.Source: §A.2 · p. 14

LLM Model Selection

• Phi-2, Gemma-2B, and TinyLlama-1.1B are quantized to 4-bit for compatibility with iPhone 14 resource constraints.Source: §A.2 · p. 14

LMM Model Selection

• LMMs are harder to serve on mobile because of ensemble structure, larger model sizes, and complex inference, so selection includes models not yet fully supported by Llama.cpp.Source: §A.2 · p. 14

• Evaluation metrics cover standard NLP, multimodality, and trust-and-safety tasks.Source: §A.3 · p. 14
• The evaluation details are provided for each of the three task categories.Source: §A.3 · p. 14
• The metric design spans both text and multimodal mobile-use evaluations.Source: §A.3 · p. 14

• Table 7 lists evaluation prompts for various NLP datasets.Source: §A.3.1 · p. 15
• Table 8 lists evaluation prompts for various multimodal datasets.Source: §A.3.1 · p. 15
• These templates specify the prompts used to evaluate the respective datasets.Source: §A.3.1 · p. 15

A.3.2 Evaluation Metrics

• A.3.2 Evaluation Metrics: The framework uses task-specific measures including EM and F1 for question answering, SP and LS for Text-to-SQL, and R1 and RL for summarization.Source: §A.3.2 · p. 15
• Win-rate is the fraction of times a predicted response is chosen over a baseline model’s prediction, with GPT-4 as the baseline; score uses GPT-4 to rate responses on a 10-point scale.Source: §A.3.2 · p. 15
• A.3.2 Evaluation Metrics: Trust and safety evaluation includes GPT-4o-judged accuracy and exact-match-based scores for datasets with single or multiple ground-truth answers.Source: §A.3.2 · p. 15

A.4.1 Utilization Experiments

• A.4.1 Utilization Experiments: CPU utilization fluctuates across datasets because loading and inference use different amounts of CPU, while memory remains nearly constant during inference.The stable memory trace indicates that models occupy most memory and sample-related memory costs are relatively limited.Source: §A.4.1 · p. 16, Figure 6 · p. 16

A.4.2 Latency Analysis for LMMs on Intel CPU

• A.4.2 Latency Analysis for LMMs on Intel CPU: 8-bit and 4-bit quantization generally improve inference speed, whereas 3-bit quantization is not faster than the original model.Source: §A.4.2 · p. 17, Table 9 · p. 17
• A.4.2 Latency Analysis for LMMs on Intel CPU: Smaller models generally have lower latency, while Mobile-VLM is significantly faster than similarly sized models.The paper attributes Mobile-VLM's lower latency to an additional convolutional layer that reduces image tokens by a factor of four [9].Source: §A.4.2 · p. 17