Source-linked AI summary
Sustainable LLM Inference for Edge AI: Evaluating Quantized LLMs for Energy Efficiency, Output Accuracy, and Inference Latency
Erik Johannes Husom, Arda Goknil, Merve Astekin, Lwin Khin Shar, Andre Kåsen, Sagar Sen, Benedikt Andreas Mithassel, Ahmet Soylu
TL;DR
Edge deployment of LLMs lacks systematic evidence about the joint energy, accuracy, and latency effects of quantization. This paper evaluates 28 quantized Ollama models on a Raspberry Pi using hardware-based energy measurements and diverse task benchmarks, finding substantial energy and latency reductions with task-dependent accuracy trade-offs.
Problem
Research has limited real-world, systematic evidence comparing quantized LLM energy consumption, accuracy, and latency across models, quantization levels, and task types on edge devices.
Method
The study benchmarks 28 quantized Ollama LLMs across diverse datasets and tasks on a Raspberry Pi 4, measuring inference energy with Joulescope and evaluating accuracy and latency.
Results
Quantization reduces energy consumption by up to 79% and inference latency by up to 69%, while accuracy varies across tasks and extreme compression can degrade complex reasoning performance.
Takeaways & Limitations
Selecting quantization requires balancing energy efficiency, inference speed, and task-specific accuracy against deployment constraints.
Takeaways & Limitations
The experiments use a single Raspberry Pi 4, so findings may not generalize to other edge hardware platforms.
Abstract
from arXiv · showhide
Deploying Large Language Models (LLMs) on edge devices presents significant challenges due to computational constraints, memory limitations, inference speed, and energy consumption. Model quantization has emerged as a key technique to enable efficient LLM inference by reducing model size and computational overhead. In this study, we conduct a comprehensive analysis of 28 quantized LLMs from the Ollama library, which applies by default Post-Training Quantization (PTQ) and weight-only quantization techniques, deployed on an edge device (Raspberry Pi 4 with 4GB RAM). We evaluate energy efficiency, inference performance, and output accuracy across multiple quantization levels and task types. Models are benchmarked on five standardized datasets (CommonsenseQA, BIG-Bench Hard, TruthfulQA, GSM8K, and HumanEval), and we employ a high-resolution, hardware-based energy measurement tool to capture real-world power consumption. Our findings reveal the trade-offs between energy efficiency, inference speed, and accuracy in different quantization settings, highlighting configurations that optimize LLM deployment for resource-constrained environments. By integrating hardware-level energy profiling with LLM benchmarking, this study provides actionable insights for sustainable AI, bridging a critical gap in existing research on energy-aware LLM deployment.
1 Introduction
This study addresses limited real-world evidence on how quantized LLMs jointly affect energy consumption, accuracy, and latency on edge devices. It systematically compares diverse models, tasks, and quantization settings using hardware-based energy measurements.
- Prior research provides limited systematic evidence on quantized LLM energy consumption across model families, quantization levels, and task types.
- Standardized comparisons of energy consumption, accuracy, and latency across edge-device LLM deployments remain lacking.
- The study evaluates 28 quantized Ollama LLMs spanning model families, parameter scales, and quantization levels.
- The evaluation covers commonsense, reasoning, mathematical, and programming tasks using datasets selected for resource-constrained edge devices.
- Hardware-based Joulescope measurements capture real-time energy consumption during inference on a Raspberry Pi 4 with 4GB RAM.
- Quantization cuts energy use by up to 79% and latency by up to 69%, while accuracy varies across tasks and extreme compression can degrade mathematical reasoning.
2 Background
The background describes quantization as a way to reduce LLM memory and computational demands, while explaining how precision, quantization type, and energy-monitoring method shape edge deployment trade-offs.
- Post-training quantization converts trained high-precision weights into lower-bit representations without modifying the original training process.
- Weight-only quantization targets layer weights and accelerates memory-bounded GEMV operations during inference decoding.
- Weight-activation quantization targets inputs and weights to optimize compute-bounded GEMM operations during prefill.
- KV-cache quantization reduces memory overhead for long texts and large batch sizes by quantizing attention keys and values.
- Lower-bit formats improve memory and power efficiency, but reduced precision can degrade accuracy and extreme quantization can add dequantization overhead.
- Ollama evaluates PTQ weight-only formats whose qX_K_Y notation encodes bit precision, K-means clustering, and variant-specific trade-offs.
- Hardware monitors directly measure system power, whereas software tools estimate consumption using telemetry and calibrated models.
3 Study Design
The study measures quantized LLM energy and performance on a Raspberry Pi using direct power monitoring, automated logging, sampled edge-device benchmarks, and standardized response prompts.
- The experiments examine how quantization affects energy consumption, accuracy-energy trade-offs, and inference speed across five research questions and datasets.
- The Raspberry Pi 4 with 4GB RAM serves as the resource-constrained edge platform for inference measurements.
- A Joulescope power analyzer measures current and voltage between the Raspberry Pi and its power supply for direct energy profiling.
- Automated scripts log generated responses and energy readings, then compute average power, total energy per query, and peak power.
- The benchmark datasets cover commonsense, reasoning, mathematical, and programming capabilities, with sampling used to keep evaluation computationally feasible.
- Prompts require answers enclosed in <ANS> tags and omit extra explanations to simplify extraction and reduce post-processing.
3.5 Accuracy Evaluation Metrics
Accuracy evaluation combines task-specific metrics with model variants spanning multiple families, parameter sizes, storage footprints, and quantization formats.
- Accuracy metrics are selected for multiple-choice, reasoning, mathematical problem-solving, and code-generation tasks.
- TruthfulQA uses Exact Match, ROUGE-L, and Cosine Similarity to assess generated answers through exact, lexical, and semantic comparisons.
- HumanEval uses pass@1 because each prompt requests a single generated code solution.
- The model set includes LLaMA 3.2, Qwen 2.5, and Gemma 2 families with parameter sizes from 494M to 2.6B.
- Variants span FP16, 8-bit, 4-bit, and 3-bit formats to compare model size, accuracy, and energy consumption.
- Quantization reduces storage requirements, while higher-bit formats retain more precision and lower-bit formats prioritize efficiency at potential accuracy cost.
3.7 Variables
The experiment varies model family, model size, quantization type and level, task type, and dataset, then measures energy, latency, throughput, and output accuracy on an edge device.
- Independent variables: Independent variables include LLM family, parameter count, model size in bytes, quantization type and bit-width, task type, and dataset.Datasets include CommonsenseQA, BIG-Bench Hard, TruthfulQA, GSM8K, and HumanEval.
- Dependent variables: Dependent variables are energy consumption, inference latency, token generation throughput, and output accuracy.Energy is measured in joules, latency is response-generation time, and throughput is tokens generated per second.
- Experimental setup: Inference runs use a Raspberry Pi 4, Joulescope power meter, and separate computer for data recording.The setup supports power monitoring across models and datasets.
- Execution procedure: Models were evaluated sequentially by dataset with continuous power logging, backups after sessions, restarts, and batching for long HumanEval responses.The procedure was designed to support repeatability and manage storage constraints.
3.9 Data Analysis
The analysis aggregates inference measurements, removes Raspberry Pi idle energy, summarizes distributions and relationships, and identifies configurations on the energy-accuracy Pareto frontier.
- Data preparation: Measurements were aggregated across multiple runs and categorized by architecture, quantization level, and dataset.The raw data include energy, token counts, processing time, and accuracy per inference operation.
- Energy correction: Idle energy was subtracted using the Raspberry Pi’s average idle power of 2.85W, equivalent to 2.85J/s.This correction isolates inference energy from background device usage.
- Output processing: Responses were extracted from <ANS> tags, cleaned when formatting deviated, and invalid outputs were filtered before task-specific answer processing.GSM8K solutions required extracting final values from detailed explanations.
- Statistical analysis: Mean, median, and variance were computed for each model variant and quantization level, followed by Pareto efficiency analysis.A configuration is Pareto-optimal when no alternative offers at least equal accuracy with lower or equal energy and a strict improvement in one dimension.
- Visualization: Box plots summarize energy distributions, scatter plots examine energy-accuracy relationships, and Pareto curves show the efficiency-accuracy boundary.Plots use medians, interquartile ranges, whiskers, fitted regression lines, and configuration-specific styling.
4.1 How does model quantization affect the energy consumption of LLM inference on an edge device? (RQ1)
Quantization generally lowers LLM inference energy on the Raspberry Pi, but energy efficiency is non-linear across quantization levels and depends on model family, quantization method, and task characteristics.
- Quantization impact: 79% lower energy results when llama3.2_1b_instruct falls from 17.60 J/token at FP16 to 3.75 J/token with q3_K_S.For qwen2.5_0.5b_instruct, energy falls 60%, from 4.42 J/token at FP16 to 1.78 J/token with q3_K_M.
- Quantization impact: 62% greater efficiency occurs for qwen2.5_0.5b q3_K_M than q3_K_S, despite identical bit precision.For qwen2.5_1.5b, q3 variants consume more energy per token than q4 variants.
- Variability: Energy consumption in llama3.2_1b_instruct_q3_K_S varies by ±3.49 J/token, whereas qwen2.5_0.5b_instruct_q3_K_M varies by ±0.83 J/token.The results indicate that highly quantized models can differ substantially in predictability.
- Task effects: 11.92 J/token for BigBenchHard versus 1.94 J/token for HumanEval shows a sixfold task-level energy difference.HumanEval’s longer responses distribute fixed inference overhead across more tokens.
- Task effects: 0.91 is HumanEval’s average response-length correlation with energy, compared with 0.26 for BigBenchHard and 0.31 for CommonsenseQA.Long-response tasks support length-based estimation more strongly than short-response tasks.
- Conclusion: The RQ1 conclusion is that selecting quantization requires considering non-linear energy effects, model family, response length, task type, and deployment constraints.Extreme quantization can sometimes increase energy usage.
4.2 What are the trade-offs between accuracy and energy efficiency across different quantization levels in LLMs deployed on an edge device? (RQ2)
Quantization affects accuracy differently across model families and tasks, creating task-dependent energy-accuracy trade-offs rather than one universally optimal configuration.
- Model sensitivity: 0.31 ± 0.36 for qwen2.5_0.5b q3_K_S is only slightly below its full-precision accuracy of 0.32 ± 0.40.The smaller qwen2.5_0.5b base model shows minimal sensitivity to quantization.
- Task-specific accuracy: 0.06 average accuracy on GSM8K indicates that aggressive quantization does not significantly degrade already-low mathematical-reasoning performance.Higher precision offers little benefit for models struggling with complex reasoning.
- Task-specific accuracy: 0.67–0.70 accuracy for gemma2_2b exceeds 0.26–0.43 for smaller models on CommonsenseQA across quantization levels.Model scale remains important for commonsense reasoning under quantization constraints.
- Overall trade-offs: Only five models lie on the average energy-accuracy Pareto frontier across datasets.These Pareto-optimal configurations span multiple model families and quantization techniques.
- Overall trade-offs: 0.45 accuracy makes gemma2_2b_instruct_q3_K_M the highest-accuracy overall model while consuming 109 Joules/response.Its aggressive q3 quantization maintains performance at low precision.
- Deployment implications: Models optimal on average may not be optimal for individual tasks, as qwen2.5_1.5b is absent from the average frontier but dominates BigBenchHard and GSM8K.Task-specific frontiers therefore support task-specific model selection.
- Task-specific trade-offs: 0.87 accuracy at 134 Joules/response makes llama3.2_1b_q4_K_M a strong HumanEval trade-off.Aggressively quantized qwen2.5_0.5b variants reach 0.70–0.77 accuracy at 114–133 Joules/response.
- Conclusion: The RQ2 conclusion is that higher precision often retains accuracy, but some aggressive quantization remains competitive and Pareto-optimal choices vary by task.Gemma2_2b benefits from higher precision in code generation, while qwen2.5_0.5b is comparatively insensitive across tasks.
4.3 How do different quantization techniques impact inference speed on an edge device? (RQ3)
Quantization generally improves edge-device inference speed, but gains vary by model, quantization level, and task complexity. Smaller models achieve higher throughput, while aggressive quantization does not always reduce latency further.
- Token generation throughput: 3.2–10.1 tokens/sec: Qwen2.5_0.5b achieves the highest throughput, with q4 and q3 variants outperforming fp16 by 2–3×.The quantized variants reach 3.2–4.1 tokens/sec compared with fp16, particularly in q4 and q3 formats.
- Token generation throughput: 1.9–4.8 tokens/sec: Llama3.2_1b Q4 variants generally exceed FP16 at 1.4–1.7 tokens/sec and q8_0 at 2.3–3.3 tokens/sec.Larger models are slower: gemma2_2b reaches 0.8–2.1 tokens/sec, while qwen2.5_1.5b reaches 2.0–4.0 tokens/sec; quantization has less throughput impact in these architectures.
- Inference latency: 56% and 69%: FP16-to-quantized formats improve latency for llama3.2_1b and qwen2.5_0.5b, respectively.Differences among q8, q4, and q3 are less consistent, and q3 can be slower because of computational overhead.
- Model effects: 33.85s avg. versus 36.33s avg.: qwen2.5_1.5b is faster than llama3.2_1b despite having more parameters.The comparison indicates that architecture and optimization influence inference speed beyond parameter count alone.
- Task effects: 94.84s avg.: HumanEval has the highest latency, exceeding the other benchmarks at 13.96–33.19s because its responses are longer.Response length and latency show a strong correlation above 0.9 across most models.
5 Threats to Validity
The study identifies threats to measurement validity, generalizability, and construct validity. These include hardware and timeout variability, evaluation on one device and limited datasets, and incomplete coverage of efficiency and quantization-robustness metrics.
- Internal validity: Hardware fluctuations, background processes, thermal throttling, and system load could introduce noise into energy and inference measurements.The study mitigated these risks through controlled conditions, cooling, synchronized clocks, and repeated measurements.
- Internal validity: A default 20-minute timeout could prematurely terminate complex inferences, although model behavior was manually verified and timeouts adjusted when necessary.
- External validity: A single Raspberry Pi 4 with 4GB RAM may not represent other edge platforms, whose architectures, power management, and optimizations can differ.The authors call for experiments across diverse edge hardware for broader generalization.
- External validity: The five benchmark datasets may not represent interactive, dialogue-based, multilingual, or other real-world LLM applications.Future work should evaluate a wider range of tasks and deployment scenarios.
- Construct validity: Energy efficiency is measured through total inference energy, without explicitly analyzing latency, throughput, or cost per inference.The authors also note that real-world evaluation should combine energy with latency and task-performance requirements.
- Construct validity: Standard benchmark results do not explicitly measure quantization-induced accuracy degradation or robustness to precision-sensitive errors.The authors propose quantization-aware robustness assessments, including arithmetic and linguistic sensitivity.
6 Related Work
Prior work addresses quantized LLM efficiency, edge deployment, and benchmarking, but often leaves energy trade-offs insufficiently examined. This study extends that literature through hardware-based profiling of diverse quantized LLMs across tasks and efficiency dimensions.
- Quantization methods: Related approaches include activation-guided weight-and-activation quantization, mobile post-training quantization, and quantization-aware training for edge inference.Agile-Quant, MobileQuant, and EdgeQAT represent complementary strategies to weight-only quantization.
- Prior findings: Prior edge studies report substantial efficiency gains, including up to 2.55× speedup and a 160× smaller footprint with a 4.1% accuracy drop.These results illustrate the literature’s emphasis on speed, memory, and accuracy.
- Research gap: Prior studies often overlook energy-consumption trade-offs across quantization techniques while targeting memory footprint, inference speed, and accuracy.
- Study contribution: This work uses real-world hardware-based energy measurements to benchmark 28 Ollama quantized LLMs and analyze energy, performance, and accuracy trade-offs.
- Benchmarking: Existing edge benchmarks evaluate combinations of memory use, execution time, power, model size, throughput, latency, and accuracy.Examples include PalmBench, the Edge-Device LLM Competition, MLPerf tools, and AI Benchmark.
- Benchmarking: The study builds on CommonsenseQA and HumanEval by adding energy consumption to analyses of quantized LLM output accuracy and efficiency on edge devices.
7 Conclusion
The study finds that quantization improves energy consumption and inference latency on edge devices, while accuracy effects vary by task and model family. Although q3 and q4 provide substantial energy savings, extreme compression can reduce performance on complex reasoning tasks.
- Quantization significantly reduces energy consumption and inference latency for edge-deployed LLMs.
- Accuracy effects vary across tasks and model families.
- q3 and q4 quantization achieve substantial energy savings.
- Extreme compression can degrade performance on complex reasoning tasks.
- Quantization strategies should be selected according to task requirements and deployment constraints.