Source-linked AI summary
A Measurement Study of LLM Inference Trade-offs Across Edge Continuum Hardware
Maysam Khatib, Moysis Symeonides, Demetris Trihinas, George Pallis, Marios D. Dikaiakos
TL;DR
LLM deployment across the edge continuum requires balancing quality, latency, footprint, and energy. This paper conducts controlled self-hosted measurements across Jetson Orin and near-edge CPU/GPU modes, with GPT-4o as a cloud reference. GPU-enabled server execution has the lowest compute-side latency, while Jetson Orin has lower measured energy, and delivery overheads can change latency-sensitive placement choices.
Problem
Edge-continuum deployment requires selecting models and execution locations under competing accuracy, latency, footprint, and energy objectives.
Method
The study uses a containerized benchmarking pipeline to evaluate open-weight models and quantization variants across Jetson Orin and near-edge CPU/GPU modes, with GPT-4o as a cloud reference.
Results
GPU-enabled server execution provides the lowest compute-side latency, while Jetson Orin shows lower measured energy; model size alone does not reliably predict accuracy or latency.
Takeaways & Limitations
Deployment decisions should consider accuracy, latency, footprint, energy, and possible streamed-token delivery overheads rather than compute-side metrics alone.
Takeaways & Limitations
The study uses sequential MMLU multiple-choice question answering and does not evaluate concurrent serving, long-context dialogue, retrieval, tools, multimodality, or long-form generation.
Abstract
from arXiv · showhide
Large language models (LLMs) are increasingly used as backends for intelligent web services, but serving them across the edge continuum requires balancing quality, latency, model footprint, and energy. This paper presents a controlled measurement study of self-hosted LLM inference across edge and near-edge deployment nodes: an NVIDIA Jetson AGX Orin and a near-edge server with CPU-only and GPU-enabled inference modes. We evaluate multiple open-weight LLMs and quantization variants using a fixed question-answering workload, and compare them against GPT-4o as a cloud-hosted accuracy and latency reference. Our benchmarking pipeline reports accuracy, model footprint, per-token decoding latency, prefill latency, and overall execution energy. The results show that GPU-enabled server execution provides the lowest compute-side latency, while Jetson Orin shows lower measured energy, consistent with its lower platform power under our setup. CPU-only execution is consistently dominated in latency for our workload and shows higher measured energy. We also show that parameter count and downloaded weight-file size alone do not reliably predict observed accuracy or latency. Finally, using Pareto-frontier analysis, we study how deployment decisions may change under possible streamed-token delivery overheads, highlighting that compute-side inference metrics alone can lead to suboptimal placement for latency-sensitive interactive web services.
1 Introduction
Edge-continuum LLM deployment requires joint decisions about model, execution location, and acceptable trade-offs across accuracy, latency, footprint, and energy. The paper introduces a controlled benchmark and shows that deployment choices are multi-objective, with streaming overheads potentially changing preferred placements.
- Edge deployment requires choosing which model to run, where to run it, and which trade-offs are acceptable for each application and query.
- Larger models may improve accuracy but increase footprint and reduce responsiveness, while quantization can reduce memory needs yet alter quality and serving overheads.
- Existing benchmarks and serving systems do not determine where an LLM-backed intelligent web service should execute across the edge continuum.
- The benchmarking pipeline records accuracy, model footprint, prefill and per-token decoding latency, and overall execution energy across controlled edge and near-edge deployments.
- No single off-the-shelf model is best across accuracy, latency, model footprint, and measured energy.
- GPU servers provide the lowest per-token latency, whereas Jetson Orin shows lower measured energy and streaming-overhead sensitivity makes on-device GPU inference more competitive for delay-sensitive settings.
2 Reference Benchmarking Architecture
The benchmarking architecture converts configurations into repeatable containerized experiments that run fixed workloads across models and devices. It collects accuracy, latency, and energy metrics while distinguishing compute-side inference from client-observed streamed-token latency.
- A configuration-driven pipeline orchestrates repeatable experiments from model, dataset, and target-device selection through unified result export.
- The workload generator streams normalized prompts to the serving API, records responses and timestamps, computes accuracy, and collects device-specific power measurements.
- The study compares Jetson AGX Orin, near-edge CPU and GPU modes, and GPT-4o as a cloud accuracy reference, while cloud executions lack comparable device-level metrics.
- MMLU supplies a controlled multiple-choice workload spanning 57 academic subjects, with accuracy aggregated over 1,531 questions using fixed zero-shot prompting and parsing.
- Energy per trial is computed as E = ¯P × T, where ¯P is average execution power and T is workload runtime.
- Streamed-token latency is modeled as Lstream = Lcompute + ddelivery, adding an effective per-token delivery overhead to measured compute-side decoding latency.
- Server energy is a partial lower-bound estimate whereas Orin energy is a full-system measurement, so absolute cross-platform comparisons are indicative rather than calibrated.
3 Observations
The observations compare model, platform, latency, energy, and Pareto trade-offs across controlled edge and near-edge inference configurations. GPU servers generally minimize latency, Orin minimizes measured energy in this setup, and delivery overhead can change deployment choices.
- GPT-4o reaches approximately 80% accuracy, while larger self-hosted models such as Mistral 7B improve accuracy at the cost of larger parameter counts and slower responses.
- Parameter count and downloaded weight-file size alone do not reliably predict observed accuracy or latency across model families, quantization levels, and serving backends.
- Server GPU provides the lowest per-token latency across models, Orin GPU is slower but below Server CPU, and CPU execution can substantially increase token response time.
- Server GPU has the lowest prefill latency, Orin is mostly around or below 1 ms per input token, and larger CPU-served models reach roughly 17–22 ms per input token.
- Orin shows the lowest measured energy per trial among evaluated self-hosted setups, while CPU-only execution has the highest values for most models.
- All CPU-only trials are dominated in the evaluated accuracy–latency space, whereas almost all Pareto-front points come from the GPU-enabled server and one Orin point.
- Adding modeled 30 ms and 60 ms per-token delivery overhead shifts server-GPU points rightward, making local Orin increasingly relevant while preserving server-GPU options for higher accuracy.
4 Related Work
Prior work standardizes inference measurement, improves model compression, and optimizes serving efficiency, but this study treats quantization as an end-to-end deployment factor rather than proposing a new quantizer or serving system.
- Benchmarking: MLPerf Inference and MLPerf Power standardize performance, accuracy, energy, and power reporting across hardware and software stacks.These efforts use common workloads, accuracy checks, and prescribed measurement procedures to improve comparability.
- Compression and quantization: SmoothQuant reduces quantization difficulty by migrating activation quantization burden to weights through an offline transformation with minimal accuracy loss.
- Compression and quantization: AWQ protects salient weight channels using activation statistics and pairs low-bit weight-only quantization with a deployment-oriented runtime for on-device inference.
- Study positioning: Unlike these methods, this study evaluates quantization as a trial factor in an end-to-end pipeline and measures its effects on deployment metrics.
- Serving systems: Orca, vLLM, and Sarathi-Serve improve serving efficiency through iteration-level scheduling, PagedAttention, and chunked prefills, respectively.
5 Conclusion, Limitations & Future Work
The study finds that edge deployment is inherently multi-objective: GPU servers minimize compute-side latency, Jetson Orin uses less measured energy, and CPU-only execution is dominated for the studied workload. Its conclusions are bounded by sequential MMLU-based evaluation, platform-specific energy telemetry, and controlled delivery-delay injection.
- Conclusion: GPU-enabled servers provide the lowest compute-side latency, Jetson Orin shows lower measured energy, and CPU-only execution is dominated in accuracy-latency space.
- Conclusion: Parameter count alone does not reliably predict deployment performance, so empirical evaluation across architectures and quantization settings remains necessary.
- Conclusion: Pareto-frontier choices depend on whether latency is measured at the compute side or at the client-observed streamed-token level.
- Limitations & Future Work: The study does not evaluate concurrent streams, batching, queueing, or tail latency, so its rankings should not be generalized directly to high-throughput multi-tenant serving.
- Limitations & Future Work: Its short MMLU multiple-choice workload does not support generalization to long-form generation, while platform-specific energy telemetry limits fully calibrated cross-platform energy comparisons.