PaperlayerView exact revision on arXiv

Source-linked AI summary

Bench360: Benchmarking Local LLM Inference from 360 Degrees

Linus Stuhlmann, Mauricio Fadel Argerich, Jonathan Fürst

arXiv:2511.16682v2cs.CLcs.AIcs.LGcs.PF

TL;DR

Local LLM deployments require balancing task quality, system performance, and resource use, while existing benchmarks provide limited guidance for practical deployments. Bench360 evaluates local configurations across tasks, scenarios, engines, and quantization schemes, and its results show that the best choice depends on the workload and serving scenario.

  • Problem

    Choosing a local model, quantization scheme, and inference framework requires balancing computing cost, energy use, and output quality for a specific task and deployment scenario; existing benchmarks often lack practical deployment metrics or adaptability.

    Source: §1 · pp. 1–2

  • Method

    Bench360 provides configurable evaluation of local LLM models, engines, quantization schemes, and deployment scenarios, combining task-specific quality with system metrics.

    Source: §1 · pp. 1–2

  • Results

    The best-performing inference engine is scenario-dependent: LMDeploy excels at single-request responsiveness, SGLang and vLLM lead batch throughput on different hardware tiers, and vLLM is most robust and energy-efficient for multi-user serving.

    Source: §5 · p. 8

  • Takeaways & Limitations

    There is no universal best configuration; choices depend on workload-specific trade-offs among quality, latency, and energy use.

    Source: §1 · p. 1, §5 · p. 8

    Experiments focus primarily on GPTQ-based quantization, and broader evaluation is constrained by inference-engine support for alternative methods.

    Source: Limitations · p. 9

Abstract

from arXiv · show

Running LLMs locally has become increasingly common, but users face a complex design space across models, quantization levels, inference engines, and serving scenarios. Existing inference benchmarks are fragmented and focus on isolated goals, offering little guidance for practical deployments. We present Bench360, a framework for evaluating local LLM inference across tasks, usage patterns, and system metrics in one place. Bench360 supports custom tasks, integrates multiple inference engines and quantization formats, and reports both task quality and system behavior (latency, throughput, energy, startup time). We demonstrate it on four NLP tasks across three GPUs and four engines, showing how design choices shape efficiency and output quality. Results confirm that tradeoffs are substantial and configuration choices depend on specific workloads and constraints. There is no universal best option, underscoring the need for comprehensive, deployment-oriented benchmarks.

1 Introduction

Local deployment requires choosing models, quantization, and inference frameworks to balance task quality, system performance, and resource use for a specific workload. Bench360 addresses this with configurable evaluation across local tasks, scenarios, engines, and quantization schemes.

1 Introduction

  • Bench360 evaluates local LLM configurations across task-specific quality and system metrics to help users identify options suited to their requirements.It is designed to replace time-consuming trial and error with standardized evaluation across configurations.Source: §1 · pp. 1–2
  • Existing benchmarks often omit on-premise metrics such as energy, cold-start behavior, and latency, or are tied to particular APIs and evaluation scenarios.Source: §1 · pp. 1–2
  • Bench360 combines system-efficiency measures with freely definable task-quality metrics such as accuracy, F1, and ROUGE.Source: §1 · p. 2
  • Bench360 supports comparisons across model families, quantization schemes, and inference frameworks including TGI, vLLM, SGLang, and LMDeploy.Source: §1 · p. 2
  • Users can add tasks and model configurations through plug-and-play support intended for reproducible experiments and continuous benchmarking.Source: §1 · p. 2

2 Related Work

Existing LLM inference benchmarks cover different combinations of task quality, systems performance, energy, and deployment settings. Bench360 is positioned as a customizable benchmark for local deployments across tasks and scenarios.

2 Related Work

  • LLMeBench (Dalvi et al., 2023) and the Open LLM Leaderboard focus on task quality, while LLMPerf (Contributors, 2024) evaluates APIs rather than local deployments.Source: §2 · p. 2
  • MLPerf covers multiple scenarios and energy measurement, whereas LLM-Inference-Bench (Chitty-Venkata et al., 2024) spans frameworks but uses perplexity rather than task-specific quality metrics.Source: §2 · pp. 2–3
  • Bench360 targets local deployments by combining tasks and scenarios with system metrics and customization of tasks, quantization, and workload scenarios.Source: §2 · p. 3

3 Bench360—Benchmarking Local LLMs from 360°

Bench360 combines modular task, workload, backend, and metrics components to evaluate local LLM configurations. Its task and serving interfaces support varied evaluations, while its metrics collector records task quality and system behavior.

3 Bench360—Benchmarking Local LLMs from 360°

  • Bench360 combines task, workload, backend, and metrics modules, with YAML configuration and modular APIs for adding tasks, engines, or workload generators.Source: §3 · p. 3

3.1 Task Engine

  • The task engine uses a common plugin interface to generate prompts and reference answers from datasets, then score outputs with task-specific quality metrics.Source: §3.1 · p. 3

3.2 Workload Controller

  • Single-stream blocks on each response, batch submits groups of b, and server dispatches independent Poisson-arrival user requests without benchmark-imposed queuing or batching.Source: §3.2 · p. 3

3.3 Backend Abstraction

  • A common backend interface runs containerized TGI, vLLM, SGLang, or LMDeploy instances and translates shared configuration into each engine’s API.Source: §3.3 · p. 4

3.4 System Metrics Collector

  • The collector records token and generation latency, throughput, GPU and host resources, energy efficiency, startup and loading times, and server queue, wait, and end-to-end latency.Source: §3.4 · p. 4

4 Experimental Evaluation

Bench360 evaluates memory-constrained deployment, serving scenarios, and system efficiency across four NLP tasks, model families, inference engines, and GPUs. Its research questions compare model and quantization choices under a 24 GB budget and engine behavior across serving regimes.

4.1 Experimental Setup

  • The evaluation covers MMLU, SQuAD2.0, CNN/DailyMail, and Spider, with task-specific quality metrics, and benchmarks four model families and four engines across L4, A10, and A30 GPUs.The study also compares selected FP16 and GPTQ-quantized variants within a 24 GB VRAM constraint.Source: §4.1 · p. 5

4.2 Memory-Constrained Deployment

  • Across tasks, larger quantized models can improve quality within 24 GB, but trade-offs vary by family; Gemma-2 alone consistently appears on both Pareto frontiers.The evaluation compares the largest same-family FP16 model that fits with the largest quantized variant.Source: §4.2 · pp. 5–6, Figure 2 · p. 6

4.3.1 Single-Stream Latency

  • LMDeploy leads single-stream responsiveness in startup and TPOT, while LMDeploy and TGI have the lowest TTFT across the three GPUs.TGI startup exceeds 260 s on A30, whereas LMDeploy has markedly shorter cold-start delays.Source: §4.3.1 · pp. 6–7, Figure 3 · p. 7

4.3.2 Batch Throughput

  • 964.4 TPS at B=128 on A30 is vLLM’s peak; SGLang delivers the highest throughput across batch sizes on L4 and A10.The batch comparisons use FP16 models, so differences chiefly reflect engine-level scheduling and execution efficiency rather than numerical compression.Source: §4.3.2 · p. 7, Table 4 · p. 7

4.3.3 Server Concurrency

  • vLLM scales to C=64 on A30 and consistently satisfies QoS through C=32 across devices, while L4 and A10 typically saturate between C=16 and C=32.At C=64, vLLM remains usable without severe tail latencies.Source: §4.3.3 · pp. 7–8

4.4 System Efficiency

  • vLLM is the most energy-efficient engine across GPUs, although its average GPU utilization is not always highest.A30 consistently reduces energy per token relative to L4 and A10; the paper notes hardware gains depend on engines exploiting added bandwidth and compute.Source: §4.4 · p. 8, Figure 4 · p. 8

5 Conclusion

Bench360 integrates task-specific quality with system metrics for local LLM evaluation. Results show substantial configuration trade-offs and no single best setup across deployment needs.

5 Conclusion

  • Bench360 evaluates local deployments by integrating task-specific quality with system metrics including latency, throughput, and energy consumption.The framework is presented as a holistic, user-focused benchmark for local inference.Source: §5 · p. 8
  • Quantization can enable larger, more accurate models on memory-constrained hardware but often incurs significant latency and energy costs, while the best engine depends on serving scenario.LMDeploy excels at single-request responsiveness; SGLang and vLLM lead batch throughput on different hardware tiers, and vLLM is most robust and energy-efficient for multi-user serving.Source: §5 · p. 8
  • No single setup is best for local inference; the ideal choice requires evaluating the model, quantization, and engine against specific deployment requirements.Bench360 is offered as a tool for navigating this decision space and optimizing deployments for individual needs.Source: §5 · p. 8

Limitations

The study's conclusions are bounded by its quantization, model and hardware, inference-engine, and deployment coverage.

Limitations

  • The evaluation focuses primarily on GPTQ quantization, with broader coverage constrained by inference-engine support.Alternative activation-aware or mixed-precision methods are proposed for future evaluation as ecosystem support matures.Source: Limitations · p. 9
  • Models fit within a 24 GB VRAM budget on mid-tier GPUs, limiting conclusions about very large models and high-end accelerators.The authors chose this setting to reflect local environments in smaller labs, organizations, and individual settings.Source: Limitations · p. 9
  • The comparison covers four engines—vLLM, SGLang, TGI, and LMDeploy—while specialized formats or hardware-specific optimizations hinder controlled comparison of other runtimes.The modular design can integrate additional engines when standardized interfaces or comparable execution paths are available.Source: Limitations · p. 9
  • The study evaluates single-GPU deployments and does not explore multi-GPU inference or CPU offloading.Both are supported by the Bench360 architecture and identified as directions for future investigation.Source: Limitations · p. 9

Ethical considerations

The authors discuss local inference's privacy benefits and dual-use risks, energy costs, and the framework's configuration workflow.

Ethical considerations

  • The authors report no ethical issues or conflicts with the ACL Code of Ethics and describe transparent local-inference trade-offs as supporting broader availability.Source: Ethical considerations · p. 9
  • Benchmarking may temporarily increase energy use; the authors argue this cost is outweighed by informed production choices and report experiments used 100% renewable energy.They connect more efficient deployment choices with reduced energy consumption and associated CO2 emissions.Source: Ethical considerations · p. 9
  • Keeping inference on-premise can enhance privacy and data sovereignty, but local deployments may lack centralized services' real-time safety filters and content moderation.Bench360 does not enforce safety guardrails; the authors say its transparent, auditable evaluations can help practitioners assess efficiency and safety trade-offs.Source: Ethical considerations · p. 9

A Bench360 Framework Details

B System Performance Metrics

Bench360 organizes system performance measures across latency, throughput, memory, and GPU utilization, using average and peak measurements for resource usage.

B.1 Latency

  • Latency is measured as TTFT, the delay from request issuance to first-token receipt; TPOT, generation time divided by output-token count; and GL, TPOT multiplied by output-token count.GL recovers the full generation time.Source: §B.1 · p. 13

B.2 Throughput

  • Generation speed is tracked in tokens per second and sentences per second, using generated-token or sentence counts and generation time.The passage defines Nt as total tokens generated, Ns as total sentences generated, and Tgen as generation time in seconds.Source: §B.2 · p. 13

B.3 GPU Memory Usage

  • GPU memory consumption is captured as both average and peak usage over the measurement period.mi denotes memory usage in MB at sample i, and n is the total number of samples.Source: §B.3 · p. 13

B.4 GPU Utilization

  • GPU utilization is captured as both average and peak over the measurement period.ui denotes GPU utilization percentage at sample i, and n is the total number of samples.Source: §B.4 · p. 13

C System Cost & Resource Metrics

The section formalizes system cost and resource metrics, covering power and energy as well as runtime GPU-memory overhead beyond model file size.

C System Cost & Resource Metrics

C.1 Power & Energy

  • The power and energy metrics define average and peak power and derive energy consumption from power readings and generation time.The definitions identify the number of samples, generation time, tokens generated, and sentences generated.Source: §C.1 · pp. 13–14

C.2 Model Size & Overhead

  • GPU-memory overhead is the additional runtime memory use beyond the model’s static weight-file size.The comparison uses average GPU memory footprint during inference and total weight-file size on disk, both in MB.Source: §C.2 · p. 14

D System Deployment & Infrastructure Metrics

The section formalizes deployment and infrastructure metrics, including cold-start latency for a fresh inference invocation.

D System Deployment & Infrastructure Metrics

D.1 Cold Start Time

  • Cold-start latency measures the total latency of a fresh inference invocation.It combines framework initialization time, model-weight loading time, and time-to-first-token latency.Source: §D.1 · p. 14

E Additional Experimental Details and Results

The experimental details fix generation and prompting conditions and document the hardware and inference frameworks used. They also describe server-side latency measurements across GPU concurrency levels against QoS thresholds.

E.1 Decoding Parameters, Prompts and Reproducibility Details

  • Generation uses temperature 0.1 and top-p 0.9, with a 64-token limit except for summarization, which uses 128 tokens.Source: §E.1 · p. 14
  • All tasks use few-shot prompts with fixed, task-specific demonstrations shared across models, engines, hardware configurations, and runs.Source: §E.1 · p. 14
  • Reproducibility uses a fixed global seed, and evaluation samples are drawn uniformly without replacement.The seed is 42 for Python random, NumPy, and PyTorch; sample counts are controlled by benchmark configuration.Source: §E.1 · p. 14

E.2 Hardware Setup and Inference Frameworks

  • Experiments ran on three KVMs with Ubuntu Linux and compare three GPUs and four local inference engines.The referenced tables describe host specifications, GPU characteristics, and the engines used.Source: §E.2 · p. 14, Table 6 · p. 15, Table 7 · p. 15
  • Figure 5 plots E2E latency and TTFT against concurrency, with QoS thresholds of 6 seconds and 2 seconds, respectively.Source: Figure 5 · p. 16
Loading 2511.16682v2…