PaperlayerView exact revision on arXiv

Source-linked AI summary

Triple-Bottom-Line Sustainability of Language Models for Edge AI: A Comparison Between SLMs and Quantized LLMs

Jainil Dharmil Shah

arXiv:2609.00665v1cs.AI

TL;DR

Edge-AI model selection must balance capability, deployment burden, energy, and safety, but comparisons often emphasize isolated metrics. The paper evaluates native SLMs and quantized LLMs with a relative, three-pillar HSS. Qwen3-30B-A3B/GGUF Q4 ranks first in the global LLM pool, while the study cautions that quantization outcomes depend on the joint configuration and that HSS is pool-relative.

  • Problem

    Edge-AI model selection lacks a unified comparison of capability, economic deployability, operational environmental cost, and behavioral safety.

    Source: §1 · p. 2

  • Method

    The study compares five BF16 SLMs with 25 LLM configurations using capability, systems efficiency, operational energy, and a five-prompt refusal-based safety proxy in a three-pillar HSS.

    Source: §1 · p. 2, §3.1 · p. 3, §3.2 · p. 3

  • Results

    Qwen3-30B-A3B/GGUF Q4 ranks first in the global LLM pool (93.38), and GGUF Q4 raises mean throughput from 22.74 to 95.50 tokens/s while reducing mean energy from 12.10 to 5.28 J/token.

    Source: §4 · pp. 4–5

  • Takeaways & Limitations

    Quantized LLMs can rank ahead in the global LLM pool, but quantization is a joint model-and-backend configuration choice rather than a guaranteed monotonic precision-efficiency trade-off.

    Source: §1 · p. 1, §4 · pp. 4–5

    HSS is relative to its comparison pool, and its social pillar relies on a five-prompt lexical refusal proxy that cannot represent the full safety distribution.

    Source: §3.2 · p. 4, §4 · p. 6

Abstract

from arXiv · show

Edge-AI model selection is commonly driven by one isolated metric - accuracy, latency, memory, energy, or safety, even though a deployable language model must balance all five. Our work focuses on answering the question whether na- tively trained small language models (SLMs) or large language models (LLMs) compressed through post-training quantization offer the more sustainable edge- deployment trade-off. We introduce a reproducible Holistic Sustainability Score (HSS) organized around the triple bottom line: an economic pillar for capability and systems efficiency, an environmental pillar for operational GPU energy and a social pillar for harmful-prompt robustness. Five BF16 SLMs and five LLMs under different quantization approaches - BF16, INT8, NF4 4-bit, GPTQ 4-bit, and GGUF Q4 produce 30 measured configurations. Capability is assessed on five zero-shot benchmarks; efficiency uses latency, throughput, peak VRAM and energy; and safety is approximated by attack success rate on five harmful prompts. Qwen3-30B-A3B/GGUF Q4 ranks first in the combined pool (93.38), followed by Mistral-Small-24B/GGUF Q4 (92.40), while Phi-4-mini/BF16 is the highest- ranked SLM in that pool (89.49). Thus, the hypothesis that native SLMs must be the most sustainable edge choice is not supported universally; optimized quantized LLMs can win overall, while SLMs remain competitive through lower resource demand. Quantization is a systems-level choice rather than a monotonic precision- efficiency trade-off and HSS remains relative to its comparison pool and proxy definitions.

1 INTRODUCTION

Edge-AI model selection needs to balance capability, deployment cost, and safety under fixed hardware, power, and response-time budgets. This study compares native SLMs with quantized LLMs using a relative, three-pillar sustainability score.

1 INTRODUCTION

  • Edge budgets make memory, latency, throughput, energy, and unsafe behavior consequential alongside predictive quality.The introduction notes that GPU memory determines whether a model can run locally, while latency and throughput affect usability and operating cost.Source: §1 · p. 1
  • Quantization can reduce model footprint, but its effects on end-to-end latency and energy depend on the algorithm, kernels, checkpoint, architecture, and backend.A smaller bit width alone does not guarantee lower latency or energy.Source: §1 · p. 1
  • Edge-AI model selection lacks a unified framework for comparing capability, deployability, operational environmental cost, and behavioral safety.The paper asks whether native SLMs or post-training-quantized LLMs offer the more sustainable edge-deployment trade-off.Source: §1 · p. 2
  • The study evaluates five BF16 SLMs and a 5 × 5 grid of LLMs and quantization cases, then compares them through three equally weighted HSS pillars.HSS is presented as a transparent, auditable decision aid that preserves raw metrics and makes the comparison population explicit.Source: §1 · p. 2, Figure 1 · p. 2

2 BACKGROUND WORK

Prior work establishes that quantization affects edge behavior and that larger quantized models can outperform smaller baselines on some tasks. The paper frames a triple-bottom-line comparison as an open question.

2 BACKGROUND WORK

  • Prior edge studies show quantization materially changes real deployment behavior, while comparisons of quantized larger models with smaller baselines find task-dependent advantages that can reverse.The cited studies do not combine native SLMs and quantized LLMs in a triple-bottom-line comparison.Source: §2 · p. 2
  • Capability is summarized by the mean of five complementary zero-shot benchmarks, with per-benchmark results retained in appendix data.The benchmarks are MMLU, ARC-Challenge, HellaSwag, GSM8K, and TruthfulQA.Source: §2 · p. 3
  • The LLM set spans dense and mixture-of-experts families around 20–33B total parameters, comparing BF16 baseline, bitsandbytes INT8/NF4, GPTQ 4-bit, and GGUF Q4 K M configurations with different kernels and, for GGUF, a different serving backend.Source: §2 · p. 3
  • HSS combines capability and systems efficiency, operational energy, and safety with equal top-level pillar weights while retaining components for users to inspect.The score is relative to its comparison pool, not an absolute measure of sustainability.Source: §2 · p. 3

3 METHODOLOGY

The methodology evaluates 30 configurations under controlled settings, measures capability, systems efficiency, energy, and a narrow safety proxy, then normalizes metrics for a three-pillar HSS.

3.1 EXPERIMENTAL MATRIX AND REPRODUCIBILITY

  • 30 configurations comprise five BF16 SLMs and 25 LLM cases across five quantization approaches, with zero-shot evaluation and deterministic generation.Capability tasks use a limit of 20 examples per task; the authors characterize the small task limit as increasing statistical uncertainty.Source: §3.1 · p. 3

3.2 MEASURED METRICS

  • Capability averages five benchmark scores, while efficiency records five-prompt latency, throughput, peak VRAM, and operational energy per query and generated token.GPU power is sampled every 50 ms, and operational energy is calculated as average power multiplied by elapsed time.Source: §3.2 · p. 3, Figure 2 · p. 4
  • Safety uses five harmful prompts and lexical refusal cues to calculate proxy ASR as 1−refusal rate, but the classifier and prompt set cannot represent the full safety distribution.The classifier cannot distinguish every safe redirection from nuanced unsafe responses.Source: §3.2 · pp. 3–4

3.3 NORMALIZATION AND HSS

  • Metric values are normalized within a comparison pool, with capability and throughput treated as higher-is-better and latency, VRAM, energy, and ASR as lower-is-better.When all values are equal, the normalized value is set to 0.5; the economic pillar averages capability, latency, throughput, and VRAM, while the environmental pillar averages query and token energy.Source: §3.3 · p. 4

P (ASRi) , (4)

The reported HSS and ranks depend on the comparison pool because changing the pool changes normalized values, even when raw measurements stay the same.

P (ASRi) , (4)

  • The study reports four comparison views: an SLM-only pool, a global LLM pool, five within-quantization LLM pools, and a combined SLM–LLM pool.Raw measurements remain unchanged, but normalized values and HSS can change with the pool; ranks are descending within each pool.Source: P (ASRi) , (4) · p. 4

4 RESULTS & DISCUSSION

Across the named normalization pools, quantized Qwen3 and Mistral lead the LLM and combined rankings, while SLMs remain competitive through lower resource demand. The results also show that model, quantization, and backend choices jointly shape the trade-off, and the combined ranking is exploratory.

4 RESULTS & DISCUSSION

  • In the SLM-only pool, Llama-3.2-3B/BF16 ranks first with HSS 92.42 despite mean capability of 0.556 versus Phi-4-mini’s 0.591.Its 9.47 s latency, 34.41 tokens/s throughput and 2.63 J/token contribute to its overall balance.Source: §4 · p. 4
  • Equal top-level HSS weights prevent the larger number of efficiency metrics from automatically outweighing environmental and social considerations.Min-max normalization is sensitive to outliers, and the social pillar uses a five-prompt lexical refusal proxy rather than a complete safety assessment.Source: Figure 3 · p. 5, §4 · p. 6
  • When the five SLMs enter the global pool, Qwen GGUF and Mistral GGUF remain first and second, showing that small models are competitive on the composite even when larger models lead capability.Source: §4 · p. 5
  • Across five models, GGUF Q4 preserves mean capability (0.524 versus 0.525 for BF16), raises mean throughput from 22.74 to 95.50 tokens/s, and lowers mean peak VRAM from 56.35 to 13.64 GB and mean energy from 12.10 to 5.28 J/token.The result reflects a joint configuration: GGUF uses llama.cpp, while other cases use Hugging Face/Transformers-based execution.Source: §4 · p. 5
  • Mistral’s GGUF case retains capability of 0.647 versus 0.658 for BF16 while improving throughput from 27.71 to 61.13 tokens/s and reducing peak VRAM from 48.57 to 15.88 GB.By contrast, gpt-oss-20B/GGUF reaches 173.86 tokens/s and 1.48 J/token, but its 0.352 capability limits global HSS to 84.11.Source: §4 · p. 5
  • Within fixed quantization pools, gpt-oss-20B wins BF16, INT8 and NF4, Mistral wins GPTQ, and Qwen3 wins GGUF; Gemma GPTQ reaches 958.71 s latency, 0.33 tokens/s, 85.01 GB peak VRAM and 493.02 J/token.The findings reject a simple fewer-bits-is-faster account; kernel support, retained high-precision modules, conversion quality and backend behavior can dominate bit width.Source: §4 · p. 5
  • In the combined pool, Phi-4-mini/BF16 is the highest-ranked SLM at 89.49, while lower VRAM and energy can offset part of the capability gap.The combined ranking is exploratory because the SLM and LLM runs used different platforms and software paths.Source: §4 · pp. 5–6

5 CONCLUSION & FUTURE WORK

The study treats edge sustainability as a joint property of model, precision, backend, and workload, and finds no universal SLM victory. It identifies expanded measurement and evaluation as future work.

5 CONCLUSION & FUTURE WORK

  • The best edge-sustainability balance depends jointly on model, precision, backend, and workload, with Qwen3-30B-A3B/GGUF Q4 leading the 25- and 30-case pools.The study also finds that the preferred model changes with the deployment choice and that SLMs remain competitive in the combined ranking.Source: §5 · p. 6
  • INT8, NF4, GPTQ, and GGUF cannot be ordered universally because edge benefits depend on architecture and execution stack.HSS is presented as a relative comparison, not a scalar capturing every societal consequence.Source: §5 · p. 6
  • Future work should expand benchmark trials, latency and energy measurements, safety evaluation, and tests of HSS robustness.The proposed additions include confidence intervals, larger adversarial sets with human or validated-classifier judgments, weight sweeps, and uncertainty propagation.Source: §5 · p. 6

A COMPLETE RESULTS

The appendix reports raw measurements for auditability and normalized pillar-score rankings for distinct comparison pools. Because normalization depends on the pool, HSS values across tables require recalculation before comparison.

A COMPLETE RESULTS

  • The appendix pairs raw measurements with normalized economic, environmental, and social pillar rankings for the stated comparison pools.The raw tables define Cap. as the mean of five benchmark scores and ASR as proxy attack success rate; efficiency values come from the corresponding runs.Source: A COMPLETE RESULTS · p. 9
  • HSS values are pool-dependent and should not be compared across tables without recalculation.The raw measurements remain unchanged across the four views, while normalized values and rankings can change with the comparison pool.Source: A COMPLETE RESULTS · p. 9
  • Tables 2 and 3 report raw measurements for the SLM-only and global 25-case LLM comparisons, respectively.Tables 4 and 5 show normalized HSS rankings for those SLM and global LLM pools; Table 6 reports independently normalized within-quantization LLM rankings, and Table 7 covers the globally renormalized combined pool.Source: Table 2 · p. 9, Table 3 · p. 9, Table 4 · p. 9

B ADDITIONAL EXPERIMENTAL DETAILS

The experimental details define zero-shot capability scoring, a fixed-prompt efficiency protocol, and a narrow safety proxy. Reproducibility artifacts and workbooks support reconstruction of the rankings.

B ADDITIONAL EXPERIMENTAL DETAILS

  • Capability scores average five zero-shot benchmark-family scores without additional task weighting.Each task uses 20 examples, and metric selection follows the evaluation harness priority order.Source: B ADDITIONAL EXPERIMENTAL DETAILS · p. 11
  • Efficiency measurements use five fixed prompts, with total latency covering all responses and GPU power and memory sampled every 50 ms.Energy per query divides operational energy by five, while energy per token divides by generated tokens.Source: B ADDITIONAL EXPERIMENTAL DETAILS · p. 11
  • Safety uses five harmful or illegal prompts and fixed-phrase refusal detection, so its social score is a narrow behavioral proxy.Saved per-prompt outputs permit manual audit; the social score uses reverse-normalized ASR.Source: B ADDITIONAL EXPERIMENTAL DETAILS · p. 11
  • LLM case directories include metrics, evaluation outputs, hardware and environment capture, configurations, and quantization audits.Formula-driven workbooks and notebooks reproduce SLM-only, LLM-only, within-quantization, and combined rankings.Source: B ADDITIONAL EXPERIMENTAL DETAILS · p. 11
Loading 2609.00665v1…