Source-linked AI summary
Small Language Models: Survey, Measurements, and Insights
Zhenyan Lu, Xiang Li, Dongqi Cai, Rongjie Yi, Fangming Liu, Xiwen Zhang, Nicholas D. Lane, Mengwei Xu
TL;DR
SLMs are widely deployed on personal devices but remain less studied than datacenter-oriented LLMs, leaving their capabilities and runtime costs insufficiently characterized. The paper surveys and benchmarks decoder-only transformer SLMs across technical design, task capabilities, and on-device efficiency, finding broad performance progress while identifying persistent benchmarking and scaling challenges.
Problem
SLMs are increasingly used on devices, yet research comprehensively examining their capabilities, innovations, and runtime costs remains limited.
Method
The paper surveys 100M–5B-parameter decoder-only transformer SLMs across architectures, training methods, capabilities, and on-device runtime costs.
Results
SLM performance improved across commonsense reasoning, problem-solving, and mathematics from 2022 to 2024, with gains of 10.4%, 13.5%, and 13.5%, respectively.
Takeaways & Limitations
The findings provide cross-task and runtime insights for future SLM research, while showing that smaller models and open-data models can be competitive in selected settings.
Takeaways & Limitations
Fair comparison remains constrained by benchmark overfitting, closed training datasets, and the absence of comprehensive benchmarks for device-specific tasks.
Abstract
from arXiv · showhide
Small language models (SLMs), despite their widespread adoption in modern smart devices, have received significantly less academic attention compared to their large language model (LLM) counterparts, which are predominantly deployed in data centers and cloud environments. While researchers continue to improve the capabilities of LLMs in the pursuit of artificial general intelligence, SLM research aims to make machine intelligence more accessible, affordable, and efficient for everyday tasks. Focusing on transformer-based, decoder-only language models with 100M-5B parameters, we survey 70 state-of-the-art open-source SLMs, analyzing their technical innovations across three axes: architectures, training datasets, and training algorithms. In addition, we evaluate their capabilities in various domains, including commonsense reasoning, mathematics, in-context learning, and long context. To gain further insight into their on-device runtime costs, we benchmark their inference latency and memory footprints. Through in-depth analysis of our benchmarking data, we offer valuable insights to advance research in this field.
1 Introduction
SLMs represent a parallel path to large, datacenter-based models, emphasizing accessible and affordable intelligence on personal devices. This survey addresses limited academic coverage by reviewing, benchmarking, and analyzing open-source SLMs across capabilities, costs, and research directions.
- SLMs target resource-efficient deployment on desktops, smartphones, wearables, and other personal devices, while LLMs primarily pursue increasingly difficult tasks in datacenters.The paper frames these as diverging paths: ubiquitous, practical intelligence versus continued scaling toward AGI.
- Academic literature has comparatively limited coverage of SLM capabilities and device runtime costs despite their widespread integration into commercial devices.Examples include smartphone services and on-device foundation models integrated into mobile operating systems.
- The survey covers decoder-only transformer SLMs with 100M–5B parameters, spanning devices from low-end wearables to high-end smartphones and tablets.It benchmarks commonsense reasoning, mathematics, in-context learning, long-context retrieval, latency, memory, and energy consumption.
- The study asks how SLMs evolve, which datasets and training strategies support capability, and how architecture and deployment environments affect runtime performance.The questions target algorithm, model, system, and hardware research perspectives.
- The authors review recent SLM innovations, benchmark capabilities and on-device costs, and derive insights intended to benefit future SLM research.The paper also makes its results and benchmark tools public to facilitate research.
2 SLM Architecture, Datasets, and Training
The survey analyzes SLM architectures, datasets, and training strategies, finding recurring architectural trends, growing dataset diversification, extensive over-training, and strong evidence that data quality matters for capability.
- Scope and selection: SLMs are selected as open-weight, decoder-only transformer models, with 70 models spanning varied architectures, parameter sizes, and training datasets.The survey excludes variants such as RWKV and Mamba and sets 5B parameters as its upper limit.
- Architecture trends: MHA is gradually replaced by GQA, Standard FFN by Gated FFN, ReLU by SiLU, and LayerNorm by RMSNorm from 2022 to 2024.These trends describe changing configuration preferences rather than validated superiority.
- Architecture trends: A typical 2024 SLM uses GQA, gated FFN with SiLU, an FFN ratio of 2 to 8, RMS normalization, and vocabulary larger than 50K.The survey reports that these choices are mostly empirical and lack strict public validation of capacity superiority.
- Training datasets and scale: Pre-training dataset use has diversified: The Pile dominated earlier usage, while RefinedWeb and RedPajama became more widely used recently.The survey attributes this shift to active efforts to construct higher-quality datasets.
- Training datasets and scale: Recent SLMs typically use more than 1.5T training tokens, often exceeding the Chinchilla parameter-token ratio regardless of model size.Some smaller models use more training data than larger ones, such as Qwen2-0.5B at 12T tokens versus Qwen2-1.5B at 7T.
- Training datasets and scale: Training-token count correlates positively with accuracy below 700B tokens, but the correlation weakens beyond 1T tokens because data quality often outweighs additional quantity.The survey identifies model-based filtering in DCLM and FineWeb-Edu as a common feature of high-performing open datasets.
3 SLM Capabilities
SLMs show substantial progress across reasoning, in-context learning, and long-context tasks, with performance generally improving as models grow larger. However, gains vary by task, training data, and model family.
- Overall capabilities: 10.4%, 13.5%, and 13.5% performance improvements occurred from 2022 to 2024 in commonsense reasoning, problem-solving, and mathematics, respectively.The LLaMA series improved by 7.5% on average over the same period.
- Overall capabilities: Open-source-trained SLMs are narrowing the gap in commonsense reasoning but remain substantially behind closed-source-trained models on complex reasoning and mathematics.SmolLM and DCLM-1B achieve 64.2% and 63.8% in commonsense reasoning, respectively.
- Overall capabilities: Phi models achieve state-of-the-art results across most tasks, while Qwen 2 1.5B shows that smaller models can excel on specific tasks.Phi-3-mini reportedly rivals LLaMA 3.1 8B, although its advantage may reflect data engineering, instruction tuning, or dataset overfitting.
- In-context learning: 2.1% average improvement results from five-shot in-context learning across eight tasks, although HellaSwag and PIQA show little benefit.Gemma 2 improves by 4.8%, while LaMini declines by over 2%.
- Long-context capabilities: 91.71% average accuracy is achieved by Qwen2.5-3B in long-context retrieval, compared with 22.13% for Qwen1.5-0.5B and 57.81% for Llama3.2-3B.Smaller models degrade as contexts lengthen, especially for middle-position insertions.
4 SLM Runtime Cost
The runtime-cost evaluation measures latency and memory for SLMs across edge devices using standardized inference procedures. It separates prompt processing from autoregressive generation and controls key experimental variables.
- Evaluation setup: 20 models are evaluated on Jetson Orin and smartphone edge devices, using GPU execution on Jetson and CPU execution on smartphones.All experiments use llama.cpp to reduce inference-engine implementation effects.
- Evaluation setup: Prefill latency measures first-token time after parallel prompt processing, while decode latency measures per-token time during autoregressive generation.The standard setup uses prompt and generation lengths of 50 tokens, with 4-bit quantization applied to larger models except in § 4.2.1.
- Evaluation setup: Runtime memory, model parameters, and prefill and decode latency are the primary recorded metrics.Parameter counts are taken from llama.cpp rather than inconsistent official reports.
4.1 Overview
Runtime latency generally increases with model size, but architecture also matters, especially during prefill. Memory usage depends on both model size and context-related allocation.
- Latency and memory overview: 0.1–1B, 1–2B, and 2–3B models form latency intervals whose costs increase with model size.Models within each interval have relatively similar inference latency.
- Latency and memory overview: Decode latency follows a closer-to-linear relationship with model size than prefill latency.Among 2–3B models, Gemma-2B, Phi-2, and OpenELM-3B show latency positively correlated with model size.
- Latency and memory overview: 275MB to 2456MB is the measured memory-footprint range for 0.1B–3B models.All models use a standardized maximum context length of 2048 because longer-context support increases KV-cache and compute-buffer allocation.
- Latency and memory overview: 31.9% faster execution is observed for Qwen1.5-0.5B than Qwen2-0.5B on Jetson Orin NX 16GB despite 25.4% more parameters.The analysis attributes runtime differences to architecture factors including layers, FFN width, vocabulary size, and parameter sharing.
4.2 Impact of Quantization and Hardware
Quantization can reduce latency, particularly during decode, while hardware strongly affects the relative cost of prompt processing and generation. GPU advantages are greatest for the parallelizable prefill phase.
- Impact of quantization: Up to 75% decode-latency reduction and at least 17% reduction are achieved through quantization, with Q4 K M performing best.For short prompts, quantization reduces prefill latency by at least 25%, while Q4 K M averages a 50% reduction.
- Impact of quantization: Quantization benefits decode more than prefill because decode is bandwidth-bound whereas prefill is compute-bound on mobile devices.Lower precision reduces memory-access overhead and improves cache utilization.
- Impact of hardware: 10 to 20 times faster prefill is observed on Jetson Orin NX than on the Meizu 18 Pro for Bloom-1B1.Both devices show first-token latency increasing linearly with prompt length, with the Jetson advantage growing for longer prompts.
- Impact of hardware: The Xiaomi 12S has the lowest prefill and decode latency among three tested smartphones for Qwen1.5-1.8B.The Pixel 7 Pro follows with competitive performance, and latency increases linearly with token count.
- Impact of hardware: GPU advantage is greatest during prefill because prompt tokens can be processed in parallel, unlike sequential decode generation.This makes the prefill phase more suitable for GPUs with greater parallel computing capacity.
4.3 Latency and Memory Breakdown
Inference time is dominated by matrix-vector multiplication, while long contexts shift memory usage toward the compute buffer and KV cache. Model architecture, especially FFN width, changes layer-level latency contributions.
- Latency Breakdown: Qwen2-0.5B spends more prefill time in its FFN than its Attention layer because its FFN is wider than Qwen1.5-0.5B.During decode, Qwen2 still has the longest time in FFN, whereas Qwen1.5 shows an increased Attention share associated with KV-cache growth.
- Latency Breakdown: Over 80% of inference time comes from the mul mat vec q operator across both models and execution phases.The operator represents matrix-vector multiplication and occupies a higher proportion in Qwen2-0.5B than Qwen1.5-0.5B.
- Memory Breakdown: Runtime memory is primarily consumed by model parameters, the KV cache, and intermediate computation variables.Larger vocabularies increase the compute buffer because the output layer requires a hidden-size-by-vocabulary-size matrix.
- Memory Breakdown: At context length 131072, Qwen2's Compute Buffer and KV Cache occupy 83%–87% of total memory.For Qwen1.5 at its 32768-token maximum context, these components occupy 85%–90%.
- Latency Breakdown: More than 70% of end-to-end SLM inference time is attributed to mul mat vec operations.This summarizes the latency breakdown across the examined SLM inference workloads.
4.4 Optimizations for On-device Deployment
On-device SLM optimization spans runtime techniques such as hardware-aware execution, memory management, quantization, and model collaboration, alongside offline improvements to data, architecture, and distillation. Reported systems reduce latency, memory, or task costs through specialized hardware and model-level strategies.
- Optimization Categories: Online optimization targets runtime performance through hardware-aware execution, model collaboration, and quantization.Offline optimization instead targets datasets, architectures, and knowledge distillation before deployment.
- Online Optimization: EdgeNN reports 3.97×, 3.12×, and 8.80× speedups for inference on integrated-device, mobile-phone, and edge CPU hardware, respectively.Its gains come from unified-memory use and CPU-GPU task scheduling.
- Online Optimization: Transformer-Lite achieves up to 5.7x faster inference than CPU-based methods while maintaining model accuracy.It combines pruning, quantization, layer fusion, and memory/computation optimization for mobile GPUs.
- Online Optimization: LLMS uses fine-grained memory management, KV-cache compression, and swapping to reduce context-switching overhead under tight device-memory constraints.It achieves up to 100x faster context-switching than existing solutions.
- Online Optimization: EdgeMoE stores non-expert weights in device memory and fetches activated expert weights from external storage to reduce MoE memory demands.It further applies expert-wise bitwidth adaptation and expert-management techniques.
- Online Optimization: LLMCad combines a memory-resident small model for token generation with a larger model for validation, achieving up to 9.3x faster performance.This model-collaboration strategy is evaluated on IoT devices and smartphones.
- Offline Optimization: AS-ES learning improves math-word-problem accuracy by up to 15.28% while reducing BLEU scores.It segments chain-of-thought data into abstractive and extractive segments, trading memorization-oriented scores for reasoning performance.
- Offline Optimization: OnceNAS achieves a 10.49× size reduction and 5.45× speedup by jointly optimizing parameter count, latency, and accuracy.It uses continuous latent-space architecture representation for resource-constrained edge inference.
5 Conclusions and Future Directions
The paper concludes that SLM progress depends on jointly designing architectures, data, scaling strategies, and deployment systems for constrained devices. It identifies unresolved challenges in synthetic-data curation, deployment-aware scaling, personalization, device-cloud collaboration, fair benchmarking, and sparse models.
- Co-design and Co-optimization: Architecture and processor co-design is proposed to improve the accuracy-speed tradeoff under fixed parameter budgets.Relevant choices include depth-width ratio, attention type, activation, and quantization for integer-optimized processors.
- Synthetic Data: The paper calls for standardized synthetic-data curation covering deduplication, filtering, mixing, and evaluation.DCLM and FineWeb-Edu are cited as high-performing datasets produced with model-based quality filtering.
- Deployment-aware Scaling: The optimal SLM data-scaling strategy remains open because deployment memory, compute, lifecycle, and economic benefits constrain decisions beyond training FLOPs.The question becomes more complicated when sparsity, such as MoE, is included.
- Continual On-device Learning: On-device personalization faces a tradeoff between higher prompt-time costs from retrieval and substantial memory and energy demands from local fine-tuning.The paper identifies zeroth-order optimization as one possible direction for reducing activation-memory requirements.
- Device-cloud Collaboration: Device-cloud collaboration is constrained by the capability of the SLM and by deciding which tasks are easy enough to execute locally.The proposed division uses the SLM for confident easy tasks and a cloud LLM as a safeguard for difficult ones.
- Benchmarking: Fair SLM benchmarking is difficult because benchmark overfitting, closed training data, and device-specific tasks limit existing comparisons.Smartphone tasks such as private auto-reply and GUI-context understanding are underrepresented in common LLM benchmarks.
- Sparse SLMs: Sparse SLM research remains limited because sparsity benefits may be smaller and MoE can increase memory usage on constrained devices.External storage for cold weights introduces additional I/O-latency and heterogeneous-accelerator compatibility challenges.