PaperlayerView exact revision on arXiv

Source-linked AI summary

A Comprehensive Survey of Small Language Models in the Era of Large Language Models: Techniques, Enhancements, Applications, Collaboration with LLMs, and Trustworthiness

Fali Wang, Zhiwei Zhang, Xianren Zhang, Zongyu Wu, Tzuhao Mo, Qiuhao Lu, Wanjing Wang, Rui Li, Junjie Xu, Xianfeng Tang, Qi He, Yao Ma, Ming Huang, Suhang Wang

arXiv:2411.03350v2cs.CLcs.AIcs.LG

TL;DR

Large language models’ scale creates deployment, privacy, latency, cost, and specialization challenges, while comprehensive coverage of small language models remains lacking. This paper surveys SLM definitions, techniques, applications, models, collaboration with LLMs, and trustworthiness, proposing a task- and resource-based definition. It presents a taxonomy spanning enhancement and deployment methods, model collections, LLM–SLM synergies, and trustworthiness evaluation.

  • Problem

    LLMs’ computational demands hinder practical deployment, while existing SLM studies lack a comprehensive survey and consensus definition.

  • Method

    The paper conducts a comprehensive survey of SLMs, proposes a task- and resource-based definition, and organizes research into taxonomies and frameworks.

  • Results

    The survey covers SLM enhancement techniques, applications and deployment, models below 7 billion parameters, LLM–SLM collaboration, and trustworthiness evaluation.

  • Takeaways & Limitations

    SLMs are presented as practical alternatives for resource-constrained and specialized applications, while collaboration can combine local SLMs with cloud-based LLMs.

  • Takeaways & Limitations

    SLMs still face credibility risks including adversarial attacks, hallucinations, and privacy breaches.

Abstract

from arXiv · show

Large language models (LLMs) have demonstrated emergent abilities in text generation, question answering, and reasoning, facilitating various tasks and domains. Despite their proficiency in various tasks, LLMs like PaLM 540B and Llama-3.1 405B face limitations due to large parameter sizes and computational demands, often requiring cloud API use which raises privacy concerns, limits real-time applications on edge devices, and increases fine-tuning costs. Additionally, LLMs often underperform in specialized domains such as healthcare and law due to insufficient domain-specific knowledge, necessitating specialized models. Therefore, Small Language Models (SLMs) are increasingly favored for their low inference latency, cost-effectiveness, efficient development, and easy customization and adaptability. These models are particularly well-suited for resource-limited environments and domain knowledge acquisition, addressing LLMs' challenges and proving ideal for applications that require localized data handling for privacy, minimal inference latency for efficiency, and domain knowledge acquisition through lightweight fine-tuning. The rising demand for SLMs has spurred extensive research and development. However, a comprehensive survey investigating issues related to the definition, acquisition, application, enhancement, and reliability of SLM remains lacking, prompting us to conduct a detailed survey on these topics. The definition of SLMs varies widely, thus to standardize, we propose defining SLMs by their capability to perform specialized tasks and suitability for resource-constrained settings, setting boundaries based on the minimal size for emergent abilities and the maximum size sustainable under resource constraints. For other aspects, we provide a taxonomy of relevant models/methods and develop general frameworks for each category to enhance and utilize SLMs effectively.

1 INTRODUCTION

Large language models offer broad capabilities but impose substantial resource, latency, privacy, and specialization costs. The survey positions small language models as efficient, customizable alternatives and proposes a task- and resource-based definition alongside a comprehensive taxonomy.

  • Motivation: Generic LLMs can underperform in specialized domains, while deploying broad capabilities for narrowly defined tasks can waste resources.The introduction specifically identifies healthcare and law as domains where fine-tuning is often needed but computationally expensive.
  • Motivation: SLMs reduce memory, storage, training, and inference demands while supporting edge deployment, privacy, faster responses, and personalization.Local processing can keep patient, business, personal-message, and calendar data on-device while enabling immediate prompt execution.
  • Motivation: SLMs are gaining attention, with higher recent Hugging Face download frequency for smaller models and increasing popularity of SLM releases over time.These trends are reported through Figures 2 and 3, based on download statistics obtained October 7, 2024, and a model-release timeline.
  • Definition: Because prior studies lack consensus on SLM size, the survey defines SLMs by specialized-task capability and resource constraints.The proposed range begins at the minimum size exhibiting emergent abilities for a specialized task and ends at the largest size manageable under limited resources.
  • Contributions: The survey organizes SLM research across enhancement techniques, applications, sub-7B models, LLM–SLM collaboration, and trustworthiness.It covers training, fine-tuning, distillation, quantization, deployment, model comparisons, synergies with LLMs, and evaluation of hallucination and privacy concerns.

2 FOUNDATIONAL CONCEPTS IN BUILDING LANGUAGE MODELS

This section introduces foundational concepts for language models, covering architecture, training, and methods for obtaining SLMs from LLMs. Advanced strategies for improving SLM performance are introduced separately.

  • The section establishes foundational concepts and background knowledge for language models.
  • It covers language-model architecture and the training process.
  • It discusses methods for obtaining SLMs from LLMs.
  • Advanced training strategies for improving SLM performance are introduced in Section 3.

2.1 Architecture of SLMs

SLMs commonly use Transformer architectures, whose self-attention captures long-range dependencies but incurs substantial inference cost. Alternative architectures and attention optimizations address efficiency, while components such as feedforward networks, positional embeddings, and normalization support model functionality under resource constraints.

  • Transformer and alternatives: Transformers use self-attention to manage long-range text dependencies, but this mechanism creates large inference costs.Subquadratic-time architectures including Mamba, Hymba, and xLSTM are introduced to alleviate this issue.
  • Attention mechanisms: Multi-Head Attention uses multiple independent heads to capture diverse information before combining their outputs through a learned projection.Each head attends to different parts of the input sequence using query, key, and value representations.
  • Attention mechanisms: MQA and GQA reduce key-value overhead by sharing key-value heads across attention heads or subgroups, while MLA compresses keys and values into a joint latent vector.MLA is described as reducing KV-cache requirements while maintaining high performance.
  • Attention mechanisms: Flash Attention minimizes memory overhead in attention calculations, enabling more efficient processing of longer sequences.The optimization targets the memory demands of standard attention calculations.
  • Other components: Feedforward networks apply two linear transformations separated by a nonlinear activation, while positional embeddings encode token order and normalization stabilizes training.The survey discusses ReLU, GeLU, SiLU, SwiGLU, RoPE, layer normalization, and RMSNorm as related SLM components.
  • Mamba: Mamba addresses Transformer’s quadratic inference cost by relying on the last hidden state and using dynamic selection to filter irrelevant information.Its selective mechanism improves state-space models whose fixed linear-invariant behavior limits input selection.
  • Mamba: Mamba-1.4B achieves 32.8% accuracy on Arc-Challenge, compared with 28.5% for Pythia-1.4B and 29.4% for RWKV-1.5B.Mamba 2 links state-space models with attention through structured semi-separable matrices and achieves 2-8x faster speeds while remaining competitive.
  • Hymba: Hymba 1.5B surpasses Llama-3.2-3B by 1.32% average accuracy, using 11.67× smaller cache size and delivering 3.49× higher processing speed.These results are reported for commonsense reasoning tasks.

2.2 Training SLMs from Scratch

Training SLMs from scratch combines pre-training, task-specific fine-tuning, and decoding strategies. The survey describes data preparation, parameter-efficient adaptation, and token-selection methods used to improve compact models.

  • Training pipeline: SLM training comprises pre-training for general knowledge, fine-tuning for specific tasks, and decoding strategies for iterative next-token selection.These stages cover acquisition, specialization, and generation.
  • Pre-training: Encoder-based models use masked language modeling, whereas decoder-based models use next-token prediction to learn language representations or token distributions.Masked modeling predicts hidden tokens from surrounding context; next-token prediction models the following token.
  • Pre-training: Data cleaning, filtering, deduplication, noise reduction, and subword tokenization improve data quality, generalization, and vocabulary coverage without increasing model size.BPE and WordPiece split text into subwords, helping manage rare and compound words.
  • Fine-tuning: Parameter-efficient fine-tuning keeps core weights largely fixed while LoRA, prefix-tuning, and adapters learn task-specific adjustments.LoRA updates weights with low-rank matrices, while prefix-tuning adds learnable prefixes and adapters insert trainable layers.
  • Decoding strategies: Greedy search selects the highest-probability token, while beam search considers multiple sequences and top-k sampling increases generation diversity.These decoding choices trade off likelihood, exploration, and creativity.

2.3 Obtain SLMs from LLMs

The survey presents pruning, knowledge distillation, and quantization as primary ways to derive SLMs from LLMs. These methods reduce size, memory, or computation while aiming to retain capabilities and performance.

  • Overview: Pruning, knowledge distillation, and quantization transfer or compress LLM capabilities for smaller models deployed in resource-constrained environments.Together, they balance size reduction, efficiency, and performance retention.
  • Pruning: Unstructured pruning removes individual weights and can achieve high sparsity, but irregular memory access may reduce hardware efficiency.SparseGPT reduces OPT-175B and BLOOM-176B to up to 60% sparsity.
  • Pruning: Structured pruning removes components such as neurons, channels, or layers, simplifying implementation without specialized hardware but potentially degrading accuracy more than unstructured pruning.ShortGPT removes low-influence layers using Block Influence scores.
  • Knowledge distillation: Knowledge distillation trains a smaller student to mimic a larger teacher, using internal states or output distributions in white-box settings and teacher outputs alone in black-box settings.Teacher-generated rationales and instruction data can improve smaller models with fewer examples.
  • Quantization: Post-training quantization compresses models without retraining but may lose precision, whereas quantization-aware training incorporates quantization during training and often achieves higher accuracy.Quantization reduces parameter precision and cuts storage and computational demands.

3 ADVANCED ENHANCEMENT STRATEGIES FOR SMALL LANGUAGE MODELS

The survey organizes SLM enhancement around training from scratch, supervised fine-tuning, advanced distillation and quantization, and techniques such as mixture-of-experts. Reported strategies improve efficiency, data utilization, and performance across compact models.

  • Fine-tuning and other enhancements: Enhancement methods include downstream fine-tuning, instruction tuning, preference optimization, advanced distillation, quantization, and mixture-of-experts.The survey describes these as approaches for improving instruction adherence, efficiency, and application-specific capability.
  • Training from scratch: Training-from-scratch enhancements span architecture design, data construction, and optimization strategy.The survey highlights parameter sharing, quality-focused data processing, and multi-round training.
  • Architecture design: MobileLLMs with 125M and 350M parameters improve performance by 2.7% and 4.3%, respectively, over previous models with equivalent parameters.They also achieve accuracy comparable to LLaMa-2-7B on API call tasks.
  • Data construction: Quality filtering improves SLM performance, and MindLLMs applies data-processing techniques to achieve improved capability acquisition.MindLLMs is available in 1.3B and 3B bilingual versions.
  • Optimization strategy: Two training rounds with a 50% sampling rate provide a reported trade-off between performance and training efficiency.A deep and thin architecture with multiple-round training improves Pangu 1.5B, producing an average performance increase of 8.87%.
  • Distillation techniques: Adapt-and-Distill, which adapts both teacher and student before distillation, is reported as more effective than three alternative sequencing strategies.The comparison includes training from scratch, distillation followed by adaptation, and adapting only the teacher before distillation.
  • Quantization: SmoothQuant supports effective 8-bit W8A8 quantization while preserving accuracy and reducing memory and computational costs.BitNet b1.58 matches full-precision performance from 3 billion parameters while reducing memory and latency costs.

4 APPLICATIONS OF SMALL LANGUAGE MODELS

SLMs are applied across NLP tasks and deployed on mobile and edge devices. Their privacy, latency, and memory advantages make runtime and memory efficiency central deployment considerations.

  • Applications: SLMs are increasingly used over LLMs because they offer enhanced privacy, faster inference, and lower memory requirements.These properties support applications handling localized data and operating under constrained resources.
  • Deployment: Mobile and edge deployment requires attention to memory and runtime efficiency, particularly on budget-constrained mobile phones.The section discusses task-specific applications alongside deployment methods for these devices.

4.1 Task-specific SLM Applications

SLMs support diverse task-specific applications, including question answering, coding, recommendation, web search, and mobile use. Fine-tuning, domain specialization, and tailored designs can improve performance while preserving efficiency.

  • Overview: SLMs contribute to question answering, coding, recommendation, web search, and mobile applications under resource constraints.The surveyed applications include generative tasks, encoding-oriented systems, and on-device deployment.
  • Question Answering: Instruction tuning transfers large-model outputs to smaller models, but behavior cloning can struggle on reasoning-intensive question answering.Explanation tuning and cautious reasoning with strategy selection address limitations in reasoning and system-instruction choice.
  • Domain-Specific Question Answering: Domain-specific SLMs can match or exceed larger models within specialized domains and suit edge deployments requiring precision.Their narrower flexibility is paired with higher accuracy and more relevant responses in the surveyed settings.

4.2 SLM Deployment on Mobile and Edge Devices

Deploying language models on mobile and edge devices requires jointly controlling memory use and runtime cost. The surveyed approaches compress models and caches, selectively load experts, and optimize inference execution.

  • Deployment Challenges: SLM deployment remains constrained by model size, memory capacity, computing workload, weight transfers, and resource scheduling.These constraints motivate compression and runtime-optimization methods for resource-constrained devices.
  • Memory Efficiency Optimization: Quantization, pruning, low-rank decomposition, and KV-cache compression reduce storage or computation, but aggressive compression can harm complex-task performance.KV-cache offloading can also add GPU–CPU switching overhead.
  • Memory Efficiency Optimization: EdgeMoE loads experts only when activated and achieves approximately 3× memory savings compared with retaining all weights in memory.Switch Transformers otherwise require 54GB of inference memory in the cited example.
  • KV Cache Compression: GEAR enables near-lossless KV-cache compression with up to 2.29× peak memory reduction through low-precision quantization, low-rank residuals, and sparse outlier correction.The method separates coherent and incoherent approximation errors.
  • Runtime Efficiency Optimization: On-device NPU inference achieves up to 43.6× speedup and 59.5× energy savings in the cited mllm-NPU system.The survey presents these as benefits of hardware-aware inference execution.
  • Runtime Efficiency Optimization: Runtime methods accelerate inference through prefill reduction, early exits, expert prediction, model collaboration, and reduced switching overhead.EdgeMoE lowers latency from approximately 0.7s to 0.3s in an eight-expert Switch Transformer example.

5 GENERIC AND DOMAIN-SPECIFIC SMALL LANGUAGE MODELS

The survey examines SLMs below 7 billion parameters across general and specialized domains. It organizes how these models are obtained, the data and tasks used, and representative model families.

  • Scope: SLMs in this survey have fewer than 7 billion parameters and are studied in both general and specific domains.The scope covers model acquisition, datasets, evaluation tasks, and representative models.
  • Acquisition: The survey categorizes SLM acquisition through compression, fine-tuning, and training from scratch.Representative models and associated datasets are summarized in the cited tables.

5.1 Generic-domain SLMs

Generic-domain SLMs combine compact architectures, curated data, and training procedures to reduce computational demands while retaining useful capability. The survey highlights component choices, architectural sharing, data quality, and lightweight-model performance.

  • Overview: Generic SLMs reduce pre-training, fine-tuning, and inference costs while supporting privacy, personalization, and rapid responses on low-power edge devices.Their compact and localized nature is presented as particularly suitable for resource-limited environments.
  • Model Landscape: The survey reviews 42 representative generic-domain SLMs, whose datasets and training techniques vary despite broadly similar architectures.Some training datasets are not openly available.
  • Architecture: GQA, RMS normalization, embedding sharing, layer sharing, shared FFNs, and hardware-aware architecture search are recurring efficiency-oriented design choices.PhoneLM searches for a resource-efficient architecture before pretraining according to hardware, capability, and safety requirements.
  • Training Data: Data sources span web, code, social media, multilingual, scientific, and educational corpora, with filtering and deduplication emphasized for training quality.The survey describes datasets including RefinedWeb, CulturaX, The Stack, FineWeb, RedPajama, and Pile.
  • Observed Trends: MobiLlama models show that shared FFNs can support strong performance below 1B parameters and reduce resource consumption on low-end hardware.The survey also reports competitive results from advanced architectures in the 1B–3B range.
  • Training and Optimization: Pruning, quantization, distillation, SFT, and DPO substantially enhance SLM performance, while data quality may matter more than data quantity or architecture.The cited DPO description frames optimization around preferred outputs using policy and reference-policy log ratios.

5.2 Domain-Specific SLMs

Domain-specific SLMs adapt compact language models to specialized knowledge and tasks across healthcare, science, chemistry, mathematics, astronomy, law, and finance. The surveyed approaches use domain corpora, instruction tuning, continual pre-training, distillation, and specialized annotation to improve performance.

  • Motivation: Domain-specific SLMs target well-defined tasks and expertise in fields such as healthcare, law, finance, and science.The survey presents specialized models as alternatives to broad general-purpose capability for domain-focused applications.
  • Healthcare: 59.9% on MedQA lets Hippo-Mistral 7B outperform Meditron 70B at 58.5%.The Hippo series uses continuous pre-training, instruction tuning, and RLHF after fine-tuning on Mistral and Llama-2.
  • Healthcare: 57.3% on MedMCQA (dev) and 69.0% on MMLU medical genetics exams are reported for the 2.7B BioMedLM after fine-tuning.BioMedLM is a GPT-style model trained on PubMed content.
  • Healthcare: 105K samples from ten sources form the IMHI dataset, enabling MentalLLaMA’s explainable mental-health analysis and top-method accuracy on its test set.MentalLLaMA is instruction-tuned from Llama-2-7B using expert-designed prompts and ChatGPT-generated explanations.
  • Science and mathematics: SciGLM improves ChatGLM3-6B-Base scientific QA accuracy by 3.06% across several science benchmarks.Its self-reflective annotation framework uses stepwise, reflective, and answer-integration prompts for scientific problems.
  • Insights: Domain adaptation commonly combines custom datasets with continual pre-training or fine-tuning, while distillation transfers general capabilities from LLMs.The survey notes that LLMs such as GPT-4 often annotate domain data and that specialized annotation frameworks support data quality.

6 SLMS FOR LLMS

SLMs can augment LLMs by improving reliability, retrieval, reasoning, fine-tuning, applications, and evaluation. The survey organizes these uses around lightweight auxiliary models that calibrate, filter, guide, or assess larger models.

  • Overview: The survey categorizes SLM support for LLMs into reliable generation, prompt extraction, fine-tuning, applications, guardianship, and evaluation.These categories address latency, retrieval noise, weak zero-shot performance, copyright risks, and evaluation difficulties.
  • Reliable generation: SLMs analyze LLM outputs or internal states to estimate confidence and detect hallucinations, with SAPLMA reporting 71% to 83% accuracy.Other lightweight detectors use token contributions or supervised data based on LLM knowledge boundaries.
  • Retrieval augmentation: SLMs act as retrieval proxies by judging query familiarity, document relevance, and ranking, thereby deciding when retrieval or reranking is needed.SlimPLM uses heuristic answers, SKR assesses knowledge limitations, and other systems use SLMs for relevance estimation or reranking.
  • Reasoning: SLMs transfer task knowledge into in-context examples and execute supervised subtasks, complementing LLMs when extensive supervised data exceeds ordinary ICL capacity.The discussion notes that ICL generally uses 16 to 32 examples, while specialized SLM training supports larger supervised-task workloads.
  • Applications and evaluation: SLM–LLM combinations improve specialized applications by assigning subtasks such as generation, knowledge integration, relation extraction, and empathetic response.The survey also describes SLM feedback and contrastive mechanisms for improving LLM behavior and evaluation.
  • Fine-tuning: S2L reduces mathematical SFT data to 11% of MathInstruct while achieving performance comparable to the full dataset.The approach uses proxy fine-tuning, high-quality data selection, and guidance for LLM-generated task data.

7 SYNERGY BETWEEN SMALL AND LARGE LANGUAGE MODELS

SLMs and LLMs can collaborate through cloud-edge and task-centric arrangements that divide work by privacy, latency, cost, and task complexity. The surveyed systems use local SLM processing and selective LLM escalation to preserve quality while reducing expensive large-model use.

  • Cloud-edge synergy: SLMs handle straightforward edge tasks rapidly, while LLMs process complex requests, enabling dynamic allocation that optimizes resource use and operational cost.This arrangement is described as both cloud-edge and task-centric synergy.
  • Cloud-edge synergy: Cloud-edge synergy keeps privacy-sensitive processing on SLMs while assigning de-identified or non-sensitive data to LLMs.The section distinguishes inference-time and training-time cloud-edge collaboration.
  • Inference: A split-learning 6G system assigns 0–10B-parameter SLMs to mobile devices and over-10B-parameter LLMs to complex cloud reasoning and planning.Its perception, grounding, and alignment modules support communication between device and server.
  • Inference: LLMCad combines lightweight token generation with high-precision verification and achieves up to 9.3× speedup for over-10-billion-parameter LLMs while maintaining accuracy.It uses token trees and speculative generation on devices such as Jetson TX2.
  • Training: CROSSLM trains SLMs locally and uses feedback on LLM-generated synthetic data to improve both task-specific data quality and SLM performance.The resulting cyclical exchange provides mutual enhancement while preserving client data privacy.
  • Task-centric synergy: Filter-then-rerank uses SLMs for easy information-extraction samples and sends difficult cases’ top-N labels to LLMs for reranking.This minimizes LLM reliance for non-difficult samples and improves performance by an average of 2.4% over previous methods.
  • Task-centric synergy: Data Shunt+ assigns easy samples to SLMs and hard outliers to LLMs, improving accuracy and cost efficiency while significantly reducing LLM queries.Its S4L and L4S modules use prompt pruning and two-stage confidence distillation.

8 TRUSTWORTHINESS IN SMALL LANGUAGE MODELS

Trustworthiness research on SLMs and other language models covers robustness, privacy, reliability, safety, and fairness. The survey emphasizes that model size does not uniformly determine trustworthy behavior and that its taxonomy is intentionally non-exhaustive.

  • Taxonomy: The survey focuses on five trustworthy scenarios: robustness, privacy, reliability, safety, and fairness.Robustness includes adversarial and out-of-distribution settings; safety includes misinformation and toxicity; reliability includes hallucination and sycophancy.
  • Findings: Smaller LMs sometimes outperform larger ones on trustworthiness, while larger models do not always show better harmlessness.These findings are reported across studies of open-source model trustworthiness and crafted factuality, fairness, and toxicity examples.
  • Evaluation: Trustworthiness benchmarks evaluate language models across dimensions including truthfulness, safety, fairness, robustness, privacy, and machine ethics.TrustLLM is described as a comprehensive benchmark with datasets and metrics for these dimensions.
  • Reliability: RAmBLA reports that larger models are less likely to hallucinate and may reject answers in uncertain situations.It evaluates biomedical assistants using robustness, high recall, and hallucination dimensions.
  • Safety: Fine-tuned 7B LMs can perform comparably to GPT-4-scale LMs as evaluators in SORRY-Bench.SORRY-Bench tests 43 LMs on requests that should be refused.
  • Robustness: Pruning and extreme quantization significantly affect language-model trustworthiness, while larger models generally show better resilience to perturbations.These findings come from studies of compression methods and adversarial reasoning datasets.
  • Scope and deployment: On-device SLMs exhibit vulnerabilities compared with their on-server counterparts, underscoring trust and ethical concerns for personal-device deployment.The survey notes that its listed dimensions reflect only the current focus and are not exhaustive.

9 FUTURE DIRECTIONS

The section identifies future directions for improving SLM efficiency, adaptability, collaboration, deployment, and trustworthiness while addressing gaps in evaluation and domain coverage.

  • Efficient Architectures: Hybrid architectures combining SSMs’ global static information with Transformers’ dynamic processing could balance effectiveness and efficiency.
  • Training Efficiency: Larger models’ faster, more monotonic activation convergence and higher proportional effective rank suggest ways to address SLM training inefficiencies.
  • Domain Specialization: Research should expand domain-specific SLMs beyond scientific and healthcare applications into law, finance, education, telecommunications, and transportation.
  • Evaluation: Future benchmarks should compare capabilities alongside latency, memory, power, CPU use, fine-tuning needs, and context-window constraints.
  • Efficient Enhancement: Efficient SLM adaptation should tailor instructions and data to intrinsic domain capabilities, latent knowledge, and the minimum data needed for fine-tuning.
  • SLM–LLM Collaboration: SLMs may collaborate with LLMs by routing tasks, evaluating outputs, reducing RAG noise, improving broader-task performance, and supporting trustworthy generation.
  • Deployment and Trustworthiness: Future work should refine cloud-edge task division, broaden collaborative research, comprehensively evaluate SLM trustworthiness, and develop trustworthy models from scratch or through compression and fine-tuning.

10 CONCLUSION

The survey addresses ambiguity surrounding SLM definitions and synthesizes their construction, enhancement, applications, collaboration with LLMs, deployment, and trustworthiness. It organizes representative general- and domain-specific models, methods, datasets, architectures, and future research insights.

  • The survey covers SLMs with up to 7 billion parameters and addresses ambiguity in their characterization.
  • It surveys foundational concepts, knowledge distillation, quantization, and strategies for adapting LLMs to SLM contexts.
  • The paper reviews general- and domain-specific SLMs, including their datasets, architectural decisions, applications, and device deployment strategies.
  • It examines SLMs as LLM-augmentation proxies for fine-tuning and as components of cloud-local and task-centric synergies.
  • The survey concludes with insights intended to guide future research on small language models.
Loading 2411.03350v2…