Source-linked AI summary
Little Brains, Big Feats: Exploring Compact Language Models
Dari Baturova, Elena Bruches, Ivan Chernov, Roman Derunets, Arsenii Fomin, Andrey Kostin
TL;DR
Small language models have received less attention despite their relevance where computational resources are limited, and Russian-language RAG benchmarks remain underdeveloped. This paper evaluates them as RAG generators across diverse datasets and finds that selected models can match or exceed larger models while using substantially fewer computational resources.
Problem
Evidence remains limited on small language models as generators for Russian-language RAG systems and on benchmarks spanning diverse domains and question types.
Method
The study constructs a diverse Russian-language question-answering benchmark from public and proprietary datasets and systematically evaluates small language models in RAG generation.
Results
Selected small language models surpassed larger models in output quality while requiring substantially fewer computational resources and operating without GPU hardware.
Takeaways & Limitations
Small language models show promise for practical RAG applications requiring resource-efficient, fully on-device deployment.
Takeaways & Limitations
The results reflect performance in Russian alone, limiting the study’s language scope.
Abstract
from arXiv · showhide
While large language models have been dominating the research landscape recently, small language models remain highly relevant across various domains; yet, they receive far less attention. In this study, we investigate how smaller language models perform during the generation stage within a Retrieval-Augmented Generation (RAG) system. To benchmark these models effectively, we utilised both open-source and proprietary datasets covering diverse subject areas and question types. Our findings demonstrate that a RAG system with small language models can be executed directly on-device without requiring any GPU hardware within a reasonable time. The experimental code and links to the supplementary materials can be accessed through the GitHub repository: https://github.com/SibNN/SLM-RAG-EVAL.
1 Introduction
This work investigates Small Language Models (SLMs) for generation in a Retrieval-Augmented Generation (RAG) framework, motivated by the substantial computing demands of generation-stage LLMs. It benchmarks SLM-based RAG using diverse Russian-language datasets and demonstrates feasible on-device operation without GPU hardware.
- Motivation: Generation-stage LLMs demand substantial computing power, motivating SLMs as an alternative when resources or budget are limited.
- Small Language Models: SLMs contain millions to several billions of parameters and are engineered for resource-constrained environments, including edge devices, smartphones, and personal computers.
- Deployment Motivation: SLMs are beneficial on edge platforms and necessary for sensitive information that cannot be transmitted through external APIs, requiring deployment on internal hardware.
- Study Design: The study investigates SLM generation in a product-oriented RAG framework with local CPU inference and evaluates models on open-source and proprietary datasets spanning subject areas and question types.
- Key Finding: SLM-based RAG systems operate efficiently without GPU hardware, making fully on-device deployment feasible for many applications.
- Contributions: The contributions comprise a Russian-language benchmark, systematic SLM benchmarking in RAG, and analysis of performance, evaluation methods, and on-device deployment considerations.
2 Related Work
This section situates the study within RAG, Small Language Model, benchmark, and generative-evaluation research. It highlights the need to assess compact generators under limited resources and the remaining challenges in evaluating RAG systems.
- Retrieval-Augmented Generation: Neural retrieval addressed semantic limitations of lexical methods such as TF-IDF and BM25, while multimodal RAG expanded retrieval across text, image, and audio inputs.Dense Passage Retrieval used dense representations, and multimodal retrieval can extend text-based models without additional resource-intensive multimodal training.
- Retrieval-Augmented Generation: RAG combines information retrieval with neural generation, conditioning outputs on external documents retrieved at inference time.Early open-domain question answering used a retrieve-and-read pipeline, while the RAG framework combined a neural retriever with a sequence-to-sequence generator.
- Small Language Models: The study complements work using large generators by examining whether Small Language Models can generate effectively in RAG under limited computational resources.SLMs are designed for low-latency, low-memory CPU, mobile-device, and edge-device inference, trading scale for efficiency while preserving practical NLP performance.
- Small Language Models: SLM research emphasizes compression, knowledge transfer, end-to-end pretraining, and strong-to-weak distillation for on-device, privacy-preserving, and industrial applications.Examples include DistilBERT, MobileBERT, TinyLlama, smaller Qwen variants, and Shakti; deployments commonly use quantisation, LoRA, adapters, synthetic data, and inference optimisations.
- Benchmarks: RAG benchmarks span broad domains and task types, but SLMs require evaluation aligned with domain specificity and limited computational resources.RAGBench contains 100,000 examples across five industrial domains, CRAG contains 4,409 question-answer pairs, and SLM-Bench covers nine NLP tasks and 23 datasets across 14 domains.
- Generative Evaluation: Automatic lexical metrics often miss semantic correctness and factual grounding, motivating LLM-as-a-Judge methods despite concerns about evaluator bias, reproducibility, and prompt sensitivity.G-Eval correlates strongly with human judgments, while pairwise frameworks such as MT-Bench and Chatbot Arena compare responses across systems.
3 Data Description
The benchmark evaluates Russian-language RAG generation using five datasets spanning diverse domains and question types, including public and proprietary sources. Its 500-sample evaluation set is heterogeneous, moderately challenging, and paired with a separate benchmark for testing LLM-as-Judge reliability.
- Dataset composition: Five Russian-language datasets combine publicly available and proprietary sources to cover diverse domains and question types.The proprietary dataset contains question, context, and reference answer fields but cannot be publicly released due to licensing restrictions.
- Dataset composition: The benchmark retains 100 examples per dataset after removing empty or incomplete samples, producing 500 evaluation samples.This sampling procedure was used to balance evaluation across datasets.
- Question characterization: The benchmark includes factoid, reasoning, evidence-based, comparison, experience-based, and instruction questions classified by Qwen3-8B.The classification follows the question types defined in the cited methodology.
- Evaluator reliability: A separate LLM-as-Judge dataset combines valid original pairs scored 1 with mismatched question, answer, golden-answer, and context combinations scored 0.The mixed benchmark tests whether an evaluator model can distinguish valid from invalid answers.
- Benchmark characteristics: Pairwise cosine similarities between datasets range from 0.06 to 0.12, indicating substantial diversity across data sources.The reported diversity reflects broad topic coverage and lexical variety in the final benchmark.
- Benchmark characteristics: Qwen3-8B rated question complexity at an overall average of 4.94 on a 1–10 scale, indicating moderately challenging questions.The assessment was intended to determine whether questions were overly simplistic.
4 Experiments
The experiments evaluated 17 locally runnable models and GPT-5-mini in RAG answer generation, using context-based comparisons and a no-context reference condition. Three judges—GPT-5-mini, Qwen3-8B, and GLM-4.7—were selected for reliable, consistent evaluation, achieving ICC = 0.96.
- Model selection and evaluation setup: 17 models met the parameter-size, model-family, open-source, GGUF, and 16 GB RAM criteria, with GPT-5-mini added as a state-of-the-art reference.All candidate models were first tested for CPU-only inference compatibility.
- Model selection and evaluation setup: Candidate models received the query and retrieved documents, while GPT-5-mini was evaluated with and without context during the generation-stage experiments.The retrieval component was assumed to have already produced the most relevant documents.
- Evaluation metrics: RAG outputs were scored on Correctness, Answer Relevance, Context Relevance, and Faithfulness using continuous judge scores in [0, 1].Context Relevance was excluded when comparing generation models because every model received the same retrieved documents.
- Judge selection: Judge selection combined F1 for correct-versus-incorrect classification, Average Bad Score, Pearson correlation with consensus, and judge-family diversification.Continuous scores were binarised at 0.5 for F1, while lower Average Bad Score indicated better identification of incorrect responses.
- Judge selection: GPT-5-mini, Qwen3-8B, and GLM-4.7 were selected as final judges, whose scores achieved ICC = 0.96.The consistency supported their use as a reliable multi-judge evaluation system.
5 Discussion
The discussion finds that compact models can deliver adequate RAG generation quality, while contextual information strongly affects answer accuracy. It also examines response-language preferences and CPU-only inference speed to inform practical model selection.
- Model performance: Small models can produce results comparable to larger models, although answer quality depends on model size.Some sufficiently compact models still provide adequate generation quality for RAG systems.
- Model performance: Qwen3-4B-Instruct-2507-Q5KM was selected for production because it balances response quality and CPU inference latency.
- Context dependence: Contextual information significantly influences generated-answer accuracy, indicating that correct responses require external context beyond inherent model knowledge.
- Response language: Because queries were written in Russian without instructing models to answer in Russian, the evaluation exposed their language preferences during generation.The analysis considered whether models consistently answered in Russian, switched to English, or produced mixed-language outputs.
- Inference speed: 50 samples were used to estimate average generation time in a CPU-only environment without GPU acceleration, revealing noticeable inference-speed differences across models.These measurements support practical model-selection decisions for local deployment.
6 Limitations
The study evaluates SLMs only as generative models within RAG, leaving embedding, ranking, and model-specific prompting outside its scope. Future work addressing these limitations could provide deeper insight into SLM capabilities.
- Evaluation Focus: The study focuses exclusively on SLMs’ text-generation performance within RAG systems.It does not evaluate their role in determining embedding techniques or ranking strategies.
- Evaluation Focus: Embedding techniques and ranking strategies within RAG systems are not investigated.These omissions constrain the study’s scope beyond generative-model performance.
- Prompt Standardisation: A uniform prompt is applied across all models and configurations, although architectures may respond better to customised prompts.Individual model prompt customisation could improve performance.
- Future Work: Addressing these shortcomings in subsequent studies could provide deeper insights into SLMs’ true capabilities.The proposed future work follows from the study’s evaluation and prompt-design limitations.
7 Conclusion
The study addresses the limited attention given to small language models by evaluating them as generators for Russian-language RAG systems using a diverse curated benchmark. Selected SLMs surpassed LLMs in output quality while using fewer computational resources, with future work targeting embedding, reranking, and broader task coverage.
- The study examines SLMs as generators for Russian-language RAG systems, addressing their limited attention relative to LLMs.
- A curated benchmark compiled from diverse source datasets enables equitable comparisons across a broad spectrum of topics.
- Selected SLMs surpassed LLMs in output quality while requiring substantially fewer computational resources.
- The analysis characterizes SLM performance through trade-offs between output quality and latency.
- Future research should improve embedding and reranking methodologies and expand SLMs to additional tasks.