Source-linked AI summary
Small Language Models are the Future of Agentic AI
Peter Belcak, Greg Heinrich, Shizhe Diao, Yonggan Fu, Xin Dong, Saurav Muralidharan, Yingyan Celine Lin, Pavlo Molchanov
TL;DR
The paper addresses whether generalist LLMs are well matched to the repetitive, specialized invocations common in agentic systems. It argues from SLM capabilities, agent architectures, and deployment economics that SLMs should handle many invocations, with heterogeneous systems retaining LLMs where general-purpose abilities are essential. The paper also outlines an LLM-to-SLM conversion algorithm and emphasizes the operational and economic significance of partial migration.
Problem
Agentic systems increasingly perform repetitive, specialized tasks, raising whether generalist LLMs should handle most invocations despite their costs and broad capabilities.
Method
The paper develops a position using SLM capability evidence, agent-architecture patterns, deployment economics, and an outline for converting agents from LLMs to SLMs.
Results
Modern SLMs can provide sufficient reasoning power for a substantial portion of agentic invocations and can match or exceed performance previously associated with larger models on selected tasks.
Takeaways & Limitations
The paper supports SLM-first heterogeneous agents: use specialized SLMs for many invocations and invoke generalist LLMs selectively when general reasoning or open-domain dialogue is essential.
Takeaways & Limitations
The paper acknowledges empirical evidence that LLMs can be superior in general language understanding and that the economic comparison remains highly case-specific.
Abstract
from arXiv · showhide
Large language models (LLMs) are often praised for exhibiting near-human performance on a wide range of tasks and valued for their ability to hold a general conversation. The rise of agentic AI systems is, however, ushering in a mass of applications in which language models perform a small number of specialized tasks repetitively and with little variation. Here we lay out the position that small language models (SLMs) are sufficiently powerful, inherently more suitable, and necessarily more economical for many invocations in agentic systems, and are therefore the future of agentic AI. Our argumentation is grounded in the current level of capabilities exhibited by SLMs, the common architectures of agentic systems, and the economy of LM deployment. We further argue that in situations where general-purpose conversational abilities are essential, heterogeneous agentic systems (i.e., agents invoking multiple different models) are the natural choice. We discuss the potential barriers for the adoption of SLMs in agentic systems and outline a general LLM-to-SLM agent conversion algorithm. Our position, formulated as a value statement, highlights the significance of the operational and economic impact even a partial shift from LLMs to SLMs is to have on the AI agent industry. We aim to stimulate the discussion on the effective use of AI resources and hope to advance the efforts to lower the costs of AI of the present day. Calling for both contributions to and critique of our position, we commit to publishing all such correspondence at https://research.nvidia.com/labs/lpr/slm-agents.
1 Introduction
Agentic AI deployment is expanding rapidly, while modern agents rely primarily on generalist LLMs accessed through centralized cloud APIs. The paper challenges using a single generalist model for comparatively simple agent requests and proposes migration toward SLMs.
- More than half of large IT enterprises use AI agents, with 21% adopting them within the last year.The agentic AI sector had more than USD 2bn in startup funding and was valued at USD 5.2bn in late 2024.
- Most modern AI agents use LLMs to make strategic decisions, control tool-based operations, decompose tasks, and perform reasoning.Agents typically access these models through centralized cloud-hosted API endpoints.
- LLM API endpoints serve diverse requests through one generalist model, supported by an estimated USD 57bn investment in hosting infrastructure.The paper notes expectations that this operational model will remain central to the industry.
- The paper challenges the industry custom of routing comparatively simple agent requests to singleton generalist LLMs.It presents a position favoring SLMs for agentic AI and outlines an LLM-to-SLM conversion algorithm.
2 Position
The paper defines SLMs operationally by practical on-device deployment and argues that most repetitive, scoped agentic subtasks favor SLMs over LLMs. It proposes SLM-first heterogeneous systems, reserving LLMs for essential general reasoning or open-domain dialogue.
- 2.1 Definitions: An SLM is defined as a language model that fits on a common consumer device and serves one user’s agentic requests with practical latency.
- 2.2 Statement: The paper’s position is that SLMs are sufficiently powerful, operationally more suitable, and more economical for most agentic language-model uses.It presents this as a value statement about the likely future of agentic AI rather than a recommendation or obligation.
- 2.2 Statement: Most agentic subtasks are repetitive, scoped, and non-conversational, favoring models that are efficient, predictable, and inexpensive.The paper associates SLMs with lower latency and reduced memory, computation, and operational costs while maintaining adequate constrained-domain performance.
- 2.2 Statement: Agent architectures commonly decompose complex goals into modular subtasks that can be handled by specialized or fine-tuned SLMs.The paper characterizes using LLMs for all such tasks as economically inefficient and environmentally unsustainable at scale.
- 2.2 Statement: When general reasoning or open-domain dialogue is essential, heterogeneous systems should use SLMs by default and invoke LLMs selectively.This combines SLM efficiency and precision with LLM generality.
- 2.2 Statement: The paper frames wider SLM adoption as supporting responsible and sustainable AI deployment amid rising infrastructure costs and environmental concerns.
3 Position Arguments
The paper argues that SLMs can replace LLMs for many agentic invocations because they provide sufficient capability with greater efficiency, flexibility, and economy. It further supports heterogeneous, modular systems that combine models of different sizes and capabilities.
- 3.1 SLMs are already sufficiently powerful for use in agents: Phi, Nemotron-H, and SmolLM2 models achieve performance comparable to substantially larger models on reasoning, code generation, tool calling, and instruction following.Reported comparisons include Phi-2 running ∼15× faster than 30bn models and Nemotron-H using an order-of-magnitude fraction of inference FLOPs versus dense 30bn LLMs.
- 3.1 SLMs are already sufficiently powerful for use in agents: Additional SLM advances include Hymba-1.5B outperforming larger 13bn models on instruction following and xLAM-2-8B surpassing frontier models on tool calling.Reasoning can also be enhanced through selective fine-tuning, self-consistency, verifier feedback, and tool augmentation.
- 3.1 SLMs are already sufficiently powerful for use in agents: Modern SLMs can handle a substantial portion of agentic invocations, making capability rather than parameter count the binding constraint.The argument draws on advances in training, prompting, and agentic augmentation.
- 3.2 SLMs are more economical: SLMs are more economical because a 7bn model is 10–30× cheaper than a 70–175bn LLM in latency, energy consumption, and FLOPs.Their smaller size also reduces parallelization needs and supports high-throughput, low-latency inference.
- 3.2 SLMs are more economical: Parameter-efficient and low-resource fine-tuning lets SLM behaviors be added, fixed, or specialized in a few GPU-hours rather than over weeks.This supports rapid adaptation of multiple specialized models for different agentic routines.
- 3.3 SLMs are more flexible: Heterogeneous, modular agentic systems can assign models of different sizes to operations with different complexity while supporting strict model–tool and model–code interfaces.The paper presents “Lego-like” composition with specialized experts, tool calling, caching, and fine-grained routing as cheaper, faster to debug, and easier to deploy.
4 Alternative Views
The paper presents objections that favor LLM generalists, including stronger general language understanding and economic advantages from centralized deployment, then rebuts or qualifies these views. It acknowledges that LLM-centric deployment has a head start while arguing that specialized SLMs remain viable for decomposed agentic subtasks.
- LLM generalists and language understanding: LLMs are reported to outperform same-generation SLMs on broad language understanding, generation, translation, and reasoning tasks.The alternative view attributes this advantage to scaling laws and a hypothesized semantic hub that supports cross-language and cross-domain generalization.
- Rebuttal: Specialized SLMs can be fine-tuned for individual tasks and scaled at inference time to reach the desired reliability, while retaining greater cross-device agility.The paper presents task-specific fine-tuning and test-time compute scaling as responses to the general-performance objection.
- LLM generalists and language understanding: Agentic decomposition can reduce complex goals into subtasks where the general abstract understanding attributed to LLM semantic hubs has limited utility.The rebuttal argues that advanced agents already decompose complex problems and inputs into simpler operations.
- Economic objections: The paper acknowledges that the relative economics of SLM and LLM deployment are highly case-specific and that the economy of scale may favor centralized LLM endpoints.It identifies endpoint utilization, infrastructure setup, and talent costs as relevant considerations.
- Economic objections: Recent inference scheduling and modularization advances are presented as countering utilization concerns, while setup-cost analyses indicate a consistent falling trend.These developments support the paper’s view that SLM economics may prevail despite current deployment advantages for LLMs.
- Equally possible worlds: The paper recognizes that LLM- and SLM-centered agentic worlds are both possible, but says LLM deployment currently benefits from practice, optimization, and industry inertia.The authors maintain that the advantages developed across their arguments could plausibly overturn this starting position.
5 Barriers to Adoption
The paper identifies substantial investment in centralized LLM infrastructure, generalist evaluation practices, and limited public awareness as barriers to SLM adoption. It characterizes these as practical hurdles rather than fundamental technological flaws.
- Barriers: Large upfront investment in centralized LLM inference infrastructure has directed industry tools and infrastructure toward that paradigm.This investment has left decentralized SLM and on-device inference comparatively underconsidered.
- Barriers: SLM development often relies on generalist benchmarks, even though agentic-utility evaluations have found studied SLMs outperforming larger models.The paper presents benchmark choice as a mismatch between general model evaluation and agentic deployment needs.
- Barriers: SLMs receive less marketing and press attention than LLMs despite their stated suitability for many industrial scenarios.The paper labels this gap a barrier to popular awareness rather than a capability limitation.
- Authors’ assessment: The authors describe barriers B1–B3 as practical hurdles, with inference scheduling, agentic benchmarks, and greater awareness reducing their effect.They specifically cite Dynamo, increasing recognition of agentic evaluation, and broader knowledge of SLM economics.
6 LLM-to-SLM Agent Conversion Algorithm
The proposed conversion process replaces suitable LLM interfaces with specialized SLMs using logged agent data, task clustering, model selection, fine-tuning, and iterative refinement. The process is intended to make this change systematic and operationally manageable.
- S1 Secure usage data collection: The algorithm begins by instrumenting non-HCI agent calls to collect prompts, responses, tool-call contents, and optionally latency metrics.The paper recommends encrypted logging, role-based access controls, and anonymization before storage.
- S2 Data curation and filtering: Collected data is curated and filtered to remove PII, PHI, and other sensitive information before specialist-model training.The paper gives 10k–100k examples as a rule-of-thumb amount that can suffice for small-model fine-tuning.
- S3 Task clustering: Unsupervised clustering of prompts and agent actions identifies recurring operations that define candidate SLM-specialization tasks.Examples include intent recognition, data extraction, document-specific summarization, and tool-related code generation.
- S4 SLM selection: Candidate SLMs are selected per task using capabilities, task-relevant benchmarks, licensing, and deployment footprint.The criteria include instruction following, reasoning, context-window size, memory, and computational requirements.
- S5 Specialized SLM fine-tuning: Selected SLMs are fine-tuned on task-specific datasets, with PEFT methods such as LoRA or QLoRA available to reduce computational and memory requirements.Full fine-tuning and knowledge distillation are also considered when resources or adaptation needs justify them.
- S6 Iteration and refinement: The SLMs and router can be retrained periodically with new data, creating a continuous improvement loop that returns to data collection or selection.This refinement step is intended to maintain performance and adapt to changing usage patterns.
7 Call for Discussion
The paper frames agentic AI as a potentially transformative development and argues that reducing AI infrastructure expense and improving sustainability is desirable. It therefore calls for exploration of cost-saving and resource-efficient deployment options.
- Call for Discussion: The authors view AI agents as promising for transformative effects on white-collar work and beyond.This frames the discussion as relevant to the future impact of agentic AI deployment.
- Call for Discussion: The authors argue that expense savings and more sustainable AI infrastructure would catalyze this transformation, making exploration of such options desirable.The claim is presented as an authorial view about the value of improving AI deployment economics and sustainability.
A Definitions
The paper defines SLMs through practical deployment characteristics rather than hardware-specific thresholds, contrasting them with larger models and examining their role in agentic systems.
- Definitions: The proposed SLM definition emphasizes timelessness, practicality, and generality rather than fixed hardware metrics such as parameter count or FLOPs.The authors seek definitions that remain useful as technology advances and reflect deployment on widely available consumer devices.
- Definitions: WD1 is presented as satisfying all three criteria, while WD2 complements the set of language models.
- Definitions: The paper uses an extremalist lens to contrast maximally small and maximally large intelligent systems.The discussion highlights physical constraints at galactic scale and limited interaction capacity at minimal scale.
- Definitions: SLMs are characterized as compact systems able to run on personal devices, use modest human interaction for training, or perform constrained, verifiable tasks.The contrast positions LLMs as requiring datacenter-scale infrastructure and organization-level training.
- Definitions: The definition is anchored in models deployable in distributed fashion with present-day technology and interactive enough to be useful to humans.Its contemporary instances can evolve as the underlying technology advances.
- Definitions: The appendix assesses SLM replacement potential in MetaGPT, Open Operator, and Cradle by examining LLM use and estimating replaceable-query percentages.
B.1 Case study 1: MetaGPT
MetaGPT assigns specialized software-development roles that invoke language models for structured and adaptive work. The assessment estimates that about 60% of its LLM queries could be handled by specialized SLMs.
- B.1 Case study 1: MetaGPT: MetaGPT emulates a software company through Product Manager, Architect, Engineer, and QA Engineer roles handling drafting, design, implementation, and testing.
- B.1 Case study 1: MetaGPT: Its agents use role-based actions, prompt templates, dynamic intelligence, and retrieval-augmented generation.
- B.1 Case study 1: MetaGPT: Routine code generation, boilerplate tasks, and predefined structured responses are identified as suitable for SLM replacement.
- B.1 Case study 1: MetaGPT: 60% of MetaGPT's LLM queries are estimated to be reliably handleable by appropriately specialized SLMs.
LLM Invocations
Open Operator uses language models for intent parsing, execution-flow decisions, and content generation. The assessment estimates that about 40% of its LLM queries could be reliably handled by specialized SLMs.
- LLM Invocations: Open Operator is a workflow automation agent whose functions include natural-language processing, decision making, and content generation.
- LLM Invocations: SLMs are suited to simple command parsing, routing, and messages generated from predefined templates.
- LLM Invocations: More complex multi-step reasoning and maintaining conversation flow and context over time remain areas where LLMs offer significant advantages.
- LLM Invocations: 40% of Open Operator's LLM queries are estimated to be reliably handleable by appropriately specialized SLMs.
B.3 Case study 3: Cradle
Cradle enables agents to operate GUI applications using screenshot input and simulated interaction. The assessment estimates that about 70% of its LLM queries could be handled by specialized SLMs.
- B.3 Case study 3: Cradle: Cradle is designed for General Computer Control, enabling agents to operate GUI applications through screenshots and simulated user interaction.
- B.3 Case study 3: Cradle: Its functions include visual-context interpretation, GUI action planning, and error handling for unexpected software states.
- B.3 Case study 3: Cradle: Repetitive GUI workflows and pre-learned click sequences are identified as suitable for SLM replacement.
- B.3 Case study 3: Cradle: 70% of Cradle's LLM queries are estimated to be reliably handleable by appropriately specialized SLMs.