Source-linked AI summary
TinyAgent: Function Calling at the Edge
Lutfi Eren Erdogan, Nicholas Lee, Siddharth Jha, Sehoon Kim, Ryan Tabrizi, Suhong Moon, Coleman Hooper, Gopala Anumanchipalli, Kurt Keutzer, Amir Gholami
TL;DR
Large LLMs enable tool-using agents but are difficult to deploy at the edge because of their size and computational demands. TinyAgent trains specialized small models with curated function-calling data and improves inference through tool retrieval and quantization. Its models achieve function-calling performance that can surpass GPT-4-Turbo while supporting local deployment.
Problem
Large LLMs’ size and computational demands limit edge deployment, creating privacy, connectivity, and latency challenges for agentic systems.
Source: §1 · p. 1Method
TinyAgent fine-tunes small open-source models on curated function-calling data and uses Tool RAG plus quantization for efficient local inference.
Source: §1 · p. 2, §3.4 · pp. 5–6, §3.5 · p. 6Results
TinyAgent’s fine-tuned models achieve function-calling performance that can surpass GPT-4-Turbo while supporting efficient edge deployment.
Source: §1 · p. 2, §3.3 · p. 5, §5 · p. 7Takeaways & Limitations
Task-specific small language models can power a locally deployed Mac assistant that executes user commands through function calling.
Source: §1 · p. 2, §5 · p. 7TinyAgent still requires human oversight because hallucinations and erroneous responses remain possible.
Source: §6 · p. 7
Abstract
from arXiv · showhide
Recent large language models (LLMs) have enabled the development of advanced agentic systems that can integrate various tools and APIs to fulfill user queries through function calling. However, the deployment of these LLMs on the edge has not been explored since they typically require cloud-based infrastructure due to their substantial model size and computational demands. To this end, we present TinyAgent, an end-to-end framework for training and deploying task-specific small language model agents capable of function calling for driving agentic systems at the edge. We first show how to enable accurate function calling for open-source models via the LLMCompiler framework. We then systematically curate a high-quality dataset for function calling, which we use to fine-tune two small language models, TinyAgent-1.1B and 7B. For efficient inference, we introduce a novel tool retrieval method to reduce the input prompt length and utilize quantization to further accelerate the inference speed. As a driving application, we demonstrate a local Siri-like system for Apple's MacBook that can execute user commands through text or voice input. Our results show that our models can achieve, and even surpass, the function-calling capabilities of larger models like GPT-4-Turbo, while being fully deployed at the edge. We open-source our dataset, models, and installable package and provide a demo video for our MacBook assistant agent.
1 Introduction
Large LLMs enable tool-using agents but are difficult to deploy locally because of their size and computational demands. TinyAgent addresses this gap by training specialized small models for private, edge-based function calling and efficient agentic execution.
1 Introduction
- Cloud-based LLM inference raises privacy, connectivity, and latency concerns that motivate local edge deployment.Uploading user data to third-party services can create privacy issues, while unstable connectivity and cloud response times can impede real-world use.Source: §1 · p. 1
- Large models retain extensive general world knowledge, making them unnecessarily large for specialized downstream applications.The paper links this parametric memorization to the scaling pressures behind capabilities such as in-context learning and complex reasoning.Source: §1 · p. 1
- TinyAgent asks whether smaller models with less parametric memory can reproduce larger models’ emergent abilities for specialized tasks.The motivating task is understanding natural-language commands and orchestrating tools and APIs.Source: §1 · pp. 1–2
- The framework trains specialized small language models on high-quality data so they can privately and securely perform edge-based function calling.The approach avoids requiring generic world knowledge while targeting natural-language understanding and tool orchestration.Source: §1 · p. 2
- TinyAgent combines function-calling enablement, curated task-specific data, fine-tuning, Tool RAG, and quantization for efficient deployment.The authors report that fine-tuning can exceed GPT-4-Turbo’s function-calling performance, while retrieval and quantization support real-time edge inference.Source: §1 · p. 2
2 Related Work
Prior work establishes tool-calling models, synthetic dataset generation, and device-control agents, but TinyAgent combines these directions for efficient local function calling. It emphasizes task-specific abstractions and deployment on MacOS rather than cloud dependence or low-level UI control.
2.1 Function Calling LLMs
- Function-calling research spans Toolformer, ReAct, Gorilla, ToolLLM, and on-device API invocation, while TinyAgent adds parallel calling and tool retrieval without architectural changes.This preserves compatibility with a wider range of open-source models.Source: §2.1 · p. 2
2.2 Dataset Synthesis
- Synthetic-data methods can produce large training sets, motivating filtered or smaller high-quality datasets for efficient task-specific function-calling training.TinyAgent builds a pipeline for systematically generating curated data aimed at robust performance with fewer examples.Source: §2.2 · p. 2
2.3 Device Control
- Android benchmarks and UFO study low-level or UI-focused device control but do not address direct deployment of small language models on the device.Their settings emphasize simulated mobile interactions or Windows applications rather than resource-constrained local execution.Source: §2.3 · p. 2
- TinyAgent formulates device control as high-level function calling instead of low-level UI actions and runs fully locally on MacOS.Task-specific abstractions are intended to support more robust and efficient execution in real-life on-device scenarios.Source: §2.3 · pp. 2–3
3 TinyAgent
TinyAgent adapts small language models for function-calling agents by combining structured planning, curated training data, tool retrieval, and quantized edge deployment. Across these components, the framework improves planning accuracy while reducing the prompt and model costs needed for local execution.
3.1 Teaching LLMs to do Function Calling
- LLMCompiler teaches small models to output function plans containing selected functions, arguments, and dependencies that can be parsed and executed.Off-the-shelf small models otherwise produce wrong functions, hallucinated names, incorrect dependencies, and inconsistent syntax.Source: §3.1 · p. 3
3.2 Dataset Generation
- 80K training examples, 1K validation examples, and 1K test examples were synthetically generated from function sets, validated with graph and type checks, at a cost of about $500.The dataset targets a Mac assistant with predefined APIs for tasks such as email, contacts, messaging, calendars, notes, reminders, files, and Zoom meetings.Source: §3.2 · pp. 3–4
3.3 Fine-tuning for Improved Function Calling Reasoning
- 83.09% success rate was achieved by the fine-tuned 7B model, about 4% higher than GPT-4-Turbo, while the 1.1B model rose from 12.71% to 78.89%.Success requires selecting the correct functions and matching the ground-truth dependency DAG.Source: §3.3 · pp. 4–5
3.4 Efficient Inference with Tool RAG
- DeBERTa-based Tool RAG retrieves 3.97 tools per query with 0.998 recall, compared with six tools and 0.968 recall for basic RAG, while reducing prompt size by about 2x tokens.The method treats tool selection as 16-way multi-label classification and includes tools whose predicted probability exceeds 50%.Source: §3.4 · pp. 5–6, Figure 3 · p. 5, Table 1 · p. 6
3.5 Fast Edge Deployment with Quantization
- 4-bit quantization with group size 32 delivers 30% better latency and a 4x model-size reduction for edge deployment.Quantization reduces storage, model footprint, and the time and resources required to load model weights into memory.Source: §3.5 · p. 6, Table 2 · p. 6
4 Putting It All Together
TinyAgent runs model inference and audio processing locally on a MacBook, enabling private, on-device commands. Its 1.1B model exceeds GPT-4-Turbo accuracy while remaining fast in local deployment.
- The MacBook assistant processes user commands through locally executed model inference and locally processed audio.Audio input uses the locally deployed Whisper-v3 model through whisper.cpp.Source: §4 · pp. 6–7
- The TinyAgent-1.1B model exceeds GPT-4-Turbo accuracy while running locally and privately on-device.Source: §4 · p. 7
5 Conclusions
TinyAgent shows that a small language model can power a Mac Siri-like assistant for processing user queries. Its function-calling, retrieval, and quantization pipeline enables edge deployment while achieving higher task success than GPT-4-Turbo.
6 Ethics Statement
The ethics statement emphasizes accessible, inclusive, and locally deployable agent systems while recognizing risks from hallucinations and synthetic-data bias. It recommends human oversight and diverse data practices to address these concerns.
- TinyAgent should support equitable access, including accessibility features such as voice recognition and text-to-speech.Source: §6 · p. 7
- Human oversight remains necessary because hallucinations and erroneous responses can occur during TinyAgent’s operation.The statement recommends checks allowing humans to review, override, or refine decisions.Source: §6 · p. 7
- Synthetic training data can carry regional and cultural biases that affect effectiveness and impartiality across demographic groups.The authors describe incorporating diverse cultural data and demographic groups to mitigate these biases.Source: §6 · p. 7