Source-linked AI summary
IoT-MCP: Bridging LLMs and IoT Systems Through Model Context Protocol
Ningyuan Yang, Guanliang Lyu, Mingchen Ma, Yiyi Lu, Yiming Li, Zhihui Gao, Hancheng Ye, Jianyi Zhang, Tingjun Chen, Yiran Chen
TL;DR
Heterogeneous IoT hardware and control complexity make LLM-IoT integration and rigorous system evaluation challenging. IoT-MCP uses a decoupled MCP-based architecture and introduces a task benchmark; experiments report high task and prompt success alongside response-time and memory measurements.
Problem
Method
IoT-MCP uses a decoupled architecture to connect LLMs with IoT devices through MCP, and IoT-MCP Bench evaluates the framework across generated tasks and system metrics.
Source: §1 · p. 2Results
The 1,254-task benchmark reports 100% Tool Execution Performance accuracy, 99% Prompt Robustness accuracy, 205 ms average response time, and 74 KB average memory usage.
Source: §1 · p. 2Takeaways & Limitations
IoT-MCP and its benchmark provide an evaluated integration framework for LLM interaction with IoT devices across 22 sensors and six MCU families.
Source: §6 · p. 8The current architecture supports sensor-based monitoring but omits actuator integration, although its decoupled design supports extension to control mechanisms.
Source: §5.1 · p. 8
Abstract
from arXiv · showhide
The integration of Large Language Models (LLMs) with Internet-of-Things (IoT) systems faces significant challenges in hardware heterogeneity and control complexity. The Model Context Protocol (MCP) emerges as a critical enabler, providing standardized communication between LLMs and physical devices. We propose IoT-MCP, a novel framework that implements MCP through edge-deployed servers to bridge LLMs and IoT ecosystems. To support rigorous evaluation, we introduce IoT-MCP Bench, the first benchmark containing 114 Basic Tasks (e.g., ``What is the current temperature?'') and 1,140 Complex Tasks (e.g., ``I feel so hot, do you have any ideas?'') for IoT-enabled LLMs. Experimental validation across 22 sensor types and 6 microcontroller units demonstrates IoT-MCP's 100% task success rate to generate tool calls that fully meet expectations and obtain completely accurate results, 205ms average response time, and 74KB peak memory footprint. This work delivers both an open-source integration framework (https://github.com/Duke-CEI-Center/IoT-MCP-Servers) and a standardized evaluation methodology for LLM-IoT systems.
1 Introduction
IoT-MCP addresses the challenges of connecting heterogeneous IoT devices with LLMs through standardized MCP interfaces. The paper also introduces a benchmark and reports strong task execution, response-time, and memory results.
1 Introduction
- Heterogeneous hardware, low-latency demands, and the lack of standardized performance metrics complicate LLM-IoT integration and evaluation.The paper identifies MCP as a protocol-agnostic interface with an extensible tool framework for LLM interaction with IoT systems.Source: §1 · p. 1
- IoT-MCP separates interaction into Local Host, Datapool and Connection Server, and IoT Devices modules.The architecture supports communication between users and LLMs, coordination of requests to MCUs, and device-side sensor operations.Source: §1 · p. 2
- The framework supports six MCU families and 22 sensors, and its design and benchmark are open-sourced.The paper lists the supported devices and their functionalities in Figure 2 and Table 1.Source: §1 · p. 2
- 1,254 benchmark tasks support evaluations reporting 100% Tool Execution Performance accuracy, 99% Prompt Robustness accuracy, 205 ms average response time, and 74 KB average memory usage.The benchmark assesses sensor-data interpretation, device-control accuracy, and response reliability.Source: §1 · p. 2
2 Related Work
Prior work covers LLM tool calling, MCP-based IoT integration, and MCP evaluation, but the paper identifies gaps in hardware breadth and system-level assessment. It positions IoT-MCP and its benchmark as responses to those gaps.
2 Related Work
- Earlier tool-use work progressed from structured reasoning [18] to API-based tools [13] [15] and zero-shot tool use with documentation [11], before MCP standardized contextual tool documentation [10].Source: §2 · p. 2
- The benchmark's sensor list covers 22 sensors and notes a special “read-all” function for sensors with multiple functions.Source: Table 1 · p. 2
- SensorMCP supports language-driven sensor operation, but its sensor dependency and training procedures limit extensibility and response speed.The passage reports approximately 30 minutes to accommodate new requirements.Source: §2 · p. 2
- Existing MCP evaluation methods primarily assess model capabilities, leaving system-level effectiveness and operational efficiency difficult to evaluate [8].MCP-RADAR provides a five-dimensional benchmark, but different server functions complicate direct comparisons.Source: §2 · pp. 2–3
- IoT-MCP combines a decoupled architecture with a benchmark for model capabilities and system performance in IoT settings.The related-work discussion identifies limited hardware compatibility, latency from integrated training, and model-focused evaluation as gaps in existing approaches.Source: §2 · p. 3
3 Architecture
The architecture responds to IoT hardware, data, and resource constraints with a decoupled system, while the benchmark evaluates system performance using generated tasks and multiple evaluation dimensions.
3.1 Design Challenges
- IoT-MCP faces challenges from unreliable MCU connections, heterogeneous sensory data, and constrained device resources that complicate concurrent requests and processing.The design discussion also notes that sensors may store and process data locally or in the cloud, complicating unified data handling.Source: §3.1 · p. 3
3.2 IoT-MCP
- The three-domain architecture isolates the Local Host from device disruptions and uses an intermediary server layer to coordinate requests and buffer them during MCU disconnections.IoT devices use lightweight, extensible microservices and multiple communication protocols; the design distributes processing loads and supports modular sensor expansion.Source: §3.2 · pp. 3–4
- The Datapool and Connection Server can be deployed locally or in the cloud according to scale, access demand, and MCU resource constraints.Source: §3.2 · p. 3
3.3 IoT-MCP Bench
- IoT-MCP Bench combines a dataset of 114 Basic Tasks and 1,140 procedurally generated Complex Tasks with a toolkit measuring success rate, average response time, and peak memory footprint.The evaluation dimensions include tool reliability, latency and connection overhead, memory use, cross-model compatibility, concurrency, prompt robustness, and deployment stability.Source: §3.3.1 · p. 4
3.3.1 Evaluation metrics.
- The benchmark's evaluation dimensions include cross-model compatibility, concurrent-load response and memory, prompt robustness, and deployment stability during simulated network interruptions.Source: §3.3.1 · p. 4
3.3.2 Dataset Generation.
- The dataset starts with 114 manually crafted tasks based on 22 sensors, then generates 1,140 complex variants using task combination, language variation, and multi-sensor coordination.The transformations are described as complexification, ambiguification, and integration; the DHT11 example progresses from a simple temperature reading to increasingly complex tasks.Source: §3.3.2 · pp. 4–5, Figure 3 · p. 4
4 Experiments and Results
Experiments evaluate IoT-MCP on ESP32-S3 microcontrollers with Claude 3.5 Haiku, reporting successful basic-task execution, response and memory measurements, robustness tests, and a 12-hour deployment.
4.1 Experimental Setup
- Experiments generally used ESP32-S3 microcontrollers, Claude 3.5 Haiku, WiFi, and the Basic Task dataset under controlled conditions.The released MCP Server designs and their supported features are listed in Table 1.Source: §4.1 · p. 5
4.2 Metrics Evaluation
- 74 KB was the average peak MCU memory footprint, versus 51 KB average idle usage; stable TCP connections were the primary memory contributor.Idle usage accounted for approximately 40–80% of average peak usage.Source: Figure 5 · p. 5, §4.2 · p. 6
- All Basic Tasks were successfully executed after each was run 10 times across supported sensors and operation types.The reported results aggregate tool-execution success and data accuracy across sensor categories.Source: §4.2 · p. 5
- 205 milliseconds was the average end-to-end response time; the Connection Server accounted for approximately 30–75% of total response time.I2C sensors, particularly the MPU6050, had notably longer response times.Source: §4.2 · p. 5
4.3 System Robustness Evaluation
- Claude models achieved the highest success rates (99–100%); rates decreased by approximately 77% for DeepSeek V3 and 84% for GPT-4.1.The reported variation primarily reflected differences in tool-calling conventions and parameter interpretation.Source: Figure 6 · p. 6, §4.3 · p. 6
- Under concurrent requests, response times remained 150–250 ms and memory usage was 55–79 KB, with linear scaling up to four requests.KY010 and KY036 sensors were selected as representative concurrency test cases.Source: Figure 6 · p. 6, §4.3 · p. 6
- Complex tasks with challenging, ambiguous inputs achieved a 99% overall success rate; substandard results occurred on LTR390 and MPU6050 servers.Tool instructions designated as “read-all” frequently generated errors.Source: Figure 7 · p. 6, §4.3 · p. 7
4.4 Real-World Deployment Validation
- In a 12-hour real-world test, 13 sensors across 7 types connected through 6 MCUs to IoT-MCP all returned continuous results, while connections could be maintained or automatically restored after unexpected disconnections.Source: Figure 8 · p. 7, §4.4 · p. 7
5 Discussions
The discussion identifies sensor-only support and the lack of dynamic workflow composition as current scope limits. The decoupled architecture is described as extendable to control mechanisms.
5.1 Hardware Scope Limitations
- Sensor-only support confines current applications to monitoring and omits actuator integration.The authors state that the decoupled architecture can be extended to control mechanisms.Source: §5.1 · p. 8
5.2 Functional Complexity Limitations
- The architecture supports basic device invocation but lacks dynamic workflow composition because LLM+MCP clients act as callers rather than designers.Proposed future capabilities include composition engines, workflow management, performance-based optimization, and safety protocols.Source: §5.2 · p. 8
6 Conclusions
IoT-MCP bridges MCP and IoT systems through three decoupled domains and is evaluated with a 1,254-task benchmark spanning 22 sensors and 6 MCUs.
6 Conclusions
- IoT-MCP comprises three domains: Local Host, Datapool and Connection Server, and IoT Devices.The framework is presented as addressing scalable control challenges.Source: §6 · p. 8
- The benchmark contains 1,254 tasks spanning 22 sensors and 6 MCUs from different families.Source: §6 · p. 8
- IoT-MCP is evaluated using Success Rate, Average Response Time, and Peak Memory Footprint.Source: §6 · p. 8