PaperlayerView exact revision on arXiv

Source-linked AI summary

Beyond Fluent Generation: A CPU Reliability Benchmark for MCP-Style Tool Calling in Sub-2B Small Language Models for Edge Deployment

Abrar Shahriar Qurat-Ul-Ain Mastoi

arXiv:2609.07370v1cs.CL

TL;DR

Tool invocation requires machine-readable calls with the right tool and arguments, beyond fluent generation. This study benchmarks five sub-2B models on controlled prompts and finds substantial differences in notebook success, with the best result depending on response recovery.

  • Problem

    Tool calls can trigger external actions, so fluent text alone is insufficient: outputs must be parseable and use the correct tool and arguments.

    Source: §1 · p. 1

  • Method

    The study benchmarks five open-weight models below 2B parameters on 100 prompts, comparing greedy and sampling decoding with recovery-based checks of tool calls.

    Source: §1 · p. 1, §3.3 · p. 3

  • Results

    Qwen2.5 checkpoints substantially outperform similarly sized Phi, Pythia, and TinyLlama checkpoints; Qwen2.5-1.5B sampling has the strongest notebook result at 79%.

    Source: §8 · p. 10

  • Takeaways & Limitations

    The evidence supports SLMs as components in bounded local tool systems, not as trusted autonomous executors.

    Source: §8 · p. 10

    The prompts supply the tool and target values, and the study omits refusal, ambiguity, adversarial, and safety cases; its resource probe also uses unspecified CPU hardware and three prompts.

    Source: §6 · pp. 9–10

Abstract

from arXiv · show

Resource constrained single-board computers including Raspberry Pi, NVIDIA Jetson Nano, Arduino UNO Q, Orange Pi, and LattePanda motivate on-device small language model (SLM) agents that reduce cloud dependence, improve data locality, and tolerate intermittent connectivity. Model Context Protocol (MCP)-style tool invocation demands more than fluent generation: an agent must emit machine-readable JSON, select the correct tool, supply all required arguments, and avoid unintended actions. We establish a platform-agnostic CPU baseline by evaluating five open-weight models below two billion parameters Phi-1.5, Pythia-1.4B, TinyLlama-1.1B-Chat, Qwen2.5-0.5B, and Qwen2.5-1.5B on 100 prompts spanning weather retrieval, web search, calculation, email composition, and task creation, under greedy decoding and nucleus sampling. A recovery parser strips Markdown fences, extracts brace-delimited substrings, and scores parseability, tool-name correctness, argument completeness, and value agreement. Under this criterion, Qwen2.5-1.5B achieves 75% (greedy) and 79% (sampling); Qwen2.5-0.5B achieves 72% (greedy) but drops to 32% under sampling. Phi-1.5 scores 0%; Pythia and TinyLlama reach at most 7%. A strict post-hoc audit finds only 5 of 1,000 raw responses directly parseable as JSON, exposing near-total dependence on output recovery. A CPU resource probe shows Qwen2.5-1.5B requires 7,960 MiB and 30.782 s mean latency; Qwen2.5-0.5B uses 3,637 MiB and 10.627 s, revealing a reliability-resource trade-off for edge deployment. These results do not cover the named boards directly or a full MCP implementation. Safe deployment requires schema validation, constrained generation, least-privilege execution, and human escalation for consequential actions.

1 Introduction

The introduction motivates evaluating whether unmodified sub-2B models can reliably produce MCP-style tool calls while accounting for CPU resource costs. It presents a controlled benchmark and distinguishes it from full MCP evaluation and direct testing on the named boards.

1 Introduction

  • External actions require more than fluent text: malformed calls can halt workflows, while valid but wrong calls can modify state, disclose information, or contact the wrong recipient.Earlier work demonstrated API calling, but correct tool names and arguments remain difficult even for substantially larger models (Schick et al., 2023; Patil et al., 2023).Source: §1 · p. 1
  • Edge SLM deployment is constrained by memory, bandwidth, compute, energy, and thermal limits, motivating a joint question about tool-call reliability and CPU resource cost.The introduction notes that parameter count alone is not a complete proxy for deployability (Lu et al., 2025; Chen et al., 2025).Source: §1 · p. 1
  • The benchmark compares JSON tool-call generation by five open-weight models below 2B parameters and separates syntax recovery, tool identification, argument completion, value agreement, and extra-argument control.It also compares greedy and sampling decoding on the same 100 prompts and reports paired tests and Wilson confidence intervals.Source: §1 · p. 1
  • The experiment is a controlled, MCP-inspired function-call baseline, not a full MCP evaluation; it omits tool discovery, client/server messaging, live execution, and multi-step trajectories.MCP benchmarks may include long tool descriptions, variable execution outcomes, intermediate observations, and single- and multi-step calls (Fan et al., 2025).Source: §1 · pp. 1–2
  • The named boards are prospective targets rather than tested devices, so the CPU experiment is a baseline for future device-specific evaluation.The introduction names Raspberry Pi, NVIDIA Jetson Nano, Arduino UNO Q, Orange Pi, and LattePanda as untested platforms.Source: §1 · p. 2

2 Related Work

Related work situates the study among edge SLMs, tool-calling evaluations, and structured-output safety research. It distinguishes an untuned checkpoint baseline on specified calls from specialized systems and more realistic agent benchmarks.

2.1 Small models and edge inference

  • Edge studies emphasize that architecture, training specialization, and output control matter alongside model size, while this comparison measures deployable checkpoint behaviour rather than a causal effect of parameter count.MobileLLM reports competitive API-calling behaviour; TinyAgent combines fine-tuning, retrieval, and quantisation; Octopus uses API-call training and conditional masking (Liu et al., 2024; Erdogan et al., 2024; Chen et al., 2025).Source: §2.1 · p. 2

2.2 Function-calling evaluation

  • Tool-use benchmarks cover planning, retrieval, stateful execution, incomplete requests, and multi-step calls, whereas every prompt here supplies the desired tool and target values.The present task therefore tests instruction adherence, copying, and structured serialization more directly than intent recognition or planning.Source: §2.2 · p. 2

2.3 Structured output and safety

  • Structured-output and safety work distinguishes syntactic guarantees from semantic appropriateness; this benchmark covers benign, fully specified calls, not adversarial safety or harmful requests.Constrained decoding still requires empirical evaluation of framework coverage, generation quality, and efficiency (Geng et al., 2025); ToolEmu motivates sandboxed testing before live execution (Ruan et al., 2024).Source: §2.3 · pp. 2–3

3 Method

The method evaluates controlled JSON-based calls across five small models and five mock tools, using recovery-based scoring alongside strict whole-response JSON checks. A separate CPU probe measures resource use; the prospective edge boards are not directly tested.

3 Method

  • The workflow moves from checkpoint selection and controlled prompt generation through decoding, JSON recovery, reliability scoring, resource measurement, and deployment interpretation.The pictured hardware platforms are prospective targets; the experiment uses a platform-agnostic CPU procedure.Source: §3 · p. 3, Figure 1 · p. 4

3.1 Models and experimental design

  • Five evaluated checkpoints run in FP32 without quantisation, under greedy decoding or sampling at temperature 0.7 and nucleus probability p = 0.9.The main experiment generates up to 150 new tokens per prompt with global seed 42.Source: §3.1 · p. 3

3.2 Benchmark prompts

  • The 100-prompt dataset balances five mock tools at 20 prompts each, with each prompt specifying a tool and every expected argument and value.The order is shuffled with seed 42 and reused across model and decoding conditions; tools are simulated rather than executed.Source: §3.2 · p. 3

3.3 Parsing and evaluation

  • Evaluation strips Markdown fences, extracts a brace-delimited candidate, parses JSON, then checks tool identity, required arguments, and value agreement; extra keys are separately recorded.Notebook success allows extra parameter keys, while schema-exact scoring additionally requires their absence; strict JSON is checked on the complete response.Source: §3.3 · p. 3
  • Rates receive two-sided 95% Wilson intervals, and paired decoding outcomes are compared with exact McNemar tests without multiple-comparison correction.These tests are described as descriptive.Source: §3.3 · p. 3

3.4 CPU resource probe

  • The separate CPU probe measures FP32 load time, process RSS, and latency across three greedy generations, while an exploratory LangGraph cell is excluded from primary evidence.The probe uses a shared, long-running process and unspecified CPU hardware; results are machine- and protocol-specific indicators, not hardware-normalised benchmarks. None of the named boards is directly tested.Source: §3.4 · pp. 3, 5, Figure 1 · p. 4

4 Results

On recovered calls, Qwen2.5-1.5B leads overall, but strict whole-response JSON is rare. Results also vary by tool, decoding strategy, and evaluation setup.

4.1 End-to-end recovered calls

  • Sampling effects differ by model: Qwen2.5-0.5B has 50 greedy-only versus 10 sampling-only successes (p = 1.62×10−7), while Qwen2.5-1.5B shows no evidence of a difference (p = .608).Sampling was run only once, so these outcomes do not estimate run-to-run stochastic variance.Source: §4.1 · p. 5

4.2 Component metrics and strict-output audit

  • Qwen2.5-0.5B greedy passes 82% at recovered JSON, 75% at correct tool selection, and 72% after argument checks; Qwen2.5-1.5B greedy fully validates all recovered objects.Recovered validity is at most 9% for Pythia, 8% for TinyLlama, and 1% for Phi.Source: §4.2 · p. 5, Table 3 · p. 7

4.1 End-to-end recovered calls

  • 79% of Qwen2.5-1.5B sampling calls succeed, versus 75% for its greedy condition and 72% for Qwen2.5-0.5B greedy; the remaining models score 0%–7%.Table 2 reports 100 prompts per condition and 95% Wilson confidence intervals.Source: §4.1 · p. 5, Table 2 · p. 6, Figure 2 · p. 6

4.2 Component metrics and strict-output audit

  • Only 5 of 1,000 complete responses are directly parseable as JSON, all from Qwen2.5-1.5B sampling, so the best application-level rate relies on recovery.A strict consumer rejects fenced or prefaced objects even when a human can understand them.Source: §4.2 · p. 6

4.3 Results by tool

  • Qwen2.5-1.5B greedy scores 0% on weather despite perfect calculation, task-creation, and email results; sampling raises weather to 70%, while Qwen2.5-0.5B greedy reaches 100% on task creation and 95% on email.The category profile shows how aggregate rates can conceal deterministic failures.Source: §4.3 · p. 6, Table 4 · p. 7

4.5 Exploratory LangGraph diagnostic

  • All ten exploratory LangGraph attempts terminate with an “id” error before a model response or tool invocation, so their 0% success reflects framework integration rather than model capability.The elapsed measurements are wrapper failure-handling times, not CPU generation latencies.Source: Table 5 · p. 7, §4.5 · p. 7

4.4 Failure modes

  • Of 658 failed evaluations, 588 (89.4%) terminate at JSON recovery, 27 (4.1%) at tool name, 40 (6.1%) at missing parameters, and 3 (0.5%) at value mismatch.Figure 3 normalizes error labels within each condition’s failures; the sequential evaluator assigns only the earliest error, potentially hiding later semantic faults.Source: §4.4 · pp. 6–7, Figure 3 · p. 8

4.6 Resource measurements

  • Within this notebook run, Qwen2.5-0.5B greedy is three points below Qwen2.5-1.5B greedy in recovered-call rate, with roughly 65% lower mean latency and 54% lower RSS.This comparison does not establish performance on other CPUs, quantisation formats, runtimes, or output lengths.Source: §4.6 · p. 7, Table 6 · p. 8

5 Discussion

Reliability varies by checkpoint and decoding strategy, while recovered-call performance does not make small language models dependable autonomous executors. The discussion recommends treating them as proposals for a supervisory layer that validates calls and controls execution.

5.1 What determines reliability?

  • Near-1.4B models range from 0% to 79%, while Qwen2.5-0.5B reaches 72% under greedy decoding, so parameter count alone does not explain reliability.Training objective, instruction-following behaviour, tokenizer, model family, and prompt compatibility are confounded.Source: §5.1 · p. 8
  • Sampling slightly benefits Qwen2.5-1.5B but lowers Qwen2.5-0.5B by 40 percentage points, demonstrating checkpoint-dependent decoding effects.The discussion calls for validating the exact model-prompt-decoder combination rather than adopting a universal temperature recommendation.Source: §5.1 · p. 8
  • Recovered success can overstate raw instruction compliance, while strict-JSON-only reporting overlooks calls recoverable by an application layer.The authors recommend a formally specified parser, schema validation, bounded repair attempts, and rejection on ambiguity.Source: §5.1 · p. 8

5.2 Deployment implications

  • Even with prompts supplying the correct tool and values, the best recovered-call rate was 79%, and strict whole-response JSON validity was 5% in that condition.The discussion notes that realistic ambiguity, missing information, distractors, changing state, and tool failures can increase errors.Source: §5.2 · p. 9
  • Treat the SLM as a fallible proposal generator and route calls through a deterministic supervisor enforcing schemas, permissions, state constraints, and human confirmation.The discussion also recommends allow-listed tools, policy checks, least privilege, and no action on failure.Source: §5.2 · p. 9

6 Limitations and Threats to Validity

The evaluation has construct, internal, statistical, systems, and external-validity limits that constrain what its results establish. In particular, it tests specified single calls rather than realistic MCP interactions, and its resource probe is not a controlled edge-hardware benchmark.

Construct validity

  • Prompts supplied the correct tool and arguments, calls were not executed or formally schema-validated, and the parser could merge objects or accept surrounding prose.No refusal, clarification, irrelevant-tool, ambiguous-intent, adversarial, or safety cases were included.Source: §6 · p. 9

Internal validity

  • Checkpoints differed in training and alignment, chat templates were not applied, sampling had one run per prompt, and missing version records limit exact replication.Generation was capped, so truncation may count as a syntax failure.Source: §6 · pp. 9–10

Statistical validity

  • The 100 prompts were manually templated rather than sampled from a documented population, so Wilson intervals describe uncertainty conditional on this item set, not real-request diversity.Tool-level cells contain 20 observations, and McNemar tests are exploratory.Source: §6 · p. 10

Systems validity

  • The resource probe used three prompts, a shared notebook process, and unspecified CPU hardware; it omitted quantised runtimes, energy, throughput, and memory-bandwidth measurements.Latency was averaged over three greedy generations after warm-up, capped at 50 new tokens; the authors caution against generalising to edge hardware without a controlled rerun.Source: §6 · p. 10, Figure 5 · p. 10

External validity

  • This was not an MCP protocol benchmark or realistic agent trajectory: it used five tools and one call per prompt, with no distractors, intermediate results, or live failures.The authors identify those missing interaction conditions as necessary for external validation.Source: §6 · p. 10

7 Future Work

Future work should expand evaluation across controlled systems conditions, model configurations, realistic tool interactions, and representative edge hardware.

7 Future Work

  • Future evaluations should record hardware and energy, repeat stochastic runs, compare model tuning and chat templates, validate schemas, test realistic interactions, and benchmark isolated quantised runtimes on edge platforms.The proposed work includes distractors, paraphrases, missing arguments, clarification, sandboxed state changes, MCP discovery and failures, context-length variation, and multi-step trajectories. On representative boards, measure peak and steady-state memory, energy, and thermal behaviour separately.Source: §7 · p. 10

8 Conclusion

The study finds large differences in JSON-based tool invocation among sub-2B models, with Qwen2.5 checkpoints outperforming the similarly sized alternatives. Its strongest result still depends on recovery, so the authors support bounded local tool systems rather than trusted autonomous execution.

8 Conclusion

  • Qwen2.5-1.5B sampling achieved 79%, the strongest notebook result, while only five of 1,000 raw generations were strict whole-response JSON.The study reports that the strongest result also carried the largest measured resource cost.Source: §8 · p. 10
  • The evidence supports SLMs as components in bounded local tool systems, not as trusted autonomous executors.The conclusion calls for constrained interfaces, schema and policy validation, least-privilege tools, confirmation for consequential actions, and human escalation when needed.Source: §8 · pp. 10–11
  • Decoding effects ranged from strongly harmful to mildly beneficial across the evaluated models.The conclusion describes substantial variation in JSON-based tool invocation among sub-2B models.Source: §8 · p. 10

Reproducibility and Data Availability

The study provides its benchmark materials and detailed results through a public project repository. A versioned archival release is recommended before final submission for long-term reproducibility.

Reproducibility and Data Availability

  • The supplied artifact includes the notebook, 100 benchmark prompts, 1,000 detailed model responses, summary metrics, error counts, and figures.The listed files include mcp_test_final.ipynb, benchmark_prompts.csv, detailed_results.csv, metrics_summary.csv, and error_counts.csv.Source: Reproducibility and Data Availability · p. 11
  • The public project repository is available, and a versioned archival release with a persistent identifier is recommended before final submission.The passage frames the archival release as a step to support long-term reproducibility.Source: Reproducibility and Data Availability · p. 11
  • The study used no human participants or personal datasets; email addresses and actions are synthetic.Source: Reproducibility and Data Availability · p. 11

Related papers

Loading 2609.07370v1…