PaperlayerView exact revision on arXiv

Source-linked AI summary

Not Every Call Needs a Frontier Model: Per-Call-Site Evaluation of Small Language Models in a Deployed Agentic Home-Automation System

Panagiotis Kasnesis, Christos Chatzigeorgiou, Lazaros Toumanidis, Amalia Contiero Syropoulou

arXiv:2610.09021v1cs.AI

TL;DR

Agentic systems make structurally different LLM calls, but are often served by one model chosen for the hardest task. This paper evaluates per-call-site model selection in a deployed home-automation system; local and hosted performance is statistically indistinguishable at four of five sites, with code generation the exception.

  • Problem

    Agentic systems issue structurally different calls with widely varying difficulty, yet one model sized for the hardest task may serve them all.

    Source: §1 · p. 1

  • Method

    The authors evaluate nine models across five call sites in deployed Wactorz using a 280-case benchmark grounded in two real installations.

    Source: §1 · p. 2, §3 · pp. 2–3

  • Results

    Four of five sites are statistically indistinguishable between the best local model and both hosted models; code generation separates them, with p = 0.002 against Sonnet 5 and p = 0.039 against Haiku 4.5.

    Source: §4 · p. 4

  • Takeaways & Limitations

    Model selection is best treated per call site, and actuation safety requires measuring both acting when asked and refusing when a device does not exist.

    Source: §7 · p. 7

    The benchmark covers two installations, while the live deployment was judged by a single unblinded rater.

    Source: §6 · p. 7

Abstract

from arXiv · show

An agentic system issues several structurally different kinds of LLM calls. It routes intent, classifies actions, grounds language in a device registry, plans multi-agent pipelines and writes the Python code those pipelines run. The difficulty of these call sites varies by an order of magnitude, yet in practice a single model, chosen for the hardest site, serves all of them. In this work, we evaluate 9 models from 0.8B to a frontier hosted model across the five call sites of a deployed open-source home-automation framework (Wactorz), using its unmodified production prompts and two real Home Assistant installations (280 cases, 2520 scored calls). We find that capability is not ordered the same way at every site, and that larger models are not uniformly better: one 4B model is worse than its 2B sibling at grounded actuation. Paired testing shows the best local model to be statistically indistinguishable from both hosted models at four of five sites. Only code generation separates them, against a small hosted model (p = 0.039) as well as a frontier one (p = 0.002). Aggregate accuracy also hides a safety failure specific to actuation, where small models resolve the accuracy/refusal trade-off in degenerate ways: one model (Gemma4 E2B) actuates on 87.2% of requests for devices the site does not own, while another refuses every request it receives. Routing each site to its best local model reaches 91.8% against 95.4% at no per-call cost. In a live deployment judged by a user, hosting only the two generative sites matches hosting everything (39/43 against 39/43) for 28% of the spend, and the actuation gap the benchmark predicted appears as exactly one case in twenty-six. Benchmark, harness and all records are released at https://github.com/waldiez/slm-callsite-eval.

1 Introduction

The paper argues that evaluating a single model for an entire agentic system obscures differences among its call sites. It measures per-site model selection in a deployed home-automation framework and tests routing policies on the live system.

1 Introduction

  • A single model sized for the hardest call site can obscure opportunities to use smaller models for easier, more frequent calls.The paper contrasts four-way intent classification under a ten-token budget with program synthesis against an unfamiliar API.Source: §1 · p. 1
  • The evaluation uses Wactorz’s deployed prompts and system rather than a simulated proxy.The first three call sites use production prompts imported from the agent code; all five prompts are reproduced in Appendix A.Source: §1 · p. 2
  • The work contributes a 280-case, two-installation benchmark and paired evaluation of nine models across five call sites.It also introduces an actuation-safety metric and tests routing end to end on the live deployment.Source: §1 · p. 2
  • The benchmark, harness, and all 2,520 records are released.The live deployment also exposes three failures that per-call benchmarking cannot see.Source: §1 · p. 2

2 Related work

Prior work discusses small models for narrow agentic calls, cost-aware per-query routing, and smart-home instruction handling. This paper distinguishes itself by measuring per-call-site substitution in a deployed system and testing routing there.

2 Related work

  • Belcak et al. [1] proposed instrumenting a running system to identify calls that smaller models can serve; this paper carries out that procedure in a deployed system.The paper reports per-site capability orderings, actuation calibration failures, and a measured cost illustration.Source: §2 · p. 2
  • Unlike per-query cascades and learned routers [2] [4] [7] [15] [20], per-call-site assignment is static, needs no runtime router, and cannot mis-route between sites.It does not adapt to a hard instance of an otherwise easy site; per-query routing could still choose models within a site.Source: §2 · p. 2
  • Smart-home studies score invalid instructions [9] or map under-specified goals to device plans [8], but neither decomposes a deployed system by call site or reports cost.Here, invalid requests name devices owned by the other installation, making a wrong actuation a substitution rather than a hallucination.Source: §2 · p. 2

3 Call sites, benchmark and setup

The setup evaluates five call sites in Wactorz’s deployed architecture, using production prompts, real installation inventories, and local and hosted models. The sites differ in output type and workload, with only planning and code generation forming a generative chain.

3 Call sites, benchmark and setup

  • Intent, HA action, and actuation produce labels or short structured objects, while planning and code generation are generative.Planner emits a pipeline specification with concrete entity ids, topics, and schedules; codegen writes Python against the framework’s agent API.Source: §3 · p. 2
  • Figure 1 shows intent as the gate to downstream sites, with planner and codegen as the only chain and actuation as a registry-heavy leaf.The actuator prompt averages 28,122 input tokens and accounts for 81% of hosted spend.Source: Figure 1 · p. 3
  • Actuation requires an exact action-set match, including refusal when the expected output is empty; code must compile and define required async functions.Planner scoring requires a valid plan resolving the supplied entity ids and topics; codegen is further assessed for running in §5.Source: §3 · p. 3
  • The 280-case benchmark spans 36 intent, 28 HA action, 116 actuator, 50 planner, and 50 codegen cases across two installations.The actuator cases include devices owned only by the other site, plausible absent-device names, and refusals.Source: §3 · p. 3
  • Nine models include Qwen3.5 and Gemma4 local sizes plus Claude Sonnet 5 and Haiku 4.5 hosted references.Gemma4 26B is a mixture of experts with roughly 4B active parameters per token; quantization-aware training for Gemma4 versus post-training quantization for Qwen3.5 is a confound.Source: §3 · p. 3
  • Four of five call sites are statistically indistinguishable between the best all-round local model and either hosted model; code generation is the exception.Table 2 reports paired McNemar tests on the same cases.Source: Table 2 · p. 4

4 Results

Results vary by call site and model size, while paired tests identify code generation as the only site separating the best local model from hosted models. Site-specific routing improves the accuracy–cost trade-off, but aggregate accuracy masks actuation calibration failures.

Results

  • Qwen3.5 4B scores below Qwen3.5 2B on actuation (50.9% vs. 62.1%), while Gemma4 26B trails Gemma4 12B on code generation (80.0% vs. 84.0%).No model wins every call-site column; three different local models lead the five sites.Source: §4 · p. 4
  • On four of five call sites, the best local model is statistically indistinguishable from both hosted models; codegen differs (p = 0.002 vs. Sonnet 5; p = 0.039 vs. Haiku 4.5).All discordant code-generation cases favoured the hosted models.Source: §4 · p. 4
  • Gemma4 26B loses on callback and lifecycle shapes—window 50, poll 67, cleanup 67, subscribe 71, versus 100 for both hosted models—supporting an API-specific rather than general synthesis interpretation.These are the same primitives that failed on real hardware; ordinary Python shapes were already matched.Source: §4 · p. 4
  • All-local per-site routing gives up 3.6 points against all-hosted, while hosting only planner and codegen gives up 1.4 points and cuts spend by 82%.These accuracies are weighted by benchmark case counts, not a deployment’s call mix.Source: Table 3 · p. 5, §4 · p. 5
  • Haiku 4.5 scores 80.6% at intent, below five of seven local models, and 89.6% across all five sites, below Gemma4 26B’s 90.7%.The models have opposite profiles: local is stronger at routing, while hosted is stronger at code generation.Source: §4 · p. 5
  • On 116 actuator cases, Qwen3.5 0.8B emits the same empty array, yielding 0.0% execution accuracy and 0.0% over-actuation; Qwen3.5 2B and 4B fail oppositely.The 2B model over-actuates on 66.7% of impossible requests; the 4B model returns an empty action list on 72.7% of actionable requests.Source: §4 · p. 5
  • Of 70 service calls on impossible requests, 53 (76%) already named a real registry entity, so valid-ID constraints alone would not prevent substitution.The authors recommend guardrails alongside a calibrated model, not in its place.Source: §4 · p. 5

5 End-to-end validation on the live deployment

In a live deployment, hosting only the planner and code-generation sites matched hosting every call site at substantially lower spend. The offline benchmark also anticipated the observed actuation gap, while live execution exposed failures the harness could not capture.

End-to-end validation on the live deployment

  • 15/20 pipeline cases passed with only planner and codegen hosted, matching all-hosted performance for 28% of the spend ($0.70 against $2.47).Excluding three system-limited cases, the configurations were indistinguishable at 39/43; running everything locally scored 12/20 on pipeline cases.Source: §5 · p. 6
  • 24/26 direct cases were identical across the middle and bottom configurations, while local actuation cost exactly one case in twenty-six (25/26 against 24/26).The offline benchmark predicted a four-point actuation difference (99.1% against 94.8%).Source: §5 · p. 6
  • The all-hosted configuration misrouted two live requests, while local configurations handled both correctly.One request was classified as conversation and left unacted; another triggered every light instead of recognizing that the site lacked a door sensor.Source: §5 · p. 6
  • Live execution exposed failures from a fresh-empty-buffer window API, an unreachable LLM client, and catalogue recipes that bypassed codegen routing.Catalogue recipes handled several requests directly, so codegen mattered less in deployment than equal-weight benchmark categories imply.Source: §5 · p. 6

6 Limitations

The evaluation is limited by single-pass sampling, two installations, and a single unblinded rater for the live run; its prompts, hardware, and benchmark weighting also constrain how broadly results apply.

Limitations

  • One pass per case at temperature 0 leaves generative-site variance unreported, although one end-to-end case passed in two configurations and failed in the third.Subgroup counts of 2–10 make those percentages indicative rather than conclusive.Source: §6 · p. 7
  • Offline ground truth covers a household and an office, and the end-to-end run covers one installation with verdicts from a single unblinded rater.Further homes and buildings are untested; blinded, multi-rater judgment is future work.Source: §6 · p. 7
  • Three intent keys and the planner entity block were revised after output inspection, and the evaluation does not compare static assignments with learned per-query routing.Both key sets are released, and the revisions moved the hosted model +0.08 and three local models −0.03.Source: §6 · p. 7
  • Equal weighting by benchmark cases differs from real call traffic, and entity-registry growth was not tested despite its dominance of the actuator prompt.The paper notes that benchmark weighting overstates codegen's deployment importance.Source: §6 · p. 7
  • Latency measurements are single-request wall-clock times without batching or energy measurement, and local models ran on a high-end DGX Spark rather than less powerful edge hardware.Hosted latency includes network round trips, and time to first token cannot be separated from decoding.Source: §6 · p. 7
  • Compound action-and-conversation requests are not evaluated, and absolute results come from one framework's prompts despite a proposed call-site taxonomy for broader agentic systems.Decomposing compound requests is future work; transfer to less powerful hardware and other frameworks remains untested.Source: §6 · p. 7

7 Conclusion

The conclusion recommends choosing models by call site rather than making a single procurement choice, and the appendix documents the production prompts and harness variants used for the evaluated sites.

7 Conclusion

  • Model selection is best treated as per-call-site routing rather than a single procurement choice, with assignments checked on the live system.The specific assignments apply to one framework and two installations.Source: §7 · p. 7

A Prompts

  • The prompt appendix uses the deployed prompts unchanged for the first three sites, while planner and codegen prompts are condensed to preserve scoring output contracts.The deployed planner and codegen prompts are assembled at call time from live registry and agent information and reach several hundred lines.Source: A Prompts · p. 10

A.1 Intent routing

  • Intent routing assigns one of four labels under a ten-token output budget, separating immediate control, Home Assistant management, reactive pipelines, and other requests.The prompt directs mixed device-control and non-Home-Assistant requests to OTHER.Source: §A.1 · p. 10

A.2 Home Assistant action classification

  • Home Assistant action classification maps requests to supported operation labels, with unclear or non-Home-Assistant requests classified as unknown.The prompt distinguishes inventory listings from device-state or existence questions, which are classified as other.Source: §A.2 · pp. 10–11

A.3 Grounded actuation

  • Grounded actuation converts language into strict JSON service-call arrays using only entities in the live registry, returning an empty array when no matching device exists.The prompt prohibits substituting a different device and treats acting on the wrong device as worse than doing nothing.Source: §A.3 · pp. 11–12

A.4 Pipeline planning

  • Pipeline planning emits complete JSON spawn configurations with concrete entities, MQTT topics, and schedules rather than descriptive plans.The prompt covers HA-actuator, scheduled, and dynamic pipeline types.Source: §A.4 · pp. 12–13

A.5 Agent code generation

  • Agent code generation outputs Python async-function code against the framework API, with explicit constraints on subscriptions, callbacks, polling, and synchronous methods.The prompt requires a callback for agent.subscribe(), which is not awaitable, and disallows a while loop inside process().Source: §A.5 · p. 13

B Acting versus refusing

The figure contrasts execution accuracy on actionable requests with over-actuation on impossible requests, showing why neither axis alone establishes deployability.

Acting versus refusing

  • Qwen3.5 2B and 4B reach almost the same joint score by failing in opposite directions: 2B never abstains, while 4B often returns an empty action list.The 2B model actuates on two thirds of impossible requests; 4B returns an empty list on 56 of 77 actionable requests and is correct on 20 of the 21 it answers.Source: Figure 2 · p. 14
  • The horizontal dimension is execution accuracy on 77 actionable requests, while the vertical dimension is the share of 39 impossible requests that produced a real-hardware call.The bottom-right region is deployable; the bottom-left can represent a model that never actuates.Source: Figure 2 · p. 14
  • A perfect over-actuation rate can belong to a model that never actuates, so the figure's axes are not meaningful in isolation.Deployment suitability depends on the joint acting-and-refusing pattern.Source: Figure 2 · p. 14

C Sub-group breakdowns

The subgroup tables break out accuracy by call site, examining installation ownership, device naming, planner feasibility, codegen API shape, intent subgroup, and HA action subgroup.

C Sub-group breakdowns

  • Table 5 breaks actuator accuracy out by installation and cross-site subgroup, including control, flip, nearmiss, same-device, and refusal cases.Flip devices are owned only by the other installation; refusal devices are owned by neither.Source: Table 5 · p. 14
  • Table 6 groups actuator accuracy by how devices are named, from direct names and locations to attributes, switches, and multi-device requests.Its refusal cases name devices the site does not own.Source: Table 6 · p. 15
  • Table 7 breaks planner accuracy out by subgroup, including infeasible requests scored on refusing to plan an unsupported pipeline..Source: Table 7 · p. 15
  • Table 8 reports code-generation accuracy by the required API shape.Source: Table 8 · p. 15
  • Table 9 reports intent-routing accuracy by subgroup.Source: Table 9 · p. 15
  • Table 10 reports Home Assistant action-classification accuracy by subgroup.Source: Table 10 · p. 16

D Where hosted spend goes

Hosted spend is concentrated at the actuator call site, whose prompt includes the site's full entity list; the table also cautions that provider-reported token counts are not comparable across model families.

D Where hosted spend goes

  • The actuator call site dominates hosted spend for both hosted models because its prompt carries the site's full entity list.The table notes that a mid-sized local model already handles this call site.Source: Table 11 · p. 16
  • Provider-reported token counts are not comparable across model families because models tokenize identical prompts differently.Source: Table 11 · p. 16
  • The table identifies the actuator call site as one a mid-sized local model already handles.Source: Table 11 · p. 16

E Model builds and serving

The local model builds share one host and runtime, while the latency and throughput comparison reports mean per-call wall-clock time and aggregate generation throughput.

E Model builds and serving

  • All seven local models use the same host and runtime, so their latency differences reflect the models rather than the serving stack.The host is an NVIDIA DGX Spark with 128 GB unified memory; serving uses llama.cpp behind LiteLLM.Source: E Model builds and serving · p. 16
  • The local-build table defines MTP as multi-token prediction and QAT as quantization-aware training, and identifies Gemma4 26B as A4B.A4B means about 4B parameters are active per token.Source: Table 12 · p. 16

F Latency and throughput

  • Table 13 reports mean wall-clock seconds per call and aggregate generation throughput; Gemma4 26B beats Gemma4 12B with about 4B active parameters per token versus 12B dense.Local latency is measured without batching; hosted latency includes network round-trip, and throughput aggregates tokens over time.Source: Table 13 · p. 17
Loading 2610.09021v1…