Source-linked AI summary
From Fixed Keys to Readable Schemas: Small Language Models for Vehicle Agent Function Calls
Hamed Jafarzadeh Asl, Yuanhao Yu, Vahid Partovi Nia
TL;DR
Vehicle assistants must map requests to valid function calls or refusals under tight on-device constraints, while vehicle function surfaces can change. This paper benchmarks Functional Token and Schema-in-Prompt representations across small language models and finds that representation determines generalization and failure modes beyond model scale alone.
Problem
Vehicle function calling must remain accurate and support refusal under constrained deployment, despite changing function surfaces that challenge how small language models represent available functions.
Method
The paper constructs a benchmark separating Seen, Unseen, and Out-of-Scope requests, then compares Functional Token and Schema-in-Prompt representations under matched training across 270M–1.7B-parameter models.
Results
The representations perform similarly on Seen Functions but diverge when the function surface changes: FT cannot predict Unseen Functions outside its learned output space, whereas SIP supports schema-mediated selection with higher inference cost.
Takeaways & Limitations
Function-surface representation is a deployment decision: FT favors stable surfaces and compact inference, while SIP accommodates changing functions without modifying the learned output representation.
Takeaways & Limitations
The benchmark is synthetic, English-only, and single-turn, and efficiency is measured on a Device Proxy rather than vehicle hardware.
Abstract
from arXiv · showhide
In-vehicle assistants must translate natural-language requests into accurate vehicle function calls under strict memory and latency constraints, making small language models (SLMs) attractive for on-device deployment. For such models, a key design choice is how the available function surface is presented. Two approaches are to represent each function with a dedicated Functional Token (FT) or provide function schemas directly in the prompt. FTs enable compact inference but are restricted to functions learned during training, whereas Schema-in-Prompt (SIP) can generalize to unseen functions at the cost of longer prompts and higher inference overhead. We introduce a benchmark of 9,822 single-turn examples spanning 79 vehicle functions derived from Android Automotive, including held-out functions and requests requiring refusal. We compare both approaches under matched fine-tuning across four SLMs from 270M to 1.7B parameters. On functions seen during training, scaling provides limited benefit: the 270M model can match the 1.7B model, while the strongest overall performance occurs at 0.6B. On held-out functions, FT achieves zero accuracy by construction, whereas SIP generalizes and improves substantially with scale. On out-of-scope requests, FT can invoke an unavailable function it was trained to emit, while SIP more reliably refuses based on the functions offered. This flexibility comes with higher memory use and latency. Our theoretical analysis explains how SIP enables generalization and why longer schema contexts increase inference cost. Overall, function-surface representation, rather than model scale alone, determines the capabilities and failure modes of SLM-based vehicle function calling.
1 Introduction
The paper asks how small in-vehicle language models should represent changing vehicle function surfaces under strict deployment constraints. It compares compact Functional Tokens with schema-conditioned prompts, emphasizing differences in generalization, refusal, and inference cost.
- Motivation: Reliable vehicle assistants must produce correct functions and arguments or refuse unsuitable requests because errors can actuate physical hardware.The available function surface also varies across vehicle models, model years, and trim levels.
- Representations: Functional Tokens provide compact inference but tie the output space to functions encoded during training.Adding a new function requires adapting the representation and model.
- Representations: Schema-in-Prompt supplies offered schemas at inference time, enabling functions unseen as training targets while increasing prompt length and computation.The model selects among functions offered for the current vehicle.
- Evaluation: The benchmark separates Seen, Unseen, and Out-of-Scope requests to compare function selection and refusal when the available function surface changes.It is derived from Android Automotive and evaluates matched settings across SLMs from 270M to 1.7B parameters.
- Contributions: The study contributes a reproducible benchmark, a controlled FT–SIP comparison across compact models, and theoretical analysis of generalization and inference cost.The comparison includes function selection, refusal, latency, and memory behavior.
2 Vehicle Function-Calling Benchmark
The benchmark converts documented Android Automotive vehicle properties into validated function schemas and evaluates selection or refusal across controlled function-surface splits. Its synthetic, single-turn design isolates representation effects while limiting ecological scope.
- Construction: The benchmark derives controllable vehicle functions from Android Automotive specifications and preserves legal locations, ranges, and enumerated argument values.Read-only properties are excluded, producing an externally specified function surface.
- Construction: Validated function calls are generated before user wording, then checked for target leakage, semantic faithfulness, and ambiguity.In a 217-example verified slice, 100.0% were judged faithful and 99.1% unambiguous.
- Evaluation design: Seen functions occur as training targets, whereas Unseen functions are withheld as targets but may appear as non-target offered schemas.Related functions are withheld together to avoid training on close counterparts from the same control family.
- Evaluation design: Out-of-Scope requests require refusal because the requested function is absent from the complete surface or omitted from that example’s offered set.The second case tests recognition of unavailability rather than operation identification alone.
- Corpus: The corpus contains 9,822 examples over a 79-function surface, with offered sets containing 4 to 24 functions and a median of 14.Seen and Unseen schemas are mixed so offered-set composition cannot reveal the split or answer by elimination.
3 Theoretical Analysis of Function-Surface Representation
The theoretical analysis distinguishes FT and SIP by their admissible task-level output sets. SIP can predict a held-out function when its schema is offered, but schema serialization increases prompt-processing and cache costs.
- Output support: Under FT, the admissible task-level output set is fixed to trained function tokens plus refusal.A function outside the trained token set cannot be recovered exactly without extending the representation and adapting the model.
- Output support: Under SIP, the admissible function set is derived from the functions offered for the current task instance plus refusal.A function withheld as a training target remains valid when its schema appears in the offered set.
- Inference cost: SIP’s schema-conditioned flexibility increases sequence length, prompt-processing cost, and key–value cache requirements relative to FT.The analysis provides a theoretical basis for measured latency and memory differences.
4 Experimental Setup
The experiments compare FT and SIP under matched training across four compact models spanning 270M to 1.7B parameters. They measure exact function-call accuracy, refusal accuracy, and execution on held-out functions.
- Models and adaptation: Four models—Gemma 3-270M, FunctionGemma-270M, Qwen3-0.6B, and Qwen3-1.7B—are adapted separately with FT and SIP using the same training corpus.The model set supports same-scale comparisons and a within-family capacity comparison.
- Training and evaluation: Matched configurations use the same examples and optimization recipe with three independent seeds per configuration.The evaluation reports exact Vehicle Function Call accuracy for Seen and Unseen Functions and correct-refusal accuracy for Out-of-Scope Requests.
- Training and evaluation: For Unseen Functions, Execution measures any valid Vehicle Function Call, while the gap from exact accuracy captures incorrect execution.This separates producing a valid call from selecting the correct held-out function.
5 Results
The results separate the effects of function-surface representation from model capacity. FT and SIP perform similarly on Seen Functions, but SIP supports Unseen Functions and more reliable availability-aware refusal at substantially higher inference cost.
- Seen Functions: Seen Function accuracy is already near saturation, with both FT and SIP performing highly across evaluated models and limited benefit from larger configurations.Qwen3-0.6B has the highest mean Seen accuracy, while FunctionGemma-270M and Qwen3-1.7B show no significant difference in any seed for SIP.
- Unseen Functions: FT achieves zero exact accuracy on Unseen Functions, whereas SIP achieves nonzero generalization whenever the schema is offered.The difference follows from FT's learned output support and SIP's ability to select from schemas supplied at inference time.
- Unseen Functions: The 1.7B configuration substantially improves Unseen accuracy over 0.6B within Qwen3.At 270M, FunctionGemma also outperforms Gemma 3 on every seed, linking schema interpretation to both capacity and prior function-calling specialization.
- Behavioral interpretation: Execution on Unseen requests can be incorrect even when a model produces a valid call, so execution rate must be distinguished from exact accuracy.For FT, every Unseen execution is necessarily incorrect because its Unseen accuracy is zero; for SIP, the execution–accuracy gap measures non-exact calls.
- Out-of-Scope Requests: SIP consistently achieves higher correct-refusal accuracy than FT on Out-of-Scope Requests, especially when a valid but unavailable function has a trained FT.SIP observes the offered Function Surface, whereas FT may emit a trained function even when it is unavailable.
- Inference cost: SIP requires 26–98× the Device Proxy time to first call and 2.4–4.2× FT's peak resident memory while adding 1,785 to 1,905 prompt tokens.The added cost follows from processing offered schemas in the input context.
6 Conclusion
The paper introduces a controlled vehicle function-calling benchmark and compares FT with SIP across changing function surfaces. It concludes that FT favors compact inference on stable surfaces, whereas SIP supports generalization and availability-aware refusal when functions change, with greater deployment cost.
- Conclusion: FT is better suited to stable Function Surfaces under tight inference constraints, whereas SIP is better suited to changing functions without modifying the learned output representation.SIP's flexibility comes with longer prompts and higher Device Proxy latency and memory use.
- Benchmark construction: Vehicle Function Call targets are generated and validated before natural-language requests are written, separating structured validity from language quality.Validation checks function identity, required arguments, types, permitted values, numerical ranges, and applicable vehicle areas.
- Function holdout: Unseen functions never appear as correct training targets, although their schemas may occur as non-target candidates in offered sets.Related functions are withheld together to reduce direct matching to nearly equivalent training targets.
- Refusal evaluation: Out-of-Scope examples require refusal when no suitable function is available, including requests for unavailable functionality and functions absent from the current offered set.The corpus contains 1,156 examples of the first type and 999 of the second.
- Quality control: Leakage filtering, independent semantic screening, and manual review address complementary benchmark-quality risks, but semantic screening is retained as diagnostics rather than automatic filtering.The manual Verified Slice found 100.0% faithfulness and 99.1% unambiguity among 217 reviewed examples.
A.9 Scope and Limitations
The benchmark isolates English, single-turn function calling with synthetic requests and does not evaluate dialogue history, changing vehicle state, dependent call sequences, additional languages, or naturally collected requests.
- Scope: The benchmark evaluates English, Single-Turn Function Calling and excludes dialogue history, changing vehicle state, and sequences of dependent Vehicle Function Calls.This scope isolates Function Surface representation from multi-turn planning and state tracking.
- Limitations: Because the utterances are synthetically generated, the validation checks do not establish that their distribution matches naturally occurring in-vehicle requests.Evaluation on naturally collected requests, additional languages, and multi-turn interactions remains future work.
B.1 Task-Level Output Support
FT fixes the task-level output support to functions assigned dedicated tokens during training, whereas SIP derives admissible outputs from schemas offered at inference. This enables SIP to represent unseen functions and condition refusal on the current function set, but increases context length and inference cost.
- Functional Token support: FT's admissible task-level outputs are the seen functions T plus refusal, so an unseen function cannot be recovered exactly without extending the representation.The evaluated FT decoder maps valid outputs to functions in T.
- Schema-in-Prompt support: SIP constructs its admissible function set from the schemas supplied at inference, allowing an unseen function to become a valid output when included in the offered set.This supports transfer from learned request–schema relations to functions never observed as training targets.
- Refusal behavior: An out-of-scope request requires refusal when no offered function can satisfy it, but FT may emit a compatible seen function unavailable in the current row.SIP exposes the offered set explicitly, so selection and refusal can condition on available functions.
- Deployment trade-off: SIP's sequence length grows with offered-schema count and size, increasing prompt-processing work and key–value cache requirements relative to FT.The longer context raises deployment cost even though it expands task-level output support.
- Generalization limits: Training-example scaling can reduce an estimation component under stated assumptions, but the theoretical bound does not guarantee monotonic empirical improvement or unseen-function generalization.Unseen-function transfer additionally relies on relations shared between request language and schema language.
B.3 Risk–Cost Trade-offs of In-Context Demonstrations
In-context demonstrations create a trade-off among attainable predictive improvement, context-selection quality, and deployment cost. More demonstrations expand the best achievable context set, but particular examples can hurt performance and lengthen inference.
- Inference-time adaptation: Demonstrations change SIP only through conditioning context while model parameters remain fixed, making their effects an inference-time adaptation.The serialized context includes requests, offered schemas, target outputs, and formatting.
- Decision margins: A positive call margin means the target outranks every competing offered function and exceeds the rejection threshold; a positive refusal margin places every offered function below it.Additional demonstrations may increase a particular margin, but arbitrary demonstrations need not do so.
- Predictive trade-off: Demonstrations can improve predictive risk only when their attainable gain exceeds the context gap introduced by the selected sequence.The context gap can reflect suboptimal selection or ordering, irrelevant examples, and conflicting examples.
- Budget effects: Increasing the demonstration budget cannot worsen the best achievable risk, but the actual context's risk may increase or decrease with selection, relevance, and ordering.The larger budget expands admissible contexts without guaranteeing that the supplied context improves.
- Deployment cost: Dense self-attention makes prompt-prefill computation scale quadratically with context length, while the broader deployment cost also depends on the selected context and cost weighting.The sequence includes the request, offered schemas, demonstrations, and formatting overhead.
C.1 Training and Inference Details
The evaluation uses matched training and inference procedures across model–representation pairs, with common examples and parsing criteria. Reported behavioral metrics distinguish exact calls, valid execution, refusal, and over-triggering.
- Training setup: All model–representation pairs use the same training examples and optimization recipe, with three independently trained seeds per configuration.Training runs for three epochs with learning rate 2 × 10^-5, batch size 16, maximum sequence length 4096, and warmup ratio 0.03.
- Inference setup: Inference uses greedy decoding with at most 128 generated tokens, and FT and SIP predictions are parsed using matched criteria.The matched parsing criteria prevent more permissive formatting handling from favoring either representation.
- Primary metrics: Seen and Unseen accuracy require exact agreement with the target Vehicle Function Call, including function and arguments.Out-of-Scope accuracy measures correct refusal instead of producing a call.
- Additional metrics: Unseen Execution counts parsed calls to valid functions, while Out-of-Scope Over-trigger counts calls produced when refusal is correct.Execution exceeds exact accuracy when a valid call is produced but does not exactly match the target.
- Aggregation: Accuracy and behavioral rates are averaged across three seeds after per-seed computation, whereas statistical tests use paired example-level correctness.Device Proxy measurements use a separate aggregation procedure.
C.3 Complete Main-Grid Results
Table 3 contains the complete numerical accuracy results for every model–representation pair in the main grid. These values support the Seen, Unseen, and Out-of-Scope comparisons presented in the accompanying figures.
- Main-grid results: Table 3 reports exact mean accuracies for all model–representation pairs in the main experimental grid.The table underlies Seen Function and Out-of-Scope results in Figure 2 and Unseen Function accuracy in Figure 3.
C.4 Paired Statistical Tests
The paired analyses test whether observed differences reflect consistent example-level disagreements, while ablations and device-proxy measurements clarify adaptation, prompt-format, and deployment-cost effects. Results indicate that representation and implementation choices affect unseen-function performance, refusal, and inference cost.
- Paired comparisons: No significant Seen Function difference was detected between FunctionGemma-270M SIP and Qwen3-1.7B SIP in any seed.The comparison provides no evidence that the larger configuration improves Seen performance, but does not establish equivalence.
- Paired comparisons: FunctionGemma-270M SIP significantly outperformed Gemma 3-270M SIP on Unseen Functions in every seed.No significant Seen Function difference was detected for this same-scale pair; the result is limited to the evaluated models.
- Refusal decomposition: For unavailable Seen Functions with trained FT tokens, FT had substantially lower correct-refusal rates than SIP across all evaluated models.The FT input did not expose the row-specific offered set, preventing direct conditioning on the function's absence.
- Refusal decomposition: When unavailable functions lacked trained FT tokens, both representations refused much more reliably, concentrating the aggregate FT–SIP refusal gap in token-available cases.This decomposition distinguishes inability to produce a function from failure to recognize that an available function is absent from the offered set.
- Ablations: Full fine-tuning lowered Qwen3-1.7B SIP Unseen Function accuracy relative to LoRA but preserved its within-Qwen ordering and improved Out-of-Scope accuracy.The ablation does not establish that LoRA is generally superior to full fine-tuning.
- Ablations: FunctionGemma's native function-calling format improved Unseen Function accuracy but reduced Out-of-Scope accuracy relative to the common SIP format.The improvement did not eliminate the larger gap between the 270M and Qwen SIP configurations.
- Deployment cost: SIP's longer schema prompts account for most of its latency difference because offered schemas are serialized in the input.FT uses a compact request-only representation, while measurements were collected on a Xeon 8580 CPU with four threads and Q4_K_M quantization.