Source-linked AI summary
Do I Need the Cloud? Uncertainty-Aware Step-Level Handoff for Small Language Model Agents
Abolfazl Younesi
TL;DR
Agent decisions can change in difficulty after tool observations, while query-level routers choose a model for the request as a whole. STEPGATE scores each local action and selectively escalates challenging steps; it reports higher success than local-only operation in single-step and multi-turn evaluations while using a minority of cloud actions.
Problem
Agent decisions change in difficulty across tool observations, but existing routers typically select a model once per query rather than for each action.
Source: §1 · pp. 1–2Method
STEPGATE combines action-structured uncertainty and risk in a gate that selectively hands challenging local SLM actions to a stronger model.
Source: §3 · pp. 3–4Results
82.7% task success with 30.8% escalation on the single-step benchmark; in multi-turn evaluation, 69.0% trajectory and 84.0% action success at a 30.0% cloud budget.
Source: §5 · p. 7Takeaways & Limitations
The results suggest selective step-level handoff offers an efficiency–reliability trade-off in the evaluated setting, with mixed local/cloud execution completing 55.0% of multi-turn trajectories.
Source: §5 · p. 7The findings are limited to benchmark tasks, scripted multi-turn observations, and the specified open-weight model pairings.
Source: §4 · p. 7
Abstract
from arXiv · showhide
Small language models (SLMs) are attractive as local agent controllers because they reduce remote inference, latency, and deployment footprint, yet structured tool errors can cause an agent step to fail. Existing routers typically select a model once per query. However, agents expose sequential decision points whose difficulty dynamically changes based on intermediate observations. We propose STEPGATE, an uncertainty-aware handoff framework that scores each local SLM action and selectively escalates challenging steps to a stronger model. On a 52-task held-out single-step BFCL-derived test split, the Qwen2.5-1.5B/7B pair attains 82.7% task success with 30.8% escalation, versus 67.3% local-only and 75.4% random escalation (which uses 33.8% escalation). In a separate multi-turn evaluation, STEPGATE achieves 69.0% trajectory success and 84.0% action success using only 30.0% cloud actions, compared with 48.0%/70.5% local-only, 60.0%/78.2% random escalation, and 57.0%/77.1% query-level routing (strong-only achieves 82.0% trajectory success at 100% cloud actions). These results suggest that step-level escalation recovers a large share of the performance gap to the stronger Qwen2.5-7B backend at a matched cloud-action rate while transmitting fewer tokens remotely. However, our evaluation is limited to one model family, a single stronger backend, and scripted tasks. Furthermore, the test sets are small, multi-turn comparisons rely on paired intervals and statistical tests, and our risk tiers serve as research annotations rather than formal safety guarantees.
1 Introduction
Agent tasks require repeated structured decisions whose difficulty can change after tool observations, making whole-query routing a poor fit. STEPGATE proposes scoring each local action and escalating only steps likely to need a stronger model.
1 Introduction
- Agent trajectories include narrow decisions such as tool selection, argument filling, observation interpretation, and choosing the next action.Structured tool errors can fail a trajectory even when a natural-language answer would remain useful.Source: §1 · p. 1
- Whole-query cloud routing can waste local capability, while committing an entire trajectory to an SLM compounds errors.The difficulty may shift from an easy local search to disambiguation and then a high-risk mutation.Source: §1 · p. 2
- The paper asks whether an SLM can predict before execution when its next action may fail and escalate only that step.The proposed signals include tool identity, arguments, schema validity, execution state, and action risk.Source: §1 · p. 2
- STEPGATE frames action-level SLM–LLM handoff as constrained risk minimization across task failure, cloud use, latency/energy, and remote-context exposure.Its stated contribution also combines action-aware uncertainty features with an adaptive gate.Source: §1 · p. 2
2 Related work
Prior work reduces inference costs through routing, uncertainty estimation, and tool-using agents, but STEPGATE focuses on choosing an executor for each structured action rather than routing a whole query.
Efficient heterogeneous inference
- STEPGATE distinguishes its routing unit—the state-action pair—from systems that route a full query or optimize where inference executes.The positioning table defines step routing, structured uncertainty, action risk, and finite-sample guarantees as comparison dimensions.Source: §2 · p. 2, Table 1 · p. 3
Uncertainty and selective prediction
- Related uncertainty work includes calibration, selective prediction, sampling disagreement, semantic entropy, and uncertainty-guided information seeking.STEPGATE adapts uncertainty signals to executable tool actions, where a single argument difference can change external consequences.Source: §2 · p. 3
Agents and tool use
- Agent and tool-use research motivates reliable execution, while STEPGATE studies calibrated pre-execution handoff from a local SLM to a stronger model.The related work spans reasoning-environment interaction, API invocation, function calling, edge agents, and standardized function-calling evaluation.Source: §2 · p. 3
3 Uncertainty-aware step-level handoff
STEPGATE reevaluates routing after each tool observation, using action-structured uncertainty and risk to choose local execution, bounded verification, or cloud handoff. Its gate is calibrated empirically, and deployment thresholds trade task success against resource and exposure budgets.
3 Uncertainty-aware step-level handoff
- At each step, the policy observes the current state, receives a structured local-model proposal, and chooses local execution, verification, or cloud escalation before reevaluating after the next observation.The state includes the user goal, interaction history, and available tools.Source: §3 · p. 3
- The action-aware representation combines token uncertainty, tool margin, schema validity, proposal consistency, trajectory state, and action-risk signals.Risk signals distinguish read-only from state-changing tools and may include destructive/open-world hints and deployment-defined severity.Source: §3 · pp. 3–4
- A lightweight gate estimates the probability that executing the SLM proposal would be judged incorrect, with model selection and calibration performed on held-out data.The calibration is empirical and carries no finite-sample misrouting bound.Source: §3 · p. 4
- The policy minimizes deployment loss across task, cloud, latency, energy, and privacy costs, or maximizes task success under a cloud-use budget.Risk-adjusted thresholds assign low-risk proposals locally, an intermediate region to bounded verification, and high-risk proposals to cloud escalation; thresholds are validation-selected.Source: §3 · p. 4, Figure 2 · p. 4
- Cloud handoffs send a compact step-specific context rather than automatically transmitting the entire local history.The request includes the subgoal, needed schemas, compact state, proposed action, uncertainty evidence, and only relevant observations.Source: §3 · p. 4
- The algorithm acquires extra consistency samples only when the initial gate is uncertain, then selects one action and continues, with retry limits and a safety-predicate fallback.Local self-verification is limited to sample-based consistency plus one bounded revision or retry.Source: §3 · p. 4
4 Experimental results
Across single-step and multi-turn tests, STEPGATE improves task outcomes over local-only and query-level routing while keeping cloud use limited. The results also show that handoff benefits depend on local-model scale, and that feature-ablation findings are not monotonic.
Experimental setup and statistical scope
- Evaluation covers one Qwen2.5 model family and stronger backend, a 52-task single-step test split, and scripted multi-turn tasks; findings do not support generalization beyond these benchmarks.The multi-turn setting uses 100 held-out trajectories with scripted observations, and the single-step router comparisons are interpreted descriptively.Source: §4 · pp. 4, 5, 7
Main result
- On 52 test tasks, STEPGATE reaches 82.7% task success at 30.8% escalation, +15.4 points over local-only and +7.3 over random escalation using 33.8%.The +7.3-point difference from random is suggestive on this small split; the passage reports overlapping individual 95% Wilson intervals.Source: Table 2 · p. 5, §4 · p. 5
Multi-turn handoff
- At 30.0% cloud actions, STEPGATE achieves 69.0% trajectory success (+21.0 points vs. local-only) and 84.0% action success; 55.0% of tasks succeed via mixed local/cloud execution.Strong-only reaches 82.0% trajectory success at 100% cloud actions; STEPGATE closes over 60% of that headroom while preserving local autonomy for 70% of steps.Source: §4 · pp. 5–6, Table 3 · p. 6
Paired analysis
- Paired multi-turn gains are +21.0 points versus local-only (p=0.0003) and +12.0 versus query-level routing (p=0.029), but comparisons with random escalation and self-consistency do not exclude zero.The random-escalation difference is +9.0 points (p=0.108), and the self-consistency difference is +3.0 points (p=0.629).Source: §4 · p. 6
Model scale and routing curves
- At a 50%-target sweep, 1.5B routing reaches 88.5% with gradient-boosted trees versus 67.3% local-only, while 3B reaches 90.4% versus 88.5% local-only.Routing offers the greatest leverage where local models leave substantial recoverable-error margin; its value saturates as local capacity nears the strong-model ceiling.Source: Figure 3 · p. 6, §4 · p. 6
Calibration and ablations
- For the 1.5B full feature set, gradient-boosted trees achieve AUROC 0.854 and ECE 0.121; logistic regression achieves AUROC 0.829 and ECE 0.212.The gate is calibrated empirically, with calibration fitted on a disjoint split because miscalibration can violate an escalation budget.Source: §4 · p. 6
- At a matched 30.8% escalation budget, removing risk metadata leaves task success identical at 82.7%; the wrong-R2+ rates are 5.8% without risk metadata and 3.8% with it.The one-step difference in wrong-R2+ outcomes is too small to be conclusive; the authors do not claim risk features improve task success.Source: §4 · p. 7
Efficiency
- For the 1.5B operating point, STEPGATE's measured end-to-end latency is 0.777s versus 2.255s for strong-only, and remote tokens are 134.5 versus 475.0.Remote-token counts are a disclosure-volume proxy, not a privacy guarantee; local energy measurements are hardware-specific.Source: §4 · p. 7
5 Conclusion
STEPGATE uses uncertainty-aware, step-level handoff for SLM agents. Its evaluations show higher success than local-only and random escalation in the tested settings, while the conclusion bounds the evidence to one model family, one stronger backend, and scripted tasks.
5 Conclusion
- 69.0% trajectory success and 84.0% action success with a 30.0% cloud budget outperform local-only and random escalation in the multi-turn evaluation.Strong-only reaches 82.0% trajectory success with 100% remote execution; 55.0% of STEPGATE trajectories complete through mixed local/cloud execution.Source: §5 · p. 7
- 82.7% task success with 30.8% escalation improves on 67.3% local-only success in the 52-task single-step benchmark.The conclusion reports the Qwen2.5-1.5B/7B pair.Source: §5 · p. 7
- The evidence is limited to one model family, one stronger backend, and scripted tasks; risk tiers are empirical annotations, not formal safety guarantees.Remote-token counts do not ensure privacy, and the individual contribution of risk features remains unestablished.Source: §5 · p. 7
A.1 Budget-matched single-step comparison and equal-budget risk ablation
This appendix block describes the single-step comparison protocol, STEPGATE’s risk- and budget-conditioned routing, calibration and risk annotations. It also distinguishes empirical calibration from conformal-style guarantees.
A.1 Budget-matched single-step comparison and equal-budget risk ablation
- Table 4 reports the 52-task single-step test with realized escalation beside each result; matched rows are retuned to 30.8% escalation.The table uses unpaired 95% Wilson intervals; the predictor ablation fixes the cloud budget at 30% and says the risk gate escalates two more of 52 steps than the full gate.Source: Table 4 · p. 9, §A.1 · p. 9, Table 5 · p. 10
A.2 Formal decision problem and routing rule
- The routing policy minimizes task, cloud, latency, energy, and remote-context costs, then uses risk- and budget-conditioned thresholds for local execution, verification, or escalation.Higher-risk actions shift thresholds toward verification and escalation; deterministic authorization policies can override routing.Source: §A.2 · pp. 9–10
A.3 Feature definitions, calibration, and labels
- The failure predictor is trained on local proposals labeled before cloud correction, with calibration selected on validation data rather than guaranteed per-deployment misrouting bounds.Its calibration protocol targets ranking and low average ECE; the passage recommends transparent logistic-regression and gradient-boosted-tree baselines.Source: §A.3 · p. 11
A.4 Background: conformal prediction and risk control
- STEPGATE’s empirical calibration does not bound misrouting on a particular deployment sample, and the implementation does not verify exchangeability across trajectory steps.The passage contrasts this with conformal-style guarantees that require exchangeability.Source: §A.4 · p. 12
A.5 Risk taxonomy: concrete examples
- Risk tiers are research annotations, and assignment depends on the tool call’s arguments as well as its tool name.Tier boundaries were fixed before test evaluation rather than tuned against downstream task success.Source: §A.5 · p. 12
A.6 Multi-turn environment and baseline specification
The multi-turn evaluation uses 100 held-out tasks in a sandboxed, scripted tool environment and compares routing methods that share the same models, schemas, and evaluator. Paired analyses assess trajectory-success differences, while larger pre-registered evaluations remain future work.
A.6 Multi-turn environment and baseline specification
- 100 held-out tasks run in a sandboxed, scripted environment spanning calendar, email, file-storage, and web/knowledge-search tools.Trajectories average 4.1 steps, with a maximum of eight; task difficulty varies by disambiguation, risk-tier requirements, and replanning after observations.Source: §A.6 · p. 13
- All methods share Qwen2.5-1.5B and Qwen2.5-7B models, tool schemas, and evaluator, differing in their routing decisions.The query-level router commits before observations, whereas STEPGATE is reevaluated after each observation; all methods use the same minimized handoff context.Source: §A.6 · p. 13
- Trajectory-success differences use paired bootstrap intervals and exact two-sided McNemar tests on identical task IDs; a larger pre-registered benchmark remains future work.The study has 100 tasks, and its unpaired Wilson intervals are for reference rather than a substitute for paired tests.Source: §A.6 · p. 13
A.7 Dataset construction, splits, and leakage prevention
These appendix recommendations specify dataset construction, handoff and tool-safety practices, and budget-matched routing comparisons. They emphasize separating evaluation from model and threshold selection and treating the gate as distinct from authorization.
A.7 Dataset construction, splits, and leakage prevention
- Use an immutable BFCL release and official evaluator where possible, separate calibration and final evaluation by task IDs, and treat the risk taxonomy as a research annotation.Distribution shift and transfer to other agent environments are unevaluated in this work.Source: §A.7 · p. 13
A.8 Prompts, handoff packet, and tool safety
- Construct minimized cloud packets from explicit task, schema, state, action, uncertainty, and necessary-observation fields, and redact sensitive values when literal content is unnecessary.State-changing tools should run in mocks or sandboxes, with independent validation, allowlists, rate limits, and required human confirmation enforcing authorization.Source: §A.8 · p. 14
A.9 Baselines and budget matching
- Match strong-model calls across routing methods by selecting baseline thresholds on validation data, and report attainable operating points rather than one favorable threshold.Repeat random escalation across seeds; query-level routing commits before the first tool action, while step-level methods may be reevaluated after observations.Source: §A.9 · p. 14
A.10 Metrics, uncertainty intervals, and statistical analysis
The section defines effectiveness, resource, uncertainty, and selective-behavior metrics, then specifies statistical and hardware-measurement practices for evaluating routers.
A.10 Metrics, uncertainty intervals, and statistical analysis
- Task success and cloud-step rate are the primary effectiveness and resource metrics, with action correctness, risk, token, latency, memory, and energy measures reported secondarily.Router discrimination uses AUROC and AUPRC, calibration uses Brier score and ECE, and selective behavior uses risk-coverage and success-cloud curves.Source: §A.10 · p. 14
- Task-level proportions use paired bootstrap 95% confidence intervals with at least 10,000 resamples; router comparisons use paired tests, and learned gates are repeated over at least three seeds.Absolute effects accompany relative improvements, and confidence-interval overlap is not a substitute for paired testing.Source: §A.10 · p. 14
- Router usefulness is assessed on success-versus-cloud-step and risky-wrong-action-versus-cloud-step frontiers, with budget-matched claims requiring equal cloud budgets within a declared tolerance.Area under the success-cloud curve is a summary; the full curve remains primary.Source: §A.10 · p. 14
A.11 Efficiency and hardware measurement protocol
- Hardware evaluations freeze device and runtime conditions, measure complete agent steps after warm-up, and report median and tail latency alongside mean latency.Energy reporting specifies counter source, sampling, integration, idle-power treatment, and tool-execution inclusion; API latency is measured separately from local latency.Source: §A.11 · p. 15
- Efficiency accounting includes local self-consistency samples and verifier calls, with local and strong-model tokens, invocations, and wall-clock time reported.Monetary cost is reported only with the exact price schedule and calculation date.Source: §A.11 · p. 15
A.12 Additional measured results
Additional measured results report gate discrimination and calibration across local-model sizes, alongside diagnostics of entropy ranking and resource-quality trade-offs; the measurements are bounded to the evaluated in-distribution setting.
A.12 Additional measured results
- The reported 30%-budget measurements characterize the evaluated in-distribution setting; robustness to unseen tool families, schema paraphrases, distractor growth, and longer trajectories is outside the study’s empirical scope.Table 8 covers three local/strong pairs.Source: §A.12 · p. 15, Table 8 · p. 15
- As entropy-ranked local coverage increases, error rates among locally executed steps rise most sharply for 0.5B and least for 3B; this is a ranking diagnostic, not a learned-gate routing curve.Table 8 covers 30%-fixed-cloud-budget results across three local/strong pairs, but its caption supplies no cell values.Source: Figure 5 · p. 15, Table 8 · p. 15
- The full logistic gate reaches AUROC 0.729/0.829/0.899 and ECE 0.154/0.212/0.135 for the 0.5B/1.5B/3B models, respectively.The reported energy measurements apply only to the evaluated hardware.Source: §A.12 · p. 15
- Figure 6 presents learned-gate reliability diagrams for Qwen2.5-0.5B, 1.5B, and 3B against a perfect-calibration diagonal, with individual bins treated cautiously because samples are finite.ECE and Brier scores are reported alongside the diagrams rather than inferred visually.Source: Figure 6 · p. 16
- Figure 7 plots measured task success against mean strong-model input-plus-output tokens per task, showing that the preferred operating point depends on local-model size.The caption frames token efficiency and task success as joint considerations.Source: Figure 7 · p. 16
A.13 Diagnostic plots from the executed runs
The diagnostic plots distinguish ranking, calibration, resource-quality trade-offs, execution costs, and failure recoverability; the section then contrasts STEPGATE’s setting with UALA, RACER, and TokUR.
A.13 Diagnostic plots from the executed runs
- Figure 5 shows entropy-ranked local error increasing with retained coverage, most steeply for 0.5B and least for 3B, a ranking result rather than STEPGATE’s success-escalation curve.The result indicates stronger entropy-based separation of easy and difficult actions for the weaker local model.Source: §A.13 · p. 16
- The learned-gate reliability diagrams’ ECE values are 0.154, 0.212, and 0.135 for the 0.5B, 1.5B, and 3B models, respectively.The diagrams assess calibration beyond whether predicted failure probabilities are correctly ordered; sparse bins should be read cautiously.Source: §A.13 · p. 16
- Escalation recovery varies by failure category, so local failure probability alone does not indicate the expected benefit of cloud handoff for every action.Figure 9 separates observed failure counts by type from recovery rates when those steps are escalated; sparse categories warrant caution.Source: §A.13 · p. 17, Figure 9 · p. 17
- Figure 7 compares measured task success with mean strong-model tokens per task, while Figure 8 and Table 9 report latency, peak memory, and power measurements specific to the evaluated hardware/runtime.The preferred resource-quality trade-off can differ by local-model size; dashed lines connect measured configurations rather than fitted curves.Source: §A.13 · p. 17, Figure 8 · p. 17, Table 9 · p. 18
- These plots are descriptive diagnostics from executed outputs, not additional experiments; the selective-risk curve is entropy-ranked and the failure analysis combines test and calibration failures.The reliability diagrams use the learned logistic gate, while the Pareto plot uses measured strong-model tokens per task.Source: §A.13 · p. 17
B Extended related work: UALA, RACER, and TokUR
- UALA uses an adaptive confidence threshold to accept a free-form answer or escalate to search or human help, whereas STEPGATE chooses an executor for a structured call after tool selection.The approaches are complementary: a UALA-style gate could decide whether to call a tool, followed by STEPGATE’s executor choice.Source: B Extended related work: UALA, RACER,… · p. 18
- RACER’s finite-sample guarantee assumes exchangeable queries and oracle correctness labels, unlike STEPGATE’s stateful steps and counterfactual failure labels; RACER selects model sets while STEPGATE commits to one executor.The passage identifies trajectory-level calibration or sequential risk control as possible extensions.Source: B Extended related work: UALA, RACER,… · p. 18
- TokUR uses low-rank weight perturbations and ensembles for token-level uncertainty, while STEPGATE’s intrinsic features use single-pass decode-time statistics to meet tight per-step latency budgets.A lightweight perturbation ensemble is described as a possible direction, traded off against added local latency.Source: B Extended related work: UALA, RACER,… · p. 18
C Limitations and broader impact
The authors limit the evaluation’s generalizability and deployment claims, noting small scripted test sets, no formal risk-control guarantee, and research-only risk annotations.
C Limitations and broader impact
- The multi-turn evaluation uses 100 scripted tasks and the single-step test has 52 held-out tasks with wide confidence intervals; differences among several routers are small relative to the sample size.The single-step baselines also have unequal realized escalation rates, with rate-matched variants reported separately.Source: C Limitations and broader impact · p. 18
- The evaluation covers one model family, one strong backend, and an in-distribution benchmark configuration, leaving robustness to other backends, distribution shift, and other benchmarks untested.Results are not stratified by risk tier or task difficulty, and the authors do not claim risk features improve task success.Source: C Limitations and broader impact · p. 19
- The empirically calibrated gate provides no finite-sample risk-control guarantee, and the study explores local self-correction only lightly, leaving the effect of deeper local deliberation unknown.The described self-correction uses sample-based consistency and one bounded retry.Source: C Limitations and broader impact · p. 19
- Risk tiers are research annotations rather than authorization policy, remote-token counts are only a disclosure-volume proxy, and the study does not evaluate leakage or adversarial settings.Energy and latency depend on hardware and runtime; strong-model escalation can fail, and correlated local/strong failures remain possible.Source: C Limitations and broader impact · p. 19