LEASH: Uncertainty-Aware Weak-to-Strong Escalation and Oversight for Computer-Use Agents

1. Introduction

• Weak-to-strong escalation balances the high cost of frontier models against the poor performance of light open-weight models, but raises questions about when to escalate and who oversees the strong agent.Long-horizon computer-use tasks can take more than an hour on average; the weak agent may also fail to ensure the strong agent’s actions achieve the intended outcome.Source: §1 · p. 2
• LEASH estimates failure risk by comparing weak-agent hidden states with past successes and failures, then uses the signal for initial routing, runtime escalation, and post-handoff oversight.A weak observer monitors the strong agent’s trajectory every three steps; high uncertainty triggers rollback and one reconsideration with higher reasoning effort.Source: §1 · p. 2
• LEASH outperforms random escalation by 6.55–11.94 points at matched escalation rates and recovers 75–102% of the weak-to-Opus performance gap at 69–81% of always-on Opus cost.For Qwen3.5-35B-A3B, success is 73.7% versus 72.9% for always-on Opus, at 19% lower cost.Source: §1 · p. 3
• LEASH frames escalation as a budget-constrained sequential stopping problem and uses one latent uncertainty signal for both escalation and oversight.Its score measures whether a hidden state lies closer to the weak agent’s past failures than to its past successes, with theoretical justification for escalation.Source: §1 · p. 3
• The framework extends the weak agent’s uncertainty signal to post-handoff oversight, with the white-box weak agent monitoring the black-box strong agent through its own hidden states.The approach uses the weak agent’s past experience without an external dataset or additional training, and without hosting another expensive runtime agent.Source: §1 · pp. 2–3

Efficient Computer-Use Agents

• For efficient computer-use agents, model routing and cascading offer a promising way to balance inference cost and performance, as demonstrated by FrugalGPT and RouteLLM (Chen et al., 2023).Source: §2 · p. 3
• Unlike routing methods based on action difficulty and confidence or trained lightweight monitors, LEASH uses the backbone agent’s internal representations for task-level routing and step-level escalation without training a dedicated monitor.Adaptive VLM Routing selects models by action difficulty and confidence; StepWise trains monitors to detect stalls and milestones for escalation.Source: §2 · p. 4

Weak-to-Strong Paradigms

• Weak-to-strong generalization studies training-time transfer from weaker supervision, whereas weak-to-strong escalation is an inference-time setting in which weak models execute by default and selectively invoke stronger models.This changes the weak–strong relationship from supervision transfer to sequential resource allocation.Source: §2 · p. 4

Uncertainty Estimation and Agent Monitoring

• Prior uncertainty and monitoring work addresses GUI grounding, question answering, or oversight through observable trajectories, while LEASH integrates representation-based uncertainty into escalation and closed-source-agent oversight.Repeated generation can be costly at every step of long-horizon trajectories; LEASH reuses backbone hidden states.Source: §2 · p. 4

3.1. Problem Formulation

• The weak agent acts until control transfers to the strong agent, and the escalation policy’s stopping time determines when handoff occurs within the shared action budget.The total escalation rate serves as a proxy for the budget, with task success as the objective.Source: §3.1 · p. 5

3.2. Reference-Based Latent Uncertainty Estimation

• LEASH builds query and action reference banks from weak-agent hidden states labeled by task and subgoal outcomes, then scores states by their relative distance to successful and failed references.Query states are taken at the task-prompt token, while action states are taken at each step’s last generated action token; a larger score means greater relative proximity to failures.Source: §3.2 · pp. 5–6

3.3. Uncertainty-Aware Weak-to-Strong Escalation

• Initial and runtime gates use query- and action-level uncertainty to route difficult tasks early or hand off after the first flagged action.Thresholds are calibrated from reference-task score quantiles to target an overall escalation rate; runtime calibration accounts for tasks retained by the initial gate and uses each trajectory’s maximum runtime score.Source: §3.3 · p. 6

3.4. Post-Handoff Oversight and Intervention

• After handoff, the weak agent acts as a shadow observer, periodically scoring its own candidate action representation in the strong agent’s context and triggering reflection when uncertainty exceeds a threshold.The approach assumes the weak model’s uncertainty also indicates greater difficulty for the strong agent on average; intervention is capped, and further interventions stop after two alter the strong agent’s proposed action.Source: §3.4 · p. 7

4.2. Task Success and Inference Cost

• 73.66% task success at $0.4117 per task with Qwen3.5-35B-A3B, versus 72.90% for always-on Opus at $0.5094 per task.LEASH escalated 64.1% of tasks; the comparison is on OSWorld.Source: Table 1 · p. 8, §4.2 · p. 8
• LEASH beats random escalation by 6.55–11.94 percentage points at closely matched escalation rates and recovers 75–102% of the weak-agent-to-Opus performance gap, versus 37–69% for random escalation.LEASH costs 69–81% as much as always-on Opus.Source: §4.2 · p. 8
• At comparable escalation rates, LEASH achieves higher success than StepWise and Progress Advantage on OSWorld with EvoCUA-8B.Both comparison methods use Opus 5 as the strong executor.Source: Table 2 · p. 8, §4.2 · p. 9

4.3. Uncertainty Estimation

• Pooled LDA performs best for initial uncertainty on Nemotron and runtime uncertainty on both tested backbones, while probability-based uncertainty is stronger initially for Qwen3.8-27B.Estimator effectiveness remains moderate and varies by backbone and execution stage.Source: §4.3 · p. 9, Table 3 · p. 9

4.4. Ablation Study

• Enabling uncertainty-guided escalation raises success from 41.15% with the Manager to 67.94%; adding Shadow monitoring and checkpoint intervention raises it further to 73.66%.The added Shadow components incur only a small additional cost.Source: §4.4 · p. 9, Table 4 · p. 10
• On Shadow-triggered tasks, Shadow-enabled success is 64.5% versus 48.4% without Shadow, while always-on Opus succeeds on 51.6% of those tasks.Opus succeeds on 74.5% of escalated tasks without a Shadow trigger.Source: §4.4 · p. 10, Table 5 · p. 10

4.5. Sensitivity Analysis

• Across escalation rates, LEASH outperforms random task routing; on escalation-prone tasks, it reaches 64.6% success at $0.67 per task versus 60.4% at $0.79 with shuffled timing.The timing comparison is on the subset most frequently escalated by LEASH.Source: §4.5 · p. 10, Figure 4 · p. 11, Figure 5 · p. 11
• Increasing the reference bank from 40 to 99 raises initial-stage AUROC from 0.701 to 0.770 for Qwen3.5-35B-A3B and from 0.633 to 0.696 for EvoCUA-8B.The reported AUROC is initial-stage uncertainty estimation.Source: §4.5 · p. 10, Table 6 · p. 11

• The analysis isolates initial routing: each task is either escalated before execution or not escalated, and the fixed budget escalates a fraction p.The score is the population version of the initial query-level score, using true class means and covariance.Source: §5 · p. 11
• Under shared-covariance Gaussian class distributions with distinct means, LEASH’s initial gate escalates the largest fraction of tasks the weak agent would fail among query-representation rules at the same rate.Both outcomes must occur with positive probability.Source: §5 · p. 11
• When escalation gain is nondecreasing in the score, or when strong-agent success is constant, the gate also maximizes task success rate.The result is part (ii) of Proposition 1.Source: §5 · p. 11
• The proof links the score to weak-agent failure log-odds and uses a Neyman–Pearson exchange argument to show no same-rate query-based rule catches more failures.The stated score identity holds up to an additive constant under the model.Source: §5 · p. 12
• The proposition does not analyze the runtime gate; the authors leave its optimal-stopping analysis to future work.The implication passage states that the result applies to the initial gate.Source: §5 · p. 12

• LEASH is a training-free framework supporting initial routing, runtime escalation, and continued post-handoff oversight.The conclusion states that a shared latent uncertainty signal can support both selective escalation and post-handoff oversight.Source: §6 · p. 12

Limitations

• The evaluation covers computer-use agents on OSWorld; broader generality remains unvalidated, and the score may not rank escalation benefit when both agents struggle.Post-handoff monitoring also faces distribution shift, and its intervention effect is not causally isolated.Source: Limitations · p. 12

A.2. Representations and Estimator Fitting

• Runtime representations use the last generated action token, while initial representations use the final prompt position; task-grouped validation avoids trajectory leakage when selecting ridge parameters.Ignoring task membership yielded a near-perfect AUROC of 0.997 in a preliminary check, attributed to leakage across steps from the same trajectory.Source: §A.2 · p. 19

A.3. Reference Banks and Threshold Calibration

• Reference banks come from held-out Manager-guided trajectories disjoint from evaluation tasks, and thresholds are calibrated from task-grouped out-of-fold scores.The final estimators are fitted on the full reference set after calibration; evaluation data are not used for fitting or threshold calibration.Source: §A.3 · p. 20

A.4. Post-Handoff Oversight

• Shadow probes run after handoff at the first strong-agent step and every third step, using the actual trajectory; candidate actions are logged but never executed or shown to the strong agent.At most six checkpoints can fire per episode, with further firings suppressed after two action-changing checkpoints; a firing makes the strong agent re-decide with raised reasoning effort, without seeing the weak model’s candidate action.Source: §A.4 · p. 20

A.1. Routing Details, Analyses and Proofs

• Under the shared-covariance Gaussian model, the distance score u(z) determines the weak agent’s failure log-odds, up to the class-prior term.The score is strictly increasing in failure risk.Source: §A.1 · p. 18
• At a fixed escalation rate, the initial gate thresholds u(z), which is strictly increasing with weak-agent failure risk under the Gaussian model.The proof models the initial routing decision before execution and compares routing rules using only z; its threshold-exchange argument follows the Neyman–Pearson lemma under a marginal escalation-rate constraint.Source: §A.1 · pp. 18–19
• At the same escalation rate, thresholding u(z) hands off a strictly larger fraction of weak-agent failures than random escalation and maximizes the conditional probability of escalating a failure.This comparison is between the initial gate and any rule based on z with the same escalation rate.Source: §A.1 · p. 19
• The gate also maximizes success gain only when escalation gain Δ(z) is a nondecreasing function of u(z); a constant strong-agent success probability is one condition that suffices.The proof defines Δ(z) as the strong agent’s success probability minus the weak agent’s success probability.Source: §A.1 · p. 19

B.1. Experimental Details

• Evaluation uses screenshot observations and a shared budget of 50 actions per task, with four lightweight backbones and Claude Opus 5 as Manager and strong executor.Always-on reference runs use the strong model alone, without the Manager or escalation framework.Source: §B.1 · p. 21
• The evaluated configurations are cumulative: Manager adds planning and verification, Escalation adds initial and runtime handoff, and LEASH adds post-handoff Shadow monitoring and intervention.The evaluation task pool contains 262 tasks.Source: §B.1 · p. 21
• Thresholds are calibrated from task-grouped out-of-fold scores on a held-out reference set, targeting 25% initial and 50% overall escalation; observed rates can differ from targets.The reference set is disjoint from the 262 evaluation tasks, and the reported realized rates range from 43–64%.Source: §B.1 · p. 21
• After handoff, Shadow probes the lightweight model every three strong-agent steps; its unexecuted actions provide scores that can prompt the strong agent to reassess while retaining control.The action budget is shared across the task.Source: §B.1 · p. 21
• Inference costs are estimated from recorded token usage using OpenRouter reference prices, including lightweight-agent, Manager, strong-executor, and monitoring calls.For locally served models, the reported costs are reference estimates rather than measured hardware costs.Source: §B.1 · p. 21

B.2. Uncertainty Estimation and Reference-Bank Sensitivity

• Pooled LDA provides the strongest overall discrimination in most settings among the compared uncertainty estimators, with performance varying by backbone and execution stage.Runtime AUROC measures discrimination of Manager-assigned subgoal-segment outcomes, not individual action correctness.Source: §B.2 · p. 21
• The standard deviation of initial-stage AUROC estimates falls from roughly 0.10 at 40 tasks to about 0.02 at 99 tasks as the reference bank grows.Smaller banks also provide coarser control of realized escalation rates because thresholds are quantiles of the same pool.Source: §B.2 · p. 21
• Table 8 compares cumulative gate configurations for Qwen3.5-35B-A3B; the first two disable post-handoff Shadow monitoring and checkpoint intervention.Each configuration retains the Manager.Source: Table 8 · p. 22

B.3. Additional Component Ablations

• The complete framework performs best, but its comparison with runtime-only escalation does not show a benefit from adding the initial gate in this setting.Runtime-only escalation achieves higher success than initial-only escalation, at higher cost.Source: §B.3 · p. 22

B.4. Inference Cost

• Runtime uncertainty scoring reuses action-generation hidden states without another model call; initial and Shadow probes require lightweight-model inference, and interventions may add strong-model computation.These costs are included in the total workflow cost reported in Table 1.Source: §B.4 · p. 22

C. Prompt Templates

• The prompts cover planning, verification, handoff, and checkpoint intervention; the diagnostic state query is logging-only, and the alternative think_reflect prompt is not used by the hidden-state checkpoint.The strict verification instruction is appended to the replanning system prompt, while Shadow candidate actions are neither executed nor shown to the strong agent.Source: C. Prompt Templates · p. 22

D. Case Study

• Across all three cases, the weak agent follows an unproductive GUI trajectory before handoff, and the strong agent changes strategy rather than continuing its actions.The examples illustrate runtime escalation identifying stalled or inefficient execution trajectories.Source: D. Case Study · pp. 22–23
• After handoff at step 11, the strong agent uses terminal-based python-pptx to standardize fonts in 27 actions; the Manager-only run exhausts its 50-action budget.The weak agent had repeatedly interacted with an unresponsive macro interface.Source: D. Case Study · p. 22
• After handoff at step 8, the strong agent uses keyboard navigation and multi-selection to create the requested photo archive; the Manager-only run creates an incorrect archive and scores zero.The weak agent repeatedly opened, inspected, and closed candidate photos; the strong agent completed the task in 27 actions.Source: D. Case Study · p. 23
• After 23 steps of browsing without installing the extension, the strong agent switches to terminal installation after handoff at step 24 and successfully installs LanguageTool.The Manager-only run exhausts its action budget without completing the installation.Source: D. Case Study · p. 23