PaperlayerView exact revision on arXiv

Source-linked AI summary

CASEL: Consequence-Aware Selective Execution for Reliable Long-Horizon Tool-Using Agents

Yanyan Zhang, Fei Shen, Ting Yan, Hao Zhang

arXiv:2610.36570v1cs.AIcs.CRcs.SE

TL;DR

Tool-using agents face an unresolved allocation question: how much scrutiny should precede an action that changes the world? CASEL selects among interventions using consequence-weighted risk and reversibility. In a controlled simulation with equal menus and costs, it reduced harmful-side-effect trajectories relative to uncertainty ranking, but the evaluation assumes intervention effectiveness.

  • Problem

    Existing safeguards leave open how much scrutiny to spend before a particular action changes the world, and uncertainty-only or mutation-only gates can treat actions with different consequences or reversibility alike.

    Source: §I · p. 1

  • Method

    CASEL is a runtime allocation policy that estimates failure probability, consequence, reversibility, ambiguity, and evidence staleness to select among intervention modes.

    Source: §I · p. 1

  • Results

    With the same menu and cost, CASEL lowers harmful-side-effect trajectories from 14.75% for Uncertainty-Menu to 7.02%, and mean normalized harm from 0.184 to 0.123.

    Source: B. Main Results · p. 4

  • Takeaways & Limitations

    In controlled simulation with equal menus and costs, consequence-weighted risk and reversibility roughly halve severe side effects relative to uncertainty ranking, while a trajectory budget did not help.

    Source: §IX · p. 6

    CASEL is an allocation policy, not a safety guarantee; the simulation ranks allocation rules under assumed intervention effectiveness and costs, not their deployment effect.

    Source: §VIII · p. 6

Abstract

from arXiv · show

Tool-using language agents increasingly modify external systems, yet most runtime safeguards allocate scrutiny by generic uncertainty scores or coarse mutation heuristics. We introduce CASEL (Consequence-Aware Selective Execution Layer), a model-agnostic runtime allocation policy that combines calibrated failure probability, state-dependent consequence, reversibility, ambiguity, and evidence freshness to choose, for each proposed tool call, among direct execution, verification, sandboxing, clarification, approval, and abstention by minimizing a per-mode cost. We formalize direct execution as selective prediction and state and test the validity conditions of a finite-sample threshold and a trajectory risk ledger. In a controlled simulation over 20 seeds with 120,000 test actions each, in which every method has the same intervention menu and the same compute cost, CASEL reduces harmful-side-effect trajectories from 14.75 percent for an uncertainty-ranked gate to 7.02 percent and mean normalized harm from 0.184 to 0.123. A gate ranked by the product of failure probability and consequence reaches similar harm (5.82 percent, 0.120) but sends 45 percent more actions to human approval, and removing reversibility-aware routing raises the rate to 15.53 percent. A trajectory-level risk budget, in contrast, increases harm at equal cost (11.82 percent). Intervention effectiveness is assumed, not measured: the results rank allocation rules under the simulator's assumptions and do not establish deployment effectiveness.

I. INTRODUCTION

Runtime safeguards leave open how much scrutiny to spend before a particular action changes the world. CASEL allocates that scrutiny using consequence-weighted risk and action reversibility, while treating existing safety mechanisms as intervention options.

I. INTRODUCTION

  • CASEL addresses the allocation question left open by existing safeguards: how much scrutiny should precede a particular action’s effect on the world?Uncertainty-only gates treat equally risky read-only and irreversible actions alike, while mutation-only gates conflate reversible edits with irreversible commitments.Source: §I · p. 1
  • The runtime layer estimates failure probability, consequence, reversibility, ambiguity, and evidence staleness, then selects a graded intervention for each proposed tool call.Direct execution is framed as selective prediction; scrutiny is meant to scale with consequence-weighted risk and match action reversibility.Source: §I · p. 1
  • CASEL’s six-mode objective exercises five modes, states validity conditions for a finite-sample direct-execution threshold, and evaluates a trajectory risk ledger.The introduction also describes a controlled evaluation with shared intervention menus and costs, ablations, stress tests, and a negative result for trajectory budgeting.Source: §I · p. 1

II. RELATED WORK

  • Prior work strengthens agents, studies tool safety, and formalizes selective prediction, while CASEL treats these mechanisms as branches in a per-call allocation policy.The related-work discussion distinguishes orthogonal approaches that call a stronger model or score whole action sequences from CASEL’s per-tool-call decision.Source: §II · pp. 1–2

III. PROBLEM FORMULATION

The formulation treats a proposed tool call as potentially invalid relative to the task, policy, tool contract, or environment. It estimates its risk and selects a feasible execution mode by balancing residual harm against intervention costs.

III. PROBLEM FORMULATION

  • An action fails when it is invalid relative to the task, policy, tool contract, or environment state, and its failure probability is conditioned on history, action, and state.The state at commitment is only partially observed through the history.Source: §III · p. 2
  • The one-step expected harmful loss is r_t = p_t c_t, where c_t is normalized consequence if the action is wrong.Runtime routing uses estimated quantities; hatted p_t, c_t, and r_t are estimates, while λ and μ weight compute and interaction costs.Source: §III · p. 2
  • Before commitment, the controller chooses among direct, verify, sandbox, clarify, approve, and abstain, minimizing expected residual harm plus compute and interaction costs over feasible modes.The feasible set excludes modes unavailable for the action’s reversibility class.Source: §III · p. 2
  • Linearity of expectation permits cumulative risk to be tracked with a ledger without assuming independence, and the paper tests whether budgeting that risk helps.Source: §III · p. 2

A. Structured Risk Signals

CASEL constructs structured estimates of action consequence, reversibility, ambiguity, and evidence freshness, then uses them to route each proposed call among feasible interventions.

A. Structured Risk Signals

  • A raw failure score is calibrated on a held-out split, and the algorithm estimates consequence, reversibility, ambiguity, and staleness before computing estimated risk.The algorithm clarifies on high ambiguity, verifies stale evidence when refresh is available, and removes direct execution above its risk threshold.Source: A. Structured Risk Signals · pp. 2–3
  • The table’s stated unit of comparison is one decision per proposed tool call.Source: Table 1 · p. 3
  • Estimated consequence combines frozen per-tool severity from auditable metadata with a bounded scope factor for the magnitude of the affected action.The metadata can include write access, privilege, data sensitivity, and value transfer; scope can reflect records, amount, or rollback cost.Source: A. Structured Risk Signals · p. 3
  • Reversibility changes which modes are feasible: sandboxing requires a reversible mutation or dry-run interface, while approval applies to compensable and irreversible actions.Direct, verify, clarify, and abstain are always feasible.Source: A. Structured Risk Signals · p. 3
  • Ambiguity captures missing intent that more sampling cannot recover and routes to clarification; stale evidence triggers a re-read.For GUI agents, perception errors can propagate into the failure score and staleness signal.Source: A. Structured Risk Signals · p. 3

B. Adaptive Intervention

Adaptive intervention estimates residual error for each feasible mode and chooses the mode with the lowest expected harm and cost.

B. Adaptive Intervention

  • A mode’s residual failure estimate is γ_m times the estimated failure probability, with γ_m reflecting mode effectiveness and step features.The simulator instead uses one constant γ_m per mode.Source: B. Adaptive Intervention · p. 3
  • The per-mode cost is J(m) = γ_m r̂_t + λk(m) + μq(m), combining residual estimated harm with compute and interaction costs.Source: B. Adaptive Intervention · p. 3
  • Ignoring interaction cost, an intervention beats direct execution when its expected harm reduction V_m exceeds its compute cost λk(m).V_m = (1 − γ_m)r̂_t.Source: B. Adaptive Intervention · p. 3

V. SELECTIVE-RISK CONTROL

The section treats direct execution as selective prediction: choose the largest-coverage set whose risk is at most a target. A finite-sample threshold can certify that risk under specified calibration conditions, but the bound applies to a defined execution set.

A. Direct-Execution Set and Finite-Sample Threshold

  • The objective is to maximize direct-execution coverage subject to conditional risk R(τ) ≤ ϵ.Here, R(τ) is expected harmful loss among records in the threshold set Aτ.Source: A. Direct-Execution Set and… · p. 3
  • For a fixed scorer and K-threshold grid selected before calibration, the rule certifies R(τ) ≤ ϵ with probability at least 1 − δ for qualifying thresholds.The calibration records must be i.i.d. from the deployment distribution and disjoint from the data used to fit the calibrator.Source: A. Direct-Execution Set and… · p. 3
  • The guarantee covers the complete policy’s directly executed set, not a further-selected subset that could concentrate high-loss actions.Because actions within a trajectory are dependent, calibration uses one record per trajectory.Source: A. Direct-Execution Set and… · p. 3

B. Trajectory Ledger

The trajectory ledger blocks direct execution when accumulated estimated risk would exceed a budget. Its expected-harm bound depends on upper-bounding true conditional harm at each directly executed step, a condition the paper says a calibrated point estimate does not ensure.

B. Trajectory Ledger

  • The ledger accumulates estimated risk only for directly executed actions and blocks a direct call that would exceed budget BH.The offline threshold rule selects a complete-policy direct set; online, direct execution also requires ˆrt ≤ ˆτ and direct to minimize J(m).Source: B. Trajectory Ledger · pp. 3–4
  • The expected direct-execution harm is bounded by BH only if each direct step’s estimated risk upper-bounds its true conditional harm and the charged total stays within budget.The result follows under the stated measurability and almost-sure budget conditions.Source: B. Trajectory Ledger · p. 3
  • The ledger is conditional accounting, not a reliability guarantee: calibrated point estimates need not upper-bound true conditional harm, and intervened actions’ residual harm is excluded.Source: B. Trajectory Ledger · p. 3

VI. CONTROLLED EVALUATION

The controlled evaluation compares routing policies in a simulated action stream with calibrated failure scores, noisy consequence estimates, and matched compute cost. Its protocol specifies action classes, feasible interventions, and trajectory-based outcome measures.

A. Protocol

  • Each seed uses a 40,000-action calibration split and 6,000 test trajectories of 20 actions, totaling 120,000 test actions.Actions are 58% read-only, 28% reversible, and 14% irreversible; consequences mainly span class-dependent ranges.Source: A. Protocol · p. 4
  • The simulator assigns residual multipliers of 1.00, 0.42, 0.15, and 0.04 to direct, verify, sandbox, and approve, respectively, as assumed constants.Clarify is exercised only in the ambiguity stress family; abstain and the staleness branch are not exercised.Source: A. Protocol · p. 4
  • Menu gates rank actions by estimated failure probability, consequence, or their product, while CASEL minimizes J(m); policies are matched to compute cost 0.486.Matched costs are set using scores but no outcome labels; Base, Always-Verify, and Mutation-Gated are fixed policies.Source: A. Protocol · p. 4
  • Task success counts trajectories without failed actions, while HSR counts trajectories with a failed action whose true consequence is at least 0.45.MNH is defined over trajectories; outcomes are averaged across 20 seeds.Source: A. Protocol · p. 4

B. Main Results

CASEL’s main results show lower harmful-side-effect rates and normalized harm than an uncertainty-ranked gate at the same menu and cost, while consequence-aware alternatives and ablations reveal trade-offs in approvals, reversibility, and risk budgeting. The section also reports improved calibration and better selective-risk ranking from combining failure probability with consequence.

B. Main Results

  • Isotonic calibration lowers mean ECE by 94.5% and Brier score by 9.3% on test actions.Figure 2 compares raw and calibrated scores; bars show means over 20 seeds with 95% CIs, and lines show paired per-seed values.Source: B. Main Results · p. 4, Figure 2 · p. 4
  • At the same menu and cost, CASEL lowers HSR from 14.75% for Uncertainty-Menu to 7.02% and MNH from 0.184 to 0.123, while task success is 1.79 points lower.The paired HSR difference is −7.73 points (95% CI [−7.87, −7.60]); the authors attribute lower task success to generic success weighting every error equally.Source: B. Main Results · p. 4
  • Risk-Menu reaches HSR 5.82% and MNH 0.120 versus CASEL’s 7.02% and 0.123, but sends 12.6% of actions to approval versus 8.7%.The authors report similar harm with about 30% fewer approvals and slightly higher task success for CASEL across the cost range.Source: B. Main Results · p. 4

C. Risk–Coverage Quality

  • Across coverage 0.1–1.0, ranking direct actions by ˆpˆc yields selective-risk AUC 0.00533, versus 0.01050 for ˆp and 0.00593 for ˆc.The product reduces AUC by 49.2% relative to uncertainty and 10.1% relative to consequence.Source: C. Risk–Coverage Quality · p. 4

D. Ablations

  • Restricting CASEL to verification raises HSR from 7.02% to 15.53% and MNH to 0.194 at the same cost.The authors state that matching interventions to commitment semantics accounts for most of the gain.Source: D. Ablations · p. 4
  • At the same compute cost, CASEL+Budget has HSR 11.82% and MNH 0.167 and is dominated across its BH sweep.A full ledger escalates low-risk actions late in a trajectory, where expected harm is small.Source: D. Ablations · p. 4
  • Using the raw score raises HSR by 1.20 points and MNH by 0.005, while task success rises by 0.82 points.All ten differences in Figure 4(c) are significant (Holm-corrected p < 10−4).Source: D. Ablations · p. 4

E. Checks of the Guarantees and Stress Tests

The section checks consequence-aware action ranking and finite-sample threshold behavior, then tests trajectory budgeting, ambiguity handling, and misspecification. The ledger’s premise often fails, and stress-test outcomes depend on the simulator’s assumptions.

  • At ϵ = 0.014, the threshold binds in all 20 seeds (coverage 62.3%, risk 0.0035, cost 0.553); with 160,000 records, it binds at ϵ = 0.009.True selective risk never exceeded ϵ.Source: E. Checks of the Guarantees and Stress… · pp. 4–5
  • Ranking actions by ˆpˆc halves selective-risk AUC relative to uncertainty-only ranking.Source: Figure 3 · p. 5
  • With equal menus, consequence-aware allocation has far lower HSR than uncertainty ranking at every cost; CASEL matches a risk-ranked gate with fewer approvals, while trajectory budgeting does not help.Source: Figure 4 · p. 5
  • 54% of directly executed steps violate the ledger premise; realized direct-execution harm exceeds budget by 10% at BH = 0.04 and 2% at 0.08, while intervened residual harm is uncovered.Source: E. Checks of the Guarantees and Stress… · p. 5
  • With intent missing in 10% of actions, CASEL lowers HSR from 19.1% to 8.4% and clarifies 9.8% of actions at τg = 0.2.Source: E. Checks of the Guarantees and Stress… · p. 5
  • Halving estimated failure probability, consequence, or intervention effectiveness raises CASEL HSR to 16.6%, 18.8%, or 21.0%, respectively, but the ordering remains stable.Absolute harm depends on intervention effectiveness and unbiased estimates.Source: E. Checks of the Guarantees and Stress… · pp. 5–6

VII. DEPLOYMENT CONSIDERATIONS

The paper outlines domain adaptation and integration with existing tools, while describing its evidence as a controlled-simulation comparison rather than deployment validation.

VII. DEPLOYMENT CONSIDERATIONS

  • Deployment requires domain-specific signals and recalibration; CASEL can intercept tool calls without retraining and reuse existing verification, sandbox, approval, and clarification tools.Source: §VII · p. 6

VIII. DISCUSSION AND LIMITATIONS

  • The simulation ranks allocation rules under assumed intervention effectiveness and costs; it does not establish deployment effectiveness.Source: §VIII · p. 6
  • Evaluations estimating real verifier and rollback effectiveness remain planned, and the paper was not experimentally compared with REFLEX or CA2I.Source: §VIII · p. 6

IX. CONCLUSION

  • In controlled simulation with equal menus and costs, consequence-weighted risk and reversibility roughly halve severe side effects versus uncertainty ranking; trajectory budgeting did not help.Source: §IX · p. 6

Related papers

Loading 2610.36570v1…