Knowing When to Ask for Help: Bayesian Self-Escalation in Hierarchical LLM Agents

Nadeem Shaikh

Source-linked AI summary

Knowing When to Ask for Help: Bayesian Self-Escalation in Hierarchical LLM Agents

Nadeem Shaikh

arXiv:2608.24087v1Aug 25, 2026cs.LGcs.AIstat.ML

TL;DR

Existing LLM systems usually delegate before reasoning or after a response is complete, leaving mid-generation handoff under-specified. The paper formulates intra-generation delegation as Bayesian optimal stopping over a learned competence posterior and derives threshold policies with calibration-based guarantees. Controlled simulations confirm the theory, while a narrow Qwen2.5-Coder validation confirms two of three preregistered predictions.

Problem
Existing LLM delegation systems generally decide before inference or after full generation, rather than determining during reasoning when a weaker model should hand off.
Method
The paper models intra-generation delegation as Bayesian optimal stopping over an online competence posterior learned from labelled trajectories.
Results
Controlled simulation confirms each theoretical prediction, while a Qwen2.5-Coder 1.5B->7B code cascade confirms two of three preregistered predictions.
Takeaways & Limitations
Self-escalation’s success is tied to calibration of the competence posterior, and real-model evidence remains limited to a narrow code-generation cascade.
Takeaways & Limitations
The real-model study uses one code cascade, model pair, greedy-decoding setting, and seed, evaluates a constant rather than optimal-stopping threshold, and leaves one preregistered prediction untested.

Abstract

from arXiv · show

Current LLM agent systems decide delegation before reasoning begins (a router picks a model) or after a response is complete (a verifier scores it and may retry). We study a third regime: an agent that recognises, during its own reasoning, that it is unlikely to succeed and transfers control to a stronger model. We formulate intra-generation delegation as a Bayesian optimal-stopping problem over a learned competence posterior -- an online estimate of the agent's eventual task success whose sufficient statistics are learned from labelled trajectories, not read off raw entropy. We derive the myopic escalation threshold in closed form, characterise the optimal policy via dynamic programming, and prove that the optimal policy is a time-varying threshold with no shape assumption on the raw signal. We further prove exponential separation of the oracle belief at the Chernoff-information rate of the signal, a regret bound governed by the calibration of the posterior, and a finite-sample guarantee: with n labelled calibration trajectories the deployed plug-in policy's regret decays as 1/sqrt(n). A controlled simulation study confirms each prediction of the theory, including the predicted 1/sqrt(n) rate. We additionally report a real-model validation on a Qwen2.5-Coder 1.5B->7B code cascade (MBPP, 257 tasks), confirming two of three pre-registered predictions: the escalation frontier dominates post-hoc routing at equal cost, and the cumulative competence belief's discrimination rises over generation.

1 Introduction

The paper introduces self-escalation, in which an LLM delegates mid-generation when its learned competence estimate indicates likely failure. It formulates this decision theoretically and evaluates predictions in simulation and a real-model cascade.

Self-escalation lets a working model recognise during reasoning that it may have left its competence region and hand off to a stronger model.
Intra-generation delegation is formulated as Bayesian optimal stopping over an online estimate of eventual task success.
Raw uncertainty is treated only as evidence; the competence posterior is learned from labelled trajectories rather than read directly from entropy.
The theory derives a closed-form myopic threshold, characterises an optimal policy by dynamic programming, and proves a time-varying threshold structure without raw-signal shape assumptions.
Calibration is identified as the binding design constraint because the method’s cost is governed by posterior calibration.
A streaming algorithm, offline threshold-schedule procedure, simulation study, preregistered protocol, and Qwen2.5-Coder cascade validation are provided.

2 Related Work

The paper positions self-escalation among deferral, LLM routing, uncertainty estimation, adaptive computation, sequential testing, and optimal stopping. Its distinction is sequential delegation within one generation using a learned competence estimate.

Learning-to-defer routes rejected inputs to an external expert, while this setting makes deferral sequentially within a single generation to a stronger model.
LLM cascades and routers generally decide at whole-response granularity, often after completing a cheap generation or generating multiple samples.
Uncertainty methods provide possible evidence streams, including token entropy, next-token margins, probes, and semantic entropy, but commonly target completed-generation detection.
Adaptive computation, early exits, speculative decoding, and self-revision reallocate computation within a system but do not supply the paper’s explicit framing of mid-generation competence-based escalation.
The oracle decision core is related to finite-horizon sequential testing, while the paper applies optimal stopping to deferral from a working model to a stronger mode.

3 Problem Formulation

The formulation models a junior agent’s eventual correctness as latent and uses sequential competence signals to update a posterior before choosing continuation or escalation. Costs include computation, escalation, and wrong answers.

A junior agent generates for T steps while latent Y records whether its completed answer would be correct, with base competence π.
At each step, the agent observes a raw competence-evidence signal such as normalised token entropy or next-token margin.
The posterior is learned and calibrated from labelled trajectories, because raw uncertainty metrics are not themselves likelihood ratios or correctness probabilities.
Under conditional-i.i.d. signals with densities f0 and f1, the likelihood-ratio statistic λ(e) updates belief through a log-odds recursion.
The objective accounts for junior-step compute cost κ, escalation cost γ, and wrong-answer loss L.
The agent chooses continue or escalate; escalation hands the query and distilled context to a senior agent, while non-escalation answers locally at step T.

4 The Escalation Policy

The escalation policy compares deferral with local completion using belief-dependent costs, then incorporates the option value of further evidence through dynamic programming. The resulting policy is threshold-based.

The forward escalation cost is γ +(1−q)L, while finishing locally costs (T−t)κ +(1−b)L at belief b.
The myopic policy escalates at step t when belief B_t falls below a time-dependent threshold τ_t^myo.
Escalation becomes more attractive with a reliable senior, costly errors, cheap escalation, or many remaining junior tokens.
The myopic rule omits the option value of continuing to obtain evidence that may resolve uncertainty without escalation.
The optimal policy is obtained from a dynamic program whose transition uses the posterior-predictive signal distribution, escalating when the escalation branch minimises expected cost.

5 Theoretical Analysis

The theory establishes that Bayesian competence beliefs support threshold-based escalation, while decision regret is governed by posterior quality. It also derives exponential belief separation and finite-sample guarantees for calibrated plug-in policies.

Belief dynamics: The competence belief is a martingale, so waiting preserves expected belief while information can still improve future decisions.The option value of continuation arises from the nonlinear future value, not from increasing expected competence.
Consistency and separation rate: Oracle belief errors decay at most as A_τe^-tC, where C is the Chernoff information of the signal pair.For fixed τ, the belief converges almost surely to the true outcome indicator when C > 0.
Optimality of a threshold policy: The optimal policy escalates when the current competence belief falls below a time-varying threshold, without assumptions on raw-signal likelihood-ratio monotonicity.FOSD-monotone belief transitions and backward induction establish the threshold structure.
Calibration is the binding constraint: Posterior miscalibration controls excess cost, making calibration more important than a more elaborate router or decision rule.The regret analysis compares decisions using fitted and true beliefs at a common threshold.
Calibration is the binding constraint: Brier score is the appropriate optimization target because it penalizes calibration and refinement, whereas ECE is primarily an alarm for large errors.A small ECE does not guarantee safety when the predictor ignores information in the filtration.
Finite-sample guarantee: With n labelled calibration trajectories, plug-in regret decays at the predicted 1/√n rate, although the bound can be pessimistic for low-mass bins.The theorem assumes a finite signal alphabet and minimum class-conditional bin probabilities; direct per-step recalibration avoids compounding.

6 Algorithm

The algorithm performs streaming belief updates during one junior-model pass and escalates when the belief crosses an offline-computed threshold schedule. It supports model-agnostic text handoff and separates posterior fitting from cost-dependent threshold computation.

Streaming inference: Algorithm 1 updates the competence belief in O(1) time per token and exits early when the belief crosses the threshold schedule.Algorithm 2 computes the schedule once offline by backward induction on a discretised belief grid.
Context handoff: DistillContext hands off the partial reasoning trace, intermediate results, tool history, and retrieved evidence as text rather than hidden activations.The escalation cost γ should include tokens reread by the senior model.
Posterior fitting: The competence posterior is fitted offline from labelled correctness trajectories using class-conditional likelihood ratios or a logistic success predictor.Signals can include running mean entropy, margin trends, and spike counts.
Modularity: The factored posterior-plus-threshold design allows prices, latency budgets, or senior-model changes to be handled by rerunning backward induction.The fitted signal model need not be recollected or retrained when only decision costs change.
Theory versus production instantiation: Production deployment can replace the explicit Bayesian filter with a learned success predictor, but the formal threshold-structure guarantee is then lost.The regret analysis remains filter-agnostic, while arbitrary learned predictors need not satisfy the FOSD transition property.

7 Simulation Study

The simulation evaluates Bayesian self-escalation under specified signal and cost conditions, finding that optimal-stopping policies dominate fixed-rule and post-hoc baselines while calibration and model assumptions shape performance.

Setup: π = 0.60, T = 40, q = 0.90, and N = 40,000 define the simulation setup, with Beta signals separating eventual successes from failures.The comparison includes junior-only, senior-only, fixed-rule, selective, and Bayesian policies.
Cost–accuracy results: 96.0% accuracy at ≈0.11 cost/query requires escalation on only 40% of queries, versus 91.0% for the fixed rule and 90.1% for always escalating.The Bayesian policy also matches post-hoc selective accuracy at lower cost by aborting local generation early.
Cost–accuracy results: The Bayesian frontier weakly dominates baselines across compute budgets, while the optimal-stopping point beats unconditional escalation on both cost and accuracy.The matched-compute gap over fixed-rule routing varies with where the budget falls on the fixed rule’s steep region.
Threshold schedules: τ^myo_1 = 0.828 causes 62% of queries to escalate at the first token and 68% overall, reaching 0.934 accuracy at cost 0.129.This illustrates the predicted early over-escalation of the literal myopic schedule; the well-performing optimal-stopping schedule is parameter-free.
Threshold schedules: Interior optimal-stopping thresholds rise from 0.02 to 0.08 because the option value of another observation outweighs the κ = 0.002 delay cost in the small-κ regime.Posterior beliefs drift upward for eventual successes and downward for failures, enabling separation within a few tokens.
Calibration: 30% confidently-wrong contamination lowers accuracy from 95.2% to 85.7% and escalation from 0.47 to 0.37, identifying calibration as the binding constraint.The contaminated failures resemble successes, so they are wrongly retained locally and create the O(L) regret event.
Sample complexity: The belief error declines from 2.9 × 10^-3 at n = 25 to 2.1 × 10^-4 at n = 3200, with log–log slope −0.53 matching the predicted 1/√n rate.The realised end-to-end cost gap is already below 0.0016 in the reported calibration experiment.
Robustness: Under correlated-Gaussian misspecification, total cost is 0.158 versus 0.166 and accuracy is 95.6% versus 95.8% in the matched i.i.d. model.The method degrades gracefully: correlation narrows its edge but does not invert it.

8 Real-Model Validation

The real-model validation applies the framework to a narrow Qwen2.5-Coder 1.5B→7B MBPP cascade and supports two pre-registered predictions: streaming escalation outperforms post-hoc routing at comparable cost, while cumulative belief discrimination rises during generation.

Setup: The validation uses Qwen2.5-Coder-1.5B-Instruct and 7B-Instruct on 257 sanitized MBPP tasks, with execution-based correctness labels and a posterior fitted to cumulative signal means.Logged signals include token entropy, next-token log-probability, and top-2 margin.
Capability gap: The junior solves 62.3% of tasks and the senior 80.9%; the senior rescues 54 of 97 junior failures, while 43 failures are shared by both models.The shared-failure floor limits the escalation ceiling to 214/257 = 83.3%.
Competence signal: AUROC 0.758 shows the competence signal is informative but imperfect, with instantaneous entropy separating eventual failures from successes early before collapsing and briefly inverting mid-generation.Confidently-wrong failures and hard-but-correct successes create the middle-band inversion, showing raw entropy is not the posterior.
Cumulative belief: Spearman ρ = 0.93 links generation fraction to cumulative-belief AUROC, which rises from 0.51 to 0.76 despite non-monotonic instantaneous signals.The cumulative belief plateaus rather than declines when fresh evidence becomes uninformative, then resumes rising.
Cost–accuracy frontier: 30.2% less total compute reaches 75% accuracy with streaming than post-hoc routing, using 14,841 versus 21,273 generated tokens.At τ=0.5, streaming escalates 37% of tasks and reaches 74.7% accuracy; aborting doomed generations offsets senior calls.
Scope: The real-model evidence confirms two of three pre-registered predictions but does not test calibration sensitivity or the optimal-stopping dynamic program.The evaluated policy is a constant threshold in one code cascade, with one model pair, greedy decoding, and a single seed.

9 Protocol for Real LLM Systems

The protocol pre-registers evaluation of Bayesian self-escalation across datasets, signals, baselines, and cost–accuracy metrics, then reports its first real-model execution on a code cascade with two predictions confirmed.

Systems: The real-model validation uses a small junior model, a stronger senior model, and per-token log-probabilities exposed by the inference stack.
Signals: Token-level entropy, next-token probability margin, and semantic-entropy probes are calibrated separately to compare signal quality.
Datasets: The evaluation covers reasoning, commonsense, code, and in-domain deployment sets, with executable tests defining eventual success for code tasks.
Metrics: The study reports cost–accuracy Pareto frontiers, escalation precision and recall, belief calibration, AUROC, and decision latency including signal-computation cost.
Falsifiable predictions: The pre-registered protocol tests whether Bayesian policies dominate query-level routing and confidence cascades at equal cost, whether cumulative-belief AUROC rises with generation, and whether calibration affects accuracy gaps.Failure of any prediction would falsify the central claims.

10 Limitations

The limitations concern signal assumptions, calibration, deployment requirements, senior-model modelling, and the narrow scope of real-model validation.

Modelling assumptions: The conditional-i.i.d. signal assumption is false for real token streams, so the update is approximate and should use a sequence model instead.The threshold structure itself requires no shape assumption on signal densities.
Calibration: Confidently wrong junior predictions can defeat every threshold policy using that signal, making calibration a central practical constraint.The paper suggests combining epistemic signals with lightweight cross-model agreement.
Deployment: Token-level signals require log-probability access, restricting deployment to self-hosted or managed endpoints that expose log-probabilities.
Senior independence: Treating senior success as constant q ignores its correlation with query difficulty; a difficulty-conditioned q(x) would tighten the decision.
Scope of the real-model study: The real-model study covers one code cascade, one model pair, greedy decoding, and one seed, evaluates a constant threshold, and leaves real-model prediction (c) and other protocol datasets open.Its belief is also a running-mean proxy rather than the recursive filter.

11 Conclusion

The paper frames asking for help during generation as Bayesian self-escalation: a junior tracks eventual success and defers when escalation has higher expected utility. It provides threshold policies, calibration-based regret analysis, simulations, and released reproducibility materials.

Conclusion: Bayesian self-escalation lets a junior model track an online posterior over eventual success and defer to a stronger model when expected utility favors escalation.
Conclusion: The framework derives a closed-form myopic threshold, characterizes optimal stopping dynamically, proves a time-varying threshold structure, and links success to belief calibration.
Conclusion: Controlled simulations confirm each theoretical prediction, while the real-model code-cascade validation confirms two of three pre-registered predictions.
Reproducibility: The simulation and real-model analysis scripts are released, and the paper and simulation scripts are permanently archived at DOI: 10.5281/zenodo.21330787.