REFLEX with Jev for Efficient Selective Control in LLM Agents

1 Introduction

• LLM agents commonly use a generative model at nearly every step, although many steps require only choosing from a fixed set.This design is flexible but assigns different kinds of computation to the same model.Source: §1 · p. 1
• The paper asks whether a specialized decision model can handle bounded choices without reducing task success, and when it can outperform a cheap generative cascade.The question concerns individual decisions within an agent's trajectory.Source: §1 · p. 1
• REFLEX uses Jev for high-confidence bounded choices and delegates low-confidence states or required generation to the strong LLM.Jev returns a bounded choice and confidence score, separating bounded control from generative reasoning within an agent trajectory.Source: §1 · p. 1
• 95% success with 72.7% fewer strong-model calls than a strong-only agent on the frozen 100-task benchmark.Across Qwen3.8-Max, Kimi K3, and DeepSeek-V4-Pro fallbacks, calls fell by 66% to 72% while success stayed within a 2-point non-inferiority margin.Source: §1 · pp. 1–2
• On external BFCL tasks, function selection is 98.4% accurate, but deciding whether to call any function is only 52.0% accurate.The harder question is often whether the agent has enough authorization or information to act; evaluating only tool selection can miss control-reliability differences.Source: §1 · pp. 1–2
• A cheap LLM cascade with self-escalation remains competitive when ordinary routing is already highly accurate.The external τ-style evaluation found a 3.7-fold cost reduction against a strong-only agent, with a statistically unresolved success difference.Source: §1 · p. 2

Cost-aware LLM routing

• Cost-aware routing methods choose a model for each query, while REFLEX alternates bounded control and generative reasoning within one agent trajectory.The cited routing work includes FrugalGPT, RouteLLM, RouterBench, and LLMRouterBench; consistently outperforming simple routing baselines remains difficult.Source: §2 · p. 2

Tool-using agents and evaluation

• Tool-use benchmarks assess function selection, execution, relevance, and multi-turn interaction, while this paper uses BFCL and τ to examine sources of difficulty.BFCL tests whether ordinary tool selection limits performance; τ tests whether selective control improves complete trajectories.Source: §2 · p. 2

Confidence and selective action

• Selective prediction balances coverage and risk, and REFLEX uses confidence to execute high-confidence bounded decisions while delegating low-confidence states.The paper reports risk-coverage alongside calibration and relates its approach to confidence estimation and abstention work.Source: §2 · p. 2

3.1 Agent control as a selective decision problem

• REFLEX separates bounded control from open-ended generation, reducing strong-model calls for typed decisions while retaining the strong LLM as fallback.High-confidence executable decisions can run directly; low-confidence or non-executable states are passed to the fallback.Source: §3.1 · p. 2, Figure 1 · p. 3

3.2 Decision heads and execution semantics

• Control selects among bounded alternatives, argument synthesis supplies structured values, and generation or reasoning handles free-form text and open-ended policy questions.In the external multi-turn setting, a cheap generative executor supplies arguments and text after the control decision, with the strong model as final fallback.Source: §3.2 · p. 3
• REFLEX targets control, and evaluation separates tool-selection, argument, premature-termination, and policy errors.It also distinguishes recoverable invalid decisions from unsafe actions or terminal failures, such as extra retrieval versus an irreversible write.Source: §3.2 · p. 3
• The evaluation reports cost per episode, cost per successful episode, and strong calls per successful task to avoid rewarding apparent savings from early failure.Risk-coverage is also compared with random escalation at matched coverage.Source: §3.2 · p. 3
• A cheap generative model with self-escalation is a central baseline; REFLEX is useful only if its typed signal identifies a subset it can handle more efficiently or reliably.The comparison does not assume a specialized decision layer will outperform the cascade.Source: §3.2 · p. 3

Experimental setup

• Cross-family validation replaces the strong fallback with Kimi K3 or DeepSeek-V4-Pro while keeping Jev, threshold, tools, and evaluator fixed.Qwen3.8-Flash and Qwen3.8-Max serve as the primary small and strong generative models; Kimi and DeepSeek use the same fixed high reasoning effort across conditions.Source: §4 · pp. 3–4
• The frozen REFLEX-Sim v1.2.0 test set contains 100 held-out tasks spanning direct actions, tool selection, workflows, policy-governed actions, and ambiguous tasks.Success is based on terminal-state assertions and policy constraints rather than exact reference trajectories; the main runs had zero harness errors.Source: §4 · p. 4
• Confidence intervals use paired cluster bootstrap resampling, with McNemar tests for paired binary episode success and preregistered RF-5C contrasts.API costs are recomputed from logged token records using dated provider and region price snapshots; simulator cost is treated as benchmark overhead in the τ comparison.Source: §4 · p. 4
• The decision-complexity intervention crosses action-set sizes K ∈ {10, 25, 50} with four ambiguity levels across 60 scenario families, yielding 1,440 decisions.A matched 240-decision intervention replaces an information-seeking competitor with an irreversible-commitment competitor while holding other stated factors fixed.Source: §4 · p. 4
• External evaluation uses BFCL decisions and a τ-style held-out multi-turn validation of 300 episodes across 60 paired tasks and five arms.The τ-style evaluation is treated as external validation rather than a leaderboard submission, with native terminal-state reward as the primary outcome.Source: §4 · p. 4

5.1 REFLEX improves the tradeoff between success and computation

• At τ = 0.5, R1 achieves 95% success with 1.12 strong calls per task versus B0’s 88% and 4.10, a 72.7% call reduction.R1 matches B0’s success at τ = 0.9 and 0.95 while using 46.8% and 43.9% fewer calls, respectively.Source: §5.1 · pp. 4–5, Figure 2 · p. 5, Table 1 · p. 5
• At τ = 0.5, all 65 episodes without escalation succeed and 250 of 251 autonomous decisions are valid; replacement rates are highest for tool selection (99%) and completion (77%).The reported AURC is 0.019 versus 0.145 for random escalation at matched coverage.Source: §5.1 · p. 5

5.2 Reliability depends on decision structure

• RF-5C accuracy falls 6.7 percentage points as K rises from 10 to 50 and 5.6 points from A0 to A3; these effects are concentrated in authorization or information decisions.Embedding similarity did not consistently track procedural distance, and a leave-risk-out sensitivity interval for the cardinality estimate included zero.Source: Figure 3 · p. 5, §5.2 · pp. 5–6

5.3 Competitor type changes the consequences of errors

• Overall accuracy is statistically unresolved between write and read competitors (0.833 versus 0.842), but write yields more irreversible-commit errors (10.0% versus 1.7%) and read more deferrals (5.8% versus 12.5%).RF-5C observed deferrals were 11.5% of errors (a lower bound), revising the exploratory general deferral claim.Source: Figure 4 · p. 6, §5.3 · p. 6

5.4 External benchmarks show where the benefits are limited

• On BFCL, function selection is 98.4% accurate versus 52.0% for deciding whether to call; on τ 2-bench, REFLEX costs 3.7× less than B0, while a cheap cascade remains competitive.The τ 2-bench success difference between REFLEX and B0 is statistically unresolved; neither comparison establishes equal success.Source: §5.4 · pp. 6–7, Figure 5 · p. 7

5.5 Strong-call reductions persist across fallback families

• Strong-model calls fall 66.1% to 71.9% across fallback families; confidence-bound non-inferiority passes for Qwen and DeepSeek, but not Kimi, while strong-token use falls 50.6% to 64.1%.Kimi’s strong-only arm succeeds on 100% of tasks versus 98% for REFLEX; the preregistered point-estimate rule was not a formal non-inferiority test.Source: §5.5 · pp. 7–8, Figure 6 · p. 8

5.6 What does the gate identify?

• The fixed gate selects the same 66-task autonomous subset across fallback families, with all 66 succeeding; during escalated episodes Jev still handles 21.3% to 26.8% of steps without a strong-model call.The trace audit identifies no incorrect autonomous decision among the audited failures.Source: §5.6 · p. 8, §5.5 · p. 8, Table 5 · p. 9

• A stable 66% subset is solved autonomously with zero failures across three fallback families, reducing computation without reproducing the fallback’s full reasoning behavior.Source: §6 · p. 9
• BFCL and τ 2-bench indicate limited router headroom when tool selection is near ceiling, while a cheap self-escalating cascade remains competitive.Source: §6 · p. 9
• Near authorization boundaries, larger candidate sets and structural ambiguity reduce reliability, while changing a near-valid competitor shifts commitment-versus-deferral errors without significantly changing accuracy.Source: §6 · p. 9
• Specialized decision layers are most suitable when trajectories contain repeated bounded choices, confidence can separate direct execution from escalation, and benchmarks account for control uncertainty.Source: §6 · p. 9
• Overall agent success can hide whether failures arise from control, arguments, or downstream reasoning, and near-perfect routing cannot establish a new layer’s value.Source: §6 · p. 10
• Agent benchmarks should annotate the correct action and why it is currently valid, including required evidence, authorization, reversibility, and whether more information is warranted.Source: §6 · p. 10

7 Conclusion

• REFLEX reduces strong-model calls by roughly two-thirds across Qwen, Kimi, and DeepSeek fallbacks, with observed success differences ranging from a two-point loss to a four-point gain.Source: §7 · p. 10
• No success decrease is statistically significant, but only Qwen and DeepSeek meet the confidence-bound non-inferiority criterion in these runs.Source: §7 · p. 10
• Controlled interventions identify authorization-dependent commitment decisions, rather than ordinary tool selection, as the main reliability constraint.Source: §7 · p. 10

Limitations

• Because the study uses one proprietary typed decision model, it does not establish whether other nongenerative decision models behave similarly.Source: Limitations · p. 10
• The RF5C action-set and ambiguity effects are concentrated in authority and risk families, while τ evaluation is a 300-episode held-out validation rather than a multi-trial leaderboard evaluation.Source: Limitations · p. 10

A Hierarchical Routing Ablation

• The ablation tested selecting a tool family before selecting a tool within it, reducing second-stage alternatives while adding a family-selection decision.Source: A Hierarchical Routing Ablation · p. 11
• The initial smoke comparison was confounded by missing family descriptions, incomplete candidate descriptions, and omitted candidates, so it did not support a clean routing-structure comparison.Source: A Hierarchical Routing Ablation · p. 11
• After implementation repairs, hierarchical accuracy improved by 5 to 10 percentage points relative to the broken implementation, but this does not establish an advantage over corrected flat routing.Source: A Hierarchical Routing Ablation · p. 11
• Corrected flat routing was already near ceiling on well-posed probes, and splitting decisions introduced potentially ambiguous family assignment, so hierarchy was treated as a negative ablation.Source: A Hierarchical Routing Ablation · p. 11

B.1 Selective reliability and confidence

• At τ = 0.5, 250 of 251 autonomous decisions were valid, while all 65 no-escalation episodes succeeded.The single invalid decision was recoverable; increasing the threshold reduced decision coverage from 0.691 to 0.439.Source: §B.1 · pp. 11–12, Table S1 · p. 13

B.2 RF-5C design and statistical contrasts

• Significant marginal decreases occurred with larger action sets and greater ambiguity, but their interaction was not significant.The leave-risk-out sensitivity interval included zero, so the aggregate cardinality result depends on the decision families in the panel.Source: §B.2 · p. 12, Table S2 · p. 13

B.3 Matched intervention on competitor type

• Read alternatives produced fewer irreversible-commit errors and more deferral errors than write alternatives, while the accuracy difference remained unresolved.The intervention held gold action, competitor count, procedural distance, and semantic similarity fixed.Source: §B.3 · p. 12, Table S3 · p. 14

B.4 BFCL function selection and relevance

• BFCL function selection was 0.984 accurate, compared with 0.520 accuracy on the separate relevance evaluation.Accuracy changed by −0.0033 between K = 64 and K = 2; none of the incorrect selections chose an injected distant distractor.Source: §B.4 · p. 12, Table S4 · p. 14

B.5 Paired comparisons on τ 2-bench

• On τ 2-bench, the reported success differences between REFLEX and B0 or B3 had intervals including zero.The B0 comparison can be reported separately as lower episode cost, but the success difference remains uncertain.Source: §B.5 · p. 13

B.6 Cross-family calls, tokens, and costs

• Strong-call reductions were 71.9% for Qwen, 71.4% for Kimi, and 66.1% for DeepSeek.Strong-token reductions were 58.9%, 64.1%, and 50.6%, respectively; verified cost reduction is reported only for Qwen, at 52.2%.Source: §B.6 · p. 13

C.1 Corrections before the final analyses

• Development audits identified evaluator and task-input issues, which were addressed through evaluator changes, task repairs, and more complete agent inputs.Offline grading corrections used recorded trajectories rather than regenerating model outputs.Source: §C.1 · pp. 13–14

C.2 Paired outcomes on REFLEX-Sim

• 95 REFLEX tasks succeeded versus 88 for B0; the paired test on nine discordant tasks gives p = 0.0391.REFLEX gained eight tasks B0 failed and lost one task B0 succeeded on.Source: §C.2 · p. 14

C.3 RF-5C error concentration and exploratory deferrals

• 192 of 208 RF-5C errors were risk or clarification errors, while retrieval, ordinary tool selection, and workflow had no errors across 720 decisions.The exploratory RF-5 study recorded 59.8% deferrals among errors, compared with 11.5%—a lower bound—in RF-5C; these different settings are not a matched rate comparison.Source: §C.3 · pp. 14–15

C.4 Procedural distance and embedding similarity

• Embedding similarity did not consistently order procedural validity: 22.2% of far distractors exceeded the pooled near-competitor median similarity of 0.322.The median similarity for two-change competitors was higher than for one-change competitors despite their greater procedural distance.Source: §C.4 · p. 15

C.5 Failure categories on τ 2-bench

• Of 300 τ 2-bench episodes, 255 succeeded and 45 failed; the audit found zero control-selection and zero argument-generation errors.Forty-one failures were terminal database-state mismatches and four were communication failures; the audit does not establish that routing cannot affect other tasks or deployments.Source: §C.5 · p. 15

C.6 Non-inferiority and decisions within escalated episodes

• Qwen and DeepSeek meet the lower-95%-bound criterion for non-inferiority at −0.02, but Kimi does not.All three families meet the preregistered point-estimate rule; that rule is not equivalent to the confidence-bound test.Source: §C.6 · p. 15
• Jev handles 26.2% of steps in Qwen episodes, 26.8% in Kimi episodes, and 21.3% in DeepSeek episodes without reaching the strong model.These measurements are within episodes that escalated at least once, not autonomous-task coverage or GMR.Source: §C.6 · pp. 15–16
• None of the 21 failed REFLEX episodes contained an incorrect autonomous decision; the trace audit attributed the observed failures to fallback handling after escalation.This attribution applies to these runs and does not imply autonomous decisions are universally error-free or escalation always resolves a task.Source: §C.6 · p. 16

C.7 Variation between repeated executions

• REFLEX retained 0.95 success in both executions, but four successes were lost and four gained; the B0 comparison changed from 0.88 to 0.91.The two executions differ in Jev retry settings, and two runs do not establish general run-to-run variance or a fixed uncertainty band.Source: §C.7 · p. 16

D Extended Cost Results and Claim Register

• At τ = 0.5, REFLEX costs $0.00763 per task and $0.00803 per successful task, versus $0.01508 and $0.01713 for B0.Strong calls per successful task decrease from 4.66 to 1.18.Source: D Extended Cost Results and Claim… · pp. 16–17
• REFLEX-Sim cost estimates exclude external user-simulator overhead, and per-condition success intervals were unavailable in the frozen output.Costs were reconstructed from frozen token counts using the pinned price table; paired-difference intervals were not substituted for the missing intervals.Source: D Extended Cost Results and Claim… · p. 17, Table S14 · p. 18
• Four of the 12 tested claims were retired, including the general deferral tendency and superiority over the cheap self-escalation baseline.A retired claim was unsupported by the corresponding evidence, not established as false.Source: D Extended Cost Results and Claim… · p. 17, Table S15 · p. 19
• The repeated-execution comparison found a three-point B0 success change, while the same-condition REFLEX runs retained 0.95 success but changed which tasks succeeded.The two runs do not estimate general run-to-run variance or justify a fixed uncertainty band.Source: Table S13 · p. 18