PaperlayerView exact revision on arXiv

Source-linked AI summary

LLMs Can Predict Failure Risk, But Struggle to Predict Which Collaboration Protocol Pays Off: Cost-Aware Protocol Routing Across Reasoning Tasks

Chih-Hsuan Yang, Jingyan Jiang, Cheng-Hau Yang, Vikram Vasudevan, Huihuo Zheng, Venkatram Vishwanath, Rajeev Thakur

arXiv:2608.14927v1cs.AIcs.CLcs.LG

TL;DR

The paper asks whether LLM systems can determine when collaboration is worth its cost and which protocol to use. Across matched protocols and benchmarks, failure risk supports initial escalation, but protocol-specific cost-aware routing remains unresolved.

  • Problem

    The study distinguishes predicting Baseline failure from identifying which stronger collaboration protocol justifies its added cost.

  • Method

    The authors compare four matched solving protocols with fixed solvers, routers, and confidence probes across benchmarks and model families.

  • Results

    Across evaluations, failure-risk estimation supports initial stay-or-escalate decisions, while protocol-specific collaboration-value prediction remains weak.

  • Takeaways & Limitations

    Same-model confidence can support initial escalation, but selecting among collaboration protocols remains an unresolved cost-aware control problem.

  • Takeaways & Limitations

    The study measures cost mainly through logged tokens, while latency, price, energy, parallelism, and broader answer quality may change deployment preferences.

Abstract

from arXiv · show

Multi-agent large language model (LLM) systems can improve reasoning by spending more computation, but deployment requires deciding when extra collaboration is worth its cost. We isolate this decision by running every problem under four protocols while holding the solver fixed within each setting: direct solving (Baseline), iterative self-correction (Single), planner-executor-reviewer collaboration (PER), and multi-agent deliberation (Broadcast). The primary benchmark comprises 4,181 competition-level math problems; paired robustness checks cover four benchmarks spanning competition math, biology, and broader science with two solver families. Across fixed policies, trained routers, and frozen LLM routers, conservative policies under-escalate, whereas higher-solve frozen routers often over-escalate. A post-answer, pre-collaboration gpt-oss-120b probe ranks Baseline failures with 0.8847 AUROC (4,151 parseable cases; 95% CI [0.8732, 0.8955]). The same score remains informative for predicting whether any collaboration helps (0.7683 AUPRC), but is much weaker for identifying PER- or Broadcast-specific value (0.1674 and 0.1041 AUPRC). Separately, the pre-answer self-confidence gate reaches 78.0% solve at 45K tokens, compared with 73.8% at 71.3K for a frozen gpt-oss-120b router and 92.4% for a retrospective fixed-order oracle. Across 10 paired model-condition settings, the oracle adds 23.2-58.3 points of retrospective coverage over Baseline, but protocol profiles vary by task. In the six settings with held-out router evaluations, oracle gaps remain 18.5-28.9 points. Confidence can therefore support initial escalation, while protocol-specific cost-aware routing remains unresolved.

1 Introduction

The study isolates the cost-aware decision of whether to escalate and which collaboration protocol to use by holding the solver fixed while evaluating matched outcomes across four protocols. It distinguishes Baseline failure-risk prediction from protocol-specific collaboration-value prediction, finding asymmetric routing errors and unresolved protocol-specific value estimation.

  • Motivation: Stronger collaboration protocols can improve accuracy but cost an order of magnitude more, making both escalation and protocol selection deployment decisions.The study contrasts direct solving with self-correction and multi-agent deliberation under sharply different computation costs.
  • Study design: 4,181 competition-level math problems were evaluated under four protocols with matched solve and cost outcomes, enabling retrospective fixed-order-oracle analysis.The benchmark holds the solver fixed within each setting and observes every protocol on every problem.
  • Routing behavior: Cost-conservative policies under-escalate, whereas higher-solve frozen LLM routers often over-escalate.This directional asymmetry is revealed by evaluating realized solve and cost outcomes rather than routing accuracy alone.
  • Prediction targets: Failure-risk prediction asks whether Baseline will fail, while collaboration-value prediction asks whether and which stronger protocol justifies its added cost.The post-answer probe supports the first target, but precision degrades for increasingly protocol-specific value targets.
  • Contributions: The matched design changes only collaboration protocol within each solver setting, separating protocol value from changes in base-model capability.Paired outcome checks extend across four benchmarks spanning competition math, biology, and broader science.

2 Task, Benchmark, and Metrics

The study evaluates cost-aware protocol routing on 4,181 competition-level math problems by comparing four protocols under a shared solver and measuring solve rate, token use, and oracle-relative excess. Routers choose among protocol actions before observing outcomes, while a retrospective fixed-order oracle provides a matched diagnostic upper bound rather than a deployable policy.

  • Protocol routing: Routers select BASELINE, SINGLE, PER, BROADCAST, or None before observing protocol outcomes.Fixed policies are degenerate routers, while learned and frozen-LLM routers use problem text and/or allowed metadata; None abstains and can be an oracle label or router prediction.
  • Benchmark: 4,181 competition-level math problems form the primary benchmark, using the clean exact-answer Omni-MATH 2 subset.Router-visible metadata includes source, domain path, numeric difficulty, and a ten-level difficulty tier.
  • Protocols: Four protocols run each problem with the same gpt-oss-120b solver: BASELINE, SINGLE, PER, and BROADCAST.BASELINE uses direct solving; SINGLE adds iterative self-correction; PER uses planner, executor, and reviewer roles; BROADCAST uses multi-agent deliberation with peer approval.
  • Oracle: The fixed-order oracle labels the first successful action under BASELINE < SINGLE < PER < BROADCAST, assigning None when all four fail.It is a matched, single-realization diagnostic upper bound rather than a deployable policy or per-instance minimum-token oracle.
  • Router evaluation: The main learned-router comparison uses a stratified 80/10/10 split containing 3,342 training, 416 validation, and 423 test problems.Tier-majority is a metadata-only sanity check, while learned routers are five-class logistic regressions with optional TF–IDF word uni- and bigrams.
  • Metrics: Evaluation reports solve rate, average tokens, and excess tokens relative to the realized oracle, with under- and over-escalation defined by deviations from its chosen action.Choosing any protocol when the oracle is None counts as over-escalation, and all such tokens are excess.

3 Routing Errors and Cost Tradeoffs

Routing policies face a cost–coverage tradeoff: cheap references improve on Baseline without matching stronger routers, while higher solve rates require costly escalation and remain below the fixed-order oracle.

  • Cheap references: 65.0% solve at 28.9K tokens makes Tier-majority a stronger cheap reference than Baseline’s 56.3% at 18.2K.Metadata-only and text+metadata logistic routers reach 60.8% and 61.2%, respectively, below Tier-majority.
  • Frozen routers: 73.8% solve at 71.3K tokens is achieved by the frozen gpt-oss-120b router, rising to 78.3% at 88.6K tokens with numeric protocol costs and routing examples.Larger cross-family frozen routers reach 80–83% solve while spending 156K–280K tokens; no evaluated router approaches the fixed-order-oracle operating point.
  • Directional errors: 27.4% under-escalation and 12.5% over-escalation characterize Tier-majority, whereas gpt-oss-120b shifts these rates to 18.0% and 33.3%.Llama and Gemma frozen routers reduce under-escalation to 6–11% while over-escalating on 63–71%.
  • Directional errors: Router gains are not interchangeable: conservative policies miss recoverable problems, while aggressive routers buy solves through costly escalation.The comparison includes fixed and heuristic policies, trained routers, frozen LLM routers, a cost-aware prompt, and the confidence policy.

4 Failure Risk Is Not Protocol Value

Failure-risk estimation can identify likely Baseline errors, but it is substantially less effective at determining which collaboration protocol will repay its cost. A pre-answer confidence gate offers a useful binary escalation policy, while protocol-specific value estimation remains unresolved.

  • Post-answer failure risk: 0.8847 AUROC: the post-answer failure-risk probe ranks likely Baseline failures among 4,151 parseable cases.The probe uses only the problem, allowed metadata, and the model’s Baseline final answer; 30 unparseable outputs are excluded.
  • Pre-answer confidence: 0.859 AUROC: the pre-answer confidence probe measures single-pass solve probability before the model sees a Baseline answer.This estimate comes from 329 usable cases in the primary 423-example test split and represents a distinct operating point from the post-answer probe.
  • Protocol-specific value: 0.1674 and 0.1041 AUPRC: failure-risk prediction weakens sharply for PER- and Broadcast-specific value.The score remains informative for whether any protocol improves on Baseline, but it is not a reliable full protocol selector.
  • Binary escalation: 78.0% solve at 45.0K tokens: the self-confidence gate outperforms the frozen gpt-oss-120b router’s 73.8% at 71.3K tokens on cost efficiency.The gate keeps Baseline at confidence ≥70 and otherwise chooses Single; overlapping solve intervals do not establish a higher solve rate.
  • Remaining limitation: 14.4 points: the confidence gate’s gap to the fixed-order oracle shows that initial escalation does not solve marginal protocol-value estimation.The gate does not choose PER, Broadcast, or None, leaving the value of each expensive protocol to be estimated after initial escalation.

5 Matched Breadth and Router Robustness

Across four benchmarks and two solver families, stronger collaboration has recoverable value in all 10 paired settings, but protocol benefits vary by task. Held-out routers improve over Baseline yet remain substantially below the retrospective oracle, while Broadcast and PER are not interchangeable.

  • Matched breadth: Four benchmarks span competition math, engineering entrance-exam STEM, college-level science, and biology multiple-choice tasks.The matched evaluation includes OmniMath 2, JEEBench, SciBench, and two LAB-Bench no-tool prompt conditions, evaluated with gpt-oss-120b and Gemma-4-31B-it.
  • Matched breadth: 23.2–58.3 points: fixed-order-oracle coverage exceeds Baseline in all 10 paired settings.Broadcast is strongest in nine settings, while PER exceeds Broadcast by 2.7 points for Gemma on SciBench.
  • Router robustness: 7.2–37.5 points: the trained router improves over Baseline in all six held-out-router settings.Its gains remain 18.5–28.9 points below the fixed-order oracle, with mixed gains relative to Tier-majority.
  • Protocol-specific value: 30.8–44.6 points: Broadcast exceeds PER on LAB-Bench when Baseline and Single both fail.The corresponding OmniMath gap is 9.6–10.7 points, but PER-only successes occur in every setting at 2.4–11.1% of these conditional subsets.

6 Discussion and Conclusion

The study distinguishes failure-risk estimation from collaboration-value estimation: confidence can support an initial stay-or-escalate decision, but cost-aware selection among collaboration protocols remains unresolved.

  • Discussion and Conclusion: Failure-risk estimation helps decide whether to remain with a cheap direct solver, whereas collaboration-value estimation must identify which expensive workflow provides sufficient marginal benefit.The matched design separates these decisions because aggregate routing accuracy can conceal the distinction.
  • Discussion and Conclusion: Conservative and aggressive routers fail in opposite directions, while the binary gate addresses only the initial stay-or-escalate decision.Broader held-out routers retain large oracle gaps, underscoring the limits of current routing approaches.
  • Discussion and Conclusion: Selecting among Single, PER, Broadcast, and None remains a cost-aware control problem rather than a solved routing task.The benchmark, directional error analysis, and explicit failure-risk/protocol-value distinction define measurable targets for future work.

7 Limitations … A.4 Held-Out Text+Metadata Routers

The paper limits its claims through targeted cross-setting robustness checks and diagnostics rather than broad universality. These analyses expose measurement, parsing, causal, and deployment-cost limitations while documenting matched evaluation coverage and held-out router procedures.

  • 7 Limitations: The main router comparison uses one solver family and one math benchmark, while broader checks span four benchmarks, three task families, and two solvers.The authors characterize these as targeted robustness checks, not evidence of broad universality.
  • 7 Limitations: Logged tokens are the primary cost measure, but latency, price, energy, parallelism, and quality beyond exact correctness may change deployment preferences.The post-answer probe excludes 30 unparseable outputs, confidence varies by model and domain, and PER–Broadcast differences do not establish causality.
  • A Cross-Setting Robustness and Diagnostics: The diagnostics use 2,000 percentile bootstrap resamples over problem identifiers for confidence intervals unless otherwise stated.This section frames the analyses as establishing the scope of the paper’s main claims.
  • A.1 Positioning Within Routing and Adaptive Inference: Related routing work is organized by whether it changes base-model selection, computation or sampling allocation, or tools, roles, and collaboration structures.Table S1 makes explicit which inference component each line of work changes.
  • A.2 Matched Protocol Coverage Across Four Benchmarks: The matched evaluation covers OmniMath 2, JEEBench, SciBench, and LAB-Bench across competition math, engineering STEM, college science, and biology.The two solver families cover all five evaluation conditions, yielding 10 matched model–condition settings.
  • A.3 Post-Answer Confidence Probes: The post-answer probe receives problem text, allowed metadata, and the model’s Baseline final answer, while correctness and protocol outcomes remain in separate scoring labels.Schema validation rejects forbidden keys, and no probe receives a reasoning trace.
  • A.3 Post-Answer Confidence Probes: 4,151 of 4,181 outputs are parseable in the primary gpt-oss OmniMath setting, whose post-answer AUROC differs from the 0.859 pre-answer probe.The post-answer run uses the full matched setting and includes the Baseline final answer, whereas the pre-answer run uses 329 cleaned estimates from a 423-example test split.
  • A.4 Held-Out Text+Metadata Routers: Held-out routers use five-class logistic regression over TF–IDF word unigrams/bigrams plus sparse allowed-metadata features, with fixed 70/15/15 stratification.Gemma router costs are unavailable because protocol-token fields are incomplete; gpt-oss average/excess tokens are 151.7K/105.5K for OmniMath, 556.4K/261.3K for LAB strict, and 266.0K/112.5K for LAB text-no-tool.

A.5 PER–Broadcast Interaction … B.1 Full Routing Comparison

The supplementary analyses show that Broadcast is stronger on average without dominating pointwise, while routing comparisons reveal a cost–solve-rate tradeoff and probe evaluations depend on carefully specified targets and parsing. Detailed OmniMath analyses further examine uncertainty, costs, embeddings, and held-out router performance.

  • A.5 PER–Broadcast Interaction: Broadcast is stronger on average, but PER-only successes occur in every setting, so global ordering does not imply pointwise dominance.The larger LAB differences are observed domain dependence, without direct causal measurements.
  • A.5 PER–Broadcast Interaction: 26.6 points is the largest Broadcast-minus-PER gap on all problems across the six listed confidence/router settings.Conditional on Baseline failure, the corresponding six gaps are 6.3, 8.5, 35.0, 30.8, 10.5, and 30.3 points; below confidence 70, they are 5.0, 8.8, 25.3, 29.7, 7.8, and 21.9 points.
  • A.6 Post-Answer Probe Prompt and Parsing: The probe’s failure score targets Baseline failure, while secondary evaluations distinguish coarse benefit from PER-first and Broadcast-only success.Broadcast-only success requires Baseline, Single, and PER to fail while Broadcast succeeds; AUPRC is relative to target prevalence.
  • B Detailed OmniMath Router Analyses: The detailed OmniMath analysis covers full routing tables, bootstrap intervals, cost-aware prompt and embedding ablations, marginal-cost accounting, token sensitivity, metadata, text signal, and confidence gates.It also documents benchmark provenance, scope checks, prompts, and cleaning.
  • B.1 Full Routing Comparison: Cost-conservative baselines are cheap but miss many solvable problems, while higher-solve frozen LLM routers improve solve rate mainly through over-escalation.Lightweight text-feature routers do not clearly dominate metadata or tier baselines, and balanced text+metadata improves macro-F1.
  • B.1 Full Routing Comparison: Bootstrap intervals use 2,000 held-out test-problem resamples with seed 42, while embedding models use normalized MiniLM representations and dev-split macro-F1 selection.Lower excess cost breaks ties for embedding-only or embedding+metadata kNN/logistic models.

B.2 Difficulty-Metadata Ablation … C.3 Protocol Execution and Compute Accounting

Across ablations, confidence policies, robustness checks, and protocol accounting, the paper finds that self-assessment helps judge one-shot solvability but does not reliably select the best collaboration protocol. The benchmark fixes four protocol outcomes under shared solver conditions, exposing substantial compute costs and unresolved cost-aware routing challenges.

  • B.2 Difficulty-Metadata Ablation: Text-only routers raise solve rate mainly by selecting costlier protocols, while source-holdout no-tier gains require substantially more tokens and over-escalation.Removing explicit difficulty fields leaves text+metadata near the early-stopped full model; no-tier learned routers can improve held-out solve rate, but at higher token use.
  • B.3 Direct Self-Assessment and Confidence Policies: Self-confidence predicts one-shot success, but confidence alone cannot identify whether SINGLE, PER, BROADCAST, or no collaboration is optimal.The reliability mechanism supports deciding whether a low-cost direct attempt is plausible, whereas oracle composition remains distributed across escalation choices.
  • B.4 Reduced Gemma-3 Actor-Stack Check: On a tier-sampled 833-problem Gemma-3 subset, SINGLE and PER improve over one-shot solving, while a Gemma-3 router selects PER on 56.8% and never selects BROADCAST or none.Against the Gemma actor oracle, this router solves 63.3%, under-escalates on 6.4%, and over-escalates on 67.0%; the subset is not a full-benchmark replication.
  • B.4 Reduced Gemma-3 Actor-Stack Check: Gemma-3 self-confidence reaches AUROC 0.787 versus 0.765 for tier alone, yet its calibration is weaker than the main gpt-oss self-assessment.Cross-model estimates are parseable but weaker and less calibrated than same-model gpt-oss self-assessment, so the signal is not evidence for arbitrary cross-solver capability estimation.
  • B.4 Reduced Gemma-3 Actor-Stack Check: Moderate utility weights λ ∈{0.4, 0.5} reproduce the Kneedle-selected held-out operating points for both confidence policies.Budget-constrained sweeps trace the expected solve-cost frontier, while the exploratory protocol-value probe remains useful but is not a clean replacement for the simpler self-confidence gate.
  • C.1 Benchmark Provenance and Metadata: The benchmark contains 4,181 matched competition-level mathematics problems, with robustness conditions spanning JEEBench, SciBench, and LAB-Bench across engineering, science, and biology tasks.The primary slice retains source, domain, difficulty, and ten difficulty-tier fields; both solver families cover all five evaluation conditions.
  • C.2 Splits and Oracle-Label Scope: Each problem is evaluated under BASELINE, SINGLE, PER, and BROADCAST, with the oracle choosing the first correct protocol in fixed cost order or none if all fail.The oracle is outcome-grounded and based on observed matched runs, making it a single-realization control target rather than a manually assigned difficulty label.
  • C.3 Protocol Execution and Compute Accounting: 4.53B logged tokens construct the full matched benchmark: 77M for BASELINE, 201M for SINGLE, 1.67B for PER, and 2.58B for BROADCAST.All traces use openai/gpt-oss-120b with deterministic decoding and a 4096-token per-call cap; router-side costs are accounted separately where applicable.

C.4 Learned Router Implementation · D Prompt and Execution Details · D.2 Cost-Aware Frozen Router Prompt Ablation

The routing and probing procedures use fixed protocol labels, structured metadata, controlled prompts, and explicit cost-order instructions. Implementation details also document confidence-parsing coverage and limitations, including non-random missingness and residual gaps between cleaned probes and routing oracles.

  • C.4 Learned Router Implementation: Learned routers use five-class logistic regression over Baseline, Single, PER, Broadcast, and None labels, with TF-IDF text features and optional metadata.Metadata includes source, domain, difficulty, and difficulty tier; ablations remove metadata or use text only.
  • C.4 Learned Router Implementation: Main learned routers select regularization, class weighting, and training epoch on validation data using dev log loss, macro-F1, and excess cost tie-breaking.The reported implementation uses scikit-learn LogisticRegression with solver=saga and a grid over C and class_weight.
  • D.1 Frozen Router Prompt: Frozen routers receive source, difficulty, difficulty tier, domain summary, and full problem text, then choose the cheapest sufficient protocol from an ordinal cost order.Allowed outputs are baseline_llm, single_agent, per, broadcast, and none; runs use temperature 0.0 and unparsable responses receive a fallback label.
  • D.2 Cost-Aware Frozen Router Prompt Ablation: The cost-aware frozen-router ablation adds numeric average costs and routing examples to the main prompt, with one fallback in the final cleaned run.The prompt lists 18K, 48K, 402K, and 622K token costs and instructs escalation only when expected solve gains justify extra cost.
  • D.5 Pre-Answer Confidence Parsing and Coverage: 77.8% of test problems yielded usable pre-answer confidence estimates after manual re-parsing, from 310 initially parsed rows to 329 cleaned rows.The remaining missing rows comprised 72 truncations and 22 HTTP 429 failures.
  • D.5 Pre-Answer Confidence Parsing and Coverage: Missing confidence estimates are not missing at random because dev fallbacks commonly reflect parse failures caused by the model entering solving mode.A completion-cue variant was excluded because naive integer parsing could extract numbers from problem statements; cleaned JSON coverage still does not create a routing oracle.

E Additional Diagnostic Figures

Additional diagnostics show that the lightweight text+metadata router overfits after early epochs and that aggregate comparisons are not driven by a single benchmark source. Together, these checks support interpreting the current routers as miscalibrated about the marginal value of escalation.

  • Learned-router diagnostics: Text+metadata development loss diverges from training loss after early epochs, consistent with overfitting and motivating early stopping.The learned-router training loss continues improving while development loss worsens in this lightweight setup.
  • Source-level robustness: The aggregate comparison is not explained by a single benchmark source.A source-level heatmap checks that the learned-router result is not driven by one benchmark source alone.
  • Interpretation: The diagnostics support the narrower interpretation that current routers are miscalibrated about the marginal value of escalation.This interpretation follows from the overfitting check and the source-level comparison.

F Data and Reproducibility

The paper documents its prompts, schemas, splits, execution settings, and uncertainty procedures, while releasing trace and aggregate-result artifacts for reproducibility. The paper’s claims use only the four protocols and benchmark/model settings defined in the main text, with restricted raw prompts handled through identifiers and reconstruction instructions.

  • Documentation: The appendix records the exact prompts, feature schemas, split rules, execution settings, and uncertainty procedures needed to interpret the reported results.The companion dataset archive is hosted on Hugging Face.
  • Reproducibility artifacts: The companion archive contains anonymized four-protocol traces, per-protocol outcome labels, matched outcome tables, and identifier registries for Baseline, Single, PER, and Broadcast.The arXiv source package also includes aggregate result tables as ancillary CSV files.
  • Scope boundaries: The archive includes experiments beyond this paper, so analyses should select only the four protocols and benchmark/model settings defined in the main text.These boundaries define which archived records support the paper’s claims.
  • Data access: Raw problem redistribution follows each upstream benchmark’s license; restricted datasets are released through identifiers and reconstruction instructions rather than raw prompts.This preserves reproducibility while respecting upstream redistribution constraints.
Loading 2608.14927v1…