PaperlayerView exact revision on arXiv

Source-linked AI summary

Jev-as-a-Judge for RAG Claim Verification: Confidence-Gated Cascading Between a Non-Generative Verifier and a Frontier LLM

Alden Do Rosario, Kirollos Atef

arXiv:2610.20634v1cs.CL

TL;DR

RAG pipelines repeatedly need a binary verdict on whether evidence supports a claim, but often obtain it from a text generator. The paper compares a non-generative decision model with a frontier LLM on paired claims, finding similar headline accuracy but different errors and costs. Its cross-fitted cascade is cheaper than the frontier model, while matched random escalation performs comparably.

  • Problem

    RAG pipelines need repeated binary evidence-support judgments, yet many produce them through a text generator and pay generation prices for prose that is converted back into a bit.

    Source: §1 · p. 1

  • Method

    The study compares a non-generative model returning a typed verdict and native probability with a frontier LLM on the same 495 human-labelled claims, then evaluates confidence-based cascading and matched random escalation.

    Source: §1 · p. 2

  • Results

    −0.56 points, Jev minus Astra, on the paired headline comparison; the cross-fitted cascade reaches 74.10% at $2.92 per thousand claims versus 73.83% at $8.93 for Astra, while matched random escalation does as well.

    Source: §1 · pp. 1–2

  • Takeaways & Limitations

    The accuracy tie conceals different errors: Jev false-verifies 23.23% of unsupported claims versus Astra’s 14.14%, so headline accuracy alone does not show which is the more permissive guard.

    Source: §1 · p. 1, §11.1 · p. 17

    The RAGTruth comparison includes only 4 unsupported claims, and both systems label all 4 correctly, limiting what this sample says about catching unsupported claims on that dataset.

    Source: §7.4 · p. 10

Abstract

from arXiv · show

Retrieval-augmented generation asks one question at volume: does this evidence support this claim? The answer is one bit, yet most pipelines obtain it from a text generator and pay generation prices for the prose wrapped around it. We compare a non-generative decision model, which returns a typed value from a declared enumeration together with a native probability and no text, against a frontier LLM configured as an engineer would deploy it for binary classification. Both answer the same 495 human-labelled claims from LLM-AggreFact, so every comparison is within-item. We report three things. First, the two systems are statistically indistinguishable on the metric this task is conventionally scored by, a paired difference of -0.56 points, Jev minus Astra, at a 187-fold cost difference. Second, that tie carries almost no information about deployment: the systems disagree on 15.2% of claims, and the cheaper one admits 23.2% of the claims annotators marked unsupported against the frontier model's 14.1%. A component whose job is to catch unsupported claims should be judged on that rate, and the conventional headline hides it. Third, because the errors differ they compose: routing the claims the cheap model is confident about and escalating the rest reaches 74.10 at USD 2.92 per thousand claims against 73.83 at USD 8.93, with the threshold chosen out of fold. But a matched random-escalation control at in-sample thresholds, spending the same budget by coin flip, does as well, so what buys the saving is the escalation share and not the probability that picks what to escalate. Measured against the cheap model it is built from, the cascade costs 61-fold more for 0.83 points, 95% CI [-3.36, 4.86], and it is the more permissive guard of the two. Equal accuracy conceals unequal errors. All predictions, prompts, pricing and code are released.

1 Introduction

RAG pipelines repeatedly need a binary evidence-support verdict, yet commonly pay generation prices for text that downstream code converts back to a bit. The paper compares a non-generative decision model with a frontier LLM on paired claims and argues that headline accuracy alone misses both their different errors and their deployment trade-offs.

1 Introduction

  • Jev returns a typed verdict and native probability without prose, while Astra is configured for binary classification; both evaluate the same 495 human-labelled claims from LLM-AggreFact.Each claim goes to both systems, making the comparison within-item.Source: §1 · p. 1
  • 15.2% of claims receive different verdicts, and Jev false-verifies 23.23% of unsupported claims versus Astra’s 14.14%, making the accuracy tie conceal unequal guards.The paper also reports Jev correct on 41 claims where Astra errs, with the reverse on 34.Source: §1 · pp. 1–2
  • −0.56 points, Jev minus Astra, 95% CI [−5.6, +4.5], with exact McNemar p = 0.49: the systems are statistically indistinguishable on the headline comparison despite a 187-fold cost difference.Table 1 compares configurations on the same 495 claims; the reported difference is within one point.Source: §1 · pp. 1–2, Table 1 · p. 2
  • 74.10% at $2.92 per thousand claims for the cross-fitted cascade versus 73.83% at $8.93 for the frontier model; matched random escalation does as well, so the saving tracks escalation share, not routing confidence.The cascade is more permissive: it false-verifies 19.22% of claims it keeps, versus 14.14% for the frontier model.Source: §1 · p. 2

2 Related Work

  • Prior factuality work established grounded claim verification and consolidated human-labelled datasets; this study changes the machine answering the task rather than its definition.The related work also situates the comparison among LLM judges, cascades, selective prediction, and confidence-based routing.Source: §2 · pp. 2–3
  • The paper’s narrower contributions are to characterize native-probability routing, measure complementary errors between headline-tied systems, and quantify optimism from in-sample threshold selection.It explicitly does not claim that confidence routing beats random routing at this sample size or transfers beyond grounded claim verification.Source: §1 · p. 2, §2 · p. 3

3 Problem Setup and Metrics

The task judges whether a claim is supported by its supplied evidence, not whether it is true in the world. The section defines macro-averaged balanced accuracy and adds metrics for unsupported claims and probabilities used to route decisions.

Decision definition

  • A claim is unsupported when its supplied evidence does not entail it, even if it happens to be true in the world.The human gold label and system decision each use the supported/not_supported distinction.Source: §3 · p. 3

Balanced accuracy

  • 79.3% pooled balanced accuracy for the decision model contrasts with 73.3% macro-averaged, so the paper follows the LLM-AggreFact macro convention throughout.The macro score is the unweighted mean of per-dataset balanced accuracies; pooling weights class-specific performance by which datasets contribute more claims.Source: §3 · p. 3

Error metrics

  • False verification—the rate of approving unsupported claims—is reported alongside verified precision because balanced accuracy can hide differences in this error.False verification is the complement of specificity; it is computed over the pooled 198 unsupported claims.Source: §3 · p. 4

Probability provenance

  • A probability’s provenance is labeled as native, logprob-derived, or none; verbalized self-reports are not treated as comparable probabilities.The decision model supplies a native probability, while the frontier model’s provider rejects logprobs, leaving no probability signal for that arm.Source: §3 · p. 4

Probability metrics

  • Selective AUROC measures whether confidence separates a model’s correct from incorrect claims, unlike AUROC against gold labels, which measures separation of supported from unsupported claims.The section also reports Brier score, negative log loss, and expected calibration error, with a second binning as an ECE sensitivity check.Source: §3 · p. 4

4 Systems Under Test

The evaluation compares a typed, non-generative decision model with a frontier LLM configured for binary classification. Their outputs, probability availability, prompts, and token-cost accounting differ in ways that shape the comparison.

Systems

  • The decision model returns an enumerated verdict and a distribution over verdicts, while the frontier model returns a binary classification without provider-supplied token probabilities.The decision model’s supported probability is native; the frontier arm therefore supplies no routing signal.Source: §4 · pp. 4–5

Token accounting

  • The decision model’s output spans 35 or 36 tokens across claims, a fixed serialization envelope rather than prose, and its provider bills those output tokens at zero.The frontier arm’s output length varies by claim; its standard- and high-effort ranges are 4–134 and 4–519 tokens, respectively.Source: Table 2 · p. 5

Configurations

  • MiniCheck-Flan-T5-Large was excluded because its port scored far below every reported configuration and the gap could not be attributed to the model rather than the port.The three-configuration table reports per-request mean token counts and costs priced using the uncached standard-tier snapshot.Source: Table 2 · p. 5

Prompt parity

  • Both systems receive the same semantic criteria, including marking claims unsupported for materially missing, contradicted, wrong, over-scoped, or embellished content.An audit found an entity-swap criterion missing from the original frontier prompt; both frontier configurations were rerun with the correction.Source: §4 · p. 5

Token accounting

  • 1,139 input tokens on average for the decision model versus 843 for the frontier arm; the 187× cost ratio mainly reflects the 238.1-fold input-price difference.Repricing the decision model’s output tokens at its vendor’s input rate changes the ratio from 187× to 181×, accounting for 3% of the advantage.Source: §4 · p. 5

5 Experimental Setup

The experiment uses paired claims from a pinned development split and estimates differences with dataset-stratified procedures. Cascade thresholds are cross-fitted, while the split and some inferential results constrain what the evaluation supports.

Data

  • 495 claims are sampled in equal numbers from the 11 LLM-AggreFact component datasets, with 297 supported and 198 unsupported labels.Both classes occur in every dataset, but the smallest minority class has only four claims.Source: §5 · p. 6

Split discipline

  • Prompts and cascade parameters are chosen on the development split, and reported intervals are development-split intervals rather than evidence from the unread test split.One threshold is selected out of fold within the development split; the benchmark’s public history means neither system can be assumed uncontaminated.Source: §5 · p. 6

Inference

  • Each configuration is evaluated on identical claims, and paired stratified bootstrap resamples claims within datasets for macro-balanced-accuracy intervals.Exact McNemar tests assess per-item correctness discordance; Holm-Bonferroni adjustment is applied across 13 reported tests, with two adjusted comparisons separating on the guard metric.Source: §5 · p. 6

Operational scope

  • No latency comparison is reported because the controlled single-stream protocol was not run; persistent model-caused failures are scored wrong rather than dropped.The prediction files are committed and timestamped before analysis, with hashes released for verification.Source: §5 · p. 7

Cascade simulation

  • Cascade thresholds use 200 repeats of five-fold stratified cross-validation, scoring each claim under a threshold fit without its fold.The simulation uses locked predictions and compares confidence-based escalation with random escalation of the same traffic share and an oracle control.Source: §5 · p. 7

6 Results: same score

On 495 paired claims, Jev and Astra are statistically indistinguishable in macro-balanced accuracy, but their errors differ, including on unsupported claims. Per-dataset results and the thin unsupported class in RAGTruth constrain how broadly those comparisons can be read.

6.1 The paired difference

  • 73.27% for Jev versus 73.83% for Astra in macro-balanced accuracy, a −0.56-point paired difference (95% CI [−5.6, 4.5]; McNemar p = 0.49), at 187-fold different costs.The interval does not establish equivalence, and the reported McNemar test is on pooled raw per-item accuracy.Source: §6 · p. 7, §6.1 · p. 7, Table 4 · p. 8

6.2 The ordering changes with the metric

  • 79.8% versus 78.4% raw accuracy favors Jev, while 73.3% versus 73.8% macro-balanced accuracy favors Astra; the metric changes the ordering.Jev’s higher raw accuracy reflects the majority supported class, while balanced accuracy does not reward that prevalence advantage in the same way.Source: §6.2 · p. 8, Table 5 · p. 8

6.3 What the interval does not support

  • The [−5.6, 4.5]-point interval permits operationally meaningful gaps in either direction, so the study claims no equivalence.The 11-dataset macro average has limited resolution, with one flipped decision in the thinnest minority class moving it more than the observed gap.Source: §6.3 · p. 8

7.1 The disagreement splits both ways

  • Jev is correct where Astra is wrong on 41 claims, and Astra is correct where Jev is wrong on 34; 66 claims defeat both.The paired discordant cells underpin the McNemar test, but its p = 0.49 does not mean the systems make identical errors.Source: §7.1 · p. 9, Table 6 · p. 9, Figure 1 · p. 9

7.2 The cheaper system is the more permissive guard

  • 23.2% of gold-unsupported claims are verified by Jev versus 14.1% by Astra, a 9.1-point gap across 198 claims.The paired stratified bootstrap interval for this gap is [4.55, 13.64] points; verified precision is 84.1% for Jev and 88.6% for Astra.Source: §7.2 · p. 9

7.3 Per dataset, the advantage changes hands

  • Across 11 datasets, Jev leads on 6 and Astra on 5, with opposing extremes of 60.5 versus 46.1 on ExpertQA and 48.7 versus 68.8 on AggreFact-CNN.Each dataset has about 45 claims; the rank order is unstable, and the signed aggregate difference masks disagreement across all 11.Source: Figure 2 · p. 10, §7.3 · p. 10

7.4 What RAGTruth cannot tell us

  • RAGTruth’s four gold-unsupported claims are too few to establish the aggregate false-verification gap; that gap is assessed across all 198 unsupported claims.Both systems label all four RAGTruth unsupported claims correctly in this run, while one earlier Jev pass missed one.Source: §7.4 · p. 10

7.5 Errors that differ can be composed

  • The 41 and 34 discordant claims make composition possible, but production routing still needs a pre-call signal for which claims Jev will get wrong.The systems are not independent: they fail together on 66 claims, versus 21.6 expected under independence.Source: §7.5 · pp. 10–11

8 Results: calibration, and who even has a probability

Jev’s native probability can support confidence gating, but it discriminates its own errors less well than supported from unsupported claims. Astra supplies no probability for routing; gating trades coverage for precision and leaves some claims unanswered.

8 Results: calibration, and who even has a probability

  • Against gold labels, Jev’s supported-claim probability reaches AUROC 0.868 and AUPRC 0.893; confidence for whether Jev itself was right reaches selective AUROC 0.722 and selective AUPRC 0.892.Routing depends on the latter measure, which is weaker at distinguishing claims than the gold-label measure.Source: §8 · p. 11

8.1 Calibration of the native probability

  • Jev’s Brier score is 0.153 versus 0.250 for a constant one-half forecast, and its expected calibration error is 0.099 over ten equal-frequency bins.The negative log likelihood is 1.133, but 62% comes from ten certain-but-wrong claims; the clipping constant is provisional, and using 0.544 yields the same predictions.Source: §8.1 · p. 11
  • Jev’s probability is under-confident through the lower half of the scale and over-confident in a narrow band near the top.In Figure 3, the largest gap is predicted 0.05 versus observed 0.29; equal-frequency bins are used because probability mass piles up at zero and one.Source: §8.1 · p. 11, Figure 3 · p. 12

8.2 Which systems have a probability at all

  • Astra provides no probability at either reasoning budget because its provider returns a 400 error for token logprobs, making cheap-first the available cascade direction.The figure notes that Astra has no probability for this configuration rather than an empty calibration axis.Source: §8.2 · p. 11, Figure 3 · p. 12

8.3 What the probability buys before any routing

  • Gated balanced accuracy is computed only on covered claims, and at the tightest gate two datasets lack a retained claim of one gold class, leaving a nine-dataset rather than eleven-dataset macro.Balanced accuracy over covered claims is not monotonic: 75.8 at 69.1% coverage is below 77.2 for the looser row; abstained claims remain unanswered.Source: Table 7 · p. 12, §8.3 · p. 12
  • At the tightest gate, Jev answers 35.2% of claims with 97.1% verified precision and 2.1% false verification; precision rises and false verification falls across the gate grid.At full coverage, 84.1% of claims Jev calls supported really are, at a 23.2% false-verification rate.Source: §8.3 · pp. 11–12

9 Confidence-gated cascading

The confidence-gated cascade answers claims with Jev when its native confidence clears a threshold and otherwise escalates to Astra. Cross-fitting estimates its operating point, while comparisons with Astra, Jev, random escalation, and an oracle bound what its cost and routing signal achieve.

9 Confidence-gated cascading

  • 74.10% macro-balanced accuracy at $2.92 per thousand claims is the cross-fitted cascade result, versus 73.83% at $8.93 for Astra alone.The threshold is chosen out of fold; matched random escalation costs $2.81 per thousand claims and reaches the same place as the frontier arm.Source: §9 · pp. 12–13

9.1 The mechanism

  • Jev answers when max(p, 1 − p) clears the threshold; otherwise the same claim and evidence go to Astra.The offline replay recomposes locked predictions and excludes orchestration, retries, and the second network hop, so its cost is a floor.Source: §9.1 · p. 13

9.2 Choosing the threshold without looking at the answer

  • 200 repeats of stratified 5-fold cross-fitting choose the threshold on remaining folds and apply it to held-out claims, then score the assembled 495 claims.The reported point averages accept and escalate shares across repeats, not the shares from any single fixed threshold.Source: §9.2 · p. 13, Figure 4 · p. 13

9.3 Testing the comparison the cost claim rests on

  • The cross-fitted cascade is 0.27 points above Astra on macro-balanced accuracy, 95% CI [−3.85, 4.15], which cannot resolve a difference on 495 claims.Its accuracy is below Astra in 68 of 200 repeats; against the high-effort frontier arm, its guard metric is 6.09 points worse, with adjusted p = 0.005.Source: §9.3 · p. 14, Table 8 · p. 14, Table 9 · p. 15

9.4 The cascade against the cheap system it is made of

  • Against Jev alone, the cascade costs 61× more for 0.83 points of macro-balanced accuracy, 95% CI [−3.36, 4.86].It false-verifies 19.22% of gold-unsupported claims versus Jev’s 23.23%; the interval excludes zero, but the exact McNemar test is marginal and not significant by the stated criterion.Source: §9.4 · p. 14

9.5 Against spending the same budget at random

  • At a matched budget, random escalation is not separable from the frontier arm: it costs $2.81 per thousand claims and scores 73.36 versus 73.83.The paired difference is −0.47 points, 95% CI [−5.44, 4.41]; the control therefore attributes the saving to escalation share, not the confidence signal.Source: §9.5 · pp. 14–15

9.6 How much room the idea has

  • Oracle routing reaches 82.9 by escalating 20.2% of claims at $1.60 per thousand, 9.6 points above Jev alone.It escalates exactly the claims Jev gets wrong and is an upper bound, not an operating point, because it requires knowing the answers.Source: §9.6 · p. 15

10 How much of a tuned cascade is real

Tuning and scoring a cascade on the same claims makes its reported accuracy optimistic. Cross-fitting reduces that selection bias, but the threshold remains unstable at this sample size.

10 How much of a tuned cascade is real

  • 75.82% in-sample versus 74.10% cross-fitted macro-balanced accuracy is a 1.72-point selection difference from fitting one threshold on the scored claims.The in-sample operating point escalates 17.2% at $1.62 per thousand, versus 29.9% at $2.92 cross-fitted.Source: §10 · p. 15

10.1 The threshold is not identifiable at this sample size

  • Across 1,000 fold-fits, the threshold selected 0.80 669 times and the grid’s highest threshold 253 times, corresponding to 17.2% and 64.8% escalation regimes.The mean 29.9% escalation rate averages across these two regimes, so it is expected spend rather than a threshold anyone would deploy.Source: §10.1 · p. 16
  • The bracket [72.1, 75.8] reflects fold-assignment and threshold-selection variation, not sampling uncertainty across claims.Its upper end includes the in-sample best, and the authors measured no other seeds; each cross-fitted mean is one fold assignment at one recorded seed.Source: §10.1 · p. 16

10.2 What this costs, and where the number belongs

  • The authors report a 3.06× cost reduction against Astra, bought at 5.08 points more false verification, while the accuracy difference lies in [−3.85, 4.15].The same-data sweep inflates accuracy by 1.72 points and quotes $1.62 per thousand where the cross-fitted cost is $2.92.Source: §10.2 · p. 16

11 Discussion

The discussion argues that verification systems should be judged on the errors operators pay for, not accuracy alone. It treats routing as a cost trade-off against the frontier model, not as a demonstrated improvement over the cheap model or evidence of transfer beyond this benchmark.

11.1 Measure the error you actually pay for

  • Jev false-verifies 23.2% of unsupported claims versus Astra’s 14.1%, despite their macro-balanced accuracy being within 0.6 points.The discussion recommends reporting false-verification rate and verified precision at the intended operating point because false verifications and false alarms carry different costs.Source: §11.1 · p. 17

11.2 Route rather than replace, only against the frontier arm

  • Against Astra, the cross-fitted cascade reaches 74.10 macro-balanced accuracy at $2.92 per thousand versus 73.83 at $8.93, but the sample cannot resolve the accuracy difference.Against Jev it costs 61-fold more for 0.83 points; at the matched random budget, the cascade is no further from the frontier arm, so the saving belongs to escalation share.Source: §11.2 · p. 17

11.3 Where else a decision model fits

  • The proposed relevance of decision models to routing, moderation, retrieval triage, and schema validation remains a hypothesis: this study measured one question on one benchmark.The authors frame those other applications as having a cheap test, not as demonstrated transfer.Source: §11.3 · p. 18

12 Limitations and Conflict of Interest

The paper concludes that similar headline accuracy can conceal different errors, and that the cascade’s cost advantage is against the frontier model rather than Jev. Its evidence is limited to a single benchmark and pair of systems, with evaluation and deployment caveats.

12 Limitations and Conflict of Interest

  • All reported numbers come from 495 paired claims on the public benchmark’s development split, with prompts, thresholds, grids, and metrics chosen while outcomes were visible.The test split was never read, and the authors state that equivalence was neither claimed nor tested.Source: §12 · p. 18

12.1 Conflict of interest

  • The first author founded and leads CustomGPT.ai, whose product includes claim verification, and the second author works there.The authors state that their commercial interest in cheap verification is why predictions, prompts, and analysis are released.Source: §12.1 · p. 19

13 Conclusion

  • The cascade saves cost against Astra but costs 61-fold more than Jev for 0.83 points, while the two base systems’ similar accuracy conceals different errors.The conclusion reports a 3.06-fold cost reduction against Astra, with 5.08 points more false verification; random escalation at a similar budget costs $2.81.Source: §13 · p. 19

A Per-dataset results

  • Per-dataset accuracy rows describe the aggregate’s composition but do not support per-dataset inference because several datasets have thin minority classes.The appendix reports that opposing Jev–Astra differences largely cancel across datasets; the two widest differences are −20.11 and 14.36 points.Source: A Per-dataset results · p. 19, Table 11 · p. 20

B Prompts

  • Both systems receive the same supported/not-supported criteria in their respective native input formats, but Jev uses a typed question with an enumerated answer rather than an output-format instruction.The frontier prompt is a text template; the Jev request carries claim and evidence as named state keys.Source: B Prompts · p. 19

B.1 Frontier prompt

The Astra prompt frames claim verification as a binary decision about whether supplied evidence supports every materially important claim component. It restricts judgment to that evidence and requires a single enum label without explanation.

B.1 Frontier prompt

  • Astra must decide whether the supplied evidence supports every materially important part of the claim.Source: §B.1 · p. 20
  • A claim is NOT_SUPPORTED when any material part is missing, partial, contradicted, numerically wrong, temporally wrong, names the wrong entity, too broadly scoped, or adds unsupported detail.Source: §B.1 · p. 20
  • The prompt limits judgment to supplied evidence and requires exactly one word, SUPPORTED or NOT_SUPPORTED, with no explanation.Source: §B.1 · p. 20

B.2 Jev question definition

The Jev question asks whether evidence supports all materially important parts of a claim. It defines the two labels by direct support versus failure to support any material part.

B.2 Jev question definition

  • Jev asks whether evidence supports all materially important parts of the claim, using a choice question with claim and evidence as inputs.Source: §B.2 · p. 20
  • SUPPORTED means the evidence explicitly states or directly entails every materially important part of the claim.Source: §B.2 · p. 20
  • NOT_SUPPORTED applies when at least one materially important part lacks support, including missing or partial evidence, contradiction, wrong numbers, dates or entities, scope overreach, or unsupported causal or detail claims.Source: §B.2 · p. 21

C Incidents and corrections

The incident log records corrections to prompts, analysis rules, and released artifacts so readers do not mistake implementation or reproducibility problems for study results. The corrections regenerated affected numbers from the locked predictions, without changing the headline result where stated.

C Incidents and corrections

  • The first Astra run’s empty responses and provider errors came from a completion-token cap consumed by internal reasoning, so the runs were repeated with a larger cap.Source: C Incidents and corrections · p. 21
  • A parity audit added the missing wrong-entity criterion to the Astra template and repeated both reasoning-effort runs; all reported Astra numbers use those repeats.Source: C Incidents and corrections · p. 21
  • The MiniCheck arm was excluded, with its predictions released so readers can diagnose the port rather than rely on the authors’ explanation.Source: C Incidents and corrections · p. 21
  • The evidence hash file was edited after timestamping, temporarily breaking verification; the stamped file was restored and additional hashes moved to an unstamped file.The prediction-file block was byte-identical across versions, so no result was affected.Source: C Incidents and corrections · p. 21
  • A code-versus-paper escalation boundary discrepancy changed escalation shares, costs, and part of the gating sweep; regenerating from locked predictions left the headline unchanged.The code escalated a claim at probability 0.2 that the stated rule would keep.Source: C Incidents and corrections · p. 21
  • The log-loss clipping constant was provisional rather than preregistered, so the preregistration claim was retracted and scores were reported at a clip matched to vendor probability resolution.Source: C Incidents and corrections · p. 22
  • The published article emphasized an in-sample cascade row of 75.82 macro-balanced accuracy at 17.2% escalation and $1.62 per thousand, superseded by cross-fitted 74.10 at $2.92.The passage calls the 1.72-point accuracy gap selection optimism; the earlier article also predates the escalation-rule fix.Source: C Incidents and corrections · p. 22
  • A stale public package contained a superseded model registry and checks that compared HTTP status rather than file bytes; the corrected check verifies published files by SHA-256.The stale package did not change results because the caps document runs locked and hashed before correction.Source: C Incidents and corrections · p. 22

D Reproduction

The reproduction workflow regenerates paper numbers and figures from locked predictions, configuration, and labels, while documenting which inputs are not regenerated or distributed. The released package includes checks intended to let readers verify its contents and prediction provenance.

D Reproduction

  • One command regenerates the paper’s numbers from locked prediction files, hashed configuration, and gold labels, while the evaluation sample is a released, hashed input rather than a regenerated draw.The gated dataset labels are not included; rerunning requires an accepted Hugging Face token, and the command fails with 401 without one.Source: D Reproduction · p. 22
  • The released example-ID list and gold labels are inputs to the reproduction command, but the sample draw is not regenerated and the labels are not redistributed.Source: D Reproduction · p. 22
  • The analysis command writes the numbers, cross-fitted cascade view, and excluded MiniCheck arm, and hashes the four result prediction files; a separate command regenerates figures from the numbers file.Source: D Reproduction · pp. 22–23
  • The public package includes predictions, prompts, registry, pricing, IDs, leaderboard capture, incident log, analysis code, and tests, with build-time checks and a reader-side SHA-256 verification script.The package builder runs shipped tests and omits output if checks fail; the builder itself is withheld, while the reader-side verifier is included.Source: D Reproduction · p. 23
  • The timestamp proof commits to the prediction-hash file, supporting that the named prediction files were fixed before analysis; auxiliary inputs are hashed separately without timestamp proof.At writing, calendar-server attestations were pending Bitcoin confirmation; the proof becomes independent of the calendars after mining and upgrade.Source: D Reproduction · p. 23

E Limitations in full

The study’s conclusions are bounded by a fixed, exploratory sample, model and prompt choices, and an offline cascade whose threshold selection remains unstable. Pricing and several statistical comparisons also have limits that constrain how broadly to interpret the reported results.

E Limitations in full

  • −0.6 points on Jev minus Astra is not evidence of equivalence: no equivalence margin or test was preregistered, and the pooled false-verification gap is a different estimand from macro accuracy.The false-verification gap is 9.1 points, with an interval of [4.55, 13.64] over 198 gold-unsupported claims; the accuracy comparison is macro-averaged over 11 datasets.Source: E Limitations in full · p. 24
  • The cross-fitted cascade avoids the 1.72-point in-sample optimism, but threshold choice remains unstable and the threshold grid, routing decision, and objective were selected with development outcomes visible.Across 1,000 fold-fits, one threshold was selected 669 times and a very different one 253 times; gate-sweep comparisons are uncorrected for multiple testing.Source: E Limitations in full · pp. 24–25
  • The cascade’s macro-balanced-accuracy advantage over Astra is not stable: it falls below Astra in 68 repeats, and 43.1% of bootstrap draws place the difference below zero.The reported guard penalty against Astra is 5.08 points, with interval [1.52, 9.09], and its adjusted p-value is 0.057.Source: E Limitations in full · pp. 24–25
  • The 495-example evaluation list is fixed and came from an exploratory Jev run in which every retained row succeeded, so it cannot establish Jev’s failure rate or be regenerated as a new draw.The claims also never approach either system’s context limit, so the sample cannot establish how either behaves on longer documents.Source: E Limitations in full · pp. 25, 27
  • The cascade is an offline replay of stored predictions, not a deployed router; its added network hop, queueing, and live failure behavior were not measured, and latency was not measured under controlled conditions.Both systems answered the 495 claims independently, and no claim was resent through a live cascade.Source: E Limitations in full · pp. 25–26
  • The 187× cost ratio is a dated model of list prices, not an invoiced or durable price advantage; negotiated rates and Jev’s free-output-token policy can change the comparison.The prices were taken from two vendors’ public lists on one date.Source: E Limitations in full · p. 26
  • One frontier family, single runs, and a prompt correction that reversed the Jev–Astra ordering limit reproducibility and model-wide inference.The repeat covers only 411 claims under a superseded prompt; contamination cannot be ruled out, and the two reasoning efforts do not test a prompt that invites reasoning.Source: E Limitations in full · pp. 26–27
  • MiniCheck scored 61.6 [56.3, 66.7] on the 495 claims, implicating its adapter rather than the checked metric layer or shared gold labels; the partial leaderboard capture cannot establish field-wide comparisons.The adapter-specific concerns are chunking, evidence truncation, and aggregation of per-chunk entailment scores.Source: E Limitations in full · p. 27

Related papers

Loading 2610.20634v1…