Confidence Is Not a Promise: A Reference Audit of Typed Decision Models

• The paper asks what a typed decider’s probability licenses, since software can act on its returned decision and confidence.The study measures both confidence as evidence about correctness and the accuracy promise made by a selective gate.Source: §1 · pp. 1–2
• The study compares five Laya checkpoints with Jev 1.13 on MASSIVE, including fine-tuning, calibration, and explanation fidelity.The paired comparison uses 2,000 Portuguese and English utterances per language and tests the vendor’s statement about probability one.Source: §1 · p. 2
• The contribution on selective gates compares point estimates, Learn then Test, and a Beta lower-bound rule, including their guarantees, coverage, and failure conditions.The analysis separates item-level uncertainty from Monte Carlo uncertainty and includes an ablation over 60 configurations.Source: §1 · p. 2
• The paper measures how typing noise affects a clean-text promise and reports that noise training raises noisy accuracy without loss on clean text.It also reports that a probability of 0.9999 does not protect a decision from a single typo.Source: §1 · p. 2
• Across 51 locales, Jev keeps a 95% promise in 8; across four further domains, fine-tuned Laya needs 250 to 8,000 labels to be non-inferior to Jev.The paper also measures drift repairs and finds that neither decider predicts the direction of the next M3 value better than a constant answer.Source: §1 · p. 2
• The reference package releases frozen answers, a datasheet, a selective-accuracy promise check, and code, alongside a protocol separating vendor claims from measured evidence.The protocol maps seven operational controls to measurements.Source: §1 · p. 2

• Prior typed-decider studies examine calibration, comparisons with other deciders, and whether decisions remain consistent when decomposed.Reported findings include Jev’s calibration and selective-prediction performance [Deußer et al., 2026], language sensitivity, and self-disagreement.Source: §2 · p. 3
• Selective prediction methods provide risk guarantees, but prior measurements also report frequent threshold-budget violations; this paper evaluates a Bayesian lower-bound rule against Learn then Test and point estimates.The paper frames its contribution as a replication for typed deciders, with uncertainty separated and the Bayesian rule’s benefits and costs measured.Source: §2 · p. 4
• Prior explanation research uses deletion metrics and warns that they can reward explanations unrelated to model reasoning; the relation between typed-decider confidence and explanation fidelity had not been measured item by item.The behavior of coalition-based explanations under rounded API probabilities was also unmeasured to the authors’ knowledge.Source: §2 · p. 4

3.1 Definition

• A typed decision model takes a state, an instruction, and caller-chosen textual options, then scores the options with a function shared across them.The caller chooses the number of options; the output can support choice, binary yes/no, or an expected ordered score.Source: §3.1 · pp. 4–5

3.2 Two ways of returning a probability

• An autoregressive language model can score options by multiplying token probabilities, but the sum over caller options may be below one because probability mass can fall on other text.The direct typed-decider interface returns a distribution on the options in one pass per question.Source: §3.2 · p. 5

3.3 The two models

• Laya uses masked option anchors and a shared scoring head, while Jev’s architecture and training data are not published.The multilingual Laya checkpoint uses mmBERT-base with 322 million parameters and truncates states beyond 1,024 tokens; Jev is documented through its interface.Source: §3.3 · p. 5

4.1 Three confidences

• For fixed option count, Jev’s c_J and Laya’s c_max induce the same threshold ordering, while c_H corresponds to c_max only for binary choices.For three or more options, c_H and c_max can rank probability vectors differently; gates therefore use c_max.Source: §4.1 · p. 5

4.2 Calibration

• Top-label calibration means that among decisions assigned confidence c, the fraction correct is c; it is a group property, not a guarantee for one decision.The paper reports proper Brier and log-likelihood scores alongside ECE, and notes that a proper training objective does not guarantee calibration on new data.Source: §4.2 · pp. 5–6

4.3 Confidence as a detector of correct answers

• Confidence is also evaluated as a ranking score for correctness, distinct from calibration; AUROC equals the probability that a randomly drawn correct answer has higher confidence than a wrong one.AUROC of 1 would be required for confidence to prove correctness.Source: §4.3 · p. 6

4.4 Explanation fidelity

• The deletion-area explanation test removes words in descending importance and compares its curve with random removal, treating the comparison as a lower standard than retraining-based benchmarks.The perturbation both defines and evaluates importance, and deletion can create inputs unlike the training data.Source: §4.4 · p. 6

4.5 The selective gate

• A selective gate accepts decisions above a confidence threshold and escalates the rest; a point-estimate scan can select a threshold whose true accuracy is near its target, making the promise hold on new data with probability close to one half under stated heuristic conditions.This is a heuristic rather than a theorem because thresholds in the scan are dependent.Source: §4.5 · p. 7

4.6 Multi-objective formulation

• With calibrated top-label confidence, accepting exactly when C exceeds 1 − Cr/Ce minimizes expected cost per item when wrong actions cost Ce and escalations cost Cr.The multi-objective formulation links weighted wrong-action and escalation costs to this Bayes threshold; class-specific thresholds use fewer calibration items per threshold.Source: §4.5 · p. 7, §4.6 · p. 9

4.5 The selective gate

• The Beta lower-bound rule requires posterior credibility that accepted-case accuracy exceeds the target, but it is not a frequentist guarantee and rare qualification can select favorable calibration samples.Its ascending scan inherits a guarantee only under conditions captured by the comparison with the fixed-sequence scan; the scans can disagree when a lower threshold qualifies after a higher one fails.Source: §4.5 · pp. 7–8

4.7 Comparing two methods: superiority, non-inferiority, equivalence and efficiency

• Method comparisons distinguish superiority, non-inferiority, equivalence, and no difference detected using prespecified margins and interval-based tests.Efficiency requires non-inferiority plus a resource reduction; preregistered comparisons are confirmatory, while uncorrected exploratory findings are leads for further work.Source: §4.7 · pp. 9–10

5 Experimental setup

• Portuguese and English use the same MASSIVE intents and options item by item, with 2,000 shuffled test utterances per language and one gold intent plus four distractors per question.The test split uses seed 13; distractors are drawn from other test intents.Source: §5 · p. 10
• Jev 1.13 answers were collected through its API and frozen, while five Laya checkpoints include multilingual, English, typed-decisions, and Portuguese fine-tuned variants.The fine-tuned variants use 11,274 MASSIVE Portuguese training utterances after removing 240 exact test duplicates.Source: §5 · p. 10, Table 2 · p. 11
• Items 0–999 calibrate temperatures and gate thresholds, while items 1,000–1,999 report performance; the gate is also tested over 1,000 random calibration/report splits.Laya temperature is refitted on each calibration half in the repeated-split evaluation.Source: §5 · p. 10
• The audit treats each locale’s 2,000 items as a plug-in population and tests thresholds across calibration draws, counting a violation when true selective accuracy falls below target.It repeats the procedure on 200 bootstrap populations to separate item variation from Monte Carlo error.Source: §5 · p. 11
• A gate passes when the one-sided 95% upper bound on its violation rate is at most 0.05, but a draw with no qualifying threshold counts as keeping the promise.The authors therefore report decisions alongside violation rates; the Learn then Test guarantee is approximate because candidate thresholds are observed confidences and Laya temperature is fitted on the same sample.Source: §5 · p. 11
• Five seeded scripts produce the reported metrics, explanations, population-gate and split-half analyses, and tables from released items and frozen model outputs.The scripts use seed 20261020.Source: §5 · p. 11

6.1 Accuracy

• About 4,000 labelled pt-PT utterances bring fine-tuned Laya to Jev’s accuracy; without fine-tuning, Laya trails Jev by about 21 points, but fine-tuning exceeds Jev by 2.6 points.The full-partition fine-tune used in Table 4 reached 0.973; the paired non-inferiority measurement used a different recipe.Source: Figure 1 · p. 12, §6.1 · p. 12, Table 4 · p. 12

6.2 Confidence as evidence

• Confidence ranks a correct answer above a wrong one with probability 0.80 to 0.96, but no system reaches AUROC 1 or certifies any single answer.At confidence 0.97 or more, Jev still made errors in Portuguese and English, as did fine-tuned Laya.Source: §6.2 · p. 13, Table 6 · p. 13
• Temperature scaling repairs global Laya calibration, yet the Portuguese multilingual checkpoint’s ECE is 0.045 overall and 0.312 in the datetime scenario.Across non-fine-tuned checkpoints, median scenario ECE is 2.8 to 3.3 times global ECE in Portuguese and 1.8 to 2.9 times in English.Source: §6.2 · pp. 13, 15

6.3 The gate

• 39% to 53% of calibration draws violated the promise under the point rule, while every bound-based rule stayed within tolerance across the reported calibration sizes.Learn then Test and the fixed-sequence Beta rule violated in at most 3.2% of draws; SGR decided about one item per 1,000 for two Laya cases at 99%.Source: §6.3 · p. 15, Table 7 · p. 16, Figure 4 · p. 16
• The ascending Beta rule kept the promise in all 24 tested combinations and was non-inferior to Learn then Test, but four combinations showed detectable gains in decisions, including two trade-offs with higher violation rates.Across the 24 comparisons, none of the decision differences remained detected after Holm correction.Source: §6.3 · pp. 16–17, Table 8 · p. 17

6.4 Why the bound works, and where it fails

• Replacing the point estimate with a bound accounts for almost the whole effect; the uniform prior kept the promise in all 24 combinations, while the Jeffreys prior broke it in five.With 250 calibration items at a 99% target, no bounded rule qualified a threshold, while the point rule still broke the promise in 50% to 65% of splits.Source: §6.4 · pp. 18–19
• The ascending scan fails when disagreement between scans brings a coverage jump near the high-coverage plateau; under sampled-to-dev shift, Learn then Test and Beta broke a 99% promise in 30% and 41% of draws.The reported shift is from sampled test rows to the dev partition.Source: §6.4 · pp. 19–20
• 147 of 190 exchangeable evaluations broke the promise under the point rule, while Learn then Test kept it in all 190; the Beta rule decided more items in 128 and fewer in none, without its guarantee.The Beta rule had one broken case and six undetermined cases.Source: §6.4 · p. 19, Table 9 · p. 20

7 When the input is noisy

• The evaluation applied five deterministic noise operators to report items, including typos at 5%, 10%, or 20% of words and a mixed profile; the exploratory runs were registered before execution.Jev was measured on fresh clean and noisy API answers; Laya was measured as released and fine-tuned.Source: §7 · p. 20, Table 10 · p. 20
• Typos and mixed noise cost Jev 0.7–2.2 accuracy points, fine-tuned Laya 2.4–5.7, and released Laya 4.6–9.6; swapping adjacent words and removing punctuation changed little.The clean-text fine-tuning advantage narrowed under noise to between −2.7 and +0.5 points, with Jev ahead by 2.7 under the heaviest mixed English noise.Source: §7 · p. 20
• A clean-text threshold breaks the 95% promise on noisy text: for fine-tuned Laya, in 11 of 12 typo and mixed conditions, with 95%–100% violation rates; for Jev, in 9 of 12, with 13%–81% rates.The loss can be only a few accuracy points because it falls on accepted items and pushes their accuracy below the target.Source: §7 · pp. 20–21
• A single greedy typo changed 27.5% of fine-tuned Laya’s 200 Portuguese decisions and 69% of released Laya’s; flipped decisions had median clean confidence 0.9999.Among flipped Portuguese decisions, 88% had clean confidence of at least 0.99; Figure 6 shows confidence groups overlapping across typo counts.Source: §7 · p. 21, Figure 6 · p. 21
• Against the same fine-tuning recipe without noise, noise training yielded mixed-profile-at-10% accuracy gains of +1.6 in Portuguese and +0.8 in English, with clean accuracy non-inferior under the 2-point margin.The comparison was paired over items and averaged over three seeds.Source: §7 · p. 21
• A gate’s calibration items must resemble its deployment traffic, and callers should not treat a 0.9999 probability as protection against a single typo.The paper also recommends adding noise to fine-tuning data, which raised noisy accuracy without a clean-accuracy loss in these exploratory runs.Source: §7 · pp. 21–22

8 When the language changes

• The locale evaluation used parallel MASSIVE utterances across 52 locales and applied an en-US threshold chosen on 1,000 calibration items to separate report items in each locale.The analysis was registered before the exploratory runs; Laya’s temperature was fitted on the en-US calibration draw.Source: §8 · p. 22
• Fine-tuning in Portuguese raised Laya’s accuracy by 14.9–30.5 points in every measured locale, but the fine-tuned model was non-inferior to en-US only in pt-PT.Jev was non-inferior to en-US in 6 of 51 locales; released Laya was non-inferior in none.Source: §8 · p. 22
• Calibrated in en-US, Jev’s 95% promise held in 8 of 51 locales and failed in the other 43, with violation rates from 0.31 to 1.00.Je v’s ECE remained between 0.009 and 0.030 across locales; the promise held only where accuracy loss was at most 1.1 points, consistent with 1.2 points of slack in en-US.Source: §8 · p. 23
• Calibrating in each locale, the Beta rule kept the promise in 29 locales for released Laya and 18 for fine-tuned Laya, with median decisions of 119 and 757 per 1,000, respectively.Learn then Test kept the promise in 35 and 20 locales, with medians of 45 and 752 decisions; Figure 7 compares locale-calibrated Laya with Jev calibrated in en-US.Source: Figure 7 · p. 22, §8 · p. 23
• For multilingual deployment, the authors conclude that each language needs its own calibration sample; low ECE or a small accuracy loss does not replace it.The promise’s slack is the accepted-item accuracy minus its target; here, the English promise could absorb about one point of loss.Source: §8 · p. 23

9 When the domain changes

• The battery covered four domains with 2,000 items each, split evenly between calibration and report; fine-tunes used 250–8,000 domain labels across three seeds.FakeRecogna was replaced before answers were collected because 71% of fake items, versus 1.4% of real items, had a verdict marker in the title.Source: §9 · p. 23
• Fine-tuned Laya was more accurate than Jev from 250 labels on financial tweets (0.926 vs 0.728) and Fake.Br (0.981 vs 0.716), needed 8,000 labels for news non-inferiority, and was never non-inferior on LIAR2.On LIAR2, neither model moves far from always answering “false”, which is right for 28.5% of the report items.Source: §9 · p. 23
• Jev’s ECE ranged from 0.076 to 0.171 across the four domains, while temperature-scaled Laya stayed between 0.006 and 0.048.The authors conclude that calibration measured on one task licenses nothing about another; they describe domain-specific temperature fitting as a low-cost repair.Source: §9 · p. 23
• Across the four domains, the point rule kept the promise in 20 of 40 cases, versus 39 for Learn then Test and 35 for the Beta rule.The Beta rule often decided more: at 0.95, fine-tunes had medians of 517 versus 331 decisions per 1,000 on news headlines and 842 versus 532 on financial tweets. Table 12 summarizes the cross-domain battery.Source: Table 12 · p. 24, §9 · p. 25
• Clean-calibrated fine-tune promises failed under typos on every seed for news headlines and financial tweets, with violation rates from 0.21 to 0.92.On Fake.Br, typos alone cost little, but the mixed profile reduced fine-tune accuracy by 34–40 points and broke its promise in every draw.Source: §9 · p. 25
• On Fake.Br, removing punctuation alone cost fine-tuned Laya 34.3 accuracy points, and a surface-feature-only model reached 0.807; fine-tuning without punctuation retained 0.931 on punctuation-free text.The original fine-tune fell to 0.606 on punctuation-free text, supporting the authors’ account that it relied on punctuation available in the training data.Source: §9 · p. 25
• With an empty state, fine-tunes answered news headlines correctly on 0.32–0.37 of report items, versus 0.17 before fine-tuning and 0.23 for Jev.Where options were fixed, empty-state answers were a single majority-class choice: neutral on financial tweets and “false” on LIAR2 for Jev and two fine-tunes.Source: §9 · p. 25
• On LIAR2, no rule accepted an item at 90% or 95%; among the 20 most confident items, Jev was right on 45% and fine-tunes on 60%–65%.The statement-only rating has a low apparent accuracy ceiling, though the ceiling itself was not observed; a two-level false/true question allowed 90% promises for a tenth of decisions.Source: §9 · p. 25

10.1 The loop and its comparators

• The loop couples an audited-accuracy trigger, model promotion, and threshold re-qualification; its monitors use accepted-item audits rather than selectively escalated cases.It discounts audited accuracy evidence across windows and triggers when the posterior probability that accuracy is below α reaches 1 − q. The study compares it with eight alternatives using the same decider, retraining procedure, audit rate, and label stream.Source: §10.1 · p. 26

10.2 Streams and measures

• The evaluation uses synthetic MASSIVE shifts and year-to-year HuffPost streams, checking promise on post-change windows and measuring cost with audits and escalations.Synthetic streams include missing scenarios, a Portuguese-to-English change, and swapped intent labels; real transitions count as degrading only when frozen-model accuracy drops by at least 2 points. The primary cost ratio is 10, and the loop’s tuned λ = 0.95 and q = 0.01 were at the preregistered grid edge.Source: §10.2 · pp. 26–27

10.3 Results

• The preregistered claim that the loop improves on the best tuned comparator is not supported: it costs more in D1 and D3 and is cheaper only in D2.On the control stream it never retrains, while the periodic arm retrains three times; where drift degrades the model, Learn then Test re-qualified each window keeps the promise more often than retraining arms, but uses more reviews. The loop is not detectably more expensive than the periodic arm under the sensitivity analysis charging 50 reviews per retrain, which does not establish equivalence.Source: §10.3 · pp. 27–28

10.4 When the drift has a pattern

• The dynamic gate predicts next-window per-bin accuracy, but its Bayesian promise requires a correctly specified model and level q/B for each of B bins.A per-bin guarantee implies the mixture accuracy among accepted items meets α, but is sufficient rather than necessary; the experimental lower-bound variant uses q = 0.05 per bin, not for their mixture. No component can anticipate a change unsupported by past labels or inputs, and the guarantee does not survive a misspecified evolution.Source: §10.4 · p. 29
• On degrading drift, faster forgetting improves the level model’s promise and cost versus tuned recalibration, while known seasonal components help on S1 only at a review and cost trade-off.On S1, seasonal components reduce windows below α from 51% to 32%, with 267 more labels per 1,000 items and cost higher by 0.10; Learn then Test reaches 0.7% below α at cost 0.49. In D1 and D3, the level model reduces below-promise windows from 57% to 28% and 38% to 23%, respectively, at lower cost than recalibration. The gain comes from forgetting faster: recalibration at λ = 0.80 matches the level model, while cycles or changes that do not degrade the model gain nothing from dynamic components.Source: §10.4 · p. 30, Table 15 · p. 30

10.5 Numeric decisions over real time

• On M3, Laya’s accuracy matches a constant answer at 0.476 as released and 0.522 after fine-tuning, while Jev reaches 0.540 and ETS reaches 0.673.Jev’s interval [0.514, 0.568] contains the 0.522 accuracy of always answering rise; ETS’s 100 most confident decisions are right in 88% of cases.Source: §10.5 · p. 31, Table 16 · p. 31
• At M3 targets of 0.70 and 0.80, threshold gates fall below target in 17% to 50% of origins, while per-bin gates never do but act on fewer decisions.At target 0.70, per-bin gates act on 282 to 382 decisions per 1,000 versus 763 to 828 for threshold gates; at target 0.60, every gate keeps the promise. Per-bin gates are more conservative because each accepted bin must reach the target, and the seasonal dynamic model adds nothing measurable over the forgetting Beta.Source: §10.5 · p. 31, Table 17 · p. 31
• At target 0.70, sliding Learn then Test falls below target in 29% of origins (622 decisions per 1,000), versus 4% for per-bin gates (266 to 294).The per-bin gates keep the demanding target more often while making fewer decisions.Source: §10.5 · p. 32
• 65% of decisions are right with exponential smoothing, while the 1,000 most confident of 34,272 test decisions are right in 97%.Source: §10.5 · p. 32
• A threshold calibrated on pooled history keeps the average but breaks the demanding target in a third to half of periods; per-bin tracking keeps it by acting on fewer decisions.Source: §10.5 · p. 32

11.1 Probability is not explanation

• Confidence does not distinguish decisions resting on one word from decisions supported redundantly, and its relation to occlusion faithfulness is weak and changes sign.The fine-tuned Portuguese example has probability 0.997 with one word accounting for 0.995; another has probability 0.9999 but no single word changes the decision.Source: §11.1 · pp. 32–33

11.2 The decision as a coalition game

• For inputs of up to eight units, a fitted account using words and pairs explains a median 0.86 to 0.93 of coalition-game variance in the four Laya cases.Exact games require a median of 64 model passes per decision, versus 514 for KernelSHAP with 256 pairs of coalitions.Source: §11.2 · pp. 33–34

11.3 Credible intervals under rounding

• Under two-decimal rounding, Wi intervals cover 58.8% of exact values in Portuguese and 55.4% in English; none of the tested methods yields calibrated word-effect intervals.When a decision and most coalitions read 1.00, the data show only that log-odds exceed logit 0.995, not by how far.Source: §11.3 · pp. 34–35

11.4 Unique, redundant and synergic evidence

• On the planted redundant-evidence cases, the pairwise Shapley interaction index reaches macro F1 0.901, versus 0.300 for the Bayesian model.The planted test supports this use for redundant evidence, not unique or synergic evidence.Source: §11.4 · p. 35

11.5 Confidence and the faithfulness of the Shapley order

• For Laya, confidence has only a slight relation to Shapley-order fidelity, too weak to select which explanations to trust.The slope is 0.004 to 0.010 per log-odds unit, while Jev’s slope sign is decided in only one of four cases.Source: §11.5 · p. 36

11.6 Explanations across options and orders

• Contrastive explanations are tied to their option set and must be recomputed when the caller changes the options.Jev shows approximate invariance in eight comparisons, but 68% to 76% of items are excluded because replacements change the decision or runner-up.Source: §11.6 · p. 36
• The exact Shapley order and HEDGE lead deletion faithfulness, while occlusion is among the two least faithful methods despite being cheapest.Across six cases, the Shapley order deletes faster than occlusion by 0.05 to 0.23 in percentile; on Jev’s rounded games, its mean percentile is 0.94 to 0.96 versus 0.71 to 0.75 for occlusion.Source: §11.6 · p. 36

11.7 An additive head

• An additive head built for exact word contributions loses 28.7 accuracy points in Portuguese and 31.8 in English.The resulting head is additive in positions rather than words.Source: §11.7 · p. 36

12 A validation protocol

• Statements are tagged by source as documentation, code, measured, or not published, because vendor documentation and model versions can change over time.The documentation changed between 2026-09-29 and 2026-10-20, including experimental image input and version 1.14.Source: §12 · pp. 36–37
• Seven operational controls are converted into measurements that require no model-internal access, except explanation fidelity.The controls apply equally to open and closed deciders.Source: §12 · p. 37, Table 20 · p. 37
• Comparisons fix a justified margin before seeing data, claim non-inferiority only by tests, and report an interval containing zero as no difference detected.Resource use is measured alongside outcomes, and uncertainty includes item sampling as well as Monte Carlo error.Source: §12 · p. 37

13 What a caller can do

• Table 21 collects what the article’s measurements say to a caller of a typed decider.The table is described as gathering the sections’ interpretations alongside their supporting measurements.Source: §13 · p. 37, Table 21 · p. 38

14 Limitations

• Most measurements use MASSIVE with five options, and each of the four additional domains is represented by one dataset.Fine-tuned results may be inflated by near-duplicate test utterances; fine-tuned checkpoints were trained only in Portuguese.Source: §14 · p. 38
• Under drift or correlated items, neither the Beta rule nor Learn then Test retains its promise without further assumptions.The reported gate evaluation uses random splits of one fixed sample, so calibration and report items are exchangeable.Source: §14 · p. 38
• The noise evaluation is synthetic, and real traffic may include untested abbreviations, mixed languages, or voice-transcription errors.The cost ratios are illustrative scenarios rather than universal deployment costs.Source: §14 · p. 38
• Jev’s two-decimal rounding limits analyses near probability one, while coalition-game gates use rounded Laya as a stand-in and assume rounding drives coverage failure.The planted explanation result supports redundant evidence only; the fine-tuning data removed exact duplicates but may retain near duplicates.Source: §14 · p. 38

15 Conclusion

• 147 of 190 exchangeable evaluations broke the promise under a point-estimate threshold, while Learn then Test kept it in all 190 with a guarantee.The ascending Beta rule decided more items on average in 128 evaluations and fewer in none, but had no guarantee.Source: §15 · p. 39
• Under drift, the Bayesian loop avoids needless retraining when change is harmless, while re-qualifying thresholds in each window keeps promises more often where the model degrades.The loop reacts late under degrading drift and does not improve on periodic retraining; the arms that most often keep the promise incur more reviews.Source: §15 · p. 39
• Exact coalition games can explain single decisions, but confidence does not select faithful explanations and API rounding removes uncertainty statements about word effects.Explanations remain tied to the pinned model version and the options used to compute them.Source: §15 · p. 39
• Across locales and domains, a promise does not automatically transfer, and the practical rule is to calibrate on traffic-like data and recheck after traffic changes.English-calibrated Jev kept a 95% promise in 8 of 51 locales; fine-tuned Laya required 250 to 8,000 labels to match Jev depending on domain.Source: §15 · p. 39

Data and code availability

• The Zenodo package includes parallel locale items, frozen Jev and Laya outputs, coalition games, drift runs, a datasheet, and promise-check code.Jev’s probabilities for options it did not select are not distributed because the vendor agreement forbids using its output to train an imitation model.Source: Data and code availability · p. 39

Use of AI-assisted tools

• AI-assisted tools were used for code development and English-language editing; experimental design, analysis, and interpretation were the author’s.Source: Use of AI-assisted tools · p. 39

A Proofs

• A confidence-entropy threshold is not generally equivalent to a maximum-probability threshold once there are three or more options.For three options, the proof gives distributions whose maximum probabilities and entropies order differently.Source: A Proofs · p. 45
• The cost-optimal calibrated gate accepts when confidence exceeds 1 − C_r/C_e, where C_e is error cost and C_r is review cost.Ties may be resolved either way.Source: A Proofs · p. 45
• With prior Beta(1, 1), the per-bin predictive mean is Laplace’s rule of succession, while the Bayesian success-run bound gives the minimum qualifying sample size.The bound is n ≥ ⌈ln q / ln α⌉ − 1.Source: A Proofs · p. 46
• The fixed-sequence scan matches Learn then Test under the stated prior and testability condition, retaining its violation guarantee.The proof bounds the ascending scan’s violation probability by the Learn then Test violation probability plus the probability that the scans disagree.Source: A Proofs · p. 46
• The proofs show that, at equal coverage, the threshold set S* maximizes conditional accuracy among accepted sets.The argument also bounds the best achievable accuracy when the input alone omits information available in a finer representation.Source: A Proofs · p. 46
• For drift gating, the accepted-bin weighted mean reaches the target when each accepted bin qualifies, and a union bound controls the chance that any accepted bin falls short.The proof also shows that the single-level update corresponds approximately to evolving Beta counts by the forgetting factor before adding new labels.Source: A Proofs · p. 46

B What each route costs

• 4,000 labels make fine-tuned Laya non-inferior to Jev, and 11,274 labels make it more accurate; the break-even API-volume boundary trades label count against label price.Training on 4,000 labels took 17 seconds on one GPU; Figure 10’s green region marks when label cost is below the avoided API spend, while local inference is not free.Source: B What each route costs · p. 47, Figure 10 · p. 47

C An additive head

• The additive head reaches 0.614–0.749 accuracy in Portuguese and 0.589–0.713 in English, versus 0.960–0.972 for the original head, and its word scores do not explain its decisions.The pure additive head loses about 30 accuracy points; the penalized residual costs about 4 points and still does not explain itself by word.Source: C An additive head · p. 48

D Repairing the re-qualification of the loop

• In exploratory drift repairs, held-out-label re-qualification reduced below-promise windows by 9.2 points in D1, while a fixed-sequence scan reduced them by 21.7 points in D3 at 124 extra labels per 1,000.The three repairs together retained the D1 gain but lost it in D3; the proposed explanation for that loss was not tested.Source: D Repairing the re-qualification of… · pp. 48–49
• Across five preregistered degrading news-stream transitions, the loop’s mean-cost difference from periodic retraining was −0.27 per item with an interval containing zero, although it had the lowest mean cost in nine of ten transitions.In the largest-drop transition, Learn then Test per window was cheapest.Source: D Repairing the re-qualification of… · p. 49

E Gate at all targets, split-half evaluation

• The split-half evaluation fits temperature and threshold on calibration items, then measures accepted-item accuracy on a separate report half across 1,000 random partitions.The passage says the promise is judged against the plug-in population in Table 7.Source: E Gate at all targets, split-half… · p. 49

F Reference table by language

• The language table reports decider accuracy and 95% promise-violation rates on 1,000 parallel report items, comparing English-calibrated with locale-calibrated settings.Jev answered the new locales’ report items alone, so locale calibration there applies to Laya only.Source: F Reference table by language · p. 49, Table 23 · p. 50

G Report cards of the battery

• The domain report cards compare accuracy, ECE, promise violations, decisions per 1,000, noisy-text conditions, and empty-state accuracy across domain blocks.The caption identifies the point, ascending Beta, and Learn then Test rules at targets of 0.90 and 0.95.Source: G Report cards of the battery · p. 51, Table 24 · p. 51