One Confidence Threshold, Unequal Coverage: Language Gaps in Typed Decision Interfaces and How Translating Inputs Reduces Them

1 Introduction

• Typed decision interfaces return probabilities for fixed multiple-choice options, letting developers apply a confidence threshold across users’ languages.The gate acts automatically above the threshold and otherwise defers to a person.Source: §1 · pp. 1–2
• The confirmatory study tests 1,000 MASSIVE intent items in 51 languages on three vendors’ interfaces; exploratory checks vary classifiers, schemas, formats, and input origins.The study also examines requests written natively in 16 African languages with English translations.Source: §1 · p. 2
• 8.6–11.7 points: among items answered correctly in both languages, the gate handles English more often on average across all three interfaces.Observed coverage is lower than English’s in every other language, while handled requests are rarely wrong.Source: §1 · p. 2
• With each language’s threshold fitted to a 5% selective-error target, English’s held-out coverage still leads the mean other language by 17.3–18.9 points.This exploratory comparison uses thresholds fitted on tuning items.Source: §1 · p. 2
• English remains ahead when inputs are human translations of requests originally written in non-English languages, by 25.0–33.4 points.The native-input check covers 16 African languages.Source: §1 · p. 2
• Translating inputs into English reduces the gap at threshold 0.9 to 0.8–1.4 points, but English retains a lead at a common 5% error target.Exploratory checks also find that per-language thresholds and conformal sets do not eliminate English’s coverage advantage.Source: §1 · pp. 2–3

2 Setup

• At threshold 0.9, the gate handles a request when its top option’s probability is at least 0.9 and defers it otherwise.The returned option is always the top option; confidence means its probability, not the interface’s separate confidence field.Source: §2 · p. 3
• The confirmatory sample contains 1,000 MASSIVE items available in all 51 languages; exploratory data add 150 MASSIVE items and 300 XNLI pairs.Each confirmatory item is represented by its language versions, yielding 51,000 requests per model.Source: §2 · p. 3
• Coverage, selective error, top-option accuracy, and top-label ECE are reported per language; the paired gap compares handling on items correct in English and language L.The paired gap averages each language’s difference in gate handling across the 50 non-English languages.Source: §2 · p. 3
• The confirmatory plan fixes the item sample, three models, threshold, scoring rules, and paired handling-gap endpoint, with one test per model.Inference resamples the same 1,000 items across languages; Holm’s procedure adjusts the three primary p-values.Source: §2 · p. 3
• Table 1 summarizes the 0.9 confirmatory comparison, including paired handling gaps, correct-but-deferred requests, and coverage standardized to a 75% correct-answer share.Its caption reports percentages or percentage points and identifies intervals and detected language differences.Source: Table 1 · p. 4
• The plan was written before the first confirmatory request, while analyses using exploratory data or unplanned confirmatory checks are labeled exploratory.Exploratory analyses are not corrected across analyses.Source: §2 · p. 4

3 Does one threshold serve the languages equally?

• 8.6–11.7 points: on items answered correctly in both languages, the gate handles English more often on average across the 50 other languages in all three interfaces.The primary endpoint meets the confirmatory success criterion for each model; Holm-adjusted p < 0.001 for each.Source: §3 · p. 4
• Coverage is below English’s in all 50 languages for each model, detectably so in 44–47; the largest observed selective error is 8.2%.For gpt-6-luna, coverage is 59% in English, 36% in Swahili, and 27% in Javanese.Source: §3 · p. 4
• For gpt-6-luna, ECE detects no language difference despite coverage trailing English by up to 32 points.The ECE differences are at most 1.8 points, with a simultaneous band of median half-width 3.6 points.Source: §3 · pp. 4–5
• 32% of Swahili requests are deferred despite a correct top label, compared with 22% of English requests.Among correctly answered requests, the gate handles 72% in English and 51% in Swahili.Source: §3 · p. 5
• In exploratory MASSIVE results, the simultaneous band places 9, 15, 23, and 35 languages below English at thresholds 0.7, 0.8, 0.9, and 0.95, respectively.No language is detected above English at these thresholds.Source: §3 · p. 5
• On XNLI, selective error reaches 29% for Urdu, and ECE differs from English in 10 of 14 languages.This contrasts with the absence of detected ECE differences for gpt-6-luna on MASSIVE.Source: §3 · pp. 4–5
• At a common 5% selective-error target, English’s held-out coverage exceeds the mean other language’s by 17.3–18.9 points across the three interfaces.The thresholds are fitted on one random half and evaluated on the other; held-out selective error is 5.1–5.2%.Source: §3 · p. 5

• On MASSIVE, English’s empirical coverage curve reaches the highest coverage at 5% selective error, tied with Turkish.Figure 2 plots observed selective error against coverage as the threshold falls, with one line per language and markers at 0.9.Source: Figure 2 · p. 6
• 6.9–9.0 points: English’s coverage lead over the mean other language remains after standardizing every language to 75% correct answers.Every 95% interval is above zero.Source: §4 · p. 6
• 25.0–33.4 points: English translations of natively written African-language requests are handled more often than their native originals on jointly correct items.Machine-translated English versions show the same direction, with mean gaps of 13.3–32.1 points across five languages.Source: §4 · p. 6, Table 4 · p. 7
• Localizing the schema leaves the mean paired handling gap above zero in all three interfaces, though its size changes by interface.On gpt-6-luna, it grows from 8.3 to 18.1 points; changes on the other interfaces are under three points and undetected.Source: §4 · pp. 6–7
• The multilingual classifier’s mean paired handling gap is 1.8 points, versus 8.6–11.7 for the interfaces, but its handled answers are wrong up to 13.9% of the time.It handles 91% of English requests, compared with 59–78% for the interfaces; the systems differ in training and architecture.Source: §4 · p. 7
• Schema rewording, asking the gate’s question alongside two others, and three open-weight models preserve English’s high coverage rank, while a small model illustrates that high coverage can coexist with selective error up to 34%.The small model handles 71–83% of XNLI requests in every language, a failure that a coverage check alone would miss.Source: §4 · p. 7

5.1 Translating inputs into English

• 0.8–1.4 points: translating inputs into English reduces the mean paired handling gap at threshold 0.9 in all three interfaces, from 8.6–11.7 points.Fewer languages fall detectably below English, though selective error rises for some languages as their coverage increases.Source: §5.1 · p. 7, Table 5 · p. 8
• The reduction remains detectable on jointly correct items, excluding exact translation matches and among the half of pairs least similar to the English utterance.18.4% of translations match the English utterance after normalization.Source: §5.1 · p. 7
• 6.7–13.7 points: English’s held-out coverage lead at a 5% error target remains after translation, down from 17.3–18.9 points.The mean other language’s held-out coverage rises by 11.3 [9.2, 13.1] points on Microsoft-Decision-1 and 8.1 [6.4, 9.9] on Perplexity Decider; the 3.6 [−3.7, 8.7] change on gpt-6-luna is not detected.Source: §5.1 · pp. 7–8

5.2 Thresholds set per language from about a hundred labels

• An English-fitted threshold misses the 5% error budget on held-out items in 12 of 51 MASSIVE languages and all 15 XNLI languages.The worst MASSIVE language reaches 11% error while English remains within budget at 0%.Source: §5.2 · p. 8
• Own thresholds leave coverage unequal and often exceed the error budget: among MASSIVE languages within budget, coverage ranges from 31% to 79%, and 47% of language–split pairs miss the budget on average.On XNLI, own thresholds exist in three languages and exceed the budget in all three.Source: §5.2 · p. 8
• Neither certification procedure certifies a threshold on the exploratory split, so the certified gate defers every request on MASSIVE and XNLI.The sample has 83 tuning items per MASSIVE language; the procedures require at least 122 or 45 handled items under their stated tests.Source: §5.2 · pp. 8–9

5.3 Thresholds certified with larger label budgets

• At 300 labels per language, fixed-sequence certification succeeds in 63% and 55% of language–split pairs on Microsoft-Decision-1 and Perplexity Decider, respectively.At 500 labels, English coverage on Perplexity Decider is 78%, against 46% for the mean other language; adding 1.00 to the threshold grid reverses the comparison for gpt-6-luna.Source: §5.3 · p. 9, Table 6 · p. 9

5.4 Per-language conformal thresholds

• Per-language conformal thresholds reach the 90% set-coverage target in every language on average, while English’s singleton-rate lead over the lowest language widens from 30 to 51 points on MASSIVE.Part of the widening follows from two-decimal probability rounding.Source: §5.4 · p. 9

• Prior multilingual calibration work examines probability values and how to repair them, while selective-prediction work examines accuracy among kept answers and confidence-based error ranking.The cited work includes pretrained and question-answering models, LLMs, and abstention strategies.Source: §6 · p. 9
• Earlier work reports that English-developed abstention strategies lose accuracy in low-resource languages and that confidence-based abstention can widen group disparities.The present study compares language versions of the same items in production interfaces.Source: §6 · p. 9
• Concurrent typed-interface studies examine per-language accuracy, risk–coverage curves, or calibration and error ranking, whereas this paper measures one fixed threshold’s coverage across 51 languages and tests per-language thresholds.Related conformal-threshold work reports a pooled 90% coverage target met on average but 77.5% coverage for Somali; another study finds tension between equal coverage and equal set size.Source: §6 · pp. 9–10

• The confirmatory study covers MASSIVE only; smaller exploratory checks cannot resolve individual language ranks, and the native-input check covers one task and 16 African languages.MASSIVE and XNLI non-English versions are translations from English; the native-input check has 31–40 human-translated items per language, and translate-test used one translator.Source: §7 · p. 10
• The study measures deferral but not human-review availability, latency, or outcomes, so it does not assess the cost of review.Source: §7 · p. 10

8 Conclusion

• At a 0.9 threshold, English answers that are also correct in another language are handled more often, while handled requests are rarely wrong.The coverage gap persists after equalizing accuracy and fitting language-specific thresholds to a 5% error target.Source: §8 · p. 10
• Translating inputs into English reduces the 0.9-threshold gap from 8.6–11.7 to 0.8–1.4 points, but English retains a 6.7–13.7-point lead at a common 5% error target.The same-direction gap appears for requests written natively in African languages when English is their human translation.Source: §8 · p. 10
• The authors recommend reporting per-language coverage at the operating threshold, alongside calibration.They also propose reporting the mean paired handling gap against a reference language.Source: §8 · p. 10

Broader Impact Statement

• A gate that defers more requests in some languages gives their speakers less automation and may mean longer waits when deferred requests go to human review.Per-language coverage reports can expose this difference before deployment.Source: Broader Impact Statement · p. 10

A Confirmatory study details

• The study’s 1,000 MASSIVE items were drawn from 2,633 eligible ids, with the pool excluding items used in exploratory work and an earlier pilot.All languages received the same items in their dataset translations.Source: A Confirmatory study details · p. 14
• The confirmatory analysis prespecified its primary endpoint, bootstrap, decision rule, retries, and minimum answered-request requirement; secondary measures were reported without a decision rule.An arm with fewer than 95% answered requests counted as failed and entered no test.Source: A Confirmatory study details · p. 14
• The highest exact 95% upper bound on selective error was 12.0% for gpt-6-luna in Javanese, and its per-language coverage correlated 0.84 with the exploratory study.The 12.0% bound corresponds to 22 errors among 272 handled requests.Source: A Confirmatory study details · p. 14
• Table 7 reports accuracy, coverage at 0.9, and dL for every language and model.dL is the English-minus-language difference in handling among items correct in both languages.Source: Table 7 · p. 15

B Methods of the exploratory additions

• Translate-test sends non-English inputs translated by gpt-5.4-mini to each interface with the unchanged English schema, then compares against the confirmatory English-input results.Translation similarity was measured by exact match and chrF; Table 8’s caption says the gap reduction holds on fixed items and without translations reproducing the English utterance.Source: B Methods of the exploratory additions · pp. 15–16, Table 8 · p. 16
• The native-input analysis pairs native utterances with human English translations in InjongoIntent and also tests machine-translated Swahili, Amharic, Hausa, Yoruba, and Zulu inputs.Each interface receives 7,644 requests; gpt-6-luna refusals are scored as deferred and wrong.Source: B Methods of the exploratory additions · p. 16
• The fine-tuned XLM-RoBERTa classifier handles 96% of correct English answers, has ECE of 9.4–14.0% in every language, and has 26 languages below English in coverage.For the interfaces, the share of correct English answers handled is 72–85%.Source: B Methods of the exploratory additions · p. 16
• Threshold-label budgets use nested tuning and held-out sets from shared item permutations, with Learn-then-Test chosen to bound selective error.The comparison notes that conformal risk control instead bounds the share of all requests that are handled and wrong.Source: B Methods of the exploratory additions · p. 16

C Full per-language tables

• Table 9 lists MASSIVE results for all 51 languages sorted by coverage under the English threshold, while Table 10 gives all 15 XNLI languages.Table 10 has 156 tuning and 144 held-out items.Source: Table 9 · p. 17, Table 10 · p. 17

D Schema translation checks

• Localizing the schema lowers coverage in more languages than it raises for gpt-6-luna, while the other interfaces’ language rankings barely move.For gpt-6-luna, mean coverage changes by −7.6 points and English’s lead over the lowest language grows from 31 to 40 points; Figure 3 plots English-schema versus localized-schema coverage.Source: D Schema translation checks · p. 17, Figure 3 · p. 18
• With the localized schema, gpt-6-luna coverage ranges from 16–59%; Swahili is handled 35% of the time versus 56% for English, and its gap narrows from 25 to 21 points.Across languages, coverage changes more than accuracy or selective error in most cases; the table caption identifies the compared metrics and conditions.Source: D Schema translation checks · p. 18, Table 11 · p. 18
• The mean paired handling gap rises from 8.3 to 18.1 points for gpt-6-luna under schema localization, while it changes from 10.7 to 8.4 for Microsoft-Decision-1 and 8.1 to 7.4 for Perplexity Decider.The interval for the gap change is [7.6, 12.0] points for gpt-6-luna, [−5.3, 0.6] for Microsoft-Decision-1, and [−2.2, 0.7] for Perplexity Decider.Source: D Schema translation checks · p. 19
• Back-translation checks find mean schema equivalence of 4.71/5, but neither judge drift nor lexical drift has a detected association with coverage changes under localization.The reported correlations are −0.14 (p = 0.35) for judge drift and −0.26 (p = 0.070) for lexical drift.Source: D Schema translation checks · p. 19
• English rewording raises mean coverage by 2.5 points and preserves the original language ordering more closely than schema localization.Its Spearman correlation with the original ordering is 0.88; the paired difference from the localized schema correlation is 0.69, with 95% interval [0.45, 0.76].Source: D Schema translation checks · p. 19
• On 150 Swahili items, the corrected schema changes coverage from 34.7% to 36.0% and the English–Swahili gap from 21.3 to 20.0 points.The coverage difference’s 95% bootstrap interval is [−5.3, +7.3]. The reproducibility appendix defines empirical and certified threshold fits, including the fixed-sequence procedure.Source: D Schema translation checks · p. 19, E Reproducibility details · p. 21

F Threshold sweep and further checks

• 6–15 points: the mean English-minus-language coverage gap on MASSIVE grows across thresholds 0.7, 0.8, 0.9 and 0.95, with Javanese lowest throughout.The English–Swahili gap is 17, 25, 31 and 32 points at those thresholds, respectively.Source: F Threshold sweep and further checks · p. 21, Figure 4 · p. 22
• 47%: own-threshold fits miss the 5% held-out error budget in language–split pairs on average across 200 overlapping splits.Misses range from 22–64% by language; the English threshold misses in 69% of pairs.Source: F Threshold sweep and further checks · p. 22

F.1 Multi-question requests

• Coverage and English-minus-language gaps are nearly unchanged by adding questions to the request: cross-format rank correlations are 0.96 on MASSIVE and 0.98 on XNLI.English has the highest coverage in both formats, and the simultaneous band detects no language above English.Source: §F.1 · pp. 22–23

G Risk–coverage details

• At common risk or coverage comparisons, English retains an advantage on XNLI, while MASSIVE errors at 50% coverage show no detected language difference.At 50% XNLI coverage, 13 of 14 languages have higher selective error than English; on MASSIVE the mean language exceeds English by 2 points, with no individual language detected.Source: G Risk–coverage details · p. 23

H Thresholds fitted to a common risk target, and a certification grid up to 1.00

• Held-out thresholds fitted to a common 5% risk target still place 7–46 languages below English, while held-out error exceeds 5% in about half of language–split pairs.Including threshold 1.00 in the certification grid can reduce certification; at 500 labels the certified gpt-6-luna coverage is 2% for English and 3% for other languages on average.Source: H Thresholds fitted to a common risk… · pp. 24–25

I Conformal comparison details

• Per-language conformal thresholds bring set coverage to at least 91% in every MASSIVE language on average, but widen the English-to-lowest singleton-rate gap from 30 to 51 points.This parallels the tension between equal set coverage and equal set size; for gpt-6-luna, probability rounding mechanically contributes to the widening.Source: I Conformal comparison details · p. 25
• Under APS, per-language calibration also widens the English-minus-lowest singleton gap, from 30 to 35 points on MASSIVE and from 27 to 41 on XNLI.The APS widening is smaller than the LAC increase from 30 to 51 points on MASSIVE.Source: I Conformal comparison details · p. 26

J Open-weight models and the small comparator

• For open-bosun and open-lumma, pooled non-English selective error is 14% and 22%, against 7% and 5% for English; open-decider and gpt-6-luna have similar observed errors across languages.Source: J Open-weight models and the small… · p. 26
• English’s lead over the median language at 0.9 is 16, 25, and 11 points for open-bosun, open-decider, and open-lumma, respectively, and 11 for gpt-6-luna.Table 17 reports English ranking first or second in coverage in every model.Source: J Open-weight models and the small… · p. 26, Table 17 · p. 27
• Coverage orderings correlate with gpt-6-luna’s at Spearman 0.76, 0.54, and 0.47 for open-bosun, open-decider, and open-lumma; their ten-lowest-language sets overlap by 6, 5, and 5 languages.The open-lumma bottom set is not ten languages: 12 languages have zero coverage and cannot be ordered among themselves.Source: J Open-weight models and the small… · p. 27
• On XNLI, only open-decider clears the threshold often enough for comparison; open-bosun handles about 3% of requests at best, and open-lumma clears it on a single item in one language.Open-decider handles 60% of English requests, compared with 68% for gpt-6-luna.Source: J Open-weight models and the small… · p. 27
• Laya-multilingual’s MASSIVE coverage tracks its weak accuracy, while on XNLI it handles 71–83% across languages despite selective error ranging from 9% to 34%.By contrast, gpt-6-luna’s MASSIVE selective error stays within 0–8% while coverage varies by language.Source: J Open-weight models and the small… · p. 27

K Further related work

• Related work contrasts multilingual calibration and abstention studies with this paper’s empirical threshold fit, which lacks the finite-sample guarantees available from conformal risk control and Learn-then-Test.The passage also notes group-conditional risk control as a route to guarantees per group.Source: K Further related work · p. 27