Source-linked AI summary
One Threshold Does Not Fit All Languages: Language-Conditional Deferral for Reliable and Efficient Low-Resource Text Classification
Bhanu Prakash Vangala, Vangala Navya
TL;DR
Multilingual classifiers with human fallback need a reliability guarantee that applies within each language, not only across a pooled mix. This paper compares pooled and language-conditional conformal thresholds; per-language calibration brings coverage near the 90% target across languages, with different review costs. It recommends calibrating, reporting, and budgeting review separately by language.
Problem
Method
The paper compares a pooled conformal threshold with a separate Mondrian threshold for each language, without changing the trained model.
Source: §2 · p. 3Results
Language-conditional calibration brought coverage to 0.891–0.910 on MasakhaNEWS and 0.900–0.910 on AfriSenti, with no languages more than two points below target.
Source: §3.1 · p. 4Takeaways & Limitations
The authors recommend calibrating and reporting coverage per language, then budgeting human review from each language’s measured deferral rate.
Source: §4 · p. 4AfriSenti’s released development and test splits are not exchangeable, so benchmark certification holds in deployment only when calibration data match the traffic actually seen.
Source: §3.3 · p. 4
Abstract
from arXiv · showhide
In the Global South, the lower-income countries of Africa, Asia, and Latin America where most of the world's languages are spoken, a deployed text classifier usually runs on ordinary CPUs, serves many languages with a single model, has few labeled examples in any of them, and relies on people to catch its mistakes. Such a system is only useful if it can promise how often it will be wrong: at most a fixed fraction of the labels it assigns on its own may be incorrect, and everything else must go to a person. Split conformal prediction delivers this promise through a single confidence threshold, normally estimated on validation data pooled across languages. We ask whether the promise reaches every language, and it does not. On MasakhaNEWS (16 African languages) and AfriSenti (12 languages plus two never seen in training), a pooled threshold meets the 90% target on average but covers Somali at 77.5%, Tigrinya at 83.7%, and the two unseen languages at 77.5% and 81.2%. Estimating one threshold per language brings every language to between 89.1% and 91.0% without retraining, and it shows how unequal the cost of the promise is: keeping it means sending 43% of Somali news and over 80% of Amharic and Xitsonga tweets to a person, against under 8% of Nigerian Pidgin news. One or two hundred labels per language are enough and the models train in minutes on one CPU core, so the fix is affordable: calibrate, report, and budget human review one language at a time.
1 Introduction
In low-resource multilingual deployments, one pooled threshold can satisfy a reliability target on average without ensuring it for any individual language. This paper tests whether per-language thresholds can deliver the promise and what they cost in labels and human review.
1 Introduction
- Deployments in the described setting use ordinary CPUs, one model across languages, hundreds rather than hundreds of thousands of labeled examples, and human review for uncertain cases.The paper evaluates its question on two African-language classification benchmarks using a single CPU core.Source: §1 · pp. 1–2
- A pooled threshold provides only a marginal guarantee across the calibration mix, leaving reliability for any single language unspecified.An aggregate dashboard can report 90% coverage while missing undercoverage in Somali.Source: §1 · p. 2
- Up to 12.5 points of undercoverage in the lowest-resource languages is repaired by estimating one threshold per language, without retraining.The approach needs only 100 to 200 labeled examples per language.Source: §1 · p. 2
2 Reliability contracts from a single cheap model
The paper seeks a low-cost deferral rule that limits wrong automatic labels in every language while exposing each language’s human-review burden. It uses split conformal prediction with either one pooled threshold or a separate threshold for each language.
2 Reliability contracts from a single cheap model
- A deployment needs a rule that limits wrong automatic labels to fraction α in every language, minimizes deferral, avoids retraining, and makes each language’s review cost visible.The pipeline issues an automatic label only when the prediction set contains exactly one label.Source: §2 · p. 2
- The single multilingual classifier uses character n-grams and word unigrams and bigrams with multinomial logistic regression; calibration depends only on its probabilities.The calibration layer therefore applies equally to an encoder or an API-served model.Source: §2 · p. 2
- Split conformal prediction takes a finite-sample quantile of calibration scores, forms a prediction set, and defers inputs whose set is not a singleton.A wrong automatic label is a miscoverage event, so its probability is at most α; the experiments use α = 0.10.Source: §2 · pp. 2–3
- Pooled calibration uses one threshold across languages, while language-conditional Mondrian calibration estimates one per language and guarantees coverage of at least 1−α within each language.The conditional policy leaves the trained model unchanged and needs the input language, which multilingual routing already requires; review costs differ by language.Source: §2 · p. 3, Figure 2 · p. 3
3 Experiments
The experiments compare pooled and language-conditional calibration across two African-language classification benchmarks, measuring per-language reliability, review cost, and calibration-data requirements. Language-conditional thresholds bring coverage close to the 90% target in every language, while revealing unequal review burdens and sensitivity to calibration-distribution match.
3 Experiments
- The single-core models train in 3.4 minutes on MasakhaNEWS and 1.3 minutes on AfriSenti; official-test accuracy averages 87.6% and 62.9%, respectively.The experiments use 16-language MasakhaNEWS and 12 trained AfriSenti languages, with Oromo and Tigrinya held out from training; the models occupy 40 MB and 25 MB, respectively.Source: §3 · p. 3
3.1 A pooled threshold keeps the contract on average and breaks it per language
- 0.891–0.910 coverage on MasakhaNEWS and 0.900–0.910 on AfriSenti under language-conditional calibration, versus pooled undercoverage including Somali at 0.775 ± 0.019 and Tigrinya at 0.837 ± 0.019.Pooled coverage is 0.902 ± 0.006 on MasakhaNEWS and 0.901 ± 0.003 on AfriSenti; the pooled quantile is dominated by languages with more calibration examples and more confident predictions.Source: §3.1 · pp. 3–4
3.2 The price of reliability is not the same in every language
- 42.9% of Somali news is deferred under conditional calibration, versus 9.5% pooled; conditional deferral is 84.7% for Amharic and 85.1% for Xitsonga tweets, versus 34.6% for Igbo.Deferral closely follows base accuracy: Spearman ρ = −0.70 on MasakhaNEWS and −0.97 on AfriSenti.Source: §3.2 · p. 4
- With pooled calibration, only 9.3% of Oromo and 16.8% of Tigrinya tweets are labeled automatically, and those labels are correct 33.8% and 37.6% of the time; language-specific thresholds defer 96.8% and 95.0%, restoring 0.90 coverage.Oromo and Tigrinya have no training data for the model; the language-specific thresholds use a few hundred examples.Source: §3.2 · p. 4
3.3 How many labels does a language need, and where must they come from?
- Coverage averages 0.904 with 20 calibration examples and 0.897–0.903 with 50 or more, while standard deviation across seeds falls from 0.06 at n = 20 to 0.02 at n = 200.On MasakhaNEWS, deferral is 17.5% at n = 20 versus 10.9% with the full calibration half; the authors report that 100 to 200 labels per language are enough.Source: §3.3 · p. 4
- Using official development splits for calibration and official test splits for evaluation drops AfriSenti aggregate coverage to 0.879, with Moroccan Arabic at 0.798, Algerian Arabic at 0.848, and Nigerian Pidgin at 0.848.The released splits are not exchangeable; benchmark certification holds in deployment only when calibration data come from the traffic the system will actually see.Source: §3.3 · p. 4
4 Recommendations and limitations
The authors recommend per-language calibration, reporting, and review budgeting, while noting limits from the model choice and language identification.
4 Recommendations and limitations
- Calibrate thresholds, report coverage and deferral, and budget human review per language; treat high deferral as a reason to collect training data rather than lower the reliability bar.The conditional policy relies on language identification, which is weaker for low-resource languages; a fine-tuned encoder would defer less than the cheap model, though the pooled-versus-conditional gap belongs to the calibration layer.Source: §4 · p. 4
- Language identification is a deployment boundary for conditional deferral because it is weaker for low-resource languages.The authors also note that a fine-tuned encoder would defer less than their cheap model, while the pooled-versus-conditional gap belongs to the calibration layer.Source: §4 · p. 4
- A high deferral rate is a reason to collect training data, not to lower the reliability bar.The authors recommend reporting coverage and deferral per language and budgeting human review accordingly.Source: §4 · p. 4