Source-linked AI summary
Discovering Translation-Worthy Languages with E-Values
Wajdi Ben Saad, Safa Madiouni
TL;DR
Multilingual translation can improve or degrade classification, but existing uniform or heuristic routing lacks statistically controlled group selection. This paper uses paired e-processes to compare direct and translated paths continuously, freeze routing before testing, and evaluate the resulting policy. The router improves held-out accuracy over direct-only by 8.14 points on SIB-200 and 16.70 points on MASSIVE, with decisions stable across thresholds and outcome-independent orderings.
Problem
Existing uniform translation and heuristic language tiers do not statistically control when group-level translation improves multilingual classification.
Method
The router monitors paired fixes and regressions with an anytime-valid e-process, selects translation at a first crossing, and freezes the policy before test evaluation.
Results
The router improves held-out accuracy over direct-only by 8.14 points on SIB-200 and 16.70 points on MASSIVE.
Takeaways & Limitations
Paired e-processes provide statistically controlled, auditable, task-specific multilingual routing decisions that remain stable across thresholds and 50 outcome-independent orderings.
Takeaways & Limitations
The process targets positive paired accuracy rather than a practical improvement margin or intervention cost, and selective translation may reinforce disparities across linguistic groups.
Abstract
from arXiv · showhide
Choosing when to translate multilingual documents is a central routing problem in text classification: translation can improve predictions for some languages while degrading others or adding unnecessary computation. Uniform translation and heuristic language tiers do not provide statistically controlled route selection. We introduce a language-level router based on paired e-processes that continuously compares direct and translation-assisted classification before freezing a routing policy. A familywise-controlled threshold of 280 bounds the probability of any false route across 14 eligible languages per dataset by 0.05. On SIB-200 and MASSIVE, the router selects translation for 4 of 15 languages and 14 of 15 locales, improving held-out accuracy over direct classification by 8.14 and 16.70 percentage points, respectively. All 28 decisions remain stable across 50 outcome-independent orderings and relative to the per-group threshold. Our results demonstrate that paired e-processes enable statistically controlled, anytime-valid, and auditable multilingual classification routing.
1 Introduction and related work
The paper frames multilingual translation as a group-level routing problem where translation may help or hurt, and proposes paired e-processes to select interventions only after accumulating statistically valid evidence. The resulting discovery–freeze–test protocol produces auditable, task-specific routing decisions rather than universal language tiers.
- Motivation: Translation benefits vary by language, task, and pipeline, making fixed resource tiers insufficient evidence for routing decisions.The paper contrasts group-level routing with instance-level strategy learning and aims to assign a predefined group to translation only after evidence accumulates.
- Method: Paired e-processes continuously monitor discordant correctness outcomes between direct and translated classification paths.Translation is favored when fixes are more likely than regressions among examples where the two paths disagree.
- Protocol: The discovery–freeze–test protocol converts first crossings into an auditable routing policy evaluated on held-out data.All routes are frozen before test evaluation, preventing test outcomes from changing the selected policy.
- Evaluation: The study evaluates independently discovered routers on SIB-200 topic classification and MASSIVE intent classification.The contribution is a paired-evidence formulation of group routing rather than a new e-process.
- Implication: The resulting routing boundary is task-specific rather than a universal language tier.This claim is tested across the two multilingual classification settings.
2 Method
The method reduces paired direct-versus-translation comparison to discordant outcomes, monitors them with an anytime-valid e-process, and routes at the first threshold crossing. Familywise allocation uses threshold 280 across 14 eligible groups, while fixed ordering and freezing make the resulting policy auditable.
- 2.1 Sign-sufficient paired reduction: For each group, correctness indicators compare a default path D with an intervention path T on held-out discovery examples.The method records whether each path predicts the reference label correctly.
- 2.1 Sign-sufficient paired reduction: Discordant examples become fixes or regressions, and q_g is the fix probability among discordances.Concordant pairs contribute equally to both accuracies, so the sign of the paired accuracy difference is determined by discordances.
- 2.2 Discordance e-process: The e-process tests H0,g: q_g ≤ 1/2 using a fixed normalized mixture over alternatives p > 1/2.Under the stated sequential null condition, likelihood-ratio processes are nonnegative supermartingales and their mixture retains the e-process property.
- 2.3 Anytime validity and evidence allocation: The first crossing τ_g = inf{t : E_g,t ≥ 1/α_g} remains valid under continuous inspection and stopping.This anytime-valid rule permits routing when evidence first reaches the threshold rather than only at a fixed horizon.
- 2.3 Anytime validity and evidence allocation: The familywise allocation α_g = .05/14 yields threshold 280 and bounds the probability of at least one false route by .05.Threshold 20 is retained only as a per-group sensitivity analysis.
- 2.4 Evidence time, routing, and freezing: Evidence accumulates in discordance time, while approximately t/d_g paired examples are needed to reach t discordances.This requirement depends on both intervention advantage q_g and the frequency d_g of informative comparisons.
- 2.4 Evidence time, routing, and freezing: A fixed seed and example identifier determine the order, and routes are frozen before test evaluation after the first crossing or retained default decision.The frozen route includes the threshold, selected groups, models, and backends.
3 Empirical evaluation
The evaluation tests independently discovered routers on SIB-200 and MASSIVE, combining Monte Carlo calibration with held-out accuracy, latency, and robustness analyses. The router selects translation selectively on SIB-200 and broadly on MASSIVE, with stable decisions and improved held-out accuracy over direct-only classification.
- Evaluation setting: 15 languages and 15 locales are evaluated with separate discovery and test sets, comparing direct-only, translation-only, fixed-tier, and paired-test routing policies.Each dataset has 14 translation-eligible non-English groups; English remains on the direct path.
- Calibration and power: 0.26% is the empirical null probability of crossing threshold 280 by 500 discordances, while power reaches 78.76% at q = .60.The corresponding per-group threshold 20 has higher power but weaker error control, illustrating the evidence–power trade-off.
- Held-out confirmation: 8.14 points is the SIB-200 accuracy gain over direct-only, while MASSIVE gains 16.70 points and translates every eligible locale.On SIB-200, selective routing translates 26.67% of examples and reduces mean latency by 70.5% versus translation-only; MASSIVE matches translation-only latency.
- Robustness and interpretation: All observed routing decisions remain unchanged across thresholds, α values, and 50 outcome-independent orderings.The exact paired-test router selects the same groups, so the contribution is valid continuous monitoring and reproducible policy freezing rather than superior endpoint selection.
4 Conclusion and limitations
The paper concludes that paired e-processes can replace fixed language heuristics with auditable, task-specific routing while keeping statistical evidence separate from downstream utility. It also identifies missing intervention-cost and practical-margin criteria and warns that selective translation may reinforce disparities.
- Limitations: The process targets positive paired accuracy for each group–task population but does not encode a practical improvement margin or intervention cost.
- Limitations: Selective translation improves computational efficiency but may reinforce performance disparities across linguistic groups.
- Conclusion: Paired e-processes provide auditable, task-specific routing decisions while keeping statistical evidence distinct from downstream utility.
A Technical appendices and supplementary material
The supplementary materials provide frozen policy results, latency measurements, confidence intervals, threshold-crossing analyses, and a confirmatory quality–translation trade-off. Experiments used cached or stored predictions for later analyses, with hardware-specific latency reporting.
- Subsequent statistical analyses reused cached predictions and required no additional model inference or translation.
- Frozen application-level policy results report translation and accuracy as percentages.
- Measured policy latency is reported in seconds per example from steady-state complete-path measurements, excluding model loading and cold starts.The times are hardware-specific.
- Held-out e-router accuracy gains are accompanied by percentile 95% confidence intervals based on 2,000 paired bootstrap replicates.Examples are resampled within language.
- The supplementary analyses include a synthetic threshold-crossing power curve and a confirmatory quality–translation trade-off.Figure 2 compares threshold-crossing probabilities, while Figure 3 addresses the quality–translation trade-off.
NeurIPS Paper Checklist
The checklist reports that the paper documents its methodology, reproducibility materials, statistical analyses, compute resources, asset licensing, and societal-impact considerations. It also states that the study uses public resources and does not involve human participants or a high-risk released asset.
- Experimental setting and reproducibility: The paper specifies datasets, splits, model families, routing rules, statistical thresholds, and evaluation metrics, with exact configurations and backend details in the supplement and code.
- Open access to data and code: A public code package includes environment specifications, configuration files, scripts, and reproduction instructions for experiments using public datasets.
- Experiment statistical significance: Policy-level accuracy gains use percentile 95% confidence intervals from 2,000 paired bootstrap replicates, supplemented by null-crossing and power simulations.
- Experiments compute resources: The supplement reports hardware, accelerator, memory, batching, worker settings, execution strategy, and wall-clock times for final runs.
- Ethics and responsible release: The study uses established public benchmark datasets and pretrained models without human-subject data collection or a released high-risk model.
- Societal impacts: The checklist identifies selective translation's computational-efficiency benefit alongside a risk of reinforcing performance disparities across linguistic groups.