PaperlayerView exact revision on arXiv

Source-linked AI summary

Structure, Association, and Decision Value: Representation-Based Difficulty Estimation for Adaptive Inference in African-Language NLI

Toheeb Ogunade

arXiv:2608.19003v1cs.CL

TL;DR

The paper asks whether representation statistics can estimate example-level value for adaptive inference in multilingual African-language NLI. Across clean-language evaluations and frozen checkpoints, it finds that tested signals do not make adaptive routing preferable to always-expensive inference under the tested budgets.

  • Problem

    The study addresses whether statistically significant representation–difficulty associations provide enough incremental information to support useful adaptive-routing decisions.

  • Method

    The paper evaluates frozen multilingual NLI checkpoints across 15 clean African-language configurations and compares representation statistics with two marginal-benefit targets.

  • Results

    Under the tested models, signals, and compute budgets, evaluated signals do not make adaptive routing preferable to always-expensive inference, although an oracle reaches 0.688 at 60% compute.

  • Takeaways & Limitations

    A representation statistic can be significant for one computational-benefit target yet irrelevant to another, making it a poor decision variable for routing in this setting.

  • Takeaways & Limitations

    Because checkpoints were frozen and seed-averaged marginal-gain targets were unavailable, measured relationships may reflect checkpoint idiosyncrasy rather than example difficulty.

Abstract

from arXiv · show

We ask whether internal representation statistics can provide useful example-level difficulty signals for adaptive inference in multilingual African NLP, and find that they cannot in this setting. Studying natural language inference across 15 African languages with frozen off-the-shelf checkpoints, we report four results. First, AfriXNLI's English configuration shares 1,047 of its 1,050 examples verbatim with XNLI evaluation data, and one widely used NLI checkpoint scores 1.000 on that test split, consistent with XNLI test exposure. Because AfriXNLI is derived from XNLI, its English, French and Swahili configurations cannot serve as clean evaluations for XNLI-trained models. Second, parameter count does not reliably order capability across African languages: our larger checkpoint is better in seven languages and worse in eight, with no significant aggregate difference. Third, across three multilingual representation spaces, angular dispersion is consistently more language-determined than effective rank, so pooled correlations can inflate one and mask the other. Fourth, the association that survives language control depends on the target: effective rank predicts probability gain from escalation but not whether escalation changes the prediction, while cheap-model confidence shows the opposite pattern; the two targets correlate at only 0.655. Under the tested models, signals, and compute budgets, no evaluated signal makes adaptive routing preferable to always-expensive inference, although an oracle exceeds it by 11 accuracy points at 60% of the compute. Our central methodological finding is that a representation statistic can be statistically significant for one notion of computational benefit while being irrelevant to another, and therefore be a poor decision variable.

1 Introduction · 2 Background and Related Work

The paper shows that statistically persuasive representation–difficulty associations can still fail as decision signals for adaptive inference. It therefore examines evaluation validity, capability ordering, language confounding, target dependence, and routing utility before drawing negative conclusions.

  • 1 Introduction: The study tests whether a cheap model’s internal representation can identify inputs worth escalating in low-resource multilingual NLI, where compute is scarce and performance varies widely.The paper presents this as a setting where assumptions behind representation-based difficulty methods are least often checked.
  • 1 Introduction: Before assessing representation geometry, the evaluation must separate example difficulty from language identity and benchmark contamination; the initial setup failed both checks.Correcting these issues changed the overall answer.
  • 1 Introduction: 1,047 of AfriXNLI’s 1,050 English examples overlap verbatim with XNLI, while contamination linked to checkpoint training splits is exposed by development–test accuracy gaps.This evaluation-validity analysis does not require access to training data.
  • 1 Introduction: Parameter count does not reliably order accuracy across African languages, undermining the compute hierarchy presupposed by a model cascade.The capability-ordering stage therefore tests whether larger models are consistently better across languages.
  • 1 Introduction: Across three multilingual representation spaces, language identity can dominate some geometry statistics, causing pooled analyses to manufacture or mask relationships.The surviving association also depends on which notion of computational benefit is measured.
  • 2 Background and Related Work: AfriXNLI extends XNLI by translating a subset into 16 African languages, while AfroXLMR adapts XLM-R through continued pre-training on African-language corpora.This translated benchmark construction creates the data lineage examined for contamination.
  • 2 Background and Related Work: Representation geometry offers cheap forward-pass summaries for difficulty or quality, while established cascades use calibrated cheap-model confidence to govern escalation.The paper contributes an examination of representation statistics as routing signals, not a new routing mechanism.

3 Experimental Design

The experiments use a clean 15-language AfriXNLI set, frozen checkpoints, calibrated predictions, and representation features to study association and routing. Evaluation controls for language structure and uncertainty while explicitly flagging target construction from a single checkpoint pair.

  • Data and benchmark: 15 language configurations form the clean set, with 6,750 development and 9,000 test examples after excluding English, French, and Swahili.Each configuration has 450 development and 600 test examples with perfectly balanced three-way labels.
  • Models and features: All checkpoints are frozen without fine-tuning; three provide compute-ladder rungs, while three provide representation statistics only.mDeBERTa-base serves in both roles, and xlm-roberta-base was added post hoc as a replication source.
  • Calibration: Each model is temperature-calibrated on the pooled clean development split of 6,750 examples before all reported results are evaluated on test.The fitted temperatures are 4.116 for MiniLM, 1.704 for mDeBERTa, and 2.723 for XLM-R-large.
  • Target construction: Both ∆prob and ∆correct are constructed from a single frozen checkpoint pair rather than the seed-averaged quantity specified by the design.The paper marks results that depend on these targets explicitly.
  • Representation statistics: Representation analysis uses hidden states from layers 4, 8, and 12, producing 27 features across three sources, three layers, and three statistics.Features are computed over non-padding tokens from each example’s hidden-state matrix.
  • Association and uncertainty: Associations use within-language partial Spearman correlations with controls, combined by Fisher-z meta-analysis, while uncertainty uses bootstrap, subsampling, permutation, and clustered resampling.Accuracy comparisons resample languages and then examples within language; routing is evaluated leave-one-language-out.

4 Stage I: Evaluation Validity

AfriXNLI’s English, French, and Swahili configurations are not clean evaluations for XNLI-trained checkpoints because the benchmark reuses XNLI evaluation examples. We therefore exclude these configurations, while noting residual cross-lingual lineage contamination in the 15 remaining African-language configurations.

  • Benchmark lineage: 1,047 of AfriXNLI’s 1,050 unique English premise–hypothesis pairs (99.7%) occur verbatim in XNLI evaluation splits, and its 450-example English development split exactly matches XNLI validation.AfriXNLI documentation states that it translates XNLI and retains the original English and French subsets.
  • Split-specific exposure: MiniLM and mDeBERTa perform far better on development than test configurations, including mDeBERTa’s 1.000 English-dev accuracy versus 0.890 English-test accuracy.Both model cards report training on XNLI development data, and AfriXNLI development is XNLI validation.
  • Split-specific exposure: XLM-R-large scores 0.998 on English development and 1.000 on English test, consistent with test exposure but not establishing it because its training splits are undocumented.The checkpoint answers every English test item correctly, but the study cannot verify inaccessible training data.
  • Exclusion and limitations: The English, French, and Swahili configurations cannot serve as clean evaluations or high-resource controls, so the study excludes them and analyzes the 15 remaining configurations.The exclusion follows from benchmark lineage regardless of whether XLM-R-large saw XNLI test data.
  • Exclusion and limitations: The remaining 15 African configurations retain benchmark-lineage contamination because their translated pairs derive from XNLI examples seen in other languages, potentially benefiting strong cross-lingual transfer.The paper treats this as a limitation rather than a controlled variable because no uncontaminated African-language NLI benchmark is available.

5 Stage II: Capability Is Not Ordered by Parameter Count

Parameter count does not reliably order capability across African languages: larger checkpoints win in some languages and lose in others. Thus, adaptive-inference hierarchies must be verified per language rather than assumed from model size.

  • Capability ordering: The aggregate difference of +0.022 in mDeBERTa’s favour is not statistically distinguishable from zero, with a 95% confidence interval of [−0.035, +0.083].The bootstrap gives P(diff > 0) = 0.772, so parameter count does not determine which model is better.
  • Capability ordering: Seven languages favour mDeBERTa and eight favour XLM-R-large, with per-language accuracy differences spanning −0.145 ≤ ∆acc ≤+0.220.XLM-R-large is 14.5 points better on Amharic and 12.7 on Oromo, while mDeBERTa is 22.0 points better on Igbo and 20.0 on Shona.
  • Implications for adaptive inference: For mDeBERTa→XLM-R-large, mean ∆prob = −0.011: escalation helps on 37.5% of examples and hurts on 39.7%.Escalation therefore switches models rather than reliably purchasing additional capability.
  • Implications for adaptive inference: Only the MiniLM→mDeBERTa pair forms a usable ladder, with approximately 8× the floating-point operations and mean ∆prob = +0.099.This holds only for the 11 of 15 languages in which the cheap rung exceeds chance; routing experiments are consequently restricted to those languages.
  • Methodological consequence: The compute hierarchy required by adaptive inference cannot be assumed from model size and must be verified per language before routing claims are attempted.The ordering varies across languages, while the study does not identify which linguistic, script, or resource properties explain that variation.

6 Stage III(a): Language Identity Dominates Representation Geometry

Across three multilingual representation spaces, angular dispersion is consistently more language-determined than effective rank, showing that pooled representation statistics can partly restate language identity. However, the stronger hypothesis that language dependence quantitatively predicts pooling bias does not replicate.

  • Variance decomposition: η2 measures the proportion of a statistic’s total variance lying between languages rather than within them, with values near one indicating near-constant language-level behavior.The analysis uses 15 clean languages and evaluates variance distribution before introducing any difficulty target.
  • Cross-space ordering: Across three representation spaces, angular dispersion is the most language-determined statistic and effective rank the least, with spectral concentration intermediate throughout.The ordering is averaged over layers {4, 8, 12} and holds for the task-tuned, African-adapted, and generic multilingual models.
  • Cross-space ordering: η2 = 0.583, 0.484, 0.542 for angular dispersion versus 0.075, 0.133, 0.151 for effective rank across the three spaces.Angular dispersion at layer 8 of mDeBERTa reaches η2 = 0.757.
  • Interpretation: Roughly half of angular dispersion’s variation, and three quarters at some layers, is language-related rather than example-related, so pooled analyses partly re-describe language identity.Such a statistic can resemble a difficulty signal to the extent that languages differ in difficulty, regardless of the later correlation target.
  • Limits of replication: Spearman(η2, |ρpooled| − |ρwithin|) = +0.415 with permutation p = 0.032, but its bootstrap 95% confidence interval is [−0.031, +0.737] and the relationship is source-dependent.The association is +0.917 for mDeBERTa but +0.133 for both AfroXLMR and XLM-R-base; therefore η2 is not claimed to predict pooling bias as a general quantitative law.

7 Stage III(b): Representation Signals Depend on What “Benefit from Computation” Means

The representation signal associated with computational benefit depends on whether benefit means probability gain or a changed decision. Effective rank predicts the former, confidence predicts the latter, and their associations are not interchangeable for routing.

  • Language control: +0.296 pooled angular-dispersion correlation with ∆prob falls to +0.061 under language control, while effective rank changes from −0.009 to −0.127.Pooling inflates the language-determined statistic’s apparent relationship and masks the language-neutral statistic’s association.
  • Effective rank: −0.127 is effective rank’s within-language association with ∆prob, but its association with ∆correct is only −0.027 and non-significant after correction.For ∆prob, the association is strongest in the study and survives multiple robustness checks; against ∆correct, p = 0.28.
  • Cheap-model confidence: −0.099 is cheap-model confidence’s strongest association with ∆correct, whereas its association with ∆prob is +0.003 and unrelated.Confidence therefore shows the mirror image of effective rank across the two benefit targets.
  • Decision implications: −0.066 is angular dispersion’s association with ∆correct, despite +0.029 against ∆prob; the opposite targets therefore select different signals.The ∆correct association has p = 1.6 × 10−8, while the ∆prob association has p = 0.13.
  • Limitations: The dual-target analysis was post-hoc, based on one cheap–expensive model pair, and not replicated across model pairs.The authors consequently ask whether associations provide enough incremental information for useful routing, not merely whether they are statistically significant.

8 Stage IV: From Statistical Association to Adaptive Routing

Adaptive routing initially optimized the wrong target: probability gain rather than the decision-relevant correctness change. After correction, geometry and confidence were statistically indistinguishable, yet neither practical method beat always-expensive inference despite an oracle showing substantial attainable benefit.

  • Objective alignment: The initial router trained on ∆prob but was evaluated on prediction correctness, optimizing a quantity different from the measured objective.The mismatch motivated retraining with ∆correct while holding the rest of the experiment fixed.
  • Objective alignment: +0.655 is the correlation between ∆prob and ∆correct, while confidence and effective rank predict opposite targets within language.Confidence is essentially unrelated to ∆prob (ρ = +0.003) but strongest for ∆correct (ρ = −0.099); effective rank shows the opposite profile.
  • Corrected comparison: +0.007 at 20% and +0.011 at 60% are the geometry router’s gains after retraining on ∆correct, narrowing its gap to confidence.At 20%, the confidence−geometry gap is −0.0015 to −0.0024; at 60%, it is +0.0059 to +0.0147 across λ ∈{1, 10, 100, 1000}.
  • Routing value: 0.545 versus 0.577 at 60% and 0.565 at 80% show that no practical routing method exceeded always-expensive inference.Under the tested signals, adaptive routing did not pay for itself.
  • Routing value: 0.688 at 60% is the oracle’s accuracy, eleven points above always-expensive inference while using 40% less computation.The approximately 14-point gap between the best practical method and the oracle indicates that the routing problem is learnable, but neither tested signal captures enough benefit.

9 Discussion

The discussion separates representation structure, statistical association, and decision usefulness, showing that multilingual evaluation can conflate them. It recommends contamination-aware evaluation, measurement-based model selection, language-controlled representation reporting, and target-specific routing validation.

  • Core interpretation: Representation geometry, statistical association, and decision usefulness are distinct, so multilingual evaluation can conflate them.Each investigative stage separates a pair of concepts that might otherwise be treated as one.
  • Evaluation validity: AfriXNLI English, French, or Swahili results with XNLI-trained checkpoints are not interpretable as capability measures without contamination-aware reporting.The discussion does not claim AfriXNLI is invalid; the issue is pairing its preserved source-language subsets with source-trained checkpoints.
  • Model selection: Six of fifteen languages show model-ordering differences exceeding ten accuracy points, with larger and smaller checkpoints each winning in different languages.The larger model is not globally inferior, and the aggregate difference is indistinguishable from zero; cascade ordering must therefore be measured per language.
  • Representation confounding: Angular dispersion is consistently more language-dependent than effective rank across three representation spaces, motivating η2 reporting alongside pooled multilingual correlations.The evidence establishes an ordering of statistics by language dependence and explains how pooling can mislead, but does not establish a law linking η2 to pooling bias.
  • What “benefit from computation” means: Effective rank predicts ∆prob but not ∆correct, whereas cheap-model confidence predicts ∆correct but not ∆prob; the targets correlate at only +0.655.A signal can be informative for one operational definition of computational benefit while being irrelevant or oppositely related to the evaluated objective.
  • Practical routing: 0.688 is achieved by the oracle at 60% compute versus 0.577 for always-expensive inference, yet no evaluated practical method exceeds always-expensive inference.The roughly 14-point gap between achievable and observed routing performance remains an open problem under the tested models, signals, and compute budgets.

10 Limitations

The study’s limitations concern target construction, benchmark lineage, checkpoint and representation coverage, task scope, post-hoc analyses, and oracle interpretation. Together, these barriers prevent identifying the optimal adaptive signal or generalizing the conclusions broadly.

  • The marginal-gain targets: ∆prob is useful for descriptive analysis, but ∆prob̸ = ∆correct means it should not be treated as a universal measure of computational benefit.It is one operationalisation among several, despite offering greater statistical power than a ternary target.
  • The marginal-gain targets: Fixed checkpoints prevented seed-averaged marginal-gain targets, so measured ∆prob relationships do not establish stability across independently trained model instances.Variation reflects examples and languages within fixed checkpoints, with some variance attributable to checkpoint idiosyncrasy.
  • Benchmark lineage: The 15 remaining configurations are translated XNLI instances, so cross-lingual contamination cannot be ruled out and no comparable uncontaminated benchmark estimates its size.Excluding English, French, and Swahili removes direct surface overlap, but the benchmark lineage remains shared with checkpoint exposure.
  • Checkpoint selection and representation statistics: The three checkpoints and three similar-depth encoders are limited samples, so capability ordering and representation-statistic findings should not be generalized across model families.Effective rank, spectral concentration, and angular dispersion also cover only a limited set of geometric descriptors, computed over single-sequence token dimensions.
  • Single task, coverage, and post-hoc analyses: The study covers only natural language inference, while routing uses 11 of 15 languages and one MiniLM→mDeBERTa pair; post-hoc analyses remain unreplicated.The post-hoc analyses were XLM-R-base as a third representation source and the dual-target analysis, both explanatory rather than confirmatory.
  • Oracle interpretation: The oracle is not deployable: its 14-point gap measures opportunity under perfect allocation, not a deficit attributable to any practical method.Because the oracle observes the outcome it predicts, it provides an upper bound rather than evidence that deployment can approach that level.

11 Conclusion

The study finds that representation statistics did not make adaptive routing preferable to always-expensive inference under the tested conditions. It concludes that representation structure, statistical association, and decision usefulness are distinct, while multilingual evaluation can conflate them.

  • Evaluation validity: 1,047 of AfriXNLI’s 1,050 English examples are verbatim XNLI matches, so its English, French, and Swahili configurations are not clean evaluations for XNLI-trained models.The checkpoints commonly used for evaluation are trained on XNLI.
  • Adaptive routing: Under the tested models, signals, and compute budgets, evaluated signals did not make adaptive routing preferable to always-expensive inference.The conclusion is limited to the tested models, signals, and compute budgets.
  • Adaptive routing: 0.688 for an oracle at the 60% budget versus 0.577 for always-expensive inference shows that a substantial, learnable allocation problem remains.The oracle result indicates available headroom even though the evaluated signals did not exploit enough of it.
  • Methodological conclusion: Representation structure, statistical association, and decision usefulness are distinct properties that multilingual evaluation can conflate.This is the paper’s narrower methodological conclusion.

Reproducibility · Ethics and Data Statement

The study is reproducible from released code, configurations, cached outputs, and a results ledger without fine-tuning or accelerator access. It uses only public benchmarks and checkpoints, with no human subjects or new data collection, and frames contamination as an evaluation-construction caution.

  • Reproducibility: Code, configurations, cached model outputs, and a results ledger are available at the paper-v1 tag.The ledger records every reported number with its sample, statistic, uncertainty, and limitation.
  • Reproducibility: No model was fine-tuned, and all inference was performed once on a laptop without a discrete GPU.The resulting outputs were cached for subsequent analyses.
  • Reproducibility: Every analysis, including bootstrap and permutation procedures, re-runs from cached outputs in minutes without an accelerator.This supports reproducing the reported analyses from the released caches.
  • Ethics and Data Statement: The work uses only publicly released benchmarks and publicly released model checkpoints.No new data collection was conducted.
  • Ethics and Data Statement: The study involves no human subjects and no new data collection.Its data sources are existing public benchmarks and checkpoints.
  • Ethics and Data Statement: The contamination findings concern the interaction between benchmark construction and checkpoint training data.Both aspects were documented by their authors.
  • Ethics and Data Statement: The analysis is intended as a caution about combining documented artefacts in evaluation, rather than criticism of their releasers.The paper also notes that undetected contamination is likel…

A Per-Language Capability Differences · B Stability of the Effective-Rank Association

The section compares per-language capability differences using δ = mDeBERTa − XLM-R-large and tests whether the effective-rank association remains stable across resampling variants. All resampling variants remain negative, while the permutation null is centered at zero and has a much smaller maximum magnitude.

  • A Per-Language Capability Differences: δ = mDeBERTa − XLM-R-large is the reported per-language accuracy-difference metric.The comparison is defined separately for each language.
  • A Per-Language Capability Differences: The per-language capability analysis is framed as an accuracy comparison between mDeBERTa and XLM-R-large.The supplied table identifies the two checkpoints and the direction of subtraction.
  • A Per-Language Capability Differences: The reported δ values are intended to expose capability differences that may vary by language.Table 9 is explicitly organized by language rather than as a single pooled comparison.
  • B Stability of the Effective-Rank Association: All resampling variants of the effective-rank association remain negative.This persistence across resampling variants supports the stability check reported in Table 10.
  • B Stability of the Effective-Rank Association: The permutation null is centered at zero rather than at the observed negative association.The null distribution therefore provides a zero-centered reference for the resampling analysis.
  • B Stability of the Effective-Rank Association: The permutation null’s maximum magnitude is five times smaller than the effect.This comparison is reported directly as part of the resampling-variant analysis.
Loading 2608.19003v1…