PaperlayerView exact revision on arXiv

Source-linked AI summary

Register Bias in Complexity-Based Large Language Model Routing

Simran Koul

arXiv:2609.17542v1cs.CLcs.CY

TL;DR

The paper asks whether complexity-based LLM routing is fair across linguistic registers and audits that question with authentic and controlled parallel data. It finds that length-based routing downshifts non-standard-register queries, while pervasive model bias—not significant routing-induced harm on this benchmark—drives the measured quality disparity.

  • Problem

    The paper investigates whether routing meaning-equivalent queries by cheap complexity estimates is fair across linguistic registers.

  • Method

    It audits four tercile-based complexity routers on 37,704 authentic learner pairs and controlled parallel register variants, then evaluates quality across device, edge, and cloud models.

  • Results

    Length-based routing systematically downshifts non-standard-register queries because omitted function words make them shorter, while other signals show no consistent effect.

  • Takeaways & Limitations

    Routing compounds exposure for users whose queries every tested tier, including a frontier cloud model, already answers less accurately.

  • Takeaways & Limitations

    The routing-induced quality harm is null on a short-question benchmark, so workloads with more length variation are needed to test whether that null is benchmark-specific.

Abstract

from arXiv · show

Large language model services increasingly route each query to one of several models of differing capability, using a cheap estimate of query complexity to send easy queries to small models and hard queries to large ones. I show that this routing step is not register neutral: text written in a non-standard English register, African American English or the English of second-language writers, is systematically assigned a lower-capacity tier than a meaning-equivalent standard-English version of the same query. The effect is driven by a specific, common routing signal, input length, because non-standard registers omit function words and thus look shorter and therefore simpler; other complexity signals do not carry it. I demonstrate the disparity on 37,704 authentic learner sentence pairs and on a controlled parallel corpus. I then measure the quality consequence on a device, edge, and cloud model ladder and find that the harm is driven by pervasive model bias, every tier, including a frontier cloud model, answers non-standard-register queries significantly less accurately, while the marginal quality cost of the routing decision itself is not significant on this benchmark. Complexity-based routing thus compounds the exposure of the users that the models already serve worst.

I. INTRODUCTION

This paper audits whether complexity-based LLM routing assigns meaning-equivalent queries to different capability tiers by linguistic register. It finds that non-standard-register queries are downshifted because function-word omissions make them shorter, while the routing disparity compounds existing model bias.

  • Complexity-based routing assigns meaning-equivalent African American English and second-language queries to lower-capacity tiers than standard-English versions.
  • Token length carries the disparity robustly, whereas readability and syntactic-depth signals do not and can even reverse.
  • The audit uses authentic human text and a controlled parallel corpus to examine register-dependent routing differences.The authentic evidence comprises 37,704 learner pairs.
  • Non-standard registers omit function words, making queries shorter and causing length-based routers to treat them as simpler.
  • Even a frontier cloud model is significantly register-biased, while routing’s marginal quality cost is not significant on this benchmark.

II. RELATED WORK

Prior work studies cost-aware routing, geographic or language fairness, and single-model dialect bias. This paper instead audits register variation within English and separates routing disparity from underlying model bias.

  • Existing routing work reduces serving cost by matching queries to models of appropriate strength.
  • Fairness-aware routing has addressed geography and distinct languages, but not register variation within English.
  • Dialect-bias research establishes that a single LLM’s accuracy and behavior can vary across English varieties.
  • This paper’s distinct contribution is auditing the routing layer above models rather than re-establishing single-model dialect bias.
  • The study uses token length, readability, and dependency depth as representative complexity proxies used by real routers.

III. METHOD

The method builds tercile routers from four standard complexity signals and evaluates them on authentic learner pairs plus controlled register variants of matched questions. Routing disparity is measured by paired downshift and upshift counts.

  • Four tercile routers map measured query complexity to three capability tiers using token length, dependency depth, Flesch-Kincaid grade, and Gunning fog.Thresholds are fitted at the 33rd and 66th percentiles of standard-English reference texts.
  • For each pair, the study compares assigned tiers and counts downshifts, upshifts, and ties under each complexity signal.
  • The authentic dataset contains 37,704 learner-original and standard-corrected sentence pairs from W&I+LOCNESS.Beginner, intermediate, and advanced learner strata are used, with native LOCNESS as a control.
  • The controlled dataset transforms 300 Natural Questions into African American English and Indian English variants with a rule-based transformer.Transforms that error out are dropped, leaving the controlled variants for semantic screening.

C. Semantic-equivalence gate

The semantic-equivalence gate retains controlled register variants only when an external judge confirms that every variant asks the same question as its standard-English original.

  • An out-of-pipeline judge compares meaning while ignoring grammar, dialect, and spelling, retaining only variants that pass for every register.The gate retained 279 of 299 usable questions after dropping 6.7%.
  • The judge is deliberately separate from the routed models, preventing the routing evaluation models from determining semantic equivalence.

D. Routing-disparity metric

The study compares four complexity signals by deterministically routing parallel queries into three capability tiers, then testing whether register variants are asymmetrically assigned.

  • D. Routing-disparity metric: The router maps measured query complexity to tiers, and each parallel pair is counted as a downshift, upshift, or unchanged assignment.Asymmetry is tested with paired sign tests, using exact McNemar tests for small counts and a continuity-corrected normal approximation for large counts.
  • D. Routing-disparity metric: The routing audit therefore separates deterministic assignment disparity from downstream model-quality differences.This design compares routing decisions directly before analyzing answer accuracy across model tiers.
  • D. Routing-disparity metric: Quality analysis evaluates a device, edge, and cloud ladder with deterministic, register-invariant fact-containment grading.The ladder uses Llama 3.2 1B, Llama 3.1 8B, and Claude Opus; the grader is separate from the routed models.

A. Authentic learner text

Across authentic learner text and controlled parallel questions, length-based routing systematically downshifts non-standard-register queries, whereas other complexity signals are inconsistent or reverse the pattern.

  • A. Authentic learner text: 1,272 downshifts versus 474 upshifts occurred for 37,704 learner/corrected pairs under length-based routing (paired sign test, p < 10^-3).The same signal produced significant downshifts for African American English and Indian English in 279 gated Natural-Questions parallel sets: 68 versus 42 and 101 versus 34, respectively.
  • A. Authentic learner text: Syntactic depth, Flesch-Kincaid, and Gunning fog do not show a consistent disparity and sometimes reverse it because learner errors and missing punctuation distort their scores.For example, Flesch-Kincaid produced 1,147 downshifts versus 2,194 upshifts in learner pairs and 11 versus 51 for Indian English in the controlled corpus.
  • A. Authentic learner text: The effect is weaker in 988 native LOCNESS pairs, where length routing yielded 20 downshifts versus 7 upshifts (p = 0.019).Readability signals were erratic on these short single sentences, supporting a narrow claim about token-length routing.
  • A. Authentic learner text: Length disparity is driven by omitted function words, which shorten non-standard-register text and make it appear simpler to the router.Examples include auxiliary deletion in African American English and omitted articles or prepositions in second-language writing.

V. QUALITY COST OF THE MISROUTING

The quality analysis compares register accuracy across a device, edge, and cloud ladder while separating fixed-tier model bias from routing-induced quality loss.

  • V. QUALITY COST OF THE MISROUTING: The analysis decomposes quality effects into underlying model bias and routing-induced harm.This separates disparities attributable to the models themselves from those attributable to assigning queries to different tiers.
  • V. QUALITY COST OF THE MISROUTING: Per-tier accuracy is compared by register across device, edge, and cloud models using McNemar tests against Standard English.The figure reports answer accuracy for each register on the full capability ladder.
  • V. QUALITY COST OF THE MISROUTING: Register bias persists and strengthens up the ladder: even the frontier cloud model answers African American English and Indian English significantly less accurately than Standard English.This establishes that quality disparities remain when comparing registers at fixed capability tiers.

A. Path B: model bias (tier fixed)

At fixed capability tiers, non-standard-register queries receive lower accuracy than Standard-English equivalents, and the disparity remains significant even for the frontier cloud model.

  • A. Path B: model bias (tier fixed): Holding the tier fixed, the frontier cloud model answers African American English and Indian English significantly less accurately than Standard English.The effect strengthens with capability rather than disappearing; the 1B model is too weak to show it because of a floor effect.
  • A. Path B: model bias (tier fixed): The fixed-tier comparison uses paired McNemar tests between Standard-English questions and each register variant.Results are reported in Table I and Fig. 2.
  • A. Path B: model bias (tier fixed): Routing-induced quality differences are not significant on this benchmark: African American English realized 0.430 versus 0.423 counterfactual accuracy (p = 0.80), and Indian English 0.341 versus 0.348 (p = 0.87).Natural-Questions items are short and nearly uniform in length, producing tight tier boundaries and limiting the measured marginal cost of downshifting.

C. Reading the two paths together

The quality disparity is driven by significant register bias across the model ladder, including the frontier cloud model, while routing adds no significant marginal quality cost on this benchmark. Nevertheless, routing compounds exposure by downshifting non-standard-register users toward weaker tiers, including a genuinely deployable on-device model.

  • C. Reading the two paths together: Significant model bias, including in the frontier cloud model, drives the quality cost for non-standard-register users.The routing analysis separates this underlying bias from routing-induced harm.
  • C. Reading the two paths together: Routing systematically downshifts non-standard-register queries toward weaker tiers, sending affected users toward the models they already experience as less accurate.The marginal quality cost of routing itself is not significant on this benchmark.
  • C. Reading the two paths together: The weakest tier is genuinely deployable on a phone, so the server-side disparity transfers to real hardware.Llama 3.2 1B achieved 0.175 on-device accuracy versus 0.200 server-side, with agreement on 37 of 40 questions.

VII. LIMITATIONS

The paper’s robust limitation is that exact disparity rates depend on workload, while the direction and length-based mechanism remain the reliable claims. The routing-induced null comes from a short-question benchmark and requires more length variation to test its generality.

  • VII. LIMITATIONS: Exact disparity rates are workload-specific; the robust claims concern the direction and mechanism.The controlled register variants are rule-transformed, while authenticity rests on the W&I+LOCNESS arm.
  • VII. LIMITATIONS: The routing-induced quality harm is null on the short-question benchmark, so workloads with more length variation are needed to test whether that result is benchmark-specific.Readability and syntactic-depth measures are also noisy on short single sentences, making length the reliable result.
  • VII. LIMITATIONS: The conclusion is that routing compounds exposure of users whom every tier, including a frontier cloud model, answers less accurately.The paper frames this as a fairness concern above any individual model that routers can avoid.
Loading 2609.17542v1…