The Dialect Tax: Dialectal Biases Persist throughout the Language Modeling Pipeline

Elle

Source-linked AI summary

The Dialect Tax: Dialectal Biases Persist throughout the Language Modeling Pipeline

Elle

arXiv:2608.24952v1Aug 24, 2026cs.CLcs.AIcs.LG

TL;DR

Dialectal performance gaps are well documented, but their location within the modern language-modeling pipeline is unclear. Using parallel English dialect corpora that hold meaning fixed, the paper traces the dialect tax across tokenization, training, post-training, and inference. It finds that disparities accumulate across the pipeline: character-level tokenization does not remove downstream gaps, and dialectal content produces divergent training signals and context-dependent reward preferences.

Problem
The study addresses the unclear source of systematic dialectal performance disparities within the modern language-modeling pipeline.
Method
The paper uses parallel English dialect corpora with fixed meaning to examine tokenization, pre-training, post-training, and inference across language-model components.
Results
Dialect taxes accumulate across the pipeline: dialect pairs produce more divergent gradients than unrelated SAE documents, while model and reward-model representations remain dialectally unequal.
Takeaways & Limitations
Character-level tokenization does not eliminate input, output, or accuracy disparities, so evaluating interventions across the entire model pipeline is necessary.
Takeaways & Limitations
The datasets do not capture the full situational range of dialect taxes and include synthetic rule-based text rather than only community-validated native-speaker data.

Abstract

from arXiv · show

Systematic dialectal performance gaps in language models (LMs) are well documented, but the source of these disparities within the modern language modeling pipeline remains unclear. Our study traces this "dialect tax" across the natural language processing pipeline. Using parallel English dialect corpora that hold meaning fixed while varying surface form, we first confirm that LMs recognize matched Standard American English (SAE) and dialectal texts as semantically equivalent. However, we discover further representational gaps corresponding to downstream performance gaps. Across model families and generations, modern LMs still encode dialectal texts unequally during tokenization, pre-training, post-training, and inference. Strikingly, bypassing traditional subword segmentation via a character-level counterfactual tokenizer removes neither input and output asymmetries nor dialectal accuracy gaps. During pre-training, dialect pairs induce more divergent gradient updates than pairs of entirely unrelated SAE documents, indicating that models find semantically equivalent dialectal content harder to learn from than unrelated SAE documents. During post-training, reward models show contextual, unstable dialect preferences, assigning higher values to isolated AAVE-exclusive tokens than to SAE-exclusive tokens, while full reasoning contexts receive task- and model-dependent dialect penalties. Overall, our findings suggest that the dialect tax is encoded and accumulated not by any one step in isolation, but at every step of the language modeling process.

1 Introduction

The paper traces dialectal bias across the language-modeling pipeline, asking where performance disparities arise despite longstanding evidence of societal and model biases. Using parallel English dialect texts, it finds that dialect taxes accumulate across tokenization, training, and inference rather than originating in one component.

Motivation: Prior work identifies disparities in tokenizers, language models, and reward models, including differences in accessibility cost, latency, and quality.These disparities can exacerbate economic divides in language-technology access.
Research aim: Parallel English dialect corpora let the study trace the dialect tax while holding semantic content fixed and varying surface form.Comparing English dialects also removes byte-premium confounds.
Core findings: Models recognize matched SAE and dialectal texts as semantic equivalents more strongly than translations or character-perturbed copies, yet dialectal performance gaps persist.The paper specifically reproduces reasoning gaps across current language models.
Core findings: Character-level tokenization narrows the input-side dialect gap but leaves accuracy and output behavior largely unchanged.This indicates that learned model weights retain disparities beyond subword segmentation.
Core findings: Dialect taxes arise during both pre-training and post-training, alongside disparities in gradients, hidden states, and reward-model preferences.The findings therefore span multiple stages of the modeling pipeline.
Implication: Dialectal biases accumulate across language models and reward models, so addressing them requires training approaches that evaluate the full pipeline.The authors argue against treating tokenizers as the sole source of dialect prejudice.

2 Same Meaning, Higher Taxes

The study first tests whether dialectal pairs preserve meaning before measuring dialectal disparities. Embedding results show strong semantic equivalence, supporting the interpretation that later gaps reflect surface-form differences rather than gross semantic mismatch.

Datasets: Parallel datasets pair SAE with AAVE, Appalachian, Chicano, Indian, and Singaporean English texts while varying dialectal surface form.PARALLELAAVE uses human rewrites, whereas MULTIVALUE uses rule-based synthetic translations.
Method: The embedding evaluation compares dialect pairs with character perturbations and translations using cosine similarity between text representations.The setup uses a 300M-parameter embedding model with four representation dimensions.
Results: All five MULTIVALUE dialect pairs exceed 0.98 similarity, above perturbation and translation baselines ranging from 0.659 to 0.978.The comparison uses 429 aligned sentences and Holm-adjusted p < 0.001.
Results: On PARALLELAAVE, SAE-AAVE similarity is 0.92, exceeding deletion, capitalization, and translation controls but remaining below swapping and insertion controls.The value is far above the unrelated-document null baseline of µ = 0.41.
Robustness: Similarity tracks semantic content rather than language-model surprisal, weakening the explanation that predictability drives the embedding results.Across transformations, mean similarity correlates positively rather than negatively with log median per-token surprisal.

3 Models (Still) Have Dialectal Biases

The paper evaluates tokenization and reasoning across contemporary model families. It finds persistent dialectal representation and reasoning disparities, with tokenizers producing unequal segmentation and AAVE text receiving weaker reasoning performance than SAE text.

3.1 Tokenizers Have Dialectal Biases: The study measures token bias using eight metrics, including token length and fertility, and tests parity on parallel texts conditioned on identical content.It evaluates BPE, UNIGRAM, and WORDPIECE tokenization methods.
3.1 Tokenizers Have Dialectal Biases: AAVE texts have higher mean fertility than SAE texts across all tokenizers, with an average gap of +0.07 tokens per word.The direction is marked as lower-is-better for fertility.
3.1 Tokenizers Have Dialectal Biases: Appalachian, AAVE, and Chicano English have median token lengths above SAE parity, while Indian and Singaporean English fall below parity.This ranking is stable across tokenizer families.
3.2 Models (Still) Have Dialectal Biases: The reasoning benchmark compares SAE and AAVE across math, algorithm, logic, and planning tasks using multiple open-source and GPT model families.The evaluation includes approximately 300 samples per condition.
3.2 Models (Still) Have Dialectal Biases: AAVE reasoning performance is weaker than SAE performance for every evaluated model family and size, leaving the dialect tax unresolved.The result is reported in Table 12.

4 Isolating the Tokenizer from the Model

The tokenizer counterfactual separates subword-segmentation effects from dialect disparities encoded in model behavior. Character tokenization reduces some input-side differences but does not systematically remove representational, entropy, or accuracy gaps.

Method: Character tokenization bypasses subword merges, allowing tokenizer-induced and model-induced dialect gaps to be disentangled.Gaps that disappear implicate the tokenizer, while persistent gaps implicate the language model.
Behavioral disparities: 93.0% of paired samples show higher mean per-token input entropy for AAVE than SAE under both tokenizations.The direction is statistically significant under both tokenizations, with 92.3% sign agreement across 19,995 pairs.
Behavioral disparities: Across nine models, five accuracy gaps widen and four narrow after character tokenization, producing no systematic change in gap magnitude.The paired tests are statistically insignificant, so character tokenization does not reliably narrow downstream accuracy disparities.
Behavioral disparities: The output entropy gap remains unchanged across model families after swapping tokenizers, while input and output gap changes are uncorrelated across 13,197 paired items.The reported correlation is Pearson r = +0.001, p = 0.91.
Representational differences: Across nine instruction-tuned LMs from three families, dialect remains linearly decodable from hidden states under both tokenizations.Canonical-tokenization accuracy exceeds 99% across models, while character-tokenization accuracy remains above 86%, far above the 50% chance baseline.

5 Dialect Taxes Exist Despite Training

The study traces dialect taxes through pre-training, post-training, and inference, finding persistent disparities despite semantic equivalence and character-level tokenization. Dialectal form produces divergent training signals, unstable reward preferences, and worse prediction and downstream accuracy than comparable transformations.

5.1 Pre-Training Gradients: Matched dialect pairs produce more divergent gradient updates than unrelated SAE documents, indicating that meaning-preserving dialect shifts perturb training signals strongly.Across models and tasks, the paired gradient similarity was lower than the unrelated-SAE baseline.
5.1 Pre-Training Gradients: The pre-training loss gap is largely independent of correctness, with conditional expectations differing by only 0.006 nats, far below the dialect gap.The divergence therefore occurs across problems regardless of whether the model answers correctly.
5.1 Pre-Training Gradients: Dialectal gradient divergence is comparable to substantial character corruption, including perturbations on the order of dropping 15% of characters.AAVE was more disruptive than character swaps or insertions despite preserving meaning.
5.2 Post-Training Rewards: Reward models show contextual dialect preferences: isolated dialect-exclusive tokens score higher, while full contexts produce task- and model-dependent gaps.Across token-level evaluations, the pooled dialect-versus-SAE score gap was −0.55, whereas sample-level preference directions varied by task and model.
5.3 Inference-Time Representations: Inference representations retain dialect-related differences despite similar hidden-state trajectories, and AAVE increases next-token entropy by +0.29 nats on average.Dialectal transformations resemble character noise internally, yet AAVE causes a 3.09-percentage-point accuracy loss versus 0.04 to 1.23 points for noise and translation.

6 Discussion

The dialect tax is introduced across multiple locations in the language-modeling pipeline rather than at a singular source. Because individual interventions can mitigate specific disparities without resolving the underlying gap, dialect bias requires holistic evaluation and attention.

The dialect tax is introduced at every stage of the language modeling pipeline.
Pinpointing a singular source of dialect prejudice obscures the need to address the issue holistically.
Individual interventions can mitigate specific measured disparities without resolving the underlying gap.
Interventions should be evaluated across the entire model pipeline rather than at isolated components.

Limitations

The study’s empirical scope is constrained by computational resources, available datasets, accessible models, and the absence of controlled training experiments. These limitations restrict coverage across dialects, settings, model scale, random seeds, and causal encoding points.

The experiments are limited by computational resources and available techniques.
Datasets: The datasets fix semantic meaning while varying dialectal form but do not capture the situational range where the dialect tax may appear.The resources differ in genre and construction, including human rewrites, formal benchmarks, and rule-based generation.
Models: The study uses smaller open-source models because their internals are inspectable, so closed commercial systems and larger models remain outside its verified scope.
Models: Compute and budget constraints prevented repeating every experiment across multiple random seeds, so statistical evidence relies on paired inputs and models rather than run-level significance.
Interventions: Because the study conducts no model training, it cannot identify when dialect disparities become encoded during pre-training or post-training.Controlled experiments are proposed as a continuation for testing causal mechanisms.

Ethics Statement

The study follows general NLP research principles and uses open-source datasets collected with informed consent and pseudonymized participant identities.

The study used open-source datasets whose data was collected with informed consent and pseudonymized participant identities.

A Data

The study reconstructs and standardizes several benchmark and parallel-dialect datasets, then evaluates tokenizers, language models, embeddings, and reward models across controlled text transformations.

Datasets: REDIAL provides parallel SAE–AAVE reasoning queries for the main multiple-choice benchmark experiments.The reconstructed dataset incorporates source datasets and programmatically added inference instructions.
Source benchmarks: The analysis covers programming, logic, mathematics, and planning benchmarks, with source-specific corrections, deduplication, and formatting changes.Examples include HUMANEVAL, MBPP, FOLIO, LOGICBENCH, GSM8K, SVAMP, and ASYNCHOW.
Models and tokenizers: Tokenization analyses compare BPE, UNIGRAM, and WORDPIECE using metrics defined over tokens, types, words, and characters.The study uses family-associated tokenizers and evaluates token bias through eight metrics.
Models and tokenizers: The study also uses EMBEDDINGGEMMA-300M for text representations and multiple language-model and reward-model families.Similarity queries are prefixed with a sentence-similarity task instruction.

C.4 Results on MULTIVALUE

The MULTIVALUE analysis compares paired dialect and SAE texts using perplexity-based measures and semantic-equivalence visualizations.

Semantic equivalence: Figure 8 visualizes the relationship between semantic equivalence and the change in fertility across MULTIVALUE, PARALLELAAVE, and REDIAL.The corresponding visualization is reproduced in the appendix.
Perplexity comparisons: Paired perplexity ratios compare dialects, translations, and character perturbations against matched SAE texts.A ratio above 1 indicates that a model finds the transformed item more surprising than its SAE counterpart.
Perplexity comparisons: The analysis reports both the median paired ratio and the fraction of paired items with ratios above 1.These measures summarize how often and by how much transformed texts are harder than SAE.

D.1 Tokenization Biases

Dialectal representation gaps persist across tokenization and hidden-state analyses, with patterns that remain stable across tokenizer families and correspond to income differences.

Tokenization and income: Dialect tokenization efficiency corresponds to income positioning: lower-income AAVE, Appalachian, and Chicano groups tokenize less efficiently, while higher-income Indian and Singaporean groups tokenize more efficiently than SAE.Figure 11 marks token parity at y = 1 and the US median household income at x = $80, 610.
Tokenization over time: Tokenizer rankings show strong agreement across seven tokenizers, including Kendall’s W = 0.89 for fertility and W = 0.95 for average types per word.Agreement is statistically significant at p < 0.001 on every reported metric.
Tokenization over time: No metric × dialect cell shows significant rank drift across tokenizer release dates, and dialect ranks remain constant for token count and word count.Complementary tests of gap magnitude also fail to survive correction.
Reward-model analyses: The study evaluates reward gaps using sample-level and dialect-exclusive token rewards, including per-RM, per-task, and per-corpus comparisons.Negative raw gaps indicate that a reward model scores dialect-exclusive tokens higher than SAE-exclusive tokens.
Hidden-state representations: Final-layer hidden-state similarity to SAE drops for every model on MULTIVALUE, including across pooled MULTIVALUE and PARALLELAAVE analyses.Figures report per-model layer-wise cosine similarity and final-layer comparisons across dialect transformations.