PaperlayerView exact revision on arXiv

Source-linked AI summary

DialectGen: Benchmarking and Improving Dialect Robustness in Multimodal Generation

Yu Zhou, Sohyun An, Haikang Deng, Da Yin, Clark Peng, Cho-Jui Hsieh, Kai-Wei Chang, Nanyun Peng

arXiv:2510.14949v3cs.CLcs.CVcs.LG

TL;DR

Multimodal generative models may not handle dialectal prompts as effectively as SAE prompts, a question the paper studies across six English dialects. It constructs DialectGen and an encoder-based mitigation method, finding that the method brings tested models’ performance on five dialects close to SAE performance while retaining SAE performance.

  • Problem

    The paper asks whether multimodal generative models can produce content effectively from dialectal prompts, given that dialect prompts can expose failures even when semantically equivalent SAE prompts work.

    Source: §1 · pp. 1–2

  • Method

    The paper constructs a six-dialect benchmark with dialect-speaker-verified paired prompts and designs an encoder-based method to recognize dialect lexemes while retaining SAE knowledge.

    Source: §1 · p. 2, §3.2 · p. 4

  • Results

    77.68% average dialect performance with the proposed method closely approaches the base model’s 77.91% SAE average score, with negligible degradation in SAE MSCOCO and polysemy performance.

    Source: §5.3 · p. 9

  • Takeaways & Limitations

    The tested encoder-based approach improves dialect robustness across five dialects while maintaining near-zero SAE performance cost on Stable Diffusion 1.5 and SDXL.

    Source: §1 · p. 2

    Using the Dialect Learning loss alone may compromise handling of dialect lexemes that are polysemous in SAE contexts.

    Source: §5.2 · p. 8

Abstract

from arXiv · show

Contact languages like English exhibit rich regional variations in the form of dialects, which are often used by dialect speakers interacting with generative models. However, can multimodal generative models effectively produce content given dialectal textual input? In this work, we study this question by constructing a new large-scale benchmark spanning six common English dialects. We work with dialect speakers to collect and verify over 4200 unique prompts and evaluate on 17 image and video generative models. Our automatic and human evaluation results show that current state-of-the-art multimodal generative models exhibit 32.26% to 48.17% performance degradation when a single dialect word is used in the prompt. Common mitigation methods such as fine-tuning and prompt rewriting can only improve dialect performance by small margins (< 7%), while potentially incurring significant performance degradation in Standard American English (SAE). To this end, we design a general encoder-based mitigation strategy for multimodal generative models. Our method teaches the model to recognize new dialect features while preserving SAE performance. Experiments on models such as Stable Diffusion 1.5 show that our method is able to simultaneously raise performance on five dialects to be on par with SAE (+34.4%), while incurring near zero cost to SAE performance.

1 Introduction

DialectGen examines whether multimodal generative models handle dialectal prompts, against a backdrop of dialect diversity and training-data filters that can exclude lower-resource dialects. It benchmarks six English dialects and 17 models, then introduces an encoder-based mitigation strategy that improves dialect performance while preserving SAE performance.

Motivation

  • Pre-training content filters can exclude data involving lower-resource English dialects, and prior work reports allocational harms associated with dialect performance discrepancies.The introduction notes that three out of four English speakers have a dialect background other than Standard American or British English.Source: §1 · pp. 1–2

Benchmark and findings

  • Across six dialects and 17 image and video models, DialectGen finds performance drops of up to 38.63% for open-weights image models and 48.17% for open-weights video models.DialectGen pairs semantically identical SAE and dialect prompts that differ by one synonymous lexeme, with dialect-speaker verification.Source: §1 · p. 2

Mitigation strategy

  • The encoder-based method brings Stable Diffusion 1.5 and SDXL performance on five dialects to SAE parity while keeping SAE MSCOCO performance drops below 1%.It teaches the text encoder to recognize dialect lexemes while retaining SAE polysemous lexeme knowledge, and uses an encoder-based KL regularization loss based on image-SAE caption datasets.Source: §1 · p. 2

Contributions

  • The paper’s contributions include the benchmark, a comprehensive evaluation of 17 models and five baseline mitigations, and a method aimed at improving dialect robustness while maintaining SAE performance.Source: §1 · p. 2

2 Related Works

Prior dialect-robustness research spans several NLP tasks and has begun to examine text-to-image generation, but dialectal lexical variation remains relatively underrepresented. DialectGen addresses this gap with a large-scale evaluation of lexical variation in image and video generation.

Prior work

  • Earlier dialect-variation work examined QA, NLI, dependency parsing, POS tagging, and text-to-image generation, while this paper extends the line of study to text-to-video generation.Source: §2 · p. 3

Lexical variation

  • Prior dialectal NLP work has mainly focused on grammatical variation, whereas DialectGen provides a large-scale dataset of dialectal lexical variation.Source: §2 · p. 3

DialectGen

  • The paper presents its benchmark as the first large-scale evaluation of dialect robustness in multimodal generation across six dialects and both text-to-image and text-to-video models.Source: §2 · p. 3

3 DialectGen Benchmark

DialectGen builds paired SAE and dialect prompts from selected lexical features, then uses dialect-speaker review to ensure the pairs are meaningful, equivalent, and non-ambiguous. Its final benchmark contains 4,200 validated prompts.

3.1 Dataset Construction

  • The dataset construction begins with 1,126 dictionary-sourced dialect lexemes, filters derogatory or culture-unique items, and pairs remaining lexemes with SAE equivalents of the same meaning.Source: §3.1 · p. 3
  • GPT-4o generates visual-scene prompts around each SAE lexeme, which are swapped to dialect equivalents and prepared in concise and detailed forms.Concise prompts generally have at most six words; detailed prompts generally have at least nine words, offering less or more context, respectively.Source: §3.1 · pp. 3–4
  • For dialect lexemes with an additional SAE dictionary meaning, the dataset includes an SAE polysemy prompt using that meaning unambiguously.Source: §3.1 · p. 4

3.2 Dialect Speaker Validation and Filtering

  • Dialect speakers assess whether each prompt pair is valid in the dialect, exactly synonymous with its SAE counterpart, and non-ambiguous; only pairs passing both annotators’ checks are retained.Annotators are matched to a dialect using self-identification and an assessment quiz; filtering removes 35.9% of generated prompts, leaving 4,200 validated prompts.Source: §3.2 · p. 4

4 Experiments

The experiments evaluate dialect-induced performance drops using automatic image-text alignment metrics and human ratings. Across models, drops are larger for concise prompts, vary by dialect, and correlate strongly between automatic metrics and human judgment.

4.1 Evaluation Metrics

  • Automatic evaluation uses reference-free image-text alignment metrics, including VQAScore and CLIPScore, while human evaluation rates alignment on a 0–10 scale for 5% of model outputs.The human scores are scaled by 0.1 before calculating SAE and dialect performance.Source: §4.1 · pp. 4, 6

4.2 Benchmark Experiments

  • Pearson correlations between human evaluation and VQAScore (r = 0.968) and CLIPScore (r = 0.924) are high; VQAScore aligns better with human judgment for dialect-induced drops.The correlation between VQAScore and CLIPScore is r = 0.907.Source: §4.2 · p. 6
  • All evaluated models show larger performance drops for concise prompts than for detailed prompts, which provide richer context for inferring dialect-lexeme meanings.Source: §4.2 · p. 6
  • Among video models, Wan 2.1 and CogVideoX have the largest overall drops while Cosmos-1 is most robust; among image models, DALL-E 2 and Flux.1 [dev] have the largest drops.DALL-E 3 with Prompt Rewrite and gpt-image-1 are the most robust image models.Source: §4.2 · p. 6
  • Most models show their largest dialect-wise drops for ChE and InE, significant drops for AAE and SgE, and generally no significant degradation for BrE.The paper attributes BrE’s relative robustness to its status as a higher-resource dialect in training datasets.Source: Table 2 · p. 5, §4.2 · p. 6

5 Mitigation Methods

The paper compares established mitigation baselines with an encoder-based strategy designed to improve dialect robustness while preserving performance on Standard American English prompts. The full method brings average dialect performance close to the base model’s SAE average, with negligible degradation on SAE measures.

5.2 Our Method

  • Dialect Learning aligns dialect-prompt embeddings with synonymous SAE prompts, while Polysemy Control aligns target and frozen encoder embeddings for the same SAE polysemous prompt.Dialect Learning alone may compromise recognition of dialect lexemes that have polysemous meanings in SAE contexts.Source: §5.2 · pp. 7–8

5.3 Mitigation Results

  • 77.68% average dialect performance approaches the base model’s 77.91% SAE average, with negligible degradation in SAE MSCOCO and polysemy performance.These are the results for the method incorporating all three loss components.Source: Table 3 · p. 8, §5.3 · p. 9

5.2 Our Method

  • KL regularization uses caption-to-reference-embedding similarity scores as surrogate logits to keep the target encoder’s output distribution close to the frozen encoder’s.The method samples caption-image pairs from a general SAE dataset such as MSCOCO.Source: §5.2 · pp. 8–9

5.1 Baseline Methods

  • Prompt rewriting improves average dialect performance by up to 6.8%, while decoder fine-tuning improves dialect performance by up to 5.7% but substantially reduces SAE MSCOCO and polysemy scores.Input-level rewriting does not degrade SAE MSCOCO or polysemy performance.Source: §5.3 · p. 9

5.3 Mitigation Results

  • Dialect Learning alone greatly improves dialect performance but degrades SAE MSCOCO and polysemy performance.This comparison motivates the need to retain performance beyond dialect prompts.Source: §5.3 · p. 9
  • Adding KL regularization improves SAE MSCOCO and polysemy metrics where cosine regularization does not, while preserving dialect gains.Source: §5.3 · p. 9
  • 17.43% (Text KL Reg.) and 17.76% (Image KL Reg.) are the polysemy performance gains from adding Polysemy Control.The comparison highlights this component’s role in recognizing polysemous lexemes in SAE contexts.Source: §5.3 · p. 9

6 Conclusion

DialectGen evaluates dialect robustness across 17 text-to-image and text-to-video models, finding substantial performance drops and introducing an encoder-based mitigation strategy that preserves Standard American English performance.

6 Conclusion

  • Image and video models show dialect performance drops of up to 38.63% and 48.17%, respectively, on DialectGen.The benchmark covers 17 widely used text-to-image and text-to-video models.Source: §6 · p. 9
  • The authors introduce DialectGen, a large-scale, multi-dialectal benchmark for evaluating multimodal generative models’ dialect robustness.Source: §6 · p. 9
  • An encoder-based mitigation strategy is designed to improve dialect robustness while preserving Standard American English performance.Source: §6 · p. 9

A.1 Decoder Fine-tuning

For Stable Diffusion 1.5, the proposed method produces better-aligned images than the base model and Diffusion DPO in qualitative examples, while prompt revision yields modest gains and can introduce several kinds of errors.

A.1 Decoder Fine-tuning

  • The proposed method generates higher-quality, better-aligned images than the base model and Diffusion DPO in the qualitative comparison.Diffusion DPO slightly improves alignment but more often blends target elements with irrelevant objects.Source: §A.1 · p. 15, Figure 3 · p. 15

A.2 Prompt Revision

  • Prompt revision improves dialect performance by up to 6.8% on Stable Diffusion 1.5, but translating prompts in isolation does not yield proportional end-to-end gains.Source: §A.2 · p. 16
  • Applying prompt revision blindly can cause semantic drift and add computational overhead, including for prompts already written in SAE.The passage notes systems cannot reliably detect whether an incoming prompt is dialectal or SAE.Source: §A.2 · p. 16
  • Rewriting can mistake a valid dialect lexeme for a typo, replacing it with an unrelated SAE word and moving the prompt further from its intended meaning.Source: §A.2 · p. 16
  • Rewriting may polish surrounding phrasing yet leave the key dialect word untranslated, making it ineffective but not harmful in the described case.Source: §A.2 · p. 16
  • For concise prompts, rewriting can add unsupported context or modifiers that shift the intended visual content.The examples describe over-augmentation and semantic drift, including a case where the original term needs no translation.Source: §A.2 · p. 17

B Mitigation Results on Stable Diffusion XL and Wan 2.1 Video

On Stable Diffusion XL and Wan 2.1 Video, the proposed method substantially improves dialect performance while keeping SAE performance near its baseline; common alternatives yield smaller gains and can degrade SAE performance.

B.1 Stable Diffusion XL

  • Stable Diffusion XL’s Refiner lacks a publicly accessible corresponding CLIP-style image encoder, so only Text KL Regularization is applied to it.The evaluation compares the final method against other baselines in this more complex configuration.Source: §B.1 · p. 17
  • 85.99% average dialect performance on Stable Diffusion XL surpasses the base model’s SAE score of 84.43%, with less than a 1-point drop in both SAE metrics.Table 5 reports mitigation results across SAE MSCOCO, SAE polysemy, average dialect performance, and each dialect, measured using VQAScore.Source: §B.1 · p. 17, Table 5 · p. 18

B Mitigation Results on Stable Diffusion XL and Wan 2.1 Video

  • Prompt Revision gains reach 7.8% on Stable Diffusion XL and 6.4% on Wan 2.1 Video, while decoder fine-tuning gains reach 5.3% and 3.2%, respectively.Decoder fine-tuning also noticeably degrades both SAE MSCOCO and SAE polysemy performance on both models; Diffusion DPO was not tested on Wan 2.1 Video because it is not directly applicable to video generation.Source: §B.1 · p. 17, §B.2 · p. 18

B.2 Wan 2.1 Video

  • 23.44% average dialect performance gain on Wan 2.1 Video comes with less than a 1% drop in both SAE MSCOCO and SAE polysemy performance.Table 6 reports mitigation results across overall SAE and dialect performance, measured using VQAScore.Source: §B.2 · p. 18, Table 6 · p. 18

C Complete DialectGen Model Performance Results

This section provides a dialect-wise breakdown of DialectGen benchmark performance and describes how SAE and dialect prompt performance are computed for paired prompts.

C Complete DialectGen Model Performance Results

  • Table 7 gives a complete dialect-wise performance breakdown for all evaluated text-to-image and text-to-video models using VQAScore, where higher scores are better.The breakdown complements Table 2 in the main paper.Source: C Complete DialectGen Model… · p. 18, Table 7 · p. 19
  • For each prompt pair, the evaluation generates n images using different random seeds for image models or uniformly samples n video frames.The scoring functions use reference-free image-text alignment metrics, including VQAScore and CLIPScore.Source: C Complete DialectGen Model… · p. 19
  • SAE performance is averaged over generations from the SAE prompt, while dialect performance uses the semantically paired dialect prompt.The respective values are reported as SAE and Dialect in the tables.Source: C Complete DialectGen Model… · p. 19

D Grammatical vs. Lexical Robustness in Multimodal Models

The section contrasts grammatical and lexical variation in multimodal generation, then examines how using multiple dialect lexemes affects performance.

D Grammatical vs. Lexical Robustness in Multimodal Models

  • A small-scale experiment applies grammatical variations to 300 SAE prompts in the AAE setting and evaluates generation quality with VQAScore.The rationale for focusing on lexical variation is that multimodal models often show insensitivity to grammatical or syntactic structure, which the passage associates with the bag-of-words nature of CLIP-style encoders.Source: D Grammatical vs. Lexical Robustness… · pp. 19–20
  • Lexical variations cause significant performance drops for existing text-to-image models, whereas grammatical variations do not incur significant drops.The comparison is measured in VQAScore for three models; Table 10 reports Grammatical, Lexical, and Grammatical + Lexical variations.Source: D Grammatical vs. Lexical Robustness… · p. 20, Table 10 · p. 21
  • Table 9 provides examples of grammatical variations between SAE and AAE sentences, with color highlighting corresponding linguistic features.Blue marks unique SAE features and purple marks corresponding AAE features, when applicable.Source: Table 9 · p. 20

E Compositional Impact of Multiple Dialect Lexemes on Multimodal Generative Models

  • Models experience more significant performance degradation when prompts contain more dialectal lexemes in the Indian English evaluation.The experiment uses 100 newly annotated prompt pairs, comparing one replaced lexeme with settings replacing 2–3 words; Table 11 reports VQAScore and percentage drops relative to SAE prompts.Source: E Compositional Impact of Multiple… · p. 21, Table 11 · p. 21

F Text Encoder Training Data Analysis

The analysis finds imbalanced dialect-lexeme exposure in representative training corpora and relates these patterns to observed dialect performance gaps.

F Text Encoder Training Data Analysis

  • PDS compares frequencies of non-polysemous dialect lexemes with SAE counterparts, while DSR measures dialect-sense use among polysemous lexemes.Both metrics are averaged per dialect; a balanced corpus scores 0.50 on each.Source: F Text Encoder Training Data Analysis · pp. 21–22, Table 12 · p. 22
  • Chicano English and Indian English lexemes are most underrepresented: PDS is 0.01–0.03 and DSR is 0.00 in both analyzed corpora, alongside the largest performance drops across nearly all models.The reported VQAScore drops for concise prompts are 34–55%. Table 12 summarizes PDS and DSR for LAION-2B and mC4.Source: F Text Encoder Training Data Analysis · p. 22, Table 12 · p. 22
  • British English has higher exposure, with PDS 0.37 in LAION-2B and 0.29 in mC4 and DSR 0.26 and 0.15, respectively; its dialect-induced drops are often below 10%.AAE and SgE have PDS values of 0.03–0.09 and DSR near zero (≤0.03), consistent with moderate but substantial drops of roughly 20–33% for most text-to-image models.Source: F Text Encoder Training Data Analysis · p. 22
  • mC4 has uniformly lower PDS and DSR than LAION-2B, and Wan 2.1 shows a 47.33% VQAScore drop on concise prompts.The authors suggest lower dialect-data exposure in its mC4-pretrained encoder may have contributed to Wan 2.1’s poor robustness.Source: F Text Encoder Training Data Analysis · p. 22

G Implementation Details

  • For video evaluation, the study generates a small fixed number of frames because the first frames typically suffice for prompt-fidelity judgments and prompts have limited motion.The authors note that longer sequences offer diminishing returns in this setting.Source: G Implementation Details · p. 23

J Human Annotation Details

  • The annotation process recruited 17 dialect-speaker annotators and approved 64.1% of 6,552 proposed prompts across four rounds.The remaining 35.9% were rejected by annotators.Source: J Human Annotation Details · p. 24

L Limitations

The work focuses on lexical variation and notes limits in its metric and data scope. Future directions include broader variation and impact studies, while its dataset and code support reproducibility.

L Limitations

  • The study prioritizes lexical variation because it appears more consequential for multimodal generation than grammatical variation, which remains outside its evaluation focus.Its text-image alignment evaluation uses reference-free VQAScore and CLIPScore, and human judgments correlated with them at r = 0.968 and r = 0.924, respectively.Source: L Limitations · p. 27

M Future Work

  • Future work could systematically examine representational and skin-tone shifts associated with dialect inputs, including differences observed for “brinjal” generations.The authors observed darker-skinned individuals and more upscale, decorated settings for “eggplant” generations than for “brinjal” generations with FLUX.1 [dev].Source: M Future Work · p. 27
  • Future studies could examine grammatical dialect variation and its joint effects with lexical variation on multimodal generation.The authors identify possible interactions and failure modes as subjects for further study.Source: M Future Work · p. 27
  • Future research could assess how dialect performance gaps affect downstream studies, industrial systems, and dialect-speaker groups.The cited research areas rely on multimodal models’ semantic understanding and high-fidelity generation capabilities.Source: M Future Work · p. 27
  • Extending evaluation to prompts with multiple dialect lexemes would test compositional understanding, but creating controlled data at scale is non-trivial.The authors identify this data-creation challenge as one that needs to be addressed.Source: M Future Work · p. 27

N Societal Impact

  • DialectGen may not fully or accurately represent natural usage patterns across all speaker sub-communities, and its release is governed by a Data Use Agreement.The agreement forbids uses including deception, impersonation, mockery, discrimination, hate speech, targeted harassment, and cultural appropriation; the benchmark is designed to be configurable and updatable.Source: N Societal Impact · p. 28

O Reproducibility

  • The authors provide the benchmark dataset and source code, including validated dialect–SAE prompt pairs and scripts for reproducing reported tables and figures.The paper describes dataset construction, annotation, evaluation, and mitigation procedures, with further implementation and training details in the appendix.Source: O Reproducibility · p. 28

Related papers

Loading 2510.14949v3…