Source-linked AI summary
Multilingual in Name Only? Cultural and Linguistic Weaknesses of LLMs in Urdu
Farah Adeeba, Abdul Rafae Khan, Rajesh Bhatt, Hassan Sajjad
TL;DR
The paper asks whether multilingual LLMs generate reliable Urdu stories despite limited evaluation of low-resource languages. It constructs and annotates Urdu-Stories across three models, finding surface fluency alongside grammatical, semantic, narrative, and cultural weaknesses. These results expose important limits on using current LLMs for low-resource-language generation and information retrieval.
Problem
Multilingual LLMs are insufficiently evaluated in low-resource languages, especially for long-form Urdu generation requiring linguistic, semantic, narrative, and cultural competence.
Method
The study builds Urdu-Stories, a 93-story corpus generated by three LLMs and manually annotated for linguistic, semantic, and cultural errors.
Results
The models show basic grammatical and semantic errors, incoherence, unnatural repetition, limited originality, and pervasive cultural shallowness despite surface fluency.
Takeaways & Limitations
Current multilingual LLMs are not reliably grounded in Urdu’s linguistic and cultural dimensions for open-ended content generation or information retrieval.
Takeaways & Limitations
The corpus covers 93 stories from three commercial models and one language, so broader systems and languages are needed to strengthen generality.
Abstract
from arXiv · showhide
Multilingual large language models (LLMs) are increasingly used for open-ended text generation, yet their behaviour in low-resource languages remains poorly understood. In this work, we question how correct and reliable is the generation of multilingual LLMs when used for the task of story generation. We consider Urdu language as a representative low-resource language. We generate Urdu-Stories, a corpus of 93 stories generated using three contemporary LLMs (GPT-5.1, Qwen-3-Max, DeepSeek-3.1). We manually annotate the errors present in them under a nine-label linguistic, semantic, and cultural taxonomy. Our notable findings suggest that LLMs often make basic errors of grammar and semantics. The stories lack coherence, have unnatural repetition and show pervasive cultural shallowness. We further show using few-shot prompting that the cultural and context errors largely remain unresolved. Our findings highlight the limitations of current LLMs as a reliable source of content generation and information retrieval for low-resource languages.
1 Introduction
This study examines whether multilingual LLMs can generate Urdu stories that are linguistically accurate, semantically coherent, narratively structured, and culturally grounded. Across a 93-story corpus, the findings reveal surface fluency alongside linguistic fragility, semantic and cultural weaknesses, repetition, and limited originality.
- Story generation is used as a demanding testbed because long-form narratives require sustained contextual awareness, structural coherence, and cultural sensitivity.The study addresses a gap in prior multilingual evaluation, which has largely emphasized general understanding and translation rather than long-form generation.
- Models frequently reproduce known literary stories rather than creating novel narratives, and repeatedly insert supplied titles into story bodies.The authors introduce Title Repetition Rate to quantify title anchoring, which exceeds repetition patterns in human-written stories.
- Generated stories exhibit spelling inconsistencies, grammatical instability, factual contradictions, meaningless sentences, and unprompted switching into Punjabi.These findings motivate evaluation beyond syntax or fluency alone, including semantic coherence and contextual relevance.
- Cultural shallowness appears through pervasive cross-script contamination and a 3.7:1 male-default protagonist bias.The study treats cultural fidelity as a central evaluation question because reliable language models must account for social norms and shared cultural experience.
- LLMs remain superficially fluent in Urdu but fail to achieve deeper creativity, semantic robustness, and cultural grounding.The authors frame these findings as evidence that multilinguality at the surface level does not ensure reliable low-resource-language generation.
2 Story Corpus Development
The corpus uses 31 titles from four source categories to separate cross-cultural transfer, literary reproduction, plot memorisation, and generation from unseen prompts. Three commercial LLMs produced 93 stories under a fixed zero-shot protocol, which native Urdu speakers annotated with substantial agreement.
- Title and corpus design: The corpus contains 31 titles divided into TSST, UST, Fables, and NUT categories, each targeting a distinct source-engagement or memorisation condition.The split tests cross-cultural transfer, canonical Urdu-literature reproduction, plot memorisation, and memorisation without prior text.
- Title and corpus design: All six NUT titles were author-composed, unpublished, and contextually plausible for Pakistan, providing an unseen-prompt control.This design follows a leakage-control principle intended to avoid matching published text in model training data.
- Generation protocol: GPT-5.1, Qwen-3-Max, and DeepSeek-3.1 each received every title once through a fixed zero-shot prompt requesting a 2,000-word Urdu story in a Pakistani cultural context.The models were accessed in November 2025.
- Corpus statistics: The protocol produced 93 stories totaling 151,965 words and 9,460 sentences.Per-model corpus statistics are presented in Table 1.
- Annotation: A native Urdu speaker annotated all stories using thirteen labels spanning linguistic, semantic, and entity categories.Grammar Error includes agreement subtypes, Semantic Anomaly marks syntactically valid but semantically empty sentences, and Context Mismatch marks factual contradictions.
- Annotation: Independent re-annotation of 60 stories by four additional native Urdu speakers yielded substantial overall agreement of κ = 0.81.The subset included 20 stories per model and was balanced across title categories.
3 Qualitative Analysis
Generated Urdu stories show widespread linguistic, semantic, and cultural weaknesses: models produce frequent grammatical and spelling errors, incoherence, impossible events, and culturally inappropriate details despite surface fluency.
- Cultural Context: Cultural mismatches include inappropriate Pakistani dialogue and naming conventions, religious anachronisms, and Western-context substitutions.Examples include cremating a Muslim character, confusing the Eid crescent with a full moon, and inserting culturally inappropriate settings.
- Linguistic Errors: Grammatical errors were the most frequent category, recurring across models in postposition use, agreement, and compound-sentence formation.Representative failures include omitted ergative marking, incorrect gender agreement, and inconsistent aspect in sequential narratives.
- Linguistic Errors: Linguistic errors also include model-specific spelling, spacing, and punctuation patterns that reflect English-mediated or tokenizer-specific representations of Urdu.Errors affect retroflexes, fused compounds, and English-style em-dash usage.
- Semantic Coherence: Semantic failures include syntactically well-formed but meaningless sentences, physical impossibilities, contradictions, and lost discourse state across narratives.Examples include impossible objects, inconsistent timelines, and sentences that violate real-world plausibility.
- Semantic Coherence: Kinship and gender errors reveal that models reproduce Urdu surface forms without reliably maintaining the finer-grained social and discourse distinctions encoded in Urdu.Models collapse kinship relations, misuse address forms, and lose protagonist gender over several sentences.
4 Tractability via LLM Remediation
A targeted LLM post-editing pass corrected many local linguistic errors but did not resolve failures requiring cultural knowledge, persistent discourse state, or language-register control.
- Experimental Setup: The experiment evaluated constrained post-editing on 15 stories, with five stories sampled from each model and balanced across title categories.The prompt required minimal fixes and prohibited sentence-level rewriting.
- Correction Results: Approximately 90% alignment with human annotations was achieved for grammar, agreement, spelling, and space-deletion errors using constrained single-token edits.The edits generally preserved surrounding text, while semantic-anomaly correction was lower but still useful.
- Remediation Boundary: The remediation boundary excludes long-range gender breakdown, cultural and religious anachronisms, Western-context substitutions, and Punjabi register switching.These failures require background knowledge, persistent discourse state, or commitment to the prompted language register beyond sentence-level proofreading.
5 Conclusion
The study finds that multilingual models can produce mostly fluent Urdu sentences but struggle with native linguistic, semantic, and Pakistani cultural competence in sustained stories.
- Conclusion: The models generate fluent Urdu on the surface but reproduce known source texts or unpack titles, while errors cluster on features not shared with English.The conclusion highlights failures in case, gender agreement, retroflex spelling, cultural practice, and setting.
- Conclusion: Most grammar and spelling errors can be removed by proofreading, but cultural and contextual failures remain unresolved.The conclusion contrasts correctable surface errors with failures such as cremation of a Muslim character and gas stations in Punjabi villages.
- Conclusion: The paper concludes that current multilingual models can mostly spell and form Urdu sentences but cannot yet reliably write a Pakistani story.The conclusion also notes unusually warm social worlds and a strong imbalance toward male protagonists.
Limitations
The study’s evidence is bounded by a modest corpus, restricted model and language coverage, comparison choices, and methodological limitations affecting reproducibility and measurement.
- Scope: The corpus contains 93 stories from one language and three commercial models, limiting how broadly the cross-model findings can generalize.The authors identify additional commercial and open-weight systems and other low-resource languages as important extensions.
- Comparison baseline: The human comparison uses published canonical Urdu fiction rather than prompt-matched human generations, so the baseline is not experimentally matched.The authors nevertheless treat published literature as a culturally validated benchmark for native literary writing.
- Measurement: Exact-phrase TRR can underestimate title anchoring when models repeat title components or synonyms rather than the full surface phrase.The authors leave partial-word, lemma-level, and synonym-aware matching for future work.
- Reproducibility: Commercial API outputs are non-deterministic and model versions evolve, limiting exact reproducibility.This caveat applies to repeated evaluation of the commercial systems studied.
- Model coverage: The attempted LLaMA-70B comparison was excluded because it entered repetitive loops after approximately 500 words, leaving broader open-weight coverage for future work.The excluded system therefore does not contribute analyzable results to the comparison.
F.1 Method
The study compares generated stories with source texts using overlap-based memorisation criteria and models narrative structure through character relationship graphs, with reliability assessed through annotation agreement.
- Memorisation analysis: Each generated story is compared with its source using plot, theme, character, and resolution overlap scores, plus a categorical label.The comparison covers 36 original-generated pairs across 12 titles and three models.
- Memorisation analysis: A story is labelled memorisation when at least two of three plot, theme, and resolution thresholds are met, then crossed with character overlap into a 2×2 typology.The fixed thresholds are plot ≥0.60, theme ≥0.70, resolution ≥0.80, and character overlap ≥0.50.
- Memorisation analysis: The analysis finds category-dependent memorisation: fables dominate Faithful Retelling for GPT and DeepSeek, while every Urdu-canon story is Original for every model.English classics show a mixed pattern, including one DeepSeek Chopin Faithful Retelling.
- Cross-model comparison: Pairwise overlap between model generations is highest for GPT and DeepSeek at 0.45, while QWEN is roughly equidistant from both at 0.33–0.34.The pattern is interpreted as consistent with GPT and DeepSeek mirroring the same fables more often.
- Character-network analysis: For narrative-structure analysis, GPT-5.4 extracts characters and signed pairwise relationships, which are represented as story-level weighted graphs.Edges encode positive, negative, or neutral relationships with intensity values from 1 to 3; duplicate edges are averaged.
- Character-network analysis: Density, clustering, positivity ratio, mean signed edge weight, and Jaccard character overlap summarize graph structure and model similarity.Figures 2 and 3 show per-story distributions of density and mean signed edge weight.
I Repeated Generations: Full Results
Repeated generations show that the main Urdu-story findings persist across stochastic reruns: title repetition varies moderately, while positivity, gender imbalance, and linguistic, semantic, cultural, and script errors recur.
- 0.40–0.43 full-corpus Urdu-canon TRR exceeds regenerated TRR in all nine model×run comparisons, despite moderate run-to-run variation.TRR standard deviations range from 0.027 to 0.055.
- All nine positivity ratios exceed 0.71, with SDs of 0.032–0.048, confirming a stable positivity bias across models and runs.
- Male protagonists outnumber female protagonists in every model and run, with pooled ratios of 4.1:1 across reruns and 3.7:1 in the full corpus.
- Errors persist across independent generations: every model exhibits linguistic and semantic errors in every additional run, and none produces error-free output.
- Recurring rerun errors include malformed agreement, semantically incoherent descriptions, kinship mismatches, Punjabi or Latin-script intrusions, and Western cultural substitutions.
J English-Language Control: Full Results
The English control reveals strong language-conditioned differences: Urdu stories are more positive, repeat titles more often, memorize less faithfully, and retain a male protagonist bias absent in English.
- Urdu positivity ratio is 0.807 versus 0.637 in English, while DeepSeek’s mean signed edge weight is 0.37 in Urdu versus 0.00 in English.
- English has higher faithful-retelling rates than Urdu, 38.1% versus 19.4%, and greater cross-model plot overlap, 0.517 versus 0.373.
- Urdu TRR is 3.5× to 5.8× higher than English across all models, with mean title counts of 6.46 versus 3.62 after controlling for story length.
- The Urdu male-to-female protagonist ratio of 3.7:1 reverses to 0.4:1 in English, although title selection confounds this comparison.
- In English Pakistani settings, models use Urdu loanwords as surface markers, whereas Urdu stories contain Pakistani vocabulary without comparable cultural grounding.
K Cross-Family Judge Validation: Full Results
Cross-family validation supports the robustness of categorical memorisation, positivity, and character-diversity findings, while continuous intensity estimates and some automatic error judgments remain judge-sensitive.
- Gemini reproduces positivity direction and model ordering, with all values above 0.80, and preserves near-zero median Jaccard overlap across model pairs.Mean signed edge weight is somewhat higher under Gemini, indicating greater judge sensitivity for intensity estimates.
- Gemini flagged 176 errors in GPT stories, of which 149 were confirmed by a native Urdu speaker, yielding 84.7% precision.The 27 false positives arose from legitimate spelling variants and acceptable Urdu space-deletion patterns.
- Human and automatic memorisation labels agreed on 10/13 story pairs, with all mismatches occurring for unseen-source UST titles.
M Extended Examples: Context Mismatch and Semantic Errors
Extended examples show that Urdu-story failures include incorrect kinship terms, gendered references, and culturally or religiously implausible details.
- DeepSeek and QWEN misuse Urdu kinship terms, including dādī ammā for a maternal grandmother and abbā jān for an unrelated elderly stranger.
- A DeepSeek story applies the masculine plural do nau-jawān to a mixed-gender pair that includes a girl.
- Examples of cultural and religious anachronism include morning moon-sighting and calling Pakistani restaurant front-of-house staff kepṭan.
N Space Deletion Errors
The models exhibit distinct word-fusion patterns in Urdu compounds, while the annotation and post-editing procedure targets minimal, issue-specific corrections. Space-deletion detection is restricted to missing spaces between adjacent Urdu words, excluding punctuation and formatting cases.
- Model-specific fusion patterns: GPT fuses “be” auxiliaries at 40–67%, whereas DeepSeek fuses “come” auxiliaries at 44–67%; QWEN shows almost no fusion in either family.Each model mishandles a different closed inventory of compounds while writing the remaining forms correctly.
- Post-editing procedure: The post-editing pass processes one story per call using numbered sentences and returns one JSON entry for each detected issue.The prompt enforces minimal edits and includes a self-check to suppress punctuation-adjacent false positives.
- SpaceDeletion definition: SpaceDeletion denotes two adjacent Urdu words written as one token because the intervening word boundary is missing.The label concerns word-to-word boundaries only.
- Exclusion rules: The detector excludes missing spaces around punctuation, quotes, brackets, numbers, line breaks, dialogue formatting, and long sentences.Punctuation-adjacent cases are explicitly excluded from SpaceDeletion.
- Correction rule: Corrections insert only the missing space between fused word characters and must leave punctuation, quotes, and formatting unchanged.The corrected sentence should differ only by the required space insertion.