PaperlayerView exact revision on arXiv

Source-linked AI summary

Which India Survives Translation? Narrative Homogenisation Across Indian Oral Traditions in LLMs

Paarth Singh Rathore

arXiv:2608.26123v1cs.CL

TL;DR

LLMs may flatten distinct non-Western storytelling traditions because their training data overrepresents English-language cultural narratives. This pilot compares generated stories with authentic corpora from three Indian traditions using embedding-based measures, finding partial narrative homogenisation and lower fidelity under regional-language prompting. It complements larger human-annotation studies while identifying translation and embedding-based evaluation as important scope boundaries.

  • Problem

    The study addresses limited evidence on whether narrative standardisation persists when LLMs generate stories for specific, well-documented sub-national Indian oral traditions rather than India as a single demonym.

  • Method

    The pilot compares two LLMs’ generated stories with authentic reference corpora from the Pabuji epic, Tamil Sangam poetry, and Bengali folk tales using sentence embeddings and cosine similarity.

  • Results

    Outputs remained closer to their own tradition’s references but showed 0.52–0.66 cross-tradition output similarity, while regional-language prompting consistently reduced fidelity relative to English prompting.

  • Takeaways & Limitations

    The findings support partial rather than total narrative homogenisation and extend the contested multilingual-prompting debate to narrow, lesser-documented Indian oral traditions.

  • Takeaways & Limitations

    The English-translated references and English-oriented embedding models plausibly inflate the regional-language fidelity penalty, while embeddings cannot distinguish superficial overlap from genuine structural or stylistic fidelity.

Abstract

from arXiv · show

Large language models (LLMs) are trained predominantly on English-language internet text that over-represents certain cultural narratives, raising concerns that models flatten the diversity of non-Western storytelling traditions into a single homogenized archetype. We present a pilot computational study examining this across three maximally distinct Indian regional oral and literary traditions: the Rajasthani Pabuji epic, classical Tamil Sangam poetry, and Bengali folk tales. We collected authentic reference corpora for each tradition (11, 21, and 10 passages respectively) and prompted two LLMs (Claude Sonnet and Gemini) with 54 generation requests spanning three prompt types per tradition - generic, culturally specific, and regional-language. Using Sentence-BERT embeddings and cosine similarity, we measure reference drift (how closely outputs track their own tradition's authentic texts relative to the other two) and cross-tradition convergence (how similar outputs are across traditions). We find that while outputs remain closer to their own tradition's reference than to others, cross-tradition similarity is high (0.52-0.66) relative to what the traditions' genuine distance would predict, indicating partial homogenisation. Unexpectedly, prompting in the regional language (Hindi, Tamil, or Bengali) consistently reduced fidelity to the authentic tradition relative to English prompting, by as much as 27 percentage points for Rajasthani and Bengali traditions. We discuss this against conflicting prior results on multilingual prompting and argue it reflects a difference between eliciting general cultural diversity and simulating one narrow, lesser-documented oral tradition. We position this pilot as a lightweight, scalable complement to recent large-scale human-annotation studies of Indian cultural misrepresentation in LLM-generated stories, as part of a broader doctoral research program.

I. INTRODUCTION

The paper examines whether LLMs homogenise storytelling across distinct Indian traditions and addresses this gap with a lightweight computational comparison of authentic corpora and generated stories.

  • LLMs trained predominantly on English-language internet text may overrepresent Anglo-American and Western cultural narratives.
  • Narrative research suggests that culturally marked stories can still converge on a common underlying plot structure.
  • Existing approaches either treat India as a single national demonym or require costly human-annotation infrastructure for fine-grained regional measurement.
  • The study measures narrative homogenisation by comparing generated stories with authentic reference texts from three culturally distinct Indian oral and literary traditions.
  • The traditions are the Rajasthani Pabuji epic, classical Tamil Sangam poetry, and Bengali folk tales, selected for maximal mutual distinctiveness.
  • The paper also tests whether Hindi, Tamil, or Bengali prompting improves fidelity relative to English prompting amid conflicting multilingual-prompting findings.

II. RELATED WORK

Prior research documents broad cultural bias in LLMs, including Western alignment and India-specific stereotypes, while motivating more targeted measurement of cultural narratives.

  • Cultural bias is established as part of the broader problem of training-data and alignment choices amplifying social and cultural biases.
  • Across five GPT models, cultural values clustered near English-speaking and Protestant European countries, although cultural prompting reduced distance for 71–81% of countries in later GPT-4-family models.
  • CAMeL found systematic preference for Western-associated entities in sixteen models, with Cultural Bias Scores of 40–65% against an ideal near zero.
  • IndiBias found measurable caste- and religion-linked stereotypes across ten models, with bias varying across models and not consistently decreasing in either English or Hindi.

C. Narrative Homogenisation

Related studies report narrative standardisation and cultural misrepresentation, but the present study targets specific Indian sub-national traditions and examines contested language-prompting effects.

  • Rettberg and Wigers found nearly all of 11,800 stories across 236 national demonyms converged on one plot skeleton despite nationality changes.
  • Their limited Indian sample leaves open whether standardisation persists for specific, well-documented sub-national oral traditions.
  • TALES combined community taxonomy and large-scale annotation, finding 88% of 540 stories contained at least one cultural misrepresentation.
  • AI assistance increased cross-cultural embedding similarity in Indian writing from 0.48 to 0.54, while reverse convergence was negligible.
  • Multilingual prompting findings conflict: prior work reports greater diversity, whereas other evidence places Indic-language generation closer to lower cultural fidelity.
  • The study extends this debate by finding that regional-language prompting reduces fidelity for narrow, lesser-documented oral traditions relative to English prompting.

III. METHODOLOGY

The methodology compares three maximally distinct Indian traditions using authentic English-translated reference corpora of different sizes and forms.

  • The selected traditions differ maximally in geography, language, literary form, and moral or emotional grammar while remaining sufficiently documented for English-translated corpora.
  • Rajasthani (Pabuji epic): The Pabuji corpus represents a living Rajasthani oral performance epic centred on desert heroism, cattle protection, caste loyalty, and chivalric vows.
  • Tamil (Sangam poetry): The Tamil corpus draws on Sangam poetry, a classical collection structured by the akam interior/love and puram exterior/heroic thematic division.
  • Bengali (folk tales): The Bengali corpus uses village-set folk narratives characterised by domestic settings, morally ambiguous protagonists, and frequent female-centred agency.
  • After cleaning, the reference corpus contained 11 Rajasthani, 21 Tamil, and 10 Bengali passages averaging 100–300 words each.

B. Prompting Protocol

The study prompts Claude Sonnet and Gemini across three Indian traditions using generic, culturally specific, and regional-language conditions, then evaluates outputs against tradition-specific embedding centroids. It compares own-tradition fidelity, cross-tradition convergence, prompt-type effects, and language-prompting effects.

  • B. Prompting Protocol: Three prompt types test generic, culturally specific, and regional-language cues for each tradition.The regional-language prompts use Hindi for Rajasthani, Tamil for Sangam poetry, and Bengali for Bengali folk tales.
  • B. Prompting Protocol: Each of nine prompts was issued to Claude Sonnet and Gemini in three fresh-conversation runs, producing 54 outputs.Fresh conversations prevented conversational-context cross-contamination.
  • C. Embedding-Based Analysis: Sentence-BERT all-mpnet-base-v2 embeddings represent references and outputs, with L2 normalisation making cosine similarity equivalent to dot product.The same cosine metric is used for subsequent dimensionality reduction.
  • C. Embedding-Based Analysis: Reference passages are averaged into one centroid per tradition, representing each tradition’s authentic narrative signature.These centroids provide the reference basis for comparing generated outputs.
  • C. Embedding-Based Analysis: Reference drift compares each output’s own-tradition similarity with its mean similarity to the other two tradition centroids.Other-similarity minus own-similarity yields the drift score; a positive score indicates greater similarity to non-target traditions.
  • C. Embedding-Based Analysis: Cross-tradition convergence measures pairwise cosine similarity between each tradition’s mean output embedding for each model.High similarity indicates that outputs generated for different traditions resemble one another independently of reference texts.
  • C. Embedding-Based Analysis: Prompt-type effects compare mean own-similarity across generic, culturally specific, and regional-language conditions.This tests whether culturally specific prompting improves fidelity over generic prompting.
  • C. Embedding-Based Analysis: Language-prompting effects compare pooled English prompts with regional-language prompts using mean own-similarity.Types 1 and 2 form the English baseline, while Type 3 is the regional-language condition.

IV. RESULTS

The experiment combines tradition-specific reference corpora with 54 outputs from Claude Sonnet and Gemini, while Tables I and II organise similarity results for evaluating reference drift and cross-tradition convergence.

  • IV. RESULTS: Reference corpora contain 11 Rajasthani, 21 Tamil, and 10 Bengali passages.The 54 outputs span both models and all nine prompt conditions, with three runs per condition.
  • IV. RESULTS: Table I reports mean cosine similarity to each output’s own-tradition centroid and to the mean of the other two traditions’ centroids.The table is organised by tradition and model.
  • IV. RESULTS: Table II reports pairwise cosine similarity between each model’s mean output embedding for each tradition.These pairwise comparisons evaluate resemblance among model outputs across traditions.

B. Reference Drift

Outputs remain closer to the authentic references of the traditions they target than to the other traditions, although cross-tradition output similarity remains substantial.

  • B. Reference Drift: All six tradition–model combinations show negative drift, so outputs remain closer to their own authentic references than to the other two traditions.Tamil has Claude’s largest negative drift (−0.192).
  • B. Reference Drift: Tamil shows Claude’s largest negative drift at −0.192.The paper relates this pattern to Tamil’s comparatively larger digital and scholarly footprint.
  • B. Reference Drift: Rajasthani has the smallest-magnitude drift and the highest absolute own-similarity scores, ranging from 0.626–0.659 for both models.The authors treat this result cautiously because named entities recur in prompts and reference texts.
  • C. Cross-Tradition Convergence: Cross-tradition similarity ranges from 0.518 to 0.663.Rajasthani–Bengali and Tamil–Bengali pairs consistently exceed 0.65 for both models despite distinct authentic traditions.

D. Prompt-Type Effect

Prompt type affects own-tradition similarity, with regional-language prompts performing worst across all three traditions. Direct English-versus-regional-language comparisons show substantial fidelity reductions for every tradition.

  • D. Prompt-Type Effect: Table III reports mean own-tradition similarity by prompt type.The table supports comparisons among generic, culturally specific, and regional-language prompts.
  • D. Prompt-Type Effect: The generic prompt outperforms the culturally specific prompt for Rajasthani and Tamil.For Bengali, culturally specific prompting reverses this pattern by a small margin: 0.680 versus 0.651.
  • D. Prompt-Type Effect: Bengali culturally specific prompting scores 0.680 versus 0.651 for generic prompting.This is the opposite ordering from Rajasthani and Tamil, but the margin is small.
  • D. Prompt-Type Effect: Regional-language prompting scores lowest in all three traditions by a substantial margin.This is the consistent Type 3 pattern in the prompt-type comparison.
  • D. Prompt-Type Effect: UMAP shows distinct, spatially separated tradition clusters with model outputs co-locating near their corresponding reference clusters.Both Claude and Gemini outputs appear near the relevant tradition rather than in one undifferentiated region.
  • E. Language-Prompting Effect: Table IV directly compares pooled English prompts with regional-language prompts.Types 1 and 2 are pooled as English prompts against Type 3.
  • E. Language-Prompting Effect: The regional-language fidelity drops are −0.113 for Tamil, −0.273 for Bengali, and −0.262 for Rajasthani.These values are measured against pooled English prompting.
  • E. Language-Prompting Effect: Regional-language prompting reduces own-tradition fidelity for all three traditions.The smallest drop is Tamil (−0.113), while the largest are Bengali (−0.273) and Rajasthani (−0.262).

F. Embedding Space Visualisation

The embedding analysis shows that model outputs retain some tradition-specific alignment while also converging substantially across distinct traditions. Regional-language prompting further reduced fidelity, with resource imbalance and register shifts offered as possible explanations.

  • Reference passages and model outputs form distinct tradition-specific clusters in UMAP space.Claude and Gemini outputs co-locate near the reference cluster corresponding to the prompted tradition.
  • Outputs track their own tradition’s references more closely than the other two traditions.This indicates that the models retain some tradition-specific signal rather than treating all prompts identically.
  • Cross-tradition similarity reaches 0.65 or higher between structurally and linguistically distinct traditions.The high similarity contrasts with the well-separated reference corpora and supports partial narrative homogenisation.
  • Regional-language prompting consistently reduced fidelity to the corresponding authentic corpus, with declines as large as 27 percentage points.This finding differs from studies reporting increased diversity or reduced hallucination under multilingual prompting.
  • Tamil’s greater digital and scholarly representation than Pabuji or the Bengali corpus is consistent with a resource-stratification explanation.The paper also considers regional prompts shifting outputs toward a devotional or Sanskritised register rather than the target oral-epic style.

C. The Anomalous Rajasthani Result

The apparently strong Rajasthani result should be interpreted cautiously because named-entity overlap may inflate embedding similarity. Broader design limitations—including translation-mediated comparison, proxy-based evaluation, uneven corpora, few models, and pilot-scale sampling—bound the conclusions.

  • Rajasthani has the smallest-magnitude drift and highest absolute own-similarity despite being the least digitally documented tradition.The authors caution that explicit references to Pabuji, Marwar, and cattle-protection vows may inflate similarity through lexical overlap.
  • The richer Rajasthani Type 2 prompt underperforms the bare Type 1 prompt, weakening an interpretation based on deeper knowledge activation.The pattern is more consistent with named-entity overlap driving the own-similarity signal.
  • Only Claude Sonnet and Gemini were evaluated, limiting generalisability across the broader LLM landscape.GPT-family models were excluded because of access constraints.
  • English translations and primarily English-trained embedding models may inflate the reported regional-language fidelity penalty.Native-language references and multilingual embedding models are identified as important remedies for this confound.
  • Sentence-embedding cosine similarity is a scalable proxy that cannot reliably distinguish named-entity matching from structural or stylistic fidelity.This limitation is directly relevant to interpreting the anomalous Rajasthani result.
  • The corpora contain 11, 21, and 10 passages, and the study reports 54 outputs without statistical significance testing.Uneven corpus sizes and modest per-condition samples constrain cross-tradition comparisons and the strength of the pilot’s conclusions.

VII. CONCLUSION AND FUTURE WORK

The pilot finds partial narrative homogenisation across three Indian traditions while preserving measurable own-tradition proximity, and outlines extensions to test the finding’s breadth and validity.

  • Conclusion: 0.52–0.66 cross-tradition output similarity indicates partial, rather than total, narrative homogenisation despite stronger own-tradition proximity.The study compares generated stories with authentic reference corpora from three distinct Indian oral and literary traditions.
  • Future work: The work is framed as a methodological pilot for a doctoral programme studying narrative archetypes and cultural bias in Indian regional traditions.The programme extends the AI STORIES project to regional and sub-national oral traditions.
  • Future work: Planned extensions expand traditions, passages, and model families to test whether homogenisation generalises across datasets, models, and training regimes.The proposed scope includes additional traditions and open-weight models.
  • Future work: Future work will pair native-language reference corpora with multilingual embeddings and structured human annotation to separate translation artefacts from genuine fidelity.These additions aim to test whether embedding similarity reflects narrative and structural fidelity rather than superficial lexical overlap.
  • Conclusion: Regional-language prompting consistently reduced fidelity to each tradition’s authentic narrative register, adding a narrative-specific result to an unresolved multilingual-prompting debate.This finding motivates further testing of whether the penalty is specific to narrow, lesser-documented oral traditions.
Loading 2608.26123v1…