Source-linked AI summary
Evaluating Cultural Awareness of LLMs for Haitian Creole
Christelle Clervilsson, Yanzhu Guo
TL;DR
Haitian Creole is underrepresented in digital resources, and existing benchmarks offer limited coverage of its cultural knowledge and everyday experience. This paper evaluates cultural awareness with a native-curated benchmark and entity infilling, finding a gap relative to French and recurring hardship-and-resilience portrayals of Haitian characters.
Problem
Haitian Creole is severely underrepresented in digital resources, while existing resources and benchmarks provide limited coverage of its cultural knowledge and everyday experience.
Source: §1 · pp. 1–2Method
Results
The evaluation finds a clear Haitian Creole–French cultural-awareness gap, cross-domain variation, and recurring hardship-and-resilience portrayals of Haitian characters.
Source: §5 · p. 9Takeaways & Limitations
The benchmark and evaluation framework provide resources for studying cultural awareness in low-resource languages.
Source: §5 · p. 9The manually curated benchmark is incomplete and cannot capture the full diversity of Haitian cultural practices across regions and communities.
Source: Limitations · p. 9
Abstract
from arXiv · showhide
Large language models (LLMs) exhibit substantial performance disparities between high- and low-resource languages. Beyond lower task performance, they often fail to capture the cultural norms and values of underrepresented communities. In this work, we present the first systematic evaluation of cultural awareness in LLMs for Haitian Creole, a language spoken by millions but severely underrepresented in digital resources. We assess cultural awareness along four complementary dimensions---specificity, bias, diversity, and variation---using a benchmark of culturally salient prompts curated by native speakers in a text infilling setting. Our results reveal a clear gap between cultural awareness in Haitian Creole and higher-resource French, with Haitian performance being more uneven across domains and more affected by French linguistic interference. Story generation further reveals recurring portrayals of Haitian characters through hardship and resilience, showing that even positive characterizations can encode stereotypical narratives. Our code, benchmark, and evaluation framework are publicly available.
1 Introduction
Multilingual fluency does not ensure cultural awareness, and Haitian Creole’s digital and benchmark underrepresentation leaves its cultural grounding insufficiently assessed. The paper introduces a native-curated benchmark and evaluates cultural awareness through entity infilling and story generation.
1 Introduction
- Existing Haitian Creole resources and benchmarks mainly evaluate standard NLP tasks or specialized domains, leaving cultural knowledge and everyday experience with limited coverage.Some resources rely heavily on biblical texts or specialized medical conversations.Source: §1 · p. 2
- The paper asks whether multilingual LLMs can faithfully represent Haitian culture when generating text in Haitian Creole.Haitian Creole is spoken by more than 13 million people and remains severely underrepresented in digital resources.Source: §1 · pp. 1–2
- The study builds on CAMeL to evaluate Haitian Creole cultural awareness through entity infilling across specificity, bias, diversity, and variation.It also analyzes stereotypes associated with Haitian personas through story generation.Source: §1 · p. 2
- The authors introduce a benchmark of culturally salient Haitian Creole prompts and entities curated and annotated by native speakers.Source: §1 · p. 2
2 Related Work
Prior Creole NLP work has advanced resources and standard task evaluation, while cultural-awareness research has developed cross-cultural benchmarks. This paper connects those strands by evaluating culturally grounded generation for Haitian Creole.
2 Related Work
- Prior Creole NLP work emphasizes resource construction, modeling, and task-based evaluation, but does not examine culturally grounded text or speakers’ lived experiences.Examples include Kreyol-MT and CreoleVal; prior work covers machine translation and multiple standard NLP tasks.Source: §2 · p. 2
- Cultural-awareness frameworks assess multilingual generation through entity infilling, story generation, or extracted cultural entities, alongside dimensions such as specificity and diversity.CAMeL (Naous et al., 2024) uses infilling and story generation; MAKIEVAL (Zhao et al., 2025) links extracted entities to Wikidata.Source: §2 · p. 3
- This work connects Creole NLP with cultural-awareness evaluation through a native-curated benchmark of Haitian entities and prompts.It uses the benchmark to evaluate multilingual LLMs in a low-resource language and cultural setting.Source: §2 · p. 3
3 Cultural Entity Infilling
Cultural entity infilling tests whether generated entities reflect a prompt’s cultural context. The section builds a native-speaker-curated benchmark, defines evaluation measures, and compares model and language conditions.
3 Cultural Entity Infilling
- The task asks models to replace a [MASK] with one entity from a specified category that reflects the prompt’s cultural context.Culturally contextualized prompts explicitly cue Haitian context; culturally agnostic prompts omit those cues to test default cultural references.Source: §3 · p. 3, §3.1 · p. 4
3.1 Benchmark Construction
- Existing Haitian Creole datasets were unsuitable, so the authors built a dedicated benchmark and supplemented sparse Wikidata coverage with native-speaker curation.The benchmark spans seven categories, with Wikidata retained for Authors and Locations and other categories curated from scratch.Source: §3.1 · p. 3
3.2 Model Selection
- Mistral was selected for the main experiments because it produced the most coherent Haitian Creole generations among the systems tested.Other tested models struggled to generate more than a few coherent Haitian Creole sentences or exceeded available computational resources.Source: §3.2 · p. 5
3.3 Entity matching.
- Exact matching against category reference lists makes evaluation reproducible, but valid regional or oral variants may be missed and excluded.The authors note that reference lists may not exhaustively capture culturally relevant expressions, particularly variants poorly represented online or in corpora.Source: §3.3 · p. 5
3.4 Evaluation Metrics
- Specificity and diversity assess culturally aligned outputs, bias measures non-target outputs without cultural cues, and variation measures response differences across repeated generations.Variation is calculated from ten generations per culturally contextualized prompt; diversity counts distinct listed entities among valid generated entries.Source: §3.4 · p. 5
3.5 Cultural Entity Infilling Results
- Medium Mistral has higher specificity and lower bias across most categories, while large Mistral has higher diversity among its culturally aligned outputs.Performance is uneven across domains: the medium model is stronger for Musical Groups and Authors but weaker for Festivals and Locations, where it can conflate French and Haitian Creole place names or mischaracterize celebrations.Source: Figure 6 · p. 4, §3.5 · p. 6
3.6 Comparing Cultural Awareness in Haitian and French Contexts
- Mistral shows stronger cultural awareness for French than Haitian culture, particularly in specificity and diversity, but the gap varies by category.Haitian variation may be inflated by alternate spellings of the same entity; the Food bias comparison is also sensitive to the French reference list’s exclusion of globally common foods.Source: §3.6 · pp. 6–7
- The Location gap is larger, with French forms of Haitian place names lowering Haitian specificity, while Musical Groups show no measurable Haitian bias.French generations sometimes include globally recognized groups such as BTS; the authors suggest this pattern warrants considering unequal access to global references in cultural evaluation.Source: §3.6 · p. 7
4 Story Generation
The story-generation task compares portrayals of Haitian and French characters conditioned on culturally salient entities. Haitian characters recur in narratives of adversity and resilience, while French characters receive a broader range of identities and experiences.
4 Story Generation
- The task compares how stories portray people linked to different cultural contexts and whether systematic characterization differences emerge.Source: §4 · p. 7
4.1 Experimental Setup
- Initial generations contained frequent French words and inconsistent spelling, so the prompt was iteratively refined to encourage more natural Haitian Creole.Source: §4.1 · p. 7
- Stories are conditioned on Haitian or French names, food, and musical groups to examine whether culturally salient entities influence character portrayals.The Names condition uses two lists of 150 Haitian and 150 French first names; the other categories reuse entity lists from the infilling task.Source: §4.1 · p. 7
4.2 Extracting Character Descriptors
- The analysis extracts five salient adjectives per story and uses z-scores to identify descriptors disproportionately associated with Haitian or French characters.The z-score measures an adjective's association with a cultural condition relative to its overall frequency.Source: §4.2 · p. 8
4.3 Results and Analysis
- Across Names, Food, and Musical Groups conditions, Haitian and French characters receive distinct descriptor profiles, with differences weaker outside Names.Haitian characters are often described through strength and perseverance; French characters more often receive traits such as independence, thoughtfulness, and organization.Source: §4.3 · p. 8
4.4 Key Insights
- Haitian characters are repeatedly framed through adversity and resilience, while French characters receive a broader range of identities and life trajectories.Haitian characters' positive traits often co-occur with poverty, hardship, and humble origins; French narratives include intellectual, adventurous, and everyday experiences.Source: §4.4 · pp. 8–9
- Stereotyped portrayals persist with food or musical-group cues, even without explicit nationality, and positive traits can still reinforce a narrow narrative.The analysis cautions against treating positive sentiment as evidence of unbiased representation; the surrounding narrative and circumstances matter.Source: §4.4 · p. 9
5 Conclusion
The paper introduces a benchmark and framework for evaluating Haitian cultural awareness in multilingual LLMs. Its evaluation finds a Haitian Creole–French gap, domain variation, and recurring portrayals of Haitian characters through hardship and resilience.
5 Conclusion
- The benchmark combines native-speaker-curated Haitian cultural prompts and entities with measures of specificity, diversity, bias, and variation, plus story-generation analysis.Source: §5 · p. 9
- The evaluation finds a cultural-awareness gap between Haitian Creole and higher-resource French, substantial variation across domains, and recurring Haitian portrayals through hardship and resilience.Source: §5 · p. 9
- The authors argue that cultural evaluation should examine not only language support but also how faithfully and inclusively models represent a culture.Source: §5 · p. 9
Limitations
The evaluation's conclusions are bounded by incomplete cultural inventories, Haitian Creole orthographic and contact variation, the story-analysis method, and limited model coverage.
Limitations
- Manually curated prompts and entity inventories are incomplete, and outputs outside their lists may be counted as errors even when culturally plausible.This is especially relevant to narrow categories such as Food and Festivals; the benchmark cannot capture the full diversity of Haitian practices across regions and communities.Source: Limitations · p. 9
- Orthographic variation and French contact complicate entity matching, and manual normalization introduces subjectivity into specificity, diversity, and variation metrics.Model errors cannot be attributed to cultural knowledge alone because linguistic competence and cultural representation are intertwined.Source: Limitations · p. 9
- The same Mistral model generates stories and extracts adjectives, potentially reinforcing patterns, while adjective-based analysis may miss broader narrative differences.The authors note that independent models or human annotators would provide a more robust evaluation.Source: Limitations · p. 9
- Computational constraints limit quantitative experiments to a small number of models, so findings are an initial characterization rather than a comprehensive model-family comparison.Other model interfaces supplied exploratory evidence but were not evaluated systematically.Source: Limitations · pp. 9–10
A Existing Haitian Creole Resources
Existing Haitian Creole datasets provide substantial text but were generally built for language tasks rather than culturally salient knowledge, motivating manual curation for this study.
A Existing Haitian Creole Resources
- Public Haitian Creole datasets offer broad text coverage but were not specifically curated for culturally salient knowledge and have limitations in domain coverage, data quality, or diversity.These shortcomings motivated manual curation of culturally grounded prompts and entities for the evaluation.Source: Table 5 · p. 12, A Existing Haitian Creole Resources · p. 12
B Overview of Multilingual LLMs
The model-selection overview records whether Haitian Creole is explicitly documented in each model family’s multilingual coverage and reports training scale and coverage from public documentation.
B Overview of Multilingual LLMs
- Model families are compared by documented multilingual coverage and training scale, based on publicly available model documentation.“Haitian Creole” marks explicit documentation; “Not specified” means its inclusion is not explicitly documented.Source: Table 6 · p. 13
C Error Analysis
The error analysis shows that cultural entity infilling scores depend on normalization and annotation choices, while linguistic ambiguity and culturally plausible variants complicate evaluation. Errors include generic cultural responses that are relevant to the Caribbean but not specifically Haitian.
C Error Analysis
- Orthographic variants of one entity, such as griyo, complicate evaluation in a language with non-standardized spelling conventions.Source: Table 7 · p. 13
- Haitian Creole prompts can lack grammatical cues that distinguish entity types, and its flexible expressions and unmarked adjective gender complicate alignment.Prompts were designed to reduce ambiguity and avoid uninterpretable completions.Source: Table 8 · p. 13, C Error Analysis · p. 14
- Normalization and annotation choices can substantially change measured scores, including Mistral specificity after erroneous or ambiguous Food outputs are removed.Specificity is also affected by removing “fèt” before festival names and by requiring exact proper names for Political Figures and Heroes.Source: C Error Analysis · pp. 13–14
- Some outputs are semantically incoherent because models misunderstand the prompt, while other ambiguous completions reflect Haitian Creole’s linguistic flexibility.Source: C Error Analysis · p. 14
- Food responses may be linguistically plausible but culturally unnatural: diri kole names a specific dish, unlike the literal diri ak pwa.A plausible Haitian Creole form alone does not establish that the answer names the intended culturally specific food.Source: C Error Analysis · p. 14, Figure 10 · p. 14
- Under agnostic prompts, models sometimes use French rather than Haitian Creole place names; political responses can be slogans or nicknames outside benchmark scope.These political expressions reflect aspects of Haitian political culture but introduce variants beyond the benchmark’s scope.Source: C Error Analysis · p. 14
- Generic Caribbean answers may be understandable and relevant yet fail to identify specifically Haitian entities, especially in Food.Shared ingredients and dishes make culturally specific names and preparations important for assessing Haitian cultural knowledge.Source: C Error Analysis · p. 14
D Story Generation Prompts
Story-generation prompts keep a fixed structure while varying the conditioning entity by category and cultural condition. The evaluation extracts five salient adjectives from each generated story.
D Story Generation Prompts
- The story prompt structure stays fixed while the conditioning entity varies by category: name and nationality, food item, or artist or group.Separate entity lists are used for each category and cultural condition.Source: Table 9 · p. 15
- Story evaluation extracts five salient adjectives from each generated story for subsequent analysis.Source: Table 10 · p. 15