PaperlayerView exact revision on arXiv

Source-linked AI summary

Pak3H: Evaluating the Cost of Cultural Mismatch in LLM Alignment with a Human-Contextualized Urdu Benchmark

Abdullah Hashmat, Usman Naseem, Agha Ali Raza

arXiv:2608.30065v1cs.CLcs.AI

TL;DR

Existing multilingual evaluations provide limited evidence about alignment fidelity in culturally and linguistically distinct low-resource settings, where alignment gains transfer poorly. A three-stage human-centered pipeline adapts English 3H datasets into Urdu through sample categorization, manual cultural contextualization, and translation with post-editing. Zero-shot evaluations of multiple LLMs on Pak3H reveal consistent cross-lingual alignment degradation across all three dimensions.

  • Problem

    Existing multilingual evaluations provide limited evidence about alignment fidelity in culturally and linguistically distinct low-resource settings, where alignment gains transfer poorly.

    Source: §1 · p. 1, §2 · p. 2

  • Method

    A three-stage human-centered pipeline adapts English 3H datasets into Urdu through sample categorization, manual cultural contextualization, and translation with post-editing.

    Source: §3 · p. 2

  • Results

    Zero-shot evaluations of multiple LLMs on Pak3H reveal consistent cross-lingual alignment degradation across all three dimensions.

    Source: §7 · p. 9

  • Takeaways & Limitations

    The authors conclude that human-guided localization is necessary for equitable multilingual evaluation.

    Source: §7 · p. 9

    Subsampling Alpaca and BeaverTails reduces statistical power and the ability to detect subtle patterns.

    Source: Limitations · p. 9

Abstract

from arXiv · show

Large language models (LLMs) demonstrate strong Helpfulness, Harmlessness, and Honesty (3H) alignment in English-centric settings, but these gains transfer poorly to low-resource languages due to cultural mismatches. Existing multilingual 3H benchmarks rely predominantly on automated translation or LLM based synthesis, propagating source-language biases while sacrificing local relevance. To address this gap, we introduce Pak3H1, the first human-validated, culturally contextualized Urdu benchmark suite for 3H alignment, comprising PakAlpaca (helpfulness), PakBeaverTails (harmlessness), and PakTruthfulQA (honesty). Our multi-stage pipeline integrates manual cultural adaptation and dictionary-guided post editing to prioritize native speaker judgment, ensuring both semantic fidelity and contextual authenticity. Zero-shot evaluations across multiple open and proprietary LLM architectures reveal systematic cross-lingual alignment gaps: helpfulness win rates decline under localized contexts, harmlessness guardrails break down against regional safety risks, and composite honesty metrics degrade substantially due to localized factual constraints. These findings expose structural limitations in current alignment approaches, underscoring the necessity of human-guided localization for equitable multilingual evaluation.

1 Introduction

Pak3H introduces a human-validated, culturally contextualized Urdu benchmark suite for evaluating Helpfulness, Harmlessness, and Honesty. Its zero-shot evaluations reveal cross-lingual alignment degradation and motivate human-guided cultural localization.

1 Introduction

  • Existing 3H benchmarks are rooted in English and Western norms, limiting cultural relevance and missing region-specific harms and sensitivities.TruthfulQA is largely English-only; VeritasQA relies on minimally adapted translation; Alpaca is English-centric; and BeaverTails and HH-RLHF use Western risk frameworks.Source: §1 · p. 1
  • Pak3H is the first human-validated, culturally contextualized Urdu 3H suite, adapting Alpaca, BeaverTails, and TruthfulQA into three Urdu benchmarks.Urdu is spoken by over 230 million people and remains severely underrepresented in alignment research.Source: §1 · p. 1
  • Pak3H localizes benchmark samples through categorization with majority voting and agreement validation, manual adaptation by native bilingual annotators, and dictionary-guided post-editing.The post-editing uses the official Government of Pakistan Urdu dictionary.Source: §1 · pp. 1–2
  • Zero-shot evaluations of multilingual LLMs on Pak3H reveal consistent cross-lingual alignment degradation across all three 3H dimensions.The authors describe Pak3H as the first publicly available Urdu benchmark evaluating the complete 3H framework with extensive human validation.Source: §1 · p. 2

2 Related Work

Related work reports cross-lingual alignment gains but leaves culturally faithful evaluation in low-resource settings underexamined. Pak3H addresses this gap with a native-human-annotation pipeline.

Cross-Lingual Alignment Transfer

  • English-preference tuning reports gains across 23 languages and toxicity reductions across 17, but prior evaluations measure aggregate performance rather than low-resource cultural alignment.Dang et al. (2024) report gains on Aya-23 8B (Aryabumi et al., 2024); Li et al. (2024) report mGPT 1.3B toxicity falling from 46.8% to 3.9%.Source: §2 · p. 2

Multilingual Datasets and Dataset Localization

  • Low-resource 3H datasets often use automated translation or LLM synthesis, which can propagate English biases, misalign content culturally, and lack native human validation.VeritasQA adapts TruthfulQA instances with minimal cultural verification; Okapi translates English instructions into 26 languages, while IndicLLMSuite combines translation with LLM synthesis.Source: §2 · p. 2

Cultural Adaptation and Bias Mitigation in Low-Resource Alignment

Contribution

  • Pak3H uses native human annotation, with annotator-driven categorization, manual cultural adaptation, and dictionary-guided post-editing.This contrasts with prior approaches that rely on automated translation or LLM synthesis.Source: §2 · p. 2

3 Methodology

Pak3H uses a three-stage human-centered pipeline to contextualize English 3H datasets into Urdu while preserving semantic fidelity and cultural relevance. It combines sample categorization, manual cultural adaptation, and translation with post-editing.

3 Methodology

  • The pipeline converts English 3H datasets into culturally grounded Urdu versions through sample categorization, manual cultural contextualization, and translation with post-editing.The workflow produces PakAlpaca, PakBeaverTails, and PakTruthfulQA.Source: §3 · p. 2, Figure 1 · p. 3

3.1 Stage 1: Sample Categorization

  • Annotators classified samples as Locally Contextualized, Directly Translated, or Locally Irrelevant, then used majority voting and review to resolve or discard conflicts.Because more than 75% of samples were Directly Translated, the authors report raw majority agreement of 0.685–0.911 and raw pairwise agreement of approximately 0.64–0.92, noting that the skew makes chance-corrected κ underestimate consensus. Irrecoverable Locally Irrelevant items and unresolved conflicts were discarded.Source: §3.1 · p. 3

3.2 Stage 2: Manual Cultural

  • Native bilingual annotators adapted culturally dependent samples, while independent reviewers checked meaning, cultural appropriateness, and substitutions before inclusion.Locally Irrelevant samples received a secondary adaptation and review; only those passing semantic and cultural checks were retained.Source: §3.2 · p. 3

3.3 Stage 3: Translation and Post-Editing

  • NLLB-200 1.3B produced initial Urdu translations, which annotators post-edited for accuracy, natural phrasing, and cultural fit against the Government of Pakistan Urdu dictionary.A random subset was manually inspected before full-scale processing.Source: §3.3 · p. 4

3.4 Annotator Profile and Guidelines

  • Six trained native Urdu–English undergraduate annotators from diverse Pakistani regions conducted the work, with a 150-sample pilot prompting clearer TruthfulQA DT–LI boundaries.The authors note that the cohort does not represent the full dialectal diversity of Pakistani Urdu speakers; pilot disagreement was greatest at the Directly Translated–Locally Irrelevant boundary.Source: §3.4 · p. 4
  • Guidelines required accurate Pakistani substitutions, meaning-preserving natural Urdu for idioms, and culturally familiar entities and everyday references only when intent remained intact.Annotators used real-world information rather than fabricated data; culture-specific entities could be retained or generalized when no suitable Pakistani equivalent existed.Source: §3.4 · p. 4

4 Experimental Setup

The experimental setup evaluates culturally grounded Urdu versions of three alignment benchmarks using original-protocol metrics and zero-shot model evaluations. It covers dataset construction, metric choices, and evaluation pipelines for helpfulness, harmlessness, and honesty.

4.1 Datasets

  • The benchmark suite pairs Alpaca with helpfulness, BeaverTails with harmlessness, and TruthfulQA with honesty, adapting all three English benchmarks into culturally grounded Urdu versions.TruthfulQA contains 817 questions; BeaverTails has 30,207 QA pairs, from which 3,500 unique questions were sampled; 4,500 of Alpaca’s 52,000 instruction–response pairs were sampled for contextualization.Source: §4.1 · pp. 4–5
  • Culturally neutral prompts were generally directly translated, while culturally dependent samples received full manual adaptation under the locally contextualized protocol.The dominance of directly translated samples reflects the source benchmarks’ many culturally neutral prompts.Source: §4.1 · p. 5

4.2 Evaluation Metrics

  • Helpfulness uses Win Rate, while harmlessness uses a Safety Score from Beaver-dam-7B; higher Win Rate and lower Safety Score indicate better outcomes.Metrics are percentages unless otherwise specified; MC1–MC3 and generation similarity scores use a [0,1] scale.Source: §4.2 · p. 5
  • Honesty is assessed with MC1–MC3 accuracy, BLEU and ROUGE generation scores, and a GPT-Judge Truthful–Informative score, with higher TI indicating better honesty.MC1–MC3 test preference for truthful answers over misconceptions; TI classifies responses as truthful and/or informative.Source: §4.2 · p. 5

4.3 Zero-Shot Setup

  • All evaluations are zero-shot, without task-specific fine-tuning or in-context examples; honesty uses the official TruthfulQA library and GPT-Judge-7B, and harmlessness uses Beaver-Dam-7B across 14 harm categories.The harmlessness classifier is applied to both English and Urdu outputs for direct comparison; multilingual model results are also reported for honesty.Source: §4.3 · pp. 5–6
  • Helpfulness uses AlpacaEval pairwise preferences across multiple generator–judge configurations, comparing English and Urdu outputs with language-matched references on 4,445 instructions.The configurations include gpt-4o-mini as both generator and judge, plus DeepSeekV3.2 and Llama-3-70B-Instruct generators judged by gpt-4o-mini.Source: §4.3 · p. 6

5 Results

Across PakAlpaca, PakBeaverTails, and PakTruthfulQA, contextualized Urdu reveals cross-lingual degradation in helpfulness, harmlessness, and honesty. The results associate these gaps with cultural context and localized factual constraints, underscoring the need for human-guided localization.

5.1 Helpfulness (PakAlpaca)

  • English consistently outperforms Urdu across generator–judge configurations, and the gap is statistically significant under the gpt-4o-mini baseline.McNemar’s test gives χ2(1) = 27.13 (p < 0.001), and the two-proportion z-test gives z = 8.47 (p < 0.0001). Directly translated samples show smaller degradation than locally contextualized and locally irrelevant samples; the trend also appears with independent generator setups.Source: §5.1 · p. 6

5.2 Harmlessness (PakBeaverTails)

  • 74.12%→88.74% Safety Score (unsafe responses) in contextualized Urdu indicates a substantial increase in unsafe or harmful outputs.Average maximum harm probability is lower in Urdu (62.01% vs. 68.04%), suggesting a calibration difference; manual validation found classifier–human agreement of 88% on Urdu versus 92% on English. The reported gap is attributed to overlooked cultural sensitivities, regional harms, and Urdu formality registers rather than classifier limitations.Source: §5.2 · pp. 6–7, Table 3 · p. 6

5.3 Honesty (PakTruthfulQA)

  • GPT-4o-mini remains the strongest judge across categories and languages, while the English–Urdu performance gap is most pronounced for locally irrelevant samples.Qwen2.5-7B and GPT-Judge-7B show near-zero TI Composite scores on Urdu across categories, suggesting limited reliability for low-resource language evaluation.Source: §5.3 · p. 7
  • 70.21%→50.62% TI Composite for GPT-4o-mini in contextualized Urdu, with composite scores degrading across all models.Truthfulness also falls for GPT-4o-mini (88.29%→74.11%) and Llama-3-8B (42.25%→25.00%), indicating reduced factual alignment rather than mere translation artifacts. The findings point to factual challenges for English-trained representations and a need for native multilingual preference optimization.Source: Table 4 · p. 7, §5.3 · p. 7
  • GPT-4o-mini MC1 falls from 0.783 to 0.562, while GPT-Judge-7B rises from 0.224 to 0.502; GPT-4o-mini BLEU and ROUGE-1 fall from 0.218 to 0.087 and 0.512 to 0.030.The anomalous multiple-choice increase is attributed to erratic probability-mass redistribution in low-resource tokens, not an honesty gain. English–Urdu raw n-gram scores are not directly comparable, but the within-Urdu drop against culturally adapted references indicates difficulty producing fluent, contextually appropriate, factual responses.Source: §5.3 · p. 7

6 Discussion

The discussion finds that English-centric alignment degrades across helpfulness, harmlessness, and honesty under culturally contextualized Urdu evaluation. Human-guided localization partly mitigates these gaps, while naive translation and persistent shortcomings motivate scalable approaches that retain native-speaker validation.

6 Discussion

  • Across 4,445 PakAlpaca instructions, localized Pakistani norms and institutions significantly reduce helpfulness win rates, a gap that persists across independent generator–judge setups.Directly translated tasks show only minor variance; localized contexts fall outside English training distributions.Source: §6 · pp. 7–8
  • Safety Score rises from 74.12% to 88.74% on PakBeaverTails, while lower maximum harm probability indicates more frequent safety triggers but slightly less severe individual outputs.Manual validation found 88% human-classifier agreement on Urdu text; the failures reflect underrepresentation of local risks and Urdu honorific protocols in Western safety taxonomies.Source: §6 · p. 8
  • Composite TI scores fall from 70.21% to 50.62% for gpt-4o-mini, while weaker models show unstable MC1–MC3 metrics that can mask factual non-compliance.Uniform drops in internal lexical alignment indicate untruthful fabrications rather than merely reduced translation fluency.Source: §6 · pp. 8–9

6.1 Human-Guided Localization vs. Naïve MT

  • Naive MT underperforms human-edited localization: PakTruthfulQA TI falls from 50.62% to 47.81%, and PakAlpaca win rate from 29.26% to 26.16% (DeepSeek →gpt-4o-mini).The comparison attributes the losses to semantic drift and cultural loss in raw translation.Source: Table 5 · p. 9, §6.1 · p. 9

6.2 Implications for Multilingual Alignment

  • Culturally contextualized low-resource evaluation reveals degradation across all three 3H dimensions: naive transfer amplifies safety violations, reduces helpfulness, and creates honesty trade-offs that aggregate metrics may obscure.Source: §6.2 · p. 9
  • Manual categorization, cultural substitution, and dictionary-verified post-editing partly mitigate alignment gaps through native judgment and semantic fidelity, but persistent gaps motivate scalable hybrid localization.Source: §6.2 · p. 9

7 Conclusion

Pak3H evaluations show cross-lingual alignment degradation across helpfulness, harmlessness, and honesty, motivating human-guided localization. The benchmark centers native judgment through culturally contextualized Urdu tasks and a human-centered construction pipeline.

7 Conclusion

  • Pak3H is the first human-validated, culturally contextualized Urdu benchmark suite for 3H alignment, comprising PakAlpaca, PakBeaverTails, and PakTruthfulQA.The three benchmarks address helpfulness, harmlessness, and honesty, respectively.Source: §7 · p. 9
  • Zero-shot evaluations of multiple LLMs reveal consistent cross-lingual alignment degradation across all three 3H dimensions, underscoring the need for human-guided localization.The conclusion frames localization as necessary for equitable multilingual evaluation.Source: §7 · p. 9
  • Pak3H uses a three-stage human-centered pipeline of annotator-driven categorization, manual cultural adaptation, and dictionary-guided post-editing to prioritize native judgment and semantic fidelity.Source: §7 · p. 9

Limitations

The benchmark’s evidence is constrained by subsampling, a small annotator pool, and zero-shot evaluation with potential pretraining leakage, while contextualization procedures differ across sample types.

Limitations

  • Alpaca and BeaverTails were subsampled, while all 817 TruthfulQA questions were used, limiting statistical power and making full contextualization costly.The paper notes that full-scale contextualization would substantially increase annotation cost and effort.Source: Limitations · p. 9
  • Only six undergraduate Pakistani annotators participated; despite review and agreement checks, residual subjectivity or regional bias may affect nuanced cultural, sectarian, or social adaptations.Source: Limitations · p. 9
  • Evaluations were zero-shot on contextualized datasets using popular source benchmarks, leaving potential pre-training data leakage risk; the stated scope is dataset curation and evaluation design.Source: Limitations · p. 9
  • All LC and retained LI samples were fully manually contextualized, whereas DT samples were dictionary-guided and human post-edited, introducing distributional shifts.Source: Limitations · p. 9
  • Severe, systematic degradation across evaluated models is presented as evidence that prior English exposure cannot explain the gaps, supporting robustness against surface memorization.Source: Limitations · p. 9

Ethics Statement

The work adapts English alignment benchmarks into culturally contextualized Urdu datasets for more equitable multilingual evaluation, using human-guided localization while acknowledging residual bias and managing sensitive content with annotator safeguards and responsible-use guidance.

Ethics Statement

  • The adapted Urdu datasets support more equitable multilingual evaluation of helpfulness, harmlessness, and honesty.Source: Ethics Statement · pp. 9–10
  • Human-guided localization and validation aim to reduce English-centric bias and semantic distortion, though annotator subjectivity and regional bias may persist.Source: Ethics Statement · p. 10
  • Annotators handling potentially harmful or sensitive content received a briefing and pilot, gave informed consent, could opt out, and had mental health resources available.Source: Ethics Statement · p. 10
  • Harmful, misleading, or culturally sensitive dataset content is included strictly for alignment evaluation and robustness analysis and should be used responsibly.Source: Ethics Statement · p. 10

Future Work … E Classifier Validation Qualitative Example

The appendix outlines planned full-dataset and multilingual fine-tuning studies, documents annotation and win-rate reporting, and gives a qualitative example of the Urdu moderation classifier preserving safety classifications across English and Urdu.

Future Work

  • Future work will scale the framework to the full dataset and fine-tune multilingual LLMs on contextualized data to study scalability, robustness, performance, and downstream 3H behavior.The current framework uses a subsampled dataset; the planned studies will examine how culturally grounded safety annotations affect alignment quality.Source: Future Work · p. 10

A Inter-Annotator Agreement

  • Fleiss’ κ is reported for each dataset, but Directly Translated category imbalance may depress κ despite high annotator consensus.The table caption likewise notes that high DT prevalence can skew agreement statistics.Source: A Inter-Annotator Agreement · p. 11, Table 6 · p. 11

C Conflicted Samples

  • Samples with differing LC, DT, or LI labels received a second annotation round; consensus labels were assigned, while persistently conflicted samples were discarded.Figure 5 shows examples of conflicted cases and their annotator label distributions.Source: C Conflicted Samples · p. 11

D Dataset Samples

  • The appendix provides dataset prompt examples, pairing English versions with culturally adapted Urdu variants across categories.Source: D Dataset Samples · p. 11

E Classifier Validation Qualitative Example

  • The Beaver-Dam-7B classifier preserved semantic intent and produced highly aligned safety classifications for a representative matched English–Urdu prompt-response pair.The example comes from the manual validation study and illustrates cross-lingual consistency on low-resource Urdu text.Source: E Classifier Validation Qualitative… · p. 11

B Detailed PakAlpaca Win-Rate Breakdown

  • Table 7 reports English and Urdu PakAlpaca win rates for three generator–judge configurations, across the full dataset and DT, LC, and LI samples, with differences in percentage points.The supplied passages describe the table’s scope but do not provide its cell values.Source: Table 7 · p. 12

F Evaluation Configuration and Parameters

The evaluation uses default or deterministic settings from each benchmark’s official framework to support reproducibility and comparability without parameter tuning. The protocols differ by benchmark: classification for PakBeaverTails, greedy probability-based scoring for PakTruthfulQA, and judged generation for PakAlpaca.

F Evaluation Configuration and Parameters

Related papers

Loading 2608.30065v1…