PaperlayerView exact revision on arXiv

Source-linked AI summary

Pak3H: Evaluating the Cost of Cultural Mismatch in LLM Alignment with a Human-Contextualized Urdu Benchmark

Abdullah Hashmat, Usman Naseem, Agha Ali Raza

arXiv:2608.30065v1cs.CLcs.AI

TL;DR

English-centric alignment gains transfer poorly to culturally distinct, low-resource languages, while existing benchmarks often rely on translation or synthesis with limited local adaptation. Pak3H introduces a human-validated Urdu 3H suite built through staged cultural localization, and zero-shot evaluations report degradation across all three dimensions, including sharp changes in harmlessness and honesty metrics.

  • Problem

    Existing multilingual 3H benchmarks provide limited evidence about alignment fidelity in culturally and linguistically distinct low-resource settings and often rely on translation or synthesis.

  • Method

    Pak3H builds PakAlpaca, PakBeaverTails, and PakTruthfulQA through human categorization, manual cultural adaptation, dictionary-guided post-editing, and zero-shot evaluation.

  • Results

    Zero-shot evaluations reveal consistent cross-lingual alignment degradation across all three dimensions; Safety Score rises from 74.12% to 88.74% and TI falls from 70.21% to 50.62%.

  • Takeaways & Limitations

    The findings expose limitations of English-centric alignment and highlight the need for human-guided cultural localization in multilingual evaluation.

  • Takeaways & Limitations

    Alpaca and BeaverTails are subsampled, reducing statistical power, while six annotators may leave residual subjectivity or regional bias.

Abstract

from arXiv · show

Large language models (LLMs) demonstrate strong Helpfulness, Harmlessness, and Honesty (3H) alignment in English-centric settings, but these gains transfer poorly to low-resource languages due to cultural mismatches. Existing multilingual 3H benchmarks rely predominantly on automated translation or LLM based synthesis, propagating source-language biases while sacrificing local relevance. To address this gap, we introduce Pak3H1, the first human-validated, culturally contextualized Urdu benchmark suite for 3H alignment, comprising PakAlpaca (helpfulness), PakBeaverTails (harmlessness), and PakTruthfulQA (honesty). Our multi-stage pipeline integrates manual cultural adaptation and dictionary-guided post editing to prioritize native speaker judgment, ensuring both semantic fidelity and contextual authenticity. Zero-shot evaluations across multiple open and proprietary LLM architectures reveal systematic cross-lingual alignment gaps: helpfulness win rates decline under localized contexts, harmlessness guardrails break down against regional safety risks, and composite honesty metrics degrade substantially due to localized factual constraints. These findings expose structural limitations in current alignment approaches, underscoring the necessity of human-guided localization for equitable multilingual evaluation.

1 Introduction

Pak3H addresses the poor transfer of English-centric 3H alignment to low-resource languages by introducing a human-validated, culturally contextualized Urdu benchmark suite. Its localization pipeline combines human categorization, manual adaptation, and dictionary-guided post-editing, while zero-shot evaluations reveal alignment degradation across all three dimensions.

  • Motivation and Contribution: English-centric 3H gains transfer poorly to multilingual and low-resource languages where cultural, linguistic, and contextual nuances matter for deployment.
  • Motivation and Contribution: Existing benchmarks rely on English and Western norms, translation, or synthesis, limiting cultural relevance and missing region-specific harms.
  • Motivation and Contribution: Pak3H is presented as the first human-validated, culturally contextualized Urdu benchmark suite covering helpfulness, harmlessness, and honesty.It comprises PakAlpaca, PakBeaverTails, and PakTruthfulQA, derived from Alpaca, BeaverTails, and TruthfulQA.
  • Method and Evaluation: The three-stage pipeline uses annotator categorization, manual cultural adaptation, and dictionary-guided post-editing to preserve semantic fidelity and Pakistani cultural relevance.
  • Method and Evaluation: Zero-shot evaluations across multiple multilingual LLMs reveal consistent cross-lingual alignment degradation across helpfulness, harmlessness, and honesty.

2 Related Work

Prior work shows multilingual alignment transfer and scalable dataset construction, but often evaluates aggregate performance or prioritizes translation over cultural fidelity. Pak3H targets this gap through native human annotation and culturally grounded Urdu localization.

  • Cross-Lingual Alignment Transfer: English-only preference tuning has produced multilingual gains, but prior studies assess aggregate cross-lingual performance rather than culturally distinct low-resource settings.
  • Multilingual Datasets and Dataset Localization: Existing localized 3H datasets commonly use automated translation or LLM synthesis, prioritizing scalability or semantic consistency over cultural relevance.
  • Cultural Adaptation and Bias Mitigation: Recent work indicates that translation alone may lose cultural nuance and motivates locally relevant adaptations for South Asian contexts.
  • Cultural Adaptation and Bias Mitigation: Pak3H differs from prior approaches by relying heavily on native human annotation, manual cultural adaptation, and dictionary-guided post-editing.

3 Methodology

The methodology localizes English 3H benchmarks into Urdu through staged categorization, culturally informed adaptation, review, translation, and post-editing. Native bilingual annotators and agreement checks are used to preserve semantic intent and cultural appropriateness.

  • Pipeline Overview: The pipeline proceeds through sample categorization, manual cultural contextualization, and translation with post-editing.
  • Stage 1: Sample Categorization: Annotators classify samples as Locally Contextualized, Directly Translated, or Locally Irrelevant according to Pakistani cultural adaptability.
  • Stage 1: Sample Categorization: Majority voting, agreement validation, secondary review, and discarding of irrecoverable items support reliable category assignment and semantic preservation.
  • Stage 2: Cultural Adaptation: Native bilingual annotators replace culturally specific entities and scenarios with Pakistani equivalents while preserving semantic intent and instructional structure.
  • Stage 2: Cultural Adaptation: Independent review checks semantic preservation, cultural appropriateness, and incorrect substitutions before adapted samples are included.
  • Stage 3: Translation and Post-Editing: NLLB-200 supplies initial Urdu translations, followed by systematic post-editing that corrects inaccuracies, unnatural phrasing, literal translations, and inappropriate terms.
  • Annotator Profile and Guidelines: The six-person annotation team comprised native Urdu–English bilingual undergraduates from diverse Pakistani regions, though it does not fully represent dialectal diversity.
  • Annotator Profile and Guidelines: Adaptation guidelines require accurate Pakistani substitutions, natural Urdu idioms, and culturally familiar entities when original intent is preserved.

4 Experimental Setup

The experiments construct culturally grounded Urdu versions of three established 3H benchmarks and evaluate multiple LLMs under strict zero-shot protocols. Metrics are dimension-specific, with English and Urdu outputs compared using benchmark-aligned evaluators and references.

  • Datasets: The study builds one culturally grounded Urdu benchmark per 3H dimension from Alpaca, BeaverTails, and TruthfulQA.
  • Datasets: More than 75% of samples fall into the Directly Translated category because many source prompts are culturally neutral.
  • Evaluation Metrics: All metrics are percentages unless specified, while MC1–MC3 and generation similarity scores use the [0,1] scale.
  • Evaluation Metrics: Helpfulness uses AlpacaEval Win Rate, and harmlessness uses a Beaver-dam-7B-based Safety Score where lower values indicate safer behavior.
  • Evaluation Metrics: Honesty uses MC1–MC3, BLEU, ROUGE, and a GPT-Judge-based Truthful–Informative score.
  • Zero-Shot Setup: Evaluations use strict zero-shot inference without task-specific fine-tuning or in-context examples.
  • Zero-Shot Setup: English and Urdu outputs are compared across 4,445 instructions using multiple generator–judge configurations and dimension-specific evaluation models.

5 Results

Zero-shot evaluations reveal consistent alignment differences between English and adapted Urdu across helpfulness, harmlessness, and honesty. The gaps are largest in culturally contextualized categories and persist across models and evaluation setups.

  • Helpfulness: English consistently outperforms Urdu across PakAlpaca generator–judge configurations.The comparison covers 4,445 instructions and multiple generator–judge pairings.
  • Helpfulness: Locally contextualized and locally irrelevant samples show substantially larger helpfulness drops than directly translated samples.The pattern is consistent across models, categories, and independent generator–judge pairings.
  • Harmlessness: 88.74% Safety Score in contextualized Urdu versus 74.12% indicates markedly higher rates of unsafe or harmful outputs.Average maximum harm probability is slightly lower in Urdu, suggesting a calibration difference rather than classifier failure.
  • Honesty: 70.21% →50.62%: PakTruthfulQA TI Composite declines for GPT-4o-mini in adapted Urdu.The composite decline is predominantly driven by reduced truthfulness rather than translation artifacts.
  • Honesty: GPT-4o-mini MC1 falls from 0.783 to 0.562 in Urdu, while weaker GPT-Judge-7B scores anomalously increase from 0.224 to 0.502.The divergent weaker-model pattern is attributed to erratic probability-mass redistribution in low-resource tokens.
  • Honesty: BLEU drops 0.218 →0.087 and ROUGE-1 drops 0.512 →0.030 for GPT-4o-mini in Urdu.Raw English–Urdu n-gram overlap is not directly comparable, but intra-language comparisons still indicate difficulty producing fluent, appropriate, factual responses.

6 Discussion

Discussion of the three evaluation dimensions shows that culturally contextualized Urdu exposes systematic weaknesses in English-centric alignment. Human-guided localization improves contextual fidelity relative to naive translation but does not eliminate cross-lingual degradation.

  • Helpfulness: PakAlpaca helpfulness degradation persists across independent generator–judge setups and is strongest for scenarios involving Pakistani norms, laws, and institutions.Directly translated tasks show only minor variance, whereas localized scenarios fall outside English training distributions.
  • Harmlessness: 88% human–classifier agreement on Urdu supports interpreting PakBeaverTails safety degradation as genuine model failures rather than classifier limitations.The Safety Score rises from 74.12% to 88.74%, while average harm probability declines slightly.
  • Harmlessness: Limited representation of sectarian dynamics, regional risks, and Urdu honorific protocols leaves standard Western safety taxonomies incomplete.The discussion links these omissions to more frequent safety-boundary violations with slightly lower individual-output severity.
  • Honesty: 70.21% →50.62%: GPT-4o-mini’s TI Composite declines when honesty questions are anchored to local Pakistani realities.English-trained representations fail to generalize to region-specific truthfulness constraints.
  • Honesty: Unstable MC1–MC3 behavior in weaker models can mask factual non-compliance, while lexical metrics decline uniformly in Urdu.The combined pattern indicates difficulties beyond surface translation fluency.
  • Localization: Direct machine translation underperforms human post-editing, with PakTruthfulQA TI falling from 50.62% to 47.81% and PakAlpaca win rate from 29.26% to 26.16%.The comparison attributes the baseline gap to semantic drift and cultural loss.
  • Overall discussion: Across all three 3H dimensions, English-centric alignment degrades on culturally contextualized low-resource content, while human-guided localization partially mitigates the issues.The reported pattern includes increased safety violations, reduced helpfulness, and honesty trade-offs.

7 Conclusion

Pak3H is presented as a human-validated, culturally contextualized Urdu suite covering the complete 3H framework. Its zero-shot evaluations reveal consistent alignment degradation across all three dimensions in culturally adapted low-resource settings.

  • Conclusion: Pak3H comprises PakAlpaca, PakBeaverTails, and PakTruthfulQA for helpfulness, harmlessness, and honesty.The suite is described as the first culturally contextualized Urdu benchmark suite for 3H alignment.
  • Conclusion: A three-stage human-centered pipeline combines annotator categorization, manual cultural adaptation, and dictionary-guided post-editing.The design emphasizes native judgment and semantic fidelity.
  • Conclusion: Zero-shot evaluations across multiple LLMs reveal consistent cross-lingual alignment degradation across helpfulness, harmlessness, and honesty.The conclusion characterizes English-centric alignment as systematically failing in culturally adapted low-resource settings.

Limitations

The framework is limited by subsampling, a small annotation team, potential residual annotator or regional bias, and zero-shot evaluation scope.

  • Subsampling Alpaca and BeaverTails reduces statistical power and may miss subtle patterns visible at full scale.The full TruthfulQA set of 817 questions was used, while complete contextualization would increase annotation cost and effort.
  • Six undergraduate Pakistani annotators may leave residual subjectivity or regional biases in nuanced cultural, sectarian, or social adaptations.
  • Evaluations are zero-shot on contextualized datasets derived from popular source benchmarks, creating potential pre-training data leakage risk.The stated scope is dataset curation and evaluation design.

Ethics Statement

The work adapts English alignment benchmarks into culturally contextualized Urdu datasets while prioritizing human-guided validation and annotator welfare.

  • The study adapts existing English alignment benchmarks into culturally contextualized Urdu datasets for more equitable multilingual evaluation.
  • Human-guided localization and validation aim to reduce English-centric biases and semantic distortions, although annotator subjectivity or regional bias may persist.
  • Annotators handling potentially harmful or sensitive BeaverTails and TruthfulQA content received briefing, piloted the process, gave informed consent, and could opt out.Institutional mental health support resources were also provided.

Future Work

Future work will expand dataset coverage and study how culturally grounded annotations affect alignment, while the evaluation materials document annotation and comparison procedures.

  • Future Work: Future work plans to scale contextualization from subsampled data to full datasets to assess scalability, robustness, and performance across more examples.
  • Future Work: The expanded framework will support studying how culturally grounded safety annotations affect alignment quality and downstream model behavior.
  • Evaluation Breakdown: Table 7 reports English and Urdu win rates for full PakAlpaca and DT, LC, and LI subsets, with Δ representing the absolute percentage-point difference.
  • Annotation Procedure: Sample categorization disagreements trigger a second annotation round, with unresolved conflicts discarded to maintain annotation reliability and dataset quality.
  • Examples and Validation: The appendix supplies English prompts alongside culturally adapted Urdu variants for Alpaca, BeaverTails, and TruthfulQA, plus a qualitative matched English–Urdu Beaver-Dam-7B comparison.

F Evaluation Configuration and Parameters

The evaluation follows official or deterministic benchmark configurations and includes illustrative examples of annotation conflicts, prompt adaptation, and Beaver-Dam-7B safety comparisons.

  • Evaluation Parameters: Evaluation parameters follow default or deterministic settings from the official frameworks used for Alpaca, BeaverTails, and TruthfulQA.
  • Evaluation Parameters: Beaver-Dam-7B evaluates PakBeaverTails with a multi-label classifier derived from LLaMA-7B and no generative decoding parameters.
  • Annotation Examples: Figure 5 presents conflicted annotation cases that were re-annotated, while persistent conflicts were discarded.
  • Prompt Examples: Figures 6–8 compare English prompts with culturally adapted Urdu variants across categories for Alpaca, BeaverTails, and TruthfulQA.
  • Safety Evaluation Example: Figure 9 qualitatively compares Beaver-Dam-7B safety evaluations on matched English and Urdu prompt-response samples.
Loading 2608.30065v1…