PaperlayerView exact revision on arXiv

Source-linked AI summary

Measuring Cultural Alignment Beyond the Average: A Framework for Evaluating Maternal-Health LLM Interactions in Indian Contexts

Umaira Izhar, Gunjan Arora, Pushpendra Singh

arXiv:2610.11586v1cs.CL

TL;DR

Healthcare-LLM evaluations offer limited insight into culturally situated maternal-health reasoning. MH-INDIC uses survey-grounded dimensions to assess population alignment and profile variation, then examines culturally conditioned dialogue generation. Models can approximate human averages while preserving less profile variation; human-grounded conditioning improves dialogue alignment and quality ratings.

  • Problem

    Healthcare-LLM evaluations emphasize factual correctness, safety, and fluency but provide limited insight into culturally situated maternal-health reasoning shaped by social and relational norms.

    Source: §1 · p. 1

  • Method

    MH-INDIC operationalises maternal-health reasoning through ten culturally grounded dimensions and links survey-based model evaluation with downstream dialogue generation.

    Source: §1 · p. 2

  • Results

    Models can approximate human cultural averages while failing to preserve meaningful behavioural variation across profiles, and human-grounded conditioning improves downstream profile alignment and dialogue quality.

    Source: §8 · p. 9

  • Takeaways & Limitations

    Cultural evaluation of maternal-health LLMs should distinguish average alignment from profile-level sensitivity and assess downstream dialogue grounding.

    Source: §1 · p. 2, §8 · p. 9

    The cohort is predominantly urban and graduate-educated, so findings may not generalise to rural or lower-literacy populations; MH-INDIC is intended for research, not clinical decision-making.

    Source: §9 · p. 9

Abstract

from arXiv · show

Existing evaluation methods for healthcare LLMs primarily assess factual correctness,safety, and fluency, while providing limited insight into whether generated interactions reflect culturally situated healthcare reasoning. This limitation is particularly important in maternal health, where care decisions are shaped by social and relational norms. We introduce MH-INDIC, a culturally grounded evaluation framework for maternal-health interactions in urban and semi-urban North Indian contexts that operationalises cultural behaviour through ten dimensions of maternal-health reasoning. Using a 26-item survey administered to 102 pregnant and postpartum women from urban and semi-urban North India, we evaluate ten LLMs. We distinguish population level cultural alignment from profile-level behavioural variation. Although several models approximate the human population-level distribution, all evaluated systems exhibit substantially lower variation across demographic and household profiles than the human cohort, revealing a gap between aggregate alignment and profile-conditioned sensitivity. As a downstream application of MH-INDIC, we use the strongest-aligned proprietary and open-source models to generate culturally conditioned maternal-health dialogues under zero-shot, self-conditioned, and human-grounded prompting. Human-grounded conditioning produces stronger profile alignment and dialogue quality ratings, suggesting that measured cultural profiles can improve the cultural grounding of generated interactions

1 Introduction

Maternal-health decisions are shaped by relational and social norms that existing healthcare-LLM evaluations capture only partially. MH-INDIC evaluates culturally situated maternal-health reasoning and connects survey-based alignment to dialogue generation.

1 Introduction

  • Existing healthcare-LLM evaluations emphasize accuracy, safety, fluency, and broad cultural appropriateness, but offer limited insight into socially situated maternal-health reasoning.Maternal-health decisions involve spouses, elders, community health workers, and informal knowledge sources, as well as norms around authority, uncertainty, emotional disclosure, and self-sacrifice.Source: §1 · pp. 1–2
  • MH-INDIC extends Hofstede’s framework with maternal-health-specific relational and social dimensions, covering ten forms of maternal-health reasoning in urban and semi-urban North India.The dimensions include authority, gendered caregiving, uncertainty, self-care, agency, care beliefs, bodily disclosure, and mental-health suppression.Source: §1 · p. 2
  • The study grounds its evaluation in a 26-item expert-consulted survey of 102 women and compares ten LLMs using matched profiles and shared scoring.The evaluated models span proprietary, open-source, and medical-domain families; the survey responses establish the human reference.Source: §1 · p. 2
  • Strong aggregate alignment does not necessarily imply human-like profile sensitivity, while stronger alignment improves the realism of downstream synthetic maternal-health dialogues.The framework connects survey-level evaluation with dialogue generation under self-conditioned and human-grounded prompting, assessed for cultural plausibility, affective realism, and demographic consistency.Source: §1 · p. 2

2 Related Work

Prior work has addressed maternal-health AI, cultural modeling, and dialogue generation, but often without capturing relational context or connecting evaluation to generation. MH-INDIC brings these strands together through empirically defined dimensions and human–model comparison.

2 Related Work

  • Maternal-health AI has focused largely on biomedical or individual factors, with limited representation of spousal authority, elder roles, and community norms.Culturally sensitive maternal-health chatbot work is emerging, but it does not yet provide a generalisable framework for evaluating culturally situated reasoning at scale.Source: §2 · p. 3
  • Cultural NLP benchmarks often use abstract values and population averages, leaving individual and household variation and domain-specific decision pressures less examined.Dialogue research has explored personalization and value alignment, but cultural grounding is often surface-level and rarely evaluated across both reasoning and generation tasks.Source: §2 · p. 3
  • MH-INDIC links prior strands by operationalising maternal-health behaviour with empirical dimensions, comparing human and model behaviour, and connecting survey alignment to dialogue generation.The framework is designed to assess culturally situated reasoning rather than only superficially appropriate responses.Source: §2 · p. 3

3 MH-INDIC

MH-INDIC adapts Hofstede’s dimensional approach to maternal-health interactions, adding domain-specific dimensions to represent socially situated decision-making among mothers, clinicians, families, and communities.

3 MH-INDIC

  • Hofstede’s dimensions serve as an interpretable scaffold, but the framework reinterprets them at the maternal-health interaction level rather than using national cultural scores.Four additional dimensions are introduced from maternal-care and public-health literature, producing a ten-dimensional framework.Source: §3 · p. 3
  • The adapted dimensions describe authority, uncertainty, future health priorities, self-care, gender norms, and tensions between community norms and individual maternal decisions.Power Distance Index represents medical and institutional authority; uncertainty avoidance represents comfort with ambiguity and unfamiliar procedures; LTO addresses future maternal and child health.Source: §3 · pp. 3–4
  • The four added dimensions capture maternal autonomy versus family control, biomedical versus traditional care, bodily disclosure, and recognition or expression of emotional distress.Together, the dimensions form the conceptual backbone for evaluating culturally grounded maternal-health behaviour.Source: §3 · p. 4

4 Methodology

The methodology uses matched human and LLM surveys to score cultural alignment, distinguish aggregate agreement from profile-conditioned variation, and generate culturally conditioned dialogues. The pipeline proceeds from survey design through evaluation to dialogue assessment.

4 Methodology

  • The four-stage pipeline comprises survey design, matched administration to humans and LLMs, behavioural indicator construction and scoring, and culturally grounded dialogue generation.Human responses provide the reference for evaluating alignment and selecting a model for downstream generation.Source: §4 · p. 4, Figure 2 · p. 5

4.1 Survey Design and Administration

  • The survey maps Likert-scale, multiple-choice, and open-ended items to ten dimensions, with direct and social-contextual indicators refined by two domain experts.Structural analyses assess interpretability rather than requiring high internal consistency for behaviourally heterogeneous, formative dimensions; the English survey was administered to 102 women and ten LLMs under matched persona conditioning.Source: §4.1 · p. 4

4.2 Behavioural Indicator Construction and Scoring

  • Responses are converted into dimension-level representations using predefined coding, normalized ordinal scores, and aggregation rules designed to preserve relational dynamics.MAFC is decomposed into spouse- and elder-mediated authority, while MHS separates personal disclosure from community recognition; identical scoring is applied to human and model responses.Source: §4.2 · p. 4

4.3 Aggregate and Profile-Conditioned Alignment

  • Aggregate alignment measures closeness to population-level human tendencies, whereas profile-conditioned sensitivity measures whether models preserve variation across profiles.Model response standard deviations across profile-conditioned evaluations are compared with analogous human subgroup variation; lower aggregate-distance values indicate stronger alignment with the human reference.Source: §4.3 · p. 5

4.4 Culturally Grounded Dialogue Generation

  • The strongest-aligned open-source and proprietary models generate dialogues under zero-shot, self-conditioned, and human-grounded prompting.Each dialogue targets PDI, with MAFC and GN held as secondary cues; paired 20-turn conversations are assessed by blinded experts.Source: §4.4 · pp. 5–6

5 Experiments

The experiments compare ten LLMs with a human maternal-health reference and test whether cultural conditioning shapes generated dialogues. They assess alignment across cultural dimensions and rate dialogue alignment and quality.

5.1 Human-LLM cultural alignment:

  • Ten LLMs across proprietary, open-source, medical-domain, and Hindi-adapted families are evaluated on cohort-derived maternal profiles, with results generally reported at T = 0.3.Thirty profiles are selected across seven demographic attributes; responses are generated greedily at T = 0.0 or by majority voting over five samples at T = 0.3 and T = 0.7.Source: §5.1 · p. 6

5.2 Dialogue Generation.

  • Claude Sonnet 4.6 and Qwen3 30B generate dialogues for three real cohort profiles and two synthetic profiles under different conditioning strategies.The profiles include autonomous, midrange, conflicting, fully autonomous, and fully constrained orientations; blinded experts assess cultural plausibility, affective realism, and demographic consistency.Source: §5.2 · p. 6

5.3 Evaluation Metrics

  • Cultural alignment is measured against the human cohort, while dialogue ratings cover ten cultural dimensions and three conversation-quality measures.Dialogue evaluation uses a 1–5 Likert scale; PAS compares rated orientation dimensions with profile targets, and two domain experts and two LLM judges rate each dialogue independently.Source: §5.3 · pp. 6–7

6 Results and Analysis

The results distinguish models’ fit to population averages from their ability to preserve profile-level variation. They also compare dialogue conditioning strategies and examine expert–LLM-judge agreement.

6.1 Aggregate Cultural Alignment

  • At T = 0.3, Claude Sonnet 4.6 has the lowest Euclidean distance (0.253) and MAE (.061), and the highest cosine similarity (.968) and Spearman correlation (.527).Its T = 0.0 Euclidean distance is also lower than GPT-5’s (0.262 vs. 0.345).Source: §6.1 · p. 7
  • Humans show conditional sensitivity of 0.199 across ten dimensions, compared with 0.119 for strongest-aligned Claude Sonnet 4.6.All systems show substantially weaker profile-conditioned sensitivity than the human cohort despite several proprietary models approximating population-level averages.Source: §6.1 · p. 7

6.2 Dimension-Level Behavioural Deviations

  • Models’ largest deviations involve household authority, gender roles, and self-sacrifice, while BTB, LTO, and PDI remain comparatively stable across models.MAFC, GN, and IVR overstate the constrained pole by up to +0.41, +0.41, and +0.50, respectively; MHS understates distress suppression by up to −0.32.Source: §6.2 · p. 7
  • Models also misrepresent relational and emotional distinctions: they compress or over-separate elder- and spouse-mediated authority and struggle to distinguish experienced distress from visible expression.The human ED:SC ratio is approximately 2.3, compared with 1.5–2.0 for open-source and medical-domain models and 5.5 for Claude and 20.7 for Gemini 3 Pro.Source: §6.2 · pp. 7–8

6.3 Dialogue Generation and Cultural Grounding

  • Human-grounded prompting produces the highest PAS across both models and evaluation panels, while zero-shot generation has the weakest expert-rated PAS.Expert-rated dialogue quality follows the same regime ordering; Claude generally scores higher on PAS, naturalness, and overall quality than Qwen3, which is slightly better on profile consistency.Source: §6.3 · p. 8
  • Human-grounded conditioning raises expert-rated PAS over zero-shot by +0.31 for Claude and +0.21 for Qwen3, but the LLM-judge effect is weaker, particularly for Claude.In conflicting profile P18, Claude preserves clinic deference and household autonomy under human-grounded conditioning, whereas Qwen3 shifts toward the constrained pole.Source: §6.3 · p. 8

6.4 Expert Assessment

  • Expert–LLM-judge agreement is substantial overall, but judges can over-anchor on prompts and miss distinctions in under-rendered or culturally conflicting dialogues.The authors conclude that LLM-as-a-judge provides useful aggregate signals but cannot substitute for expert review in those cases.Source: §6.4 · p. 8

7 Discussion

Across the North Indian cohort, models reproduce population-level averages more readily than profile-specific behavioural variation. The discussion connects this gap to demographic priors and limits in relational inference.

7 Discussion

  • The marginal-conditional gap across evaluated systems is consistent with models relying on population-level priors that reproduce human averages without capturing rules distinguishing individual women.The authors note that factors beyond the demographic and household attributes used to define profiles may also partly account for the gap.Source: §7 · p. 9
  • The largest gaps involve household authority, gender norms, self-sacrifice, and mental-health suppression, which require relational inference that current models flatten into a demographic template.The authors warn that generated dialogues may appear to represent cultural profiles while reflecting only a narrow prior.Source: §7 · p. 9
  • The discussion argues that marginal alignment should be reported alongside profile-conditioned sensitivity and that domain adaptation does not necessarily transfer to cultural reasoning.It also says LLM-as-a-judge evaluation requires expert calibration on culturally conflicting or under-rendered dialogues.Source: §7 · p. 9

8 Conclusion

The conclusion presents MH-INDIC as a framework for maternal-health reasoning in urban and semi-urban North Indian contexts. Its findings contrast alignment to cultural averages with the preservation of profile-specific variation, and report stronger downstream results under human-grounded conditioning.

8 Conclusion

  • MH-INDIC evaluates maternal-health reasoning in urban and semi-urban North Indian contexts.Across survey-level alignment and dialogue generation, models approximate cultural averages but fail to preserve meaningful variation across demographic and household profiles.Source: §8 · p. 9
  • The largest failures involve socially situated behaviours around family negotiation, emotional suppression, and culturally mediated care decisions.The conclusion reports that human-grounded conditioning substantially improves downstream profile alignment and dialogue quality.Source: §8 · p. 9
  • Unaddressed cultural misalignment could yield interactions or recommendations that do not reflect household authority, emotional context, or care-seeking realities.The authors say this could limit usefulness for the populations the systems aim to serve.Source: §8 · p. 9

9 Ethics and Limitations

The section situates findings within the study’s North Indian cohort and research-only scope, describes the authors’ positionality, and explains how cohort-derived and synthetic profiles structure the experiments.

9 Ethics and Limitations

  • Findings may not generalize beyond a predominantly urban, graduate-educated cohort to rural, lower-literacy, or other underrepresented populations.MH-INDIC omits financial constraints, healthcare accessibility, and infrastructure disparities; dialogue evaluation covered three dimensions in 22 dialogues, and aligned generation may reinforce restrictive norms.Source: §9 · p. 9
  • The survey was voluntary, IRB-approved, anonymized, and intended for research evaluation rather than clinical decision-making.Source: §9 · p. 9

A Positionality

  • The authors describe their Indian origins, regional familiarity, and computer science, HCI, and mental-health expertise as informing MH-INDIC’s design.Source: A Positionality · p. 11

B Maternal Profile Statistics

  • Thirty cohort-derived profiles were selected by farthest-first traversal across seven demographic fields, starting from the rarest configuration and retaining respondent links for human-grounded dialogue prompts.Source: B Maternal Profile Statistics · p. 11
  • 0.199 human conditional sensitivity versus 0.119 for Claude Sonnet 4.6 at T = 0.3; all LLMs fall below the human reference.Source: Figure 4 · p. 11
  • Five dialogue profiles span cultural postures, including conflicting, mixed, and autonomous cohort cases plus synthetic controls at both extremes.The synthetic controls fix every dimension at 0.0 or 1.0 and share P18’s demographic frame.Source: B Maternal Profile Statistics · p. 11

C Psychometric Analysis

This section describes MH-INDIC’s dimension patterns and scoring conventions, alongside the demographic profiles and response-format instructions used to elicit culturally situated dialogues.

C Psychometric Analysis

  • Cronbach’s α ranged from 0.19 to 0.63 across seven multi-item dimensions, and no inter-dimension correlation exceeded |r| = 0.41.Most associations were weak (|r| < 0.2); the strongest links were IVR-PDI (r = 0.41), MAFC-GN (r = 0.39), and MAFC-LTO (r = 0.37). BTB and CNID, and the limited-item SVO dimension, are treated as exploratory and interpreted cautiously.Source: C Psychometric Analysis · p. 12, Figure 5 · p. 13

D Scoring Criteria

  • Responses are encoded and normalized to [0, 1], reverse-coded so higher scores indicate the constrained or restrictive pole, and aggregated using fixed, dimension-specific rules.Missing or non-applicable indicators are excluded, and the scoring procedures are applied identically to human and model-generated responses without post-hoc tuning.Source: D Scoring Criteria · pp. 13–14
  • MHS combines personal disclosure and community recognition sub-dimensions, with higher values indicating greater suppression.The composite used in the main alignment analysis is the mean of available sub-dimensions; the decomposition supports the analysis in Figure 6.Source: D Scoring Criteria · p. 13

E Dialogue Generation Prompts

  • The dialogue prompts condition a woman’s behavior on her profile and conversational cues, eliciting family dynamics without presuming control, conflict, or independence.The client prompt requires consistent, natural behavior across the conversation, while the assistant prompt adapts support to the woman’s revealed social reality and avoids pushing confrontation or autonomy.Source: E Dialogue Generation Prompts · pp. 14–16
  • The prompt profile describes a self-employed Hindu woman aged 31–40 in Mumbai, graduate-educated, living with her husband, in her first pregnancy, with household income of 5–30 lakh.The profile also includes a child age of 20 months.Source: E Dialogue Generation Prompts · p. 16
  • The prompt presents answer choices as option placeholders, from {option_1} through {option_2} and onward.Source: E Dialogue Generation Prompts · p. 16
  • Response instructions vary by question type and require exact option text without explanation, a comma-separated list for multiple choices, or a single digit.Source: E Dialogue Generation Prompts · p. 16

F Temperature Robustness

The model ranking and separation among four model families remain stable across the tested temperatures.

F Temperature Robustness

  • Claude Sonnet 4.6 ranks best on Euclidean distance, MAE, cosine similarity, and Spearman ρ at all three temperatures, with model families remaining separated.The four families show no cross-family overlap.Source: F Temperature Robustness · p. 16

G Cohort Demographics

The cohort provides an urban and semi-urban North Indian maternal reference population, and the survey operationalizes ten dimensions through mixed item formats and a shared scoring procedure.

G Cohort Demographics

  • 98 participants (96%) were aged 21–40, 50 were Hindu and 49 Muslim, and 98 were graduate-educated.The household split was 54 husband-only households and 47 joint or extended households including in-laws.Source: G Cohort Demographics · p. 17
  • The empirical reference is urban, predominantly graduate-educated, Hindu/Muslim, and North Indian; less-educated, rural, and religious-minority populations require further validation.Most respondents had prior pregnancy experience (N = 73), and Delhi (N = 20) and Mumbai (N = 12) were the most represented locations.Source: G Cohort Demographics · p. 17

H Survey Instrument

  • Survey items use Likert, multiple-choice, and free-text formats mapped to the ten MH-INDIC dimensions.The instrument covers questions 1–11, with the item-to-dimension mapping in Appendix D.Source: H Survey Instrument · p. 17, Figure 9 · p. 20
  • The scoring pipeline codes and normalizes related responses, averages them, then inverts scores to align higher values with the constrained or restrictive pole.The same procedure is applied to human and LLM-generated responses.Source: Figure 7 · p. 18
Loading 2610.11586v1…