Source-linked AI summary
Perspectives on Cross-Lingual Consistency in LLMs for Medical Questions
Minh Duc Bui, Mario Sanz-Guerrero, Abteen Ebrahimi, Sagi Shaier, Peter Herbert Kann, Manuel Mager, Katharina von der Wense
TL;DR
Multilingual medical NLP debates whether answers should stay consistent across languages or adapt to cultural contexts, but evidence about stakeholder preferences, user benefit, and item-level distinctions is limited. The paper synthesizes the literature, surveys stakeholders, and tests LLM personas; it finds divided preferences and that LLMs fail to reproduce them.
Problem
The literature treats cross-lingual variation either as error or as potentially appropriate adaptation, while lacking evidence on stakeholder preferences, user outcomes, and which items warrant each approach.
Source: §1 · pp. 1–2Method
The paper synthesizes two stances in multilingual medical NLP, surveys stakeholders, and evaluates whether LLM personas reproduce their perspectives.
Source: §6 · p. 9Results
Anthropologists favor adaptation, medical and NLP respondents are divided, and LLMs overestimate consistency preference among medical and NLP personas while failing to reproduce cross-national variation.
Source: §4.4 · p. 7, §6 · p. 9Takeaways & Limitations
The survey instrument cannot separate views about medical content from communication style, motivating follow-up studies that distinguish these dimensions.
Source: Limitations · p. 9The binary survey cannot determine whether respondents weighted medical content, communication style, or both.
Source: Limitations · p. 9
Abstract
from arXiv · showhide
Should multilingual LLMs answer medical questions consistently across input languages, or adapt responses to cultural cues? Existing multilingual medical benchmarks usually assume that medically correct answers should remain consistent across languages and treat cross-lingual variation as model error. In contrast, cultural adaptation research argues that appropriate medical answers may legitimately differ across contexts. We review the multilingual medical NLP literature through these two perspectives, we identify three gaps: limited stakeholder perspectives (e.g., of medical professionals), a lack of empirical evidence on which approach better serves users, and no benchmarks capable of distinguishing universally correct from culture-specific cases. To address the first gap, we survey 356 participants across three stakeholder groups (medical, NLP, and anthropology professionals) in three countries (Germany, Spain, and the United States). Anthropologists consistently favor adaptation, while medical and NLP respondents remain divided, with notable divergence between U.S. and European medical professionals. LLMs prompted with profession and country personas fail to reproduce this variation, overestimating cross-lingual consistency preference among NLP and medical personas. We conclude that neither consistency nor adaptation can currently be considered clearly preferable, highlighting the need for empirical evidence on which approach better serves users across cultural contexts.
1 Introduction
Multilingual medical NLP is divided over whether correct answers should remain invariant across languages or adapt to cultural context. The paper synthesizes these stances, identifies gaps, surveys stakeholders, and tests whether LLMs can represent their views.
1 Introduction
- Translated multilingual benchmarks often preserve the source-language gold answer and treat cross-lingual differences as model errors.This reflects the consistency assumption that correct answers are language-invariant.Source: §1 · p. 1
- Cultural adaptation research argues that appropriate medical answers may legitimately differ across contexts, and evaluations grounded in one setting can penalize accurate responses from another.Language and culture are described as entangled, though not interchangeable.Source: §1 · p. 1
- The literature synthesis identifies three gaps: limited stakeholder input, no evidence that either stance improves user outcomes, and no benchmark separating universally consistent from context-specific items.Examples include drug mechanisms and local guidelines.Source: §1 · p. 2
- The stakeholder survey finds anthropologists favor adaptation, while medical and NLP respondents are divided; LLM personas overestimate consistency preferences among medical and NLP stakeholders.The survey also finds cross-national variation among medical respondents.Source: §1 · p. 2
2 Background
Language is commonly used as a proxy for culture in multilingual NLP, although it is not a reliable one. Medical guidelines also vary across regions, making a single correct answer less assured.
2.1 Language vs. Culture
- Language and culture are intertwined, but language is rarely a reliable proxy for culture.The same language may be used across distinct cultural contexts.Source: §2.1 · p. 2
- When richer cultural context is absent, multilingual NLP often defaults to language as its proxy because language is always present.Healthcare systems, local guidelines, and belief systems require explicit specification that is frequently missing.Source: §2.1 · p. 2
2.2 Cultural Differences in Medical Practices
- Despite a shared evidence base, medical guidelines are not generally identical worldwide.US and German-speaking osteoporosis guidelines both target fracture risk but use different risk-assessment tools, with the latter incorporating roughly 50 additional variables.Source: §2.2 · p. 2
- Guideline differences reflect sociocultural, socioeconomic, and preference-related factors as well as scientific evidence, complicating the idea of one correct answer.The passage presents these differences as a challenge for multilingual NLP systems.Source: §2.2 · pp. 2–3
3 Two Stances on Cross-Lingual Answer Consistency in Medical NLP
The literature contrasts a consistency stance, which treats medical answers as language-invariant, with adaptation approaches that account for cultural context. Its synthesis identifies unresolved questions about stakeholders, user outcomes, and which medical items should remain consistent or adapt.
3.1 The Consistency Stance
- Translated benchmarks and cross-language evaluations commonly treat a source-language answer as the reference and deviations as errors.MedExpQA and CasiMedicos retain Spanish physicians’ answers across languages [Alonso et al., 2024; Goenaga et al., 2024], while XLingHealth frames consistency with English answers as a safety criterion [Jin et al., 2024].Source: §3.1 · p. 3
- Consistency is often assumed rather than established, although cultural differences in health systems and treatment expectations make output changes ambiguous as evidence of bias.The best answer, if any single answer exists, remains an open question.Source: §3.1 · p. 4
3.2 The Adaptation Stance
- Adaptation research spans native-resource construction and claims that medical content itself can differ across settings.Examples include locally grounded materials [Qiu et al., 2024; Restrepo et al., 2025] and benchmarks reflecting Traditional Chinese Medicine [Wang et al., 2024a].Source: §3.2 · p. 4
- Adaptation also concerns communication: effective delivery may vary even when medical content stays consistent.Participatory workshops argue that culturally appropriate health AI should account for relational context and material constraints, not only language or surface conventions [Peters et al., 2025].Source: §3.2 · p. 5
3.3 What the Literature Does Not Tell Us
- Stakeholder preferences are largely assumed, and no prior work jointly surveys medical professionals and NLP researchers across multiple regions on cross-lingual answer consistency.The paper notes that such choices involve value judgments about whose medical norms should prevail.Source: §3.3 · p. 5
- Neither stance has been tested for downstream user benefit, and the literature lacks evidence about which specific items should be consistent or adapted.The unresolved distinction includes drug mechanisms versus regionally varying screening guidelines.Source: §3.3 · p. 5
4 Surveying Stakeholders
This section tests whether the consistency–adaptation debate extends to professionals in medicine, NLP, and anthropology through a survey spanning three countries.
4.1 Target Population
- The survey targets medical, NLP, and anthropology professionals in Germany, Spain, and the US, with countries chosen to span distinct healthcare systems.Respondents could complete the questionnaire in English, Spanish, or German.Source: §4.1 · p. 6
4.2 Recruitment
- Recruitment combined professional networks and mailing lists with Prolific, yielding a final sample of 348 respondents.Prolific participants underwent profession-based screening and attention checks.Source: §4.2 · p. 6
4.3 Survey
- Respondents chose between language-invariant answers grounded in scientific knowledge and answers reflecting medical approaches within the language’s cultural framework.The binary format operationalized the two stances from the literature; whether either stance applies to individual items remains an open question.Source: §4.3 · p. 6
4.4 Results and Discussion
- 65% of German medical respondents preferred consistency; 5 of 6 profession–country groups did not differ significantly from chance.The German medical group was the only exception to the binomial-test result against the 50% baseline.Source: Figure 2 · p. 6, §4.4 · p. 7
- Anthropologists favored adaptation in all three countries, with support never below 75%; each result differed significantly from chance.The paper relates this stable preference to anthropologists’ attention to cultural variation and suspicion of universalizing claims.Source: Figure 2 · p. 6, §4.4 · p. 7
- 57% of US medical respondents favored adaptation, versus 35% in Germany and 47% in Spain; the US–Germany difference was significant.German and Spanish medical respondents tended toward consistency, unlike their US counterparts.Source: Figure 2 · p. 6, §4.4 · p. 7
- US PubMed publications had the highest share of culture-related articles, followed by Spain and Germany, mirroring the survey ranking.The comparison covers publications from 2015–2025 and medical professionals’ support for adaptation.Source: §4.4 · p. 7, Figure 3 · p. 7
4.5 Implications of the Survey Findings
- The authors conclude neither stance has clear consensus and call for user-centered investigation; the sample covers only three Western, Global-North systems.They note a significant US–Germany difference and suggest broader geographic sampling would extend the evidence.Source: §4.5 · p. 7
5 LLM Simulation of Survey Opinions
The paper tests whether LLMs prompted with stakeholder personas can represent human opinions about consistency in multilingual medical answers, then considers implications for system design.
5.1 Experimental Setup
- The simulation asks whether LLMs reproduce survey opinions by presenting the same question under profession and country personas.The survey is posed in English, German, or Spanish; model responses are averaged across 20 sampling-based runs per condition.Source: §5 · pp. 7–8, §5.1 · p. 8
5.2 Results
- Human medical respondents vary by country, while LLM consistency preferences shift by prompt language, with English highest and German lower.English prompts yield consistency rates of 0.80 and 0.84 for NLP and medical personas; German prompts yield 0.67 and 0.69.Source: Figure 4 · p. 8, §5.2 · pp. 8–9
- LLMs select consistency more often than human respondents under NLP and medical personas, matching human opinions in only one of three cases.The consistency proportions are 80.0% and 83.6% for LLMs, versus 61.2% and 53.7% for humans, respectively.Source: §5.2 · p. 8
5.3 Practical Implications
- Persona-prompted LLMs cannot substitute for stakeholder surveys when making design choices, and system-prompt steering alone may not reproduce stakeholder variation.The paper identifies cross-lingual alignment on parallel medical QA as a consistency route and regional QA plus country-specific guideline retrieval as an adaptation route.Source: §5.3 · p. 9
6 Conclusion
The paper synthesizes competing views on cross-lingual consistency and adaptation, surveys stakeholders, and tests whether LLMs reproduce their opinions.
6 Conclusion
- The literature review identifies gaps in stakeholder involvement, user-outcome evidence, and benchmarks distinguishing invariant from culturally adaptable questions.The survey finds anthropologists favor adaptation while medical and NLP respondents are divided; LLMs fail to reproduce stakeholder distributions and cross-country variation.Source: §6 · p. 9
Limitations
The survey was designed to detect systematic stakeholder agreement or disagreement, and its conclusions have several stated limits.
Limitations
- The binary survey question cannot reveal whether respondents based their choices on medical content, communication style, or both.The authors recommend follow-up work that separates these dimensions.Source: Limitations · p. 9
- Profession-by-country group sizes surface reported patterns but do not support fine-grained demographic comparisons.Source: Limitations · p. 9
- The sample covers only the Global North, leaving uncertain whether the findings hold in low- and middle-income countries.Source: Limitations · p. 9
Ethical Statement
The paper reports ethical safeguards and risks, then describes its crowdworker screening, survey form, and limited demographic collection.
Ethical Statement
- The project received IRB approval, obtained informed consent, paid crowdworkers $16.60 per hour, and collected no personally identifiable information.The authors also acknowledge potential misuse of findings to justify medically harmful outputs under the guise of adaptation.Source: Ethical Statement · p. 9
A.1 Filtering Crowdworkers
- Crowdworker eligibility was checked through platform prescreening, survey-level domain confirmation, and profession-specific verification for NLP researchers and anthropologists.Anthropologist responses were manually reviewed; NLP researchers completed a fill-in-the-blank prompt.Source: §A.1 · pp. 12–13
A.2 Survey Form
- All participants saw the main survey question, and native speakers from each country verified the translated survey versions.Source: Figure 5 · p. 13, §A.2 · p. 13
A.3 Demographic Details
- The survey collected demographics related only to profession.Source: §A.3 · p. 13
A.4 Free-Text Responses
Free-text comments most often distinguished stable medical content from communication that may adapt to readers. The PubMed analysis selected culture-related publications with authors affiliated in one country.
A.4 Free-Text Responses
- Of 348 respondents, 60 commented and 44 comments were relevant; the authors grouped them inductively into six recurring themes.Because responses were few and some touched on multiple themes, the authors treat theme counts as indicative rather than exact.Source: §A.4 · p. 13
- The most frequent comment theme held that medical content should stay consistent across languages while communication may adapt to the reader.The comments echo the content–communication distinction discussed in Section 3.2.Source: §A.4 · p. 13
B PubMed Crawling
- The PubMed corpus covered 2015–2025 publications whose authors were all affiliated with institutions in the same country.The authors selected culture-related MeSH terms to form the culturally grounded subset.Source: B PubMed Crawling · p. 13
- Figure 6 provides detailed results for PubMed publications, while Figure 3 presents the comparison with the survey.The reported comparison is to the survey; the detailed results are in Figure 6.Source: B PubMed Crawling · p. 13, Figure 6 · p. 13
C Detailed Model Experiments
The detailed-experiments materials identify the prompt setup and the scope of three tables covering participant demographics, comment themes, and cross-language consistency results.
C Detailed Model Experiments
- The experiments ran on two H100 GPUs, and one inference pass took about two minutes for the largest models.Source: §C.1 · p. 13
- Table 3 presents participant demographics by culture and job role, including degree counts and graduation-year statistics.Its labels define N as count, SD as standard deviation, Min/Max as range, and abbreviate degree categories.Source: Table 3 · p. 14
- Table 4 lists themes in the free-text comments, assigning each comment to its primary theme.Source: Table 4 · p. 14
- Table 5 presents consistency and adaptation results for the no-persona condition across languages; its caption does not provide individual language names or numerical scores.The full prompt is provided in Figure 7.Source: §C.1 · p. 13, Table 5 · p. 14
C.2 Detailed Results
Without personas, model responses show substantial variance and only weak consensus, with a slight overall preference for consistency. The detailed tables cover results across languages, countries, and professions.
C.2 Detailed Results
- Without personas, responses show substantial variance and weak consensus, with a slight overall preference for consistency.Figure 8 presents the proportion selecting consistency, averaged across models.Source: §C.2 · p. 14, Figure 8 · p. 14
- The survey prompt was presented in English, German, and Spanish, with response-option order randomized for each prompt instance.Source: Figure 7 · p. 14
- Table 6 reports consistency-versus-adaptation results across languages, countries, and professions.Detailed results with personas are reported in Table 6.Source: §C.2 · p. 14, Table 6 · p. 15