Source-linked AI summary
Better to Ask in English: Cross-Lingual Evaluation of Large Language Models for Healthcare Queries
Yiqiao Jin, Mohit Chandra, Gaurav Verma, Yibo Hu, Munmun De Choudhury, Srijan Kumar
TL;DR
The paper investigates whether healthcare LLMs provide equally safe and useful responses across languages, addressing limited evidence about multilingual human–LLM interactions. It proposes XLingEval and XLingHealth, and finds pronounced disparities between English and non-English performance across evaluation criteria. These findings highlight equity concerns for multilingual healthcare information, although the study relies partly on a smaller healthcare model and machine-translated data.
Problem
The paper addresses limited evidence about the quality and safety of multilingual human–LLM healthcare interactions amid English-centric model development.
Method
The paper proposes XLingEval, evaluating correctness, consistency, and verifiability across four languages and three healthcare datasets, alongside the XLingHealth benchmark.
Results
LLM responses showed pronounced language disparities across correctness, consistency, and verifiability, with non-English performance generally lower than English performance.
Takeaways & Limitations
The findings raise questions about whether multilingual healthcare accessibility claims adequately account for linguistic equity.
Takeaways & Limitations
The study used a smaller healthcare-focused model for comparison and machine-translated multilingual datasets because suitable multilingual resources were unavailable.
Abstract
from arXiv · showhide
Large language models (LLMs) are transforming the ways the general public accesses and consumes information. Their influence is particularly pronounced in pivotal sectors like healthcare, where lay individuals are increasingly appropriating LLMs as conversational agents for everyday queries. While LLMs demonstrate impressive language understanding and generation proficiencies, concerns regarding their safety remain paramount in these high-stake domains. Moreover, the development of LLMs is disproportionately focused on English. It remains unclear how these LLMs perform in the context of non-English languages, a gap that is critical for ensuring equity in the real-world use of these systems.This paper provides a framework to investigate the effectiveness of LLMs as multi-lingual dialogue systems for healthcare queries. Our empirically-derived framework XlingEval focuses on three fundamental criteria for evaluating LLM responses to naturalistic human-authored health-related questions: correctness, consistency, and verifiability. Through extensive experiments on four major global languages, including English, Spanish, Chinese, and Hindi, spanning three expert-annotated large health Q&A datasets, and through an amalgamation of algorithmic and human-evaluation strategies, we found a pronounced disparity in LLM responses across these languages, indicating a need for enhanced cross-lingual capabilities. We further propose XlingHealth, a cross-lingual benchmark for examining the multilingual capabilities of LLMs in the healthcare context. Our findings underscore the pressing need to bolster the cross-lingual capacities of these models, and to provide an equitable information ecosystem accessible to all.
1 INTRODUCTION
The paper addresses safety and equity gaps in multilingual healthcare dialogue by proposing XLingEval, which evaluates correctness, consistency, and verifiability across languages. Experiments reveal substantial disparities between English and non-English responses, motivating stronger cross-lingual capabilities.
- LLM healthcare use raises safety concerns because incorrect or incomplete information can have life-threatening consequences.
- XLingEval evaluates healthcare dialogue responses for correctness, consistency, and verifiability across English, Hindi, Chinese, and Spanish.
- 18.12% fewer GPT-3.5 answers were more comprehensive and appropriate in non-English languages than in English across three datasets.
- GPT-3.5 was 5.82 times more likely to produce incorrect responses in non-English languages than in English.
- Consistency declined by up to 9.1% in Spanish, 28.3% in Chinese, and 50.5% in Hindi relative to English, while verifiability fell by 14.6% in Chinese and 23.4% in Hindi.
- XLingHealth complements the framework as a cross-lingual healthcare benchmark, while the framework is intended to support evaluation by policymakers, practitioners, and healthcare professionals.
2 THE XLINGHEALTH BENCHMARK
XLingHealth is a cross-lingual benchmark for clinical healthcare questions built from expert-curated datasets representing general-public health inquiries. Its English-origin data and machine-translated multilingual versions shape the benchmark’s scope.
- XLingHealth is a cross-lingual benchmark for clinical health inquiry built from three prominent healthcare question-and-answer datasets.
- The benchmark uses HealthQA, LiveQA, and MedicationQA, covering general health, NIH-related FAQs, and drug-related consumer concerns.
- The datasets target questions commonly asked by the general public and use answers provided by medical professionals.
- The datasets were originally English-language, creating a multilingual-data constraint for cross-lingual healthcare evaluation.
3 CORRECTNESS
The correctness evaluation compares LLM answers with expert-curated ground truth using automated classification and human validation. Results show pronounced language disparities, with Hindi exhibiting the largest discrepancy, followed by Chinese and Spanish.
- Correctness combines contradiction with comprehensiveness and appropriateness, evaluated through automated comparisons against ground truth and human validation.
- 38.62% fewer Hindi, 11.90% fewer Chinese, and 10.76% fewer Spanish HealthQA answers were more comprehensive and appropriate than English answers.
- GPT-3.5 produced 3 contradictory HealthQA answers in English versus 47 in Hindi, 14 in Chinese, and 5 in Spanish.
- MedAlpaca showed similar English–non-English disparity, alongside many answers that were neither contradictory nor similar to ground truth because it could not produce multilingual text effectively.
- Language disparity appeared across all four automated correctness labels, with Hindi showing the most prominent discrepancy, followed by Chinese and Spanish.
- Human validation found 94.20% average correlation for English and 95.14% for Spanish between GPT-3.5 labels and majority human labels.
4 CONSISTENCY
The consistency protocol measures whether repeated LLM answers remain similar at lexical, semantic, and topic levels across languages and temperature settings. GPT-3.5 was generally more consistent in English, with disparities especially pronounced for Chinese and Hindi.
- Consistency Protocol: The protocol prompts each question ten times in English and a non-English language, then compares responses across surface, semantic, and topic dimensions.Temperature τ controls generation randomness, while lexical, embedding-based, and topic-model metrics quantify similarity.
- Consistency Metrics: Surface-level consistency uses n-gram overlap and response length, while semantic consistency uses BERTScore and sentence-embedding similarity.N-gram similarity is a Jaccard similarity over the n-grams in two answers.
- Topic Consistency: Topic consistency is estimated with LDA and HDP, with HDP automatically inferring the number of topics from data complexity and volume.LDA requires a predefined topic count and can produce closely aligned or duplicated topics when that count is large.
- Numerical Results: GPT-3.5 showed higher consistency in English than other languages across sim2-gram, BERTScore, and simsent metrics.For BERTScore, English scores were 0.9206 / 0.6160 / 0.5299 across τ=0.0 / 0.5 / 1.0, versus 0.8454 / 0.5536 / 0.4860 for Chinese.
- Numerical Results: Significant consistency disparities appeared across language pairs except English-Spanish on LiveQA at τ=0.0.The analysis used Tukey’s HSD tests with 95% confidence intervals and mean differences.
5 VERIFIABILITY
The verifiability evaluation tests whether LLMs can distinguish correct from erroneous healthcare answers across languages. GPT-3.5 performed better in English and Spanish than in Chinese and Hindi, while temperature changes had little effect.
- Evaluation Protocol: The protocol gives the model question-answer pairs in English or a non-English language and asks it to assign a binary correctness label.Predictions are compared with ground truth across accurate and inaccurate claims.
- Evaluation Setup: Dataset-specific negative examples were used, and evaluation reported macro-precision, macro-recall, macro F1-score, accuracy, and AUC.HealthQA supplies one positive and nine negative answers per question, while LiveQA and MedicationQA use four randomly sampled negatives.
- Results: At τ=1.0 on HealthQA, macro F1 was 0.85 for English and Spanish, compared with 0.73 for Chinese and 0.65 for Hindi.AUC followed the same pattern: 0.92 / 0.87 for English / Spanish versus 0.68 / 0.62 for Chinese / Hindi.
- Results: On LiveQA, GPT-3.5 achieved 0.73 in English versus 0.66 / 0.62 / 0.67 on the three non-English datasets.The passage describes this as a sharp decrease relative to English.
- Results: Performance remained relatively stable across temperature settings, with standard deviations below 0.01.This stability was reported alongside the cross-language verifiability comparisons.
6 RELATED WORKS
Prior medical benchmarks may not reflect real-world human–LLM interactions, while English-centric training raises doubts about multilingual capability and equitable access.
- Existing medical benchmarks may not capture how people use LLMs for symptom evaluation, health precautions, or medical-term clarification.
- The study addresses whether LLMs can support inclusive, real-world healthcare interactions across languages.
- LLM development remains heavily English-centric, with LLaMA 2 sourcing nearly 90% of pretraining data from English texts.
7 DISCUSSION
The discussion links persistent non-English weaknesses in correctness, consistency, and verifiability to equity concerns and proposes clearer safeguards and adaptable evaluation.
- Equity and accessibility of healthcare information: Healthcare LLM responses are repeatedly weaker in non-English languages across correctness, consistency, and verifiability, challenging claims of equitable access.
- Equity and accessibility of healthcare information: Trustworthiness cues could make limited non-English healthcare capabilities and potential harms more prominent to users.
- Generalized framework for evaluating LLM Applications: XLingEval can evaluate other models and domains, but domain-specific metrics require tailoring to requirements such as legal precedence.
- Generalized framework for evaluating LLM Applications: The disparity is attributed primarily to limited non-English data resources and imbalanced training-data distributions.
- Future of LLMs in Healthcare: Future healthcare LLMs require safeguards such as knowledge bases and semi-automated training-data quality verification.
8 CONCLUSION AND LIMITATIONS
The paper presents XLingEval and XLingHealth to evaluate multilingual healthcare LLMs, reports language disparities, and acknowledges important model and dataset limitations.
- XLingEval evaluates healthcare LLMs on accuracy, consistency, and verifiability across English, Chinese, Hindi, and Spanish.
- The experiments reveal disparities in LLM responses across the four evaluated languages, motivating stronger cross-lingual capabilities.
- XLingHealth is introduced as a benchmark for assessing multilingual healthcare capabilities.
- The study compares GPT-3.5 with the smaller healthcare-focused MedAlpaca because no comparable open-access multilingual general-purpose LLM was available.
- The absence of multilingual healthcare datasets required machine translation, which may limit translation quality.
A DETAILS OF DATASET CONSTRUCTION
The benchmark is constructed from translated healthcare questions whose quality is evaluated by fluency and meaning, using the best-performing translation model for each target language.
- The benchmark addresses the lack of multilingual healthcare QA datasets by curating translated versions and evaluating their quality.
- Translation quality is assessed on fluency and whether the target-language version faithfully conveys the English meaning.
- ChatGPT performs best for English-to-Chinese and English-to-Spanish translation, while Google Translate performs best for English-to-Hindi translation.
- The dataset uses the best-performing translation model for each non-English language to maximize translation quality.
- Correctness evaluation compares LLM answers with expert ground truth using contradiction, appropriateness, and comprehensiveness dimensions.
- Human correctness evaluation instances contain a question, expert answer, GPT-3.5 response, and reasoning supporting the classification label.
C.2 Results
The verifiability evaluation uses five classification metrics and reports human-annotation consistency for multilingual assessment. Annotation coverage differed across languages, with fewer medical experts recruited for Chinese and Hindi.
- Three annotators evaluated English and Spanish batches, whereas Chinese batches used two and Hindi batches used one.The authors note difficulties securing three medical experts for every batch in Chinese and Hindi.
- Five verifiability metrics were used: macro precision, macro recall, macro F1-score, accuracy, and AUC.Macro precision and recall average class-specific precision and recall; macro F1 is their harmonic mean.
D.2 Results of Unpaired t-tests
Unpaired t-tests found statistically significant cross-lingual performance differences for most language pairings, with English–Spanish differences emerging as temperature increased.
- For most language pairings, p-values remained below α, indicating statistically significant cross-lingual performance discrepancies.The English–Spanish comparison at τ=0.0 was generally statistically consistent, but significant differences emerged as τ approached 1.0.
D.3 Results on MedAlpaca
MedAlpaca shows severe multilingual weaknesses across correctness, consistency, and verifiability, despite stronger lexical consistency than GPT-3.5. Its language consistency is lowest in Spanish and varies with temperature.
- Consistency: MedAlpaca’s topical-level consistency declined compared with GPT-3.5, while lexical consistency increased through higher sim1-gram and sim2-gram scores.
- Consistency: MedAlpaca-30b had the lowest language consistency in Spanish, and language consistency varied across temperatures.The benchmark metric measures alignment between the response language and the source language; the associated table reports general decreases as temperature rises except for Spanish.
- Verifiability: MedAlpaca-30b performed poorly in claim authentication because predictions were highly imbalanced toward the negative class, producing low macro-precision, macro-recall, macro-F1, and AUC scores in most languages.
- Correctness: For HealthQA, MedAlpaca’s comprehensive-and-appropriate answers decreased by approximately 92.23%, 97.93%, and 93.26% for Spanish, Chinese, and Hindi versus English.Similar trends were observed on LiveQA and MedicationQA, while non-English answers were often labeled neither contradictory nor similar to the ground truth.