PaperlayerView exact revision on arXiv

Source-linked AI summary

PAINT: A Framework for Multilingual Pain Representation from Medical Licensing Exams to Real-World Clinical Texts

Boya Zhang, Yuanchao Ma, Jiageng Wu, Krittaphas Chaisutyakorn, Rodrigo Gameiro, Ahram Han, Chien-Yu Huang, Nikhil Jaiswal, Marianna Leite, Miriam Mora-Mau, Jaime Pineda-Moreno, Siena Placino, Ryohei Yamamoto, Yugang Jia, Jie Yang, Leo Anthony Celi, Douglas Teodoro

arXiv:2610.18269v1cs.CL

TL;DR

Pain documentation varies across languages, document types, and patient demographics, while existing pain resources remain limited in linguistic and clinical diversity. PAINT pairs exam and real-world clinical text across four languages under a shared schema and validates its ensemble labels with clinicians. The validation reports high ensemble performance across pain fields and demographic extraction.

  • Problem

    Existing pain resources remain limited in linguistic and clinical diversity, despite the importance of studying variation in pain assessment, treatment, and documentation across patient groups and languages.

    Source: §1 · pp. 1–2

  • Method

    PAINT pairs medical examination and real-world clinical texts under a multilingual schema and uses a three-model agreement ensemble for pain and demographic annotations.

    Source: §1 · p. 2, §4 · pp. 3–4

  • Results

    Clinician validation found ensemble macro-F1 of 0.91–1.00 on BRIDGE-Pain and demographic extraction accuracy of 92–100% across five clinical datasets.

    Source: §5 · p. 5

  • Takeaways & Limitations

    Pairing exam and clinical text under a common schema enables within-language comparisons, with clinician-validated annotations and an open-weight labeling pipeline.

    Source: §1 · p. 2, §7 · p. 6

    Corpus purpose, language, country, and healthcare system co-vary across partitions, so their label distributions characterize each corpus rather than estimate prevalence or support cultural claims about pain experience.

    Source: Limitations · p. 7

Abstract

from arXiv · show

Pain is one of the most common and clinically important symptoms, reflecting disease onset, progression, and treatment response. However, pain is difficult to accurately capture and interpret from medical text, as its expression varies across languages, documentation types, and patient demographics, and is often embedded in terminology, negation, temporality, and contextual descriptions. Existing pain-related studies remain largely monolingual or limited to narrow clinical settings. In this study, we introduce PAINT, a multilingual framework for studying pain representation across medical exams and real-world clinical texts. PAINT includes nine pain-focused dataset partitions spanning English, Spanish, Portuguese, and Japanese, with clinician annotations in all four languages. We curated a structured schema covering pain relevance, pain presence, anatomical location, pain descriptors, and demographic information. To capture multilingual pain representations, we leverage large language models (LLMs) and adopt an ensemble strategy to improve performance. Ten clinicians across four languages validate the labels: the ensemble reaches macro-F1 of 0.91-1.00 on pain relevance and pain presence and 92-100% accuracy on demographics, with inter-annotator $\kappa$ of 0.82-1.00. We release the WorldMedQA-Pain ensemble labels, the extraction pipeline, validation scripts, and clinician annotation sheets for publicly licensed corpora, providing a foundation for pain-centered clinical NLP and for studying pain documentation across languages and healthcare contexts.

1 Introduction

PAINT addresses the lack of a unified multilingual pain resource spanning medical education and clinical practice. It pairs exam and clinical text under a shared schema, with clinician-validated labels, an extraction pipeline, and documentation of label reliability and coverage.

1 Introduction

  • Existing pain resources are limited in linguistic and clinical diversity, and none provides a unified multilingual representation across medical education and clinical practice.Prior annotated corpora focused on English clinical text.Source: §1 · p. 2
  • PAINT pairs medical licensing examination questions with real-world clinical records under a common schema across Portuguese, Japanese, Spanish, and English.The pairing enables within-language comparisons between exam and clinical text.Source: §1 · p. 2
  • The resource includes clinician-validated labels and a released extraction pipeline, with validated labels also serving as an evaluation set for pain extraction.The evaluation set is especially useful for testing systems in non-English settings where such data are scarce.Source: §1 · p. 2
  • The labeling pipeline is cross-lingual and cross-corpus, transfers from WorldMedQA-Pain to BRIDGE-Pain, and runs locally on a single GPU.On five clinical corpora, clinician review confirms macro-F1 of at least 0.91 for pain relevance and presence and at least 92% demographic accuracy.Source: §1 · p. 2
  • PAINT documents partition-level sample sizes, consensus coverage, label rates, demographic-field coverage, annotation instructions, and inter-annotator disagreements.This documentation lets users judge reliability by field and partition before reuse.Source: §1 · p. 2

2 Related Work

Related work describes pain disparities and multilingual clinical resources, but existing resources do not represent pain as a standardized variable for cross-lingual analysis. PAINT builds on LLM extraction and ensemble reasoning to produce multilingual pain annotations.

Pain disparities and clinical bias

  • Studies report disparities in pain assessment and treatment across race, gender, age, and language, and demographic bias in LLM-assisted pain management (Singhal et al., 2016; Hoffmann and Tarzian, 2001; Eccleston et al., 2021; Rambachan et al., 2023; Omar et al., 2026, 2025).Pierson et al. (2021) found that patient-reported pain models explained more racial disparities in reported pain than radiologist-based severity grading.Source: §2 · p. 2

Multilingual clinical resources

  • Medical exam benchmarks and clinical corpora cover multilingual knowledge and routine documentation, but none provides pain as a standardized structured variable for cross-lingual analysis.Berke et al. (2026) separately propose a computational theory of intuitive pain reasoning.Source: §2 · p. 3

LLM-based clinical information extraction

  • PAINT builds on few-shot extraction, chain-of-thought prompting, self-consistency, and ensemble reasoning by using two multilingual prompting pipelines and agreement among three models for silver annotations.The cited foundations include Agrawal et al. (2022), Wei et al. (2022), Wang et al. (2023), and Chang et al. (2024).Source: §2 · p. 3

3 Multilingual Medical Data Collection

The data collection pairs four medical licensing examination partitions with five real-world clinical-text partitions. Together, these corpora provide the exam and clinical material used for PAINT’s multilingual pain representation.

WorldMedQA-Pain

  • WorldMedQA-Pain contains 2,963 clinician-annotated instances from four licensing examinations in Brazilian Portuguese, Japanese, Spanish, and English.The partitions are REVALIDA, Japan’s National Medical Licensing Examination, MIR, and PLAB.Source: §3 · p. 3

BRIDGE-Pain

  • BRIDGE-Pain contains 17,795 clinical documents across five partitions, retaining complete MEDIQA and Cantemist data and down-sampling three other sources to 5,000 records each.The sources cover English dialogues and ICU summaries, Spanish oncology reports, Brazilian Portuguese electronic health records, and Japanese medication-safety reports.Source: §3 · p. 3

Data collection

  • Both resources use the multilingual annotation approach, with the BRIDGE-Pain labels produced by the ensemble pipeline and validated by clinicians on sampled records.WorldMedQA-Pain labels are clinician-annotated under the multilingual protocol.Source: §3 · p. 3

4 Pain and Demographic Annotation

PAINT separates whether pain is mentioned from whether it is present, then applies multilingual model-based extraction and agreement rules. The method also documents disagreement, attribute aggregation, and normalization choices.

Extraction pipeline

  • The extraction uses two chain-of-thought prompting pipelines, one for pain attributes and one for demographics, processing source texts in their original languages.Hierarchical constraints on pain labels are enforced in the prompts.Source: §4 · p. 3

Annotation schema

  • Pain relevance captures any mention, while pain presence requires pain to be asserted in the patient; location and descriptors are conditioned on pain presence, and demographics are recorded when stated.A denial or intervention-related mention can be relevant without being pain-present.Source: §4 · p. 3
  • The schema treats explicit denials and painless-type derivatives as examples of pain mentions that are not asserted in the patient.Table 1 presents cases agreed on by all three models.Source: Table 1 · p. 4, §4 · p. 4

Model selection and ensemble

  • Gemma-4-E4B, Qwen3-8B, and DeepSeek-R1-Distill-Qwen-32B were selected as the three highest-performing open-weight models and combined into a consensus ensemble.Their macro-F1 scores ranged from 0.87 to 0.97 for pain relevance and presence across the four WorldMedQA-Pain partitions.Source: §4 · p. 4

Ensemble strategy

  • For pain relevance and presence, the ensemble labels an instance only when all three models agree on both fields; otherwise it abstains, prioritizing precision over recall.Ensemble coverage is the fraction of instances with non-abstaining predictions.Source: §4 · p. 4

Attribute aggregation

  • Demographic values use majority voting over non-null predictions, while locations and descriptors are aggregated item by item under a two-of-three agreement rule.For sparse attributes, this retains greater coverage than requiring unanimous agreement.Source: §4 · p. 4

The abstain subset

  • On the abstain subset, no system exceeds 0.75 macro-F1 for pain presence, while GPT-5 reaches 0.80–0.98 for pain relevance.The reported pattern indicates that abstentions are associated primarily with uncertainty about pain presence, and the consensus subset is enriched for more reliably determined presence cases.Source: §4 · p. 4

Normalisation

  • Demographic normalization records age in years, maps gender to male or female using a cross-lingual dictionary, preserves ethnicity and occupation as free text, and assigns null to unnormalizable values.Source: §4 · p. 4

5 Clinician Validation

Ten clinicians validated ensemble labels across WorldMedQA-Pain and BRIDGE-Pain; pain-field and demographic results were generally high, alongside high inter-annotator agreement.

Validation design

  • Ten clinicians, two per dataset, independently reviewed 100 records per BRIDGE-Pain dataset and 30 records per WorldMedQA-Pain partition.The validation protocol included overlapping subsets of 50 BRIDGE-Pain records and 10 WorldMedQA-Pain records for independent annotator reliability assessment.Source: §5 · pp. 4–5

WorldMedQA-Pain evaluation

  • WorldMedQA-Pain model benchmarks distinguish full-corpus individual-model scores from ensemble and GPT-5 evaluation on the three-model consensus subset.Consensus coverage is the fraction of the full corpus retained; Table 3 separately reports results for non-consensus (abstain) rows.Source: Table 2 · p. 5, Table 3 · p. 5

Clinician validation

  • 0.91–1.00 macro-F1 on BRIDGE-Pain pain fields was comparable to 0.92–0.99 on WorldMedQA-Pain; demographic accuracy was 92–100% across five clinical datasets.Inter-annotator agreement was κ = 0.88–1.00 for pain relevance, κ = 0.87–1.00 for pain presence, and 90–100% for demographics; wherever defined, demographic κ ranged from 0.82 to 1.00.Source: §5 · p. 5, Table 5 · p. 6

Metrics

  • Table 5 compares the ensemble with clinician annotations in panel (a) and reports clinician agreement on overlapping records in panel (b).Pain fields use macro-F1 in panel (a) and Cohen’s κ in panel (b); demographic fields use accuracy and percentage agreement, respectively.Source: Table 5 · p. 6, §5 · p. 6

6 Availability and Dataset Statistics

The paper releases labels, extraction and validation materials, and documents partition-level statistics, with access conditions varying by corpus license.

Availability

  • The release includes WorldMedQA-Pain consensus labels, the extraction pipeline, prompts, schema, validation and agreement scripts, and clinician annotation sheets for publicly licensed corpora.Annotation sheets for credentialed corpora are available on request; BRIDGE-Pain labels cannot be redistributed, but authorized users can regenerate them with the pipeline.Source: §6 · p. 6
  • WorldMedQA-Pain clinician gold labels are planned for a subsequent version; other fields beyond pain relevance and presence are released as per-model raw outputs for aggregation with scripts.The released consensus labels cover pain relevance and pain presence.Source: §6 · p. 6

Dataset statistics

  • Partition-level label distributions differ substantially and should not be treated as comparable prevalence estimates.Tables 6 and 7 summarize the released labels per partition.Source: §6 · p. 6

7 Conclusion

PAINT brings pain representation in licensing examinations and real-world clinical text together across four languages, with clinician-validated annotations and an open-weight labeling pipeline. It is intended as infrastructure for multilingual clinical information extraction and studying documentation variation across languages and care settings.

7 Conclusion

  • PAINT is presented as a multilingual resource spanning medical licensing examinations and real-world clinical text in English, Spanish, Portuguese, and Japanese.Its contribution includes clinician-validated pain and demographic annotations and an open-weight labeling pipeline.Source: §7 · p. 6
  • PAINT is intended as a benchmark for multilingual clinical information extraction and infrastructure for studying pain documentation and attached demographic detail across languages and care settings.The stated scope includes examining whether large language models reproduce known disparities in pain assessment.Source: §7 · p. 6
  • Table 6 defines partition sample budgets and coverage within rows where all three models agree on both pain fields.The three largest partitions are down-sampled to 5,000 records, while Cantemist and MEDIQA are included in full.Source: Table 6 · p. 7
  • Table 7 reports WorldMedQA-Pain partition statistics, with pain rates derived from the three-model ensemble and location representing non-empty pain-location items.Clinician-annotated labels validate the pain rates; they are not identified as the source of those rates.Source: Table 7 · p. 7

Limitations

PAINT’s partition-level labels are not comparable prevalence estimates, and the resource captures clinician- and examination-writer documentation rather than patient reports. Cross-language and cross-partition comparisons also face data-access and language-variety constraints.

Limitations

  • Partition label distributions characterize their corpora, not prevalence; differences should be read as corpus-purpose effects rather than cultural differences in pain experience.Corpus purpose, language, country, and healthcare system co-vary across the nine partitions.Source: Limitations · p. 7
  • PAINT captures pain documented by clinicians and examination writers, not patient reports, and none of its source corpora contain numeric pain scores.The schema could accommodate structured intensity scores and patient-reported pain if corpora with those data become available.Source: Limitations · p. 7
  • The five BRIDGE-Pain partitions cannot be redistributed, so users need access to the source data to regenerate the clinical half with the released pipeline.Source: Limitations · p. 7
  • English exam-versus-clinical comparisons are confounded by UK-versus-US language, and the Brazilian Portuguese partitions do not represent European Portuguese vocabulary.Source: Limitations · p. 7

Ethics Statement

PAINT’s ethics and release practices preserve source-corpus restrictions, while its stated use is research rather than clinical decision-making. The section also describes the inference setup and cross-language differences in pain descriptors.

Ethics Statement

  • The annotation study received MIT IRB exemption, and released clinician sheets omit source text or are access-restricted according to corpus licensing.Ten clinicians—nine physicians and one pharmacist—fluent in the source languages conducted the annotation.Source: Ethics Statement · pp. 7–8
  • PAINT is intended for multilingual clinical NLP, information extraction, and fairness auditing, not diagnosis, treatment recommendations, or automated triage.Demographic results for restricted clinical datasets are reported only as aggregate statistics, apart from seventeen de-identified annotation disagreements in Appendix E.Source: Ethics Statement · p. 8

A Compute and infrastructure

  • PAINT uses inference without training or fine-tuning, running three 4-bit open-weight models locally on an A100 or L4 GPU and querying GPT-5 by API.Source: A Compute and infrastructure · p. 11

B Pain descriptor vocabulary by language

C Age composition per dataset

The age-composition figure displays both bin sample sizes and each bin’s share of its partition, while warning that extreme age bins are thin in several partitions. The prompt templates adapt a shared English structure across four languages.

C Age composition per dataset

  • Extreme age bins are thin in several partitions, a constraint to keep in mind when stratifying by age.Figure 2 normalizes age composition; cell numbers show bin sample sizes and colors show each bin’s share of the partition.Source: C Age composition per dataset · p. 11

D Prompts

  • The English pain and demographic extraction prompts are paired with Japanese, Spanish, and Brazilian Portuguese versions that substitute language-specific terms and examples.The variants reuse the four-step English template, with substitutions authored or reviewed by fluent collaborators against each language’s pain lexicon.Source: D Prompts · p. 11

E Clinician validation

Clinician validation supports high agreement between PAINT’s ensemble labels and clinician judgments, while the appendices document how labels were checked, who annotated them, and where validation results apply.

E Clinician validation

  • 0.98 macro-F1 on both binary pain fields and over 97% on every demographic field in pooled BRIDGE-Pain validation, on records where clinicians agree.The pooled validation covers about 750 records per field; pain-field scores compare ensemble outputs with clinician annotations, while demographic scores are clinician-confirmed accuracy.Source: E Clinician validation · p. 11, Table 9 · p. 12
  • Clinicians judged whether displayed ensemble values matched source text, supplied corrections when wrong, and excluded unjudged records from agreement calculations.Blank values counted as correct when the text stated no such information; normalization rules varied by field, and occupation strings were compared exactly after lower-casing and trimming whitespace.Source: E Clinician validation · p. 12
  • Cantemist annotators agreed on all 50 overlapping records (κ = 1.00 on all five fields), while exam demographic validation was treated as confirmatory because vignette details are explicit and near-ceiling.WorldMedQA-Pain demographic validation used 30 records per partition, with 10-record overlaps and κ of 0.82–1.00 where defined.Source: E Clinician validation · p. 12, Table 10 · p. 16

G Annotation provenance

  • Clinician validation was conducted individually by ten coded clinicians, with separate sheets and no visibility into other annotators’ labels or whether their judgments matched the pipeline.The same clinician code denotes the same person across partitions; Spanish annotators were native speakers, and English annotators used English clinically.Source: G Annotation provenance · pp. 14–15

H Released spot-check protocol

  • Spot checks sample 30–50 records per language and field when ensemble-resolved coverage exceeds 5%, but assess accuracy only on resolved rows, not recall on the abstain subset.Clinicians rate outputs as matches, diverges, or ambiguous; reporting includes match rates with Wilson 95% confidence intervals, ambiguity rates, and common failure modes.Source: H Released spot-check protocol · p. 16

I WorldMedQA-Pain annotation protocol

  • WorldMedQA-Pain exam items are labeled in two ordered steps: pain relevance includes denials, while pain presence requires pain to be asserted in the patient.Items unresolved by the procedure are adjudicated by the team; language-specific lexicons use authoritative references and country annotator review.Source: I WorldMedQA-Pain annotation protocol · p. 17

Related papers

Loading 2610.18269v1…