CasiMedicos-Arg: A Medical Question Answering Dataset Annotated with Explanatory Argumentative Structures

• Medical QA benchmarks largely emphasize answering clinical multiple-choice questions, while doctors must also explain and argue for their predictions.Recent models report high accuracy, but explanation generation and analysis remain insufficiently addressed.Source: §1 · pp. 1–2
• Most Medical QA benchmarks are available only in English, limiting evaluation of models’ abilities in other languages.Source: §1 · p. 2
• CasiMedicos-Arg provides 558 multilingual clinical documents with doctors’ explanations of correct and incorrect predictions, annotated with 5021 claims, 2313 premises, 2431 support relations, and 1106 attack relations.The dataset covers English, French, Italian, and Spanish and supports argument mining and explanation generation research.Source: §1 · p. 2

2.1 Medical Question Answering

• Medical QA benchmarks mainly organize English datasets around multiple-choice evaluation, with sizes ranging from hundreds to approximately 187K questions.Long-form answers exist in several datasets, but they are not consistently constructed as clinician-authored ground truth.Source: §2.1 · p. 2
• MIRAGE evaluates retrieval-augmented generation for Medical QA, and MEDRAG improves some zero-shot results by grounding generation in specific, up-to-date contexts, though effects vary across models.Source: §2.1 · p. 2
• Existing Medical QA resources generally lack manually annotated argumentative explanations for both correct and incorrect predictions, especially beyond English.CasiMedicos-Arg is introduced to address this gap with premises, claims, support, and attack relations.Source: §2.1 · p. 3

2.2 Explanatory Argumentation in the Medical Domain

• Explanatory-language research has generated rationales jointly with predictions or labels using models such as bi-LSTM, GPT-2, RoBERTa, and T5.These approaches span natural-language inference, question answering, and general explanation generation.Source: §2.2 · p. 3
• Medical explanation systems include template-based generation using Human Phenotype Ontology knowledge and methods for evaluating reasons in clinical texts.These efforts aim to make explanations more verifiable or assessable in medical settings.Source: §2.2 · p. 3
• Despite this progress, existing explanation approaches are generally not grounded in argumentation models and lack verified medical knowledge at their core.The paper identifies these shortcomings as especially important because medical explanations must justify decisions in a sensitive domain.Source: §2.2 · p. 3

3.1 Argumentation Annotation Guidelines

• The guidelines annotate Claims, Premises, Support, Attack, discourse Markers, and medical labels for Disease, Treatment, and Diagnostics.Claims and Premises structure arguments, while medical labels identify the types of answers and arguments being discussed.Source: §3.1 · pp. 4–5
• The annotation distinguishes case-specific evidence as Premises from general medical knowledge as Claims, including statements that support or attack a main claim.Long statements containing distinct evidence are split into independent premises so their relations to specific claims remain separable.Source: §3.1 · p. 4

3.2 CasiMedicos Real Case Example

• A worked pediatric case illustrates explanations that justify the correct answer while also ruling out alternative treatments and diagnoses.The example includes markers, premises, claims, and explicit reasoning about follow-up, mammography, biopsy, and hormonal treatment options.Source: §3.2 · pp. 5–6

3.3 Annotation Process and Results

• Annotation proceeded through training, reconciliation, and full-dataset labeling, with agreement assessed on 100 cases at instance and token levels.Annotators first worked on 10 cases, then reconciled Claim and Premise descriptions before completing the dataset.Source: §3.3 · p. 6, §3.4 · p. 6

3 CasiMedicos-Arg Annotation

• 558 cases were retained after excluding 64 cases lacking gold explanations or containing confusing answer–argument relations.The source corpus contained 622 medical cases before annotation filtering.Source: §3.3 · p. 6

3.4 Inter-Annotator Agreement (IAA)

• Instance-level F1 was 0.765 for Claims and 0.659 for Premises, with mean F1 0.669 across labels; token-level mean F1 reached 0.880.Token-level Claim agreement was 0.915, while Diagnostics was lowest at 0.638.Source: §3.4 · p. 6, Table 1 · p. 6, Table 2 · p. 6

3.5 Annotation Results

• The explanations contain many more Claims than Premises because doctors often rely on general knowledge, omit case evidence, or leave premises implicit.The dataset analysis identifies explanations based on clinical information without repeating it and explanations whose warrants are implied rather than stated.Source: §3.5 · pp. 6–7, Table 3 · p. 7, Table 4 · p. 7
• Support relations occur about twice as often as Attack relations, while conclusions based only on attacks are associated with lower confidence.The relation distribution is reported in Table 5.Source: §3.5 · p. 7, Table 5 · p. 7

4.2 Sequence Labelling with LLMs

• The benchmark evaluates encoder-only, encoder-decoder, and decoder-only models for multilingual argument-component detection.Models include mBERT, mDeBERTa, Medical mT5, LLaMa2, and Mistral.Source: §4 · p. 7, §4.2 · p. 8

4.1 Multilingual CasiMedicos-Arg

• English annotations were projected to French, Italian, and Spanish using word alignments, automatic post-processing, and native-speaker manual revision.The post-processing corrected sequence misalignments before expert review.Source: §4.1 · pp. 7–8

4.2 Sequence Labelling with LLMs

• Experiments compare monolingual training, English-to-target model transfer, and multilingual data augmentation across the four languages.Multilingual augmentation pools training data from all languages before evaluating each language separately.Source: §4.2 · p. 8
• The study reports sequence-level F1-macro results, averaging three randomly initialized runs, with Table 6 covering English settings.Hyperparameters were selected separately for model families, including different epoch counts.Source: Table 6 · p. 8, §4.2 · p. 8

English and multilingual results

• Multilingual pooling improves every model over English-only gold-data training, while decoder-only generative models outperform the other architectures overall in English.Medical mT5 models perform nearly as effectively as the decoder-only models.Source: §5 · p. 8, Table 7 · p. 9

Multilingual results

• Mistral achieves the highest F1-macro scores for Spanish, French, and Italian, although multilingual training yields no substantial additional improvement for Mistral.The comparison is reported in Table 7 for data-transfer and cross-lingual model-transfer settings.Source: §5 · p. 9, Table 7 · p. 9
• Cross-lingual model transfer is the least effective setting, whereas Medical mT5 obtains the best results in that setting.The authors relate this pattern to Medical mT5’s training on multilingual medical data.Source: §5 · p. 9

Overall findings

• The results provide competitive baselines for argument-component detection and support the validity of the annotations and English-to-other-language projection strategy.The conclusion links the baselines to both annotation validity and multilingual data-transfer performance.Source: §5 · p. 9

Dataset contribution

• CasiMedicos-Arg contains 558 parallel documents in French, English, Italian, and Spanish with doctors’ gold reference explanations.The dataset targets evaluation of explanations that justify medical predictions.Source: §6 · p. 9
• The annotations comprise 5,021 claims, 2,313 premises, 2,431 support relations, and 1,106 attack relations.These annotations enrich the reference explanations with argumentative structure.Source: §6 · p. 9

Empirical conclusion

• Inter-annotator agreement and argument-component baselines support annotation validity, while experiments show an advantage for multilingual data transfer.The conclusion presents multilingual transfer as the strongest supported experimental finding.Source: §6 · p. 9

Scope

• The dataset covers four languages, but the authors wanted broader language-family and morphological diversity and were constrained by available native expertise.The limitation concerns both language selection and the expertise needed to manually correct projected labels and translations.Source: Limitations · p. 9

Dataset size

• The dataset contains 558 documents, which the authors identify as a size limitation despite its similarity to other widely used Medical QA benchmarks.The authors describe increasing the dataset size as a short-term objective while contextualizing its current scale against related resources.Source: Limitations · pp. 9–10

Scope

• The experiments report multilingual data-transfer as superior to cross-lingual model transfer with currently available LLMs.This comparison qualifies the language-related limitation by identifying a supported transfer strategy that performed better in the reported experiments.Source: Limitations · p. 10

Future work

• Future work must further study explanation generation and how to evaluate explanations in the highly specialized medical domain.The authors characterize medical-domain explanation evaluation as a crucial unsolved issue.Source: Limitations · p. 10

Dermatology clinical case

• A dermatology case describes tense hand blisters after sun exposure in a man with alcohol abuse, hepatitis C, and slight malar hypertrichosis.The listed diagnostic options include epidermolysis bullosa acquisita, porphyria cutanea tarda, phototoxic reaction, contact dermatitis, and acute intermittent porphyria.Source: A Appendix. CasiMedicos Real Cases · p. 13
• Porphyria cutanea tarda is supported by photoexposed blistering on the hands, hypertrichosis, excess alcohol use, and hepatitis C infection.The explanation also describes uroporphyrin elevation associated with reduced uroporphyrinogen decarboxylase activity.Source: A Appendix. CasiMedicos Real Cases · p. 13

Pediatrics clinical case

• A pediatrics case concerns a six-month-old infant with respiratory distress, retractions, rhonchi, prolonged expiration, and slightly decreased bilateral air entry.The infant is observed in hospital, and the question asks which complementary tests are appropriate.Source: A Appendix. CasiMedicos Real Cases · p. 13
• For the pediatrics case, the stated recommendation is not to request complementary tests unless clinical worsening occurs.This recommendation appears as the answer associated with the respiratory presentation.Source: A Appendix. CasiMedicos Real Cases · p. 13

Correction counts

• Table 8 reports the number of corrections introduced during post-processing after automatic label projection.The supplied material provides the table title but not its entries or numerical results.Source: Table 8 · p. 13, B Number of corrections after… · p. 13