Source-linked AI summary
Lessons from Natural Language Inference in the Clinical Domain
Alexey Romanov, Chaitanya Shivade
TL;DR
Clinical NLI remains difficult because specialized medical text is knowledge-intensive, while clinical datasets are limited by privacy and costly expert annotation. The paper introduces MedNLI, an expert-annotated dataset grounded in patient histories, and evaluates open-domain transfer learning and domain-knowledge integration. The experiments report performance gains from domain-specific embeddings and ontology-based modeling, while also identifying annotation artifacts and difficult numerical and domain-specific reasoning.
Problem
Clinical-domain NLI lacks large labeled datasets and poses specialized reasoning challenges because medical data is sensitive and clinical text uses a distinct sublanguage.
Method
The paper constructs MedNLI from clinician-annotated MIMIC-III patient histories and studies open-domain transfer learning plus integration of medical terminology and ontology knowledge.
Results
The experiments provide strong baselines and report performance gains from domain-specific word embeddings and integrating domain ontologies into model training.
Takeaways & Limitations
The released dataset, code, and benchmarks are intended to support further research in clinical NLP and clinical NLI.
Takeaways & Limitations
MedNLI has annotation artifacts, and a premise-oblivious classifier achieves an F1 of 61.9 on it.
Abstract
from arXiv · showhide
State of the art models using deep neural networks have become very good in learning an accurate mapping from inputs to outputs. However, they still lack generalization capabilities in conditions that differ from the ones encountered during training. This is even more challenging in specialized, and knowledge intensive domains, where training data is limited. To address this gap, we introduce MedNLI - a dataset annotated by doctors, performing a natural language inference task (NLI), grounded in the medical history of patients. We present strategies to: 1) leverage transfer learning using datasets from the open domain, (e.g. SNLI) and 2) incorporate domain knowledge from external data and lexical sources (e.g. medical terminologies). Our results demonstrate performance gains using both strategies.
1 Introduction
The paper introduces clinical-domain NLI as an underexplored, knowledge-intensive problem and addresses limited public training resources with MedNLI and supporting modeling studies.
- NLI determines whether a hypothesis can be inferred from a premise, a task known as recognizing textual entailment.
- SNLI provides a high-quality NLI benchmark but is limited to Flickr captions and mostly short, simple sentences.
- Clinical NLP lacks large labeled datasets because sensitive patient data prevents crowdsourced annotation and expert labeling is costly.
- Prior biomedical textual-entailment datasets were constrained, and none of the datasets used in those studies were publicly available.
- Clinical text presents distinct challenges, including abbreviations, inconsistent punctuation, misspellings, and a specialized medical sublanguage.
- MedNLI is introduced as a publicly available, expert-annotated clinical NLI dataset alongside open-domain transfer-learning and domain-knowledge integration studies.
2 The MedNLI dataset
MedNLI was constructed from de-identified MIMIC-III clinical notes, with clinicians generating and validating inference pairs from Past Medical History sections. The resulting resource contains 14,049 sentence pairs and reflects both clinical reasoning challenges and annotation limitations.
- 2.1 Premise sampling and hypothesis generation: Clinicians generated definitely true, definitely false, and maybe true hypotheses from each selected premise using an annotation prompt.
- 2.1 Premise sampling and hypothesis generation: MedNLI uses MIMIC-III clinical notes as premises and clinician-generated hypotheses grounded in patients’ Past Medical History.MIMIC-III includes records from 38,597 patients and 2,078,705 clinical notes.
- 2.2 Annotation collection: De-identification artifacts sometimes made premises unusable and increased the difficulty of clinical inference.
- 2.2 Annotation collection: κ = 0.78 measured agreement between clinicians, with disagreements mainly distinguishing “Definitely true” from “Maybe true.”
- 2.3 Dataset statistics: 14,049 unique sentence pairs were created from 4,683 premises and split into training, development, and testing subsets without premise overlap.
- 2.3 Dataset statistics: MedNLI concepts span clinical categories such as medications, diseases, symptoms, and devices, which were mapped to UMLS terminology and semantic types.
3 Models
The paper evaluates baseline NLI architectures, transfer-learning strategies, and knowledge-directed attention for the clinical domain. Models range from feature-based and bag-of-words systems to recurrent architectures incorporating token interactions and medical knowledge.
- Baseline models: The baseline comparison includes a feature-based system, BOW, InferSent, and ESIM, with consistent hyperparameters used to compare relative performance.The study’s stated goal is to explore relative gains from proposed methods rather than outperform existing models.
- Feature-based system: 35 handcrafted features combine standard NLP signals with UMLS-based similarity, including shortest-path distances between medical concepts.The feature groups include BLEU, token counts, negation, TF-IDF, edit distance, embedding similarity, and UMLS similarity.
- Neural architectures: BOW represents each sentence by summing token vectors, concatenates premise and hypothesis vectors, and feeds them to a multilayer neural network.The model uses ReLU rather than tanh because it trained faster and achieved better results.
- Neural architectures: InferSent uses bidirectional LSTM encoders and max pooling, then combines sentence vectors with p, h, |p − h|, and p ∗ h.This interaction vector is designed to contain the information needed to decide the relationship between the input sentences.
- Neural architectures: ESIM uses two bidirectional LSTMs, pairwise attention, attended hidden states, pooling, and a combination scheme similar to InferSent.The architecture computes attention between premise and hypothesis tokens before a second LSTM and pooling operations.
- Transfer learning and domain knowledge: Transfer learning is tested through direct, sequential, and multi-target approaches using open-domain NLI data before clinical-domain evaluation.Multi-target transfer separates shared, source-domain, and target-domain components, while knowledge-directed attention uses SNOMED-CT concept-path lengths.
- Transfer learning and domain knowledge: Knowledge-directed attention computes an n×m matrix whose cells encode shortest SNOMED-CT paths between hypothesis and premise concepts.The resulting knowledge-directed attention is concatenated with token representations so the composition layer can learn their relative importance.
4 Results and discussion
Transfer learning and domain-specific embeddings improved MedNLI performance, while retrofitting degraded it. Knowledge-directed attention and ensembles provided additional gains, supporting methods that incorporate clinical information through flexible relationships.
- Transfer learning: Sequential and multi-target transfer learning improved all models, with maximum gains of 2.4%, 0.9%, and 0.3% for BOW, InferSent, and ESIM, respectively.
- Transfer learning: The best accuracy for every model came from sequential transfer using MultiNLI’s “slate” domain, although ESIM remained below its baseline.
- Word embeddings: MIMIC-III-trained embeddings significantly increased accuracy for all models, whereas fastText Wikipedia embeddings fine-tuned on MIMIC-III gave InferSent a 3.1% boost.
- Word embeddings: The six-model ensemble produced a 0.4% accuracy gain over the best base models without requiring additional data or training.
- Retrofitting: Retrofitting hurt performance, potentially because UMLS contains complex relations unlike the synonymy-dominated relations in WordNet.
- Knowledge-directed attention: Because MedNLI concept pairs rarely have direct knowledge-graph links, knowledge-directed attention was useful for relationships of any length, benefiting both models with fastText[MIMIC-III].
5 Discussion
The discussion identifies persistent reasoning challenges in clinical NLI, especially numerical, domain-specific, abbreviation-based, and clinical-knowledge inferences. It also outlines potential applications and concludes that MedNLI, domain-specific embeddings, and ontology integration support further clinical NLP research.
- Error analysis: Neutral is the hardest class for all models, with most errors involving confusion between entailment and neutral.
- Error analysis: Knowledge-directed attention gains arise mostly from the neutral class, where 87% of improved predictions were previously classified as entailment.
- Error analysis: Numerical reasoning, determining abnormal values, domain-specific inferences, nonstandard abbreviations, and clinical knowledge remain major error sources.
- Error analysis: Negations are interpreted correctly up to 75%, while other tested linguistic features showed no significantly harder subset.
- Error analysis: Semantic-type analysis found no standout error patterns, although correct cases frequently involved Finding or Disease or Syndrome concepts.
- Applications: MedNLI supports potential automation of clinical-trial eligibility assessment and clinical-guideline compliance monitoring through NLI.
- Conclusion: The paper presents MedNLI as a public expert-annotated dataset, reports gains from domain-specific embeddings, and introduces ontology integration to advance clinical NLP.