Source-linked AI summary
MedSTS: A Resource for Clinical Semantic Textual Similarity
Yanshan Wang, Naveed Afzal, Sunyang Fu, Liwei Wang, Feichen Shen, Majid Rastegar-Mojarad, Hongfang Liu
TL;DR
Clinical STS has few resources, despite the need to manage redundant information in EHRs. The authors construct MedSTS from Mayo Clinic clinical notes and have experts annotate a subset. Their comparison reports that clinical STS data are more complex than general-domain STS data, in part because of medical terminology.
Problem
Clinical STS systems and resources are scarce, although STS could help identify redundant EHR information relevant to clinical decision-making.
Source: §1 · pp. 2–3Method
The authors build MedSTS from unique Mayo Clinic clinical-note sentences, select pairs using averaged surface-similarity measures, and have two clinical experts annotate a subset.
Source: §3.1 · p. 5, §3.3 · p. 6Results
The MedSTS clinical STS dataset is more complex than general-domain STS datasets, which the authors attribute to medical terminology and the complexity of synonyms and medical-concept hierarchies.
Source: §5 · p. 11Takeaways & Limitations
Clinical STS systems for MedSTS should consider medical-domain thesauri alongside advanced similarity techniques used for general-domain data.
Source: §5 · p. 11The resource is relatively small, comes from one institution, uses conventional STS guidelines with limited clinical properties, and was annotated by only two experts.
Source: §5 · p. 12
Abstract
from arXiv · showhide
The wide adoption of electronic health records (EHRs) has enabled a wide range of applications leveraging EHR data. However, the meaningful use of EHR data largely depends on our ability to efficiently extract and consolidate information embedded in clinical text where natural language processing (NLP) techniques are essential. Semantic textual similarity (STS) that measures the semantic similarity between text snippets plays a significant role in many NLP applications. In the general NLP domain, STS shared tasks have made available a huge collection of text snippet pairs with manual annotations in various domains. In the clinical domain, STS can enable us to detect and eliminate redundant information that may lead to a reduction in cognitive burden and an improvement in the clinical decision-making process. This paper elaborates our efforts to assemble a resource for STS in the medical domain, MedSTS. It consists of a total of 174,629 sentence pairs gathered from a clinical corpus at Mayo Clinic. A subset of MedSTS (MedSTS_ann) containing 1,068 sentence pairs was annotated by two medical experts with semantic similarity scores of 0-5 (low to high similarity). We further analyzed the medical concepts in the MedSTS corpus, and tested four STS systems on the MedSTS_ann corpus. In the future, we will organize a shared task by releasing the MedSTS_ann corpus to motivate the community to tackle the real world clinical problems.
1. Introduction
Clinical EHR text can become redundant and difficult for providers to manage, while clinical STS resources are scarce. The paper introduces MedSTS, a Mayo Clinic sentence-pair resource intended to support clinical STS research.
1. Introduction
- Clinical EHR documentation can accumulate poorly organized or erroneous material, increasing providers’ cognitive burden when tracking complex problems and maintaining decision-making continuity.The paper links this growth to copying, templates, and smart phrases.Source: §1 · p. 2
- A prior inpatient-note study found 76% of text redundant; its best method identified new information with precision 0.74, recall 0.83, and Fscore 0.78.The cited study used statistical language models and expert-derived gold standards.Source: §1 · p. 3
- MedSTS pairs clinical sentences and includes an expert-annotated subset scored from 0 to 5 for semantic similarity.The authors plan to use MedSTS_ann to motivate a shared medical STS task.Source: §1 · p. 3
- A resource gap in clinical STS motivates MedSTS, since few clinical STS systems have been developed and the authors identify lack of resources as the main reason.The paper presents STS as a way to find highly similar snippets for removal during automated summarization.Source: §1 · p. 3
2. Background
STS assigns a scaled similarity score to text snippets based on meaning, and is related to paraphrase detection and textual entailment. The background describes its NLP uses, shared-task datasets, and established similarity approaches.
2. Background
- STS scores how similar two snippets are in meaning, but differing wording and ambiguous word combinations make the task challenging.It is closely related to paraphrase detection and textual entailment.Source: §2 · p. 4
- STS supports applications including information retrieval, question answering, information extraction, and textual summarization.Methods have used word-to-word similarity, vector-space models, text corpora, semantic resources, and supervised combinations of features.Source: §2 · p. 4
- SemEval STS tasks used sentence pairs from varied public datasets, scored equivalence from 0 to 5, and evaluated systems by Pearson correlation with human scores (Agirre, Diab et al. 2012, Agirre, Cer et al. 2013, Agirre, Banea et al. 2014, Agirrea, Baneab et al. 2015, Agirrea, Baneab et al. 2016).The datasets have also been used for tasks such as sentence embeddings and machine-translation evaluation (Wieting and Gimpel 2017; Magnolini, Vo et al. 2016).Source: §2 · pp. 4–5
3. Methods
The methods construct MedSTS from de-identified Mayo Clinic clinical sentences, select candidate pairs using surface similarity, and have clinical experts annotate a subset. The process yields a large sentence-pair resource and an expert-scored subset.
3.1 Data Collection
- 14.9 million unique sentences from 3 million de-identified Mayo Clinic notes, containing 361.9 million tokens, formed the source corpus after frequency-based PHI filtering.The study received institutional review board approval.Source: §3.1 · p. 5
3.2 Sentence Pairs Selection
- 174,629 sentence pairs entered MedSTS after the authors averaged three surface-similarity scores and retained pairs scoring at least 0.45.The scores used sequence matching, cosine similarity, and normalized Levenshtein distance; Fig. 1 shows the sentence-pair distribution.Source: §3.2 · pp. 5–6, Figure 1 · p. 6
- The authors sampled equal numbers of pairs from five surface-similarity ranges between 0.45 and 0.95 to create a 1,250-pair MedSTS_ann subset.This subset was intended for a future shared task.Source: §3.2 · p. 6
3.3 Annotation
- Two clinical experts independently rated MedSTS_ann sentence pairs on a six-point semantic-equivalence scale, and their average scores served as the gold standard.The scale runs from 0 for completely dissimilar sentences to 5 for completely similar sentences; Table 1 provides score explanations and examples.Source: §3.3 · pp. 6–8, Table 1 · p. 8
4. Results
MedSTS covers four selected UMLS semantic types, with signs and symptoms more frequent than procedures and medications. The annotated subset shows high annotator agreement, while baseline performance is lower on MedSTS_ann than on most general-domain STS datasets.
4.1 Corpus Analysis
- MedSTS contains 5,299 sign-and-symptom types, versus 1,222 disorders, 634 procedures, and 41 medications.MedSTS_ann likewise has more unique signs and symptoms (334) than disorders (164), procedures (124), or medications (20).Source: §4.1 · p. 8
4.2 Annotation Results
- The annotated dataset contains 54,161 word tokens, averages 51 words per sentence, and has weighted Cohen’s Kappa agreement of 0.67.The score distribution assigned by the annotators is shown in Fig. 3.Source: §4.2 · p. 9, Figure 3 · p. 10
4.3 Baseline system results
- Baseline systems performed worse on MedSTS_ann than on most general-domain STS datasets, indicating greater clinical STS complexity.The comparison used Pearson correlation coefficients across the baseline methods and datasets.Source: Table 2 · p. 10, §4.3 · p. 10
5. Discussion
The discussion presents MedSTS as a resource for studying clinical semantic similarity and EHR redundancy, while noting constraints in its data and annotations. It recommends domain-specific thesauri and outlines planned extensions to the resource and clinical STS systems.
5. Discussion
- The resource aims to support NLP methods that reduce EHR redundancy and potentially improve system usability, portability, and generalizability.The authors describe limited access to clinical data, due to privacy and confidentiality, as a barrier to exploration.Source: §5 · p. 11
- The authors propose refining the corpus by note type or section because its current construction does not distinguish them and semantic-type counts are uneven.They suggest adding procedure-focused surgical or therapy notes and medication-section sentences to balance the resource.Source: §5 · p. 11
- Baseline comparisons suggest clinical STS is more complex than general-domain STS; the authors recommend medical thesauri alongside advanced similarity techniques.They attribute the challenge to medical terminology, including synonymous terms and hierarchies of medical concepts (Peterson, Pakhomov et al. 2007).Source: §5 · p. 11
- Clinical STS may help with patient cohort identification by mapping a user query to multiple semantically similar equivalent formulations.The discussion also connects STS with clinical summarization and reducing redundant information associated with cognitive burden.Source: §5 · p. 12
- The authors identify limited size, single-institute data, conventional guidelines with little clinical adaptation, and only two expert annotators as limitations.The sensitive patient data prevented using Amazon Mechanical Turk crowdsourcing for annotation.Source: §5 · p. 12
- The authors plan to remove PHI, split MedSTS_ann into training and testing sets, and organize a shared task with Pearson-correlation evaluation.Participating teams would sign a Mayo Data Use Agreement and be allowed three system submissions.Source: §5 · p. 12
- The authors plan to extend a general-English STS system with clinical features and embeddings learned from a large clinical corpus.They hypothesize that adding clinical-domain features could improve the system for clinical STS tasks.Source: §5 · p. 12