Source-linked AI summary
Word-level Text Highlighting of Medical Texts for Telehealth Services
Ozan Ozyegen, Devika Kabe, Mucahit Cevik
TL;DR
Medical information overload makes it difficult for healthcare professionals to efficiently review patient information. This paper evaluates three approaches to word-level highlighting in medical texts. Its n-gram LSTM is reported as the best-performing model and successfully highlights important medical information.
Problem
Information overload leaves medical professionals spending time searching and filtering information, including patient messages in telehealth.
Source: §1 · p. 2Method
The study evaluates TF-IDF, TF-IDF combined with LIME, and neural-network methods for word-level medical-text highlighting.
Source: §1 · p. 5, §4 · p. 14Results
The n-gram LSTM is reported as the best-performing model and successfully highlights important information in medical texts, while LSTM models outperform TF-IDF and LIME methods on medical chat highlighting.
Source: §5.1 · p. 21, §6 · p. 27Takeaways & Limitations
The study provides a baseline and a numerical evaluation of medical-text highlighting models across two medical text datasets.
Source: §1 · p. 5
Abstract
from arXiv · showhide
The medical domain is often subject to information overload. The digitization of healthcare, constant updates to online medical repositories, and increasing availability of biomedical datasets make it challenging to effectively analyze the data. This creates additional work for medical professionals who are heavily dependent on medical data to complete their research and consult their patients. This paper aims to show how different text highlighting techniques can capture relevant medical context. This would reduce the doctors' cognitive load and response time to patients by facilitating them in making faster decisions, thus improving the overall quality of online medical services. Three different word-level text highlighting methodologies are implemented and evaluated. The first method uses TF-IDF scores directly to highlight important parts of the text. The second method is a combination of TF-IDF scores and the application of Local Interpretable Model-Agnostic Explanations to classification models. The third method uses neural networks directly to make predictions on whether or not a word should be highlighted. The results of our experiments show that the neural network approach is successful in highlighting medically-relevant terms and its performance is improved as the size of the input segment increases.
1. Introduction
The paper addresses information overload in telehealth by proposing word-level highlighting of patient messages to help doctors focus on important information. It compares TF-IDF, TF-IDF with LIME, and neural-network approaches, and studies highlighting across medical datasets.
1. Introduction
- The work focuses on text highlighting and extractive summarization for an online chat service connecting patients and doctors.The introduction situates this focus amid information overload and increasing telehealth demand.Source: §1 · pp. 2, 4
- The paper proposes highlighting important words and short segments in patient messages to help doctors read and respond to queries faster.The introduction frames this as a mechanism for supporting doctors’ responses to patients.Source: §1 · p. 5
- Three approaches are investigated: TF-IDF scoring, TF-IDF with LIME explanations, and neural networks that predict whether each word should be highlighted.LIME interprets models after classification, while TF-IDF indicates how much a datapoint supports a prediction.Source: §1 · p. 5
- The study presents itself as the first word-level medical-text highlighting study and releases one dataset to encourage further work.It also establishes a baseline and evaluates five models using three approaches on two medical text datasets.Source: §1 · pp. 5–6
2. Background
The background reviews extractive summarization, highlighting, and methods for selecting important text. It distinguishes prediction-based highlighting from approaches that score or retrieve text for summaries.
2. Background
- Extractive summarization selects text from the source, and is commonly sentence-level; sentence selection can be treated as information retrieval using scores.The passage notes that most extractive-summarization research focuses on sentence-level work.Source: §2 · p. 6
- TF-IDF is described as a statistic for reflecting a word’s or text segment’s importance to a document in a corpus.Laban et al. [15] use it to identify keywords for training a reinforcement-based abstractive summarization model.Source: §2 · p. 7
- Prior clinical-record methods include repeated-sentence, case-based, translation, and composite approaches to select summary sentences.The composite method combines the other three, normalizes scores, and selects top sentences.Source: §2 · pp. 7–8
- Domain-concept methods identify and score medical keywords, whereas this paper’s methods predict on text and highlight according to the prediction.The related work also describes concept frequency as a way to score sentences for biomedical summaries.Source: §2 · pp. 8–9
- Other reviewed approaches use deep learning to enhance features for extractive summaries or LIME to identify features indicative of a classification.The paper also uses LIME to extract words relevant to a patient’s condition after classifying a message.Source: §2 · pp. 9–10
3. Datasets
The study uses a medical-chat dataset and MTSamples transcriptions to develop and evaluate medical-text highlighting. The chat data supplies existing issue-category labels and manually annotated highlighting samples; the transcription set is evaluation-only.
3. Datasets
- The study uses a proprietary patient-doctor chat dataset and the open-source MTSamples transcription dataset.The MTSamples reports cover multiple medical specialties and work types.Source: §3 · p. 10, §3.2 · p. 12
3.1. Medical Chat Dataset
- The medical chat dataset contains 33,699 exchanged messages, and each conversation is tagged with an issue category derived from the patient’s questionnaire.The categories describe the type of patient issue and support deriving classification and highlighting datasets.Source: §3.1 · pp. 10–11
3.1.1. Medical Chat Classification Dataset
- The classification dataset uses existing issue-category labels to build a highlighting model without additional highlighting annotations.It omits pain management, which has few samples, and “other,” which contains many mislabeled samples.Source: §3.1.1 · p. 11
3.1.2. Medical Chat Highlighting Dataset
- The manually labeled highlighting subset comprises 350 randomly selected conversations, split into 300 training and 50 test conversations.Each word receives a binary highlighted/unhighlighted label; doctors’ guidelines require highlighting numbers, body parts, and patient gender, and test samples were annotated by all three annotators.Source: §3.1.2 · p. 12
3.2. MTSamples Highlighting Dataset
- MTSamples contributes 100 selected transcriptions for evaluation only, and the dataset is released to encourage further highlighting research.The original collection has over 2,300 transcriptions, but most consist almost entirely of medical terms; inter-annotator agreement is reported as Krippendorf’s alpha α = 0.85.Source: §3.2 · pp. 12–13
3.3. Data Preprocessing
- Preprocessing keeps stopwords because removing them lowers performance and prevents highlighting words such as “of” or “my.”Other steps include tokenization, removing URLs, lowercasing, and removing non-Unicode characters and line breaks.Source: §3.3 · pp. 13–14
4. Methodology
The methodology compares direct TF-IDF highlighting, LIME explanations of classification models, and neural models fine-tuned for word-level highlights.
4. Methodology
- The three approaches train TF-IDF directly, use classifier feature importances through LIME, or pretrain neural models on medical terms before fine-tuning them on highlighted chats.Five models are trained across these methodologies.Source: §4 · p. 14
4.1. TF-IDF
- TF-IDF scores use n-gram frequency in a document and inverse document frequency; conversation-level documents performed better than individual messages, while larger training subsets did not improve results.The model uses unigrams, bigrams, and trigrams, and conversation-level documents performed better mainly because individual messages can be very short.Source: §4.1 · pp. 15–16
4.2. LIME
- LIME explains predictions from trained classifiers using a local linear approximation, whose feature coefficients indicate sample-level importance.LR and SVM used TF-IDF n-grams, while LSTM used Word2Vec; LSTM explanations were omitted because importance was spread across more words, reducing highlighting performance.Source: §4.2 · pp. 16–17
4.3. LSTM
- The proposed three-layer Bidirectional LSTM predicts highlights for each word using pretrained PubMed embeddings and a contextual sequence representation.The model uses 256 LSTM cells, sigmoid activations, and dropout of 0.2; bidirectional cells process positive and negative time directions.Source: §4.3 · pp. 17–18
- The LSTM was trained either on one word at a time or on a full message, and medical-term pretraining was followed by fine-tuning on annotated chats.Medical-term sources included UMLS and ICD codes, alongside non-medical words; experiments also varied the number of annotated chat samples.Source: §4.3 · pp. 18–19
5. Results
Across Medical Chat Highlighting and MTSamples, LSTM models outperform the alternatives, with the N-gram LSTM slightly ahead of the unigram model on chat data. Visualizations show both successful highlighting and errors, while fine-tuning results favor the N-gram model except before chat-data fine-tuning.
5.1. Medical Text Highlighting Results
- Evaluation uses threshold-specific precision and recall alongside multi-threshold ROC-AUC and PR-AUC, which produce the same model ranking.The authors handpicked a reasonable threshold for each model; they also note PR-AUC is preferred when classes are severely imbalanced or the minority class is more important.Source: §5.1 · p. 20
- LSTM models outperform the alternatives on both datasets; on Medical Chat Highlighting, the N-gram LSTM performs slightly better than the unigram model, while on MTSamples LIME beats TF-IDF.Medical Chat evaluation uses patient messages only; TF-IDF and LIME miss dates and numbers because those features were not considered important by either method.Source: §5.1 · pp. 21–22, Table 1 · p. 22, Table 2 · p. 22
- Figures 3 and 4 show that the N-gram LSTM performs similarly well on Medical Chat and MTSamples.The figures present ROC and precision-recall curves.Source: §5.1 · p. 22, Figure 3 · p. 23, Figure 4 · p. 23
5.2. Visualizing the predictions
- The visualized predictions correctly identify gender, age, symptoms, and drug information, but Figure 6 misses some date-related information and adjectives.The authors observed inconsistent labeling of phrases such as “last week” in the training data and note that further training-set verification can improve model performance.Source: §5.2 · p. 24, Figure 5 · p. 24, Figure 6 · p. 25
5.3. Fine-tuning LSTM models
- The N-gram LSTM outperforms the unigram LSTM at every fine-tuning level except the medical-terms-only starting point.Without fine-tuning on Medical Chat Highlighting, the models perform significantly worse because the Medical Terms dataset contains terms, not full chat messages.Source: §5.3 · pp. 25–26
6. Conclusion and Future Work
The study explores word-level highlighting of medical text for telehealth, reporting that its n-gram LSTM highlights important information with high precision and recall. Future work includes evaluating the models with doctors and exploring active learning for dataset labeling.
6. Conclusion and Future Work
- The study explores word-level medical text highlighting for telehealth and describes it as the first such study, proposing two new methodologies.The approaches leverage large medical term corpora, relevant metadata, and manually labelled samples.Source: §6 · pp. 26–27
- Figure 7 shows PR-AUC increasing as LSTM models are fine-tuned with more annotated data.Source: Figure 7 · p. 27
- The n-gram LSTM successfully highlights important information in medical texts, with high precision and recall.The reported result is based on evaluation of five highlighting models on two medical text datasets.Source: §6 · p. 27
- Evaluating the models with telehealth users, including doctors, is identified as future research.The authors propose this evaluation to gain insights into designing word-level highlighting for telehealth applications.Source: §6 · p. 27
- The authors identify active learning as another future direction for improving dataset preparation and labeling.They also suggest using it to select dataset samples more carefully for labeling.Source: §6 · p. 28