PaperlayerView exact revision on arXiv

Source-linked AI summary

Launching into clinical space with medspaCy: a new clinical text processing toolkit in Python

Hannah Eyre, Alec B Chapman, Kelly S Peterson, Jianlin Shi, Patrick R Alba, Makoto M Jones, Tamara L Box, Scott L DuVall, Olga V Patterson

arXiv:2106.07799v1cs.CL

TL;DR

Clinical NLP projects face labor-intensive integration of isolated modules, and existing clinical tools provide limited support for rule-based development. medspaCy adds clinical-text components to spaCy, and its authors aim to reduce development burden for projects using rules or combining them with other NLP methods.

  • Problem

    Clinical NLP code reuse is difficult because connecting isolated modules into cohesive pipelines is labor-intensive, and existing clinical toolkits provide limited rule-based support.

    Source: Introduction · pp. 1–2

  • Method

    medspaCy is a spaCy-based library of clinical text components designed to integrate rule-based and statistical methods.

    Source: Introduction · p. 2

  • Results

    The authors aim to improve cNLP development productivity and reduce development burden for projects using rule-based NLP or integrating rules with other methods.

    Source: Conclusion · p. 7

  • Takeaways & Limitations

    medspaCy capabilities supported analysis of 9,040 open-ended suicide-prevention survey responses for a task force report due within three months.

    Source: Veteran Suicide Risk and Intervention… · p. 7

    UMLS licensing prevents redistribution of its databases; Windows QuickUMLS setup requires Anaconda and additional installation steps.

    Source: UMLS Mapping · p. 4

Abstract

from arXiv · show

Despite impressive success of machine learning algorithms in clinical natural language processing (cNLP), rule-based approaches still have a prominent role. In this paper, we introduce medspaCy, an extensible, open-source cNLP library based on spaCy framework that allows flexible integration of rule-based and machine learning-based algorithms adapted to clinical text. MedspaCy includes a variety of components that meet common cNLP needs such as context analysis and mapping to standard terminologies. By utilizing spaCy's clear and easy-to-use conventions, medspaCy enables development of custom pipelines that integrate easily with other spaCy-based modules. Our toolkit includes several core components and facilitates rapid development of pipelines for clinical text.

Introduction

Clinical NLP development has faced labor-intensive integration of isolated modules, while existing clinical tools provide limited support for rule-based methods. medspaCy addresses this need with a spaCy-based library of clinical text processing components.

Introduction

  • Connecting isolated modules into cohesive clinical NLP pipelines remains labor-intensive, challenging code reuse across projects.Source: Introduction · p. 1
  • NLTK’s list-of-strings approach and lack of a unifying architecture have hindered its ability to scale to large datasets.Source: Introduction · p. 1
  • spaCy supports modular pipelines that integrate rule-based and statistical methods with custom components for specialized tasks.Source: Introduction · p. 1
  • Clinical-language differences limit scispaCy’s performance on clinical narratives, while scispaCy and MedaCy lack adequate rule-based development support beyond spaCy’s native functionality.Source: Introduction · p. 2
  • medspaCy is a spaCy-based library of core components designed to combine statistical and symbolic methods for clinical text processing.Source: Introduction · p. 2

Methods

medspaCy builds on spaCy’s API, keeping its components usable alone or as modules in larger pipelines. It offers defaults and customization while exposing component-generated information through spaCy attributes.

Framework Overview

  • medspaCy minimally extends spaCy’s API so components can run standalone or within larger pipelines that include modules from other sources.Source: Framework Overview · p. 2
  • Component outputs are exposed through custom attributes registered at the Doc, Span, or Token level for later analysis.Source: Framework Overview · p. 2
  • Default initialization supports quick prototyping, while optional parameters and shareable resources support customization and reuse across projects.Source: Framework Overview · p. 2

MedspaCy Components

medspaCy brings together components for end-to-end clinical text processing, including custom tokenization and rule-based sentence detection adapted to clinical documentation.

MedspaCy Components

Tokenization

  • The clinical tokenizer adds rules for punctuation and inconsistent whitespace common in clinical notes while retaining spaCy’s non-destructive text handling.Source: Tokenization · p. 2

Sentence Detection

  • PyRuSH provides rule-based clinical sentence detection that handles templated text, concept-value statements, inconsistent punctuation, and meaningful whitespace boundaries.Many downstream components operate at the sentence level, and detectors trained on non-clinical text typically perform poorly on clinical data.Source: Sentence Detection · p. 3

Section Detection

Clinical documents have structured, document-specific sections whose meaning depends on where statements appear. medspaCy detects section titles and can preserve nested structure and modify entity attributes by section.

Section Detection

  • Clinical document types have distinct structures and section schemes, and the same statement can have different meanings depending on its section.Source: Section Detection · p. 4
  • medspaCy detects clinical sections by rule-based matching of section titles, using default rules adapted from SecTag and expanded through practice.Source: Section Detection · p. 4
  • The sectionizer can change entity attributes based on section context and preserve nested sections as a traversable hierarchy.Source: Section Detection · p. 4

Concept Extraction

MedspaCy extends spaCy rule-based concept extraction with clinical pattern-matching features and adapts QuickUMLS for UMLS concept mapping. UMLS licensing constrains the resources distributed with the toolkit.

Concept Extraction

  • Concept rules specify match patterns, semantic categories, and metadata, with multi-token regular expressions and direct entity attributes such as temporality or vocabulary codes.The additional regular-expression features also support compatibility with existing cNLP resources and knowledge bases.Source: Concept Extraction · p. 4

UMLS Mapping

  • QuickUMLS was adapted as a spaCy component for efficient approximate dictionary matching to UMLS concepts.Its matching leverages SimString; the paper cites comparable speed and accuracy to other existing systems.Source: UMLS Mapping · p. 4
  • UMLS licensing prevents database redistribution, so medspaCy ships resources generated from a publicly available sample and documents how to create QuickUMLS resources after installation.Windows setup requires Anaconda and additional steps; Linux and MacOS can use the provided sample immediately after installation.Source: UMLS Mapping · p. 4

Contextual Analysis

MedspaCy adapts ConText to spaCy for assigning contextual attributes to clinical concepts, with customizable rules and utilities for pipeline deployment and output analysis.

Contextual Analysis

  • ConText links clinical entities to linguistic modifiers, typically within a sentence, to assign attributes such as negation, temporality, certainty, and experiencer.MedspaCy implements default rules that can be customized, including modifier directionality and linking logic.Source: Contextual Analysis · pp. 4–5
  • Users can customize ConText rules and attributes, including adding use-case-specific semantic categories beyond defaults such as negation.Rules support pattern matching and user-defined modifier properties.Source: Contextual Analysis · p. 5

Utilities

  • MedspaCy utilities transform methods into spaCy-compatible components and convert processed Docs into tabular formats such as Pandas, SQL, or CSV.The conversion supports attributes and spans for entities, sections, and context.Source: Utilities · p. 5
  • A pipeline controller supports large-scale deployment by batching input and output documents and handling standard database operations through pyodbc.It can create tables and write queries for selected columns.Source: Utilities · p. 5

Visualization

MedspaCy adapts spaCy visualizations to display extracted clinical information and links between entities and contextual modifiers.

Visualization

  • One visualization highlights extracted entities, contextual modifiers, and section titles in the text.Figure 3 presents an example using highlights.Source: Visualization · p. 5, Figure 3 · p. 5
  • A second visualization uses directed arrows to show links between entities and their ConText modifiers.Figure 3 also illustrates context arrows.Source: Visualization · p. 5, Figure 3 · p. 5
  • The visualizations can present final model output or explain model logic, while intermediate displays can assist debugging during development.The paper notes rapid update-test-inspect cycles in interactive settings such as Jupyter Notebook.Source: Visualization · p. 5

Use Cases

VA operational and research projects used medspaCy, including symptom-based biosurveillance and COVID-19 surveillance. The COVID-19 testing system supported rapid deployment and identified patients beyond those found through other methods.

Use Cases

  • MedspaCy-supported rapid development was essential to several VA projects aiding the COVID-19 pandemic response.The toolkit was used in multiple operational and research-oriented VA projects.Source: Use Cases · p. 5

Chief Complaint Surveillance

  • Since early 2019, a VA biosurveillance pipeline has processed emergency-department triage symptoms into UMLS concepts and was used to identify potential patients before SARS-CoV-2 testing or the term COVID-19 existed.It preprocesses text with selected EMT-P regular expressions and maps spans to standard concepts using QuickUMLS.Source: Chief Complaint Surveillance · pp. 5–6

COVID-19 Surveillance

  • Over 36% of all identified COVID-positive patients were identified by this method alone; evaluation reported precision of 82.5% and recall of 94.2%.The system processed documents for over 3 million patients between January 1 and June 15, 2020.Source: COVID-19 Surveillance · p. 6
  • Iterative development permitted deployment by January 21, 2020, and the system remained operational throughout the pandemic, processing over 63 million clinical documents one year after initial deployment.Terminology changes required adding concepts and patterns nearly every day.Source: COVID-19 Surveillance · p. 6

VA COVID-19 Screening

VA pipelines extract structured information from COVID-19 screenings and inpatient nursing assessments. The screening pipeline processed millions of documents, while a sectionizer turns templated questions and answers into headers and bodies.

VA COVID-19 Screening

  • Standardized COVID-19 screenings became frequent VA EMR documents in 2020, with some inpatients receiving multiple screenings daily.Source: VA COVID-19 Screening · p. 6
  • The pipeline identifies screenings as standalone documents or subsections, then extracts up to 17 questions and associated responses.Extracted topics include exposures, prior tests or diagnoses, and symptoms.Source: VA COVID-19 Screening · p. 6
  • 6.8 million screening documents for 914,000 Veterans who received a COVID-19 test have been processed, and the dataset is available to VA researchers.The data are available through the VA COVID-19 Shared Data Resource.Source: VA COVID-19 Screening · p. 6

Inpatient Nursing Templated Text Processing

  • Rules generated from empty templates let the sectionizer identify each nursing question as a header and capture its answers as the section body.Figure 4 depicts highlighted headers as template questions and the body text as responses.Source: Inpatient Nursing Templated Text… · p. 6, Figure 4 · p. 6

Veteran Suicide Risk and Intervention Analyses

MedspaCy supported a rapid analysis of suicide-prevention responses, a developing housing-stability pipeline, and hands-on clinical NLP education. These applications span completed analysis, planned use, and teaching.

Veteran Suicide Risk and Intervention Analyses

  • 9,040 open-ended responses were analyzed and summarized within three months, and the insights informed development of the PREVENTS task force report.The work used iterative rule development, exploratory data analysis, visualization, and UMLS knowledge resources.Source: Veteran Suicide Risk and Intervention… · p. 7

Homelessness and Housing Stability Identification

  • A developing pipeline extracts homelessness and housing-stability mentions, aggregates them over time, and is intended to evaluate SSVF outcomes and identify at-risk individuals.Source: Homelessness and Housing Stability… · p. 7

Education

  • MedspaCy is used in clinical data science teaching, where students create rules and modify components to build systems for processing clinical documentation.Its simple installation and common interface let students with little programming experience apply NLP to clinical data.Source: Education · p. 7

Future work

The authors aim for medspaCy to improve cNLP productivity and reduce development burden. Future directions include task-specific pipelines and additional machine-learning utilities.

Future work

  • Future work could release task-specific medspaCy pipelines with components, resources, and pretrained models for clinical NLP tasks.Potential sources include public resources, shared tasks, and previously developed ontologies.Source: Future work · p. 7
  • Future work could add machine-learning utilities, including default-pipeline word embeddings or text feature-extraction tools.These additions would let users more fully leverage both statistical and rule-based cNLP.Source: Future work · p. 7

Conclusion

  • The authors aim to improve cNLP productivity and reduce development burden for projects using rule-based NLP or integrating rules with other methods.Source: Conclusion · p. 7
Loading 2106.07799v1…