Source-linked AI summary
NegBio: a high-performance tool for negation and uncertainty detection in radiology reports
Yifan Peng, Xiaosong Wang, Le Lu, Mohammadhadi Bagheri, Ronald Summers, Zhiyong Lu
TL;DR
NegBio addresses the difficulty of distinguishing negative and uncertain findings from positive findings in radiology information extraction. It uses universal-dependency patterns and subgraph matching to determine trigger scope, and it significantly outperformed NegEx across the evaluated datasets. The authors identify double negation and incomplete annotation of negations and uncertainties as important boundaries.
Problem
Negative and uncertain findings are frequent in radiology reports, but distinguishing them from positive findings remains challenging for information extraction.
Method
NegBio applies dependency-based rules and subgraph matching to identify the syntactic scope of negation and uncertainty around medical findings.
Results
NegBio achieved significant improvement over NegEx on all evaluated datasets.
Takeaways & Limitations
NegBio can improve information-extraction precision by removing negative and uncertain findings from extracted positive findings.
Takeaways & Limitations
NegBio misses double negation, producing more false negatives, and two radiology corpora lack complete annotations for negative and uncertain findings.
Abstract
from arXiv · showhide
Negative and uncertain medical findings are frequent in radiology reports, but discriminating them from positive findings remains challenging for information extraction. Here, we propose a new algorithm, NegBio, to detect negative and uncertain findings in radiology reports. Unlike previous rule-based methods, NegBio utilizes patterns on universal dependencies to identify the scope of triggers that are indicative of negation or uncertainty. We evaluated NegBio on four datasets, including two public benchmarking corpora of radiology reports, a new radiology corpus that we annotated for this work, and a public corpus of general clinical texts. Evaluation on these datasets demonstrates that NegBio is highly accurate for detecting negative and uncertain findings and compares favorably to a widely-used state-of-the-art system NegEx (an average of 9.5% improvement in precision and 5.1% in F1-score).
Methods
NegBio is presented as a pipeline for processing radiology reports, with the NegBio component applied after medical findings are pre-tagged. Figure 1 summarizes this overall workflow.
- NegBio takes sentences with pre-tagged medical-finding mentions as input and checks whether each finding is negative or uncertain.
- When MetaMap is used alone, the system skips NegBio and produces labels directly.
- Figure 1 depicts the overall NegBio pipeline.
Medical findings recognition
The approach labels reports in two passes: identifying medical findings and UMLS concepts, then removing findings classified as negative or uncertain. It focuses on 14 common disease-finding types selected by radiologists.
- Reports are processed in two passes: finding and UMLS concept detection followed by NegBio filtering.
- MetaMap identifies findings and their corresponding UMLS concepts in the first pass.
- The system focuses on 14 common disease-finding types selected by radiologists from a clinical perspective.
- NegBio is applied to identified findings to rule out negative and uncertain findings.
Universal dependency graph construction
NegBio constructs universal dependency graphs from parsed report sentences, representing words and grammatical relations as labeled vertices and directed edges. The graph captures dependency structure for subsequent pattern matching.
- A universal dependency graph represents grammatical relationships in a sentence as a directed graph.
- Graph vertices carry word, part-of-speech, and lemma information.
- Graph edges encode typed dependencies from governors to dependents, including nsubj and conj relations.
- In the example graph, “Lung” is the subject and “acute infiltrates” and “pleural effusion” are coordinating findings.
- Each report is sentence-split and tokenized with NLTK, parsed with a biomedical Bllip model, and converted to universal dependencies using Stanford dependencies.
Negation and uncertainty detection
NegBio defines negation and uncertainty rules over universal dependency graphs and searches for matching chains or subgraphs around finding mentions. This makes scope depend on syntactic context rather than fixed word distance.
- NegBio searches from a finding mention’s head word and labels the finding negative or uncertain when it matches a predefined dependency pattern.
- Rules can be dependency chains or subgraphs, enabling matches for constructions such as “no evidence of” and “not excluded”.
- Subgraph matching makes negation and uncertainty scope depend on syntactic context rather than word distance.
- The scope comprises all vertices covered by the matched subgraph.
- The released source code contains the complete set of rules.
Evaluations on findings detection in an end-to-end system
NegBio was evaluated in an end-to-end system that extracts only positively asserted findings, using OpenI and ChestX-ray datasets. It improved performance over MetaMap and NegEx, including higher precision and F1-score on both datasets.
- Evaluation design: The evaluation compared MetaMap alone, MetaMap with NegEx, and MetaMap with NegBio, using precision, recall, and F1-score for positive findings.
- End-to-end evaluation: Precision increased from 13.8% to 77.2% when negation and uncertainty detection was added to the extraction system.
- End-to-end evaluation: F1-score increased from 23.8% to 80.7% after adding negation and uncertainty detection.
- OpenI: 89.8% precision, 85.0% recall, and 87.3% F1-score were achieved by NegBio on OpenI.
Experiments on negation detection
NegBio was additionally tested for negation detection on BioScope and PK, covering radiology reports and general clinical text. It outperformed NegEx, with especially large gains on BioScope and smaller gains on PK.
- Evaluation design: The negation experiments used corpora with annotated negated expressions and evaluated NegBio against NegEx.
- Datasets: BioScope contained 977 radiology reports with 466 annotated negative scopes, while PK contained 116 general clinical documents.
- BioScope: 25.5% higher precision and 13.6% higher F1-score were achieved by NegBio than by NegEx on BioScope.
- PK: 2.7% higher precision and 0.2% higher F1-score were achieved by NegBio than by NegEx on PK.
Discussion
NegBio improved performance over NegEx across datasets by using syntactic structures to constrain negation and uncertainty scopes, while remaining limited by recognition, parsing, double-negation, and annotation challenges.
- NegBio achieved significant improvement over NegEx on all evaluated datasets.
- Syntactic rules improved precision by preventing regular-expression scopes from overextending across complex sentence structures.NegBio correctly preserved positive findings that NegEx incorrectly marked negative because of the broad “not .*” pattern.
- Uncertainty detection contributed to higher precision, with disabling its rules reducing F-score by 7.4% on OpenI and 2.5% on ChestX-ray.
- NegBio recalls were comparable to NegEx for document-level positive-finding detection, while recall gains were limited by small datasets and restricted text variation.
- Errors arose from imperfect named-entity recognition, inaccurate parsing of long noun phrases, and missed double negation.On OpenI, 30% of “Nodule” mentions were not correctly recognized; double negation remains an open challenge.
- Because OpenI and ChestX-ray annotate only positive findings, all negation and uncertainty cases could not be evaluated, leaving full annotation as future work.
Conclusion
The paper proposes NegBio for detecting negative and uncertain findings in radiology reports and evaluates it on public and newly constructed corpora. It reports significant improvement over the state of the art and presents the open-source tool as useful for healthcare informatics research and applications.
- NegBio determines negative and uncertain findings in radiology reports and can improve information-extraction precision.
- The evaluation used two publicly available corpora and a newly constructed corpus.
- NegBio achieved significant improvement over the state of the art on all datasets.
- As an open-source tool, NegBio may contribute to healthcare informatics research and real-world applications.