Source-linked AI summary
HunFlair: An Easy-to-Use Tool for State-of-the-Art Biomedical Named Entity Recognition
Leon Weber, Mario Sänger, Jannes Münchmeyer, Maryam Habibi, Ulf Leser, Alan Akbik
TL;DR
Biomedical NER tools are often evaluated on small datasets even though they are applied to varied texts, and existing tools can be difficult to integrate. HunFlair combines pretrained biomedical representations and multiple corpora in Flair; it generally outperforms standalone tools on unseen corpora, with a 7.26 pp average F1 gain. It is designed for easy use and provides 23 consistently formatted corpora for further research.
Problem
NER tools are commonly trained and evaluated on small gold-standard datasets, although real-world texts vary in topic, entity distribution, genre, and type, a mismatch that can cause severe performance drops.
Source: §1 · p. 1Method
HunFlair provides Flair-integrated models for five entity types, using pretrained biomedical language representations and entity-specific training on multiple gold-standard corpora.
Source: §1 · p. 1, §2 · p. 2, §1.2 · p. 6Results
7.26 pp average F1 gain; HunFlair outperforms standalone competitors in all but one comparison on three unseen corpora, and sets a new state of the art on one corpus while performing on par on others.
Source: §3.1 · p. 3, §3.2 · p. 3Takeaways & Limitations
HunFlair is easy to install and use, and its 23 consistently formatted biomedical NER corpora support researchers and practitioners in training models and exploring new approaches.
Source: §1 · p. 2, §4 · p. 3For Gene on CRAFT, tokenized-output alignment loses non-ASCII symbols and whitespace, and fuzzy matching is used to align predicted entities with the original text.
Source: §2 · p. 9
Abstract
from arXiv · showhide
Summary: Named Entity Recognition (NER) is an important step in biomedical information extraction pipelines. Tools for NER should be easy to use, cover multiple entity types, highly accurate, and robust towards variations in text genre and style. To this end, we propose HunFlair, an NER tagger covering multiple entity types integrated into the widely used NLP framework Flair. HunFlair outperforms other state-of-the-art standalone NER tools with an average gain of 7.26 pp over the next best tool, can be installed with a single command and is applied with only four lines of code. Availability: HunFlair is freely available through the Flair framework under an MIT license: https://github.com/flairNLP/flair and is compatible with all major operating systems. Contact:{weberple,saengema,alan.akbik}@informatik.hu-berlin.de
1 Introduction
Biomedical NER tools face a mismatch between small evaluation datasets and varied real-world texts, while HUNER is difficult to integrate into major NLP frameworks. HunFlair brings five entity-type models into Flair, combining broad corpus training with simple installation and use.
1 Introduction
- NER tools are often trained and evaluated on small gold-standard datasets but applied to texts that vary in topic, entity distribution, genre, and type.This mismatch can lead to severe performance drops.Source: §1 · p. 1
- HunFlair provides Flair-based NER models for Cell Line, Chemical, Disease, Gene, and Species.The models combine character-level pretraining with joint training on multiple gold-standard corpora.Source: §1 · p. 1
- The integrated tool can be installed with one command and applied with a few lines of code, while its consistently formatted 23 corpora support users training their own models.Figure 1 also provides an overview of installation and a sample application with annotated output.Source: §1 · p. 2, Figure 1 · p. 2
2 HunFlair
HunFlair is built within Flair, which supports mixing character, word, and document embeddings. Its entity-specific BiLSTM-CRF models use pretrained biomedical representations and training data pooled across matching corpora.
2 HunFlair
- Flair supports intuitive training and distribution of sequence-labeling, text-classification, and language models, with character, word, and document embeddings that can be mixed and matched.The framework is reported to achieve state-of-the-art performance in several NLP research challenges.Source: §2 · p. 2
- HunFlair uses a Flair character-level language model pretrained on roughly 24 million PubMed abstracts and 3 million PMC full texts, together with fastText word embeddings.Named-entity prediction uses a BiLSTM-CRF model.Source: §2 · p. 2
- HunFlair trains a distinct model for each entity type on the union of training sets from gold-standard corpora annotated for that type.The stated aim is to improve performance across text genres and biomedical subdomains.Source: §2 · p. 2
3 Results
HunFlair generally outperforms standalone biomedical NER tools on unseen corpora and is competitive with research prototypes. Pretraining across multiple gold-standard corpora improves F1 across all entity types.
3 Results
- The evaluation compares HunFlair with off-the-shelf biomedical NER tools and research prototypes, which differ in their intended balance of ease of use and performance.The prototype comparisons are based on reported scores.Source: §3 · p. 2, §3.2 · p. 3
3.1 Comparison to off-the-shelf NER tools
- 7.26 pp average F1 gain; HunFlair outperforms standalone competitors in all but one comparison on three unseen corpora.None of the evaluation corpora was used to train HunFlair or the competitor tools.Source: §3.1 · p. 3
3.2 Comparison to research prototypes
- HunFlair sets a new state of the art on one corpus and performs on par with reported research-prototype scores on the others.The comparison covers BioBERT, SciBERT, CollaboNet, and SciSpacy.Source: §3.2 · p. 3
- 0.8 pp for chemicals to 4.75 pp for cell lines; finetuning on multiple gold-standard corpora improves F1 across all entity types.The evaluation compares pretrained and nonpretrained HunFlair on all 23 NER corpora.Source: §3.2 · p. 3
4 Conclusion
The authors conclude that HunFlair is a state-of-the-art biomedical NER tagger that performs strongly on unseen corpora. Its pretrained models, usability, and bundled corpora support future research.
4 Conclusion
- HunFlair is presented as a state-of-the-art biomedical NER tagger built on pretrained domain-specific language models and performing well on unseen corpora.The conclusion describes its performance advantage over other tools as often large.Source: §4 · p. 3
- HunFlair is easy to install and use, and includes 23 biomedical NER corpora in a single format to support future research.Table 1 reports F1 scores for several tools across three unseen corpora and the Chemical, Disease, Gene, and Species entity types.Source: §4 · p. 3, Table 1 · p. 3
- Table 1 marks the best results in bold for each reported comparison.Its F1-score scope covers multiple off-the-shelf tools, three unseen corpora, and four entity types.Source: Table 1 · p. 3
Funding
The authors acknowledge funding from HEIBriDS and the German Research Council.
Funding
- Leon Weber and Jannes Münchmeyer were funded by HEIBriDS, while Maryam Habibi was funded by the German Research Council.The German Research Council funding was grant LE-1428/7-1.Source: Funding · p. 3
1 Training of HunFlair
HunFlair training combines large-corpus embeddings with entity-specific taggers trained on harmonized biomedical NER corpora. Its training data span varied biomedical text genres, while the tagger uses a BiLSTM-CRF sequence-labeling architecture.
1 Training of HunFlair
- Training proceeds in two stages: first train word embeddings on unlabeled text, then train the NER tagger on manually labeled corpora.HunFlair uses character-level Flair embeddings and fastText embeddings.Source: §1 · p. 5, §1.1 · p. 5
1.1 Embeddings
- Flair and fastText embeddings are trained on roughly 14 billion biomedical tokens, with Wikipedia adding another 2.6 billion tokens for fastText.The biomedical corpus comprises roughly 3 million PMC full texts and 25 million PubMed abstracts.Source: §1.1 · p. 5
1.2 Gold standard NER pre-training
- 23 manually curated biomedical NER corpora spanning patents, abstracts, and full texts are harmonized for entity-specific model training.Each model uses corpora annotated for its entity type; training and validation sets combine the corresponding HUNER splits.Source: §1.2 · pp. 6–7
- A BiLSTM-CRF tagger represents words with HunFlair language-model and fastText embeddings, then predicts one IOBES label per word.The tagger uses a single-layer Bi-LSTM with a hidden size of 256 and a CRF output layer.Source: §1.2 · p. 6
2 Evaluation against off-the-shelf tools
HunFlair is evaluated against biomedical NER tools on held-out corpora, with scoring designed to accommodate differences in tool preprocessing. It performs better than competitors except for Species on BioNLP under the reported protocol, while entity-boundary and offset handling affect comparisons.
2 Evaluation against off-the-shelf tools
- Evaluation uses CRAFT, BioNLP13 Cancer Genetics, and PDR, none of which were used to train HunFlair or its competitors.This approximates application to completely unseen text.Source: §2 · p. 8
- Models receive complete documents and generate text-offset predictions, which are matched to gold offsets with a one-character tolerance at either span boundary.F1 is reported; the tolerance accommodates small offset differences from preprocessing special characters.Source: §2 · p. 8
- HUNER’s tokenized output loses non-ASCII symbols and whitespace, complicating alignment; any-overlap matching narrows the reported Gene-on-CRAFT difference from Weber et al. (2019).The evaluation protocol also differs from Weber et al. (2019), which used homogeneously preprocessed corpora.Source: §2 · pp. 8–9
- HunFlair outperforms all competitors except for Species on BioNLP under the reported evaluation protocol.SciSpacy boundary errors produce strikingly different scores under strict versus any-overlap matching.Source: §2 · p. 9
3 Evaluation against state-of-the-art models
HunFlair is compared with reported state-of-the-art results on JNLPBA, NCBI Disease, and CDR. It sets a new state of the art on one corpus and performs on par on the others.
3 Evaluation against state-of-the-art models
- HunFlair sets a new state of the art on one corpus and performs on par with other state-of-the-art models on the others.The comparison covers JNLPBA (Gene annotations only), NCBI Disease, and CDR; scores are macro-averaged F1.Source: §3 · pp. 9–10
- For this comparison, the authors use large BioWordVec embeddings and remove the three evaluation corpora from the pretraining set.The baselines include BioBERT, SciBERT, CollaboNet, and SciSpacy.Source: §3 · p. 9
4 Effects of pretraining
Pretraining on multiple gold-standard corpora improves average results across all five entity types, chiefly through higher recall, though some cases show lower F1 or precision.
4 Effects of pretraining
- The experiment compares the pretrained tagger with a randomly initialized LSTM that still uses pretrained Flair and fastText embeddings.It uses large BioWordVec embeddings and excludes test portions of the corpora from pretraining.Source: §4 · p. 10
- 0.8 pp for chemicals to 4.75 pp for cell lines: pretraining raises average results across all entity types.The comparison is between the pretrained and vanilla taggers; Table SM 5 reports precision, recall, F1, and their gains.Source: §4 · pp. 10–11
- 28 of 34 cases have higher recall with pretraining, and recall improves by over 4.0 pp in eight cases.The authors attribute most performance improvements to better recall and say this indicates better coverage of entities and surface forms, plus adaptability to other biomedical subdomains.Source: §4 · p. 10
- Six cases have slightly lower F1, by at most 1.05 pp; five of those also have lower precision.Precision is lower in ten cases where F1 increases as well; the authors suggest additional training entities may occasionally produce imprecise predictions.Source: §4 · p. 10