PaperlayerView exact revision on arXiv

Source-linked AI summary

CollaboNet: collaboration of deep neural networks for biomedical named entity recognition

Wonjin Yoon, Chan Ho So, Jinhyuk Lee, Jaewoo Kang

arXiv:1809.07950v2cs.CLcs.LG

TL;DR

BioNER datasets cover limited subsets of entity types, and biomedical terms can be polysemous. CollaboNet connects entity-type specialist models so they can share predictions; it achieved state-of-the-art results across four BioNER datasets while reducing false positives and entity-type errors.

  • Problem

    BioNER datasets often annotate only a small subset of entity types, and polysemous biomedical terms can be mislabeled across types.

    Source: Background · p. 2

  • Method

    CollaboNet connects single-task NER models trained on different entity-type datasets, using collaborator outputs as additional input while models take turns as target and collaborator.

    Source: Background · p. 2, CollaboNet · p. 5

  • Results

    CollaboNet achieved state-of-the-art F1, precision, and recall on four BioNER datasets and reduced false positives, especially false biomedical entities.

    Source: Discussion · p. 8, Conclusion · p. 10

  • Takeaways & Limitations

    BioNER is a building block for downstream relation extraction and biomedical entity search.

    Source: Background · p. 1

    Exact-match evaluation makes results sensitive to inconsistent modifier boundaries, which can count boundary variants as false positives and false negatives.

    Source: Discussion · p. 8

Abstract

from arXiv · show

Background: Finding biomedical named entities is one of the most essential tasks in biomedical text mining. Recently, deep learning-based approaches have been applied to biomedical named entity recognition (BioNER) and showed promising results. However, as deep learning approaches need an abundant amount of training data, a lack of data can hinder performance. BioNER datasets are scarce resources and each dataset covers only a small subset of entity types. Furthermore, many bio entities are polysemous, which is one of the major obstacles in named entity recognition. Results: To address the lack of data and the entity type misclassification problem, we propose CollaboNet which utilizes a combination of multiple NER models. In CollaboNet, models trained on a different dataset are connected to each other so that a target model obtains information from other collaborator models to reduce false positives. Every model is an expert on their target entity type and takes turns serving as a target and a collaborator model during training time. The experimental results show that CollaboNet can be used to greatly reduce the number of false positives and misclassified entities including polysemous words. CollaboNet achieved state-of-the-art performance in terms of precision, recall and F1 score. Conclusions: We demonstrated the benefits of combining multiple models for BioNER. Our model has successfully reduced the number of misclassified entities and improved the performance by leveraging multiple datasets annotated for different entity types. Given the state-of-the-art performance of our model, we believe that CollaboNet can improve the accuracy of downstream biomedical text mining applications such as bio-entity relation extraction.

Background

BioNER faces limited, entity-type-specific training data and ambiguity when biomedical terms can denote different entity types. CollaboNet combines entity-type experts so they can share information and reduce false positives.

Background

  • BioNER supports downstream biomedical text-mining applications, including relation extraction and complex entity search.Examples include drug-drug interactions and disease-treatment relations.Source: Background · p. 1
  • Dictionary- and rule-based approaches can achieve high precision but struggle with unseen entities, polysemy, or the manual feature and maintenance demands described for those methods.Deep learning approaches, including BiLSTM-CRF models with character-level embeddings, have shown promising BioNER results.Source: Background · p. 2
  • Gold-standard BioNER datasets often annotate only one or two entity types, leaving each type with a small portion of the available annotations.NCBI covers diseases, whereas JNLPBA covers genes and proteins.Source: Background · p. 2
  • Multi-task models can gain recall through broader entity coverage, but combining entity types can make them harder to distinguish and lower precision.Prior MTL work leveraged datasets for related tasks and achieved performance comparable to state-of-the-art single-task models.Source: Background · p. 2
  • Polysemous terms such as BRCA1 and VHL can be mislabeled as diseases when used as gene names, increasing false positives.The same term may denote different entity types depending on textual context.Source: Background · p. 2
  • CollaboNet connects single-task models trained on different entity-type datasets, letting each expert use collaborator outputs to inform its predictions.Models take turns as target and collaborator, sharing multi-domain information to address false positives from polysemy.Source: Background · p. 2

Methods

The methods define a sequence-labeling task and build a BiLSTM-CRF model from biomedical word and character-level representations. CollaboNet’s component models use recurrent sequence representations for biomedical NER.

Methods

  • The methods overview presents a BiLSTM-CRF model for BioNER and introduces CollaboNet as a set of such models.The figures depict the single-task model and CollaboNet structure.Source: Methods · p. 3, Figure 1 · p. 3, Figure 2 · p. 4

Problem Definition

  • The task maps an input word sequence to corresponding labels, represented with the BIOES scheme.BIOES labels mark beginnings, insides, ends, outside tokens, and single-token entities.Source: Problem Definition · p. 3

Word Embedding (WE)

  • Each word is represented with a word embedding, using biomedical-corpus embeddings that prior work found useful for BioNER.The paper denotes the embedding for word w_t as x_t in R^d_word.Source: Word Embedding (WE) · p. 3

Character Level Word Embedding (CLWE)

  • A CNN derives character-level word embeddings to add morphological information, and the model concatenates these with biomedical word embeddings.The CNN applies convolution over character windows and takes element-wise maximum values for the character-level representation.Source: Character Level Word Embedding (CLWE) · pp. 3–4

Long Short-Term Memory (LSTM)

  • Forward and backward LSTMs encode the embedding sequence, and their states are concatenated to form BiLSTM representations.The resulting bidirectional states are used in the BiLSTM-CRF architecture.Source: Long Short-Term Memory (LSTM) · p. 4

Bidirectional LSTM with Conditional Random Field (BiLSTM-CRF)

CollaboNet combines dataset-specific BiLSTM-CRF experts that exchange information during training. A BiLSTM-CRF models label sequences using recurrent representations and a CRF layer.

Bidirectional LSTM with Conditional Random Field (BiLSTM-CRF)

CollaboNet

  • CollaboNet trains multiple BiLSTM-CRF single-task models as entity-type experts, transferring their knowledge to other experts.Each model is trained on a specific dataset.Source: CollaboNet · p. 5
  • Each expert is first trained independently until its performance converges, then serves in turn as a target or collaborator.During a target phase, collaborator models generate inferences but are not trained.Source: CollaboNet · p. 5
  • Weighted max pooling aggregates collaborator outputs, with trainable weights, to provide information for training the target model.The target model is trained for one epoch before the target changes.Source: CollaboNet · p. 5

Experiments

The experiments used six biomedical datasets, exact-match entity evaluation, and selected embedding, optimizer, and model settings. A post-processing step correcting invalid BIOES sequences improved precision and F1.

Datasets

  • The six reported datasets cover disease, chemical, and gene/protein entities and were constructed from MEDLINE abstracts.The study used BC2GM, BC4CHEMD, BC5CDR-chem, BC5CDR-disease, JNLPBA, and NCBI.Source: Datasets · pp. 5–6

Metric

  • True positives required exact matches between predicted and gold entity spans under BIOES notation.JNLPBA's training data was split to form a development set, and its sentences were re-separated using the original dataset.Source: Datasets · p. 5, Metric · p. 6
  • Correcting invalid BIOES sequences improved precision by about 0.1 to 0.5% and F1 by about 0.04 to 0.3%.Precision, recall, and F1 were the evaluation measures.Source: Metric · p. 6

Settings and hyperparameters

  • The models used 200-dimensional biomedical word embeddings, AdaGrad, 300-unit forward and backward LSTMs, and dropout.Training phases after preparation took 3,000 minutes, or 1,500 minutes without BC4CHEMD.Source: Settings and hyperparameters · p. 6

Results

CollaboNet achieves higher precision and F1 than most single-task models across all datasets, improves average precision and recall, and outperforms Wang et al.’s multi-task model on four of six datasets. The single-task model also outperforms baseline single-task models on average across precision, recall, and F1.

Results

  • The comparison reports single-task and CollaboNet results against baselines; Wang et al.’s models were rerun on two BC5CDR datasets for fair comparison, with scores averaged across six datasets.The reported mean uses the arithmetic mean; the STM was run with 10 random initializations.Source: Results · p. 6

Performance of single-task models

Performance of CollaboNet

  • CollaboNet outperforms Wang et al.’s multi-task model on 4 of 6 datasets and achieves higher precision and F1 than most single-task models on all datasets.On average, CollaboNet improves both precision and recall.Source: Performance of CollaboNet · p. 7

Discussion

Discussion reports higher macro-average performance and fewer false positives, especially false biomedical entities. It also examines entity-type errors, annotation inconsistency, case-study examples, and planned extensions.

Discussion

  • CollaboNet reduced false positives, especially false biomedical entities, as indicated by negative values in the difference column.The comparison reports error types for STM and CollaboNet.Source: Discussion · p. 8, Table 4 · p. 8
  • Bio-entity errors accounted for 30.8% of STM errors and up to 49.3% in JNLPBA, and can alter DDI results unlike span errors, which have a minor effect.The discussion notes that bio-entity errors are difficult to detect and fix, even for biomedical researchers.Source: Discussion · p. 8
  • Mixed annotation conventions can limit exact-match scores: datasets differ on whether modifiers belong within entity spans, and CollaboNet’s gains may not seem significant relative to its added complexity.The authors say inconsistent annotations may lower CollaboNet’s results despite its outperforming STM.Source: Discussion · p. 8
  • 736 of CollaboNet’s 3,966 errors were bio-entity errors (18.6%); CollaboNet also performed better than STM at detecting polysemy and other entity types, with higher macro-average performance than baselines.The error analysis covered five datasets and models selected for their best experimental performance.Source: Discussion · pp. 7–8

Case study

  • Chemical case studies show collaborator gene/protein predictions helping the chemical model avoid labeling globulin and ACE entities as chemicals.The single-task chemical model misidentified these examples; CollaboNet used information from the gene/protein model.Source: Case study · p. 8
  • Collaborator information helped the gene/protein model reject LMB and cHD as gene/protein entities, improving precision for other entity-type models.The model used information from chemical and disease collaborators, respectively.Source: Case study · p. 9
  • Expert-reviewed examples exposed missed or questionable ground-truth labels, which the authors believe their system can correct.Examples included unlabeled gene/protein and chemical entities, as well as a general term judged not to be a named entity.Source: Case study · pp. 9–10

Future works

  • Future work plans to add entity types and datasets, including CRAFT, LINNAEUS, and Variome, and apply CollaboNet to downstream text-mining systems.The authors cite entity search engines such as BEST as a possible application.Source: Future works · p. 10

Conclusion

CollaboNet combines multiple single-task BiLSTM-CRF models for BioNER and reports state-of-the-art performance across four datasets, while noting resource costs and potential for extension.

Conclusion

  • CollaboNet achieved state-of-the-art performance on four BioNER datasets in F1 score, precision, and recall.The conclusion also reports the highest F1 scores when leveraging multiple datasets.Source: Conclusion · p. 10
  • Multiple single-task models exchange information to improve predictions and distinguish polysemous or orthographically similar biomedical entities.The models consist of BiLSTM-CRF models, and the conclusion contrasts their collaborative design with recently proposed multi-task models.Source: Conclusion · p. 10
  • CollaboNet requires substantial memory and time, although its structure allows researchers to add expert models for other entity types.The conclusion notes that existing multi-task models also require substantial memory and time.Source: Conclusion · p. 10

Abbreviations

  • The glossary expands model and task shorthand, including BiLSTM, CRF, MTL, MTM, and STM.It defines BiLSTM as bidirectional long short-term memory, CRF as conditional random field, MTL as multi-task learning, MTM as multi-task model, and STM as single-task model.Source: Abbreviations · p. 10

Funding

This section records study funding, author contributions, resource availability, supplement context, and the competing-interests declaration.

Funding

  • The study and publication costs were supported by specified National Research Foundation of Korea and government-agency grants.The passage lists NRF-2017M3C4A7065887, 2016M3A9A7916996, and NO. C1202-18-1001 among the funding identifiers.Source: Funding · p. 10

Availability of data and materials

About this supplement

  • The article appeared in BMC Bioinformatics Volume 20 Supplement 10, the proceedings of DTMBIO 2018.The supplement covers the 12th International Workshop on Data and Text Mining in Biomedical Informatics.Source: About this supplement · p. 10

Authors’ contributions

  • WY, CHS, JL, and JK conceived the idea; their roles also included model design and development, experiments, analysis, writing, and supervision.All authors read and approved the manuscript.Source: Authors’ contributions · p. 10

Competing interests

Publisher’s Note

The publisher states its neutrality regarding jurisdictional claims in published maps and institutional affiliations.

Publisher’s Note

  • Springer Nature states it remains neutral regarding jurisdictional claims in published maps and institutional affiliations.Source: Publisher’s Note · p. 10
Loading 1809.07950v2…