PaperlayerView exact revision on arXiv

Source-linked AI summary

ML-Net: multi-label classification of biomedical texts with deep neural networks

Jingcheng Du, Qingyu Chen, Yifan Peng, Yang Xiang, Cui Tao, Zhiyong Lu

arXiv:1811.05475v2cs.IRcs.CLcs.LGstat.ML

TL;DR

Multi-label classification poses a challenge because documents can receive an indeterminate number of labels and selecting the proper label set is difficult. ML-Net combines label prediction with label-count prediction to use document context in setting the output. The authors report best-to-date performance in the classification tasks, while noting computational limitations and that label counts may depend on other information, such as hierarchy.

Abstract

from arXiv · show

In multi-label text classification, each textual document can be assigned with one or more labels. Due to this nature, the multi-label text classification task is often considered to be more challenging compared to the binary or multi-class text classification problems. As an important task with broad applications in biomedicine such as assigning diagnosis codes, a number of different computational methods (e.g. training and combining binary classifiers for each label) have been proposed in recent years. However, many suffered from modest accuracy and efficiency, with only limited success in practical use. We propose ML-Net, a novel deep learning framework, for multi-label classification of biomedical texts. As an end-to-end system, ML-Net combines a label prediction network with an automated label count prediction mechanism to output an optimal set of labels by leveraging both predicted confidence score of each label and the contextual information in the target document. We evaluate ML-Net on three independent, publicly-available corpora in two kinds of text genres: biomedical literature and clinical notes. For evaluation, example-based measures such as precision, recall and f-measure are used. ML-Net is compared with several competitive machine learning baseline models. Our benchmarking results show that ML-Net compares favorably to the state-of-the-art methods in multi-label classification of biomedical texts. ML-NET is also shown to be robust when evaluated on different text genres in biomedicine. Unlike traditional machine learning methods, ML-Net does not require human efforts in feature engineering and is highly efficient and scalable approach to tasks with a large set of labels (no need to build individual classifiers for each separate label). Finally, ML-NET is able to dynamically estimate the label count based on the document context in a more systematic and accurate manner.

BACKGROUND

Multi-label classification assigns texts one or more labels, making label selection challenging when the number of labels is indeterminate. ML-Net combines label prediction with document context in an end-to-end framework for biomedical texts.

BACKGROUND

  • Multi-label classification assigns each text one or more labels; PubMed indexing typically gives new publications a dozen relevant MeSH terms.Source: BACKGROUND · p. 4
  • Binary relevance decomposes classification by label but assumes label independence, whereas label powerset models potential correlations between labels.Source: BACKGROUND · p. 4
  • Both binary relevance and label powerset can have low throughput when the number of distinct labels becomes extremely large.Source: BACKGROUND · p. 4
  • Neural approaches score labels and commonly use thresholding to separate relevant from irrelevant labels, requiring threshold optimization while ignoring document context.Source: BACKGROUND · p. 5
  • ML-Net combines label prediction and label decision, using confidence scores and document context to predict labels end-to-end without a separate decision step.Source: BACKGROUND · p. 5
  • The study evaluates ML-Net on three public biomedical text tasks—two literature tasks and one clinical task—and compares it with traditional and deep-learning baselines.Source: BACKGROUND · p. 5

DEEP NEURAL NETWORK

This section describes ML-Net’s document encoding and joint prediction of label confidences and document-conditioned label counts, alongside an evaluation across biomedical literature and clinical notes. It also outlines comparisons with binary-relevance classifiers and neural-network variants, as well as task-specific embeddings and model settings.

DEEP NEURAL NETWORK

  • ML-Net combines hierarchical attention document encoding with one network for per-label confidence scores and another for label-count estimates.Source: DEEP NEURAL NETWORK · p. 6
  • HAN represents text through word- and sentence-level attention networks, using a bidirectional RNN for sequential context and attention to emphasize salient portions.Source: Text pre-processing · p. 8
  • The label predictor uses ReLU outputs and log-sum-exp pairwise (LSEP) loss to model pairwise ranking errors among relevant and irrelevant labels.Source: Text pre-processing · p. 8
  • ML-Net estimates each document’s label count from its contextual vector, addressing thresholding that ignores document context.The count predictor is a multilayer perceptron with a Softmax output.Source: Label count estimation network · p. 9
  • At prediction time, ML-Net ranks label confidence scores and returns the top K labels, with K supplied by the count-prediction network.Source: Label count estimation network · p. 9

EVALUATION DESIGN

  • The literature tasks use 1,580 cancer-hallmark abstracts annotated for 10 hallmarks and 3,661 chemical-exposure abstracts annotated across 32 taxonomy classes.Source: EVALUATION DESIGN · p. 10
  • The ICD-9 clinical-note task is an extreme multi-label problem, with over 7,000 unique labels and 37 assigned labels per document on average after augmentation.Source: EVALUATION DESIGN · pp. 10–11
  • Example-based metrics score each test instance separately; true positives require exact labels for the literature tasks but allow ancestor, descendant, or identical ICD-9 codes.Source: EVALUATION DESIGN · pp. 11–12
  • For tasks 1 and 2, ML-Net used 200-dimensional PubMed word2vec embeddings, an LSTM recurrent unit, 50-dimensional hidden and attention outputs, and dropout of 0.5.The maximum permitted label counts were 5 and 8 for tasks 1 and 2, respectively.Source: System implementation · p. 12
  • For task 3, the model used 300-dimensional MIMIC III word2vec embeddings and configured its label-count network for 7,024 ICD codes, with a maximum of 70 predicted labels.The label-count network used 7,024, 7,024, and 128 neurons across its layers.Source: System implementation · p. 13
  • Traditional baselines used TF-IDF features and a separate binary classifier per label; SVM was reported because it performed best among the tested algorithms in all three tasks.The compared algorithms included logistic regression, random forest, and extra trees.Source: System implementation · p. 14
  • ML-Net’s deep-learning comparator replaced hierarchical attention with Kim’s CNN while retaining the label prediction network, and global-threshold variants were evaluated for both architectures.Thresholds were selected on validation data for tasks 1 and 2, and training data for task 3, which lacked a validation set.Source: System implementation · p. 14

RESULTS

ML-Net achieved the best F-scores on the first two tasks, while diagnosis-code assignment favored thresholding over label-count prediction. Results also varied by task: the reported diagnosis-code table identifies binary relevance as having the best scores.

RESULTS

  • ML-Net had the best F-score on tasks 1 and 2; for cancer hallmarks it beat the baseline by over 14%, while only label-count-prediction models beat binary relevance on chemical exposure.On both tasks, label-count prediction performed better than thresholding; hierarchical attention also achieved a slightly higher F-score than convolutional neural networks.Source: RESULTS · p. 15
  • For diagnosis-code assignment, ML-Net with thresholding performed better than the label-count-prediction network; the authors suspect ICD-9 hierarchy influences label counts more than document context.They suggest that adding codes based on hierarchical relations can make the label-set count depend on the ICD-9 structure rather than the document.Source: RESULTS · p. 15
  • For diagnosis-code assignment, Table 3 identifies the binary-relevance scores as the best results.Source: Table 3 · p. 16

DISCUSSION & CONCLUSIONS

ML-Net achieves the best performance to date on the discussed classification tasks, while the discussion identifies diagnosis-code distribution and closely related code predictions as challenges. The authors also note limitations in tuning and in the label-count network’s ability to use label hierarchy.

DISCUSSION & CONCLUSIONS

  • Task 3 performs much worse than the first two, and unbalanced code frequencies and train–test co-occurrence differences negatively affect performance.The top 100 codes account for 115,268 of 215,805 occurrences; code co-occurrence rankings also differ between training and test sets.Source: DISCUSSION & CONCLUSIONS · p. 16
  • The system more readily predicts diagnosis codes that are close to one another, sometimes including code pairs together when clinicians might choose only one.For example, predicted co-occurrence counts for codes 412 and 414.01 were 155, versus 88 in the gold standard; for 413.9 and 414.01, they were 139 versus 49.Source: DISCUSSION & CONCLUSIONS · pp. 16–17
  • ML-Net reduces feature-engineering effort and avoids separate classifiers for each label, particularly for collections with over 1,000 labels.Its label-prediction network is combined with a label-count prediction network, which estimates the output count from document context and avoids manual threshold search.Source: DISCUSSION & CONCLUSIONS · p. 17
  • ML-Net achieves the best performance to date in these classification tasks.The discussion also credits its combined label-prediction and label-count networks with avoiding manual threshold search and estimating label counts from document context.Source: DISCUSSION & CONCLUSIONS · p. 17
  • Limited computation prevented thorough hyperparameter tuning, so the current settings may not be optimal; the label-count network also cannot model label hierarchy.The count network takes the document vector as input, although label counts might also depend on hierarchical label structure.Source: DISCUSSION & CONCLUSIONS · p. 17
  • The authors identify improved document encoding and evaluation on larger-scale biomedical classification tasks, including automatic MeSH indexing, as future directions.They mention replacing the word-level network with a sentence encoder and sentence embedding as one possible architecture change.Source: DISCUSSION & CONCLUSIONS · pp. 17–18

DECLARATIONS

The authors acknowledge support from NCBI and NIH programs and CPRIT. They state that the content is the authors’ responsibility and does not necessarily represent the named institutions’ official views.

FUNDING

  • Research was supported by the NCBI Scientific Visitors Program, the NIH Intramural Research Program, and CPRIT’s Training program.Source: FUNDING · p. 23

DISCLAIMER

  • The authors state that the content is their responsibility and does not necessarily represent the official views of the named institutions.Source: DISCLAIMER · p. 23
Loading 1811.05475v2…