PaperlayerView exact revision on arXiv

Source-linked AI summary

Drug-Drug Interaction Extraction from Biomedical Text Using Long Short Term Memory Network

Sunil Kumar Sahu, Ashish Anand

arXiv:1701.08303v2cs.CL

TL;DR

The paper addresses the difficulty of extracting drug-drug interactions from rapidly growing biomedical literature using existing knowledge bases and handcrafted features. It introduces three Bi-LSTM models using word and position embeddings with max or attentive pooling, and reports that Joint AB-LSTM outperforms existing methods on the SemEval-2013 DDI task. The models are competitive without explicit feature engineering, although long sentences with many drug mentions remain difficult.

  • Problem

    Rapidly growing biomedical literature makes it difficult for existing DDI knowledge bases and handcrafted-feature systems to keep pace with available information.

  • Method

    The paper proposes B-LSTM, AB-LSTM, and Joint AB-LSTM models using word and position embeddings, Bi-LSTM encoding, and max or attentive pooling.

  • Results

    Joint AB-LSTM outperforms all existing methods on the SemEval-2013 DDI classification task, while B-LSTM and AB-LSTM perform competitively with state-of-the-art methods.

  • Takeaways & Limitations

    LSTM-based models provide a competitive DDI extraction approach using simple latent word and distance features rather than explicit feature engineering.

  • Takeaways & Limitations

    The models struggle with drug pairs in long sentences containing many other drug entities, and the filtering assumptions remove 144 positive training instances.

Abstract

from arXiv · show

Simultaneous administration of multiple drugs can have synergistic or antagonistic effects as one drug can affect activities of other drugs. Synergistic effects lead to improved therapeutic outcomes, whereas, antagonistic effects can be life-threatening, may lead to increased healthcare cost, or may even cause death. Thus identification of unknown drug-drug interaction (DDI) is an important concern for efficient and effective healthcare. Although multiple resources for DDI exist, they are often unable to keep pace with rich amount of information available in fast growing biomedical texts. Most existing methods model DDI extraction from text as a classification problem and mainly rely on handcrafted features. Some of these features further depend on domain specific tools. Recently neural network models using latent features have been shown to give similar or better performance than the other existing models dependent on handcrafted features. In this paper, we present three models namely, {\it B-LSTM}, {\it AB-LSTM} and {\it Joint AB-LSTM} based on long short-term memory (LSTM) network. All three models utilize word and position embedding as latent features and thus do not rely on explicit feature engineering. Further use of bidirectional long short-term memory (Bi-LSTM) networks allow implicit feature extraction from the whole sentence. The two models, {\it AB-LSTM} and {\it Joint AB-LSTM} also use attentive pooling in the output of Bi-LSTM layer to assign weights to features. Our experimental results on the SemEval-2013 DDI extraction dataset show that the {\it Joint AB-LSTM} model outperforms all the existing methods, including those relying on handcrafted features. The other two proposed LSTM models also perform competitively with state-of-the-art methods.

1. INTRODUCTION

The paper targets automatic extraction of drug-drug interactions from rapidly expanding biomedical text, where existing knowledge bases and handcrafted-feature methods face scalability and feature-engineering limitations. It proposes Bi-LSTM models with max or attentive pooling, with Joint AB-LSTM achieving the strongest reported performance.

  • 1. INTRODUCTION: 27 million PubMed citations and approximately 0.8 million annual additions make manually updating structured DDI knowledge bases challenging.DrugBank and Stockley are cited as structured DDI resources.
  • 1. INTRODUCTION: DDI extraction identifies how two drugs in a sentence are related, including Advice, Effect, Mechanism, and generic Interaction categories.The paper illustrates these interaction types with examples from the SemEval dataset.
  • 1. INTRODUCTION: The work focuses on the second SemEval-2013 Task-9 task, which classifies drug-drug interactions from biomedical text.The earlier challenge assumed that drug names were already provided.
  • 1. INTRODUCTION: Bi-LSTM models encode whole-sentence context before pooling, addressing limitations of CNN n-gram features when relevant clues are distant.B-LSTM uses max pooling, whereas AB-LSTM uses attentive pooling; Joint AB-LSTM combines both.
  • 1. INTRODUCTION: Joint AB-LSTM outperforms all existing DDI extraction models, while B-LSTM and AB-LSTM remain competitive with state-of-the-art methods.The models are evaluated on the SemEval-2013 DDI extraction task.

2. MODEL ARCHITECTURE

The architecture represents each sentence using word and drug-distance embeddings, then produces fixed-length features for classification. Three models use these representations with Bi-LSTM encoding and distinct pooling designs.

  • 2. MODEL ARCHITECTURE: Each model receives word, distance-to-first-drug, and distance-to-second-drug features, then computes class scores for final DDI classification.The two drug names are assumed to be given in the sentence.
  • 2. MODEL ARCHITECTURE: The three architectures are presented together as the paper’s B-LSTM, AB-LSTM, and Joint AB-LSTM model family.The supplied figure caption identifies Figure 1 as a block diagram of all three models.
  • 2. MODEL ARCHITECTURE: Word and position features are mapped into real-valued vectors through embedding matrices before subsequent neural processing.Pre-trained word vectors are used for the word embedding matrix, while other feature matrices are randomly initialized.

Bi-LSTM Layer

The Bi-LSTM layer processes sentence feature vectors in both forward and backward directions, producing contextual outputs for each word while using LSTM gates and memory to regulate information.

  • Bi-LSTM Layer: LSTM uses input, output, and forget gates plus a memory cell to regulate information across the sentence.The gates use sigmoid activation, while the final output is computed from the updated memory state.
  • Bi-LSTM Layer: Each time step receives a feature vector for one word and computes hidden, cell, and Bi-LSTM output states.The input sequence has one vector per sentence word, and the hidden layer has size N.
  • Bi-LSTM Layer: The Bi-LSTM output concatenates the forward and backward hidden states for each word.This produces a contextual representation that incorporates information from both reading directions.
  • Bi-LSTM Layer: Bi-LSTM produces an output for each word by combining forward and backward LSTM processing.The backward LSTM processes the sentence with reversed word order.

Pooling Layer

The pooling layer converts variable-length Bi-LSTM word representations into fixed-length features using either max pooling or attention-weighted aggregation.

  • Pooling Layer: Pooling produces fixed-length features from variable-length word features.The models experiment with max pooling and attentive pooling.
  • Pooling Layer: Max pooling selects the dimension-wise maximum across the complete sequence of Bi-LSTM outputs.It assumes each Bi-LSTM output contains information from the complete sentence.
  • Pooling Layer: Max pooling may miss complementary DDI clues when they occur in different clauses or far apart in a sentence.The cited example contains separate clues indicating interaction effect and increased plasma concentration.
  • Pooling Layer: Attentive pooling assigns sentence-specific weights to word features and combines the Bi-LSTM outputs linearly.The attention vector has one weight per sentence position, allowing relevant context words to occupy different positions across sentences.

Fully Connected and Softmax

The pooled fixed-length vector is transformed and passed through a class-sized fully connected layer, whose softmax output gives normalized probabilities for each class.

  • Fully Connected and Softmax: The pooled vector undergoes a tanh transformation before entering the fully connected neural layer.The fully connected layer has a number of nodes equal to the number of classes.
  • Fully Connected and Softmax: Softmax converts the fully connected output into a normalized probability score for each class.The output layer uses parameters W_o and b_o, with C denoting the number of classes.

Training and Implementation

The proposed models use Bi-LSTM-based pooling variants for classification and are trained with cross entropy and Adam; implementation uses Python and TensorFlow.

  • Training and Implementation: All three models are trained with cross entropy loss and Adam optimization using batches of 200.The implementation uses Python with TensorFlow, and code for reproducing results is available online.
  • Training and Implementation: B-LSTM applies max pooling to Bi-LSTM outputs before fully connected and softmax classification.This model is represented in Figure 1a.
  • Training and Implementation: AB-LSTM applies attentive pooling to Bi-LSTM outputs before fully connected and softmax classification.Attention weights are obtained from the attentive-pooling mechanism shown in Figure 1b.
  • Training and Implementation: Joint AB-LSTM uses separate Bi-LSTM modules with max pooling in one and attentive pooling in the other, then concatenates both feature sets for classification.Both modules receive the same feature vectors and produce outputs for every sentence word.

3. DATASET DESCRIPTION

The SemEval-2013 DDI classification dataset contains annotated drug-pair interactions from Medline and DrugBank, with preprocessing and filtering designed for five-way classification. Filtering removes selected negative or structurally redundant instances while retaining all positive test instances.

  • 3. DATASET DESCRIPTION: The corpus includes four interaction types: Advice, Effect, Mechanism, and Int, covering recommendations, effects or pharmacodynamic mechanisms, pharmacokinetic mechanisms, and unspecified interactions.
  • 3. DATASET DESCRIPTION: The dataset labels each drug pair as Advice, Effect, Mechanism, Int, or Negative, with all possible pairs separately annotated when sentences contain more than two drugs.The data combine Medline abstracts and DrugBank texts.
  • 3.1 Pre-processing: Preprocessing tokenizes and lowercases sentences, normalizes digits, and replaces target and other drug mentions with DRUG-A, DRUG-B, and DRUG-N.
  • 3.2 Negative Instance Filtering: The original positive-to-negative instance ratio is 1:5.9, motivating negative-instance filtering before model training.
  • 3.2 Negative Instance Filtering: Filtering removes self-interactions, kind-of or special-case drug pairs, and pairs in the same coordinate structure using string matching and regular expressions.
  • 3.2 Negative Instance Filtering: Filtering removed 144 positive training instances but eliminated no positive test instances, including 54 Mechanism, 65 Effect, 49 Int, and 6 Advice instances.

4. EXPERIMENT DESIGN

The experiments train and evaluate three LSTM models using the challenge evaluation scheme and compare them with conventional, kernel-based, and neural baselines. The design also examines performance changes caused by negative-instance filtering and uses a held-out validation split for tuning.

  • 4. EXPERIMENT DESIGN: The models are trained and evaluated on drug-pair instances using the SemEval challenge evaluation scheme.
  • 4. EXPERIMENT DESIGN: Table 4 reports performance improvement after negative-instance filtering using relative F1-score change as its comparison measure.
  • 4. EXPERIMENT DESIGN: Table 5 compares class-wise precision, recall, and F1 changes between models trained on filtered and complete datasets.The table marks decreases, increases, and near-unchanged values with ↓, ↑, and ∼.
  • 4. EXPERIMENT DESIGN: An 80%/20% split of the original training data supplies training and validation sets because no separate development set is available.Hyperparameters are tuned on the validation portion.
  • 4. EXPERIMENT DESIGN: The study compares three proposed LSTM models with conventional-feature, kernel, and neural-network baselines.

5. RESULTS AND DISCUSSIONS

The proposed LSTM models generally outperform competing methods, with Joint AB-LSTM achieving the strongest aggregate results, while filtering improves F1 scores. Analyses show benefits from embeddings and longer-context modeling, but long sentences with repeated drug entities remain difficult.

  • 5.1 Effect of Negative Instance Filtering: Filtering negative instances improved F1 scores for all three LSTM models, producing more than 3% relative improvement for each.The authors attribute the gains to cleaner data and reduced class imbalance.
  • 5.2 Comparison with Baseline Methods: B-LSTM and Joint AB-LSTM outperformed existing models on the complete dataset, although no LSTM model consistently exceeded the others at the chosen significance level.Both models significantly outperformed CNN*1, with reported p-values of 0.0007 and 0.01, respectively.
  • 5.2 Comparison with Baseline Methods: Joint AB-LSTM obtained the best filtered-dataset F1 score of 69.39%, improving relatively by 3.6% over Kim2.The comparison used precision, recall, and F1 score on the test data.
  • 5.2 Comparison with Baseline Methods: Joint AB-LSTM achieved the best macro-average F1, while Advice was easiest and Int was most difficult to detect across models.Insufficient training data was proposed as an explanation for the weaker Int-class performance.
  • 5.3 Feature Analysis: Removing position embeddings caused a 1.1% relative decrease, while replacing pretrained word vectors and removing position embeddings caused a 4.6% decrease.These results indicate contributions from both word and position embeddings in Joint AB-LSTM.
  • 5.4 LSTM vs CNN models: LSTM models performed better than CNN on instances with longer sentences or greater separation between the targeted drugs.Incorrectly classified instances also tended to have longer sentences, larger entity separation, and multiple drug entities.

6. CONCLUSION

The study proposes three LSTM-based models for DDI classification and finds that Joint AB-LSTM achieves state-of-the-art performance, while the other models remain competitive.

  • Joint AB-LSTM achieves state-of-the-art performance for DDI classification on the SemEval-2013 DDI extraction dataset.
  • B-LSTM, AB-LSTM, and Joint AB-LSTM use word and distance embeddings with Bi-LSTM-based representation learning.
  • AB-LSTM and Joint AB-LSTM additionally use neural attention to obtain higher-level feature representations.
  • B-LSTM and AB-LSTM perform competitively with existing state-of-the-art methods.
  • Imbalance, noise, repeated drug names, and longer sentences adversely affect model predictions, while Advice interactions are easiest to predict.
Loading 1701.08303v2…