Source-linked AI summary
Extraction of Pharmacokinetic Evidence of Drug-drug Interactions from the Literature
Artemy Kolchinsky, Anália Lourenço, Heng-Yi Wu, Lang Li, Luis M. Rocha
TL;DR
The paper addresses the limited automatic extraction of experimentally specific DDI evidence, focusing on pharmacokinetic evidence needed for causal-mechanism assessment and downstream investigations. It evaluates literature-mining classifiers on annotated PubMed abstracts and sentences, finding high performance across both tasks and robustness across many configurations. The results support automated identification of relevant abstracts and evidence-bearing sentences, within the evaluated corpus and cross-validation setting.
Problem
Existing literature mining does not adequately extract experimental DDI evidence reported differently across study types, despite pharmacokinetic evidence supporting causal-mechanism assessment and putative DDIs.
Method
The study uses manually annotated PubMed abstract and sentence corpora to evaluate linear classifiers with textual, metadata, named-entity, dictionary, and feature-transform inputs.
Results
High performance was reached for abstract classification and evidence-sentence extraction, with abstract F1≈0.93, MCC≈0.74, iAUC≈0.99 and sentence F1≈0.76, MCC≈0.65, iAUC≈0.83.
Takeaways & Limitations
BLM can identify PubMed abstracts reporting pharmacokinetically supported DDIs and extract evidence sentences for further pharmacological and pharmaco-epidemiological investigation.
Takeaways & Limitations
The evaluation does not perform resampling or re-weighting despite imbalanced abstract and sentence datasets, and its testing procedure relies on separated cross-validation documents.
Abstract
from arXiv · showhide
Drug-drug interaction (DDI) is a major cause of morbidity and mortality and a subject of intense scientific interest. Biomedical literature mining can aid DDI research by extracting evidence for large numbers of potential interactions from published literature and clinical databases. Though DDI is investigated in domains ranging in scale from intracellular biochemistry to human populations, literature mining has not been used to extract specific types of experimental evidence, which are reported differently for distinct experimental goals. We focus on pharmacokinetic evidence for DDI, essential for identifying causal mechanisms of putative interactions and as input for further pharmacological and pharmaco-epidemiology investigations. We used manually curated corpora of PubMed abstracts and annotated sentences to evaluate the efficacy of literature mining on two tasks: first, identifying PubMed abstracts containing pharmacokinetic evidence of DDIs; second, extracting sentences containing such evidence from abstracts. We implemented a text mining pipeline and evaluated it using several linear classifiers and a variety of feature transforms. The most important textual features in the abstract and sentence classification tasks were analyzed. We also investigated the performance benefits of using features derived from PubMed metadata fields, various publicly available named entity recognizers, and pharmacokinetic dictionaries. Several classifiers performed very well in distinguishing relevant and irrelevant abstracts (reaching F1~=0.93, MCC~=0.74, iAUC~=0.99) and sentences (F1~=0.76, MCC~=0.65, iAUC~=0.83). We found that word bigram features were important for achieving optimal classifier performance and that features derived from Medical Subject Headings (MeSH) terms significantly improved abstract classification. ...
Introduction
The paper targets the underdeveloped automatic extraction of pharmacokinetic evidence for DDIs, which supports causal-mechanism assessment and downstream pharmacological and pharmaco-epidemiological studies. It evaluates biomedical literature mining for identifying relevant PubMed abstracts and extracting evidence-bearing sentences.
- Motivation: BLM can uncover possible DDIs in published literature and clinical databases for subsequent in vitro, in vivo, and clinical investigation.
- Motivation: Pharmacokinetic evidence is important for identifying or dismissing causal mechanisms behind DDIs and supporting putative DDIs found in patient records.The paper focuses on evidence such as Ki, IC50, and AUCR from in vitro and in vivo experiments.
- Research gap: Existing DDI information-extraction methods do not specifically extract experimental evidence or distinguish different kinds of evidence.The paper therefore targets a more specific evidence-extraction task than identifying interacting drugs alone.
- Objectives: The study addresses both identifying PubMed abstracts containing pharmacokinetic DDI evidence and extracting sentences containing that evidence.It also assesses abstract classification using an updated corpus and compares classifiers and evaluation methods.
- Objectives: The experiments evaluate automated methods for identifying pharmacokinetic DDI evidence and report high performance across many classifier configurations.
Materials and Methods
The pipeline builds annotated abstract and sentence corpora, represents PubMed text and metadata with unigram and bigram features, and evaluates multiple linear classifiers and transformations. Cross-validation estimates generalization and selects hyperparameters while feature transforms and dimensionality reduction modify the document representations.
- Pipeline: The pipeline selects documents, hand-labels ground truth, extracts and normalizes textual features, computes unigram/bigram matrices, and uses nested cross-validation for hyperparameters.
- Corpora: The abstract corpus contains 1203 pharmacokinetics-related PubMed abstracts, including 909 DDI-relevant and 294 DDI-irrelevant abstracts.Relevant abstracts reported the presence or absence of drug interaction supported by explicit pharmacokinetic parameter data.
- Feature representation: Features include title and abstract text plus author, journal, MeSH, registry-number, and secondary-source metadata, processed with tokenization and Porter stemming.
- Feature representation: Bigram runs add word bigrams to unigram features, increasing the number of coefficients estimated from training data.
- Corpora: The sentence corpus contains 4600 sentences from 428 abstracts, manually labeled for explicit pharmacokinetic evidence of DDI.
- Classifiers: The study evaluates six linear classifiers, including VTT, SVM, logistic regression, Naive Bayes, regularized LDA, and diagonal LDA.Regularized LDA uses SVD and covariance shrinkage, whereas diagonal LDA sets off-diagonal covariance entries to zero.
- Feature transforms: Feature representations use raw occurrences, IDF, TFIDF, length normalization, and PCA projections with 100 to 1000 components.
- Feature transforms: PCA reduces the number of estimated parameters, while feature transforms can make class-separating surfaces more linear and reduce nondiscriminating-feature weights.
Results
The classifiers performed strongly on both abstract and sentence tasks, with bigram features generally improving performance and selected metadata and named-entity features providing additional gains. Abstract classification achieved the strongest reported results, while sentence classification was also effective but showed task-specific feature behavior.
- Abstract classification performance: F1≈0.93, iAUC≈0.98, and MCC≈0.73 were achieved by the best abstract classifier, improving substantially over preliminary results.The earlier preliminary results were F1≈0.8, iAUC≈0.88, and MCC≈0.6.
- Abstract classification performance: Bigram features improved every classifier according to RP3, with LDA using bigrams achieving the best overall abstract performance.LDA was significantly better than other classifiers for MCC, while Logistic Regression and SVM were not significantly different from it for some measures.
- Pharmacokinetics DDI Features in abstract classification: MeSH metadata and interaction-related terms were among the most relevant abstract features, whereas genetics terminology and the MeSH term Phenotype were associated with irrelevant abstracts.Relevant features also included administration protocols and study-design terms such as oral, day, mg, treatment, alone, and combination.
- Impact of NER and PubMed metadata on abstract classification: F1≈0.93, MCC≈0.74, and iAUC≈0.99 were reached by top abstract classifiers after adding NER and dictionary features.BICEPP consistently produced the best improvement across tested classifiers, followed by the i-Drugs dictionary.
- Evidence sentence extraction performance: LDA with bigrams performed best on sentence classification by RP3 and led according to F1 and MCC, while SVM shared top status on iAUC.Classifiers accounting for feature covariances generally performed well, whereas dimensionality reduction decreased performance for the strongest non-transformed classifiers.
- Evidence sentence extraction performance: Drug-name counts from BICEPP, DrugBank, and i-Drugs significantly improved iAUC for nearly all sentence classifiers, while dLDA benefited little from NER features.In sentence features, quantitative changes and interaction expressions were relevant, but generic pharmacokinetic terms were often irrelevant; “interact with” was relevant whereas “interact” was not.
Discussion
The study shows that linear-classifier pipelines can identify pharmacokinetic DDI evidence at both abstract and sentence levels, with high performance across multiple configurations. Feature analyses identify useful textual and metadata signals while also defining practical limits from metadata availability and modest feature gains.
- F1≈0.93, MCC≈0.74, and iAUC≈0.99 were reached for abstract classification, compared with F1≈0.76, MCC≈0.65, and iAUC≈0.83 for evidence sentence extraction.
- Unigram-plus-bigram features significantly outperformed unigrams alone, although their iAUC and MCC advantages were no larger than one percent.
- A sophisticated LDA classifier dominated both tasks, while feature transforms and PCA improved several classifiers but not the overall best performance.
- Abstract features associated with DDI evidence included interaction terms, enzyme inhibitors, experimental results, numerical values, clearance changes, and significance expressions.
- At sentence level, relevant evidence was distinguished from protocol, generic interaction, and biomedical terminology within otherwise pharmacokinetics-focused abstracts.
- MeSH omission significantly reduced abstract iAUC only slightly, to approximately 0.98, and newly published articles should not rely on unavailable metadata.
- NER and dictionary features produced modest gains for several classifiers, while unigram and bigram statistics alone still extracted relevant-class information.
- Relevant and irrelevant materials shared pharmacokinetic feature statistics, making classification nontrivial but practically relevant for labeling PubMed documents.
Extraction of Pharmacokinetic Evidence of Drug-drug Interactions from the Literature: SUPPORTING INFORMATION
The supporting information lists Artemy Kolchinsky, Anália Lourenço, Heng-Yi Wu, Lang Li, and Luis M. Rocha.
- Artemy Kolchinsky is listed among the supporting-information authors.
- Anália Lourenço, Heng-Yi Wu, and Lang Li are listed among the supporting-information authors.
- Luis M. Rocha is listed among the supporting-information authors.
1 Abstract performance
Abstract classification performance was high across several linear classifiers, with LDA achieving the strongest reported overall results. Feature-transform analyses compared classifier configurations, while PCA separated covariance-aware from naive classifiers.
- Classifier performance: LDA with bigram features achieved the highest abstract-run MCC (.728) and accuracy (.897), while its F1 was .931 and iAUC was .984.Logistic Regression and SVM with bigrams had closely comparable F1 and iAUC values.
- Feature transformations: Feature transforms and dimensionality reductions were evaluated against no-transform, no-reduction configurations, with significance assessed using one-tailed tests.Naive Bayes was excluded because it applies only to binary data, while VTT was tested only with sparse transforms.
- Classifier analysis: PCA of classifier hyperplanes separated covariance-aware classifiers—LDA, SVM, and Logistic Regression—from classifiers that did not use feature covariance information.The projection summarized differences among transform-classifier combinations through their loadings on the first two principal components.