Source-linked AI summary
A Brief Survey of Text Mining: Classification, Clustering and Extraction Techniques
Mehdi Allahyari, Seyedamin Pouriyeh, Mehdi Assefi, Saied Safaei, Elizabeth D. Trippe, Juan B. Gutierrez, Krys Kochut
TL;DR
The rapid growth of unstructured text makes machine processing difficult. This paper surveys fundamental text-mining techniques and biomedical applications, providing a rough overview of current progress.
Problem
The rapid growth of unstructured text creates a need for methods that enable computers to process and analyze it effectively.
Method
The paper reviews fundamental text-mining tasks and techniques, including preprocessing, representation, classification, clustering, and biomedical applications.
Results
The survey provides a brief overview of widely used text-mining algorithms, techniques, and approaches in biomedical domains.
Takeaways & Limitations
The paper offers a rough overview of current progress in text mining within the scope of its brief introduction.
Abstract
from arXiv · showhide
The amount of text that is generated every day is increasing dramatically. This tremendous volume of mostly unstructured text cannot be simply processed and perceived by computers. Therefore, efficient and effective techniques and algorithms are required to discover useful patterns. Text mining is the task of extracting meaningful information from text, which has gained significant attentions in recent years. In this paper, we describe several of the most fundamental text mining tasks and techniques including text pre-processing, classification and clustering. Additionally, we briefly explain text mining in biomedical and health care domains.
1 INTRODUCTION
The introduction motivates text mining by the rapid growth of largely unstructured text and the difficulty machines face in understanding it. It frames text mining within knowledge discovery and outlines the iterative process and interdisciplinary foundations used to extract useful patterns.
- Motivation: Text mining has gained attention because text data are rapidly expanding across social networks, patient records, health care insurance, and news outlets.An IDC report predicts data volume will reach 40 zettabytes by 2020, a 50-time increase from the beginning of 2010.
- Motivation: Unstructured text is easy for humans to process but significantly harder for machines, creating a need for methods that can extract useful information and knowledge.The introduction describes the volume of text as an invaluable information source and an avalanche requiring effective processing algorithms.
- Knowledge Discovery: Knowledge discovery is an iterative and interactive process involving application and data understanding, preparation and preprocessing, modeling, and evaluation.The CRoss Industry Standard Process for Data Mining model identifies these primary steps, while the paper treats data mining as the modeling phase of KDD.
- Text Mining Scope: Text mining extracts high-quality information from structured, semi-structured, and unstructured resources while spanning information retrieval, natural language processing, and information extraction.The introduction distinguishes text mining’s pattern-finding purpose from information retrieval’s emphasis on access and notes NLP techniques such as part-of-speech tagging and syntactic parsing.
2 TEXT REPRESENTATION AND ENCODING
Text mining requires structured document representations for analysis, most commonly bag-of-words and numeric vector models. Preprocessing and term weighting further prepare documents by reducing noise, normalizing words, and emphasizing informative terms.
- Text representation: Bag-of-words represents documents by term occurrence counts while ignoring word order, producing vectors suitable for dimension-reduction techniques such as LSI, PLSA, and topic models.These techniques are identified as three main dimension-reduction approaches used in text mining.
- Preprocessing: Preprocessing is a key component of text-mining algorithms and can noticeably influence classification success alongside feature extraction, feature selection, and classification.A traditional text-categorization framework comprises preprocessing, feature extraction, feature selection, and classification.
- Preprocessing: Tokenization breaks character sequences into word or phrase tokens, while filtering removes stop words and potentially uninformative very common or very rare terms.The resulting token list is used for further processing, and filtering targets words with limited content or distinguishing information.
- Preprocessing: Lemmatization groups inflected word forms into a single item, whereas language-dependent stemming obtains the root of derived words and is often preferred in practice because part-of-speech labeling is tedious and error prone.The most widely used stemming method in English is the stemmer published in.
- Vector Space Model: The Vector Space Model converts documents into numeric vectors whose term weights indicate word importance, supporting efficient analysis in text-mining algorithms and information-retrieval systems.Boolean and TF-IDF are the two main term-weighting models described for VSM.
- Vector Space Model: TF-IDF normalizes term frequency by inverse document frequency, decreasing weights for collection-frequent terms so distinctive, relatively infrequent words influence document matching more.Documents can then be represented by vectors of term weights, with cosine similarity used to compare documents.
3 CLASSIFICATION
Text classification assigns predefined labels to documents using models such as probabilistic classifiers, nearest-neighbor methods, decision trees, and SVMs. Performance is evaluated on held-out documents using accuracy, precision, recall, and F-1 scores.
- Classification overview: Text classification assigns predefined class labels to documents, including hard, soft, and multiple-label assignments.Hard classification explicitly assigns a label, whereas soft classification assigns a probability value.
- Evaluation: Classification performance is measured by comparing predicted labels with true labels on a held-out test set.Accuracy is the portion of correctly classified documents, while precision, recall, and F-1 are common evaluation metrics.
- Probabilistic classifiers: Probabilistic classifiers estimate model parameters from training examples and use Bayes rule to select the class most likely to have generated a new example.Naive Bayes represents documents using word distributions; Bernoulli models ignore word frequency, whereas multinomial models capture it.
- Nearest neighbor classifier: Nearest-neighbor classification infers a test document’s class from labels of similar training documents using distance-based measures.The method assumes documents in the same class are more likely to be close according to similarity measures such as cosine similarity.
- Support Vector Machines: Support Vector Machines find linear separators between classes and are robust to the high dimensionality of sparse text data.SVMs seek a hyperplane with maximum margin and often require little feature selection because they select support vectors.
4 CLUSTERING
Text clustering groups similar documents using similarity functions and requires text-specific representations because text is high-dimensional, sparse, correlated, and variable in length. The section surveys hierarchical, k-means, and probabilistic topic-modeling approaches, including LDA’s intractable exact inference and approximate alternatives.
- Clustering overview: Text clustering finds groups of similar documents using a similarity function and supports applications including classification, visualization, and document organization.Clusters may operate at different granularities, including documents and paragraphs.
- Clustering overview: Text representations are high-dimensional and sparse, contain correlated words, and require normalization because documents vary in length.The vocabulary may be on the order of 10^5 words, while an individual document may contain only a few hundred words; sparsity is more severe for tweets.
- Hierarchical clustering: Hierarchical clustering builds a hierarchy through divisive top-down splitting or agglomerative bottom-up merging based on document similarity.Agglomerative clustering starts with one cluster per document and repeatedly merges the most similar clusters.
- k-means clustering: k-means partitions n documents into k clusters, but finding an optimal solution is NP-hard and the method is sensitive to the initial choice of k.Efficient heuristics converge rapidly to a local optimum, and agglomerative clustering can help determine an initial k.
- Probabilistic clustering: Topic modeling creates probabilistic generative models in which documents are mixtures of topics and topics are probability distributions over words.The main topic models identified are Probabilistic Latent Semantic Analysis and Latent Dirichlet Allocation; LDA adds a Dirichlet prior that supports modeling new unseen documents.
- Probabilistic clustering: LDA’s exact posterior inference is intractable because of the denominator, so approximate methods such as variational inference and Gibbs sampling are used.Gibbs sampling uses Markov Chain Monte Carlo to approximate the posterior with an empirical distribution.
5 INFORMATION EXTRACTION
Information extraction automatically converts unstructured or semi-structured text into known structured information. Its two fundamental tasks are named entity recognition and relation extraction, commonly addressed with statistical learning methods.
- 5 INFORMATION EXTRACTION: Information extraction automatically extracts structured information from unstructured or semi-structured text, with the target information specified beforehand.It is described as a limited form of full natural language understanding.
- 5 INFORMATION EXTRACTION: A sentence about Microsoft, Bill Gates, Paul Allen, and its founding date can yield founder and founding-date relations.The extracted relations are FounderOf(Bill Gates, Microsoft), FounderOf(Paul Allen, Microsoft), and FoundedIn(Microsoft, April - 4 1975).
- 5 INFORMATION EXTRACTION: Information extraction comprises two fundamental tasks: named entity recognition and relation extraction, with statistical learning methods representing the state of the art.These tasks are widely studied across information retrieval, natural language processing, and Web mining, and applied in biomedical text mining and business intelligence.
- Named Entity Recognition: Named entity recognition locates and classifies word sequences denoting real-world entities into predefined categories such as person, organization, and location.Dictionary string matching is insufficient because dictionaries can be incomplete and omit forms of entities.
- Relation Extraction: Relation extraction identifies semantic relations between entities and commonly frames their categorization as a classification problem over fixed relation types.Named entity recognition serves as a preprocessing step for relation extraction and also supports applications such as question answering.
6 BIOMEDICAL ONTOLOGIES AND TEXT MINING FOR BIOMEDICINE AND HEALTHCARE
Biomedical text mining addresses the rapid growth of biomedical literature by transforming unstructured text into machine-understandable knowledge. The section introduces biomedical ontologies and their use in information extraction, clustering, and named entity recognition.
- Biomedical literature is growing exponentially, making it difficult for scientists to assimilate new publications and track relevant research.
- Text mining and statistical machine learning are widely used to manage biomedical information overload across tasks including protein structure prediction, gene clustering, hypothesis generation, and clinical diagnosis.
- Biomedical ontologies: An ontology formally defines domain-specific terms and relationships that describe and represent an area of knowledge.
- Biomedical ontologies: UMLS unifies over 100 dictionaries, terminologies, and ontologies in its Metathesaurus and provides a semantic network of semantic types and relations.
- Information extraction and NER: Biomedical information extraction typically preprocesses scientific and clinical text, while named entity recognition locates and classifies proteins, genes, and diseases.
- Information extraction and NER: NER is challenged by rapidly expanding entities, synonyms, and frequent acronyms, but systems generally achieve good results using precision, recall, and F-score evaluation.
7 DISCUSSION
The article provides a brief overview of fundamental text-mining algorithms and techniques, including approaches for biomedical applications. It also emphasizes the challenge of identifying relevant articles within rapidly expanding scientific literature.
- 7 DISCUSSION: The paper surveys fundamental algorithms and techniques extensively used in text mining.It presents a rough overview of current progress rather than a comprehensive account of all methods.
- 7 DISCUSSION: The discussion includes important text-mining approaches applied in the biomedical domain.This extends the survey beyond general text-mining methods to biomedical applications.
- 7 DISCUSSION: The high and rapidly growing volume of scientific literature makes identifying articles pertinent to researchers’ interests difficult.Daily additions to online archives improve access to information while complicating article selection.