Source-linked AI summary
A Comprehensive Survey on Word Representation Models: From Classical to State-Of-The-Art Word Representation Language Models
Usman Naseem, Imran Razzak, Shah Khalid Khan, Mukesh Prasad
TL;DR
This survey addresses how complex, noisy text can be represented effectively for NLP and machine-learning tasks. It synthesizes representation methods from classical approaches through contextual and transformer-based language models, along with preprocessing and applications. The survey concludes that representation research has progressed from feature engineering toward deep-learning and state-of-the-art models, while continued work is expected on unlabeled data, reinforcement learning, and multimodal learning.
Problem
NLP needs effective ways to represent rich, unstructured, and low-quality text so it can be analyzed and used by machine-learning frameworks.
Method
The paper surveys preprocessing, classical text representation, distributed and contextual word vectors, transformer-based language models, classifiers, metrics, and applications.
Results
The survey covers a progression from classical feature engineering to deep-learning representations, including GloVe, FastText, contextual vectors, and transformer-based language models.
Takeaways & Limitations
The paper expects future NLP research to make better use of unlabeled data and develop more model designs, reinforcement-learning applications, and multimodal learning.
Takeaways & Limitations
Continuous word representations have limitations including out-of-vocabulary words, for which assigning a shared UNK vector can degrade results when OOV counts are large.
Abstract
from arXiv · showhide
Word representation has always been an important research area in the history of natural language processing (NLP). Understanding such complex text data is imperative, given that it is rich in information and can be used widely across various applications. In this survey, we explore different word representation models and its power of expression, from the classical to modern-day state-of-the-art word representation language models (LMS). We describe a variety of text representation methods, and model designs have blossomed in the context of NLP, including SOTA LMs. These models can transform large volumes of text into effective vector representations capturing the same semantic information. Further, such representations can be utilized by various machine learning (ML) algorithms for a variety of NLP related tasks. In the end, this survey briefly discusses the commonly used ML and DL based classifiers, evaluation metrics and the applications of these word embeddings in different NLP tasks.
1 INTRODUCTION
The survey examines how low-quality, unstructured text can be converted into vector representations for machine-learning-based analysis. It reviews methods from classical feature engineering through state-of-the-art representational learning approaches.
- Motivation: Unstructured text from social media, forums, articles, clinical notes, and reviews is rich in information but noisy and difficult to process.Informal language, abbreviations, punctuation, emoticons, slang, URLs, and spelling mistakes complicate analysis.
- Applications: Text representations support applications including email filtering, sentiment analysis, sarcasm detection, hate-speech detection, question answering, and biomedical text mining.Machine-learning models use learned representations to classify low-quality text or obtain information from it.
- Scope and contribution: The survey focuses on representing low-quality text as numeric vectors for easier understanding and use in machine-learning frameworks.It treats documents, sentences, words, and characters as entities whose relationships can be inferred as vectors.
- Scope and contribution: The survey traces text representation from manually engineered features and bag-of-words methods to neural representation learning and state-of-the-art language models.It covers methods operating at character, word, sentence, and document levels.
- Text classification pipeline: Text classification requires preprocessing raw text and extracting numeric features before applying an appropriate classifier.Feature extraction transforms raw data into feature vectors that machines can understand.
2 TEXT PRE-PROCESSING
The survey presents preprocessing as a necessary stage for cleaning noisy text before representation and classification. It reviews tokenization, normalization, punctuation and number handling, lemmatization, part-of-speech tagging, and empirical findings on technique combinations.
- Preprocessing motivation: Preprocessing removes or transforms unwanted text elements such as stop-words, punctuation, incorrect spellings, and slang that may harm classification performance.The survey frames cleaning as especially important for noisy text datasets.
- Preprocessing techniques: Tokenization divides documents into sentences and sentences into words, symbols, phrases, or other tokens.It is usually applied as the first standard preprocessing step in NLP tasks.
- Preprocessing techniques: Emoticons, emojis, abbreviations, and slang can be replaced with associated meanings to preserve sentiment and opinion information.These transformations are discussed particularly for short social-media text.
- Preprocessing techniques: Lemmatization uses lexical knowledge to transform inflected words into base forms rather than merely chopping word endings.The survey distinguishes it from stemming and also discusses part-of-speech tagging.
- Empirical findings: Studies report that preprocessing combinations must be chosen carefully because inappropriate combinations can degrade classification performance.Reported effects vary across techniques, datasets, and classifiers.
3 FEATURE EXTRACTION METHODS
The survey traces feature extraction from classical frequency-based representations to distributed, contextual, and transformer-based language models. It emphasizes the shift toward automatically learned low-dimensional representations that better capture semantic, syntactic, contextual, and global co-occurrence information.
- Classical Models: Classical feature extraction models represent text as vectors based primarily on word frequency, including categorical and weighted representations.Categorical methods include one-hot encoding and BoW, while weighted methods include TF and TF-IDF.
- Classical Models: One-hot encoding assigns each vocabulary term a binary vector, with the vector dimension determined by vocabulary size.The corresponding word index is marked with 1 and other positions with 0.
- Classical Models: BoW and TF-IDF ignore word order, semantics, and syntactic information, while large vocabularies produce high-dimensional sparse representations.TF-IDF therefore functions primarily as a lexical-level feature.
- Representation Learning: These classical limitations motivated representation learning methods that automatically discover features in lower-dimensional spaces for downstream tasks.The survey links this shift to the need for representations that better preserve semantic and syntactic information.
- Distributed Representations: Word2Vec emphasizes local context, whereas GloVe uses a global co-occurrence matrix to represent word relationships.GloVe matrix entries X_ij encode the frequency with which words w_i and w_j co-occur within a context window.
- Contextual and Transformer-Based Models: Contextual and transformer-based models extend representation learning by addressing context and complex word attributes across NLP tasks.The survey discusses ULMFiT’s three-stage transfer-learning process, XLNet’s bidirectional-context prediction, ALBERT’s parameter efficiency, and BART’s denoising objectives.
- Contextual and Transformer-Based Models: BART matches RoBERTa’s performance on GLUE and SQuAD and achieves state-of-the-art results on several text-generation tasks.Its best performance uses sentence shuffling together with text infilling.
4 CLASSIFICATION TECHNIQUES
The survey reviews commonly used machine-learning and deep-learning classifiers for text classification, emphasizing their designs, uses, and limitations. It covers Naive Bayes, decision trees, recurrent architectures, convolutional networks, and related models.
- ML-based classifiers: Naive Bayes classifiers use Bayes theorem and assume that each pair of classified features is independent of the others.They are widely used in information retrieval and text classification.
- ML-based classifiers: Decision trees classify text by passing instances through root and decision nodes until reaching a leaf representing the assigned class.They are easy to describe and visualize but can be sensitive to small data changes and overfit.
- Deep-learning classifiers: Deep-learning models use multiple nonlinear processing layers and generally require substantial training data to learn semantically effective textual representations.The survey presents them as achieving strong results across classification tasks.
- Deep-learning classifiers: RNNs model sequential text but are vulnerable to vanishing and exploding gradients during backpropagation.LSTMs were introduced to address these gradient and long-term dependency issues more effectively.
- Deep-learning classifiers: CNNs use stacked feature-map filters and pooling layers to reduce dimensionality while retaining salient features.Text classification can face very high dimensionality because text may have many channels.
5 EVALUATION METRICS
The survey describes standard evaluation tools for text classification, beginning with confusion-matrix outcomes and then defining accuracy, precision, recall, and F1 score. These metrics quantify different aspects of prediction correctness.
- Confusion matrix: A confusion matrix presents classification efficiency using true positives, true negatives, false positives, and false negatives.These entries distinguish correctly and incorrectly predicted positive and negative instances.
- Accuracy: Accuracy is the ratio of correctly predicted observations to the total number of observations.Its formula uses TP, TN, FP, and FN.
- Precision: Precision is the ratio of true-positive observations to all positive predictions, TP+FP.It focuses on the correctness of positive predictions.
- Recall: Recall is the ratio of true-positive observations to overall relevant observations, TP+FN.It measures the positive instances identified relative to the total positive observations.
- F1 score: F1 score is a weighted average of recall and precision that incorporates both false positives and false negatives.The survey gives F1 as 2*Recall*Precision/(Recall+Precision).
6 APPLICATIONS
The survey reports that language models have expanded from early feature extraction and information retrieval into many domains and text-classification applications.
- Applications: Language models are used in medicine, social sciences, healthcare, psychology, law, and engineering, among other domains.Applications include information retrieval, sentiment analysis, recommender systems, summarization, question answering, machine translation, and named entity recognition.
7 CONCLUSION
The survey synthesizes classical feature-engineering methods, deep-learning models, word embeddings, contextual vectors, and transformer-based language models for representing text as vectors. It concludes that these approaches capture rich textual information for machine-learning and NLP tasks, while future work is expected to improve model designs and use of unlabeled and multimodal data.
- Conclusion: The survey introduces algorithms that capture rich information in text and represent it as vectors for traditional machine-learning frameworks.It first discusses classical feature engineering and then deep-learning-based models.
- Conclusion: The surveyed embedding methods include Word2Vec, GloVe, FastText, Context2Vec, CoVe, and ELMo.These methods include both word embeddings and contextual word vectors.
- Conclusion: The survey also presents transformer-based state-of-the-art language models trained on general and domain-specific corpora.The paper describes these models as still being in development.
- Conclusion: Future NLP research is expected to pursue better use of unlabeled data, improved model designs, reinforcement-learning applications, and multimodal learning.The survey gives dialogue systems as an example of reinforcement-learning applications and notes that language can correlate with other signals.