A Comprehensive Survey on Word Representation Models: From Classical to State-Of-The-Art Word Representation Language Models

1 INTRODUCTION

• The survey situates text representation in applications including email filtering [20], sentiment and opinion mining, hate-speech detection, and biomedical text mining [109].Other cited applications include irony and sarcasm detection, document organization, question answering, and content mining.Source: §1 · p. 2
• The survey focuses on representing text as continuous vectors at character, word, sentence, or document level, capturing relationships among entities for machine-learning use.The representations support tasks such as clustering, dimensionality reduction, and text classification.Source: §1 · p. 3
• The classification pipeline preprocesses text and extracts features before applying a classifier, but the survey limits its focus to representation rather than detailed classifier comparisons.The pipeline also faces general data issues such as class imbalance and missing labels.Source: §1 · pp. 2–4
• It traces representation methods from manually engineered features and bag of words to neural-network methods that learn embeddings.The survey covers variations of these methods and discusses future directions for text representation.Source: §1 · p. 3
• Informal, noisy text complicates representation: short-text preprocessing addresses abbreviations, emoticons, spelling errors, and URLs, while feature extraction converts raw text into numeric vectors.Although social platforms provide large corpora, labeling them is difficult; emoticons and hashtags have been used as sentiment and emotion cues [69].Source: §1 · pp. 2–4

2.1 Preliminaries related to Text Pre-processing

• Tweet preprocessing can tokenize text, handle URLs, mentions and hashtags, segment hashtag words, and preserve emoji or emoticon meanings for classification.Tokenization is described as a standard first step; URLs may be replaced with tags, mentions removed, and hashtag phrases segmented.Source: §2.1 · pp. 5–6
• Normalize informal text by expanding abbreviations, slang, contractions and elongated words, correcting spelling, and converting numeric shorthand before removing numbers.The section cautions that removing numbers too early can discard information in forms such as “2maro” and “gr8.”Source: §2.1 · pp. 6–7
• Other techniques include removing punctuation and stop-words, lower-casing, stemming or lemmatizing, POS tagging, and handling negation, though case and punctuation can carry useful signals.Stemming can reduce features; lemmatization uses lexical knowledge, while negation may be marked with a “NEG_” prefix or handled using antonyms.Source: §2.1 · pp. 7–8

2.2 Related work on text pre-processing methods

• Studies find preprocessing effects task-dependent: some transformations and carefully chosen combinations improve classification, while other removals or combinations can hurt.Reported results vary: for example, found URL retention and negation transformation helpful but stemming and lemmatization harmful, while found lemmatization helpful and punctuation removal unhelpful; the survey notes no recommendation work on improving text quality.Source: §2.2 · pp. 8–9

3.1 Classical Models

• One-hot encoding assigns each vocabulary term a unique binary dimension, while Bag-of-Words sums token vectors but ignores word order, grammar, and semantic relationships.These representations grow with vocabulary size and can produce sparse, high-dimensional vectors.Source: §3.1 · pp. 10–11
• TF normalizes a word’s document frequency by document length, while TF-IDF uses inverse document frequency to reduce the influence of common terms.Because TF-IDF builds on Bag-of-Words, it does not capture word order, semantics, or syntax and is suited to lexical-level features.Source: §3.1 · p. 11

3.2 Representation Learning

• Representation learning automatically discovers features for downstream tasks through supervised input-output training, unsupervised learning from unlabeled data, or self-supervised learning with automated labels such as masked-word prediction.Self-supervised learning predicts an aspect of the input from other parts of it.Source: §3.2 · p. 12

3.2.1 Distributed Representations.

• Word embeddings map vocabulary items to low-dimensional vectors that represent similarities between words used in similar contexts and can serve as a neural model’s first layer.The survey describes this approach as following the distributional hypothesis.Source: §3.2 · p. 12, §3.2.1 · p. 13
• Word2Vec’s CBOW predicts a center word from context, whereas Skip-gram predicts context from a center word; Skip-gram suits scarce data and rare words, while CBOW is faster for repeated words [96, 102].Negative sampling updates only sampled output vectors; hierarchical softmax uses a Huffman tree, with the former suited to repeated words and the latter to less frequent words.Source: §3.2.1 · pp. 13–15
• GloVe uses global word co-occurrence statistics to complement Word2Vec’s local-window focus, while FastText represents n-grams to capture subword relations and improve representations of rare and OOV words [15].GloVe builds a co-occurrence matrix and factorizes it to obtain vectors.Source: §3.2.1 · pp. 15–16
• Context2Vec uses a bidirectional LSTM for sentence context; CoVe transfers a translation-pretrained encoder with GloVe vectors [90], while ELMo combines forward and backward language-model layers for context-sensitive word vectors [125].ELMo provides different representations for the same word in different sentences.Source: §3.2.1 · pp. 16–18
• Continuous embeddings improve classification by capturing semantic and syntactic information, but static Word2Vec and GloVe miss polysemy, FastText addresses OOV words with n-grams, and current SOTA models struggle with low-quality text.The survey also notes that Word2Vec and GloVe assign random vectors to words unseen during training.Source: §3.2.1 · pp. 16, 20
• Transformer-based models progress from ULMFiT’s three-stage fine-tuning to GPT’s left-to-right prediction and BERT’s bidirectional masked-token and next-sentence objectives.ULMFiT uses general LM pre-training, target-task LM fine-tuning, and classifier fine-tuning; BERT masks 15% of tokens. GPT is unidirectional, while BERT uses parallel attention rather than recurrence.Source: §3.2.1 · pp. 21–23
• ALBERT reduces BERT’s parameter burden, while XLNet and RoBERTa alter pre-training to address bidirectional context and improve efficiency; XLM and StructBERT add cross-lingual and structural objectives.An ALBERT configuration matching BERT-large has 18 times fewer parameters and trains about 1.7 times faster; XLNet uses permutation prediction and two-stream attention, while RoBERTa removes NSP and trains longer.Source: §3.2.1 · p. 24
• The surveyed models include DistilBERT, MegatronLM, ERNIE, UniLM, ELECTRA, MASS, and T5, using distillation, scaling, knowledge masking, multiple objectives, token replacement, masked sequences, and text-to-text encoder-decoder pretraining.DistilBERT [141] reduces BERT’s size by 40% while retaining 97% of its language-understanding proficiency and running 60% quicker; Sci-BERT [9] and BioBERT [79] are biomedical domain-specific models.Source: §3.2.1 · pp. 25–26
• BART matches RoBERTa on GLUE and SQuAD and achieves state-of-the-art results on several text-generation tasks, with sentence shuffling plus text infilling performing best.BART corrupts input sequences using noise functions and trains a transformer to reconstruct them.Source: §3.2.1 · pp. 25–26

3.3 Related work on Word representation methods

• Traditional n-gram and bag-of-words features are simple and economical but miss word order and semantics, while neural representations and recursive or paragraph models can improve classification over BoW.Recursive models capture long dependencies but rely on parsing; paragraph representations avoid parsing. Word2Vec and GloVe embeddings are widely used for low-dimensional vectors that capture semantic relationships.Source: §3.3 · p. 27
• Pretrained and contextual embeddings support neural text classifiers, but OOV handling, polysemy, domain mismatch, and sentiment remain challenges addressed by robust, knowledge-enriched, sentiment-aware, and domain-specific models.Studies report improved classification from pretraining and contextual embeddings; hybrid sentiment representations incorporate polarity or lexicon information, while contextual and domain-specific LMs capture attributes including semantics, OOV words, context, and syntax.Source: §3.3 · pp. 27–29

4.1 ML based classifiers

• Naive Bayes is simple, scalable, and suited to high-dimensional inputs and limited training data, but its feature-independence assumption conflicts with interrelated language and can yield poor results.The passage also identifies data scarcity as a limitation when estimating feature likelihoods.Source: §4.1 · pp. 29–30
• SVMs separate classes with a hyperplane, adding dimensions when needed, but kernel choice and large-data training are costly; logistic regression outputs probabilities and can fail on continuous or interdependent data.Logistic regression works well for categorical results, but its predictions can suffer when independent values are chosen incorrectly.Source: §4.1 · pp. 30–31
• Decision trees route examples through learned feature decisions and are interpretable with little tuning, but can overfit; random forests combine bootstrapped trees and feature subsets at computational cost.Random forests can handle missing values and outliers, but many trees require more training time and memory and can slow prediction.Source: §4.1 · pp. 31–32

4.2 Deep learning based classifiers

• Deep models require substantial training data; RNNs suit sequential text classification but face vanishing or exploding gradients, while LSTMs mitigate vanishing gradients and retain longer dependencies.GRUs are a simpler RNN variant with two gates and no internal memory cell.Source: §4.2 · pp. 32–33
• CNNs build text features with stacked convolutional maps and pooling, but text classification can involve so many channels that dimensionality becomes a challenge.Pooling reduces layer size while retaining features, and the resulting maps are flattened before the fully connected final layer.Source: §4.2 · pp. 33–34

Confusion matrix

• The confusion matrix presents classification efficiency through true and false positive and negative counts.TP and TN are correctly predicted positive and negative instances; FP and FN are wrongly predicted positive and negative instances.Source: §5 · p. 34, Table 5 · p. 34

Accuracy

• Accuracy is the ratio of correctly predicted observations to the total observations.The section defines accuracy using TP and TN as the correctly predicted instances.Source: §5 · pp. 34–35

Precision

• Precision is the ratio of true positive observations to all positive predictions, TP+FP.Source: §5 · p. 35

Recall

• Recall is the ratio of true positive observations to TP+FN.Source: §5 · p. 35

F1 score

• F1 score is described as a weighted average of recall and precision that incorporates false positives and false negatives.Source: §5 · p. 35

• Historically, language models extracted features for text classification, particularly in information retrieval systems.Source: §6 · p. 35
• Language models are now used in domains including medicine, social sciences, healthcare, psychology, law, and engineering.Source: §6 · p. 35
• Applications include information retrieval, sentiment analysis, recommender systems, summarization, question answering, machine translation, named entity recognition, and adversarial attacks and defenses.Source: §6 · p. 35

7 CONCLUSION

• The survey presents algorithms that represent text as vectors for traditional frameworks, covering classical feature engineering, deep-learning models, and word embeddings.It discusses Word2Vec, GloVe, FastText, Context2Vec, CoVe, and ELMo.Source: §7 · pp. 35–36
• Deep learning’s success stems from stacked nonlinear processing that learns representations at multiple abstraction levels, and it supports semantic analysis, text classification, and NLP tasks.The survey also notes deep learning’s established capabilities in computer vision and speech recognition.Source: §7 · p. 35
• Coverage includes transformer language models trained on general and domain-specific corpora, which the survey describes as still developing.The authors expect better use of unlabeled data and further model designs to drive learning-based NLP research.Source: §7 · p. 36
• The authors expect more NLP applications to use reinforcement learning, such as dialogue systems, and more research on multimodal learning because language is often grounded in or correlated with other signals.Source: §7 · p. 36