Source-linked AI summary
Representation learning for very short texts using weighted word embedding aggregation
Cedric De Boom, Steven Van Canneyt, Thomas Demeester, Bart Dhoedt
TL;DR
Very short texts are noisy and sparse, making semantic comparison difficult for applications such as event detection and recommendation. The paper learns tf-idf-based weights for semantic word embeddings and introduces a median-based loss for aggregating short-text representations. Across Wikipedia and Twitter experiments, the method outperforms baselines and works with different embeddings without retraining.
Problem
Very short texts have sparse vocabulary, while tf-idf-based representations struggle to capture their semantic meaning.
Method
The method learns tf-idf-based weights for semantic word embeddings and uses a novel median-based loss to construct short-text representations.
Results
The method outperforms the baselines in Wikipedia and Twitter experiments and is applicable to different word embeddings without retraining.
Takeaways & Limitations
The learned weighted representations retain most of the text's semantic information and can be applied out-of-the-box.
Takeaways & Limitations
The Twitter ground truth is noisy, with a manually labeled error rate of 28%, making lower error rates difficult to achieve.
Abstract
from arXiv · showhide
Short text messages such as tweets are very noisy and sparse in their use of vocabulary. Traditional textual representations, such as tf-idf, have difficulty grasping the semantic meaning of such texts, which is important in applications such as event detection, opinion mining, news recommendation, etc. We constructed a method based on semantic word embeddings and frequency information to arrive at low-dimensional representations for short texts designed to capture semantic similarity. For this purpose we designed a weight-based model and a learning procedure based on a novel median-based loss function. This paper discusses the details of our model and the optimization methods, together with the experimental results on both Wikipedia and Twitter data. We find that our method outperforms the baseline approaches in the experiments, and that it generalizes well on different word embeddings without retraining. Our method is therefore capable of retaining most of the semantic information in the text, and is applicable out-of-the-box.
1. Introduction
Very short texts require representations that capture semantic similarity despite sparse, noisy vocabulary. The paper combines semantic word embeddings with learned tf-idf-based weighting and evaluates the approach on Wikipedia and Twitter data.
- Motivation: Short messages are difficult to compare because their sparse vocabulary makes similarity detection daunting at scale.This challenge matters for applications including event detection and news recommendation.
- Approach: The proposed representation uses semantic word embeddings and learns word weights from tf-idf information.The weighted embeddings are aggregated into an overall representation of each short text fragment.
- Contributions: The paper introduces weighted embedding representations, a median-based loss function, and related/non-related text-pair datasets from Wikipedia and Twitter.The loss is designed for minibatch learning to mitigate outlier effects.
- Findings: The technique is reported to outperform most baselines and remain applicable across different word embeddings without additional model training.This supports direct out-of-the-box use in different contexts.
2. Related work
Prior short-text representations struggle with sparse exact overlap, computational cost, fixed-length requirements, or retraining needs. The paper situates weighted embedding aggregation as a simple, flexible alternative that preserves semantic information without requiring fixed-length input.
- Limitations of existing representations: Tf-idf similarity relies on exact word overlap, which becomes less reliable for shorter texts and ignores synonyms and semantic relatedness.It is also vulnerable to homonym-related effects.
- Semantic-space methods: LSI and LDA incorporate semantic information but require computationally expensive training and inference on large vocabularies.Skip-gram embeddings offer another semantic-space approach based on predicting surrounding words.
- Neural and paragraph representations: Neural sentence encoders can require fixed-length inputs or aggregation, while some recurrent approaches still require retraining for different embeddings.Paragraph2vec also requires retraining for unseen word groups at test time.
- Aggregation methods: Mean, max, and min aggregation are simple and out-of-the-box, but they discard word order while avoiding fixed-length input requirements.These methods are often combined with multilayer perceptrons or convolutional networks.
3. Methodology
The method represents short texts by sorting words by inverse document frequency, weighting their embeddings, and averaging them into fixed-dimensional vectors. It learns these weights with pairwise semantic-similarity objectives, including a median-based loss designed to reduce imbalance and distribution skew, and extends the approach to variable-length texts.
- Semantic similarity objective: Related and non-related text pairs are compared by a distance threshold, with related pairs intended to lie closer and non-related pairs farther apart.The threshold classifier labels a pair when d(t1, t2) ≤ θ and a non-pair when d(t1, t2) > θ.
- Basic architecture: Words are sorted by decreasing idf, their embeddings are multiplied by learned rank-based weights, and the weighted vectors are averaged into one text representation.The same weights are used to represent both texts in each training pair.
- Loss functions: The contrastive-based loss can produce unbalanced losses and skew distance distributions, potentially increasing overlap even while the loss decreases.Its optimization may focus on easily shifted examples while ignoring other pairs, worsening threshold-based predictions.
- Loss functions: The median-based loss uses a balanced minibatch median distance as an approximate split point and a scaled cross-entropy objective.It treats related pairs above the median and non-related pairs below it as misclassified, while using the median's robustness to outliers.
- Texts with variable length: The fixed-length method is extended to texts of length m ≤ nmax by subsampling and linearly interpolating among nmax learned weights.The interpolated weights replace the original weights in the text-representation equation, and their derivatives are propagated during learning.
4. Data collection
The study constructs related and non-related text pairs from Wikipedia and Twitter to evaluate short-text representations. Wikipedia provides structured fixed-length excerpts, while Twitter supplies noisy event-linked messages gathered under heuristic pairing rules.
- Dataset design: The datasets include Wikipedia text pairs for methodological development and Twitter message pairs for testing practical streaming use.Wikipedia offers structured data for initial experiments; Twitter provides a contrasting noisy medium.
- Wikipedia: Wikipedia related pairs are formed from two n-word excerpts taken from the same paragraph with a two-word gap between them.The supplied passage specifies preprocessing and fixed-length pair construction from the March 4th 2015 English Wikipedia dump.
- Twitter: Twitter messages are limited to 140 characters and commonly contain spelling errors, abbreviations, and slang.The paper contrasts this medium with formal, linguistically correct Wikipedia text.
- Twitter: Twitter related pairs require shared event hashtags, temporal proximity within 15 minutes, and sufficient word dissimilarity.The pairing heuristics require at least 0.5 hashtag Jaccard similarity, fewer than 0.5 word Jaccard similarity, and at least five non-metadata words.
- Twitter: The Twitter dataset contains 15,000 training pairs, 20,000 validation pairs, and 13,645 test pairs, but its heuristic ground truth has a 28% manual-label error rate.The authors therefore characterize the dataset as very noisy and expect performance below roughly 28% error to be difficult.
5. Experiments
Experiments evaluate the methodology with optimal split error and Jensen-Shannon divergence, comparing learned representations against tf-idf and embedding-aggregation baselines. Training uses weighted embeddings with minibatch optimization and regularization, followed by validation-based threshold selection and test evaluation.
- Evaluation: The evaluation uses optimal split error and Jensen-Shannon divergence to distinguish related from non-related text-pair distributions.Higher JS divergence indicates a greater difference between the two distributions.
- Evaluation: Euclidean distance is used throughout because prior work found better separation between related and non-related pairs than cosine distance.The gradient of the Euclidean distance used in Equation (7) is described as straightforward to calculate.
- Evaluation: The experimental procedure trains word weights on the train set, selects the split point θ on validation data, and evaluates predictions on the test set.This separates weight learning, threshold selection, and final evaluation.
- Baselines: Baselines include tf-idf cosine similarity, mean or maximum embedding aggregation, minimum/maximum concatenation, top 30% idf aggregation, and idf-weighted means.The embedding baselines aggregate dimensions using mean, maximum, or minimum operations.
- Learning procedure: The learning procedure uses minibatch stochastic gradient descent with L2 regularization on the word weights.The total batch loss combines the minibatch loss with a regularization term weighted by λ.
- Learning procedure: The experiments use a batch size of 100 text couples, λ = 0.001, initial learning rate η = 0.01, and weight initialization at 0.5.The learning rate is lowered to 0.001 when average epoch loss deteriorates, and training stops when the inter-epoch loss difference falls below 0.05%.
13 STOP
Experiments on Wikipedia and Twitter show that the proposed representations outperform baselines across fixed- and variable-length settings, while median-based learning improves robustness and transfers across embeddings.
- Weight analysis: Weights decrease monotonically with idf-ranked words; from the 16th word onward they are close to zero, and a 20-word text contains eight non-informative words on average.The same decreasing pattern appears for texts of 10 and 30 words.
- Wikipedia results: Our method significantly outperforms all baselines on length-20 Wikipedia texts for both losses (p < 0.001, two-tailed binomial test).A plain tf-idf vector competes with the simplest widely used baselines.
- Wikipedia results: Approx 2.25% better performance from concatenating minimum and maximum vectors than using maximum vectors alone indicates semantically relevant sign information in embeddings.
- Wikipedia results: For variable-length Wikipedia texts, the median-based loss improves the contrastive-based loss by almost 5% in split error while the method remains the overall best performer (p < 0.001, two-tailed binomial test).The experiment varies text length between 10 and 30 words.
- Embedding transfer: Without retraining on Google News embeddings, the model achieves the best error rate among embedding baselines (p < 0.001, two-tailed binomial test), while median loss again beats contrastive loss by approx 5%.Tf-idf is the clear winner in this setting and suffers little from vocabulary reduction.
- Twitter results: On Twitter, the median-based method performs best, but its improvement over the idf-weighted baseline is not statistically significant (p > 0.05, two-tailed binomial test).Tf-idf performs worst because tweet vectors are much sparser than Wikipedia vectors.
6. Conclusion
The paper concludes that learned idf-based weighting of word embeddings produces effective representations for very short texts. The method supports fixed- and variable-length inputs, different embeddings without retraining, and outperforms widely used baselines on Wikipedia and Twitter data.
- The method derives vector representations for very short text fragments by learning idf-based weights for word embeddings.
- A novel median-based loss function mitigates outlier effects, while subsampling and linear interpolation extend the method to variable-length texts.
- The approach applies out-of-the-box to different word embeddings without retraining and outperforms widely used baselines on Wikipedia and Twitter data.