Source-linked AI summary
Portuguese Word Embeddings: Evaluating on Word Analogies and Natural Language Tasks
Nathan Hartmann, Erick Fonseca, Christopher Shulby, Marcos Treviso, Jessica Rodrigues, Sandra Aluisio
TL;DR
The paper asks whether word analogies adequately evaluate Portuguese word embeddings. It trains 31 models across four algorithms and tests them on analogies, POS tagging, and sentence similarity; the results favor task-specific evaluation over analogy scores.
Problem
Word analogies may be inappropriate for evaluating embeddings, motivating comparisons on downstream NLP tasks.
Method
The study trains 31 Portuguese embedding models with FastText, GloVe, Wang2Vec, and Word2Vec, evaluating them on analogies, POS tagging, and sentence similarity.
Results
Intrinsic analogy results and extrinsic task results were not aligned; Wang2Vec performed very well across evaluations, while GloVe led analogies but performed worst with FastText on POS tagging and sentence similarity.
Takeaways & Limitations
Word analogies are not appropriate as the sole evaluation of word embeddings, so task-specific evaluations appear to be a better option.
Takeaways & Limitations
POS accuracies were well below earlier reports, probably because the embedding vocabularies did not split clitic pronouns from verbs, producing many out-of-vocabulary words.
Abstract
from arXiv · showhide
Word embeddings have been found to provide meaningful representations for words in an efficient way; therefore, they have become common in Natural Language Processing sys- tems. In this paper, we evaluated different word embedding models trained on a large Portuguese corpus, including both Brazilian and European variants. We trained 31 word embedding models using FastText, GloVe, Wang2Vec and Word2Vec. We evaluated them intrinsically on syntactic and semantic analogies and extrinsically on POS tagging and sentence semantic similarity tasks. The obtained results suggest that word analogies are not appropriate for word embedding evaluation; task-specific evaluations appear to be a better option.
1. Introduction
Word embeddings provide efficient, meaningful representations for words, motivating their widespread use in NLP. Because analogy scores may not reflect downstream performance, the paper evaluates Portuguese embeddings with both analogy and task-specific benchmarks.
- Word embeddings represent words as real-valued vectors learned from large, non-annotated corpora.These representations can capture syntactic, semantic, and morphological knowledge.
- Embedding methods include co-occurrence-based and predictive approaches.GloVe is an example of the first family, while predictive methods estimate neighboring words.
- Word analogy benchmarks test whether vector algebra recovers a fourth word from three related words.The relationships may be syntactic or semantic, such as tense or country–capital pairs.
- Word analogies are computationally cheap and popular but may be inappropriate for evaluating embeddings.Task-specific evaluation instead compares models on downstream NLP tasks.
- The paper trains and evaluates Portuguese embeddings across Brazilian and European variants using intrinsic and extrinsic tasks.It also makes 31 models and the corpus-preprocessing script publicly available.
2. Training Corpus
The study constructs a large, multi-source Portuguese corpus intended to represent the language across Brazilian and European varieties. Preprocessing reduces vocabulary size and filters noisy or infrequent material before training.
- The corpus combines several sources to create a large, multi-genre representation of Portuguese.The collection includes Brazilian and European texts.
- Word types occurring fewer than five times are replaced with UNKNOWN to reduce vocabulary size.Numerals, URLs, and emails are also normalized to shared tokens.
- Tokenization uses whitespace and punctuation while paying special attention to hyphenation.Clitic pronouns such as “machucou-se” remain intact.
- Sentences with fewer than five tokens are removed to reduce noisy content.This preprocessing reduced LX-Corpus from 1,723,693,241 to 714,286,638 tokens.
- Table 1 reports the sources and statistics of the collected corpora.
3. Embedding Methods
The paper trains 31 Portuguese word-embedding models with four algorithmic families. These methods differ in whether they model co-occurrence, neighboring words, word order, or character n-grams.
- The study trains 31 models using GloVe, Word2Vec, Wang2Vec, and FastText.
- GloVe trains word vectors and biases from a context-word co-occurrence matrix.Each matrix entry represents the probability of one word occurring close to another.
- Word2Vec uses CBOW to predict an omitted middle word or Skip-Gram to predict neighboring words.Its single-weight-matrix architecture supports fast log-linear training.
- Wang2Vec modifies Word2Vec to incorporate word order for stronger syntactic representation.Continuous Window preserves context order, while Structured Skip-Gram uses position-specific parameters.
- FastText represents words by summing character n-gram vectors with surrounding-word vectors.The method therefore targets morphological information.
4. Evaluation
The paper evaluates Portuguese word embeddings through intrinsic analogies and extrinsic POS tagging and semantic similarity tasks. Results show that analogy performance does not consistently predict task performance.
- Evaluation design: The study evaluates embeddings on syntactic and semantic analogies, POS tagging, and sentence semantic similarity for Brazilian and European Portuguese.Embeddings were trained in dimensions ranging from 50 to 1,000.
- Intrinsic evaluation: GloVe performed best on average in intrinsic evaluation, while FastText led syntactic analogies and GloVe led semantic analogies.Wang2Vec followed FastText on syntactic analogies and GloVe on semantic analogies.
- Intrinsic evaluation: CBOW models, except Wang2Vec, achieved very low results on semantic analogies.The authors speculate that ignoring word order limits semantic representation.
- POS tagging: Wang2Vec achieved the best POS tagging performance, while larger dimensionality generally improved accuracy.The 1,000-dimensional Word2Vec models were slightly worse than the 600-dimensional models.
- Semantic similarity: Word2Vec CBOW achieved the best European Portuguese semantic similarity result, while Wang2Vec Skip-Gram achieved the best Brazilian Portuguese result.Neither FastText nor GloVe surpassed Hartmann’s prior results.
- Overall findings: The study reports a lack of correlation between syntactic and semantic analogy performance and corresponding NLP task performance.This finding supports evaluating embeddings directly on downstream tasks.
5. Related Work
Prior Portuguese embedding studies evaluated analogy benchmarks and downstream tasks using different corpora, algorithms, and language variants. Their findings include strong performance from Word2Vec models, corpus-size effects, and no clear penalty from mixing variants.
- Portuguese embedding studies: Only a few studies had attempted to evaluate Portuguese word embeddings before this work.Rodrigues et al. trained Skip-Gram Word2Vec models and released analogy benchmarks for Brazilian and European Portuguese.
- Portuguese embedding studies: The Rodrigues benchmark contains five semantic analogy types and nine syntactic analogy types for Portuguese.Semantic categories include capitals, currencies, cities and states, and family relations.
- Portuguese embedding studies: Word2Vec CBOW with 300 dimensions achieved 21.7% syntactic, 17.2% semantic, and 20.4% overall analogy accuracy in Sousa’s study.The study used a Wikipedia sample focused on education-related articles.
- Downstream evaluation: Skip-Gram achieved the best results across all POS-tagging tests in Fonseca et al.’s comparison of three vector-space models.The comparison included Word2Vec Skip-Gram, HAL, and a neural method.
- Language variants: Larger training corpora generally produced better embeddings, and mixing Brazilian and European Portuguese did not decrease tagger performance.The finding came from extrinsic POS-tagging analyses across language variants.
6. Conclusions and Future Work
Intrinsic and extrinsic evaluations produced conflicting rankings of Portuguese word embeddings. Wang2Vec performed well across tasks, while future work targets preprocessing and broader evaluation.
- Intrinsic and extrinsic evaluation results were not aligned with each other.
- GloVe achieved the best results on syntactic and semantic analogies but the worst, together with FastText, on POS tagging and sentence similarity.
- Wang2Vec vectors yielded very good performance across the evaluations, suggesting usefulness for a variety of NLP tasks.
- All trained models and the corpus-preprocessing script are available for download.
- Future work includes testing different tokenization and normalization patterns, lemmatizing selected word categories, and evaluating more NLP tasks.The authors expect lemmatization to reduce vocabulary and enable more efficient processing.