Source-linked AI summary
BPEmb: Tokenization-free Pre-trained Subword Embeddings in 275 Languages
Benjamin Heinzerling, Michael Strube
TL;DR
BPEmb addresses the limited representation of rare and unseen words by providing pre-trained BPE subword embeddings across many languages. It constructs embeddings from BPE units without tokenization and evaluates them on fine-grained entity typing, where they perform competitively with alternative subword approaches and better for some languages while using far fewer resources. The paper also identifies limits from evaluating only a subset of languages and from ignoring context.
Problem
Representing rare or unseen words is difficult, and BPE embeddings were commonly trained per task rather than published for general use.
Method
BPEmb applies BPE to Wikipedia text and pre-trains embeddings for the resulting subword symbols in 275 languages, evaluating them on fine-grained entity typing.
Results
BPEmb performs as well as, and for some languages better than, alternative subword approaches while requiring no tokenization and being orders of magnitude smaller.
Takeaways & Limitations
BPEmb may support subword representation under resource constraints, including potential use on mobile devices.
Takeaways & Limitations
The evaluation covers only a few of the 275 languages and uses an artificial setup that disregards context, so transfer to other tasks requires further study.
Abstract
from arXiv · showhide
We present BPEmb, a collection of pre-trained subword unit embeddings in 275 languages, based on Byte-Pair Encoding (BPE). In an evaluation using fine-grained entity typing as testbed, BPEmb performs competitively, and for some languages bet- ter than alternative subword approaches, while requiring vastly fewer resources and no tokenization. BPEmb is available at https://github.com/bheinzerling/bpemb
1. Introduction
BPEmb addresses the challenge of representing rare and unseen words by publishing pre-trained BPE subword embeddings for 275 languages. The collection is evaluated for fine-grained entity typing and is competitive with alternative subword approaches while being more compact and tokenization-free.
- Subword methods address rare or unseen words by representing them through parts such as characters, n-grams, or byte pairs.Earlier systems commonly replaced unknown words with a generic UNK token.
- BPEmb publishes pre-trained byte-pair embeddings for 275 languages.
- BPEmb is evaluated in a fine-grained entity typing task.
- BPEmb performs as well as alternative approaches and better for some languages, while requiring fewer resources and no tokenization.
2. BPEmb: Byte-pair Embeddings
BPEmb applies Byte Pair Encoding to untokenized Wikipedia text and pre-trains embeddings for the resulting subword symbols across 275 languages. It provides multiple merge-operation settings and embedding dimensions to support controlled comparisons.
- BPE iteratively merges the most frequent symbol pair, producing variable-length subword units.
- BPE requires no preprocessing and can be applied directly to untokenized text.
- BPEmb pre-trains embeddings for BPE symbols from sufficiently large Wikipedias in 275 languages using GloVe.
- BPEmb varies merge operations across 1000 to 200000 and embedding dimensions across 25 to 300.
3. Evaluation: Comparison to FastText and Character Embeddings
The evaluation compares BPEmb with FastText and character embeddings for multilingual fine-grained entity typing. Entity mentions are transformed into subword sequences, embedded, and encoded with averaging, LSTM, or CNN architectures.
- FastText represents words using token embeddings enriched with character n-gram embeddings, including 3- to 6-grams for unknown words.
- Character embeddings represent mentions as sequences of character unigrams, with embeddings learned for the k most frequent characters.
- BPE segmentation can produce meaningful or meaningless units because it is based on frequency, as illustrated for Japanese, Chinese, and English text.
- Fine-grained entity typing assigns one or more of 89 entity types to an entity mention.The task targets rare and long-tail entities that may lack strong representations in token-based embeddings.
- Entity mentions are converted into subword sequences, their embeddings are looked up, and the sequence is encoded into a one-dimensional vector.The encoder can average the sequence or use an LSTM or CNN.
- The evaluation uses entity mentions from Wikidata mapped to Freebase types, with 80000 training and 20000 test instances for English.
4. Results and Discussion
BPEmb outperforms other subword units on English entity typing and remains competitive across languages, while using substantially less memory. Performance varies with language, merge operations, embedding dimension, and architecture.
- 4.1. Subwords vs. Characters vs. Tokens: 0.624 ± 0.029 mean score for BPE-RNN, versus 0.617 ± 0.007 for FastText-RNN on English entity typing.BPEmb outperformed all other subword units across architectures; the FastText difference was significant at p < 0.001.
- 4.1. Subwords vs. Characters vs. Tokens: Figure 1 summarizes 1000+ hyper-parameter trials using score distributions, medians, and quartiles for each subword unit and architecture.The distributions are smoothed and cut at their highest and lowest scores.
- 4.1. Subwords vs. Characters vs. Tokens: 11 MB for 100k BPE embeddings with dimension 25, compared with 6 GB for FastText’s 3 million embeddings with dimension 300.BPEmb performed well with low embedding dimensionality and could match FastText with a fraction of its memory footprint.
- 4.1. Subwords vs. Characters vs. Tokens: Figure 2 examines how BPE merge-operation count and embedding dimension affect English entity typing.These two hyper-parameters are also associated with the compact representation discussed for BPEmb.
- 4.2. Multilingual Analysis: For high-resource languages, FastText and BPEmb perform equally except for a significant English improvement; Tibetan and Lao show small BPEmb gains.Results are more varied for medium- to low-resource Asian languages, with Khmer affected by Unicode control-character inconsistencies.
5. Limitations
The evaluation covers only a few of BPEmb’s 275 languages and uses an artificial context-free setup, so transfer to other tasks remains untested.
- 5. Limitations: Evaluation covered only a few of the 275 available languages because of limited computational resources.The authors identify broader multilingual evaluation as an open limitation.
- 5. Limitations: The entity-typing setup disregards context, although contextual clues can determine an entity’s type.The authors state that how the results transfer to other tasks requires further study.
7. Conclusions
BPEmb provides subword embeddings trained on Wikipedias in 275 languages and performs competitively with other subword approaches. Its lack of tokenization and much smaller size may support use under resource constraints.
- 7. Conclusions: BPEmb is a collection of subword embeddings trained on Wikipedias in 275 languages.
- 7. Conclusions: BPEmb performs as well as, and for some languages better than, other subword-based approaches.The evaluation included entity typing comparisons with alternative approaches.
- 7. Conclusions: BPEmb requires no tokenization and is orders of magnitude smaller than alternative embeddings.The authors identify potential use under resource constraints, including mobile devices.