Source-linked AI summary
Character-Aware Neural Language Models
Yoon Kim, Yacine Jernite, David Sontag, Alexander M. Rush
TL;DR
Neural language models can suffer from missing subword information, especially for rare words and morphologically rich languages. This paper replaces input word embeddings with character-based representations fed into an RNN-LM. The model matches English PTB state of the art with 60% fewer parameters and outperforms word- and morpheme-level LSTM baselines across several morphologically rich languages.
Problem
Word-level neural language models overlook subword structure, making rare-word representations poorly estimated and limiting performance in morphologically rich languages.
Method
The model uses character-level CNN representations as inputs to a recurrent neural network language model, without input word embeddings or morphological tagging.
Results
The model matches existing state of the art on English PTB with approximately 60% fewer parameters and outperforms word-level and morpheme-level LSTM baselines in six morphologically rich languages.
Takeaways & Limitations
Character-only inputs can encode rich semantic and orthographic features while supporting competitive word-level language modeling with fewer parameters.
Abstract
from arXiv · showhide
We describe a simple neural language model that relies only on character-level inputs. Predictions are still made at the word-level. Our model employs a convolutional neural network (CNN) and a highway network over characters, whose output is given to a long short-term memory (LSTM) recurrent neural network language model (RNN-LM). On the English Penn Treebank the model is on par with the existing state-of-the-art despite having 60% fewer parameters. On languages with rich morphology (Arabic, Czech, French, German, Spanish, Russian), the model outperforms word-level/morpheme-level LSTM baselines, again with fewer parameters. The results suggest that on many languages, character inputs are sufficient for language modeling. Analysis of word representations obtained from the character composition part of the model reveals that the model is able to encode, from characters only, both semantic and orthographic information.
Introduction
Traditional count-based language models suffer from data sparsity, while neural models address it but overlook subword structure. The paper proposes a character-based language model that avoids word embeddings and performs competitively with fewer parameters.
- Count-based language models estimate n-gram probabilities from counts, but rare n-grams remain poorly estimated because of data sparsity.Smoothing does not fully resolve the estimation problem for rare n-grams.
- Neural language models reduce n-gram sparsity by representing words as learned vectors and feeding them into neural networks.These word embeddings also place semantically similar words near one another in vector space.
- Word-level neural models overlook subword structure, causing poorly estimated rare-word embeddings and high perplexities, especially in morphologically rich languages.The issue also affects domains with dynamic vocabularies, such as social media.
- The proposed model uses a character-level CNN as input to an RNN language model, without morphological tagging or word embeddings at the input layer.Because word embeddings account for much of the parameter count in neural language models, this design substantially reduces model size.
- On English PTB, the model matches existing state of the art with approximately 60% fewer parameters.
- On Arabic, Czech, French, German, Spanish, and Russian, the model outperforms word-level and morpheme-level LSTM baselines with fewer parameters.
Model
The model replaces conventional word embeddings with character-derived representations, processes them through a highway network and multilayer LSTM, and predicts words with a softmax. Character convolutions capture n-gram features while highway layers model their interactions and adaptively carry input dimensions.
- Character-level Convolutional Neural Network: The model replaces conventional word embeddings with a character-level CNN using max-over-time pooling as the RNN-LM input.Character embeddings are arranged into a word matrix, convolved with filters, and pooled into a fixed-dimensional representation.
- Recurrent Neural Network: The resulting word representation is supplied to a multilayer LSTM, which maintains hidden and memory states to model sequential dependencies.LSTM memory cells are additive over time, alleviating gradient vanishing, while additional layers receive the preceding layer’s hidden state.
- Recurrent Neural Network Language Model: An affine transformation followed by a softmax converts the LSTM hidden representation into a distribution over the next word, trained by sequence negative log-likelihood.Predictions remain word-level, so word embeddings are still used in the output layer.
- Character-level Convolutional Neural Network: Multiple filters of varying widths produce features that capture character n-grams, with max pooling selecting the highest-valued feature for each filter.A filter’s width determines the size of the character n-gram it can detect.
- Highway Network: The highway network transforms the character-CNN representation while allowing some input dimensions to pass directly through carry gates.Its transform gate and complementary carry gate adaptively combine transformed and original features.
Experimental Setup
The experiments evaluate language models with perplexity across corpora of different languages and sizes, using PTB for tuning and analysis before transferring the chosen hyperparameters to morphologically rich languages. Comparisons include word-, morpheme-, and character-based models, with training and optimization choices adapted to dataset scale.
- Evaluation: Perplexity evaluates models on corpora spanning different languages and dataset sizes.The test corpora statistics are reported separately, and lower perplexity is the evaluation criterion.
- Evaluation: Hyperparameters, introspection, and ablations are conducted on the English Penn Treebank using standard training, validation, and test splits.The PTB version contains approximately 1 million tokens and a 10k-word vocabulary.
- Multilingual evaluation: The PTB-tuned model is applied without language-specific hyperparameter tuning to Czech, German, French, Spanish, Russian, and Arabic.Non-Arabic data use ACL Workshop on Machine Translation splits, while Arabic data come from News-Commentary.
- Optimization: Hierarchical softmax partitions the vocabulary into approximately equal random clusters to accelerate training on large datasets.The probability of a word is decomposed into selecting its cluster and then selecting the word within that cluster; this was unnecessary for DATA-S.
Results
Results show that character-level models perform competitively with fewer parameters on English and outperform word- and morpheme-based alternatives on the evaluated multilingual datasets. The strongest emphasis is on richer-morphology settings, where character models remain smaller while achieving better perplexity.
- English Penn Treebank: The large character model is on par with existing PTB state of the art despite having approximately 60% fewer parameters.The small model significantly outperforms other neural language models of similar size.
- Scope of results: English is used for comparison with prior work, whereas the main contribution is evaluated on languages with richer morphology.The paper notes that English is relatively simple from a morphological standpoint.
- Baseline design: Morpheme models have strictly more parameters than word models because their inputs include word embeddings in addition to morpheme embeddings.The morphological baseline uses an unsupervised morphological tagger as a preprocessing step.
- DATA-S: Character-level models outperform word-level counterparts on DATA-S despite being smaller.They also outperform morphological logbilinear and LSTM models, although gains over morphological LSTMs are more measured.
- DATA-L: Character models again outperform word- and morpheme-based models on DATA-L.Only small models are trained on DATA-L because of memory constraints.
Discussion
The discussion examines how character-based representations, highway layers, and corpus conditions affect performance and interpretability. Character models capture orthographic and semantic structure, outperform word models across tested German settings, and trade speed for fewer parameters.
- Learned Word Representations: Highway layers enable character representations to encode semantic relationships beyond surface similarity.After highway layers, orthographically distinct words such as “you” and “we” become neighbors, while “while” and “though” are also placed near each other.
- Learned Word Representations: Character representations cover out-of-vocabulary words and can accommodate non-standard spelling.OOV words are positioned near words with the same part of speech, while “looooook” suggests potential text-normalization applications.
- Learned Word Representations: Character n-gram filters generally do not correspond to valid morphemes, despite initially being expected to detect morphemes.PCA analysis nevertheless shows differentiation among prefixes, suffixes, and other character n-grams.
- Highway Layers: Highway layers substantially improve performance, while a one-layer MLP performs poorly as a replacement.The authors hypothesize that highway networks adaptively combine local CNN features; one to two highway layers were important, whereas additional layers generally produced similar performance.
- Effect of Corpus/Vocab Sizes: The character model outperforms the word-level model across all tested German corpus and vocabulary sizes, although gains become less pronounced with larger corpora.The comparison uses perplexity reductions when moving from small word-level to character-level models.
- Further Observations: Character inputs alone can outperform combined word-and-character representations, but character processing is slower than word lookup.On PTB, the large character model trained at 1500 tokens/sec versus 3000 tokens/sec for the word-level model; precomputation can trade memory for scoring speed.
Related Work
Prior neural language models represent words with learned vectors, factor embeddings, morphemes, or characters. The paper’s character-composition approach avoids morphological tagging while addressing limitations of word-level and purely character-level models.
- Neural Language Models: Neural language models replace count-based probability estimation with learned word-vector parameterizations.Their induced embeddings place semantically close words near one another.
- Factored and Morpheme-Based Models: Factored neural language models represent words using shared factors such as morphemes, word shape, or part-of-speech annotations.Morpheme-based variants compose word representations from learned morpheme embeddings.
- Character-Level Models: Morpheme-based models require morphological tagging as preprocessing, whereas purely character-level models avoid tagging and can generate novel words.Purely character-level models, however, were generally outperformed by word-level models in prior work.
- Character Composition: Character CNN features have improved several NLP tasks, including tagging, named entity recognition, and text classification, sometimes without word embeddings.Other work also uses character RNNs or bidirectional LSTMs, leaving the relative performance of CNN and LSTM composition models open.
- This Work: The paper’s model uses a character CNN instead of conventional word embeddings and does not require morphological tagging.The CNN output is used as the input to a recurrent language model.
Conclusion
The paper presents a word-prediction language model driven only by character inputs and reports stronger or comparable performance with fewer parameters. Its learned representations encode semantic and orthographic information, while broader task applicability remains future work.
- Conclusion: The model uses only character-level inputs while retaining word-level predictions and outperforms word- and morpheme-embedding baselines with fewer parameters.The conclusion questions whether word embeddings are necessary as inputs for neural language modeling.
- Conclusion: Character composition representations encode both semantic and orthographic features from characters alone.Using the CharCNN and highway layers for representation learning remains an avenue for future work.
- Conclusion: The architecture’s viability for other tasks, such as neural machine translation, remains an open question.The paper specifically suggests investigating it as an encoder/decoder architecture.