Source-linked AI summary
Finding Function in Form: Compositional Character Models for Open Vocabulary Word Representation
Wang Ling, Tiago Luís, Luís Marujo, Ramón Fernandez Astudillo, Silvio Amir, Chris Dyer, Alan W. Black, Isabel Trancoso
TL;DR
Traditional word representations use independent vectors for word types, limiting compactness and representation of unseen or morphologically related words. The paper composes character embeddings with bidirectional LSTMs to produce word vectors, achieving state-of-the-art results in language modeling and POS tagging, with especially pronounced benefits in morphologically rich languages.
Problem
Independent word vectors require parameters for every word type and cannot naturally represent unseen words or relationships among related forms.
Method
The C2W model composes shared character embeddings with bidirectional LSTMs to generate fixed-dimensional word representations without an explicit word lookup table.
Results
The model achieves state-of-the-art performance in language modeling and POS tagging, with especially pronounced improvements in morphologically rich languages.
Takeaways & Limitations
Character composition can learn lexical and morphological information compactly while remaining effective for POS tagging and language modeling across languages.
Takeaways & Limitations
The language-model softmax remains closed-vocabulary, so out-of-vocabulary words cannot be directly addressed at test time.
Abstract
from arXiv · showhide
We introduce a model for constructing vector representations of words by composing characters using bidirectional LSTMs. Relative to traditional word representation models that have independent vectors for each word type, our model requires only a single vector per character type and a fixed set of parameters for the compositional model. Despite the compactness of this model and, more importantly, the arbitrary nature of the form-function relationship in language, our "composed" word representations yield state-of-the-art results in language modeling and part-of-speech tagging. Benefits over traditional baselines are particularly pronounced in morphologically rich languages (e.g., Turkish).
1 Introduction
The paper argues that independent word vectors are problematic for morphologically rich languages and proposes character composition to model both regular and non-compositional form–function relationships. Experiments evaluate this approach in language modeling and POS tagging, finding strong performance and especially large gains in morphologically rich languages.
- Motivation: Independent word vectors fail to represent relationships between related word forms and cannot generate representations for unseen words.The paper identifies these limitations as especially problematic in morphologically rich languages.
- Motivation: Orthographic similarity is neither necessary nor sufficient for functional similarity, so the model must capture both regular morphology and non-compositional effects.Examples include butter versus batter and rich versus affluent.
- Results: The character-based model generates similar representations for semantically and syntactically similar words despite orthographic differences.The introduction gives October and January as an example.
- Results: The model improves over word lookup tables using only a fraction of their parameters in two tasks.The experiments cover language modeling and part-of-speech tagging across several languages.
- Results: The model achieves state-of-the-art POS-tagging performance, including a new best result in English.Performance improvements are especially dramatic in morphologically rich languages.
2 Word Vectors and Wordless Word Vectors
Traditional word lookup tables assign independent parameters to word types, creating problems for vocabulary size, unseen words, and morphological generalization. The paper instead motivates composing smaller linguistic units, ultimately choosing characters because form–meaning relationships are partly regular and partly arbitrary.
- Independent Parameters: Word lookup tables assign each vocabulary word an independent vector, with d×|V| parameters overall.A 20-million-word-type English Wikipedia vocabulary would require a separate vector for each type.
- Independent Parameters: Independent lookup tables do not encode that recurring form changes, such as adding s, can correspond to recurring linguistic transformations.They therefore cannot construct representations for unseen forms from observed components such as French and -ification.
- Compositional Representations: Related word forms are not treated as fully independent in the paper’s account of lexical knowledge.The discussion connects this view to debates about how inflectional processes are represented.
- Compositional Representations: Composing smaller pieces into word representations offers an alternative to storing a separate parameter vector for every word type.Prior approaches compose morphemes, which are meaning- or syntax-bearing units.
- Compositional Representations: The paper chooses characters as compositional units, while acknowledging that lexical form and meaning have a non-trivial relationship.Morphological processes can be regular, whereas pairs such as lesson–lessen and coarse–course can remain difficult to interpret compositionally.
3 C2W Model
The C2W model replaces a word lookup table with a bidirectional LSTM that composes character representations into a fixed-dimensional word vector. Character embeddings are processed in both directions, and the resulting states are combined to produce the representation, with caching used for efficiency.
- Model Architecture: The C2W model takes a word type as input and outputs a d-dimensional vector that can replace a word-lookup representation in a network.Its bidirectional LSTMs are designed to learn complex non-local dependencies over character sequences.
- Character Inputs: Each character is represented by a one-hot vector and projected through a shared character lookup matrix into a dC-dimensional embedding.The projection can capture similarities among character types, such as vowels and consonants.
- Bidirectional Composition: The LSTM updates hidden states iteratively while its gates control input inclusion, forgetting, and relevance to the current state.The cell memory is propagated through the sequence and combined linearly at each time step.
- Bidirectional Composition: A forward LSTM processes the character sequence and a backward LSTM processes its reverse, producing forward and backward state sequences.The two LSTMs use different parameter sets.
- Output Representation: The word representation is formed by combining the forward and backward states using learned combination parameters.The composition model therefore returns the same kind of word-level vector interface as a lookup table.
- Efficiency: Caching repeated composed word representations reduces redundant computation during training and inference.The cache size can be adjusted to trade memory use against computational performance.
4 Experiments: Language Modeling
The experiments test whether character-composed word representations can support language modeling despite irregular form–function relationships, while addressing out-of-vocabulary words and parameter efficiency. C2W consistently outperforms word lookup tables, with especially pronounced gains in Turkish, while retaining comparable training speed.
- Model: The language model composes preceding word representations with an recurrent LSTM and predicts the next word using a vocabulary-sized softmax.C2W replaces the word lookup table with a character-composed representation while preserving the surrounding language-model architecture.
- Evaluation caveat: The closed-vocabulary softmax cannot directly address test-time OOV words, and absolute perplexity is not informative across models with different vocabulary sizes.Relative perplexity remains useful for comparing how well models predict words from their contexts.
- Experimental setup: The evaluation covers English, Portuguese, Catalan, German, and Turkish using Wikipedia data spanning different morphological typologies.The setup uses 1 million training words per language and restricts the output vocabulary to the 5000 most frequent words.
- Results: C2W always outperforms word lookup tables in language-model perplexity, with improvements especially pronounced in Turkish.The paper links the Turkish advantage to suffix-dependent meaning differences, such as evde versus evden.
- Efficiency: Approximately 300 words per second are processed during training by both representations despite the C2W model's character composition.The vocabulary softmax is the main bottleneck, and caching avoids composing the same word type twice within a batch.
- Nonce-word representations: The character model can produce embeddings for nonce words and generalizes better than lookup tables that use unknown tokens for OOV words.Similarity is measured against in-vocabulary words using cosine similarity.
5 Experiments: Part-of-speech Tagging
The POS tagger composes character-based word representations with a bidirectional word-level LSTM, then predicts tags from combined contextual states. Across languages, C2W generally outperforms word lookup tables, with especially strong Turkish results, while retaining a compact parameterization.
- Bi-LSTM Tagging Model: Character-based word features are fed into a bidirectional LSTM over words, whose combined states are projected to POS-tag probabilities.The model can instead use hand-engineered features, and its forward, backward, and combined-state sizes are hyperparameters.
- English PTB Results: 97.29% test accuracy is achieved by the bidirectional LSTM C2W model on English PTB, surpassing the word lookup table.Regular RNNs perform approximately 2% worse than LSTM counterparts, while the bidirectional LSTM gives the best result among compared recurrent variants.
- Model Size: The C2W model uses roughly 4k character-lookup parameters plus about 150k compositional parameters, versus approximately 2 million parameters for a 50-dimensional word table.The comparison uses approximately 40k lowercased PTB word types and about 80 character types.
- Multiple Languages: Across five languages, C2W consistently performs better than word lookup tables and is comparable to or slightly better than Stanford’s POS tagger.On Turkish, the improvement is significantly higher than 4%, despite using the same training data and no handcrafted features.
- Benchmark Comparison: On English PTB, C2W without feature engineering is on par with the current state-of-the-art system, while adding handcrafted features improves results further.The benchmark comparison distinguishes feature engineering and additional data usage.
- Limitations and Extensions: Words lacking morphological cues, such as snake, may require contextual evidence or pretrained embeddings to determine their POS tags.The paper illustrates this with cat, dog, and snake, where related pretrained representations can transfer noun evidence.
6 Related Work
Prior work augments word lookup tables with lexical features, while character-compositional models learn such features directly. The C2W model addresses compactness and character-order limitations through subword regularities.
- Contribution: The authors position their work as learning representations without word lookup tables while reducing redundancy in those tables.The model combines automatic lexical-feature learning with parameter compactness.
- Character-based representations: Character-based models learn lexical features directly from word forms rather than relying only on predefined feature sets.Morphological analyzers restrict models to defined features, whereas compositional character models can extract meaningful lexical features automatically.
- Character-based representations: Summing character embeddings can collapse words with different orders, while convolutional models reduce this problem by combining character windows.The related models differ in how much character order they preserve.
- Model compactness: The C2W approach learns subword regularities that support more compact word representations as training data and model size grow.This connects character composition to broader efforts to compact large language-modeling and translation models.
- Applications: Earlier work also found similar improvements from the approach in dependency parsing, especially in morphologically rich languages.This extends the reported application scope beyond language modeling and POS tagging.
7 Conclusion
The conclusion presents C2W as a character-based replacement for explicit word lookup tables. It reports competitive POS tagging without manual lexical features and sensitivity to both morphological and non-compositional patterns.
- Model: C2W builds word embeddings without an explicit word lookup table by using characters as atomic units.The resulting embeddings remain sensitive to lexical aspects within words.
- Results: On POS tagging, character-only models achieve comparable or better results than state-of-the-art systems without manually engineered lexical features.The conclusion frames this as automatic learning of features from task-specific data.
- Interpretation: Success in languages with impoverished morphological cues indicates that the model learns non-compositional aspects of how letters fit together.Both language modeling and POS tagging benefit from morphological cues, but the conclusion emphasizes broader form-based learning.
- Availability: The implementation code for the C2W model, language model, and POS tagger is publicly available.The paper gives a GitHub repository for these implementations.