Source-linked AI summary
Bilingual Character Representation for Efficiently Addressing Out-of-Vocabulary Words in Code-Switching Named Entity Recognition
Genta Indra Winata, Chien-Sheng Wu, Andrea Madotto, Pascale Fung
TL;DR
Code-switching Twitter NER must handle context-dependent meanings, noisy text, and entities in a different language from the matrix language. The paper uses a hierarchical LSTM model with bilingual character representations, transfer learning, and preprocessing; it reports 62.76% F1-score for English-Spanish without gazetteers or knowledge-based information.
Problem
Code-switching tweets present context-dependent word meanings, noisy text that contributes to OOV words, and entities expressed in a language other than the matrix language.
Source: §1 · p. 1, §3.2 · p. 2Method
Results
62.76% F1-score was achieved for English-Spanish without gazetteers or knowledge-based information.
Source: §5 · p. 4Takeaways & Limitations
The model can be easily adapted to another language pair without gazetteers or knowledge-based information.
Source: §3.3 · p. 3
Abstract
from arXiv · showhide
We propose an LSTM-based model with hierarchical architecture on named entity recognition from code-switching Twitter data. Our model uses bilingual character representation and transfer learning to address out-of-vocabulary words. In order to mitigate data noise, we propose to use token replacement and normalization. In the 3rd Workshop on Computational Approaches to Linguistic Code-Switching Shared Task, we achieved second place with 62.76% harmonic mean F1-score for English-Spanish language pair without using any gazetteer and knowledge-based information.
1 Introduction
Code-switching Twitter NER is challenged by context-dependent word meanings, noisy spelling, and entities in a language different from the matrix language. The paper combines bilingual character representations with transfer learning to address OOV words and adapt to bilingual domains.
1 Introduction
- Code-switching tweets combine context-dependent word meanings, social-media noise that contributes to OOV words, and entities expressed in a language other than the matrix language.For example, “cola” can mean a product or a Spanish “queue,” while “Westland Mall” is an English entity in a Spanish phrase.Source: §1 · p. 1
- A bilingual character bidirectional RNN captures character-level information to tackle OOV words in code-switching NER.The model also uses an LSTM for long-range dependencies in word and character sequences.Source: §1 · p. 1
- Transfer learning from monolingual pre-trained word vectors adapts the model to different domains in a bilingual setting.Source: §1 · p. 1
2 Related Work
Prior NER work explored neural decoders, character-level features, and language-specific resources. The cited studies include BiLSTM and CRF approaches, as well as multitask learning with gazetteers.
2 Related Work
- BiLSTM was introduced for NER after CNNs had been used as word decoders.The passage names Collobert et al. (2011) for CNNs and Huang et al. (2015) for BiLSTM.Source: §2 · p. 1
- Neural character-level features replaced hand-crafted features, and character embeddings were used to address OOV words in NER, POS tagging, and language modeling.The cited work also used CRF decoders to improve results and Stack memory-based LSTMs for sequence chunking.Source: §2 · p. 1
- Aguilar et al. (2017) combined POS tagging with NER in multitask learning and used gazetteers for language-specific knowledge.Source: §2 · p. 1
3 Methodology
The methodology combines bilingual word and character representations with preprocessing and transfer learning to address OOV words in noisy English-Spanish tweets. It uses a bidirectional LSTM architecture, repairs some predicted tag sequences, and evaluates the model on the ENG-SPA tweet dataset.
3.1 Dataset
- The ENG-SPA Twitter dataset has nine IOB-formatted entity labels, with Person, Location, and Product most frequent and Time, Event, and Other least common.The paper identifies Other as the least clustered category.Source: §3.1 · pp. 1–2
3.2 Feature Representation
- Preprocessing replaces hashtags, mentions, and URLs with placeholders and normalizes OOV tokens using capitalization, lowercasing, and repeated-character removal heuristics.The authors report that FastText and preprocessing reduce the OOV rate, especially when English and Spanish vocabularies are concatenated.Source: §3.2 · p. 2
3.3 Model Description
- The model concatenates fixed FastText word vectors with bilingual character representations, then passes the combined sequence through a bidirectional LSTM to predict entity tags.The word representation uses English and Spanish vectors; the character encoder uses the last hidden state for each word.Source: §3.2 · p. 2, §3.3 · p. 3
3.4 Post-processing
- Post-processing inserts a missing I-tag between matching B- and I-tags and changes a B-tag when paired with an I-tag of another category; it improved development-set predictions.Figure 3 is identified as showing examples of the post-processing.Source: §3.4 · pp. 3–4, Figure 3 · p. 4
3.5 Experimental Setup
- The models use 200 hidden units, batch size 64, 300-dimensional word embeddings, 150-dimensional character embeddings, and dropout of 0.4 across LSTMs.The best model was selected using the development set and evaluated on the test set with harmonic mean F1-score.Source: §3.5 · p. 4
4 Results
For ENG-SPA tweets, adding pretrained word vectors and character features improved performance, with character features helping after Char-RNN dropout. FastText transfer learning further improved results and reduced OOV words; the gap to first place was approximately 1%.
4 Results
- FastText transfer learning significantly improved model performance and reduced OOV words in the development and test sets.The margin between the model and the first-place system was approximately 1%.Source: §4 · p. 4
- Character-level features improved performance only after dropout was applied to the Char-RNN.Source: §4 · p. 4
- Spanish FastText subword representations did not improve results because OOV words often contained special characters that could produce noisy vectors, and most were not entity words.Source: §4 · p. 4
5 Conclusion
The paper concludes that a hierarchical bidirectional LSTM with bilingual character representations addresses OOV words in English-Spanish code-switched tweets. Token replacement, normalization, and transfer learning further reduce OOV rates and improve performance, with a 62.76% F1-score achieved without gazetteers or knowledge-based information.
5 Conclusion
- The hierarchical bidirectional LSTM uses a bilingual character RNN to address OOV words.Source: §5 · p. 4
- Token replacement, token normalization, and transfer learning further reduce OOV rates and significantly improve performance.Source: §5 · p. 4
- 62.76% F1-score was achieved for English-Spanish without gazetteers or knowledge-based information.Source: §5 · p. 4