Source-linked AI summary
Neural Cross-Lingual Named Entity Recognition with Minimal Resources
Jiateng Xie, Zhilin Yang, Graham Neubig, Noah A. Smith, Jaime Carbonell
TL;DR
The paper studies unsupervised cross-lingual NER when target languages lack annotated data and lexical and word-order differences impede transfer. It learns translations through shared bilingual embeddings and adds self-attention, achieving state-of-the-art or competitive results with fewer resources while exposing challenges in Uyghur.
Problem
Unsupervised cross-lingual NER must transfer knowledge without labeled target-language data despite lexical mapping and word-order differences across languages.
Method
The method finds discrete word translations through nearest neighbors in a shared bilingual embedding space and incorporates self-attention into the neural architecture.
Results
The combined methods achieve state-of-the-art or competitive cross-lingual NER results on commonly tested languages with lower resource requirements than past approaches.
Takeaways & Limitations
Embedding-based translation and self-attention provide a low-resource approach for cross-lingual NER, while the paper evaluates its challenges in extremely low-resource Uyghur.
Takeaways & Limitations
Uyghur results expose challenges for multilingual embedding methods when embeddings are low quality, languages are distant, and text distributions diverge.
Abstract
from arXiv · showhide
For languages with no annotated resources, unsupervised transfer of natural language processing models such as named-entity recognition (NER) from resource-rich languages would be an appealing capability. However, differences in words and word order across languages make it a challenging problem. To improve mapping of lexical items across languages, we propose a method that finds translations based on bilingual word embeddings. To improve robustness to word order differences, we propose to use self-attention, which allows for a degree of flexibility with respect to word order. We demonstrate that these methods achieve state-of-the-art or competitive NER performance on commonly tested languages under a cross-lingual setting, with much lower resource requirements than past approaches. We also evaluate the challenges of applying these methods to Uyghur, a low-resource language.
1 Introduction
The paper addresses unsupervised cross-lingual NER by combining low-resource lexical translation with self-attention to handle cross-language word-order differences. It reports strong Spanish and Dutch results, competitive German performance, and an evaluation on Uyghur.
- Motivation: Unsupervised cross-lingual NER transfers labeled knowledge from a high-resource source language to a target language with few or no labels.The paper focuses on the extreme setting where no labeled target-language data is available.
- Lexical mapping: The proposed lexical mapping projects multilingual embeddings into a shared space, retrieves nearest-neighbor translations, and trains on the translated data.This combines the low resource requirements of embeddings with dictionary-style discrete translations and character-level information.
- Word-order robustness: Self-attention is added to the neural architecture to account for word-order differences between source and target languages.The mechanism allows information within an encoded sequence to be reordered.
- Results: The approach outperforms previous state-of-the-art results on Spanish and Dutch and remains competitive on German without parallel resources.The best results occur when self-attention is added, and similar results are obtained with different seed dictionaries.
2 Approach
The approach transfers NER without target-language labels by projecting bilingual embeddings into a shared space, translating source words with nearest neighbors, and training on translated data. It also uses self-attention to address cross-lingual word-order differences, while acknowledging that imperfect embedding alignment remains challenging.
- Problem setting: Unsupervised transfer uses source-language labels, monolingual corpora in both languages, and a small or unsupervisedly induced dictionary.No labeled target-language data is used.
- Lexical mapping: The method trains separate monolingual embeddings, projects them into a shared space, translates source words by nearest-neighbor search, and trains NER on the translated data.English entity tags are copied onto the corresponding translated target words.
- Bilingual embeddings: An orthogonal transformation aligns dictionary word pairs while preserving dot-product similarity within and across languages.The shared representations are formed as X′ = XV and Y′ = YU after solving the Procrustes problem.
- Bilingual embeddings: The alignment is refined through mutual-nearest-neighbor dictionaries and repeated Procrustes solutions before translation.The paper uses k = 3 refinement steps.
- Lexical mapping: CSLS selects each source word’s target translation while addressing hubness in the shared embedding space.The metric adjusts cosine similarity using average similarities to neighboring words.
- Discussion: Imperfect cross-lingual embedding alignment can still yield correct translations when the correct target word remains closer than incorrect alternatives.The method uses the shared space for translation rather than directly modeling it in the NER system.
- Word-order robustness: Order-invariant self-attention is incorporated into the neural architecture to allow information reordering across languages with different word orders.The NER architecture is a self-attentive Bi-LSTM-CRF model.
3 NER Model Architecture
The NER architecture combines character- and word-level neural representations with self-attention before a linear-chain CRF. Self-attention supplies sentence-level context that is less dependent on word positions.
- Hierarchical CRF: The hierarchical CRF model combines character-level and word-level neural networks with a linear-chain CRF for label dependencies and inference.The character network captures subword information, while the word network produces context-sensitive representations.
- Hierarchical CRF: Each word representation concatenates character information with its word embedding before entering the word-level Bi-LSTM.The model uses bidirectional LSTMs at both character and word levels.
- Self-Attention: Self-attention is added above the word-level Bi-LSTM to provide every word with a context feature vector based on all sentence words.The context vectors are obtained irrespective of word positions, introducing flexibility for word-order differences.
- Self-Attention: Queries and keys are computed from word-level hidden representations using a single-layer MLP with a tanh transformation.The parameter matrix has dimensions d×d, where d is the hidden-state size.
- Self-Attention: The attention mask (E − I) prevents each word’s attention weights from centering on itself, encouraging sentence-level context.E is an all-one matrix and I is the identity matrix.
- Self-Attention: The attention outputs are concatenated with the original hidden representations and supplied as the final inputs to the CRF layer.Each final representation pairs a word-level state with its corresponding self-attention output.
4 Experiments
The experiments test unsupervised English-to-target NER transfer across European benchmark languages and Uyghur, using bilingual-embedding translation, self-attention, and multiple resource settings. Results are strong for Spanish and Dutch, competitive for German, and competitive for Uyghur with substantially fewer resources, while revealing failures for some embedding approaches.
- Experimental setup: The experiments vary dictionaries, embedding-based translation procedures, comparison baselines, ablations, and the low-resource Uyghur setting.Seed dictionaries include identical strings, adversarial learning, and provided dictionaries.
- Experimental setup: The evaluation transfers English NER models to Spanish, Dutch, and German on CoNLL data, then tests the approach on Uyghur.Models are trained on translated English data and evaluated in the target language; Uyghur evaluation uses 199 annotated documents.
- Cross-lingual results: The proposed method outperforms previous state-of-the-art results on Spanish and Dutch and remains competitive on German without parallel resources.Different seed dictionaries produce similar results, while adding self-attention gives the best results.
- Dictionary comparison: Using the neural model on cheap-translation data outperforms the baseline on all three languages, with smaller dictionaries and sometimes no dictionary.Self-attention also helps on data whose word order is not correct.
- Ablation study: The ablation shows that translation using target-space word embeddings and target-language character sequences outperforms the common-space variant.Target character sequences help especially for German, where capitalization patterns differ from English.
- Uyghur case study: On Uyghur, the method is competitive with a much smaller dictionary and no Uyghur Wikipedia knowledge, especially when embedding translation supplements the comparison system’s dictionary.Adversarial learning and identical-character methods fail at around 10 F1, attributed to low-quality embeddings, distant languages, and Arabic script.
5 Related Work
Cross-lingual NER work transfers knowledge through annotation projection, language-independent features, or multilingual training. These approaches differ in their resource requirements, especially whether target-language labels are needed.
- Annotation projection: Annotation projection creates target-language training data by projecting source annotations through parallel corpora.
- Language-independent transfer: Language-independent transfer trains models with delexicalized features such as word clusters, gazetteers, Wikipedia categories, or projected word embeddings.
- Language-independent transfer: These transfer methods train on the source language and apply the resulting model directly to the target language.
- Multitask learning: Multitask approaches share parameters across languages but usually require some target-language training data for bootstrapping.
- Bilingual word embeddings: Bilingual word embeddings can be learned through parallel-corpus methods or joint training with cross-lingual objectives.
6 Conclusion
The paper proposes shared-space translation and self-attention for unsupervised cross-lingual NER. Combined, these methods achieve state-of-the-art or competitive results with lower resource requirements, while also exposing challenges in Uyghur.
- The paper finds word translations in a shared embedding space built from a seed lexicon to address lexical mapping.
- The model adds self-attention to alleviate word-order divergence across languages.
- Combined methods achieve state-of-the-art or competitive results on commonly tested languages with lower resource requirements than past approaches.
- The paper evaluates the challenges of applying these methods to the extremely low-resource language Uyghur.