PaperlayerView exact revision on arXiv

Source-linked AI summary

Weakly Supervised Cross-Lingual Named Entity Recognition via Effective Annotation and Representation Projection

Jian Ni, Georgiana Dinu, Radu Florian

arXiv:1707.02483v1cs.CLcs.IR

TL;DR

The paper tackles the need for expensive, language-specific NER annotation when extending a trained system to a new language. It proposes annotation and representation projection with co-decoding, and reports that the combined systems outperform three other weakly supervised approaches on CoNLL data.

  • Problem

    Cross-lingual NER needs large amounts of costly, time-consuming, and difficult-to-obtain human annotation in each target language.

  • Method

    The paper combines heuristic data selection for annotation projection, word-embedding representation projection for direct model transfer, and two co-decoding schemes.

  • Results

    The combined systems outperform three state-of-the-art cross-lingual NER approaches on CoNLL NER test data.

  • Takeaways & Limitations

    The approaches provide a strong baseline for cross-lingual NER without human annotation in target languages.

  • Takeaways & Limitations

    Annotation projection can be harmed by non-parallel comparable data, while CRFs cannot handle several million training words in the reported setting with over 50 entity types.

Abstract

from arXiv · show

The state-of-the-art named entity recognition (NER) systems are supervised machine learning models that require large amounts of manually annotated data to achieve high accuracy. However, annotating NER data by human is expensive and time-consuming, and can be quite difficult for a new language. In this paper, we present two weakly supervised approaches for cross-lingual NER with no human annotation in a target language. The first approach is to create automatically labeled NER data for a target language via annotation projection on comparable corpora, where we develop a heuristic scheme that effectively selects good-quality projection-labeled data from noisy data. The second approach is to project distributed representations of words (word embeddings) from a target language to a source language, so that the source-language NER system can be applied to the target language without re-training. We also design two co-decoding schemes that effectively combine the outputs of the two projection-based approaches. We evaluate the performance of the proposed approaches on both in-house and open NER data for several target languages. The results show that the combined systems outperform three other weakly supervised approaches on the CoNLL data.

1 Introduction

The paper addresses cross-lingual NER without human annotation in the target language by combining annotation projection, representation projection, and co-decoding. Its combined systems outperform three state-of-the-art cross-lingual approaches on CoNLL NER test data.

  • 1 Introduction: The paper targets cross-lingual NER without human annotation in a new target language, addressing the cost and difficulty of manually creating language-specific training data.Supervised NER systems require substantial manually annotated data and language-specific resources.
  • 1 Introduction: A heuristic, language-independent selection scheme chooses higher-quality projection-labeled NER data from noisy comparable corpora.The scheme is intended to improve target-language accuracy when alignment quality is low and projected labels are noisy.
  • 1 Introduction: Representation projection maps target-language word embeddings into a source-language space so a trained source-language NER system can decode the target language without retraining.The approach creates a universal representation across languages.
  • 1 Introduction: Two co-decoding schemes combine outputs from the two projection-based systems to produce more accurate predictions than either individual system.The paper evaluates the approaches on in-house and open datasets across several target languages.
  • 1 Introduction: The combined systems outperform three state-of-the-art cross-lingual NER approaches on CoNLL NER test data.The comparison covers approaches from Täckström et al. (2012), Nothman et al. (2013), and Tsai et al. (2016).

2 NER Models

The paper introduces CRF, MEMM, and neural-network NER models for sequence labeling. These models differ in normalization, computational trade-offs, and the representations used for cross-lingual transfer.

  • 2 NER Models: NER is formulated as sequence labeling, inferring an NER tag for each word in an input sequence.The paper introduces three NER model families used in its experiments.
  • 2.1 CRFs and MEMMs: A general-order CRF makes each label depend on a fixed number of preceding labels, with feature functions weighted by learned parameters.When the order is one, the model is a linear-chain CRF.
  • 2.1 CRFs and MEMMs: CRFs model conditional probabilities over whole label sequences, whereas MEMMs normalize conditionally at each token.CRFs better handle label bias, while order-o CRF training is more computationally expensive than order-o MEMM training.
  • 2.2 Neural Networks: The neural model uses the target word and surrounding context with locally normalized predictions conditioned on previously assigned tags.Gazetteers and character-level representations are omitted because they may not transfer readily across languages.
  • 2.2 Neural Networks: NN1 uses word embeddings directly, while NN2 adds a prototype-based smoothing layer that computes cosine similarities and returns a weighted prototype average.With smoothing, NN2 tends to have more balanced precision and recall than NN1.

3 Annotation Projection Approach

The annotation projection approach transfers English NER labels to target-language sentences in comparable corpora, then selects higher-quality projected data for target-language training. Experiments show that selection is especially beneficial when alignment quality is low and projection labels are noisy.

  • 3 Annotation Projection Approach: The approach uses comparable English-target-language corpora and trains target-language second-order MEMMs on selected projection-labeled data.Experiments cover Japanese, Korean, German, and Portuguese, using manually annotated target-language test data and coordinate search for thresholds.
  • 3 Annotation Projection Approach: The method applies an English NER system to source sentences and projects recognized entity tags to aligned target-language spans.The resulting target-language sentences receive automatically projected labels without human annotation.
  • 3 Annotation Projection Approach: Projection quality depends on both the English NER system and sentence alignment, while nonparallel comparable data can reduce target-language accuracy.This motivates filtering noisy projected examples rather than using all available weakly labeled data.
  • 3.1 Data Selection Scheme: Annotation quality is estimated from the relative frequency of projected tags for each entity, with sentence quality averaging the scores of projected entities.The frequency table records entities, projected tags, and empirical probabilities; for example, GPE is the most frequent tag for Estados Unidos in the Portuguese snapshot.
  • 3.1 Data Selection Scheme: A language-independent heuristic selects projected sentences using annotation-quality and entity-count thresholds.The thresholds trade off selected-sentence quality, annotation information, and the total number of selected sentences.
  • 3.2 Accuracy Improvements: +12.2 F1 for Japanese and +13.7 F1 for Korean were achieved with data selection over no selection when alignment quality was low.The improvement was statistically significant for Japanese, Korean, and Portuguese.

4 Representation Projection Approach

Representation projection maps target-language word embeddings into English space so one English NER model can decode multiple languages without retraining.

  • Representation projection maps words from any language into English, enabling one universal NER system without retraining for new languages.The approach trains an English NER system using word embeddings and applies it to projected target-language inputs.
  • 4 Representation Projection Approach: The word-embedding model concatenates surrounding context words and applies distance-decaying context weights when constructing representations.The embeddings are trained separately for English and target languages before cross-lingual mapping.
  • 4.2 Cross-Lingual Representation Projection: The method learns a weighted linear mapping from target-language embeddings to English embeddings using word-aligned dictionary pairs.Pair weights reflect relative translation frequencies, giving more frequent pairs greater importance.
  • 4.2 Cross-Lingual Representation Projection: The mapping projects a target-language vector v into English space as M_f→e v, where the English NER model then predicts entity tags.Unknown target-language words use matching English embeddings when available or an NER-trained UNK vector otherwise.
  • 4.2 Cross-Lingual Representation Projection: The dictionary is limited to approximately 5K–6K word pairs because larger dictionaries may harm direct-transfer performance.

5 Co-Decoding

Co-decoding combines annotation- and representation-projection outputs to improve NER predictions by exploiting their complementary confidence and precision–recall behavior.

  • 5 Co-Decoding: The two systems use different training data and models, allowing their output views to be combined into a potentially more accurate prediction.
  • 5 Co-Decoding: Exclude-O confidence co-decoding addresses the problem that confidence scores from the two weakly supervised systems may not be directly comparable.The comparison is especially problematic between O tags and entity tags.
  • 5 Co-Decoding: The rank-based scheme keeps all annotation-projection entities, then adds nonconflicting representation-projection entities to improve recall.It prioritizes the annotation-projection system because it tends toward higher precision, while representation projection contributes additional entities.
  • 5 Co-Decoding: Representation-projection entities are added only when the annotation-projection output assigns O to their entire span, avoiding conflicts between overlapping or differently tagged entities.Conflicts include overlapping spans or identical spans with different NER tags.

6 Experiments

The experiments evaluate annotation projection, representation projection, and two co-decoding schemes on in-house and CoNLL NER data. Combined projection systems achieve the strongest reported weakly supervised results across the evaluated settings.

  • Evaluation data: The evaluation covers fine-grained in-house NER data for four languages and CoNLL data for Spanish, Dutch, and German.The in-house test sets contain over 50 entity types, whereas CoNLL uses PER, ORG, LOC, and MISC.
  • In-house NER data: The rank-based AP+NN2 system achieves the highest weakly supervised F1 for Korean, German, and Portuguese, and the second highest for Japanese.It improves over the better individual projection system by 2.2 to 7.4 F1 score.
  • In-house NER data: On in-house data, AP has relatively high precision and low recall, while NN2 produces more balanced precision and recall than NN1.NN2 includes a smoothing layer, and rank-based co-decoding is reported as the more effective combination scheme in this setting.
  • Evaluation data: Weakly supervised systems perform below target-language supervised learning, but pre-annotation improves annotation speed by 40%-60%.The authors frame weak supervision as useful when target-language annotation is unavailable and for reducing annotation cost when annotation is feasible.
  • CoNLL NER data: The exclude-O confidence-based AP+NN1 system achieves the highest development-set F1 for Spanish and Dutch and the second highest for German.It improves over the better individual projection system by 1.5 to 3.4 F1 score.
  • CoNLL NER data: The CoNLL test comparison reports that the proposed top systems outperform three prior state-of-the-art cross-lingual NER approaches.The comparison uses confidence- or rank-based co-decoding of AP and NN1 selected using development data.

7 Related Work

Related work covers annotation projection, multilingual resources, direct model transfer, and cross-lingual embedding construction. These approaches differ in whether they create weak labels or transfer language-independent representations and models.

  • Annotation projection: Traditional annotation projection transfers NER tags across language pairs using parallel corpora or translations.A related variant projects tag expectations and uses them as constraints for generalized-expectation model training.
  • Multilingual resources: Wikipedia-based approaches generate weakly labeled multilingual NER data using entity categories, Freebase attributes, or multilingual entity-type mappings.The mappings can provide decoding constraints or dictionary features for multilingual NER systems.
  • Direct model transfer: Direct model-transfer methods train source-language NER systems with cross-lingual word clusters or language-independent labels and features.These resources are derived from monolingual data, aligned parallel data, cross-lingual wikification, or multilingual Wikipedia.
  • Cross-lingual embeddings: Cross-lingual embedding methods either align separately trained monolingual spaces with seed dictionaries or learn inter-lingual representations from mixed-language corpora.The available supervision can include aligned sentences or documents.

8 Conclusion

The paper develops annotation and representation projection approaches for cross-lingual NER without target-language human annotation. Their combined systems outperform three state-of-the-art cross-lingual NER approaches and provide a strong baseline.

  • Conclusion: The paper combines effective annotation projection, representation projection, and two co-decoding schemes for weakly supervised cross-lingual NER.The systems target settings with no human annotation in the target languages.
  • Conclusion: The combined systems outperform three state-of-the-art cross-lingual NER approaches.The authors present them as a strong baseline for building cross-lingual NER systems without target-language human annotation.
Loading 1707.02483v1…