PaperlayerView exact revision on arXiv

Source-linked AI summary

Named Entity Recognition without Labelled Data: A Weak Supervision Approach

Pierre Lison, Aliaksandr Hubin, Jeremy Barnes, Samia Touileb

arXiv:2004.14723v1cs.CLcs.LGstat.ML

TL;DR

NER models can be difficult to adapt when target-domain labelled data is unavailable. The paper uses labelling functions and an HMM to create unified annotations for training a sequence model. It reports an improvement of about 7 percentage points in entity-level F1 and outperforms a contextualised-embedding domain-adaptation approach.

  • Problem

    Applying NER in target domains without labelled data makes transfer-learning methods difficult to use.

    Source: §1 · p. 1

  • Method

    Labelling functions annotate target-domain documents, an HMM unifies their outputs while accounting for their accuracies and confusions, and a sequence model trains on the resulting annotations.

    Source: §1 · p. 1

  • Results

    The method boosts NER by about 7 percentage points on entity-level F1 and outperforms Han and Eisenstein (2019)'s unsupervised domain-adaptation approach using contextualised embeddings.

    Source: §5 · p. 9

  • Takeaways & Limitations

    Document-level labelling functions contribute by exploiting texts' internal coherence and narrative structure.

    Source: §5 · p. 9

    The aggregation model does not yet account for potential correlations between labelling functions, some of which can be noisy; evaluation beyond NER remains future work.

    Source: §5 · p. 9

Abstract

from arXiv · show

Named Entity Recognition (NER) performance often degrades rapidly when applied to target domains that differ from the texts observed during training. When in-domain labelled data is available, transfer learning techniques can be used to adapt existing NER models to the target domain. But what should one do when there is no hand-labelled data for the target domain? This paper presents a simple but powerful approach to learn NER models in the absence of labelled data through weak supervision. The approach relies on a broad spectrum of labelling functions to automatically annotate texts from the target domain. These annotations are then merged together using a hidden Markov model which captures the varying accuracies and confusions of the labelling functions. A sequence labelling model can finally be trained on the basis of this unified annotation. We evaluate the approach on two English datasets (CoNLL 2003 and news articles from Reuters and Bloomberg) and demonstrate an improvement of about 7 percentage points in entity-level $F_1$ scores compared to an out-of-domain neural NER model.

1 Introduction

The paper addresses NER in target domains without labelled data by using weak supervision to annotate documents and train a sequence model. It contributes a broad set of labelling functions and an HMM that unifies their noisy outputs.

1 Introduction

  • Labelling functions annotate target-domain documents, an HMM combines their outputs while modelling their accuracies and confusions, and a sequence labelling model trains on the unified annotations.The final model is trained using cross-entropy loss.Source: §1 · p. 1
  • The approach targets settings where no labelled target-domain data is available, making transfer-learning methods difficult to apply.Source: §1 · p. 1
  • Labelling functions can often be ported across domains, unlike manual annotations that must be repeated for each target domain.Source: §1 · p. 2
  • The labelling-function collection spans neural NER models trained on different domains, gazetteers, heuristics, and document-level constraints.The paper also proposes a weak-supervision model for sequence labelling that can include probabilistic predictions.Source: §1 · p. 2

2 Related Work

Related work covers domain adaptation, aggregation of multiple annotation sources, weak supervision, and ensemble learning. The paper distinguishes its approach by applying weak supervision to sequence labelling and aggregating labelling functions once to create training data.

2 Related Work

  • Domain-adaptive fine-tuning uses language-model losses on source and target text before fine-tuning on labelled source data; it outperforms several strong WNUT 2016 NER baselines.Other domain-adaptation work includes pivot-based feature adaptation, low-dimensional transformations, general and domain-dependent features, and multi-task learning.Source: §2 · p. 2
  • Crowd-annotation aggregation methods estimate annotator trustworthiness or jointly learn sequence models, including HMM-based approaches for POS tagging and NER.Other cited approaches use classifier combinations or Bayesian aggregation with variational EM.Source: §2 · p. 2
  • Weak-supervision frameworks combine sources to train a discriminative model, but typically assume independent data points, limiting their adaptation to sequence labelling.The paper contrasts its approach with biomedical NER work that generates candidate spans through an ad-hoc mechanism.Source: §2 · p. 2
  • Safranchik et al. (2020) use linked HMMs and linking rules to group sequence elements into same-tag spans, whereas this paper uses document-level relations and no linking rules.Source: §2 · pp. 2–3
  • Unlike ensemble methods that combine classifiers at prediction time, this approach aggregates labelling functions once to create training data, avoiding repeated execution of costly models and gazetteers.Source: §2 · p. 3

3 Approach

The approach builds target-domain annotations from diverse labelling functions, aggregates their probabilistic outputs with an unsupervised HMM, and trains a sequence labeller on the resulting annotations. It combines local signals with document-level consistency and models uncertainty in both aggregation and training.

3 Approach

  • Labelling functions annotate target-domain documents, an HMM aggregates their outputs into unified labels, and a sequence labeller is trained on those annotations.The aggregation model is estimated without target-domain labelled data.Source: §3 · p. 3, §3.2 · p. 4

3.1 Labelling functions

  • The labelling-function collection combines out-of-domain sequence models, knowledge-base gazetteers, and heuristics for entity types and formats.The experiments use models trained on Ontonotes, CoNLL 2003, the Broad Twitter Corpus, and SEC filings.Source: §3.1 · pp. 3–4
  • Document-level functions exploit textual coherence: repeated strings receive label-frequency distributions, while later substring mentions inherit an earlier entity's label distribution.The consistency function uses the relative frequencies of labels assigned to same-string spans in the document.Source: §3.1 · p. 4

3.2 Aggregation model

  • The aggregator is an HMM whose latent states are underlying labels, whose emissions are conditionally independent labelling-function distributions, and whose states follow Markov transitions.Labelling functions can emit probabilistic predictions, and each has an associated emission model.Source: §3 · p. 3, §3.2 · p. 4
  • Baum-Welch estimates transition and emission parameters without target labels, while restricting each token's candidate states to labels emitted by at least one function aids EM convergence.Informative Dirichlet priors can use counts and transition frequencies from the most reliable labelling function.Source: §3.2 · pp. 4–5

3.3 Sequence labelling model

  • The sequence labeller is trained to minimize expected loss over the HMM's posterior label marginals, equivalent to cross-entropy against its probabilistic labels.The approach does not constrain which sequence-labelling model architecture is used.Source: §3.3 · p. 5

4 Evaluation

The evaluation tests weakly supervised NER on CoNLL 2003 and Reuters/Bloomberg data without using target-domain labelled data. Results examine label-function subsets, aggregated labels, and the trained neural model, alongside practical behavior and efficiency.

4 Evaluation

  • The evaluation uses two English datasets without target-domain labelled data: CoNLL 2003 and Reuters/Bloomberg news sentences.Reuters/Bloomberg data were crowd-annotated; QUANTITY labels were removed because crowd annotations for that label were highly inconsistent.Source: §4 · p. 5, §4.1 · p. 6

4.2 Baselines

  • Baselines include an Ontonotes-trained neural model, majority voting, Snorkel-style aggregation, domain adaptation systems, and alternative mixture models.Other tested neural architectures performed similarly or worse than the transition-based model; ACC and CV estimates did not converge and were omitted from the results tables.Source: §4.2 · p. 6, §4.3 · p. 7

4.3 Results

  • The neural sequence labeller trained on aggregated labels performs close to the HMM-aggregated labels on both datasets.The authors conclude that knowledge from labelling functions can be injected into a standard neural model without substantial loss.Source: §4.3 · p. 7, Table 1 · p. 8, Table 2 · p. 8

4.4 Discussion

  • Without informative priors, Reuters/Bloomberg performance is poor, with F1 at 0.12; informative and non-informative priors perform similarly on CoNLL 2003.The authors cite this contrast as evidence that informative priors are useful for small datasets.Source: §4.4 · p. 7
  • An Ontonotes-trained model labels “Heidrun” as a product, while a document-level function counterbalances the error; “Status Weekly” remains mislabeled as an organisation.The discussion also reports the greatest pairwise overlap among functions predicting all labels, with specialized functions often overlapping less.Source: §4.4 · pp. 7–8

4.3 Results

  • Document-level functions raise entity-level F1 from 0.702 to 0.716 on CoNLL 2003.The authors attribute the gain to the contribution of document-level relations in NER.Source: §4.3 · p. 7, Table 1 · p. 8

4.4 Discussion

  • HMM parameter estimation takes under 30 minutes on all CoNLL 2003 data, and the estimated model applies to new texts with one forward-backward pass.The authors say this can scale to hundreds of thousands of documents and can be ported to other languages, with heuristics and gazetteers adapted to the target language.Source: §4.4 · p. 8

5 Conclusion

The paper combines diverse labelling functions with an HMM to train a neural sequence labeller without target-domain labels. On CoNLL 2003 and Reuters/Bloomberg news, it reports an entity-level F1 improvement of about 7 percentage points and highlights document-level functions.

5 Conclusion

  • The method combines data-driven NER models, gazetteers, heuristics, and document-level entity relations, then uses an HMM to merge their outputs before neural sequence-labeller training.The HMM parameters are estimated with Baum-Welch; functions may specialize in labels or produce probabilistic predictions.Source: §5 · p. 9
  • About 7 percentage points higher entity-level F1 is reported on CoNLL 2003 and Reuters/Bloomberg news, outperforming Han and Eisenstein’s (2019) unsupervised domain adaptation approach.The approach uses document-level functions that exploit texts’ internal coherence and narrative structure.Source: §5 · p. 9
  • Document-level labelling functions are singled out for using the internal coherence and narrative structure of texts.Source: §5 · p. 9
  • Future work identifies potential function correlations, noisy functions and subset selection, and evaluation on sequence-labelling tasks beyond NER as open issues.Source: §5 · p. 9

A Labelling functions

The appendix inventories the labelling functions used in the experiments, covering model-based, gazetteer, heuristic, and document-level functions. It also distinguishes raw neural-model predictions from predictions with shallow post-processing.

A Labelling functions

  • The inventory spans neural models, gazetteers, heuristic detectors, and document-level functions.Source: A Labelling functions · p. 14
  • Neural labelling functions include models trained on the Broad Twitter Corpus, SEC filings, CoNLL 2003, and OntoNotes 5.0, with listed post-processed variants.Source: A Labelling functions · p. 14
  • Gazetteer functions include case-sensitive and case-insensitive lists from Wikipedia, company names, Crunchbase, Geonames, and DBPedia products.Some variants are restricted to multitoken entities.Source: A Labelling functions · p. 14
  • Heuristic functions detect entity types and proper-name patterns using cues such as casing, part-of-speech tags, rarity, and dependency relations.The inventory also includes a probabilistic parser for dates, times, money, percentages, and cardinal or ordinal values.Source: A Labelling functions · p. 14
  • The neural models are listed in raw and shallow-postprocessed versions, with post-processing correcting known recognition errors such as currency-adjacent amounts classified as MONEY.Source: Table 3 · p. 14

B Label matching problem

The appendix addresses how to match latent mixture-model states to NER labels. In the experiments, the authors initialize class probabilities and compare correlation-based label-matching choices.

B Label matching problem

  • The baseline mixtures of multinomials face a label-matching problem when assigning latent states to NER labels.Source: B Label matching problem · p. 15
  • The experimental procedure initializes class probabilities using label frequencies from the most reliable labelling function to encourage EM to explore the nearby mode.Source: B Label matching problem · p. 15
  • States are matched to labels by pairwise correlations using the most reliable function, majority voting, or the Dirichlet dependent mixture model.Source: B Label matching problem · p. 15
  • Correlations below 0.1 trigger the O label, and the majority-voter mapping performed best among the tested label-matching techniques.Source: B Label matching problem · p. 15

C Detailed results

The appendix presents CoNLL 2003 evaluation results broken down by NER label, alongside token- and entity-level metrics. It identifies these as detailed results corresponding to the main paper’s micro-averaged results.

C Detailed results

Related papers

Loading 2004.14723v1…