PaperlayerView exact revision on arXiv

Source-linked AI summary

Learning to Contextually Aggregate Multi-Source Supervision for Sequence Labeling

Ouyu Lan, Xiao Huang, Bill Yuchen Lin, He Jiang, Liyuan Liu, Xiang Ren

arXiv:1910.04289v2cs.CLcs.LG

TL;DR

Sequence labeling depends on high-quality annotations, yet gold labels are costly and sources of imperfect supervision may have different strengths. ConNet learns source-specific representations and contextually aggregates them, with reported improvements over existing methods in crowd annotation and unsupervised cross-domain adaptation.

  • Problem

    Sequence labeling relies on high-quality annotations, while sources can have different strengths across situations and their relative value may depend on sentence context.

    Source: §1 · p. 1

  • Method

    ConNet separates shared annotator-invariant patterns from source-specific representations, then uses a context-aware attention mechanism to aggregate them for each sentence.

    Source: §1 · p. 1, §4 · p. 3

  • Results

    ConNet outperforms existing methods in crowd annotation and unsupervised cross-domain adaptation, with state-of-the-art performance on real-world crowdsourcing datasets.

    Source: §1 · p. 2

  • Takeaways & Limitations

    The framework also shows promising results on text classification, beyond the sequence-labeling applications evaluated.

    Source: §6 · p. 9

    In the crowd-annotation setting, annotators share an input distribution and target sentences come from the same domain.

    Source: §3 · p. 3

Abstract

from arXiv · show

Sequence labeling is a fundamental framework for various natural language processing problems. Its performance is largely influenced by the annotation quality and quantity in supervised learning scenarios, and obtaining ground truth labels is often costly. In many cases, ground truth labels do not exist, but noisy annotations or annotations from different domains are accessible. In this paper, we propose a novel framework Consensus Network (ConNet) that can be trained on annotations from multiple sources (e.g., crowd annotation, cross-domain data...). It learns individual representation for every source and dynamically aggregates source-specific knowledge by a context-aware attention module. Finally, it leads to a model reflecting the agreement (consensus) among multiple sources. We evaluate the proposed framework in two practical settings of multi-source learning: learning with crowd annotations and unsupervised cross-domain model adaptation. Extensive experimental results show that our model achieves significant improvements over existing methods in both settings. We also demonstrate that the method can apply to various tasks and cope with different encoders.

1 Introduction

ConNet addresses the challenge of learning from imperfect supervision when gold labels are costly or unavailable, using source-specific representations and context-aware aggregation. The paper reports better performance than existing methods in crowd annotation and unsupervised cross-domain adaptation.

  • The key challenge is aggregating imperfect annotations from multiple sources without knowing the target domain’s ground-truth label sequences.The sources may be crowd annotations or annotations from other domains.Source: §1 · p. 1
  • Because sources have different strengths across situations, their relative importance should depend on each sentence’s context.Source: §1 · p. 1
  • ConNet models shared annotator-invariant patterns separately from source-specific biases, then dynamically aggregates source representations to form a consensus.Its source-specific representations capture each annotator’s strengths, and attention selects sources for unseen sentences.Source: §1 · pp. 1–2
  • ConNet outperforms strong baselines across crowd-annotation and unsupervised cross-domain adaptation tasks, achieving state-of-the-art performance on real-world crowdsourcing datasets.The introduction also reports improvements across tasks and encoders.Source: §1 · p. 2

2 Related Work

Related work spans neural sequence labeling, crowd annotation, and unsupervised domain adaptation. Prior approaches include feature-engineered graphical models, methods for estimating or modeling annotator behavior, and techniques for transferring knowledge across domains.

  • Traditional CRFs rely on feature engineering, while CNN and BLSTM models support end-to-end learning; this paper uses a BLSTM-CRF base tagger (Ma and Hovy, 2016a; Lample et al., 2016).Source: §2 · p. 2
  • Skene (1979) proposed the pioneering approach to aggregate crowd annotations and estimate true labels; later works used Expectation-Maximization (EM) to jointly learn the classification model and annotator behavior.Source: §2 · p. 2
  • Crowd-learning work treats crowd annotations as noisy gold labels and models annotator-specific bias, while human-curated annotations can also contain label noise (Rodrigues and Pereira, 2018; Wang et al., 2019).Source: §2 · p. 2
  • Some unsupervised adaptation methods use target-domain unlabeled data for tri-training, while metadata domain vectors depend heavily on prior domain knowledge (Saito et al., 2017; Ruder and Plank, 2018).Source: §2 · p. 2
  • Autoencoder adaptation seeks shared knowledge through target-input reconstruction, whereas adversarial methods learn domain-invariant features but do not exploit domain-specific information (Ghifary et al., 2016; Fernando et al., 2015; Long et al., 2014; Ming Harry Hsu et al., 2015).Source: §2 · p. 3

3 Multi-source Supervised Learning

The section formulates multi-source sequence labeling as training a target tagger from imperfect annotations, with the target either sharing the sources’ distribution or differing from it. It distinguishes same-domain crowd annotation from unsupervised cross-domain adaptation.

3 Multi-source Supervised Learning

  • The learning objective is to use imperfect source annotations to predict tag sequences in a target corpus, which may share the source distribution or differ substantially.Source: §3 · p. 3

Application I: Learning with Crowd Annotations

  • In crowd annotation, workers label subsets of the same dataset, with no input-distribution shift and target sentences drawn from that same domain.Individual annotators may make mistakes or skip sentences.Source: §3 · p. 3

Application II: Unsupervised Cross-Domain Model Adaptation

  • In cross-domain adaptation, annotated source domains are available but the target domain has no labels, and source models may not match target-domain label distributions.Source: §3 · p. 3

4 Consensus Network

CONNET separates shared sequence-labeling behavior from source-specific representations, then uses sentence context to aggregate those representations into a consensus. Its framework is organized around a BLSTM-CRF base model, decoupling and aggregation phases, inference, and applications to crowd and cross-domain supervision.

4.1 The Base Model: BLSTM-CRF

  • A bidirectional LSTM encodes each input sequence, and a CRF uses emission and transition scores to score and distribute probability over tag sequences.The emission matrix scores tags at each word, while the transition matrix scores consecutive tags.Source: §4.1 · pp. 3–4

4.2 The Decoupling Phase: Learning annotator representations

  • Each source gets a matrix that transforms shared emission and transition scores, modeling its annotation biases while shared BLSTM-CRF parameters are trained across sources.The paper treats each annotator as a noisy version of the shared model, with matrix entries modeling possible tag-labeling errors.Source: §4.2 · p. 4

4.3 The Aggregation Phase: Dynamically Reaching Consensus

  • Sentence embeddings drive attention weights that aggregate source-specific matrices into a consensus representation, with attention supervised by target predictions combined using source training-set F1.The resulting representation gives more weight to sources related to the current sentence and requires training only the attention weight matrix in this phase.Source: §4.3 · pp. 4–5

4.4 Parameter Learning and Inference

  • At inference, CONNET computes sentence embeddings, assigns source weights, aggregates source representations, and applies only the resulting consensus matrix to infer labels.The paper states that specialist knowledge helps deal with more complex instances.Source: §4.4 · p. 5

4.5 Model Application

  • CONNET treats annotators as sources for crowd learning and domains as sources for unsupervised adaptation, and is also described as extendable to other tasks and encoders.The method is also incorporated as a feature for controlling crowd-annotation quality in AlpacaTag and LEAN-LIFE.Source: §4.5 · p. 5

5 Experiments

Experiments evaluate CONNET on crowd-annotated NER and unsupervised cross-domain tasks, with additional tests using transformer and MLP encoders. The reported results show strong performance on real and simulated crowd data and across cross-domain tasks.

5.1 Datasets

  • Crowd experiments use real-world AMT annotations and AMTC splits, simulated annotators, and cross-domain POS, NER, and sentiment-classification datasets.AMT annotator F1 scores against ground truth range from 17.60% to 89.11%.Source: §5.1 · pp. 5–6

5.4 Learning with Crowd Annotations; 5.5 Cross-Domain Adaptation Performance

  • CONNET significantly outperforms other methods on AMT and AMTC F1 and has the highest average score across POS, NER, and classification tasks.Table 1 covers real-world crowd-sourced NER; Table 2 covers cross-domain data.Source: Table 1 · p. 6, §5.4 · p. 7, Table 2 · p. 8

5.2 Experiment Setup

  • Sequence-labeling tasks use BLSTM-CRF, classification uses an MLP, and results average three runs with different random initializations.The setup uses early stopping based on development performance where available.Source: §5.2 · p. 6

5.3 Compared Methods

  • Comparisons include wrapper methods that aggregate labels and joint models that incorporate sources, alongside Gold models trained with target-domain ground truth.The wrapper baselines include token- and sequence-level voting and probabilistic label aggregators; joint baselines model annotators within taggers.Source: §5.3 · pp. 6–7

5.4 Learning with Crowd Annotations

  • Both partial decoupling variants improve AMT results, and AWV is effective supervision; test-time AWV and BLSTM-CRF on AMV labels score 77.35(±0.08) and 78.93(±0.13), respectively.Figure 4 compares decoupling-phase and aggregation-phase variants.Source: Figure 4 · p. 7, §5.4 · pp. 7–8
  • On simulated crowds, CONNET improves over MVT-SLM and is competitive with Crowd-Cat, especially at low reliability, and outperforms baselines as annotator count varies.The count comparison uses annotators with reliability r = 1/50.Source: Figure 5 · p. 7, §5.4 · pp. 8–9

5.6 Analyzing Learned Attention

  • Average source-to-target attention reflects domain relatedness: bn and nw contribute reciprocally, bn contributes to bc and mz, while web and tc contribute little.Figure 6 presents averaged attention scores for each source-target domain pair.Source: Figure 6 · p. 9, §5.6 · p. 9

5.5 Cross-Domain Adaptation Performance

  • Context-aware feature-level voting outperforms context-independent MTL-BEA, while Tri-Training on concatenated source data also performs worse than CONNET.Crowd-Add is unstable and performs poorly on NER; the paper attributes this partly to its uncontrollable crowd-vector size.Source: §5.5 · p. 9

6 Conclusion

CONNET learns a sequence tagger from multi-source supervision for crowd annotation and cross-domain adaptation. Experiments report superiority over prior models in these settings and promising results on text classification.

6 Conclusion

  • CONNET applies to crowd annotations and cross-domain adaptation, and the framework also shows promising results on text classification.Source: §6 · p. 9
  • The model learns fine-grained representations of each source and dynamically aggregates them for each unseen target sentence.Source: §6 · p. 9
  • Experiments show CONNET is superior to previous crowdsourcing and unsupervised domain-adaptation sequence-labeling models.Source: §6 · p. 9

A.1 Case study on learning with crowd annotations

A case study on AMTC examines whether CONNET’s attention selects annotators suited to the entity types in each sentence. The authors report that attention favors annotators with competitive performance and relevant preferences.

A.1 Case study on learning with crowd annotations

  • The AMTC case study uses a real-world dataset with 47 annotators to examine how attention selects annotators.Source: §A.1 · p. 12
  • Sentences 1–12 contain one specific entity type each, while sentences 13–16 contain multiple entity types.Source: §A.1 · p. 12
  • Attention gives more weight to annotators with competitive performance and preferences for the entity type included in a sentence.Source: §A.1 · p. 12
  • For ORG, the top three selected annotators had lower ORG expertise than their PER and LOC expertise, yet were the three annotators with the highest ORG expertise.Source: §A.1 · p. 12

B Result of ConNet with Transformer Encoder

The authors evaluate CONNET with Transformer-CRF on crowd-annotation NER and cross-domain POS. They report that it outperforms the other baselines in both tasks and applications.

B Result of ConNet with Transformer Encoder

Loading 1910.04289v2…