PaperlayerView exact revision on arXiv

Source-linked AI summary

Using Similarity Measures to Select Pretraining Data for NER

Xiang Dai, Sarvnaz Karimi, Ben Hachey, Cecile Paris

arXiv:1904.00585v2cs.CL

TL;DR

The paper asks how to select useful pretraining data when source–target similarity is otherwise chosen by intuition. It proposes three cost-effective similarity measures and evaluates them across 30 NER data pairs, finding distinct benefits for pretrained language models and word vectors depending on source similarity.

  • Problem

    There is limited systematic evidence for selecting appropriate unlabeled data to pretrain word vectors or language models for a target NER task.

    Source: §1 · p. 1

  • Method

    The study measures target vocabulary coverage, language-model perplexity, and word-vector variance to quantify source–target similarity and evaluates their relation to pretrained-model effectiveness.

    Source: §1 · p. 1, §4.1 · p. 4, §4.3 · p. 4

  • Results

    Across the study, similarity measures predict pretrained-model usefulness; language models benefit more from similar sources, while word vectors depend strongly on vocabulary intersection and can be better with dissimilar data.

    Source: §1 · p. 2, §7.1 · p. 6, §8 · p. 9

  • Takeaways & Limitations

    Similarity should guide pretraining-data selection, with tenor considered alongside field rather than relying on field-based intuition alone.

    Source: §1 · p. 2, §8 · p. 9

    The study leaves the combined effect of source-data size and similarity for future work and focuses on transfer from one source without multilingual similarity.

    Source: §4.2 · p. 4, §7.3 · p. 9

Abstract

from arXiv · show

Word vectors and Language Models (LMs) pretrained on a large amount of unlabelled data can dramatically improve various Natural Language Processing (NLP) tasks. However, the measure and impact of similarity between pretraining data and target task data are left to intuition. We propose three cost-effective measures to quantify different aspects of similarity between source pretraining and target task data. We demonstrate that these measures are good predictors of the usefulness of pretrained models for Named Entity Recognition (NER) over 30 data pairs. Results also suggest that pretrained LMs are more effective and more predictable than pretrained word vectors, but pretrained word vectors are better when pretraining data is dissimilar.

1 Introduction

The paper addresses the lack of systematic guidance for selecting pretraining data by proposing cost-effective similarity measures and testing their predictive value for NER.

  • Practitioners’ heuristic choices include generic data, similar fields, or similar tenors, with decisions varying by individual experience.Source: §1 · p. 1
  • Experiments compare pretraining choices across 30 source–target data pairs spanning different fields and tenors.Source: §1 · p. 1
  • The study proposes quantitatively measuring source–target similarity to predict how pretraining data affects final NER accuracy.The measures target vocabulary coverage, language-model perplexity, and changes in word vectors after target-data training.Source: §1 · pp. 1–2
  • The paper systematically investigates language models pretrained on varied data sources, which it identifies as a first study of its kind.Source: §1 · p. 2
  • Models pretrained on a modest amount of similar data outperform models trained for weeks on very large generic data.Source: §1 · p. 2

2 Related Work

Prior work shows that pretrained representations can improve downstream NLP, while this paper focuses on predicting which single external source to select for a specific NER target.

Text Similarity

  • The study extends similarity analysis to the corpus level and examines its implications for unsupervised pretraining.Source: §2 · p. 2

Pretrained Word Vectors

  • Earlier studies found that pretrained word vectors improve downstream tasks, but source data, training algorithms, and hyperparameters all affect their effectiveness.Source: §2 · p. 2

Pretrained Language Models

  • Pretrained language models have improved generalization and task accuracy, including when trained on smaller, non-generic corpora.Source: §2 · p. 2

Transfer Learning

  • Domain adaptation asks how to transfer between a given source and target, whereas this work asks which single source to choose for a specific target.Source: §2 · p. 2

Named Entity Recognition

3 What Human Intuition Indicates

A practitioner survey shows that people favor biologically related research papers over tenor-matched online reviews as pretraining data for medication-forum NER.

  • The survey compared medication forum posts as target data with biology-and-health papers and broad online reviews as source alternatives.Source: §3 · p. 3
  • Participants rated unsupervised pretraining on S1 useful for supervised named entity recognition learning on T at 73%, versus 27% for S2; scores were significantly higher for S1.Ratings used a five-point Likert scale; Wilcoxon signed-rank test: Z = 43.0, p < 0.001.Source: §3 · p. 3
  • Participant ratings were significantly higher for S1 than S2: 73% agreed or strongly agreed S1 would be useful, compared with 27% for S2 (Z = 43.0, p < 0.001).Source: §3 · p. 3
  • Although small and variable across practitioners, the results suggest that intuition prioritizes field over tenor, motivating quantitative similarity measures.Source: §3 · p. 3

4 Similarity Measures

The paper quantifies source–target similarity through vocabulary overlap, language-model perplexity, and word-vector changes, mapping these measures to field, tenor, or both.

4 Similarity Measures

  • The measures operationalize intuitions that field similarity affects vocabulary, while tenor similarity affects writing style and linguistic regularities.Tenor concerns discourse participants, relationships, purposes, and associated stylistic features such as punctuation, normalization, politeness, and emotiveness.Source: §4 · p. 3

4.1 Target Vocabulary Covered

  • Target Vocabulary Covered measures field similarity as the percentage of target vocabulary also present in the source.A content-word variant, VCcR, restricts both vocabularies to nouns, verbs, and adjectives.Source: §4.1 · pp. 3–4

4.2 Language Model Perplexity

  • Language-model perplexity trains on source data and evaluates target sentences, with lower perplexity indicating greater similarity in vocabulary and surface structure.The study uses Kneser-Ney-smoothed 5-gram models and treats PPL as a proxy for tenor as well as field.Source: §4.2 · p. 4

4.3 Word Vector Variance

  • Word Vector Variance measures combined field and tenor similarity by comparing source-trained vectors after they are further trained on target data.Smaller variance indicates that similar context surrounds the same words in both datasets.Source: §4.3 · p. 4

5 Data Sets

The study uses five varied source datasets and six NER target datasets, controlling source size while examining fields, tenors, and NER’s sensitivity to word representations.

5 Data Sets

  • Five source datasets span clinical, biomedical, local-business, and diverse Wiki fields and popular reporting, notes, scholarly, review, and encyclopedia tenors.Sources are sampled to approximately 100 million tokens to isolate source-size effects, which are analyzed separately.Source: §5 · pp. 4–5
  • Six target NER datasets are CADEC, CoNLL2003, CRAFT, JNLPBA, ScienceIE, and WetLab.They were selected to study how source–target similarity affects pretrained word vectors and language models on NER.Source: §5 · p. 5
  • NER is used because token-level labeling is sensitive to word representations, and removing pretrained word vectors has previously caused large performance drops.The task is also described as a popular structured NLP benchmark.Source: §5 · p. 5

6 Experimental Setup

The experiments pretrain word vectors and language models on different source datasets, then evaluate their effects on NER using a common supervised sequence tagger.

6 Experimental Setup

  • The study separately pretrains word vectors and language models on different sources, then compares their effectiveness across target NER datasets.This directly tests how source-data choice affects pretrained representations.Source: §6 · p. 5
  • NER uses a two-layer BiLSTM-CRF with CNN character representations and micro-average F1 evaluation.The supervised architecture follows Lample et al. (2016), with character modeling based on Ma and Hovy (2016).Source: §6 · p. 5
  • Word vectors use default word2vec settings, replace the embedding initialization, and are jointly trained with the remaining model parameters.The None baseline randomly initializes word embeddings.Source: §6 · p. 5
  • Character-based pretrained language models provide contextualized word representations from all layers to the second BiLSTM layer.Target words are converted into character sequences before being fed to the language models.Source: §6 · p. 5

7 Experimental Results

Across five source–target similarity measures and NER experiments, similarity generally identifies useful pretraining data, with tenor often complementing or outweighing field. Small similar sources can match or exceed much larger generic corpora, while source-size effects depend on similarity.

7.1 Similarity Between Source and Target Data Sets

  • Tenor was reflected more strongly than field by PPL and WVV, making Yelp more similar to patient-written CADEC than medically focused PubMed or MIMIC.ScienceIE also matched PubMed through shared scholarly tenor despite different fields.Source: §7.1 · p. 6
  • Similarity measures showed high agreement across source–target comparisons, with Fleiss’s kappa of 0.733 for PPL, WVV, and TVC.The study compared 60 binary source-similarity decisions across five sources and six targets.Source: §7.1 · pp. 6–7

7.2 Impact of Pretraining Data

  • Similar source data generally produced better pretrained word vectors and LMs, although no single source improved every target and word vectors had a JNLPBA exception.The measures were predictive of pretraining effectiveness; vocabulary coverage was most informative for word vectors, while LM performance correlated more strongly with similarity measures.Source: §7.2 · pp. 7–8
  • LMs provided more benefit than word vectors for similar sources, whereas word vectors outperformed LMs for dissimilar sources such as MIMIC–CoNLL2003 and PubMed–CRAFT.This pattern was observed across the cited similar and dissimilar source–target pairs.Source: §7.2 · p. 8
  • Small similar sources achieved competitive or better performance than publicly available models trained on much larger generic corpora.The results indicate reduced computational cost without sacrificing performance; a 1-billion-token LM can take three weeks to train on 32 GPUs.Source: §7.2 · p. 8, Table 5 · p. 8

Comparison to other hyper-parameter settings

7.3 Controlling for source data size

  • With PubMed source subsets, 500 MB appeared sufficient for similarity calculation, and larger size raised absolute F1 by less than 0.5 after sufficient data.For dissimilar PubMed–CoNLL2003, word vectors remained better than LMs regardless of source size; for similar PubMed–JNLPBA, LMs remained better.Source: §7.3 · p. 9

8 Conclusions

The paper develops cost-effective similarity measures for choosing pretraining data and finds that their usefulness differs between pretrained word vectors and language models.

  • Figure 2 examines how source-data size affects the effectiveness of pretrained models for NER.Source: Figure 2 · p. 9
  • Three measures—Target Vocabulary Covered, Language Model Perplexity, and Word Vector Variance—quantify source–target similarity for selecting pretraining data.Source: §8 · p. 9
  • Pretrained word-vector effectiveness strongly depends on vocabulary overlap with the target data, whereas language models benefit more from similar sources.Source: §8 · p. 9
  • The study positions these measures as a step toward systematically guiding pretraining-data choices for NER and future NLP tasks.Source: §8 · p. 9

Related papers

Loading 1904.00585v2…