PaperlayerView exact revision on arXiv

Source-linked AI summary

WRENCH: A Comprehensive Benchmark for Weak Supervision

Jieyu Zhang, Yue Yu, Yinghao Li, Yujing Wang, Yaming Yang, Mao Yang, Alexander Ratner

arXiv:2109.11377v2cs.LGcs.AIcs.CLstat.ML

TL;DR

Weak supervision can reduce the need for manual training labels, but private or custom datasets, hidden variation in supervision sources, and inconsistent protocols complicate evaluation. WRENCH standardizes comparisons using 22 datasets, varied supervision sources, and a modular framework. Its analyses find that designing general-purpose weak supervision methods remains challenging, with tagging-scheme choices associated with up to a 10% change in F1 score on sequence-tagging tasks.

  • Problem

    Weak supervision evaluation is hindered by private or custom datasets, hidden variation in labels and sources, and inconsistent protocols.

    Source: §1 · pp. 1–2

  • Method

    WRENCH benchmarks weak supervision with 22 real-world datasets, varied real, synthetic, and procedural supervision sources, and a modular evaluation framework.

    Source: §1 · p. 2

  • Results

    The authors conclude that designing general-purpose weak supervision methods remains challenging; for sequence tagging, tagging-scheme choices can change F1 by up to 10%.

    Source: §8 · p. 10, §7 · p. 10

  • Takeaways & Limitations

    WRENCH provides a foundation for studying weak supervision and uses procedural labeling-function generators to examine supervision-source properties.

    Source: §8 · p. 10

    WRENCH currently excludes weak supervision methods that require additional clean labeled data.

    Source: §2 · p. 3

Abstract

from arXiv · show

Recent Weak Supervision (WS) approaches have had widespread success in easing the bottleneck of labeling training data for machine learning by synthesizing labels from multiple potentially noisy supervision sources. However, proper measurement and analysis of these approaches remain a challenge. First, datasets used in existing works are often private and/or custom, limiting standardization. Second, WS datasets with the same name and base data often vary in terms of the labels and weak supervision sources used, a significant "hidden" source of evaluation variance. Finally, WS studies often diverge in terms of the evaluation protocol and ablations used. To address these problems, we introduce a benchmark platform, WRENCH, for thorough and standardized evaluation of WS approaches. It consists of 22 varied real-world datasets for classification and sequence tagging; a range of real, synthetic, and procedurally-generated weak supervision sources; and a modular, extensible framework for WS evaluation, including implementations for popular WS methods. We use WRENCH to conduct extensive comparisons over more than 120 method variants to demonstrate its efficacy as a benchmark platform. The code is available at https://github.com/JieyuZ2/wrench.

1 Introduction

Weak supervision reduces reliance on costly manual labels, but comparison across approaches is hindered by private or custom datasets, hidden variation in supervision sources, and inconsistent evaluation protocols. WRENCH addresses these gaps with standardized datasets, supervision sources, and evaluation tools, and uses them to compare more than 120 method variants.

1 Introduction

  • Weak supervision uses programmatic sources such as heuristics, knowledge bases, and pretrained models instead of manually labeled training data.The paper represents these sources as labeling functions that produce potentially overlapping and conflicting votes.Source: §1 · p. 1
  • Private or custom datasets, hidden variation in labels and weak supervision sources, and inconsistent evaluation protocols make comparisons difficult to standardize.Datasets sharing a name and base data may still differ in their labels and supervision sources.Source: §1 · p. 2
  • WRENCH provides 22 diverse datasets, standardized supervision sources, and a modular framework for evaluating weak supervision methods.Its sources include real, synthetic, and procedurally generated weak supervision sources, and its framework includes implementations of popular methods.Source: §1 · pp. 2–3
  • WRENCH comparisons cover more than 120 method variants: 83 for classification and 46 for sequence tagging.The authors report extensive experiments and plan to update the benchmark with more datasets and methods.Source: §1 · p. 3

2 Related Work

Related work spans several forms of weak supervision and efforts to generate supervision sources automatically. WRENCH focuses on benchmarking methods that combine multiple sources without requiring labeled data.

Weak Supervision

  • Weak supervision draws on approaches including distant supervision, crowdsourcing, co-training, pattern-based supervision, and feature annotation.It has been applied to classification and sequence tagging to reduce annotation effort.Source: §2 · p. 3

Weak Supervision Sources Generation

  • Prior source-generation methods optimize or augment labeling functions, whereas WRENCH’s procedural generators study how source properties affect weak supervision methods.WRENCH assumes access to a fully labeled dataset to generate diverse supervision sources.Source: §2 · p. 3

The Scope of this Benchmark

  • WRENCH excludes methods that use additional clean labeled data and focuses on methods that do not require labeled data.The authors plan to consider those excluded methods in future work.Source: §2 · p. 3

3 Background: Weak Supervision

Weak supervision trains an end model from multiple potentially noisy sources, either through separate label-model and end-model stages or jointly. WRENCH formalizes this setting for classification and sequence tagging without observed ground-truth training labels.

3 Background: Weak Supervision

  • Weak supervision trains an end model from multiple weak sources that may be inaccurate, correlated, or noisy, using either a two-stage or joint approach.In the two-stage approach, source integration is separated from end-model training.Source: §3 · p. 3

3.1 Problem Setup

  • WRENCH represents supervision sources as labeling functions that label some data points or abstain, and studies classification and sequence tagging without ground-truth training labels.The benchmark can be extended to settings with ground-truth labels, which it does not currently observe during end-model training.Source: §3.1 · pp. 3–4

3.2 Two-stage Method

  • Two-stage methods combine labeling-function outputs with a label model to produce soft or hard labels for training an end model.Examples of label models include Majority Voting and probabilistic graphical models.Source: §3.2 · p. 4

3.3 One-stage Method

  • One-stage methods train the label model and end model simultaneously, typically using neural networks for aggregation and final prediction.The paper calls the model used by these methods a joint model.Source: §3.3 · p. 4

4 Wrench Benchmark Platform

WRENCH combines real-world datasets, procedural labeling-function generators, and a modular collection of baseline methods to support standardized weak supervision comparisons. Its design lets users study varied source properties and compose many two-stage method variants.

4 Wrench Benchmark Platform

  • WRENCH collects 22 publicly available real-world datasets and their user-provided labeling functions across varied topics and source forms.The datasets include generic and specialized domains, while labeling functions include keywords, regular expressions, knowledge bases, and human-provided rules.Source: §4 · pp. 4–5
  • Procedural labeling-function generators let users vary supervision sources on synthetic or real-world data and study how source properties relate to method performance.The unified interface is designed to make it easy to add generators for new types of labeling functions.Source: §4 · p. 5
  • WRENCH’s modular interfaces let users pair label models with end models and choose soft or hard labels, yielding more than 100 two-stage variants.The platform also includes publicly available and popular methods for systematic comparisons across the 22 datasets.Source: §4 · p. 5, Table 2 · p. 5

5 Labeling Function Generators

WRENCH provides synthetic and procedural labeling-function generators for studying how supervision-source properties affect weak-supervision methods. The examples show that label-model comparisons vary with LF accuracy variance, propensity, dependency structure, and dataset.

5 Labeling Function Generators

  • The generators support finer-grained study of how weak-supervision source qualities relate to method performance, using synthetic labels and procedurally generated real LFs.The examples in this section are constrained to binary classification.Source: §5 · p. 5

5.1 Synthetic Labeling Function Generator

  • WRENCH’s synthetic generator varies LF accuracy and propensity, including their variability, while generating outputs independently of input data.The initial generator follows a classic model with LF outputs conditionally independent given the unseen true label.Source: §5.1 · p. 5
  • Label-model performance diverges more with larger accuracy radius or lower LF propensity; increasing propensity makes performance rise and eventually converge.The synthetic study identifies accuracy variance and propensity as factors shaping comparative performance.Source: §5.1 · p. 6, Figure 3 · p. 6

5.2 Procedural Labeling Function Generator

  • Procedural generators build candidate LFs from features and labels, then retain those meeting user-set individual and group thresholds.The resulting LFs mimic user-provided functions while allowing fine-grained control over their attributes.Source: §5.2 · p. 6
  • The procedural study examines top-accuracy, high-correlation, and high-data-dependency LFs on Yelp and Youtube; the latter properties matter because conditional independence and uniform accuracy are often violated in practice.Correlation is measured by conditional mutual information given ground-truth labels; data-dependency by variance in LF accuracy across data clusters.Source: §5.2 · p. 6
  • On Yelp and Youtube, label models perform similarly with top-k accurate LFs, but correlated or data-dependent LFs produce large performance gaps favoring DP, MeTaL, and FS.For correlated LFs, DS performs much better on Yelp than Youtube relative to MV.Source: §5.2 · p. 6, Figure 4 · p. 7

5 Labeling Function Generators

  • The findings support benchmarking across varied datasets and LF types because LF properties materially affect comparative WS-method performance.The synthetic and procedural examples show differences across LF accuracy variance, propensity, correlation, and data-dependency.Source: §5.2 · pp. 6–7, §5.1 · p. 6

6 Benchmark Experiments

WRENCH evaluates classification and sequence-tagging methods using shared protocols across datasets, including gold-label and weak-supervision comparisons. Results vary by dataset: no classification method consistently wins, while HMM-based label models lead on most sequence-tagging datasets, with weak-label quality and coverage qualifying performance.

6.1.1 Evaluation Protocol

  • Classification comparisons include label models on test data, end models trained with weak labels, joint models, and gold-trained end models, totaling 83 methods.The evaluation includes two-stage variants using soft or hard labels.Source: §6.1.1 · p. 7, §6 · p. 7
  • Classification uses each dataset’s prior-work metric, with pretrained BERT features for LR and MLP on textual data; privacy-limited video datasets use extracted image-classifier features.The Spouse dataset has no gold methods because ground-truth training labels are unavailable.Source: §6.1.1 · p. 7

6.1.2 Evaluation Results

  • No weak-supervision method consistently outperforms others across classification datasets; fine-tuned language models are the best end-model option for textual data, and COSINE can improve their performance.Table 3 shows the best gold method and top three weak-supervision methods for each dataset; values are averaged over five runs.Source: §6.1.2 · pp. 7–8, Table 3 · p. 8

6.2.1 Evaluation Protocol

  • For sequence tagging, token-level label models are compared with whole-sequence HMM and CHMM models; end models use either a CRF layer or token-classification head.The reported end-model result uses the better of the two settings.Source: §6.2.1 · p. 8
  • Sequence-tagging evaluation uses entity-level F1 with BIO labels, converting token-level predictions to entity-level predictions when needed.For words split into BERT subwords, prediction uses the first token’s result.Source: §6.2.1 · p. 8

6.2.2 Evaluation Results

  • HMM-based label models achieve the best sequence-tagging performance on 7 of 8 datasets, while majority voting is superior on MIT-Restaurants, where weak-label coverage is very small.Table 4 reports F1 scores averaged over five runs.Source: §6.2.2 · p. 8, Table 4 · p. 9
  • Training neural end models directly on weak labels does not guarantee gains, especially for LSTMs trained from scratch when LF quality is poor.Noisy, incomplete weak labels can lead end models to overfit, leaving a significant gap from gold-label training.Source: §6.2.2 · pp. 8–9

7 Discussion and Recommendation

The discussion recommends careful method categorization and baseline selection, and identifies supervision quality, label handling, and model choices as factors shaping weak-supervision performance.

7 Discussion and Recommendation

  • Fair comparisons require matching methods to appropriate baselines; COSINE, an end model, was compared with a label model without coupling MeTaL to an end model.The authors describe that comparison as hardly fair and effective.Source: §7 · p. 9
  • Noisy or low-coverage labeling functions leave a large gap from fully supervised performance, so their quality should be checked before applying weak-supervision models.The paper cites Basketball as noisy and MIT-Restaurants as having limited coverage; direct adoption may fail to satisfy performance needs or even hurt performance.Source: §7 · p. 9
  • Soft labels outperform hard labels in most cases, especially with deeper end models, which the authors relate to label smoothing.The comparison is based on average performance across tasks; the authors suggest label smoothing can prevent deep models from overfitting to noisy training data.Source: §7 · p. 9
  • COSINE's superiority suggests training end models on uncovered data as well as covered data, motivating exploration of semi-supervised training strategies.Covered data are examples receiving at least one weak signal; the paper presents use of uncovered data as a future direction.Source: §7 · p. 9
  • Up to 10% F1-score differences arise from tagging-schema choice; IO is recommended for datasets with few entity types, while other datasets show no clear winner.With more complex schemes such as BIO, the label model may predict incorrect label sequences, particularly when entity types are few.Source: §7 · p. 10
  • For classification, MeTaL and MV are the strongest label-model choices on average and deeper end models perform better; for sequence tagging, CHMM leads label models and pretrained language models suit end models.CHMM is the only compared label model that combines labeling-function outputs with data features such as BERT embeddings.Source: §7 · p. 10

8 Conclusion and Future Work

WRENCH provides a benchmark for weak supervision, while its conclusion frames general-purpose methods as challenging and the benchmark as a foundation for further study. The authors outline planned extensions and document its data, intended users, maintenance, and licensing.

8 Conclusion and Future Work

  • WRENCH plans to add datasets with varying supervision-source dependencies to support benchmarking of dependency-structure learning.The authors note that dependency structure is frequently ignored and that misspecification can cause serious performance drops.Source: §8 · p. 10
  • Planned extensions include active generation and repurposing of supervision sources to reduce annotation effort.The authors also plan to incorporate these tasks and methods into WRENCH.Source: §8 · p. 10
  • WRENCH currently focuses on classification and sequence tagging, with plans to add other weak-supervision applications.Source: §8 · p. 10
  • Predicting model performance from dataset and supervision-source properties remains an open challenge; the authors expect generators and new source measurements to contribute.The measurements named are correlation and data-dependency.Source: §8 · p. 10

A.1 Dataset Documentations

  • Dataset documentation uses separate JSON files for train, validation, and test splits, with each data point recording its id, label, weak labels, and data.Source: §A.1 · p. 17

A.2 Intended Uses

  • WRENCH is intended for machine-learning researchers developing weak-supervision methods and data scientists applying algorithms that require manual annotations.Source: §A.2 · p. 17

A.3 Hosting and Maintenance Plan

  • The codebase and datasets are hosted through GitHub, and the authors commit to maintaining and developing the open-source project for at least five years.They plan to grow it with new tasks and datasets and welcome external contributors.Source: §A.3 · p. 17

A.4 Licensing; A.5 Author Statement

  • The work is licensed under Apache 2.0, all datasets were publicly released by prior work, and the authors accept responsibility for rights violations.Source: §A.4 · p. 17, §A.5 · p. 17

A.6 Limitations

The limitations section notes that important weak-supervision tasks and datasets remain outside WRENCH. It also describes potential misuse and the scope of the accompanying dataset statistics.

A.6 Limitations

  • Important weak-supervision tasks and datasets remain to be included, so WRENCH is an ongoing effort rather than an exhaustive benchmark.Source: §A.6 · p. 17

A.7 Potential Negative Societal Impacts

  • Using model predictions directly for spam detection without rigorous prior validation may lead to negative outcomes; the benchmark involves no human subjects or personally identifiable information.Source: §A.7 · p. 17

B.1 Detailed Statistics and Visualization

B.2 Classification Datasets

The appendix documents classification and sequence-tagging datasets alongside their weak-supervision sources, spanning varied domains, task types, and source-generation approaches.

B.2 Classification Datasets

  • Classification examples include Census, with synthetic decision-list labeling functions, and Spouse, whose training gold labels are unavailable for measuring labeling-function accuracy.Census uses a held-out sample as proxy human knowledge; Spouse weak-supervision sources are generated by Snorkel.Source: §B.2 · pp. 18–19
  • Yelp and IMDb provide sentiment datasets, while YouTube and TREC cover spam detection and question classification with weak-supervision sources drawn from prior work.Source: §B.2 · p. 18
  • Relation-classification datasets include CDR, SemEval, and ChemProt, whose sources include tutorial labeling functions, 164 heuristic rules, and 26 keyword-based rules, respectively.Source: §B.2 · p. 19
  • Classification coverage also includes video-frame datasets with ResNet-101 features, SMS spam detection, and AGNews, using prior-work or manually generated weak-supervision sources.Basketball, Commercial, and Tennis Rally share labeling functions; SMS uses keyword and regular-expression rules, while AGNews uses keyword rules.Source: §B.2 · pp. 18–19

B.3 Sequence Tagging Datasets

  • Sequence-tagging datasets include CoNLL-03 and Wikigold, which share labeling functions, plus BC5CDR, NCBI-Disease, and Laptop-Review with sources selected or reused from prior work.Source: §B.3 · pp. 19–20
  • MIT-Restaurant and MIT-Movies use curated weak-supervision sources, while Ontonotes combines detectors, knowledge-base matching, keywords, and regular expressions across multiple domains.Ontonotes has 18 entity types and 17 weak-supervision sources; its validation set uses the first 5,000 sentences.Source: §B.3 · p. 20

C Compared Methods

WRENCH compares label, end, and joint models for classification and sequence tagging, and adapts classification label models to tagging with a label-correction technique. Treating unmatched tokens as O rather than correcting labels generally lowers performance.

C.1.1 Label Model

  • Classification label models include majority and weighted voting, Dawid-Skene, data programming, MeTaL, and FlyingSquid.Their methods include vote aggregation, expectation maximization, factor-graph modeling, matrix-completion-style parameter recovery, and a triplet method, respectively.Source: §C.1.1 · pp. 20–21

C.1.2 End Model

  • Classification end models include logistic regression, multilayer perceptrons, BERT, RoBERTa, and COSINE, which uses self-training and contrastive learning to improve pretrained language-model end models.BERT and RoBERTa are limited to textual datasets; COSINE variants are denoted BC and RC.Source: §C.1.2 · pp. 21–22

C.1.3 Joint Model

  • Denoise is a joint model that aggregates weak labels with an attention network and uses a neural classifier to leverage data features, training both components end to end.Source: §C.1.3 · p. 22

C.2.1 Label Models

  • For sequence tagging, HMM infers latent true labels from independently observed noisy labels, while CHMM predicts token-wise transition and emission probabilities from BERT embeddings.Source: §C.2.1 · p. 23

C.2.2 End Model

  • Sequence-tagging end models include LSTM-CNNs-CRF, which combines character and word features with adjacent-label constraints, and BERT with a linear prediction layer or CRF.Source: §C.2.2 · p. 23

C.2.3 Joint Model

  • ConNet trains parallel CRF layers for labeling sources, then aggregates their transitions using attention scores to produce a unified label sequence.Source: §C.2.3 · p. 23

D.1 Label Correction Technique

  • Label correction generally performs better for sequence tagging: treating unmatched tokens as O yields much smaller F1 gains when it wins than the gains from label correction when that wins.The exception is Dawid-Skene, which performs better without modification on 4 of 8 datasets.Source: §D.1 · p. 23

D.2 Comparision of IO and BIO Tagging Schema

The IO-versus-BIO comparison finds that the better tagging scheme depends on the dataset, so WRENCH uses IO in its experiments. The section also reports implementation and search settings and points to detailed classification and sequence-tagging comparisons.

D.2 Comparision of IO and BIO Tagging Schema

  • For datasets with few entity types, IO yields higher F1; elsewhere there is no clear winner across datasets, so WRENCH uses IO by default.IO excels on Ontonotes and MIT-Restaurants, while BIO performs better on other datasets.Source: §D.2 · pp. 23–24, Table 9 · p. 24

E.1 Hardware and Implementation

  • Experiments use Python and PyTorch, AdamW with a linear learning-rate scheduler, validation-based early stopping, and CPUs or 64 Nvidia V100 GPUs.Text classification features use BERT [CLS] embeddings; relation classification uses concatenated [CLS] and entity-token embeddings, with other features and pretrained models also supported.Source: §E.1 · p. 24

E.2 Hyper-parameter Search Space

  • Each model’s hyperparameters are selected by validation-set grid search, with three initialization runs per trial and five runs for final evaluation.Source: §E.2 · p. 24

E.3 Parameters for studies in Sec. 5

  • Procedural studies specify generated labeling-function settings for Figures 3 and 4, including accuracy, propensity, and n-gram feature parameters.Figures 3(a) and 3(b) each use 10 labeling functions; Figure 4 sets minimum propensity to 0.1 and uses 1- to 2-gram features.Source: §E.3 · pp. 24–25

F.1 Classification

  • Detailed classification comparisons are reported in Tables 11–12, including comparisons among pretrained-language-model methods on textual datasets.Table 11 averages each metric over five runs.Source: §F.1 · p. 25, Table 11 · p. 27, Table 12 · p. 28

F.2 Sequence Tagging

  • Detailed sequence-tagging results are reported in Table 13 as F1, precision, and recall with standard deviations, averaged over five runs.Source: §F.2 · p. 25, Table 13 · p. 29

Related papers

Loading 2109.11377v2…