PaperlayerView exact revision on arXiv

Source-linked AI summary

Snorkel: Rapid Training Data Creation with Weak Supervision

Alexander Ratner, Stephen H. Bach, Henry Ehrenberg, Jason Fries, Sen Wu, Christopher Ré

arXiv:1711.10160v1cs.LGstat.ML

TL;DR

Creating labeled data is costly, especially for deep-learning systems that need large expert-curated datasets. Snorkel lets users write labeling functions, models their noisy and correlated outputs without ground truth, and uses probabilistic labels to train discriminative models. Across user studies, deployments, and open-source datasets, it reports faster development and performance that exceeds heuristic methods while approaching large hand-curated training sets.

  • Problem

    Deep-learning systems require large labeled datasets, but creating expert-curated training data is expensive and can require relabeling as project goals change.

  • Method

    Snorkel uses user-written labeling functions, a generative model to combine their outputs without ground truth, and probabilistic labels to train discriminative models.

  • Results

    Snorkel reduces training cost and difficulty, exceeds prior weak-supervision methods, and approaches the quality of large hand-labeled training sets across reported evaluations.

  • Takeaways & Limitations

    Weak supervision provides an improved way for developers in industry, research labs, and government agencies to build machine-learning models.

  • Takeaways & Limitations

    Training assumes access to unlabeled data plus a small labeled development set and a blind held-out evaluation set.

Abstract

from arXiv · show

Labeling training data is increasingly the largest bottleneck in deploying machine learning systems. We present Snorkel, a first-of-its-kind system that enables users to train state-of-the-art models without hand labeling any training data. Instead, users write labeling functions that express arbitrary heuristics, which can have unknown accuracies and correlations. Snorkel denoises their outputs without access to ground truth by incorporating the first end-to-end implementation of our recently proposed machine learning paradigm, data programming. We present a flexible interface layer for writing labeling functions based on our experience over the past year collaborating with companies, agencies, and research labs. In a user study, subject matter experts build models 2.8x faster and increase predictive performance an average 45.5% versus seven hours of hand labeling. We study the modeling tradeoffs in this new setting and propose an optimizer for automating tradeoff decisions that gives up to 1.8x speedup per pipeline execution. In two collaborations, with the U.S. Department of Veterans Affairs and the U.S. Food and Drug Administration, and on four open-source text and image data sets representative of other deployments, Snorkel provides 132% average improvements to predictive performance over prior heuristic approaches and comes within an average 3.60% of the predictive performance of large hand-curated training sets.

1. INTRODUCTION

Snorkel addresses the cost and difficulty of creating labeled training data by letting subject matter experts program weak supervision sources and automatically combining their outputs. It introduces an end-to-end data-programming system with flexible labeling-function interfaces, modeling tradeoffs, and strong results across deployments and open-source datasets.

  • Motivation: Tens of thousands to millions of labeled examples are often needed for peak deep-learning performance, while expert-created training sets are expensive and may require relabeling as goals change.Practitioners therefore increasingly use cheaper but noisier weak-supervision sources.
  • Motivation: Weak-supervision sources can overlap and conflict, requiring accuracy and correlation estimation without ground truth and preservation of label-quality lineage for the end model.Naive source selection or majority voting can produce poor or tied labels, whereas source-aware weighting can improve training-label accuracy.
  • Contributions: Snorkel provides a flexible labeling-function interface for heterogeneous supervision strategies and supports weak supervision as a REPL-like interaction model for machine learning.The interface accommodates sources operating over spans, documents, images, and multiple contextual resources.
  • Contributions: 2.8× faster model building and 45.5% higher predictive performance were achieved by subject matter experts versus seven hours of hand labeling in a user study.The reported result validates the productivity and predictive-performance effects of Snorkel’s interface.
  • Contributions: Up to 1.8× pipeline speedups were obtained by an optimizer that selects when to model source accuracies and correlations versus using majority vote.The optimizer predicts the modeling advantage within 2.16 accuracy points on average and can save up to 61% of training time while retaining 60%–70% of correlation-learning benefit.
  • Contributions: Snorkel is presented as the first end-to-end implementation of data programming, improving over heuristic baselines across deployments and open-source datasets while approaching large hand-curated training sets.Reported improvements average 110% in two deployments and 153% on four representative open-source datasets, with performance within 3.60% of large hand-curated sets in the latter evaluation.

2. SNORKEL ARCHITECTURE

Snorkel replaces hand-labeling with a three-stage weak-supervision workflow: users write labeling functions, Snorkel models their outputs, and a discriminative model trains on probabilistic labels. Its architecture supports structured inputs and flexible labeling-function interfaces, while enabling generalization beyond heuristic coverage.

  • Architecture: Snorkel’s workflow has three stages: writing labeling functions, modeling their accuracies and correlations, and training a discriminative model.The workflow is designed around data programming and uses weak supervision throughout.
  • Architecture: The pipeline assumes unlabeled training data plus smaller labeled development and held-out test sets for development and evaluation.These labeled sets can be orders of magnitude smaller than a training set.
  • Architecture: Labeling functions map data points to labels or abstentions, producing a matrix that Snorkel synthesizes into probabilistic training labels.The resulting labels are used to train a discriminative model.
  • Data Model: Snorkel represents complex inputs as hierarchies of contexts, such as documents, sentences, and spans, and passes Candidate objects to labeling functions.In the chemical-disease example, a Candidate is a tuple of two spans.
  • Labeling Functions: Users can implement labeling functions as arbitrary code or use declarative operators for patterns, heuristics, external knowledge bases, and other weak-supervision sources.The interface was shaped by interactions with users across bioinformatics, defense, industry, and other areas.
  • Discriminative Model: As unlabeled data increases, Snorkel-trained discriminative models can reduce generalization error at the same asymptotic rate as supervised models using additional hand-labeled data.A neural network trained on Snorkel’s probabilistic labels also correctly identifies an example on which all 33 labeling functions abstain.

3. WEAK SUPERVISION TRADEOFFS

Snorkel’s generative model offers its greatest predictive gains in medium-density labeling settings, while majority vote can suffice at low or high density. The paper develops heuristics and an optimizer to select modeling complexity and trade predictive performance against computational cost.

  • Label-density regimes: At low label density, an optimal generative model rarely differs from majority vote because conflicting non-abstaining labels are infrequent.This limits the available modeling advantage when most points have at most one non-abstaining label.
  • Label-density regimes: At high label density, majority vote converges exponentially toward an optimal solution when the average labeling-function accuracy exceeds 50%.The result provides an upper bound on the expected advantage of optimal weighting as average label density increases.
  • Label-density regimes: Medium label density is the regime where modeling labeling-function accuracies is expected to deliver the greatest predictive gains.Many data points have a small number of disagreeing labels, so estimated accuracies can strongly affect predictions.
  • Label-density regimes: Equivalent overall label densities can yield different modeling benefits, as shown by EHR and Chem, where accuracy modeling helps EHR more than Chem.This motivates using per-example positive-to-negative label ratios rather than average label density alone.
  • Automatically choosing a modeling strategy: The optimizer uses labeling-function outputs to choose between majority vote and a generative model, correctly selecting the strategy across the reported cases.Its predicted advantage serves as a guide for deciding when generative modeling can be skipped.
  • Modeling structure: As the correlation threshold decreases, predictive performance generally improves initially, while computational cost rises linearly with the number of modeled correlations.An elbow point is selected as a practical tradeoff between performance and runtime.

4. EVALUATION

Snorkel is evaluated across real-world deployments and open-source tasks, where it improves over weak-supervision baselines and approaches hand-labeled performance. Additional experiments show benefits from probabilistic labels, heterogeneous labeling functions, and user-written supervision.

  • 132% average predictive-performance improvement over distant-supervision baselines is reported across the evaluation.
  • 3.60% average gap to hand-supervision performance is reported across all tasks, including 2.11% on relation extraction and 5.08% on cross-modal tasks.
  • 43.15% average recall improvement shows that discriminative models generalize beyond heuristics encoded in labeling functions.
  • 5.81% average improvement results when discriminative models train on Snorkel’s probabilistic labels rather than unweighted labeling-function outputs.
  • Labeling function ablation: Adding structure-based labeling functions contributed an additional F1 point in the ablation study.
  • User study: 8 subjects matched or outperformed hand-labeled models, with average F1 of 30.4 versus 20.9 for hand supervision.

5. RELATED WORK

Related work addresses weak-supervision source combination, source-accuracy estimation, alternative supervision paradigms, and related data-management problems. Snorkel is positioned as a system for managing weak supervision directly from subject-matter-expert input.

  • Combining Weak Supervision Sources: Weak-supervision research studies how to combine multiple noisy sources and estimate source accuracy without a gold standard.
  • Other Forms of Supervision: Semi-supervised, active, transfer, self-training, and co-training methods use different assumptions or training arrangements than Snorkel’s weak-supervision setting.
  • Other Forms of Supervision: Snorkel focuses on soliciting higher-level, noisier input directly from subject-matter experts while managing weak-supervision sources.
  • Related Data Management Problems: Data fusion and truth discovery estimate source reliability and likely truth using probabilistic models with latent variables.

6. CONCLUSION

Snorkel presents a paradigm in which users provide labeling functions to create training data without manually resolving noise and conflicts. The reported evaluations indicate improved performance and reduced training-data creation difficulty across real deployments.

  • Snorkel uses labeling functions to capture domain knowledge and resources as higher-level supervision for creating training datasets.
  • The system manages noise and conflicts inherent in combining weak-supervision sources without requiring users to handle them directly.
  • Evaluations report that Snorkel exceeds prior weak-supervision methods and approaches the quality of large hand-labeled training sets.
  • Deployments in industry, research labs, and government agencies are reported as evidence of real-world impact.

A.2 Proof of Proposition 1

The proof analyzes when perfectly weighted majority voting can outperform simple majority voting under a low-label-density model. Under non-adversarial labeling functions, the expected advantage is bounded quadratically in label density.

  • The analysis assumes a fixed probability of emitting a non-zero label and non-adversarial labeling functions with accuracy above 50%.
  • The proof upper-bounds modeling advantage by counting cases where perfectly weighted majority vote is correct while majority vote is incorrect.
  • Perfectly weighted and simple majority votes can disagree only when at least one pair of labels disagrees.
  • Under the independent model, the expected advantage scales at most quadratically in label density.

A.3 Explanation of Theorem 1

The theorem analyzes a symmetric Dawid–Skene-style setting with constant labeling propensity and mean worker accuracy above one-half. It then uses a previously established corollary to bound the mean error rate and expected modeling advantage.

  • Modeling assumptions: The analysis treats each crowd worker as a labeling function with class-dependent accuracy that is independent across workers.This is the binary-classification Dawid–Skene setting used for the theorem.
  • Modeling assumptions: The symmetric setting assumes every worker has the same accuracy parameter, αj ≡ α+.
  • Modeling assumptions: A constant probability sampling strategy assumes every labeling function labels every instance with the same probability, P(Λi,j ≠ ∅) = pl.The sampling probability matrix therefore has identical entries.
  • Bound: When the mean labeling-function accuracy satisfies ᾱ > 1/2, Corollary 9 supplies an upper bound on the mean error rate.The passage also gives the equivalent condition w̄* > 0.
  • Bound: The resulting error bound is substituted into an upper bound on the expected modeling advantage EΛ,y,w∗[A∗], using d̄ = npl.

A.4 Proof of Proposition 2

The proposition develops a tractable conditional upper bound for deciding whether to learn the generative model or use majority vote. The accompanying discussion connects this decision to empirical modeling advantage as labeling functions are added.

  • Proposition: The proposition bounds modeling advantage conditional on the observed label matrix Λ, assuming labeling-function weights lie within a fixed range.When the bound is small, the generative model can be skipped in favor of unweighted majority vote.
  • Proof: The proof upper-bounds expected advantage by the expected cases where weighted majority vote is correct while majority vote is incorrect.
  • Empirical comparison: Figure 6 compares generative-model and majority-vote advantage on random subsets of CDR labeling functions with different sizes.
  • Empirical comparison: The optimizer selects majority vote during early development and the generative model later, as the number of labeling functions increases.
  • Proof: The probabilistic formulation expresses P(yi = y′ | Λi, w) using a sigmoid, while omitting labeling-propensity factors because they do not depend on y.
  • Proof: The proof removes the y′ placeholder and simplifies notation to match the paper’s main-body formulation.

A.5 Modeling Advantage Notes

The modeling-advantage analysis finds that the benefit of the generative model grows with the number of labeling functions, while the reported F1 treatment reflects known negative class imbalance.

  • Optimizer behavior: The modeling advantage grows as more labeling functions are added, allowing the optimizer to save execution time during early iterative development.
  • Metric caveat: F1 counts generative-model no-label outputs as negatives because of known negative class imbalance in relation extraction.This convention gives the generative model the benefit of the doubt and can hide some -1 versus 0 differences.
  • Metric caveat: Empirical modeling advantage nevertheless counts cases where the generative model emits -1 while majority vote emits 0, because they affect end-model training.

B.1 Data Set Details

The dataset details report candidate counts for unlabeled training and hand-labeled development and test splits, alongside participant-level performance analyses. Development and test sets may be much smaller than training sets.

  • Dataset splits: Dataset sizes are reported as numbers of candidates for unlabeled training, hand-labeled development, and test splits.
  • Dataset splits: Development and test sets can be orders of magnitude smaller than the corresponding training sets.
  • User study analysis: Figures 7 and 8 compare participant score distributions and participant-background groups against hand-labeled-data baseline models.
  • User study analysis: Figure 8 also reports descriptive statistics for user factors grouped by the predictive performance of participants’ end models.

C. IMPLEMENTATION DETAILS

Snorkel provides an accessible notebook-based interface for writing flexible labeling functions, inspecting model errors, and scaling execution across workers or clusters. Its implementation also supports preprocessing and candidate extraction through common interfaces and external NLP tools.

  • User Study: In the user study, most participants matched or exceeded a model trained on 7 hours, or 2,500 instances, of hand-labeled data.The best-performing user had an MS or Ph.D. in any field, strong Python skills, and intermediate-to-advanced machine-learning experience; text-mining experience added no benefit.
  • Interface Implementation: Users write arbitrary Python labeling functions in Jupyter, with additional primitives and generators for declarative weak supervision.The interface targets subject matter experts without advanced programming skills and supports pattern-matching, distant-supervision, and weak-classifier heuristics.
  • Interface Implementation: The Viewer organizes development-set candidates into true-positive, false-positive, true-negative, and false-negative buckets for iterative error analysis.These buckets help subject matter experts identify recurring patterns in model errors.
  • Execution Model: Labeling functions execute independently on discrete candidates, enabling embarrassingly parallel processing through database-backed Python workers.A master notebook process distributes candidate primary keys, while workers materialize candidates through the ORM layer.
  • Execution Model: Snorkel integrates with Spark so labeling functions can run across a cluster while communicating only the function closure and resulting labels after candidate data is cached.This reduces repeated distribution costs during the iterative workflow.
  • Execution Model: The same execution model supports preprocessing and candidate extraction, with wrappers for Stanford CoreNLP and SpaCy and named-entity-recognition-based candidate definition.These utilities use a common class interface.
Loading 1711.10160v1…