Source-linked AI summary
Snorkel DryBell: A Case Study in Deploying Weak Supervision at Industrial Scale
Stephen H. Bach, Daniel Rodriguez, Yintao Liu, Chong Luo, Haidong Shao, Cassandra Xia, Souvik Sen, Alexander Ratner, Braden Hancock, Houman Alborzi, Rahul Kuchhal, Christopher Ré, Rob Malkin
TL;DR
Hand-labeling is a costly bottleneck for machine-learning applications. The paper introduces Snorkel DryBell, which uses diverse organizational knowledge as weak supervision and extends Snorkel for flexible ingestion, cross-feature serving, and scalable execution. Across three Google classification tasks, it produces classifiers comparable to those trained with tens of thousands of hand-labeled examples and transfers non-servable resources to servable models with an average 52% performance improvement.
Problem
Hand-labeled training data is costly and slow to produce, while changing applications quickly reduce the value of existing labeled sets.
Method
Snorkel DryBell ingests organizational resources as labeling functions, transfers supervision across feature sets, and uses scalable weak-supervision execution to train deployment models.
Results
Across three Google classification tasks, Snorkel DryBell trained classifiers that would otherwise require tens of thousands of hand-labeled examples and improved performance when transferring non-servable resources to servable models.
Takeaways & Limitations
Weak supervision can connect heterogeneous organizational resources to new machine-learning problems and support industrial development without relying exclusively on manual labeling.
Takeaways & Limitations
The generative model assumes conditional independence among labeling functions, although richer graphical structures are left for future work.
Abstract
from arXiv · showhide
Labeling training data is one of the most costly bottlenecks in developing machine learning-based applications. We present a first-of-its-kind study showing how existing knowledge resources from across an organization can be used as weak supervision in order to bring development time and cost down by an order of magnitude, and introduce Snorkel DryBell, a new weak supervision management system for this setting. Snorkel DryBell builds on the Snorkel framework, extending it in three critical aspects: flexible, template-based ingestion of diverse organizational knowledge, cross-feature production serving, and scalable, sampling-free execution. On three classification tasks at Google, we find that Snorkel DryBell creates classifiers of comparable quality to ones trained with tens of thousands of hand-labeled examples, converts non-servable organizational resources to servable models for an average 52% performance improvement, and executes over millions of data points in tens of minutes.
1 INTRODUCTION
Snorkel DryBell addresses the cost and delay of hand-labeling by using organizational knowledge as weak supervision at industrial scale. It extends Snorkel with flexible ingestion, cross-feature serving, and scalable execution for production applications.
- Hand-labeled training sets are costly because they require edge-case instructions, expert training, long turnaround times, and repeated relabeling as applications change.
- The system extends weak supervision from simple pattern matchers to diverse organizational resources and production-scale workflows.
- Snorkel DryBell uses existing organizational knowledge as weak supervision for industrial content and event classification applications.
- Its three central design principles are flexible ingestion of organizational resources, cross-feature production serving, and scalable sampling-free execution.
- 11.5% average relative F1 improvement over classifiers trained on small ∼15K-example development sets reached quality equivalent to using 80K labels.
2 BACKGROUND
The paper builds on weak supervision by combining labeling functions, unsupervised generative modeling, and discriminative training without ground-truth labels. Snorkel DryBell extends this framework to organizational resources, cross-feature serving, and massive-scale execution.
- Modern machine learning models often require massive hand-labeled datasets, motivating cheaper but noisier weak supervision methods.
- Snorkel lets users express multiple programmatic supervision sources as labeling functions that may vary in accuracy, coverage, and correlation.
- The generative model estimates labeling-function quality from agreement and disagreement patterns without using ground-truth labels.
- The resulting probabilistic labels train a discriminative classifier with a noise-aware expected-loss objective.
- Snorkel DryBell adds template-based organizational-resource ingestion, cross-feature serving, and architecture designed for massive scale, including 6.5M data points in one application.
3 CASE STUDIES: WEAK SUPERVISION FOR RAPID DEVELOPMENT
Three Google case studies show how Snorkel DryBell combines heterogeneous organizational resources to develop or update classifiers rapidly. It also transfers supervision from non-servable features to real-time models and estimates source quality at scale.
- Case studies: The case studies cover two content-classification tasks and one real-time event-classification task using diverse organizational labeling sources.
- Content classification: 684,000 unlabeled content examples were weakly supervised with ten labeling functions using heuristics, internal named-entity models, and a topic model.
- Content classification: Previously positive labels slightly reduced end-model F1 after a category change, illustrating rapid depreciation of labeling investments when strategy shifts.
- Real-time event classification: 140 non-servable weak supervision sources trained a deep neural network over real-time event features, identifying 58% more events and improving an internal metric by 4.5%.
- Source quality: Estimating labeling-function accuracies helped identify previously unknown low-quality sources without manually tuning every source combination.
4 CROSS-FEATURE MODEL SERVING
Snorkel DryBell transfers knowledge from non-servable feature sets to new models over real-time or otherwise servable features. In Google applications, this was essential because weak-supervision sources were not directly applicable to the target event-level features, while non-servable sources improved final performance.
- Cross-feature model serving: Snorkel DryBell transfers knowledge from models over aggregate statistics, expensive crawlers, internal models, or graphs to models over external, cheap, or servable features.The paper frames this as a form of transfer learning or transductive model distillation, although the feature sets differ.
- Cross-feature model serving: In the real-time events case study, none of the weak-supervision sources directly applied to the event-level, real-time, servable features of interest.The system instead used those sources to train a new model defined over the servable features.
- Cross-feature model serving: 52% average gains in final performance came from incorporating signals from non-servable sources in the two content applications.These sources included expensive internal models and high-latency web-crawler features that were effectively non-servable.
5 SYSTEM ARCHITECTURE
Snorkel DryBell combines templated MapReduce labeling-function execution with a sampling-free generative model and production serving. Its architecture supports diverse Google resources, distributed computation, and transfer from non-servable labeling features to servable models.
- Architecture: Snorkel DryBell redesigns Snorkel for millions of examples using templated labeling functions and distributed computation.The system decouples labeling-function execution and generative modeling around templates and a distributed compute environment.
- Labeling Function Template Library: Templated C++ classes abstract distributed-filesystem I/O and MapReduce execution while letting subclasses define labeling-function pipelines.The templates support default C++ functions and pipelines that integrate additional services such as NLP models.
- Labeling Function Template Library: Engineers write simple functions that use organizational resources to compute each example’s labeling-function vote.The library supports methods that select inputs for services such as NLP models and interpret their outputs.
- Sampling-Free Generative Model: The generative model combines noisy labeling-function outputs into probabilistic training labels for downstream production systems.Snorkel DryBell uses a conditionally independent generative model and estimates labeling-function accuracies and abstention propensities.
- Sampling-Free Generative Model: The sampling-free implementation exceeds 100 gradient steps per second versus fewer than 50 examples per second for Gibbs sampling, yielding a 2× speedup.This comparison uses ten labeling functions and batch size 64 on a single compute node.
- Production Serving: TFX trains servable discriminative models from Snorkel DryBell’s probabilistic labels, enabling transfer from non-servable labeling resources to production features.The resulting models can use compact, production-available features rather than the broader feature representations used by labeling functions.
6 EXPERIMENTS
Snorkel DryBell was evaluated on content and real-time event classification tasks using Google data, comparing weakly supervised classifiers with hand-labeled and simpler weak-supervision baselines. The results show stronger predictive performance, substantial reductions in hand-labeling needs, and benefits from non-servable resources and generative weighting.
- Topic and Product Classification: Snorkel DryBell-produced training data yielded higher test-set F1 than classifiers trained directly on hand-labeled development sets for both content classification tasks.The resulting discriminative classifiers also outperformed the corresponding generative models.
- Trade-Off Between Weak Supervision and Hand-Labeled Data: Roughly 80K hand-labeled examples were needed on the topic task to match the weakly supervised classifier’s predictive accuracy.Figure 5 compares relative F1 as the number of hand-labeled examples increases.
- Ablation Study: 52% average performance improvement resulted from incorporating non-servable Google resources into labeling functions for the end discriminative classifier.The ablation removed labeling functions depending on non-servable resources and retained only pattern-based rules.
- Ablation Study: 4.8% average performance improvement resulted from using generative-model weights instead of equal weights for labeling-function votes.The comparison used otherwise identical logistic regression classifiers for the topic and product tasks.
- Real-Time Events: 58% additional events of interest were identified by Snorkel DryBell compared with the baseline Logical-OR approach, with a 4.5% internal-metric improvement in event quality.The comparison used the same deep neural network architecture and labeling-function set.
- Real-Time Events: Snorkel DryBell produced a smoother, more nuanced score distribution than Logical-OR, avoiding nearly absolute-confidence predictions and yielding more interpretable and usable outputs.The baseline over-estimated event scores, whereas Snorkel DryBell’s distribution was more consistent with the expected distribution.
7 DISCUSSION
The discussion identifies diverse-resource ingestion, transfer from non-servable to servable features, and rapid labeling-function development as central to industrial weak supervision. It also describes broader organizational and model-serving implications while noting implementation lessons for different users.
- Industrial Design Principles: Labeling functions provide an effective abstraction for integrating heterogeneous organizational resources, including classifiers, structured knowledge, heuristics, and expensive models.These resources differ in both their information and how they are maintained and executed.
- Industrial Design Principles: Denoising labeling functions transfers knowledge from non-servable resources to servable models, which is crucial when production services impose latency agreements.Servable models allow engineers to meet prediction-time constraints while using richer resources during supervision.
- Industrial Design Principles: High-throughput labeling-function development supports rapid human-in-the-loop iteration, whereas waiting for human annotators can slow classifier development.The content classification applications depended on rapidly iterating on labeling functions.
- Organizational Implications: As business strategies change, machine learning teams may need to update training examples for classifiers that identify newly relevant content.The discussion presents low-latency classifier development as increasingly important for organizations dependent on machine learning.
- Organizational Implications: Snorkel DryBell enables organizations to leverage existing resources, including other classifiers, to create new classifiers instead of maintaining a separate hand-labeled set for each one.The paper connects this code-as-supervision paradigm to teams responsible for hundreds or more classifiers.
- Lessons for Other Organizations: Cross-feature transfer uses non-production-servable features to create training data for models defined over production-servable features.The paper describes multiple representations of the same data as useful for weakly supervising models subject to service-level agreements.
- Implementation Lessons: Automatic-differentiation optimization was faster to develop, easier to distribute, and faster to execute than an MCMC sampling approach.The comparison is presented as an implementation lesson from adapting Snorkel for different use cases.
8 RELATED WORK
Related work places Snorkel DryBell among methods that learn from limited labeled data and combine or infer information from multiple sources. The paper distinguishes weak supervision from semi-supervised, transfer, self-training, co-training, pseudo-labeling, zero-shot, data-fusion, truth-discovery, and data-cleaning approaches.
- Weak Supervision: Weakly supervised learning uses multiple noisy but inexpensive label sources as an alternative to hand-labeled training data.Snorkel DryBell is presented as part of research seeking to learn and infer with limited labeled data.
- Machine Learning Related Work: Semi-supervised learning combines labeled and unlabeled data to discover structure that can improve extrapolation from limited labeled examples.The paper lists this as a related machine-learning approach rather than equating it with weak supervision.
- Machine Learning Related Work: Transfer learning, self-training, co-training, pseudo-labeling, and zero-shot learning reduce reliance on labeled data through different forms of reuse or learner-generated labels.These methods are identified as related approaches for learning with limited labeled data.
- Data Systems Related Work: Data fusion and truth discovery estimate the accuracy of conflicting records and integrate them into likely correct records, while data cleaning identifies and corrects dataset errors.HoloClean is cited as a weakly supervised system for learning to correct errors.
9 CONCLUSION
The paper reports initial large-scale industrial deployment results for Snorkel DryBell. It concludes that weak supervision can match the training-data needs of classifiers otherwise requiring tens of thousands of hand-labeled examples while connecting organizational resources to new ML problems.
- Conclusion: Weak supervision trained classifiers that would otherwise require tens of thousands of hand-labeled examples in a large-scale industrial setting.The conclusion reports initial deployment results for the Snorkel DryBell framework.
- Conclusion: Snorkel DryBell connected a wide range of organizational resources to new machine learning problems and improved predictive accuracy.The paper concludes that these results indicate a potential significant role for weak supervision in industrial ML development.