PaperlayerView exact revision on arXiv

Source-linked AI summary

Interactive Weak Supervision: Learning Useful Heuristics for Data Labeling

Benedikt Boecking, Willie Neiswanger, Eric Xing, Artur Dubrawski

arXiv:2012.06046v2cs.LGcs.AIstat.ML

TL;DR

Obtaining labeled data can be costly, while crafting useful heuristics takes time and foresight. IWS proposes labeling-function candidates for expert feedback and uses those judgments to select weak-supervision sources; across many tasks, it reports stronger downstream test performance than Snuba and standard active learning per user annotation.

  • Problem

    Obtaining labeled data is often costly, and creating effective weak-supervision heuristics requires time, creativity, and foresight.

    Source: §1 · pp. 1–2

  • Method

    IWS proposes candidate labeling functions, learns from experts’ feedback about them, and uses selected functions to produce probabilistic labels for training an end classifier.

    Source: §1 · p. 2, §3.2 · p. 4

  • Results

    Across many tasks, IWS achieves superior downstream test-set predictive performance to Snuba and standard active learning when measured by the number of user annotations.

    Source: §5 · p. 9

  • Takeaways & Limitations

    A small number of expert interactions can select good weak-supervision sources from a large candidate pool and yield competitive end classifiers.

    Source: §5 · p. 9

    The approach’s current evaluation covers text and image data, and suitable interpretable labeling-function families may be difficult to find for some tasks and data types.

    Source: §3.1 · p. 4, §5 · p. 9

Abstract

from arXiv · show

Obtaining large annotated datasets is critical for training successful machine learning models and it is often a bottleneck in practice. Weak supervision offers a promising alternative for producing labeled datasets without ground truth annotations by generating probabilistic labels using multiple noisy heuristics. This process can scale to large datasets and has demonstrated state of the art performance in diverse domains such as healthcare and e-commerce. One practical issue with learning from user-generated heuristics is that their creation requires creativity, foresight, and domain expertise from those who hand-craft them, a process which can be tedious and subjective. We develop the first framework for interactive weak supervision in which a method proposes heuristics and learns from user feedback given on each proposed heuristic. Our experiments demonstrate that only a small number of feedback iterations are needed to train models that achieve highly competitive test set performance without access to ground truth training labels. We conduct user studies, which show that users are able to effectively provide feedback on heuristics and that test set results track the performance of simulated oracles.

1 INTRODUCTION

Obtaining labeled data can be costly, and designing useful weak-supervision heuristics takes time and foresight. IWS asks experts to judge candidate labeling functions, then uses the selected functions to train a classifier; its reported results compare favorably with active learning.

1 INTRODUCTION

  • Labeled data can be scarce, while having annotators inspect individual samples is often expensive and time-consuming.Privacy concerns, required expertise, and shifting problem definitions can add to the cost.Source: §1 · p. 1
  • Creating effective labeling functions requires time, random inspection for ideas, and foresight to find functions that cover enough data and add predictive value.The first few functions may be easier to devise than later ones.Source: §1 · pp. 1–2
  • IWS proposes candidate labeling functions for expert feedback, learns which are likely to be accurate, and uses the resulting set to train a weakly supervised end classifier.An unsupervised label model estimates latent class labels from the selected functions.Source: §1 · p. 2
  • After 200 LF annotations, IWS’s mean test set AUC requires at least three times as many active-learning iterations annotating data points.The average user response time was also shorter for LF queries than for active-learning queries on data points.Source: §1 · p. 2

2 RELATED WORK

Prior work has combined weak supervision with active learning, used explanations to generate heuristics, and developed systems for automatically generating labeling functions. These approaches provide related strategies for obtaining or constructing weak supervision sources.

2 RELATED WORK

3 METHODS

The methods define how labeling functions generate probabilistic training labels, then introduce IWS to elicit expert judgments about candidate functions and select a useful subset. The selection strategy varies with whether the final set is unrestricted, size-limited, or restricted to expert-validated functions.

3 METHODS

  • The methods focus on binary end classification, and suitable labeling-function families may not exist for every task or data type.The authors argue that when such a family exists, experts can choose one and judge its generated candidates instead of creating every function from scratch.Source: §3 · p. 3, §3.1 · p. 4

3.1 PRELIMINARIES

  • The data-programming label model combines noisy function outputs to estimate latent labels, modeling function accuracy and propensity while assuming conditional independence given the class.A noise-aware end classifier is trained using the resulting probabilistic labels.Source: §3.1 · p. 3
  • Labeling-function families generate expert-interpretable candidates from parameterized templates, such as word lookups, regular expressions, trees, image detectors, or time-series motifs.For text, candidate parameters can be sampled from unlabeled data; the family must be chosen with domain-expert input.Source: §3.1 · pp. 3–4

3.2 INTERACTIVE WEAK SUPERVISION

  • IWS sequentially queries experts about candidate labeling functions, estimates their usefulness, selects a final subset, and passes it through a label model to train an end classifier.The classifier is trained on probabilistic labels with a noise-aware loss.Source: §3.2 · pp. 4–5
  • Experts judge whether a candidate function is useful, and the feedback model assumes that the probability of a positive judgment increases monotonically with the function’s accuracy.The model represents binary feedback as Bernoulli and links its probability to latent accuracy.Source: §3.2 · p. 4
  • For an unrestricted final set, IWS targets functions above an accuracy threshold and uses level-set estimation to explore candidates near that threshold.The described algorithm uses a straddle acquisition function and estimates posterior accuracy from feedback.Source: §3.2 · p. 5
  • When the final set is size-limited, IWS selects functions above the accuracy threshold and ranks them by an accuracy-coverage trade-off.The resulting selection is a thresholding and sorting operation.Source: §3.2 · p. 6
  • When only expert-validated functions may be used, IWS treats selection as active search and seeks to maximize validated functions within the query budget.Its acquisition score is the posterior probability that a function receives a positive usefulness label.Source: §3.2 · p. 6

4 EXPERIMENTS

The experiments evaluate interactive weak supervision against active learning, Snuba, and full-label training across text and image tasks. Oracle and real-user studies assess downstream classifier performance and whether users can judge candidate labeling functions.

4 EXPERIMENTS

  • The experiments cover six binary text tasks from Amazon, IMDB, and BiasBios, plus COCO tasks using captions or images to classify person, sports, and vehicle labels.Compared approaches include IWS-LSE with unbounded or bounded function sets, IWS-AS with validated functions, active learning, Snuba, and full training labels.Source: §4 · p. 7
  • Text labeling functions use unigram matches, while image functions use mutual-nearest-neighbor clusters extended through shared nearest neighbors.Each extended image cluster defines a labeling function that assigns the same label to its members.Source: §4 · pp. 7–8

4.2 USER EXPERIMENTS ON TEXT

  • Human IWS-AS performance closely tracked the simulated oracle after about 100 iterations; matching its 200-iteration mean test AUC required about 600 active-learning iterations on Amazon and over 1,000 on IMDB.The figure’s iterations correspond to expert annotations, with IWS querying labeling functions and active learning querying samples.Source: Figure 3 · p. 8, §4.2 · p. 9

4.1 ORACLE EXPERIMENTS

  • The simulated oracle marks a labeling function useful when its accuracy is at least 0.7.The oracle experiments examine how downstream classifiers perform when function feedback follows this threshold.Source: §4.1 · p. 8

4.2 USER EXPERIMENTS ON TEXT

  • In the real-user text study, users judged high-accuracy functions useful with few mistakes, and Amazon and IMDB users mislabeled about 2% and 9% of sampled items, respectively.Users saw a heuristic description and intended label, with document snippets available when needed.Source: §4.2 · pp. 8–9

4.1 ORACLE EXPERIMENTS

  • IWS-LSE significantly outperformed other approaches on five of six text datasets, matched Snuba on one, and approached full-label performance; image-task results were competitive.The COCO comparisons cover labeling functions generated from captions and directly from images.Source: §4.1 · p. 8, Figure 4 · p. 9

5 CONCLUSION

The paper concludes that a small number of expert interactions can select useful weak-supervision sources and produce competitive end classifiers. It also distinguishes IWS settings by whether final labeling-function sets are unrestricted, size-limited, or restricted to expert-reviewed functions.

5 CONCLUSION

  • A small number of expert interactions can select useful labeling functions from a large candidate pool and yield competitive classifiers; experiments with real users found models matching simulated-oracle performance.The conclusion reports stronger downstream test performance than Snuba and standard active learning on many tasks, measured by user annotations.Source: §5 · p. 9
  • Current evidence covers text and image data; broader task types need suitable labeling-function families, and some may not offer the interpretability experts need to judge them.Large class counts may also produce very large candidate sets that require more efficient search.Source: §5 · p. 9

A PSEUDOCODE FOR DIFFERENT IWS SCENARIOS

  • The pseudocode distinguishes unrestricted final function sets, size-bounded sets that need no user validation, and sets limited to functions reviewed by experts.The expert-review constraint may apply for security or legal reasons.Source: A PSEUDOCODE FOR DIFFERENT IWS SCENARIOS · p. 14

B.1 USER EXPERIMENTS ON IMAGES WITH IMAGE LABELING FUNCTIONS

In the COCO sports image study, users judged image labeling functions consistently and well. With human feedback, IWS performance closely tracked the simulated oracle after about 100 iterations.

B.1 USER EXPERIMENTS ON IMAGES WITH IMAGE LABELING FUNCTIONS

  • After about 100 iterations, image-task IWS performance with human feedback closely tracked simulated-oracle performance, and users judged labeling-function accuracy consistently and well.The study used mutual-nearest-neighbor image labeling functions for COCO sports classification.Source: §B.1 · p. 15
  • Users assessed image functions by inspecting clusters averaging 7.9 images and 15 sampled images from extended clusters; developing better image primitives remains an open challenge.The study set k1 to 20 and k2 to 1,500, with performance reported as robust to changes in these parameters.Source: §B.1 · p. 15

B.2 EXPERIMENT AND IMPLEMENTATION DETAILS

The experiments use text and image datasets and specify the models, features, and initialization used in evaluation. The appendix also defines three IWS settings according to which labeling functions can enter the final set.

B.2 EXPERIMENT AND IMPLEMENTATION DETAILS

  • Text experiments cover six binary tasks across Amazon reviews, IMDB reviews, and four occupation pairs from Bias in Bios.The Amazon subset has 200,000 reviews split into 160,000 training and 40,000 test points; IMDB has 25,000 training and 25,000 test samples.Source: §B.2 · p. 15
  • COCO supplies the cross-modal caption-and-image and image-only tasks, with its 4,952-image validation set reserved as test data.Those test images are never used elsewhere in the pipeline.Source: §B.2 · p. 15
  • IWS models LF usefulness with a bagging ensemble of 50 two-hidden-layer MLPs, while uncertain user responses receive half weight.The end classifier is also an MLP; text models use SVD-based features, and image models use pretrained ResNet-18 features.Source: §B.2 · pp. 15–16

B.3 FULL IWS RESULTS AND ALL BASELINES

  • Figure 6 plots mean test AUC against iterations for IWS-LSE and IWS-AS alongside active learning, Snuba, all-label training, and random-acquisition IWS.The appendix distinguishes unrestricted final LF sets, size-limited sets, and sets restricted to expert-inspected LFs; one iteration represents one expert label, but query effort varies by application.Source: Figure 6 · p. 16, §B.3 · p. 16

B.2 EXPERIMENT AND IMPLEMENTATION DETAILS

  • IWS begins with eight LF queries: four LFs with accuracy between 0.7 and 0.75 and four with arbitrary accuracy; active learning starts with the same number of labeled samples.Source: §B.2 · p. 16

B.4.1 INTERFACE AND EXPERIMENT PROMPT

The user study presents candidate labeling functions with task context and guidance for judging usefulness, including an option to abstain. It also records feedback confidence and tracks model and LF statistics across iterations.

B.4.1 INTERFACE AND EXPERIMENT PROMPT

  • Users could choose “I don’t know” when unsure rather than answer the LF query.Source: §B.4.1 · p. 16
  • Figure 7 illustrates the prompt and answer options; users judged an LF useful if they expected better-than-random accuracy and otherwise labeled it bad.Source: §B.4.1 · p. 16, Figure 7 · p. 17
  • Users received the classification task and document domain, a description of the term-based LF, and examples of better-than-random and arbitrary heuristics.Source: §B.4.1 · p. 17
  • For each query, users could inspect four random documents where the LF applied, but were told to consider the LF before viewing them.Source: §B.4.1 · p. 17
  • Uncertainty was recorded alongside binary LF labels and used as a training weight: 1 for certain responses and 0.5 otherwise.Source: §B.4.1 · p. 17

B.4.2 ADDITIONAL STATISTICS OF USER EXPERIMENTS

  • Figure 8 tracks each user's downstream test AUC, the number of LFs accepted as useful, and maximum positive correlation with previously accepted LFs.The correlation calculation considers entries where at least one LF is nonzero.Source: §B.4.2 · p. 17

B.5.1 IWS WITH REAL USERS, SENTIMENT CLASSIFICATION

The appendix gives examples of term-based heuristics that real users marked useful for IMDB and Amazon sentiment tasks, including frequently accepted terms, high-accuracy terms, and randomly selected examples.

B.5.1 IWS WITH REAL USERS, SENTIMENT CLASSIFICATION

  • Frequently useful IMDB terms include positive words such as “wonderful” and “brilliant” and negative words such as “worst” and “boring.”Source: §B.5.1 · p. 17
  • The IMDB examples with the highest underlying accuracy include terms such as “flawless” and “breathtaking” for one class and “unwatchable” for the other.Source: §B.5.1 · p. 17
  • Amazon examples include frequently useful positive terms such as “wonderful” and “favorite” and negative terms such as “terrible” and “garbage.”Source: §B.5.1 · p. 19
  • The Amazon examples also include terms annotated useful with the highest underlying accuracy.Source: §B.5.1 · p. 19
  • Two randomly selected sets of user-approved IMDB terms include both sentiment-associated words and less direct terms such as “expected,” “reason,” and “mistake.”Source: §B.5.1 · p. 19

B.5.2 IWS WITH AN ORACLE, OCCUPATION CLASSIFICATION

The appendix presents oracle-selected term heuristics for four occupation-classification tasks and an ablation of the LF-partition threshold used by IWS-LSE.

B.5.2 IWS WITH AN ORACLE, OCCUPATION CLASSIFICATION

  • For journalist versus photographer, frequently useful oracle terms distinguish photography and studio from journalism and reporting.Source: §B.5.2 · p. 19
  • For painter versus architect, the example terms contrast construction and design vocabulary with painting and gallery vocabulary.Source: §B.5.2 · p. 19
  • For professor versus physician, useful terms include clinical cues such as “medical” and “surgery,” contrasted with academic cues such as “teaching” and “published.”Source: §B.5.2 · p. 19
  • For professor versus teacher, oracle-selected terms include “students” and “classroom” for one class and “research” and “journals” for the other.Source: §B.5.2 · p. 19
  • Across datasets, test performance is very similar after about 100 iterations over a wide range of LF-partition thresholds.The plotted IWS-LSE-ac setting uses ˜m = 100.Source: Figure 9 · p. 20

B.6 ABLATION OF IWS PARAMETER SETTINGS

The ablations show good performance across a wide range of LF-accuracy thresholds and final-set sizes, with a threshold too close to 1.0 potentially underperforming by stifling exploration.

B.6 ABLATION OF IWS PARAMETER SETTINGS

  • IWS-LSE-a and IWS-LSE-ac perform well across a wide range of LF-accuracy thresholds, though a threshold too close to 1.0 can underperform.The authors suggest that a near-1.0 threshold may stifle exploration within the limited user-query budget.Source: §B.6 · p. 20
  • Final LF-set sizes from ˜m = 50 to 200 produce good results for IWS-LSE-ac.The final-set size is bounded at each iteration by the number of LFs annotated as useful plus ˜m.Source: §B.6 · p. 20, Figure 10 · p. 21
  • IWS-LSE-ac ranks LFs using both estimated accuracy and observed coverage, via (2αj −1) ∗ˆlj.The final-set size parameter ˜m controls the set-size bound alongside the number of LFs annotated as useful.Source: §B.6 · pp. 20–21

C ON LABELING FUNCTION PROPERTIES

This section analyzes how LF accuracy and propensity affect probabilistic estimates of latent labels and the choice of label-model parameters. Under its stated assumptions, it identifies accuracy–coverage trade-offs and conditions relevant to those estimates.

C ON LABELING FUNCTION PROPERTIES

  • The label-model estimate directly depends on the canonical accuracy parameters θ(1), not propensity parameters θ(2); binary probabilistic labels simplify to a sigmoid.The analysis assumes binary classification and introduces equal-across-classes LF accuracy and evenly divided errors independent of the true class.Source: C ON LABELING FUNCTION PROPERTIES · pp. 22–23
  • LF ranking can use the trade-off term (2αj −1)lj, combining accuracy αj with labeling propensity lj, also called coverage.The section identifies this term as useful when accuracy is known or estimated and empirical coverage is used to estimate propensity.Source: C ON LABELING FUNCTION PROPERTIES · pp. 23–24

C.1 WHAT ARE THE OPTIMAL LABEL MODEL PARAMETERS?

  • The optimal-parameter analysis considers label models with accuracy factors alone and with both accuracy and LF-propensity factors.The section frames these as two cases for determining optimal θ parameters.Source: §C.1 · p. 25

C.1.1 ASSUMING ACCURACY FACTORS ONLY

  • With accuracy factors alone, P(Hj = Y) combines an LF’s accuracy with its propensity to label rather than abstain.The analysis defines the LF output H and class variable Y and considers a model that only represents LF accuracy.Source: §C.1.1 · p. 25

C.1.2 ASSUMING ACCURACY FACTORS AND LF PROPENSITY FACTORS

  • When accuracy and propensity factors are modeled, the binary-case optimal accuracy parameter θ(1)*j is positive only if αj > 0.5.Here αj is the conditional probability that the LF is correct when it does not abstain.Source: §C.1.2 · pp. 26–27
Loading 2012.06046v2…