Language Models in the Loop: Incorporating Prompting into Weak Supervision

1 Introduction

• Limited labeled training data motivates programmatic weak supervision, which combines noisy heuristic labelers on unlabeled examples to create training labels.Labelers may vote or abstain, and a label modeling stage resolves conflicts.Source: §1 · pp. 1–2
• Prompt wording sensitivity and limits on instruction complexity make one prompt insufficient for some nuanced decision boundaries.The paper motivates a framework that lets subject matter experts refine task contours with multiple prompts.Source: §1 · p. 2
• The framework maps prompt responses to labels or abstentions, then uses a label model to combine votes into training data for an end model.Prompts let users express complementary heuristics, including fuzzy concepts that can be difficult to encode in code.Source: §1 · p. 2
• Prompt-based labeling functions produce more accurate end models than original code-based labeling functions and often outperform direct task prompting across GPT-3 and T0 models.The authors attribute gains to complementary heuristics and report that the prompts usually make weakly correlated mistakes.Source: §1 · p. 3
• 20.2 percentage points is the average improvement from prompting pre-trained models with multiple heuristic tasks over directly prompting them to solve the task.This is the reported zero-shot comparison.Source: §1 · p. 3
• 7.1 percentage points is the average improvement from translating code labeling functions into prompts when using T0++.The comparison is between weakly supervised models using prompted functions and those using the code functions.Source: §1 · p. 3

2.1 Weakly Supervised Machine Learning

• Programmatic weak supervision combines heuristic labeling functions that can disagree or abstain, then models their outputs to infer labels without ground truth.The label model can account for dependencies among heuristic sources; many such methods are integrated into Snorkel.Source: §2.1 · p. 3
• Weak supervision has been extended to structured, partial, hierarchical, and other supervision settings, and can combine with semi-supervised, self-supervised, and active learning.This work differs from representation-based approaches by using large pre-trained models directly as zero-shot labeling functions.Source: §2.1 · p. 4

2.2 Language Models and Prompting

• Prompting poses tasks as natural-language text and interprets a language model’s predicted words as task solutions, either zero-shot or after fine-tuning.Prior work also addresses prompt sensitivity and methods to improve zero-shot performance.Source: §2.2 · p. 4
• Prior language-model supervision work generates or edits examples, uses alternative prompt views, or proposes new labeling functions from labeled data.In contrast, this paper uses multiple prompted tasks as supervision and requires no changes to existing weak-supervision pipelines.Source: §2.2 · p. 4

3.1 Workflow

• The workflow targets subject-matter experts with abundant unlabeled data, a small manually labeled development set, and potentially API-only access to large models.The development set supports prompt and hyperparameter choices; the approach can also integrate with hand-engineered programs.Source: §3.1 · p. 5, §3 · p. 5
• A prompted labeling function pairs a prompt template with a label map that converts model responses into label votes or abstentions.Experts can assess and refine prompts using unlabeled examples and the small labeled development set.Source: §3.1 · p. 6
• Aggregated weak-supervision votes produce probabilistic labels for training a smaller classifier that can operate independently of the underlying language model.The label model combines votes, and the end model is trained against its probabilistic estimates.Source: §3.1 · p. 6

3.2 Developing Prompted Labeling Functions

• Multiple prompts can collect related signals because task supervision is multifaceted and models may know useful signals without knowing how to integrate them.Separate prompts capture salient task components rather than relying on a single long, nuanced instruction.Source: §3.2 · p. 6
• 100% precision and 45% recall for RegEx compare with 76% precision and 58% recall for the prompted function.The prompt finds more true positives with meaning similar to “check out,” but also misfires on some false positives.Source: Figure 3 · p. 7, §3.2 · p. 7
• Prompts can express ambiguous concepts that are difficult to enumerate in code, while the three examples range from broad intent to a specific phrase.The intermediate channel-reference prompt can capture indirect expressions such as “Like and subscribe!”Source: §3.2 · p. 7

3.3 Calibration

• Contextual calibration estimates scaling weights from predictions on null inputs and applies the resulting transformation to prompted labeling-function predictions.This addresses model probability biases without requiring labeled data to learn a recalibration transformation.Source: §3.3 · p. 7

4 Experimental Study

• 20.2 percentage points average improvement over direct zero-shot prompting came from models built with prompted labeling functions; T0++ prompted labelers improved 7.1 percentage points over code-based functions.The latter comparison uses the best language model, T0++.Source: §4 · p. 8

4.1 Datasets

• The evaluation focuses on WRENCH’s YouTube, SMS, and Spouse text-classification tasks, excluding four datasets used in T0++ training.Table 1 summarizes dataset statistics and defines P(positive) as positive-label class balance across gold-labeled splits.Source: §4.1 · p. 8, Table 1 · p. 8

4.2 Translating WRENCH Labeling Functions into Prompts

• Because WRENCH’s labeling functions had already been developed, the study translates existing rules into Yes/No prompts rather than developing prompts from scratch.The authors note this is more restricted than initial prompt development, and illustrate replacing family-keyword enumeration with a general question about whether two people are family members.Source: §4.2 · p. 8

4.3 Comparing Programmatic Labelers

• The comparison uses original WRENCH labelers, one-prompt zero-shot labeling, and prompted versions of WRENCH labelers, with votes denoised and used to train RoBERTa end models.The original-labeler approach uses majority vote; gold-labeled test splits evaluate model performance.Source: §4.3 · p. 9

4.4 Large Language Models

• The prompted labeling functions are evaluated with GPT-3 Instruct engines Ada, Babbage, and Curie, plus T0++; DaVinci is omitted because of cost constraints.T0++ is an 11B-parameter model based on the T5 architecture.Source: §4.4 · pp. 9–10

4.5 Evaluation Metrics

• Evaluation uses precision, recall, F1, and accuracy, with accuracy or F1 selected according to each WRENCH benchmark's default metric.Reported metrics are means and standard errors across six training runs with different random seeds.Source: §4.5 · p. 10

4.6 Results

• Table 3 compares zero-shot and prompted weak supervision across four large language models using calibrated prompts.It reports performance metrics as means and standard errors over six training replicates.Source: Table 3 · p. 10
• Prompted weak supervision outperforms zero-shot by an average of 18.2% across language models and datasets, with results ranging from -26.7% to 100%.Table 3 reports performance for four language models; scores are means and standard errors over six training replicates.Source: §4.6.1 · p. 10, Table 3 · p. 10

4.6.1 Prompted Weak Supervision

• With T0++, prompted weak supervision approaches or exceeds WRENCH labeling-function models and improves Spouse performance by 25 F1-score points.Across datasets, T0++ gains an average of 39.5% over zero-shot; InstructGPT prompted weak supervision often reduces performance, with gains consistently observed only on YouTube.Source: §4.6.1 · p. 10

4.6.2 Prompt Calibration

• On SMS, T0++ combines noisier, higher-coverage prompts to reach 91.8 F1, just 0.6 points below the WRENCH labeling functions' 92.4 F1.WRENCH keyword labelers have high precision but under 2% coverage; combining and denoising multiple prompts yields the end-model score.Source: §4.6.2 · pp. 10–11, Figure 4 · p. 11
• For T0++, calibration decreases prompt accuracy by an average of 1.5 points while increasing coverage by 2.4 points; InstructGPT accuracy falls 2.0–10.5 points as coverage rises 40–69.7 points.Babbage and Ada prompts can approach 100% coverage when they give the same answer across examples; T0++ alone consistently yields more conservative labelers.Source: §4.6.2 · pp. 11–12, Figure 5 · p. 12

4.6.3 Diversity Measures

• YouTube T0++ prompts show more disagreement and less correlated errors than InstructGPT prompts, suggesting more complementary label information and greater ensemble efficiency.The paper reports similar patterns on other datasets.Source: §4.6.3 · pp. 12, 14, Figure 6 · p. 13

5 Discussion and Conclusion

• Multiple prompted labeling functions improve over underspecified zero-shot classification and make task insight and language-model knowledge inspectable.The authors describe this as a mechanism for contextualizing task insight and querying knowledge in large language models.Source: §5 · p. 14
• Prompts may improve as their underlying language models improve, and the translated prompts likely underestimate potential performance because new prompts were not developed and refined.The study focused on translating existing labeling functions.Source: §5 · p. 14
• T0++ consistently outperforms the InstructGPT family in prompted weak supervision.The authors suggest that future work could generate prompted pretraining data more closely aligned with SME prompt design; querying a model may also improve the underlying model [37].Source: §5 · p. 14
• Calibration gains are consistent across prompt collections but inconsistent and brittle for individual prompts.The authors expect improved calibration methods to help SMEs refine individual labeling-function thresholds.Source: §5 · p. 14

A.1 GPT-3 API Costs

• Table 5 reports estimated OpenAI API query costs for labeling WRENCH training sets.The supplied passage does not give the cost values.Source: Table 5 · p. 20
• The cost estimates concern training-set labeling, not test-set evaluation.Source: Table 5 · p. 20
• The estimates cover the InstructGPT family of language models.Source: Table 5 · p. 20

A.2.1 End Model Generalization

• The RoBERTa end model provides consistent improvements over direct evaluation of the same calibrated zero-shot prompts.Source: §A.2.1 · p. 20
• The zero-shot prompt is used as a labeling function by programmatically labeling the training split before training the RoBERTa end model.Source: §A.2.1 · p. 20
• Table 6 compares direct zero-shot test classification with using the same prompt to label training data for a RoBERTa end model.The table caption identifies the comparison; no cell values are supplied here.Source: Table 6 · p. 20, §A.2.1 · p. 20

A.2.2 Zero Shot Labeling Function

• T0++ improved by an average of 0.2 F1 points and InstructGPT Ada by 2.6 F1 points when the zero-shot prompt was added as a labeling function, while Curie dropped 2.8 and Babbage 0.8 F1 points on average.Source: §A.2.2 · p. 20
• Adding the zero-shot prompt as another labeling function generally negatively impacted performance, despite gains for T0++ and InstructGPT Ada.Source: §A.2.2 · p. 20
• Table 7 presents results for prompted weak-supervision models with the zero-shot prompt added as an additional labeling function.The supplied passage states the comparison's scope but not table values.Source: §A.2.2 · p. 20

A.3.1 Contextual Calibration

• Calibration improves prompted-labeler performance, with the largest gains where uncalibrated prompts perform pathologically.Source: §A.3.1 · p. 20
• InstructGPT models performed very poorly in many zero-shot and prompted weak-supervision experiments.Source: §A.3.1 · p. 20
• Tables 9 and 10 outline contextual-calibration benefits for zero-shot and prompted weak-supervision baselines, respectively.Source: §A.3.1 · p. 20
• Table 7 concerns adding the zero-shot prompt as an additional labeling function in prompted weak supervision.The supplied caption identifies the setup, not its outcomes.Source: Table 7 · p. 21
• Figures 12 and 13 show class-conditional calibration changes alongside accuracy changes across datasets and language models.Source: §A.3.1 · p. 21
• For T0++, calibration produces little-to-no accuracy change for prompts labeling the negative class.Source: §A.3.1 · p. 21
• Table 8 presents the Table 3 performance metrics for uncalibrated prompts.The supplied passage does not provide the values.Source: §A.3.1 · p. 20, Table 8 · p. 21

• Tables 11, 12, and 13 contain the complete set of translated WRENCH labeling functions.Source: §A.4 · p. 21

• Tables 9 and 10 report contextual-calibration performance impact for zero-shot and prompted weak-supervision models, with mean and standard error across six training replicates.Source: Table 9 · p. 22, Table 10 · p. 23
• YouTube and Spouse accuracy-versus-coverage plots compare calibrated results above with uncalibrated results below, using color for language model and marker style for class label.Source: Figure 7 · p. 22, Figure 8 · p. 25
• Rules typically label less than 2% of the training set but have high precision; SMS and Spouse show low overall disagreement, while YouTube has higher disagreement but limited correlated errors.Source: Figure 9 · p. 26
• SMS and Spouse prompt-pair diversity measures use color intensity to show the percentage of training examples labeled by each pair of prompts.Source: §A.5 · p. 21, Figure 10 · p. 27, Figure 11 · p. 28
• Figures 12 and 13 break down accuracy and coverage changes from contextual calibration by negative and positive class label, respectively.Source: Figure 12 · p. 29, Figure 13 · p. 30