A Survey on Programmatic Weak Supervision

1 Introduction

• Deep learning often depends on massive labeled datasets, but manually collecting, curating, and labeling them is prohibitively time-consuming and labor-intensive.Active, semi-supervised, and transfer learning still require clean labeled data for satisfactory performance.Source: §1 · p. 1
• The introduction asks whether cheaper sources such as knowledge bases, crowdsourced labels, and heuristic rules can be combined with broader weak-supervision inputs.The question follows discussion of distant supervision, crowd labels, and heuristic rules as alternatives to manual annotation.Source: §1 · p. 1
• PWS frameworks combine weak supervision sources, encoded as labeling functions (LFs), to generate training labels [Ratner et al., 2016; Ratner et al., 2017].LFs are user-defined programs that label subsets of data; their votes may be noisy and conflicting, so label models aggregate them before end-model training.Source: §1 · p. 1
• The paper surveys PWS formulations, label, end, and joint models, complementary limited-label approaches, applications, and future research directions.It also notes work on automatic or interactive LF generation and on learning dependencies among LFs.Source: §1 · pp. 1–2

2 Preliminary

• PWS applies m labeling functions to unlabeled data X to form an n × m label matrix L, then uses L and X to produce an end model f_w.The formal setting allows some methods to take a dependency graph of sources as input.Source: §2 · p. 2
• Two-stage methods aggregate L into soft or hard training labels with a label model, then use those labels to train the end model.The survey reviews label models and end models separately.Source: §2 · p. 2
• One-stage methods train label aggregation and final prediction simultaneously, typically using separate neural networks, and are called joint models.The paper describes this as a direct approach to handling weak labels.Source: §2 · p. 2

3 Labeling Functions

• LFs are the weak-supervision sources at the core of PWS, and may label only a subset of data points.The survey considers their types, development, and potential dependencies.Source: §3 · p. 2, §3.1 · p. 2

3.1.1 User-written Heuristics

• Heuristic LFs encode domain knowledge as rules, including keyword or regex matches for text and object or visual-property rules for images [Ratner et al., 2017; Meng et al., 2018; Awasthi et al., 2020].Source: §3.1.1 · p. 2

3.1.2 Existing Knowledge

• Existing knowledge sources include knowledge bases for distant supervision [Liang et al., 2020], pre-trained models, and noisy third-party tools.Examples include matching relation mentions against DBPedia, repurposing models with different label spaces, and using TextBlob or NER taggers as LFs.Source: §3.1.2 · pp. 2–3

3.1.3 Crowd-sourced Labels.

• Crowd-sourcing contributors can each be represented as an LF that noisily labels data points [Ratner et al., 2017; Lan et al., 2020].The survey’s example uses contributors’ five-category sentiment labels for weather-related tweets.Source: §3.1.3 · p. 3

3.2 Labeling Function Generation

• LF development typically draws on subject-matter experts, domain knowledge, and a small sample of unlabeled data called a development set.Because exploring development data and designing LFs can be challenging and time-consuming, the survey groups effort-reduction approaches as automatic, interactive, or guided generation.Source: §3.2 · p. 3
• Automatic generation learns LFs from a small labeled set or seed LFs, so these approaches still require initial supervision.Snuba learns weak classifiers; TALLOR compounds seed rules; GLaRA augments seed LFs using semantic relationships.Source: §3.2 · p. 3
• Interactive generation queries users about candidate LFs and adapts toward high-precision functions; it has been reported to outperform instance-level active learning at lower annotation cost.Darwin and IWS generate candidates using n-grams or context-free grammar information [Galhotra et al., 2021; Boecking et al., 2021].Source: §3.2 · pp. 3–4
• Guided generation curates the development set to help experts explore data and develop informative LFs, soliciting supervision at the functional rather than individual-label level.Source: §3.2 · p. 4

4 Label Model

• LF dependencies matter for recovering latent labels, yet some label models assume conditional independence between LFs.Source: §4 · p. 4

4.1 LF Dependency Structure

• Dependency handling ranges from user-specified relations to automatic learning from labels or static analysis of LF code, then incorporation into graphical models.Source: §4.1 · p. 4

4.2 Label Model for Classification

• Classification label models range from parameter-free majority voting to probabilistic graphical models that fit LF outputs, with DP, MeTaL, and FlyingSquid differing in parameterization and estimation.Source: §4.2 · p. 4
• CAGE supports continuous LFs, NPLM partial-label LFs, and PLRM indirect LFs, expanding the supervision sources usable in PWS.Source: §4.2 · pp. 4–5

4.3 Label Model for Sequence Tagging

• Sequence-tagging methods model token dependencies with HMM variants and can combine supervision at different resolutions using graphical models.Source: §4.3 · p. 5

4.4 Label Model for General Learning Tasks

• UWS extends PWS to ranking, regression, and learning in hyperbolic manifolds through an efficient method-of-moments approach in embedding space.Source: §4.4 · p. 5

• End models train discriminative downstream predictors on probabilistic labels, and noise-aware losses can account for residual label noise.Source: §5 · p. 5
• COSINE uses self-training to generate pseudo-labels for data uncovered by labeling functions.Source: §5 · p. 5
• Other methods designed for learning with noisy labels can also serve as end models.Source: §5 · p. 5

6 Joint Model

• Joint models train label and end models end-to-end so they can enhance each other, commonly using neural label models for feature-conditioned aggregation.Source: §6 · p. 5
• Neural joint models can implicitly capture LF dependencies but generally cannot incorporate pre-given dependency structure.Source: §6 · p. 5

6.1 Joint Model for Classification

• Classification joint methods include posterior-network agreement, constrained min-max objectives, small-clean-set denoising, and teacher-student self-training.Source: §6.1 · p. 5

6.2 Joint Model for Sequence Tagging

• For sequence tagging, ConNet aggregates source-specific CRF transitions using LF-quality-conditioned attention, while DWS uses hard EM with a CRF layer.Source: §6.2 · pp. 5–6

• PWS can provide initial or expanded labels for active learning, while active learning can query examples likely to expose label-model errors or improve the model.Source: §7 · p. 6
• Transfer learning and PWS are described as complementary: transfer reduces but does not eliminate labeled-data needs, which PWS can supply.Source: §7 · p. 6
• Semi-supervised learning can use PWS-generated labels on unlabeled data, and PWS methods can use self-training; the survey identifies combining clean and weak labels as future work.Source: §7 · p. 6

Extend to More Complex Tasks

• Most PWS methods support classification or sequence tagging, while complex reasoning tasks such as question answering, navigation, and scene graph generation demand more labeled data.These tasks can draw on text, images, and tables, but current PWS methods consider labeling functions from only one modality.Source: §8 · p. 6

Extend the Scope of Usable LFs

• Future PWS could incorporate additional labeling-function sources, including physical rules, to reduce human effort in training-data curation.The paper notes attempts to broaden usable sources, while identifying physical rules as another potential source for complex tasks.Source: §8 · p. 6

Ethical and Trustworthy AI

• Code-generated PWS labels can be inspected, audited, governed, and edited to reduce bias, but improper use can also introduce more direct bias into training data.The paper calls for systematic study of PWS's ethical and governance implications.Source: §8 · p. 6

9 Conclusion

• The survey introduces existing approaches for each component of the PWS workflow.It frames PWS as a way to produce large-scale annotations with minimal human effort despite the cost and time of manual annotation.Source: §9 · p. 6
• The paper describes how PWS can interact with methods from related fields for downstream applications and lists datasets and recent applications from the literature.Source: §9 · p. 6
• The survey discusses current PWS challenges and future directions to inspire further research.Source: §9 · p. 6