Source-linked AI summary
Training Complex Models with Multi-Task Weak Supervision
Alexander Ratner, Braden Hancock, Jared Dunnmon, Frederic Sala, Shreyash Pandey, Christopher Ré
TL;DR
Weak supervision is cheaper than hand labeling but difficult to combine when sources have unknown accuracies, correlations, and different task granularities. MeTaL models sources as labeling related sub-tasks and uses matrix completion to recover their accuracies without labeled data, improving end-model accuracy across three fine-grained classification problems.
Problem
Weak supervision must combine sources with unknown accuracies, correlations, and different task granularities without ground-truth labeled data.
Method
MeTaL models weak supervision sources as labeling related sub-tasks and recovers their accuracies with a matrix completion-style estimator using task and dependency structures.
Results
Across three fine-grained classification problems, MeTaL improved accuracy by 20.2 points over traditional supervised learning, 6.8 points over majority vote, and 4.1 points over Data Programming.
Takeaways & Limitations
The framework provides higher-quality training supervision for end models while supporting diverse sources with multiple granularities and correlations.
Takeaways & Limitations
The class-conditional noisy-label model learns one accuracy parameter per emitted label and assumes incorrect labels are uniform over true labels.
Abstract
from arXiv · showhide
As machine learning models continue to increase in complexity, collecting large hand-labeled training sets has become one of the biggest roadblocks in practice. Instead, weaker forms of supervision that provide noisier but cheaper labels are often used. However, these weak supervision sources have diverse and unknown accuracies, may output correlated labels, and may label different tasks or apply at different levels of granularity. We propose a framework for integrating and modeling such weak supervision sources by viewing them as labeling different related sub-tasks of a problem, which we refer to as the multi-task weak supervision setting. We show that by solving a matrix completion-style problem, we can recover the accuracies of these multi-task sources given their dependency structure, but without any labeled data, leading to higher-quality supervision for training an end model. Theoretically, we show that the generalization error of models trained with this approach improves with the number of unlabeled data points, and characterize the scaling with respect to the task and dependency structures. On three fine-grained classification problems, we show that our approach leads to average gains of 20.2 points in accuracy over a traditional supervised approach, 6.8 points over a majority vote baseline, and 4.1 points over a previously proposed weak supervision method that models tasks separately.
1 Introduction
MeTaL addresses the difficulty of combining weak supervision sources with unknown accuracies, correlations, and granularities. It models sources as labeling related sub-tasks and recovers their accuracies without labeled data.
- Hand-labeled data is prohibitively slow, expensive, and static for complex modeling problems, motivating cheaper but noisier weak supervision.
- The central challenge is estimating unknown accuracies when sources are correlated and label different tasks or granularities.
- MeTaL treats each weak supervision source as labeling a related sub-task and exploits task relationships to obtain additional agreement and disagreement signals.
- A matrix completion-style algorithm recovers source accuracies from observed source behavior and dependency structure without labeled data.
- 20.2 points in accuracy over traditional supervised learning, 6.8 points over majority vote, and 4.1 points over Data Programming were achieved across three fine-grained classification problems.
2 Related Work
MeTaL extends weak supervision and crowdsourcing ideas to multi-task, multi-granular, and correlated sources. It also targets scalable theory and settings with few sources labeling many items.
- Weak Supervision: Prior weak supervision methods use generative or other models but do not handle multi-granularity or multi-task supervision adequately.
- Crowdsourcing: The framework relates to crowdsourcing but addresses correlated sources, multi-task supervision, and few labelers labeling many items.
- Multi-Task Learning: MeTaL focuses on generating supervision for multi-task models rather than on the particular end-model architecture.
- Related Methods: The work combines matrix structure estimation and concentration bounds to estimate classifier accuracies under structural constraints.
3 Programming Machine Learning with Weak Supervision
Programmatic weak supervision applies noisy labeling sources to unlabeled data, combines overlapping and conflicting outputs with a label model, and trains an end model. MeTaL formulates diverse sources as labeling related sub-tasks.
- Programmatic weak supervision applies user-provided sources to unlabeled data, combines their noisy labels, and uses the resulting labels to train an end model.
- MeTaL formulates each weak supervision source as labeling one or more related sub-tasks, including tasks at different granularities.
- In a fine-grained NER example, sources range from coarse PERSON-versus-ORGANIZATION labels to fine-grained entity types with unknown accuracies.
- Sources are arbitrary black-box functions that may label or abstain on selected tasks, while the framework estimates their accuracies and combines their outputs.
- The source dependency graph represents conditional dependencies among source outputs given the latent task labels.
4 Modeling Multi-Task Weak Supervision
MeTaL models weak supervision sources as labeling related sub-tasks and recovers their unknown accuracies without ground-truth labels. Its matrix completion-style estimator leverages task and dependency structure, with theoretical and empirical results covering scalability, generalization, and diverse source behaviors.
- Accuracy Estimation: The framework recovers unknown source accuracies without labeled data using observed agreements, disagreements, and the dependency graph Gsource.Its matrix completion-style objective estimates label-model parameters from observable covariance structure and graph-induced sparsity.
- Accuracy Estimation: Identifiability conditions establish when the source accuracies have a unique recoverable solution, while Algorithm 1 estimates them from unlabeled data.The approach assumes a triangulated dependency graph with singleton separator sets in the main analysis, while describing extensions for non-singleton separators.
- Multi-Task Formulation: MeTaL formulates each weak supervision source as labeling a subset of related classification tasks governed by a task structure.The task structure defines feasible combinations of task labels, including hierarchical subsumption relationships and inapplicable labels.
- Theoretical Analysis: Theoretical analysis connects label-model estimation error to end-model generalization error, which scales as n^-1/2 with n unlabeled data points.This matches the asymptotic scaling of traditionally supervised learning methods with respect to unlabeled rather than labeled data.
- Extensions: MeTaL supports abstaining and unipolar sources through coverage extensions and class-conditional accuracies, improving accuracy by 2.8 points for unipolar modeling.Class-conditional parameters allow different accuracies for different emitted labels and capture sources that emit one class or abstain.
5 Experiments
Experiments evaluate MeTaL on three fine-grained classification problems using diverse weak supervision sources and compare it with supervised, voting, and data-programming baselines. MeTaL improves accuracy across these comparisons, while ablations examine unipolar correction, joint task modeling, and end-model generalization.
- Datasets: The evaluation covers entity classification, relation classification, and document classification with weak supervision available at coarse and fine granularities.The datasets contain 3k–63k unlabeled training examples and 200–350 labeled development examples.
- End Model Protocol: The end model uses shared bidirectional LSTM and linear layers with a separate task head for each task.Hyperparameters were selected for each application and then fixed.
- Core Validation: MeTaL outperforms traditional hand-labeled supervision by 20.2 points, hierarchical majority vote by 6.8 points, and data programming by 4.1 points in accuracy.The comparisons use the same end-model architecture and evaluate the broad fine-grained classification task.
- Ablations: 2.8 points of average accuracy come from modeling unipolar sources in MeTaL.The authors identify unipolar sources as especially common for fine-grained tasks.
- Ablations: 1.3 points of average accuracy come from jointly modeling tasks rather than estimating source accuracies separately for each task.This ablation measures the empirical impact of the proposed joint multi-task modeling.
- End Model Generalization: The end model improves over direct label-model predictions by 3.4 points on average, with the largest gain reaching 7 points on the dataset with 63k unlabeled examples.The result supports generalization beyond the provided weak supervision and scaling with additional unlabeled data.
6 Conclusion
The paper concludes that MeTaL integrates diverse multi-task weak supervision through matrix completion-style accuracy recovery and theoretical scaling analysis. Its experiments show empirical gains, while future work targets learning task relationships and broader settings with limited labeled data.
- Conclusion: MeTaL integrates weak supervision sources with different granularities, accuracies, and correlations.The framework treats sources as labeling related sub-tasks.
- Conclusion: A scalable matrix completion-style algorithm recovers unknown source accuracies, and theoretical bounds characterize scaling with unlabeled data.The problem setup uses multi-task labels, source outputs, coverage sets, and dependency structures.
- Future Work: Future work aims to learn the task relationship structure and address a broader range of settings where labeled training data is a bottleneck.The conclusion identifies these as extensions of the presented framework.
- Problem Setup: The label model maps source labels to probabilities over feasible task-label vectors for supervising an end model.Sources may output noisy, incomplete, overlapping, or conflicting labels.
A.3 Our Approach: Modeling Multi-Task Sources
The approach estimates a latent label model from unlabeled source outputs by combining graphical-model structure with generalized covariance analysis. A matrix completion-style formulation recovers source-related parameters and enables probabilistic training labels.
- The method models weak supervision sources and the latent variable Y using their conditional independence structure Gsource.
- Generalized covariance analysis separates observable clique information from unobserved parameters linked to the label model.
- The sparsity pattern of the inverse covariance enables block-matrix inversion and a matrix completion-style objective.
- After estimating the label-model parameters, the method recovers class balance and computes probabilistic training labels Pµ(Y|λ).
- The initial exposition focuses on junction trees with singleton separator sets, while extensions address non-singleton separators and rank-one reductions.
A.3.1 Defining a Multi-Task Source Model
MeTaL represents multi-task weak supervision with a chordal source dependency graph and augmented clique statistics. Known inverse-covariance sparsity converts estimation into structured matrix completion, with simplifying assumptions enabling rank-one recovery.
- Gsource is a user-provided pairwise-interaction graph over Y and source outputs, with chordal augmentation used when necessary.
- Indicator statistics over graph cliques form sufficient statistics µ whose expectation is estimated without observing ground-truth labels.
- Observable cliques exclude Y, while separator-set cliques capture the junction-tree structure used to partition the generalized covariance matrix.
- Under singleton separators, covariance decomposition yields a rank-(r −1) completion form that can be represented as a rank-one term after simplifying assumptions.
- Inverse generalized covariance zeros are determined by whether clique pairs belong to the same maximal clique, providing the structural constraints for completion.
- Uniform conditional probability across incorrect labels avoids modeling class-wise errors and reduces the recovery problem to one parameter per observable element.
A.4 Example: Hierarchical Multi-Task Supervision
Hierarchical multi-task supervision organizes labels from coarse to fine levels, with parent tasks constraining child-task applicability. A source’s lowest labeled task determines its full hierarchical output.
- A task hierarchy connects higher-level general labels to lower-level specific labels through directed parent-child constraints.
- In the example, Y1 distinguishes PERSON from BUILDING, while Y2 and Y3 refine the applicable branch into DOCTOR or HOSPITAL labels.
- Inapplicable descendant tasks receive the value N/A when parent-task assignments make them irrelevant.
- A descendant label can determine the other hierarchy labels, such as DOCTOR implying PERSON and excluding BUILDING.
- Each source can be viewed as labeling one task because its lowest non-zero, non-N/A label determines the entire output vector.
- Modeling local accuracies for covered non-N/A tasks can reduce hierarchical supervision models to rank-one form.
B Theoretical Results
The theory characterizes identifiability and estimation error through the inverse source-dependency structure, covariance conditioning, and unlabeled sample size. It also connects label-model estimation to end-model generalization under explicit assumptions.
- Identifiability: The rank-one parameters are identifiable up to sign when the inverse-graph linear system has a solution.
- Identifiability: An odd cycle in every connected component of the inverse augmented graph makes the completion system invertible.
- Identifiability: One known parameter sign per connected component resolves the remaining sign ambiguity.
- Estimation error: The estimation bound depends on graph structure, including bottlenecks measured by a Cheeger-like quantity and conditioning through the smallest eigenvalue of the observed matrix.
- Estimation error: For conditionally independent sources, the relevant structural quantity scales as 1/√(m −2), reducing the covariance-estimation cost from m^2 to m in the bound.
- Generalization: The generalization analysis assumes an optimal label model matches the source-label distribution and that source labels are sufficient for the target label.
- Generalization: End-model training minimizes noise-aware loss over unlabeled data, with empirical optimization error represented by a decreasing function γ(n).
B.4 Proof of Theorem 2
The proof bounds estimation error by combining concentration of observed statistics with successive perturbation analyses. It then propagates these bounds through matrix inversion and the recovery of the model parameters.
- Concentration bound: The proof starts by bounding the error in the empirically estimated observable covariance matrix using matrix concentration.The observed covariance is constructed from n unlabeled samples, and its estimation error is treated as a perturbation.
- Perturbation analysis: Successive perturbation bounds control the effects of covariance estimation error on matrix inverses and intermediate quantities.The analysis introduces additive perturbations Δ and a relative perturbation δ, then bounds their norms through blockwise and Woodbury inversion steps.
- Intermediate bounds: The proof bounds intermediate normalization and quadratic-form terms using spectral constraints on the observable and source covariance matrices.These bounds rely on properties such as Σ_O ⪰ 0, Σ_S < 1, and lower bounds on relevant normalization constants.
- Final recovery: The resulting error bounds are combined to control the recovered parameter vector from the estimated covariance and expected statistics.The final steps incorporate the error from estimating observable expectations and algebraically expand the rank-one estimate into the full parameter vector.
- Final recovery: The rank-one estimate is expanded so that its entries correspond to probabilities of correct or incorrect source-clique outputs.Weight tying copies the relevant values across associated entries in the full parameter representation.
C.3 Dataset Statistics
The experiments use three fine-grained classification datasets with distinct task structures and dataset-specific advantages. Training data is treated as unlabeled, while development and test data retain ground-truth labels.
- Dataset statistics: Development and test examples have ground-truth labels, whereas training examples are treated as unlabeled for the weak-supervision experiments.
- Dataset statistics: OntoNotes contains over 63k relevant data points, enabling empirical analysis of performance scaling with unlabeled data.
- Dataset statistics: OpenI is a real-world medical-triage dataset whose supervision sources were created with direct domain-expert input.Its weak supervision sources naturally occur at multiple levels of granularity.
- Dataset statistics: Three fine-grained classification tasks are evaluated, with dataset splits and sub-task structures summarized in Table 4.The datasets cover named entity recognition, relation extraction, and medical document classification.
C.4 Task Accuracies
Table 5 reports the individual task accuracies of MeTaL and majority vote on the test sets.
- Task accuracies: Table 5 compares MeTaL and majority vote using individual task accuracies computed from test-set predictions.Ties are broken randomly when evaluating each model.
C.5 Ablation Study: Unipolar Correction and Joint Modeling
The ablation separates the gains from unipolar correction and joint multi-task accuracy modeling relative to data programming. Together, these components produce a 4.1-point accuracy gain over the data programming baseline.
- Unipolar correction: 2.8 accuracy points come from applying the unipolar correction to the data programming baseline.
- Joint modeling: 1.3 additional accuracy points come from jointly modeling accuracies across multiple tasks within MeTaL.
- Aggregate effect: 4.1 accuracy points is the aggregate gain of MeTaL over the data programming baseline.Table 6 reports micro accuracy averaged over 10 trials with 95% confidence intervals.