Source-linked AI summary
Training Classifiers with Natural Language Explanations
Braden Hancock, Paroma Varma, Stephanie Wang, Martin Bringmann, Percy Liang, Christopher Ré
TL;DR
BabbleLabble addresses the limited information in conventional labels by having annotators provide natural-language explanations. It parses those explanations into labeling functions that generate noisy labels for unlabeled data, and users achieve comparable F1 scores up to 100× faster across three relation extraction tasks. The results also show that a simple rule-based parser suffices in this weak-supervision setting.
Problem
Conventional annotation provides only one bit per binary-classification example, motivating methods that extract more information from each annotation.
Method
BabbleLabble parses natural-language explanations into labeling functions, executes them on unlabeled data, and uses the resulting weakly supervised training set to train a classifier.
Results
Users achieved comparable F1 scores up to 100× faster with explanations than with individual labels across three relation extraction tasks.
Takeaways & Limitations
A simple rule-based semantic parser can support this framework because filtering, nearby logical forms, and noisy-label combination make imperfect functions usable.
Takeaways & Limitations
Once enough labeled examples are available, traditional supervision dominates because ground-truth labels are preferable to noisy labeling-function outputs; the efficiency benefit targets settings with more unlabeled than labeled data.
Abstract
from arXiv · showhide
Training accurate classifiers requires many labels, but each label provides only limited information (one bit for binary classification). In this work, we propose BabbleLabble, a framework for training classifiers in which an annotator provides a natural language explanation for each labeling decision. A semantic parser converts these explanations into programmatic labeling functions that generate noisy labels for an arbitrary amount of unlabeled data, which is used to train a classifier. On three relation extraction tasks, we find that users are able to train classifiers with comparable F1 scores from 5-100$\times$ faster by providing explanations instead of just labels. Furthermore, given the inherent imperfection of labeling functions, we find that a simple rule-based semantic parser suffices.
1 Introduction
BabbleLabble increases the information conveyed by each annotation by pairing labels with natural-language explanations that become labeling functions over unlabeled data. Across relation extraction tasks, this approach enables comparable classifier performance substantially faster than individual labeling, while a simple rule-based parser is sufficient.
- Motivation: Standard labeling provides only one bit per example, motivating richer supervision from annotators who have already read and understood each example.The framework addresses how to obtain more information from the same annotation effort.
- Approach: Natural-language explanations capture supervision that cannot be reduced easily to highlighting input spans, including absent words or combinations of words.The approach supports explanations involving conditions such as the absence of a word or the presence of at least two words.
- Approach: BabbleLabble parses natural-language explanations into labeling functions that label many unlabeled examples for classifier training.The explanations represent heuristic mappings from examples to labels, producing a large weakly supervised dataset.
- Results: A simple rule-based semantic parser achieved nearly identical downstream performance to a perfect parser in the weak-supervision framework.The framework filters incorrect labeling functions and combines noisy supervision, making exact semantic parsing unnecessary in the reported setting.
- Results: Using labeling functions rather than logical-form features improved F1 by 9.5 points, a 26% relative improvement, and F1 scaled with available unlabeled data.The comparison concerns using functions to label a larger training set versus feeding explanation-derived features directly to a classifier.
- Results: Users trained classifiers with comparable F1 scores up to 100× faster using explanations instead of individual labels across three relation extraction tasks.The evaluation covered spouse extraction, disease-causing chemical extraction, and protein-kinase interaction extraction.
2 The BabbleLabble Framework
BabbleLabble converts natural-language explanations into labeling functions, filters and aggregates their noisy labels over unlabeled data, then trains a discriminative classifier. Its parser and aggregation pipeline are designed to handle imperfect, overlapping explanations without requiring ground-truth labels for filtering.
- Framework overview: The framework parses natural-language explanations into logical forms representing labeling functions, which are applied to many unlabeled examples.The parser seeks coverage over potentially useful functions rather than a single correct parse.
- Semantic parser: The rule-based parser uses lexical, unary, and compositional grammar rules, can ignore unrecognized tokens, and uses a domain-independent predicate grammar.The grammar can be extended with task-specific primitives for other settings.
- Filter bank: Candidate labeling functions are filtered using example consistency, coverage behavior, redundancy, correlation, and specificity criteria.The filter bank removes inconsistent functions without additional labels and keeps only the most specific surviving function from each explanation.
- Filter bank: Over three tasks, the filter bank removes over 95% of incorrect parses, while remaining incorrect parses have average end-task accuracy within 2.5 points of corresponding correct parses.
- Label aggregator: The label aggregator combines conflicting labeling-function outputs using a factor graph that models labeling propensity and accuracy rather than simple majority vote.It produces probabilistic labels from the observed label matrix and latent true labels.
- Discriminative model: The resulting noisy training set is used to train an arbitrary discriminative model, allowing the classifier to learn informative features absent from the labeling functions.The reported experiments use logistic regression as the discriminative model.
3 Experimental Setup
BabbleLabble is evaluated on three relation extraction tasks using datasets from news, biomedical, and real-world Parkinson’s disease research. The experiments compare explanation-based supervision with traditional labels and use standardized implementation and evaluation procedures.
- Tasks: The evaluation covers Spouse, Disease, and Protein relation extraction tasks.Each task predicts whether two entities participate in the relationship of interest.
- Evaluation: BabbleLabble achieves the same F1 as traditional supervision while using fewer inputs by factors ranging from over 5 for Protein to over 100 for Spouse.The comparison uses 30 explanations versus specified numbers of individually labeled examples.
- Datasets: The Spouse dataset uses news sentences with highlighted names and majority-vote labels from three annotators.Thirty explanations were randomly sampled from 200 explanations generated by 10 graduate students.
- Datasets: The Disease dataset uses PubMed abstracts labeled for whether a chemical causes a disease.Explanations were produced by translating existing Python labeling functions into natural language.
- Datasets: The Protein task targets protein–kinase interactions related to Parkinson’s disease with domain-expert annotators.Annotators had biological expertise but minimal programming experience.
- Implementation: Parsing 360 explanations takes approximately two seconds on a single core, and reported F1 scores average 40 random-seed runs.Hyperparameters were selected by random search over 30 configurations on a held-out development set.
4 Experimental Results
The experiments show that explanations provide high-bandwidth supervision, that imperfect parses can remain useful, and that using logical forms as labeling functions outperforms treating them as classifier features. Performance benefits depend on abundant unlabeled data and do not persist once enough ground-truth labels are available.
- 4.1 High Bandwidth Supervision: BabbleLabble reaches a given F1 with far fewer user inputs than traditional supervision, by as much as 100 times on Spouse.Thirty explanations took about as long to collect as 60 labels, while explanations generate many noisy labels from unlabeled examples.
- 4.1 High Bandwidth Supervision: Traditional supervision dominates once labeled examples are sufficiently numerous because ground-truth labels are preferable to noisy labeling-function outputs.BabbleLabble is most efficient when unlabeled data substantially exceeds labeled data.
- 4.1 High Bandwidth Supervision: Among incorrect parses, 4% arise from unsupported concepts, 2% from human errors, and the remainder from unrecognized paraphrases.The examples include unsupported references such as “the subject of the sentence” and paraphrases outside the parser’s supported wording.
- 4.1 High Bandwidth Supervision: The filter bank removes over 95% of incorrect labeling functions, leaving final sets that are 86% correct on average.Filtering uses semantic consistency with the originating example and pragmatic checks on degenerate or duplicate behavior.
- 4.2 Utility of Incorrect Parses: Nearby labeling functions with one predicate changed improve test accuracy over perfect-parser starting points 57% of the time.This indicates that user-described signals can be useful starting points without being optimal labeling functions.
- 4.3 Using LFs as Functions or Features: Data programming exceeds feature-based use of logical forms by 9.5 F1 points with the rule-based parser and 4.5 points with a perfect parser.The comparison evaluates explanations as labeling functions versus classifier features.
- 4.3 Using LFs as Functions or Features: Data programming performance improves as unlabeled examples increase even when the number of labeling functions remains constant.The discriminative model also captures correlated patterns absent from the explanations, producing a 4.3 F1-point improvement over direct labeling-function predictions.
5 Related Work and Discussion
The paper connects natural-language supervision with semantic parsing and weak supervision. Its broader discussion frames labels as a low-bandwidth communication channel and positions natural language explanations as a richer source of training signal.
- Natural-Language Supervision: Related work converts natural language into classifier features or concepts, whereas this paper converts explanations into labeling functions.The distinction places BabbleLabble between natural-language learning and weak-supervision approaches.
- Semantic Parsing: BabbleLabble reverses a common weak-supervision direction by obtaining weak supervision signal from semantic parsers.Prior work often learns semantic parsers from weak supervision.
- Weak Supervision: Distant supervision and multi-instance learning use knowledge bases to probabilistically impute training sets.Other weak-supervision methods aggregate multiple noisy sources with generative models.
- Discussion: The paper describes labels as low-bandwidth communication and natural language as a higher-bandwidth channel between annotators and learning algorithms.In relation extraction, one explanation can be “worth” 100 labels, while extension to other tasks remains an open direction.
Reproducibility
The paper reports that its code, data, and experiments are available through CodaLab, with refactored code and interactive tutorials available on GitHub.
- Released Materials: Code, data, and experiments are available on the CodaLab platform.The paper provides a CodaLab worksheet link for the released materials.
- Released Materials: Refactored code includes simplified dependencies, performance and speed improvements, and interactive tutorials on GitHub.The release is presented as an additional implementation resource.
A Predicate Examples
The rule-based semantic parser uses predicates for logical combinations, comparisons, text properties, entity relations, positional constraints, and list or set operations.
- Logical predicates express conjunction, disjunction, negation, and quantifier-like conditions such as any, all, and none.
- Comparison predicates encode equality, inequality, and ordering relationships between variables.
- Text predicates detect casing, capitalization, prefixes, suffixes, or substrings.
- Entity and numeric predicates describe people, locations, dates, numbers, and organizations using sentence-based relations.
- List, set, counting, containment, intersection, mapping, filtering, and alias predicates express structured relations among words or predefined lists.
- Distance and positional predicates constrain where words or entities occur relative to one another.
B Sample Explanations
The paper presents sample natural-language explanations that users provide for labeling decisions across the evaluated tasks.
- Users provide natural-language explanations for each task’s labeling decisions.
Spouse
The spouse examples show explanations expressed as rules over entity positions, nearby words, and relation-specific cues. They include both positive and negative labeling conditions.
- Users refer to the first and second people in a sentence as X and Y.
- A positive spouse label can require “and” between X and Y and “marriage” immediately after person1.
- Another positive rule identifies person Y when it is preceded by “beau”.
- Negative rules reject labels when spouse-related words are absent or when more than two people and an actor or actress cue appear in specified positions.
Disease
The disease examples express relation labels through lexical order, trigger phrases, and negation windows around the disease and chemical entities.
- A positive disease relation can place the disease immediately after the chemical while including “induc” or “assoc” in the chemical name.
- Another positive rule requires a word containing “develop” before the chemical and “following” between the disease and chemical.
- Positive labels can also use causal phrases such as “induced by”, “caused by”, or “due to” between the chemical and disease.
- A negative label is assigned when “none”, “not”, or “no” occurs within 30 characters to the disease’s left.
Protein
The protein labeling rules identify positive relations using nearby amino-acid names or connective words, while excluding molecular-biology terms and certain punctuation patterns.
- A relation is labeled true when “Ser” or “Tyr” occurs within 10 characters of the protein.
- A relation is labeled true when “by” or “with” links the protein and kinase without nearby negation, with fewer than 10 intervening words.
- A relation is labeled false when the sentence contains “mRNA”, “DNA”, or “RNA”.
- A relation is labeled false when two commas separate the protein and kinase with fewer than 30 characters between them.