Source-linked AI summary
Reinforcement Learning for Relation Classification from Noisy Data
Jun Feng, Minlie Huang, Li Zhao, Yang Yang, Xiaoyan Zhu
TL;DR
Distant supervision makes relation classification scalable but assigns noisy relation labels and does not identify which sentence expresses a relation. The paper jointly trains a reinforcement-learning instance selector and a sentence-level relation classifier, reporting better performance than state-of-the-art baselines on noisy data. It also notes that extending the instance-selection solution to other noisy-data tasks remains future work.
Problem
Distant-supervision methods classify sentence bags rather than individual sentences, cannot map relations to sentences, and suffer from noisy labeling.
Method
The model jointly trains an instance selector using reinforcement learning and a relation classifier that predicts sentence-level relations and provides selection rewards.
Results
The model filters noisy sentences and performs sentence-level relation classification better than state-of-the-art baselines on noisy data.
Takeaways & Limitations
Instance selection with reinforcement learning benefits noisy-data relation classification and supports sentence-level extraction on cleansed data.
Takeaways & Limitations
Generalizing the instance-selection solution to other noisy-data or distant-supervision tasks is left for future work.
Abstract
from arXiv · showhide
Existing relation classification methods that rely on distant supervision assume that a bag of sentences mentioning an entity pair are all describing a relation for the entity pair. Such methods, performing classification at the bag level, cannot identify the mapping between a relation and a sentence, and largely suffers from the noisy labeling problem. In this paper, we propose a novel model for relation classification at the sentence level from noisy data. The model has two modules: an instance selector and a relation classifier. The instance selector chooses high-quality sentences with reinforcement learning and feeds the selected sentences into the relation classifier, and the relation classifier makes sentence level prediction and provides rewards to the instance selector. The two modules are trained jointly to optimize the instance selection and relation classification processes. Experiment results show that our model can deal with the noise of data effectively and obtains better performance for relation classification at the sentence level.
Introduction
Distant supervision and bag-level relation classification leave sentence-level mappings unresolved and remain vulnerable to noisy bags. The paper proposes jointly trained instance selection and relation classification using reinforcement learning.
- Motivation: Distant supervision labels every sentence mentioning an entity pair with the pair's knowledge-base relation, even when the sentence does not express it.The paper illustrates this with a noisy sentence assigned the BornIn relation despite not describing birth.
- Motivation: Bag-level methods cannot map individual relations to sentences and are sensitive to bags whose sentences are all noisy.They retain at least one sentence from each bag, including bags containing no sentence describing the relation.
- Motivation: 53% of 100 investigated sample bags contained no sentences describing the relation, illustrating the prevalence of fully noisy bags.The paper states that such bags decrease relation-classification performance.
- Proposed Model: The proposed model uses an instance selector to choose high-quality sentences before a relation classifier predicts relations at the sentence level.The model is designed to filter an entire bag when all its sentences are labeled incorrectly.
- Proposed Model: Instance selection is formulated as reinforcement learning, using feedback from the relation classifier instead of explicit sentence-level annotations.The selector searches through sentence choices and receives rewards based on the utility of selected sentences as a whole.
Related Work
Prior relation-classification work includes supervised neural and handcrafted-feature methods, but large-scale training requires expensive high-quality labels. Distant supervision expands data while introducing noisy labels, motivating multi-instance and sentence-selection approaches.
- Supervised Methods: Traditional supervised relation-classification methods rely on handcrafted features or automatically generated neural features and require high-quality labeled data.The paper identifies the expense of obtaining large amounts of labeled data as a motivation for distant supervision.
- Neural Methods: Neural relation-classification approaches include convolutional, recursive, and long short-term memory networks, with attention used for heterogeneous contexts.These approaches still require substantial labeled data for training.
- Distant Supervision: Distant supervision assumes that all sentences mentioning the entities in a fact triple describe that triple's relation, creating noisy labels.The paper presents distant supervision as a response to the need for large-scale training data.
- Noise Handling: Multi-instance learning addresses distant-supervision noise by modeling relation classification over bags of sentences mentioning the same entity pair.Sentence-level attention can down-weight incorrect sentences, but these methods remain centered on bag-level prediction.
- Paper's Position: The paper's framework selects correct sentences with reinforcement learning before predicting relations, targeting limitations of prior bag-level methods.Those limitations include missing exact relation-to-sentence mappings and inability to handle entirely noisy bags.
Methodology
The framework jointly trains an instance selector and a sentence-level relation classifier: the selector filters noisy sentences, while classifier feedback guides selection.
- Framework: The framework decomposes relation classification into instance selection and relation classification, using selected sentences to produce sentence-level predictions.The selector chooses training instances from noisy sentence–relation pairs, and the classifier predicts the semantic relation for each sentence and entity pair.
- Instance Selector: The instance selector uses reinforcement learning to decide whether each sentence in a bag should be selected.Its state encodes the current sentence, previously selected sentences, and the entity pair; actions are sampled from a policy function.
- Joint Training: Training is divided into bags so the selector receives feedback after each bag, after which selected sentences are merged into a cleansed dataset.The cleansed dataset trains the relation classifier at the sentence level.
- Instance Selector: Selection receives a delayed terminal reward from the relation classifier based on the utility of the selected sentences.The reward is zero before selection finishes and evaluates the average likelihood assigned to selected sentences; an empty selection uses the average likelihood over training data.
- Relation Classifier: The relation classifier takes one filtered sentence rather than a sentence bag, enabling sentence-level classification on cleansed data.This contrasts with prior models that classify the bag as a whole.
- Joint Training: The two modules are trained jointly, using policy gradients for the selector and gradient descent for the CNN classifier, with slowly updated target networks for stability.The target policy and target CNN parameters are updated by linear interpolation with τ ≪1.
Experiment
The experiments evaluate sentence-level relation classification and the instance selector using manual labels, model comparisons, and selection analyses. CNN+RL improves over baseline models, selected data improves classification, and the selector filters noisy sentences and bags.
- Evaluation settings: The evaluation predicts one relation label per sentence and uses manually labeled sentences, with accuracy and macro-averaged F1 as metrics.The dataset for manual evaluation contains 300 randomly chosen sentences.
- Baselines: The baselines include sentence-level CNN, bag-level CNN+Max, and bag-level CNN+ATT models.CNN+Max selects the most correct sentence in each bag, while CNN+ATT uses sentence-level attention to down weight noisy sentences.
- Relation classification results: CNN+RL outperforms CNN, CNN+Max, and CNN+ATT, while sentence-level models perform better than bag-level models for sentence-level prediction.The results attribute CNN+RL’s advantage over CNN to filtering noisy data by instance selection.
- Selected-data evaluation: Models trained on selected data achieve much better performance than counterparts trained on the original data for both CNN and CNN+ATT.The comparison is reported in Figures 3 and 4 and is interpreted as evidence that the selector filters noisy sentences and distills high-quality sentences.
- Selection accuracy: The instance selector achieves 74% accuracy on 300 sampled training sentences, correctly selecting 45 sentences and rejecting 177 noisy instances.The selector chose 64 sentences and rejected 236 instances in the manual assessment.
- Selection strategy comparison: Reinforcement-learning selection performs much better than greedy selection on held-out evaluation when the number of selected instances is held constant.The relation classifier is kept untouched while replacing RL selection with greedy selection.
- Noisy bag filtering: In a sample of 100 deleted sentence bags, 86% consist entirely of noisy sentences, supporting the selector’s ability to filter such bags.The paper presents this finding as evidence that the instance selector can exclude noisy sentences effectively.
- Selection examples: Examples show CNN+RL selecting correct sentences and filtering bags containing only noisy sentences, where other compared methods fail.Table 2 compares CNN+RL and CNN+Max selection indicators with CNN+ATT attention weights.
Conclusion and Future Work
The proposed reinforcement-learning model filters noisy sentences and improves sentence-level relation classification over state-of-the-art baselines. Its potential generalization to other noisy-data or distant-supervision tasks remains future work.
- The model filters noisy sentences and performs sentence-level relation classification better than state-of-the-art baselines from noisy data.It uses an instance selector and relation classifier within a reinforcement learning framework.
- The instance-selection solution could generalize to other tasks that use noisy data or distant supervision, but the paper leaves this extension for future work.Sentiment classification on noisy data is given as a possible example.