Source-linked AI summary
Robust Distant Supervision Relation Extraction via Deep Reinforcement Learning
Pengda Qin, Weiran Xu, William Yang Wang
TL;DR
Distant supervision alleviates annotation costs but produces noisy training samples, and existing selection or soft-attention methods leave false positives insufficiently addressed. The paper uses model-independent deep reinforcement learning to make hard selection decisions, redistribute false positives into negative examples, and improve neural relation extractors across experiments.
Problem
Distant supervision produces noisy instances, while one-best selection and soft attention remain suboptimal for handling false positives.
Method
A model-independent deep reinforcement learning agent learns whether to retain or remove each candidate instance and reconstructs a cleaned training set without supervised information.
Results
The method brings consistent performance gains across various deep neural network relation extraction models and achieves strong performance on the New York Times dataset.
Takeaways & Limitations
False positives are relocated into negative examples, while the plug-and-play framework can be applied across relation extraction pipelines.
Takeaways & Limitations
A single agent for all relation types performed worse than using one agent per relation.
Abstract
from arXiv · showhide
Distant supervision has become the standard method for relation extraction. However, even though it is an efficient method, it does not come at no cost---The resulted distantly-supervised training samples are often very noisy. To combat the noise, most of the recent state-of-the-art approaches focus on selecting one-best sentence or calculating soft attention weights over the set of the sentences of one specific entity pair. However, these methods are suboptimal, and the false positive problem is still a key stumbling bottleneck for the performance. We argue that those incorrectly-labeled candidate sentences must be treated with a hard decision, rather than being dealt with soft attention weights. To do this, our paper describes a radical solution---We explore a deep reinforcement learning strategy to generate the false-positive indicator, where we automatically recognize false positives for each relation type without any supervised information. Unlike the removal operation in the previous studies, we redistribute them into the negative examples. The experimental results show that the proposed strategy significantly improves the performance of distant supervision comparing to state-of-the-art systems.
1 Introduction
Relation extraction supports information extraction and downstream NLP applications, but distant supervision introduces noisy training instances. The paper proposes deep reinforcement learning to identify false positives and improve robust relation extraction.
- Relation extraction predicts relations between entities and supports knowledge graphs, structured search, sentiment analysis, question answering, and summarization.
- Distant supervision reduces annotation costs by using knowledge-base entity pairs to select noisy training instances from unlabeled data.
- Prior approaches select one sentence or assign soft attention weights, but may miss valuable instances and inadequately address false positives.
- The proposed framework uses deep reinforcement learning to dynamically recognize false positives during distant supervision.
- The method is model-independent and can be applied to state-of-the-art relation extractors.
- Experiments report performance gains for recently proposed neural relation extractors on the widely used New York Times dataset.
2 Related Work
Early distant supervision methods modeled multiple relations or relied on noise suppression, while later neural methods selected or softly weighted candidate sentences. These strategies still risk false positives or losing valuable training instances.
- Mintz et al. combined dependency paths and feature aggregation for distant supervision but introduced false positives when entity pairs had multiple relations.
- Hoffmann et al. jointly modeled multiple relations, and Surdeanu et al. proposed multi-instance multi-label learning to improve performance.
- Early approaches did not explicitly remove noisy instances, instead relying on the model to suppress noise.
- Neural methods that select one plausible instance per entity pair can miss valuable training instances.
- Attention-based selection assigns soft weights to noisy instances, but the paper argues that false positives require a different treatment.
3 Reinforcement Learning for Distant Supervision
The framework uses a policy-based deep reinforcement learning agent to make sentence-level retain-or-remove decisions for noisy distant-supervision data. Rewards from relation-classifier F1 guide false-positive recognition, after which filtered instances are redistributed into negative examples.
- Agent and task: The agent learns, for each relation type, whether to remove or retain each distantly supervised candidate instance.Its goal is to identify false positives while reconstructing a cleaned training set.
- States: The state combines the current sentence representation with the average representation of sentences removed in earlier states.Word and position embeddings encode sentence information, and the current sentence receives greater weight.
- Rewards: The reward is based on the change in relation-classifier F1 between adjacent epochs, rather than on a binary reward.Positive reward is assigned when F1 improves; continuous reward magnitude reflects the F1 difference, with averaging over recent epochs used to reduce randomness.
- Agent retraining: The method compares removed-instance sets across epochs and assigns rewards to the set associated with the better F1 outcome.The difference between removed parts in successive epochs determines which actions receive positive or negative reinforcement.
- Dataset redistribution: The learned relation-specific agents act as false-positive indicators, and entity pairs whose aligned sentences are all classified as false positives are moved to the negative set.This redistributes rejected positive candidates rather than simply discarding them.
4 Experiments
Experiments evaluate the policy-based reinforcement learning method on the Riedel dataset using held-out evaluation and relation-classifier performance. The method redistributes training instances and improves F1 scores and aggregate PR-curve performance, with statistically significant improvements reported.
- Policy-based agent: The policy-based RL agents redistribute training data by moving detected false-positive samples into the negative set.The experiments test whether the agents can generate relation indicators that produce a more suitable training dataset.
- Dataset and evaluation: The experiments use the Riedel dataset, aligned from Freebase entity pairs and the New York Times corpus, with 2005–2006 for training and 2007 for testing.The dataset contains 52 actual relations plus the NA relation, and evaluation follows the held-out protocol.
- Policy-based agent: Agents are trained for 10 relation types with more than 1,000 positive instances because of randomness on the smaller-scale data.The study compares original, pre-trained-agent, and retrained RL-agent datasets using F1 scores.
- Results: After retraining with policy-based RL, F1 scores improve further, including cases where pre-trained agents perform poorly.The authors interpret these results as evidence that the retraining strategy can steer agents toward better policies.
- Results: The redistributed dataset improves CNN- and PCNN-based relation extractors, while AUC comparisons and t-tests report significant gains over previous studies.The reported t-test p-values are all less than 5e-02.
- False-positive analysis: False-positive rates vary across relations and are concentrated among entity pairs with few sentences, whose percentages for one to five sentences are [55.9%, 32.0%, 3.7%, 4.4%, 0.7%].The analysis also identifies duplicated sentences and entity-pair misalignment as sources of false positives.
5 Conclusion
The paper proposes a policy-based deep reinforcement learning framework that learns to relocate false-positive samples and better use unlabeled data. It is classifier-independent and improves distant supervision relation extraction across strong baselines on the New York Times–Freebase dataset.
- 5 Conclusion: The framework uses deep reinforcement learning to systematically learn a selection and redistribution strategy for false-positive samples.It contrasts with prior approaches that use one instance per entity pair or soft attention weights.
- 5 Conclusion: Figure 5 presents the scale of the removed part for each relation type using relation IDs from Table 2 on the horizontal axis.
- 5 Conclusion: The framework is classifier-independent, making it a plug-and-play technique for relation extraction pipelines.
- 5 Conclusion: The method boosts distant supervision relation extraction across various strong deep learning baselines on the widely used New York Times–Freebase dataset.