Source-linked AI summary
Zero-Shot Relation Extraction via Reading Comprehension
Omer Levy, Minjoon Seo, Eunsol Choi, Luke Zettlemoyer
TL;DR
Relation extraction traditionally requires predefined relation types and training examples, limiting extraction of unseen relations. The paper reduces extraction to reading comprehension using natural-language questions, large-scale distantly supervised data, and answerability modeling. Experiments show strong generalization to question paraphrases and zero-shot extraction of unseen relations, though the zero-shot problem remains far from solved.
Problem
Existing crowdsourced and distantly supervised systems cannot extract relation types that were neither specified nor observed during training.
Method
The paper maps each relation slot to natural-language questions, trains a reading-comprehension model with crowdsourced questions and distant supervision, and defines unseen test relations through questions.
Results
The approach incurs a 4% relative F1 reduction on new question paraphrases and reaches an F1 of 41% for previously unseen relations.
Takeaways & Limitations
Relation extraction can generalize to relations defined on-the-fly in natural language, while zero-shot extraction remains an open challenge for information extraction and machine reading.
Takeaways & Limitations
458 relations had zero questions after verification, and the algorithm assumes a single answer span although less than 5% of the data has multiple answers.
Abstract
from arXiv · showhide
We show that relation extraction can be reduced to answering simple reading comprehension questions, by associating one or more natural-language questions with each relation slot. This reduction has several advantages: we can (1) learn relation-extraction models by extending recent neural reading-comprehension techniques, (2) build very large training sets for those models by combining relation-specific crowd-sourced questions with distant supervision, and even (3) do zero-shot learning by extracting new relation types that are only specified at test-time, for which we have no labeled training examples. Experiments on a Wikipedia slot-filling task demonstrate that the approach can generalize to new questions for known relation types with high accuracy, and that zero-shot generalization to unseen relation types is possible, at lower accuracy levels, setting the bar for future work on this task.
1 Introduction
The paper reframes relation extraction as reading comprehension by expressing relations as natural-language questions, enabling extraction of previously unseen relation types. It combines crowdsourced questions, distant supervision, and answerability modeling to support large-scale and zero-shot learning.
- Relation extraction is reframed as answering natural-language reading-comprehension questions associated with relation slots.A non-null answer to a relation-specific question both asserts the fact and fills its slot value.
- Zero-shot extraction defines a new relation at test time through questions, without labeled examples for that relation.The model is trained on N relation types and receives only slot-defining questions for relation R_N+1 during testing.
- 120 Wikidata relations and roughly 10 crowdsourced questions per relation produced over 30,000,000 question-sentence-answer examples.Distant supervision supplies relation data, while crowdsourcing gathers and verifies relation-defining questions.
- The reading-comprehension model is extended to determine when a question has no answer in the text.This addresses the relation-extraction setting, where the requested answer may be absent.
- 4% relative F1 reduction accompanies generalization to new question paraphrases, while unseen-relation extraction rises from virtually zero to an F1 of 41%.The analysis attributes zero-shot generalization partly to shared typing information and relation-paraphrase detection, while noting many cases remain unsolved.
2 Related Work
The paper positions its zero-shot slot-filling task as harsher than settings using trigger words or partially observed relations. Its approach instead uses natural-language questions to describe canonical relation slots.
- The target setting provides labeled examples for N relation types but only parametrized questions for an unseen relation R_N+1.No manifestation of the test relation is observed during training.
- Unlike trigger-word approaches for event identification, this work uses questions because they better describe slot-filling relations.The comparison is between seed-trigger similarity and question-based descriptions of relation slots.
- Open information extraction avoids relation-specific training data but can treat different phrasings as different relations.This paper instead seeks a canonical slot value independent of the source text’s phrasing.
- Universal-schema methods can predict unseen entities for seen relations, whereas this work targets relations with no observed manifestation during training.The paper characterizes its zero-shot scenario as substantially more challenging.
3 Approach
The approach converts slot-filling relations into parametrized natural-language questions, instantiates them for entities, and applies a trained reading-comprehension model to sentence spans. For a new relation, defining the information need as a question is sufficient to run extraction.
- Slot filling seeks text spans A in a sentence s such that the relation R(e, a) holds, including the empty set when no answer is present.For example, occupation of Steve Jobs yields spans such as businessman, inventor, and industrial designer.
- Schema querification replaces R(e, ?) with a relation-level template q_x, then instantiates it with each relevant entity.The template “What did x do for a living?” becomes an entity-specific question such as “What did Steve Jobs do for a living?”.
- The converted dataset trains a reading-comprehension model to return answer spans within a sentence for a question.The model outputs the set of spans it judges to answer the instantiated question.
- At test time, a new relation is represented by questions, which are instantiated with candidate entities and run over sentences.Each non-null answer a for entity e yields the extracted relation R_N+1(e, a).
- The method presents a natural-language API requiring an information need expressed as a question, training data, and a reading-comprehension model.It is intended to avoid requiring linguistic knowledge or a predefined schema for each new relation.
4 Dataset
The dataset converts relation-slot filling into reading-comprehension examples by combining distant supervision with crowdsourced question templates. Schema-level annotation produces a large, verified dataset while negative examples model unanswerable question–sentence pairs.
- Data construction: Slot-filling examples are converted into reading-comprehension instances containing a relation, question, sentence, and answer set.Distant supervision aligns Wikidata relations with Wikipedia text before queries are converted into natural-language questions.
- Schema querification: Annotators create question templates by matching masked entities and underlined answer spans across example sentences.Templates are verified on additional sentences and retained when answered correctly in a majority of examples.
- Dataset scale: 1,192 high-quality question templates spanning 120 relations yield over 30,000,000 question-sentence-answer examples.The templates are joined with slot-filling data by relation and instantiated with matching entities.
- Negative examples: Negative examples pair a question for one relation with a sentence expressing another relation, producing over 2 million examples.About a third of these negative examples contain good distractors according to the authors’ analysis.
5 Model
The model extends neural reading comprehension to relation extraction, where answers may be absent from the sentence. It predicts answer spans or a null answer using augmented start and end distributions with learned answerability thresholds.
- Task formulation: The system receives a sentence and question and returns either an answer span within the sentence or no answer.This accommodates relation-extraction examples in which the queried relation is not expressed in the text.
- Base model: The approach starts from BiDAF, which encodes sentence and question context and predicts answer-span start and end positions.BiDAF uses recurrent encoders and attention to align the question with the sentence.
- Answerability: A trainable bias appended to both confidence vectors adds a null position, allowing softmax distributions to represent unanswerable questions.The final elements of the augmented vectors encode confidence that the answer has no start or end.
- Decision rule: The model predicts no answer when the null-answer probability exceeds the probability of the best candidate span.This learned bias acts as a threshold on the model’s raw confidence scores.
- Decision rule: A global confidence threshold pmin can additionally classify low-confidence best answers as unanswerable.The authors use this threshold to examine performance across broader answerability settings.
6 Experiments
The experiments evaluate generalization to unseen entities, question templates, and relations. The reading-comprehension approach performs well on unseen entities and templates, while zero-shot extraction of unseen relations remains substantially harder.
- The experiments test generalization to unseen entities, unseen question templates, and unseen relations.These settings isolate whether the approach transfers across entities, paraphrases, and relation types.
- 6.1 Unseen Entities: All model variants generalize well to new entities and texts, while outperforming the RNN Labeler and Miwa & Bansal systems.Performance varies little among KB Relation, NL Relation, Multiple Templates, and Question Ensemble; Single Template performs significantly worse.
- 6.1 Unseen Entities: Only 18% of sampled model errors were pure system errors, with annotation errors accounting for 36% and inaccurate span selection for 42%.The authors estimate a performance ceiling slightly above 90% F1 in this setting.
- 6.2 Unseen Question Templates: Unseen question templates incur roughly a 3.5-point F1 loss relative to previously observed templates.The result indicates that the approach can generalize to new descriptions of known relations.
- 6.3 Unseen Relations: Training on multiple question templates substantially improves unseen-relation performance, with an additional advantage from asking multiple questions at test time.Multiple phrasings help the model learn answer-type paraphrases shared across relations.
7 Analysis
The analysis examines which cues support solving positive and negative examples in unseen-relation extraction. Type-based cues are more common than relation cues, and global type cues are easier for the model to exploit than specific ones.
- Among 100 analyzed examples, 60 contained answers in the sentence and 40 were negative examples.The sample was used to analyze how the method extracts unseen relations.
- 35% of negative examples contained distractors that had the correct answer type but were not the correct answer.The example concerns a sentence mentioning Sarah Palin when the question asks whom John McCain married.
- Type cues accounted for 58% of selected cues, compared with 42% for relation cues; relation-specific cues made up half of all cues.Global cues accounted for one third and verbatim cues for one sixth, although the method exploited these cues only partially.
- The model handled verbatim, global, and specific relation cues with similar accuracy, but detected global type cues much better than specific type cues.The relation-cue comparison showed no clear trend because of the small sample size.
- The authors identify global type detection and relation paraphrase detection as the primary sources of generalization to new relations.Global type detection is learned across many relations, while paraphrase detection likely relies on pretrained word embeddings.
8 Conclusion
The paper reduces relation extraction to reading comprehension, enabling generalization to unseen relations defined on-the-fly in natural language. Zero-shot relation extraction remains far from solved, motivating further work.
- The approach enables relation extraction for unseen relations defined on-the-fly in natural language.
- Zero-shot relation extraction remains an open challenge for information extraction and machine reading.
- The authors make their code and data publicly available to support future research.