Source-linked AI summary
Neural Open Information Extraction
Lei Cui, Furu Wei, Ming Zhou
TL;DR
Conventional Open IE systems depend on hand-crafted patterns and other NLP tools, which can propagate errors. This paper trains an encoder-decoder neural Open IE model on high-confidence bootstrapped tuples and finds significantly better precision and recall than other systems, with comparable extraction speed. The approach is scoped to binary extractions, while more complex tuple forms remain future work.
Problem
Conventional Open IE systems rely on hand-crafted patterns from other NLP tools, creating cascading errors that can produce incorrect tuples.
Method
The neural Open IE model uses an encoder-decoder framework trained on highly confident binary extractions bootstrapped from a state-of-the-art Open IE system.
Results
The neural Open IE system performs significantly better in precision and recall than other systems while reducing dependence on other NLP tools.
Takeaways & Limitations
The neural approach can generate high-quality tuples without hand-crafted patterns from other NLP tools and achieves comparable computational efficiency.
Takeaways & Limitations
The study considers binary extractions, leaving n-ary and nested extractions for future research.
Abstract
from arXiv · showhide
Conventional Open Information Extraction (Open IE) systems are usually built on hand-crafted patterns from other NLP tools such as syntactic parsing, yet they face problems of error propagation. In this paper, we propose a neural Open IE approach with an encoder-decoder framework. Distinct from existing methods, the neural Open IE approach learns highly confident arguments and relation tuples bootstrapped from a state-of-the-art Open IE system. An empirical study on a large benchmark dataset shows that the neural Open IE system significantly outperforms several baselines, while maintaining comparable computational efficiency.
1 Introduction
Open IE extracts arguments and relation phrases without a predefined ontology, but conventional systems often rely on hand-crafted patterns that propagate errors. The paper proposes a neural encoder-decoder approach trained from high-confidence bootstrapped tuples and reports stronger benchmark performance with reduced tool dependence.
- Motivation: Open IE generates triples or n-ary propositions by extracting both arguments and relation phrases without a predefined ontology schema.Such structured representations support applications including knowledge base construction, question answering, and text comprehension.
- Motivation: Conventional Open IE systems built on hand-crafted patterns from syntactic parsing suffer cascading and compounded errors across processing stages.The paper identifies reducing incorrect tuple extraction as a central motivation.
- Approach: The proposed neural Open IE system casts extraction as sequence-to-sequence generation with an encoder-decoder framework.The input is a sentence, while the output is a tuple sequence marked with argument and relation placeholders.
- Approach: Training pairs are bootstrapped from highly confident tuples produced by a state-of-the-art Open IE system.This supplies training examples for learning tuple generation directly.
- Results: Experiments on a large benchmark dataset report significantly better precision and recall than other systems while reducing dependence on other NLP tools.The paper also releases a large set of bootstrapped training examples and conducts comprehensive comparisons.
2 Methodology
The methodology formulates Open IE as sequence generation from sentences to tuple sequences, using recurrent encoder-decoder networks with attention and copying. The setup focuses on binary extractions whose arguments and relations are drawn from input spans.
- Problem Definition: The task models a sentence X and tuple sequence Y, decomposing P(Y|X) into autoregressive conditional probabilities.X is the input word sequence and Y is the extracted tuple sequence.
- Problem Definition: The implementation considers binary extractions and requires argument and relation phrases to be sub-spans of the input sentence.Consequently, the output vocabulary consists of the input vocabulary plus placeholder symbols.
- Encoder-Decoder Architecture: A recurrent encoder-decoder maps a variable-length input sequence to a compressed representation and generates a variable-length output sequence.Both encoder and decoder use recurrent neural networks; the described encoder and decoder are stacked LSTMs.
- Encoder-Decoder Architecture: The decoder uses its hidden state and context vector to calculate output-word probabilities and select the highest-probability word.The encoder uses a three-layer stacked LSTM to produce hidden representations for the input.
- Attention: Attention computes a context vector from encoder states, while joint optimization maximizes the output sequence log probability conditioned on the input.The alignment model scores how well input positions match output positions.
- Copying Mechanism: The copying mechanism combines sequence generation with attention-based copying because output words largely come directly from the input vocabulary.This design addresses the task’s need to reproduce input spans as tuple arguments and relations.
3 Experiments
Experiments train the neural Open IE model from high-confidence OPENIE4 tuples and evaluate it against established systems on a large benchmark. The neural system achieves the best precision-recall performance while remaining computationally comparable to conventional approaches.
- Evaluation: The test benchmark contains 3,200 sentences with 10,359 extractions and compares the neural system with OLLIE, ClausIE, Stanford OPENIE, PropS, and OPENIE4.Evaluation uses precision and recall.
- Results: The neural Open IE system performs best on the precision-recall curve, and its top-5 outputs achieve the best AUC score of 0.473.The comparison uses the evaluation script from Stanovsky and Dagan (2016).
- Results: Only 11.4% of neural Open IE extractions agree with OPENIE4 extractions, while the neural system still achieves a higher AUC than OPENIE4.The authors attribute this difference to learning arguments and relations across many highly confident training instances.
- Computational Cost: The neural approach takes 172s on the 3,200-sentence test set using an M60 GPU, compared with about 160s for the most efficient conventional system, OLLIE.The reported running times are therefore comparable under the stated hardware settings.
4 Related Work
Related Open IE systems developed from sequential labeling, syntactic and lexical constraints, contextual modeling, and rule-based extraction, but conventional systems rely heavily on hand-crafted patterns. This work applies an end-to-end LSTM sequence-to-sequence approach to binary Open IE, bypassing those patterns.
- Earlier Systems: TEXTRUNNER introduced Open IE as sequential labeling, while REVERB added syntactic and lexical constraints and more than doubled the area under its precision-recall curve.R2A2 then added an argument identifier to improve argument extraction.
- Earlier Systems: OLLIE expanded extraction to noun- and adjective-mediated relations and incorporated contextual information to improve extraction precision.Earlier generations primarily focused on verb-mediated relations.
- Later Systems: Later systems addressed limitations of binary, verb-mediated extraction by incorporating attributes, noun-mediated relations, numerical sentences, and conjunction splitting.SRLIE, OPENIE4, and OPENIE5 represent successive extensions of extraction coverage.
- This Work: The paper uses an LSTM-based sequence-to-sequence approach for binary Open IE as an end-to-end neural alternative to hand-crafted patterns.The authors present this as the first end-to-end neural treatment of the Open IE task and associate it with reduced error propagation.
5 Conclusion and Future Work
The paper concludes that its neural Open IE approach produces high-quality tuples without hand-crafted patterns, while identifying more complex extraction types as future work.
- The encoder-decoder model is trained on highly confident binary extractions bootstrapped from a state-of-the-art Open IE system.This training strategy enables the model to generate high-quality tuples without hand-crafted patterns from other NLP tools.
- Experiments report very promising results on a large benchmark dataset.
- Future work will extend the neural approach to n-ary and nested extractions.The paper also mentions convolutional sequence-to-sequence and transformer models as possible avenues for improving performance.