Source-linked AI summary
A Survey on Open Information Extraction
Christina Niklaus, Matthias Cetto, André Freitas, Siegfried Handschuh
TL;DR
Open IE research lacks a consistently defined task and a large, reproducible evaluation basis, despite aiming for domain-independent and scalable extraction. This survey classifies the field’s approaches, reviews how they address contextual and structural challenges, critiques evaluation practices, and identifies future research directions. It concludes that standardized large-scale evaluation, broader language coverage, and attention to inter-proposition relationships remain important open areas.
Problem
Open IE lacks a generally accepted tuple definition and established large-scale gold standard, limiting objective, reproducible cross-system evaluation.
Method
The survey classifies Open IE systems into learning-based, rule-based, clause-based, and inter-proposition approaches, then reviews their evaluation procedures and research directions.
Results
The survey identifies contextual and structural approaches, including methods that capture attribution, nested dependencies, or compact semantically enriched extractions.
Takeaways & Limitations
Future work should pursue large-scale objective evaluation, broader language coverage, and approaches that better represent relationships among propositions.
Takeaways & Limitations
Evaluation remains fragmented across small domain-dependent corpora and nonstandard datasets, preventing clear conclusions about portability across text genres.
Abstract
from arXiv · showhide
We provide a detailed overview of the various approaches that were proposed to date to solve the task of Open Information Extraction. We present the major challenges that such systems face, show the evolution of the suggested approaches over time and depict the specific issues they address. In addition, we provide a critique of the commonly applied evaluation procedures for assessing the performance of Open IE systems and highlight some directions for future work.
1 Introduction
Open IE converts natural-language text into relational tuples without prespecifying target relations, while aiming to operate efficiently across heterogeneous domains. The field’s systems illustrate different extraction choices and increasingly varied outputs.
- Task and goals: Information extraction represents text as relational tuples containing arguments and a semantic relation phrase.Traditional IE targets predefined relations using handcrafted patterns or patterns learned from labeled examples.
- Task and goals: Open IE detects relations automatically in a single corpus pass while minimizing hand-crafted patterns and training data.Systems may require only a small set of hand-tagged seeds or manually defined extraction patterns.
- Core challenges: Domain-independent Open IE favors shallow parsing because deep parsers and broad NER systems can fail across heterogeneous Web genres.The survey specifically identifies POS taggers as preferable to syntactic or dependency parsers for this setting.
- Core challenges: Scalable Open IE requires computational efficiency, motivating shallow linguistic features such as POS tags over parse-tree-derived features.The efficiency requirement is tied to extracting from very large text collections.
- Evolution and examples: TEXTRUNNER first implemented these criteria, but later systems often used small domain-dependent corpora and dependency parsers.These choices weaken the original assumptions of domain independence and efficiency.
2 Open IE Systems
Open IE systems extract relational tuples using learned or hand-crafted patterns, while later approaches broaden relation structure and model sentence context. The surveyed systems target recurring problems including incomplete, uninformative, incoherent, overly specific, and context-insensitive extractions.
- Learning-based and rule-based systems: Open IE approaches use hand-crafted or automatically learned patterns to extract relational tuples consisting of argument phrases and semantic relations.Learning-based systems include TEXTRUNNER, WOE, OLLIE, and ReNoun; rule-based systems include REVERB, KRAKEN, and EXEMPLAR.
- Learning-based Systems: OLLIE bootstraps dependency-parse pattern templates from high-precision REVERB seed tuples and adds attribution and clausal modifiers to represent contextual information.It also extends extraction beyond verb-based relations to relationships mediated by nouns and adjectives.
- Rule-based Systems: REVERB uses POS-based constraints and a lexical constraint to reduce incoherent, uninformative, and overly specific extractions while covering about 85% of verb-based relational phrases.Unlike candidate-pair-first approaches, it identifies relational phrases before determining their arguments.
- Rule-based Systems: KRAKEN and EXEMPLAR extract n-ary relations with hand-crafted dependency-based rules, with KRAKEN gathering the full set of arguments for each relational phrase.KRAKEN produces tuples of arbitrary arity, while EXEMPLAR assigns semantic roles to arguments using Semantic Role Labeling.
- Rule-based Systems: PROPS and PredPatt transform dependency parses into directed graphs that explicitly represent proposition or predicate-argument structure, making proposition extraction more direct.PROPS uses a rule-based converter, while PredPatt adds ARG dependencies between predicate heads and argument heads while preserving original Universal Dependency relations.
- Systems Capturing Inter-Proposition Relationships: OLLIE addresses context-insensitive extraction by representing attribution and clausal modifiers, while MinIE shortens overly specific extractions to core constituents for downstream semantic applications.These developments target the interpretation and usability of propositions in complex assertions and downstream tasks.
3 Evaluation
Open IE evaluation lacks a clear formal tuple specification and an established large-scale gold standard, making objective cross-system comparison difficult. Proposed benchmarks address this gap, but evaluation remains constrained by domain coverage and inconsistent use of recall.
- Evaluation challenges: A missing formal specification of valid relational tuples has prevented an established, large-scale gold-standard corpus for objective comparison.Consequently, systems were predominantly evaluated manually on small-scale datasets.
- Evaluation challenges: Existing evaluations often use small, domain-dependent corpora, limiting conclusions about portability across genres.News, Wikipedia, and Web domains dominate the evaluation sets.
- Evaluation challenges: Many procedures emphasize precision while ignoring recall or substituting proxies such as yield and coverage.This absence of a standard procedure makes performance comparisons difficult to replicate.
- Benchmarking: Stanovsky and Dagan introduced a large gold benchmark corpus to standardize Open IE evaluation around consensual task principles.The principles include assertedness, minimal propositions, and completeness with an open lexicon.
- Scope: Open IE systems commonly focus on verbal predicates, leaving noun- and adjective-mediated relations outside their scope.This limitation conflicts with the task’s goal of extracting relations from heterogeneous corpora rather than only predefined relation classes.
- Benchmarking: RelVis supports quantitative precision, recall, and F2-score evaluation alongside manual qualitative error analysis.Its error analysis distinguishes six common classes of inaccurate extractions.
4 Open Research Questions
Open IE still has unresolved questions in evaluation, language coverage, and extraction normalization. Benchmark adoption remains limited, most systems focus on English, and canonicalization and coreference resolution are largely underexplored.
- Evaluation: Large-scale, objective, reproducible comparison remains limited because most approaches use proprietary datasets over small, domain-dependent corpora.Although benchmark frameworks have been proposed, neither had been widely adopted apart from Cetto et al. (2018).
- Multilinguality: Most Open IE approaches focus on English, leaving portability and transferability to other languages as open research directions.Only a few systems address German, multiple Romance languages, or cross-language patterns.
- Normalization and coreference: Canonicalizing relational phrases and arguments has received little attention, despite potential benefits for textual entailment and knowledge-base population.Coreference resolution is another largely ignored capability that may assist interpretation of extracted propositions.
5 Conclusion
The paper surveys Open IE methods, organizing their evolution and the problems they address, while examining evaluation gaps and identifying future-work directions.
- The survey classifies Open IE systems into learning-based, rule-based, clause-based, and inter-proposition approaches.
- It traces how Open IE approaches evolved over time and which specific problems they tackle.
- It reviews how Open IE performance has been evaluated and identifies gaps in commonly used procedures.
- The paper concludes by identifying directions for future work in Open IE.