Source-linked AI summary
Simple Large-scale Relation Extraction from Unstructured Text
Christos Christodoulopoulos, Arpit Mittal
TL;DR
Knowledge bases rely on facts that structured sources cover only for a limited set of relations, leaving many facts in unstructured text. The paper introduces a distant-supervision method for large-scale relation extraction and separates HypeNET’s features from its architecture. With similar features, fastText performs on par with HypeNET while training much faster.
Problem
Structured sources cover a limited set of high-importance relations, while many facts are expressed in unstructured text.
Source: §1 · p. 1Method
Results
fastText performs on par with HypeNET using almost the same input features, reducing training time from around 75 minutes to less than a minute.
Source: §6.2 · p. 6Takeaways & Limitations
The experiments suggest that dynamic-length context modeling did not bring gains over fastText’s fixed n-gram window in this comparison.
Source: §7 · p. 7
Abstract
from arXiv · showhide
Knowledge-based question answering relies on the availability of facts, the majority of which cannot be found in structured sources (e.g. Wikipedia info-boxes, Wikidata). One of the major components of extracting facts from unstructured text is Relation Extraction (RE). In this paper we propose a novel method for creating distant (weak) supervision labels for training a large-scale RE system. We also provide new evidence about the effectiveness of neural network approaches by decoupling the model architecture from the feature design of a state-of-the-art neural network system. Surprisingly, a much simpler classifier trained on similar features performs on par with the highly complex neural network system (at 75x reduction to the training time), suggesting that the features are a bigger contributor to the final performance.
1. Introduction
Structured sources provide accurate facts but cover a limited set of high-importance relations, leaving many facts in unstructured text. The paper examines relation extraction at large scale and introduces a new distant-supervision method.
1. Introduction
- Structured sources are accurate, but cover a limited set of high-importance relations, while many facts remain implicit or explicit in unstructured text.The paper motivates extracting facts from text to enrich knowledge bases.Source: §1 · p. 1
- Relation extraction identifies relations between resolved entities and normalizes them by mapping them to a knowledge-base schema.Fact extraction also involves entity recognition and resolution, which map surface strings to knowledge-base entities.Source: §1 · p. 1
- The paper analyzes a state-of-the-art model on Wikidata and Alexa KB datasets and proposes distant supervision with reduced entity-resolution noise for large-scale RE.The target scale is billions of facts.Source: §1 · p. 1
2. Related work
Earlier relation-extraction work focused on events and a limited set of relations, while later approaches combined linguistic features and distant supervision. This paper follows HypeNET while extending evaluation to multiple relations and large-scale knowledge bases.
2. Related work
- Earlier MUC and ACE extraction focused on events in limited domains, with only 3 relations in MUC and 24 in ACE.The cited account reports 7k relation instances for 40k entity mentions in ACE.Source: §2 · p. 1
- Unlike OpenIE systems that extract entity and relation phrases, this work targets entities and relations normalized to a knowledge base.Distant supervision assumes a sentence mentioning two entities known to share a relation in a knowledge base is evidence for that relation.Source: §2 · p. 2
- Prior distant-supervision work reduces label noise either by combining multiple weak-supervision sources or by aggregating support sentences for each entity pair.The paper follows the support-sentence aggregation line of work.Source: §2 · p. 2
2.1. HypeNET
- HypeNET combined dependency-path information with entity representations and was applied to instance-of extraction, while this work targets multiple relations at much larger scale.The authors use Wikidata and Alexa KB, and note that HypeNET's training examples covered only one relation.Source: §2.1 · p. 2
3. Distant supervision
Applying the existing distant-supervision approach to large knowledge bases produced noisy annotations, in part because entity names can be ambiguous. The paper proposes page-specific gazetteers and a revised annotation pipeline to reduce that noise.
3. Distant supervision
- Large knowledge bases yield ambiguous entity denotations and poor distant-supervision annotations when the existing HypeNET technique is applied.For example, “Chicago” can refer to either a city or a Broadway musical.Source: §3 · p. 2
3.1. Page-specific gazetteers
- A page-specific gazetteer collects denotations of entities one KB hop from a Wikipedia page’s main entity, reducing extracted entities while improving non-NP coverage and entity-resolution precision.Considering entities more than one hop away is proposed as a way to increase recall; the figure illustrates the page-specific gazetteer pipeline and a one-hop KB neighborhood.Source: §3.1 · p. 2, Figure 1 · p. 3, Figure 2 · p. 3
3.2. Annotation pipeline
- The annotation pipeline uses greedy longest-string matching with the page-specific gazetteer to assign KB IDs, replacing the earlier noun-phrase candidate matching approach.The earlier approach also filtered infrequent paths and paths longer than five tokens.Source: §3.2 · p. 3
- For Alexa KB labeling, relation class constraints precede Bloom-filter membership checks; the system keeps negatives at a 4:1 ratio to positives.The Bloom filters use a 0.001 false-positive rate; Wikidata labels are checked directly as relation properties.Source: §3.2 · p. 4
4. Isolating HypeNET features
The paper separates HypeNET’s feature design from its neural architecture by encoding dependency-path and entity information as discrete features for simpler classifiers. It compares MaxEnt, fastText, and HypeNET implementations using those features.
4. Isolating HypeNET features
- The feature set encodes each support’s dependency path with lemma, Brown-cluster prefix, part of speech, dependency relation, and direction, alongside entity strings and cluster prefixes.Features from different supports are concatenated into one feature list.Source: §4 · p. 4
4.1. Using a MaxEnt classifier
- The MaxEnt experiment uses the discrete features with MALLET’s default LBFGS optimizer and a Gaussian prior variance of 1.The classifier is a standard Maximum Entropy model.Source: §4.1 · p. 4, §4 · p. 4
4.2. Using the fastText model
- fastText uses token n-grams as inputs to a linear classifier with a rank constraint and hierarchical softmax; the experiments set the n-gram count to 4.Other fastText settings remain at their defaults.Source: §4.2 · p. 4
4.3. Using the HypeNET model
- HypeNET encodes dependency-path tokens with linguistic embeddings and LSTMs, averages the path representations, adds entity embeddings, and classifies with a softmax layer.The implementation trains lemma embeddings initialized with GloVe and replaces the unidirectional LSTM with a bidirectional LSTM.Source: §4.3 · p. 4, Figure 3 · p. 4
5. Evaluation
Evaluation spans several relations and knowledge bases using distant-supervision examples. On Alexa KB’s instance-of task, the resulting model slightly exceeded the basic model.
- Evaluation covered instance of, birthplace of, and part of; for Alexa KB, part of was replaced with applies to.Source: §5 · pp. 4–5
- 94.29±0.21 F-score versus 94±0.15 for the basic model on Alexa KB instance of at threshold 0.5 across three trials.Source: §5 · p. 4
- Models used 50K training and 10K validation and test examples per relation, except part of with 22K training; each example grouped supporting sentences for a distant-supervision triple.Source: §5 · p. 5
6. Results and discussion
The experiments compare distant-supervision label quality and RE model choices, finding that fastText matches HypeNET while training much faster. Further analyses examine training-data size, support grouping, satellite nodes, support selection, and feature ablations.
6.1. Distant supervision
- 1% false positives and 39% false negatives were produced by the page-specific gazetteer, versus 67% and 3%, respectively, for Shwartz et al. (2016).On Wikidata, the new method had around 1% false positives and 5% false negatives; many false negatives involved relations such as subclass, treated separately here. The gazetteer also addressed spurious entity matching and non-standard noun phrases.Source: §6.1 · pp. 5–6
6.2. Model comparison
- About 75 minutes to under a minute of training time, fastText performs on par with HypeNET using almost the same features.MaxEnt shows that the features alone are insufficient; the authors attribute the best results to fastText’s and HypeNET’s higher-dimensional feature representations.Source: §6.2 · p. 6
6.3. Training data size
- Above 90% F-score, HypeNET and fastText perform even with 1,000 training examples for instance of on Alexa KB.F-scores increase with more training examples; Figure 5 uses a 0.5 confidence threshold, and its training-size counts refer to entity pairs rather than individual support sentences.Source: §6.3 · p. 6, Figure 5 · p. 6
6.4. Grouping supports
- The effect of grouping supports is much smaller for instance of than for birthplace of and applies to.The authors suggest that supports for the latter two relations may be more diverse, making grouping more useful to the classifier.Source: §6.4 · p. 6, Table 2 · p. 6
6.5. Using satellite nodes
- Satellite nodes offer the clearest advantage for birthplace of; performance without them differs only marginally for instance of and applies to.The authors suggest immediate entity context is less informative for the latter relations because their expressions are syntactically more constrained.Source: §6.5 · p. 6, Table 3 · p. 6
6.6. Using all supports
- Using the five most frequent supports slightly increases performance except for birthplace of, despite many examples having more than five supports.On average, more than 18K training examples and 3K test examples contain more than five supports.Source: Table 4 · p. 6, §6.6 · p. 7
6.7. Feature ablation
- Dependency-parser structure is critical for instance of and applies to, while full-sentence features match the best dependency-supported version for birthplace of.The authors point to relations sometimes expressed without clear lexical cues as one explanation; birthplace of has cues such as “born in” or city names.Source: Table 5 · p. 7, §6.7 · p. 7
7. Conclusion
The conclusion reports that a simple fastText model achieves similar performance to HypeNET, while dynamic-length context modeling brings no gains. It also identifies broader relation coverage and distant-supervision noise as priorities for future work.
7. Conclusion
- Replacing HypeNET’s network architecture with fastText achieves similar performance.The paper presents a new mechanism for generating distant-supervision data based on its large-scale KB.Source: §7 · p. 7
- Dynamic-length context modeling brought no gains in the experiments.HypeNET uses LSTMs for longer dynamic contexts, whereas fastText uses a fixed n-gram window.Source: §7 · p. 7
- Experiments on grouping support and satellite node features provide a basis for building RE systems for more relations.Source: §7 · p. 7
- Scaling to about 1k relations will require combining the outputs of multiple relation-group systems into one classifier.The authors describe broader relation coverage as an immediate future focus.Source: §7 · p. 7
- Distant supervision remains noisy because sentences mentioning both entities do not always express their relation.The authors propose expanding manual annotations to training and validation sets to assist or replace distant supervision.Source: §7 · p. 7