Source-linked AI summary
CoType: Joint Extraction of Typed Entities and Relations with Knowledge Bases
Xiang Ren, Zeqiu Wu, Wenqi He, Meng Qu, Clare R. Voss, Heng Ji, Tarek F. Abdelzaher, Jiawei Han
TL;DR
Joint extraction of typed entities and relations remains difficult because human annotation is costly while distant supervision introduces context-agnostic label noise and existing pipelines ignore cross-task dependencies. CoType addresses this with domain-independent mention segmentation and joint, noise-robust embeddings that model entity-relation interactions. Across three public datasets, it improves state-of-the-art performance and is reported to be robust across domains.
Problem
Joint typed entity-relation extraction needs domain-independent learning without expensive human annotation, but distant supervision introduces context-agnostic false type labels and existing methods overlook cross-task dependencies.
Method
CoType mines entity mentions with domain-agnostic segmentation and jointly embeds mentions, text features, and type labels using a noise-robust objective with entity-relation cross-constraints.
Results
Experiments on three public datasets demonstrate improved entity-typing and relation-extraction performance and robust domain independence.
Takeaways & Limitations
CoType provides a single distant-supervision framework for jointly estimating types of linkable and unlinkable entity and relation mentions across different domains.
Takeaways & Limitations
The framework does not address label noise from wrongly linked knowledge-base entities and assumes human-curated target type hierarchies.
Abstract
from arXiv · showhide
Extracting entities and relations for types of interest from text is important for understanding massive text corpora. Traditionally, systems of entity relation extraction have relied on human-annotated corpora for training and adopted an incremental pipeline. Such systems require additional human expertise to be ported to a new domain, and are vulnerable to errors cascading down the pipeline. In this paper, we investigate joint extraction of typed entities and relations with labeled data heuristically obtained from knowledge bases (i.e., distant supervision). As our algorithm for type labeling via distant supervision is context-agnostic, noisy training data poses unique challenges for the task. We propose a novel domain-independent framework, called CoType, that runs a data-driven text segmentation algorithm to extract entity mentions, and jointly embeds entity mentions, relation mentions, text features and type labels into two low-dimensional spaces (for entity and relation mentions respectively), where, in each space, objects whose types are close will also have similar representations. CoType, then using these learned embeddings, estimates the types of test (unlinkable) mentions. We formulate a joint optimization problem to learn embeddings from text corpora and knowledge bases, adopting a novel partial-label loss function for noisy labeled data and introducing an object "translation" function to capture the cross-constraints of entities and relations on each other. Experiments on three public datasets demonstrate the effectiveness of CoType across different domains (e.g., news, biomedical), with an average of 25% improvement in F1 score compared to the next best method.
1. INTRODUCTION
CoType addresses joint extraction of typed entities and relations across domains without costly human-annotated data by combining distant supervision with joint, noise-robust modeling. It targets context-dependent label noise and cross-task dependencies, and experiments report improved performance across three public datasets.
- Motivation: Manual annotation for many entity and relation types is expensive and error-prone, motivating domain-independent extraction with minimal or no human supervision.The paper focuses on domain-specific corpora such as news, scientific publications, and social media content.
- Challenges: Distant supervision can assign false positive types because knowledge-base mappings are context-agnostic.For example, Barack Obama’s mention may receive politician and artist, although only person is correct in context.
- Approach: CoType detects entity mentions with domain-agnostic segmentation and jointly models entity and relation typing under distant supervision.Its framework uses positive knowledge-base examples for segmentation and learns from corpus features, type labels, and mention interactions.
- Approach: The joint embedding objective models mention-type association, mention-feature co-occurrence, and entity-relation cross-constraints in a noise-robust way.The framework uses a partial-label loss for noisy associations and a translation-based objective for entity-relation interactions.
- Results: Experiments on three public datasets show significantly improved entity-typing and relation-extraction performance with robust domain independence.The authors present this result as evidence that CoType improves state-of-the-art systems across domains.
2. BACKGROUND AND PROBLEM
The task uses a POS-tagged corpus, knowledge base, and target type specifications to create automatically labeled training data and infer types for unlabeled relation mentions. It focuses on binary relations and acknowledges limited linkability and reliance on external entity linking and curated hierarchies.
- Inputs: CoType takes a POS-tagged corpus, a knowledge base, a target entity-type hierarchy, and a target relation-type set as inputs.The target types are subsets of the corresponding types available in the knowledge base.
- Definitions: An entity mention is a token span representing an entity, while a binary relation mention is an ordered pair of entity mentions in a sentence.The relation mention representation is z = (m1, m2, s).
- Training data: Distant supervision links extracted mentions to knowledge-base entities and assigns their knowledge-base types as candidate labels.Candidate relation types come from relations between mapped entities, while candidate entity types come from their knowledge-base type facts.
- Candidate sets: Candidate relation mentions include linkable mentions, unlinkable true relations, and false relation mentions with no target relation expressed.The unlabeled set contains both unlinkable true relations and false candidates.
- Task definition: The task estimates a relation type or None for each unlabeled relation mention and a single entity type-path or None for each argument.The formal problem uses automatically labeled data and the mention’s local context.
- Scope: The framework does not address errors from wrongly mapped knowledge-base entities and assumes human-curated target type hierarchies are available.Both entity-linking limits and hierarchy generation are explicitly outside the study’s scope.
3. THE COTYPE FRAMEWORK
CoType addresses noisy, context-agnostic distant supervision by jointly modeling entity and relation mentions, text features, and type labels. Its framework generates candidates through domain-agnostic segmentation, learns mutually constrained embeddings, and infers types from the learned spaces.
- Challenges: Distant supervision introduces false candidate types because KB associations are context-agnostic, while relation mentions and entity arguments also have dependent types.These challenges make independently trained or cascading subtasks vulnerable to incorrect labels and error propagation.
- Joint Entity and Relation Embedding: CoType uses weakly supervised partial-label learning to model mention-type relevance from local context features.The most relevant type is progressively estimated during learning rather than treating every candidate type as correct.
- Model Learning and Type Inference: The learned embeddings support type inference for unlinkable test mentions by searching target relation types and entity type paths.Objects that are close in each embedding space tend to share types.
- Candidate Generation: A POS-constrained segmentation algorithm mines phrase and POS-pattern quality to detect candidate entity mentions from a POS-tagged corpus.It trains random-forest classifiers for phrase and POS-pattern quality, then finds the best segmentation using Viterbi Training with linear corpus-size complexity.
- Candidate Generation: Candidate relation mentions are generated from ordered pairs of detected entity mentions, with lexical and contextual features extracted for each mention and its arguments.The procedure forms both argument orders for each entity pair in a sentence before applying distant supervision.
- Joint Entity and Relation Embedding: CoType jointly embeds mentions, text features, and type labels in entity and relation spaces, using translation-based loss to capture entity-relation interactions.The global objective allows entity and relation embeddings to mutually influence one another, helping constrain errors across components.
4. EXPERIMENTS
Experiments evaluate CoType on three domain-diverse datasets against distant-supervision, embedding, neural, supervised, and joint-extraction baselines. CoType consistently achieves stronger extraction and classification performance, remains robust across corpus sizes, and scales linearly.
- Data and setup: Experiments use NYT, Wiki-KBP, and BioInfer corpora spanning news, Wikipedia, and biomedical text, with manually annotated evaluation data.Training corpora are heuristically labeled through distant supervision, while evaluation partitions contain manually annotated relation types and entity arguments.
- Data and setup: The comparison includes distant-supervision classifiers, graph and embedding methods, MultiR, FCM, and DS-Joint, alongside CoType variants.CoType-RM removes entity-relation modeling, while CoType-TwoStep represents a pipeline-style optimization.
- Entity recognition and typing: 8% improvement in Micro-F1 over the next best method on NYT accompanies CoType’s lead on all entity-recognition and typing metrics across all three datasets.The reported gains are attributed to noise-robust candidate-type modeling and joint entity-relation embeddings.
- Relation classification: Over 10% enhancement on both NYT and BioInfer over the next best method is reported for relation classification accuracy.CoType-RM supports the value of partial-label noise modeling, while the full model’s advantage over CoType-RM and CoType-TwoStep supports translation-based cross-constraints.
- Relation extraction: CoType outperforms all other methods on F1 score across the three datasets in end-to-end relation extraction.Its domain-agnostic segmentation addresses false-negative tags in distant-supervision data, while joint modeling improves over the incremental variant; precision-recall curves preserve good recall with decent precision.
- Robustness and scalability: CoType performs best at every tested BioInfer training-corpus sampling ratio, and its runtime follows a linear trend while handling the full dataset without significant time cost.All three methods improve as the sampling ratio increases.
5. RELATED WORK
Related work spans incremental extraction, weak and distant supervision, joint structured models, embedding methods, and noisy-label learning. CoType combines distant supervision with joint modeling of entity-relation interactions and noisy labels.
- Entity and relation extraction: Most existing systems incrementally recognize and type entities before extracting relation mentions, whereas some joint methods integrate both tasks globally.Existing joint approaches use sequence labeling, relation-argument constraints, or factor graphs, but require human-annotated corpora and existing entity detectors.
- Supervision: Weak supervision bootstraps from manually specified seeds or patterns, while distant supervision aligns text with a knowledge base to generate candidate entity and relation types.Distant supervision assigns KB-derived types to detected mentions and mention pairs, even when those assignments are context-inappropriate.
- CoType’s positioning: CoType combines noisy distant supervision with joint modeling of entity-relation interactions and other signals to address domain restriction and error propagation.This positions it against human-annotation-dependent joint extraction and separately solved distant-supervision methods.
- Embeddings and noisy labels: Embedding research commonly assumes links or labels are correct, whereas CoType models true links and labels for entity and relation mentions in unstructured contexts.Its objects differ from structured KB embeddings because the focus is on context-dependent mentions.
- Embeddings and noisy labels: Partial-label and multi-label multi-instance learning address noisy candidate labels, but CoType additionally models entity-relation interactions.The paper compares its full model with variants to separate these contributions.
6. CONCLUSION
The conclusion presents CoType as a domain-independent framework for jointly extracting typed entities and relations with distant supervision. Its segmentation, global embedding, noise-robust objective, and mutual-dependency modeling support effectiveness across domains, while future work targets false negatives, type hierarchies, and joint test-time inference.
- Conclusion: CoType mines entity mentions with a domain-agnostic segmentation algorithm and formulates joint entity-relation typing as a global embedding problem.The framework is designed for domain-independent extraction with distant supervision.
- Conclusion: A noise-robust objective models noisy type labels, while the framework captures mutual dependencies between entity and relation mentions.Experiments are described as demonstrating effectiveness and robustness across text corpora from different domains.
- Future work: Future work includes reducing false-negative type labels, modeling type correlations in hierarchies, and jointly inferring test entity and relation types.These directions are identified as extensions of the presented framework.