Source-linked AI summary
Improved Relation Extraction with Feature-Rich Compositional Embedding Models
Matthew R. Gormley, Mo Yu, Mark Dredze
TL;DR
Relation extraction needs representations that combine generalizable lexical information with structural context while handling varied annotations. The paper introduces FCM, which composes word embeddings with unlexicalized features, and reports strong results across ACE 2005 and SemEval-2010. FCM remains easy to implement and supports arbitrary input structure and global composition information.
Problem
Lexical features hinder generalization to unseen words, while embeddings alone are insufficient because relation extraction also requires broader linguistic context and varied annotations.
Method
FCM composes word embeddings with unlexicalized linguistic features over arbitrary sentence annotations and predicts relations from the resulting sentence embedding.
Results
FCM achieves state-of-the-art relation extraction results on unseen domains from ACE 2005 and SemEval-2010, especially when combined with a feature-based log-linear model.
Takeaways & Limitations
FCM provides an easy-to-implement compositional model that handles arbitrary input types and global information while supporting strong relation extraction performance.
Takeaways & Limitations
The reported ACE 2005 evaluation uses gold named entities to facilitate comparison with prior work.
Abstract
from arXiv · showhide
Compositional embedding models build a representation (or embedding) for a linguistic structure based on its component word embeddings. We propose a Feature-rich Compositional Embedding Model (FCM) for relation extraction that is expressive, generalizes to new domains, and is easy-to-implement. The key idea is to combine both (unlexicalized) hand-crafted features with learned word embeddings. The model is able to directly tackle the difficulties met by traditional compositional embeddings models, such as handling arbitrary types of sentence annotations and utilizing global information for composition. We test the proposed model on two relation extraction tasks, and demonstrate that our model outperforms both previous compositional models and traditional feature rich models on the ACE 2005 relation extraction task, and the SemEval 2010 relation classification task. The combination of our model and a log-linear classifier with hand-crafted features gives state-of-the-art results.
1 Introduction
Relation extraction benefits from combining generalizable word embeddings with unlexicalized linguistic context. FCM composes these signals over arbitrary sentence annotations and achieves strong results on ACE 2005 and SemEval-2010.
- Motivation: Lexical features help relation extraction but make generalization to unseen words difficult, while embeddings alone are insufficient without broader linguistic context.Prior systems combine lexical information with contextual features such as entity position and dependency-path membership.
- Model: FCM combines unlexicalized linguistic context with word embeddings to represent relations and generalize to unseen data.The model supports arbitrary linguistic structure expressed through hand-crafted features.
- Model: The model combines learned embeddings with hand-crafted features so lexical indicators can generalize while structural roles remain distinguishable.Examples include distinguishing relevant occurrences between entities or on their dependency path from irrelevant occurrences.
- Model: FCM decomposes an annotated sentence, builds feature-enriched substructure embeddings, sums them into a sentence embedding, and predicts the relation with a softmax.Its composition uses features associated with substructures and can incorporate global information.
- Results: FCM with a log-linear model obtains the best reported ACE-2005 coarse-grained cross-domain results, while its SemEval-2010 results are competitive with the best reported results.The paper presents FCM as a new compositional embedding model for relation extraction.
2 Relation Extraction
Relation extraction identifies relations for entity pairs in sentences, including directed relation types. The task combines difficult annotation and data characteristics, motivating contextual modeling of words and their structural roles.
- Task definition: Relation extraction identifies what relation exists, if any, for every pair of entity mentions in an input sentence.Each instance includes the sentence, two entity mentions, and associated annotations.
- Task definition: Relations are directed, so Rel(M1, M2) and Rel(M2, M1) are treated as different relations.ACE 2005 also contains many negative examples, whereas SemEval reduces the number of negative examples artificially.
- Task setting: Standard ACE 2005 relation extraction uses long sentences with multiple named entities, known entity types, and unknown relation directions.Earlier relation-classification work often used short sentences, one relation per sentence, or assumed entity order was known.
- Motivation: Whether a word indicates a relation depends on contextual roles such as being between entities, on their dependency path, or to their left or right.These complementary properties motivate representations that capture both lexical semantics and sentence structure.
3 A Feature-rich Compositional Embedding Model for Relations
FCM composes word embeddings with hand-crafted features over annotated sentence substructures, then predicts relations using a log-bilinear model. Its outer-product representation supports arbitrary annotations, global information, and parameter sharing across similarly embedded words with similar sentence functions.
- Combining Features with Embeddings: The framework is general enough to construct annotated sentence embeddings for tasks combining embeddings with hand-engineered lexical features.Although the paper focuses on relation extraction, the stated framework is not restricted to that task.
- Combining Features with Embeddings: FCM represents each word-based substructure as the outer product of hand-crafted features and a dense word embedding.For relation extraction, substructures correspond to words, while features can encode local context and global information from sentence annotations.
- The Log-Bilinear Model: The model sums substructure embeddings into an annotated sentence embedding and applies a softmax layer to predict the relation label.The full pipeline decomposes the sentence, extracts and combines substructure features, sums the resulting embeddings, and predicts an output label.
- Discussion of the Model: FCM generalizes across words with similar embeddings only when those words share similar functions in the sentence.The outer product assigns feature-specific weights, such as distinguishing words on a dependency path from similar words elsewhere.
- Discussion of the Model: FCM is a smoothed version of traditional lexical conjunctions that preserves expressive feature interactions while generalizing to low-frequency features.Replacing a one-hot word representation with its embedding yields the smoothed lexical feature construction.
- Discussion of the Model: Inference in FCM requires O(snd) products with sparse features, compared with O(C · nd^2) for typical CNNs and RNNs.Here, s is the average number of non-zero per-word features, n is sentence length, d is embedding dimension, and C is model-dependent.
4 Hybrid Model
The hybrid model combines FCM with an existing log-linear model by multiplying their scores and normalizing the resulting distribution.
- Hybrid Model: The hybrid model combines FCM with an existing log-linear model to integrate compositional embeddings and hand-crafted features.The model is introduced by combining the two component distributions rather than replacing the log-linear model.
- Hybrid Model: The combined model multiplies the FCM and log-linear probabilities, with a normalization constant ensuring a valid distribution.The log-linear component uses the conventional exp(θ · f(x, y)) form over a feature vector f(x, y).
5 Training
FCM training optimizes a cross-entropy objective using stochastic updates and backpropagation, with optional fine-tuning of word embeddings. The hybrid model remains trainable by backpropagation because its sub-models have separate parameters.
- Objective: FCM training optimizes a cross-entropy objective over the training data.For each instance, stochastic training uses the loss log P(y|x; T, e).
- Optimization: Backpropagation computes parameter gradients by repeatedly applying the chain rule.The gradients are defined using the model’s predicted label probabilities and indicator functions.
- Embedding updates: When word embeddings are parameters, FCM fine-tunes them jointly with the model.This corresponds to the log-bilinear version of the model.
- Hybrid training: The hybrid model can also be trained by backpropagation because its sub-models have separate parameters.
6 Experimental Settings
The experiments use feature-rich FCM configurations, pretrained word embeddings, and established relation-extraction datasets and baselines. ACE 2005 evaluates domain adaptation, while SemEval 2010 evaluates relation classification under a prior comparable setting.
- Features: FCM features describe words, target entities, and their dependency path using compound feature constructions.The feature vector includes entity types, in-between indicators, and on-path indicators.
- Features: Entity types are used to distinguish embedding functions for relation prediction, such as person–vehicle pairs for ART relations.The experiments use ACE named-entity tags or SemEval WordNet supertags for entity types.
- Embedding setup: All experiments use 200d word embeddings trained on Gigaword 5.0 with CBOW and negative sampling.The embeddings use a window size of c=5 and 15 negative words.
- Models: The evaluated models include FCM with or without fine-tuning, a rich-feature log-linear baseline, and hybrid models combining FCM with the log-linear model.
- Datasets and evaluation: ACE 2005 uses a domain-adaptation setting, while SemEval 2010 Task 8 determines the relation type or no relation between two entities.ACE training uses the union of news domains, with separate development and evaluation domains; SemEval uses 10-fold cross-validation on training data.
- Datasets and evaluation: The study focuses on ACE 2005 rather than the common ACE 2004 cross-validation convention because that convention uses the dataset for both training and evaluation.
7 Results
FCM performs strongly across ACE 2005 and SemEval 2010, often surpassing competing compositional and feature-rich models. Its results also show the importance of feature design and embedding choice, while fine-tuning can introduce over-fitting.
- ACE 2005: On ACE 2005, baseline + FCM obtains the highest F1 across out-of-domain test domains and the best reported results for the task.The improvement is significant relative to the other baselines.
- ACE 2005: FCM uses only a dependency parse yet obtains better Avg. F1 than a log-linear model using over 50 feature templates and several linguistic resources.The comparison includes POS tagging, dependency parsing, chunking, and constituency parsing in the log-linear model.
- SemEval 2010 Task 8: With either WordNet or NER features, FCM outperforms the RNN and MVRNN; with NER features and fine-tuning, it also outperforms a CNN and an RNN/MVRNN hybrid.FCM uses fewer linguistic resources than many close competitors.
- SemEval 2010 Task 8: FCM + linear obtains comparable results to concurrent task-specific models, while their base compositional models remain below the best FCM result.Replacing the word embeddings with task-specific embeddings raises the result to 83.7%.
- Ablation analysis: The HeadOnly ablation reaches F1 score = 14.30%, while removing HeadEmb causes the largest degradation, showing the importance of the rich binary feature set.In-between is the second most important feature template, and removing entity type features also significantly hurts performance.
- Effects of the word embeddings: Task-specific embeddings produce the best performance, improving results by 0.7%, whereas increasing embedding dimension does not necessarily improve performance because of over-fitting.With the same initial embeddings, FCM usually outperforms competing models without hyperparameter changes; DepNN is the exception.
8 Related Work
Prior compositional models capture sentence structure but are constrained by fixed substructures and limited annotation use. FCM instead combines compositional embeddings with flexible linguistic features, allowing multiple word properties without changing the model structure.
- Compositional models for sentences: CNNs and RNNs model fixed substructures such as chains or trees, limiting their ability to capture arbitrary combinations of word order, dependency, and named-entity annotations.These approaches also ignore differences in the functions of words appearing in different contexts.
- Compositional models with annotation features: Specializing an RNN to the minimal subtree spanning two target entities improves task fit but does not generalize easily to other NLP tasks.The paper presents feature enhancement and embedding-feature engineering as more general specialization strategies.
- Compositional models with annotation features: Feature-enhanced compositional models and engineered embedding features improve over pure compositional models, but prior approaches typically use only limited information for each word.These approaches include context-specific weights, embedding concatenations, and hand-engineered features added to log-linear models.
- Compositional models with annotation features: FCM can incorporate multiple useful word properties without changing the model structure, addressing the limited feature use of prior approaches.Dependency-path methods share a motivation with FCM’s On-path features.
- Task-specific enhancements for relation classification: Task-specific enhancements form an orthogonal research direction, including task-specific word embeddings and ranking-based loss functions for relation classification.The paper compares these approaches with FCM in its experiments.
9 Conclusion
FCM is a compositional embedding model that handles arbitrary input types and global information while remaining easy to implement. It achieves near-state-of-the-art performance alone and state-of-the-art results when combined with traditional feature-based log-linear models.
- FCM derives sentence-level and substructure embeddings from word embeddings for relation extraction.
- FCM handles arbitrary input types and global information for composition while remaining easy to implement.
- FCM alone attains near-state-of-the-art performance on several relation extraction tasks.
- Combining FCM with traditional feature-based log-linear models produces state-of-the-art results.
- Future improvements focus on task-specific embeddings or loss functions and applications beyond relation extraction.The paper identifies dependency parsing, semantic role labeling, paraphrasing, and TAC-KBP as future application areas.
Appendix 1: Experiments on ACE 2005 where Gold Entity Types Are Unknown
In the low-resource ACE 2005 setting, gold entity spans are available but entity types are unknown. FCM performance decreases without entity types, while Brown clusters and WordNet supersense tags improve FCM, with WordNet achieving the best reported result in this setting.
- Without entity types, FCM-only performance falls to 48.15%.
- Removing entity-type features reduces the baseline model to 40.62%, below the reduced FCM-only model.
- Adding head embeddings greatly improves the baseline, while Baseline + FCM also improves performance but is not significantly better than Baseline + HeadOnly.The authors suggest FCM becomes less efficient at using context embeddings when entity-type information is unavailable.
- Brown clusters improve FCM when entity types are unknown, and WordNet supersense tags produce the best reported result in this low-resource setting.The Brown-cluster FCM outperforms the reported Plank and Moschitti single-model results under the same feature source.
- The best single FCM model is comparable to Nguyen et al.'s best single system, while their combination method is slightly better.
- The evaluation uses ACE 2005 out-of-domain test sets with gold entity spans but unknown entity types.Relations are generated from entity pairs with three or fewer intervening entity mentions, and evaluation covers seven coarse relation types.