Source-linked AI summary
Classifying Relations by Ranking with Convolutional Neural Networks
Cicero Nogueira dos Santos, Bing Xiang, Bowen Zhou
TL;DR
Relation classification systems still often rely on costly handcrafted features. This paper introduces CR-CNN, a convolutional network that classifies by ranking, and reports state-of-the-art performance on SemEval-2010 Task 8 with F1 84.1 without such features.
Problem
State-of-the-art relation classification systems still rely on costly handcrafted features.
Method
CR-CNN uses convolution, learned class embeddings, and a pairwise ranking loss to classify relations.
Results
F1 84.1: CR-CNN outperforms the state-of-the-art on SemEval-2010 Task 8 without costly handcrafted features.
Takeaways & Limitations
Omitting the artificial class Other improves precision and recall, while word embeddings alone achieve state-of-the-art results on the text span between target nominals.
Takeaways & Limitations
The text-span experiment assumes that most information needed for classification appears between the two target nouns.
Abstract
from arXiv · showhide
Relation classification is an important semantic processing task for which state-ofthe-art systems still rely on costly handcrafted features. In this work we tackle the relation classification task using a convolutional neural network that performs classification by ranking (CR-CNN). We propose a new pairwise ranking loss function that makes it easy to reduce the impact of artificial classes. We perform experiments using the the SemEval-2010 Task 8 dataset, which is designed for the task of classifying the relationship between two nominals marked in a sentence. Using CRCNN, we outperform the state-of-the-art for this dataset and achieve a F1 of 84.1 without using any costly handcrafted features. Additionally, our experimental results show that: (1) our approach is more effective than CNN followed by a softmax classifier; (2) omitting the representation of the artificial class Other improves both precision and recall; and (3) using only word embeddings as input features is enough to achieve state-of-the-art results if we consider only the text between the two target nominals.
1 Introduction
Relation classification supports complex NLP applications, but competitive neural systems still depend on costly handcrafted features. The paper introduces CR-CNN to address this task without those features.
- Relation classification is used as an intermediate step in question-answering and automatic knowledge base construction.
- Relation classification identifies the relationship between two marked nominals in a sentence.The SemEval-2010 Task 8 dataset benchmarks this task.
- Deep neural approaches aim to reduce handcrafted features but still use WordNet, dependency-parser, or named-entity-recognizer features for state-of-the-art results.
- CR-CNN performs relation classification by ranking and uses a pairwise ranking loss to reduce the impact of artificial classes.
2 The Proposed Neural Network
CR-CNN converts text into a fixed-sized sentence representation, scores classes through learned class embeddings, and trains by pairwise ranking. Word position embeddings, hard-negative selection, and ranking loss support the architecture and training procedure.
- Architecture: CR-CNN transforms tokenized words into vectors, applies convolution, and scores each relation class by a dot product with its learned class embedding.
- Word Position Embeddings: Word position embeddings encode each word’s relative distances to the two target nouns and can be concatenated with word embeddings before convolution.
- Sentence Representation: The convolutional layer applies the same matrix to successive word windows and uses max pooling to create a fixed-sized sentence representation.
- Training Procedure: Training compares a correct class against an incorrect class using a logistic pairwise ranking loss.
- Training Procedure: The loss encourages scores above m+ for correct classes and below −m− for incorrect classes.
- Training Procedure: Updating only selected class pairs can make ranking-based training efficient for tasks with very many classes.
- Training Procedure: Selecting the highest-scoring incorrect class supplies an informative negative for stochastic gradient updates.
3 Experimental Setup
Experiments use the SemEval-2010 Task 8 benchmark, official macro-F1 evaluation, pretrained word embeddings, and cross-validated CR-CNN hyperparameter tuning.
- Dataset: SemEval-2010 Task 8 contains 10,717 examples covering nine relation types plus the artificial relation Other.The dataset includes relations such as Cause-Effect, Component-Whole, and Message-Topic.
- Dataset: The benchmark is partitioned into 8,000 training instances and 2,717 test instances.
- Evaluation: Systems are scored with the official scorer using macro-averaged F1 over the nine actual relations, excluding Other and accounting for directionality.
- Input representations: Word embeddings are initialized through unsupervised skip-gram pre-training on the December 2013 English Wikipedia snapshot using word2vec.
- Neural network hyperparameters: CR-CNN hyperparameters are tuned with 4-fold cross-validation, with best results obtained after approximately 10–15 training epochs depending on configuration.Learning rates between 0.03 and 0.01 give relatively similar results.
4 Experimental Results
Experiments show that CR-CNN benefits from word position embeddings on full sentences, while text spans between target nouns achieve near-state-of-the-art performance using only word embeddings. CR-CNN also improves over CNN+Softmax and prior systems, including when omitting the artificial class Other.
- Word Position Embeddings and Input Text Span: F1 rises from 74.3 to 84.1 when word position embeddings are added to full-sentence CR-CNN input.With only the text span between target nouns, word position embeddings have a smaller effect, and word embeddings alone achieve F1 82.8.
- Impact of Omitting the Embedding of Other: F1 increases by 1.4 when the embedding for the noisy artificial class Other is omitted, improving precision and recall for natural classes.The Other class groups infrequent relation types, making its embedding difficult to define and potentially noisy.
- CR-CNN versus CNN+Softmax: CR-CNN improves F1 by 1.6 over CNN+Softmax while outperforming it in both precision and recall.Both models use the same data, word embeddings, and word position embeddings for a fair comparison.
- Comparison with the State of the Art: F1 reaches 84.1, establishing a new state of the art on SemEval-2010 Task 8 without complicated lexical-resource or NLP-tool features.The comparison includes systems using support vector machines, recursive neural networks, and dependency-based features.
- Comparison with the State of the Art: CR-CNN achieves F1 82.8 using word embeddings as its only input features, exceeding the closest such result, 80.6, by 2.2 F1 points.This result uses only the text span between the target nouns.
- Most Representative Trigrams for Each Relation: The most informative trigrams differ with relation direction; “away from the” supports Entity-Origin(e1,e2), while “the source of” supports the reverse direction.Trigram contributions are computed by tracing convolutional representations back to input positions and ranking aggregate contributions.
5 Related Work
Relation classification research has progressed from traditional multiclass methods to deep neural architectures, but many strong systems still depend on syntactic or lexical resources. CR-CNN differs through pairwise ranking and by omitting artificial-class embeddings.
- Relation classification is commonly treated as a multiclass problem using varied machine-learning techniques.
- Deep neural approaches include recursive and convolutional models that compose sentence representations from words, parse trees, position features, and lexical features.
- State-of-the-art deep systems still use external resources or tools, including WordNet, dependency parsers, named entity recognizers, dependency trees, and named entities.
- Yu et al.'s FCM achieves slightly higher accuracy than Zeng et al.'s method only when syntactic information is used.
- CR-CNN uses pairwise ranking rather than softmax multiclass classification and omits artificial-class embeddings instead of treating all classes equally.These differences distinguish its scoring and class-representation strategy from the compared CNN and RNN approaches.
6 Conclusion
The paper presents CR-CNN as a ranking-based CNN for relation classification, combining class embeddings with a new ranking loss and a strategy for artificial classes. It reports state-of-the-art performance without costly handcrafted features and identifies representative relation contexts.
- CR-CNN achieves a new state-of-the-art on SemEval-2010 Task 8 without costly handcrafted features.
- The model combines CNN classification by ranking with learned class embeddings and a new rank loss function.
- Omitting artificial-class embeddings provides an effective way to handle artificial classes in CR-CNN.
- Using only the text between target nominals is almost as effective as using word position embeddings.
- The model can extract the most representative contexts for each relation type.