Source-linked AI summary
Combining Recurrent and Convolutional Neural Networks for Relation Classification
Ngoc Thang Vu, Heike Adel, Pankaj Gupta, Hinrich Schütze
TL;DR
Relation classification requires assigning predefined relations to marked entity pairs, and the paper examines how CNN and RNN architectures can improve this task. It introduces extended middle context, connectionist bi-directional RNNs, and ranking-based optimization, then combines CNNs and RNNs by voting. The combined models achieve new state-of-the-art results on SemEval 2010.
Problem
Relation classification assigns sentences with two marked entities to predefined relations, motivating improved neural architectures for the SemEval 2010 benchmark.
Method
The paper develops extended middle context for CNNs, connectionist bi-directional RNNs with ranking loss, and a voting ensemble of CNN and RNN models.
Results
The combined CNN-RNN voting model achieves an F1 score of 84.9 and performs better than the two neural-network types alone.
Takeaways & Limitations
The models achieve new state-of-the-art results on the SemEval 2010 relation-classification benchmark without linguistic features.
Abstract
from arXiv · showhide
This paper investigates two different neural architectures for the task of relation classification: convolutional neural networks and recurrent neural networks. For both models, we demonstrate the effect of different architectural choices. We present a new context representation for convolutional neural networks for relation classification (extended middle context). Furthermore, we propose connectionist bi-directional recurrent neural networks and introduce ranking loss for their optimization. Finally, we show that combining convolutional and recurrent neural networks using a simple voting scheme is accurate enough to improve results. Our neural models achieve state-of-the-art results on the SemEval 2010 relation classification task.
1 Introduction
The paper studies CNNs and RNNs for relation classification and proposes architectural and optimization changes for both. It combines the two model types with voting and reports new state-of-the-art results on SemEval 2010.
- Relation classification assigns sentences containing two marked entities to a predefined set of relations, with neural networks improving performance on SemEval 2010 data.
- The study investigates recurrent and convolutional neural networks, including their combination.
- Extended middle context is a CNN representation that uses all sentence regions while emphasizing the text between the relation arguments.
- Connectionist bi-directional RNNs combine intermediate hidden layers for final decisions, and ranking loss is introduced for RNN optimization in relation classification.
- A simple voting combination of CNNs and RNNs achieves new state-of-the-art results on the SemEval 2010 benchmark dataset.
2 Related Work
Prior work moved relation classification from traditional classifiers toward neural architectures, exploring CNNs, recursive networks, and recurrent models. CNN and RNN studies varied context, filters, lexical features, pooling, and output objectives.
- Earlier SemEval relation-classification systems used support vector machines or maximum entropy classifiers, while later neural networks outperformed results on the same dataset.
- CNN-based approaches used context between relation arguments, lexical features, different filter sizes, and ranking layers.
- Other neural approaches included recursively parsed sentence representations and temporal structured bi-directional RNNs with pooling.
3 Convolutional Neural Networks (CNN)
The CNN represents sentences through convolution and max-pooling, while the paper introduces extended middle context to emphasize relation-relevant text without discarding surrounding context.
- CNNs convolve word-embedding matrices with filters and use max-pooling to retain the most informative n-gram activation for subsequent processing.
- Extended middle context splits a sentence around two relation arguments and forms two inputs combining the middle region with either the left or right context.
- Repeating the middle context directs attention toward it, while independent convolution and max-pooling layers process the two contexts before concatenation.
- The design uses two-dimensional filters spanning embedding dimensions and multiple window sizes from 2 to 5.
4 Recurrent Neural Networks (RNN)
The RNN models sentence-level relation classification using forward and backward processing, with architectural variants compared across directionality, hidden-layer connections, and input representation. The connectionist variant links combined hidden states so intermediate layers contribute to the final decision.
- Unlike word-level prediction in traditional RNNs, these models process the whole sentence before predicting an output vector.
- Bi-directional processing incorporates both preceding and succeeding words when modeling each relation-classification position.
- The architecture has jointly trained forward, backward, and combined passes for sentence processing.
- Connectionist bi-directional RNNs add a connection from the previous combined hidden layer so all intermediate hidden layers can inform the final decision.
- Experiments compare the connectionist model with uni-directional and ordinary bi-directional RNNs lacking additional hidden-layer connections.
5 Model Training
The models represent words with embeddings and position information, and ranking loss trains models to separate true relations from competing labels.
- Input representations: Words use concatenated word-embedding and position-feature vectors, with embeddings updated during model training.Word2vec embeddings were trained on English Wikipedia, while position embeddings were randomly initialized.
- Ranking loss: Ranking loss maximizes the distance between the true label y+ and the best competing label c− for each data point.The objective penalizes prediction errors using separate margins for correct and incorrect classes.
- Ranking loss: The loss uses class scores sθ(x)y+ and sθ(x)c−, with γ controlling error penalization and m+ and m− defining class margins.The reported settings are γ = 2, m+ = 2.5, and m− = 0.5.
- Ranking loss: The Other class is not assigned a learned pattern and is trained using only the loss term that increases its difference from the best competing label.
6 Experiments and Results
Experiments evaluate CNNs, RNNs, and their voting-based ensemble on SemEval 2010 using macro F1. Architectural variations improve individual models, while their combination reaches 84.9 F1 and state-of-the-art performance without linguistic features.
- Experimental setup: The SemEval 2010 task 8 dataset contains 8,000 training sentences, 2,717 test sentences, and 19 manually labeled relations; evaluation uses macro F1.
- CNN performance: The CNN baseline scores 73.0 F1, rising to 78.6 after adding 5-dimensional position features.This is comparable to the reported 78.9 result from Zeng et al. without linguistic features.
- CNN performance: CNN experiments successively test multi-window convolution, a ranking layer, extended middle context, and larger word embeddings with comparable parameter counts.Table 1 reports F1 scores for these component changes.
- RNN performance: The uni-directional RNN baseline scores 61.2 F1, while position indicators produce 73.4 F1 on the SemEval test set.The difference between position indicators and position embeddings with entity flags is not statistically significant.
- RNN performance: Bi-directionality, connectionist hidden-layer connections, ranking prediction, and larger embeddings each yield statistically significant RNN improvements.Additional hidden-layer connections and the ranking layer have especially large impacts on performance.
- Combination and comparison: 84.9 F1 is achieved by voting across CNN and RNN models, exceeding either neural-network type alone.The authors attribute the improvement to complementary information from sentence-wide RNN weighting and CNN n-gram extraction.
- Combination and comparison: The proposed ER-CNN and R-RNN obtain state-of-the-art results on SemEval 2010 without linguistic features.
7 Conclusion
The paper studies architectural and feature choices for CNNs and RNNs in relation classification without linguistic features. It introduces new CNN context and recurrent designs, then shows that simple CNN–RNN combination improves benchmark results.
- Conclusion: The study investigates features and architectural choices for convolutional and recurrent neural networks without using linguistic features.
- Conclusion: For CNNs, the paper introduces a new context representation designed for relation classification.
- Conclusion: For RNNs, it introduces connectionist recurrent networks and reports the first experiments with ranking recurrent neural networks.
- Conclusion: A simple combination of CNNs and RNNs improves results and achieves new state-of-the-art performance on the SemEval 2010 task 8 benchmark.