Source-linked AI summary
Improved Relation Classification by Deep Recurrent Neural Networks with Data Augmentation
Yan Xu, Ran Jia, Lili Mou, Ge Li, Yunchuan Chen, Yangyang Lu, Zhi Jin
TL;DR
Relation classification needs models that can move beyond the shallow architectures commonly used by neural approaches. The paper proposes deep recurrent neural networks with directionality-based data augmentation, achieving an 86.1% F1-score on SemEval-2010. Analysis indicates that different layers learn representations at different abstraction levels, while augmentation supports training deeper models.
Problem
Existing neural relation-classification models are usually shallow and may not explore representations at different abstraction levels.
Method
The paper uses DRNNs over shortest dependency paths and augments data by reversing dependency sub-path order according to relation directionality.
Results
86.1% F1-score was achieved on SemEval-2010 with DRNNs and data augmentation.
Takeaways & Limitations
Different DRNN layers learn representations at different abstraction levels, and directionality-based augmentation provides additional samples without external data resources.
Abstract
from arXiv · showhide
Nowadays, neural networks play an important role in the task of relation classification. By designing different neural architectures, researchers have improved the performance to a large extent in comparison with traditional methods. However, existing neural networks for relation classification are usually of shallow architectures (e.g., one-layer convolutional neural networks or recurrent networks). They may fail to explore the potential representation space in different abstraction levels. In this paper, we propose deep recurrent neural networks (DRNNs) for relation classification to tackle this challenge. Further, we propose a data augmentation method by leveraging the directionality of relations. We evaluated our DRNNs on the SemEval-2010 Task~8, and achieve an F1-score of 86.1%, outperforming previous state-of-the-art recorded results.
1 Introduction
Relation classification has benefited from neural networks, but existing models are typically shallow. This paper proposes deep recurrent networks and directionality-based data augmentation, achieving improved SemEval-2010 performance.
- Motivation: Neural relation-classification models are typically shallow, motivating the question of whether deeper architectures are beneficial.Prior neural approaches include one-layer CNNs or RNNs, while deep architectures may support information integration and abstraction.
- Deep recurrent networks: DRNNs classify relations by exploring representation spaces at different abstraction levels and granularities.The model uses the shortest dependency path as its RNN backbone.
- Deep recurrent networks: Different DRNN layers learn different representations: lower layers mix information, while higher layers locate relation-relevant information more precisely.The analysis connects RNN units to the ultimate classification to visualize these layer-specific roles.
- Data augmentation: Directionality-based augmentation reverses the order of two dependency sub-paths to create samples with inverse relationships without external resources.The two sub-paths correspond to subject-predicate and object-predicate components.
- Results: 86.1% F1-score was achieved on SemEval-2010 with DRNNs and augmentation, while performance decreased at excessive depth because larger models require more training samples.Without augmentation, depth-3 DRNNs achieved 84.2% F1-score; augmentation alleviated data sparsity and supported deeper models.
2 Related Work
Prior relation-classification methods include feature-based, kernel-based, recursive, recurrent, convolutional, and hybrid neural models. These approaches use manually designed features, predefined similarities, parse structures, or specialized training objectives.
- Traditional methods: Traditional relation-classification methods are mainly feature-based or kernel-based.Feature-based systems use lexical, syntactic, and semantic features, whereas kernel methods rely on predefined similarity measures.
- Recursive and recurrent models: Recursive neural networks model relations using constituency trees, important phrases, or shortest dependency paths.Prior work progressively restricts recursive models to structures such as the shortest dependency path.
- CNN-based models: CNN-based approaches apply convolutional architectures to relation classification, including ranking-loss and negative-sampling variants.The negative-sampling method is designed for shortest-dependency-path-based CNNs.
- Hybrid models: Hybrid approaches combine CNNs with recursive or recurrent networks.These models represent another neural architecture family for the task.
3 The Proposed Methodology
The proposed methodology combines deep recurrent neural networks over shortest dependency paths with multiple information channels and directionality-based data augmentation. The architecture integrates representations across layers and channels, while augmentation addresses the limited size of the training dataset.
- Overall architecture: DRNNs process shortest dependency paths with two RNNs separated by the marked entities’ common ancestor.The SDP serves as the model backbone for capturing relation-relevant information.
- Overall architecture: Four information channels—words, part-of-speech tags, grammatical relations, and WordNet hypernyms—provide complementary inputs.The channels are processed over the dependency-path substructures.
- Recurrent networks on shortest dependency paths: The shortest dependency path reduces irrelevant information and emphasizes grammatical relations between words, which are suitable for relation classification.The path is used instead of a raw word sequence or whole parse tree.
- Deep recurrent neural networks: The model stacks multiple recurrent hidden layers to capture information at different abstraction levels.Each layer treats the previous layer as input, and cross connections from lower-layer previous states enhance information propagation.
- Data augmentation: The augmentation targets overfitting risk because the SemEval-2010 dataset contains only several thousand samples for training deep RNNs.The method also permits using probabilities for a relation, its inverse, or both during decoding.
- Data augmentation: Directionality-based augmentation swaps the two entity-to-ancestor subpaths to create samples labeled with inverse relations, without external data.For example, Content-Container becomes Container-Content after the subpaths are reordered.
- Deep recurrent neural networks: Max pooling gathers information from recurrent and embedding layers before concatenation, hidden-layer integration, and softmax classification.The implementation uses 40 pooling outputs that are concatenated and fed to a hidden layer.
- Training objective: The training objective combines cross-entropy losses for original and augmented samples with Frobenius-norm regularization.The objective includes the inverse relation r−1 and a regularization coefficient λ.
4 Experiments
Experiments evaluate DRNNs on SemEval-2010 Task 8, examine augmentation choices and CNN/RNN depth, and analyze information propagation across layers. Results show that selective augmentation supports deeper RNNs and improves performance, while deep CNNs hurt performance.
- Dataset: SemEval-2010 Task 8 provides 8,000 training sentences, 2,717 test sentences, 800 validation samples, and 19 labels including directed and Other relations.Official measures exclude the undirected Other class.
- Data augmentation: Augmenting only directed relations improves performance by 1.9%, whereas augmenting all relations degrades it by 0.7% and augmenting only Other degrades it by 1.1%.The authors attribute the degradation to unfavorable noise in the Other class.
- RNNs versus CNNs: Deep CNNs hurt performance in this task, while RNN performance continues increasing through four hidden layers.The authors plausibly relate CNN vulnerability to padding effects on short shortest dependency paths.
- Overall performance: The DRNN model achieves 86.1% F1-score with augmentation and 84.2% without augmentation, with the latter reported as the highest score without special treatment of Other.The results are presented as evidence for contributions from both deep networks and data augmentation.
- Information propagation: Information spreads across a wider range in hidden layers, while higher layers focus more on words relevant to the target relation.Examples show higher-level attention shifting toward entities or their common ancestor depending on the sample.
5 Conclusion
The paper proposes DRNNs and directionality-based data augmentation for relation classification. On SemEval, the model reaches its highest F1-measure of 86.1% at depth 4.
- DRNNs explore representation spaces across different abstraction levels using several RNN layers.The model is designed to integrate information relevant to target relations at higher levels.
- The paper designs data augmentation by leveraging the directionality of relations.
- 86.1% F1-measure is achieved on the SemEval dataset with a depth of 4.