Source-linked AI summary
Deep Residual Learning for Weakly-Supervised Relation Extraction
Yi Yao Huang, William Yang Wang
TL;DR
Relation extraction faces a lack of labeled training data, while the effects of residual learning and deeper CNNs on noisy NLP tasks were not well understood. The paper applies residual learning to CNNs for distantly supervised relation extraction. It reports state-of-the-art performance and finds that identity mapping can improve performance even with nine CNN layers.
Problem
Relation extraction lacks labeled training data, and it was unclear whether deeper CNNs could distill signals from noisy distant-supervision inputs.
Source: §1 · p. 1Method
The paper designs a deep CNN for distantly supervised relation extraction using residual learning, identity mapping, and shortcut feedback.
Source: §1 · pp. 1–2Results
The authors report state-of-the-art performance and significant improvement from identity mapping with nine CNN layers in noisy relation extraction.
Source: §1 · pp. 1–2, §4 · p. 5Takeaways & Limitations
The results suggest that deeper CNNs can have positive effects on noisy NLP problems, including distantly supervised relation extraction.
Source: §4 · p. 5
Abstract
from arXiv · showhide
Deep residual learning (ResNet) is a new method for training very deep neural networks using identity map-ping for shortcut connections. ResNet has won the ImageNet ILSVRC 2015 classification task, and achieved state-of-the-art performances in many computer vision tasks. However, the effect of residual learning on noisy natural language processing tasks is still not well understood. In this paper, we design a novel convolutional neural network (CNN) with residual learning, and investigate its impacts on the task of distantly supervised noisy relation extraction. In contradictory to popular beliefs that ResNet only works well for very deep networks, we found that even with 9 layers of CNNs, using identity mapping could significantly improve the performance for distantly-supervised relation extraction.
1 Introduction
Relation extraction lacks labeled training data, and it was unclear whether deeper CNNs could extract useful signals from noisy distant-supervision inputs. This paper introduces residual learning for distantly supervised relation extraction and reports improved performance with moderately deep CNNs.
1 Introduction
- Earlier neural relation-extraction models were typically shallow, leaving unclear whether deeper CNNs could distill signals from noisy distant-supervision inputs.Distant supervision uses knowledge-base facts to select noisy instances from unlabeled data.Source: §1 · p. 1
- The paper introduces residual learning for weakly supervised relation extraction and proposes identity mapping with shortcut feedback as applicable to CNN variants.The authors describe incorporating word and position embeddings into a deep residual network.Source: §1 · pp. 1–2
- The authors report that deep residual CNNs outperform CNNs by a large margin and achieve state-of-the-art performance for distantly supervised relation extraction.They also report that even a moderately deep CNN can substantially improve over vanilla CNNs.Source: §1 · pp. 1–2
2 Deep Residual Networks for Relation Extraction
The section builds ResCNN for distantly supervised relation extraction by combining word and entity-relative position embeddings with convolutional features and residual shortcuts. Max pooling and a softmax layer turn those features into relation probabilities.
2 Deep Residual Networks for Relation Extraction
- ResCNN is presented as a deep residual architecture for distantly supervised relation extraction.Figure 1 depicts the model architecture.Source: §2 · p. 2, Figure 1 · p. 3
2.1.2 Position Embeddings
- Each word is represented by concatenating its word embedding with position embeddings for its distances to the two target entities.The relative distances are mapped to real-valued vectors using position-embedding matrices; distances beyond selected minimum and maximum values are bounded.Source: §2.1 · p. 2, §2.1.2 · p. 2
2.1.1 Word Embeddings
- The model represents each word with a real-valued vector and encodes these representations in a word-embedding matrix over a fixed vocabulary.Source: §2.1.1 · p. 2, §2.1 · p. 2
2.1 Vector Representation
- For each sentence, the model concatenates the word and position embeddings of all words into a vector representation.Source: §2.1.2 · p. 2, §2.1 · p. 2
2.2 Convolution
- A convolutional filter applies to a window of h concatenated word vectors to generate a feature, producing a sequence of features across the sentence.The filter is applied at each possible window; the output sequence has length n − h + 1.Source: §2.2 · p. 2
2.3 Residual Convolution Block
- Each residual block uses two convolutional layers followed by ReLU, with same-size padding and shortcut connections that add the block input to its output.The architecture repeats these blocks.Source: §2.3 · p. 2
2.4 Max Pooling, Softmax Output
- Max pooling selects each filter’s highest feature, and the resulting features feed a fully connected softmax layer that predicts relation probabilities.Dropout masks the pooled feature vector during training; at test time, weights are scaled by the keep probability and used without dropout.Source: §2.4 · pp. 2–3, Figure 1 · p. 3
3 Experiments
Experiments on NYT-Freebase compare CNN variants and show that residual shortcuts improve deeper CNNs despite overfitting without them. ResCNN-9 outperforms models without instance selection and is on par with PCNN+ATT, while ResCNN-9 dominates at small P@N.
3.1 Experimental Settings
- The model comparisons include CNN-B, CNN+ATT, PCNN+ATT, CNN, deeper CNN-x variants, and ResCNN-x variants with residual identity shortcuts.CNN-x uses x convolutional layers; ResCNN-x adds residual identity shortcuts.Source: §3.1 · p. 3
- The experiments use the NYT-Freebase dataset, which has 522,611 training sentences and 172,448 test sentences; the authors describe it as suitable for training deeper CNNs.Evaluation uses held-out precision/recall curves and P@N.Source: §3.1 · p. 3, §3.2 · pp. 3–4
3.2 NYT-Freebase Dataset Performance
- ResCNN-9 beats CNN-B and CNN and dominates the precision/recall curve overall, while CNN-9 performs much worse than CNN.The authors report overfitting when adding layers without residual learning.Source: §3.2 · p. 4, Figure 2 · p. 4
- ResCNN-9 outperforms all models that do not select training instances and is on par with PCNN+ATT, even without piecewise max-pooling or instance-based attention.Table 2 reports P@N comparisons across models.Source: Table 2 · p. 4, §3.2 · p. 4
- ResCNN-5 performs worse than CNN-5, but ResCNN-9 and ResCNN-13 significantly improve over CNN-5 and ResCNN-5 as depth increases.The authors report that even a 9-layer CNN can benefit from identity mapping in noisy input settings.Source: §3.2 · p. 4
- ResCNN-9 dominates performance at small P@N values (N = 1, 5, 10, 20, 50), where the authors assess predictions in the higher-probability range.The comparison is reported in Table 3.Source: §3.2 · p. 4, Table 3 · p. 5
4 Conclusion
The paper concludes that deeper convolutional models can distill signals from noisy inputs in distantly supervised relation extraction. Shortcut connections and identity mapping significantly improve performance.
4 Conclusion
- Deeper convolutional models help distill signals from noisy inputs, and shortcut connections with identity mapping significantly improve performance.The conclusion says these findings align with a recent study suggesting deeper CNNs have positive effects on noisy NLP problems.Source: §4 · p. 5