Source-linked AI summary
How Transferable are Neural Networks in NLP Applications?
Lili Mou, Zhao Meng, Rui Yan, Ge Li, Yan Xu, Lu Zhang, Zhi Jin
TL;DR
Neural-network transfer in NLP remains unclear because prior studies report inconsistent results, despite its relevance for data-limited tasks. The paper conducts systematic experiments across six datasets and evaluates INIT, MULT, and neural-layer transfer. It finds that transferability depends largely on semantic relatedness, with conclusions supported within the tested settings.
Problem
Prior neural NLP transfer studies are limited and inconsistent, leaving unclear how transferable neural networks are across NLP tasks.
Method
The paper conducts experiments across six datasets, evaluates INIT and MULT, and analyzes transferability across neural layers.
Results
Transferability depends largely on source-target semantic relatedness; hidden-layer transfer can slightly degrade performance by 0.2–0.4% for semantically different tasks.
Takeaways & Limitations
In the studied scenarios, semantic relatedness is a key consideration when transferring neural NLP models, unlike the reported pattern in image processing.
Takeaways & Limitations
The evidence covers two neural models, six datasets, and two experiment groups, so generalization is stated only for similar scenarios.
Abstract
from arXiv · showhide
Transfer learning is aimed to make use of valuable knowledge in a source domain to help model performance in a target domain. It is particularly important to neural networks, which are very likely to be overfitting. In some fields like image processing, many studies have shown the effectiveness of neural network-based transfer learning. For neural NLP, however, existing studies have only casually applied transfer learning, and conclusions are inconsistent. In this paper, we conduct systematic case studies and provide an illuminating picture on the transferability of neural networks in NLP.
1 Introduction
The paper asks how transferable neural networks are in NLP, where prior transfer-learning findings are inconsistent. It focuses on semantic relatedness, neural layers, and the INIT and MULT transfer methods.
- Transfer learning uses source-domain knowledge to improve a data-limited target task and mitigate neural-network overfitting.
- Unlike image processing, NLP transferability is unclear because discrete language tokens are more semantically structured than neighboring image pixels.
- Prior neural NLP studies report inconsistent transfer results, ranging from 0.04–0.21% error reduction to an accuracy increase from 71.3% to 80.8%.
- The study distinguishes transfer between semantically similar or different tasks and investigates transferability across neural layers.
- It compares parameter initialization (INIT) with multi-task learning (MULT), including their combination.
2 Datasets
The experiments use six open datasets organized into sentence-classification and sentence-pair-classification groups. Transfer settings compare semantically equivalent tasks with semantically different tasks while keeping neural topology aligned.
- The study conducts two experiment series using six open datasets, with large datasets as sources and small datasets as targets.
- Experiment I: Sentence classification: Experiment I covers sentence classification, including sentiment and question-type objectives.
- Experiment II: Sentence-pair classification: Experiment II covers sentence-pair classification, including entailment, contradiction, neutrality, and whether two sentences share the same meaning.
- Transfer scenarios: Semantically equivalent transfers are IMDB→MR and SNLI→SICK, whereas semantically different transfers are IMDB→QC and SNLI→MSRP.
- Transfer scenarios: The paper argues that NLP transferability is more sensitive to semantic relatedness than transfer in image processing.
3 Neural Models and Settings
The study uses unified CNN and LSTM-RNN architectures across datasets so transfer can be examined independently of task semantic equivalence. Baselines are established for fair comparison rather than state-of-the-art performance.
- Each experiment group uses one neural model across three datasets, keeping the architecture the same for transfer analysis.
- Neural architectures: Experiment I uses an LSTM-RNN with a softmax classifier applied to the last word’s hidden state.
- Neural architectures: Experiment II uses a Siamese CNN with window size 5, max pooling, concatenated sentence vectors, and a hidden-layer softmax output.
- Training settings: Embeddings are pretrained with word2vec, while embeddings and hidden layers are 100 dimensional.
- Training settings: The models are compared against no-transfer baselines trained with five random parameter initializations and reported as comparable to related models.
- Training settings: The goal is a fair comparison of transfer-learning methods and settings, not state-of-the-art performance.
4 Transfer Methods
The paper studies INIT, MULT, and their combination as two ways to transfer source-domain knowledge into target-task training. INIT transfers pretrained parameters, while MULT jointly trains on both domains with a weighted objective.
- INIT: INIT trains on source data, uses the tuned parameters to initialize the target model, and may fine-tune them with target labels.
- MULT: MULT simultaneously trains source- and target-domain samples using gradient information from both domains.
- Practical considerations: Gradient-based optimization of the MULT objective is nontrivial because learning-rate choices can cause instability or local-optimum trapping.
- Combination: The experiments also evaluate a combination that first uses source pretraining for initialization and then jointly trains on source and target data.
- Method rationale: MULT can regularize the target error surface, whereas INIT supplies meaningful starting parameters without changing that surface.
5 Results of Transferring by INIT
INIT transfer succeeds for semantically equivalent tasks but generally fails for semantically different ones, with transferability varying substantially by layer and training stage. The output layer is dataset-specific, embeddings can transfer across semantic differences, and early transfer may outperform transfer after source convergence.
- Overall Performance: Approximately 6% improvement appears when transferring between semantically equivalent tasks, IMDB→MR and SNLI→SICK.This result is reported for INIT using parameters selected by highest source validation accuracy.
- Overall Performance: For semantically different tasks, transferring hidden layers provides no improvement, while freezing transferred features produces very low target performance.Training accuracies under locked settings were 48.2% for QC and 65.5% for MSRP.
- Overall Performance: Transferability in NLP is more sensitive to task semantics than in image processing, and prior positive evidence may have been driven mainly by word embeddings.In Collobert and Weston (2008), NER, POS, and CHK contributed only 0.04–0.21% accuracy improvement when pretrained embeddings were already used.
- Layer-by-Layer Analysis: The output layer is mainly dataset-specific: freezing it reduces performance, whereas randomly reinitializing it can match or exceed the baseline.Transferring output-layer parameters yields little, if any, gain.
- Layer-by-Layer Analysis: In IMDB→MR, embeddings and hidden layers each improve accuracy by 3%, whereas SNLI→SICK gains mainly from the hidden layer.The paper relates this difference to sentiment lexicons versus semantic compositionality, respectively.
- Layer-by-Layer Analysis: For semantically different tasks, embeddings are the only observed transferable parameters, benefiting the target by 2.7x and 1.8x standard deviations.The reported benefits correspond to IMDB→QC and SNLI→MSRP, respectively.
- How does learning rate affect transfer?: A large learning rate accelerates target training without necessarily damaging transferred knowledge, although validation may still be needed when resources allow.With α = 0.3, accuracy rises quickly and peaks earlier; α = 0.01 trains slowly but can reach comparable peak performance after about 100 epochs.
- When is it ready to transfer?: The best transfer stage need not coincide with the source model’s best validation performance: SNLI→SICK peaks at 78.98% after epoch 2, versus 72.65% source accuracy then and 76.26% at epoch 23.Later transfer performance decreases gradually by approximately 2.7%, while IMDB→MR shows better alignment between source and transfer performance.
6 MULT, and its Combination with INIT
Multi-task transfer is generally comparable to parameter initialization, while combining the two does not provide additional peak-performance gains. Transfer benefits depend on semantic relatedness between source and target datasets.
- Experimental setup: λ balances source and target tasks, with λ = 1 using only T and λ = 0 using only S.The experiments first tuned λ at 0.1 granularity and used finer tuning from 0.01 to 0.09 in selected experiments.
- Results: MULT does not improve target-task accuracy for QC and MSRP, whereas transfer is consistently positive for MR and SICK.These results support semantic relatedness as a major determinant of neural NLP transferability.
- Results: MULT’s peak performance is slightly below INIT in Experiment I but higher in Experiment II, placing the methods in the same ballpark.
- MULT+INIT: MULT+INIT remains effective across different λ values and consistently outperforms non-transferring by a large margin.The pretrained source-task parameters provide sufficient source information for multi-task training.
- MULT+INIT: Combining MULT and INIT yields no further gain because its peak performance is not higher than either MULT or INIT.
7 Concluding Remarks
The paper presents systematic evidence that neural NLP transferability depends largely on source–target semantic relatedness, while acknowledging that empirical conclusions are scenario-dependent. Its experiments provide insight into neural model transferability but motivate further work on robust multi-task learning.
- Conclusions: Experiments on six datasets show that neural NLP transferability depends largely on the semantic relatedness of source and target tasks.
- Conclusions: The study analyzes neural layers and evaluates parameter initialization and multi-task learning as transfer methods.
- Scope: The conclusions may vary with models, tasks, and datasets, although results were mostly consistent across two experiment groups, six datasets, and two neural models.The authors note an exception in Section 5.4.
- Conclusions: The authors consider the work generalizable to similar scenarios.
- Future work: Future work includes analyzing different MULT strategies and developing an effective, robust multi-task learning method.