PaperlayerView exact revision on arXiv

Source-linked AI summary

Transfer Learning for Speech and Language Processing

Dong Wang, Thomas Fang Zheng

arXiv:1511.06066v1cs.CLcs.LG

TL;DR

Speech and language processing needs transfer learning because its data are diverse, imbalanced, and dynamic. The paper reviews transfer across languages, speakers, and models, and reports results showing broad utility, including faster neural document-vector inference and up to 26% NER improvement. The review remains limited in scope and leaves many transfer-learning questions open.

  • Problem

    Speech and language data are diverse, imbalanced, dynamic, and inter-linked, while existing transfer-learning work lacks a unified understanding across methods and conditions.

  • Method

    The paper reviews prominent deep-learning transfer-learning applications in speech recognition and language processing, focusing on transfer across languages, speakers, and models.

  • Results

    The review reports transfer-learning results including up to 26% performance improvement for multilingual NER models and document vectors with hundreds-of-times-faster inference than LDA.

  • Takeaways & Limitations

    Transfer learning is presented as an important and rapidly evolving approach for speech and language processing.

  • Takeaways & Limitations

    The review touches only limited areas, omits many historical contributions, and focuses especially on recent deep-learning directions.

Abstract

from arXiv · show

Transfer learning is a vital technique that generalizes models trained for one setting or task to other settings or tasks. For example in speech recognition, an acoustic model trained for one language can be used to recognize speech in another language, with little or no re-training data. Transfer learning is closely related to multi-task learning (cross-lingual vs. multilingual), and is traditionally studied in the name of `model adaptation'. Recent advance in deep learning shows that transfer learning becomes much easier and more effective with high-level abstract features learned by deep models, and the `transfer' can be conducted not only between data distributions and data types, but also between model structures (e.g., shallow nets and deep nets) or even model types (e.g., Bayesian models and neural models). This review paper summarizes some recent prominent research towards this direction, particularly for speech and language processing. We also report some results from our group and highlight the potential of this very interesting research field.

I. INTRODUCTION

Transfer learning reuses auxiliary resources to improve target-task learning, addressing the diversity and imbalance of speech and languages. This review surveys promising deep-learning approaches, focusing on speech recognition and document classification.

  • Transfer learning uses auxiliary data, models, or labels to enhance learning for a target task.
  • Speech and language diversity, variation, imbalance, and dynamics make learning from multiple resources necessary.
  • Prior work spans speaker adaptation, multilingual speech recognition, cross-language document classification, and sentiment analysis.
  • The paper reviews promising transfer-learning methods, especially recent deep-learning approaches, and presents results from the authors’ research team.
  • The review focuses on speech recognition and document classification rather than enumerating all transfer-learning methods.

A. Categories of transfer learning

Transfer learning methods are categorized by how source and target data and tasks relate. The taxonomy uses feature spaces, feature distributions, label spaces, and models as conditional factors.

  • Transfer learning reuses source-domain experience or knowledge to improve learning in a target domain.
  • The paper treats multitask learning, lifelong learning, knowledge transfer, model adaptation, concept drift, and covariance shift as transfer-learning implementations under different conditions.
  • The taxonomy represents data through feature space X and distribution P(X), and tasks through label space Y and model M(x).
  • Source–target categories depend on whether conditional factors are shared or different, with ‘+’ denoting same and ‘-’ denoting different.
  • Table I provides a rough categorization because it does not clearly reflect whether source and target data are labelled.

1) Model adaptation and incremental training:

Transfer can address changing data distributions, heterogeneous feature spaces, and differing tasks through adaptation, correspondence learning, common representations, or analogy-oriented methods.

  • 1) Model adaptation and incremental training:: Adaptation may be supervised or unsupervised; unsupervised adaptation generates labels with the source model before updating.
  • 2) Heterogeneous transfer learning:: Heterogeneous transfer assumes correspondence between different source and target feature domains, such as speech and text.
  • 2) Heterogeneous transfer learning:: Correspondence can be established through pivot words, instance-level co-occurrence probabilities, or learned common representations.
  • 2) Heterogeneous transfer learning:: Most discussed approaches assume that the source and target label spaces remain unchanged; differing tasks are substantially more difficult.

3) Multiview co-training:

Multiview co-training uses paired source and target features during training while retaining only target features at runtime. Multitask learning instead transfers across related tasks with different labels.

  • 3) Multiview co-training:: Co-training trains separate source- and target-feature models, alternately generating labels for unlabelled data and updating the other model.
  • 3) Multiview co-training:: Multiview co-training assumes each training instance has source and target features, but only target features are available at runtime.
  • 3) Multiview co-training:: Model transfer uses a source model with the same feature and label spaces to initialize or improve a different target model.
  • 4) Model transfer:: Multitask learning applies when feature spaces match but task labels differ, using task relatedness for mutual regularization.
  • 5) Multitask learning:: A key multitask-learning issue is determining whether two tasks are relevant enough to learn together.

C. Transfer learning in deep learning era

Deep learning reframes transfer learning as the transfer of robust, high-level representations across domains and tasks. Its architecture separates joint representation learning from target-domain adaptation and can support transfer even without adaptation data.

  • Deep representation learning: Deep models learn multi-level features that are robust to variation, share parameters hierarchically, and can be fine-tuned for specific tasks.Feature learning can also use unlabelled data, reducing supervision requirements.
  • Deep representation learning: Representation learning unifies many transfer-learning approaches by learning high-level features shared across multiple data sources and tasks.Knowledge and models are transferred through shared features rather than only through direct parameter reuse.
  • Architecture: The architecture jointly projects heterogeneous source inputs into a common semantic space, then updates the model using target data that may be labelled or unlabelled.The target distribution may differ from the original target-domain distribution used during joint training.
  • Architecture: The framework covers multitask learning, structural correspondence learning, latent representation learning, and conventional model adaptation under different training and adaptation configurations.Its feature extractor can also be trained in an unsupervised manner.
  • Examples: Unsupervised denoising-autoencoder features enabled sentiment models trained on book reviews to transfer directly to DVD reviews and outperform competing approaches.The result supports the domain-independence of high-level abstract features.
  • Examples: Very few labelled examples can adapt general high-level features to recognize unseen objects, while task vectors can support zero-data and zero-shot learning for related tasks.The task vector encodes task information and helps estimate relationships among tasks.

III. TRANSFER LEARNING IN SPEECH PROCESSING

Speech transfer learning addresses variation across languages, speakers, and models by exploiting shared structure and learned representations. In multilingual speech processing, shared DNN layers and task-related supervision support low-resource recognition and enhancement.

  • Scope: Speech transfer learning is organized around transfer across languages, speakers, and models because speech varies with language, speaker, channel, environment, and emotion.The review selects these three areas as prominent applications.
  • Cross-lingual and multilingual transfer: Shared linguistic patterns and universal phone sets strengthen multilingual models, especially for low-resource languages, compared with monolingual training.The review discusses this advantage in speech recognition and speech enhancement.
  • Cross-lingual and multilingual transfer: Multilingual DNNs share lower layers across languages while retaining language-specific higher layers, reflecting increasingly language-dependent representations at depth.This design implements transfer through a shared feature extractor and language-specific classifiers.
  • Cross-lingual and multilingual transfer: In tandem systems, multilingual DNNs produce posterior or bottleneck features that are used to train conventional GMM-HMM systems.The features are described as relatively language-independent.
  • Cross-lingual and multilingual transfer: Hybrid and tandem multilingual systems can be combined by adding multilingual bottleneck features to original inputs or combining bottleneck modules.These configurations retain both learned features and hybrid acoustic modeling.
  • Multitask learning: Multitask training of phone and grapheme recognition improved performance in three low-resource South African languages.The work also compared multitask training with conditional training.
  • Multitask learning: Learning speech content and speaker accent jointly improved English ASR for both British and Chinese accents.When accent information is used as an auxiliary output, the pronunciation vector is needed only during training.
  • Speech enhancement: An English-trained DNN enhancement model was demonstrated to be effective for Chinese speech because noise and reverberation were assumed to be largely language-independent.The transfer is applied directly across languages.

B. Speaker adaptation

Speaker adaptation for deep networks is difficult because distributed parameters lack a simple low-dimensional adaptation form. Proposed solutions add speaker representations or constrain which network components are updated, while adaptation gains depend on model and data scale.

  • Motivation: Adapting deep DNNs is difficult because their parameters form a compact distributed model that cannot be readily updated by simple transformations such as MLLR.Recent work shows adaptation is possible under structural constraints.
  • Speaker-vector methods: Speaker codes can be learned with the network and later inferred for a new speaker while the network remains fixed.The code may be augmented to input and hidden layers, with transform matrices providing an extension.
  • Speaker-vector methods: Externally produced speaker vectors, including i-vectors, need not be adapted and can be trained unsupervised on large unlabelled databases.Their disadvantage is that they do not incorporate phone information.
  • Direct model adaptation: Direct DNN adaptation constrains updates to selected components such as the input layer, hidden activations, or output layer.The review notes comparisons among different adaptation locations.
  • Recurrent models: LSTMs can be adapted using speaker-adaptive front ends or speaker-dependent layers.This extends constrained adaptation beyond DNN architectures.
  • Empirical scope: Adaptation methods improved performance for small networks with medium training sets, but gains were insignificant for large networks trained on large datasets.Large multi-speaker DNNs already model speaker variation relatively well.
  • Related applications: The techniques also transfer to speech synthesis, where MAP and MLLR adaptation can produce speaker-specific voices and support multilingual synthesis.The cited applications include HMM-based and DNN-based speech synthesis.

C. Model transfer

Model transfer trains a child model using knowledge from a teacher model, including transfers from complex to simple models and, in the authors’ work, from weaker to stronger models. It can also serve as a pre-training strategy.

  • Model transfer: Model transfer learns a child model from an existing teacher model whose learned knowledge guides the child’s training.The approach extends the earlier multitask-learning idea into deep-learning settings.
  • Model transfer: Teacher outputs or hidden units can regularize simpler child models, helping them approach complex-model performance for applications such as mobile speech recognition.Examples include transferring from complex DNNs or RNNs to smaller models and FitNet’s hidden-unit regularization.
  • Model transfer: A weak DNN can teach a stronger, more complex RNN, showing that model transfer is not limited to complex-to-simple transfers.The authors report that the RNN can be learned effectively with DNN regularization.
  • Pre-training: Model-transfer pre-training worked in scenarios where RBM and layer-wise discriminative pre-training did not, and combining the two pre-training methods provided additional gains when data were not abundant.These findings are reported for the authors’ experimental setting.

IV. TRANSFER LEARNING IN LANGUAGE PROCESSING

Language-processing transfer learning mainly addresses cross-lingual and cross-domain reuse by constructing shared representations or correspondences between languages, while recent methods increasingly use embeddings and deep models.

  • Cross-lingual and multilingual transfer learning: Cross-lingual transfer reuses knowledge across languages by constructing correspondences based on shared semantic structures.The review describes bilingual lexicons, sentence or document embeddings, shared latent topics, clusters, and projections as correspondence mechanisms.
  • Cross-lingual and multilingual transfer learning: Bilingual lexicons can translate a classifier trained in one language for use in another, including English-to-Chinese document classification.The same strategy was also applied to multilingual language modeling, where foreign words must be addressed.
  • Cross-lingual and multilingual transfer learning: Shared latent structures allow multilingual models to learn common topics or word clusters without requiring word alignment.Multilingual LDA uses a common latent topic space, while multilingual word clusters combine monolingual data with parallel-data correspondence.
  • Cross-lingual and multilingual transfer learning: 26% performance improvement was observed on NER with a multilingual model using cross-lingual word clusters.A later extension used the clusters to directly transfer an NER model from the source language to the target language.
  • Cross-lingual and multilingual transfer learning: Sentence and document embeddings transfer knowledge using paired texts without word alignment, and can support multilingual classification and translation.Noise-contrastive training and joint use of monolingual and bilingual data are described as related extensions.
  • Cross-lingual and multilingual transfer learning: Neural and projection-based methods map representations from different languages into common spaces for classification or direct model transfer.Examples include linear or orthogonal word-vector transforms, heterogeneous feature augmentation, and deep features transformed using parallel data.

B. Cross-domain transfer learning

Cross-domain transfer learning in this review concerns differences in feature modalities, especially transferring between visual and textual representations through feature combination, correspondence estimation, or shared latent spaces.

  • Cross-domain transfer learning: Feature-domain transfer differs across modalities such as audio and image features, whereas application domains differ primarily in data distributions.The review focuses on transfer between different feature modalities rather than application-domain distributions.
  • Cross-domain transfer learning: Multimodal systems can combine heterogeneous features at the feature level or combine scores from models trained on different features.The review gives visual-textual semantic relatedness as an example of both strategies.
  • Cross-domain transfer learning: Co-occurrence data can estimate image-text correspondence, enabling target features to be transferred into a source domain with a strong classifier.The approach formulates transfer with a Markov chain and risk-minimization inference and was tested on text-aided image classification.
  • Cross-domain transfer learning: Kernelized canonical correlation analysis maps visual and textual representations into a shared latent semantic space for image segmentation and labelling.This is presented as a common-latent-space approach to cross-domain transfer.
  • Cross-domain transfer learning: Neural networks can embed images and words in one low-dimensional space, allowing word embeddings to improve image classification even without image training data for some classes.Deep learning is presented as a way to learn high-level representations shared across modalities.

C. Model transfer

Model transfer applies knowledge distillation or cross-model supervision to transfer information between models with different sizes, representations, or underlying modeling paradigms.

  • Model transfer: Knowledge distillation transfers information from a large neural sentiment classifier to a smaller network through outputs or word vectors.The smaller model can use the large network's output as supervision or directly receive its large word-vector representations.
  • Model transfer: A two-layer neural network learned LDA-like document vectors from raw TF inputs, with inference hundreds of times faster than LDA.The neural model was supervised by document-vector representations produced by the Bayesian LDA model.

V. PERSPECTIVE AND CONCLUSIONS

The review concludes that transfer learning is increasingly important for diverse, imbalanced, dynamic, and inter-linked speech and language data, while many mechanisms and applications remain unresolved.

  • Perspective and conclusions: The review surveys applications and argues that transfer learning is essential because speech and language are diverse, imbalanced, dynamic, and inter-linked.The paper presents this conclusion after a deliberately limited review emphasizing recent deep-learning directions.
  • Perspective and conclusions: The review covers only limited areas and omits many historical contributions, especially while emphasizing recent deep-learning research.It also calls for collaboration across researchers working on speech, language, visual, and other heterogeneous resources.
  • Perspective and conclusions: Transfer learning can share learning across domains and tasks or migrate a model from one setting to another.It may use supervised labeled data or unsupervised unlabeled data, and transfer instances, representations, structures, or models.
  • Perspective and conclusions: Deep learning appears to substantially reduce the risk of negative transfer by using related-domain data to learn abstract features.The review contrasts this with earlier settings in which data from a different distribution could not be used as flexibly.
  • Perspective and conclusions: Transfer learning can improve speed, generalization, and intelligibility, although the review presents these as broad potential benefits.The conclusion also states that internal representations, parameters, and features can all be transferred.
  • Perspective and conclusions: Researchers still lack clear answers about what transfer changes internally and when transfer will help or hurt.The review identifies model-dependent inspection and incomplete prediction of transfer outcomes as open questions.
Loading 1511.06066v1…