PaperlayerView exact revision on arXiv

Source-linked AI summary

Efficient Transfer Learning Schemes for Personalized Language Modeling using Recurrent Neural Network

Seunghyun Yoon, Hyeongu Yun, Yuna Kim, Gyu-tae Park, Kyomin Jung

arXiv:1701.03578v1cs.CLcs.AI

TL;DR

Personalized LSTM-RNN language modeling is constrained by private data that users do not transfer and the limited computing resources of mobile devices. The paper transfers a general model to a target model using fast schemes, including fixed-first-n layers, and retrains it locally with user data. The authors report similar user-style output and higher generalization performance for surplus-layer and fixed-n schemes than the baseline while training fewer parameters.

  • Problem

    Personalized LSTM-RNN training requires data and computing resources, but users seldom transfer private message data and mobile devices may not have sufficient resources.

    Source: Introduction · p. 1

  • Method

    The paper transfers a general LSTM-RNN to a target model and fine-tunes it with limited user data using schemes such as surplus layers or fixed first n layers.

    Source: Introduction · p. 1, Fast Transfer Learning Schemes · p. 2

  • Results

    Surplus-layer and fixed-n schemes show higher generalization performance than the baseline while training fewer parameters, and the output is similar to the user's style.

    Source: Conclusion · p. 6

  • Takeaways & Limitations

    The transferred model can be retrained in a local environment without sending personal data out of the device.

    Source: Architecture for Personalized Language… · p. 2

    The personalizing-data assumption is illustrated with drama-character corpora of 8,406 and 9,194 lines, which the authors assume could be gathered on a user device.

    Source: Datasets · p. 3

Abstract

from arXiv · show

In this paper, we propose an efficient transfer leaning methods for training a personalized language model using a recurrent neural network with long short-term memory architecture. With our proposed fast transfer learning schemes, a general language model is updated to a personalized language model with a small amount of user data and a limited computing resource. These methods are especially useful for a mobile device environment while the data is prevented from transferring out of the device for privacy purposes. Through experiments on dialogue data in a drama, it is verified that our transfer learning methods have successfully generated the personalized language model, whose output is more similar to the personal language style in both qualitative and quantitative aspects.

Introduction

Personal language modeling with LSTM-RNNs is constrained by private data that users do not transfer and the limited computing resources of mobile devices. The paper proposes fast transfer learning schemes and reports that personalized outputs better match the target user's dialogue than other users' dialogue.

  • Personalized LSTM-RNN training is limited by private message data that users seldom transfer and the computing demands of training deep models on phones.LSTM-RNNs are used for language modeling, but training them requires substantial data and computing power.Source: Introduction · p. 1
  • Fast transfer learning copies the first n layers of a general model into a target model, then fine-tunes it with relatively little user data.The proposed schemes include freezing layers or adding a surplus layer.Source: Introduction · p. 1
  • The personalized model's output is more similar to the same user's dialogue than to other users' dialogue.The experiments evaluate sentence completion and message-reply prediction using literary-style and drama-script data.Source: Introduction · p. 1
  • The paper presents efficient, privacy-preserving transfer learning for RNN-based language models and tests its applicability to short-message reply prediction in a mobile-like setting.The contribution statement characterizes this as the first research on transfer learning for RNN-based language models with privacy preservation.Source: Introduction · pp. 1–2

Architecture for Personalized Language Model

The architecture supports sentence completion and message-reply prediction, with personalizing transfer performed in a small local environment. The paper describes a whole-model retraining baseline and two schemes that update fewer parameters.

Architecture for Personalized Language Model

  • The general model is trained with abundant data in a powerful environment, then retrained locally so personal data need not leave the device.Applying this transfer approach to mobile devices is an assumption stated by the authors.Source: Architecture for Personalized Language… · p. 2

Sentence Completion Language Model

  • The sentence-completion model predicts the next word from a given word sequence and repeats prediction until it emits an end-of-sentence signal.Its described architecture uses a 1,000-dimension word embedding and three LSTM layers with 100 cells each.Source: Sentence Completion Language Model · p. 2

Message-Reply Prediction Language Model

  • The message-reply model encodes the message and uses its last hidden state as context when generating the response sequence.The implementation uses a 1,000-dimension embedding and three LSTM layers with 100 cells each.Source: Message-Reply Prediction Language Model · p. 2

Fast Transfer Learning Schemes

  • The surplus-layer scheme trains only an inserted layer, while fixed-first-n keeps the first n LSTM layers fixed and trains the remaining parameters.The paper also compares both approaches with a baseline that retrains the whole model using private data.Source: Fast Transfer Learning Schemes · p. 2

Measures

Perplexity does not measure how closely generated text matches a target style, and BLEU is not directly applicable to personalized output. The paper uses cross entropy between generated and target word distributions, which is lower for dialogues from the same character than for different characters.

Measures

  • Perplexity measures how well a language model predicts a sample, but not how well its output matches a target language style.Source: Measures · p. 2
  • BLEU is not directly applicable to measuring personalized model-output quality because it compares one language with a target language.Source: Measures · p. 2
  • Cross entropy compares the word distribution of model output with the target corpus, using distributions derived from normalized word-count histograms.For each test message, the language model produces output whose word distribution is compared with that of a target corpus.Source: Measures · p. 3
  • Lower cross entropy is observed for same-character dialogue comparisons than for different-character comparisons, supporting its use as a measure of style similarity.The paper tests the measure on dialogues among the main characters of Friends.Source: Measures · p. 3, Table 1 · p. 3

Datasets

The experiments use formal written, Bible, drama-dialogue, and character-specific corpora to represent different language styles and personalize models. The character corpora contain thousands of dialogue lines and are treated as data that could be gathered on a user device.

Datasets

  • The WMT’14 English corpus provides 21,000,000 sentences of high-quality formal written language from bilingual web pages.Source: Datasets · p. 3
  • The English Bible corpus contains 31,102 sentences and differs from WMT’14 in frequent vocabulary and average sentence length.Source: Datasets · p. 3
  • The Friends drama corpus contains 69,000 dialogue sentences, with adjacent utterances from different characters linked for message-reply training.Source: Datasets · p. 3
  • The Chandler and Rachel corpora contain 8,406 and 9,194 lines, respectively, and are assumed to approximate user data available on a device.Source: Datasets · p. 3

Experiments

The paper evaluates transfer learning in sentence completion and message-reply prediction, using written-to-spoken adaptation and character-specific personalization. The reported results include style changes in generated completions, lower validation perplexity for two proposed schemes than the baseline, and more similar output style after personalization.

Experiments

  • The experiments cover sentence completion from literary-style to spoken-style data and message-reply prediction using drama scripts and character data.Source: Experiments · pp. 3–4

Literary-Style to Spoken-Style Sentence Completion

  • The sentence-completion experiment uses a WMT’14 general model trained for about 10 days, followed by about 4 hours of transfer training for each scheme.The experiments used a Titan-X GPU and a GeForce GT 730 GPU, and the vocabulary contained 50,002 words.Source: Literary-Style to Spoken-Style… · p. 4
  • Personalized completions differ in length and style from the general model, with the general model tending to produce longer sentences than the Friends-personalized model.The paper interprets this difference as reflecting spoken-language characteristics, which are usually briefer than official-document language.Source: Literary-Style to Spoken-Style… · p. 4, Table 2 · p. 4
  • The Bible-personalized model produces output with more Bible-like vocabulary and sentence styles.Source: Literary-Style to Spoken-Style… · p. 4, Table 2 · p. 4

General-Style to Personal-Style Message-Reply Prediction

  • Schemes 2 and 3 continue decreasing in validation perplexity and reach lower values than scheme 1, which begins increasing around epoch 10.The paper attributes scheme 1's increase to overfitting and reports that the proposed schemes save computing power with reduced parameters.Source: General-Style to Personal-Style… · pp. 4–5, Figure 1 · p. 5
  • With scheme 2, perplexity decreases as more personal training sentences are used and beats the Friends 5-gram model from the 2,000-sentence cases.Source: General-Style to Personal-Style… · p. 5, Table 4 · p. 5
  • The proposed schemes outperform the n-gram models with n=3 and n=5 on the validation dataset.Table 3 compares the proposed schemes with Chandler- and Friends-trained n-gram models; scheme 1 is overfit, with the caption reporting a lowest value of 48.17.Source: Table 3 · p. 5, General-Style to Personal-Style… · p. 5
  • Cross entropy is lowest against the Chandler corpus among the schemes, indicating that the personalized model output is more similar in style to its target corpus than the general model.The comparison uses 753 test messages and includes Friends, Chandler, and Bible target corpora.Source: General-Style to Personal-Style… · p. 5

Related Work

Prior work established RNN/LSTM language modeling and extended RNNs to response generation, while transfer learning and mobile-model compression were studied separately. The paper distinguishes its focus on personalizing RNN language models from those approaches.

  • Earlier RNN and LSTM work modeled sequential language and supported machine translation, with qualitative language-modeling examples on web corpora.Source: Related Work · p. 5
  • Neural Responding Machine work applied RNNs to message replies, reporting 40% accuracy for contextually and syntactically proper responses; it trained on pooled data rather than personalizing a general model.Source: Related Work · pp. 5–6
  • Transfer-learning studies proposed general-to-domain model transfer and found lower layers tended toward general features, but did not apply this to RNN language models.Source: Related Work · p. 6
  • Mobile-model studies pruned unnecessary connections using powerful machines before releasing compressed models, whereas this paper investigates training on-device to keep user data private.Source: Related Work · p. 6

Conclusion

The conclusion presents transfer schemes for personalized LSTM-RNN language models that can be trained locally with reduced parameters. The authors report user-style similarity and identify analysis of LSTM cells and language styles as future work.

  • The surplus-layer and fixed-n-layer schemes showed higher generalization performance while training fewer parameters than the baseline.Source: Conclusion · p. 6
  • The proposed schemes generate personalized language models in the user’s local environment to preserve privacy.Source: Conclusion · p. 6
  • The authors conclude that personalization can use less user data than traditional deep neural network training, and plan to study links between user language styles and LSTM cells.Source: Conclusion · p. 6

Related papers

Loading 1701.03578v1…