Source-linked AI summary
Shakespearizing Modern Language Using Copy-Enriched Sequence-to-Sequence Models
Harsh Jhamtani, Varun Gangal, Eduard Hovy, Eric Nyberg
TL;DR
Automating stylistic transformation is needed because manual adaptation is tedious, and this paper targets Modern English to Shakespearean English conversion. It combines a pointer-based sequence-to-sequence model with dictionary-constrained embeddings, demonstrating gains over baselines.
Problem
Manual transformation to a desired style is tedious, while automated solutions for quickly adapting text to different styles remain scarce.
Method
The paper uses a sentence-level sequence-to-sequence neural model with a pointer network for copying input words and dictionary-based retrofitting of pre-trained embeddings.
Results
Copy models outperform their SimpleS2S counterparts by at least 7-8 BLEU points, demonstrating effectiveness over baselines.
Takeaways & Limitations
Input copying and dictionary constraints are useful for stylistic transformation with shared but non-identical source-target sides and sparse parallel data.
Takeaways & Limitations
The paper demonstrates transformation only to Shakespearean English; other stylistic variations remain to be explored.
Abstract
from arXiv · showhide
Variations in writing styles are commonly used to adapt the content to a specific context, audience, or purpose. However, applying stylistic variations is still by and large a manual process, and there have been little efforts towards automating it. In this paper we explore automated methods to transform text from modern English to Shakespearean English using an end to end trainable neural model with pointers to enable copy action. To tackle limited amount of parallel data, we pre-train embeddings of words by leveraging external dictionaries mapping Shakespearean words to modern English words as well as additional text. Our methods are able to get a BLEU score of 31+, an improvement of ~6 points above the strongest baseline. We publicly release our code to foster further research in this area.
1 Introduction
The paper addresses automated transformation from Modern English to Shakespearean English, a style-adaptation task that is difficult because the registers, vocabulary, grammar, senses, orthography, and domains differ. It proposes an end-to-end neural approach using copying and dictionary-informed embeddings.
- Automating stylistic adaptation matters because manually transforming text for different audiences and purposes is tedious.
- The task transforms Modern English into Shakespearean-style English, using “Original” for Shakespearean sentences and “Modern” for their paraphrases.
- The two styles use diachronically disparate registers, with Early Modern English differing through vocabulary, grammar, semantic senses, and orthography.
- A sentence-level sequence-to-sequence neural model with a pointer network enables direct copying of words from the input.
- Dictionary mappings between Shakespearean and modern words retrofit pre-trained embeddings, helping the model perform with limited parallel data.
2 Dataset
The dataset pairs line-by-line Shakespearean sentences with Modern English paraphrases across selected plays, with separate training, validation, and test splits. Examples compare outputs from copying, non-copying, and statistical models.
- The dataset contains line-by-line Modern English paraphrases for 16 of Shakespeare’s 36 plays.The source is the educational site Sparknotes.
- The training split contains 18,395 sentences from 14 plays, while validation uses 1,218 Twelfth Night sentences and testing uses 1,462 Romeo and Juliet sentences.
- Examples: Table 1 compares Modern paraphrases, Shakespearean originals, and outputs from COPY, SIMPLES2S, and STAT models.
- Examples: Direct copying from the source helps the Copy models generate better outputs than the SimpleS2S models in the shown examples.
- The Original side has longer sentences and a larger vocabulary than the Modern side, with slightly higher entropy in its word-frequency distribution.
3 Method Overview
The system encodes a Modern English sentence with a bidirectional LSTM and decodes it using a mixture of an RNN vocabulary model and a pointer network.
- A bidirectional LSTM encodes the input Modern English sentence.
- The decoder combines an RNN that predicts vocabulary words with a pointer network that predicts words from the input.
- The two decoder probabilities are weighted together, with weights computed from the previous decoder hidden state and encoder outputs.
- The input and output are token sequences, with the input length denoted T_enc and the output length denoted T_dec.
4 Token embeddings
The model uses shared vocabulary embeddings for Modern and Shakespearean tokens, pretrained with task text, optional external text, and dictionary constraints. Several embedding strategies are evaluated, including plain, PTB-augmented, and dictionary-based pretraining.
- Vocabulary and pretraining: The encoder and decoder use embeddings over the union of Modern and Shakespearean vocabularies, with sharing between them in the best-performing setting.Shared representations exploit tokens common to both vocabularies.
- Vocabulary and pretraining: Embeddings are pretrained on training sentences, optionally adding PTB text and a Shakespearean-to-Modern dictionary.The external resources provide additional text and cross-style token information.
- Embedding strategies: Plain pretraining trains embeddings on the union of Modern and Original training sentences without additional data.This is the simplest pretraining strategy.
- Embedding strategies: PTB pretraining adds all PTB sentences to the training split when learning word embeddings.The procedure uses external text in addition to the paper’s parallel training sentences.
- Embedding strategies: Retrofitting incorporates dictionary constraints into pretrained embeddings by mapping Shakespearean words to modern English words.The final dictionary contains 1524 pairs, and the authors use an off-the-shelf retrofitting implementation.
- Embedding strategies: Fixed pretrained embeddings outperform trainable embeddings in the reported experiments.The fixed-versus-trainable comparison is motivated by overfitting concerns with limited supervised data.
5 Method Description
The proposed model encodes modern-English input with a bidirectional LSTM and decodes with a mixture of an attentional RNN and a pointer network. Shared attention and a learned mixture allow vocabulary generation and direct input copying.
- Encoder model: The encoder represents the input with forward and reverse recurrent states, combined by addition rather than concatenation to avoid extra parameters.The parameter-saving choice is intended for the paper’s low-data setting.
- Attention: At each decoder step, attention uses the previous decoder state, encoder states, and a sentinel vector.The sentinel attention weight controls the contribution of the decoder RNN to output probabilities.
- Pointer model: The pointer module uses location-based attention to copy tokens directly from the input sequence.This addresses vocabulary overlap, proper nouns, and rare words that a sequence-to-sequence model may not predict.
- Output prediction: The decoder RNN and pointer module share attention weights while producing distributions over vocabulary words and input words, respectively.Their probabilities are combined at every decoder step.
- Output prediction: Training feeds the ground-truth previous output, whereas test-time prediction feeds the previously predicted output.This distinguishes teacher-forced training from autoregressive inference.
- Output prediction: The final token probability is a weighted sum of decoder-LSTM and pointer-model probabilities.The pointer probability is nonzero only for words occurring in the input; setting g = 0 yields the SimpleS2S model.
6 Loss functions
The model is trained with cross-entropy loss over decoder-time token distributions and additionally evaluated with a supervised-attention sentinel loss.
- Training objective: Cross-entropy loss trains the model using predicted token distributions at each decoder time step.The loss is defined over data pairs (x, y) and vocabulary words.
- Training objective: Sentinel loss provides an additional supervised-attention objective derived from the sentinel mechanism.The experiments report the effect of adding sentinel loss as +SL.
7 Experiments
Experiments use Shakespearean–modern paraphrases from 16 plays, compare against copying, dictionary, and SMT baselines, and evaluate both BLEU and PINC. PINC measures paraphrase novelty but not fluency.
- Dataset: The preprocessing lowercases and tokenizes sentences with NLTK’s PUNKT tokenizer, while mapping æ to modern character equivalents such as ae.These steps normalize historical character forms before modeling.
- Baselines: The input-replication baseline achieves BLEU of 21+ because copying is valid when both source and target are English.This baseline is called As-it-is in the comparison discussion.
- Baselines: Dictionary word-by-word replacement performs worse than As-it-is because it replaces aggressively without context and misses one-to-many mappings and long-range dependencies.The dictionary is augmented with common second-person forms such as thou, thy, and thyself.
- Baselines: SMT baselines are trained with MOSES and GIZA++, using a 4-gram target language model and MERT tuning on validation data.The experiments also consider dictionary and PTB information during SMT training and report the best approach.
- Evaluation: BLEU is the primary evaluation metric, while PINC evaluates how much generated paraphrases differ from their source.PINC uses n-grams up to length 4; higher values indicate greater novelty, but it does not measure fluency.
7.4 Training and Parameters
The models use Adam optimization, selected embedding and LSTM sizes, greedy decoding, and optional UNK post-processing during training and inference.
- The models use minibatches of 32 and Adam with a 0.001 learning rate, momentum parameters 0.9 and 0.999, and ϵ = 10^-8.
- The ME configuration, with 192-dimensional embeddings and LSTM states, performs best on validation BLEU across models.Larger 384-384 and 512-512 configurations fail to converge or perform very poorly.
- At test time, greedy decoding selects the most likely target sentence.The authors observed no improvement from beam search.
- UNK post-processing raises BLEU by about 0.1-0.2 for neural models; the best model reaches 31.26 from 31.12, a 0.14 increase.
8 Results
Results support the Copy architecture, dictionary retrofitting, embedding choices, and external-data pretraining, while showing higher novelty but some fluency and repetition issues.
- Copy models outperform their SimpleS2S counterparts by at least 7-8 BLEU points.
- Retrofitted configurations generally outperform corresponding Plain configurations, with the best configuration gaining at least 11 BLEU.The comparison is between Copy.Yes.RetroExtFixed and Copy.Yes.PlainExtFixed.
- Fixed embeddings generally outperform trainable embeddings; Copy.Yes.RetroExtFixed scores 31.12 BLEU versus 20.95 and uses 5.25M versus 9.40M parameters.
- Pretraining with external data improves Copy.Yes.RetroExtFixed by more than 2 BLEU points over Copy.Yes.RetroFixed.
- Pretraining harms BLEU for SimpleS2S models but improves it for Copy models; Copy.No.PlainVar exceeds Copy.No.NoneVar by 1.8 BLEU.
- Neural models produce higher PINC scores than statistical and dictionary approaches, indicating greater output novelty relative to the source.PINC does not measure fluency.
- Adding sentinel loss reduces BLEU by a point or two in Copy+SL configurations.
- Without copying, SimpleS2S fails to predict “saint” and “francis” and drifts after incorrectly predicting “flute”; Copy outputs generally resemble ground truth more closely but can repeat or lose fluency.
9 Related Work
Prior style-adaptation systems use statistical phrase tables or targeted language-model modifications, whereas this work uses an end-to-end neural approach for broader transformations.
- Earlier systems adapt style with phrase-table statistical machine translation or language models that insert adjectives and adverbs.
- The authors describe prior automated style-adaptation solutions as limited in transformation scope, while their method handles generic word addition and deletion.
- Pointer networks can copy input words directly into neural sequence-to-sequence outputs but cannot generate words absent from the input.
- Pointer-based methods have been applied to extractive summarization and question answering, while recurrent-pointer mixtures have also been used for language modeling.
- This paper applies an end-to-end trainable neural network to Modern English-to-Shakespearean English style transformation, distinct from the cited machine-translation settings.
10 Conclusion
The paper proposes pointer-network and LSTM mixture models for transforming Modern English into Shakespearean English, showing benefits from input copying and dictionary constraints. Its demonstrated scope remains limited to Shakespearean style, while other stylistic variations are left for future work.
- The paper uses a mixture model of pointer network and LSTM to transform Modern English text into Shakespearean style English.
- Input-copying mechanisms and dictionary constraints are useful for shared but non-identical source-target sides with sparse parallel data.
- The demonstrated transformation is limited to Shakespearean style English.
- Other stylistic variations, including formality, politeness, and fancier words or expressions, remain to be explored.