Source-linked AI summary
Style Transfer Through Back-Translation
Shrimai Prabhumoye, Yulia Tsvetkov, Ruslan Salakhutdinov, Alan W Black
TL;DR
Style transfer seeks to change stylistic properties without changing meaning or intent, but controlled language generation must separate semantic content from style. This paper grounds a latent representation in back-translation and uses style-specific generators for sentiment, gender, and political slant; it reports improved style transfer, meaning preservation, and fluency over baselines.
Problem
Controlled language generation seeks to separate semantic content from stylistic dimensions while rephrasing text for different styles.
Method
The method uses machine-translation-based back-translation to learn a latent content representation that preserves meaning and weakens author-style attributes, followed by separate style-specific generators.
Results
The model outperforms state-of-the-art baselines in style transfer accuracy, meaning preservation, and fluency across the evaluated tasks.
Takeaways & Limitations
The approach provides a non-parallel style-transfer method evaluated on sentiment, gender, and political slant, with political slant introduced as a new task.
Takeaways & Limitations
Generated sentences do not always adequately preserve meaning, and sentiment is difficult to modify independently of propositional content.
Abstract
from arXiv · showhide
Style transfer is the task of rephrasing the text to contain specific stylistic properties without changing the intent or affect within the context. This paper introduces a new method for automatic style transfer. We first learn a latent representation of the input sentence which is grounded in a language translation model in order to better preserve the meaning of the sentence while reducing stylistic properties. Then adversarial generation techniques are used to make the output match the desired style. We evaluate this technique on three different style transformations: sentiment, gender and political slant. Compared to two state-of-the-art style transfer modeling techniques we show improvements both in automatic evaluation of style transfer and in manual evaluation of meaning preservation and fluency.
1 Introduction
The paper presents back-translation style transfer for preserving sentence meaning while reducing stylistic properties, then evaluates it across sentiment, gender, and political slant. It reports improvements over state-of-the-art baselines in style accuracy, meaning preservation, and fluency.
- Approach: Back-translation rephrases sentences to reduce stylistic characteristics before separate style-specific generators produce the desired style.The pipeline first back-translates the sentence, then applies separate generators for style transfer.
- Tasks: The approach targets sentiment modification, gender transfer, and political slant transfer, including a newly proposed political slant task.Political slant transfer preserves meaning while confounding classifiers of political slant.
- Results: 12% absolute improvement was attained for political slant transfer, and up to 7% absolute improvement in modification of sentiment over state-of-the-art baselines.Style transfer was evaluated with classifiers trained on held-out data.
- Results: Manual evaluation found better meaning preservation than baselines for gender and political slant, while fluency was better in all experiments.Meaning preservation was evaluated using A/B testing, and fluency was evaluated by humans.
2 Methodology
The method uses back-translation to derive a style-agnostic latent representation, then trains style-specific generators guided by classifier feedback. It combines reconstruction and classification objectives to generate fluent sentences in a desired style while preserving meaning.
- 2.1 Meaning-Grounded Representation: Back-translation produces a latent code intended to preserve sentence meaning while weakening author-specific style attributes.The source sentence is translated into a target language and encoded by a fixed back-translation encoder.
- 2.2 Generator Learning: Separate style-specific decoders condition on the latent code and receive classifier-loss feedback targeting the decoder’s intended style.The framework uses multiple decoders, with the classifier target determined by which style decoder generated the output.
- 2.1 Meaning-Grounded Representation: The same fixed translation and back-translation encoder can be reused across different style-transfer tasks without learning separate latent-code models.The translation systems and back-translation encoder are independent of the task-specific data.
- 2.2.1 Style Classifiers: A supervised CNN classifier predicts style and evaluates generated samples, with style-specific lexicons added to improve classification accuracy.The classifier accepts discrete or continuous tokens so generator outputs can provide feedback during training.
- 2.2.2 Generator Learning: Bidirectional LSTM decoders generate token sequences conditioned on the back-translation representation, with global attention and a copy mechanism supporting generation.Attention aligns target states with source states and helps replace unknown characters.
- 2.2.2 Generator Learning: Because discrete tokens block gradient propagation, training uses a temperature-controlled continuous softmax approximation that anneals toward discrete generation.The generators also use reconstruction loss, while the generative loss combines reconstruction and classification terms through a balancing parameter.
3 Style Transfer Tasks
The paper evaluates style transfer for gender, political slant, and sentiment, using datasets and task-specific setups for modifying style while preserving intent. The section also motivates these tasks through privacy, bias-mitigation, and comparison with prior work.
- Motivation: The tasks have practical applications because linguistic choices can reveal demographic and personal attributes, while style transfer may help synthesize balanced training data.The motivation includes online anonymity, personal security, and reducing bias in NLP technologies.
- Task overview: The three tasks transfer gender, political slant, and sentiment, with political-slant transfer introduced as a new evaluation task.The methodology is presented as applicable to other styles, social categories, bias types, and multi-class settings.
- Gender: Gender transfer uses Yelp reviews labeled for two genders and filters out gender-neutral or weakly gender-indicative sentences.Gender is treated as a binary variable because continuous-valued gender annotations were unavailable.
- Political slant: Political-slant transfer uses public Facebook comments responding to members of the United States Senate and House, labeled Democratic or Republican by the recipient’s party.The task preserves a commenter’s intent while modifying observable political affiliation.
- Sentiment: Sentiment transfer uses Yelp reviews to generate a similar review with the opposite sentiment, following prior state-of-the-art experimental setups.The experiments replicate the models and setups of Hu et al. (2017) and Shen et al. (2017).
4 Experimental Setup
The experiments compare the back-translation style-transfer model with a cross-aligned auto-encoder and use English–French translation data for the latent representation. The setup specifies recurrent generators and encoders, attention, a CNN classifier, and task-specific settings.
- Baseline: The model is compared against Shen et al.’s cross-aligned auto-encoder, using style-specific decoders to align generated sentences with style distributions.The released sentiment model is used directly, while separate models are trained for gender and political slant.
- Translation data: English–French and French–English translation systems are trained on WMT15 data for the back-translation component.The data combine Europarl v7, news commentary v10, and common crawl corpora, with approximately 5.4M English–French parallel examples.
- Model configuration: The generators and encoders are two-layer LSTMs with 300-dimensional inputs and 500-dimensional hidden states.The generator samples sentences of maximum length 50 and uses global attention vectors of size 500.
- Evaluation scope: The Hu et al. model is omitted from political-slant results after obtaining nearly random style-transfer accuracy of 50.98% on the authors’ political-slant datasets.The authors report reproducing Hu et al.’s results on its original tasks and data.
- Model configuration: The CNN style classifier uses 100 filters of size 5 with max pooling and receives word embeddings plus two style-lexicon membership features.The classifier input size is 302, and the balancing parameter λc is set to 15.
5 Results
The paper evaluates style-transfer accuracy, meaning preservation, and fluency against cross-aligned auto-encoder baselines using held-out classifiers and human judgments. BST improves style transfer on political slant and sentiment, preserves meaning better on gender and political slant, and is more fluent overall, especially for longer sentences.
- Evaluation setup: Style-transfer accuracy is measured with classifiers trained on held-out data, while meaning preservation and fluency are evaluated by human judgments.The evaluation covers three dimensions: desired-style accuracy, semantic preservation, and readability or naturalness.
- Translation quality: 32.52 BLEU for English–French and 31.11 BLEU for French–English support the translation systems used to ground BST representations.The authors chose French because it is close to English and has abundant parallel data, allowing them to focus on style generation.
- Preservation of meaning: BST models outperform CAE models on meaning preservation for gender and political slant, while sentiment modification preserves meaning worse than the baseline.Human evaluation used pairwise comparisons with an optional no-preference response; sentiment is difficult because its content is intertwined with meaning and intent.
- Preservation of meaning: 47.27% of annotators preferred BST for long sentences, compared with 12.5% preferring CAE, and BST significantly outperforms CAE on longer, harder inputs.The human evaluation divided examples into short and long sentence groups.
- Fluency: BST outperforms the baseline overall in fluency and generates significantly more fluent longer sentences.Fluency was rated from 1, unreadable, to 4, perfect, using randomly selected generated sentences.
6 Related Work
Related work approaches non-parallel style transfer through latent-variable generation, cross-aligned auto-encoders, style-specific decoders or embeddings, lexical replacement, and adversarially learned representations. The paper also connects style transfer to paraphrase generation and machine translation with non-parallel data.
- Non-parallel style transfer: Hu et al. use variational auto-encoders with discriminators to learn disentangled latent representations for controllable attributes such as sentiment and tense.Their method generates sentences from a latent content representation and a style code.
- Non-parallel style transfer: Shen et al. analyze non-parallel style transfer theoretically and propose cross-aligned auto-encoders with discriminators, focusing on sentiment and word decipherment.Other related approaches include style-specific decoders, style embeddings, lexical replacement, and adversarially trained style-agnostic representations.
- Paraphrase and translation: The work relates to paraphrase generation using phrase-based or neural back-translation and to machine translation with non-parallel data.This connection places back-translation within existing methods for generating meaning-preserving alternative formulations.
7 Conclusion
The paper proposes back-translation for non-parallel style transfer, using machine-translation techniques to learn latent content representations that weaken style attributes. It reports broad gains over a cross-aligned autoencoder baseline, while acknowledging imperfect meaning preservation.
- It applies the method to gender, political slant, and sentiment style transfer using non-parallel text.
- The model outperforms a cross-aligned autoencoder baseline on political slant and sentiment transfer.
- It also outperforms the baseline in all fluency experiments and in meaning-preservation experiments for gender and political slant.
- The authors acknowledge that generated sentences do not always adequately preserve meaning.
- Future work considers debiasing, anonymizing author traits, and improving representations through multilingual pivoting.
A Supplementary Material
The supplementary material documents experiment tables for transfer accuracy, human meaning-preservation evaluation, and generated examples. It identifies the compared models and summarizes the organization of these evaluations.
- Tables 7–9 report style-transfer accuracy for gender, political slant, and sentiment modification.
- The meaning-preservation evaluation used four tests with 20 random samples for each style-transfer type and did not assess style appropriateness or fluency.
- Table 12 presents human preferences for meaning preservation as percentages.
- Table 13 gives gender, political-slant, and sentiment transfer examples; its caption states that BST outputs are generally grammatically better structured.