Source-linked AI summary
Disentangled Representation Learning for Non-Parallel Text Style Transfer
Vineet John, Lili Mou, Hareesh Bahuleyan, Olga Vechtomova
TL;DR
Disentangling style and content in neural text-generation representations is challenging, especially for non-parallel style transfer. The paper combines multi-task and adversarial objectives with bag-of-words content features, achieving substantially better transfer strength, content preservation, and fluency than previous state-of-the-art work.
Problem
The paper addresses the challenge of disentangling style and content in neural text-generation latent spaces for non-parallel style transfer.
Method
The approach separates latent style and content spaces using multi-task and adversarial objectives, approximating content with style-neutral bag-of-words features.
Results
The disentangled space achieves substantially better style-transfer strength, content preservation, and language fluency than previous state-of-the-art work.
Takeaways & Limitations
The learned disentangled representation can be directly applied to style transfer on non-parallel, style-labeled corpora.
Takeaways & Limitations
The method assumes that each sentence has a binary sentiment label identifying its style.
Abstract
from arXiv · showhide
This paper tackles the problem of disentangling the latent variables of style and content in language models. We propose a simple yet effective approach, which incorporates auxiliary multi-task and adversarial objectives, for label prediction and bag-of-words prediction, respectively. We show, both qualitatively and quantitatively, that the style and content are indeed disentangled in the latent space. This disentangled latent representation learning method is applied to style transfer on non-parallel corpora. We achieve substantially better results in terms of transfer accuracy, content preservation and language fluency, in comparison to previous state-of-the-art approaches.
Introduction
The paper addresses disentangling style and content in text-generation latent spaces using an autoencoder with auxiliary multi-task and adversarial objectives. It applies the learned representations to non-parallel text style transfer and reports improved transfer strength, content preservation, and language fluency.
- Problem and approach: The model encodes sentences with an autoencoder and seeks latent representations disentangled with respect to style and content.The latent representation is divided into separate style and content spaces.
- Problem and approach: The approach combines multi-task and adversarial objectives to enforce separation between artificially divided style and content spaces.Multi-task losses ensure desired information is present, while adversarial objectives support separation.
- Problem and approach: BoW features focused on style-neutral, non-stopwords provide a content approximation that strengthens disentanglement through BoW multi-task and adversarial losses.These losses complement traditional style-oriented auxiliary losses.
- Application: The learned disentangled latent space supports style transfer on non-parallel, style-labeled corpora, transforming sentences to preserve content while changing style.Parallel corpora are difficult to obtain because they require sentence pairs with the same content and differing styles.
- Evaluation: Experiments on two customer review datasets show that style and content spaces are disentangled, with substantially better style-transfer strength, content preservation, and language fluency than previous results.Ablation tests indicate that the auxiliary losses combine effectively and each plays its own role in disentangling the latent space.
Related Work
Prior work has explored disentangled latent spaces extensively in image processing, while NLP studies have addressed style control through sentiment, adversarial alignment, Wasserstein penalties, and parallel formality-transfer data. This paper instead encodes style and content from the input and uses auxiliary losses to enforce disentanglement for non-parallel style transfer.
- Disentangled representations: Image-processing research has disentangled features such as rotation and color, while image statistics can capture characteristics including artistic style.Other work also uses data augmentation to learn a disentangled latent space.
- NLP style transfer: In NLP, style is vague, so researchers often treat sentiment as a salient style and use discriminators to reconstruct sentiment and content.The passage notes that this reconstruction does not establish that the latent space is disentangled.
- NLP style transfer: Prior NLP approaches align content representations with adversarial discriminators or a Wasserstein-distance penalty, while another uses sequence-to-sequence models on a parallel formality corpus.The paper characterizes the Wasserstein penalty as more indirect than its bag-of-words-based auxiliary losses and identifies parallel formality transfer as beyond its scope.
- Paper's distinction: This paper encodes both style and content spaces from the input and designs auxiliary losses so each space encodes only its desired information.The resulting disentangled representation can be directly applied to non-parallel text style-transfer tasks.
Approach · Autoencoder · Style-Oriented Losses
The approach uses an autoencoder with auxiliary multi-task and adversarial losses to disentangle style and content representations for non-parallel text style transfer. It implements deterministic and variational autoencoders, then encourages sentiment information in the style space and its removal from the content space.
- Approach: The model is built on a sequence-to-sequence autoencoder with multi-task and adversarial losses for the style and content spaces.The approach is presented for style transfer in natural language generation.
- Autoencoder: A deterministic autoencoder encodes an input sequence into a compact hidden representation and uses a decoder RNN to reconstruct the original sentence.The encoder uses a GRU, and reconstruction is trained with sequence-aggregated cross-entropy loss.
- Variational Autoencoder.: The variational autoencoder adds a KL-divergence penalty and reconstructs from sampled posterior latents to smooth the latent space.Its prior is the standard normal distribution N(0, I), while the encoder predicts the posterior parameters.
- Style-Oriented Losses: The auxiliary objectives seek to decompose the latent vector as h = [s; c], where s represents style and c represents content.The autoencoding losses remain the primary training objective, while auxiliary losses disentangle the latent space.
- Multi-Task Loss for Style.: The multi-task loss makes the style vector discriminative for sentiment by training the style classifier alongside the autoencoding objective.This jointly incentivizes sentence decoding and sentiment prediction from s.
- Multi-Task Loss for Style.: For non-parallel corpora, sentiment is treated as the style and each sentence receives a binary positive or negative sentiment label.The style space is trained to predict this label through a two-way softmax classifier and cross-entropy loss.
- Adversarial Loss for Style.: The adversarial loss trains an adversary to predict style from c while tuning the encoder so that c is not discriminative for style.The adversary’s gradients do not flow back to the autoencoder; the encoder maximizes the adversarial objective with the discriminator fixed, favoring a uniform predicted style distribution.
- Adversarial Loss for Style.: Combining adversarial and multi-task losses is presented as a promising approach that achieves better style-transfer performance than previous state-of-the-art methods.The adversarial objective uses entropy penalization because it extends readily to multi-category classification and supports the content-oriented losses.
Content-Oriented Losses
The method adds content-oriented auxiliary losses because style-oriented objectives do not constrain content encoding, allowing content to leak into the limited-capacity style space. It approximates content with bag-of-words features, trains the content space to predict them, and adversarially removes them from the style space.
- Motivation: Style-oriented losses do not constrain content encoding, so content can flow into the limited-capacity style space.The paper identifies this limitation in its approach and in most previous work.
- Bag-of-Words Content Representation: Content is approximated with bag-of-words features representing word-occurrence probabilities.For content modeling, the method excludes stopwords and style-specific words, including sentiment words for sentiment transfer.
- Content Multi-Task Loss: The content multi-task loss trains the content space to encode bag-of-words information using a softmax classifier and cross-entropy against the target distribution.Unlike autoencoding, it considers only content words and applies the objective only to the content space.
- Content Adversarial Loss: The content adversarial loss trains a style-space discriminator to predict bag-of-words features while maximizing its entropy to remove content information from the style space.The discriminator updates its own parameters, whereas the adversarial objective updates the encoder parameters.
Training Process
Training combines reconstruction with auxiliary multi-task and adversarial objectives for style and content, while alternating discriminator and model-loss optimization. Hyperparameters balance the autoencoding and auxiliary terms.
- Loss formulation: The overall autoencoder loss combines reconstruction, multi-task style and content objectives, and adversarial style and content objectives.The auxiliary objectives are weighted alongside the autoencoding loss in J_ovr.
- Loss formulation: The λ hyperparameters balance the autoencoding loss against the auxiliary losses.Separate coefficients weight the multi-task and adversarial objectives for style and content.
- Optimization procedure: Training alternates between optimizing discriminator losses J_dis(s) and J_dis(c) and optimizing the model’s overall loss J_ovr.This alternating procedure is specified in Algorithm 1.
Generating Style-Transferred Sentences
The paper applies its disentangled latent space to generate sentences that preserve input content while changing style. Target-style transfer is performed by combining an inferred target-style vector with the input’s encoded content for decoding.
- Generating Style-Transferred Sentences: The disentangled latent space enables style transfer that retains generally the same meaning while changing properties such as sentiment.This is presented as a direct application of the proposed representation.
- Generating Style-Transferred Sentences: For an input sentence, the method encodes separate style and content vectors, then estimates the vector for the desired target style.The input is represented by disentangled vectors s* and c*, while the target style is represented by an empirical estimate ŝ.
- Generating Style-Transferred Sentences: The inferred target-style vector is concatenated with the encoded content vector before decoding the style-transferred sentence.This decoding procedure is illustrated in Figure 1b.
Experiments
Experiments on Yelp and Amazon reviews evaluate latent-space disentanglement and non-parallel sentiment transfer. The model separates style from content and outperforms competing approaches across transfer strength, content preservation, and language fluency.
- Datasets: Experiments use Yelp and Amazon review datasets with binary sentiment labels for disentanglement training and sentiment-transfer evaluation.Yelp has 444101/63483/126670 train/validation/test reviews, while Amazon has 559142/2000/2000; maximum lengths are 15 and 20 words, respectively.
- Latent-space disentanglement: The 128-dimensional content vector is only slightly better than majority guessing for style classification, whereas the 8-dimensional style vector achieves substantially higher accuracy without improvement from combining both.These classification results verify that style information is concentrated in the style space rather than the content space.
- Latent-space disentanglement: t-SNE plots show clean style separation and indistinguishable content representations, while VAE latent spaces are smoother and more continuous than DAE spaces.The plots use the disentangled style and content spaces learned on Yelp.
- Sentiment transfer evaluation: Evaluation combines automatic metrics and blind human judgments of transfer strength, content preservation or similarity, and generated-language quality.Human evaluation was conducted on Yelp with 6 evaluators using a 1–5 Likert scale.
- Sentiment transfer results: The model outperforms competing methods in transfer accuracy, cosine similarity, word overlap, language fluency, and human-rated performance, while its empirically estimated style vector reliably controls sentiment without losing content.The style-embedding baseline reconstructs exact source sentences 24% of the time, compared with less than 6% for all other models.
- Sentiment transfer results: 90% transfer accuracy results from combining multi-task and adversarial style losses, outperforming previous methods by 10% and showing complementary effects of the two losses.Using either loss alone achieves reasonable transfer accuracy and cosine similarity.
Conclusion
The paper proposes disentangling style and content in neural latent spaces using multi-task and adversarial objectives with bag-of-words content approximation. Experiments show the resulting representation supports text style transfer with stronger transfer, content preservation, and fluency than prior state-of-the-art work.
- Conclusion: The approach combines multi-task and adversarial objectives to separate content and style information in neural latent spaces.Content information is approximated with bag-of-words features from style-neutral, non-stopword vocabulary.
- Conclusion: Qualitative and quantitative experiments show that the latent space separates into distinct style and content parts.The disentangled representation can be directly applied to text style-transfer tasks.
- Conclusion: The method achieves substantially better style-transfer strength, content-preservation scores, and language fluency than previous state-of-the-art work.These improvements are reported for text style-transfer tasks using the disentangled latent space.
Supplemental Material · A. Bag-of-Words (BoW) Vocabulary Ablation Tests
The ablation tests examine how the BoW vocabulary used for auxiliary content losses affects performance. Excluding sentiment words and stopwords performs better on every quantitative metric.
- A. Bag-of-Words (BoW) Vocabulary Ablation Tests: The experiments measure the effect of BoW vocabulary selection on auxiliary content losses.The vocabulary choice is the experimental variable in the tests.
- A. Bag-of-Words (BoW) Vocabulary Ablation Tests: Table 6 presents ablation tests evaluating different BoW vocabulary choices.These tests assess the vocabulary used for the auxiliary content losses.
- A. Bag-of-Words (BoW) Vocabulary Ablation Tests: Excluding sentiment words and stopwords performs better on every quantitative metric.The passage reports consistent improvement across all quantitative evaluations, without providing the individual values here.
B. t-SNE plots of Ablation Tests
This section presents t-SNE plots examining style and content embeddings under the baseline model and with each auxiliary loss added independently.
- Figure 3 shows t-SNE plots of style and content embeddings without auxiliary losses, using VAE latent embeddings with only JAE.
- Figure 4 examines VAE latent embeddings with JAE + Jmul(s).
- Figure 5 examines VAE latent embeddings with JAE + Jadv(s).
- Figure 6 examines VAE latent embeddings with JAE + Jmul(c).
- Figure 7 examines VAE latent embeddings with JAE + Jadv(c).