Source-linked AI summary
Style Transformer: Unpaired Text Style Transfer without Disentangled Latent Representation
Ning Dai, Jianze Liang, Xipeng Qiu, Xuanjing Huang
TL;DR
Existing unpaired style-transfer models struggle to separate style from semantics and preserve long-range content through latent representations and RNNs. Style Transformer avoids assumptions about disentangled latents and uses Transformer attention, achieving competitive or better performance with better content preservation on two datasets. The training process still faces gradient-estimation challenges for discrete generated sentences, while multiple-attribute adaptation remains future work.
Problem
Disentangled latent representations may retain style information, while RNN-based latent mediation weakens long-term dependency handling and content preservation.
Method
Style Transformer uses a novel training algorithm without disentangled-latent assumptions and employs Transformer self-attention for style transfer.
Results
The model achieves competitive or better performance than prior approaches on two datasets, with better content preservation on both.
Takeaways & Limitations
Avoiding disentangled latent representations enables Style Transformer to preserve content better across the evaluated datasets.
Takeaways & Limitations
Discrete generated sentences prevent direct discriminator-gradient propagation, requiring gradient-estimation methods such as REINFORCE or Gumbel-Softmax.
Abstract
from arXiv · showhide
Disentangling the content and style in the latent space is prevalent in unpaired text style transfer. However, two major issues exist in most of the current neural models. 1) It is difficult to completely strip the style information from the semantics for a sentence. 2) The recurrent neural network (RNN) based encoder and decoder, mediated by the latent representation, cannot well deal with the issue of the long-term dependency, resulting in poor preservation of non-stylistic semantic content. In this paper, we propose the Style Transformer, which makes no assumption about the latent representation of source sentence and equips the power of attention mechanism in Transformer to achieve better style transfer and better content preservation.
1 Introduction
Unpaired text style transfer must change stylistic properties while preserving style-independent content, but disentangled latent representations and RNN-based generation create substantial difficulties. Style Transformer addresses these concerns without assuming disentangled latents and uses Transformer attention for transfer and content preservation.
- Unpaired style transfer changes stylistic properties while retaining style-independent content because paired sentences with matching content and different styles are difficult to construct.
- Disentangled latent representations may still retain recoverable style information, making stylistic separation from sentence semantics difficult to judge.
- Disentanglement is unnecessary when a decoder can generate the desired style from an entangled representation by overwriting the original style.
- Fixed-size latent vectors struggle to capture rich semantic information in long text, while RNN encoders and decoders weakly capture long-range dependencies and preserve content.
- Style Transformer introduces a training algorithm without disentangled-latent assumptions and applies Transformer self-attention to improve style transfer and content preservation.
- Experimental results show that the approach generally outperforms other approaches on two style transfer datasets, with especially strong content preservation.
2 Related Work
Related work largely follows latent-representation or non-latent approaches, but both lines make limited use of attention for referring to long-term history or the source sentence. Style Transformer instead uses Transformer self-attention without assumptions about the source latent representation.
- One line of prior work learns latent representations to manipulate generated-sentence style, commonly using encoder-decoder architectures and disentanglement strategies.
- Fixed-size latent representations improve interpretability and manipulation but prevent models from using source-sentence information after encoding.
- Other approaches avoid manipulating latent representations, including cycled reinforcement learning, content-word and phrase replacement, and unsupervised machine translation.
- Both prior lines make few attempts to use attention for long-term history or source-sentence reference, although attention is established as important for capturing long-term dependencies.
- Style Transformer follows the non-latent line by making no assumption about the source latent representation and using Transformer as its basic module.
3 Style Transformer
This section introduces the paper’s style-transfer task and then develops the proposed model from that problem definition.
- The section first introduces the style-transfer task before discussing the proposed model based on its problem definition.
3.1 Problem Formalization
Style is defined by the shared characteristic or dataset distribution of sentences, and transfer rewrites an arbitrary sentence into a desired style while preserving its original information.
- A style is the characteristic shared by sentences in one dataset, equivalently defined by that dataset’s distribution.
- Given K datasets, each dataset defines a distinct style denoted by s(i).
- Style transfer rewrites an arbitrary sentence x into a sentence with desired style s while preserving information from x as much as possible.
3.2 Model Overview
Style Transformer learns a mapping from an input sentence and desired style to a transferred sentence. Because parallel corpora are unavailable, discriminator-based approaches provide supervision from non-parallel data.
- The model learns fθ(x, s), mapping sentence x and style control variable s to a transferred sentence.
- Unpaired style transfer lacks direct supervision for transferring x to a different style.
- Two discriminator-based approaches create supervision from non-parallel corpora.
- The Style Transformer and discriminator are combined through an overall learning algorithm.
3.3 Style Transformer Network
The Style Transformer uses a standard Transformer encoder-decoder with an added style embedding. It predicts output tokens autoregressively from sentence representations and style conditioning.
- The Transformer encoder maps input sentence x to a sequence of continuous representations z.
- The decoder estimates the conditional probability of output sentence y through autoregressive factorization.
- At each time step, a softmax classifier computes the probability of the next token from decoder logits.
- An additional style embedding conditions the encoder on both input sentence x and style control variable s.
- The network denotes its predicted output sentence as fθ(x, s).
3.4 Discriminator Network
The discriminator supplies style supervision when the target style differs from the source style. The paper considers conditional and multi-class discriminator architectures alongside Transformer-based reconstruction.
- Direct supervision is unavailable when the selected style differs from the sentence’s original style, so a discriminator learns supervision from non-parallel corpora.
- Cycle reconstruction preserves source information by reconstructing x from transferred sentence by under the original style label.
- Style control trains the discriminator to provide supervision that encourages the transferred sentence to match desired style bs.
- The discriminator is another Transformer encoder that distinguishes sentence styles and provides style supervision to the Style Transformer.
- Conditional Discriminator: A conditional discriminator judges whether a sentence has a proposed corresponding style, while real and reconstructed sentences are positive and transferred sentences are negative.
- Multi-class Discriminator: The multi-class discriminator predicts among K styles plus a fake-sample class, with transferred sentences labeled as class 0.
- Figure 2 distinguishes same-style reconstruction from different-style training using Transformer reconstruction and discriminator-based style reconstruction.
3.5 Learning Algorithm
Training alternates discriminator learning with Style Transformer learning. The model combines self-reconstruction, cycle reconstruction, and style-controlling losses, while using a soft distribution to preserve gradient continuity.
- Training alternates discriminator learning and Style Transformer network learning.
- Discriminator Learning: The discriminator distinguishes real and reconstructed sentences from transferred sentences using cross-entropy classification loss.
- Discriminator Learning: Conditional and multi-class discriminators use different sentence-labeling protocols.
- Style Transformer Learning: For matching styles, self-reconstruction trains the model to reconstruct x by minimizing negative log-likelihood.
- Style Transformer Learning: For differing styles, cycle reconstruction preserves input information while style-controlling loss prevents the network from merely copying the input.
- Overall Training: Training alternates nd discriminator-update steps with nf Style Transformer-update steps in a GAN-like procedure.
- Gradient Handling: Because discrete samples block direct discriminator gradients, the method feeds the downstream network the softmax distribution as a soft generated sentence.
4 Experiment
Experiments evaluate Style Transformer on Yelp and IMDb using automatic and human measures of style control, content preservation, and fluency. The model generally performs competitively, with stronger content preservation, while ablations show distinct roles for its losses and discriminator training samples.
- Datasets: Experiments use Yelp and IMDb review datasets, with sentiment labels and human reference sentences available for Yelp test examples.
- Evaluation: Evaluation measures style control, content preservation, and fluency through sentiment accuracy, BLEU-based metrics, and perplexity.Human evaluation supplements automatic metrics because unpaired style transfer lacks parallel data.
- Automatic Evaluation: Style Transformer achieves competitive overall automatic performance and better content preservation on both datasets, while its conditional model improves style control over the multi-class model.Both Style Transformer models generate sentences with relatively low perplexity.
- Human Evaluation: Human evaluation results conform mainly to automatic results, with Style Transformer models outperforming DAR and CtrlGen in content preservation.The comparison uses more than 400 human evaluation reviews across the two datasets.
- Ablation Study: Removing self-reconstruction causes single-word outputs, removing cycle reconstruction reduces BLEU, and removing discriminator loss produces copied outputs without style modification.The reported ablation results associate the three losses with readability, source-information preservation, and style supervision, respectively.
- Ablation Study: Using only real sentences in discriminator training substantially harms style accuracy, whereas removing real sentences causes only a small performance drop from the full model.The final model therefore uses a mixture of real and generated sentences for discriminator training.
5 Conclusions and Future Work
The Style Transformer achieves competitive or better performance than previous state-of-the-art approaches on two text style transfer datasets, especially in content preservation. Future work includes extending the model to multiple attributes and combining it with back-translation.
- Conclusions: Style Transformer achieves competitive or better performance than previous state-of-the-art approaches on two text style transfer datasets.
- Conclusions: The model achieves better content preservation on both datasets without assuming a disentangled latent representation.
- Future Work: Future work will adapt Style Transformer to multiple-attribute settings and investigate combining back-translation with its training algorithm.
- Conclusions: The paper presents a Yelp case study whose colors mark good transfer, bad transfer, and grammar errors.
- Conclusions: The paper reports a model ablation study on the Yelp dataset.