Source-linked AI summary
Transforming Delete, Retrieve, Generate Approach for Controlled Text Style Transfer
Akhilesh Sudhakar, Bhargav Upadhyay, Arjun Maheswaran
TL;DR
Text style transfer must preserve content while changing stylistic attributes, but parallel style corpora are unavailable and prior methods have important training and quality limitations. The paper introduces Transformer-based deletion and generation within Delete Retrieve Generate, outperforming state-of-the-art systems across sentiment, gender, and political-slant transfer while evaluating GLEU as a metric aligned with human judgments.
Problem
Text style transfer lacks parallel corpora whose paired sentences differ only in style, while existing adversarial and Delete Retrieve Generate methods have documented training, control, quality, and robustness limitations.
Method
The paper combines a Delete Retrieve Generate framework with a Transformer-based Delete Transformer and Generative Style Transformer, using pretrained language modeling for target-style rewriting.
Results
The models outperform state-of-the-art systems on sentiment, gender, and political-slant transfer across the evaluated datasets.
Takeaways & Limitations
GLEU balances target-style matching and content retention and correlates with human evaluations, while the models produce natural sentences that retain core content.
Takeaways & Limitations
Prior framework components remain vulnerable to removing core content, missing source attributes, unfluent generation, and failure on longer inputs.
Abstract
from arXiv · showhide
Text style transfer is the task of transferring the style of text having certain stylistic attributes, while preserving non-stylistic or content information. In this work we introduce the Generative Style Transformer (GST) - a new approach to rewriting sentences to a target style in the absence of parallel style corpora. GST leverages the power of both, large unsupervised pre-trained language models as well as the Transformer. GST is a part of a larger `Delete Retrieve Generate' framework, in which we also propose a novel method of deleting style attributes from the source sentence by exploiting the inner workings of the Transformer. Our models outperform state-of-art systems across 5 datasets on sentiment, gender and political slant transfer. We also propose the use of the GLEU metric as an automatic metric of evaluation of style transfer, which we found to compare better with human ratings than the predominantly used BLEU score.
1 Introduction
The paper addresses unsupervised text style transfer without parallel corpora by improving the Delete Retrieve Generate framework with Transformer-based deletion and generation. The resulting system outperforms state-of-the-art methods across sentiment, gender, and political-slant transfer.
- Parallel corpora pairing sentences that differ only in style are scarce, motivating methods that transfer style without them.
- Adversarial latent-representation methods are difficult to train, inflexible in content-style trade-offs, sparse, low-quality, and limited in fine-grained style control.
- The Delete Retrieve Generate framework removes source attributes, retrieves target-style attributes, and generates a rewritten sentence from the resulting content and attributes.
- The proposed system uses a Delete Transformer to improve attribute removal and a Generative Style Transformer to replace the LSTM-based Generate mechanism.
- The models outperform state-of-the-art systems on sentiment, gender, and political-slant transfer.
2 Our Approach
The approach learns to transfer an input sentence into a target style through deletion, optional attribute retrieval, and generation. It defines style transfer as producing target-style output while preserving non-stylistic content.
- The model learns P(y|x, stgt) so the output y has target style stgt while retaining the input sentence's non-stylistic information.
- Delete separates an input into non-stylistic content c and stylistic attributes a, with the original sentence reconstructible from both.
- Retrieve optionally selects target attributes atgt from the corpus of sentences in the target style.
- Generate produces target-style text either from content c alone or from content c combined with retrieved attributes atgt.
2.1 Delete
The Delete stage identifies style-bearing tokens using attention-based importance scores, removes a tunable fraction of them, and evaluates whether content is preserved while style attributes are extracted.
- Delete Transformer: A style classifier’s attention weights serve as token importance scores for distinguishing stylistic attributes from content.The Delete Transformer uses a selected attention head and layer to obtain directly usable style-related scores.
- Delete Transformer: The method examines [CLS]-to-token attention across head-layer pairs to identify tokens significant for style classification.The selected attention scores are extracted for every token and compared across Transformer heads and layers.
- Attribute extraction: The top γ|x| tokens by importance are removed, producing a reduced sentence x′ while allowing γ to control deletion proportion.The parameter γ is tuned per dataset, and |x| denotes the number of input tokens.
- Delete Transformer: The selected head-layer pair is chosen by averaging a style-separation score over a validation set of reduced sentences.The final pair is selected from all combinations of attention heads and layers.
- Evaluation: Human evaluation found that the method deleted all style attributes in 89% of examples and wrongly removed non-style attributes in 12%.The evaluation used 200 random sentiment-transfer test sentences; the comparison method achieved 67% and over 29%, respectively.
2.2 Retrieve
The Retrieve stage selects a sentence from the target-style corpus whose content is closest to the source content, using a distance metric to obtain compatible target attributes.
- Retrieve: A target-style sentence is retrieved by minimizing content distance between the source content and candidate target-style contents.Closer contents are expected to have compatible attributes because they occur in similar contexts.
- Retrieve: The method compares TF-IDF, averaged GloVe, and Universal Sentence Encoder representations for retrieval.TF-IDF vector similarity produced the best retrieval results among the tested mechanisms.
2.3 Generate
The Generate stage uses Transformer-based GST variants to rewrite reduced content into the target style, optionally conditioning generation on retrieved attributes for finer control.
- 2.3 Generate: GST is a decoder-only Transformer based on an unsupervised generative language model and trained to produce fluent target-style sentences from content and attributes.Masked attention lets GST attend only to preceding tokens while modeling content and retrieved attributes.
- Variants of GST: B-GST receives content and target style, whereas G-GST additionally receives retrieved target-style attributes.B-GST can generate freely when retrieval is incompatible; G-GST supplies explicit attribute information.
- Variants of GST: G-GST enables fine-grained control by allowing target attributes to be manually specified during inference without the Retrieve component.It can also reduce sparsity when the target corpus contains sentences similar to the source corpus.
- Variants of GST: Special tokens mark target style, content boundaries, output position, and, for G-GST, the start of retrieved attributes.G-GST places retrieved attributes before the content in its input representation.
- Training: Without parallel data, GST variants are trained with reconstruction objectives that recover each sentence from its content and, where applicable, attributes or source style.B-GST uses source style, while G-GST uses its own attributes during reconstruction.
- Training: G-GST replaces attributes with randomly selected source- or target-style attributes in 10% of training examples to prevent trivial reconstruction.The noise consists of 5% source-style and 5% target-style replacements.
3 Experiments
The experiments use five datasets with established train-dev-test splits and compare against adversarial and Delete Retrieve Generate baselines. Human evaluations report preferences across content preservation, fluency, target-style match, and overall success.
- 3.1 Datasets: Experiments use five datasets, including YELP, AMAZON, CAPTIONS, POLITICAL, and GENDER.YELP, AMAZON, and CAPTIONS follow Li et al. (2018), while POLITICAL and GENDER follow Prabhumoye et al. (2018).
- 3.1 Datasets: The datasets retain the train-dev-test splits established by the corresponding prior works.Human gold-standard references are also available for the YELP, AMAZON, and CAPTIONS test sets.
- 3.2 Comparison to Previous Works: Comparisons include three adversarially trained models and two Delete Retrieve Generate models on Yelp, Amazon, and Captions.The compared models are StyleEmbedding, MultiDecoder, CrossAligned, DeleteOnly, and DeleteAndRetrieve.
- 3.2 Comparison to Previous Works: Table 2 reports human preferences for content preservation, fluency, target-style match, and overall success.Annotators compare generated sentences given the same source sentence and may prefer either output or neither.
- 3.2 Comparison to Previous Works: Table 3 reports human evaluation results for content preservation and fluency.The table caption defines these as the two reported evaluation dimensions.
4 Evaluation of Results
The evaluation combines human judgments with automatic measures of style strength, content preservation, fluency, and overall quality. B-GST generally outperforms prior systems in human evaluations, while GLEU better tracks those judgments than several isolated learned metrics.
- Evaluation setup: Human evaluations compare generated sentences on content preservation, style transfer strength, fluency, and overall success.Annotators rate outputs against the source sentence across these four criteria.
- Automatic metrics: GLEU evaluates source, reference, and generated sentences together, rewarding necessary changes and retained content.Unlike BLEU, it penalizes wrongly changed source words and rewards successful changes and retention.
- Human evaluation: B-GST outperforms state-of-the-art systems by a good margin on almost all human-rated parameters across all datasets.The reported advantage is specifically strongest for B-GST among the authors’ models.
- Metric interpretation: GLEU scores correlate with the authors’ human evaluations, whereas perplexity and target-style accuracy do not fully correlate with human judgments.The authors caution that learned metrics depend heavily on training-data distributions and should not be considered in isolation.
- Result analysis: B-GST generated sentence lengths closely correlate with source lengths, unlike the prior D&R model.The comparison is shown in Figure 2.
- Metric interpretation: Automatic metrics remain useful because they can scale evaluation to larger sets of models and datasets.The authors also report consistent BLEU advantages and strong, consistent perplexity for B-GST across datasets.
5 Related Work
Related work includes latent-representation, reinforcement-learning, machine-translation, and Delete Retrieve Generate approaches. These methods face convergence, sparsity, contextual deletion, or quality limitations, while prior Transformer use differs in its adversarial setting.
- Latent representations: Latent-representation approaches use adversarial networks to separate content and style, including cross-aligned auto-encoders, VAEs, and encoder-decoder frameworks.These methods learn shared or disentangled representations for content and stylistic attributes.
- Non-latent approaches: Reinforcement-learning and unsupervised machine-translation approaches suffer from sparsity and convergence issues that can produce low-quality sentences.These approaches do not rely on latent representations to separate content and attributes.
- Non-latent approaches: Delete Retrieve Generate is another non-latent approach, alongside reinforcement-learning and unsupervised machine-translation methods.The passage identifies DRG as part of this alternative group without assigning it the same stated limitation.
- Attribute extraction: Prior attention-based attribute extraction methods do not adequately model sentence context or exploit a Transformer’s contextual capacity.The cited comparison includes Li et al.’s salience deletion method.
- Transformer-based work: Dai et al. use Transformers for style transfer in an adversarial generator-discriminator setting.This distinguishes that prior Transformer approach from the framework discussed elsewhere in the paper.
6 Conclusion
The paper concludes that Generative Style Transformer outperforms state-of-the-art systems on sentiment, gender, and political slant transfer. Its approach combines the DRG framework, massively pretrained language models, and Transformer networks.
- Conclusion: Generative Style Transformer outperforms state-of-the-art systems on sentiment, gender, and political slant transfer.This is the paper’s stated cross-task conclusion.
- Conclusion: The model combines the Delete Retrieve Generate framework with massively pretrained language models and the Transformer network.These components define the paper’s overall approach.