Source-linked AI summary
A Dual Reinforcement Learning Framework for Unsupervised Text Style Transfer
Fuli Luo, Peng Li, Jie Zhou, Pengcheng Yang, Baobao Chang, Zhifang Sui, Xu Sun
TL;DR
Unsupervised style transfer must preserve content without parallel data, yet existing content–style separation is difficult. The paper introduces DualRL, which trains dual one-step mappings with reinforcement-learning rewards for style accuracy and content preservation. It reports over 8 BLEU points of average improvement across two datasets and favorable human evaluations, while noting training complexity from pre-training and annealed pseudo teacher-forcing.
Problem
Existing unsupervised methods rely on difficult content–style separation, and implicit style expressions challenge direct removal of explicit style words.
Method
DualRL trains source-to-target and target-to-source one-step mappings with reinforcement learning, using dual rewards for style accuracy and content preservation without parallel data.
Results
Over 8 BLEU points averaged across two datasets were achieved over other systems, while human evaluation found the best average score and more than 10% successfully transferred instances.
Takeaways & Limitations
DualRL empirically demonstrates that dual one-step mappings trained with automatically generated supervision can outperform previous approaches on sentiment and formality transfer.
Takeaways & Limitations
Pre-training and annealed pseudo teacher-forcing make training complicated, motivating pure reinforcement-learning training from scratch.
Abstract
from arXiv · showhide
Unsupervised text style transfer aims to transfer the underlying style of text but keep its main content unchanged without parallel data. Most existing methods typically follow two steps: first separating the content from the original style, and then fusing the content with the desired style. However, the separation in the first step is challenging because the content and style interact in subtle ways in natural language. Therefore, in this paper, we propose a dual reinforcement learning framework to directly transfer the style of the text via a one-step mapping model, without any separation of content and style. Specifically, we consider the learning of the source-to-target and target-to-source mappings as a dual task, and two rewards are designed based on such a dual structure to reflect the style accuracy and content preservation, respectively. In this way, the two one-step mapping models can be trained via reinforcement learning, without any use of parallel data. Automatic evaluations show that our model outperforms the state-of-the-art systems by a large margin, especially with more than 8 BLEU points improvement averaged on two benchmark datasets. Human evaluations also validate the effectiveness of our model in terms of style accuracy, content preservation and fluency. Our code and data, including outputs of all baselines and our model are available at https://github.com/luofuli/DualLanST.
1 Introduction
Unsupervised text style transfer must change style while preserving content, but existing two-step methods struggle when style is implicit or content–style separation is unreliable. DualRL instead learns interacting one-step mappings in both directions and trains them with reinforcement learning.
- Motivation: Text style transfer rephrases input into a desired style while preserving its original content, despite parallel data being difficult to collect.Applications include sentiment transformation and formality modification.
- Limitations of prior work: Most existing unsupervised methods separate content from style before generating text in the target style.Approaches use either adversarially learned content representations or direct removal of style-specific words.
- Limitations of prior work: Implicit style, such as negative sentiment in “The only thing I was offered was a free dessert!!!”, cannot be handled by simply removing explicit style words.This limits the application range of direct neutralization methods.
- Proposed framework: DualRL directly learns one-step source-to-target and target-to-source mappings instead of explicitly separating content and style.The two sequence-to-sequence models form a dual structure for unsupervised transfer.
- Proposed framework: The two mapping models interact through quality feedback signals that reward style change and content preservation during reinforcement-learning training.The forward model transfers informal text to formal text, while the backward model performs the reverse mapping.
- Contributions: DualRL addresses pre-training and generation-quality challenges without parallel data and outperforms state-of-the-art systems in automatic and human evaluations.The architecture is described as generic and adaptable to other sequence-to-sequence generation tasks lacking parallel data.
2 Dual Reinforcement Learning for Unsupervised Text Style Transfer
DualRL learns source-to-target and target-to-source style-transfer mappings directly from non-parallel corpora. It combines style and reconstruction feedback in reinforcement learning, while pseudo-parallel teacher-forcing addresses training and generation challenges.
- DualRL framework: DualRL directly learns two one-step mappings between non-parallel corpora instead of separating content and style first.The forward model maps source style to target style, while the backward model performs the reverse transfer.
- Reward design: A style-classifier reward evaluates target-style accuracy, while a reconstruction reward estimates content preservation through backward reconstruction.The reconstruction reward is the probability that the backward model recovers the original source from the transferred sentence.
- Reward design: BLEU-based back-translation was tested as a content-preservation measure but showed poor performance in primary experiments.The framework therefore uses reconstruction probability rather than this BLEU-based reward for content preservation.
- Reward design: The final reward is the harmonic mean of style accuracy and content preservation, with β controlling their trade-off.The two transfer models are trained alternately with policy gradients to maximize expected reward.
- Training strategy: DualRL uses pseudo-parallel data and annealed teacher-forcing to address reinforcement learning’s warm-start, fluency, and readability challenges without parallel data.Teacher-forcing updates become less frequent through an exponentially increasing interval during training.
3 Experiments
The experiments cover sentiment and formality transfer using YELP and GYAFC, with additional YELP references created to improve evaluation reliability.
- Sentiment transfer: The model is evaluated on sentiment transfer using YELP restaurant reviews, with ratings above 3 labeled positive and those below 3 negative.
- Formality transfer: Formality transfer uses the family and relationships domain of GYAFC, whose alignments are reserved for evaluation rather than training.
- Evaluation data: YELP evaluation gains three additional human references per test sentence because its single-reference setup makes automatic evaluation less reliable.
3.3 Training Details
Training uses LSTM encoder-decoder transfer models, separate pre-training and dual-learning optimization settings, and comparisons against established unsupervised systems.
- Model and optimization: Both transfer models are basic LSTM-based encoder-decoder models with 256 hidden units and 300-dimensional embeddings learned from scratch.
- Model and optimization: Adam uses learning rates of 10^-3 for pre-training and 10^-5 for dual learning, with batch sizes 32 and 128, respectively.
- Baselines: The comparison includes StyleEmbed, MultiDec, CrossAlign, BackTrans, Template, Retri, Del, DelRetri, Unpaired, and UnsuperMT.
3.5 Evaluation Metrics
Evaluation combines automatic measures of style accuracy and content preservation with human ratings of style, content, and fluency.
- Automatic evaluation: Automatic evaluation uses a pre-trained TextCNN style classifier for target-style accuracy and BLEU against human references for content preservation.
- Automatic evaluation: Geometric and harmonic means, G2 and H2, summarize overall automatic performance from style accuracy and BLEU.
- Human evaluation: Three linguistically trained annotators rate target-style accuracy, content preservation, and fluency from 1 to 5.
- Human evaluation: A transferred text is successful when it receives a rating of 4 or 5 on all three human-evaluation criteria.
3.6 Results and Discussions
DualRL achieves the strongest overall automatic and human-evaluation performance, although classifier accuracy does not fully align with human judgments across datasets.
- Automatic results: Over 8 BLEU points averaged across two datasets separate DualRL from the other systems, while DualRL also achieves the best overall automatic scores G2 and H2.
- Automatic results: DualRL does not achieve the highest style-classifier accuracy because retrieval can favor ACC and small edits can mislead the classifier.
- Human results: DualRL obtains the best human average score and more than 10% successful transferred instances averaged across the two datasets.
- Human results: All systems perform better on YELP than GYAFC, indicating that formality transfer is more challenging than sentiment transfer.
- Correlation analysis: BLEU significantly correlates with content preservation but not fluency, while ACC-style correlations vary by dataset and G2/H2 correlate with human averages.
3.7 Ablation Study
The ablation study shows that RL improves style accuracy but can harm readability, while MLE preserves readability without directly controlling style. Combining RL and MLE achieves the strongest balance on the YELP dataset.
- Ablation results: RL-only training increases style accuracy but decreases BLEU-based content preservation and fluency.RL can reward sentences that satisfy the style classifier while remaining low-quality or less readable.
- Ablation results: MLE-only training produces higher BLEU-based content preservation and fluency than RL-only training, but its style accuracy declines.MLE acts as a conditional language-model objective and lacks a mechanism for directly controlling style accuracy.
- Ablation setup: RL denotes removing specified algorithm steps, while MLE denotes removing a different set of steps from Algorithm 2.These definitions identify the training components removed in the ablations.
- Combined training: The RL+MLE combination achieves the best BLEU score without compromising style accuracy.Compared with MLE-only training, it improves H2/G2 by over 3.5 absolute points and raises success rate by 13%.
3.8 Case Study
The case study compares representative systems on YELP and GYAFC examples, highlighting different trade-offs among content preservation, style transfer, and fluency.
- System comparisons: CrossAlign tends to sacrifice content preservation because it learns a style-independent content representation through adversarial training.Template and Del-Retri better preserve content but may fail when style is expressed implicitly.
- System comparisons: DualRL achieves a better balance among preserving content, changing style, and improving fluency.
3.9 Error Analysis
The error analysis identifies analogies and metaphors of sentiment as a typical failure case, because preserving literal content and transferring style can conflict.
- Failure cases: Analogies and metaphors in sentiment are difficult for the proposed system and existing methods to handle.The example “canned tuna fish” refers figuratively to overcooking rather than its literal content.
- Failure cases: These examples create a trade-off between preserving the original content and transferring the style.
4 Related Work
Related work addresses unsupervised text style transfer through style-independent content representations or pseudo-parallel data, but the lack of parallel data remains a central challenge.
- Motivation: The lack of parallel data remains the major challenge in text style transfer.Parallel data consists of aligned sentences with the same content but different styles and is difficult to collect.
- Content representation: Early approaches learn style-independent content representations using adversarial networks, VAEs, or style discriminators.
- Pseudo-parallel data: Other approaches construct pseudo-parallel data through back-translation and adapt unsupervised machine-translation methods.Learning from pseudo-parallel data is accompanied by data-quality problems.
5 Conclusion and Future Work
The paper concludes that direct one-step mappings trained with DualRL effectively address unsupervised text style transfer without explicit content-style separation. It also identifies training complexity from pre-training and annealed pseudo teacher-forcing as an open issue, while suggesting broader applications for the generic architecture.
- DualRL trains direct source-to-target and target-to-source mappings using automatically generated supervision, avoiding explicit content-style separation.
- Experiments on sentiment and formality transfer show significant improvements over previous approaches.
- Pre-training and annealed pseudo teacher-forcing are effective but make training complicated, motivating reinforcement learning from scratch.
- Future work may extend DualRL to other unsupervised sequence-to-sequence generation tasks lacking parallel data.