Source-linked AI summary
Unpaired Sentiment-to-Sentiment Translation: A Cycled Reinforcement Learning Approach
Jingjing Xu, Xu Sun, Qi Zeng, Xuancheng Ren, Xiaodong Zhang, Houfeng Wang, Wenjie Li
TL;DR
Sentiment-to-sentiment translation seeks to change sentiment while preserving content, but supervised parallel data are unavailable. The paper addresses this with cycled reinforcement learning that collaborates between neutralization and emotionalization modules. On two review datasets, the approach significantly outperforms state-of-the-art systems, especially in semantic preservation, while automatic evaluation metrics remain limited.
Problem
Sentiment-to-sentiment translation must alter underlying sentiment while preserving non-emotional semantic content, but the lack of parallel training data makes this difficult.
Method
Cycled reinforcement learning collaborates between neutralization and emotionalization modules to separate semantic content from sentiment and train with unpaired data.
Results
The approach significantly outperforms state-of-the-art systems on two review datasets, especially in semantic preservation.
Takeaways & Limitations
The method substantially improves content preservation for unpaired sentiment-to-sentiment translation.
Takeaways & Limitations
Human evaluation shows a smaller baseline gap than automatic evaluation, and BLEU can be noisy because overlapping n-grams include function words.
Abstract
from arXiv · showhide
The goal of sentiment-to-sentiment "translation" is to change the underlying sentiment of a sentence while keeping its content. The main challenge is the lack of parallel data. To solve this problem, we propose a cycled reinforcement learning method that enables training on unpaired data by collaboration between a neutralization module and an emotionalization module. We evaluate our approach on two review datasets, Yelp and Amazon. Experimental results show that our approach significantly outperforms the state-of-the-art systems. Especially, the proposed method substantially improves the content preservation performance. The BLEU score is improved from 1.64 to 22.46 and from 0.56 to 14.06 on the two datasets, respectively.
1 Introduction
Sentiment-to-sentiment translation must change sentiment while preserving non-emotional content, but parallel-data scarcity makes this difficult. The proposed cycled reinforcement learning approach separates semantic content from sentiment and enables training on unpaired data, improving content preservation.
- Task and challenge: Sentiment-to-sentiment translation changes a sentence’s underlying sentiment while preserving its non-emotional semantic content.The task is framed as a special style-transfer problem with applications including review transformation and news rewriting.
- Task and challenge: Existing methods often change sentiment but fail to preserve semantic content because emotional and semantic information are mixed in one dense hidden vector.For example, a food review can be transformed into a sentence about a different topic.
- Proposed approach: The cycled reinforcement learning approach uses neutralization and emotionalization modules to separate non-emotional semantic information from sentiment.Neutralization filters emotional words, while emotionalization adds sentiment to the neutralized semantic content.
- Proposed approach: Cycled training reconstructs an emotional input after neutralization, then uses policy-gradient rewards from generated-text quality to improve neutral-word selection.The rewards assess target-sentiment matching and content preservation, guiding the neutralization module.
- Contribution: The method enables training with unpaired data containing only reviews and sentiment labels.This addresses the bottleneck created by the lack of supervised parallel data.
- Results: The approach significantly outperforms state-of-the-art systems, especially on content preservation.The supplied introduction states the qualitative comparison but does not provide the associated BLEU values.
2 Related Work
Prior unpaired language style-transfer methods use latent representations, adversarial training, or attribute discriminators, but perform poorly at preserving non-emotional semantic content in sentiment translation. This work instead uses collaboration between neutralization and emotionalization modules rather than an inverse translation task.
- Unpaired style transfer: Earlier unpaired style-transfer methods combine variational auto-encoders with attribute discriminators, adversarial networks, or content-style representation separation.These approaches target effective style control or removal of style information from input content.
- Limitations of prior work: When applied to sentiment-to-sentiment translation, previously described models have poor preservation of non-emotional semantic content.Content preservation is identified as an indispensable evaluation metric for this task.
- Proposed distinction: The proposed method builds supervised training pairs by reconstructing the original sentence despite lacking parallel data.This connects the method to back reconstruction while using a different architectural strategy.
- Proposed distinction: Unlike back reconstruction in machine translation, the approach does not introduce an inverse task; it collaborates between neutralization and emotionalization modules.The two modules jointly support sentiment translation and semantic-content preservation.
3 Cycled Reinforcement Learning for Unpaired Sentiment-to-Sentiment Translation
The method combines neutralization and emotionalization modules in a cycled reinforcement-learning framework. A sentiment classifier supports pre-training, while rewards encourage neutralization that preserves content and enables sentiment changes.
- Overview: The approach uses a neutralization module to extract non-emotional semantic content and an emotionalization module to add sentiment.The two modules are trained together through cycled reinforcement learning.
- Neutralization Module: The neutralization module identifies non-emotional words with an LSTM, then discretizes attention weights to select the retained semantic content.The selected content is passed to the emotionalization module, and cross-entropy loss is used during neutralization pre-training.
- Pre-training: The self-attention sentiment classifier supplies attention-based supervision for pre-training the neutralization module.Its attention weights are used to identify emotional and neutral words; classifier accuracy reaches 89% and 90% on the two datasets.
- Emotionalization Module: The emotionalization module uses an encoder and two sentiment-specific decoders to reconstruct or transform sentences from neutralized content.The input sentiment determines whether the positive or negative decoder is used.
- Cycled Reinforcement Learning: Cycled training reconstructs the original sentence with its source sentiment and uses opposite sentiment to produce the translation target.Policy gradients optimize the neutralization module because its discrete word choices are not differentiable.
- Reward: The reward combines sentiment confidence and BLEU through their harmonic mean, guiding neutralization and subsequently improving emotionalization.A simpler continuous-attention pre-training method performs much worse than explicitly removing emotional words with discrete attention weights.
4 Experiment
The evaluation uses unpaired positive and negative reviews from Yelp and Amazon. Reviews are filtered and converted into sentence-level text-sentiment pairs for experimentation.
- Datasets: Experiments evaluate the method on two review datasets containing user ratings, with positive and negative reviews kept unpaired.Ratings above three define positive reviews, while ratings below three define negative reviews.
- Preprocessing: The datasets are converted to sentence-level examples by filtering reviews exceeding 20 words and extracting each review’s first sentence.The first sentence is selected because it usually expresses the review’s core idea.
- Yelp: The processed Yelp dataset contains 1.43M training, 10K validation, and 5K testing pairs.
- Amazon: The processed Amazon dataset contains 367K training, 10K validation, and 5K testing pairs.
4.2 Training Details
Training settings are selected using validation performance, with fixed classifier training, model dimensions, optimization parameters, and gradient clipping.
- Training Details: Hyperparameters are tuned on validation sets, and the self-attention sentiment classifier is trained for 10 epochs on both datasets.
- Training Details: The experiments use hidden size 256, embedding size 128, vocabulary size 50K, learning rate 0.6, and batch size 64.
- Training Details: The reward weight β is 0.5, Adagrad is used for optimization, and gradients are clipped when their norm exceeds 2.
4.3 Baselines
The evaluation compares the proposed method with established style-transfer baselines, including CAAE and MDAL.
- Baselines: The study compares its proposed method with state-of-the-art systems.
- CAAE: CAAE uses refined alignment of latent representations in hidden layers for style transfer and serves as an adapted baseline.
- MDAL: MDAL uses multiple decoders and adversarial learning to separate style and content representations in hidden layers.
4.4 Evaluation Metrics
The evaluation combines automatic metrics for sentiment transfer, content preservation, and overall performance with human judgments of transformed text. Automatic scores provide indications of transfer quality but cannot fully assess transferred-text quality.
- Automatic Evaluation: ACC measures whether generated sentences match the designated sentiment using a pretrained TextCNN classifier.TextCNN reaches 89% and 88% accuracy on the two datasets.
- Automatic Evaluation: BLEU measures content preservation by comparing transferred sentences with source sentences through overlapping lexical n-grams.It is used as the content-preservation metric for the transferred text.
- Automatic Evaluation: G-score summarizes overall performance as the geometric mean of ACC and BLEU.It is used as a single-number measure combining sentiment transformation and content preservation.
- Human Evaluation: Human evaluation supplements automatic metrics because quantitative evaluation cannot accurately assess transferred-text quality.Annotators score transformed sentences for sentiment and semantic similarity without knowing which system produced them.
4.5 Experimental Results
The proposed method achieves the best overall performance on Yelp and Amazon, mainly by substantially improving content preservation. Human evaluations corroborate this advantage, while also revealing remaining tension between sentiment transformation and semantic preservation.
- Automatic evaluation: 1.17 and 1.64 BLEU for CAAE and MDAL on Yelp, versus 0.56 and 0.27 on Amazon, indicate poor baseline content preservation.BLEU evaluates semantic content preservation, and irrelevant generated text worsens overall performance even when sentiment is correct.
- Automatic evaluation: The proposed method achieves the best overall performance on both datasets, with BLEU improving from 1.64 to 22.46 and from 0.56 to 14.06.The authors attribute these gains to explicitly filtering emotional words, preserving extracted content, and using transferred-text quality as reinforcement-learning reward.
- Human evaluation: Human semantic scores increase from 3.87 to 5.08 on Yelp and from 3.22 to 4.67 on Amazon.Human evaluation reports improved semantic preservation and the best overall performance for the proposed model.
- Human evaluation: With improved content preservation, the proposed method has lower sentiment accuracy than CAAE on both datasets.The authors identify simultaneous promotion of sentiment transformation and content preservation as an issue requiring further study.
- Evaluation limitations: Automatic and human evaluations agree, but human evaluation shows a smaller performance gap between the baselines and the proposed method.The authors note that sentiment classifiers can produce noisy results, while BLEU counts function-word overlap alongside content-word overlap.
4.6 Incremental Analysis
The incremental analysis finds that the full method performs best, with pre-training and cycled reinforcement learning complementing each other. The neutralization module improves sentiment transformation while slightly reducing content preservation, and accurately removes emotional words with little non-emotional information loss.
- The full method with all components achieves the best overall performance.
- G-score improves from 32.77 to 34.66 and from 26.46 to 27.87 on the two datasets with pre-training.
- G-score improves from 34.66 to 42.38 and from 27.87 to 31.45 when cycled reinforcement learning is combined with pre-training.
- The neutralization mechanism greatly improves sentiment transformation but slightly reduces content preservation.
- The neutralization module accurately removes emotional words with little loss of non-emotional information.
4.7 Error Analysis
The method still has failure cases involving sentiment-conflicted and neutral sentences. These cases arise from incomplete sentiment removal or failure to add the target sentiment.
- Sentiment-conflicted sentences reveal that the original sentiment is not always removed completely.
- Unseen emotional words and implicitly expressed sentiment contribute to failures on sentiment-conflicted inputs.
- For neutral sentences, the decoder sometimes generates semantic content without adding the target sentiment.
- Handling complex sentiment expressions and developing a better sentiment-aware decoder remain future-work directions.
5 Conclusions and Future Work
The paper addresses unpaired sentiment-to-sentiment translation with cycled reinforcement learning trained without parallel data. Experiments on two review datasets show substantial improvements over state-of-the-art systems, especially in semantic preservation, while finer-grained sentiment-strength changes remain future work.
- The proposed cycled reinforcement learning approach enables sentiment translation without parallel training data.
- Experiments on two review datasets show that the method substantially outperforms state-of-the-art systems, especially in semantic preservation.
- Future work will explore fine-grained translation that changes sentiment strength in addition to reversing sentiment.