Source-linked AI summary
Style Transfer in Text: Exploration and Evaluation
Zhenxin Fu, Xiaoye Tan, Nanyun Peng, Dongyan Zhao, Rui Yan
TL;DR
Language style transfer is constrained by limited parallel data and unreliable evaluation, motivating methods that can learn from non-parallel corpora. The paper proposes two adversarially trained models that separate content and style, along with metrics for transfer strength and content preservation. Across paper-news title and sentiment-transfer tasks, the models support non-parallel style transfer, while content preservation correlates highly with human judgments.
Problem
Language style transfer lacks sufficient parallel data and reliable evaluation metrics, limiting how models and outputs can be assessed.
Method
The paper proposes multi-decoder and style-embedding models that separate content and style representations using adversarial training, plus metrics for transfer strength and content preservation.
Results
Experiments on paper-news title and positive-negative review transfer show that the proposed models learn from non-parallel data and that content preservation correlates highly with human judgments.
Takeaways & Limitations
The proposed models and evaluation metrics provide a basis for studying text style transfer with non-parallel corpora.
Takeaways & Limitations
The paper reports both metrics separately because a scenario-dependent weighted integration of transfer strength and content preservation remains future work.
Abstract
from arXiv · showhide
Style transfer is an important problem in natural language processing (NLP). However, the progress in language style transfer is lagged behind other domains, such as computer vision, mainly because of the lack of parallel data and principle evaluation metrics. In this paper, we propose to learn style transfer with non-parallel data. We explore two models to achieve this goal, and the key idea behind the proposed models is to learn separate content representations and style representations using adversarial networks. We also propose novel evaluation metrics which measure two aspects of style transfer: transfer strength and content preservation. We access our models and the evaluation metrics on two tasks: paper-news title transfer, and positive-negative review transfer. Results show that the proposed content preservation metric is highly correlate to human judgments, and the proposed models are able to generate sentences with higher style transfer strength and similar content preservation score comparing to auto-encoder.
Introduction
Text style transfer matters for NLP applications but remains behind other domains because parallel corpora and reliable evaluation metrics are scarce. This paper addresses these gaps with non-parallel models and metrics for transfer strength and content preservation.
- Language style transfer supports applications including paper-to-news title conversion and poetry generation.
- Parallel data is difficult to obtain for style-transfer tasks, motivating algorithms that learn without parallel data.Only a small number of academic news reports have corresponding papers.
- Separating textual content from style remains an open research problem in NLP.
- The proposed evaluation metrics measure transfer strength and content preservation despite the lack of ground-truth references.
- Two models use multi-task learning and adversarial training to learn style transfer from non-parallel data.
- Experiments on paper-news and positive-negative review transfer found that the models have different strengths and that content preservation correlates highly with human judgments.
Contributions
The paper contributes a paper-news title dataset, evaluation metrics covering transfer strength and content preservation, and two models for learning style transfer without parallel corpora.
- The paper composes a paper-news title dataset to facilitate research in language style transfer.
- It proposes evaluation metrics for transfer strength and content preservation that are highly correlated with human evaluation.
- It proposes and evaluates two models that learn style transfer without parallel corpora, with different advantages under different scenarios.
Related Work
Prior work spans image style transfer, text transfer with parallel data, non-parallel sentiment methods, style-controlled generation, and adversarial representation separation. The paper distinguishes its approach through source-conditioned transfer and content-preservation evaluation.
- Computer-vision methods separate and recombine image content and style, but single sentences or short articles provide insufficient style information for direct NLP transfer.
- CycleGAN performs unpaired image translation, whereas discrete text makes cycle training difficult to implement.
- Modern-to-Shakespearean text transfer uses parallel data and a paired-word dictionary, resources unavailable in most style-transfer tasks.
- Earlier non-parallel text methods used variational autoencoders, but reported evaluation was limited or insufficient for style transfer.
- Style analysis and style-controlled generation model linguistic style or condition generation, but do not transfer style from source sentences.
- Adversarial networks have been used to learn domain-invariant, shared, and private representations across domains or tasks.
Model
The paper builds two seq2seq style-transfer models around content representations separated from style. Multi-decoder uses style-specific decoders, while style-embedding conditions one decoder on a learned style vector.
- Model: Both models encode an input sentence into content representations intended to contain content information without style information.
- Multi-decoder Model: The multi-decoder model uses one decoder per style to generate outputs in the corresponding style.
- Background: Auto-encoder Seq2seq Model: The auto-encoder baseline uses an encoder representation and decoder to recover the input sequence, with y set equal to x for style transfer.
- Background: Auto-encoder Seq2seq Model: GRU encoders control information flow with reset and update gates, while the decoder predicts each next token from previous tokens using softmax probabilities.
- Multi-decoder Model: The multi-decoder generation loss sums the losses of its style-specific decoders, and its final objective combines generation with two adversarial losses.
- Style-embedding Model: Style embeddings are jointly trained with the encoder-decoder generation objective, while adversarial losses remain shared with the multi-decoder formulation.
- Style-embedding Model: The style-embedding model concatenates content c with a style embedding e and feeds the result to a single decoder.
- Training: Training uses Adadelta, validation perplexity for parameter selection, alternating decoder updates for multi-decoder, and joint shuffled-data training for style-embedding.
Evaluation
The paper evaluates style transfer using separate metrics for transfer strength and content preservation, addressing the lack of parallel ground-truth references. Transfer strength uses a classifier, while content preservation compares source and target sentence embeddings; the paper reports that a weighted combination is left for future work.
- Evaluation Metrics: The proposed evaluation framework measures two distinct aspects of style transfer: transfer strength and content preservation.This avoids relying only on output similarity to unavailable parallel references.
- Transfer Strength: Transfer strength measures whether the target style is transferred using an LSTM-sigmoid classifier.The accuracy is defined as Nright/Ntotal, where Nright counts correctly transferred test cases.
- Transfer Strength: The classifier assigns paper and news styles using a 0.5 output threshold.Outputs at or below 0.5 correspond to paper, while outputs above 0.5 correspond to news.
- Content Preservation: Content preservation compares source and target sentence embeddings using cosine distance.Sentence embeddings use max, min, and mean pooling of word embeddings.
- Evaluation Metrics: A single F1-style integration is not considered suitable because different applications may prioritize transfer strength or content preservation.The paper reports both metrics separately and leaves weighted integration for future work.
Experimental Setup
The experiments use two non-parallel datasets—paper-news titles and positive-negative reviews—with fixed training, validation, and test partitions. The study explores multiple parameter settings and trains separate classifiers and embedding-based metrics for evaluation.
- Datasets: The experiments evaluate the proposed methods on paper-news title transfer and positive-negative review transfer using non-parallel corpora.Both datasets are divided into training, validation, and test sets.
- Datasets: Validation and test sets each contain 2,000 sentences, with the remaining data used for training.The partition is kept the same for model training and evaluation.
- Datasets: The paper-news dataset combines academic paper titles with science-and-technology news titles filtered from a repository containing 422,937 titles.The news portion was filtered to 108,503 science-and-technology titles.
- Datasets: The review dataset samples 400,000 positive and 400,000 negative reviews from Amazon product reviews.The source collection spans multiple product domains and covers reviews from 1996 to 2014.
- Parameter Exploration: Because the paper is exploratory, it compares several embedding, hidden-state, and style-embedding parameter settings rather than selecting one best configuration.The explored ranges differ between the paper-news and review tasks.
- Evaluation Setup: The transfer-strength classifier reaches 98.8% validation accuracy for paper-news titles and 84.8% for positive-negative reviews.In both tasks, classifier training stops after two epochs.
- Evaluation Setup: Content preservation uses pretrained 100-dimensional word embeddings, filtering sentiment words for the positive-negative review task.A positive and negative word dictionary performs the filtering.
Results and Analysis
The proposed content-preservation metric correlates highly with human judgments, while model results reveal a trade-off between transfer strength and content preservation across both tasks.
- Comparison with Human Judgments: 0.5656 Spearman correlation (p-value<0.0001) indicates high agreement between content preservation scores and human judgments.The comparison used 200 paper-news transferred pairs rated by three people.
- Overall Results: Higher transfer strength is negatively correlated with content preservation in both tasks and across all models.The proposed models’ trade-off curves appear less steep than the auto-encoder’s.
- Paper-News Title Transfer: In paper-news transfer, multi-decoder achieves stronger transfer, while style-embedding preserves more content; neither is a clear overall winner.Both proposed models achieve considerably high scores on both metrics.
- Paper-News Title Transfer: 0.2–0.6 transfer strength and 0.89–0.95 content preservation show that style-embedding covers a broad range across hyper-parameters.The multi-decoder model generally produces higher transfer strength but lower content preservation, motivating different model choices for different requests.
- Positive-Negative Review Transfer: In positive-negative review transfer, multi-decoder performs better than style-embedding on both metrics.The result is described as the red line lying above and to the right of the green line.
- Content Preservation Baseline: 0.609 and 0.863 are the estimated lower bounds of content preservation for the paper-news and positive-negative review datasets, respectively.The proposed models achieved much higher content preservation scores than these lower bounds.
- Qualitative Study: The qualitative cases show auto-encoder outputs nearly identical to inputs, whereas the proposed models alter selected words or phrases while preserving most content.Both proposed models perform better on positive-negative transfer than on paper-news transfer in the sampled examples.
Conclusions
The paper studies style transfer from non-parallel corpora by proposing two models, two evaluation metrics, and two datasets. Experiments support learning style transfer from non-parallel data and show that content preservation correlates highly with human judgment, while broader evaluation remains future work.
- The paper proposes two models, two evaluation metrics, and two datasets for studying style transfer with non-parallel corpora.
- Experiments show that the proposed models can learn style transfer from non-parallel data and that content preservation correlates highly with human judgment.
- Future work will develop more comprehensive metrics, including sentence fluency, and conduct a thorough human evaluation study.
Appendix: Dimension Influence
Across the paper-news experiments, increasing model dimensions generally improves sentence recovery and content preservation, while reducing transfer strength in the auto-encoder and making style-embedding dimension especially influential.
- Auto-encoder: Increasing auto-encoder dimensions improves source-sequence recovery and content preservation but reduces transfer strength.The passage attributes the lower transfer strength to decreasing indeterminacy.
- Multi-decoder: Increasing multi-decoder dimensions similarly improves the model’s ability to recover the input sentence.
- Style-embedding: Style-embedding dimension has a larger influence than encoder dimension in most paper-news settings.Larger style-embedding dimensions preserve more content and improve decoder sentence recovery.