PaperlayerView exact revision on arXiv

Source-linked AI summary

Evaluating Style Transfer for Text

Remi Mir, Bjarke Felbo, Nick Obradovich, Iyad Rahwan

arXiv:1904.02295v1cs.CL

TL;DR

Text style transfer lacks standard evaluation practices and agreement on which aspects to measure. This paper examines human and automated evaluation across style intensity, content preservation, and naturalness; it reports stronger human-score correlations for its automated metrics and tradeoffs among model aspects.

  • Problem

    Style transfer research lacks standard evaluation practices and a clear definition of which aspects to evaluate.

    Source: §1 · p. 1

  • Method

    The paper evaluates style intensity, content preservation, and naturalness, examining human scoring reliability and proposing automated measures including direction-corrected Earth Mover’s Distance and adversarial classification.

    Source: §4.3 · p. 4, §6 · p. 8

  • Results

    The proposed automated metrics correlated more strongly with human scores than existing automated metrics on a binary sentiment dataset, while content preservation and naturalness declined as transfer intensity increased across models.

    Source: §5.4 · p. 7, §6 · p. 8

  • Takeaways & Limitations

    Automation can facilitate evaluation when collecting human scores is infeasible because of cost or limited time.

    Source: §6 · p. 8

    The experiments focus on the binary case because creating a high-quality, multi-label style-transfer evaluation dataset is out of scope.

    Source: §5 · p. 4

Abstract

from arXiv · show

Research in the area of style transfer for text is currently bottlenecked by a lack of standard evaluation practices. This paper aims to alleviate this issue by experimentally identifying best practices with a Yelp sentiment dataset. We specify three aspects of interest (style transfer intensity, content preservation, and naturalness) and show how to obtain more reliable measures of them from human evaluation than in previous work. We propose a set of metrics for automated evaluation and demonstrate that they are more strongly correlated and in agreement with human judgment: direction-corrected Earth Mover's Distance, Word Mover's Distance on style-masked texts, and adversarial classification for the respective aspects. We also show that the three examined models exhibit tradeoffs between aspects of interest, demonstrating the importance of evaluating style transfer models at specific points of their tradeoff plots. We release software with our evaluation metrics to facilitate research.

1 Introduction

Text style transfer changes an input’s style while seeking to retain its content, but the field lacks standard evaluation practices and agreement on which aspects to assess. The paper examines more reliable human and automated evaluation and highlights tradeoffs among aspects of model performance.

1 Introduction

  • Style transfer changes an input attribute while retaining nonattribute-related content, though this definition assumes style and content words can be distinguished.The paper gives sentiment changes as examples and notes that the distinction is a simplifying assumption.Source: §1 · p. 1
  • The field lacks standard evaluation practices and a clear definition of which aspects should be evaluated.Existing evaluations often rely on human ratings, which can be expensive and time-consuming.Source: §1 · p. 1
  • The paper studies more reliable human measures and automated methods, and shows that the examined models trade off aspects of performance.It also releases software with its evaluation metrics and argues for evaluating models at specific points on tradeoff plots.Source: §1 · p. 1

2 Aspects of Evaluation

The paper evaluates style transfer along three aspects: transfer intensity, content preservation, and naturalness. Considering all three helps distinguish models that succeed on one aspect but perform poorly on another.

2 Aspects of Evaluation

  • The evaluation framework considers style transfer intensity, content preservation, and naturalness.Intensity measures style difference, content preservation measures input-output content similarity, and naturalness concerns whether output appears human-written.Source: §2 · p. 1
  • Models should be compared across all three aspects because strong performance on one can coexist with weak performance on another.A model that reconstructs input well may preserve content and naturalness but achieve little or no style transfer.Source: §2 · pp. 1–2
  • Past evaluations combine human and automated techniques in different ways, complicating comparisons across papers.Table 1 summarizes the evaluation techniques used in prior work.Source: Table 1 · p. 2

3 Related Work

Prior style-transfer evaluations use differing combinations of human and automated measures, making scores difficult to compare. The paper reviews limitations of commonly used metrics for style transfer, content preservation, and naturalness.

3 Related Work

  • Prior evaluations of CAAE, ARAE, and DAR use different combinations of human and automated measures, instructions, and scales.The authors focus on these three models partly because human evaluation is costly.Source: §3 · p. 2
  • Target-style scoring may not be comparable across papers because evaluations use different classifiers.The approach counts outputs classified as having the target style.Source: §3 · p. 2
  • BLEU penalizes intended differences between input and output, despite style transfer requiring changes to words.Prior work used BLEU to measure content preservation, although it was designed for machine translation evaluation.Source: §3 · p. 2
  • Human naturalness ratings use varied labels, while grammaticality alone can miss semantically nonsensical text and may not match input quality.Human-written inputs may themselves be imperfectly grammatical or readable.Source: §3 · p. 2
  • Perplexity measures language-model prediction of reference text, but style-transfer outputs need not be gold-standard texts and its correlation with human judgments is unknown.The paper treats fluency as a subset of naturalness.Source: §3 · p. 2

4 Methods

The methods construct a dataset-specific style lexicon and define human and automated evaluations for style transfer intensity, content preservation, and naturalness. The evaluation designs account for stylistic changes while comparing outputs with their inputs or with human text.

4.1 Construction of Style Lexicon

  • The style lexicon is built from the highest-absolute-weight features of a logistic regression trained on dataset style labels, favoring precision over recall to avoid removing content words.Static lexica may not scale beyond binary sentiment and may miss context-dependent stylistic meaning.Source: §4.1 · p. 3

4.2 Human Evaluation

  • Human raters compare input and output style, judge content similarity after style words are masked, and choose which text is more natural without knowing which is the output.The style-intensity comparison uses a 1–5 scale from identical to completely different styles.Source: §4.2 · p. 3

4.3 Automated Evaluation

  • Direction-corrected Earth Mover’s Distance measures movement between input and output style distributions, including movement toward a target even when it is not the top predicted style.The score is negated when the relative style change moves away from the target; the distribution-based metric supports binary and non-binary styles.Source: §4.3 · p. 4
  • Content-preservation metrics are tested on style-removed or style-masked texts, with BLEU, METEOR, Word Mover’s Distance, and greedy embedding matching among the candidates.Masking replaces lexicon words with a placeholder; removal deletes them, reducing penalization for expected style changes.Source: §4.3 · p. 4
  • Adversarial unigram and neural logistic regression classifiers distinguish human inputs from model outputs, with greater ability to fool a classifier indicating greater naturalness.The methods also use language-model perplexity to assess adversarial classification’s effectiveness as a metric.Source: §4.3 · p. 4

5 Experiments and Results

Experiments on Yelp binary sentiment transfer compare CAAE, ARAE, and DAR using human and automated evaluations across style transfer intensity, content preservation, and naturalness. The results show that models trade off these aspects, so comparisons depend on where each model is evaluated on its tradeoff plot.

5 Experiments and Results

  • The experiments focus on CAAE, ARAE, and DAR using transfers based on the Yelp binary sentiment dataset; high-quality multi-label evaluation data is out of scope.Three Amazon Turk raters evaluated 244 texts per aspect per model, half transferred in each sentiment direction.Source: §5 · pp. 4–5

5.1 Style Transfer Models

  • The evaluated models vary their training parameters to produce different content-preservation and style-intensity outcomes; DAR also varies its removal threshold at test time without retraining.CAAE varies adversarial-loss weight ρ, ARAE varies adversarial-loss weight λ, and DAR tests several γ values after training.Source: §5.1 · p. 5

5.2 Human Evaluation

  • Style masking improves inter-rater reliability for content-preservation judgments, while relative naturalness judgments are more reliable than absolute judgments.The content-preservation kappa is 0.297 for style-masked texts versus 0.173 for unmasked texts; relative naturalness judgments exceed absolute judgments across thresholds τ ∈ {2, 3}.Source: §5.2 · pp. 5–6, Table 5 · p. 5, Table 6 · p. 5

5.3 Automated Evaluation

  • Direction-corrected Earth Mover’s Distance correlates more strongly with human style-intensity scores than target-style scoring across fastText and textcnn classifiers.For content preservation, WMD generally correlates most strongly across text-modification settings, with style masking yielding higher average correlation; adversarial classifiers agree more with relative than absolute human naturalness judgments, while sentence-level perplexity has no significant correlation.Source: §5.3 · pp. 6–7

5.4 Aspect Tradeoffs

  • Across the examined models, increasing style transfer intensity tends to reduce content preservation and naturalness; ARAE and DAR perform comparably for the tested hyperparameters and show less naturalness decrease than CAAE.The plots use direction-corrected EMD, normalized inverse WMD, and the percentage of outputs judged more natural than inputs; the ideal tradeoff may vary by dataset.Source: §5.4 · pp. 7–8

6 Discussion

The discussion argues that evaluation should specify style-transfer aspects and use human measures and automated metrics that better align with human judgments. It also highlights tradeoffs among aspects and identifies further evaluation settings to explore.

6 Discussion

  • The proposed metrics showed stronger correlations with human scores than existing automated metrics on a binary sentiment dataset.The metrics are direction-corrected EMD, WMD on style-masked texts, and adversarial classification for the respective aspects.Source: §6 · p. 8

6.1 Human Evaluation

  • For content preservation, masking style words addresses rater-dependent style identification, while relative scoring for naturalness achieved higher inter-rater reliability than absolute scoring.The style lexicon is constructed conservatively to avoid capturing content words, which are critical to content-preservation evaluations.Source: §6.1 · p. 8

6.2 Automated Evaluation

  • Direction-corrected EMD accounts for a spectrum of styles, supports binary and non-binary datasets, and penalizes outputs whose style distribution moves away from the target.Existing external classifiers can provide style distributions for the input and output texts.Source: §6.2 · p. 8
  • WMD compares input and output word embeddings, while BLEU may correlate less strongly with human judgment because style transfer typically has only one reference text.The single reference reduces the likelihood of n-gram overlap that BLEU relies on.Source: §6.2 · p. 8
  • Agreement between adversarial classifiers and human relative naturalness scores supports classifiers for automated evaluation; tradeoff plots enable simultaneous model comparisons across aspects.Intersections or near intersections indicate models can achieve similar results at the corresponding hyperparameters.Source: §6.2 · pp. 8–9

6.3 Future Research

  • Sentence-level perplexity of the output alone is not meaningful for automated naturalness evaluation; a proposed extension would also consider input perplexity.The proposed direction parallels the use of both input and output texts in automated naturalness classifiers.Source: §6.3 · p. 9
  • The paper calls for evaluation on datasets with different styles or numbers of style classes, noting that less distinct style and content vocabularies may yield naturally low content-preservation scores.Such scores are described as a dataset-specific trend that can be visualized in tradeoff plots.Source: §6.3 · p. 9
Loading 1904.02295v1…