PaperlayerView exact revision on arXiv

Source-linked AI summary

A Probabilistic Formulation of Unsupervised Text Style Transfer

Junxian He, Xinyi Wang, Graham Neubig, Taylor Berg-Kirkpatrick

arXiv:2002.03912v3cs.CLcs.LG

TL;DR

Parallel data is scarce for text style transfer, while existing unsupervised approaches often rely on heuristic objectives or restrictive assumptions. This paper models non-parallel data as a partially observed parallel corpus and learns with amortized variational inference. Its approach improves on prior unsupervised objectives across style-transfer tasks and matches the state of the art in unsupervised machine translation.

  • Problem

    Parallel corpora are difficult to obtain for text style transfer, while existing unsupervised methods often use heuristic objectives or restrictive independence assumptions.

    Source: §1 · pp. 1–2

  • Method

    The model treats non-parallel data from two domains as a partially observed parallel corpus and uses amortized variational inference to derive a practical learning objective.

    Source: §1 · p. 2, §3.2 · p. 4

  • Results

    The probabilistic objective outperforms manually defined unsupervised objectives across style-transfer tasks and matches the state of the art in unsupervised machine translation.

    Source: §1 · p. 2, §6 · p. 10

  • Takeaways & Limitations

    The results support probabilistic principles as a useful guide for designing objectives in deep neural systems.

    Source: §1 · p. 2

    REINFORCE was difficult to stabilize and did not yield reasonable results in the reported comparison.

    Source: §5.4 · p. 9

Abstract

from arXiv · show

We present a deep generative model for unsupervised text style transfer that unifies previously proposed non-generative techniques. Our probabilistic approach models non-parallel data from two domains as a partially observed parallel corpus. By hypothesizing a parallel latent sequence that generates each observed sequence, our model learns to transform sequences from one domain to another in a completely unsupervised fashion. In contrast with traditional generative sequence models (e.g. the HMM), our model makes few assumptions about the data it generates: it uses a recurrent language model as a prior and an encoder-decoder as a transduction distribution. While computation of marginal data likelihood is intractable in this model class, we show that amortized variational inference admits a practical surrogate. Further, by drawing connections between our variational objective and other recent unsupervised style transfer and machine translation techniques, we show how our probabilistic view can unify some known non-generative objectives such as backtranslation and adversarial loss. Finally, we demonstrate the effectiveness of our method on a wide range of unsupervised style transfer tasks, including sentiment transfer, formality transfer, word decipherment, author imitation, and related language translation. Across all style transfer tasks, our approach yields substantial gains over state-of-the-art non-generative baselines, including the state-of-the-art unsupervised machine translation techniques that our approach generalizes. Further, we conduct experiments on a standard unsupervised machine translation task and find that our unified approach matches the current state-of-the-art.

1 INTRODUCTION

Unsupervised text style transfer is attractive because parallel data is hard to obtain, while existing neural approaches often rely on heuristic objectives. The paper introduces a probabilistic alternative that performs strongly across style-transfer tasks and matches the state of the art in unsupervised machine translation.

1 INTRODUCTION

  • Parallel corpora are scarce for many sequence-transduction applications, particularly text style transfer, which changes superficial properties while aiming to preserve content.The evaluated tasks include formality transfer, author imitation, word decipherment, sentiment transfer, and related-language translation.Source: §1 · p. 1
  • Recent unsupervised systems use adversarial discriminators or hand-designed losses such as backtranslation, but objective design remains heuristic and adversarial training can be unstable.The cited work includes strong unsupervised machine-translation and style-transfer results.Source: §1 · p. 2
  • Probabilistic models make data assumptions explicit, but classical sequence models often impose strong independence assumptions that limit performance on complex tasks.Variational approximation offers a way to train probabilistic models without requiring those strong assumptions.Source: §1 · p. 2
  • The model’s natural probabilistic objective outperforms manually defined unsupervised objectives across style-transfer tasks and matches the state of the art in unsupervised machine translation.The authors present this as evidence that probabilistic principles can guide objective design in deep neural systems.Source: §1 · p. 2

2 UNSUPERVISED TEXT STYLE TRANSFER

Unsupervised text style transfer transforms text between domains while aiming to preserve its content, but parallel examples are often unavailable. The paper frames non-parallel corpora as a partially observed parallel corpus whose missing counterparts are latent variables to infer.

2 UNSUPERVISED TEXT STYLE TRANSFER

  • Text style transfer changes a text’s domain while aiming to preserve its underlying content, and is commonly studied in the unsupervised setting because parallel corpora are difficult to collect.The section gives formality transfer, sentiment transfer, word decipherment, and author imitation as examples.Source: §2 · p. 2
  • The model treats non-parallel corpora from two domains as a partially observed parallel corpus, with latent sentences completing the missing pairs.Its goal is to infer the latent counterpart in the other domain from each observed corpus.Source: §2 · p. 2
  • Figure 1 distinguishes shaded observed variables from unshaded latent variables in the proposed bitext-completion model.The generator is an encoder-decoder, and the latent-sequence prior is a pretrained language model.Source: Figure 1 · p. 3

3 THE DEEP LATENT SEQUENCE MODEL

The deep latent sequence model treats observed sentences as generated from latent parallel sentences in the other domain. It uses amortized variational inference to learn the model, with parameter sharing and gradient approximations addressing key training challenges.

3.1 MODEL STRUCTURE

  • Each observed sentence is generated from a latent parallel sentence in the opposite domain, using a domain-specific prior and transduction distribution.The model treats the unobserved halves as latent variables, marginalizing them during learning and inferring them at test time.Source: §3.1 · p. 3
  • Encoder-decoder transduction models and recurrent language-model priors avoid the strong independence assumptions common in classical generative sequence models.The flexible neural parameterization makes exact inference intractable.Source: §3.1 · p. 3

3.2 LEARNING

  • Amortized variational inference supplies an ELBO surrogate because the neural model’s marginal likelihood cannot be computed by dynamic programming.The inference networks approximate the true posteriors, and the KL terms act as language-model-prior regularizers on induced sentences.Source: §3.2 · p. 4
  • Inference-network parameters are tied to the opposite-direction generators, coupling learning across domains and reducing the model to two encoder-decoders.The shared encoder and domain-conditioned decoder further tie both transfer directions.Source: §3.2 · pp. 4–5, Figure 2 · p. 4
  • For gradient approximation, the model uses Gumbel-softmax straight-through gradients for KL terms and greedy decoding without gradients for reconstruction.The authors report that REINFORCE has high variance and that tested stochastic estimators did not outperform stop-gradient for reconstruction.Source: §3.2 · p. 5
  • An initial-epoch self-reconstruction loss addresses the encoder-decoder’s difficulty generating realistic sentences before unsupervised training provides a useful signal.The loss decays from 1.0 to 0.0 over the first k epochs, with k usually less than 3 in the reported experiments.Source: §3.2 · p. 5

4 CONNECTION TO RELATED WORK

The section relates the ELBO to unsupervised transduction methods: backtranslation appears as its reconstruction loss, while the objective adds an entropy term absent from earlier language-model regularizers.

4 CONNECTION TO RELATED WORK

  • Backtranslation appears naturally in the ELBO as its reconstruction-loss term, rather than only as data augmentation.Source: §4 · p. 5
  • Earlier pretrained-language-model methods use likelihood rewards or treat the language model as an adversary; the paper connects these heuristics to its ELBO.Source: §4 · p. 5
  • Earlier language-model regularizers omit −Hq; the ELBO’s entropy term discourages peaked transductions that repeat similar sentences and helps bypass bad local optima.The paper reports that this regularization is important and improves performance.Source: §4 · pp. 5–6

5 EXPERIMENTS

Across the evaluated style-transfer tasks, the model outperforms UNMT on reference-BLEU while achieving comparable classification accuracy; the experiments also examine task-specific behavior and gradient-training choices. The evaluation includes five style-transfer tasks and standard unsupervised machine translation.

5.1 DATASETS AND EXPERIMENT SETUP

  • Experiments cover sentiment, word-substitution decipherment, formality transfer, author imitation, related-language translation, and standard unsupervised machine translation.Sentiment transfer is evaluated for attribute control, content preservation, and fluency; its metrics are classification accuracy, self-BLEU, and output perplexity.Source: §5 · p. 6, §5.1 · p. 6
  • The decipherment test uses fully enciphered text, while the translation experiments include Bosnian–Serbian and WMT’16 German–English.The German–English experiment uses 5M non-parallel sentences per language and is described as substantially harder than the style-transfer tasks.Source: §5.1 · pp. 6–7

5.2 RESULTS

  • Across the style-transfer tasks, the model matches UNMT on classification accuracy and outperforms it on reference-BLEU; it also consistently beats BT+NLL.Reference-BLEU is described as the most important metric because it compares generations against gold parallel data. BT+NLL can be competitive, but very low PPL sometimes accompanies repeated, overly simple outputs.Source: Table 1 · p. 7, Table 2 · p. 7, §5.2 · p. 8

5.3 FURTHER ABLATIONS AND ANALYSIS

  • Removing shared parameters between transduction directions on decipherment yields extremely low BLEU and no meaningful outputs.The ablation replaces the shared setup with two encoder-decoder models.Source: §5.3 · p. 8
  • As domain-classifier accuracy rises, gains over UNMT are smaller on decipherment and English–German translation, whose vocabularies differ.The authors suggest LM-prior discrimination regularization may be less necessary when domains are very different.Source: §5.3 · p. 8, Figure 3 · p. 8
  • On author imitation, the model preserves source content better than UNMT and achieves higher word F1 across POS categories, especially nouns.The authors suggest better noun generation likely contributes to content preservation.Source: §5.3 · p. 8, Figure 4 · p. 8, Table 3 · p. 8
  • For sentiment transfer, sampling-based gradient approximation underperforms greedy approximation on both ELBO and task evaluations.After training, a single-sample ELBO estimate has standard deviation below 0.3 across 10 test repetitions; final reported ELBO values use sampling-based estimates.Source: §5.3 · p. 9, Table 4 · p. 9

5.4 COMPARISON OF GRADIENT PROPAGATION METHODS

  • On sentiment transfer, stopping reconstruction-loss gradients to the inference network yields better ELBO than Gumbel Softmax and REINFORCE.The authors attribute the result to stop-gradient’s lower variance and better bias–variance balance for discrete latent sequences; REINFORCE was difficult to stabilize.Source: §5.4 · p. 9, Figure 5 · p. 10

6 CONCLUSION

The paper unifies prior unsupervised text style transfer work through a probabilistic generative formulation and reports substantial improvements across five tasks.

6 CONCLUSION

  • The paper proposes a probabilistic generative formulation that unites past work on unsupervised text style transfer.Source: §6 · p. 10
  • The formulation provides a different way to reason about unsupervised objectives in this domain.Source: §6 · p. 10
  • The model yields substantial improvements on five text style transfer tasks, with bigger gains when the styles are harder to distinguish.Source: §6 · p. 10

A.1 MODEL CONFIGURATIONS.

The experiments use a shared attentional encoder-decoder architecture, with pooling and noise choices varying by task.

A.1 MODEL CONFIGURATIONS

  • UNMT, BT+NLL, and the proposed method use an attentional encoder-decoder architecture across experiments.Source: §A.1 · p. 13
  • The encoder and decoder are one-layer LSTMs with hidden size 512; embeddings are size 128 and readout states receive dropout at rate 0.3.Source: §A.1 · p. 13
  • Max pooling over encoder hidden states precedes the decoder; window size controls retained information, with size 1 equivalent to attention and a large window corresponding to no attention.Source: §A.1 · p. 13
  • UNMT uses its default denoising noise, while BT+NLL and the proposed method add it to self-reconstruction only for decipherment and related-language translation.Source: §A.1 · p. 13

A.2 HYPERPARAMETER TUNING.

Hyperparameter tuning varies pooling, self-reconstruction scheduling, and the NLL or KL weight.

A.2 HYPERPARAMETER TUNING

  • Pooling window sizes are {1, 5}, and self-reconstruction patience k is {1, 2, 3}.Source: §A.2 · p. 13
  • UNMT and BT+NLL also test leaving the self-reconstruction loss unannealed, as in the unsupervised machine translation task.Source: §A.2 · p. 13
  • The NLL weight for BT+NLL and KL weight for the proposed method are varied over {0.001, 0.01, 0.03, 0.05, 0.1}.Source: §A.2 · p. 13

A.3 SENTIMENT TRANSFER EXAMPLE OUTPUTS

The sentiment-transfer examples note that BT+NLL often produces very short, simple sentences.

A.3 SENTIMENT TRANSFER EXAMPLE OUTPUTS

  • BT+NLL tends to produce extremely short and simple sentences in the sentiment-transfer examples.Source: §A.3 · p. 13

A.4 REPETITIVE EXAMPLES OF BT+NLL

BT+NLL can achieve low perplexity on sentiment and formality transfer by generating overly simple, repetitive sentences. The appendix provides examples of its repetitive formality outputs alongside random sentiment-transfer examples.

A.4 REPETITIVE EXAMPLES OF BT+NLL

  • BT+NLL’s low perplexity on sentiment and formality transfer reflects overly simple, repetitive sentences.The paper identifies these as representative tasks where the baseline has low perplexity.Source: §A.4 · p. 13
  • Table 6 is captioned as containing random sentiment-transfer examples.Source: Table 6 · p. 14
  • Table 7 presents repetitive examples from the BT+NLL baseline on formality transfer.Source: §A.4 · p. 13, Table 7 · p. 14
Loading 2002.03912v3…