Source-linked AI summary
Style Transfer from Non-Parallel Text by Cross-Alignment
Tianxiao Shen, Tao Lei, Regina Barzilay, Tommi Jaakkola
TL;DR
Non-parallel style transfer must disentangle content from style when only differently styled corpora are available. The paper learns cross-aligned latent representations so transferred sentences match the target style distribution, and evaluates this approach across three tasks with strong reported performance. Its identifiability analysis also shows that some latent content distributions, such as isotropic Gaussians, cannot support recovery of rotational effects.
Problem
Non-parallel text-generation tasks require changing presentation while preserving content, but content and style interact and must be disentangled from corpora with shared content distributions.
Method
The method encodes sentences into rich style-independent content representations, decodes them with style-dependent generators, and aligns original and cross-generated sentence distributions.
Results
Across sentiment modification, word-substitution decipherment, and word-order recovery, the method is effective, including 41.5% human-evaluated successful sentiment transfers and BLEU scores of 57.4 and 26.1.
Takeaways & Limitations
Cross-alignment can learn non-parallel transfer across multiple tasks by using distributional constraints on latent representations and transferred sentence populations.
Takeaways & Limitations
With isotropic Gaussian latent content, rotational style effects cannot be recovered; decipherment evaluation also assumes the model learns a word-substitution function from data distributions.
Abstract
from arXiv · showhide
This paper focuses on style transfer on the basis of non-parallel text. This is an instance of a broad family of problems including machine translation, decipherment, and sentiment modification. The key challenge is to separate the content from other aspects such as style. We assume a shared latent content distribution across different text corpora, and propose a method that leverages refined alignment of latent representations to perform style transfer. The transferred sentences from one style should match example sentences from the other style as a population. We demonstrate the effectiveness of this cross-alignment method on three tasks: sentiment modification, decipherment of word substitution ciphers, and recovery of word order.
1 Introduction
The paper addresses style transfer and related generation tasks when only non-parallel corpora are available, requiring content preservation while changing presentation. It proposes cross-alignment of latent content representations and evaluates the approach across three tasks.
- Non-parallel generation tasks must preserve source content while rendering it under constraints such as style, plaintext, or ciphertext.
- The method assumes corpora share a content distribution and learns style-independent content vectors that can be decoded into another style.
- Cross-generated sentences provide distributional constraints by requiring transferred examples to match the target-style corpus as a population.
- The model is evaluated on sentiment modification, word-substitution decipherment, and word-order recovery using non-parallel data.
- 41.5% of review sentences preserved content while transferring sentiment under human evaluation, versus 41.0% for Hu et al.'s control-gen baseline.
- BLEU scores reached 57.4 for decipherment and 26.1 for word-order recovery, with gains of 50.2 and 20.9 over a comparable method without cross-alignment.
2 Related work
Related work includes non-parallel style transfer in vision and indirect-signal approaches to natural-language generation. The paper also draws on methods for training adversarial models with discrete recurrent-network samples.
- Computer-vision approaches separate content and style features or learn generative mappings between two domains without parallel examples.
- Natural-language generation commonly uses parallel sentences, while related non-parallel methods guide generation through indirect signals such as property classifiers.
- The paper uses Professor-Forcing to address adversarial training over discrete recurrent-network samples and the gap between teacher forcing and self-feeding.
- Continuous-relaxation techniques approximate discrete sampling so the training procedure can be optimized through back-propagation.
3 Formulation
The formulation models two differently styled corpora as observations generated from shared latent content and style variables, then asks when transfer is identifiable from marginals alone. Examples show that recoverability depends on the complexity and distinctiveness of the latent content distribution.
- 3 Formulation: The generative process samples style y and content z independently, then generates datapoint x from p(x|y, z).
- 3 Formulation: The observed corpora share a content distribution but use unknown styles, and the goal is to estimate transfer functions in both directions.
- 3 Formulation: The central feasibility question is whether the joint distribution of two styled variables can be recovered when only their marginal distributions are observed.
- 3 Formulation: Recovery requires distributions generated by different styles to be distinct; otherwise identical marginals can correspond to different joint distributions.
- 3.1 Example 1: Gaussian: With isotropic Gaussian content z ∼ N(0, I), orthogonal style transformations produce identical distributions, so rotational effects cannot be recovered.
- 3.1 Example 1: Gaussian: A more complex Gaussian-mixture content distribution can make affine transformations uniquely determined and transfer recoverable from respective marginals.
- 3.2 Example 2: Word substitution: In the word-substitution example, transfer reduces to inferring a word alignment between two realizations of the same bigram language model.
- 3.2 Example 2: Word substitution: Finding the alignment is equivalent to finding permutation matrix P such that P^T M1 P ≈ M2, linking the problem to graph isomorphism and its computational difficulty.
4 Method
The method encodes sentences into style-independent latent content representations and decodes them under a target style. Adversarial alignment operates either on latent distributions or across transferred and real sequences to support non-parallel style transfer.
- 4 Method: The encoder infers content z from a sentence and style, while the generator reconstructs or renders a sentence from style y and content z.Together, E and G form an auto-encoders when applied to the same style, enabling reconstruction training.
- 4 Method: Unlike a VAE, the standard auto-encoder preserves richer information in z and lowers uncertainty in p(x|y,z), supporting meaningful style flipping.The authors argue that forcing z toward a simple prior can make non-parallel transfer less effective.
- 4.1 Aligned auto-encoder: The aligned auto-encoder matches aggregated latent representations from the two styles with an adversarial discriminator while minimizing reconstruction loss.Its encoder, generator, and discriminator are trained in a min-max objective.
- 4.2 Cross-aligned auto-encoder: The cross-aligned auto-encoder aligns real sentences from one style with transferred sentences from the other using discriminators D1 and D2.D1 distinguishes real x1 from transferred x2, while D2 distinguishes real x2 from transferred x1.
- 4.2 Cross-aligned auto-encoder: To train through discrete generation, the model feeds softmax word distributions with temperature γ and matches hidden-state sequences using Professor-Forcing.Teacher-forced real sequences are compared with self-fed transferred sequences, whose hidden states are smoothly distributed.
- 4.2 Cross-aligned auto-encoder: Cross-aligning the whole hidden-state sequence prevents initial latent misalignment from propagating through recurrent generation toward the target domain.Figure 2 contrasts teacher forcing for x1 with self-feeding for transferred x2; the first aligned variant compares only initial states.
5 Experimental setup
The experiments evaluate non-parallel transfer on sentiment modification, word-substitution decipherment, and word-order recovery. Evaluation combines pretrained-classifier metrics, human judgments, BLEU, and comparisons with frequency-based and parallel-data baselines.
- Sentiment modification: Sentiment modification uses Yelp reviews labeled positive above rating three and negative below rating three, despite document-level annotations for sentence-level modeling.Transferred sentences are evaluated with a pretrained TextCNN classifier and human judgments of fluency, sentiment, and overall quality.
- Sentiment modification: The pretrained TextCNN reaches 97.4% accuracy on the simplified sentiment-transfer dataset.Classifier-based evaluation measures how often transferred sentences receive the intended sentiment.
- Sentiment modification: Human evaluation ranks 500 randomly selected test sentences for fluency and sentiment, using fluency scores from 1 to 4.Overall transfer quality is assessed comparatively by showing judges a source and two transferred sentences.
- Word substitution decipherment: Word-substitution decipherment replaces every plaintext word with a cipher token through a one-to-one key and evaluates transfer with BLEU.Performance is reported against the percentage of substituted vocabulary; a frequency baseline and a parallel-trained machine-translation system provide comparisons.
- Word order recovery: Word-order recovery transfers between original English sentences and shuffled English sentences to recover or produce word order from non-parallel training data.Parallel testing data is constructed as in the decipherment experiment, while the transfer model learns the shuffle function from distributions.
6 Results
The experiments evaluate non-parallel style transfer across sentiment modification, word-substitution decipherment, and word-order recovery. Cross-alignment preserves source content more effectively in sentiment transfer and outperforms comparison models on the other tasks.
- Sentiment modification: Our model achieved better overall sentiment transfer than control-gen despite lower sentiment accuracy and fluency.The comparison attributes control-gen’s advantage to direct sentiment-classifier guidance, while the proposed model better preserves grammatical structure and source meaning.
- Word substitution decipherment: Cross-aligned decipherment consistently outperformed competing models across substitution rates, with the advantage increasing as substitution became harder.At 100% substitution, variational and aligned auto-encoders performed poorly enough that the passage rules out their use for this task.
- Word order recovery: 26.1 Bleu was achieved by the cross-aligned auto-encoder for word-order recovery, while other models performed no better than no transfer.Even the parallel-data machine-translation model reached only 64.6 Bleu, and valid alternative reorderings could still be penalized.
- Sentiment modification: Table 3 presents each input sentence followed by outputs from Hu et al. (2017) and the cross-aligned auto-encoder.The samples support qualitative comparison of sentiment transfer outputs from the two systems.
- Decipherment and word order recovery: Table 4 reports Bleu scores for word-substitution decipherment and word-order recovery.The table organizes quantitative results across the two non-sentiment tasks.
7 Conclusion
The paper formulates style transfer from one style to another as decipherment using only non-parallel data. It aligns latent sentence populations and demonstrates the method on sentiment transfer, word substitution decipherment, and word ordering.
- Conclusion: The method forces distributional alignment over latent sentence representations when only non-parallel data from two styles is available.The paper assumes the two collections arise from a latent-variable generative model.
- Conclusion: The method is demonstrated on sentiment transfer, word-substitution decipherment, and word-ordering tasks that permit quantitative evaluation.The conclusion identifies recovering a joint distribution from marginal distributions as an open question for future research.
A Proof of Lemma 1
The lemma analyzes when different affine transformations of a Gaussian-mixture latent variable can produce the same observed distribution. With at least two components having different covariance matrices, the transformation is uniquely determined.
- Assumptions: The proof assumes a Gaussian-mixture latent distribution with K ≥ 2 and at least two distinct covariance matrices.The observation model is an affine transformation with isotropic Gaussian noise.
- Proof setup: Equality of the observed distributions under two transformations imposes componentwise constraints on their parameters.The proof considers distinct affine transformations and uses invertibility of the transformation family.
- Conclusion: When two mixture covariances differ, the only solution is A−1A′ = I, so A = A′ and b = b′.Thus distinct affine transformations cannot yield the same observed distribution under the lemma’s assumptions.