Content preserving text generation with attribute controls

1 Introduction

• The lack of parallel sentence pairs that differ along stylistic dimensions makes attribute-conditioned generation an important challenge for sequence-to-sequence methods.Text is also harder to transfer than images because distortions are perceptible rather than negligible.Source: §1 · p. 1
• The model combines auto-encoding and back-translation losses for content preservation with an adversarial discriminator for attribute compatibility and realistic generation.The discriminator is described as more data efficient and more scalable to multiple attributes with several classes than prior methods.Source: §1 · p. 1
• The evaluation adds content-preservation and fluency metrics alongside attribute compatibility, plus human evaluation, addressing the risk that label-matching measures reward vacuous outputs.The paper notes that prior evaluations do not penalize outputs that match the desired attribute while ignoring input content.Source: §1 · pp. 1–2
• The model is evaluated on writing-style transfer with parallel data, and the authors report simultaneous control of multiple attributes without parallel data as a first demonstration to their knowledge.The parallel-data task evaluates content generation, attribute compatibility, and realism together.Source: §1 · p. 2

Related work

• Conditional text-generation work has controlled length [13], voice, honorifics, and sentiment [15] [16], while a VAE-based approach conditions generation on a sentence and attribute labels [18].The VAE approach uses discriminators for attribute compatibility, but its reconstructions can differ from the input sentence.Source: §2 · p. 2
• Non-parallel style-transfer methods mask stylistic words [21], match hidden-state distributions adversarially [22], or use machine-translation back-translation [23].Source: §2 · p. 2
• Unlike prior discriminator approaches that use a classifier or discriminator per attribute or value, this model uses one discriminator to assess realism and compatibility with a set of attributes.The authors report handling multiple attributes simultaneously, whereas prior work mostly focused on one or two.Source: §2 · p. 2
• Auto-encoding can favor copying, while back-translated samples can be noisy or incorrect; the proposed reconstruction loss interpolates between these losses to reduce sensitivity to those issues.Source: §2 · p. 2
• The task is to generate a sentence that preserves the input's content while matching specified attribute values, such as mood or tense.Figure 1 depicts this input-to-output task.Source: Figure 1 · p. 3

3 Formulation

• The formulation allows K attributes and defines content as sentence information not captured by those attributes.Source: §3 · p. 3

3.1 Model Overview

• An RNN encoder produces a content representation, and a decoder generates a sentence conditioned on that representation and the requested attribute values.Source: §3.1 · p. 3

3.2 Content compatibility

• Autoencoding reconstructs the input from its own encoding and labels, while back-translation reconstructs it after generation under different attribute values.Source: §3.2 · p. 3
• Interpolation between input and generated-sentence representations combines autoencoding and back-translation reconstruction, discouraging trivial copying and encouraging similar content representations.The interpolated loss reduces to either component at the corresponding extremes of the binary interpolation vector.Source: §3.2 · pp. 3–4

3.3 Attribute compatibility

• The adversarial discriminator encourages realistic, attribute-compatible generation; mismatched real sentence-label pairs are added as fake examples to discourage ignoring either input.Source: §3.3 · p. 4

3.4 Discussion

• Hard-sampling generated words during training avoids the soft-sampling and inference mismatch described for discrete text generation.Source: §3.4 · p. 4
• Unlike multiple class-specific discriminators, the joint sentence-attribute discriminator is described as more data-efficient and as exploiting correlations among attributes and their values.Source: §3.4 · p. 5

4.1 Training and hyperparameters

• Across tasks, models use a GRU encoder and decoder with attribute embeddings; validation selects the interpolation probability and adversarial-loss weight.The encoder hidden state size is 500, the decoder size is 700, and the attribute embedding size is 200.Source: §4.1 · p. 5

4.2 Metrics

• Evaluation uses attribute classifiers for label accuracy, back-translation BLEU for content preservation, and generated-sentence perplexity for fluency.The content metric averages BLEU for sentences translated to the other sentiment domain and back, in both directions.Source: §4.2 · pp. 5–6

4.3 Sentiment Experiments

• The proposed model outperforms prior methods on all reported sentiment metrics, while human ratings indicate more realistic outputs that better reflect conditioning information.Qualitative examples also show outputs more related to the input than those from the compared models.Source: Table 1 · p. 6, §4.3 · p. 6

4.4 Monolingual Translation

• In semi-supervised Old-to-Modern English translation, the model improves by more than 2 BLEU points over purely supervised baselines, while its unsupervised version falls short.The authors attribute the semi-supervised result to finding sentence alignments by exploiting unlabelled data.Source: Table 3 · p. 6, §4.4 · p. 7

4.3 Sentiment Experiments

• Human assessments agree with the objective evaluations, and qualitative examples show the model producing sentences more related to the input than the compared models.Table 2 assesses attribute and content percentages alongside fluency on a five-point Likert scale.Source: §4.3 · pp. 6–7, Table 2 · p. 6, Table 4 · p. 7

4.5 Ablative study

• The interpolated reconstruction loss alone produces a reasonable model, while adding adversarial loss improves attribute compatibility and the proposed combination preserves content more gracefully.Figure 3 plots attribute compatibility against content compatibility as training progresses; its top-right region is desirable.Source: §4.5 · p. 8, Figure 3 · p. 8

4.6 Simultaneous control of multiple attributes

• Generated sentences achieve attribute accuracies of 98% for mood, 98% for tense, 90% for voice, and 97% for negation; voice control tends to distort content.The authors note that voice can require global changes such as switching subject-verb-object order.Source: §4.6 · pp. 8–9, Table 5 · p. 8

5 Conclusion

• The model uses reconstruction and adversarial losses to encourage content preservation, attribute compatibility, and realistic sequence generation.Source: §5 · p. 9
• Experiments and metrics demonstrate that the model reflects the conditioning information.Source: §5 · p. 9
• The model extends to simultaneous multiple-attribute control, while continuous values and larger semantic and syntactic attribute sets remain future work.Source: §5 · p. 9

A Qualitative comparison

• The passages identify these tables as sample comparisons but do not state a comparative outcome.Source: A Qualitative comparison · pp. 12–13, Table 6 · p. 12, Table 7 · p. 13
• Table 6 shows restaurant-review test queries modified to express opposite sentiment by Ctrl gen [18], Cross-align [22], and the proposed model.Source: Table 6 · p. 12, A Qualitative comparison · p. 12
• Table 7 shows movie-review test queries modified to express opposite sentiment by Ctrl gen [18], Cross-align [22], and the proposed model.Source: Table 7 · p. 13, A Qualitative comparison · p. 13

B Human Evaluation

• Sections B.1, B.2, and B.3 describe the human-evaluation setup.Source: B Human Evaluation · p. 14
• The evaluations used Amazon Mechanical Turk (AMT) to obtain annotations.Source: B Human Evaluation · p. 14
• The annotations concerned generated sentences.Source: B Human Evaluation · p. 14

B.1 Content compatibility

• Annotators choose candidate sentences with the same semantic content as the reference but the opposite sentiment.Source: §B.1 · p. 14
• The instructions allow “No preference” when neither candidate meets both criteria, including when all candidates are irrelevant or have incorrect sentiment.Source: §B.1 · p. 14
• For the example “This is a great movie!”, “This is not a good movie” and “This is a bad movie” are given as suitable answers.Source: §B.1 · p. 14

• Choose the best sentiment according to the stated criterion.Source: §B.2 · p. 14
• A positive sentence conveys positive sentiment, illustrated by liking the food.Source: §B.2 · p. 14
• A negative sentence conveys negative sentiment, while a neutral sentence carries no sentiment information.Source: §B.2 · p. 14

• Fluency is rated using scores from 1 to 5.Source: §B.3 · p. 14
• A score of 5 means grammatical with no errors; 4 allows one or two mistakes.Source: §B.3 · p. 14
• A score of 3 indicates partly grammatical, somewhat coherent text with glaring errors.Source: §B.3 · p. 14
• Scores of 2 and 1 describe increasingly severe problems, from incoherence to complete ungrammaticality.Source: §B.3 · p. 14

• Soft sampling uses an exponentially decaying temperature, starting at 1 and held at 0.01 after reaching that value; the experiment uses Yelp.Source: C Sampling strategy · p. 15
• Soft-sampling models are judged highly attribute-compatible, but have considerably poorer content compatibility and lower fluency.Source: C Sampling strategy · p. 15
• This result supports the authors’ claim that training and inference behavior are mismatched when soft-sampled sequences are used for training.Source: C Sampling strategy · p. 15
• Table 8 compares soft and hard sampling; higher values are better for accuracy and content compatibility, while lower perplexity is better.Source: Table 8 · p. 15