PaperlayerView exact revision on arXiv

Source-linked AI summary

"Mask and Infill" : Applying Masked Language Model to Sentiment Transfer

Xing Wu, Tao Zhang, Liangjun Zang, Jizhong Han, Songlin Hu

arXiv:1908.08039v1cs.CL

TL;DR

Non-parallel sentiment transfer seeks to change a sentence’s sentiment while retaining its content, a challenge for prior RNN-based generation. This paper masks sentiment tokens and uses an attribute-conditioned masked language model to infill them. On two review datasets, the authors report better transfer accuracy and semantic preservation than state-of-the-art systems.

  • Problem

    Non-parallel sentiment transfer must change a sentence’s sentiment while preserving its semantic content, and prior RNN encoder-decoder approaches have limited ability to capture deep and long-range dependencies.

    Source: §1 · p. 1

  • Method

    The two-stage Mask and Infill approach masks sentiment tokens and uses an attribute-conditioned pretrained MLM to fill those positions from context and target sentiment.

    Source: §1 · p. 2

  • Results

    On two review datasets, the authors report that their approach outperforms state-of-the-art systems in transfer accuracy and semantic preservation.

    Source: §6 · p. 6

  • Takeaways & Limitations

    The approach enables non-parallel sentiment transfer training without parallel training data.

    Source: §6 · p. 6

Abstract

from arXiv · show

This paper focuses on the task of sentiment transfer on non-parallel text, which modifies sentiment attributes (e.g., positive or negative) of sentences while preserving their attribute-independent content. Due to the limited capability of RNNbased encoder-decoder structure to capture deep and long-range dependencies among words, previous works can hardly generate satisfactory sentences from scratch. When humans convert the sentiment attribute of a sentence, a simple but effective approach is to only replace the original sentimental tokens in the sentence with target sentimental expressions, instead of building a new sentence from scratch. Such a process is very similar to the task of Text Infilling or Cloze, which could be handled by a deep bidirectional Masked Language Model (e.g. BERT). So we propose a two step approach "Mask and Infill". In the mask step, we separate style from content by masking the positions of sentimental tokens. In the infill step, we retrofit MLM to Attribute Conditional MLM, to infill the masked positions by predicting words or phrases conditioned on the context1 and target sentiment. We evaluate our model on two review datasets with quantitative, qualitative, and human evaluations. Experimental results demonstrate that our models improve state-of-the-art performance.

1 Introduction

Non-parallel sentiment transfer must change a sentence’s sentiment while preserving its meaning, but prior RNN encoder-decoder approaches struggle with sentence quality and long-range dependencies. The paper recasts transfer as masked infilling, then reports strong results on two review datasets.

Motivation

  • Prior non-parallel transfer methods use RNN encoders and decoders, whose limited representations and short-range prediction can hinder realistic sentence generation.The task lacks parallel sentence pairs with the same content and different attributes.Source: §1 · p. 1

Approach

  • Mask and Infill replaces sentiment tokens with target expressions using a deep bidirectional MLM, rather than generating a new sentence from scratch.The mask stage identifies sentiment tokens; the infill stage predicts replacements from context and target sentiment.Source: §1 · p. 2
  • The fused marker-identification method combines attention and frequency-ratio signals to filter false markers, discover single sentiment words, and fall back to attention when needed.Attention tends to favor single words, while frequency-ratio dictionaries can favor longer phrases and depend on dictionary quality.Source: §1 · p. 2
  • AC-MLM conditions masked-token prediction on sentiment attributes, while a pretrained sentiment classifier constrains outputs and soft-sampling guides optimization.Soft-sampling approximates sampled word vectors to back-propagate gradients through discrete generation.Source: §1 · p. 2

Results

  • The authors report state-of-the-art accuracy, BLEU, and human-evaluation performance, and say their approach outperforms most state-of-the-art models on BLEU and accuracy.Evaluation covered Yelp and Amazon using quantitative, qualitative, and human evaluations.Source: §1 · p. 2

2 Related Work

The related work contrasts earlier RNN-based non-parallel transfer systems with this paper’s use of a pretrained masked language model for generation. It also distinguishes generative use of pretrained models from prior fine-tuning for discriminative tasks.

2.1 Non-parallel Style Transfer

  • Earlier non-parallel style-transfer systems separate content and attribute through latent representations or feature-based methods, then generate with RNN decoders.The paper contrasts those systems with pretrained MLMs for capturing longer linguistic structure and language representation.Source: §2.1 · p. 2

2.2 Fine-tuning on Pre-trained Language Model

  • Where prior pretrained-language-model fine-tuning addressed discriminative tasks, this work applies a pretrained masked language model to generation.Figure 2 depicts a Mask module and an Infill module with reconstruction and discrimination parts.Source: §2.2 · p. 2, Figure 2 · p. 3

3 Approach

The approach section previews an overview followed by separate descriptions of masking and infilling.

3 Approach

  • The section is organized around an approach overview, the mask process, and the infill process.The infill process is described in Section 3.3.Source: §3 · p. 3

3.1 Overview

The model takes a sentence and a target sentiment attribute and produces a sentence intended to retain the original semantic content. Its mask and infill modules identify attribute markers, replace them with AC-MLM, and use a discriminator to improve target-attribute compatibility.

3.1 Overview

  • The model maps an input sentence and target sentiment attribute to an output sentence that retains the input’s semantic content while exhibiting the target attribute.The training corpus consists of labeled sentences paired with their attributes.Source: §3.1 · p. 3
  • The mask module uses a candidate phrase vocabulary to find potential attribute markers and masks them before sending the sentence to the infill module.The infill module fills the masked positions via AC-MLM.Source: §3.1 · p. 3
  • A pretrained classifier discriminates generated sentences for target-attribute compatibility, and soft-sampling guides optimization despite discrete language generation.The overview refers to implementation details in Algorithm 1.Source: §3.1 · p. 3

3.2 Identify and Mask Attribute Marker

The section identifies sentiment markers with frequency-ratio and attention-based methods, then combines them to mask markers while retaining sentence content. The combined method uses attention-based re-masking when vocabulary matching leaves too little content or finds no markers.

Fusion-Method

  • The fusion method combines a candidate phrase vocabulary with attention-based re-masking when matching finds no markers or leaves insufficient content.Sentences with too little content are reprocessed using positions with high attention weights.Source: §3.2 · p. 3, Fusion-Method · p. 3

3.2 Identify and Mask Attribute Marker

  • The paper introduces a frequency-ratio method and an attention-based method, then proposes a fusion method combining the strengths of both.Source: §3.2 · p. 3

Frequency-ratio Method

  • Frequency-ratio scoring selects n-grams whose smoothed class-specific score exceeds threshold γc as candidate attribute markers.The score compares an n-gram’s frequency in the target-attribute dataset with its frequency in other attribute datasets.Source: Frequency-ratio Method · p. 3

Attention-based Method

  • The attention-based method encodes each sentence with a bidirectional LSTM and treats words above the sentence’s average attention weight as attribute markers.Its attention weights reflect each word’s contribution to the sentence attribute.Source: Attention-based Method · p. 3

3.3 Text Infilling with MLM

AC-MLM infills masked sentiment positions using both sentence context and a target sentiment label. It is trained to reconstruct original masked words, with a classifier constraint and soft-sampling used to promote target-label compatibility.

Attribute Conditional Masked Language Model

  • AC-MLM conditions masked-token predictions on both sentence context and a target attribute by adding attribute embeddings to the original input embeddings.The method uses the content tokens as context and the target label as the attribute condition.Source: Attribute Conditional Masked Language… · p. 4

Reconstruction

  • Without parallel sentiment pairs, AC-MLM learns to reconstruct the original sentence from its content and original attribute, minimizing masked-word reconstruction error.Once trained, it predicts a distribution for each masked position from sentence content and a target attribute, then infills those positions.Source: Reconstruction · p. 4

Pre-trained Classifier Constraint

  • AC-MLM-SS adds a pretrained sentiment-classifier constraint so generated sentences receive the target attribute label.Its objective combines reconstruction and discrimination losses, balanced by η.Source: Pre-trained Classifier Constraint · p. 4
  • Soft-sampling approximates each sampled token with a differentiable probability vector, allowing discriminator gradients to pass through discrete samples.The resulting soft-generated sentence is evaluated for fitness to the target attribute.Source: Pre-trained Classifier Constraint · p. 4

4 Experiment

The experiments evaluate the approach on Yelp and Amazon review sentences labeled positive or negative. They compare it with existing sentiment-transfer systems and related baselines.

4 Experiment

  • The experiment section outlines dataset and baseline introductions, comparison with state-of-the-art systems, and analysis of experimental results and components.Source: §4 · p. 4

4.1 Datasets

  • The evaluation uses Yelp business-review sentences and Amazon product-review sentences, each labeled positive or negative and randomly split into training, validation, and test sets.Dataset statistics are presented in Table 1.Source: §4.1 · p. 4

4.2 Baselines

  • The comparison includes CrossAligned, StyleEmbedding, MultiDecoder, CycledReinforce, TemplateBased, RetrievalOnly, DeleteOnly, DeleteAndRetrieval, and LM+Classifer.Source: §4.2 · p. 4

4.3 Experiment Details

The experiment details specify the marker-identification settings and the infill model configuration. They use pretrained BERTbase for MLM and fine-tune it before adding the discriminator constraint.

Mask step

  • The frequency-ratio method uses n-grams up to 4-gram with smoothing λ=1, while the attention classifier reaches 97% accuracy on Yelp and 82% on Amazon.For fusion, sentences with fewer than five content tokens are reprocessed by the attention-based method.Source: Mask step · p. 4

Infill step

  • The infill step uses pretrained BERTbase with attribute embeddings replacing segment embeddings, fine-tunes it for 10 epochs, then trains six more epochs with the discriminator constraint.The discriminator is a CNN classifier with filter sizes 3, 4, and 5; hyperparameters are selected by validation-set grid search.Source: Infill step · p. 5

4.4 Evaluation

Automatic evaluation compares accuracy and BLEU across two evaluation setups, while human ratings assess grammaticality, content preservation, and target-attribute match. The fusion AC-MLM-SS model offers the strongest combined automatic performance and best human-evaluation results among the paper’s models.

Automatic Evaluation

Human Evaluation

  • Fusion AC-MLM-SS achieves the best human-evaluation results across grammaticality, content preservation, and target-attribute match.Three non-author annotators rated outputs on a 1–5 scale; 200 examples per target attribute were sampled for each dataset.Source: Human Evaluation · p. 5

5 Analysis

The analysis examines accuracy–BLEU trade-offs, masking strategies, and the benefits of masked language models. It also connects the approach to copying-and-generating systems.

Trade-off between Content and Attribute

  • Figure 3 presents the trend of BLEU as accuracy increases, and the analysis describes the two indicators as a trade-off.The loss-weight hyperparameter η can adjust this trend.Source: Figure 3 · p. 6, §5 · p. 6
  • The reconstruction and discriminative losses balance contextual compatibility against attribute compatibility, and η controls the BLEU–accuracy trade-off.The classifier loss raises accuracy by encouraging attribute-compatible words, while decreasing diversity.Source: §5 · pp. 5–6

Comparing Three Mask Methods

  • Fusion masking performs best overall on automatic evaluation, with slight gains on high-quality Yelp and larger gains on lower-quality Amazon.On Amazon, fusion combines the strengths of attention-based and frequency-ratio methods.Source: §5 · p. 6

Benefits of Masked Language Model

  • Compared with LSTM-decoder models, the masked-language-model approach substantially improves BLEU on both datasets; against Yang et al., it achieves comparable performance.The paper contrasts fine-tuning pre-trained BERT with training a target-domain language model from scratch, which requires a large corpus and substantial time.Source: §5 · p. 6

Connection to PG-Net

  • The approach combines copying and generation: it preserves source content while generating stylistic words by masking source tokens rather than pointing to them.The paper relates this design to pointer-generator networks for text summarization.Source: §5 · p. 6

6 Conclusions and Future Work

The paper concludes that its two-stage mask-and-infill approach addresses non-parallel sentiment transfer and outperforms state-of-the-art systems on two review datasets. It proposes fine-grained sentiment transfer and applying masked language models to other natural-language-generation tasks as future work.

6 Conclusions and Future Work

  • On two review datasets, the non-parallel mask-and-infill approach outperforms state-of-the-art systems in transfer accuracy and semantic preservation.Future work includes transfer across more than two sentiments and applying masked language models beyond style transfer.Source: §6 · p. 6

Related papers

Loading 1908.08039v1…