PaperlayerView exact revision on arXiv

Source-linked AI summary

Text Style Transfer for Bias Mitigation using Masked Language Modeling

Ewoenam Kwaku Tokpo, Toon Calders

arXiv:2201.08643v1cs.CL

TL;DR

Biased text can affect fairness in NLP applications, while text style-transfer approaches face challenges including scarce parallel data and information loss. This paper combines explicit keyword replacement with latent content representation for debiasing, reporting good content preservation and fluency while maintaining good style-transfer accuracy across both datasets.

  • Problem

    Biased text can affect fairness in NLP applications, while text style-transfer approaches face scarce parallel data and content-loss challenges.

    Source: Bias in textual data · p. 1, Style transfer · p. 2

  • Method

    The model combines explicit keyword replacement with latent content representation to transfer biased text into a neutral style while retaining semantic content.

    Source: Style transfer · p. 2

  • Results

    On both datasets, the approach produces good content-preservation and fluency scores while maintaining good style-transfer accuracy.

    Source: Conclusion · p. 7

  • Takeaways & Limitations

    The reported evaluation covers bias mitigation and gender obfuscation using content preservation, fluency, and style-transfer accuracy.

    Source: Conclusion · p. 7

    Sequence-to-sequence models can lose information during compression and may struggle to generate high-quality representations or long meaningful sentences.

    Source: Back-translation style transfer model… · p. 3

Abstract

from arXiv · show

It is well known that textual data on the internet and other digital platforms contain significant levels of bias and stereotypes. Although many such texts contain stereotypes and biases that inherently exist in natural language for reasons that are not necessarily malicious, there are crucial reasons to mitigate these biases. For one, these texts are being used as training corpus to train language models for salient applications like cv-screening, search engines, and chatbots; such applications are turning out to produce discriminatory results. Also, several research findings have concluded that biased texts have significant effects on the target demographic groups. For instance, masculine-worded job advertisements tend to be less appealing to female applicants. In this paper, we present a text style transfer model that can be used to automatically debias textual data. Our style transfer model improves on the limitations of many existing style transfer techniques such as loss of content information. Our model solves such issues by combining latent content encoding with explicit keyword replacement. We will show that this technique produces better content preservation whilst maintaining good style transfer accuracy.

Introduction

Bias in text can affect fairness in NLP applications, including recruitment and other language-system settings. The paper proposes text style transfer for debiasing, combining keyword replacement with latent content representation to address content loss while maintaining transfer accuracy.

Bias in textual data

  • Biased text matters because NLP systems can reproduce discriminatory patterns, while biased job-ad wording can reduce its appeal to applicants outside the favored gender group.The reported job-ad effect held regardless of the gender stereotype traditionally associated with the job.Source: Bias in textual data · p. 1

Style transfer

  • Style transfer for text debiasing remains underexplored, and existing approaches face scarce parallel data, content loss, and fluency challenges.Keyword replacement can lose information in deleted words, while sequence-to-sequence models can lose information during encoding and struggle to reconstruct fluent text.Source: Style transfer · pp. 1–2
  • The proposed model combines explicit keyword replacement with latent content representation to address limitations of both approaches while maintaining good style-transfer accuracy.The paper also proposes identifying attribute tokens with a text-classification explanation model and training a token embedder with a dual objective.Source: Style transfer · p. 2
  • The model is trained exclusively on nonparallel data to convert biased text to a neutral version while retaining significant content information.The authors note that parallel corpora are relatively hard to obtain.Source: Style transfer · p. 2

2. A novel way of improving content preservation and flu-

The paper presents combining keyword replacement with latent content information as a way to improve content preservation in text style transfer while retaining good transfer accuracy.

2. A novel way of improving content preservation and flu-

  • Combining keyword replacement with latent content information is presented as a way to improve content preservation while retaining good style-transfer accuracy.The paper identifies its approaches to generating latent content representation and identifying attribute tokens as additional novelties.Source: §2 · p. 2
  • The paper describes its combination of keyword replacement and latent content information as a response to a key content-preservation challenge in text style transfer.The passage characterizes the improvement in content preservation as significant.Source: §2 · p. 2
  • The authors make the code and data used in the work available.Source: §2 · p. 2

Related work

The related work reviews text style-transfer approaches spanning auto-encoders, back-translation, and explicit keyword replacement. It notes trade-offs involving information loss and sentence quality across these approaches.

Related work

  • Image style transfer combines image content and style, but its techniques do not transfer directly to text; NLP style-transfer work has addressed sentiment and machine translation.The paper focuses its review on models trained without parallel data.Source: Related work · p. 2

Auto-encoder sequence-to-sequence models

Aligned auto-encoder model and Cross-aligned auto-encoder model:

  • Aligned auto-encoders use adversarial learning for a disentangled representation; cross-aligned auto-encoders additionally align transferred and target representation distributions in both directions.The cross-aligned model uses two discriminators, each distinguishing real representations from transferred ones.Source: Aligned auto-encoder model and… · p. 2

Back-translation style transfer model (BST):

  • Back-translation style transfer encodes input through translation, then decodes a target-style text and uses adversarial learning to enforce target style.It approximates discrete token training using a continuous approximation of softmax values.Source: Back-translation style transfer model… · p. 3
  • Sequence-to-sequence models can lose information through compression and may struggle to produce high-quality representations or long meaningful sentences.The passage notes they may be more effective when substantial rephrasing is needed.Source: Back-translation style transfer model… · p. 3

Explicit Style Keyword Replacement

  • Explicit keyword replacement identifies attribute markers, deletes them, and predicts target-style replacements, preserving more input information by deleting only a small portion.The methods can still fail to capture information carried by deleted tokens.Source: Explicit Style Keyword Replacement · p. 3, B-GST and G-GST · p. 3

DeleteOnly model and Delete&Retrieve model:

  • DeleteOnly and Delete&Retrieve remove attribute tokens, retrieve possible replacements from similar target-style sentences, then generate target-style text from context using an RNN.Delete&Retrieve supplies retrieved replacement words to the RNN; DeleteOnly does not.Source: DeleteOnly model and Delete&Retrieve… · p. 3

B-GST and G-GST:

  • B-GST and G-GST use transformer-classifier attention to identify attribute words and GPT for generation; the reported limitation is failure to capture deleted-token information.G-GST follows the retrieve approach, while B-GST corresponds to DeleteOnly.Source: B-GST and G-GST · p. 3

Methodology

The methodology frames debiasing as transforming biased text into neutral text while preserving semantic content. The model is described as having four main components.

Methodology

  • The model aims to convert biased text into a neutral version while preserving its semantic content.Source: Methodology · p. 3
  • The paper denotes neutral style as sa and biased style as sb, and represents the biased-text corpus as Xb.Source: Methodology · p. 3
  • The model consists of four main components, which the paper introduces as illustrated in Figure 1.Source: Methodology · p. 3

1. Attribute Masker:

The Attribute Masker identifies bias-responsible attribute words and replaces them with [MASK] before the text reaches the Token Embedder.

1. Attribute Masker:

  • Attribute words responsible for bias are masked with [MASK] before the resulting text is sent to the Token Embedder.The masking stage identifies the words to replace.Source: §1 · p. 3

2. Token Embedder:

The Token Embedder creates embeddings for masked tokens that are intended to capture semantic information from their context.

2. Token Embedder:

  • The Token Embedder generates masked-token embeddings intended to carry semantic information from context tokens.These embeddings are produced for the masked tokens in the input.Source: §2 · p. 3

3. Latent-content Encoder:

The Latent-content Encoder represents the original text while disentangling its content from biased style.

3. Latent-content Encoder:

  • The Latent-content Encoder encodes the original text into a latent content representation and aims to disentangle it from biased style.The encoder takes unmasked text as input.Source: §3 · p. 3

4. Token Decoder:

The model routes masked text and latent content through four components; an example illustrates the end-to-end process, with latent content debiased before it is added to token embeddings.

4. Token Decoder:

  • The Token Decoder averages token embeddings with the latent content representation, then uses the resulting embeddings to predict tokens.The section also points to an example illustrating the end-to-end process.Source: §4 · p. 3

Attribute Masker

  • The authors choose LIME (Ribeiro, Singh, and Guestrin 2016) for attribute-word identification, reporting good and intuitive explanations.They contrast frequency-based identification [Li et al. 2018] and attention weights [Sudhakar, Upadhyay, and Maheswaran 2019], noting that attention-based identification has been found ineffective (Jain and Wallace 2019). A BERT classifier is trained on labeled biased and neutral text; LIME token weights are thresholded to select words for masking with [MASK].Source: Attribute Masker · pp. 3–4

Token Embedder

  • The Token Embedder generates an embedding for each masked-input token, intended to capture its semantic information from context.The paper sets each embedding to dimension 768.Source: Token Embedder · p. 4

Training the Token Embedder

  • The Token Embedder is fine-tuned with masked language modeling, predicting randomly masked words from unmasked context.About 20% of training-corpus words are masked, and the appended classification layer is later reused as the Token Decoder.Source: Training the Token Embedder · p. 4

Latent-content Encoder

  • The Latent-content Encoder maps the original unmasked sentence to a latent representation intended to retain content while removing bias.It mean-pools token embeddings from the original text to form that representation.Source: Latent-content Encoder · p. 4

Training the Latent-content Encoder

  • The encoder’s dual objective aligns its latent representation with a source-content encoding while training it to be classified as neutral.The Source Content Encoder and Bias Detector are pretrained; the former encodes content and biased style, while the latter predicts style, and the balance between objectives is controlled by λ.Source: Training the Latent-content Encoder · pp. 4–5

Token Decoder

  • The Token Decoder combines each token embedding with the latent representation to predict a vocabulary distribution.For training, the decoder uses the source representation to reconstruct masked original tokens from neutral-text data.Source: Token Decoder · p. 5, Training the Token Decoder · p. 5

Training the Token Decoder

  • The Token Decoder is trained with a dual objective that includes a classifier-based constraint favoring neutral output.Because output tokens are discrete, training uses soft sampling with a temperature that controls probability smoothness.Source: Training the Token Decoder · p. 5

Experiments

The experiments focus on gender-bias mitigation and gender obfuscation, using Jigsaw and Yelp data. They compare the model with prior style-transfer systems using automated measures of transfer, content preservation, and fluency.

Experiments

  • Experiments cover gender-bias mitigation and gender obfuscation, with gender obfuscation converting female-authored text to look male-authored.The authors note that the approach is adaptable to other biases, such as racial bias.Source: Experiments · p. 5

Dataset

  • The experiments use two datasets, including Jigsaw comments labeled for bias toward or against demographics.Dataset statistics are presented in Table 1.Source: Dataset · p. 5, Jigsaw dataset · p. 5

Jigsaw dataset:

  • Jigsaw texts with gender labels of at least 0.5 are treated as gender-biased, while texts below 0.5 form the neutral class.Multi-sentence texts are split into single-sentence examples.Source: Jigsaw dataset · p. 5

Yelp dataset:

Evaluation models and metrics

  • The model is compared with six style-transfer systems, using transfer accuracy, semantic content similarity, and language-model perplexity to assess performance.The comparison models were selected for their use in style-transfer research and reproducibility; transfer accuracy is assessed with separate BERT classifiers reported as 85% accurate for Jigsaw and 80% for Yelp.Source: Evaluation models and metrics · pp. 5–6

Hyper-parameter settings

  • The two tasks use different sequence sizes and decoder balancing parameters, while both use λ = 0.5 and an LIME threshold of µ > 0.1.Both models are trained for four epochs with temperature τ = 1; the authors leave comprehensive hyperparameter tuning to future work.Source: Hyper-parameter settings · p. 6

Results and discussion

Across the two tasks, models tend to trade off transfer accuracy against content preservation and fluency. The authors report that their model balances these metrics, while ablations illustrate how latent content information and class constraints relate to those outcomes.

Results and discussion

  • The reported models generally achieve better transfer accuracy on bias mitigation than on gender obfuscation, which the authors describe as more difficult and indistinct.They contrast the less distinct cues involved in predicting author gender from writing with clearer biased-versus-neutral cues.Source: Results and discussion · p. 6

Gender bias mitigation

  • The model improves content preservation and fluency over the compared systems on gender-bias mitigation while achieving the second-highest transfer accuracy.Delete-Only has the best transfer accuracy but lags in content preservation and fluency.Source: Gender bias mitigation · p. 6

Gender obfuscation

  • In Table 3, BST has the best style-transfer accuracy but the worst content-preservation score on Yelp.The authors describe this as a trade-off between metrics.Source: Gender obfuscation · p. 6, Table 3 · p. 7

General observation

  • Across both datasets, strong performance on one metric tends to come at the expense of others, while the model maintains a balance that can be shifted toward application priorities.The authors state that adjusting balancing parameters can prioritize a desired metric.Source: General observation · pp. 6–7

Ablation analysis

  • Without-LR removes latent content representation but retains the class constraint, whereas Without-LR&SS removes both components.The ablation evaluates these variants on Yelp.Source: Ablation analysis · p. 7, Table 4 · p. 7
  • Without-LR has 96.62% content preservation, described as significantly low, while Without-LR&SS produces the best fluency score.Without-LR often replaces female-attribute words with “pistol”; the authors attribute the other variant's fluency to prioritizing plausible replacements without the class constraint.Source: Ablation analysis · p. 7

Conclusion

The model combines explicit keyword replacement with latent content representation and is reported to perform well on bias mitigation and gender obfuscation. The authors propose several extensions, including other languages and attribute types.

Conclusion

  • On both datasets, the approach achieves good content-preservation and fluency scores while maintaining good style-transfer accuracy.The evaluation covers bias mitigation and gender obfuscation and uses content preservation, fluency, and style-transfer accuracy metrics.Source: Conclusion · p. 7
  • The model combines explicit keyword replacement with latent content information, using a text explainer to identify attribute tokens and a method for generating latent content representations.Source: Conclusion · p. 7
  • Future work includes expanding to other languages, exploring adversarial learning, and examining attributes beyond tokens, such as sentence length.Source: Conclusion · p. 7
  • The authors plan to test the model as a preprocessing technique for training fair language models, which they believe could reduce biases in automated language systems.Source: Conclusion · p. 7

Related papers

Loading 2201.08643v1…