Source-linked AI summary
Pay Attention to Your Tone: Introducing a New Dataset for Polite Language Rewrite
Xun Wang, Tao Ge, Allen Mao, Yuki Li, Furu Wei, Si-Qing Chen
TL;DR
Polite language rewriting must preserve a sentence’s message while making it euphemistic and polite, a more demanding task than slight token- or phrase-level edits. The paper builds POLITEREWRITE through human–GPT-3.5 collaboration, and preliminary experiments find lower BLEU scores than on other rewriting tasks. The dataset combines 10K collaborative pairs with 100K GPT-3.5 pairs without human review.
Problem
Polite rewriting requires deep understanding and sentence-level changes to preserve an impolite sentence’s message while expressing it euphemistically and politely.
Source: §1 · p. 1Method
Results
Preliminary experiments find POLITEREWRITE more challenging than other rewriting tasks, with much lower BLEU scores; it contains 10K collaborative pairs and 100K GPT-3.5 pairs without human review.
Source: §1 · pp. 1–2Takeaways & Limitations
The authors hope POLITEREWRITE will contribute to research on more challenging natural language generation and resource annotation with large pretrained models.
Source: §5 · p. 5
Abstract
from arXiv · showhide
We introduce \textsc{PoliteRewrite} -- a dataset for polite language rewrite which is a novel sentence rewrite task. Compared with previous text style transfer tasks that can be mostly addressed by slight token- or phrase-level edits, polite language rewrite requires deep understanding and extensive sentence-level edits over an offensive and impolite sentence to deliver the same message euphemistically and politely, which is more challenging -- not only for NLP models but also for human annotators to rewrite with effort. To alleviate the human effort for efficient annotation, we first propose a novel annotation paradigm by a collaboration of human annotators and GPT-3.5 to annotate \textsc{PoliteRewrite}. The released dataset has 10K polite sentence rewrites annotated collaboratively by GPT-3.5 and human, which can be used as gold standard for training, validation and test; and 100K high-quality polite sentence rewrites by GPT-3.5 without human review. We wish this work (The dataset (10K+100K) will be released soon) could contribute to the research on more challenging sentence rewrite, and provoke more thought in future on resource annotation paradigm with the help of the large-scaled pretrained models.
1 Introduction
Polite language rewrite must preserve meaning while recasting impolite sentences euphemistically, making it more demanding than word- or phrase-level rewriting. The paper introduces POLITEREWRITE, built with a human–GPT-3.5 annotation process, to support research on this challenge.
1 Introduction
- Unlike word- or phrase-level substitution, polite rewriting requires sensitivity to subtle emotional and sentiment changes while conveying similar information.The task aims to express an impolite sentence euphemistically and politely.Source: §1 · p. 1
- POLITEREWRITE contains 10K human–GPT-3.5 collaborative sentence pairs and 100K GPT-3.5 rewrites without human review.For the collaborative pairs, GPT-3.5 generates rewrites and human experts accept them or request edits.Source: §1 · p. 1
- Preliminary experiments find POLITEREWRITE more challenging than other rewriting tasks, with much lower BLEU scores.The paper presents this as an initial finding, not a reported numerical comparison.Source: §1 · pp. 1–2
- The dataset is intended to guide generation models away from verbal abuse and offensive expression.Source: §1 · p. 2
- The collaborative annotation paradigm is presented as reducing manual effort while maintaining high annotation quality.The paper identifies it as the first human–GPT-3.5 collaborative annotation paradigm for a sentence-rewrite dataset.Source: §1 · p. 2
2 POLITEREWRITE
POLITEREWRITE uses a human–GPT-3.5 annotation loop to build gold and silver polite-rewrite data. The reported process reduces annotation effort while retaining quality, and dataset statistics support the claim that polite rewriting requires substantial revision.
Dataset composition
- POLITEREWRITE contains 10K human-reviewed gold pairs and 100K silver pairs generated by fine-tuned GPT-3.5 without thorough human review.The silver data was produced after the collaborative annotation results prompted further generation.Source: §2 · pp. 2–3
Source selection
- Source sentences are first selected as negative by a UNILM sentiment analyzer fine-tuned on SST-2, then screened by GPT-3.5 using 8-shot in-context learning for actual impoliteness.The selected candidates proceed to collaborative annotation.Source: §2 · p. 2
Annotation workflow
- Annotation cycles from GPT-3.5 seed rewrites for 1K samples through human acceptance or revision, model fine-tuning, and generation for the next batch.Some sentences may be removed as unsuitable for polite rewrite.Source: §2 · p. 2
Annotation results
- The final annotation iteration accepted about 50% of GPT-3.5 rewrites unchanged, improved another 20%, discarded about 15% for sensitive content, and achieved over 10× annotation efficiency.About 10% were judged unsuitable for polite rewrite.Source: §2 · p. 3
Annotation quality
- Over 90% of 500 sampled sentences were indistinguishable to a native-speaker judge as human-only or collaboratively annotated, suggesting collaboration did not harm annotation quality.One additional human annotated the sample; another English native speaker made the distinction.Source: §2 · p. 3
Dataset challenge
- Leaving source sentences unchanged yields around 25 BLEU on POLITEREWRITE, versus 51 BLEU for unchanged inputs in GYAFC formality transfer, indicating more extensive revisions are needed.The paper uses these no-revision scores as evidence that POLITEREWRITE is more challenging.Source: §2 · p. 3
3 Benchmark
The benchmark evaluates Transformer and BART baselines using gold data, with and without silver data, and includes human ratings. BART with Gold+Silver reaches 37.5 BLEU, and the reported results indicate gains from silver data and a positive role for pretraining.
Experimental setup
- The 10K gold set is randomly split into 5K training, 3K development, and 2K test examples; gold training data is used with silver data for neural baselines.The listed baselines include Transformer-base and BART.Source: §3 · pp. 3–4
Benchmark results
- 37.5 BLEU is achieved by the best fine-tuned BART model trained on Gold+Silver data, which outperforms the Transformer trained from scratch.Adding 100K silver data also yields a significant performance gain.Source: §3 · p. 4, Table 6 · p. 4
Human evaluation
- Human–BLEU correlation was 0.68, below the 0.84 human–human correlation, yet the authors considered it persuasive as a reliable automatic evaluation metric for polite-language rewriting.Two English native speakers rated anonymous outputs from the four models on a 1-to-5 scale, with 5 the best polite rewrite and 1 the worst.Source: §3 · p. 4
4 Related Work
The related work frames rewriting as a family of tasks including grammatical correction, style transfer, and translation. It situates POLITEREWRITE amid labor-intensive data creation and prior uses of pretrained models for annotation and generation.
Rewrite tasks
- Rewrite tasks take text as input and produce desired text, with grammatical error correction, style transfer, and machine translation given as examples.Source: §4 · p. 4
Data creation
- Data creation for rewrite models remains labor intensive, while prior augmentation approaches include back translation and transfer learning.Source: §4 · p. 4
Pretrained models
- Prior work applied GPT-3.5 to data generation across domains, and this paper uses its knowledge with minimum human labor for a challenging neural rewrite task.The passage also notes few-shot generation with a small number of handcrafted examples (Brown et al., 2020).Source: §4 · pp. 4–5
5 Conclusion
The conclusion presents POLITEREWRITE as a challenging sentence-rewrite dataset annotated collaboratively by humans and GPT-3.5. The authors hope it will contribute to research on challenging generation and inform future resource annotation paradigms.
Conclusion
- POLITEREWRITE is introduced as a more challenging sentence-rewrite dataset annotated through collaboration between humans and GPT-3.5.Source: §5 · p. 5
- The authors hope the work will contribute to research on challenging natural language generation and prompt further thought about annotation with large pretrained models.Source: §5 · p. 5