Source-linked AI summary
Dear Sir or Madam, May I introduce the GYAFC Dataset: Corpus, Benchmarks and Metrics for Formality Style Transfer
Sudha Rao, Joel Tetreault
TL;DR
Style transfer has lacked datasets, benchmarks, and reliable automatic metrics beyond English-to-Shakespeare transformation. This paper introduces the 110K-pair GYAFC corpus, adapts rule-based, PBMT, and NMT approaches as benchmarks, and finds that existing metrics often align poorly with human judgments while neural systems can make larger changes at the expense of meaning.
Problem
Style transfer research lacked comparable training and evaluation data, benchmarks, and reliable automatic metrics beyond the English-to-Shakespeare dataset.
Method
The paper constructs GYAFC and evaluates rule-based, PBMT, and NMT approaches for informal-to-formal and formal-to-informal transfer, using automatic and human assessments of formality, fluency, and meaning preservation.
Results
110K informal/formal sentence pairs make GYAFC the largest dataset for any style, while neural systems often make larger successful rewrites but can change meaning, and prior metrics correlate poorly with human judgments.
Takeaways & Limitations
Formality style transfer gains a public corpus and strong MT-based benchmarks, while metric development should prioritize measures that better reflect human judgments.
Takeaways & Limitations
Crowdsourced evaluations contained some errors, especially in the overall ranking of systems.
Abstract
from arXiv · showhide
Style transfer is the task of automatically transforming a piece of text in one particular style into another. A major barrier to progress in this field has been a lack of training and evaluation datasets, as well as benchmarks and automatic metrics. In this work, we create the largest corpus for a particular stylistic transfer (formality) and show that techniques from the machine translation community can serve as strong baselines for future work. We also discuss challenges of using automatic metrics.
1 Introduction
The paper addresses limited resources for formality style transfer by introducing a large corpus, benchmark models, and a broader evaluation of automatic metrics.
- Style-transfer research had only one available dataset beyond which equivalent or larger resources were lacking, blocking progress.
- GYAFC contains 110K informal/formal sentence pairs, making it the largest dataset for any style.
- The authors adapt PBMT and NMT approaches as strong benchmarks for formality style transfer.
- Models are evaluated for formality, fluency, and meaning preservation using automatic metrics compared against human judgments.
- The work primarily studies informal-to-formal transfer while also evaluating formal-to-informal transfer and releasing its data, outputs, and results.
2 Related Work
Prior work applied machine translation and style-control methods to style transfer, but lacked a comparably large formality corpus and thorough metric validation.
- Early parallel style-transfer work used lexical replacement or treated modern-to-Shakespearean transformation as sequence-to-sequence translation.
- Nonparallel approaches controlled attributes such as sentiment, tense, linguistic style, or formality through latent representations, conditioning, or constrained decoding.
- Formality detection had been studied at lexical, sentence, and document levels, and this paper uses a sentence-level classifier for dataset creation and evaluation.
- Previous style-transfer evaluations reused BLEU, PINC, and other automatic metrics, motivating comparison of metrics with human judgments across three criteria.
3 GYAFC Dataset
GYAFC is built from filtered Yahoo Answers data and expert-reviewed rewrites, then analyzed for edit patterns, formality shifts, and sentence-length changes.
- 3.1 Creation Process: The corpus uses Yahoo Answers L6 because it contains many informal sentences across domains and permits redistribution.
- 3.1 Creation Process: After removing questions, URLs, and sentences outside 5–25 words, 40 million Yahoo Answers sentences remained.
- 3.1 Creation Process: Researchers sampled 53,000 informal sentences from each of Entertainment & Music and Family & Relationships and collected formal rewrites through Mechanical Turk.
- 3.1 Creation Process: Four experts reviewed worker rewrites for quality, while expert workers produced multiple references for tune and test sets.
- 3.2 Analysis: Formal rewrites had mean character edit distance 28.85, mean formality rising from −1.06 to 0.12, and mean sentence length increasing from 11.93 to 12.56.
- 3.2 Analysis: Among 50 rewrites, phrasal paraphrases occurred in 47%, punctuation edits in 40%, contraction expansions in 12%, capitalization in 46%, and normalization in 10%.
4 Models
The paper benchmarks rule-based, PBMT, and NMT approaches in both transfer directions, modifying translation methods for limited parallel data and style-specific edits.
- The benchmark suite contains rule-based, PBMT, and NMT approaches applied to informal-to-formal and formal-to-informal transfer.
- Rule-based systems apply style-specific edits such as capitalization, punctuation normalization, and contraction expansion.
- PBMT is trained on rule-based outputs and uses self-training to increase data size and encourage generalization beyond hand-coded rules.
- The NMT baseline is an attention-based bidirectional LSTM encoder-decoder with Yahoo Answers pretrained embeddings.
- NMT Copy adds a copying mechanism to preserve text stretches that should remain unchanged.
- NMT Combined augments training with PBMT-generated data and back-translation to address the limited size of parallel data.
5 Evaluation
The evaluation combines human judgments of formality, fluency, meaning preservation, and overall ranking with corresponding automatic metrics. It also examines how well these automatic measures reflect human assessments.
- Human-based Evaluation: Human evaluation rates outputs for formality, fluency, meaning preservation, and overall system ranking.The study uses 500 test sentences from two domains and collects five judgments per sentence per criterion.
- Human-based Evaluation: Formality judgments use a -3 to +3 scale, while fluency judgments use a 1 to 5 scale with a separate fragment option.Formality ranges from Very Informal to Very Formal; fluency ranges from Perfect to Incomprehensible, plus Other for incomplete sentences.
- Human-based Evaluation: Meaning preservation is judged by rating source–rewrite similarity on the STS-style scale described for the paired sentences.Workers compare each source sentence with its reference rewrite or model output.
- Human-based Evaluation: Overall rankings order reference and model rewrites by formality while taking fluency and meaning preservation into account.The model rank averages its ranks across five judgments for a subset of 500 test sentences.
- Human-based Evaluation: The authors report that most annotations appeared correct, but crowdsourced collection produced some errors, especially in system-level rankings.This caveat concerns the human judgments reviewed by the paper’s authors.
- Automatic Metrics: The automatic evaluation uses dedicated formality, fluency, and meaning measures alongside BLEU, PINC, and TERp.The formality classifier is retrained on judgments from 5000 additional sentences, while HE15 supplies sentence-level meaning similarity.
6 Results
Human and automatic evaluations reveal distinct trade-offs among the five informal-to-formal systems, while metric correlations expose weaknesses in several automatic measures.
- Human evaluation: NMT Baseline and NMT Copy achieve significantly higher human-judged formality, while the rule-based model preserves meaning best.
- Human evaluation: NMT Combined is the only model with a combined human-evaluation score statistically greater than the rule-based approach.
- Overall rankings: PBMT and NMT Combined outperform the rule-based model overall in the F&R domain, while the rule-based system ranks third and exceeds both NMT Copy and NMT Baseline.
- Automatic evaluation: Automatic formality and meaning scores align with human judgments, but automatic fluency favors NMT Baseline rather than the human-preferred NMT Combined model.The fluency discrepancy may reflect the H14 classifier’s training on more formal essays.
- Metric correlation: Formality and fluency metrics correlate moderately with human judgments, whereas meaning correlates comparatively poorly; PINC correlates very poorly and favors lexical edits over meaning preservation.
- Manual analysis: PBMT and NMT Combined make more conservative, usually correct changes, preserving meaning at the expense of smaller formality improvements.
- Manual analysis: All models struggle with highly informal or disfluent inputs, sentence completions, and unknown or misspelled tokens, despite handling low-level edits effectively.
7 Conclusions and Future Work
The paper advances formality style transfer with a public corpus, MT-based benchmarks, and metric analysis. Human judgments slightly prefer PBMT, while neural systems often make larger successful rewrites at a cost to meaning.
- The paper creates a large public training and evaluation corpus, adapts MT techniques as benchmarks, and analyzes automatic metrics for style transfer.
- Humans slightly prefer PBMT, whereas NMT Baseline and NMT Copy often make successful larger rewrites that other models cannot, sometimes changing meaning.
- Previously used metrics correlate poorly with human judgments; formality and fluency correlate best, motivating combinations with other metrics.
8 Supplementary material
The supplementary material provides additional details about the dataset, models, and results, including formal-to-informal evaluations.
- The supplement adds details supporting the dataset, models, and results introduced in the main paper, including formal-to-informal results.
8.1 Dataset
The supplementary dataset analysis examines manual and automatic frequencies of edits used to make sentences more formal.
- Manual and automatic analyses compare the frequencies of capitalization, punctuation, and normalization edits across GYAFC.Automatic analysis finds higher capitalization and punctuation percentages, and much higher normalization percentages, than manual counting.
8.2 Models
The models combine rule-based transformations with PBMT and NMT training strategies designed to extend learning beyond hand-crafted rules and limited parallel data.
- Rule-based model: The rule-based system applies capitalization, casing, contraction, slang, uppercasing, and character-repetition transformations, with some reverse-direction rules applied proportionally.Repeated characters are collapsed, while formal-to-informal uppercasing and repetition follow their frequencies in the training split.
- PBMT model: PBMT is trained on rule-based outputs so it can learn generalizations beyond the manually specified transformations.A baseline trained directly on GYAFC mainly reproduces the crafted rules, motivating this training setup.
- PBMT model: Self-training enlarges the parallel corpus by translating in-domain source-style sentences with PBMT and retraining on the resulting pairs.Additional pairs are sub-selected when their edit distance exceeds 10 to favor translations involving more edits.
- Training data: The models use data selection to train a larger language model on out-of-domain sentences closer to the target-style in-domain data.Combined models include duplicated GYAFC data, self-training data, and, for NMT Combined, back-translated data.
- NMT model: The NMT implementation uses OpenNMT-py with a 50K vocabulary, 300-dimensional embeddings, attention-based unknown-token replacement, and Yahoo Answers GloVe initialization.The supplied model description identifies the toolkit settings and pretrained embeddings used at test time and training time.
8.3 Results
Results differ by domain and transfer direction: F&R follows the main-domain pattern, whereas formal-to-informal transfer favors simple rules and exposes weaknesses in reference-based metrics.
- F&R domain: In the F&R domain, model-performance observations across human and automatic metrics are similar to those in E&M, but scores are higher overall.The original informal sentences and model outputs receive higher formality scores than their E&M counterparts.
- Formal-to-informal task: The reverse task uses reused data rather than newly collected informal rewrites, making it fundamentally different from informal-to-formal evaluation.This distinction changes the evaluation setting for the second transfer direction.
- Formal-to-informal task: For formal-to-informal transfer, the rule-based model beats the other models on human and automatic formality, fluency, and meaning criteria, except automatic meaning.PBMT Combined wins the automatic meaning metric, while NMT Combined and PBMT Combined lead BLEU and TERp, and NMT Baseline leads PINC.
- Metric analysis: Reference-based BLEU, TERp, and PINC correlate very poorly with the overall human ranking on the formal-to-informal task.The paper relates this mismatch to the greater variability of informal reference rewrites.
- Formal-to-informal task: Simple lexical transformations such as ‘just’ to ‘juz’ and ‘because’ to ‘cuz’ help the rule-based method outperform other models in sample outputs.The authors conclude that surpassing rules in this direction requires models tailored to its distinct challenges.