The Perils of Using Mechanical Turk to Evaluate Open-Ended Text Generation

1 Introduction

• Similarity metrics such as BLEU and ROUGE are mostly uninformative for open-ended generation, although such metrics are still commonly reported.The passage attributes their limited usefulness to the huge space of possible outputs compared with constrained tasks such as machine translation.Source: §1 · p. 1
• The study surveys 45 papers from 2018–2020 and evaluates paragraph-length WritingPrompts stories with AMT workers and English teachers using a common five-point rating setup.The AMT task used 200 examples and three annotators per example.Source: §1 · p. 1
• AMT ratings did not reliably distinguish model-generated from human-written stories unless workers rated both types side-by-side; separate-day runs also varied enough to suggest references were lower quality than GPT-2.The paper reports that 42% of workers on average spent fewer than 40 seconds per task, and filtering them changed overall ratings while reducing datapoints.Source: §1 · pp. 1–2
• Even expert raters found model-generated stories harder to judge: they spent more time per example than on references, and agreement dropped.The paper notes that strict worker qualifications did not prevent many AMT workers from reading too quickly.Source: §1 · p. 2
• The authors urge researchers to use expert raters when possible; if AMT is necessary, they recommend rating references alongside generated outputs and heavily filtering workers.The side-by-side ratings are intended to improve calibration.Source: §1 · p. 2

2 A survey of papers that evaluate open-ended text generation with AMT

• The survey focuses on AMT evaluations of open-ended English generation, including stories, metaphors, paraphrases, puns, sarcasm, and style- or attribute-transferred sentences.It covers papers published from 2018 to 2020 at ACL, NAACL, or EMNLP and excludes more established tasks such as machine translation and summarization.Source: §2 · p. 2
• Surveyed papers used varied evaluation criteria, most commonly fluency or grammaticality (19), overall quality (12), and relevance (10).For stories, coherence appeared in 9 papers and likability in 4.Source: §2 · p. 2
• Twenty-four papers used a five-point Likert scale, and 19 of those labeled only the endpoints; ranking two or more outputs was the next most common format, used by 23 papers.Other formats included different Likert scales, pass/fail tasks, and output-prompt matching.Source: §2 · p. 3
• Fourteen papers did not report the number of raters and six did not report the number of items; among the rest, 16 used three AMT workers per item.The most common evaluation size was 100 items, reported by 14 papers.Source: §2 · p. 3
• Among reviewed papers, 32 did not report AMT worker qualifications and 35 did not report worker compensation, findings the authors say add to reproducibility woes.Only 11 papers mentioned restricting workers to English-speaking countries or applying a language test, despite all evaluations using English text.Source: §2 · p. 3
• Reported text lengths varied from single sentences (28 papers) to short paragraphs (7) and longer paragraphs (14), with longer paragraphs most common for story generation.Source: §2 · p. 3

3.1 Experimental Setup

• Experiments used 200 prompts, three judgments per HIT, and ratings of grammar, coherence, likability, and relevance on a five-point scale.Experiments varied whether workers saw human-written stories, GPT-2 stories, or both; each experiment used a different worker set.Source: §3.1 · pp. 3–4

3.2 AMT Evaluations of Reference Text

• The strictest worker qualifications yielded the highest coherence and relevance scores and brought ratings closer to teachers, but substantial deviations remained.Qualifications included English-speaking countries, approval rate above 90%, and at least 1,000 approved HITs; agreement remained low, including for likability.Source: §3.2 · p. 4
• Repeated qualified-worker runs varied by day: Day 2 ratings were lower across the board, and Day 3 relevance was 3.46 versus 3.71 on Day 1.Grammar and coherence means were similar on Days 1 and 3, while the second day had poor overall inter-annotator agreement.Source: §3.2 · p. 5, Table 1 · p. 5
• 42% of ratings were filtered out on average using a low-bar median-time cutoff of 40 seconds.AMT-reported time could be inflated by workers accepting multiple HITs: one reported 360-second mean corresponded to a 22-second actual mean and 13-second median.Source: §3.2 · pp. 5–6
• Workers from non-English-speaking countries rated coherence, relevance, and grammar significantly lower than identically qualified workers from English-speaking countries.Only 11 surveyed papers reported filtering workers by country or language knowledge, despite evaluating English text.Source: §3.2 · p. 6

3.3 Evaluating Machine-Generated Text

• When rated side by side, GPT-2 stories scored lower than references on coherence, relevance, and likability, but similarly on grammar; separate ratings often failed to distinguish them.For side-by-side AMT ratings, coherence was 3.39 vs. 3.83, relevance 2.70 vs. 3.49, and likability 2.99 vs. 3.48; grammar was 3.82 vs. 3.83. Separate ratings sometimes gave GPT-2 and reference stories similar scores.Source: §3.3 · pp. 6–7, Table 2 · p. 7

Recruiting English teachers

• Three teachers with English-teaching qualifications rated the same 200 human-written and 200 GPT-2 stories using the AMT task interface.The teachers were recruited from the authors’ personal networks and each had an English-teaching degree or CELTA certificate.Source: §4 · p. 7

Unlike AMT workers, teachers rate reference stories higher than GPT-2 generated ones

• Teachers rated human-written stories above GPT-2 stories for coherence (M=4.38 vs. M=3.73), relevance (M=3.82 vs. M=2.54), and likability (M=3.69 vs. M=2.96), but not grammar (M=4.50 vs. M=4.56).All differences for coherence, relevance, and likability had p<0.001.Source: §4 · pp. 7–8

Teachers need to see many examples to properly calibrate their ratings

• Teachers said 10–20 stories were needed to calibrate ratings and suggested a standardized rubric and showing GPT-2 output beside references.They reported struggling more with GPT-2 stories and needing to recalibrate their scale when rating them.Source: §4 · p. 8

Coherence is difficult to rate for machine-generated text

• Teachers found coherence hardest to judge in GPT-2 stories, while relevance was easiest because many stories quickly departed from the prompt.Their average rating time rose from 69.8 seconds for human-written stories to 87.3 seconds for GPT-2 stories (p<0.05).Source: §4 · p. 8

Resolving teacher disagreement

• After re-rating and discussing 60 high-disagreement stories, teachers reached consensus on 80% of them.Disagreements included lapses of judgment, missing world knowledge, and differing views about slang and grammaticality.Source: §4 · p. 8

Replicating the study on Upwork

• Upwork teachers also rated human-written stories higher than GPT-2 stories for coherence, relevance, and likability (all p’s<0.001).Their inter-annotator agreement was higher than that of teachers recruited from the authors’ personal networks.Source: §4 · pp. 8–9

Human evaluation of text quality

• Most prior human-evaluation studies address machine translation, summarization, or very short outputs rather than longer open-ended text.The cited examples include outputs that are often only one sentence long.Source: §5 · p. 9
• Creative-text studies use crowd workers and often omit reference texts, while other work asks workers to choose between or distinguish human and machine text.Examples include stories and other less-constrained texts.Source: §5 · p. 9

Data collection using AMT

• Prior AMT research reports quality risks including positive qualification bias, concentrated response production, masked worker locations, and lower-quality data.The cited work cautions that approval rate and country of residence may not adequately filter spammers.Source: §5 · p. 9

Recommendations & Conclusion

• Even expert raters find open-ended generated text challenging, and AMT variance, poor calibration, and cognitively demanding tasks can yield misleading conclusions.The authors give the example of human-written text appearing lower quality than GPT-2 output.Source: §6 · p. 9
• Strict worker qualifications alone do not address the root of AMT evaluation problems.The authors recommend filtering by observed time per HIT, limiting items per worker, testing language proficiency, and providing training HITs.Source: §6 · p. 9
• Presenting machine-generated text alongside human references can improve rating calibration.The authors also urge researchers to use expert raters, such as linguists or language teachers, whenever possible.Source: §6 · p. 9
• Expert rating was not much more expensive in the reported comparison: $144 for AMT, $187.50 for English teachers, and $262.5 for Upwork teachers to rate 200 stories.These are the study’s reported costs for each group.Source: §6 · p. 9

Ethical Considerations

• Evaluation texts from social media and web-trained language models may contain racist, sexist, or otherwise vulgar content, so participants should be warned.The authors report that they provided such warnings in their own research.Source: §7 · p. 9
• The authors ensured crowd workers earned at least $14 per hour by assuming 50–55 seconds per HIT.They note that workers’ average hourly pay was substantially higher because average HIT completion time was low.Source: §7 · pp. 9–10
• The English teachers averaged around $20 per hour, excluding mediation.The authors report hourly wages for both crowd workers and experts.Source: §7 · pp. 9–10

Qualifications

• Bracketed counts in Table A1 indicate the number of papers or experiments that employed each measure.The supplied passage does not include the table’s specific counts.Source: Table A1 · p. 15

A Questions Used for the Survey

• It also recorded rating labels, attribute definitions, and instructions given to raters.These questions cover how rating criteria were presented.Source: A Questions Used for the Survey · p. 15
• The questionnaire asked about worker qualifications and quality controls, plus the numbers of rated items and systems.These questions address annotation quality controls and evaluation scale.Source: A Questions Used for the Survey · p. 15
• The survey asked about raters per item, inclusion of ground truth, task type, and rated-text length.These questions capture how the evaluations were set up.Source: A Questions Used for the Survey · p. 15

B Collecting Ratings on Upwork

• Three Upwork teachers, all holding TEFL, TESOL, or CELTA certificates, rated 200 human-written and 200 GPT-2 stories for $175.They used the AMT platform and rated human-written stories first, then GPT-2 stories after a break of a few days.Source: B Collecting Ratings on Upwork · p. 15
• Human-written stories received higher ratings than GPT-2 stories on coherence, relevance, and likability.The Upwork Welch’s t-test table reports this comparison, consistent with the prose description.Source: B Collecting Ratings on Upwork · p. 15, Table A4 · p. 17

C Details on Post-rating Interviews

• In mediation, teachers agreed on only about 20% of previously disputed ratings; discussion often led them to understand the other perspective and adjust ratings.Disagreements included differences in world knowledge, prompt interpretation, treatment of author comments, and sentence connections.Source: C Details on Post-rating Interviews · p. 17
• Two teachers re-evaluated 60 stories on disputed coherence, grammar, and relevance ratings; likability was excluded as subjective.Each mediation meeting lasted about two hours.Source: C Details on Post-rating Interviews · p. 17
• After each batch, each teacher completed an approximately 10-minute interview about rating time, process, rereading, and calibration.The questions included what constituted the ends of the rating scale.Source: C Details on Post-rating Interviews · p. 17

4. How often did you take breaks?

• Teachers were also asked which attribute was easiest or most difficult to calibrate.Source: §4 · p. 17

• Teachers rated human-written stories higher than GPT-2 stories for coherence, relevance, and likability.Source: Table A5 · p. 18
• The group-effect ANOVA tables report partial eta squared of 0.14, described as a large effect, for grammar, coherence, relevance, and likability ratings of reference texts.Source: Table A6 · p. 18, Table A8 · p. 18, Table A10 · p. 19
• Reference-story ratings differed by rater group: non-English-speaking workers differed from English-speaking workers on grammar, coherence, relevance, or likability, with additional day-to-day differences.The reported pairwise differences include grammar on Days 1 and 3, relevance on all three days, and likability on Day 2; coherence ratings also differed by rater group.Source: Table A7 · p. 18, Table A9 · p. 19, Table A11 · p. 20
• On Days 1–3, AMT workers rated human-written stories higher on selected qualities: coherence and relevance on Day 1, relevance and likability on Day 2, and coherence on Day 3.Source: Table A14 · p. 20, Table A15 · p. 20, Table A16 · p. 20
• Workers from non-English-speaking countries rated GPT-2 stories higher than human-written stories for grammar, coherence, and relevance.Source: Table A17 · p. 21
• When both stories appeared in one HIT, AMT workers rated GPT-2 stories lower for coherence, relevance, and likability, matching teachers’ ratings.Source: Table A18 · p. 21