PaperlayerView exact revision on arXiv

Source-linked AI summary

DynaSent: A Dynamic Benchmark for Sentiment Analysis

Christopher Potts, Zhengxuan Wu, Atticus Geiger, Douwe Kiela

arXiv:2012.15349v1cs.CL

TL;DR

Sentiment models score highly on available benchmarks yet still fall short in real-world systems. DynaSent combines naturally occurring sentences with crowdworker-created challenges for a dynamic ternary benchmark; its Neutral examples and validation variation can help separate neutrality, mixed sentiment, and uncertainty.

  • Problem

    Sentiment models achieve strong results on available benchmarks but still fall short in real-world systems and display clear shortcomings.

    Source: §1 · p. 1

  • Method

    DynaSent combines naturally occurring sentences with examples created by workers trying to fool a strong sentiment model, and its sentences are multiply validated by crowdworkers.

    Source: §6 · p. 11

  • Results

    DynaSent’s large Neutral set and validation variation can help distinguish neutrality from mixed and uncertain sentiment.

    Source: §1 · p. 2

  • Takeaways & Limitations

    The authors propose creating additional DynaSent rounds when future models solve the current ones, keeping the benchmark responsive to new models and attacks.

    Source: §1 · p. 1

    Crowdsourced texts may inherit worker biases, and the naturally occurring and prompt-based texts still come from people who write online reviews, a special group.

    Source: §2.3 · p. 4

Abstract

from arXiv · show

We introduce DynaSent ('Dynamic Sentiment'), a new English-language benchmark task for ternary (positive/negative/neutral) sentiment analysis. DynaSent combines naturally occurring sentences with sentences created using the open-source Dynabench Platform, which facilities human-and-model-in-the-loop dataset creation. DynaSent has a total of 121,634 sentences, each validated by five crowdworkers, and its development and test splits are designed to produce chance performance for even the best models we have been able to develop; when future models solve this task, we will use them to create DynaSent version 2, continuing the dynamic evolution of this benchmark. Here, we report on the dataset creation effort, focusing on the steps we took to increase quality and reduce artifacts. We also present evidence that DynaSent's Neutral category is more coherent than the comparable category in other benchmarks, and we motivate training models from scratch for each round over successive fine-tuning.

1 Introduction

DynaSent is introduced as a dynamic English ternary sentiment benchmark intended to address the gap between strong benchmark scores and real-world utility. It combines naturally occurring sentences with model-challenging crowdworker examples and uses prompt-based collection to reduce artifacts.

1 Introduction

  • DynaSent addresses the gap between benchmark performance and real-world utility in English ternary sentiment analysis.The authors note that present-day models perform strongly on available benchmarks but still fall short in real-world deployments.Source: §1 · p. 1
  • DynaSent is intended to expand with new models, modeling goals, and adversarial attacks, with future rounds created when models solve current ones.The paper frames this as a continuing benchmark rather than a fixed test set.Source: §1 · p. 1
  • DynaSent combines naturally occurring sentences and crowdworker-created examples designed to fool a strong sentiment model across two rounds.The process uses an initial model to find challenging existing examples, then a model trained on Round 1 data to challenge workers in Round 2.Source: §1 · pp. 1–2
  • Prompt-based Round 2 examples modify naturally occurring sentences and are more like natural sentences in length and lexical diversity than unprompted examples.The authors propose this alongside Round 1 harvesting to mitigate creative ruts and dataset artifacts, while acknowledging that fundamental bias sources remain.Source: §1 · p. 2
  • DynaSent’s Neutral examples and validation variation can help distinguish neutrality from mixed and uncertain sentiment, while the category’s variability favors training models from scratch over fine-tuning.The paper argues that the comparable category in the Stanford Sentiment Treebank blurs these phenomena and that Neutral variability is an obstacle to fine-tuning.Source: §1 · p. 2

2 Related Work

Related work situates DynaSent among sentiment benchmarks, adversarial datasets, and crowdsourcing methods. It highlights the limits of review-heavy benchmarks, the value of challenging examples, and the biases and label-inference issues that accompany crowdsourced data.

2.1 Sentiment Benchmarks

  • Most sentiment benchmarks rely on product and service reviews, a plentiful source that can also limit progress through over-reliance.DynaSent begins moving away from this domain but remains rooted in it; other benchmarks include phrase-labeled news, Twitter sentiment datasets, and sentiment lexicons.Source: §2.1 · pp. 2–3

2.2 Challenge and Adversarial Datasets

  • Challenge and adversarial datasets probe model weaknesses, and DynaSent is among the large datasets that support both evaluation and full-scale training.The authors model their iterative human-and-model process on Adversarial NLI, where workers create examples that fool a strong model but make sense to other annotators.Source: §2.2 · p. 3

2.3 Crowdsourcing Methods

  • Crowdsourced workers may not represent the wider population, and DynaSent’s natural and prompt-based texts still come from a special group of online reviewers.The authors acknowledge that such texts may inherit biases, while noting that crowdsourcing can also produce artifacts through shared worker strategies.Source: §2.3 · pp. 3–4
  • DynaSent treats the true 3/5 majority as the gold label when available, leaves other examples unlabeled, and includes full response distributions in the release, which Model 1 uses for training.Source: §2.3 · p. 4

3 Round 1: Naturally Occurring Sentences

Round 1 harvests naturally occurring sentences that challenge Model 0, validates them, and uses the resulting data to train Model 1. Its construction combines model assessment, worker validation, and two ways of using annotation distributions.

3.1 Model 0

  • Round 1 begins with Model 0, which is trained on public datasets and evaluated on SST-3, Yelp, and Amazon assessment data.The model uses RoBERTa-base with a three-way sentiment classifier; star ratings are binned into negative, neutral, and positive.Source: §3.1 · p. 4, §3 · p. 4
  • Model 0 performs well on Positive and Negative but less well on Neutral, whose source categories mix neutrality, mixed sentiment, and uncertainty.The authors argue that DynaSent makes progress toward a more coherent Neutral category.Source: §3.1 · pp. 4–5, Table 2 · p. 5

3.2 Harvesting Sentences

  • Half of the harvested sentences come from 1-star reviews predicted Positive or 5-star reviews predicted Negative, targeting likely model errors.The authors note that reviews can contain sentiment contrary to their star rating, motivating human validation.Source: §3.2 · p. 5

3.3 Validation

  • Each corpus sentence receives five retained worker labels after iterative quality screening; workers whose disagreement exceeded 80% were removed from the potential pool.Additional labels were collected for sentences annotated by unqualified workers, while the final corpus retains responses from the highest-rated workers.Source: §3.3 · p. 6

3.4 Round 1 Dataset

  • 80,488 Round 1 training examples have Positive, Negative, or Neutral majority labels, each requiring at least three of five votes.The dataset design also targets balanced classes, chance performance for Model 0, and at least four annotators' agreement for included majority labels.Source: Table 4 · p. 6, §3.4 · p. 7
  • Distributional labels retain all five responses as training information, and the authors report slightly better models with this approach.It makes use of examples without a majority label as well as the full labeling information.Source: §3.4 · p. 6

3.5 Estimating Human Performance

  • Human F1 is estimated by comparing each of five synthetic annotators with the gold label; the estimate is conservative because it weights lone dissenters heavily.The authors suggest comparable aggregate F1 as a useful signal that a model has solved the round and a new round can begin.Source: §3.5 · p. 7

4 Round 2: Dynabench

Round 2 uses Dynabench to collect human-validated examples aimed at fooling a sentiment model. The authors report low worker success, while constructing dev and test splits designed for chance model performance and strong label agreement.

4 Round 2: Dynabench

  • Dynabench is an open-source browser platform where workers create examples that fool a model but still make sense to humans.The Round 2 effort begins a dynamic sentiment benchmark.Source: §4 · p. 7

4.1 Model 1

  • Model 1 upsampled SST-3 threefold and DynaSent twofold, giving DynaSent roughly equal example weight to all other datasets combined.The authors report a large performance jump on Round 1 alongside a small drop on external datasets, which they associate with shifting the label distribution.Source: §4.1 · p. 8

4.2 Dynabench Interface

  • Workers chose a target label and tried to write a sentence that Model 1 misclassified but other humans would label with the target.They could try repeatedly and receive feedback on the model’s predictions.Source: §4.2 · p. 8

4.3 Methods

  • The authors kept the Prompt condition because it yielded more naturalistic and linguistically diverse examples than No Prompt.Dev and test prompts were unique; a small number of prompts were repeated elsewhere.Source: §4.3 · p. 8

4.4 Validation

  • Round 2 examples received five validation responses, with 395 workers participating.The validation process matched the one used for Round 1.Source: §4.4 · p. 8

4.5 Round 2 Dataset

  • 19% of Round 2 attempts fooled Model 1, compared with 47% in Round 1; mixed-sentiment examples and task confusion contributed to the difficulty.The authors also retained attempts after workers reached the maximum number of tries.Source: §4.5 · pp. 8–9, Table 7 · p. 9
  • Round 2 dev and test contain Prompt examples with unique prompts, majority labels supported by at least four of five workers, and Model 1 performance at chance.No Prompt examples were kept in train and flagged as such.Source: §4.5 · p. 9

4.6 Estimating Human Performance

  • Round 2 human F1 estimates are conservative; 116 of 244 workers never disagreed with the gold label on examples they rated.The authors use these estimates to help decide whether to continue improving the round or begin new ones.Source: §4.6 · p. 9

5 Discussion

The discussion examines prompted sentence collection, the coherence of the Neutral category, and fine-tuning behavior across DynaSent and external datasets.

5.1 The Role of Prompts

  • Prompt use yielded a roughly bimodal edit-distance distribution, with peaks for slight edits and substantial deviations, and almost no literal reuse or complete disregard.No Prompt workers repeatedly used idiosyncratic strategies, so those examples were included only in training and made identifiable for special handling.Source: §5.1 · pp. 9–10
  • Prompt examples were generally longer and more lexically diverse than No Prompt examples, and more like Round 1 examples.Vocabulary diversity was assessed by repeatedly sampling 100 examples per condition, then five words per example.Source: §5.1 · p. 9

5.2 The Neutral Category

  • On SST dev, the original Neutral labels often fell into Mixed or No Majority under the authors’ validation, and aligned less well than Positive or Negative labels.The authors argue that middle-scale review or reader ratings can blur neutrality with mixed or uncertain sentiment.Source: §5.2 · p. 10

5.3 Fine-Tuning

  • Fine-tuning Model 0 on Round 1 data produced worse models, generally because of very low Neutral performance, while Model 1 was trained from scratch.The authors suggest training from scratch is less susceptible to label-shift issues, though they say label shift may still contribute to weaker external Neutral performance.Source: §5.3 · pp. 10–11
  • Neutral F1 on Round 1 dev rises with more examples while scores on Model 0’s original development sets drop sharply, a pattern associated with artifacts or label shifts.The authors hypothesize that the difference between DynaSent’s Neutral category and other datasets’ Neutral categories contributes to this pattern.Source: §5.3 · p. 11

6 Conclusion

DynaSent’s first two rounds form the start of a dynamic sentiment benchmark built to respond to advances in modeling and concerns about real-world deployment. The authors propose collecting another round when future models solve both current rounds.

6 Conclusion

  • DynaSent combines 121,634 naturally occurring and Dynabench-created sentences, all multiply validated by crowdworkers.The conclusion describes the dataset as the first stage of an ongoing benchmark effort.Source: §6 · p. 11
  • The authors argue that prompting workers to modify natural examples can help them avoid creative ruts and reduce resulting dataset biases and artifacts.The conclusion presents this as a rationale for the Prompt setting on Dynabench.Source: §6 · p. 11
  • The authors propose launching another data-collection round once future models solve both existing rounds.The new round would aim to fool those models and continue advancing the benchmark.Source: §6 · p. 11

A Model 0

The appendix reports Model 0’s training configuration and describes the interface used for validation tasks.

A Model 0

  • Model 0 imports pretrained RoBERTa-base weights and uses 12 layers, 12 heads, a 768-unit hidden size, and a maximum sequence length of 128.Training used a 2e−5 learning rate, batch size 8 per GPU, and three epochs.Source: A Model 0 · p. 16

B.1 Validation Interface

  • Validation workers saw one item at a time in a ten-item HIT and received US$0.25 per HIT, with payment regardless of whether labels were retained.The interface is used for both Round 1 and Round 2.Source: §B.1 · p. 16

B.2 Worker Selection

The authors describe iterative worker selection for validation and report the distribution of worker workloads across the two rounds.

B.2 Worker Selection

  • Workers labeling in 3–5K-example batches were removed if they disagreed with the majority more than 80% of the time, and affected examples were revalidated.The process was repeated iteratively; many Round 1 examples received more than five labels, with only top-ranked workers’ labels retained.Source: §B.2 · p. 16
  • The authors acknowledge that excluding workers by majority disagreement can introduce unwanted biases, since disagreement may reflect a genuinely different perspective.They note that disagreement can also stem from inattentiveness, reading comprehension, or task-understanding issues.Source: §B.2 · p. 16

B.3 Worker Distribution

  • Round 1 validation had a median of 45 examples per worker and a mode of 11; Round 2 had a median of 20 and a mode of 1.Source: §B.3 · p. 16

B.4 Worker Agreement with Gold Labels

Across both rounds and the development and test sets, substantial numbers of workers agreed with the gold label on every case they labeled, and more than half exceeded 95% agreement.

B.4 Worker Agreement with Gold Labels

  • More than half of workers exceeded 95% agreement with the gold label across both rounds’ development and test sets.Substantial numbers agreed with the gold label on every case they labeled.Source: §B.4 · p. 16

C Additional Details on Dynabench Task

The appendix shows the Dynabench interface for the Prompt condition and describes the task instructions and comprehension checks workers completed before starting.

C.1 Interface for the Prompt Condition

  • Figure 7 shows an example of the Dynabench interface in the Prompt condition.Source: §C.1 · p. 16

C.2 Instructions

  • Figure 8 contains the task instructions, and workers had to answer comprehension questions correctly before starting.The questions are listed in Table 11.Source: §C.2 · p. 16

C.3 Data Collection Pipeline

The collection pipeline combines worker qualifications, onboarding and validation checks, and incentives for generating examples that fool the model; it also includes selected comparisons of SST-3 labels with new validation labels.

C.3 Data Collection Pipeline

  • Workers could make up to ten attempts and averaged two attempts before generating an example they claimed fooled the model; each task paid US$0.30.Source: §C.3 · pp. 16–17
  • Workers had to confirm examples when the model predicted incorrectly, and received a US$0.30 bonus for each example later validated as truly fooling the model.Source: §C.3 · p. 17
  • Workers were temporarily disallowed for failing onboarding questions within five attempts or consistently failing to produce validated fooling examples.Source: §C.3 · p. 17
  • Task eligibility required U.S. residence, English proficiency, at least 1,000 completed Mechanical Turk tasks with 98% approval, and no temporary disallowance.Source: §C.3 · p. 17
  • The collection tool was Mephisto, an open-source package adapted for Amazon Mechanical Turk.Source: §C.3 · p. 17

D SST-3 Validation Examples

E Randomly Selected Corpus Examples

The appendix provides randomly selected training examples from both rounds and a sample of SST-3-neutral cases that received a non-neutral validation label.

E Randomly Selected Corpus Examples

Loading 2012.15349v1…