Towards Linguistically Generalizable NLP Systems: A Workshop and Shared Task

1 Introduction

• NLP systems can work extremely well when problems are carefully designed and test data resembles training data.The conditions under which systems work well may be unclear to researchers and the models themselves.Source: §1 · p. 1
• Linguistic knowledge is considered critical in multiple phases of the NLP pipeline.Source: §1 · p. 1

2. Annotation schema design

• System architecture design and feature design are identified as NLP-related design areas.Source: §2 · p. 1

5. Generalization beyond training data

• The workshop and shared task were designed to bring NLP and linguistics researchers together and test system generalization beyond training-data distributions.Source: §5 · p. 1
• The paper describes motivations, design decisions, data, participating systems, scoring, results, and lessons learned.Source: §5 · p. 1
• The authors make available datasets collected for and created during the task to support future shared tasks of this type.Source: §5 · p. 1

2 Motivation: Robust NLP Systems

• Under the iid PAC model, phenomena rare in a corpus may be treated as not worth addressing, and systems are not typically built to handle tail phenomena.The authors add that iid-based learning and simplistic loss functions encourage systems to ignore phenomena they do not adequately capture.Source: §2 · pp. 1–2
• NLP systems are brittle on infrequent phenomena, unlike human language users who make subtle distinctions with little distributional support.The authors give ignoring infrequent negation as a computational example.Source: §2 · p. 2
• Iid development data is unlikely to exhibit every linguistic phenomenon of interest, and released systems may face adversarial testing by users.Source: §2 · p. 2
• Prior work created targeted, annotated evaluations for linguistic phenomena and probed linguistic knowledge in models and their representations.The cited examples include Rimell et al. (2009), Bender et al. (2011), Marelli et al. (2014), Kádár et al. (2016), Li et al. (2015), Adi et al. (2016), and Ettinger et al. (2016).Source: §2 · p. 2
• The workshop builds on that foundation by generating data to identify boundaries of systems’ linguistic capacities and welcoming related research contributions.Source: §2 · p. 2

3 Shared Task: Build It Break It, The Language Edition

• The shared task has building, breaking, and scoring phases; builders are scored on withstanding attacks, while breakers are scored on identifying system boundaries.Source: §3 · p. 2

3.1 Task Selection

• To balance linguistic demands and participation barriers, organizers ran sentiment analysis and QA-SRL, where a sentence-question input maps to an answer span.QA-SRL tests semantic role labeling without predefined roles or highly trained annotators.Source: §3.1 · p. 3

3.2 Building

• Builders received training and development data, could choose whether to use it, and ran their own systems rather than submitting code.Organizers ultimately had builder teams submit predictions in both phases.Source: §3.2 · p. 3

3.3 Breaking

• A pair breaks a system when it succeeds on one item and fails on the other; breakers were asked to make the contrast specific enough to attribute the difference to an identifiable cause.Breakers were encouraged to provide a rationale explaining each change and its implications for assessing linguistic capacity.Source: §3.3 · p. 3
• Breakers worked from provided dataset sentences and made targeted changes, keeping QA-SRL questions unchanged while modifying the sentence and, when appropriate, the answer.Source: §3.3 · pp. 3–4
• The examples illustrate anticipated failures from lexical sentiment cues and pronoun resolution, including a QA-SRL prediction of “they” instead of UCD.The resulting minimal pairs constituted the test set sent to builders for predictions and scoring.Source: §3.3 · p. 4

4.1 Training Data

• Sentiment training comprised 6,921 sentences and 166,738 phrases; neutral sentences were removed, and phrase-level labels were optional for participants.Sentences valued below 0.4 were negative and those above 0.6 were positive.Source: §4.1 · p. 4
• Breaker edits had to preserve the original sentence’s answer as a substring for its accompanying question.Source: §4.1 · p. 4
• QA-SRL training used 5,149 Wikipedia sentences, each paired with a predicate, question, and answer.Source: §4.1 · p. 4

4.2 Blind Development Data

• Blind development data included 500 sentiment sentences and 814 QA-SRL sentences, with initial builder predictions shared with breakers as references.Both sets were annotated by crowdsourcing, following the respective training-data annotation methods.Source: §4.2 · p. 4

4.3 Starter Data for Breakers

• Breakers received an additional 500 sentiment sentences and 814 QA-SRL items, collected and annotated like the corresponding blind-development data.Source: §4.3 · p. 4

4.4 Test Data

• Builder systems were evaluated on breaker-created minimal pairs labeled by breakers, with crowd-sourced labels available to check substantial deviations.Source: §4.4 · p. 4

5.1 Builder Teams: Sentiment

• Sentiment submissions included Strawman, an ensemble of five bag-of-ngrams classifiers, and Melbourne CNNs trained on phrase-level or sentence-level labels.Source: §5.1 · p. 5
• Supplementary sentiment tests used a tree-composing RNTN, a variable-length DCNN, and a linear bag-of-ngrams classifier with n up to 3.Source: §5.1 · p. 5

5.2 Breaker Teams: Sentiment

• Utrecht and Ohio State breakers targeted sentiment weaknesses with combinations of stance, world-knowledge, pragmatic, semantic, morphosyntactic, and syntactic changes.Source: §5.2 · p. 5
• Melbourne breakers used reinforcement learning with reversed labels to select sentiment edits, then applied substitutions with human checks for grammaticality and labeling.Source: §5.2 · p. 5
• Vilnius breakers fine-tuned sequence-to-sequence models on BIBI data and manually selected 100 changed sentences deemed meaningful for submission.Their two-stage training began by teaching the model to replicate movie-review sentences.Source: §5.2 · pp. 5–6

5.3 Builder Team: QA-SRL

• With no external QA-SRL builder submission, organizers supplied a logistic-regression system using skipgrams and dependency-based candidate answers and features.Source: §5.3 · p. 6

5.4 Breaker Team: QA-SRL

• The UMass QA-SRL breaker team used linguistically motivated edits including coordination, preposition changes, word-sense shifts, and word-order changes.Source: §5.4 · p. 6

• A minimal pair broke a system when it received one correct and one incorrect prediction, rewarding breakers for targeting system boundaries.Source: §6 · p. 6
• Breaker scores averaged each system’s breaking percentage weighted by its development-set accuracy, reflecting the system’s general strength.Source: §6 · p. 6
• Builder scoring used average F score across original and modified sentences, plus the percentage of sentence pairs that broke the system.Source: §6 · p. 6

7.1 Aggregate Results

• 0.528 average F1 puts Strawman first among builders, while the phrase-based CNN leads on breaking percentage at 24.39% of test pairs.The bag-of-ngrams model and DCNN follow the phrase-based CNN on breaking percentage; the sentence-based CNN ranks last by a fair margin.Source: Table 1 · p. 6, §7.1 · p. 7
• Utrecht ranks first among breaker teams by the chosen scoring metric, followed closely by OSU.The table reports aggregate breaker scores; aggregate results do not show which weaknesses particular minimal pairs target.Source: Table 2 · p. 6, §7.1 · p. 7, §7.2 · p. 7

7.2 Detailed Results

• Utrecht and OSU perform roughly on par across systems, with Utrecht’s largest margin over OSU on Strawman.Their comparable performance may reflect a comparable variety of linguistically diverse, targeted attacks.Source: §7.2 · p. 7
• Melbourne’s automatically generated test set breaks most systems less often than the other sets, but performs on par against the sentence-based CNN.VTeX has the lowest overall breaking percentages by a substantial margin, and is especially weak against the sentence-based CNN compared with other breaker teams.Source: §7.2 · p. 7

7.3 Item-based Results

• Most successful test pairs break systems together, but targeted changes can support more specific conclusions than diffuse ones.The OSU substitution involving “just willing enough” supports a conclusion about systems’ difficulty inferring the phrase’s positive effect in a previously negative context; Melbourne’s word-reordering pair makes the precise cause harder to determine.Source: §7.3 · pp. 7–8
• Adding “pain” breaks Strawman and bag-of-ngrams, while most other systems handle the change; substituting “unnerving” and “hilarious” breaks four systems in different directions.RNTN predicts both items incorrectly in the first pair; on the second, Strawman and PCNN predict both items correctly.Source: §7.3 · pp. 7–8
• Melbourne’s “thrill” pair breaks Strawman, and its word-reordering pair troubles every system in some way; the VTeX pair fools bag-of-ngrams on the altered sentence.On Melbourne’s word-reordering pair, bag-of-ngrams and RNTN fail on both items; on VTeX’s pair, DCNN and RNTN err on the original.Source: §7.3 · p. 8
• More confident conclusions would require testing system predictions on multiple pairs targeting the same linguistic phenomenon.The authors identify this as a goal for future iterations and analyses.Source: §7.3 · p. 8

8 Lessons for the Future

• QA-SRL drew less participation than sentiment analysis, making task choice and strategies such as adding a breaking phase to existing shared tasks important.The authors also call for strategies that encourage buy-in from both builders and breakers.Source: §8 · p. 8
• Questionable breaker labels can skew builder-performance assessment, so future iterations should validate labels externally, potentially only for contested examples.The proposed restriction to contested examples is intended to minimize cost and time.Source: §8 · p. 8
• Some breaker submissions changed sentences substantially, motivating clearer minimal-pair definitions and external review of whether submitted pairs are permissible.The authors identify pair conformity as important to this task.Source: §8 · p. 8
• Allowing breakers to create domain-appropriate sentences from scratch, alongside edits to existing data, may broaden the phenomena they target and improve targeting precision.The current task required breakers to draw from existing data.Source: §8 · p. 8
• A longer schedule could make participation more feasible, particularly for builders who had to run systems and submit predictions in two phases.The authors describe the task timeline as shorter than optimal and burdensome for builders.Source: §8 · p. 10
• Anonymous system submissions are proposed as a way to address participants’ possible reluctance to have their work broken.The authors identify this as a potential future change.Source: §8 · p. 10

9 Conclusion

• The first workshop and shared task began exploring NLP systems’ limits with respect to specific linguistic phenomena.The authors describe this as an initial exploration.Source: §9 · p. 10
• The first iteration also yielded lessons intended to inform future shared-task iterations and related tasks.The paper reports describing the task, its results, and lessons for future work.Source: §9 · p. 10
• The authors express confidence that testing linguistic phenomena beyond training-data distributions can contribute to NLP systems’ robustness and quality.This is presented as the potential contribution of tasks emphasizing such testing.Source: §9 · p. 10