PaperlayerView exact revision on arXiv

Source-linked AI summary

The Natural Stories Corpus

Richard Futrell, Edward Gibson, Hal Tily, Idan Blank, Anastasia Vishnevetsky, Steven T. Piantadosi, Evelina Fedorenko

arXiv:1708.05763v1cs.CL

TL;DR

Naturalistic corpora often omit rare syntactic constructions that can distinguish language-processing theories. Natural Stories provides fluent edited narratives containing such constructions, with parses and self-paced reading data. The reading times show expected effects of frequency, trigram probability, and word length.

  • Problem

    Naturalistic corpora are unlikely to contain many low-frequency syntactic constructions used to distinguish processing theories.

    Source: §1 · p. 2

  • Method

    The authors created fluent edited English narratives with low-frequency constructions and released corrected parses, self-paced reading times, and word-aligned audio.

    Source: §1 · pp. 2–3

  • Results

    Increased frequency and trigram probability lead to faster reading times, while increased word length leads to slower reading times.

    Source: §3.3.2 · p. 9

  • Takeaways & Limitations

    The corpus is intended to help form a suite of test sets exposing models’ behavior on uncommon constructions beyond fully naturalistic corpora.

    Source: §4 · p. 9

Abstract

from arXiv · show

It is now a common practice to compare models of human language processing by predicting participant reactions (such as reading times) to corpora consisting of rich naturalistic linguistic materials. However, many of the corpora used in these studies are based on naturalistic text and thus do not contain many of the low-frequency syntactic constructions that are often required to distinguish processing theories. Here we describe a new corpus consisting of English texts edited to contain many low-frequency syntactic constructions while still sounding fluent to native speakers. The corpus is annotated with hand-corrected parse trees and includes self-paced reading time data. Here we give an overview of the content of the corpus and release the data.

1 Introduction

Naturalistic corpora support broad model comparisons, but often omit the rare constructions that distinguish theories of language processing. The Natural Stories Corpus combines broad-coverage texts with targeted, fluent constructions and releases parses, reading times, and aligned audio.

1 Introduction

  • Naturalistic corpora often omit rare constructions that can distinguish processing theories, while hand-crafted experiments make broad coverage comparisons difficult.Standardized-text comparisons make it easier to evaluate model coverage across phenomena.Source: §1 · p. 2
  • The Natural Stories Corpus combines targeted, low-frequency syntactic structures with broad-coverage comparability across models in fluent English narratives.The texts are designed to remain fluent and coherent while including psycholinguistically interesting constructions.Source: §1 · p. 2
  • The release includes hand-corrected Penn Treebank-style phrase parses, automatically converted Universal Dependencies parses, self-paced reading times, and word-aligned audio.Source: §1 · pp. 2–3

2 Related Work

The paper situates Natural Stories among corpora that offer different combinations of naturalistic text, varied syntax, narrative context, and psychometric measurements. English and German resources provide contrasts in corpus scale and whether materials form connected narratives.

2 Related Work

  • The Dundee Corpus provides a prominent English naturalistic-text comparison, with newspaper editorials and eyetracking data from 10 participants.It contains 51,501 word tokens in 2,368 sentences.Source: §2 · p. 3
  • The Potsdam Sentence Corpus was designed for varied syntactic structures, but its German sentences are isolated and presented in random order.It includes cloze probabilities and eyetracking data for each word.Source: §2 · p. 3
  • The MIT Corpus shares Natural Stories’ goal of collecting data on structurally varied sentences, using four narratives with reading-time and fMRI data (Bachrach et al., ms).Source: §2 · p. 3
  • The UCL Corpus offers typical English narrative sentences with self-paced reading and eyetracking, but uses isolated sentences rather than connected narratives.Its aim complements Natural Stories’ focus on low-frequency constructions.Source: §2 · pp. 3–4

3 Corpus Description

Natural Stories comprises edited English narratives designed to include low-frequency constructions, with corrected parses and self-paced reading data. The corpus includes marked constructions more often than a Dundee sample, and its reading-time data show expected effects of frequency, probability, and word length.

3.1 Text

  • The corpus contains 10 edited stories with 10,245 lexical word tokens and 485 sentences, designed to include many low-frequency syntactic constructions.Mean sentence length is 21.1 lexical words, around the Dundee corpus mean of 21.7.Source: §3.1 · pp. 4–5

3.2 Parses

  • The texts were parsed automatically and hand-corrected; the release includes Penn Treebank-style phrase structure trees and automatically converted Universal Dependencies parses.Source: §3.2 · p. 7

3.3 Self-Paced Reading Data

  • Self-paced reading data came from 181 native English speakers; passes with fewer than five comprehension answers correct and reaction times outside 100–3000 ms were excluded.Each story had six comprehension questions.Source: §3.3 · p. 7

3.3.1 Inter-Subject Correlations

  • Inter-subject reliability was assessed by correlating each participant’s story-level reaction times with the average from all other participants.This produces one ISC statistic per participant for each story.Source: §3.3.1 · p. 8, Figure 4 · p. 8

3.3.2 Psycholinguistic Sanity Checks

  • Increased frequency and trigram probability both lead to faster reading times, while greater word length leads to slower reading times.The authors fit separate mixed-effects regression models for the three predictors.Source: §3.3.2 · pp. 8–9, Table 3 · p. 9

3.4 Syntactic Constructions

  • Natural Stories has especially high rates of nonlocal VP conjunction, nonrestrictive SRCs, idioms, adjective conjunction, noncanonical ORCs, local NP/S ambiguities, and it-clefts relative to a Dundee sample.The authors coded 200 randomly selected Dundee sentences for comparison.Source: §3.4 · p. 9

4 Conclusion

The Natural Stories Corpus combines fluent edited English with rare or difficult constructions, providing a test set that fully naturalistic corpora cannot. Figure 5 compares marked-construction rates in Natural Stories and sampled Dundee sentences.

4 Conclusion

  • The Natural Stories Corpus contains fluent edited English designed to include many rare or hard-to-process constructions.The authors see it as part of a suite of test sets that can expose model behavior on uncommon constructions unavailable in fully naturalistic corpora.Source: §4 · p. 9
  • The authors hope the corpus will support further data collection and annotation.Source: §4 · p. 9
  • Figure 5 reports rates of marked constructions in the Natural Stories corpus and in 200 randomly sampled sentences from the Dundee corpus.Source: Figure 5 · p. 10

A Features coded for Section 3.4

The coded features cover local ambiguities, dependencies, questions, idioms, quotation, conjunction, and relative-clause types; coding was performed by Suniyya Waraich, Edward Gibson, and Richard Futrell.

A Features coded for Section 3.4

Loading 1708.05763v1…