Source-linked AI summary
The Effect of Different Writing Tasks on Linguistic Style: A Case Study of the ROC Story Cloze Task
Roy Schwartz, Maarten Sap, Ioannis Konstas, Li Zilles, Yejin Choi, Noah A. Smith
TL;DR
The paper examines whether writing-task constraints affect style, using original stories and coherent or incoherent endings as a case study. It finds measurable stylistic differences across these conditions, and a model combining style features with story context achieves a new state of the art on the story cloze task.
Problem
The paper asks whether writing tasks, in addition to personal traits and mental states, affect linguistic style, a question relevant to understanding variation in written language.
Source: §1 · p. 1Method
Results
Takeaways & Limitations
The findings suggest that task framing can affect how people write, and that author instructions merit attention when designing NLP tasks.
Source: §1 · p. 2, §7 · p. 7Further investigation is required to validate the hypothesis that the task-style link involves differences in authors’ mental states.
Source: §7 · p. 7
Abstract
from arXiv · showhide
A writer's style depends not just on personal traits but also on her intent and mental state. In this paper, we show how variants of the same writing task can lead to measurable differences in writing style. We present a case study based on the story cloze task (Mostafazadeh et al., 2016a), where annotators were assigned similar writing tasks with different constraints: (1) writing an entire story, (2) adding a story ending for a given story context, and (3) adding an incoherent ending to a story. We show that a simple linear classifier informed by stylistic features is able to successfully distinguish among the three cases, without even looking at the story context. In addition, combining our stylistic features with language model predictions reaches state of the art performance on the story cloze challenge. Our results demonstrate that different task framings can dramatically affect the way people write.
1 Introduction
This study asks whether writing-task constraints affect style, using the story cloze task’s original, coherent-ending, and incoherent-ending writing conditions. It finds measurable style differences and reports that combining stylistic features with context reaches a new state of the art on the story cloze challenge.
1 Introduction
- A linear classifier using stylistic features distinguishes original, coherent, and incoherent endings to a large degree without story context, with 64.5–75.6% binary classification results.The three conditions arise from writing an entire story, adding a coherent ending to a given context, and adding an incoherent ending.Source: §1 · pp. 1–2
- People use different styles for coherent versus incoherent endings and for writing an entire story versus completing another author’s context.The authors suggest these differences may relate to cognitive load and mental state, while noting that the mental-state hypothesis needs further investigation.Source: §1 · p. 2
- 75.2% accuracy, an additional 2.8% gain, gives the final model combining story context and stylistic features a new state-of-the-art result.The stylistic-feature classifier alone reaches 72.4%, 2.3% below the prior state of the art.Source: §1 · p. 2
- The study argues that NLP task designers should attend to author instructions and reports a new state-of-the-art result on the commonsense story cloze challenge.The authors connect this implication with similar findings by Cai et al. (2017).Source: §1 · p. 2
2 Background: The Story Cloze Task
The story cloze task uses original stories and alternative endings to evaluate commonsense story understanding. Its construction creates distinct writing conditions: original stories are written as a whole, while right and wrong endings are generated for a supplied context.
2 Background: The Story Cloze Task
- The story corpus contains 49,255 five-sentence stories collected on Mechanical Turk, whose workers were asked to write coherent, self-contained stories without a prescribed topic.The task’s alternative endings were rated as parts of whole stories; only stories meeting the specified coherence and neutrality criteria were retained.Source: §2 · p. 2
- The official task asks a system to choose the right ending from two stories that differ only in their endings; training contains original stories, while development and test use revised stories for different original stories.The task was proposed as an evaluation framework for commonsense story understanding, among other uses.Source: §2 · p. 2
- The benchmark was challenging: its published score first exceeded 60% one year after introduction, while nine earlier surface-level and narrative-informed baselines reached roughly chance level.The authors report that their style-feature classifier reaches 72.4% on the task.Source: §2 · pp. 2–3
- Original endings are part of coherent five-sentence stories written by one author, whereas right endings are written after an author reads four sentences and supplies a fifth.The training set therefore contains only positive examples, with original endings generated differently from development and test right endings.Source: §2 · p. 3
- The same author writes both right and wrong endings, but is instructed to make the former coherent and the latter incoherent with the preceding sentences..Source: §2 · p. 3
3 Surface Analysis of the Story Cloze Task
Surface comparisons show systematic differences among original, right, and wrong endings in sentence length, frequent parts of speech, and word use.
3 Surface Analysis of the Story Cloze Task
- Original endings average 11 words per sentence, compared with 8.75 for right endings and 8.47 for wrong endings.Right endings are slightly longer than wrong endings.Source: §3 · p. 3
- Original and right endings use pronouns more frequently than wrong endings.The authors note that fewer pronouns have also been associated with deceptive text.Source: §3 · p. 3
- Original endings use “and” more than right endings, and substantially more than wrong ones; original and right endings favor “!” while wrong endings tend toward words such as “hates.”The authors next test whether these observed differences can distinguish the ending types.Source: §3 · p. 3
4 Model
The model tests whether writing constraints affect style by classifying individual endings from their linguistic features, without using the preceding story context.
4 Model
- The model is designed to measure whether different writing constraints are reflected in style using simple methods for style recognition..Source: §4 · p. 3
- A logistic regression classifies individual endings as right versus wrong or original versus new (right), using words from the ending alone and not earlier story text.The classifier treats each ending as an instance rather than classifying paired endings.Source: §4 · p. 3
- The feature set includes sentence length, word n-grams of one to five words, and character 4-grams.Content words in the word n-grams are replaced with their part-of-speech tags to distinguish high- and low-frequency words.Source: §4 · p. 4
5 Experiments
The experiments test whether stylistic features distinguish endings written under different task constraints, including when story context is ignored. On the story cloze test, combining stylistic features with a neural language model yields a new state-of-the-art result.
5 Experiments
- The experiments compare right with wrong endings, original with right endings, and original with wrong endings.For right-versus-wrong classification, endings are classified individually rather than as coupled pairs; original-versus-right results average five classifiers trained on balanced samples.Source: §5 · p. 4
5.1 Results
- Across all three experiments, the context-ignoring model performed above the 50% chance baseline, indicating that writing task affected author style.Because the classifier ignores the preceding four sentences, it cannot capture story coherence.Source: §5.1 · p. 5, Table 2 · p. 5
5.2 Story Cloze Task
- 72.4% accuracy, 2.3% below state of the art, was achieved by the stylistic classifier on the story cloze test without story-corpus training data or story context.The story cloze task presents two endings and asks which is right; the classifier adapts its individual ending predictions to make that choice.Source: §5.2 · p. 5, Table 3 · p. 5
- 67.7% accuracy came from the language-model score ratio, compared with 55% from scoring endings by story-conditioned probability alone.The ratio is intended to capture how unsurprising an ending is given the preceding story while controlling for the ending's inherent surprisingness.Source: §5.2 · p. 6
- 75.2% accuracy, a new state of the art, followed from combining stylistic features with neural language model predictions, a 2.8% absolute gain.The combined model adds the language model's numerator, denominator, and ratio features to the linear classifier.Source: Table 3 · p. 5, §5.2 · p. 6
6 Further Analysis
The further analysis examines which feature types and individual stylistic features distinguish writing tasks. It reports patterns in endings and a safeguard for identifying salient features.
6.1 Most Discriminative Feature Types
- Character n-grams came within 0.6% of the full model, while word n-grams yielded 61.2%, 3.3% worse than the full model.Table 4 reports Experiment 1 results for different feature subsets.Source: §6.1 · p. 6, Table 4 · p. 6
6.2 Most Salient Features
- To identify salient features, the authors examined extreme logistic-regression coefficients and retained only features occurring in at least 5% of training endings.Large coefficients can reflect rare features, so the frequency threshold mitigates that concern.Source: §6.2 · p. 6
- In Experiment 1, incoherent endings more often began with a proper noun and ended with a common noun, while coherent endings more often ended with a past-tense verb.Table 5a presents the salient features and their corpus frequencies.Source: §6.2 · p. 6, Table 5 · p. 7
- Coherent endings used more coordination, adjectives, and infinitives, whereas incoherent endings more often used gerunds.The authors suggest the adjective pattern may reflect more descriptive writing for coherent stories.Source: §6.2 · p. 6
- For original endings in Experiment 2, greater sentence length was the most salient feature, and exclamation marks were a strong indicator.The authors suggest exclamation marks may reflect greater enthusiasm when authors write their own text.Source: §6.2 · p. 6, Table 5 · p. 7
- Features associated with incoherent endings in Experiment 1 also predicted new coherent endings in Experiment 2, helping explain why original-versus-incoherent classification was easier.For example, beginning with a proper noun was more common in incoherent than coherent endings, and more common in coherent endings than original stories.Source: §6.2 · p. 7
7 Discussion
The discussion interprets the observed style differences as potentially related to authors’ mental states and draws implications for dataset instructions. It also notes that the mental-state link needs further validation.
The effect of writing tasks on mental states
- Authors used different styles for coherent versus incoherent endings and for writing an entire story versus completing another author’s story.The paper suggests the task may impose a different mental state, expressed in ways simple automatic tools can detect.Source: §7 · p. 7
- The proposed connection between writing tasks, mental states, and style remains a hypothesis requiring further investigation.The discussion situates it alongside prior work linking writing tasks and mental states to style variation.Source: §7 · p. 7
Design of NLP tasks
- Although the story cloze task controlled topic, temporal and causal diversity, and author effects, significant style differences remained across its training and test sets and labels.The task design is described as carefully controlled, but the passage reports residual style differences.Source: §7 · p. 8
- The authors recommend careful author instructions, especially for unnatural tasks; COPA instructions asked annotators to be brief and avoid proper names and slang.They also suggest rating naturally occurring sentences rather than asking annotators to generate text.Source: §7 · p. 8
- The story-cloze classifier reached 53.2% accuracy on COPA, close to random; the smaller dataset may partly explain this result.COPA had 1,000 examples, compared with 3,742 for story cloze.Source: §7 · p. 8
8 Related Work
The related work situates the study within stylometry and machine-reading research. It presents task-induced style variation as an additional methodological challenge for machine-reading datasets.
Writing style
- Prior stylometry research relates writing style to age and gender, personality (Ireland and Mehl, 2014), and other individual differences.The cited work also examines native language and individual-specific styles.Source: §8 · p. 8
- Earlier studies used stylistic features to detect deception, while this paper examines style differences between coherent and incoherent story endings.The paper characterizes this as a subtler writing-task contrast.Source: §8 · p. 8
Machine reading
- Machine-reading datasets can have methodological problems from noisy automatic generation tools; this study identifies task-induced style differences as another potential confound.The discussion cites prior dataset-generation concerns (Chen et al., 2016).Source: §8 · p. 8
9 Conclusion
The conclusion reports measurable stylistic differences between writing tasks in the story cloze setting. It connects those findings to the final model’s state-of-the-art result and to implications for cognitive research and dataset design.
9 Conclusion
- A simple linear model revealed measurable style differences both between writing an original story and continuing someone else’s, and between coherent and incoherent endings.Source: §9 · pp. 8–9
- Those style differences, combined with story context, allowed the final model to achieve state-of-the-art results on story cloze.Source: §9 · p. 9
- The findings motivate further study of how writing prompts affect authors’ mental states and responses, and offer lessons for NLP dataset design.Source: §9 · p. 9