PaperlayerView exact revision on arXiv

Source-linked AI summary

Evaluating Creative Short Story Generation in Humans and Large Language Models

Mete Ismayilzada, Claire Stevenson, Lonneke van der Plas

arXiv:2411.02316v5cs.CLcs.AI

TL;DR

Whether LLMs can write genuinely creative stories remains underexplored, particularly for longer, more complex responses. This study compares 60 people and 60 LLMs on cue-word story writing and gathers automated and human/LLM judgments. LLM stories are more linguistically complex, but humans score higher on novelty, surprise, and diversity, while non-experts and LLM judges often favor LLM stories.

  • Problem

    LLM story creativity remains understudied, and it is unclear whether findings from short-response creativity tasks generalize to longer, more complex responses.

    Source: Introduction · p. 1

  • Method

    The study compares 60 humans and 60 LLMs writing roughly five-sentence stories from three cue words, using automated creativity metrics and expert, non-expert, and LLM judgments.

    Source: Introduction · p. 2, Story Generation Data Collection · p. 3

  • Results

    LLM stories are more linguistically complex, while human stories show greater novelty, surprise, and diversity; experts align with automated metrics, but non-experts and LLM judges rate LLM stories as more creative.

    Source: Introduction · p. 2, Discussion · p. 8

  • Takeaways & Limitations

    Expert judgments track diversity and surprise, whereas non-expert and LLM judgments correlate with lexical complexity, which need not reflect semantic complexity.

    Source: Discussion · p. 8

    The study does not test alternative decoding or prompting strategies, and its reference-based novelty scores depend heavily on the selected story corpus.

    Source: Limitations · p. 9

Abstract

from arXiv · show

Story-writing is a fundamental aspect of human imagination, relying heavily on creativity to produce narratives that are novel, effective, and surprising. While large language models (LLMs) have demonstrated the ability to generate high-quality stories, their creative story-writing capabilities remain under-explored. In this work, we conduct a systematic analysis of creativity in short story generation across 60 LLMs and 60 people using a five-sentence cue-word-based creative story-writing task. We use measures to automatically evaluate model- and human-generated stories across several dimensions of creativity, including novelty, surprise, diversity, and linguistic complexity. We also collect creativity ratings and Turing Test classifications from non-expert and expert human raters and LLMs. Automated metrics show that LLMs generate stylistically complex stories, but tend to fall short in terms of novelty, surprise and diversity when compared to average human writers. Expert ratings generally coincide with automated metrics. However, LLMs and non-experts rate LLM stories to be more creative than human-generated stories. We discuss why and how these differences in ratings occur, and their implications for both human and artificial creativity.

Introduction

Whether LLMs can produce genuinely creative stories remains underexplored, especially for longer responses than the short tasks used in much prior creativity research. This study compares people and LLMs on cue-word story writing and finds differences in both automated creativity metrics and judgments by experts, non-experts, and LLMs.

Introduction

  • The creative story-writing capability of LLMs remains understudied, despite models producing high-quality stories.The paper frames creativity as involving novelty, usefulness, and surprise.Source: Introduction · p. 1
  • Prior LLM creativity evaluations often use short-response tasks, leaving their results’ generalizability to longer, more complex responses underexplored.The cited examples include tasks asking people and models to generate creative uses for everyday objects.Source: Introduction · p. 1
  • LLM stories show greater linguistic complexity, but fall short of average human stories in novelty, diversity, and surprise.The study evaluates 60 humans and 60 instruction-finetuned LLMs using a roughly five-sentence story task based on three cue words.Source: Introduction · p. 2
  • Non-experts and LLMs rate LLM-generated stories as more creative, while expert judgments correlate positively with automated metric results.The study collects fine-grained creativity judgments from expert and non-expert human raters and LLMs.Source: Introduction · p. 2

Related Work

Prior creativity evaluation includes expert judgments, theoretical frameworks, and brief psychometric tasks, while story-generation research has compared LLMs with professional writers and average humans. This study’s closest predecessor used one model and non-expert ratings; the present work broadens the comparison to populations of models and people, multiple creativity dimensions, and additional judge types.

Creativity Evaluation

  • Creativity Evaluation commonly uses the Consensual Assessment Technique, which relies on collective judgments from human experts.The level of expertise required is debated, with most evidence favoring quasi-experts or experts over non-experts as raters.Source: Related Work · p. 2
  • Prior evaluations include theoretical frameworks and brief psychometric tests, while this work studies creativity in longer story-generation responses.The Alternative Uses Test asks participants to generate creative uses for everyday objects such as a brick.Source: Related Work · p. 2

Creative Story Evaluation

  • Earlier creative-story studies compared models with award-winning or experienced writers, whereas this work focuses on average human writers.Chakrabarty et al. (2023) found LLMs fell short of experienced writers; Tian et al. (2024a) found LLM stories lacked diversity, suspense, and tension.Source: Related Work · p. 2
  • Compared with Orwig et al. (2024), this study evaluates 60 distinct LLMs, multiple creativity dimensions, expert and non-expert humans, and three LLM judges.Orwig et al. (2024) compared humans with one model, varied its temperature, and collected non-expert and GPT-4 ratings.Source: Related Work · p. 2

Methods

The study collected short stories from humans and diverse LLMs using three-word prompts, then evaluated them with automated creativity metrics and human and LLM judges. The evaluation covered novelty, surprise, diversity, linguistic complexity, and judged creativity.

Story Generation Data Collection

  • Four cue-word sets, 60 models, and 60 people supplied stories; after filtering out stories with fewer than 3 or more than 7 sentences, 431 remained for evaluation.The cue sets varied in semantic distance, and models were prompted zero-shot with temperature 0.7 and top_p 0.95.Source: Story Generation Data Collection · p. 3

Story Evaluation by Automated Metrics

  • Automated measures assessed novelty and surprise through semantic distance, diversity through lexical and semantic measures, and linguistic complexity through lexical and syntactic measures.Complexity measures included unique-word count, average word and sentence length, readability, part-of-speech ratios, dependency-path length, and constituency-tree depth.Source: Story Evaluation by Automated Metrics · pp. 3–5

Story Evaluation by Judges

  • Two blinded expert judges rated all 431 stories, 96 non-experts rated a subset of 273, and Claude, Gemini, and GPT-4 rated all stories.Experts rated creativity, originality, surprise, and effectiveness and classified stories as human- or AI-created; non-expert ratings used the median, while expert and LLM ratings used means for creativity-related measures.Source: Story Evaluation by Judges · p. 5

Results

Human stories outperformed models in novelty, surprise, and diversity, while models produced more complex language. Creativity ratings and author-identification accuracy differed across expert, non-expert, and LLM judges.

Results of Story Evaluation by Automated Metrics

Results of Story Evaluation by Judges

Which automated metrics predict creativity evaluation by the three groups of judges?

  • Expert ratings rose with semantic diversity, surprise, and lexical diversity, while non-expert and LLM ratings favored lexical complexity and were negatively related to semantic diversity, novelty, and surprise.Source: Which automated metrics predict… · p. 7

Discussion

The discussion contrasts models’ linguistic complexity with humans’ stronger creativity-related dimensions, and examines why judges evaluate the stories differently. It connects these differences to evaluation practice and possible directions for improving LLM creativity.

  • The study compares five-sentence cue-word stories from humans and LLMs using automated creativity measures and judgments from expert, non-expert, and LLM raters.Source: Discussion · pp. 7–8
  • Automated metrics showed greater linguistic complexity in LLM stories but greater human novelty, surprise, and lexical and semantic diversity, revealing gaps across major creativity dimensions.Source: Discussion · p. 8
  • Expert judgments correlated with automated metrics, but non-experts and LLMs rated LLM stories more creatively, reflecting different criteria for creativity.Source: Discussion · p. 8
  • Experts focused on diversity and surprise, whereas non-experts and LLMs tracked lexical complexity, which does not necessarily imply semantic complexity.Source: Discussion · p. 8
  • The authors suggest complexity may create an illusion of creativity for untrained raters and call for more comprehensive creativity evaluation.Source: Discussion · pp. 8–9

Limitations

The study’s conclusions are bounded by its decoding and prompting setup, reference-based novelty scoring, rater counts, and the scope of its creativity metrics.

Limitations

  • Alternative decoding and prompting strategies were not explored because evaluating open-ended outputs across many models was costly.The study used a setup from prior work intended to favor creative output and idea diversity.Source: Limitations · p. 9
  • Novelty scores depend heavily on the chosen reference corpus, which comprised human- and AI-written stories for each item set.The authors suggest that future studies could validate findings by including more stories in reference-based metrics.Source: Limitations · p. 9
  • The study used two (quasi-)expert and five non-expert raters per story; more raters or professional creative writers could improve reliability.The authors note that quasi-experts may be as reliable as experts, or sometimes more so.Source: Limitations · p. 9
  • The creativity metrics cannot fully capture broader cultural and social dimensions of creativity or the depth of truly original, imaginative language use.Source: Limitations · p. 9

Ethics

The paper reports ethics approval, public study materials, and declarations concerning conflicts of interest and AI use.

Ethics

  • The authors declare no conflicts of interest and no use of AI-assisted technologies in the research or article creation.Source: Ethics · p. 9
  • A local ethics board approved the research on September 11, 2024 (ID: FMG-10319).Source: Ethics · p. 9
  • All study materials are publicly available.Source: Ethics · p. 9

Related papers

Loading 2411.02316v5…