Source-linked AI summary
Creative Writing with an AI-Powered Writing Assistant: Perspectives from Professional Writers
Daphne Ippolito, Ann Yuan, Andy Coenen, Sehmon Burnam
TL;DR
Prior creative-writing AI research largely relied on amateur writers and narrow, contrived evaluations. This paper studies 13 diverse professional writers using Wordcraft over eight weeks, finding promise for brainstorming and story development but persistent difficulties with voice, control, and story understanding. It concludes that writing assistants should account for writers’ diverse goals and expertise.
Problem
Prior creative-writing AI studies mostly evaluated systems with amateurs or crowd workers on narrowly defined tasks, leaving professional writers’ perspectives underexamined.
Method
The study commissioned 13 published writers from diverse backgrounds to use Wordcraft, an NLG-powered text editor and chatbot, over an extended writing period.
Results
Participants saw potential for brainstorming, story-detail generation, world-building, and research assistance, but reported weak voice, uninteresting suggestions, and difficult control.
Takeaways & Limitations
AI writing assistants may make parts of creative writing easier, faster, and more enjoyable when development involves the audiences and writers who will use them.
Takeaways & Limitations
The small study could not disentangle weaknesses of Wordcraft’s interface and implementation from limitations of its underlying language model.
Abstract
from arXiv · showhide
Recent developments in natural language generation (NLG) using neural language models have brought us closer than ever to the goal of building AI-powered creative writing tools. However, most prior work on human-AI collaboration in the creative writing domain has evaluated new systems with amateur writers, typically in contrived user studies of limited scope. In this work, we commissioned 13 professional, published writers from a diverse set of creative writing backgrounds to craft stories using Wordcraft, a text editor with built-in AI-powered writing assistance tools. Using interviews and participant journals, we discuss the potential of NLG to have significant impact in the creative writing domain--especially with respect to brainstorming, generation of story details, world-building, and research assistance. Experienced writers, more so than amateurs, typically have well-developed systems and methodologies for writing, as well as distinctive voices and target audiences. Our work highlights the challenges in building for these writers; NLG technologies struggle to preserve style and authorial voice, and they lack deep understanding of story contents. In order for AI-powered writing assistants to realize their full potential, it is essential that they take into account the diverse goals and expertise of human writers.
1 Introduction
This paper examines how professional writers use AI-powered creative-writing assistance, addressing gaps in prior evaluations with amateurs and narrow tasks. It finds promise for brainstorming and story development, alongside challenges involving voice, control, and story understanding.
- Study motivation and contribution: Professional writers were targeted because they possess established writing processes and distinctive voices, making them well positioned to assess integration into creative workflows.
- Study motivation and contribution: The study commissioned 13 published writers from diverse backgrounds to use Wordcraft over eight weeks within their own writing processes.Participants provided stories, journals, and interview feedback.
- Wordcraft: Wordcraft combined a text editor with NLG-powered controls and a conversational chatbot supported by LaMDA.
- Study motivation and contribution: Participants saw Wordcraft as useful for brainstorming, generating story details, world-building, and research assistance, with the interface considered as important as the language model.
- Challenges: Participants reported that generated suggestions often lacked distinctive voice, were uninteresting, and were difficult to control for specific writing tasks.
2 Related Work
Prior creative-writing AI research has used classical story-generation methods and newer neural language models, but evaluations have usually been short, narrow, and conducted with amateurs. This work extends that literature by studying professional writers over eight weeks.
- Earlier systems: Earlier story-generation systems used symbolic planning and graph traversal, often allowing users to specify initial goals and conditions.
- Neural language models: Recent creative-writing tools commonly have language models append content to the story, although this turn-taking paradigm may not match human writers’ workflows.
- Evaluation gaps: Most prior user studies evaluated narrow functionality in contrived settings with amateur writers, while comparable professional-writer studies lasted under two hours.
3 The Wordcraft Tool
Wordcraft is a text editor that uses LaMDA to generate suggestions from users’ stories through editing controls and a chatbot. Its controls rely on in-context prompts containing examples of the requested task.
- Wordcraft architecture: Wordcraft provides NLG-powered controls that generate suggestions based on the text in the editor.
- Wordcraft architecture: Users can choose among several suggestions, insert selected outputs into the main text, and save suggestions for later use.
- Prompting: Wordcraft uses in-context learning by prompting LaMDA with conversational examples of the requested task and the user’s story or selected text.
- Wordcraft architecture: The system supports editing operations such as elaboration and rewriting in addition to story continuation, alongside a conversational chatbot.
- Prompting: LaMDA performed all Wordcraft generation, and the authors expected the findings to be broadly relevant to other similarly sized language-model families.
4 Methods
The authors studied how 13 published writers with diverse backgrounds used Wordcraft to create stories. The eight-week, minimally guided study combined participant stories, journals, and interviews, which were coded qualitatively.
- Study Design: The study recruited 13 published writers with diverse writing, personal, and technology backgrounds.
- Study Design: Participants wrote 1,000–1,500-word stories with Wordcraft without an initial tutorial, documenting noteworthy generations, frustrations, useful capabilities, and imagined workflow changes.
- Study Design: Researchers conducted two 45-minute interviews with each participant, before and after the workshop, and collected the completed story and journal.
- Analysis: Two authors coded journals and interview notes, resolving disagreements through discussion across codes covering use cases, workflows, system models, interface, controls, and chatbot feedback.
5 Initial Hopes and Expectations
Participants wanted AI assistance that enhanced, rather than replaced, their creative process by expanding ideas, supporting world-building, and facilitating access to information. They also hoped to probe how well the technology could represent marginalized and under-represented perspectives.
- Participants primarily wanted brainstorming support that enhanced their own thoughts and ideas without offloading the creative process.
- Writers sought help expanding existing ideas through character details, story-arc refinement, world-building, rewrites, humor, and stylistic analysis.
- Participants hoped to use language models to access information, leverage large bodies of writing, and connect notes or books across sources.
- Nearly all participants wanted to test the model’s ability to represent marginalized and under-represented ideas and groups.
6 Emergent Workflows
Through prolonged, relatively unconstrained exploration, participants developed diverse workflows for brainstorming, world-building, research, editing, and collaboration. Productive use often required learning Wordcraft’s specialized abilities, adapting to its limitations, and curating unexpected suggestions.
- Participants developed emergent workflows beyond the tool’s explicitly designed tasks, making the interface itself part of the technology.
- 6.1 Idea Generation and Brainstorming: Wordcraft supported brainstorming from minimal seed text, conversational plot and world-building discussions, and generation of story details without requiring exact suggestions to be incorporated.
- 6.3 Wordcraft as Improv Partner: As co-writer, Wordcraft supplied suggestions that writers curated and edited, but productive collaboration required relinquishing control and accepting disconnected ideas.
- 6.3 Wordcraft as Improv Partner: Some participants abandoned initial story directions and tailored their writing around Wordcraft’s affordances, producing stories they found more interesting, original, and unexpected.
- Writers used Wordcraft as a research assistant or beta reader, but its chatbot became less useful as stories developed because suggestions rarely accounted for existing content.
- 6.5 Theme and Variations: Generating themed lists gave participants reusable material for characters, magical items, settings, and alternative passages.
7 System Limitations Experienced by Participants
Participants encountered limitations in Wordcraft’s control, stylistic flexibility, originality, story understanding, and reliability. These constraints made it difficult to use the system for professional creative-writing tasks.
- 7.1 Difficulty Maintaining a Style and Voice: Wordcraft struggled to maintain authors’ desired styles and voices, especially across stories with multiple points of view.Writers observed a bland, elementary default voice that often produced schematic or boring prose.
- 7.2 Suggestions too Easily Revert to Tropes and Repetition: Suggestions often reverted to clichéd tropes, repetition, and biased patterns, making useful brainstorming require extensive filtering.Participants found that the system pushed unconventional premises toward familiar genre conventions and produced many uninteresting suggestions.
- 7.3 The Fickleness of Working with Large Language Models: Wordcraft behaved unpredictably on structured tasks, with small changes in prompts or exemplars affecting whether it followed requested formats and styles.Participants reported inconsistent handling of story-pitch lists, poetic constraints, and author-style rewrites; fantasy-oriented exemplars also biased its outputs.
- 7.3 The Fickleness of Working with Large Language Models: Participants with prior language-model experience could better work around Wordcraft’s fickleness but still wanted finer-grained controls.Their familiarity helped them adjust request wording, while the absence of more precise control knobs remained frustrating.
- 7.4 Lack of Understanding: The system lacked deep understanding of story contents, limiting its usefulness for higher-level writing tasks and making chatbot interactions inaccurate.Participants described the model as having a small memory and superficial understanding of their stories.
- 7.5 Unclear Provenance: Professional writers also faced reputational concerns because Wordcraft did not reveal the sources of its suggestions.Several participants reported needing to web-search generated text before incorporating it into their work to reduce plagiarism risk.
8 Discussion
The discussion shows that professional writers have heterogeneous needs for AI writing assistants, while current systems struggle with intentionality, distinctive voice, control, and alignment with creative goals.
- 8.1 The Need for Taste and Intentionality: The model’s lack of taste and intentionality produced clichés and generic tropes because it lacked a narrative agenda and learned from both good and bad text.Its local next-word prediction objective does not provide the evaluative judgment that writers use to distinguish interesting language.
- 8.2 The Tradeoff Between Safety, Sensibility, and Good Writing: Safety-oriented tuning reduced toxic or incoherent generations but also discouraged transgressive characters and complex or nonsensical metaphors valued in literary writing.The authors distinguish conversational safety and sensibility from the risk-taking sometimes required for creative work.
- 8 Discussion: The paper recommends personalized or bespoke tools because writers, novice users, language learners, designers, and other audiences have distinct needs and risks.Participants identified possible benefits for beginners and designers alongside concerns about cheating, identity, and job automation.
- 8 Discussion: Writers differed widely in whether they wanted AI as a persona-based collaborator, a utilitarian tool, or an extension of their own style.Some participants imagined beta readers, brainstorming partners, or co-writers, while others preferred tools that accelerated writing without simulating conversation.
- 8.3 Writers are Diverse and So are Their Needs: A useful co-writing AI should complement a writer’s existing strengths rather than simply match human performance or replicate an established style.Writers who already have prose skills may need ideas, while those with clear plots may need expressive language or dialogue.
- 8 Discussion: Participants found Wordcraft useful in multiple ways, but no one found it entirely suitable to their goals, underscoring the need for flexible interfaces.Writers leveraged the system differently, and the study argues against a one-size-fits-all co-writing solution.
9 Study Limitations
The study’s conclusions may reflect Wordcraft’s interface and implementation as well as limitations of its underlying language model.
- 9 Study Limitations: Wordcraft received limited professional-writer feedback during development, and the study cannot cleanly separate interface weaknesses from model limitations.A writer-involved design process or different interface might have produced different author reactions.
10 Conclusion
The study finds that AI writing tools are unlikely to replace writers soon but may make parts of creative writing easier, faster, and more enjoyable.
- 10 Conclusion: AI writing assistants may improve demanding parts of creative writing, but their development should involve intended users and target time-consuming, unpleasant tasks.The conclusion applies this promise to skilled and amateur writers while emphasizing audience involvement in tool and model development.
A Author Bios
The author-bios material is presented in Table 5.
- A Author Bios: Table 5 provides participant biographies.The table is identified as containing the biographies supplied for each participant.