Source-linked AI summary
Sparks: Inspiration for Science Writing using Language Models
Katy Ilonka Gero, Vivian Liu, Lydia B. Chilton
TL;DR
Science writing is creative yet constrained, but little work examines how language-model text is used by writers. The paper introduces theory-informed sparks—sentences generated to inspire science writers—and evaluates their quality and use. Sparks outperform an off-the-shelf baseline in coherence and diversity, approach a human gold standard, and support several writing activities for 13 PhD students.
Problem
The paper asks how language-model outputs can support writers in a creative but constrained science-writing task.
Method
The system generates sparks with GPT-2, custom decoding, and prompts informed by narratology and expository theory.
Results
Sparks are more coherent and diverse than a competitive baseline and approach a human-created gold standard.
Takeaways & Limitations
13 PhD students used sparks to craft detailed sentences, find engaging angles, and understand common reader perspectives.
Takeaways & Limitations
The exploratory user study involved 13 participants with varied science-writing experience and topics.
Abstract
from arXiv · showhide
Large-scale language models are rapidly improving, performing well on a wide variety of tasks with little to no customization. In this work we investigate how language models can support science writing, a challenging writing task that is both open-ended and highly constrained. We present a system for generating "sparks", sentences related to a scientific concept intended to inspire writers. We find that our sparks are more coherent and diverse than a competitive language model baseline, and approach a human-created gold standard. In a study with 13 PhD students writing on topics of their own selection, we find three main use cases of sparks: aiding with crafting detailed sentences, providing interesting angles to engage readers, and demonstrating common reader perspectives. We also report on the various reasons sparks were considered unhelpful, and discuss how we might improve language models as writing support tools.
1 INTRODUCTION
The paper asks how language-model outputs can support creative but constrained science writing. It introduces sparks, generated sentences intended to inspire domain experts, and evaluates their quality and use by writers.
- The research question asks how language-model outputs can support writers in a creative but constrained writing task.
- The system generates sparks, sentences related to a scientific concept and intended to inspire writers.
- Sparks are generated with GPT-2 and a custom decoding method designed to encourage specific and diverse outputs.
- The evaluations compare sparks with a competitive baseline and a human-created gold standard on diversity and coherence.
- In an exploratory study, 13 PhD students wrote tweetorials with the system and reported how they used the sparks.
- The reported contributions include a spark-generation system, a custom decoding method, quality evaluation, and an exploratory study identifying three main use cases.
2 RELATED WORK
The related work situates the paper among language-model generation, writing support, and science communication research. It emphasizes limited study of how generated text is used by writers and motivates theory-informed prompting.
- Language models generate text by predicting the likelihood of the next word from a prefix and can be trained on unannotated text.
- Large language models have become more general-purpose, supporting many tasks without requiring a separately trained model for each task.
- Known generation problems include repetitive, vague, false, and contradictory outputs because language models lack a model of truth.
- Generative Writing Support: Few natural-language-generation technologies are studied in terms of how writers use them, including generated text for science writing.
- Science Communication on Social Media: Science communication has expanded through digital networks and social media, enabling scientists and students to disseminate knowledge directly to the public.
- Narrative and Expository Structure: The system draws on narrative and expository discourse theories because tweetorials combine both forms.
3 FORMATIVE STUDY
The formative study examined difficulties in writing engaging tweetorial introductions and informed design goals for sparks. Participants struggled to generate compelling, suspenseful, and varied ideas even with tutorials.
- 3 FORMATIVE STUDY: The formative study asked participants to write first tweets for technical topics after receiving a technique for creating compelling introductions.
- 3.1 Methodology: 10 Computer Science students completed the study, selected topics, wrote first tweets, and discussed their writing process in interviews.
- 3.1 Methodology: The tutorial process involved brainstorming concrete situations, turning them into reader questions, and selecting the most engaging question.
- 3.2 Results: 9 out of 10 participants identified making a topic interesting to a general audience as the task’s most difficult part.
- 3.2 Results: Participants found creating concrete examples and engaging questions difficult, and tutorials did not fully support the task.
- 3.2.2 Participants struggled to come up with ideas that created suspense.: Participants often produced tweets lacking suspense because they did not introduce a compelling problem or gap in readers’ understanding.
- 3.2.2 Participants struggled to come up with ideas that created suspense.: Participants writing about HTTP converged on Google or Twitter examples, suggesting reliance on similar, easy-to-reach ideas.
- 3.3 Design Goals: These findings motivated sparks that provide idea-starters rather than complete ideas, while aiming for coherent and diverse outputs.
4 SYSTEM DESIGN
The system uses GPT-2 with a custom decoding method and hand-crafted prompts to generate concise, diverse sparks for science writing. Its design combines normalized inverse word frequency, top-50 token restriction, unique first tokens with beam search, and context-aware prompt templates.
- Model and decoding: GPT-2 generates sparks from prompts, with decoding and prompt engineering prioritized over fine-tuning for output quality.The authors found science-writing fine-tuning made little difference compared with modifying decoding or prompts.
- Model and decoding: Normalized inverse word frequency increases the likelihood of infrequent words during decoding, using a Vox news corpus to estimate word frequencies.The method is presented as an application of normalized inverse word frequency during decoding rather than training.
- Model and decoding: The system restricts decoding to the top 50 tokens, reducing incoherencies from promoting tokens far down the original probability distribution.This top-k restriction is used without sampling.
- Model and decoding: To increase diversity while retaining beam-search coherence, the system forces the first token of each output to be unique.The authors report that this simple strategy was more effective in testing than more sophisticated alternatives they tried.
- Model and decoding: Sparks are limited to 10 tokens after the prompt and stop when a sentence is complete, keeping outputs succinct and fast to generate.The method is implemented with Hugging Face Transformers.
- Prompt design: Hand-crafted prompts span expository, instantiation, goal, causal, and role categories, while writers can also add their own prompts.Prompts use topic and context-area information to interpret technical terms and support multiple writing angles.
5 STUDY 1: SPARK QUALITY
Study 1 evaluates spark quality across computer science, environmental science, and biology by comparing custom decoding with a baseline and human-created outputs. The custom method improves over the baseline but varies substantially by topic and prompt, remaining below human performance.
- Overall results: The custom decoding produces more coherent and diverse sparks than the baseline, but does not reach human-created output quality.The same underlying language model is used for the custom and baseline conditions.
- Output characteristics: Example outputs show that human responses are longer, more specific, and more diverse, while both language-model conditions are shorter and more repetitive.The custom method nevertheless improves overall output quality relative to the baseline.
- Caveat: Large variation in diversity and coherence means average improvements may not predict the quality of an individual spark for a given prompt and topic.The authors note that a particular output can be very high quality or much lower quality.
- Topic variation: Custom-decoding quality varies widely by topic: computer security averages 3.7 coherence, whereas automata theory averages 2.1.Human-created outputs are more consistent, with no topic below 3 in average coherence.
6 STUDY 2: USER EVALUATION
Study 2 examines how 13 STEM students use sparks while writing tweetorials on self-selected expert topics. Participants used sparks to craft sentences, find reader-relevant angles, and anticipate audience perspectives, but usefulness varied substantially.
- Analysis: The authors analyzed interviews thematically across helpfulness, unhelpfulness, and concerns about incorporating machine-generated text.Themes were reported when mentioned by more than 20% of participants, except for ownership concerns because responses were highly variable.
- Interaction patterns: Participants differed sharply in engagement and perceived usefulness: some generated and starred many sparks, while others used only a few.The average participant generated 17.2 sparks and starred 5.1, but starring more sparks did not necessarily indicate greater usefulness.
- Overall assessment: Participants’ interactions followed three archetypes—prompt and flow, flow with breaks, and shifting—and many thought sparks could speed writing despite some low-quality outputs.Some participants found the system completely unhelpful, while others valued the helpful sparks despite occasional nonsensical suggestions.
- Helpful uses: Participants used sparks to craft detailed sentences, often finding it faster to adapt spark language than to write from scratch.Some participants reported that even familiar ideas could be reached more quickly with sparks.
- Helpful uses: Sparks provided angles for making specialized research more engaging and relevant to general readers.Participants used sparks to connect topics such as glacier retreat with sea-level rise and identify ways to make research feel relevant.
- Helpful uses: Sparks also exposed likely reader assumptions and helped writers consider what background knowledge or misconceptions their audiences might bring.Examples involved assumptions about sexism and readers’ familiarity with NP-completeness.
7 DISCUSSION
The discussion identifies how sparks support both convergent and divergent aspects of science writing, while highlighting limitations in controllability, knowledge breadth, graceful failure, and user variation.
- Use cases: Participants used sparks for divergent thinking by connecting their topics with ideas that could interest readers.Sparks also reminded participants how readers might interpret their topics.
- Use cases: Sparks helped participants turn implicit, detailed, and messy knowledge into concise sentences, supporting convergent thinking.Participants often knew what they wanted to explain but used sparks to recognize correct or well-phrased formulations.
- Use cases: Unlike much storytelling research focused on generating new ideas, this work also addresses the convergent task of tying ideas together satisfactorily.The discussion relates this distinction to science writing, where participants often knew the destination but not where to start.
- Technical development: Future systems should improve controllability, broaden language-model knowledge, and make failures more graceful and understandable.The discussion emphasizes natural-language prompt control, better understanding of training data, and methods for detecting why low-quality outputs occur.
- Limitations: Participants varied in their comfort with prompting and in how they interpreted inaccurate, redundant, or irrelevant outputs.Some treated one or two poor sparks as a dead end, while others repurposed them or used them as signals about their topic.
- Limitations: The exploratory study had 13 participants with different science-writing experience and topics, limiting how carefully effects of writing style or topic granularity could be studied.The authors suggest larger participant sets for examining these differences and note that the system used one language model and one decoding method.
8 CONCLUSION
The paper investigates language-model support for creative but constrained science writing and introduces sparks, sentences about scientific concepts intended to inspire writers. Sparks were higher quality than a baseline and approached a human-created gold standard, while 13 PhD students used them to craft sentences, engage readers, and understand reader perspectives.
- The paper investigates how language models can support writers in the creative but constrained task of science writing.
- The system generates sparks: sentences about a scientific concept intended to inspire writers.
- Sparks were higher quality than a baseline system and approached a human-created gold standard.
- In an exploratory study with 13 PhD students, participants used sparks to craft detailed sentences, engage readers, and understand common reader perspectives.
- The paper discusses future language-model writing support, technical development priorities, and how users might learn to interact with models in writing contexts.
A.1 Full List of Topics Studied
The study sampled topics from biology, environmental science, and computer science glossaries.
- The biology topics included endergonic reactions, genetic drift, decomposition, dynein, circadian rhythm, placebos, ethology, osmosis, reproductive biology, and bioenergetics.
- Topics were randomly sampled from Wikipedia glossaries for biology, computer science, and environmental science.
- The environmental-science topics included biocapacity, resource productivity, forage, polypropylene, open-pit mining, soil conditioner, incineration, green marketing, coir, and old-growth forests.
- The computer-science topics included source code, automata theory, computer security, control flow, Boolean expressions, double-precision floating-point format, linear search, software development, hash functions, and cyberbullying.
B.1 Survey Questions
The survey asked about participants’ graduate-study background and their experience writing or posting technical topics for general audiences.
- Participants were asked which year of their graduate program they were in.
- Participants were asked what kind of graduate program they were enrolled in.
- Participants were asked which discipline they studied.
- Participants were asked how often they wrote about technical topics for general audiences.
- Participants were asked how often they posted about technical topics on Twitter.
B.2 Interview Questions
The interview asked participants how sparks influenced their writing, how they compared with familiar resources, and what concerns or stopping points arose.
- Participants were asked whether sparks helped and to connect helpful sparks to the writing they eventually produced.
- The interview compared sparks with Wikipedia, Google Search, and other frequently used resources.
- Participants were asked how existing prompts differed from custom prompts.
- Questions examined unhelpful or incorrect sparks and participants’ reactions to them.
- The interview also asked what made participants stop generating sparks and whether they had concerns about ownership or agency.
- Participants could add unprompted comments or questions at the end of the interview.