PaperlayerView exact revision on arXiv

Source-linked AI summary

Choice Over Control: How Users Write with Large Language Models using Diegetic and Non-Diegetic Prompting

Hai Dang, Sven Goller, Florian Lehmann, Daniel Buschek

arXiv:2303.03199v1cs.HCcs.CL

TL;DR

The paper asks how users guide LLM writing through narrative text and external instructions, and studies this with an interface combining both forms. In a study with 129 crowd workers, participants preferred selecting among multiple suggestions to controlling them with instructions, while using both prompting forms. The study's short exposure leaves long-term use for future investigation.

  • Problem

    The paper examines how users write with LLMs using narrative text and external instructions, distinguishing these forms to study their interaction.

    Source: §1 · p. 2

  • Method

    The authors introduced diegetic and non-diegetic prompting and studied 129 Prolific workers writing with interfaces varying suggestion count and instruction availability.

    Source: §1 · p. 2

  • Results

    Users preferred choosing among multiple suggestions over controlling them with non-diegetic prompts, which they used to request inspiration, topics, or facts.

    Source: §1 · p. 2

  • Takeaways & Limitations

    Users combine narrative text and external instructions to guide LLMs, and suggestion timing indicates that they also use their draft as guidance.

    Source: §1 · p. 2, §10 · p. 14

    Participants spent five minutes with each UI, so future studies should investigate long-term use as writers gain experience.

    Source: §9.5 · p. 13

Abstract

from arXiv · show

We propose a conceptual perspective on prompts for Large Language Models (LLMs) that distinguishes between (1) diegetic prompts (part of the narrative, e.g. "Once upon a time, I saw a fox..."), and (2) non-diegetic prompts (external, e.g. "Write about the adventures of the fox."). With this lens, we study how 129 crowd workers on Prolific write short texts with different user interfaces (1 vs 3 suggestions, with/out non-diegetic prompts; implemented with GPT-3): When the interface offered multiple suggestions and provided an option for non-diegetic prompting, participants preferred choosing from multiple suggestions over controlling them via non-diegetic prompts. When participants provided non-diegetic prompts it was to ask for inspiration, topics or facts. Single suggestions in particular were guided both with diegetic and non-diegetic information. This work informs human-AI interaction with generative models by revealing that (1) writing non-diegetic prompts requires effort, (2) people combine diegetic and non-diegetic prompting, and (3) they use their draft (i.e. diegetic information) and suggestion timing to strategically guide LLMs.

1 INTRODUCTION

The introduction distinguishes narrative text that guides an LLM from external instructions, then studies how users combine these forms while writing. In a study of 129 crowd workers, users preferred choosing among multiple suggestions over controlling suggestions with instructions.

1 INTRODUCTION

  • Diegetic prompts are part of the user's narrative, while non-diegetic prompts are external instructions that guide generation without becoming part of the resulting document.The paper introduces this distinction as a lens on how users influence LLMs while writing.Source: §1 · p. 2
  • The study asks how people write with LLMs using diegetic and non-diegetic prompting, a distinction useful for examining user interaction even though both reach the model as text..Source: §1 · p. 2
  • A remote study had 129 Prolific workers write five stories across four UI variants and a baseline, varying instruction availability and suggestion count.The paper also contributes a UI combining text continuations with zero-shot prompt input.Source: §1 · p. 2
  • With multiple suggestions available, users preferred choosing among them over controlling suggestions with non-diegetic prompts.They used non-diegetic prompts to ask for inspiration, topics, or facts; composing them also required effort.Source: §1 · p. 2

2 RELATED WORK

The related work spans LLM prompting and writing interfaces, positioning this study's focus on non-expert prompting within systems that combine draft text and external instructions.

2 RELATED WORK

  • The paper connects work on NLP prompting with HCI writing interfaces and locates its diegetic/non-diegetic lens among existing systems..Source: §2 · p. 2

2.1 Prompting in Large Language Models

  • LLMs can perform tasks from natural-language prompts, but optimizing prompts is difficult and requires extensive experience.Few-shot examples can steer a model, while zero-shot instructions omit examples.Source: §2.1 · p. 2

2.1.1 Prompt Engineering.

  • Prompt-engineering methods include paraphrasing and task-specific templates, whereas this study examines how non-experts use zero-shot prompts while writing.Engineered prompts are generally designed for model performance rather than human consumption.Source: §2.1.1 · p. 2

2.1.2 Prompting Interfaces.

  • Prior prompting interfaces make prompts a central interaction output; this work instead integrates prompt input into a writing-focused text editor.Examples include AI Chains, PromptMaker, prompt programming environments, and Story Centaur.Source: §2.1.2 · p. 2

2.2 Writing Interfaces for LLMs

  • The section surveys design factors in interfaces that use LLMs for text generation..Source: §2.2 · p. 2

2.2.1 Scope of Suggestions.

  • The paper focuses on phrase suggestions, which prior work describes as new ideas for writing, rather than single-word suggestions often evaluated by keystroke reduction..Source: §2.2.1 · p. 3

2.2.2 Display of Suggestions.

  • The study shows single suggestions inline, multiple suggestions in a popup list, and uses an explicit hotkey to request them.Prior work also explored implicit triggers such as inactivity or utility functions.Source: §2.2.2 · p. 3, §2.2.3 · p. 3

2.3 Diegetic and Non-Diegetic Prompting in Existing Writing Interfaces

  • Existing writing systems mainly use preceding draft text as diegetic input, while some combine it with external prompts or other non-diegetic information.This study includes the full written draft and lets users add custom prompts to guide the LLM.Source: §2.3 · p. 3

3 INTERACTION CONCEPT

The interaction concept combines diegetic text continuation and non-diegetic instructions in one editor, allowing users to work with both without taking their hands off the keyboard.

3 INTERACTION CONCEPT

  • The UI closely integrates diegetic and non-diegetic prompting in the same editor, keeping both available without requiring users to take their hands off the keyboard..Source: §3 · p. 3

3.1 Inline (Single) Suggestions (Figure 1 top row)

  • For inline single suggestions, TAB displays a green preview at the caret; ENTER accepts it, while ESC or continued typing cancels it.The green styling distinguishes unaccepted preview text from document text.Source: §3.1 · p. 3

3.2 Multiple Suggestions (Figure 1 bottom row)

  • For multiple suggestions, the interface presents three phrases in a list navigable by arrow keys or mouse, with ENTER confirming a selection..Source: §3.2 · p. 3

3.3 Pop-up Textbox for Non-Diegetic Prompts (Figure 1 right column)

  • When requesting suggestions, users may type an optional instruction in a popup above the caret, submit it, and revise it to update suggestions.TAB or ENTER submits the instruction; another ENTER accepts the selected suggestion.Source: §3.3 · p. 3

4 PROTOTYPE IMPLEMENTATION

The prototype implemented the integrated writing interface as a web system using GPT-3 text-davinci-edit-001 for text suggestions.

4 PROTOTYPE IMPLEMENTATION

  • The prototype implementation section directs readers to Figure 1 and Appendix A for screenshots of the web interface..Source: §4 · p. 3

4.1 Web System

  • The ReactJS and CKEditor5 client sent suggestion requests to a FastAPI server, which forwarded the full draft and an instruction to OpenAI's text-davinci-edit-001.The authors selected the model and API because they were reportedly trained for text plus a separate instruction.Source: §4.1 · p. 4

4.2 Language Model Prompts

  • The system used “Complete the sentence” by default and appended a user's instruction when provided, then trimmed generated output to show only the continuation.Other tested default instructions produced longer or less consistent text.Source: §4.2 · p. 4

4.3 Information Box

  • An information box showed an image of the interface and explained the available action keys..Source: §4.3 · p. 4

5 METHOD

The method combines questionnaires and interaction logging with qualitative coding of open responses and checks of participant-written texts.

5 METHOD

  • The study’s methods were selected in line with related human-AI writing studies.Source: §5 · p. 4

5.1 Questionnaires

  • Questionnaires assessed participants’ backgrounds and gathered feedback after each UI variant, followed by optional open comments about the system and experience.The initial questionnaire covered demographics and prior experience with writing features and language models.Source: §5.1 · p. 4

5.2 Interaction Logging

  • Interaction logs captured key and mouse events during writing, including suggestion-trigger positions and AI instructions.Events included timestamps, task identifiers, and the current editor text.Source: §5.2 · p. 4

5.3 Coding of Open Questions

  • Final-questionnaire open comments were coded through an inductive codebook, shared coding, cross-checking, and discussion to resolve disagreements.Two researchers developed the codebook from 20 participants, coded those responses plus 32 more, then one researcher coded the remainder and another checked it.Source: §5.3 · p. 4

5.4 Evaluation of User Written Text

  • Participant-written texts were checked for spelling and grammar errors and reviewed for alignment with selected prompts.LanguageTool counted mistakes; three researchers independently assessed prompt connection, and one reviewed comments for consistency.Source: §5.4 · p. 4

6 USER STUDY

The within-subject study compared four suggestion interfaces and a no-suggestion baseline across five writing tasks, with task order counterbalanced.

6.1 Study Design

  • Five UI variants crossed one or three suggestions with optional instructions, alongside a no-suggestion baseline; their order was fully counterbalanced.The study measured interaction behavior and questionnaire responses.Source: §6.1 · p. 4

6.2 Participants

  • The main study involved 129 Prolific participants, screened for English fluency and access to a keyboard-equipped computer.A six-person pre-study preceded it; participants were aged 18–70, with a median age of 32.Source: §6.2 · p. 4

6.3 Procedure

  • Participants completed five writing tasks in counterbalanced order, then answered a final questionnaire; the session averaged 45 minutes and 41 seconds.Source: §6.3 · p. 4

6.3.1 Topic Selection.

  • For each task, participants selected a topic; they could repeat selections but had to choose at least two different topics overall.Topic order was randomized and topics were shown one at a time to encourage variety.Source: §6.3.1 · p. 5

6.3.2 Writing Task.

  • Participants wrote about their selected topic for five minutes and were encouraged to try all features while also writing their own text.The timer was a reminder rather than a hard cutoff, and participants completed a questionnaire after each task.Source: §6.3.2 · p. 5

7 RESULTS

Across the study, participants favored multiple suggestions, while optional instructions raised acceptance for single suggestions; preferences and suggestion-request timing also varied across interfaces.

7.1 Suggestion Acceptance

  • Across 3,097 suggestion sessions (mean 6.47, SD 4.00, in tasks with suggestions enabled), acceptance was lower with one than three suggestions, while instructions narrowed the gap.Acceptance means were 0.45 for one suggestion without instructions and 0.65 with instructions; the overall means were 0.55 for one suggestion and 0.74 for three.Source: §7.1 · p. 5, §7 · p. 5, Figure 2 · p. 6

7.2 Task Completion Time / 7.3 Text Length

  • Texts were about 6–10 words shorter with single suggestions or instructions and longer with three suggestions; three-suggestion tasks took longer than baseline.Mean word counts were s0=136, s1=130, s3=140, ino=138, and iyes=131; suggestion-UI task times did not differ significantly.Source: §7.3 · p. 5, §7.2 · p. 5, Figure 2 · p. 6

7.4 Moments of Suggestion Requests

  • Single suggestions were requested at sentence starts less often (21.70% vs 31.02%), about 1.5 words later mid-sentence (10.93 vs 9.48), and less often after transitions (11.80% vs 14.03%).Instruction availability did not significantly affect the mid-sentence word position or transition-word request rate.Source: §7.4.1 · p. 5, §7.4.2 · p. 5, §7.4.3 · p. 6

7.5 Perception of the Tasks and UIs

  • Ratings varied across users, but single suggestions were rated more distracting and manually editing, and less helpful, controllable, and diverse than lists of three.Suggestions generally received favorable ratings and were perceived to have almost no grammatical or factual errors.Source: §7.5 · p. 6

7.5.3 Interactions of Number and Instruction.

  • Instructions improved perceived influence (x=1.92) but increased distraction (x=2.03) and manual editing (x=1.54); distraction rose less from one to three suggestions when instructions were available.Most participants did not find suggestion UIs distracting; instruction-focused ratings did not differ significantly between one and three suggestions.Source: §7.5 · p. 6, §7.5.3 · pp. 6–7

7.6 Instruction Usage and Content

  • Participants sent 397 instructions, using one in about every fifth eligible suggestion request; keywords or keyword lists were most common (171 uses), followed by imperatives (59) and questions (12).The average was 3.08 instructions per participant.Source: §7.6 · p. 7

7.7 Evaluation of Text Quality

  • The mean spelling-and-grammar mistake rate was 0.0025 per word, and 23 of 645 texts (about 3.5%) did not align with selected topics.Most off-topic texts (13 of 23) were written for the “shapeshifter” topic.Source: §7.7 · p. 7

8 OPEN FEEDBACK

Open feedback revealed a preference for multiple suggestions, while participants described both benefits and costs of non-diegetic instructions. Their comments also show how writers used suggestions for inspiration and guided them through both instructions and their drafts.

8.1 Comments on Suggestions

  • Most participants preferred multiple suggestions, valuing the greater chance of finding a fit, while single suggestions were seen as more intuitive, faster, and less distracting.Fourteen participants described benefits of both interfaces, including the possibility that multiple options would offer a better suggestion than accepting the first one.Source: §8.1 · pp. 7–8
  • Participants used suggestions for inspiration, to overcome writer’s block, or to finish sentences, but reported mixed quality and sometimes substantial editing.One participant said instructions could produce more detailed and interesting sentences when the direction was already known, though the suggestions often still needed changes.Source: §8.1 · p. 8

8.2 Comments on Instructions

  • Opinions on instructions were divided: participants valued guidance, inspiration, and delegating small tasks, but some found instructions difficult, disruptive, or less natural.Some participants used instructions to get suitable suggestions or words; others said composing them interrupted writing flow or made the interaction feel less like collaboration.Source: §8.2 · p. 10

8.3 Control and Influence

  • Participants associated control with both multiple choices and instructions, while some instead guided suggestions by shaping their draft before requesting them.Others disliked having to construct lead-in sentences, and some said they deliberately positioned requests where the story could develop in different directions.Source: §8.3 · p. 10

8.4 Learnability

  • Thirty-three participants raised learnability challenges, and 25 said they used instructions little because they struggled to think of them.Participants described the instruction field as clunky and said devising prompts could feel stressful or difficult.Source: §8.4 · p. 10

8.5 Distraction

  • Some participants found suggestions distracting when they already knew what they wanted to say; instructions could be useful when they felt stuck.One participant specifically linked writing without instructions to better focus and more natural sentence flow.Source: §8.5 · p. 10

8.6 Perception of the AI and Expectations

  • Participants viewed the AI either as an efficiency tool or as a source of inspiration, and some read suggestions for ideas even when they did not accept them.Those seeking efficiency could find instructions onerous, while those seeking inspiration asked for content or explored where the AI might take a story.Source: §8.6 · p. 10

9 DISCUSSION

The discussion argues that writers generally preferred choosing among suggestions, while also using instructions and draft context to guide generation. It highlights the effort of non-diegetic prompting and differing expectations about whether AI should propose ideas or transcribe them.

9.1 Choice vs. Control

  • In short creative and argumentative writing, participants preferred choosing among multiple suggestions over writing instructions; instructions improved acceptance of single suggestions but did not match multiple suggestions.The authors suggest that instructions were situationally useful and motivate further exploration of interfaces combining choice and control.Source: §9.1 · pp. 10, 12

9.2 Guiding Suggestions with Diegetic Prompts

  • With single suggestions, writers supplied about 1.5 more words in the preceding sentence and requested suggestions less often at sentence starts or after transition words.The authors interpret these request locations as evidence that writers use draft context and timing to guide continuations, especially when only one suggestion is shown.Source: §9.2 · p. 12

9.3 Challenges of Integrating Non-Diegetic Prompts

  • Participants’ expectations of prompting’s benefits and costs shape whether they prefer multiple choices or instructions.The discussion identifies choice versus control as the central trade-off, with instructions offering a way to guide single suggestions.Source: §9.3 · p. 12, §9.1 · p. 12

9.3.1 Non-Diegetic Prompts Interrupt the Writing Process.

  • Switching from narrative writing to non-diegetic instructions requires writers to shift attention toward composing directions for the system, whereas diegetic prompting avoids that shift.The discussion connects this interruption to participants’ comments and questionnaire responses about distraction and difficulty thinking of instructions.Source: §9.3.1 · p. 12

9.3.2 Non-Diegetic Prompts can be Hard to Write.

  • Many participants found effective non-diegetic prompts difficult to write, although they experimented with different styles.The observed styles did not go beyond comments that could also be meaningful to a human co-author; the authors note they might evolve with longer use.Source: §9.3.2 · p. 12

9.4.1 Two Perspectives on the Main Role: Proposer vs Transcriber.

  • Some participants saw AI as a transcriber for efficiency, while others saw it as a proposer of inspiration; settled opinions on argumentative topics could make divergent proposals distracting.The authors suggest topic and familiarity may influence this mindset and recommend accounting for such differences when selecting writing topics to calibrate metrics.Source: §9.4.1 · pp. 12–13

9.4.2 Non-diegetic Prompts Reflect Users’ Perception of the AI.

  • Non-diegetic prompts ranged from requests for inspirational ideas and word choices to transcription, advice, opinions, and information lookup.The authors note that prompts for advice or information may shift users’ perception of AI toward that of a writing collaborator.Source: §9.4.2 · p. 13

9.5 Limitiations and Reflections on Methodology

The authors reflect on limits in study duration, model access, and online observation, then extend the diegetic/non-diegetic lens beyond writing and draw implications for LLMs and interfaces.

9.5 Limitiations and Reflections on Methodology

  • Five minutes per UI leaves long-term use and how prompts evolve with experience untested.The authors call for future studies of long-term use.Source: §9.5 · p. 13
  • GPT-3 API access left the authors unable to identify the best model settings, while model changes beyond their control limit exact replicability.Suggestions could also repeat or echo instruction text, although they were rated highly overall.Source: §9.5 · p. 13
  • The online study enabled interaction logging at scale but limited direct observation and in-the-moment questions; think-aloud work could probe decisions about suggestions and instructions.The authors note that they nevertheless received rich qualitative feedback.Source: §9.5 · p. 13
  • Provided examples may have biased instruction styles, and the study tested only one popup design for integrating instructions.The pre-study found examples useful for getting started, while participants also experimented beyond them.Source: §9.5 · p. 13
  • The authors call for systematically evaluating and curating open writing tasks that cover varied topics, text types, and aspects beyond input speed.They describe open tasks as important for HCI research on writing tools.Source: §9.5 · p. 13

9.6 Beyond Writing: Diegetic and Non-diegetic Interaction in Generative Systems

  • The diegetic/non-diegetic distinction can help analyze generative interactions beyond text-to-text writing, including visual-to-text and visual-to-visual systems.Examples include sketches guiding language models and abstract shapes guiding image models.Source: §9.6 · p. 13

9.7 Implications for LLMs and User Interfaces

  • LLMs for writing support should handle a broader range of non-diegetic inputs, or interfaces should guide users toward supported styles.Keyword-style prompts were more common in the study, while PEER was trained on imperative-style prompts.Source: §9.7 · p. 14
  • Because switching between diegetic and non-diegetic writing has cognitive and interaction costs, future work should study ways to make such switches easier and more efficient.The authors note that even the best model cannot eliminate these costs.Source: §9.7 · p. 14

10 CONCLUSION

The paper frames writing with LLMs as using both draft-based and external prompts. In the GPT-3 study, participants preferred choosing among suggestions, while the findings point to effort and strategic use of both prompt types.

10 CONCLUSION

  • With the GPT-3 interface, participants preferred choosing from multiple suggestions over writing instructions, while also using draft text to guide suggestions.The paper argues that users shape text through both diegetic and non-diegetic information.Source: §10 · p. 14
  • Writing instructions requires effort, and single suggestions benefited from both diegetic and non-diegetic prompting.Writers also used their draft and suggestion timing to guide LLMs strategically.Source: §10 · p. 14
  • The authors encourage further study of these prompt types across writing tools and other domains, and release their prototype and study materials.Source: §10 · p. 14

A APPENDIX

The appendix provides supplementary interface screenshots and an overview of events logged during the user study.

A APPENDIX

  • The appendix includes a table overviewing interaction events logged during the user study.Source: A APPENDIX · p. 15, Table 3 · p. 16
  • The writing-interface screenshot identifies the info box, selected topic, and editor with the current text and an inline suggestion.Source: Figure 8 · p. 16
  • The topic-selection panel lets users browse topics and indicate when they are ready to write about one.Source: Figure 9 · p. 17

Related papers

Loading 2303.03199v1…