Look Before You Prompt, and After: Scaffolding Human-AI Collaboration in Software Tutorial Creation

1 Introduction

• LLM-assisted tutorial creation requires writers to steer output toward the intended audience, organizational style, and correct software context, then verify it.Without addressing these issues, tutorials can be confusing, ineffective, or misleading.Source: §1 · pp. 2–3
• Writers spend considerable effort steering model behavior and verifying generated code and prose, with prose verification described as mostly manual.The work includes communicating the audience, organizational standards, and prompt goals to the model and checking its output.Source: §1 · p. 3
• Interviews with 17 technical writers surfaced requirements for tracking and structuring content across LLM responses, controlling prompt context, and verifying generated code and prose.The requirements also address keeping interdependent artifacts consistent.Source: §1 · p. 3
• The authors developed dBlocks through a human-centered design process, iteratively refining the design with five technical writers before a within-subjects lab study.The lab study compared the final design with participants’ existing LLM tools.Source: §1 · p. 3
• Participants reported significantly higher confidence in tutorials produced with dBlocks than in their preferred LLM-assisted authoring workflows.Writers also verified code while drafting and organized work into scoped sections with section-specific conversations.Source: §1 · p. 3
• The authors frame LLM-assisted knowledge work as shifting practitioners from content creators to curators, while noting that this study concerns software tutorial creation.They suggest the pattern may also occur where practitioners use LLMs to produce artifacts for which they are accountable.Source: §1 · p. 4

2.1 Creating Software Documentation

• Software documentation involves iterative updates, while tutorials also require pedagogical choices about what to explain and how much prior knowledge to assume.Tutorials vary widely in design, complicating general-purpose creation tools.Source: §2.1 · p. 4

2.2 Tool Support for Creating Software Tutorials

• Existing tutorial tools automate screenshots, support collaborative annotations, or synchronize content with code changes, but still leave substantial manual effort.LLM tutorial tools address output management, context control, and dependencies; this work emphasizes context access and content verification.Source: §2.2 · p. 5

2.3 LLM-Assisted Writing

• Tutorial writing combines technical accuracy, sequential instructional clarity, and pedagogical judgment, demands that current writing-assistant research does not specifically address.Writers must maintain consistency among explanations, source programs, code snippets, and execution outputs.Source: §2.3 · pp. 5–6
• LLMs can amplify tutorial inconsistency when dependencies among content elements are not managed, and verification against code, explanations, and outputs can require more effort than writing from scratch.The authors connect this need to domain-specific interfaces for verification and managing relationships among sections.Source: §2.3 · p. 6

2.4 Human-Centered Design Approaches for Software Engineering

• The paper follows human-centered design activities to understand use context, specify requirements, produce design solutions, and evaluate them with users.Its studies combine qualitative and quantitative methods.Source: §2.4 · p. 6

3.1 Study Design

• The original study examined technical writers’ documentation-quality practices, especially review processes and their challenges.Source: §3.1 · p. 7

3.1.1 Recruitment.

• Participants were recruited through Write The Docs Slack and the Technical Writer Forum on LinkedIn, communities frequented by experienced technical writers.Source: §3.1.1 · p. 7

3.1.2 Study Procedure.

• The original interview protocol covered documentation practices, tools, review processes, and LLM use, including validation and workflow changes.Source: §3.1.2 · p. 7
• Interviews were remote, screen-recorded, and lasted 45–70 minutes; participants demonstrated their tutorial-writing practices while sharing screens.Transcripts were produced using Microsoft Teams’ automated transcription.Source: §3.1.2 · p. 7
• The secondary analysis selected 17 interviewees who reported using LLMs professionally to create instructional material.Source: §3.1.2 · p. 7

3 Formative Study

• The authors re-analyzed 17 interviews to identify patterns specific to LLM-assisted tutorial workflows.The interviews were drawn from a broader study of how technical writers establish and maintain documentation quality.Source: §3.1.3 · p. 7, §3.1.2 · p. 7, §3 · p. 7

3.1.3 Analysis.

• The first author developed preliminary codes and candidate themes, then all three authors reviewed and refined them in regular meetings.The first author had conducted and coded the original study’s interviews.Source: §3.1.3 · p. 7
• The authors report that the 17 transcripts provided sufficient depth to identify recurring patterns in LLM-assisted writing practices.Source: §3.1.3 · p. 7

3.2.1 Writers Assemble and Refine Tutorial Drafts Across Many LLM Responses.

• Writers combined parts from multiple responses and selectively revised problematic passages while preserving content that already worked.They described output quality as varying both across and within responses, sometimes prompting for different tutorial sections separately.Source: §3.2.1 · p. 8
• Writers need to select and edit response content, track reusable material across prompts, and preserve consistency between interdependent code and prose.Source: §3.2.1 · p. 8

3.2.2 Writers Extend and Restrict Context to Shape LLM Outputs.

• Writers expanded context with project documentation, style guides, and prior work, while also restricting prompts to selected resources.Prompts also specified output requirements such as word limits, audience, and tone.Source: §3.2.2 · p. 9
• Writers need context controls that expose available resources and let them choose which resources accompany each prompt.Source: §3.2.2 · p. 9

3.2.3 Writers Verify LLM Outputs Using Their Domain Expertise, Authoritative Sources, and Execution.

• Writers regarded themselves as accountable for the final product and used time saved during drafting to verify generated content.Participants described cross-checking information and researching claims rather than relying on the LLM alone.Source: §3.2.3 · p. 9
• Writers checked technical content against domain knowledge and authoritative sources, and treated execution as definitive for code examples, configuration files, and API calls.Source: §3.2.3 · p. 9
• Because chat interfaces lacked evaluation support, writers performed verification outside the chat and needed it integrated into their writing workflow.Source: §3.2.3 · pp. 9–10

• The preliminary dBlocks design was guided by the requirements derived from the formative study.Source: §4 · p. 10
• Five writers participated in a study to iterate on the preliminary design and identify improvements.Source: §4 · p. 10
• The authors refined the tool based on participant feedback and used the resulting final version in the summative study.Table 2 summarizes how design components evolved through the process.Source: §4 · p. 10

4.1.1 Blocks.

• Blocks divide a tutorial into discrete sections and keep each section’s content and LLM conversation localized for focused iteration.Source: §4.1.1 · pp. 10–11

4.1.2 Scratchpads.

• Separate text and code scratchpads support different work: combining and staging text drafts versus running code to inspect execution outcomes.The code scratchpad enables verification without switching to external tools.Source: §4.1.2 · p. 11

4.1.3 Context Manager.

• The globally scoped context manager displays resources from every block and lets writers select and preview what accompanies each prompt.Available context includes chat messages, scratchpads, and content-editor text.Source: §4.1.3 · p. 11

4.2.1 Method.

• Five technical writers tried the preliminary design by creating a short Python tutorial and discussing feature usefulness and desired improvements.Sessions included interviews about existing workflows, a roughly 20-minute creation task, and a post-study interview.Source: §4.2 · p. 11, §4.2.1 · p. 11

4.2.2 Finding 1: Participants bypassed scratchpads and worked directly with the content editor.

• Participants bypassed the text scratchpad and moved LLM output directly into the content editor, where they could see how content fit together.They were accustomed to copying outputs into the editor and described moving content between areas as friction.Source: §4.2.2 · p. 12
• Participants mainly used the code scratchpad for execution rather than iterative editing, so the design moved code verification into the content editor.The scratchpads were removed while retaining code execution in the editor.Source: §4.2.2 · p. 12

4.2.3 Finding 2: Participants needed flexibility in manipulating block boundaries.

• Participants preferred drafting holistically in one block before making structural decisions, but wanted to reorganize content across blocks afterward.The design added split and merge operations for document-level reorganization.Source: §4.2.3 · p. 12
• Participants also found opening the context manager to manage individual elements introduced friction, leading to inline shortcuts for context management.The shortcuts let users manage context elements directly.Source: §4.2.3 · p. 12

4.3.1 Overview.

• dBlocks provides a tutorial library for creating or reopening tutorials, with assembled-content preview and Markdown export.The final design followed findings and refinements from the pilot study.Source: §4.3.1 · p. 12

4.3.2 Organizing and Restructuring Content with Blocks.

• Writers can split a block at a content point or merge it with the block below, while preserving both conversation histories for navigation.Splitting creates separate blocks; merging appends the lower block's content.Source: §4.3.2 · p. 13, Figure 3 · p. 13

4.3.3 Creating and Verifying Content in the Block Workspace.

• Each block combines a content editor with its own chat, and users can copy generated snippets or full responses into the editor.Code can be run from the editor, with output or error messages displayed inline below each code block.Source: §4.3.3 · p. 13, Figure 4 · p. 14

4.3.4 Selecting Context for Prompts with the Context Manager.

• The context manager lets users select editor text, chat messages, and uploaded reference material, preview the full context, and add elements through inline shortcuts.Users can upload Markdown files and create named snippets from portions of them.Source: §4.3.4 · p. 14

4.3.5 System Implementation.

• The web application combines selected context with the user's prompt and uses Judge0 to run submitted code in isolated, containerized sandboxes.Code execution supports Python, JavaScript, C++, and Go; LLM interactions use the Anthropic API with Claude 3.5 Haiku.Source: §4.3.5 · p. 15

5 Summative User Study

• Researchers conducted a within-subjects study comparing dBlocks with participants' preferred LLM tools.The study was designed to understand how the design choices shape writers' workflows in practice.Source: §5 · p. 15
• The study examines how dBlocks compares with writers' existing LLM tools when creating software tutorials.The authors use both quantitative and qualitative measures to address this question.Source: §5 · p. 15
• The study was approved by the university's ethical review board.Source: §5 · p. 15

5.1 Study Design

• Each of 16 participants created the same Python tutorial using dBlocks and their unrestricted preferred LLM tools in a counterbalanced within-subjects design.Participants had at least one year of Python experience, used LLMs at least biweekly for six months, and had documentation-creation experience.Source: §5.1 · p. 15, §5.1.1 · p. 15, §5.1.2 · p. 16

5.1.2 Task and Procedure.

• Participants created a beginner tutorial on extracting dates from free-form text using the re and datetime libraries, with at least two executable code examples.The task left structure and depth to participant discretion and supplied official library documentation.Source: §5.1.2 · p. 16
• The authors note that 15-minute tutorial creation omits the review cycles and organizational quality criteria involved in professional tutorial work.Source: §5.1.2 · p. 16
• Each condition allowed 15 minutes; participants completed modified NASA-TLX ratings, including confidence, followed by interviews about writing, context management, and verification.Participants received a short tutorial and familiarization task before using dBlocks.Source: §5.1.2 · p. 16

5.1.3 Analysis.

• The analysis combined reflexive thematic analysis of interviews, coding of screen recordings, and Wilcoxon signed-rank comparisons of NASA-TLX ratings.Rank-biserial correlation was used as the effect size; the paired-test choice reflected the within-subjects design and sample size of 16.Source: §5.1.3 · p. 17

5.2 Findings

• 3.31 vs. 2.56: participants reported higher confidence with dBlocks than control (W=8.0, p=.022, r=.76).Ratings are shown in Figure 7; perceived success was marginally higher, while frustration, mental demand, and effort did not differ significantly.Source: §5.2 · p. 17, Figure 7 · p. 18

5.2.1 Inline Execution Reduces Friction in Checking Generated Code Quality.

• 15 of 16 participants agreed or strongly agreed that inline execution was useful, the highest usefulness rating among dBlocks’ three features.Four participants explicitly connected confidence in the tutorial to being able to execute code.Source: §5.2.1 · p. 18, §5.2 · p. 18
• 15 of 16 participants executed code inline while generating, whereas 10 of 14 who generated code in control did not attempt execution.Four dBlocks participants encountered errors; two recovered by adding the error to context and re-prompting, while two fixed the code manually.Source: §5.2.1 · p. 19

5.2.2 Blocks Help Writers Organize LLM Conversations Around Specific Tutorial Sections.

• All 16 participants created multiple blocks, each pairing tutorial content with the LLM conversation that generated it, to scope conversations to specific parts of the tutorial.When returning to earlier work, participants could directly access the chat that produced the content. Eleven participants used splitting to decompose long LLM outputs into smaller units.Source: §5.2.2 · pp. 19–20
• 12 of 16 participants agreed or strongly agreed that blocks were useful for organizing tutorial content and splitting or reorganizing long LLM outputs.Participants described blocks as helping them locate section-specific conversations and keep separate work contexts from mixing.Source: §5.2.2 · pp. 19–20, §5.2.3 · p. 20

5.2.3 Context Manager Gives Writers Explicit Control Over Prompt Context.

• Writers used context controls to exclude conversation history and select portions of external reference material, but manual selection added friction.Some forgot to add context and received responses that missed their intended context; five participants selected specific documentation portions for prompts.Source: §5.2.3 · pp. 20–21

6.1 Creating Software Tutorials in the Age of LLM-Mediated Documentation

• Tutorial writers should remain engaged with generated content, since LLMs do not have intentions or relate to learners’ actual learning experiences.The discussion notes that generated content can also contain security vulnerabilities.Source: §6.1 · p. 21

6.2 Designing LLM-Integrated Tools for SE Workflows

• For SE tools, the discussion recommends scoping LLM interactions to artifact structure, making context inspectable and selectable, and placing verification next to generation.The authors argue that these requirements may apply across SE authoring workflows, while calling for investigation of how scoping varies with artifacts and user preferences.Source: §6.2 · pp. 21–22

6.3 Using Human-Centered Design Methods to Create Tools for SE

• Human-centered design surfaced evolving requirements and showed why working prototypes with real LLM output matter: writers rejected a scratchpad because it hid how content fit together.The authors recommend human-centered design methods for SE tools whose practitioners’ strategies are still forming and vary across individuals and organizations.Source: §6.3 · p. 22

6.4 Study Limitations

• Reported confidence reflects writers’ perception, not tutorial quality; code execution confirms code runs but does not validate surrounding prose.The authors call for independent quality evaluation through expert review or reader-comprehension studies.Source: §6.4 · pp. 22–23
• Each condition lasted 15 minutes, participants knew their preferred tools, and dBlocks received only brief instruction; the authors say these constraints affected conditions differently.They argue the shorter familiarity period and time limit likely disadvantaged dBlocks, making its observed confidence difference more notable.Source: §6.4 · p. 23

7 Conclusion

• Interviews with 17 technical writers informed requirements for LLM-assisted tutorial creation, and human-centered design guided dBlocks’ iterative development.The paper argues that the requirements may generalize beyond tutorials to other LLM-assisted SE tasks such as programming.Source: §7 · p. 23
• The evaluation found that dBlocks’ features let writers engage more deeply with tutorial creation and improved their reported confidence.The authors describe documentation work in practice, longer projects, and organizational constraints as settings where these shifts remain to be tested.Source: §7 · p. 23
• The paper presents its study as a starting point for investigating whether its findings hold across longer projects and organizational constraints.It encourages future work to build on the requirements when designing tools for SE tasks.Source: §7 · p. 23