CommSketch: How Speaking while Sketching Steers Human--AI Design Ideation

1 Introduction

• Rough, incomplete sketches leave AI to infer context that may be needed to understand what marks represent, how a design should work, and why features matter.Those inferences can diverge from intent, requiring designers to explain, correct, or reframe ideas.Source: §1 · p. 2
• Prompts, reference images, and later corrections can clarify intent, but often separate communication from sketching or follow an AI misunderstanding.Source: §1 · p. 2
• Concurrent speech could make intent available as a sketch develops, including designers’ evolving understanding of what marks mean.Unlike a prompt before or after drawing, speech can accompany continued sketching.Source: §1 · p. 2
• Because speech can include tentative or abandoned ideas, its role must be examined through how human and AI contributions unfold.Source: §1 · p. 2
• The study addresses how speaking while sketching steers ideation and shapes perceptions of creativity support and human–AI collaboration.It compares AI grounding in sketches and concurrent speech with sketches alone.Source: §1 · p. 2
• The authors report a 24-participant comparative study and an empirical account of speech’s role in intent expression, alignment, and guiding AI contributions.They also identify design opportunities around dynamic alignment, multimodal expression, and co-creativity.Source: §1 · p. 2

2.1 Multimodal Intent Communication in Human–AI Interaction

• Multimodal systems draw on speech, gestures, interface actions, and visible artefacts, while sketch-based ideation presents emerging visual forms and intent together.The challenge is to make emerging meaning available without losing the openness of sketching.Source: §2.1 · pp. 2–3

2.2 AI-Supported Early-Stage Design Ideation

• Generative design tools support prompt-based creation and refinement, while research prototypes also let designers compose inputs or refine generated images through design dimensions.Sketch-centered systems such as Inkspire and SketchAI preserve sketching in the generative workflow.Source: §2.2 · p. 3

2.3 Speaking While Sketching in Human–AI Ideation

• Sketches represent form and spatial organisation, while speech can express identities, functions, rationales, intended experiences, and contextual requirements.Speech during sketching may be tentative, fragmented, self-directed, exploratory, or revised rather than planned as an AI command.Source: §2.3 · p. 3
• Concurrent sketch-and-speech systems have explored interactive authoring and intent expression; Shi et al. found speech significantly improved alignment between generated images and stated intentions.Earlier work also reports friction when designers must interrupt sketching to formulate lengthy prompts, or when AI lacks awareness of evolving activity.Source: §2.3 · p. 3
• This study examines how concurrent speech relates to participants’ subsequent responses and the development of design ideas using a functioning AI system.Source: §2.3 · p. 3

3.1 Study Design

• In a between-subjects study, Sketch-Only grounded AI interpretations in sketches, while CommSketch also used concurrent speech chosen freely by participants.No think-aloud protocol was imposed.Source: §3.1 · p. 4

3.2.1 Low-Friction Human–AI Ideation.

• AI support was embedded in the sketching environment so designers could move between developing ideas and engaging with the assistant without reformulating ideas as standalone prompts.Source: §3.2.1 · p. 4

3.2.2 Seamless Capture of Speaking While Sketching.

• CommSketch recorded and transcribed speech in the background during canvas activity, with a visible peripheral recording indicator.Recording paused during chatbot interaction and resumed on return to the canvas.Source: §3.2.2 · p. 4

3.2.3 Inspectable AI Understanding of Design.

• Opening the chatbot triggered an inspectable interpretation based on the sketch alone or the sketch and speech transcript, which participants could confirm, correct, question, or build upon.Interpretation occurred at user-triggered checkpoints, not continuously while sketching.Source: §3.2.3 · p. 4, §3.3.2 · p. 4

3.2.4 Generative Support for Idea Development.

• The chatbot supported text and image generation, and participants could discuss generated outcomes or import images onto the canvas for further development.Source: §3.2.4 · p. 4, §3.3.2 · p. 5

3.3 Prototype

• Both conditions used the same canvas, chatbot, generative functions, and workflow; CommSketch additionally included speech transcripts in AI interpretation.Participants could switch between canvas and chatbot, with speech capture resuming on return to the canvas.Source: §3.3 · p. 4, §3.3.3 · p. 5

3.4 Participants

• Twenty-four participants were randomly assigned to Sketch-Only or CommSketch, with 12 in each condition.Participants had prior design-ideation experience through coursework or professional practice.Source: §3.4 · p. 5

3.5 User Study

• Participants freely ideated toaster designs for 30 minutes, then completed questionnaires and a retrospective interview within an approximately one-hour session.They chose when and what to say in CommSketch.Source: §3.5 · p. 5

3.6 Data Collection and Analysis

• The analysis combined participant-level questionnaire comparisons, interaction-sequence coding, and directed qualitative content analysis of interviews.Source: §3.6 · p. 6

3.6.1 Questionnaires.

• Seven-point questionnaires measured six creativity-support items and seven human–AI collaboration items, with participant-level conditions compared using two-sided Mann–Whitney U tests.Holm correction was applied separately to the two item families, using an adjusted significance threshold of pHolm < .05.Source: §3.6.1 · p. 6

3.6.2 Human–AI Interaction Sequences.

• Interaction coding examined expressed design intent, responses to AI interpretations, and how human and AI contributions shaped participant-confirmed ideas.Intent coding covered sketches and annotations in Sketch-Only and additionally concurrent speech in CommSketch.Source: §3.6.2 · p. 7

3.6.3 Interviews and Directed Qualitative Content Analysis.

• Researchers analysed retrospective interviews across conditions to examine intent expression, communication, visualisations, and responses to AI understanding or misunderstanding.Two researchers independently coded the interviews; Cohen’s κ for interview analysis was .71.Source: §3.6.3 · pp. 7–8

4.1 Creativity Support: Natural Expression of Design Intent and Efficient Visualisation

• Expressiveness, Enjoyment, and Results Worth Effort had higher median ratings in CommSketch (δ= .56, .56, and .50), but none of the six items remained significant after Holm correction.These were the three largest observed effects among the six adapted creativity-support items.Source: §4.1 · p. 8, Table 4 · p. 9
• Experience-related intentions appeared in 9 of 12 CommSketch participants versus 2 of 12 Sketch-Only participants, while form appeared in all 12 and function in 12 versus 10.Concurrent speech offered another channel for expressing qualities such as portability, ease of use, safety, and humour alongside the sketch.Source: §4.1.1 · p. 8, Figure 5 · p. 9
• Eight CommSketch participants described speech as natural or synchronous, or as reducing typing, separate explanations, or later recall; three Sketch-Only participants described coordinating sketching with separate explanations and chatbot interaction.Participants also reported that speaking while sketching let them express functional details and reasoning as they emerged.Source: §4.1.2 · pp. 9–10, §4.1.1 · p. 9
• All 12 CommSketch participants and 11 of 12 Sketch-Only participants described a useful role for AI visualisation, while additional effort could still be needed to refine sketches, explain intent, or correct mismatches.Six CommSketch participants described turning rough or simple sketches into usable representations; participants in Sketch-Only could instead direct effort toward making sketches or prompts explicit.Source: §4.1.3 · p. 10

4.2 Human–AI Collaboration: Establishing a Shared Design Premise for Alignment and Co-evolution

• Alignment was higher in CommSketch and remained significant after Holm correction (δ= .69, pHolm = .028); Communication and Controllability did not remain significant.Table 5 compares human–AI collaboration ratings; Communication and Controllability had higher medians in CommSketch, but neither survived correction.Source: §4.2 · p. 10, Table 5 · p. 11
• In Sketch-Only, unclear sketches sometimes led AI to misidentify the basic design premise, prompting participants to restate what the object was before interaction could proceed.Three Sketch-Only participants and one CommSketch participant recommended that AI clarify an ambiguous object, concept, or function before proceeding.Source: §4.2.1 · p. 11

4.2.2 Alignment: From Repairing AI Interpretations to Act-

• Reframe was most frequent for 25.0% of Sketch-Only participants and none in CommSketch, while Execute was most frequent for 50.0% in CommSketch versus 16.7% in Sketch-Only.Sketch-Only reframing often repaired or redirected misunderstandings; more CommSketch participants acted on an interpretation rather than repairing it.Source: §4.2.2 · p. 11
• Participants in both conditions directed execution and selectively retained AI contributions, while CommSketch examples also show participants inviting suggestions and incorporating them as ideas evolved.Co-evolution was the most frequent Creative Agency pattern for 4 of 12 CommSketch participants (33.3%) versus 1 of 12 Sketch-Only participants (8.3%); this describes idea development, not perceived controllability.Source: §4.2.3 · p. 12

5.3 Supporting Human–AI Co-creativity through Speech in Design Ideation

• CommSketch examples show designers inviting AI suggestions and directing their incorporation, while Sketch-Only examples show human-led correction before visualization.Figure 11 contrasts C6’s co-evolution pattern with S10’s human-led agency pattern.Source: Figure 11 · p. 13, §5.3 · p. 14

5.1 Toward Dynamic Alignment as Design Intent Evolves

• Participants reported higher perceived alignment in CommSketch (pHolm = .028), and examples suggest appropriate interpretations gave subsequent AI contributions a more useful basis.The authors frame alignment as developing through interaction as designers inspect and correct interpretations.Source: §5.1 · p. 13
• Future work could connect speech to canvas edits and regions, letting designers inspect active intentions and helping AI determine when to update, clarify, or leave ambiguity unresolved.Erasure alone need not indicate rejection; the same idea may be redrawn or only one feature reconsidered.Source: §5.1 · p. 13

5.2 Supporting Multimodal Expression of Emerging Design Intent

• CommSketch participants expressed experience-related intentions such as portability, ease of use, and safety, while median Expressiveness ratings were higher but not significant after Holm correction (pHolm = .097).Participants also articulated functional details and reasoning alongside rough visual forms.Source: §5.2 · pp. 13–14
• Speech accommodated tentative exploration as well as communication of formed ideas, motivating interfaces that accept incomplete or hesitant expression without immediate commitment.The toaster task elicited readily verbalised use qualities, though some experiences may remain difficult to put into words.Source: §5.2 · p. 14
• Spatial and embodied design could extend expression beyond speech and sketch to gestures and bodily actions that convey spatial-temporal experience.The discussion calls for studying which modality combinations support intentions across design tasks, including tentative alternatives.Source: §5.2 · p. 14

5.3 Supporting Human–AI Co-creativity through Speech in Design Ideation

• Designers may need to shape AI’s role—from implementing specified features to proposing possibilities—and select suggestions to retain and incorporate into revisions.The authors propose evaluating how such exchanges affect exploration and the creativity of resulting designs.Source: §5.3 · p. 14
• Speech may let designers invite AI input before their intentions have settled into a fully specified design.A participant described requesting generation or opinions before finishing a thought; expression and alignment provide additional context for this account.Source: §5.3 · p. 14

6 Limitations and Future Work

• The 24-person sample, predominantly students with design coursework or training, may limit precision and generalisability.Further studies should include users with varied design expertise and familiarity with AI.Source: §6 · p. 14
• The simple individual toaster task may not capture longer or more complex projects involving trade-offs such as portability, capacity, and manufacturing cost.Collaborative sessions could also examine negotiating a shared direction, requiring speaker and contribution attribution.Source: §6 · p. 14
• The implementation combined the current sketch with an accumulated transcript but did not link speech to strokes or edits, limiting tracking of which intentions remain relevant as sketches change.Future work could associate utterances with sketch elements and revisions.Source: §6 · p. 14
• Transcribed speech may contain recognition errors and omits intonation and emphasis, whose effects on AI interpretation were not systematically evaluated.Direct audio input could test whether retaining vocal cues helps interpret emerging intentions.Source: §6 · p. 14
• The study did not systematically examine how speaking habits affect interaction or perceived benefits, including differences in comfort verbalising tentative ideas.Future work could assess whether switching between silent and spoken sketching better supports individual preferences.Source: §6 · pp. 14–15
• Practical adoption depends on integrating speech with existing design platforms and preserving its context when sketches are revised or transferred.Everyday workflow evaluation could assess whether integration reduces the effort of re-explaining ideas across tools and project stages.Source: §6 · p. 15

7 Conclusion

• In a study with 24 participants, concurrent speech made intended experiences and reasoning available alongside emerging visual forms and helped participants guide AI visualisations and contributions.The authors report support for perceived alignment and interaction examples of directing AI as ideas developed.Source: §7 · p. 15

A Interaction Coding Codebooks

• The codebooks separately address expressed design intent, the immediate response to AI interpretation, and development of the full idea.Examples are condensed descriptions of Results cases, not additional verbatim quotations.Source: A Interaction Coding Codebooks · p. 17

• Each available canvas submission within a participant-confirmed idea is coded for expressed design intentions, using sketches and annotations in Sketch-Only and those plus concurrent speech in CommSketch.Source: §A.1 · p. 17
• Form, function, and experience may be expressed through different or related evidence, and the categories are not mutually exclusive.Source: §A.1 · p. 17
• Experience requires evidence of intended use, sensory, or affective quality; functional detail or design rationale alone does not qualify.Source: §A.1 · p. 17
• Participant-level summaries count each participant once per category expressed across their available submissions.Source: §A.1 · p. 17

• A participant’s follow-up is read together with the preceding AI interpretation, and codes describe how the response acts on that interpretation.Source: §A.2 · p. 18
• Responses distinguish proceeding, reinforcing an intention, adding detail, repairing the premise, or inviting exploration, regardless of image-generation requests.Source: §A.2 · p. 18
• Later chatbot exchanges inform full-idea Creative Agency coding rather than the coding of an immediate follow-up.Source: §A.2 · p. 18

• Each participant-confirmed idea is assigned one Creative Agency pattern based on its canvas submissions, AI interpretations, participant responses, and generated outputs.Source: §A.3 · p. 19
• Classification follows development across the full idea sequence, not a single follow-up or who produced the final image.Source: §A.3 · p. 19
• Participant summaries use each person’s most frequent pattern and retain ties; Human-only and AI-led were not most frequent for any participant.Source: §A.3 · p. 19
• Human-led and Co-evolution examples illustrate the patterns, and all four patterns were retained in coding.Source: §A.3 · p. 19