Source-linked AI summary
How to Prompt? Opportunities and Challenges of Zero- and Few-Shot Learning for Human-AI Interaction in Creative Applications of Generative Models
Hai Dang, Lukas Mecke, Florian Lehmann, Sven Goller, Daniel Buschek
TL;DR
Generative models can create high-fidelity content, but users often struggle to control them and write effective prompts. This paper analyzes prompting as a human-AI interaction paradigm, using brainstorming and creative-writing design exploration to propose interface goals and sketches. It concludes that adequate interfaces are needed to realize prompting’s potential for ad-hoc creative tools, expression, inspiration, and feedback.
Problem
Generative models are difficult for users to control, while effective prompt writing remains largely trial and error and lacks systematic HCI interface research.
Method
The paper analyzes prompting opportunities and challenges through two brainstorming sessions with five HCI researchers and illustrates proposed interface directions for creative writing.
Results
The analysis proposes four design goals for user interfaces supporting zero- and few-shot prompting and illustrates them with concrete creative-writing UI sketches.
Takeaways & Limitations
Adequate prompting interfaces could support ad-hoc creative-tool creation, creative expressiveness, inspiration, and feedback without requiring task-specific model retraining.
Takeaways & Limitations
Prompting interfaces must address unresolved challenges in representing custom tools, supporting iterative exploration, and avoiding strict hidden rules that make natural-language interaction resemble coding.
Abstract
from arXiv · showhide
Deep generative models have the potential to fundamentally change the way we create high-fidelity digital content but are often hard to control. Prompting a generative model is a promising recent development that in principle enables end-users to creatively leverage zero-shot and few-shot learning to assign new tasks to an AI ad-hoc, simply by writing them down. However, for the majority of end-users writing effective prompts is currently largely a trial and error process. To address this, we discuss the key opportunities and challenges for interactive creative applications that use prompting as a new paradigm for Human-AI interaction. Based on our analysis, we propose four design goals for user interfaces that support prompting. We illustrate these with concrete UI design sketches, focusing on the use case of creative writing. The research community in HCI and AI can take these as starting points to develop adequate user interfaces for models capable of zero- and few-shot learning.
1 INTRODUCTION
Deep generative systems can create high-fidelity media and support dynamic human-AI creative collaboration, but controlling their outputs remains difficult. Prompting offers ad-hoc task specification, yet effective prompting remains largely trial and error and lacks systematic HCI guidance.
- Deep generative systems synthesize high-fidelity media and may enable more collaborative, dynamic co-creative work between humans and AI.
- Natural-language prompts can steer generative systems without task-specific fine-tuning, using zero-, one-, or few-shot examples.Zero-shot prompts provide no examples; one-shot and few-shot prompts provide one or more examples.
- For end users, writing effective prompts is largely a trial-and-error process influenced by factors such as word order.
- The paper addresses the lack of systematic HCI research by identifying prompting opportunities and challenges and proposing four design goals for supporting interfaces.
2 RELATED WORK
Prior work studies interactive prompting tools and prompt-engineering strategies, but practical guidance remains fragmented. The paper positions its contribution around interfaces that help non-technical users formulate, combine, and apply prompts without task-specific model training.
- The paper highlights prompt composition and combination as a design direction for reducing task complexity in interactive applications.
- Interactive systems help users construct prompts with few-shot examples, templates, or sequences of subtasks.
- Prompt engineering systematically constructs prompts to improve generative-model output, including changes to word order, context, and task instructions.
- Other approaches generate or optimize prompt alternatives algorithmically, including neighboring prompts, task-trigger words, and non-word context tokens.
- Prompt-style input/output can also support fine-tuning, but this differs from using prompts for ad-hoc end-user-specified tasks without task-specific training.
3 METHOD
The authors used two one-hour brainstorming sessions with five HCI researchers to explore prompting opportunities, challenges, and interactive creative-writing support. They clustered ideas, reviewed existing systems, and identified a three-phase creative-writing process for a support tool.
- Two one-hour brainstorming sessions with five HCI researchers collected speculative opportunities and challenges surrounding prompt design for creative expression.
- The first session gathered individual ideas, clustered similar proposals, and identified overarching topics through discussion.
- The second session reviewed interactive systems and creative-writing goals to develop scenarios for prompt-writing support.
- The researchers identified a three-phase creative-writing process that an ideal prompt-writing support tool should cover at minimum.
4 OPPORTUNITIES
The paper identifies prompting opportunities for ad-hoc tool creation, expanded creative expression, and inspiration or feedback. These opportunities include reusable zero- and few-shot tools, cross-form or cross-modal transformation, and rapid exploration of creative variations.
- 4.1 End-User Programming of Creative Tools: Prompting can let users define new tools at runtime without retraining the model, including through natural-language declarations or examples.
- 4.1 End-User Programming of Creative Tools: Users may build reusable libraries of generative tools through zero-shot declarations or few-shot programming by example.
- 4.1 End-User Programming of Creative Tools: Prompting can support verbal delegation in human-AI tasks, positioning AI as an active contributor rather than a predefined assistant or toolbox.
- 4.2 Extending and Augmenting Creative Expressiveness: Prompts can express vague artistic concepts and transform content across styles, formats, and modalities, including text-to-image generation and novel-to-screenplay conversion.
- 4.3 Providing Inspiration and Feedback: Prompts may help overcome creative blocks, simulate feedback, optimize drafts, and explore multiple text variations through high-level directions and examples.
5 CHALLENGES
The paper identifies challenges that limit prompting as a practical Human-AI interaction paradigm, spanning guidance, representation, iteration, usability, access, portability, and bias. These challenges motivate interfaces that make prompting more discoverable, expressive, efficient, and adaptable.
- 5.1 Trial and Error / Lack of Guidance: Prompt design remains largely trial and error, with few discovered strategies translated into dedicated interfaces for non-technical users.Existing work includes elaboration, trigger-word optimization, word-order changes, and prompt collections, but practical support remains limited.
- 5.2 Representation of Tasks and Effects / Prompt IO: Prompt-based tools may need representations beyond default icons, along with explanations that help users understand or remember their functions.The paper identifies discoverability and explanation as open interface requirements for custom AI tools.
- 5.2 Representation of Tasks and Effects / Prompt IO: Current prompts emphasize verbal thinking, while creative practices also rely on visuals, tacit knowledge, and skills that are difficult to incorporate or visualize.The paper leaves open how images should enter prompts or how prompts should be integrated into graphical interfaces.
- 5.2 Representation of Tasks and Effects / Prompt IO: Prompt interfaces often model execution as one-time runs, although creative work requires exploring multiple potential outcomes and input-output mappings.The paper identifies iterative exploration as an unresolved interface challenge.
- 5.2 Representation of Tasks and Effects / Prompt IO: Strict structural formats or hidden rules can make creative prompt writing resemble code rather than natural-language interaction.Without adequate interaction and interface support, natural language may not reduce the burden of controlling generative systems.
- 5.3 Performance, Access, and Generalization: Large-system delays may prevent quick iterations, while access requirements can create gatekeeping for practitioners outside large companies.The paper also notes that prompt performance may not generalize across models, potentially creating system lock-in, and that generated content may reproduce language-model biases.
6 DESIGNING USER INTERFACES THAT SUPPORT PROMPTING
The paper proposes four design goals for prompting interfaces, illustrated through creative-writing examples that help users formulate, combine, apply, and represent prompts. These designs aim to make prompting more accessible while accommodating uncertainty, reuse, computational delay, and evaluation.
- Overview: Four design goals organize interfaces that support prompting, with creative writing as the primary use case and a starting point for broader future work.The goals were identified from literature and a design session; individual designs may address multiple goals.
- 6.1 Supporting Users in Formulating Prompts: Automatic parsing can convert natural-language input into editable task parameters and a refined model prompt, while predefined building blocks offer a more manual formulation path.The automatic approach may be limited by how many elements the system recognizes; free entry remains supported in the building-block approach.
- 6.2 Supporting Users in Combining Prompts: Prompt-combination interfaces can display multiple story lines, let users select outputs as context, and support parallel exploration before choosing a direction.A separate prompt view, similar to computational notebooks, is another way to organize multiple prompts.
- 6.3 Supporting Users in Applying Prompts: Reusable prompt tools can be saved in a toolbar and applied to selected text, with generated output presented through an interaction flow.The example uses a symbol for a stored transformation and a dialog for the resulting output.
- 6.2 Supporting Users in Combining Prompts: Narrative-tree interfaces support prompt exploration by showing alternative responses and carrying selected responses forward as context for subsequent prompts.This organization helps users explore different directions before settling on a final storyline.
- 6.3 Supporting Users in Applying Prompts: Asynchronous prompt execution lets writers continue working elsewhere while computationally costly calls generate or extend text.The interface can notify users when results are ready, helping preserve writing flow during delays.
- 6.4 Representing Prompts in User Interfaces: Prompt views can be separated from text views so users switch between editorial work and evaluating generated output.Prompts may also appear as symbols or annotations attached to text selections or paragraphs.
7 CONCLUSION
The conclusion frames prompting as a promising interaction paradigm for creative human-AI work, with opportunities for end-user programming, expression, inspiration, and feedback. It argues that adequate interfaces are needed to realize these possibilities and identifies the presented designs as starting points for future HCI-AI research.
- Opportunities: Prompting may support end-user programming of creative tools, creative expressiveness, and inspiration and feedback in human-AI creative work.The paper presents these as opportunities motivating further investigation of prompting.
- Opportunities: Prompting may democratise creative activities by enabling people without specific training to create personalised tools and work more creatively with text.The conclusion presents this as a potential benefit rather than an established outcome.
- Interface directions: Prompt interfaces could support asynchronous collaboration by allowing users to continue working while computationally costly language-model prompts run.The paper illustrates this with prompts inserted into documents to extend text in the background.
- Interface directions: Separating prompt and text views can help users switch between editorial and evaluative tasks when working with generated text.The conclusion’s interface examples include a split presentation modeled on markdown, LaTeX, or code editors.
- Future work: Realizing prompting’s creative potential requires adequate interfaces and continued research at the intersection of HCI and AI.The paper positions its design directions as starting points for that future work.