Source-linked AI summary
AI-Instruments: Embodying Prompts as Instruments to Abstract & Reflect Graphical Interface Commands as General-Purpose Tools
Nathalie Riche, Anna Offenwanger, Frederic Gmeiner, David Brown, Hugo Romat, Michel Pahud, Nicolai Marquardt, Kori Inkpen, Ken Hinckley
TL;DR
Chat-based prompting makes it difficult to explore and refine ambiguous creative intent through linear exchanges. AI-Instruments reify prompts as manipulable objects, using reflection and grounding to support generation and steering. Four image-generation probes and qualitative reactions from 12 participants suggest the approach addresses interaction challenges, while multi-part artifacts remain an area for further research.
Problem
Chat-based prompting can make it difficult to explore and refine ambiguous intent or pursue alternative directions in creative work.
Source: §1 · p. 2Method
AI-Instruments embody prompts as reusable graphical objects and use reflection and grounding to support exploring alternatives and generating content from examples.
Source: §1 · p. 2, §4.1 · p. 6, §4.2 · p. 6Results
The authors report that initial perceptions from 12 content creators suggest the model's principles can inform the design of interaction techniques and assess existing ones.
Source: §8.1 · p. 15Takeaways & Limitations
The authors argue that reappropriating instrumental interaction can inform expressive AI-Instruments for intent expression, discovering possibilities, and steering generation.
Source: §9 · p. 16Applying instruments to artifacts composed of multiple content pieces requires additional research into how to integrate their different aspects.
Source: §8.2 · p. 16
Abstract
from arXiv · showhide
Chat-based prompts respond with verbose linear-sequential texts, making it difficult to explore and refine ambiguous intents, back up and reinterpret, or shift directions in creative AI-assisted design work. AI-Instruments instead embody "prompts" as interface objects via three key principles: (1) Reification of user-intent as reusable direct-manipulation instruments; (2) Reflection of multiple interpretations of ambiguous user-intents (Reflection-in-intent) as well as the range of AI-model responses (Reflection-in-response) to inform design "moves" towards a desired result; and (3) Grounding to instantiate an instrument from an example, result, or extrapolation directly from another instrument. Further, AI-Instruments leverage LLM's to suggest, vary, and refine new instruments, enabling a system that goes beyond hard-coded functionality by generating its own instrumental controls from content. We demonstrate four technology probes, applied to image generation, and qualitative insights from twelve participants, showing how AI-Instruments address challenges of intent formulation, steering via direct manipulation, and non-linear iterative workflows to reflect and resolve ambiguous intents.
1 Introduction
Chat-based prompting makes it difficult to formulate, refine, and steer ambiguous intents through nonlinear creative workflows. The paper adapts instrumental interaction into AI-Instruments, using reification, reflection, and grounding, and demonstrates four probes alongside initial reactions from 12 participants.
1 Introduction
- Chat-based prompting challenges users to formulate and disambiguate intent, steer model responses, and navigate nonlinear creative workflows.Users may also be unsure what they want to achieve or what AI can do.Source: §1 · p. 2
- AI-Instruments adapt instrumental interaction for generative AI through reification of intent, reflection on alternatives, and grounding in content or other instruments.Reflection covers both ambiguous user intent and alternative AI responses.Source: §1 · p. 2
- Four image-generation probes—Fragments, Transformative Lenses, Generative Containers, and Fillable Brushes—demonstrate complementary ways to create and compose instruments and content.Outputs from one instrument can become inputs to another, and instruments can be used to create meta-instruments.Source: §1 · p. 2
- The paper extends instrumental interaction to generative AI and reports initial reactions from 12 users comparing direct manipulation through AI-Instruments with textual prompting.The authors present the model as both a way to assess and inspire interaction techniques.Source: §1 · p. 2
2 Related Work
The related work frames five human-AI interaction challenges and reviews approaches spanning prompt support, direct manipulation, and creative practice. It motivates a general interaction model that extends instrumental interaction with principles of reflection and grounding.
2 Related Work
- Prior work addresses human-AI interaction challenges through techniques that often focus on particular problems, while this paper seeks a general model for direct manipulation across them.The challenges include intent formulation, disambiguation, resolution, steering, and workflow.Source: §2 · p. 3, §2.1 · p. 4
2.1 Human-AI Interaction
- Research on prompting and graphical interfaces addresses intent formulation, disambiguation, resolution, and steering, while chat-based iteration remains linear and trial-and-error.Existing work also explores non-linear workflows, including canvases and graph or tree representations.Source: §2.1 · p. 3
2.2 Interaction Models in the Era of AI
- The paper extends instrumental interaction for generative AI by adding reflection and grounding to its use of instruments as mediators between users and content.The resulting model is presented as a generative theory intended to inspire novel techniques.Source: §2.2 · p. 4
2.3 Content Generation and Creativity Support
- Creative design practices use ambiguity, alternatives, and iterative evaluation; reflection surfaces intent and possible results, while grounding extracts aspects of materials for reuse.Combining instruments also offers horizontal movement through composed tools and workflows.Source: §2.3 · p. 4
3 Example Walkthrough
The walkthrough illustrates how AI-Instruments support exploration and refinement through linked image-generation operations. One example explores bird styles and scene composition; another modifies a castle image through fragments, color variations, and local brush edits.
3 Example Walkthrough
- A generative container produces style variations from an image, then bird variations in a consistent style; a transformative lens adds and layers landscape content around the selected bird.The lens uses inpainting to generate a landscape around the bird.Source: §3 · pp. 4–6, Figure 2 · p. 5
- For the castle illustration, generated prompt fragments can be varied, color fragments reused, and a Fillable Brush applied to revise a selected image region.The brush prompt and painted region guide local changes.Source: Figure 3 · p. 5, §3 · p. 6
4 Instrumental Interaction with AI
AI-Instruments adapt instrumental interaction to generative AI by making user intent manipulable as graphical objects. Reification, reflection, and grounding support working with intent and generated content.
4 Instrumental Interaction with AI
- AI-Instruments adapt instrumental interaction by reifying user intent rather than fixed commands, with reification, reflection, and grounding as interconnected principles.The paper describes an AI-instrument as an AI-powered mediator between users and domain objects.Source: §4.1 · p. 6, §4 · p. 6
4.1 Reification of User Intent
- Reifying inputs and outputs as reusable graphical elements enables direct manipulation and movement between abstract goals and concrete changes.Users can articulate interaction phrases and use lasso selections to specify the scope of intent.Source: §4.1 · p. 6
4.2 Reflection
- Reflection surfaces facets of ambiguous intent and multiple interpretations of AI responses to help users steer generation toward a satisfying result.Fragments can help users refine, pivot on, or add aspects to intent; response alternatives can vary through model parameters, input variations, or different contexts.Source: §4.2 · p. 6
4.3 Grounding
- Grounding instantiates instruments from desired-outcome examples or other instruments, with AI segmentation helping users select and retain specific content aspects such as style.Instruments can also be grounded in other instruments to explore related instruments.Source: §4.3 · p. 7
5 Examples of AI-Instruments
The four AI-Instruments probes make prompts and content into manipulable objects, using reflection and grounding to support iterative image creation. Fragments, lenses, containers, brushes, and palettes offer complementary ways to vary intent, compose content, and generate or organize instruments.
5.1 Fragments
- Fragments turn prompts into [type, value] cards that users can vary, add, remove, and reuse across content as composable pieces of intent.On-demand suggestions expose value variations and new fragment types; adding or removing fragments regenerates the content.Source: §5.1 · p. 7
5.2 Transformative Lenses
- Transformative Lenses layer prompts and image content to complete, recombine, or regenerate images, while moving and layering elements supports iterative composition.Unlike undo, reverting a lens-content layering triggers regeneration and produces a slightly different rendering.Source: §5.2 · pp. 8–9
5.3 Generative Containers
- Generative Containers pair a prompt with a 2x2 grid of results, supporting reflection on response possibilities and variations grounded in examples or fragments.The current probe generates four variations per container and accepts one grounding example as input.Source: §5.3 · pp. 9–10
5.4 Fillable Brushes
- Fillable Brushes apply AI prompts to a scoped image region for style or content changes, and can be grounded in existing content to populate an editable prompt.Approximate brush strokes are converted into precise object selections using Segment Anything.Source: §5.4 · p. 10
5.5 Generated Instruments and Meta-Instruments
- Palettes store or generate collections of instruments and content, helping organize the elements created through instrument-on-instrument workflows.They can use collected content as examples or generative seeds and help users start from existing artifacts rather than a blank canvas.Source: §5.5 · p. 11
6 Implementation
The probes run on a web platform that coordinates language and image-generation models. Their implementation combines model-specific processing with modules that translate instrument actions into prompts and generated images.
6 Implementation
- The web-based system uses a JavaScript and HTML front end with fabric.js and a Node.js back end to manage content, files, and model communication.The sketched interface was chosen to focus study participants on concepts rather than surface details.Source: §6 · p. 11
- GPT-4o handles text transformations and image analysis, while Stable Diffusion with stacked ControlNet models generates images and preserves selected source content or style.The pipeline also uses masks, inpainting or outpainting, and parameters such as CFG scale and denoising strength.Source: §6 · p. 11
- Instrument modules decompose and vary prompts, compose lens inputs, generate container variations, and adjust brush outputs through ControlNet weights.For brushes, Segment Anything produces a mask from stroke control points; GPT-4o crafts a combined prompt before Stable Diffusion inpainting.Source: §6 · p. 12
7 Study
The study compared four image-generation and editing probes with chat prompting, using participant tasks and reflections to examine the interaction model and challenges it addressed.
7 Study
- The qualitative study compared four AI-instrument probes with a chat-based prompting probe through image-generation and editing tasks, then analyzed participant comments.The study examined perceptions of reification, reflection, and grounding as well as the interaction challenges addressed by each instrument.Source: §7 · p. 12, §7.3 · p. 13
7.1 Procedure
- Participants completed a 60-minute study, first using chat prompting and then completing two to three tasks per instrument after video demonstrations.All participants used the probes in the same order; the study aimed to gather insights on the overall interaction model rather than compare instruments.Source: §7.1 · pp. 12–13
7.2 Participants
- The study recruited 12 frequent generative-AI users from a large organization, with varied interests in content creation and none reporting audio or video generation.Source: §7.2 · p. 13
7.3 Material and Analysis
- Researchers coded 156 participant statements, including 28 that concerned technical-probe limitations, and collected preferences for five generic content tasks.Source: §7.3 · p. 13
7.4 Insights on Model Principles
- All 12 participants valued grounding (45 comments), reflection-in-intent (31), and reflection-in-response (31) as advantages over prompting.Participants also praised fine-grained selection and the ability to save and reuse brushes and lenses.Source: §7.4 · p. 13
- Reported trade-offs included chat’s accessibility and speed, reduced divergence with grounded lenses, lost edit history, and probe usability issues.Participants noted that 28 of 48 aspects they did not value concerned probe limitations rather than the interaction model.Source: §7.4 · p. 14
7.5 Challenges Addressed by AI-instruments
- Participants described instruments as helping formulate and disambiguate intent, explore alternatives, steer selected image regions, and support non-linear, iterative workflows.Brushes and lenses supported fine-grained edits; participants also favored different instruments for exploration and chaining prompts.Source: §7.5 · pp. 14–15
8 Discussion and Future Work
The discussion extends instrumental interaction for generative AI and considers how its principles might apply beyond image creation, alongside design challenges and open research needs.
8.1 Revisiting and Extending the Instrumental Interaction Model
- The authors extend reification to natural-language intents, reflection to both user intent and model response, and grounding to extracting and reusing intent, including through meta-instruments.Reflection reframes polymorphism using LLM concept-translation capabilities without hand-coding controls for every content type.Source: §8.1 · p. 15
- AI-generated instruments raise a design trade-off between unchecked workspace proliferation and concentrating capabilities in highly abstract instruments.The authors suggest composing and organizing instruments and iteratively refining content and meta-instruments.Source: §8.1 · p. 15
- The authors’ probe insights suggest the principles can inform new interaction techniques and help assess existing ones, while lower-level interaction choices can shape user experience.Specific interactions may differ across applications, modalities, and contexts.Source: §8.1 · p. 15
- Direct-manipulation reversibility must be considered alongside nondeterministic AI outputs; history and versioning are feasible, but their desirability may depend on users and use cases.Source: §8.1 · pp. 15–16
8.2 Beyond Images, Applying AI-Instruments to Other Forms of Content
- For text, the principles generally held in preliminary probes, but users may need help skimming and comparing many generated outputs.The authors propose bolding text or providing summaries or excerpts in Generative Containers.Source: §8.2 · p. 16
- Applying instruments to heterogeneous artifacts, such as slides combining titles and images, requires further research into integrating content at different scopes.The authors identify this and designing for heterogeneous content as future research directions.Source: §8.2 · p. 16
9 Conclusion
The paper operationalizes instrumental interaction for generative AI through reification, reflection, and grounding. It argues that this theory can inform expressive AI-Instruments and inspire further theoretical work in HCI+AI.
9 Conclusion
- Operationalizing reification, reflection, and grounding for generative AI is presented as a basis for expressive instruments that support intent expression, discovery, and steering.The authors argue these principles can drive a new generation of AI-Instruments.Source: §9 · p. 16
- Generative AI’s polymorphic and non-deterministic behavior can be leveraged by these instruments to create new expressive HCI+AI experiences.Source: §9 · p. 16
- The authors hope their operationalization of instrumental interaction will inspire new or re-appropriated theories for HCI+AI.Source: §9 · p. 16