DirectGPT: A Direct Manipulation Interface to Interact with Large Language Models

• Conversational LLM editing is slow and demanding because users must reference output elements precisely and choose wording around model limitations.Mistakes can be difficult to detect and persist in the conversation history.Source: §1 · p. 2
• DirectGPT localizes prompt effects by letting users interact with output objects instead of describing them precisely in language.Selected objects constrain the prompt’s area of effect, reducing ambiguity and preserving other content.Source: §1 · p. 2
• The interface provides continuous, manipulable output representations with highlighted changes and immediate feedback during edits.These mechanisms help users notice differences between generated iterations.Source: §1 · p. 2
• Previous prompts become reusable toolbar commands, while undo and redo support rapid, reversible trial-and-error.This replaces repeated prompt construction with templated snippets and recoverable operations.Source: §1 · p. 2
• 50% faster and 25% more successful task completion with DirectGPT, using 50% fewer and 72% shorter prompts than ChatGPT.The comparison covered text, code, and vector image editing tasks.Source: §1 · p. 2

2.1 What is Direct Manipulation?

• Direct manipulation combines continuous object representation, physical actions or labelled buttons, and rapid, incremental, reversible operations with immediate visibility.Source: §2.1 · p. 2

2.2 Five Issues With Prompting that Motivate the use of Direct Manipulation

• Prompting is limited by ambiguity, conversational verbosity, unpredictable effects, trial-and-error cost, and poor prompt reuse.Direct interfaces address these issues by referring to objects directly, enforcing constraints, enabling undo, and separating actions from objects.Source: §2.2 · pp. 2–3

2.3 Systems to Help Craft Better Prompts

• Prompt-engineering systems test, evaluate, vary, or chain prompts, whereas DirectGPT uses LLMs to manipulate objects such as text and images directly.Source: §2.3 · p. 3

2.4 Prompting through More Direct Interactions

• Existing generative-AI interfaces use predefined buttons, sliders, gestures, and physical metaphors, while DirectGPT dynamically generates toolbar commands from user interactions.Source: §2.4.1 · p. 3, §2.4.2 · p. 3

2.4.3 Reversible Operations Instead of Continuous Conversations.

• Traditional LLM interfaces lack explicit operation-level recovery, whereas DirectGPT applies conventional undo mechanisms to reverse edits.Source: §2.4.3 · p. 3

2.4.4 Immediate Feedback Instead of Word-by-Word Generation.

• DirectGPT uses spatial input to provide immediate loading feedback on elements about to be modified rather than relying only on word-by-word generation.Source: §2.4.4 · p. 3

2.5 Blending Language and Direct Manipulation

• Language and pointing have previously been combined for image editing, but earlier systems supported narrower command structures or task-specific pipelines.DirectGPT extends the combination to text, code, and image editing while covering selection, reuse, control, verification, and trial-and-error.Source: §2.5 · pp. 3–4

• DirectGPT is described through a use-case scenario followed by a detailed account of its direct-manipulation mechanisms and implementation.Source: §3 · p. 4

• Selecting words and prompting “synonym” replaces only those words, then creates a reusable synonym toolbar command for later edits.Source: §3.1 · p. 4
• Drag-and-drop references let users specify drawing locations and objects directly, replacing words such as “here” and “there” with visual references or coordinates.Source: §3.1 · p. 4, Figure 3 · p. 5
• The generated drawing is refined through reusable commands, clicks, and object references to add symmetric branches and circles with copied sizes.Source: §3.1 · p. 5, Figure 3 · p. 5

3.2 Direct Manipulation Principles for LLMs

• The interface combines direct manipulation with ordinary prompting to preserve expressive prompt-based interaction across text, code, and image-editing domains.Its design replaces a single prompt containing both verb and noun with complementary commands and object interactions.Source: §3.2 · p. 5, §3.2.2 · p. 5

3.2.1 Continuous Representation of the Last Output.

• DirectGPT preserves continuously represented text, code, and rendered images so users can inspect changes spatially and temporally.Source: §3.2.1 · p. 5, §3.2 · p. 5

3.2.2 Physical Actions Through Prompt-Object Interactions.

• Selecting objects localizes a prompt’s effect, while dragging objects into the prompt supplies unambiguous references without describing them in words.The system supports both noun-verb and verb-noun ordering and can constrain edits to selected objects.Source: §3.2.2 · pp. 5–6

3.2.3 Rapid Operations Through Prompts Reused as Tools.

• Executed prompts become toolbar tools that preserve the verb while allowing users to substitute one or multiple nouns through subsequent object selections.For example, a two-location line-drawing prompt can be reused by clicking the tool and then specifying each location.Source: §3.2.3 · p. 6, Figure 4 · p. 6

3.2.4 Immediate Targeted Feedback.

• Separating verbs from nouns lets DirectGPT show targeted loading pulses on selected or mentioned objects while a prompt executes.Users can verify whether the highlighted objects match their query and stop generation when they do not.Source: §3.2.4 · pp. 6–7

3.2.5 Reversible Operations Through Undo Mechanisms.

• DirectGPT provides undo and redo at the user’s operation granularity, including reverting a single command applied simultaneously to multiple elements.Source: §3.2.5 · p. 7

3.3 Implementation

• The prototype supports text, code, and rendered SVG images, using ReactJS, Prism.js, and the OpenAI API with “gpt-3.5-turbo”.Its source code and live demo are available.Source: §3.3 · p. 7, §3.3.2 · p. 7

3.3.1 Localizing the Effect of a Prompt.

• DirectGPT localizes edits by asking the LLM to rewrite only the selected passage and replacing that passage with the response.The full surrounding passage remains in the model context, with the selection represented as “<blank>”.Source: §3.3.1 · p. 7

3.3.2 Referring to Objects in Prompts.

• For text references, DirectGPT surrounds the selected text with unique delimiters so repeated identical content remains unambiguous to the engineered prompt.Source: §3.3.2 · p. 7
• For vector-image references, DirectGPT assigns unique IDs to SVG elements and substitutes those IDs for object-words in engineered prompts.Source: §3.3.2 · p. 7

4 USER STUDY: DIRECTGPT VS CHATGPT

• The study compared DirectGPT with ChatGPT for literary-text, JavaScript-code, and vector-image editing while measuring intent, control, recovery, and prompt reuse.Source: §4 · p. 7
• The ChatGPT baseline closely reproduced the original interface, prompting behavior, conversation context, rendering, and streaming responses.Both conditions used the same gpt-3.5-turbo model through the OpenAI Chat API.Source: §4 · p. 7

4.1 Participants and Apparatus

• Twelve participants aged 20–34, all experienced with programming and ChatGPT, completed the remote study.The sample included six self-identified women and six men.Source: §4.1 · pp. 7–8
• Image activities used starting drawings and four tasks: gradient colourization, adding elements, removing elements, and flipping the image upside down.Source: Figure 5 · p. 8

4.2 Procedure

• Each participant used both interfaces across text, code, and image activities, completing four editing tasks per activity after interface tutorials.Tasks were time-limited to three minutes and followed by a closeness-to-target rating, usability questionnaire, and interviews.Source: §4.2 · p. 8

4.3 Design and Tasks

• The study covered literary text, JavaScript, and vector images, with four tasks per content spanning localized edits and one global modification.Source: §4.3 · pp. 8–9

4.4 Data Analysis

• Participants rated 5-point statements for DirectGPT and ChatGPT; Figure 6 shows the responses, mean differences, and 95% confidence intervals.Source: Figure 6 · p. 9

4.3 Design and Tasks

• The within-subject design counterbalanced interface, activity, and content order using a Graeco-Latin square.Each participant received a unique ordering, with interface and content versions balanced across positions.Source: §4.3 · p. 9
• The study collected interaction logs, subjective ratings, and task measures from 2 interfaces × 3 activities × 4 tasks × 12 participants = 288 tasks.Source: §4.3 · p. 9

4.4 Data Analysis

• Analysis reported p-values and 95% confidence intervals for mean differences, using studentized bootstrap intervals and Wilcoxon signed-rank tests.Bootstrapped intervals used 10,000 replications.Source: §4.4 · p. 9

4.5.1 Closeness to target.

• DirectGPT increased closeness to the target by .95 points on a 5-point scale, from M=3.89 to M=4.84 (p<0.001).The effect was significant for text and vector images but not code.Source: §4.5.1 · p. 9

4.5.2 Time.

• 1min 1s shorter completion times favored DirectGPT overall, including a 46s advantage after excluding prompt-execution and output-generation time.The time-limit stopping rate was 8% with DirectGPT versus 35% with ChatGPT.Source: §4.5.2 · p. 9

4.5.3 Prompt Quantity.

• 1.77 fewer prompts per trial favored DirectGPT, with means of 1.90 versus 3.67 for ChatGPT (p<0.001).The reduction was significant for text, images, and code.Source: §4.5.3 · p. 9

4.5.4 Prompt Verbosity.

• 15.3 fewer words per prompt favored DirectGPT, with means of 5.83 versus 19.98 for ChatGPT (p<0.001).Prompt-length reductions occurred across text, code, and image activities.Source: §4.5.4 · p. 9

4.5.5 Usability.

• For the specific given tasks, DirectGPT scored 92 on the System Usability Scale versus 53 for ChatGPT, a mean difference of 39.Source: §4.5.5 · p. 9

4.5.6 Questionnaires.

• Participants rated DirectGPT higher on all seven usability statements, with the largest effect concerning ease of specifying the object to modify.Other gains covered output control, mental demand, mistake recovery, action clarity, action specification, and prompt reuse.Source: §4.5.6 · p. 10
• Participants used direct-manipulation mechanisms for 84% of DirectGPT prompts, including localization, object references, and reuse of tools generated from prior prompts.68% of prompts were localized, 14% referenced dropped objects, and 20% reused a previous prompt.Source: §4.5.7 · p. 10

5 DISCUSSION

• 50% faster completion, with 50% fewer and 72% shorter prompts, reflected clearer intent specification with DirectGPT.Participants unanimously found specifying the object to modify easy.Source: §5 · p. 10
• DirectGPT made output control clearer: participants produced results closer to targets and found controlling AI output easier.Participants also rated the effects of their actions as clearer on average.Source: §5 · p. 10
• Undo made error recovery easier, while reusable toolbar commands were used for 20% of prompts but showed the smallest comparative advantage.Half of participants also found prompt reuse easy with ChatGPT.Source: §5 · p. 10
• DirectGPT preserves prompting-only workflows, allowing users to switch between prompting and direct manipulation according to task and perceived benefit.All participants completed global tasks without direct selections.Source: §5 · p. 10

5.1 Observations and Participants’ Comments

• Participants used more concise, less conversational prompts with DirectGPT, which they said felt more like a tool than a chat.They attributed this to selecting specific parts and using concise prompts.Source: §5.1 · pp. 10–11
• DirectGPT’s benefits were largest for text and images, whereas code was similarly easy to reference in both interfaces because its structure supported precise descriptions.Participants still needed more time, prompts, and words with ChatGPT for code tasks.Source: §5.1 · p. 11
• Participants most frequently valued localizing the effect, followed by reusable prompts, dropping objects into prompts, and undoing.All participants reported a preference for DirectGPT.Source: §5.1 · p. 11

5.2 Limitation

• The findings may depend on the underlying LLM, participant expertise, task type, and interface design; exploratory tasks or less technical users may benefit differently.The study focused on editing tasks, and complex feedback may misrepresent which elements will ultimately change.Source: §5.2 · pp. 11–12

6 CONCLUSION

• DirectGPT localizes prompts, refers to objects, reuses commands, and provides undo operations for LLM-based editing.The approach is intended to integrate LLMs and other generative models into traditional software for co-creation.Source: §6 · p. 13
• Participants were faster, more successful, and preferred DirectGPT over ChatGPT for text, code, and image editing tasks.Source: §6 · p. 13

A.1.1 Localizing the Effect of a Prompt

• Physical selection of a text span generates an instruction that asks the LLM to rewrite only that selected part.DirectGPT reinserts the returned fragment into the overall text.Source: §A.1 · p. 15, §A.1.1 · p. 15

A.1.2 Referring to Textual Objects in a Prompt

• Dropping marked words into a prompt lets users request synonym replacements while specifying that the remaining text stay identical.The generated prompt uses delimiters to identify the selected words.Source: §A.1.2 · p. 15
• DirectGPT rewrites the whole text when the operation is intended to apply globally rather than only to selected fragments.The appendix contrasts localized rewriting with whole-text rewriting.Source: §A.1.2 · p. 15
• Delimiter choices are designed to keep marked text separate from surrounding tokens, with the appendix reporting slightly better empirical behavior than standard XML tags.Source: §A.1.2 · p. 15

A.1.3 Referring to Vector Objects in a Prompt

• The system uses dropped vector objects to replace pronouns such as “this” and “that” with references to specific SVG elements.The example drops two circles into a prompt asking for a line between them.Source: §A.1.3 · p. 16
• For vector edits, generated prompts identify target elements by unique IDs and request modified SVG code containing the desired change.DirectGPT ensures element IDs are unique, modifying them when necessary.Source: §A.1.3 · p. 16