PaperlayerView exact revision on arXiv

Source-linked AI summary

Direct Manipulation and Natural Language Programming, Together at Last?

Parker Ziegler, David Minh-Duy Cao, Justin Lubin, Sarah E. Chasins

arXiv:2608.26359v1cs.PLcs.HC

TL;DR

Natural language programming often treats programs as text, leaving open how editing paradigms can be unified. This paper introduces a shared structured-edit framework for direct manipulation and natural language, then evaluates it in cartokitDM+NL; participants overwhelmingly preferred direct manipulation, using natural language for 6.14% of edits.

  • Problem

    Natural language programming largely reinforces text-based program editing, motivating unified systems that combine it with structure-aware approaches.

  • Method

    The paper models programs as sequences of structured edits, using an edit language shared by direct manipulation and natural language through constrained decoding in cartokitDM+NL.

  • Results

    Participants overwhelmingly favored direct manipulation, performing just 6.14% of edits through natural language in the within-subjects study.

  • Takeaways & Limitations

    Direct manipulation scaffolded task decomposition and incremental editing while helping participants reason about how program-output changes corresponded to program transformations.

  • Takeaways & Limitations

    Because the system was evaluated with geospatial-visualization practitioners, the findings may not generalize to other domains and may depend on mature direct-manipulation tools.

Abstract

from arXiv · show

Decades of programming languages research has contributed novel approaches to program editing that go beyond modifying text, including direct manipulation programming, structure editing, and automated refactoring tools. However, the rapid growth of natural language programming largely reinforces a view of programs as text and program editing as (unstructured) text transformation. How can we develop unified programming systems that bridge the gap between these approaches, supporting multiple editing paradigms in concert? We take a first step toward answering these questions by introducing a framework that enables program editing via both direct manipulation and natural language, and instantiate this framework in a variant of the $\texttt{cartokit}$ direct manipulation programming system. Our key insight is to treat programs as sequences of structured edits and to use an edit language as a shared interface for both direct manipulation and natural language interactions, leveraging constrained decoding to support the latter. Using our instantiation, we conducted a within-subjects study ($N$=18) to understand how the combination of direct manipulation and natural language as editing modalities changes the programming process compared to each modality alone. Perhaps surprisingly, we found that study participants overwhelmingly chose to edit via direct manipulation when both modalities were available, performing just 6.14% of edits via natural language. Our thematic analysis of study sessions revealed that direct manipulation aided task decomposition, encouraged incremental editing, and helped mitigate known challenges in natural language programming related to understanding model capabilities and model-generated code. Our edit-based framework and study findings lay out a possible pathway for future research on programming systems that blend natural language with alternative editing modalities.

1 Introduction

The paper proposes unifying direct manipulation and natural language programming through structured edit sequences, using an edit language as their shared interface. In a cartokit-based system and within-subjects study, participants overwhelmingly preferred direct manipulation when both modalities were available.

  • Motivation: The paper addresses how programming systems can integrate direct manipulation and natural language editing, which currently rely on different views of programs.Prior editing systems exploit program structure, whereas natural language programming largely treats programs as unstructured text.
  • Structured edits: Programs are modeled as sequences of structured edits applied to a starting expression, giving edits precise syntax and semantics.Edit languages define how programs and outputs evolve at fine granularity.
  • Unified framework: The framework uses structured diffs as a shared edit language: direct manipulation generates diffs, while an LLM translates natural language prompts into diff sequences.The resulting diffs can be interpreted using cartokit’s existing diff semantics.
  • Study: The authors instantiate the framework in cartokitDM+NL and evaluate unified direct manipulation and natural language programming in a within-subjects study with N=18.The study compares the combined system with direct manipulation and natural language programming alone.
  • Findings: 6.14% of edits used natural language, indicating that participants overwhelmingly favored direct manipulation when both modalities were available.Direct manipulation scaffolded task decomposition and incremental editing, while natural language was used mainly for repetitive edits or guidance.

2 Background

Patch-recon synchronizes program text and program output through structured diffs. Its design supports incremental, efficient, and correctness-preserving updates while leaving the representation of diffs open to different editing systems.

  • Patch-recon: Patch-recon handles direct manipulation interactions by synchronizing updates to both the program and its output.A user interaction produces a diff that the system uses to modify the program and output.
  • Patch-recon: The patch operation takes a diff and current program P and incrementally updates P to P′.Its type is Diffs_L × Prog_L → Prog_L.
  • Patch-recon: The recon function takes a diff and current output value and incrementally updates that output.Its type is Diffs_L × Val_L → Val_L.
  • Correctness: A proof of patch-reconciliation correspondence establishes that semantic output updates emulate syntactic program updates.This supports parallel incremental updates while keeping program and output aligned.
  • Diffs: Patch-recon is agnostic about diff representation, which may encode AST operations, user actions, CRDT operations, POSIX diffs, or other structured program information.The framework does not prescribe what information a diff must contain.
  • Constrained decoding: Constrained decoding restricts language-model token sampling so generated sequences can be completed into valid outputs satisfying specified constraints.Completion analysis checks whether a selected token can lead to a valid completion and backtracks when necessary.

3 Unifying Direct Manipulation and Natural Language via Program Edits

The paper extends systems with structured edits by translating natural language prompts into constrained sequences of diffs. Because both modalities produce the same edit representation, natural language edits can reuse direct manipulation infrastructure and its incremental update properties.

  • Extension: A system with diffs, patch, recon, and patch-reconciliation correspondence can be extended to support natural language program editing.The extension replaces generation of unstructured program text with generation of structured diff sequences.
  • Constrained generation: Constrained decoding enforces that each generated diff belongs to the system’s valid diff language.The formal decoder output is a sequence of diffs satisfying the system’s constraints Φ.
  • Shared interface: Diffs serve as a shared language that tools can generate from either direct manipulation interactions or natural language prompts.This shared interface allows the two editing modalities to converge on the same program-editing pipeline.
  • Consequences: Natural language edits inherit patch-recon’s incremental application and correctness properties because they produce the same diffs as direct manipulation.Model latency can still affect the overall performance of natural language-generated diffs.

4 Implementation

The authors implemented cartokitDM+NL by adding constrained natural language diff generation to cartokit while reusing cartokit’s patch-recon infrastructure. The implementation used GPT-5 and added 780 lines of code to a roughly 13,000-line codebase.

  • Implementation: cartokitDM+NL adds natural language editing to cartokit’s geospatial visualization programming system.The implementation uses constrained decoding and JSON Schema encodings for cartokit’s diffs.
  • Implementation: 780 LOC were added to a ≈13,000 LOC cartokit codebase, including 625 TypeScript lines for constrained decoding and 155 lines for interface components.The constrained decoding implementation used Structured Outputs in the OpenAI API with GPT-5.
  • Infrastructure reuse: GPT-5-generated diff sequences reuse cartokit’s existing patch-recon processing, matching diffs dispatched by direct manipulation interactions.This preserves incremental updates, deterministic JavaScript generation, and incremental undo/redo for natural language edits.

5 Experiment Design

The study examined three programming conditions with 18 geospatial data users, combining quantitative task measures, telemetry, workload ratings, and qualitative session analysis. Participants completed reproduction and exploration tasks under counterbalanced condition orders.

  • Study Setup: The within-subjects study involved 18 geospatial data users programming interactive maps with DM+NL, DM-only, and NL-only tools.The conditions were cartokitDM+NL, cartokit, and GitHub Copilot with GPT-5.
  • Study Setup: Researchers measured task completion, time-on-task, and telemetry quantitatively, while using reflexive thematic analysis to assess editing behavior and correctness assessment.The study also collected post-task NASA-TLX workload ratings and conducted semi-structured post-study interviews.
  • Tasks and Procedure: The reproduction task required participants to recreate a published Washington Post map in JavaScript using supplied instructions, an image, and GeoJSON data.Participants had to execute their code in StackBlitz to complete the task.
  • Tasks and Procedure: The exploration task asked participants to use one or both supplied geospatial datasets to author a JavaScript program producing a map they considered desirable.Participants could end the task early by declaring success or abandoning it.
  • Measures: Overall workload was estimated with Raw TLX, the unweighted mean of six NASA-TLX subscale ratings after each task.The subscales were Performance, Effort, Frustration, Mental Demand, Temporal Demand, and Physical Demand.

6 Quantitative Results

Quantitative results compared task completion, timing, and perceived workload across direct-manipulation, combined, and natural-language conditions. Direct manipulation matched the combined condition on completion and workload measures while natural-language-only performance was weaker on reproduction tasks.

  • Task Completion and Timing: 100% of participants completed the Reproduction task in both DM+NL and DM conditions, compared with 50% using GitHub Copilot.Nine of 18 participants completed the NL condition, while six abandoned it before the 30-minute threshold.
  • Task Completion and Timing: The median Reproduction completion time was 7m11s for DM+NL and 5m17s for DM, while the NL estimate was >30m with 95% CI = [24m25s, >30m).The NL estimate required survival analysis because completion was right censored at 50%.
  • Task Completion and Timing: Participants were 4m23s slower with DM+NL than DM, 11m9s faster than GitHub Copilot, and 19m47s faster with DM than GitHub Copilot.The reported 95% CIs were [-1m58s, 7m36s], [-21m45s, -10m40s], and [-22m40s, -12m35s], respectively.
  • NASA Task Load Index: On Reproduction tasks, mean Raw TLX scores were 4.86 (SE ±0.40) for DM+NL, 4.33±0.51 for DM, and 10.02±0.71 for GitHub Copilot.DM+NL and DM workload scores were not statistically different, whereas DM+NL scores were markedly lower than NL scores.
  • NASA Task Load Index: Participants’ DM+NL Raw TLX scores were not statistically different from DM scores for either task type, but were markedly lower than NL scores.The authors suggest direct manipulation may have helped alleviate perceived workload associated with natural-language-only editing.

7 Qualitative Results

Participants used the GUI to scaffold edits from high-abstraction choices toward localized styling changes, while direct manipulation’s incremental feedback supported program understanding. In contrast, large natural-language edits were harder to verify and could introduce errors, regressions, or loss of progress.

  • GUI scaffolding: Participants began with high-abstraction GUI edits and progressively moved toward low-abstraction styling adjustments.High-abstraction edits changed structures such as layer types and encoding channels, while low-abstraction edits adjusted colors, opacities, and symbol-size ranges.
  • GUI scaffolding: 14 participants navigated the layer-editing GUI sequentially on Reproduction tasks, treating each control as a distinct editing choice.On Exploration tasks, participants used top-level Layer Type controls to transition quickly between larger program changes.
  • Natural-language risks: Large natural-language prompts could produce regressions, remove desired functionality, or introduce runtime errors, undermining participants’ sense of control.One prompt caused an out-of-memory exception, and follow-up prompting produced a legend without a map.
  • Incremental editing: Direct manipulation provided live code updates that participants described as helpful for learning and understanding program behavior.Participants valued seeing code update as they changed values through the interface.
  • Incremental editing: Direct manipulation’s discrete, incremental edits helped participants identify which generated JavaScript locations changed after each interaction.This continuity supported reasoning about how output changes corresponded to program changes.
  • Incremental editing: Participants tried to incrementalize natural-language programming by manually slicing generated code or requesting only small snippets.Manual slicing enabled incremental verification but introduced errors when participants misidentified slices or dependencies.

7.2 What Caused Participants to Switch between Direct Manipulation and Natural Language When Both Were Available?

When both modalities were available, participants primarily used direct manipulation and switched to natural language selectively for automation, replay, or ambitious edits. They also returned to direct manipulation when natural-language failures or ambiguous changes undermined trust and control.

  • DM Only: Several participants used direct manipulation exclusively because prompting felt unnecessary when they already knew how to achieve their goal through the GUI.This occurred in both Reproduction and Exploration tasks.
  • DM→NL: Some participants used natural language to automate repetitive edits or replay symbology adjustments on new data.This included treating the LLM like a parameterized record-and-replay or programming-by-demonstration system.
  • DM→NL: Participants were especially likely to prompt when direct manipulation made repetitive work tedious, such as matching color bars across election-data layers.Participants reported little motivation to prompt when the clickable interface already made an edit easy.
  • DM→NL: Some participants used natural language for “moonshot” prompts involving complex geospatial analyses they presumed direct manipulation could not perform.These prompts targeted operations associated with desktop GISs, spatial databases, or Python libraries.
  • NL→DM: Many participants abandoned the LLM after failures such as unknown diffs or timeouts, with early failures strongly shaping perceived capability and utility.Direct manipulation often offered an easier route to progress when the model failed.
  • NL→DM: Participants turned to direct manipulation to correct or repair partial or undesired diffs produced by ambiguous natural-language prompts.Direct manipulation served as a rapid way to fix model missteps after partial success.

7.3 Experience with Direct Manipulation Editing Helped Mitigate Known Challenges with Natural Language Editing

Experience with direct manipulation helped participants structure prompts, infer model capabilities, and interpret natural-language edits. The GUI supplied both decomposition boundaries and technical vocabulary, while its visible controls helped participants match model behavior to intended changes.

  • Prompt decomposition: Participants who first used cartokit often decomposed GitHub Copilot prompts along cartokit’s abstraction boundaries.Prompts referenced layer types, encoding channels, and classification methods.
  • Prompt decomposition: One participant’s prompt followed the GUI’s sequence of choosing a layer type, data property, color scheme, and classification method.The prompt’s structure mapped almost directly to prior direct-manipulation interactions.
  • Prompt decomposition: The GUI provided decomposition scaffolding and cartographic terminology that participants reused in natural-language prompts.Three of four near-novices who completed the Reproduction task with GitHub Copilot had first seen cartokit or cartokitDM+NL.
  • Model capability: Participants used direct-manipulation features to infer which program edits they could ask the LLM to make.They were uncertain about what the model could see and which interface components received prompts.
  • Model capability: Keeping layer controls visible gave participants a reference for what was possible and helped bridge ambiguity between users and models.The interface effectively announced a language of interaction through its available controls.
  • Interpreting model output: Participants used the GUI to understand which edits the LLM had applied after a prompt changed the map.This supported abstraction matching, or learning the model’s rules through discovery.

7.4 How Did Participants Assess Correctness?

Participants primarily assessed correctness through visible outputs, but the two systems constrained what errors could occur differently. Direct manipulation supported program inspection and visible maps, whereas natural-language generation could produce invisible outputs, verbose code, or plausible but unintended edits.

  • Assessing correctness: Output examination was participants’ primary proxy for correctness in all conditions, but it was not always sufficient.The study examined visual outputs alongside participants’ interactions with generated programs.
  • Soundness guarantees: The DM+NL and DM diff systems prevented invalid programs and states in which the map was not visible.Participants therefore did not enter an obvious debugging phase or express concerns about generated-code correctness in those systems.
  • Assessing correctness: Unexpected direct-manipulation outputs prompted some participants to inspect the underlying program more closely.One participant investigated why proportional symbols appeared uniformly sized despite expecting variation.
  • Natural-language limitations: In the NL condition, more than half of participants encountered generated programs that produced no visible output, often because of runtime or data-reference errors.Near-novices often lacked a concrete output with which to communicate the problem to the model.
  • Natural-language limitations: More than half of participants said the volume of generated code made it harder to assess whether requested functionality had been implemented.Large-scale changes also made true errors harder to localize in unfamiliar programs.
  • Natural-language limitations: Natural-language edits added uncertainty about the scope and completeness of model-triggered changes, even when the updated map looked plausible.One participant failed to notice that the model selected votes_total instead of the requested pct_dem_lead property.

8 Limitations and Threats to Validity

The paper’s validity is bounded by rapidly changing coding-agent capabilities, task selection designed around human participation, domain specificity, and a participant population experienced with direct-manipulation tools.

  • 8.1 Emergence of Coding Agents: Coding-agent results may age quickly as new models and tools emerge, so the authors focus analysis on programming paradigms rather than particular systems.They expect paradigm-level findings to remain relevant as models evolve.
  • 8.1 Emergence of Coding Agents: Claude Code completed two of three Reproduction tasks unaided, but needed author guidance for the third because its browser setup could not adequately render WebGL.Completion times were 6m36s, 11m30s, and 20m36s, excluding time spent authoring the initial prompt.
  • 8.1 Emergence of Coding Agents: The study selected tasks achievable with every tool while still requiring human participation, so different tools would warrant a different task set.The authors speculate that similarly selected future studies may produce comparable findings, but this remains untested.
  • 8.1 Emergence of Coding Agents: Better models may reduce regressions or invisible outputs without resolving programmers’ difficulty interpreting the scope, bounds, or desirability of agent-triggered edits.The authors identify blending agent code generation with direct manipulation as an open research opportunity.
  • 8.2 Domain Specificity: The system was evaluated with geospatial practitioners, limiting generalization beyond geospatial visualization and possibly other domains with mature direct-manipulation tools.The authors note this domain specialization is shared by prior direct-manipulation programming systems.
  • 8.3 Participant Preferences: The 18 participants’ stronger GIS and vector-graphics experience than LLM-programming experience may have contributed to their preference for direct manipulation.The authors caution that findings may not extend to communities embedded in different tooling ecosystems; prior LLM experience was not related to Reproduction performance.

9 Related Work

Related work spans multimodal interfaces, constrained code generation, program-editing systems, programming by demonstration, and program comprehension, while this paper addresses an unstudied combination of direct manipulation and natural-language editing.

  • 9.1 Multimodal Programming Systems: No prior programming system or user study had combined multimodal direct manipulation with natural-language editing before this work.The paper presents its study as the first empirical investigation of this combination.
  • 9.2 User Studies of Direct Manipulation Programming: Direct-manipulation programming has produced systems for SVG, HTML, games, and geospatial visualization, but lacked user-study evidence before this paper.The authors position their results as initial evidence for presumed scaffolding benefits.
  • 9.3 Multimodal Editing Interfaces: Earlier multimodal systems used natural language to generate interface widgets or edit templates for visualization, web styling, vector graphics, and animation.DynaVis, for example, generated dynamic widgets from prompts using constrained HTML and JavaScript templates.
  • 9.4 Constrained Decoding: Constrained decoding research improves syntactic and semantic correctness, but generally targets full program synthesis rather than structured program-edit generation.The paper treats these approaches as complementary because diffs form an edit language.
  • 9.5 LLMs for Program Editing: LLM-based repair, refactoring, migration, structure editing, live programming, and edit prediction systems generally emit unstructured host-language token sequences.This distinguishes the paper’s structured-edit approach from many existing program-editing tools.
  • 9.6 Related Perspectives: Participants’ natural-language replay of repetitive tasks connects the system to programming-by-demonstration, while GUI exploration connects it to program-comprehension research.The paper relates traces of repeated actions to generalization and GUI use to successive top-down refinement of program hypotheses.

10 Discussion and Conclusion

The discussion argues that shared structured edits can support complementary editing modalities while preserving direct-manipulation stability and extending incremental performance benefits to natural-language editing.

  • 10.1 Architecture: The edit-language and patch-reconciliation architecture provides one possible foundation for multimodal programming systems, with two highlighted areas of strength.The authors present it as one architecture among many possible designs.
  • 10.1 Architecture: Patch-reconciliation correspondence prevented invalid programs in DM and DM+NL, reducing debugging time and keeping direct manipulation functional after LLM edits.The latter benefit held regardless of which edits the LLM dispatched.
  • 10.1 Architecture: Allowing an LLM to rewrite programs arbitrarily could introduce constructs without GUI mappings or violate invariants required by existing direct-manipulation functionality.The authors suspect this could cause progressive degradation of the editing environment.
  • 10.2 Performance: Integrating natural-language edits with patch-recon through constrained decoding transfers incremental performance benefits to natural-language editing with minimal overhead.The stated overhead is network latency and model inference time.
  • 10.3 Natural Language Programming: The results complicate claims that natural-language programming necessarily requires less prior knowledge or creates a low learning barrier for end users.The paper calls for deeper study of these motivations for natural-language interfaces.
  • 10.3 Direct Manipulation and Natural Language Programming, Together at Last?: The study provides initial empirical evidence that direct manipulation can scaffold programming and supports treating direct manipulation and natural language as complementary rather than oppositional.The proposed design direction centers on a shared language of structured edits.

Data-Availability Statement

The authors provide cartokitDM+NL as open-source software and make a production deployment available for use.

  • Data-Availability Statement: cartokitDM+NL is open source and freely available, with a production deployment also provided online.Researchers may request access to natural-language editing or build the system using their own OpenAI API key.
Loading 2608.26359v1…