PaperlayerView exact revision on arXiv

Source-linked AI summary

"What It Wants Me To Say": Bridging the Abstraction Gap Between End-User Programmers and Code-Generating Large Language Models

Michael Xieyang Liu, Advait Sarkar, Carina Negreanu, Ben Zorn, Jack Williams, Neil Toronto, Andrew D. Gordon

arXiv:2304.06597v1cs.HC

TL;DR

Non-expert end-user programmers face an abstraction-matching problem because only some natural-language utterances reliably guide code-generating models. The paper proposes grounded abstraction matching, which systematically translates generated code back into predictable utterances, and finds that this approach improves users’ coping strategies, mental models, confidence, and trust when using the system.

  • Problem

    Non-expert end-user programmers struggle to formulate natural-language intents that code-generating models can reliably translate because they lack knowledge of the underlying code generator and available APIs.

  • Method

    The paper proposes grounded abstraction matching, systematically generating grounded utterances from translated data-analysis code and showing them to spreadsheet users.

  • Results

    The grounded approach improves end-users’ strategies for coping with model failures and, over time, produces more consistent mental models, greater confidence, and greater perceived trust.

  • Takeaways & Limitations

    Grounded examples can help non-expert users learn effective utterances and reason about, recover from, and edit failures in natural-language programming systems.

  • Takeaways & Limitations

    The system generates grounded utterances for only a subset of Pandas/Python APIs and assumes a single well-defined relational data table.

Abstract

from arXiv · show

Code-generating large language models translate natural language into code. However, only a small portion of the infinite space of naturalistic utterances is effective at guiding code generation. For non-expert end-user programmers, learning this is the challenge of abstraction matching. We examine this challenge in the specific context of data analysis in spreadsheets, in a system that maps the users natural language query to Python code using the Codex generator, executes the code, and shows the result. We propose grounded abstraction matching, which bridges the abstraction gap by translating the code back into a systematic and predictable naturalistic utterance. In a between-subjects, think-aloud study (n=24), we compare grounded abstraction matching to an ungrounded alternative based on previously established query framing principles. We find that the grounded approach improves end-users' understanding of the scope and capabilities of the code-generating model, and the kind of language needed to use it effectively.

1 INTRODUCTION

The paper frames abstraction matching as the challenge of expressing a well-formed intent at the level of detail and language an LLM can reliably translate into code. It proposes grounded abstraction matching, which maps a user utterance to an action and back into an editable, systematic utterance that helps users learn effective language.

  • The abstraction matching problem: Abstraction matching asks how users select natural-language utterances that reliably map well-formed intents to satisfactory system actions.The challenge involves matching utterance granularity, detail, vocabulary, and grammar to the system’s available actions.
  • The abstraction matching problem: Existing remedies include examples, tutorials, restricted vocabularies, and interface elements, but they limit generalization, require learning time, or reduce flexibility.The paper motivates an alternative because examples may not reflect users’ interests or support wider utterance generalization.
  • Grounded abstraction matching: Grounded abstraction matching maps a naturalistic utterance to a system action and back into an editable example that consistently invokes that action.The returned utterance grounds the abstraction level at which the system expresses its solutions.
  • Grounded abstraction matching: In the spreadsheet setting, users’ utterances are translated into data-analysis code, then into consistent grounded utterances that users observe, edit, and learn from.The interaction loop is illustrated with extracting the year from date strings.
  • Study setting and contributions: The study focuses on spreadsheet data analysis because non-expert programmers lack formal programming training and may not know the code generators or APIs needed to formulate reliable queries.The paper reports a user study with n=24 comparing grounded and ungrounded abstraction matching techniques.
  • Study setting and contributions: The grounded approach improves users’ recovery from system failures and increases their confidence and sense of control when using the system.The paper also discusses tensions between prompt language and explanation language, including differences between natural-language system dialects.

2 BACKGROUND AND RELATED WORK

Related work positions abstraction matching within broader natural-language interaction barriers, including forming intents, selecting actions, and operating tools. Prior LLM research shows that broader language interpretation can make effective utterances more difficult for end users to learn, while technical approaches mainly target model performance rather than user-facing interaction.

  • Natural language interfaces: Natural-language interfaces impose multiple barriers, including forming an achievable intent, selecting an action, and operating the selected tool correctly.Abstraction matching is distinct from earlier intent-formation and tool-selection problems.
  • Natural language interfaces: Programming languages constrain syntax and APIs, whereas natural-language interfaces require users to discover how their wording maps onto available system actions.The paper distinguishes abstraction matching from the later use and coordination barriers involved in operating tools.
  • Natural language interfaces: The paper explicitly leaves conversational breakdowns, turn-taking, and self-repair outside its scope.These issues emerge when natural-language interaction is conceptualized as a longer dialogue.
  • Natural language programming: LLMs broaden the range of naturalistic utterances they can interpret, but this makes the subset of utterances that reliably control them more difficult and unpredictable for end users to learn.The paper calls this condition fuzzy abstraction matching.
  • Technical and design strategies: Prompt engineering modifies user text to improve downstream model performance, but it is not intended as a user-facing interaction-design solution.The paper distinguishes technical prompt strategies from approaches that help end users formulate queries.
  • Natural language programming: Earlier natural-language programming systems addressed interaction through clarification questions, error feedback, intent tests, naturalized commands, and guided step-by-step generation.These approaches frame conversion from natural language to code as an interactive dialogue.

3 GROUNDED ABSTRACTION MATCHING

Grounded abstraction matching systematically converts generated code into an equivalent, editable natural-language command tied to the user’s actual intent. The design is intended to teach a predictable command space without directly addressing intent formation, model explanation, or broader transparency questions.

  • Core design: Grounded abstraction matching translates the system action generated from a query back into a naturalistic utterance that consistently invokes the same action.The utterance is grounded both in the actual generated code and in the user’s original intent.
  • Core design: Unlike ungrounded tutorial examples, grounded utterances have a structured correspondence to system actions and can help users infer how related commands work.Ungrounded examples may be disconnected from both the system’s capabilities and the user’s own intent.
  • Core design: The approach carves out a predictable space of utterances without forcing users to use only those utterances.Utterances that match system capabilities are described as more likely to produce the desired result.
  • Scope and distinctions: The paper does not directly address how users form well-formed intents, decompose problems, or compare natural-language programming with spreadsheet alternatives.The grounded approach may help with intent understanding indirectly, but these are not its explicit targets.
  • Scope and distinctions: Grounded utterances are not model explanations: they function as equivalent commands, rather than explaining why or how the model generated its output.The design also does not cover broader explainable-AI concerns such as confidence, data, process, or alternative outputs.

4 SYSTEM DESIGN AND IMPLEMENTATION

The systems translate spreadsheet queries into Python, execute the generated code, display results, and support subsequent query updates. System I additionally translates generated code into editable grounded utterances, generated through type-directed program analysis, while the prototype’s grounded utterances were equivalent to outputs about 85% of the time.

  • Ungrounded comparison: System II replaces grounded utterances with support based on established prompting practices, including problem decomposition and output constraints.The two systems otherwise share the same code-generation and execution pipeline.
  • Shared pipeline: Both systems combine a user query and spreadsheet table into a Codex prompt, execute generated Python, and display resulting columns, rows, tables, or values.The shared pipeline also lets users revise queries and submit a new prompt.
  • Grounded interaction: System I represents generated Python as editable grounded steps that users can revise and resubmit as new queries.The interface shows the result alongside a grounded utterance representing the generated code.
  • Grounded utterance generation: Grounded utterances are generated by translating Python syntax into a task-centric representation that uses types to resolve identifiers and select context-sensitive language.The representation retains algorithmic detail while reducing ambiguity in the concrete Python code.
  • Round-trip stability: Approximately 85% of outputs were equivalent in both the 126-query synthetic dataset and the 191-query study dataset where grounded utterances could be generated.The authors describe this stability as sufficient for evaluating grounded abstraction matching, while noting room for improvement.
  • Design limitations: The grounded utterance language remained an open design space, with the prototype sufficient for concept evaluation but requiring further exploration.The authors explicitly identify the utterance design as an area for future work.

5 USER STUDY: HOW DOES GROUNDING AFFECT ABSTRACTION MATCHING?

The study used a between-subjects design to examine how grounding affects query formulation, recovery from failures, and users’ perceptions of code-generating systems. Participants completed three spreadsheet tasks while researchers analyzed query episodes, think-aloud data, interviews, telemetry, and questionnaire responses.

  • Study aims: The study examined system failures, query rewrites, and how grounding affects perceived utility, confidence, trust, and mental models.These questions were investigated through participant queries, think-aloud data, and questionnaire responses.
  • Participants: Twenty-four participants with varied spreadsheet and programming experience were recruited across 11 industries.The sample included participants with basic and advanced spreadsheet experience, as well as differing formula expertise.
  • Experimental design: Participants were stratified and randomly assigned to grounded or ungrounded conditions in a balanced between-subjects design.The groups were similar in gender and prior spreadsheet, formula, and programming expertise.
  • Tasks and procedure: Participants completed three think-aloud tasks based on real-world spreadsheet questions, each using a roughly 25-row data table.The tasks involved counting Super Bowl wins, calculating average mission duration, and checking multiple house criteria.
  • Tasks and procedure: Task completion was judged by participants’ own reports, and experimenters intervened only to help recover from implementation bugs.Some apparent successes included slight task misinterpretations and were still counted as successes.
  • Data segmentation and analysis: Query episodes captured each cycle of submitting a query, inspecting output, and deciding how to proceed.The transcripts were segmented into episodes and augmented with queries, generated code, outputs, grounded utterances, and errors.
  • Data segmentation and analysis: Failure modes and rewrite strategies were coded through iterative open coding, independent coding, and negotiated resolution of disagreements.The final analysis used a codebook developed from an independently coded sample and then applied to the full episode set.

6 RESULTS

Grounded and ungrounded participants completed tasks with similar time, query counts, cognitive load, and standardized usability, but grounding changed how they interpreted failures and rewrote queries. Grounded utterances supported debugging, partial-answer recognition, scope adjustment, and language restructuring.

  • Task completion and queries: All participants completed every task, with no significant difference in completion time between grounded and ungrounded conditions.Grounded participants averaged 32 minutes 38 seconds versus 27 minutes 51 seconds ungrounded; the difference was not significant (t(X) = 1.592, p = 0.1256).
  • Task completion and queries: Participants issued similar numbers of queries in both conditions, and the difference was not statistically significant.The grounded approach could increase query counts without necessarily harming performance because qualitative strategies differed.
  • Usability and cognitive load questionnaires: Participants reported low-to-moderate cognitive load, high perceived performance, and high usability across both systems, with no significant quantitative condition differences.The authors interpret this result alongside high success rates and similar attempt counts, while noting qualitative differences in recovery strategies and mental models.
  • Model failures: The analysis identified twelve model-failure types grouped into technical, input, output, and logic failures, which could co-occur within one query episode.Technical failures were uncommon, occurring in 55 query episodes (18.8% of the total). Output failures included extra, missing, or improperly overwritten columns, while logic failures involved incorrect algorithms or partial answers.
  • Query rewriting: Participants used sixteen rewriting strategies across scoping, elaboration, language restructuring, and intent shaping, with code-like and system-like language improving model output.The grounded condition enabled system-breakdown and system-like-language restructuring strategies, which were unavailable in the ungrounded condition.
  • Rewriting strategies differ between grounded and ungrounded: Grounded utterances helped participants recognize partial answers, choose achievable command granularity, and evaluate when to adapt queries or start over.The overall code-frequency distributions differed significantly between conditions (χ2(13) = 70.3, p = 7.1 · 10^-7).

7 LIMITATIONS

The study’s limitations constrain the generality of its grounded-utterance findings across APIs, interaction designs, datasets, durations, and user populations.

  • Scope and implementation: Grounded utterances covered only a frequency-selected subset of Pandas/Python APIs, and the system assumed a single well-defined relational data table.The study reported 13 explanation failures across 159 grounded-condition queries and leaves multiple-table handling for future work.
  • Alternative designs: The comparison used reframing principles as the ungrounded alternative, although examples, tutorials, and other interface designs could produce different results.The authors also chose an interface that did not require users to encounter information beyond the text in the interface.
  • Interaction design: The prototype restricted responses to rewriting and resubmitting queries, excluding data-oriented feedback and model-parameter controls from the failure-recovery design space.This restriction supported close inspection of language use but does not represent the full range of possible decision-support loops.
  • Data scale: Study tables contained 20–30 rows, leaving large datasets—where row-by-row output verification becomes impossible—for future investigation.Larger datasets would also have increased debugging complexity and potentially confounded the grounding comparison.
  • Duration: A lab study cannot establish whether observed advantages persist, erode, or compound over days, weeks, or months of use.The authors propose diary studies or experience sampling for longitudinal validation.
  • Participants: Many participants had prior formula or programming expertise, although the interaction avoided direct code inspection or authoring and may generalize to some non-programmers.The sample represented spreadsheet users who write formulas rather than all spreadsheet users.

8 DISCUSSION

The discussion connects grounded utterances to learning, failure recovery, and design trade-offs while identifying risks from model behavior, individualized query styles, and vendor-specific dialects.

  • Tutorial value: Grounded utterances can be reused on future tasks and can teach canonical ways to express intent, similar to example code in API documentation.Participants described them as useful for retaining keywords and expressing desired operations in English.
  • Tutorial value: Grounded utterances can support logical thinking and problem decomposition because their step-by-step structure reflects primitive API calls in generated code.The authors also connect this value to LLMs’ ability to translate reasonably specified intents into logical code.
  • Tutorial value: Although LLM-generated solutions may be inefficient or encode poor practices, editable grounded steps helped users reason about failures and correct model mistakes.The discussion identifies alternative visualizations as a direction for future educational tools.
  • Genres of naturalistic commanding: A single grounded-utterance exposure could quickly change users’ subsequent grammar and vocabulary, while individuals developed distinct querying styles despite similar success rates.The resulting styles may support flexible interaction but can hinder collaboration when queries serve as spreadsheet documentation.
  • Genres of naturalistic commanding: Different systems may generate different naturalistic-commanding styles, creating potential vendor-specific dialect confusion across spreadsheet and database software.The authors identify shared norms and standards as a possible future response.
  • Implications for design: Grounded abstraction matching is presented as a general technique, demonstrated here with short Python data-analysis programs using Pandas.The discussion also points to spreadsheet presentation scripting, Vega-Lite, voice assistants, chatbots, and search engines as possible settings.
  • Implications for design: The paper frames grounded abstraction matching as an early exploration rather than a straightforward prescription for interface design.The authors caution that direct design implications are not always appropriate.
  • Implications for design: Grounded utterances may simultaneously function as tutorial examples, input interpretation, and explanations of system behavior.The authors suggest testing whether feedback can serve this “double duty” without overwhelming spreadsheet users.

9 CONCLUSION

The paper defines grounded abstraction matching as systematic generation of effective natural-language examples and evaluates it for spreadsheet data analysis by non-expert end-user programmers.

  • Conclusion: Abstraction matching is the core challenge of selecting natural-language utterances likely to be understood correctly by a system.The paper situates this challenge across natural-language interfaces.
  • Conclusion: Grounded abstraction matching systematically generates and shows grounded examples of effective utterances, instantiated in a spreadsheet data-analysis system.The approach is designed for non-expert end-user programmers.
  • Conclusion: Compared with an ungrounded alternative, grounded interaction expanded users’ strategies for coping with model failures and was associated with more consistent mental models, confidence, and perceived trust over time.The conclusion identifies dialogue and non-spreadsheet contexts as directions for future work.

B SYSTEM IMPLEMENTATION DETAILS

The implementation translates generated Python through intermediate representations into grounded utterances, then evaluates round-trip stability alongside the formative and main user-study pipeline.

  • Utterance generation: The utterance-generation algorithm separates Python-to-task representation translation, explanation-tree construction, and final utterance layout.Its parameterization supports either a single sentence or a sequence of utterances.
  • Utterance generation: The algorithm operates over syntax and API constructs rather than coding style, supporting single- or multi-line programs.The implementation begins with a Python abstract syntax tree and transforms successive tree representations.
  • Supported operations: The supported Pandas subset covers basic data wrangling, including indexing, vectorized operators, variable declarations, dataframe methods, aggregations, and series methods.The implementation also handles selected string operations and related dataframe or series transformations.
  • Supported operations: The implementation uses labeled tuple types so outputs such as dataframe shape can be verbalized with meaningful element labels such as rows and columns.This labeling informs utterances generated for subsequent subscript operations.
  • Round-trip evaluation: The benchmark submits a table and utterance, generates code and output, derives a grounded utterance when possible, then resubmits it to compare results.The two passes are denoted C1/O1 and C2/O2 in the benchmark description.
  • Round-trip evaluation: Round-trip stability is measured by code-generation equality and output equivalence, evaluated on synthetic data and participants’ study queries.Output equivalence ignores column-name differences, and Table 11 summarizes the results.
  • Formative study: The formative study explored spreadsheet data-analysis challenges with an early prototype that generated executable Python using Codex and inserted results into the spreadsheet.Its minimal interface consisted of a query box and displayed execution results.
  • Formative study: The formative sample comprised five self-reported spreadsheet formula users, one with prior C++ experience, recruited through UserTesting.com.The authors do not claim that this sample represents all users.

C.3 Results

Participants valued natural-language spreadsheet queries as an efficient alternative to developing formulas, but unexpected outputs often left them without a clear way to recover.

  • Participants found natural-language queries more efficient than developing formulas or searching online when the system worked.
  • Unexpected or obviously incorrect outputs created a gap between participants’ natural-language intent and the spreadsheet results.

C.3.1 It was hard for participants to recover from system errors.

Participants struggled to recover from system errors because they lacked reliable mental models of the system’s capabilities and effective query language.

  • Participants reformulated their intent twice on average after errors before abandoning the system.
  • Participants had poor notions of what the system could do and how to phrase queries reliably, even after repeated use.
  • Participants wanted guidance about effective utterances, grammar, vocabulary, column names, and specificity.
  • Participants struggled to generalize successful queries because they could not identify which wording, abstraction, operations, or spreadsheet elements mattered.
  • Participants distrusted unstable natural-language results compared with familiar spreadsheet formulas that produced anticipated outcomes.

C.3.3 Participants expressed a lack of trust towards system results.

The exploratory study connected abstraction matching with error recovery, mental-model, and trust problems, motivating grounded abstraction matching.

  • The exploratory study provided evidence of abstraction-matching, error-recovery, mental-model, and trust issues in this application context.
  • The study’s tasks involved spreadsheet data about cities, astronauts, and houses.

D.2 Task data

The study materials included task datasets, failure examples, and examples of query-rewriting strategies for spreadsheet analysis.

  • The study included full data tables for Super Bowl, astronaut, and house tasks.
  • The materials documented examples of input, output, and logic failures encountered by participants.
  • The materials presented examples of scoping, elaboration, language-restructuring, and intent-shaping changes.
  • Intent-shaping examples included queries about counting New Orleans wins and creating a per-mission space-flight column.
Loading 2304.06597v1…