SheetCopilot: Bringing Software Productivity to the Next Level through Large Language Models

1 Introduction

• Complex software control lacks a standardized model-application interaction framework and a comprehensive performance benchmark.Challenges include representing software interfaces in natural language, generating safe commands, and monitoring state changes and errors during multi-step tasks.Source: §1 · pp. 1–2
• Over 44.3% of benchmark tasks received completely correct SheetCopilot solutions, surpassing traditional programming approaches by a wide margin.The agent generated fully executable command sequences for 87.3% of problems.Source: §1 · p. 2
• SheetCopilot translates high-level natural-language spreadsheet requests into executable command sequences through a general model-software interaction framework.The agent decomposes requests into step-by-step plans and issues commands through the spreadsheet application.Source: §1 · p. 2
• The authors developed a benchmark suite of interactive spreadsheet tasks and an automated pipeline to evaluate model comprehension, planning, and software-interface execution.The evaluation resources are intended to support systematic assessment of model and interface performance.Source: §1 · p. 2

• Tool-augmented LLM systems generate action sequences that are parsed into tool API calls, while robot-planning work elicits mid-level step sequences [14, 3, 16, 15, 8, 25].Source: §2 · p. 2
• SheetCopilot plans atomic actions from sheet state, revises them using external action documentation, and executes extracted arguments in simulation.Source: Figure 1 · p. 3
• Spreadsheet language-interface studies focus on limited task subsets such as formula generation, while benchmarking general spreadsheet manipulation remains limited.Source: §2 · p. 3

3 Dataset and Evaluation

• The benchmark addresses limited task coverage and standardized evaluation in prior spreadsheet interfaces through a curated dataset and reproducible automated framework.The dataset compilation includes gathering, filtering, consolidating, adapting, and manually annotating tasks.Source: §3 · pp. 3–4

3.1 Diverse Seed Task and Workbench Collection

• 67 representative seed tasks span six spreadsheet categories, and 28 practical or hand-generated worksheets form the evaluation workbench.The categories are Entry and Manipulation, Management, Formatting, Charts, Pivot Tables, and Formulas; the dataset overview depicts diverse instructions and operations.Source: §3.1 · p. 4, Figure 2 · p. 4

3.2 Core Set Collection

• 221 tasks form the core set, combining adapted and simplified seed tasks with author-composed tasks and multiple reference solutions.GPT-4 generated 1669 adapted instructions; simplification reduced average token length from 47.1 to 33.81.Source: §3.2 · pp. 4–5

3.3 Task Evaluation by Execution

• A generated solution is correct when its executed spreadsheet matches the necessary properties of any candidate ground-truth sheet, rather than matching solution wording.Evaluation ignores sheet properties unrelated to the task, such as non-chart details in a chart task.Source: §3.3 · p. 5

4.1 Prompting LMs as a SheetCopilot

• The prompt combines a role description, atomic-action interface, output requirements, and a multi-round example to guide planning and interaction.The example illustrates the observe-propose-revise-act loop.Source: §4.1 · p. 5

4.2 Atomic Action as A Bridge for LMs and Software

• Atomic actions bridge model plans and software execution through named APIs, typed arguments, usage documentation, and examples, with platform-implementable functions.Their granularity is chosen to remain expressive yet concise, using actions drawn from spreadsheet Q&As and selected to cover collected tasks.Source: §4.2 · p. 5

4.3 State Machine-based Task Planning

• The state machine uses observing, proposing, revising, and acting stages to adapt plans to model or software feedback rather than rely on increasingly difficult open-loop planning.Observations describe sheet state; proposed actions can be revised using validation or execution errors and retrieved usage guidance before execution.Source: §4.3 · p. 6
• With SheetCopilot, GPT-3.5-Turbo outperforms the method that generates and runs VBA code by a large margin.The compared LLMs show strong Exec@1 and Pass@1; GPT-3.5-Turbo has the highest Exec@1, while GPT-4 has the best Pass@1 and efficiency.Source: Table 1 · p. 7

4.4 Hallucination Mitigation

• Hallucination mitigation checks output formatting and substitutes distant synonyms for action names to reduce confusion with the model’s internalized action knowledge.The paper describes hallucinations including undefined actions and illegal arguments.Source: §4.4 · p. 7

5.1 Benchmark Protocol

• The benchmark evaluates GPT-3.5-Turbo, GPT-4, and Claude at a 4096-token limit, using 221 tasks overall and a diverse 10% subset for GPT-4 and Claude.Exec@1 measures exception-free execution, Pass@1 measures functional correctness, and A50/A90 summarize action-count ratios at the 50th and 90th percentiles; lower values indicate fewer actions.Source: §5.1 · p. 7, §5.2 · p. 7

5.2 Comparing Task Planning Ability of Different LLMs

• GPT-4 significantly outperforms GPT-3.5-Turbo and Claude in Pass@1 and A50/A90, but its Exec@1 suffers from token-limit overruns on difficult tasks.Both GPT models reach 100% Exec@1 and Pass@1 in Management and Entry & manipulation; the three models have their best efficiency in different categories.Source: §5.2 · pp. 7–8, Figure 3 · p. 8

5.3 Ablation Studies of State Machine

• The four-component combination raises Exec@1 by 30.7% over baseline, with feedback and the external action document contributing to functional correctness and efficiency.Sheet-state feedback increases Exec@1 and Pass@1; error feedback gives the highest Exec@1 when individually applied, while combining feedback improves Pass@1 but slightly increases A50/A90. The action document improves Exec@1 and Pass@1, and adding usage examples reaches the highest performance.Source: Table 2 · p. 8, §5.3 · pp. 8–9
• 9.5% higher Exec@1 and 28.0% higher Pass@1: SheetCopilot with GPT-3.5-Turbo outperforms the VBA-based method.The paper presents prompting LLMs with SheetCopilot as a better alternative to directly translating natural-language requests to VBA code.Source: §5.3 · p. 9

5.4 Ablation Study of Atomic Action Names

• Using synonyms for atomic action names increases Pass@1 and lowers A50/A90, demonstrating that users can define their own actions for the method.Source: §5.4 · p. 9

5.6 Atomic Action at Different Granularity

• On chart and format tasks, integrated CreateChart actions improve efficiency but reduce correctness, while splitting SetFormat raises Exec@1 and Pass@1.The paper attributes CreateChart’s lower functional correctness to complex documentation and token-limit overruns; finer-grained SetFormat actions are easier for the LLM to understand and produce fewer hallucinations.Source: Table 4 · p. 10, §5.6 · p. 10

5.5 The Influence of LLM Temperature on Task Plan Generation

• Across temperatures from 0.0 to 1.0, SheetCopilot’s metrics remain stable, with only slight deviations from the temperature-0.0 values.Source: Figure 4 · p. 9, §5.5 · p. 10

• SheetCopilot performs spreadsheet manipulation with a high pass rate and acceptable efficiency, outperforming VBA-based methods.Source: §6 · p. 10
• Closed-loop control and external document insertion bring clear improvements over the baseline.Source: §6 · p. 10
• Finer-grained atomic actions notably improve functional correctness over integrated high-level actions.Source: §6 · p. 10
• Using atomic-action names dissimilar to official names achieves a surprising performance gain.Source: §6 · p. 10
• The paper proposes an observe-propose-revise-act spreadsheet agent and a realistic, diverse dataset of typical non-expert tasks.Source: §6 · p. 10

A.1 Details of Seed Task Collection

• Filtering removed 2,432 invalid Q&A pairs from 16,006, then clustering selected 220 representative questions and 67 supported seed tasks.Questions containing irrelevant-task keywords were removed; category-based filtering produced 13,574 pairs, which were clustered within category combinations before selecting tasks supported by the evaluation environment.Source: §A.1 · p. 14

A.2 Details of Workbench Collection

• Evaluation workbooks start data at A1, disallow blank columns within tables, and put column headers in the first row.Each workbook also receives a short context paragraph describing its usage and domain and providing useful formulas for manipulation.Source: §A.2 · p. 14

A.3 Details of Core Set Collection

• Core tasks were adapted to workbook elements, verified against four criteria, and simplified into natural-language instructions.GPT-4 adapted seed tasks using workbook summaries, headers, row numbers, and author-written exemplars. GPT-3.5 verification required realism, clarity, relevance, and completeness; simplification preserved intent and action order while using headers and specifying insertion locations. Of 1,515 valid adapted tasks, six per category combination were selected, then polished and checked for simulator support.Source: §A.3 · pp. 14, 16
• The final core set contains 221 tasks: 184 retained after polishing and simulator checks, plus 37 author-written tasks.Six tasks per category combination were randomly selected from simplified tasks (402 total); multiple verified successful solutions were retained as references for each task.Source: §A.3 · p. 16

A.4 Selecting the 10% Datast Used for Comparing the LLMs

• The 20-task LLM comparison subset was selected from the 221-task core set to approximately preserve category percentages and atomic-action counts.The authors describe this selection as avoiding distribution shift to a large extent.Source: §A.4 · p. 16

B.1 Collecting Atomic Actions

• ChatGPT classified Q&A pairs into six spreadsheet task categories and extracted the atomic actions needed, labeling irrelevant or overly vague questions Invalid.The categories cover entry and manipulation, management, formatting, formulas, charts, and pivot tables; a question may receive multiple labels.Source: §B.1 · pp. 16–17
• Task instructions were paraphrased for non-experts while preserving intent and action order, avoiding built-in function names and unnecessary column-index references.The guidance also required clear headers and indices for new columns and varied wording across paraphrases.Source: §B.1 · p. 22
• 44 of 80 action-name clusters were selected as atomic actions based on core-task needs and evaluation-environment restrictions.Generated action names were embedded in 768 dimensions, reduced to 64 dimensions, and clustered into 80 groups; manual inspection found the cluster-center names matched built-in Excel features.Source: §B.1 · p. 24

B.2 Collecting Synonyms for the Official Atomic Action Names

• For each atomic action, GPT-4 generated 10 synonym candidates, and the candidate least cosine-similar to the official name was used in the naming ablation.The ablation investigated the impact of alternative names in response to the action-name confusion issue.Source: §B.2 · p. 24

C Details of SheetCopilot Implementation

• The implementation details cover both the state machine and the prompt used to generate task solutions.Source: C Details of SheetCopilot Implementation · p. 24

C.1 State Machine Implementation

• The state machine observes sheet state, revises proposed actions after validation or execution errors, and repeats until the model outputs “Done!”.An external document prompts revision of the initial action; invalid actions return the process to proposing or revising, depending on the stage.Source: §C.1 · p. 24

C.2 Prompting LLMs to Generate Task Solutions

• The initial multi-turn prompt includes an overall goal, atomic actions, requirements, an exemplar, and the task to solve.Source: §C.2 · p. 24

• The examples show that combinations of available actions can attain the same goals as the missing actions, but less efficiently.Source: §D.1 · p. 26
• With SetConditionalFormat removed, SheetCopilot filters matching rows, formats them, then restores the data.Source: §D.1 · pp. 24, 27
• With CreatePivotTable removed, it substitutes SUMIF calculations for a pivot table, producing functionally correct results despite redundant calculation.Source: §D.1 · pp. 24, 27

D.2 Detailed Analysis of Atomic Action at Different Granularity

• The ablation tests whether combining chart-property actions improves efficiency and whether splitting SetFormat into finer-grained actions changes performance.The chart integration combined property-setting with CreateChart; SetConditionalFormat was removed during the format experiment because it overlapped with SetFormat.Source: §D.2 · p. 27
• Integrating chart creation and property-setting reduced A50 and A90 but decreased Exec@1 by 12.6% and Pass@1 by 6.1%; splitting SetFormat improved both correctness metrics.The ablation used GPT-3.5-Turbo on 43 chart-related and 41 format-related tasks.Source: §D.2 · p. 27
• The integrated CreateChart action’s complex, lengthy documentation contributed to argument errors and token-limit overruns, while finer-grained format actions were easier to understand and produced fewer hallucinations.Original SetFormat also prompted hallucinated arguments such as “criteria=...”; error feedback did not prevent repeated erroneous outputs.Source: §D.2 · p. 27

E SheetCopilot Manipulation Example

• In a sales-data example, SheetCopilot used sheet-state feedback, atomic-action documentation, and error feedback to revise its solution correctly.The example covers completing sales data, analyzing results, and visualizing a trend.Source: E SheetCopilot Manipulation Example · p. 27

F Analysis of failure cases

• GPT-3.5-Turbo and GPT-4 share similar failure patterns, whereas Claude’s pattern differs; Claude nevertheless performs competitively with GPT-3.5-Turbo under a prompt tuned for GPT-3.5-Turbo.The comparison of failure proportions uses the 10% evaluation subset.Source: F Analysis of failure cases · pp. 30–31
• Across the models, formula-and-range mistakes include missing absolute references when autofilling; GPT-3.5-Turbo often predicts formulas poorly, GPT-4 often returns incomplete solutions, and Claude often uses wrong formulas and ranges.All three models sometimes replace pivot tables with UNIQUE and SUMIF/AVERAGEIF/COUNTIF combinations but omit absolute references, producing incorrect calculations.Source: F Analysis of failure cases · p. 31
• Formula and range failures account for over 50% of GPT-3.5-Turbo’s full-coreset failure cases, while wrong actions and incomplete solutions together account for 33.7%.Other described failures include repeated actions, missed arguments, disobeyed requirements, and API implementation limits.Source: F Analysis of failure cases · pp. 31–32
• GPT-3.5-Turbo’s wrong ranges include autofill ranges and chart sources; wrong actions include misusing filters, while incomplete solutions omit required steps such as setting a chart title.Repeated outputs occur when it fails to revise its plan after error feedback; some API-related failures could be fixed by upgrading implementations.Source: F Analysis of failure cases · p. 32

G Demo

• Video demonstrations of SheetCopilot are available on the project website.Source: G Demo · p. 32

H Limitations

• Unoptimized token use limits SheetCopilot to short-horizon tasks; the authors suggest efficient prompting or larger-context models as future directions.Source: H Limitations · p. 32
• State feedback covers basic cell information but omits charts, pivot tables, filters, and frozen panes, so the agent may create charts or pivot tables repeatedly without observing their outcomes.Source: H Limitations · p. 32
• The manually curated dataset is not fully fledged, and multiple correct solutions may exist; collecting more ground-truth solutions could make performance estimates more accurate and stable.Source: H Limitations · p. 32
• The evaluation environment lacks support for some Excel built-in features, restricting which task categories SheetCopilot can handle.Source: H Limitations · p. 32

I Broader Impact

• The authors identify potential broader risks: automation may increase reliance on tools and disadvantage people lacking access or proficiency, while inherited LLM biases may affect spreadsheet interactions.Source: I Broader Impact · pp. 32–33
• Errors on sensitive data forms such as tax forms or financial statements may have undesirable consequences, motivating safeguards and oversight.Source: I Broader Impact · p. 32