PaperlayerView exact revision on arXiv

Source-linked AI summary

A Benchmark and Framework for Evaluating Next Action Predictions in Spreadsheets

Tejas Agrawal, Vu Le, Sumit Gulwani, Gust Verbruggen

arXiv:2606.13802v1cs.SEcs.AIcs.HCcs.LG

TL;DR

Spreadsheet authoring involves many low-level actions, yet generalized next-action prediction lacks benchmarks and edit histories. This paper constructs a 52-trajectory benchmark and online evaluation, finding that baseline performance indicates the task is learnable.

  • Problem

    Generalized spreadsheet action suggestions lack benchmarks because public corpora provide limited edit histories and spreadsheet actions vary spatially, temporally, and compositionally.

  • Method

    The paper curates 52 human-validated spreadsheet creation trajectories totaling 11,907 operations and evaluates predictors online with acceptance and dynamically updated trajectories.

  • Results

    GPT-5 with reasoning saves 33% of user actions versus 18% for GPT-5 mini, indicating that next-action prediction in spreadsheets is learnable.

  • Takeaways & Limitations

    The benchmark and online evaluation provide actionable insights for improving spreadsheet predictors through abstention, triggers, stopping criteria, and related metrics.

Abstract

from arXiv · show

Predictive code completion greatly accelerates how quickly developers work. In spreadsheets, despite being much more common, such auto-completion features are virtually non-existent. To address this gap, we introduce a benchmark for systems that observe a sequence of user actions in a spreadsheet and predict future actions. Two challenges are (1) the absence of edit histories in public spreadsheet corpora and (2) the complex space of spreadsheet actions (spatial, temporal, composite). To address (1), we manually curate 52 sequences of 12K actions that recreate spreadsheets from public corpora, seeded by parametrized heuristics and LLM refinement. To address (2), we propose an online evaluation that expects a prediction after each user action, accepts or rejects that prediction, updates the future actions upon acceptance, and repeats this until the target spreadsheet is obtained. We use multiple baseline predictors (including zero-shot LLMs, fine-tuned SLMs, and classical models) and analyze different properties that our benchmark teaches us, including but not limited to: properties of saved actions and false positives, efficiency, effect of user profiles, effect of triggers, and effect of context.

1. Introduction

The paper introduces a benchmark and online framework for predicting generalized spreadsheet actions, addressing missing edit histories and spreadsheets’ complex action space. It contributes 52 human-validated creation trajectories totaling 11,907 operations and demonstrates the framework with baseline systems and utility-focused metrics.

  • Motivation: The framework observes low-level spreadsheet actions and suggests subsequent actions to accelerate spreadsheet authoring.It targets generalized action suggestions beyond the specific spreadsheet scenarios supported by earlier assistants.
  • Online evaluation: The online evaluation predicts zero or more actions after each history, accepts or rejects the prediction, and dynamically updates future actions after acceptance.This evaluates end-to-end behavior while modeling user acceptance and measuring utility at simulation, prediction, and action levels.
  • Benchmark dataset: 52 human-validated spreadsheet creation trajectories contain 11,907 operations, with 35–821 steps per trajectory and 229 steps on average.The trajectories are manually curated from static workbooks using heuristics and large language models for heuristic seeding.
  • Online evaluation: The online design exposes compounding errors, prevents repeated easy suggestions from inflating stepwise accuracy, and tests whether systems can repair prior suggestions.Suggestions alter the state from which later predictions continue, unlike fixed-time-step teacher-forced evaluation.
  • Baseline findings: 33% of actions are saved by GPT-5 with reasoning versus 18% by GPT-5 mini in the reported baseline comparison.The paper uses zero-shot LLMs, fine-tuned SLMs, and classical machine learning models to demonstrate how its metrics support targeted improvement.

2. Benchmarks

The benchmark reconstructs spreadsheet-building trajectories through symbolic generation, LLM refinement, and human annotation, yielding 52 sheets with 11,907 operations. An oracle experiment finds that 68% of ground-truth properties appear in model predictions, establishing an empirical predictability ceiling.

  • Trajectory reconstruction: The reconstruction pipeline combines symbolic cold-start generation, LLM-assisted refinement, and human annotation to produce varied user trajectories from public workbooks.The first two stages reduce annotation effort and provide initial trajectories, while human annotators correct remaining unnatural subsequences.
  • Trajectory reconstruction: 0.69 mean normalized edit distance separates preannotation from final trajectories, with 19 of 52 trajectories effectively rewritten from scratch at distance >0.8.The median normalized edit distance is 0.77.
  • Dataset statistics: 52 sheets contain 11,907 operations, with sequence lengths ranging from 35 to 821, a mean of 229, and a median of 164.Only one sheet is used per workbook to maximise diversity, and Figure 2 reports operation-category composition and file-size distributions.
  • Predictability ceiling: 68% of all ground-truth properties appear in the union of oracle model predictions, with a per-trajectory median of 66.3%.The oracle uses full operation history and the corresponding sheet image at each step across four frontier reasoning models; 44 of 52 trajectories exceed 50%.

3. Online Evaluation

The online evaluation performs on-policy rollouts in which each accepted prediction changes the remaining future actions, while adaptive updates and final patches preserve the target worksheet state. It evaluates predictions at multiple granularities and summarizes full trajectories using user effort, acceptance, precision, and coverage metrics.

  • Online evaluation: The evaluation loop judges assistants under real-user conditions by predicting after each action and rolling forward through changing worksheet states.Each accepted suggestion affects the actions remaining for the user.
  • Online evaluation: After acceptance, the update step removes satisfied operations, prepends inverses for false positives, and verifies that the adapted sequence still reaches the target.A final patch simulates the adapted future and synthesizes corrections for residual discrepancies.
  • Metrics: At cell-property granularity, predictions classify explicitly modified properties as true positives, false positives, or false negatives against the target state.A false positive introduces an unwanted edit that the user must undo.
  • Metrics: User actions saved is the primary full-trajectory metric, complemented by acceptance rate, macroaverage precision, and predictability coverage.UAS measures percentage reduction in user effort; PCOV measures produced predictable cell-property pairs relative to the oracle union.

4. Experimental Setup

The experiments compare three solver families—zero-shot LLMs, fine-tuned SLMs, and classical predictors—under evaluation settings that vary prediction granularity, invocation controls, context, and acceptance behavior.

  • Solvers: Three solver families are evaluated: zero-shot LLMs, fine-tuned SLMs, and classical machine-learning predictors.The classical models are trained n-gram, online n-gram, LSTM, and XGBoost; the SLMs are SmolLM2 models with 135M and 360M parameters.
  • Solvers: Zero-shot LLMs receive recent operation history and valid operation syntax, with GPT-5 as the default model unless specified otherwise.The evaluated models are GPT-5-R, GPT-5, GPT-5-R mini, and GPT-5 mini; “-R” denotes low reasoning mode.
  • Evaluation settings: The benchmark uses multi-action prediction for variable-length completions and single-action repredict for one action at a time until rejection.Multi-action prediction tests both prediction quality and stopping, whereas single-action repredict removes the stopping burden from solvers that cannot decide when to stop.
  • Evaluation controls: Evaluation varies prediction stride, context window, maximum actions per prediction, and context shortening.The maximum is m=∞ for multi-action prediction and m = 1 for single-action repredict; s=1 invokes the solver after every user action.
  • Acceptance heuristics: Acceptance heuristics combine precision and savings constraints, including GREEDY, P90, P60, and ALWAYS.GREEDY accepts suggestions that advance the goal despite minor errors; P90 and P60 prioritize apparent correctness; ALWAYS accepts every prediction as an upper-bound and stress test.

5. Experiments

Experiments show that next-action prediction is learnable, with stronger and fine-tuned models performing better, but practical utility depends critically on abstention, triggering, context, prediction scope, and action type. Accepted predictions arrive in burst-like streaks, while prediction quality improves as trajectories provide more context.

  • Acceptance and utility: The ALWAYS heuristic produces −19.2% UAS at 9.3% precision, showing that useful systems must abstain and stop predicting when uncertain.Strict acceptance filters also reduce utility: GREEDY achieves 22.3% UAS, versus 20.3% for GREEDY-2 and 17.5% for HYBRID-2; P90 reaches only 17.0% UAS.
  • Triggers and context: At stride s=1, UAS reaches 22.3% despite 20.0% acceptance, whereas s=8 raises acceptance to 36.5% but collapses UAS to 9.8%.In the single-action setting, s=1 →8 raises AR from 30.9% to 43.7% while lowering UAS from 27.4% to 10.6%.
  • Action difficulty: Content-heavy operations are accepted more often than presentational operations, while fine-tuning improves number format, fill, border, and autofill by roughly 10 points.GPT-5 dominates input, whereas the online n-gram remains competitive on locally repetitive structural categories such as merge, border, and paste.
  • Acceptance streaks: Acceptance streaks have median length 1 but mean 2.21, with 5% lasting at least seven accepts, explaining how modest acceptance rates yield larger user-action savings.Other solvers show the same burst-like pattern, with mean streak lengths between 1.6 and 2.3.
  • Context and prediction scope: Expanding context from 32 to 128 prior operations raises multi-action UAS from 22.3% to 27.6%, while longer contexts provide no meaningful additional gain.Unlimited prediction scope performs best; m=1 raises acceptance to 30.6% but yields only 20.3% UAS, and acceptance rises from ∼12.5% early to ∼24% late.

6. Related Work

Related work places spreadsheet next-action prediction between natural-language automation and modeless action-to-action assistance, drawing on established code-completion and GUI-automation systems. It also motivates interactive, user-centred evaluation because offline metrics may diverge from actual utility.

  • Spreadsheet automation: Spreadsheet automation spans natural-language-to-action systems, which translate explicit commands into operations, and action-to-action systems, which suggest subsequent actions.SheetCopilot and SpreadsheetBench benchmark explicit-command approaches, while TableTalk finds prompt formulation can exceed direct-manipulation costs for routine edits.
  • Modeless edit suggestion and code completion: Modeless edit suggestions are established in code editing, progressing from repetitive symbolic edits to full-line and multi-line code completion.Examples include Blue-Pencil, IntelliCode, and GitHub Copilot.
  • Modeless edit suggestion and code completion: Prior evaluation work informs this benchmark’s design through practical language-model code-completion methodologies, suggestion-timing studies, and sequential GUI-automation benchmarks.Mozannar et al. motivate investigating prediction stride and acceptance heuristics, while ASSISTGUI provides a related state-modifying automation setting.
  • Interactive evaluation: Interactive assistants require user-centred evaluation under realistic conditions because offline metrics can diverge from actual user utility and fail for interacting recommendations.This concern is framed across information retrieval and recommender systems, including the unobserved-truth problem.

7. Conclusion

The paper introduces the first benchmark for next-action prediction in spreadsheets, comprising 52 synthetic-and-refined tasks spanning 12K actions. Online evaluation yields actionable insights into abstention, triggers, stopping criteria, and other predictor improvements.

  • Benchmark: The benchmark contains 52 tasks spanning 12K actions, created through synthetic generation and refinement.These tasks constitute the paper’s benchmark for next-action prediction in spreadsheets.
  • Evaluation: Online evaluation measures how users perceive using next-action predictors.The evaluation framework supports user-oriented assessment of predictions during spreadsheet interaction.
  • Findings: Experiments show the benchmark provides actionable insights into abstention, cheap triggers, stopping criteria, and other predictor improvements.The conclusion identifies these areas as opportunities for improving predictors.

Impact Statement

The paper aims to advance machine learning through spreadsheet next-action prediction, while encouraging less energy-intensive methods than LLM-based baselines.

  • Impact Statement: The authors identify no specific societal consequences requiring emphasis.They note that the work may have many potential societal consequences but do not highlight any individually.
  • Impact Statement: The authors encourage less energy-intensive methods, although LLMs appear in baselines to demonstrate the benchmark.The statement distinguishes benchmark-demonstration use of LLMs from the preferred direction for solving the problem.

A. Dataset Construction Details … A.3. LLM Refinement

The dataset reconstructs spreadsheet user trajectories by combining VLM annotation, symbolic action generation, sampled preferences, LLM refinement, and validation. This pipeline models regional structure, dependencies, bulk operations, formatting reuse, and more human-like action sequences.

  • A. Dataset Construction Details: Trajectory reconstruction combines semantic spreadsheet annotation, symbolic low-level action generation, LLM naturalization, and manual refinement.The process begins with metadata annotation, generates actions such as value insertion and formatting, then reorders and combines actions before manual refinement.
  • A.1. Worksheet Annotation: The VLM annotates regions, dependencies, pasted ranges, and similarly formatted regions to model how users build related spreadsheet structures.It distinguishes format-only from full-region copying and uses screenshots, cell-region information, formulas, and merged-cell data as input.
  • A.1. Worksheet Annotation: Regions receive type classifications and optional closing operations, while region dependencies encode temporal ordering beyond direct formula references.Examples include completing a summary table after its source data and applying final highlighting or header bolding after structural construction.
  • A.2. Heuristic trajectories: Heuristics generate functionally correct actions for individual cells and pasted regions, then merge compatible cell operations into range operations.Contiguous identical formatting can become one action, while border operations are merged when a single-button action can apply them.
  • A.2. Heuristic trajectories: Sampled preference settings diversify trajectories by controlling operation merging, region ordering, and within-region operation ordering.The generator can produce spreadsheet-specific ordering conventions, including different sequences for font colors and font properties.
  • A.3. LLM Refinement: An LLM judge-editor loop refines rigid symbolic sequences toward more human-like trajectories while preserving the target spreadsheet state.The judge inspects screenshots, action sequences, and prior changes, focusing on dependencies, granularity, formatting, layout, repetition, defaults, borders, and pasting.
  • A.3. LLM Refinement: When the judge provides feedback, an editor rewrites the sequence and execution-based validation checks whether the resulting state matches the target.Editing is retried up to 2 times when validation fails, allowing noise such as ineffective operations.
  • A.3. LLM Refinement: Observed refinements consolidate adjacent formatting, remove stray formatting outside content, and eliminate number or date formats from text cells.These changes target fragmented operations, unused spreadsheet areas, and semantically inappropriate formatting.

A.4. Human Refinement … D. Solver Prompts

The benchmark trajectories undergo substantial human refinement to improve naturalness, while solver implementations span classical sequence models, fine-tuned SmolLM2 variants, and standardized zero-shot LLM prompts. These sections specify the refinement practices, model designs, training setup, and prompt structure.

  • A.4. Human Refinement: Refinements grouped formatting with styled content, established structure before detail, split bulk inputs into cell actions, and trimmed ranges to actual content.These changes target more natural operation ordering and avoid implicit empty-cell padding.
  • A.4. Human Refinement: All 52 trajectories received human annotation, with mean normalized Levenshtein distance 0.69 and 19 (37%) near-completely rewritten.Annotators watched spreadsheets being built action-by-action and corrected unnatural subsequences that heuristics and prompts sometimes produced.
  • B. Classical Model Details: The classical baselines are n-GRAM, ONLINE n-GRAM, LSTM/GRU, and XGBOOST, sharing featurization and decoding while differing in next-operation mapping.Their shared structured records include operation type, range geometry, and value-type buckets.
  • B. Classical Model Details: The four classical solvers cover local repetition, corpus-learned short patterns, tabular feature interactions, and distributed sequence representations as low-cost non-LLM references.ONLINE n-GRAM uses no training, while n-GRAM, XGBOOST, and LSTM learn from progressively different representations of operation history.
  • C. Fine-tuning Setup: Two SmolLM2 variants, with 135M and 360M parameters, are fine-tuned on sequences from 11,997 disjoint public workbooks using up to 32 prior operations.Training uses a causal language-modelling objective on contiguous trajectory slices generated by the benchmark pipeline.
  • C. Fine-tuning Setup: Fine-tuning uses the evaluation representation, stride-16 sampling, 99th-percentile length filtering, AdamW, fp16, effective batch 32, and a 2048-token maximum.The 360M and 135M variants train for 6 and 5 epochs, respectively, with the lowest-evaluation-loss checkpoint selected.
  • C. Fine-tuning Setup: Fine-tuned models receive a minimal continuation prompt and emit predicted operations one per line.The template asks models to complete spreadsheet-building action sequences by identifying and extending key patterns.
  • D. Solver Prompts: All zero-shot LLM solvers use one Jinja2-templated system prompt, switching between single-action and multi-action prediction through the boolean single.The template contains an operation-syntax reference, while the user prompt supplies the active sheet when applicable and recent operation history.

E. Variance and Reproducibility

The study measures run-to-run variance for the default GPT-5 configuration across five runs per evaluation setting under fixed evaluation conditions. Variability is small, supporting the use of a representative run elsewhere in the paper.

  • Variance measurement: Five runs per setting re-evaluate the default GPT-5 configuration, with temperature 0 and fixed prompts, hyperparameters, and evaluation-loop seed.The reruns quantify per-run UAS and acceptance rate (AR), along with their mean and standard deviation.
  • Observed variance: UAS varies by under one point in the single-action setting and under two points in the multi-action setting.The multi-action setting permits longer, more variable continuations.
  • Observed variance: Acceptance rate is even tighter across the two evaluation settings.
  • Reporting choice: The paper therefore reports a single representative run elsewhere.

F. Some more insights · G. Visualizing trajectories · H. Extended example

The benchmark reveals how acceptance depends on operation context, prediction length, and category bias, while trajectory visualizations and a detailed trace show how accepted, rejected, and corrected predictions affect spreadsheet completion. The extended example illustrates trigger-based prediction, rejection, ground-truth continuation, and state adaptation after acceptance.

  • F. Some more insights: Predictions after input or autofill operations achieve ∼24% acceptance, roughly twice the ∼10–12% rate after MERGE or ALIGN.Autofill is predicted less often, suggesting stronger reliability during predictable data-entry sequences.
  • F. Some more insights: Most files require fewer than 10 correctional operations regardless of acceptance rate, although a few outliers across the spectrum incur 20+ corrections.The passage reports no monotonic relationship between correction count and acceptance rate.
  • F. Some more insights: The model under-predicts input, producing 33.0% versus 57.1% in ground truth, while over-predicting align at 14.1% versus 6.9% and border at 12.3% versus 6.3%.Align accounts for 15.9% of rejected versus 8.0% of accepted predictions; border accounts for 13.6% versus 8.0%.
  • F. Some more insights: Acceptance rises with prediction length, from 18% for single-operation predictions to 31% for 11–15 operations and 51% for 15+-operation predictions.Short predictions dominate volume, whereas longer predictions more often capture strong repeating patterns.
  • G. Visualizing trajectories: Progress curves show cumulative acceleration from accepted predictions relative to manual completion, while prediction timelines show when predictions succeed or fail.The visualizations use green markers for accepted predictions and compare assisted progress with a manual baseline.
  • H. Extended example: The zero-shot LLM prompt switches between single-action reprediction and multi-action continuation based on whether num_op_to_pred equals 1.The full operation value-format rules are reproduced in the released code, while the figure shows an abridged template.
  • H. Extended example: The extended trace shows a low-precision prediction rejected at t=10 and a perfect-precision prediction accepted at t=13, saving 2 operations.Acceptance adapts future ground truth by undoing false positives, synthesizing missing operations, correcting mismatches, and retaining true positives.
Loading 2606.13802v1…