Source-linked AI summary
Spreadsheet-RL: Advancing Large Language Model Agents on Realistic Spreadsheet Tasks via Reinforcement Learning
Banghao Chi, Yining Xie, Mingyuan Wu, Jingcheng Yang, Jize Jiang, Zhaoheng Li, Shengyi Qian, Minjia Zhang, Klara Nahrstedt, Rui Hou, Xiangjun Fan, Hanchao Yu
TL;DR
Existing spreadsheet agents often struggle with complex, multi-step workflows. Spreadsheet-RL trains specialized agents with outcome-based reinforcement learning in Excel and improves Pass@1 on general and domain-specific spreadsheet tasks, raising Qwen3-4B-Thinking-2507 from 12.0% to 23.4% on SpreadsheetBench.
Problem
Existing spreadsheet agents rely on general-purpose LLMs and prompting, limiting reliable execution of complex, multi-step spreadsheet workflows.
Method
Spreadsheet-RL combines outcome-based on-policy reinforcement learning, realistic Excel interaction, scalable task construction, and a multi-turn spreadsheet agent harness.
Results
Spreadsheet-RL improves Pass@1 on both general and domain-specific spreadsheet tasks, raising Qwen3-4B-Thinking-2507 from 12.0% to 23.4% on SpreadsheetBench.
Takeaways & Limitations
Spreadsheet-RL establishes outcome-based reinforcement learning as a practical and effective post-training paradigm for spreadsheet automation.
Takeaways & Limitations
Experiments focus on relatively lightweight open-source models and do not report results for larger dense or mixture-of-experts models.
Abstract
from arXiv · showhide
Spreadsheet systems (e.g., Microsoft Excel, Google Sheets) play a central role in modern data-centric workflows. As AI agents grow increasingly capable of automating complex tasks, such as controlling computers and generating presentations, building an AI-driven spreadsheet agent has emerged as a promising research direction. Most existing spreadsheet agents rely on specialized prompting over general-purpose LLMs; while this design has potentials on simple spreadsheet operations, it struggles to manage the complex, multi-step workflows typical of real-world applications. We introduce Spreadsheet-RL, a reinforcement learning (RL) fine-tuning framework designed to train specialized spreadsheet agents within a realistic Microsoft Excel environment. Spreadsheet-RL features an automated pipeline for scalable collection of paired start-goal spreadsheets from online forums, as well as domain-specific evaluation tasks in areas such as finance and supply chain management, which we compile into the new Domain-Spreadsheet benchmark dataset. It also includes a Spreadsheet Gym environment designed for multi-turn RL: Spreadsheet Gym exposes extensive Excel functionality through a Python sandbox, along with a refined harness that incorporates a comprehensive tool set and carefully designed tool-routing rules for spreadsheet tasks. Through comprehensive experiments, we show that Spreadsheet-RL substantially enhances AI agent's performance on both general and domain-specific spreadsheet tasks: it improves Qwen3-4B-Thinking-2507's Pass@1 on SpreadsheetBench from 12.0% to 23.4%, and raises Pass@1 from 8.4% to 17.2% on our curated Domain-Spreadsheet dataset. These results highlight Spreadsheet-RL's strong potential for generalization and real-world adoption in spreadsheet automation, and broadly, its promise for advancing LLM-based interactions with data interfaces in everyday work.
1 Introduction
Spreadsheet-RL addresses the limitations of prompt-driven spreadsheet agents by introducing an end-to-end, open-source RL framework built for realistic, multi-step Excel workflows. Its data pipeline, Spreadsheet Gym environment, and training framework improve Qwen3-4B-Thinking-2507’s SpreadsheetBench Pass@1 from 12.0% to 23.4%.
- Motivation: Spreadsheet systems support widely adopted personal and professional data-centric workflows, motivating AI agents for spreadsheet automation.Examples include travel planning, household budgeting, financial modeling, and data presentation.
- Limitations of Prior Work: Existing spreadsheet agents rely on proprietary, powerful LLMs and prompting strategies, limiting their approach beyond simple spreadsheet operations.The cited systems include SheetCopilot, SheetAgent, and ChatGPT Agent, using models such as GPT-4o.
- Challenges: Spreadsheet-specific RL is difficult because realistic initial–final spreadsheet pairs are costly to scale, while weak starting policies lack step-by-step supervision.A specialized harness is therefore needed to provide a structured action space and workflow prior for meaningful initial success.
- Framework: Spreadsheet-RL combines automated realistic task collection, Spreadsheet Gym’s multi-turn Excel-and-code-sandbox environment, and an asynchronous RL training framework.The framework supports outcome-based rewards across domains including finance, human resources, and supply chain management, with advanced Excel functionality.
- Results: 12.0% to 23.4% Pass@1: Spreadsheet-RL improves Qwen3-4B-Thinking-2507 on SpreadsheetBench.The evaluation uses GRPO objectives and includes SpreadsheetBench and the curated Domain-Spreadsheet benchmark.
- Significance: Spreadsheet-RL establishes outcome-based RL as a practical spreadsheet-automation paradigm and releases an end-to-end reproducible foundation for future research.Released components include the data, environment, harness, training pipeline, and model.
2 Related Work
Prior spreadsheet-workflow automation spans narrowly scoped techniques and newer natural-language AI agents. Recent benchmarks and data-collection methods include paired spreadsheets, synthesized workbook tasks, and spreadsheet-adjacent table-centric QA evaluations.
- Spreadsheet Workflow Automation: Early spreadsheet automation targeted specific tasks, including string processing, code-smell detection, and clustering related cells.These approaches covered a wide variety of spreadsheet-manipulation techniques but focused on well-scoped operations.
- Spreadsheet Workflow Automation: More recent systems such as SheetCopilot and SheetAgent use AI agents to translate natural-language requests into spreadsheet operations.The supplied passage states that these agents interact with spreadsheets while formulating desired operations in natural language.
- Benchmark Datasets for Spreadsheets: SpreadsheetBench collects 912 paired initial–final spreadsheets from online forums, with verification by 20 experts.The dataset is presented as a benchmark for evaluating spreadsheet workflows.
- Benchmark Datasets for Spreadsheets: SheetCopilot synthesizes spreadsheet tasks from 28 workbooks, while SheetAgent evaluates with table-centric QA benchmarks including WikiTableQuestions and TabFact.These works represent distinct benchmark construction and evaluation strategies for spreadsheet workflows.
3 Spreadsheet-RL
Spreadsheet-RL trains spreadsheet agents through multi-turn reinforcement learning in a realistic Microsoft Excel environment. It combines automated paired-spreadsheet construction, a spreadsheet-native interaction harness, and outcome-based GRPO rewards.
- RL training: GRPO optimizes the policy with verifiable outcome-based rewards, comparing the predicted final spreadsheet Dpred against oracle DO after multi-turn tool interaction.Rollout prompts encourage verification observations to mitigate sparse terminal rewards, while the RL objective remains outcome-based.
- Task formulation: Each task pairs initial spreadsheet(s) Di and instruction T with oracle final spreadsheet DO, while manipulation regions M are used only for reward computation.The agent interleaves reasoning with programmatic interaction to produce the post-operation workbook.
- Automated task construction: The Spreadsheet Data Agent builds paired initial–final spreadsheet corpora from public ExcelForum posts and solution discussions, using coding agents to execute procedures in real Excel.Seed tasks target advanced operations including complex formulas, formatting, pivot tables, and VBA/macros.
- Spreadsheet Gym: Spreadsheet Gym provides multi-turn interaction with real Microsoft Excel, a spreadsheet-native tool set, and a Python sandbox for auxiliary computation and stateful spreadsheet edits.Excel supports advanced features such as dynamic array formulas, enabling realistic and complex execution semantics.
- Agent harness: The agent harness routes operations to specialized tools, permits concurrent read-only inspection, serializes writes, and requires iterative inspection, editing, verification, and saving.This spreadsheet-native action space directly encodes common spreadsheet semantics instead of requiring brittle ad hoc Python implementations.
4 Domain-Spreadsheet Benchmark for Generalization Evaluation
Domain-Spreadsheet is a 1,660-task benchmark designed to evaluate spreadsheet agents on domain-specific knowledge and professional analytical workflows across six application areas. Its tasks reflect realistic professional workflows and support evaluation of reinforcement-learning generalization across domains.
- Benchmark Scope: 1,660 spreadsheet tasks span finance, supply chain, human resources, sales, and real estate, with finance divided into beginner, intermediate, and advanced levels.The benchmark is domain-specific and contrasts with operation-centric open-source spreadsheet benchmarks.
- Data Collection: Domain-Spreadsheet curates domain concepts and professional templates from knowledge areas represented by mainstream professional certifications.The collection addresses domain knowledge that is not readily available in public spreadsheet forums and is costly to annotate manually.
- Workflow Realism: Finance tasks model realistic workflows including comparable-company analysis with trading multiples, Value-at-Risk computation, and debt-service coverage-ratio modeling.These tasks are used to evaluate Spreadsheet-RL’s generalizability to different domains.
5 Spreadsheet-RL Evaluation
Spreadsheet-RL improves spreadsheet-agent performance through staged interaction and tool-interface enhancements followed by RL post-training, achieving higher Pass@1 on SpreadsheetBench and Domain-Spreadsheet. RL training also increases reward and accuracy while reducing response length and interaction turns.
- Evaluation Protocol: Evaluation uses Pass@1 under exact spreadsheet manipulation–region success, with numerical-cell tolerance, exact text matching, and canonicalized formula-string or evaluated-value comparison.SpreadsheetBench contains 912 unique tasks, each instantiated into three similar test cases, while Domain-Spreadsheet contains 1,660 unique tasks across five domains.
- SpreadsheetBench Evaluation: 23.4% Pass@1: Spreadsheet-RL raises Qwen3-4B-Thinking-2507 from 12.0% on SpreadsheetBench, after spreadsheet-native harnessing reaches 15.6% and comprehensive tool access reaches 19.3%.The staged gains combine spreadsheet-native interaction design, comprehensive tool access, and RL post-training.
- Domain-Spreadsheet Generalization: 17.2% Pass@1: Spreadsheet-RL improves overall performance from 8.4% on 1,660 Domain-Spreadsheet evaluation rollouts, with the largest gains on finance workflows.The evaluation covers domain-specific tasks despite training solely on operation-focused discussion-forum data.
- RL Training Dynamics: By step 60, smoothed training reward rises from roughly 0.21 to 0.33 and SpreadsheetBench accuracy from 19.3% to 23.4%.Over the same run, mean response length drops from approximately 16k to approximately 11k, while mean interaction turns fall from roughly 20.
6 Conclusion
Spreadsheet-RL is presented as the first end-to-end reinforcement-learning method specifically designed to train spreadsheet agents. It replaces reliance on prompt-driven, general-use LLM advances with outcome-based, on-policy learning through environment interaction for long-horizon spreadsheet workflows.
- Spreadsheet-RL is presented as the first end-to-end RL method specifically designed for training spreadsheet agents.
- Unlike prior prompt-driven spreadsheet agents, it uses outcome-based, on-policy RL to learn long-horizon workflows through environment interaction.
- The framework unifies large-scale, domain-specific spreadsheet task construction with a multi-turn interaction approach.
A Appendix · A.1 Limitations
Spreadsheet-RL is presented as an open foundation for reinforcement-learning post-training in spreadsheet workflows, but current experiments are limited to lightweight open-source models. Results for larger dense and mixture-of-experts models are not reported, leaving broader scaling for future work.
- A.1 Limitations: Spreadsheet-RL provides an open research foundation for studying RL post-training in spreadsheet-based data workflows.
- A.1 Limitations: Resource constraints limited the current experiments to relatively lightweight open-source models.
- A.1 Limitations: Training results are not reported for larger dense models or mixture-of-experts (MoE) models.
- A.1 Limitations: Scaling Spreadsheet-RL to larger model families is left as an important direction for future work.
A.2 Why SheetAgent Is Not Included as a Direct Baseline
SheetAgent is recognized as an important prior spreadsheet-agent design, but it is not rerun as a direct quantitative baseline because its system is tightly coupled to multiple implementation components and settings.
- Baseline exclusion: SheetAgent is not included as a direct quantitative baseline in Table 1.The paper describes SheetAgent as an important prior spreadsheet-agent design but does not report a direct rerun.
- System coupling: Its Planner, Informer, and Retriever interact through model-generated code and retrieval from a configured code corpus.These components are part of SheetAgent’s coupled design.
- System coupling: The system is tightly coupled to a particular model backend, prompt stack, code-retrieval component, and spreadsheet execution setting.Changes in the LLM API, spreadsheet runtime, or dataset interface can substantially affect the system’s behavior.
A.3 Broader Impact
Spreadsheet-RL is intended to make spreadsheet automation more accessible, reproducible, and reliable while supporting open research on productivity software. The authors caution that spreadsheet agents pose risks in high-stakes settings and should be treated as a research foundation requiring human review before deployment.
- Potential benefits: Spreadsheet-RL supports automation of repetitive workflows across finance, supply chain, human resources, sales, and personal productivity.The released data, environment, harness, training pipeline, and model are intended to support future research on open, verifiable productivity-software agents.
- Risks and limitations: Spreadsheet agents can produce incorrect formulas, unintended structural edits, or subtle formatting errors that affect downstream decisions, especially when users over-trust automated outputs.Publicly collected or synthesized training data may also contain biases or domain gaps.
- Risks and limitations: Spreadsheet-RL is presented as a research foundation rather than a fully deployable decision-making system, with practical deployment requiring human review.This caution follows the potential for spreadsheet errors, over-trust, biases, and domain gaps.
A.4 Complete Spreadsheet-Native Tool Harness Prompt
The harness directs an AI assistant to edit Excel workbooks toward the requested end state, treating the workbook itself as the answer. It specifies specialized tools for inspection, editing, verification, fallback, and concurrent read-only calls.
- Role: The assistant edits Excel workbooks to satisfy the user’s requested end state, without answering conceptually.The workbook itself is the answer.
- Tool router: Inspection uses find_cells for headers, anchors, or text and inspect_range for small relevant ranges.
- Tool router: Edits use fill_formula for formulas, clear_range for blank cells, and delete_rows/delete_columns when rows or columns must be physically removed.
- Verification and fallback: Verification uses recalculate_and_read after custom formula edits, while code_interpreter handles uncovered custom work or specialized-tool failures.
- Tool calling: Read-only calls may be issued concurrently in one assistant turn when useful, up to 20 total calls.
A.5 Inspect-Modify-Verify Example Rollout … A.14 Training Config and Hyperparameters
Spreadsheet-RL combines spreadsheet-native inspection, editing, verification, isolated rollout workspaces, and Excel-faithful asynchronous evaluation with group-relative policy optimization. Its rollout analyses and training configuration show structured tool use, improved planning behavior, and a 4B training setup designed for efficient spreadsheet automation.
- A.5 Inspect-Modify-Verify Example Rollout: The inspect-modify-verify rollout first checks matching keys, inspects the source value, edits the target, and verifies the resulting spreadsheet state.The example begins by inspecting Sheet1!B1 and Sheet2!B1, which both contain “124 bent street,” before reading Sheet2!D1.
- A.6 Qualitative Case Studies: Pre-RL Tool Interface Failures: Minimal Python interfaces caused structural-deletion and formula-templating failures, motivating structured tools that make spreadsheet semantics explicit.Deleting columns during forward iteration can skip targets, while manually templating VLOOKUP formulas risks escaping and row-reference errors.
- A.7 Qualitative Case Studies: Post-RL Rollout Behavior: Step 50 produced more concise rollouts and clearer alternative plans, debugging, and protocol-compliant scaffolds than step 0 across 2,726 SpreadsheetBench rollouts.Mean assistant-output length decreased from 51,732 to 38,965 characters; examples contrast explicit planning and stable debugging with speculative or crash-and-recover behavior.
- A.8 Harness Details: Spreadsheet Gym provides find_cells, inspect_range, formula filling, clearing, structural deletion, recalculation, and code-interpreter fallback for spreadsheet-native workflows.Inspection targets small relevant regions, fill_formula translates references, clear_range preserves structure, and recalculate_and_read supports faithful verification.
- A.9 RL-Training Compatibility and Workspace Isolation: Each rollout receives an isolated seeded workbook workspace, with locked initialization, constrained file resolution, automatic cleanup, and external sandbox or Excel backends retained for specialized execution.The filesystem isolates trajectory-specific spreadsheet artifacts rather than replacing arbitrary-code sandboxing.
- A.10 Cell-Level Evaluation Details: Evaluation compares normalized post-recalculation answer-region values with an oracle, rounding numbers and numeric strings to 2 decimal places and requiring exact matches otherwise.Dates are rounded to the nearest day, times converted to HH:MM, and empty strings treated as equivalent to None.
- A.12 GRPO Objective; A.13 Training Data Statistics; A.14 Training Config and Hyperparameters: GRPO samples response groups and updates the policy using group-relative normalized outcome rewards with clipping, while training data tracks spreadsheet sizes and operation distributions.The 4B configuration uses temperature 0.6, top-p 0.95, top-k 20, 64 tasks, global batch size 64, learning rate 1 × 10−6, and two AdamW updates per step on 1×4 H100 GPUs.