Source-linked AI summary
SpreadsheetBench: Towards Challenging Real World Spreadsheet Manipulation
Zeyao Ma, Bohan Zhang, Jing Zhang, Jifan Yu, Xiaokang Zhang, Xiaohan Zhang, Sijia Luo, Xi Wang, Jie Tang
TL;DR
SpreadsheetBench targets the gap between simplified benchmarks and complex real-world spreadsheet manipulation by using authentic forum questions, diverse spreadsheets, and multi-test-case evaluation. Across model types, scores range from 0.05% to 23.65% under its OJ-style metric, underscoring the benchmark’s difficulty, while data leakage remains a construction concern.
Problem
Existing spreadsheet benchmarks inadequately represent real user demands because they rely on simplified spreadsheets, synthetic or manually constructed instructions, and often single-test evaluation.
Method
The benchmark uses 912 authentic forum instructions, flexibly organized spreadsheets, and an Online Judge-style metric requiring solutions to pass multiple spreadsheet test cases.
Results
Model scores range from 0.05% to 23.65% under the proposed OJ-style evaluation metric, with some methods scoring 0%.
Takeaways & Limitations
The benchmark exposes substantial difficulty in real-world spreadsheet manipulation and indicates that coding ability and multi-round prompting matter for improving correct responses.
Takeaways & Limitations
Forum-derived datasets may be susceptible to data leakage because many LLMs are pretrained on large web-text corpora.
Abstract
from arXiv · showhide
We introduce SpreadsheetBench, a challenging spreadsheet manipulation benchmark exclusively derived from real-world scenarios, designed to immerse current large language models (LLMs) in the actual workflow of spreadsheet users. Unlike existing benchmarks that rely on synthesized queries and simplified spreadsheet files, SpreadsheetBench is built from 912 real questions gathered from online Excel forums, which reflect the intricate needs of users. The associated spreadsheets from the forums contain a variety of tabular data such as multiple tables, non-standard relational tables, and abundant non-textual elements. Furthermore, we propose a more reliable evaluation metric akin to online judge platforms, where multiple spreadsheet files are created as test cases for each instruction, ensuring the evaluation of robust solutions capable of handling spreadsheets with varying values. Our comprehensive evaluation of various LLMs under both single-round and multi-round inference settings reveals a substantial gap between the state-of-the-art (SOTA) models and human performance, highlighting the benchmark's difficulty.
1 Introduction
SpreadsheetBench addresses the gap between simplified spreadsheet benchmarks and the complex, context-rich demands of real users by combining authentic forum instructions, flexibly organized spreadsheets, and multi-test-case evaluation. Its results show that current models remain far from reliable spreadsheet assistants.
- Current spreadsheet agents remain far from truly helpful assistants because existing benchmarks do not reflect challenging real-world user demands.Users may find these tools useful for reducing online-search effort, but they do not necessarily improve task completion time or success rates.
- Real forum questions are more complex than synthetic queries because they often include previous attempts, incorrect answers, encountered issues, and requested output examples.
- Existing benchmarks use overly simplified spreadsheets, whereas SpreadsheetBench includes multiple tables, non-standard relational structures, textual cells, and non-textual elements.Its data organization includes nested, incomplete, and missing headers, as well as colors and other non-textual content.
- Single-test evaluation can reward solutions tailored to one spreadsheet, motivating evaluation across multiple spreadsheets with similar structures but varying values and corner cases.The benchmark therefore associates each instruction with multiple input-output spreadsheet test cases that a solution must all pass.
- SpreadsheetBench contains 912 instructions and 2,729 test cases spanning 10 manipulation categories, large tables, multiple tables, and non-standard relational tables.The benchmark averages three test cases per instruction; 35.7% of spreadsheets contain multiple tables and 42.7% contain non-standard relational tables.
- Model performance ranges from 0.05% to 23.65% under the OJ-style metric, with some methods scoring 0%, while multi-round prompting can improve correctness by using spreadsheet data and compiler feedback.
2 SPREADSHEETBENCH
SPREADSHEETBENCH formulates spreadsheet manipulation as code generation over flexible, complex spreadsheets and constructs the benchmark through sourcing, filtering, formatting, and multi-test-case construction. It combines real-world forum questions with diverse spreadsheet structures and OJ-style evaluation across multiple input-output files.
- 2.1 Task Formulation: The task asks an LLM to generate code that modifies cells or entire sheets, producing spreadsheets compared against ground-truth answers.The dataset represents instructions, spreadsheets, and answers, while solutions are generated rather than stored because correctness is judged from final outputs.
- 2.2 Benchmark Construction: Its spreadsheets support flexible organization, including multiple tables, free-form information, and non-textual elements within a single sheet.This structure differs from a single regular relational table and increases the complexity of spreadsheet understanding and manipulation.
- 2.2 Benchmark Construction: SPREADSHEETBENCH is built through data sourcing, filtering, formatting, and test-case construction from spreadsheet-manipulation questions collected from online forums.The pipeline identifies sources, filters questions for solvability, relevance, feasibility, and representativeness, then creates self-contained instructions and evaluation data.
- 2.2 Benchmark Construction: 80% of spreadsheets have 1–49 rows and 1–13 columns, while long-tailed sizes, multiple tables, and non-standard relational tables create challenging cases.More than one-third contain multiple tables in one sheet, and nearly half contain nested, incomplete, or missing headers.
- 2.4 Evaluation Metrics: OJ-style evaluation applies each generated solution to multiple spreadsheets with similar structures but different values, testing robustness beyond one example.A solution must handle common and corner cases across test cases rather than merely matching one spreadsheet’s output.
3 Experiments
Experiments evaluate diverse model classes under single- and multi-round settings using code-based spreadsheet interaction and an OJ-style benchmark. Results show that complex spreadsheet structure, coding demands, and robustness to changed content remain major challenges.
- Experimental Setup: Models are evaluated across TableQA, open-source coding and general models, close-source models, and spreadsheet-specific products.The evaluation includes TaPEx, TaPas, Binder, CodeQwen, DeepseekCoder, Mixtral, Llama 3, GPT-3.5, GPT-4o, SheetCopilot, and Copilot in Excel.
- Overall Performance: Copilot in Excel achieves roughly 20% accuracy, while GPT-4o reaches around 17%, leaving a substantial gap from Excel-expert performance.Table 2 reports representative-model performance on SPREADSHEETBENCH, and the authors characterize current systems as inadequate for complex real-world manipulation.
- Model Analysis: Coding-oriented models outperform open-source general models, highlighting coding capacity as important for spreadsheet manipulation.Spreadsheet solutions primarily rely on code, and the analysis contrasts models such as DeepseekCoder with Llama-3 (70B).
- Inference Settings: Most models improve with multi-round interaction, but GPT-4o does not benefit because retrieving already-provided rows can duplicate spreadsheet content.GPT-3.5 improves from rounds 2 to 4, whereas GPT-4o’s initial solutions already have relatively high executability and accuracy.
- Robustness Analysis: Harder multi-test-case restrictions modestly reduce performance, indicating that generated solutions may not remain effective when spreadsheet contents change.The OJ-style evaluation tests robustness across altered spreadsheet values rather than relying on a single fixed spreadsheet.
- Spreadsheet Complexity: Performance declines on spreadsheets with more rows and columns, multiple tables, or non-standard structures.The benchmark analysis divides examples into subsets based on spreadsheet size and structural properties; expanding rows from 5 to 10 does not notably improve performance, possibly because of context length.
4 Conclusion
SPREADSHEETBENCH is a rigorous benchmark built from authentic spreadsheet-manipulation requests and diverse files, with multiple test suites to assess robustness and corner cases.
- 4 Conclusion: SPREADSHEETBENCH contains 912 authentic forum instructions, diverse formatted spreadsheets, multiple and irregular tables, and multiple test suites.The benchmark is designed to evaluate LLM capacity for spreadsheet manipulation more comprehensively than existing benchmarks.
Checklist
The checklist records affirmative reporting on claims, limitations, ethics, reproducibility materials, and asset use, while noting that error bars were not reported because API-based repetition was costly.
- Checklist: The paper states that its main claims, limitations, societal impacts, ethics review, and asset-use information are addressed in the indicated sections and appendices.The checklist points readers to Section 1, Appendix A, and Appendix A.4 for these disclosures.
- Checklist: The authors report providing code, data, instructions, training details, and compute information needed to reproduce the main experiments.These materials are identified in Section 3 and Appendix C.
- Checklist: Error bars were not reported because repeating API-based model and method executions was excessively costly.The checklist marks this item as No and gives execution cost as the reason.
A.1 Limitation
The benchmark’s main limitations concern data selection and test-case construction, with exclusions for difficult-to-formalize posts and incomplete coverage of corner cases.
- A.1 Limitation: Data selection excludes posts without acknowledged responses or those that are difficult to formalize, although some could be suitable after human annotation.This exclusion may narrow coverage of potentially valuable spreadsheet-manipulation questions.
- A.1 Limitation: The large number of benchmark questions prevents meticulous construction of corner cases for every question.Annotators are nevertheless asked to remain vigilant for possible corner cases during annotation.
- A.1 Limitation: The benchmark’s data construction includes filtering inappropriate content and additional checks for personally identifiable information.The authors also describe annotator treatment and consent as part of the data-collection process.
B.1 Data Source
SPREADSHEETBENCH is constructed from real-world Excel forums and blogs, using sequential annotation stages to convert authentic spreadsheet questions into testable benchmark instances.
- Data Sourcing: Four Excel forums and blogs—ExcelForum, MrExcel, ExcelGuru, and Chandoo—provide the benchmark’s real-world spreadsheet questions and files.The sources were selected for popularity, regular updates, and spreadsheet-focused content.
- Data Selection: Data selection removes posts unrelated to spreadsheet manipulation and retains posts with corresponding spreadsheet files and acknowledged solutions.Keywords, LLM filtering, file presence, and solution verification are used in the selection pipeline.
- Data Formatting: Data formatting transforms each raw post into a self-contained question, instruction type, answer position, and corresponding spreadsheet file.This makes the collected examples testable under the benchmark’s evaluation metrics.
- Test Case Generation: Test-case generation creates two additional cases by modifying answer-position values and rerunning solutions when necessary.Formula answers update after position changes, while other solutions such as VBA may require rerunning the solution.
B.3 Data Examples
The benchmark’s examples span cell- and sheet-level manipulations, contextual user requirements, irregular table structures, multiple tables or sheets, and varied deployment baselines.
- Data Examples: Examples include extracting text into a specified cell range and creating a duty-assignment grid across a two-dimensional output range.The examples illustrate both cell-level and sheet-level manipulation tasks.
- Data Examples: Realistic instructions may include prior actions, current formulas, encountered errors, detailed requirements, and manipulation examples.These contextual elements distinguish the examples from simpler synthetic instructions.
- Data Examples: The spreadsheets contain non-standard relational tables, multiple tables on one sheet, and multiple sheets holding source data and example answers.One example searches across named sheets using a sheet name and date range.
- Deployment Environment and Model Information: The experiments evaluate baseline models across several model categories and deploy open-source models with PyTorch, Transformers, and vLLM.Spreadsheet-specific products are evaluated in Google Sheets and Microsoft Excel.
C.2 Details of Experimental Setup
The experimental setup uses standardized inference hyperparameters and compares baseline models under single-round and multi-round prompting settings.
- Experimental Setup: All baseline LLMs use temperature 1 and top_p 1 across both inference settings.The study presents separate prompts for single-round and multiple-round inference.
- Experimental Setup: Table 5 reports the impact of different inference settings on GPT-3.5 and GPT-4o performance.The comparison covers the study’s single-round and multi-round inference configurations.
D.1 Analysis of Inference Settings
Inference-setting analysis finds benefits from multi-round interaction in repeated experiments, while ReAct-based spreadsheet retrieval reduces performance relative to execution feedback alone.
- Inference Settings: Execution feedback alone achieves the highest performance among the compared multi-round configurations.The settings differ by adding execution feedback, ReAct retrieval, and five prompt rows.
- Inference Settings: Adding ReAct retrieval lowers performance because current LLMs cannot effectively use additional spreadsheet data or flexibly decide what to read.Models may redundantly reread rows already supplied in the prompt.
D.2 Analysis of Impact Factors
GPT-4o performs best with a moderate amount of tabular context and improves with additional inference rounds, but larger inputs can overwhelm row comprehension and retrieval.
- GPT-4o improves as prompt rows increase from 1 to 5, but performance declines at 10 rows in two settings.The authors attribute this pattern to difficulty processing and comprehending relatively large numbers of rows.
- GPT-4o shows an overall upward performance trend as interactive inference rounds increase across all three settings.The multi-round results are averaged across repeated experiments and are reported as more stable than an earlier result.
- At four and five rounds, settings (3) and (4) become comparable, indicating limited retrieval beyond the spreadsheet’s first five rows in ReAct.This comparison is presented as evidence that current LLMs struggle to retrieve information beyond the first five rows.
D.3 Analysis of Evaluation Metrics
Human evaluation supports the reliability of the exact-cell-match metrics: accepted results were never judged incorrect, while a small fraction of rejected results were judged correct.
- The evaluation uses human checking of 50 sampled instructions with three test cases each to assess exact-match metric reliability.The authors report four test-case- and instruction-level false discovery and omission indexes for this validation.
- 0% of automatically accepted test cases and instructions were judged incorrect by annotators.The test-case and instruction-level false discovery rates were both zero, indicating no observed false positives in the sampled evaluation.
- 3.8% of failed test cases and 4% of instructions contained correct predictions rejected by the automatic metric.These false omissions arise when generated code preserves the intended answer but adds extra content, such as additional cells or table headers.
- The benchmark’s examples cover non-standard tables, multiple tables within or across sheets, colors, and corner cases involving empty cells.The figures illustrate failures from misunderstanding table structure, incomplete content reading, and treating an empty cell as zero.