PaperlayerView exact revision on arXiv

Source-linked AI summary

TableLLM: Enabling Tabular Data Manipulation by LLMs in Real Office Usage Scenarios

Xiaokang Zhang, Sijia Luo, Bohan Zhang, Zeyao Ma, Jing Zhang, Yang Li, Guanlin Li, Zijun Yao, Kangli Xu, Jinchang Zhou, Daniel Zhang-Li, Jifan Yu, Shu Zhao, Juanzi Li, Jie Tang

arXiv:2403.19318v3cs.CL

TL;DR

Office users face laborious, error-prone tabular tasks, and their needs span document and spreadsheet formats with different operation demands. TABLELLM uses reasoning-extended training data and cross-way validation for an 8B model; it performs comparably to GPT-4o on document-embedded data and surpasses it on spreadsheet-embedded data.

  • Problem

    Office tabular tasks are laborious and error-prone, and the target setting includes query, update, merge, and chart operations across documents and spreadsheets.

    Source: §1 · p. 1, §3 · p. 3

  • Method

    TABLELLM fine-tunes an 8B model using reasoning-extended benchmark examples and cross-way validation of automatically generated training data.

    Source: §1 · p. 2, §6 · p. 8

  • Results

    TABLELLM performs comparably to GPT-4o and surpasses it in the spreadsheet-embedded scenario.

    Source: §1 · p. 2, §6 · p. 9

  • Takeaways & Limitations

    The authors anticipate that the dataset, model checkpoint, and code will provide a cost-effective solution for enhancing LLM capabilities for tables.

    Source: §6 · p. 9

    Synthetic training data may not capture real-world table diversity, leaving gaps for less common or more complex table structures.

    Source: Limitations · p. 9

Abstract

from arXiv · show

We introduce TableLLM, a robust large language model (LLM) with 8 billion parameters, purpose-built for proficiently handling tabular data manipulation tasks, whether they are embedded within documents or spreadsheets, catering to real-world office scenarios. We propose a distant supervision method for training, which comprises a reasoning process extension strategy, aiding in training LLMs to understand reasoning patterns more effectively as well as a cross-way validation strategy, ensuring the quality of the automatically generated data. To evaluate the performance of TableLLM, we have crafted benchmarks tailored to address both document and spreadsheet formats as well as constructed a well-organized evaluation pipeline capable of handling both scenarios. Thorough evaluations underscore the advantages of TableLLM when compared to various existing general-purpose and tabular data-focused LLMs. We have publicly released the model checkpoint, source code, benchmarks, and a web application for user interaction. Our codes and data are publicly available at https://github.com/TableLLM/TableLLM.

1 Introduction

Office users face laborious, error-prone tabular tasks across documents and spreadsheets, while existing approaches have difficulty covering long tables, diverse operations, and hybrid text-table queries. TABLELLM combines scenario-specific handling with reasoning-extended training data and cross-way validation; it performs comparably to GPT-4o on document-embedded data and outperforms it on spreadsheet-embedded data.

1 Introduction

  • Office users favor tableQA, revision, chart creation, and matching across Excel/CSV and Word/PDF, including long tables.These preferences point to diverse operations and format-specific processing needs, including hybrid text-table queries for documents and update or merge operations for spreadsheets.Source: §1 · p. 1
  • Existing approaches struggle to cover real-office needs: internal-parameter answer extraction is inadequate for long tables and diverse spreadsheet operations, while spreadsheet code struggles with hybrid text-table queries.Prior work has also addressed tasks such as fact verification, column annotation, table matching, and schema augmentation.Source: §1 · pp. 1–2
  • TABLELLM fine-tunes Llama3.1 (8B) with reasoning-extended benchmark data and cross-way validation of automatically generated data.It handles document-embedded tables through an inner-parameter-driven approach and spreadsheet-embedded tables through code-driven methods.Source: §1 · p. 2
  • TABLELLM performs comparably to GPT-4o on document-embedded data and outperforms GPT-4o on spreadsheet-embedded data.The authors assess both scenarios using tableQA instances from existing benchmarks and additional annotated manipulation instances.Source: §1 · p. 2
  • The authors release an 8B open-source model, source code, benchmarks, and an online application for tabular data manipulation.The listed contributions present these resources as a means to support accessibility and collaboration.Source: §1 · p. 2

2 Related Work

Prior table research has developed approaches for table question answering and, increasingly, manipulation operations. TABLELLM positions itself as addressing both question answering and manipulation through synthetic training data and table-oriented cross-validation.

2 Related Work

  • TableQA research spans representation learning, LLM fine-tuning, and prompting with tools such as SQL and Python.Examples include encoder and attention designs, unified models trained on multiple benchmarks, and multi-step prompting strategies.Source: §2 · p. 2
  • Table-manipulation work addresses insert, update, and delete operations, often translating questions into SQL or invoking spreadsheet functions and programming interfaces.These methods commonly target lengthy, regular tables in spreadsheet or database settings.Source: §2 · pp. 2–3
  • TABLELLM targets both table question answering and manipulation, using synthetic-data fine-tuning and cross-validation designed for tables.The authors contrast this scope with prior work focused on tableQA or spreadsheet manipulation.Source: §2 · pp. 2–3

3 Problem Definition

The paper frames its task as developing an LLM for query, update, merge, and chart operations on tabular data embedded in documents and spreadsheets. The dominant operations differ by setting: document tables primarily call for queries, while spreadsheets also call for modification and chart generation.

3 Problem Definition

  • Query selects data through filtering, aggregation, grouping, or sorting; update modifies, deletes, or adds data.Query covers most tableQA scenarios.Source: §3 · p. 3
  • Merge combines two tables, while chart visualizes data using bar, pie, or line charts.Document-embedded data primarily calls for querying; spreadsheet data often also requires modification and chart generation.Source: §3 · p. 3
  • The task is to develop an LLM that performs query, update, merge, and chart operations on document- and spreadsheet-embedded tables.The paper defines tabular data as data organized in a table or grid.Source: §3 · p. 3

4 TABLELLM

TABLELLM combines distant-supervision data construction with training prompts tailored to document and spreadsheet scenarios. Its data pipeline extends benchmark reasoning and generates new examples with cross-way validation, and its single trained model is also offered through a web application.

4 TABLELLM

  • The data pipeline extends benchmark reasoning, generates new question-answer pairs, and validates generated pairs; model training uses scenario-specific prompts.The overview distinguishes document-embedded and spreadsheet-embedded scenarios.Source: §4 · p. 3, Figure 3 · p. 4

4.1 Distant Supervision Data Construction

  • Existing benchmark answers are often too short to train models for complex manipulation, so the authors enrich their reasoning processes.For document-table queries, they extend short answers from WikiTQ, FeTaQA, and TAT-QA with textual reasoning and supplement some tables with generated descriptions; inputs are limited to fewer than 500 tokens.Source: §4.1 · p. 3
  • For spreadsheet manipulation, the authors use WikiSQL and Spider examples and expand their question-answer pairs with Pandas code rather than SQL queries.They validate code by checking executed outcomes against benchmark answers and exclude Spider multi-table queries and queries yielding null results.Source: §4.1 · pp. 3–4
  • To broaden limited benchmark coverage of update, merge, and chart tasks, the authors generate new questions from tabular data and cross-check answers from two solution approaches.For document-table questions, they compare internal-parameter answers with executed Pandas-code answers; code results are clustered using ROUGE-L, and a representative answer is selected.Source: §4.1 · pp. 4–5
  • For spreadsheet questions, two independently generated code solutions are executed and their outcomes compared as cross-way validation.The theoretical result states that consistency checking improves on single inference when both methods have correctness probability above 1/2, with independent distributions superior in expectation.Source: §4.1 · p. 5

4.2 Model Training

  • The single model receives document and spreadsheet training examples in a 1:1 ratio, with distinct prompts for the two data sources.Document inputs include table text and full table content, whereas spreadsheet inputs include headers and a subset of rows; the authors state that one model can alleviate deployment pressure.Source: §4.2 · p. 5

4.3 Model Deployment as Web Application

  • The web application accepts Word, PDF, Excel, and CSV inputs and returns tables, charts, or text in response to user instructions.It parses Word and PDF files into CSV for visualization and supports merging two spreadsheets under specified conditions.Source: §4.3 · p. 5

5 Experiment

The experiment evaluates TABLELLM across document- and spreadsheet-embedded tasks using established query benchmarks and human-annotated manipulation tests. It reports spreadsheet gains over competing models, near-GPT-4o document performance, and benefits from diverse training data.

5.1 Test Set Creation

  • The evaluation combines document query tests from WikiTQ, FeTaQA, and TAT-QA with spreadsheet tests from WikiSQL and Spider, plus human-annotated manipulation cases from 50 training-distinct long tables.Source: §5.1 · pp. 5–6

5.2 Evaluation Approach

  • Evaluation uses exact match for code-executed queries and modified tables, DeepSeek-V3 thresholds of 7 for inferred answers and 5 for chart-code comparisons.Source: §5.2 · p. 6

5.3 Comparison Methods

  • Baselines cover pre-training and fine-tuning, general, coding-specific, and prompt-driven LLMs, with scenario-specific prompts for general and coding models.Source: §5.3 · p. 6

5.4 Overall Experimental Results

  • TABLELLM generally surpasses other methods in spreadsheet-embedded tasks, including GPT-3.5 and GPT-4o, while performing close to GPT-4o on document-embedded tasks.Source: Table 2 · p. 7, §5.4 · p. 7

5.5 Ablation Studies on Training Data

  • 16.8% and 18.8% increases over base on WikiTQ and TAT-QA came from mixed extended-plus-generated data.Extended reasoning alone improved performance by 15.6% and 17.8%, while generated data alone improved it by 14.3% and 16.2%; original benchmark training underperformed the untrained base.Source: Table 3 · p. 7, §5.5 · p. 8
  • Cross-way validation outperformed same-way and self-check validation on document-embedded tabular data.Source: Table 4 · p. 7, §5.5 · p. 8

5.6 Training Strategy Investigation

  • Figure 4(a) shows log-linear performance gains with data size, while Figure 4(b) indicates that a 5:5 document-to-spreadsheet ratio balances performance across both data types.Source: Figure 4 · p. 8, §5.6 · p. 8

6 Conclusion

TABLELLM is presented as an 8B model for tabular manipulation in real office document and spreadsheet scenarios. The paper reports performance comparable to GPT-4o overall and higher in spreadsheet-embedded scenarios.

6 Conclusion

  • TABLELLM (8B) is designed for tabular data manipulation in real office scenarios across document-embedded and spreadsheet-embedded data.The paper says it gathered actual office requirements to identify these scenarios.Source: §6 · p. 8
  • The model uses extended reasoning processes and cross-way validation on automatically generated training data.Source: §6 · pp. 8–9
  • TABLELLM performs comparably to GPT-4o and surpasses it in the spreadsheet-embedded scenario.The conclusion anticipates that its dataset, model checkpoint, and code will offer a cost-effective solution for enhancing LLM capabilities for tables.Source: §6 · p. 9

Limitations

The paper notes that synthetic training data may not cover the diversity and difficulty of real-world tables. It also describes the training-data licenses, web application and trial, and survey respondents and tasks.

Limitations

  • Training data based on existing benchmarks may miss less common or complex tables, and some generated questions may be too easy to strengthen reasoning.The authors identify possible biases or gaps and suggest categorizing problem-generation difficulty as future work.Source: Limitations · p. 9

A License

  • Training uses WikiTQ, FeTaQA, and TAT-QA for document scenarios and WikiSQL, Spider, and GitTables for spreadsheet scenarios; the training data and model are released under CC BY-SA 4.0.Source: A License · p. 12

B Web Application

  • The web application accepts Word/PDF documents and Excel/CSV spreadsheets, supports query, chart, and two-table merge workflows, and collected 2,000 trial use cases.Of the use cases, 1,560 involved spreadsheets and 440 involved documents; 1,293 of 1,473 feedback responses were thumbs up.Source: B Web Application · p. 12

1. What is your occupation?

  • Survey respondents represented roles including research and development specialists, teachers, administrators, students, marketing professionals, and human resources personnel.Research and development specialists were 36.69% of the 507 respondents.Source: C The Survey Details · p. 12, §1 · p. 13

C The Survey Details

  • Survey results show TableQA was most sought-after (80.28%), followed by Table revision (71.40%); participants preferred Excel, Word, PDF, and CSV formats, and long tables exceeding 50 rows.Table revision involves creating, updating, and deleting tables.Source: C The Survey Details · p. 12

A Excel B Word C HTML D CSV

The survey questionnaire covers practical table work, including manipulation, cleaning, analysis, table-to-text tasks, augmentation, matching, and extraction.

D Verification of Cross-way Validation

The appendix formalizes when cross-way consistency checking can improve answer accuracy. It proves a non-decrease when response accuracy exceeds 1/2 and an expected improvement when responses come from independent distributions.

D Verification of Cross-way Validation

  • For responses with accuracy p>1/2, consistency checking is at least as accurate as single inference, with an expected benefit when the response distributions are independent.The theorem compares the posterior probability that both answers are correct after they agree with the accuracy of one response.Source: D Verification of Cross-way Validation · p. 14

Step 3: Final Inequality

Step 4: Final Inequality Derivation

  • For independent response distributions, the proof decomposes consistent-but-wrong answers by error type and uses a zero expected covariance term.The derivation identifies k as the mean error probability across error types.Source: Step 4: Final Inequality Derivation · pp. 15–16

Step 2: Comparing Independent and Identical Distributions

  • For identical response distributions, the proof applies Cauchy–Schwarz to bound consistency among incorrect answers before comparing with the independent case.The comparison proceeds through the expected probability of consistency given that at least one response is wrong.Source: Step 2: Comparing Independent and… · p. 16

E Error Analysis

The error analysis samples incorrect results from both document-embedded and spreadsheet-embedded tasks and classifies their failure types. The section also reports the training environment and optimization settings.

E Error Analysis

  • The document-embedded analysis examines 170 results that differ significantly from ground truth and classifies errors as question understanding, computational, intermediate-answer, or incomplete-answer errors.The categories distinguish misunderstanding the question, incorrect comparisons or calculations, non-final responses, and partial answers.Source: E Error Analysis · p. 16
  • The spreadsheet-embedded analysis samples 230 significantly different results across table QA, revision, chart, and matching tasks, and tracks question-understanding, data-type, and unrunnable-code errors.The sample includes 64 table-QA, 126 revision, 18 chart, and 22 matching results; revision includes update, insert, and delete cases.Source: E Error Analysis · p. 16, Table 7 · p. 17

F Training Environment and Settings

  • Training used a 5e-6 learning rate, batch size 4 per GPU, and four-step gradient accumulation for a total batch size of 128 across eight GPUs.The experiments ran with PyTorch 2.1.2 on CentOS Linux 7 using eight NVIDIA A800 80GB GPUs.Source: F Training Environment and Settings · p. 17

G Training Data for TABLELLM

The training-data section reports the scale and composition of TABLELLM’s distant-supervision data. The accompanying sections describe prompts for data generation and for baseline inference.

G Training Data for TABLELLM

  • TABLELLM’s distant-supervision training set contains 73,157 instances, expanded from an initial 4K trial with a 1:1 ratio of document-embedded to code-embedded data.Additional training data for the code-embedded scenario was added to improve its test-set performance.Source: G Training Data for TABLELLM · p. 17

H Prompts for Automatically Generating Dataset

  • A question-generation prompt covers both spreadsheet-embedded and document-embedded training data, while templates generate table-merge instructions.The prompts and templates are shown in Figures 13 and 14, respectively.Source: H Prompts for Automatically Generating… · p. 17

I Prompts for Baselines

  • Baseline inference uses prompts tailored to spreadsheet tasks for several LLMs, while document-task prompts match TABLELLM’s and other models’ established prompts are retained.The spreadsheet prompt is used for Llama3.1, GPT-3.5, GPT-4, CodeLlama, and Deepseek; TableLlama, StructGPT, Binder, and DATER use their established prompts.Source: I Prompts for Baselines · p. 17

J Meta Evaluation of DeepSeek-V3

The authors compare DeepSeek-V3’s response-quality ratings with human annotations on a 400-instance sample, reporting low false-positive and false-negative rates and examining implications for extended training data.

J Meta Evaluation of DeepSeek-V3

  • The meta-evaluation sampled 400 instances, compared DeepSeek-V3 scores with human ratings, and used correctness thresholds of 7 for document and 5 for spreadsheet data.The sample included 200 document instances and 200 spreadsheet instances; update and merge were omitted because they use exact match.Source: J Meta Evaluation of DeepSeek-V3 · p. 17
  • 1.5% false positives and 2.25% false negatives on the mixed test set support using DeepSeek-V3 to assess response quality.Rates were 1% and 2% for document-embedded data, and 2% and 2.5% for spreadsheet-embedded data, respectively.Source: J Meta Evaluation of DeepSeek-V3 · p. 17
  • False positives can occur when a long TABLELLM response contains the reference answer as a proper subset, leading DeepSeek-V3 to score an incorrect response highly.False negatives often arise when the answer is not stated at the beginning or end, or when the response is too long.Source: J Meta Evaluation of DeepSeek-V3 · p. 18
  • False positives may compromise the accuracy of model-generated responses used in training, while false negatives are excluded from extended training data.The paper reports that false-positive instances account for 1.5% and says false negatives do not detract from answer quality because they are excluded.Source: J Meta Evaluation of DeepSeek-V3 · p. 18
Loading 2403.19318v3…