PaperlayerView exact revision on arXiv

Source-linked AI summary

Large Language Model for Table Processing: A Survey

Weizheng Lu, Jing Zhang, Ju Fan, Zihao Fu, Yueguo Chen, Xiaoyong Du

arXiv:2402.05121v3cs.AIcs.CL

TL;DR

Table-processing tasks are important in everyday work, but table structure and user intent pose challenges beyond plain-text modeling. This survey reviews tasks and methods for LLMs and VLMs, including training and prompting approaches. It finds that prompting strong LLMs often performs best on table tasks, while table-structure understanding remains difficult.

  • Problem

    Tables’ two-dimensional structure and task-specific needs challenge LLMs primarily trained on sequential text, despite the importance of table tasks in everyday activities.

    Source: §1 · pp. 1–2

  • Method

    The survey reviews table tasks and user scenarios alongside LLM and VLM representation, training, and prompting approaches, including LLM-powered agents.

    Source: §1 · pp. 2–3

  • Results

    Across many table tasks, prompting strong LLMs like GPT performs best, while LLM training excels on only one of four benchmarks.

    Source: §7.1 · p. 20, §7.2 · p. 21

  • Takeaways & Limitations

    The survey identifies challenges in table processing to inform and guide future research directions.

    Source: §9 · p. 22

    Accurate chain-of-thought is difficult to obtain: human annotation can be prohibitively expensive, automatic generation can be unreliable for complex table reasoning, and repeated inference is time-consuming.

    Source: §8.2 · p. 22

Abstract

from arXiv · show

Tables, typically two-dimensional and structured to store large amounts of data, are essential in daily activities like database queries, spreadsheet manipulations, web table question answering, and image table information extraction. Automating these table-centric tasks with Large Language Models (LLMs) or Visual Language Models (VLMs) offers significant public benefits, garnering interest from academia and industry. This survey provides a comprehensive overview of table-related tasks, examining both user scenarios and technical aspects. It covers traditional tasks like table question answering as well as emerging fields such as spreadsheet manipulation and table data analysis. We summarize the training techniques for LLMs and VLMs tailored for table processing. Additionally, we discuss prompt engineering, particularly the use of LLM-powered agents, for various table-related tasks. Finally, we highlight several challenges, including diverse user input when serving and slow thinking using chain-of-thought.

1 Introduction

Table processing matters across everyday data tasks, but tables’ multidimensional structure and reasoning demands challenge models trained primarily on sequential text. This survey organizes LLM research across table tasks, representations, training, and prompting, including emerging tasks and user perspectives.

1 Introduction

  • Tables’ rows, columns, and schemas challenge LLMs trained primarily on sequential text, while table tasks can require numerical operations, multistep reasoning, and external tools.Table processing also requires understanding schemas, locating data in two dimensions, and executing queries such as SQL.Source: §1 · p. 2
  • The survey organizes LLM table-processing research into four interrelated categories: table types and tasks, data representation, training, and prompting.The taxonomy aims to provide a systematic review of technological developments and current research directions.Source: §1 · p. 2
  • Its coverage includes table QA and spreadsheet manipulation, with a technical and end-user perspective on tasks and table-data lifecycles.It also categorizes instruction-tuning, data synthesis, chain-of-thought, ReAct, and LLM-powered agent approaches, and compiles datasets, benchmarks, training corpora, papers, code, and datasets.Source: §1 · pp. 2–3
  • Earlier surveys often focused on smaller-model pretraining or fine-tuning, individual tasks, or narrower coverage, leaving LLM prompting and agent-based approaches unaddressed.Examples include surveys focused on table QA, table reasoning, prediction, generation, understanding, or NL2SQL.Source: §1 · p. 3

2 Table Types and Table Tasks

This section frames table processing around four table formats, their differing structures and use cases, the tasks performed on them, and users’ lifecycle needs. It connects these distinctions to table-specific task demands and three ways AI systems can be integrated into user interactions.

2.1 Table Definition

  • Tables are two-dimensional grids of cells whose columns represent attributes and whose rows contain records; their schema captures names, types, constraints, and relationships.The survey focuses mainly on semi-structured and structured tables.Source: §2.1 · p. 3
  • Spreadsheet, web, database, and document tables serve different scenarios, so their AI applications vary from spreadsheet copilots and web-table QA to NL2SQL and document-table extraction.Spreadsheet layouts can be irregular, web tables carry page context, database tables have explicit schemas, and document tables may need identification and conversion to table-native formats.Source: §2.1 · pp. 4–5, Figure 2 · p. 4

2.2 Differences Between Table and Text

  • Unlike text’s usual left-to-right sequence, tables require horizontal and vertical reading, rely on schemas or headers, and may retain meaning under row or column permutations.These contrasts are summarized from Li et al..Source: §2.2 · p. 5

2.3 Table Tasks

  • Table tasks span QA, fact verification, table-to-text, cleaning, population, entity linking, type annotation, spreadsheet manipulation, NL2SQL, data analysis, and document detection and extraction.The tasks broadly address table-, spreadsheet-, database-, and document-related work, using direct table understanding, code, SQL, or extraction.Source: §2.3 · p. 5

2.4 Data Lifecycle and End-users’ Perspective

  • End users prioritize productivity, while their roles shape needs: common users seek querying and manipulation, whereas data engineers need preparation and modeling tools.The survey organizes table work into a five-stage lifecycle.Source: §2.4 · p. 6, §2.4.1 · p. 6

2.4.1 Data Lifecycle

  • The lifecycle covers data entry, cleaning, CRUD, analysis, and visualization, linking table tasks to stages from creating or structuring data through downstream use.Entry includes creating table structure and converting unstructured sources; cleaning addresses errors, inaccuracies, missing values, and duplicates; analysis can involve feature engineering, outlier detection, machine learning, and tools such as SQL or Python.Source: §2.4.1 · pp. 6–7

2.4.2 End-users’ Experience

  • AI interaction can be Beside an application, Inside it as the product’s core, or Outside it orchestrating across applications; users tend to interact through natural language.User experience is strongly linked to the underlying table AI system.Source: §2.4.2 · p. 7, Figure 4 · p. 7

3 Table Data Representation

Table processing models need machine-readable inputs that preserve table structure and semantics. The survey describes text serialization and schema prompts, alongside visual and layout representations for image tables.

3.1 Text Representation

  • HTML and natural language with separators (markdown or CSV) are the most effective evaluated text-table formats; spreadsheets may need anchor-based JSON encoding for irregular layouts.The anchor-based method identifies heterogeneous rows and columns, then encodes cell locations, values, and schemas.Source: §3.1 · p. 8
  • Prompts for NL2SQL can represent table schemas in plain text or CREATE TABLE statements, and foreign-key information and concise-answer rules can benefit the task.Foreign keys can suggest relationships among relational tables.Source: §3.1 · p. 8

3.2 Visual and Layout Representation

  • Representing tables as images can facilitate complex reasoning for VLMs, and visual cues and layout information require corresponding visual and layout embeddings.The survey also describes OCR and PDF-parser preprocessing, and converting HTML tables or spreadsheets into images.Source: §3.2 · pp. 8–9
  • For image tables, visual embeddings combine image, position, and segment information, while layout embeddings encode spatial coordinates on separate x and y axes.LayoutLM and TableVLM normalize and discretize coordinates to integers from 0 to 1000.Source: §3.2 · p. 9

4 Table Training

The survey reviews table-model training from pre-LLM structural adaptations to LLM and VLM approaches. It covers instruction tuning, data construction, code-based methods, continued pre-training, and combinations of models.

4.1 Pre-LLM Era

  • Pre-LLM table models adapted architectures, encodings, and objectives, but some were task-limited or required annotated fine-tuning data.Examples include TaPas, TaBERT, TURL, TABBIE, TaPEx, and RESDSQL.Source: §4.1 · p. 9

4.2.1 What’s New in Table LLM Training

  • Instruction tuning for table tasks includes table tuning, code tuning, and hybrid approaches, while automatic dataset construction uses templates or distillation from stronger LLMs.The survey describes instruction tuning as guiding models with instructions in prompts.Source: §4.2.1 · pp. 9–10

4.2.2 Table Tuning

  • Table tuning combines existing table datasets into instruction examples, and TableLlama and Table-GPT illustrate template-based collection and synthesis-then-augment data construction.TableLlama collects 14 datasets covering 11 table tasks; Table-GPT uses 18 synthesis processes and augmentations at instruction, table, and output levels.Source: §4.2.2 · pp. 10–11
  • TableLlama and Table-GPT are reported to generalize robustly from seen table tasks to unseen tasks after table tuning.The survey also reports that Table-GPT’s augmentations can prevent overfitting.Source: §4.2.2 · p. 11

4.2.3 Code Tuing

  • Code tuning generates executable SQL or Python for table manipulation, and code-instruction methods are reported to boost table-task performance, especially in data analysis.The survey notes code LLMs’ strong leaderboard positions on DS-1000 and InfiAgent-DABench, and describes synthetic-data approaches including OSS-Instruct and DAAgent.Source: §4.2.3 · pp. 11–12

4.2.4 Hybrid of Table and Code

  • Studies including StructLM report that code LLMs used as foundation models achieve superior performance on table tasks compared with general-purpose and math LLMs.The comparison is described as an ablation on tabular datasets.Source: §4.2.4 · p. 12

4.2.5 Continue Pre-training

  • CodeS uses continued pre-training of small models with SQL-related, natural-language, and NL-to-code data to improve NL2SQL capabilities.The approach addresses deployment costs associated with small models while targeting their weaker code-generation or reasoning abilities.Source: §4.2.5 · p. 13

4.2.6 SLMs + LLMs

  • ZeroNL2SQL combines an encoder-decoder SLM that generates SQL sketches with an LLM that completes and corrects them; table VLMs commonly pre-train the visual encoder before fine-tuning the full model.The survey also divides VLM table-task approaches into detection/extraction, end-to-end processing, and hybrid systems.Source: §4.2.6 · p. 13

5 Table Prompting

Table-task prompting addresses reasoning, arithmetic, lookup, and error-correction challenges by using iterative reasoning and agent workflows. The survey describes agents that observe tables and user intent, plan and execute actions, and update their state.

5 Table Prompting

  • Table-task prompting must address complex reasoning, arithmetic, factual lookup, and correction of erroneous decisions, while enabling reflection and external Python or SQL tools.The survey identifies these as challenges for LLMs handling table tasks.Source: §5 · pp. 13–14

5.1 Common Workflow of LLM-powered Agents

  • Agents iterate from observing table data and user intent to decomposing tasks, planning and executing actions, and updating state until expectations are met.The survey names SheetAgent, Chain-of-Table, ReAcTable, TAPERA, and E5 as examples of this workflow.Source: §5.1 · p. 14
  • Agent memory stores environment observations or past actions for future planning, and table agents typically record planning history as action outcomes and table states.The survey says this memory can support consistency and effectiveness.Source: §5.1 · p. 14
  • The survey focuses its agent discussion on planning and action modules, while treating memory as a component that stores table-state and action history.The described agent workflow repeats until expectations are met.Source: §5.1 · p. 14

5.2 Planning

Table-agent planning prompts an LLM to choose actions from task instructions, table state, the current query or subproblem, and planning history. The survey covers task decomposition and reflection or revision strategies.

5.2 Planning

  • Planning breaks complex problems into smaller subproblems and reflects on or revises previous decisions.The planning module prompts the LLM to choose an action at each step.Source: §5.2 · p. 14, §5.2.1 · p. 14

5.2.1 Formalizing Planning

  • At each step, the planning module uses the task instruction, current table state, query or subproblem, and prior planning history to formulate an action.The action is executed on the target table, yielding a new observation and output that update state and history.Source: §5.2.1 · pp. 14–15

5.2.2 Complex Task Decomposition

  • Task decomposition breaks complex table problems into simpler subtasks, such as identifying relevant tables and columns and producing intermediate subqueries for NL2SQL.Dater also decomposes large web-table evidence and complex questions into smaller evidence and subquestions.Source: §5.2.2 · p. 15

5.2.3 Reflection and Revision

  • Agents can try alternative approaches and reflect on past actions, but an erroneous action can significantly affect subsequent analysis and processing.SheetCopilot and SheetAgent refine past actions; SelfEvolve asks an LLM to debug code that fails in the Python interpreter.Source: §5.2.3 · p. 15
  • Self-consistency and voting can improve accuracy on complex table QA and NL2SQL tasks, but studies report that they take time and increase cost.Revision methods also retrieve code examples or debug non-executable code to prevent errors such as incorrect APIs or arguments.Source: §5.2.3 · p. 15

5.3 Action

The survey describes how table-processing agents translate user requests into tool actions, with approaches tailored to table QA and NL2SQL, spreadsheet work, and multiple table tasks.

5.3 Action

  • Actions mediate between LLMs and tools such as database engines, spreadsheets, and Python interpreters, so systems must make error-free API calls for complex tasks.The survey frames tool access as more than converting natural-language text into software APIs.Source: §5.3 · p. 15

5.3.1 Table QA and NL2SQL

  • Table QA and NL2SQL agents use Python or SQL actions; ReAcTable adds an observation-action-reflection loop that can generate intermediate tables when direct queries cannot answer.It can generate and execute SQL, Python, or a direct answer, then refine table data through the loop.Source: §5.3.1 · pp. 15–16

5.3.2 Spreadsheet Manipulation and Data Analysis

  • Spreadsheet agents model operations as atomic VBA actions or use a Planner and Informer with Python and SQL for manipulation and evidence retrieval.SheetCopilot’s atomic actions include an API name, arguments, description, and usage examples; SheetAgent uses SQL queries to support its Planner.Source: §5.3.2 · p. 16

5.3.3 Multiple Table Tasks

  • For multiple table tasks, frameworks explore data and schemas before planning and optimizing actions; examples include StructGPT readers, UniDM context conversion, and TAP4LLM preprocessing.ChatPipe uses a Deep Q-Network to recommend data-analysis and feature-engineering operations.Source: §5.3.3 · pp. 16–17

5.4 Other Techniques

The section covers few-shot example selection and role-play as additional prompting techniques.

5.4.1 Few-shot Examples Selection

  • Few-shot selection balances relevance and diversity, while context-length limits can force trade-offs in how examples’ questions, schemas, and SQL are organized.DAIL-SQL removes example schemas to balance example quality against token quantity.Source: §5.4.1 · p. 17

5.4.2 Role-play

  • Role-play assigns an LLM a specified role, while Tapilot-Crossing uses agents with distinct roles to simulate real-world data-analysis scenarios.Its roles include Administrator, Client, Data Scientist, and AI Chatbot.Source: §5.4.2 · p. 17

6 Resources

The survey catalogs datasets, benchmarks, and software for table processing, including resources that address real-world workload, robustness, dataset quality, and table tools.

6.1 Datasets and Benchmarks

  • RobuT finds that perturbations degrade all table methods, while closed-source LLMs such as GPT show greater robustness.The intended robustness tests include row or column swaps, synonyms or abbreviations for column names, and changes in a target cell’s position.Source: §6.1 · pp. 17–18
  • Analysis of BIRD found inaccuracies in some gold SQLs and noise in some natural-language queries, exposing potential unreliability in human-labeled table datasets.After correcting errors, complex prompting methods such as DIN-SQL might be less effective than simple zero-shot prompting.Source: §6.1 · pp. 18–19
  • ScienceBenchmark and SpreadsheetBench target real-world workloads, drawing on domain-specific databases and user questions from four Excel forums, respectively.ScienceBenchmark augments hundreds of human-labeled NL2SQL pairs into thousands of data points.Source: §6.1 · p. 19
  • The survey identifies large metadata and schema resources, including SchemaPile with 221k database schemas and 1.7 million table definitions, and ComplexTable with over 1 million image tables.These resources support schema understanding, column-type annotation, downstream training, and visual table tasks.Source: §6.1 · p. 19

6.2 Open-source Software

  • Open-source table software includes database-interaction tools, PandasAI for natural-language DataFrame operations, and RetClean for LLM-based data cleaning.The survey notes that most of the discussed tools focus on NL2SQL and table QA.Source: §6.2 · p. 19

7 Analysis and Discussion

The discussion compares training and prompting methods across table tasks, considering accuracy, cost, and deployment trade-offs. It reports that prompting strong LLMs often performs best, while training and agent approaches bring distinct costs and constraints.

7.1 Discussion on LLM Training

  • Training-based methods offer enterprises control and private deployment without third-party data leakage, but substantial training costs and accuracy concerns remain.Fine-tuning a 7B model typically requires eight 80GB GPUs; training time also depends on the number of tokens.Source: §7.1 · p. 20
  • Across four benchmarks, LLM training excels on only one, whereas prompting GPT-4 is superior.The comparison covers four benchmarks in table QA and NL2SQL; Table 3 also reports model parameters and training tokens.Source: §7.1 · p. 20, Table 3 · p. 20
  • Results vary by task: 7B TableLlama does not outperform task-specific fine-tuning on WiKiSQL, and no clear LLM-versus-SLM winner emerges across NL2SQL metrics and domains.On data analysis, Lemur and DAAgent do not surpass GPT-4; Table-GPT outperforms GPT-3.5 and ChatGPT across table-related tasks, but its training cost is prohibitive for ordinary enterprise users seeking private deployment.Source: §7.1 · pp. 20–21

7.2 Discussion on LLM Prompting

  • LLM-powered agents combine LLM capabilities with external-tool flexibility, while prompting strong LLMs such as GPT still performs best on many table tasks.Agent prompting can require repeated rounds, and ReAct with execution feedback for spreadsheet manipulation takes substantial time and increases financial costs.Source: §7.2 · p. 21
  • Hard-coded prompt templates crafted through trial and error may not generalize across pipelines, LLMs, domains, or data inputs.Prompting through third-party providers can also raise enterprise data-leakage concerns, though open-source or privately fine-tuned models may gradually mitigate this issue.Source: §7.2 · p. 21
  • On TIS, most LLMs score around 50% on table-structure understanding, while GPT-4 reaches 66.1%.The benchmark probes complex hierarchies and locating specific regions in two-dimensional tables.Source: §7.2 · p. 21

8 Challenges and Future Directions

The survey identifies diverse user inputs and slow, deep reasoning as challenges for table LLMs, with future work needed on domain adaptation and accurate, efficient chain-of-thought. These challenges span ambiguous queries, varied table data, and prompting trade-offs.

8.1 Diverse User Input When Serving

  • Ambiguous or off-topic user queries, including broad requests such as “help me analyze this table,” complicate serving table systems.Users may also struggle to formulate queries accurately even when they know the table schema and content.Source: §8.1 · p. 21
  • Proprietary, diverse schemas and content challenge transfer to unfamiliar domains, while existing datasets are often simple and costly to expand for real-world business coverage.The survey proposes cost-effective synthesis of high-quality training data tailored to domain needs.Source: §8.1 · pp. 21–22

8.2 Slow and Deep Thinking

  • Accurate chain-of-thought can bridge the gap between instructions and answers, but human annotation is prohibitively expensive and LLM-generated traces may be inaccurate on complex table logic.The survey identifies cost-effective ways to ensure accurate chain-of-thought as an open research direction.Source: §8.2 · p. 22
  • Hard-coded chain-of-thought prompting workflows transfer weakly, while repeated inference is time-consuming, creating an accuracy-versus-inference-time trade-off.The survey calls for prompting methods to reconsider workflow construction and balance inference time with accuracy.Source: §8.2 · p. 22

9 Conclusion

The survey reviews LLMs for table processing across tasks, user perspectives, and technical approaches, and collects resources while identifying future research challenges.

9 Conclusion

  • The survey categorizes table tasks from academic and end-user perspectives, reviews representation, training, and prompting techniques, and discusses datasets, benchmarks, software, and research challenges.Its scope spans table question answering, spreadsheet manipulation, and data analysis.Source: §9 · p. 22

Related papers

Loading 2402.05121v3…