PaperlayerView exact revision on arXiv

Source-linked AI summary

Table Meets LLM: Can Large Language Models Understand Structured Table Data? A Benchmark and Empirical Study

Yuan Sui, Mengyu Zhou, Mingjie Zhou, Shi Han, Dongmei Zhang

arXiv:2305.13062v5cs.CLcs.AIcs.IR

TL;DR

Whether LLMs genuinely understand tables remains under-studied, and evidence on effective input designs is limited. This paper benchmarks structural understanding across seven tasks and finds that performance varies with input design, while self-augmented prompting offers a simple way to improve it.

  • Problem

    Comprehensive evidence on whether LLMs understand tabular structure and on which input designs suit tabular tasks remains limited.

    Source: §1 · pp. 1–2

  • Method

    The paper introduces the SUC benchmark across structural-understanding tasks, compares prompt and input designs, and proposes self-augmented prompting to identify relevant values or ranges before answering.

    Source: §1 · p. 2, §4 · p. 5

  • Results

    LLMs show basic but imperfect structural understanding; HTML with format explanations and role prompts achieves the highest SUC accuracy, 65.43% across seven tasks.

    Source: §1 · p. 2, §5.2.1 · p. 7

  • Takeaways & Limitations

    The benchmark findings provide guidance on tabular input choices, and the authors present self-augmented prompting as a simple, generic approach for improving structural understanding.

    Source: §1 · p. 2, §7 · p. 9

    The method is primarily designed for languages with limited morphology, such as English.

    Source: ETHICAL CONSIDERATIONS · p. 9

Abstract

from arXiv · show

Large language models (LLMs) are becoming attractive as few-shot reasoners to solve Natural Language (NL)-related tasks. However, the understanding of their capability to process structured data like tables remains an under-explored area. While tables can be serialized as input for LLMs, there is a lack of comprehensive studies on whether LLMs genuinely comprehend this data. In this paper, we try to understand this by designing a benchmark to evaluate the structural understanding capabilities of LLMs through seven distinct tasks, e.g., cell lookup, row retrieval and size detection. Specially, we perform a series of evaluations on the recent most advanced LLM models, GPT-3.5 and GPT-4 and observe that performance varied with different input choices, including table input format, content order, role prompting, and partition marks. Drawing from the insights gained through the benchmark evaluations, we propose $\textit{self-augmentation}$ for effective structural prompting, such as critical value / range identification using internal knowledge of LLMs. When combined with carefully chosen input choices, these structural prompting methods lead to promising improvements in LLM performance on a variety of tabular tasks, e.g., TabFact($\uparrow2.31\%$), HybridQA($\uparrow2.13\%$), SQA($\uparrow2.72\%$), Feverous($\uparrow0.84\%$), and ToTTo($\uparrow5.68\%$). We believe that our open source benchmark and proposed prompting methods can serve as a simple yet generic selection for future research. The code and data of this paper will be temporality released at https://anonymous.4open.science/r/StructuredLLM-76F3/README.md and will be replaced with an official one at https://github.com/microsoft/TableProvider later.

1 INTRODUCTION

The paper investigates whether LLMs structurally understand tables and which input designs help, introducing the SUC benchmark and self-augmented prompting. It reports downstream gains when self-augmentation is combined with carefully chosen inputs.

  • Table serialization and input choices are flexible, with no grounded consensus on common-sense or exhaustive designs, complicating assessment of LLMs’ tabular understanding.Source: §1 · pp. 1–2
  • TabFact improved 2.31%, HybridQA 2.13%, SQA 2.72%, Feverous 0.84%, and ToTTo 5.68% over baseline methods with self-augmented prompting and carefully chosen input choices.Source: §1 · p. 2
  • The SUC benchmark compares structural-understanding capabilities across input designs and prompt variants, including format, format explanation, role prompting, partition marks, and zero- or one-shot approaches.Source: §1 · p. 2
  • Self-augmented prompting is model-agnostic and prompts LLMs to generate intermediate structural knowledge, such as critical values or ranges, using their internal knowledge.Source: §1 · p. 2
  • LLMs show basic structural understanding but remain imperfect even at table-size detection, while input-design combinations create performance gaps that persist with GPT-4.Source: §1 · p. 2

2 PRELIMINARIES

The paper focuses primarily on flat relational tables while noting the diversity of table structures and value formats that complicates LLM reasoning. It reviews table serialization methods and explains how long inputs are handled in its experiments.

2.1 Table Structure

  • Flat relational tables are the main focus: rows represent records and columns fields, while the paper also discusses hierarchical tables such as ToTTo [29].Tables can also vary in orientation and range from flat to hierarchical [41].Source: §2.1 · p. 2
  • Text, numbers, dates, formulas, and other formats encode different information, and the gap between tabular data and natural language challenges LLM table reasoning.Text includes headers, notes, captions, and cells; numbers can express arithmetic relationships, distributions, and trends, sometimes through spreadsheet formulas [12].Source: §2.1 · p. 2

2.2 Table Serialization & Splitting

  • Table serialization converts tables to sequential text; approaches include row-by-row serialization, special header and row tokens, two-way row/column serialization, and attribute-value templates.Examples include TaPas, MATE [14], TableFormer [26], TUTA [37], TURL [11], TaPEx [25], TABBIE [20], and TableGPT.Source: §2.2 · pp. 2–3
  • To avoid information loss and structural disruption from truncating long tables, experiments use random row sampling above a token threshold and append a 1-shot example within remaining capacity.The paper compares commonly used serialization methods as baselines in Sec §3.Source: §2.2 · p. 3

3 SUC BENCHMARK

The SUC benchmark is designed to compare table-input choices and assess how well LLMs understand table structure. It evaluates structural capabilities through tasks using data collected from public table datasets and standardized input designs.

3 SUC BENCHMARK

  • The benchmark investigates which input designs best support table understanding and how much structural understanding LLMs already possess.It also analyzes trade-offs among combinations of input designs.Source: §3 · p. 3

3.1 Structural Understanding Capabilities

  • Structural understanding is divided into partition and parsing, which distinguish tables from supplementary information and identify table layouts, and search and retrieval of information at specific positions.The benchmark examines whether LLMs can parse formats including CSV, JSON, XML, markdown, HTML, and XLSX.Source: §3.1 · p. 3

3.2 Task Design

  • Tasks assess table boundaries, dimensions, and merged-cell indices to test whether models can detect table structure and parse structural information.Merged-cell detection uses hierarchical spreadsheet tables.Source: §3.2 · p. 4
  • Lookup tasks retrieve a cell’s value or position, while row and column retrieval tasks return the values for a specified row or column.Predicted row or column lists are correct when they match the ground-truth list.Source: §3.2 · p. 4

3.3 Data Collection and Reformatting of SUC

  • The benchmark uses Wikipedia tables from TabFact, FEVEROUS, SQA, HybridQA, and ToTTo, sampling 1,500 tables per task setting and reserving ToTTo for merged-cell detection.Samples retain structural portions, use unique questions with reference answers, and exclude questions GPT-3.5 consistently answers correctly; SUC is designed for one-shot in-context evaluation.Source: §3.3 · p. 4

3.4 Evaluation

  • Evaluation compares CSV, JSON, XML, markdown, HTML, and XLSX with a separator-token baseline, and tests grammar explanations, partition marks, role prompts, and format explanations as augmentations.Task accuracy is measured with constrained answer formats; regular expressions parse the remaining answers that do not follow formatting instructions.Source: §3.4 · p. 5

4 STRUCTURAL PROMPTING

The section proposes self-augmented prompting to add structural knowledge to LLM inputs, motivated by imperfect table understanding and the impact of input design. Combined with carefully chosen input choices, structural prompts improve performance on downstream tabular tasks.

Structural Prompting

  • Unlike CoT and Zero-shot-CoT [22], self-augmented prompting elicits intermediate structural knowledge, is model-agnostic, and can integrate with prompting methods such as self-consistency [36].The approach draws on LLMs’ internal knowledge to generate constraints such as format specifications.Source: §4 · p. 5
  • Self-augmented prompting invokes an LLM twice: first to identify crucial values, patterns, or ranges, then to answer the original task using those extracted details.The first prompt’s results are incorporated into the second to guide the final answer.Source: §4 · pp. 5–6
  • Manual prompt engineering adds structural information such as cell addresses, row and column counts, table size, and merged-cell positions to the input.This creates a more structure-aware in-context learning environment for downstream tasks.Source: §4 · p. 6
  • Appending table size and merged-cell positions improves downstream-task performance, while structural prompting with carefully chosen input designs benefits a variety of tabular tasks.The method is motivated in part by poor table-size detection.Source: §4 · pp. 5–6

5 EXPERIMENTS

Experiments evaluate structural understanding on SUC and five downstream tabular tasks, showing that input design and self-augmentation affect performance.

5.1 Experiment Settings

  • Experiments evaluated five typical tabular downstream tasks using GPT-3.5 and GPT-4, defaulting to text-davinci-003 unless otherwise specified; settings were temperature 0, top_p 1, and n 1.Source: §5.1 · p. 6

5.2.1 Benchmark Highlights.

  • 3.26% overall improvement over 1-shot came from self-generated key-range and value identification; self-augmented options also exceeded 1-shot in accuracy and BLEU.Manually prompting table size slightly affected downstream performance: removing that option decreased overall performance by 0.4%.Source: Table 4 · p. 7, §5.2.2 · p. 8

5.2 Results

  • 65.43% overall accuracy was highest with HTML, format explanations, and role prompts without order changes; removing prompt examples especially hurt performance.The result covers seven structural-understanding tasks.Source: §5.2.1 · p. 7

5.2.1 Benchmark Highlights.

  • 6.76% improvement over NL+Sep came from markup languages, with HTML specifically outperforming NL+Sep; the authors suggest LLM familiarity with code and web data may explain this.Source: §5.2.1 · p. 7
  • Zero-shot HTML input reduced overall accuracy by 30.38%, while placing external questions or statements after tables decreased performance across tasks by 6.81%.The paper notes that in-context learning is particularly important for size detection and merged-cell detection.Source: §5.2.1 · p. 7
  • Partition marks and format explanations reduced Cell Lookup performance but benefited merged-cell detection and improved overall downstream-task performance.Source: §5.2.1 · pp. 7–8, Table 4 · p. 7

5.2.2 Downstream Tasks.

  • Manually labeled format explanations could hurt FEVEROUS performance, whose irregular tables pose challenges; self-augmentation can generate more comprehensive cues from patterns.Source: §5.2.2 · p. 8

6 RELATED WORK

The section situates LLM few-shot reasoning and reviews intermediate prompting methods, including chain-of-thought variants and generated knowledge.

In-context Learning with LLMs

  • GPT-3, Instruct-GPT, and Codex [5] show few-shot reasoning on natural language tasks, shaped by model size, training data, and computing power.Recent studies [9] [10] [30] propose LLM training methods; these models also show emergent ability on tasks they were not specifically fine-tuned for, unlike smaller language models.Source: §6 · p. 8

Intermediate of Prompt Engineering

  • CoT [38] uses examples with explanations but yields gains chiefly with models near 100B parameters; smaller models may reason illogically and underperform standard prompting.CoT prompting’s performance boost is typically proportional to model size.Source: §6 · p. 8
  • Zero-shot-CoT appends “Let’s think step by step,” while self-consistency [36] votes across multiple chains and improves arithmetic, commonsense, and symbolic reasoning.Self-consistency can improve results even when regular CoT is ineffective.Source: §6 · p. 8
  • Liu et al. [24] prompt an LLM to generate potentially useful information about a question before producing an answer.Source: §6 · pp. 8–9

7 CONCLUSION

The paper introduces a benchmark for comparing table-input designs and finds that LLMs have basic structural-understanding capabilities. It also offers downstream-task guidance and proposes self-augmented prompting, which uses LLM self-knowledge to generate additional knowledge.

7 CONCLUSION

  • The benchmark compares input designs to study LLMs’ structural understanding of tables.The authors describe the benchmark as a tool for studying structural understanding of tables and potentially other structured data.Source: §7 · p. 9
  • The input-design comparisons indicate that LLMs have basic capabilities for understanding table structure.The authors characterize this finding as surprising.Source: §7 · p. 9
  • The paper offers guidance for applying benchmark insights to downstream tasks and proposes self-augmented prompting using LLM self-knowledge to generate additional knowledge.The authors describe the method as simple, generic, and effective.Source: §7 · p. 9

ETHICAL CONSIDERATIONS

The authors identify metadata interpretation as an important challenge left for future research, and note that their method is designed primarily for English and that scaling to longer texts remains to be explored.

ETHICAL CONSIDERATIONS

  • Interpreting metadata and inferring its significance from context is challenging and task-dependent, so the authors leave it for future exploration.They cite column type prediction and dimension/measure classification as examples of downstream tasks on which this capability depends.Source: ETHICAL CONSIDERATIONS · p. 9
  • The method is designed primarily for languages with limited morphology, such as English, while its scalability to longer texts remains future work.Source: ETHICAL CONSIDERATIONS · p. 9

A APPENDIX

The appendix identifies full-results tables for downstream tasks and the benchmark, and lists the benchmark’s task and metric headings.

A APPENDIX

  • Table 8 is titled as presenting the full results of the downstream tasks.The supplied passage does not include the table’s cell values.Source: Table 8 · p. 11
  • Table 9 is titled as presenting the full results of the benchmark.The supplied passage does not include the table’s cell values.Source: Table 9 · p. 11
  • The benchmark headings list cell lookup, reverse lookup, column retrieval, row retrieval, size detection, and merged cell detection, with accuracy and delta metrics.The headings repeat Acc and Δ for each task.Source: A APPENDIX · p. 11

Related papers

Loading 2305.13062v5…