PaperlayerView exact revision on arXiv

Source-linked AI summary

Table-GPT: Table-tuned GPT for Diverse Table Tasks

Peng Li, Yeye He, Dror Yashar, Weiwei Cui, Song Ge, Haidong Zhang, Danielle Rifinski Fainman, Dongmei Zhang, Surajit Chaudhuri

arXiv:2310.09263v1cs.CLcs.AIcs.DB

TL;DR

Language models remain suboptimal on table tasks, which involve two-dimensional relational structures unlike their predominantly one-dimensional text pretraining. The paper continues training models on diverse table tasks synthesized from real tables; Table-GPT outperforms vanilla GPT-3.5 and ChatGPT across a wide range of tasks, including unseen ones, while responding to new table-task instructions.

  • Problem

    Language models pretrained predominantly on natural-language text remain suboptimal on table-understanding tasks.

    Source: §1 · p. 1

  • Method

    Table-tuning continues training language models on diverse (instruction, table, completion) examples synthesized from real tables.

    Source: §1 · p. 1

  • Results

    Table-GPT consistently outperforms vanilla GPT-3.5 and ChatGPT across a wide range of table tasks, including unseen tasks, and responds to novel table-task instructions.

    Source: §1 · pp. 1–2

  • Takeaways & Limitations

    Table-GPT can serve as a table foundation model and a better starting point than vanilla GPT for downstream task-specific fine-tuning and prompt engineering.

    Source: §1 · p. 3

    The synthesized table-task list is not exhaustive and is presented as a starting point for further task synthesis.

    Source: §4.2 · p. 8

Abstract

from arXiv · show

Language models, such as GPT-3.5 and ChatGPT, demonstrate remarkable abilities to follow diverse human instructions and perform a wide range of tasks. However, when probing language models using a range of basic table-understanding tasks, we observe that today's language models are still sub-optimal in many table-related tasks, likely because they are pre-trained predominantly on \emph{one-dimensional} natural-language texts, whereas relational tables are \emph{two-dimensional} objects. In this work, we propose a new "\emph{table-tuning}" paradigm, where we continue to train/fine-tune language models like GPT-3.5 and ChatGPT, using diverse table-tasks synthesized from real tables as training data, with the goal of enhancing language models' ability to understand tables and perform table tasks. We show that our resulting Table-GPT models demonstrate (1) better \emph{table-understanding} capabilities, by consistently outperforming the vanilla GPT-3.5 and ChatGPT, on a wide-range of table tasks, including holdout unseen tasks, and (2) strong \emph{generalizability}, in its ability to respond to diverse human instructions to perform new table-tasks, in a manner similar to GPT-3.5 and ChatGPT.

1 INTRODUCTION

The introduction argues that language models struggle with relational tables because text pretraining does not match tables’ two-dimensional structure and permutation properties. It proposes table-tuning and reports improved performance across table tasks, including unseen tasks, while retaining instruction-following generalizability.

1 INTRODUCTION

  • Text is one-directional and order-sensitive, whereas relational tables are two-dimensional and largely invariant to row and column permutations.The authors identify vertical reading within a column as important for many table tasks.Source: §1 · pp. 1–2
  • Up to 74% of missing-cell identification tests failed, and GPT-3.5 failed on more than half of column-finding tests.The probes use real tables; missing-cell identification was repeated over 1,000 randomly sampled tables.Source: §1 · p. 2
  • Column-order swaps can change language-model responses even when the table’s semantic meaning should remain unchanged.The authors suggest that pretraining on order-sensitive text may contribute to this behavior.Source: §1 · p. 2
  • Table-tuning continues training model weights on diverse (instruction, table, completion) examples synthesized from real tables, rather than changing prompts for each task.The authors describe this as complementary to prompt engineering.Source: §1 · pp. 1–2
  • Table-GPT substantially outperforms 175B GPT-3.5 and ChatGPT across seen and unseen table tasks, and responds to novel table-task instructions.The authors also report that it excels in zero-shot and few-shot settings.Source: §1 · pp. 1–3
  • The authors propose task-, table-, instruction-, and completion-level data augmentation, and position Table-GPT as a starting point for downstream task-specific optimization.They state that the augmentation techniques help avoid overfitting and ensure generality.Source: §1 · pp. 2–3

2 PRELIMINARIES

The section reviews encoder- and decoder-style language models and their use for table tasks. It contrasts task-specific fine-tuning and prompt engineering with table-tuning, which aims to improve table-task ability while retaining general instruction-following.

2.1 Language models

  • Encoder-style models commonly use task-specific labeled data for downstream tasks, while decoder-style models can adapt to new tasks through natural-language instructions and optional examples.Task-specific labels may need to be obtained again when inputs or outputs change.Source: §2.1 · p. 3

2.2 Language models for table tasks

  • Encoder-style table models perform well on table tasks but generally need labeled fine-tuning to transfer across datasets or tasks, such as CTA label sets of 78 versus 107 types.The passage identifies TURL, TaBERT, Ditto, and Doduo as examples of encoder-style table models.Source: §2.2 · pp. 3–4
  • Table-tuning continues training underlying language-model weights once, rather than optimizing prompts or tuning separately for each task, and is complementary to prompt engineering.Carefully engineered instructions and examples can still benefit both vanilla GPT and Table-GPT.Source: §2.2 · p. 4
  • Table-tuning targets table understanding and table-task performance, while aiming to retain the ability to follow human instructions on new table tasks.The figure contrasts this target with instruction-tuning’s emphasis on generalizing to diverse instructions.Source: §2.2 · p. 4, Figure 5 · p. 4

3 CAN LANGUAGE MODELS “READ” TABLES?

The section probes whether language models can reliably read relational tables, emphasizing vertical column reading and order sensitivity. Its tests find weak accuracy on missing-cell identification and column-finding, alongside sensitivity to column order in table tasks.

3 CAN LANGUAGE MODELS “READ” TABLES?

  • Unlike sequential text, tables are two-dimensional and often require reading vertically within columns; swapping rows or columns generally leaves table meaning unchanged.The passage connects vertical reading to data imputation, error detection, and table question answering.Source: §3 · pp. 4–5
  • The missing-cell test removes a random cell and asks the model to identify its row and column, with variants that retain or remove the column separator.Removing the separator reflects a common, challenging CSV-parsing issue.Source: §3 · p. 5
  • Column-direction missing-cell accuracy was 0.38 with and 0.26 without column separators; with separators and few-shot demonstrations, accuracy was 0.51.The tests used 1,000 randomly sampled real tables.Source: §3 · p. 5, Table 1 · p. 5
  • GPT-3’s accuracy on the simpler Column-Finding test was 0.46, in which it had to name the column containing a value appearing once in the table.The test was repeated on 1,000 randomly sampled real tables.Source: §3 · p. 5
  • Language-model predictions for entity matching, error detection, and schema matching can be sensitive to the order in which table columns are presented.The passage notes that even slight column reordering can affect predictions.Source: §3 · p. 5

4 TABLE-TUNING FOR TABLE-GPT

This section presents Table-GPT’s table-tuning approach, which uses augmented instruction–table–completion examples synthesized from real tables to improve table-task training, and frames the model as a candidate table foundation model evaluated across four usage settings.

4.1 Overall approach: Synthesis-then-Augment

  • The proposed synthesize-then-augment pipeline creates table-task training data by sampling real tables and task types, synthesizing instruction–table–completion instances, then augmenting them.The resulting dataset covers 18 table-related tasks, including holdout tasks reserved for evaluating generalization; it combines synthesized tasks with existing benchmark data when automatic ground truth is unavailable.Source: Table 2 · p. 6, §4.1 · p. 7

4.2 Synthesize diverse table-tasks

  • The approach synthesizes easy table-understanding, augmentation, and manipulation tasks from real tables sampled from 2.9M high-quality web tables and 188K database tables extracted from BI data models to target task diversity.These tasks are intended as training data, not tests, to help models better read and understand tables.Source: §4.2 · p. 7
  • Experiments indicate that synthesized tasks on diverse tables improve task and data diversity and generalizability, while an ablation without them causes a substantial quality drop.The authors describe their task list as a starting point rather than an exhaustive set, suggesting that more diverse tasks could further improve table-tuning.Source: §4.2 · p. 8
  • For established tasks with limited benchmark training data, the authors synthesize new test cases from real tables, including data transformation, schema matching, imputation, error detection, and list extraction.The generated cases use task-specific procedures, such as removing a value for imputation or injecting a typographic error for error detection.Source: §4.2 · p. 8

4.3 Augment synthesized table-tasks

  • Augmentation expands synthesized examples through paraphrased instructions, row and column permutations or sampling, and completions that add reasoning steps.Table-level variants preserve semantic meaning and the completion, aiming to make models more stable to semantic-preserving table operations.Source: §4.3 · p. 9
  • For complex entity-matching tasks, reason-then-answer prompting outperforms direct answers qualitatively, motivating reasoning-augmented completions to encourage step-by-step reasoning on similar tasks.Direct answers are more prone to mistakes; GPT generates the augmented completions by adding reasoning to known yes/no answers.Source: §4.3 · p. 9
  • For error detection, chain-of-thought prompting does not resolve vanilla models’ false positives, so the method uses available ground truth to construct reasoning-augmented completions.These completions explain why a predicted value is misspelled and point to its correct version.Source: §4.3 · p. 9
  • Explanation-based augmentation grounds model predictions in actual explanations, substantially reducing false positives and improving result quality.Source: §4.3 · p. 10
  • Template-level augmentation mixes zero-shot and few-shot task templates, while task-level augmentation synthesizes new table-task types to increase training-data diversity.Source: §4.3 · p. 10

4.4 Table-GPT as “table foundation models”

  • Table-tuning trains language models on serialized instructions and tables as prompts, with completions as targets, minimizing completion loss subject to regularization.Source: §4.4 · p. 10
  • The proposed foundation-model evaluation compares table-tuned and original models in zero-shot, few-shot, task-specific prompt-tuning, and task-specific fine-tuning settings.The authors expect table-tuned models to perform better in most or all settings if table-tuning improves table understanding and manipulation.Source: §4.4 · p. 10

5 EXPERIMENTS

The experiments compare table-tuned models with their vanilla GPT-3.5 and ChatGPT counterparts across held-out and seen table tasks, and examine task-specific optimization, sensitivity, and augmentation ablations. Table-tuned models outperform their vanilla counterparts across most tests, with benefits also reported under task-specific optimization and multiple training conditions.

5.1 Experiment Setup

  • The evaluation compares paired GPT-3.5 and ChatGPT models on four unseen tasks and five seen tasks, with test datasets held out from table-tuning.The unseen tasks were not used in table-tuning; for seen tasks, test datasets were kept separate from training data.Source: §5.1 · p. 10

5.2 Quality Comparisons: Unseen + Seen tasks

  • Table-tuned models outperform vanilla counterparts in 98/104 tests across 26 datasets, two base models, and zero-shot and few-shot settings.The remaining tests comprise 3 ties and 3 losses.Source: Figure 8 · p. 11, Figure 9 · p. 11, §5.2 · p. 12

5.3 Benefits on task-specific optimizations

  • Table-GPT-3.5 consistently outperforms GPT-3.5 on the five best prompt-engineered templates for column-type annotation.The prompts were selected using 200 labeled examples from the Efthymiou dataset.Source: §5.3 · p. 12, Figure 10 · p. 12
  • With the same amount of task-specific training data, Table-GPT-3.5 dominates GPT-3.5 on column-type annotation and table question answering.The reported comparison also indicates that Table-GPT-3.5 requires fewer labeled examples to reach the same performance.Source: §5.3 · p. 12, Figure 11 · p. 12

5.4 Sensitivity Analysis

  • Performance across tasks rises as the number of training tasks increases, while performance on seen and unseen tasks improves with more training data before plateauing.With one training task, performance on other tasks generally falls below vanilla GPT-3.5.Source: §5.4 · p. 13, Figure 12 · p. 13, Figure 13 · p. 14
  • On unseen tasks, table-tuning yields little benefit for Ada, Babbage, and Curie, but a significant benefit for larger GPT-3.5 and ChatGPT models; Table-GPT-3.5 also leads GPT-3.5 by more than 10 percentage points across five prompt templates.The prompt-template comparison reports variation for both models, with Table-GPT-3.5 consistently ahead.Source: §5.4 · p. 13, Figure 14 · p. 14

5.5 Ablation Studies

  • Removing synthesized tasks significantly lowers average performance on seen and unseen tasks, while removing column permutations also lowers both averages.Removing prompt variations slightly lowers average performance, and removing chain-of-thought lowers performance on seen tasks.Source: §5.5 · pp. 13–14, Table 6 · p. 13

6 CONCLUSIONS AND FUTURE WORK

The paper proposes table-tuning: continuing to fine-tune pretrained language-model weights to improve table understanding and table-task performance while retaining versatility on unseen tasks. The authors present it as an initial step toward further work on models optimized for tables and table-related tasks.

6 CONCLUSIONS AND FUTURE WORK

  • Table-tuning continues to fine-tune pretrained language-model weights to improve table understanding and table-task performance while retaining versatility in following instructions for unseen tasks.The authors hope this initial work will serve as a springboard for further research on table-optimized models.Source: §6 · p. 14

A TASK DETAILS

This appendix presents prompts and sample outputs for a range of table tasks, including lookup and question answering, schema interpretation and matching, annotation and error detection, table modification and transformation, querying, summarization, extraction, filtering, entity matching, imputation, and sorting. Examples show different input layouts and prompting setups, including zero-shot and few-shot formats, with structured outputs and, in some cases, explanations or step-by-step reasoning.

Missing Value Identification (Column No Sep)

  • Missing-value prompts ask for the column containing the single blank cell, returning its name as JSON across zero-shot and few-shot examples.Examples cover tables with and without separators between cells.Source: A TASK DETAILS · pp. 16, 20

Missing Value Identification (Row Sep)

  • Other missing-value prompts ask for the row ID of the row containing the blank cell, returning that ID as JSON in row-separated and unseparated examples.Both zero-shot and few-shot examples are shown.Source: A TASK DETAILS · pp. 17–19

Column Identification

  • A separate prompt asks which column contains a supplied cell value and returns the column name as JSON.Examples include numeric and textual values in tables with named columns.Source: A TASK DETAILS · pp. 21–22

A TASK DETAILS

  • Examples cover cell-to-column lookup and natural-language table questions, with answers returned as JSON; one question asks how many nations appear.A separate prompt requests an explanation along with the answer.Source: A TASK DETAILS · pp. 23–24

Column Type Annotation Zero-Shot

  • Column type annotation asks for one candidate type that fits every column instance, or None when no candidate fits; examples include zero-shot and few-shot prompts.Few-shot examples label a building-name column as Building, while other columns receive Company or None.Source: A TASK DETAILS · pp. 25–26

Error Detection Zero-Shot

  • Error detection prompts request high-confidence erroneous cells, or None when no error is found, and examples identify misspelled entries or report no apparent errors.One example flags “Banetn” and “Central MJava”; another concludes that all cells appear correct.Source: A TASK DETAILS · pp. 29–30

Row/Column Swapping Zero-Shot

  • Row/column swapping examples ask the model to exchange specified row positions or column positions and return the resulting table.The zero-shot example swaps the second and third rows; the few-shot examples move the Group and % columns.Source: A TASK DETAILS · pp. 30–31

Schema Matching Zero-Shot

  • Schema matching asks for a mapping from every Table A column to a corresponding Table B column or None, returned as a list of header pairs.An example maps IATA to Airport Code and Country to Nation, while mapping ICAO to None.Source: A TASK DETAILS · p. 32

Column Mapping

  • A column-mapping example returns JSON pairs that associate source headers with corresponding headers or “None.”The shown mappings include Barony to Subdivision and County to None.Source: A TASK DETAILS · p. 33

Header Value Matching

  • Header value matching selects a candidate header for each column and returns the choices as a list, with both zero-shot and few-shot examples.Examples map table columns to headers such as crew, Portsmouth Number, and Type, or Name, Count, and Custom.Source: A TASK DETAILS · pp. 33–34

Column Augmentation

  • Column augmentation appends a generated column to the right of the input table in zero-shot and few-shot prompts.Examples add a Note column with statuses, an Album column, or a km/h column.Source: A TASK DETAILS · pp. 34–35

Natural-Language to SQL

  • Natural-language-to-SQL prompts ask for a SQL query answering a table question, with both zero-shot and few-shot examples returning JSON.The examples query a rank, a set score, or a team’s losses.Source: A TASK DETAILS · pp. 36–37

Table Summarization

  • Table-summary prompts request a JSON title or summary capturing the input table’s main information.Completions include identifying tables about Promi Big Brother, I Survived a Zombie Apocalypse, and Pennsylvania municipalities.Source: A TASK DETAILS · pp. 37–38

List Extraction

  • List extraction prompts transform a list into a plain-text, multi-column table using vertical bars and newline-separated rows.The examples return tables in JSON under the table key.Source: A TASK DETAILS · pp. 38–39

Row Augmentation

  • Row augmentation prompts generate and append a row at the bottom of a table in both zero-shot and few-shot settings.The examples extend a receptions ranking and product sales or fantasy-football tables.Source: A TASK DETAILS · pp. 39–41

Row/Column Filtering

  • A row/column filtering prompt selects the first row and returns a table containing the selected rows.The supplied example presents a table and asks for this selection.Source: A TASK DETAILS · p. 41

Row/Column Filtering Few-Shot

  • Few-shot row/column filtering asks for a selected column and returns that column's values as a table.The example selects Option_content, returning five False values.Source: A TASK DETAILS · p. 41

Row-to-Row Transformation Zero-Shot

  • The zero-shot row-to-row transformation prompt infers a pattern from input-output pairs and returns the final row's output value as JSON.In the example, the output follows the second component of the input, yielding 15.Source: A TASK DETAILS · p. 42

Entity Matching Zero-Shot

  • Entity-matching examples answer Yes when differently priced records describe the same product and No when product details differ.The examples compare descriptions, colors, and prices, including same-entity cases despite price differences.Source: A TASK DETAILS · p. 44

Data Imputation Zero-Shot

  • Data-imputation prompts request only a missing cell, with examples filling values as Korn and CCS.The examples use a music-credit table and tables containing area and ID columns.Source: A TASK DETAILS · pp. 44–45

Row/Column Sorting

  • Zero-shot row sorting orders records alphabetically by Gauge, while few-shot column sorting arranges headers in descending alphabetical order.The examples show Gauge values reordered and the Rank, State, and % columns rearranged as State, Rank, and %.Source: A TASK DETAILS · pp. 45–46

Related papers

Loading 2310.09263v1…