PaperlayerView exact revision on arXiv

Source-linked AI summary

TableLlama: Towards Open Large Generalist Models for Tables

Tianshu Zhang, Xiang Yue, Yifei Li, Huan Sun

arXiv:2311.09206v3cs.CLcs.AIcs.DB

TL;DR

Existing table methods often rely on table pretraining, specialized architectures, limited task coverage, or simplifying assumptions. This paper constructs TableInstruct and fine-tunes LongLoRA-based Llama 2 (7B) into TableLlama for diverse realistic table tasks. TableLlama performs comparably to or better than SOTA on almost all in-domain tasks and gains 5-44 absolute points over the base model on six out-of-domain datasets.

  • Problem

    Existing table-based methods often require table pretraining or specialized architectures, support limited table types and tasks, or make strong simplifying assumptions.

  • Method

    The paper constructs TableInstruct, a diverse instruction-tuning dataset, and fine-tunes LongLoRA based on Llama 2 (7B) to develop TableLlama.

  • Results

    TableLlama achieves similar or better performance than SOTA on almost all in-domain tasks and gains 5-44 absolute points over the base model on six out-of-domain datasets.

  • Takeaways & Limitations

    TableInstruct and TableLlama provide an open-source first step toward generalist models for diverse table-based tasks and improved generalization to unseen datasets and tasks.

  • Takeaways & Limitations

    TableInstruct omits tasks such as data imputation and table classification, so performance may vary with the complexity and specifics of unseen tasks and datasets.

Abstract

from arXiv · show

Semi-structured tables are ubiquitous. There has been a variety of tasks that aim to automatically interpret, augment, and query tables. Current methods often require pretraining on tables or special model architecture design, are restricted to specific table types, or have simplifying assumptions about tables and tasks. This paper makes the first step towards developing open-source large language models (LLMs) as generalists for a diversity of table-based tasks. Towards that end, we construct TableInstruct, a new dataset with a variety of realistic tables and tasks, for instruction tuning and evaluating LLMs. We further develop the first open-source generalist model for tables, TableLlama, by fine-tuning Llama 2 (7B) with LongLoRA to address the long context challenge. We experiment under both in-domain setting and out-of-domain setting. On 7 out of 8 in-domain tasks, TableLlama achieves comparable or better performance than the SOTA for each task, despite the latter often has task-specific design. On 6 out-of-domain datasets, it achieves 5-44 absolute point gains compared with the base model, showing that training on TableInstruct enhances the model's generalizability. We open-source our dataset and trained model to boost future work on developing open generalist models for tables.

1 Introduction

Existing table-based methods often depend on table pretraining, specialized architectures, narrow task or table coverage, or simplifying assumptions. The paper addresses this gap by introducing TableInstruct and TableLlama for realistic, diverse table tasks and evaluates their in-domain and out-of-domain capabilities.

  • Existing methods commonly require table pretraining or specialized architectures, support limited table types and tasks, or make strong simplifying assumptions.
  • The paper asks whether LLMs and instruction tuning can produce a generalist model that handles diverse realistic table tasks and unseen tasks.
  • TableInstruct contains 14 datasets and 11 tasks, with 8 training tasks and separate held-out datasets and tasks for out-of-domain evaluation.
  • TableInstruct combines realistic tables and tasks, including Wikipedia tables, scientific-report spreadsheets, complex semantic types, entity linking, numerical reasoning, table QA, and fact verification.
  • TableLlama is an open-source generalist model created by fine-tuning LongLoRA based on Llama 2 (7B) on TableInstruct for long-context table tasks.
  • Compared with task-specific SOTA models, TableLlama achieves similar or better performance on almost all in-domain tasks and gains 5-44 absolute points on six out-of-domain datasets over the base model.

2 TableInstruct Benchmark

TableInstruct is designed as a unified benchmark for one generalist model across diverse, realistic table tasks. It spans multiple task categories, standardized instruction-based inputs, and held-out datasets for out-of-domain evaluation.

  • TableInstruct aims to bridge multiple complex task-specific models and one generalist model without extra task-specific model-design efforts.
  • The benchmark selects representative tasks requiring different abilities, including table interpretation, augmentation, question answering, and fact verification.
  • TableInstruct includes 14 datasets covering 11 tasks, with eight datasets for training and in-domain evaluation and six held-out datasets for four out-of-domain tasks.
  • Its task categories include table interpretation, table augmentation, question answering, fact verification, dialogue generation, and data-to-text.
  • The benchmark uses real-world Wikidata entities and includes established datasets such as HybridQA, KVRET, FEVEROUS, ToTTo, WikiSQL, and WikiTQ for out-of-domain testing.
  • Each instance maps a manually designed instruction, serialized table input with metadata, and question to an output.

3 Experimental Setup

The experiments fine-tune a long-context LongLoRA model and compare TableLlama with baselines, specialized table models, and closed-source LLMs. Evaluation uses an 8192-token input length alongside task-specific implementation and sampling choices.

  • Model Construction: LongLoRA is selected because vanilla long-context fine-tuning becomes costly at the dataset scale, while shifted short attention approximates longer-context training with less computation.
  • Model Construction: TableLlama is obtained by fine-tuning LongLoRA on TableInstruct, with comparisons against SOTA models that may use table pretraining or specialized architectures.
  • Training Details: Training and inference use an 8192-token input length, with training performed on 48 A100 80GB GPUs for two epochs.
  • Training Details: Inference output length is set to 64 for most listed tasks and 512 for row population.
  • Evaluation: Downsampled test sets are used for column type annotation and entity linking, while row-population examples with more than 500 candidate entities are filtered out.
  • Baselines: GPT-4 evaluation uses the gpt-4-1106preview version, while GPT-3.5 evaluation uses gpt-3.5turbo-1106.

4 Result Analysis

TableLlama performs competitively across in-domain table tasks and improves zero-shot transfer to unseen datasets, while row population remains a notable weakness. Ablations suggest mixed-task instruction tuning, especially with table QA, supports broader generalization.

  • In-domain results: TableLlama matches or exceeds SOTA on almost all in-domain tasks without table pretraining or special table architecture.Most performance gaps are within 3 absolute points; gains reach 17.71 absolute points for entity linking, schema augmentation, HiTab, and FeTaQA.
  • In-domain results: 8 points over SOTA is reported for entity linking, where candidate entity names and descriptions may support identification.The proposed explanation is explicitly framed as a hypothesis about language models understanding candidate descriptions.
  • Limitations: TableLlama performs below SOTA by a large margin only on row population, which requires ranking many highly related candidate entities by their relation to a seed entity.The reported challenge is understanding the inherent relation between the athlete and candidate competitions.
  • Out-of-domain results: 5-44 points of gain over the base model occurs across 6 out-of-domain datasets, indicating transfer from TableInstruct to unseen table tasks and datasets.FEVEROUS shows the largest gain among the six datasets; the authors suggest same-task dataset transfer may be easier than cross-task transfer.
  • Evaluation caveat: Out-of-domain comparisons are zero-shot for TableLlama, whereas the reported SOTAs use full-dataset training, limiting direct comparability.Table 3 also marks systems using special table architectures, table pretraining, and a 500-example GPT-4 sample.
  • Open-source versus closed-source: TableLlama narrows or surpasses closed-source-model performance on 4 of 6 out-of-domain datasets, but closed-source models remain stronger on complex table QA.On in-domain tasks, TableLlama also outperforms the compared closed-source LLMs.
  • Ablation study: QA-trained models transfer best in ablations: HiTab-trained models exceed 20 points on 7 of 13 unseen datasets, while FeTaQA-trained models exceed 10 points on 7.Combining tasks also improves FEVEROUS accuracy to 73.77 versus 56.15 for TabFact-only training, and improves zero-shot transfer across table QA datasets.

5 Related Work

Prior table-learning work relies on specialized architectures, table pretraining, or task-specific designs, whereas this work unifies diverse realistic table tasks through instruction tuning and an open-source generalist model.

  • Existing table models use specialized architectures or table pretraining to encode table structure and knowledge.
  • These approaches remain data-specific and downstream task-specific, requiring tailored modeling efforts.
  • TableInstruct unifies diverse realistic table-based tasks, while TableLlama provides a one-for-all language model to reduce extra model-design effort.
  • Instruction tuning trains LLMs with instruction-output pairs to improve capabilities, controllability, and adaptation without extensive retraining or architecture design.

6 Conclusion

The paper presents TableInstruct and TableLlama as an open-source first step toward generalist LLMs for diverse table tasks, with strong table understanding and generalization results.

  • The paper constructs TableInstruct and develops TableLlama as open-source resources for generalist table-based modeling.
  • Experiments across in-domain and out-of-domain settings show that TableLlama gains strong table understanding and generalization abilities.

7 Limitations

Although TableInstruct covers 14 datasets across 11 table tasks, it omits some task types, and TableLlama’s performance may vary on unseen tasks and datasets.

  • TableInstruct does not include some table-based tasks, including data imputation and table classification.
  • On unseen table tasks and datasets, TableLlama’s performance may vary with their complexity and specific characteristics.

A Existing SOTA Models

Existing state-of-the-art table systems commonly use task-specific pretraining, architectures, modules, decoding strategies, or individually fine-tuned models.

  • TURL is a BERT-like encoder pretrained on 570K tables and requires task-specific modules fine-tuned on labeled data.
  • HiTab, TabFact, HybridQA, and WikiTQ state-of-the-art systems use specialized structural, alignment, fusion, or pretrained execution designs.
  • WikiSQL state-of-the-art modeling combines denoising objectives with clause-sensitive execution-guided decoding for SQL generation and answering.
  • FeTaQA, KVRET, and ToTTo use T5-3B fine-tuned on individual training data, while FEVEROUS uses the 20B FLAN UL2 model.

B.1 Data Selection

TableInstruct’s data selection prioritizes diversity, realisticness, and reliability across table-based tasks and sources, while separating in-domain training/evaluation from out-of-domain evaluation.

  • Diversity: TableInstruct covers 14 datasets spanning 11 tasks to provide diverse table-based task coverage.
  • Realisticness: The collection uses Wikipedia tables and NSF reports, including both simple and hierarchical tables with complex structures.
  • Reliability: Existing datasets widely used in NLP and database research provide the collection’s reliability basis.
  • TableInstruct is divided into in-domain training/evaluation and out-of-domain evaluation sets under constraints designed to keep task distributions disjoint.
  • Task selection emphasizes table interpretation, augmentation, structural understanding, and numerical reasoning.

B.2 Data Annotation

TableInstruct converts heterogeneous source datasets into a unified instruction-tuning format by combining task instructions, serialized tables, questions, and ground-truth responses.

  • Raw tables are stored in JSON, CSV, or text files, then paired with annotated instructions and questions based on task metadata.
  • Each example serializes the table and uses the ground truth as the response for model training and evaluation.

B.3 Quality Control

Quality control combines source-dataset cleaning, manual checks, annotator cross-checking, and detailed statistics describing TableInstruct’s contents.

  • The collected datasets had already been cleaned by previous authors before TableInstruct annotation.
  • 30 instances per task were randomly sampled for double-checking after annotation, with two annotators performing cross-checks.
  • Table 5 reports average text lengths, table dimensions, source types, task types, and whether tables are hierarchical.

D Case Study

The case-study prompts instantiate TableInstruct’s diverse tasks as instruction-following problems over serialized tables, metadata, candidates, highlighted cells, dialogue history, or passages.

  • Column Type Annotation: Column type annotation selects semantic types for a chosen column from candidate labels using page, section, and caption context.
  • Relation Extraction: Relation extraction selects relations between chosen table columns from candidate relations using contextual metadata.
  • Entity Linking: Entity linking maps a table-cell mention to a knowledge-base entity selected from candidates containing names, descriptions, and types.
  • Row Population: Row population ranks candidate entities for a selected column using page, section, caption, and header metadata.
  • Schema Augmentation: Schema augmentation ranks possible table headers given a caption, seed header, and header candidates.
  • Table Question Answering: Hierarchical and highlighted-cell question answering requires answers from tables, including complex hierarchical structures or designated cells.
  • Table Fact Verification: Fact verification distinguishes whether a statement is entailed or refuted by a table.
  • HybridQA and Dialogue: HybridQA answers questions using tables and passages, while table-grounded dialogue generation responds from dialogue history and tables.
Loading 2311.09206v3…