TableGPT2: A Large Multimodal Model with Tabular Data Integration

1.1 Motivation behind TableGPT2

• Current LLM workflows lack reliable access to external data, while NL2SQL and processing large tables in extended contexts remain inadequate for complex tabular tasks.The introduction characterizes relying on models without current market data, patient records, or full financial datasets as impractical, and describes cell-heavy context ingestion as inefficient.Source: §1.1 · p. 4
• Over 70% of global data is estimated to be structured tabular data, motivating large-scale modeling of tables together with schema metadata.The authors connect this data opportunity to their encoder’s capture of schema-level and cell-level information.Source: §1.1 · p. 4
• BI systems need flexible natural-language interactions, but existing LLMs and NL2SQL or table-comprehension approaches struggle with dirty schemas, missing values, complex business logic, and multi-step reasoning.The cited shortcomings also include computational efficiency, tabular understanding, and alignment with BI schemas and user demands.Source: §1.1 · pp. 4–5

1.2 Upon the Previous Version of TableGPT

• TableGPT2 advances TableGPT with scaled-up data and training, redesigned components, and technical enhancements for robustness and broader BI applicability.It offers 7B- and 72B-parameter Qwen2.5-based models, trained with over 86 billion CPT tokens, more than 437.5K table-language samples, and over 2.36M query-table-output tuples.Source: §1.2 · p. 5

1.3 General Introduction of TableGPT2

• TableGPT2’s training emphasizes coding, reasoning, tool use, and diverse BI-specific instructions across table querying, code generation, visualization, and analysis tasks.The SFT data includes over 20 combinations of table metadata inputs; query and table augmentation aim to improve robustness to noisy or incomplete tables and varied prompts.Source: §1.3 · pp. 5–7
• TableGPT2 combines a Qwen2.5 language framework with a separate tabular-reading modality and provides an agent workflow with retrieval, secure code execution, and evaluation components.The tabular module produces specialized embeddings concatenated with text-token embeddings; the agent runtime supports modular data-analysis steps and optional VLM support.Source: §1.3 · pp. 5, 7, 8
• The semantic encoder processes entire tables into compact column embeddings, using bidimensional attention without positional embeddings, hierarchical features, and column-wise contrastive learning.A Q-former-style adapter aligns column and text embeddings, while special <tab> and </tab> tokens distinguish tabular features during joint instruction tuning.Source: §1.3 · p. 7
• Across widely used tabular and general benchmarks, TableGPT2 improved on average by 35.20% for 7B and 49.32% for 72B, while retaining robust general-purpose performance.The comparison was limited to open-sourced benchmark-neutral models such as Qwen and DeepSeek [3]. The authors also introduce RealTabBench, which emphasizes unconventional tables, anonymized fields, and complex queries.Source: §1.3 · p. 8

• The open-source repository provides standalone decoder weights, a holistic agent workflow, and a portion of RealTabBench data.The decoder weights are on Hugging Face and the workflow is available through GitHub.Source: §2 · p. 8
• The standalone decoder works effectively when queries are unambiguous, column names are present, and tables are regular, as in most existing benchmarks.Source: §2 · p. 8
• The encoder-decoder configuration is recommended for complex, real-world production scenarios because encoder pretraining enhances table comprehension and robustness.Source: §2 · p. 8
• The encoder model is still being prepared, including complete integration with DeepSpeed for distributed training and vLLM for efficient inference.Source: §2 · p. 8
• Only a subset of RealTabBench is publicly released; the rest is reserved for a future public benchmark evaluating LLMs in realistic BI environments.Source: §2 · p. 8

3 Continual Pretraining from Qwen

• CPT adds college-level textbooks, data-analysis materials, and code to general corpora to support BI tasks while preserving broad language capabilities.The targeted skills include data comprehension, complex analysis, and code generation.Source: §3 · p. 8

3.1 Data Collection

• Coding data from StackOverflow and GitHub was deduplicated, while general data included finance, mathematics, and biology textbooks and Kaggle-sourced analysis materials.The curation emphasized numbers, code snippets, and tables to support reasoning across textual and structured data.Source: §3.1 · p. 9

3.2 Data Curation

• CPT data underwent document- and token-level filtering, including code tagging across 54 categories and Selective Language Modeling to retain valuable training tokens.Document filtering removed low-quality general content and sensitive or toxic corpora; code selection used FastText, Ray thread-pool processing, and heuristics. With Qwen2.5-7B-Instruct as reference, tokens were labeled at a 0.6 threshold, and below-threshold tokens were masked from loss computation.Source: §3.2 · pp. 9–10

3.3 Training Strategy

• Training used Qwen-2.5-Base with 8192-token contexts, a cosine schedule peaking at 3e-5, 3% warm-up, and BFloat16 mixed precision.Documents were shuffled, merged, and length-grouped into batches to reduce truncation and preserve complete samples where possible.Source: §3.3 · p. 10

4.1 Motivation

• Serialized tables can obscure structure, consume many tokens, and expose LLM reasoning to noise, ambiguity, and context bias; the proposed encoder provides a structure-aware, token-efficient alternative.Partial table context may mislead interpretation, while real-world tables are often noisy and ambiguous.Source: §4.1 · pp. 10–11

4.2 Model Architecture

• The encoder and adapter produce compact column representations from cell embeddings, using row- and column-wise attention without positional embeddings to preserve relational-table permutation invariance.A Q-Former-style adapter aggregates cell embeddings into k representations per column matched to the LLM embedding dimension.Source: §4.2 · p. 11
• The model dynamically inserts column embeddings alongside textual metadata, such as schemas and example values, so the LLM receives high-level table insights without requiring lossless compression.The hybrid representation combines column embeddings with metadata and specified values at the LLM’s embedding layer during inference.Source: §4.2 · p. 11

4.3 Training

• Training combines contrastive encoder pre-training, five table-language feature-alignment tasks, and joint instruction tuning of the encoder, adapter, and LLM.Pre-training pairs embeddings of the same columns across two same-schema snapshots with different sampled cell values. Feature alignment uses column prediction, cell prediction, question generation, table titling, and row summarization; the feature-alignment set contains 437K samples.Source: §4.3 · p. 12

5.1 Data Collection

• To add process-level supervision missing from existing TableQA and NL2SQL datasets, the authors collected 115K tables and 479K samples, alongside general, coding, and table-related data.SFT also includes single- and multi-turn interactions, while maintaining a balanced ratio of general data to support both general and table-specific performance.Source: §5.1 · pp. 12–13

5.1.1 Table Collection

• Collected tables span domains such as government, academia, finance, education, and healthcare, and include both regular database-like structures and irregular tables with merged cells or hierarchies.Sources include database, web, Excel, academic-task, special-format, and pre-test-task tables.Source: §5.1.1 · p. 13

5.1.2 Query Collection Based on Tables

• Queries cover eight BI categories—retrieval, insertion, deletion, modification, computation, statistics, visualization, and fuzzy queries—with fuzzy queries prompting clarification when information is missing.Queries are intended to reflect table contents while varying in instruction type and avoiding repetition.Source: §5.1.2 · p. 13
• Query generation synthesizes base questions with large LLMs, then uses human revision, category-specific prompts, domain jargon, table classifications, and retrieval-augmented generation to improve relevance and accuracy.Generated queries from both multi-turn scenarios are also reviewed and refined by human labelers.Source: §5.1.2 · p. 14
• Multi-turn query generation produced around 160K query sets across approximately 120K tables, covering 2–3 rounds and reflection/refinement or context-dependent follow-up scenarios.Some sets are truncated to a single query-answer sample when quality screening or generation or execution errors prevent completing the sequence.Source: §5.1.2 · p. 14

5.1.3 Answer Generation

• Single-turn answers target executable Python code, while multi-turn answers use reflection and refinement or context-aware dialogue; code execution checks and human review screen outputs.The process uses preprocessing to normalize table fields and values, postprocessing to filter non-executable code, and a code sandbox to verify answers.Source: §5.1.3 · pp. 14–15

5.1.4 Overall SFT Data Composition

• After 18 collection and refinement rounds, the final SFT dataset contains approximately 2.36M query-answer samples.Its composition mixes query types, agent calls, programming languages, and table tasks.Source: §5.1.4 · p. 15
• SFT tasks include Python and SQL code generation, querying, visualization, statistical testing, predictive modeling, comprehension, table generation, missing-value imputation, and table-based QA.The dataset combines general tasks with coding and table operations, including single- and multi-turn interactions, parsing, and transformation.Source: §5.1.4 · pp. 15–16

5.2.1 Table Selection and Column Manipulation

• Table filtering standardizes table orientation and removes duplicate, uninformative, overly long, incomplete, or undersized data and columns.Horizontal tables are transposed or removed; filters include fields over 100 characters, tables with more than 30% missing values, tables with fewer than 5 rows or 2 columns, and columns whose first value repeats the column name.Source: §5.2.1 · p. 16

5.2.2 Table-query-answer Tuple Cleaning

• Executable Python and SQL outputs are checked against table data, while heuristic filters remove mismatches, wrong code types, and comment-only outputs.Executor scripts filter errors including key errors and data-type conversion errors; regular-expression rules also remove generated table data that differs from the input.Source: §5.2.2 · p. 17
• Multiple models score samples on input accuracy, code detail, and output clarity; only samples above a threshold proceed through staged filtering.The pipeline first selects using queries plus table information, then filters the complete query, table-information, and code/text-output structure.Source: §5.2.2 · p. 17
• Manual sampling, fixed-set evaluation, and human review check executable accuracy and identify recurring errors or missing task capabilities.For executable code, reviewers manually verify 200–500 tuples; if accuracy is below 0.95, the team revises generation and filtering scripts. Human review also inspects validation-set failures and potential gaps such as function calls or multi-turn accuracy.Source: §5.2.2 · p. 17

5.3 Data Augmentation

• More than 20 table-info input constructions combine varied serialization formats with field descriptions, schema information, and value enumeration.Formats include HTML, XML, df.markdown(), df.to_string(), JSON, and custom serializations.Source: §5.3 · p. 17
• Removing quoted field or table names from BI queries reduces key-errors, especially in realistic BI cases, by avoiding direct query-to-schema mapping.The method targets ambiguity in BI-related queries.Source: §5.3 · p. 17
• Anonymizing table fields and categorical values improves robustness to dirty or noisy table data.Source: §5.3 · p. 18
• Combining single-turn and multi-turn QA data across prompt formats and output structures reduces sensitivity to system or user instructions and improves generalization.Source: §5.3 · p. 18
• Randomly adding chart-plotting instructions or summary explanations during data generation diversifies outputs.Source: §5.3 · p. 18

5.4 SFT with the Encoder

• Standard input-output SFT covers general-purpose tasks without structured tables, aligning common textual user queries with generated responses.Details of these general-task datasets are provided in Table 3.Source: §5.4 · p. 18
• For table-related SFT, a query-based transformer inspired by Q-Former [24] combines encoder embeddings of table information with the query so the decoder can produce answers from table-query-answer tuples.The mechanism is trained on table-query-answer tuples to process structured data efficiently.Source: §5.4 · p. 18
• The encoder is pretrained self-supervised on tasks such as schema inference, and general and table-related SFT are combined to handle unstructured and structured inputs while keeping TableGPT2’s general abilities.The model dynamically switches between the two modes depending on the task.Source: §5.4 · p. 18
• The experiments compared freezing the decoder while training the encoder and projection layers with end-to-end training of the entire model on general and table-related data.The configurations were explored to assess how to balance training between encoder and decoder components.Source: §5.4 · p. 18

6 Agent Framework

• The framework combines prompt engineering, a code sandbox, and agent evaluation with RAG-supported preparation and iterative agent actions (Figure 3).The workflow can optionally normalize input tables, generate SQL or Python, and use a VLM to analyze figures; execution may iterate based on intermediate results.Source: §6 · pp. 18–19, Figure 3 · p. 19

6.1 Prompt Engineering

• Prompt engineering pairs system instructions with RAG infrastructure to provide task context and guide agent responses.RAG retrieves relevant enterprise data and documents through embedding and reranking, limiting unnecessary context.Source: §6.1 · p. 19, §6.1.1 · p. 20, §6.2 · p. 20

6.1.1 System Instruction

• Layered system instructions define the agent’s role, analysis scope, response format, code-execution guidance, and limits for non-data questions.The instructions aim to align responses with real-world business requirements while supporting safety and compliance.Source: §6.1.1 · pp. 19–20

6.2 Retrieval-Augmented Generation

• RAG uses embedding and reranking to retrieve relevant schemas, business documents, and context, helping the agent generate code or interpret data within its context window.It supports schema and relationship retrieval across tables as well as domain-specific document retrieval for business analysis.Source: §6.2 · p. 20

6.3 Code Sandbox

• The sandbox runs agent-generated code in independent containers managed by Kubernetes, with resource quotas, timeouts, and automatic destruction of idle environments.Each execution is isolated from the host, and the sandbox can be destroyed and recreated after errors or improper behavior.Source: §6.3 · p. 21

7.1.1 Data Setups

• The evaluation combines existing benchmarks with RealTabBench, a new dataset of 360 real-world BI tables and 6,000 realistic, intricate queries.The existing benchmarks cover 27.7K tables and 88.9K test samples; the overall benchmark spans six major tasks across seven domains.Source: §7 · p. 21, §7.1.1 · pp. 22–23
• RealTabBench tests ambiguity and irregular layouts, and evaluates answer consistency, information completeness, and security.Final scores combine human reviewers and an Evaluation LLM.Source: §7.1.1 · p. 23

7.2 Results

• Without extensive training on any specific benchmark training set, TableGPT2 achieves significantly better results than nearly all other LLM-based methods and, on certain benchmarks, results superior or comparable to GPT-4o.Source: §7.2 · p. 24
• Over 60% absolute improvement in execution accuracy over the Qwen2.5 family is reported for HiTab’s hierarchically structured tables.Source: §7.2 · p. 24, Table 5 · p. 24

7.3 Qualitative Results

• Figure 4 presents qualitative comparisons across irregular, general, and ambiguous table scenarios.Source: §7.3 · p. 24, Figure 4 · p. 25

7.4 Miscellaneous

• TableGPT2 maintains strong performance on coding and language benchmarks, with no degradation in general competency.The evaluation includes MBPP, HumanEval, CMMLU, and MMLU.Source: §7.4 · pp. 24–25, Table 7 · p. 24

8 Outlook

• TableGPT2 achieves state-of-the-art table-task performance, but the authors say it is not yet ready to resolve real-world BI deployment challenges.They identify a gap between current capabilities and reliable use in production systems.Source: §8 · pp. 25–26

8.1 Coding in Specific Domains

• Production coding may require interpreters for enterprise-specific DSLs or pseudocode, balancing infrastructure fit and safety against flexibility.Encapsulated languages such as Logic-LM [70] are proposed as adaptable frameworks; an earlier TableGPT2 version used prompt-guided, SFT-reinforced hybrid DSL and general-code output.Source: §8.1 · p. 26

8.2 Multi-agent Design

• Automated agency design [71] routes queries through DAGs of specialized LLMs, while the authors report that complex real-world projects typically need more than two models.Their equity-and-fund app assigns intent recognition, code generation and tool use, and domain analysis to separate models; despite the promise of unified models, one LLM is currently insufficient for complex workflows, as MetaGPT [72] also suggests.Source: §8.2 · pp. 26–27

8.3 Tables Are Versatile

• Excel- and Pages-like tables contain irregular structures that remain difficult to handle, motivating pretraining for versatile formats overlooked by many corpora.The authors integrated a separately fine-tuned LLM to normalize irregular tables, but say substantial improvement remains possible.Source: §8.3 · p. 27

Data collection

• Data collection is attributed to Aowen Wang, Hao Chen, Kaizhe Shou, and Zhiqing Xiao.Source: Contribution List · p. 27

Continual pretraining

• Continual pretraining is attributed to Haokai Xu, Haoze Li, Qi Zhang, and Xiaomeng Hu.Source: Contribution List · p. 27

Supervised Fine-tuning

• Supervised fine-tuning is attributed to Ga Zhang, Jing Yuan, Liangyu Zha, Qingyi Huang, and Saisai Yang.Source: Contribution List · p. 27

Table encoder

• The table encoder is attributed to Chao Ye, Haoxuan Lan, Jiaming Tian, Lin Long, Liyao Li, Xijun Gu, and Xinjie Sun.Source: Contribution List · p. 27

Agent workflow

• The agent workflow is attributed to Aofeng Su, Chen Zhou, Junlin Zhou, Tao Zhang, and Xiang Li.Source: Contribution List · p. 27

Benchmark

• Benchmarking is attributed to Guangcheng Zhu, Pengzuo Wu, and Yuhang Yang.Source: Contribution List · p. 27