Source-linked AI summary
StructGPT: A General Framework for Large Language Model to Reason over Structured Data
Jinhao Jiang, Kun Zhou, Zican Dong, Keming Ye, Wayne Xin Zhao, Ji-Rong Wen
TL;DR
LLMs struggle to reason reliably over structured data whose schemas differ from their pretraining, motivating a unified zero-shot approach. StructGPT separates evidence collection from reasoning with specialized interfaces and iterative invoking-linearization-generation, and experiments across three structured-data types and eight datasets show improved performance comparable to supervised-tuning baselines.
Problem
LLMs may generate unfaithful information and may not fully understand structured-data formats and schemas, limiting reliable reasoning over such data.
Method
StructGPT uses specialized interfaces to collect relevant structured-data evidence, then iteratively invokes, linearizes, and generates with LLMs for reasoning.
Results
Experiments on eight datasets across KGQA, TableQA, and Text-to-SQL show improved zero-shot performance, with results comparable to full-data supervised-tuning methods.
Takeaways & Limitations
The framework provides a unified paradigm for improving LLM reasoning across knowledge graphs, tables, and databases.
Takeaways & Limitations
Evaluation uses instruction-following ChatGPT and Davinci-003 and covers only structured-data question answering, leaving other models and scenarios for future work.
Abstract
from arXiv · showhide
In this paper, we study how to improve the zero-shot reasoning ability of large language models~(LLMs) over structured data in a unified way. Inspired by the study on tool augmentation for LLMs, we develop an \emph{Iterative Reading-then-Reasoning~(IRR)} approach for solving question answering tasks based on structured data, called \textbf{StructGPT}. In our approach, we construct the specialized function to collect relevant evidence from structured data (\ie \emph{reading}), and let LLMs concentrate the reasoning task based on the collected information (\ie \emph{reasoning}). Specially, we propose an \emph{invoking-linearization-generation} procedure to support LLMs in reasoning on the structured data with the help of the external interfaces. By iterating this procedures with provided interfaces, our approach can gradually approach the target answer to a given query. Extensive experiments conducted on three types of structured data demonstrate the effectiveness of our approach, which can significantly boost the performance of ChatGPT and achieve comparable performance against the full-data supervised-tuning baselines. Our codes and data are publicly available at~\url{https://github.com/RUCAIBox/StructGPT}.
1 Introduction
StructGPT addresses the difficulty of applying LLMs to structured data by separating evidence collection from reasoning through specialized interfaces and iterative interaction. Across KGQA, TableQA, and Text-to-SQL, it improves zero-shot performance and approaches supervised-tuning baselines.
- LLMs can solve many tasks zero-shot, but may generate unfaithful information and lack domain-specific or real-time knowledge.
- Structured data supplies external knowledge but its schemas and formats may not be fully understood by LLMs.
- Specialized interfaces narrow the search space and help LLMs identify relevant evidence before reasoning.
- StructGPT uses iterative reading and reasoning through an invoking-linearization-generation procedure for structured-data tasks.The framework separates collecting relevant evidence from inferring answers or subsequent steps.
- 11.4% Hits@1 on WebQSP, 4.2% accuracy on TabFact, and 4.7% execution accuracy on Spider are reported gains over zero-shot ChatGPT.
2 Related Work
Prior approaches use task-specific architectures, pretrained encoders, or linearized structured data, but commonly require parameter tuning or lack broad transferability.
- Earlier structured-data systems used specialized architectures for graphs, tables, or semantic parsing, limiting generality and transfer across tasks.
- UnifiedSKG converts multiple reasoning tasks into text-to-text inputs but requires tuning model parameters.
- Recent LLM methods typically linearize structured data for prompting, with decomposition used for complex questions or data.
3 Preliminary
The paper formulates structured-data reasoning as question answering over accessible knowledge graphs, tables, or databases, where LLMs extract evidence and generate an answer or formal expression.
- Structured Data: Structured data follows a standardized format and logical data model, enabling efficient access through formal languages or algorithms.
- Structured Data: Knowledge graphs store factual knowledge as triples linking head entities and tail entities through relations.
- Structured Data: A data table contains indexed columns and rows, with each cell storing content at a column-row position.
- Structured Data: A database consists of multiple tables whose foreign keys link columns across tables.
- Problem Statement: The problem gives an LLM a natural-language question q and accessible structured data S, requiring evidence extraction followed by answer generation.
- Problem Statement: The evaluated task types are KGQA, TableQA, and database-based Text-to-SQL.
4 Approach
StructGPT treats structured data as a black-box system accessed through specialized interfaces, then iterates interface invocation, information linearization, and LLM generation to gather evidence and solve tasks.
- Motivation: A two-stage retrieval-style approach is insufficient because LLMs have limited capacity to directly handle structured data.
- Approach Overview: StructGPT provides interfaces for accurate and efficient access to structured data and iteratively uses them for reading and reasoning.
- Interfaces for Knowledge Graph: Knowledge-graph interfaces expose neighboring relations and triples so LLMs can select relevant paths from a topic entity.
- Interfaces for Table: Table interfaces expose column names, selected column contents, and relevant sub-tables containing chosen rows and columns.
- Interfaces for Database: Database interfaces expose table and column names plus table information and foreign keys for relevant-table selection.
- Reading and Reasoning with Interfaces: Each iteration invokes an interface, linearizes extracted evidence into text, and feeds it to the LLM to select data or predict an answer.
- Reading and Reasoning with Interfaces: Prompt generation supports either selecting useful evidence from linearized information or generating an answer or executable formal language.
- KG-based Question Answering: In KGQA, two sequential interface calls select relevant relations and triples before their tail entities form the answer.
5 Experiment
Experiments evaluate StructGPT across KGQA, TableQA, and Text-to-SQL using multiple datasets, metrics, LLMs, prompting settings, and supervised baselines. Results show consistent improvements from interface-assisted iterative reading and reasoning, while error analysis identifies dataset-specific failure modes.
- Experimental Setup: KGQA evaluates top-1 answer correctness with Hits@1, while TableQA uses denotation accuracy for WTQ and WikiSQL and accuracy for TabFact.KGQA also includes manual checking for cases where LLMs generate multiple answers.
- Experimental Setup: Experiments cover KGQA, TableQA, and DB-based Text-to-SQL across eight datasets, using zero-shot and few-shot settings with Davinci-003 and ChatGPT.The evaluation includes KGQA, three TableQA datasets, and three Text-to-SQL datasets, with supplementary experiments using a later ChatGPT version.
- Results: StructGPT substantially improves Davinci-003 and ChatGPT on KGQA, with further gains from few-shot exemplars and sequential evidence extraction through interfaces.The method progressively captures more detailed evidence for answering, especially on questions requiring multi-hop reasoning.
- Results: StructGPT consistently improves both tested LLMs across all three TableQA datasets, with additional gains from 32 in-context exemplars.Iterative access to relevant table information reduces the influence of irrelevant and redundant content.
- Results: Text-to-SQL results improve consistently across three datasets in both zero-shot and few-shot settings because the method extracts relevant tables and columns before SQL generation.The approach also uses 32 in-context exemplars to improve the mapping between natural-language questions and SQL queries.
- Error Analysis: Error analysis finds selection error dominant in WebQSP at 74% and reasoning error dominant in Spider at 62%, reflecting different task difficulties.WikiSQL shows relatively uniform frequencies among generation-format, selection, and reasoning errors.
6 Conclusion
StructGPT is a general framework for improving LLM zero-shot reasoning over structured data through specialized interfaces and iterative reading-then-reasoning. Across eight datasets, it substantially improves zero-shot performance and reaches performance comparable to full-data supervised-tuning methods.
- StructGPT improves LLM zero-shot reasoning over structured data through specialized interfaces and an invoking-linearization-generation procedure.The procedure is iterated sequentially so LLMs progressively capture more useful evidence before generating answers.
- The framework is implemented for KG-based question answering, table-based question answering, and database-based semantic parsing.
- Across 8 datasets, StructGPT boosts zero-shot LLM performance by a large margin.
- StructGPT achieves performance comparable to full-data supervised-tuning methods.
- The paper includes detailed error analysis identifying weaknesses of the approach.
7 Limitations
The evaluation is limited to instruction-following LLMs and question answering over structured data, while generated answer formats remain difficult to control.
- The experiments use ChatGPT and Davinci-003, so evaluation on LLMs with weaker instruction-following ability remains necessary.
- The study evaluates only question answering tasks based on structured data.Future evaluation is proposed for data-to-text and formal-language-to-text scenarios.
- Several generated texts contain format errors because answer formats are difficult to control across datasets.The paper identifies prompt design and answer parsing as areas for improvement.
A Experiment With Latest Version of LLM
Supplementary experiments with the latest August ChatGPT version show that model performance changes across versions, while StructGPT continues to improve performance on all three tasks.
- The latest August ChatGPT version has distinct performance from the June version, confirming that ChatGPT continuously evolves.
- The evolved ChatGPT underperforms the June version on WTQ.
- StructGPT further enhances the evolved ChatGPT performance on all three tasks.The authors interpret this consistency as evidence of robustness.
B Case Study
The case study illustrates StructGPT's iterative interface use across knowledge graphs, tables, and databases, progressively selecting relevant evidence before answering or generating SQL.
- KGQA: For KGQA, StructGPT extracts Harper Lee's neighboring relations, prompts the LLM to select education, and then continues interface-based evidence extraction.
- TableQA: For TableQA, the method extracts column names, selects District and Incumbent, identifies row item 8, and generates 19th.
- Text-to-SQL: For Text-to-SQL, the method extracts table and column names, selects Dogs and Breeds, and then extracts table information for SQL generation.
- Figure 4 presents representative case studies covering KGQA, TableQA, and Text-to-SQL.