Source-linked AI summary
Can ChatGPT Replace Traditional KBQA Models? An In-depth Analysis of the Question Answering Performance of the GPT LLM Family
Yiming Tan, Dehai Min, Yu Li, Wenbo Li, Nan Hu, Yongrui Chen, Guilin Qi
TL;DR
The paper asks whether ChatGPT and related LLMs can replace traditional KBQA models despite limited large-scale evidence and difficult answer evaluation. It introduces a CheckList-inspired black-box framework with unified labeling and extended answer matching, then evaluates GPT-family models and FLAN-T5 on approximately 190,000 questions across eight datasets. GPT-family models outperform traditional models on some older datasets but lag behind current state-of-the-art models on newer datasets, while GPT-family and FLAN-T5 show similar question-type strengths and weaknesses.
Problem
Existing ChatGPT evaluations use limited test cases, while textual LLM outputs make traditional exact-match evaluation difficult for comprehensive KBQA testing.
Method
The paper combines unified question-feature labeling, extended answer matching, and CheckList-based MFT, INV, and DIR tests across eight KB-based CQA datasets.
Results
GPT-family models outperform the best traditional models on WQSP and LC-quad2.0 but lag behind current state-of-the-art models on KQApro and GrailQA; GPT-family and FLAN-T5 show similar strengths and weaknesses.
Takeaways & Limitations
The findings indicate that GPT-family LLMs have dataset- and question-type-dependent KBQA strengths but do not consistently replace current traditional KBQA models.
Abstract
from arXiv · showhide
ChatGPT is a powerful large language model (LLM) that covers knowledge resources such as Wikipedia and supports natural language question answering using its own knowledge. Therefore, there is growing interest in exploring whether ChatGPT can replace traditional knowledge-based question answering (KBQA) models. Although there have been some works analyzing the question answering performance of ChatGPT, there is still a lack of large-scale, comprehensive testing of various types of complex questions to analyze the limitations of the model. In this paper, we present a framework that follows the black-box testing specifications of CheckList proposed by Ribeiro et. al. We evaluate ChatGPT and its family of LLMs on eight real-world KB-based complex question answering datasets, which include six English datasets and two multilingual datasets. The total number of test cases is approximately 190,000. In addition to the GPT family of LLMs, we also evaluate the well-known FLAN-T5 to identify commonalities between the GPT family and other LLMs. The dataset and code are available at https://github.com/tan92hl/Complex-Question-Answering-Evaluation-of-GPT-family.git
1 Introduction
The paper develops a large-scale black-box evaluation of GPT-family LLMs for complex KBQA, motivated by limited prior testing and difficult answer matching. Across eight datasets, GPT-family models outperform traditional models on some older datasets but lag behind current state-of-the-art models on newer ones.
- Prior evaluations used few test cases and faced difficulty applying exact match because LLMs generate explanatory text rather than precise knowledge-base answers.
- The framework combines unified feature labeling, improved answer evaluation, and CheckList tests for functionality, stability, and controllability.The tests are MFT, INV, and DIR.
- The evaluation covers approximately 190,000 questions from six English and two multilingual real-world KB-based CQA datasets.The multilingual portion contains approximately 12,000 questions across 13 languages.
- GPT-family LLMs outperform the best traditional models on WQSP and LC-quad2.0 but lag behind current state-of-the-art models on KQApro and GrailQA.
- GPT-family LLMs and FLAN-T5 show similar strengths and weaknesses across different question types.
- Chain-of-thought prompting improves performance on some specific questions but can negatively affect other question types.
2 Related Work
Related work motivates combining broad benchmark-style labeling with task-specific black-box testing. The paper follows CheckList’s MFT, INV, and DIR framework while using prompting methods such as chain-of-thought.
- Chain-of-thought prompting is presented as a method that can help LLMs understand and reason more when answering questions.
- The paper combines HELM-inspired feature labeling, manually predefined testing objectives, and CheckList black-box testing for broader GPT-family evaluation.
- CheckList evaluates minimum functionality, invariance to irrelevant input changes, and directional responses to modified inputs.
- The paper emphasizes black-box evaluation because the expense of training LLMs makes white-box testing impractical.
3 Evaluation Framework
The evaluation framework standardizes question features, adapts answer matching to textual LLM outputs, and applies CheckList-based tests to complex KBQA behavior. It covers reasoning, robustness, and controllability across diverse datasets and question transformations.
- 3 Evaluation Framework: The framework assigns unified answer-type, reasoning-type, and language-type tags and evaluates LLM answers with an improved exact-match strategy.The framework is organized into unified labeling and answer evaluation components.
- 3.2 Answer evaluation: Extended answer matching extracts noun-phrase or verb-phrase candidates, expands reference answers with multilingual aliases, and applies cosine similarity.Fuzzy matching is restricted to non-NUM, DATE, and Boolean answer types.
- 3.2 Answer evaluation: 0.78 is selected as the empirical cosine-similarity threshold after manual verification of 3,000 samples and comparison of false rates.The threshold minimizes average false rates across models, while exact match alone produced 2.38%-4.17% false negatives.
- CheckList tests: MFT compares single-operation with multi-operation reasoning, while INV tests spelling-error and paraphrase robustness.
- CheckList tests: DIR tests controllability by modifying reasoning phrases, specifying answer types, and using multi-round questioning inspired by chain-of-thought.
4 Experiments
Across eight KB-based CQA datasets, the experiments compare GPT-family models with traditional KBQA systems and FLAN-T5, finding both broad gains and persistent weaknesses by dataset, language, question type, and reasoning operation.
- 4.3 Overall results: GPT-4 surpasses current SOTA traditional models on four of eight test sets, while ChatGPT does so on three, showing zero-shot performance approaching or exceeding traditional systems.Table 3 reports Accuracy or F1 depending on the dataset.
- 4.3 Overall results: Newer GPT models generally improve over earlier generations, with increasingly larger gains across generations and similar performance profiles across datasets.The GPT-family curves have nearly identical line shapes, indicating shared patterns across models.
- 4.4 Multilingual KBQA results: GPT-family models improve multilingual question answering overall, but GPT-4 fails to surpass ChatGPT in four languages, suggesting that multilingual progress may be slowing.The multilingual comparisons use MKQA and QALD-9 results and examine EM by language.
- 4.5 Feature tags based results: GPT models perform best on boolean, organization, location, set-operation, and numerical-comparison questions, but struggle with precise dates, numerical calculations, counting, and some multi-generation cases.FLAN-T5 is weaker in most categories but performs similarly to ChatGPT on boolean questions.
- 4.6 CheckList results: GPT-family stability improves through GPT-4, reaching 91.70 and approaching the reported 100 stability of traditional KBQA models, while reasoning-operation scores around 73% still indicate substantial randomness.The stability analysis uses three runs across three test cases; the reasoning-operation finding concerns the latest GPT models.
- 4.6 CheckList results: Chain-of-thought prompting improves GPT-4 most strongly and raises numerical-answer performance by more than 30.00 points, while helping set operations, filtering, and counting more than multi-hop or star-shaped questions.The reported gains vary by model and reasoning type.
5 Conclusion
The study finds that GPT models’ question-answering performance and reliability improve across versions, but important reasoning weaknesses remain. CheckList testing and CoT-inspired prompts provide evaluation and improvement directions, while broader domains and model families remain for future study.
- GPT models’ question-answering performance and reliability continuously improve across versions, approaching traditional models.
- CheckList testing shows that current LLMs still have substantial room to improve in some reasoning abilities.
- CoT-inspired prompts improve the original model’s performance on certain question types.
- Future work should test multiple domains to distinguish conclusions that are universal from those specific to open-domain KBQA.
- Future research should also evaluate additional model types, including newer large-scale open-source models.