PaperlayerView exact revision on arXiv

Source-linked AI summary

Can ChatGPT Understand Too? A Comparative Study on ChatGPT and Fine-tuned BERT

Qihuang Zhong, Liang Ding, Juhua Liu, Bo Du, Dacheng Tao

arXiv:2302.10198v2cs.CL

TL;DR

The paper addresses limited quantitative evidence about whether ChatGPT can understand language as well as it generates it. It evaluates ChatGPT on GLUE against four BERT-style models and finds task-dependent performance, with strong inference results, weaknesses in paraphrase and similarity, and further gains from advanced prompting.

  • Problem

    Quantitative evidence about ChatGPT's natural language understanding ability remains limited despite extensive study of its generation ability.

  • Method

    The paper evaluates ChatGPT on GLUE's natural language understanding tasks and compares it with four representative BERT-style models, also testing advanced prompting strategies.

  • Results

    ChatGPT excels on inference tasks, falls short on paraphrase and similarity, performs comparably to BERT-base on sentiment analysis and question answering, and improves with advanced prompting.

  • Takeaways & Limitations

    ChatGPT has understanding ability comparable to some fine-tuned BERT-style models, but remains behind the best models on average and on some NLU tasks.

  • Takeaways & Limitations

    Testing mainly uses subsets of validation sets and only the GLUE benchmark, whose task types are somewhat limited.

Abstract

from arXiv · show

Recently, ChatGPT has attracted great attention, as it can generate fluent and high-quality responses to human inquiries. Several prior studies have shown that ChatGPT attains remarkable generation ability compared with existing models. However, the quantitative analysis of ChatGPT's understanding ability has been given little attention. In this report, we explore the understanding ability of ChatGPT by evaluating it on the most popular GLUE benchmark, and comparing it with 4 representative fine-tuned BERT-style models. We find that: 1) ChatGPT falls short in handling paraphrase and similarity tasks; 2) ChatGPT outperforms all BERT models on inference tasks by a large margin; 3) ChatGPT achieves comparable performance compared with BERT on sentiment analysis and question-answering tasks. Additionally, by combining some advanced prompting strategies, we show that the understanding ability of ChatGPT can be further improved.

1 Introduction

This report asks whether ChatGPT's strong generation ability extends to natural language understanding, evaluating it against representative BERT-style models. ChatGPT excels on inference, is weaker on paraphrase and similarity, and benefits from advanced prompting while retaining performance gaps and limitations.

  • Motivation: The report evaluates whether ChatGPT performs well on natural language understanding tasks, an area with limited prior quantitative analysis.It uses the GLUE benchmark and compares ChatGPT with four representative BERT-style models.
  • Main findings: ChatGPT falls short on paraphrase and similarity tasks, particularly for negative paraphrase and neutral similarity samples.These weaknesses contrast with its stronger performance on other understanding tasks.
  • Main findings: ChatGPT outperforms all BERT-style models on inference tasks by a large margin.The report associates this result with impressive reasoning ability.
  • Main findings: ChatGPT achieves performance comparable to BERT-base on sentiment analysis and question-answering tasks.This finding places ChatGPT near a baseline fine-tuned BERT model on these task types.
  • Prompting strategies: Advanced prompting strategies consistently improve ChatGPT, with manual chain-of-thought prompting providing the greatest performance benefits.In-context learning is relatively sensitive to the provided examples, especially in the 1-shot setting.
  • Limitations: Despite improvements, ChatGPT remains behind fine-tuned RoBERTa-large in average performance and can generate contradictory or unreasonable responses.The report also notes that ChatGPT does not beat current state-of-the-art models on some NLU tasks.

2 ChatGPT vs. BERT

The report evaluates ChatGPT’s understanding on GLUE against fine-tuned BERT-style models, finding strong inference performance but substantial weaknesses on paraphrase and similarity tasks. ChatGPT reaches comparable performance to BERT-base overall, while prompting strategies and error analyses reveal both improvement opportunities and reasoning limitations.

  • 2.1 Evaluation Setting: The evaluation uses GLUE tasks spanning classification, textual similarity, inference, and question answering, with ChatGPT compared against four fine-tuned BERT-style models.ChatGPT receives task-specific prompts, while BERT and RoBERTa baselines are fine-tuned on each task’s full training set.
  • 2.2 Main Results: ChatGPT achieves comparable average performance to BERT-base (78.7% vs. 79.2%) but trails RoBERTa-large (87.8%).The results indicate basic understanding ability with substantial room for improvement.
  • 2.3 Analysis: ChatGPT performs poorly on paraphrase and similarity tasks, with performance drops of up to 24% relative to BERT-base.On MRPC, the drop reaches up to 47% for not_entailment samples; on STS-B, errors are especially large below 2.5 similarity scores and near the decision boundary.
  • 2.2 Main Results: ChatGPT is comparable to BERT-base on sentiment analysis, linguistic acceptability, and question-answering tasks.The reported tasks include SST-2, CoLA, and QNLI.
  • 2.3 Analysis: ChatGPT outperforms all BERT-style models on natural language inference tasks, including MNLI and RTE.The strongest advantage appears in entailment cases, where ChatGPT surpasses the more powerful BERT models by a clear margin.
  • 2.4 Case Study: Failure analyses show fluent but contradictory or unreasonable inference responses and confusion between coarse-grained similarity and fine-grained value differences in paraphrases.The report identifies fine-grained semantic extraction as a possible improvement target for paraphrase performance.

3 Improving ChatGPT with Advanced Prompting Strategies

The report tests whether advanced prompting strategies improve ChatGPT’s NLU performance and narrow its gap with fine-tuned BERT models. All tested strategies help, but example choice affects 1-shot results and RoBERTa-large remains stronger overall.

  • 3 Improving ChatGPT with Advanced Prompting Strategies: Advanced prompting is evaluated to improve ChatGPT’s understanding ability and narrow its performance gap with powerful BERT models.
  • 3.1 Advanced Prompting Strategies: Standard few-shot prompting feeds task-specific input-output exemplars to ChatGPT as part of the input.
  • 3.1 Advanced Prompting Strategies: Manual few-shot CoT provides intermediate reasoning demonstrations that guide ChatGPT toward step-by-step answers.
  • 3.1 Advanced Prompting Strategies: Zero-shot CoT uses a template-based prompt, including “Answer (yes or no) the question step by step.”, to elicit reasoning without manually designed demonstrations.
  • 3.2 More Results and Analyses: All advanced prompting strategies improve ChatGPT over the 78.7% zero-shot baseline; standard few-shot and zero-shot CoT add +5.1% and +5.0% average scores, while manual few-shot CoT adds up to +7.5%.Manual few-shot CoT empirically performs best and outperforms most BERT-style models except RoBERTa-large.
  • 3.2 More Results and Analyses: In-context learning is sensitive to the provided example, especially in the 1-shot setting, where ChatGPT performs worse on some tasks including CoLA, MRPC, MNLI, and RTE.The analysis examines five randomly sampled examples and relates performance to their textual similarity with test data.
  • 3.2 More Results and Analyses: More related 1-shot examples produce larger performance gains on CoLA, supporting the explanation that example–test correlation affects prompting outcomes.The authors attribute unrelated examples to lower correlation and cite prior work associating noisy 1-shot examples with catastrophic output-quality effects.
  • 3.2 More Results and Analyses: Manual few-shot CoT achieves SOTA performance among comparison models on some tasks, yet ChatGPT still trails fine-tuned RoBERTa-large, especially on MRPC.The authors report similar characteristics for ChatGPT in few-data and full-data RTE settings.

4 Related Works

The related work situates ChatGPT within Transformer-based pretrained language models and distinguishes BERT-style encoder-only models from GPT-style decoder-only models. It also notes emerging interest in ChatGPT’s capabilities, applications, ethics, and failures.

  • Transformer-based pretrained language models are grouped into encoder-only, decoder-only, and encoder-decoder architectures.
  • Combining multiple random examples may reduce noise effects, which might explain why 5-shot prompting generally works better.
  • The report calls encoder-only models such as BERT “BERT-style” and decoder-only models such as GPT “GPT-style.”
  • BERT-style models use bidirectional masked language modeling and can perform well on NLU tasks after task-specific fine-tuning.
  • Large language models exhibit few-shot and zero-shot learning abilities when model sizes are sufficiently large.
  • Research on ChatGPT increasingly examines its capabilities, applications, ethics, and failures.

5 Conclusion

The study finds that ChatGPT’s understanding ability varies across NLU tasks: it performs well on inference but struggles with paraphrase and similarity, particularly negative instances. Advanced prompting improves performance, though ChatGPT remains behind the best models on some tasks.

  • ChatGPT works well on inference tasks but falls short on paraphrase and similarity tasks, especially for negative instances.
  • Advanced prompting yields significant performance improvements and enables ChatGPT to outperform RoBERTa-large on some tasks.
  • ChatGPT has comparable understanding ability to some fine-tuned BERT-style models but does not beat the current best models on some NLU tasks.

Limitations

The evaluation is limited by partial validation-set sampling and reliance on the GLUE benchmark, whose task types are somewhat limited.

  • The study evaluates ChatGPT on only part of each task’s validation set and uses only GLUE, limiting sample coverage and task diversity.The authors propose evaluating more samples and additional NLU tasks in future work.

A.1 Details of Tasks

This section describes the GLUE benchmark tasks used to evaluate ChatGPT, covering linguistic acceptability, sentiment, semantic equivalence and similarity, inference, and question-answering.

  • CoLA classifies whether a single sentence is linguistically acceptable.
  • SST-2 predicts the sentiment expressed by a sentence.
  • MRPC, STS-B, and QQP evaluate semantic equivalence or similarity between sentence or question pairs.STS-B uses a 1-5 semantic similarity scale.
  • MNLI and RTE test textual inference by determining entailment, contradiction, or neither between premises and hypotheses.RTE specifically predicts whether the premise entails the hypothesis.
  • QNLI is a binary task that predicts whether a context sentence contains the answer to a question.The task is constructed from SQuAD.

A.2 Input Examples

This section presents ChatGPT input examples for the GLUE tasks using standard few-shot, zero-shot chain-of-thought, and manual few-shot chain-of-thought prompting.

  • The examples compare standard few-shot, zero-shot CoT, and manual few-shot CoT prompting formats.The examples are presented for each GLUE task in Tables 7 to 14.
  • Tables 7-9 provide prompting examples for CoLA, SST-2, and MRPC.
  • Tables 10-14 provide prompting examples for STS-B, QQP, MNLI, QNLI, and RTE.
Loading 2302.10198v2…