PaperlayerView exact revision on arXiv

Source-linked AI summary

Mind meets machine: Unravelling GPT-4's cognitive psychology

Sifatkaur Dhingra, Manmeet Singh, Vaisakh SB, Neetiraj Malviya, Sukhpal Singh Gill

arXiv:2303.11436v2cs.CLcs.AI

TL;DR

The paper addresses the limited evidence from existing GPT-4 cognitive-psychology evaluations by testing the model on four established datasets. It reports that GPT-4 greatly surpasses the language models used in the source studies, while noting a caveat in its HANS test data.

  • Problem

    Existing GPT-4 evaluations cover some cognitive-psychology tasks, but a broader assessment on established datasets is needed.

    Source: §1 · p. 2

  • Method

    The study evaluates GPT-4 on CommonsenseQA, MATH, SuperGLUE, and HANS using ChatGPT-Plus.

    Source: §2 · p. 2, §2.5 · p. 3

  • Results

    GPT-4 greatly surpassed the language models used in the source studies, with reported accuracy of 91.2% on SuperGLUE.

    Source: §4 · p. 4, §3 · p. 4

  • Takeaways & Limitations

    The authors present GPT-4 as a potential day-to-day resource for psychologists, conditional on thorough testing on basic reasoning and cognition tasks.

    Source: §4 · p. 4

Abstract

from arXiv · show

Cognitive psychology delves on understanding perception, attention, memory, language, problem-solving, decision-making, and reasoning. Large language models (LLMs) are emerging as potent tools increasingly capable of performing human-level tasks. The recent development in the form of GPT-4 and its demonstrated success in tasks complex to humans exam and complex problems has led to an increased confidence in the LLMs to become perfect instruments of intelligence. Although GPT-4 report has shown performance on some cognitive psychology tasks, a comprehensive assessment of GPT-4, via the existing well-established datasets is required. In this study, we focus on the evaluation of GPT-4's performance on a set of cognitive psychology datasets such as CommonsenseQA, SuperGLUE, MATH and HANS. In doing so, we understand how GPT-4 processes and integrates cognitive psychology with contextual information, providing insight into the underlying cognitive processes that enable its ability to generate the responses. We show that GPT-4 exhibits a high level of accuracy in cognitive psychology tasks relative to the prior state-of-the-art models. Our results strengthen the already available assessments and confidence on GPT-4's cognitive psychology abilities. It has significant potential to revolutionize the field of AI, by enabling machines to bridge the gap between human and machine reasoning.

1 Introduction

The introduction argues that existing GPT-4 evaluations do not provide a comprehensive assessment of cognitive psychology, motivating tests on established, more demanding datasets. The study evaluates GPT-4 on CommonsenseQA, SuperGLUE, MATH, and HANS, while describing further dataset testing as ongoing.

1 Introduction

  • The study evaluates GPT-4 on CommonsenseQA, SuperGLUE, MATH, and HANS to broaden cognitive-psychology assessment beyond the tasks tested in the GPT-4 report [4].The authors describe these datasets as needed for a comprehensive evaluation and say testing other suggested datasets is ongoing.Source: §1 · p. 2
  • The authors characterize sentence completion in HellaSwag [5] and pronoun-noun identification in WinoGrande as too simple to assess GPT-4's deeper heuristics.Source: §1 · p. 2
  • The introduction frames cognitive psychology as a way to study human information-processing mechanisms and capabilities.It also describes LLMs as text-trained systems that generate coherent, human-like responses to natural-language queries.Source: §1 · pp. 1–2

2 Datasets and Methodology

The methodology evaluates GPT-4 on four datasets spanning commonsense reasoning, mathematics, language understanding, and natural-language inference. The section describes each benchmark's task demands and relevant comparison context.

2.1 CommonsenseQA

  • CommonsenseQA contains 12,247 five-choice questions designed with ConceptNet to test commonsense knowledge; its source study reported 55.9% model accuracy and around 89% human accuracy.Source: §2.1 · p. 2

2.2 MATH

  • MATH contains almost 12,500 contest problems across seven mathematical areas and five difficulty levels, evaluated by exact match after answer normalization.Reported big-language-model accuracy ranges from 3.0% to 6.9%; human scores cited range from about 40% to 90%.Source: §2.2 · p. 2

2.3 SuperGLUE

  • SuperGLUE tests challenging language-understanding tasks, with about half its tasks below 1,000 instances and all but one below 10,000.The original study reported BERT about 20 points below humans and argued for advances in multi-task, transfer, and unsupervised or self-supervised learning.Source: §2.3 · p. 3

2.4 HANS

  • HANS probes whether NLI models rely on lexical-overlap, subsequence, or component heuristics rather than valid inference.It targets specific subsets where those heuristics should fail; the cited comparison found four NLI models' accuracy just above 0% in most situations.Source: §2.4 · p. 3

2 Datasets and Methodology

  • The study tests GPT-4 on CommonsenseQA, MATH, SuperGLUE, and HANS using ChatGPT-Plus from OpenAI.Source: §2 · p. 2, §2.5 · p. 3

2.5 Methodology

  • The methodology applies the four datasets to assess GPT-4's cognitive-psychology capabilities and reports evaluation in the results section.Source: §2 · p. 2, §2.5 · p. 3

3 Results

GPT-4's reported results span commonsense, mathematics, natural-language inference, and language understanding, with performance varying across tasks. The authors caution that the HANS result may reflect the all-non-entailment examples used in their test.

3 Results

  • The cited prior benchmarks report human accuracy of 76–97% on HANS and 40–90% on MATH, while GPT-2/GPT-3 scored below 10% on MATH.The cited HANS result also reports BERT below 10% on the non-entailment category.Source: §3 · pp. 3–4
  • GPT-4 scored 83.2% on CommonSenseQA, around 84% overall, 82% on prealgebra, 35% on geometry, 100% on HANS, and 91.2% on SuperGLUE.The prose attributes these results to Figure 1; its caption describes the datasets and their categories.Source: §3 · p. 4, Figure 1 · p. 4
  • The authors caution that GPT-4's 100% HANS result may reflect that all examples used were non-entailments; mixed-data experiments were ongoing.Source: §3 · p. 4

4 Conclusions

The conclusion presents GPT-4 as outperforming the language models used in the benchmark source studies and as a potential resource for psychologists. It links realizing those applications to thorough testing on basic reasoning and cognition tasks.

4 Conclusions

  • The authors report that GPT-4 greatly surpasses the language models used in the original studies from which the datasets were sourced.Source: §4 · p. 4
  • The conclusion presents GPT-4 as a potential day-to-day tool for psychologists and psychiatrists studying the mind and developing treatment theories and approaches.It says potential applications depend on thorough testing on basic reasoning and cognition tests.Source: §4 · p. 4
  • The authors suggest that GPT-4's development could contribute to addressing mental-health challenges through cascading benefits.Source: §4 · p. 4

Dataset/Task Sample Prompt GPT4 response

The examples pair prompts with GPT-4 responses across CommonsenseQA, MATH, and SuperGLUE, while another caption identifies an accuracy figure for cognitive-psychology tasks.

Dataset/Task Sample Prompt GPT4 response

Loading 2303.11436v2…