PaperlayerView exact revision on arXiv

Source-linked AI summary

Putting GPT-3's Creativity to the (Alternative Uses) Test

Claire Stevenson, Iris Smal, Matthijs Baas, Raoul Grasman, Han van der Maas

arXiv:2206.08932v1cs.AIcs.CLcs.HC

TL;DR

The paper tests whether GPT-3 can produce creative solutions on the Alternative Uses Test, comparing it with previously collected human responses. Across originality, surprise, semantic distance, utility, and flexibility, the results are mixed, though the authors conclude the student sample performed more creatively overall.

  • Problem

    It is unclear whether a language model trained to predict text can produce creative, useful, and surprising responses on a standard creativity task.

    Source: Introduction · pp. 1–2

  • Method

    The study compared GPT-3 and human responses to the Alternative Uses Test using expert ratings, semantic distance, and response-set flexibility.

    Source: Introduction · pp. 1–2, Procedure · p. 2

  • Results

    Humans scored higher in originality, surprise, and semantic distance, while GPT-3 scored higher in utility; humans also had higher tin-can flexibility, though GPT-3 flexibility varied more.

    Source: Do humans or GPT-3 provide more… · p. 3, Do humans or GPT-3 show more… · p. 3

  • Takeaways & Limitations

    The authors argue that comparing AI and human performance can offer insights into creativity and its measurement, even if the AUT measures only one aspect.

    Source: Discussion · p. 4

    The human sample was small, college-based, and Dutch-speaking, and translation to English may have lost creative subtleties.

    Source: Discussion · p. 4

Abstract

from arXiv · show

AI large language models have (co-)produced amazing written works from newspaper articles to novels and poetry. These works meet the standards of the standard definition of creativity: being original and useful, and sometimes even the additional element of surprise. But can a large language model designed to predict the next text fragment provide creative, out-of-the-box, responses that still solve the problem at hand? We put Open AI's generative natural language model, GPT-3, to the test. Can it provide creative solutions to one of the most commonly used tests in creativity research? We assessed GPT-3's creativity on Guilford's Alternative Uses Test and compared its performance to previously collected human responses on expert ratings of originality, usefulness and surprise of responses, flexibility of each set of ideas as well as an automated method to measure creativity based on the semantic distance between a response and the AUT object in question. Our results show that -- on the whole -- humans currently outperform GPT-3 when it comes to creative output. But, we believe it is only a matter of time before GPT-3 catches up on this particular task. We discuss what this work reveals about human and AI creativity, creativity testing and our definition of creativity.

Introduction

The paper asks whether GPT-3 can produce creative, useful, and surprising responses, and systematically compares its Alternative Uses Test performance with humans across several creativity measures.

Introduction

  • GPT-3’s creative output is tested against human responses on the Alternative Uses Test, which asks for original uses of everyday objects.The study compares expert ratings, semantic distance from the object, and flexibility across response sets.Source: Introduction · pp. 1–2
  • Creativity is framed as originality and usefulness, sometimes with surprise as an additional element (Runco and Jaeger, 2012).Surprise is a response that violates expectations and elicits interest (Boden, 2004; Simonton, 2018).Source: Introduction · p. 1
  • The study presents a systematic psychological assessment of LLM creativity using expert ratings, semantic distance, and response-set flexibility.The authors note that people tend to use a more flexible strategy on the AUT than on semantic-fluency tasks.Source: Introduction · p. 2

Sample

The sample combines previously collected responses from university students with GPT-3 AUT runs.

Sample

  • The human data comprised 823 responses from 42 University of Amsterdam students fluent in Dutch, while the GPT-3 sample comprised 144 AUT runs.The human data were previously collected, and participation was approved by the department’s ethics review board.Source: Sample · p. 2

Materials

Humans and GPT-3 were given the same three everyday objects, with GPT-3’s engine and prompt settings selected after optimization experiments.

Materials

  • The human AUT used Book, Fork, and Tin Can, asking participants to list as many creative uses as possible within two minutes.Participants entered each solution separately, and responses remained visible until the time limit.Source: Materials · p. 2
  • GPT-3 was prompted for creative uses of the same three objects after Monte Carlo experiments compared engines, prompts, and parameter settings.The optimization sought valid responses and higher snapshot creativity scores.Source: Materials · p. 2
  • The final GPT-3 administration used davinci-002, temperature .65–.80, and frequency and presence penalties of 1, collecting 820 responses over two sessions.The prompt requested 9 or 10 uses depending on the object.Source: Materials · p. 2

Procedure

Responses were standardized and rated by blinded judges, while semantic distance and response-set flexibility provided additional measures.

Procedure

  • Two trained, blinded judges rated originality, utility, and surprise on five-point scales after human and GPT-3 responses were translated and standardized.After invalid and incomplete responses were removed, inter-rater agreement was good to excellent across the three ratings.Source: Procedure · p. 2
  • Semantic distance was calculated as one minus cosine similarity between response and object embeddings, while flexibility was categories divided by responses.Three trained judges categorized responses; project data, code, translations, and protocols were made available online.Source: Procedure · pp. 2–3

Results

After cleaning, the analysis included responses from humans and GPT-3 runs; Figure 1 presents two sets of responses to the Tin Can task.

  • After cleaning, the analysis included 774 responses from 42 humans and 781 responses from 144 GPT-3 runs.Source: Results · p. 3
  • Figure 1 presents two sets of responses to the AUT Tin Can task.Source: Figure 1 · p. 3
  • The human and GPT-3 response sets in Figure 1 are presented for readers to distinguish.Source: Figure 1 · p. 3

Do humans or GPT-3 provide more original, useful or surprising responses?

Humans scored higher on originality, surprise, and semantic distance, while GPT-3 scored higher on utility; humans also had higher mean flexibility, while GPT-3 flexibility varied more.

Discussion

Humans outperformed GPT-3 on originality, surprise, semantic distance, and flexibility, while GPT-3 responses were rated more useful. The authors note limits in the AUT as a creativity measure, the GPT-3 tuning experiments, and the human comparison sample.

Discussion

  • Humans scored higher on originality, surprise, and semantic distance, while GPT-3 responses were rated more useful; the authors therefore judged the human responses more creative.They note that originality generally weighs more in creativity assessment (Diedrich et al., 2015); semantic distance is treated as a proxy for creativity.Source: Discussion · p. 4
  • Humans had higher flexibility scores, while GPT-3 scores varied more; the source of that variability is unclear and is not explained by temperature.Flexibility was computed as the number of categories in a response set divided by its total responses.Source: Discussion · p. 4
  • A central limitation is uncertainty about whether the Alternative Uses Task, a divergent-thinking task, measures creativity.The authors argue that comparing AI and human performance may still inform what creativity is and how to measure it.Source: Discussion · p. 4
  • The GPT-3 instruction-and-parameter Monte Carlo experiments were less fine-grained than the authors wanted; answers to other creativity tests also appeared verbatim from published materials.The authors suggest many creativity tests may have been present in GPT-3’s training data; the cited examples include the Remote Associates Test (Mednick, 1968) and insight problems (Sternberg and Davidson, 1982).Source: Discussion · p. 4
  • The human comparison used a small sample of native Dutch-speaking college students, and translation into English may have lost creative subtleties.Humans received minimal instructions while GPT-3’s instructions and parameters were optimized; the authors propose future comparisons using optimal instructions for both and the same language.Source: Discussion · p. 4
  • The authors conclude that GPT-3 was less creative than this psychology-student sample, although its performance was impressive and often appeared human-like.They raise whether creative process should matter in defining creativity, citing Boden (2004) and Simonton (2018).Source: Discussion · p. 4
Loading 2206.08932v1…