PaperlayerView exact revision on arXiv

Source-linked AI summary

Solving and Generating NPR Sunday Puzzles with Large Language Models

Jingmiao Zhao, Carolyn Jane Anderson

arXiv:2306.12255v1cs.CL

TL;DR

Whether LLMs can generalize to wordplay puzzles remains underexplored. The authors introduce PUZZLEQA and evaluate models on puzzle solving and generation; GPT-3.5 solves 50.2% of free-response problems, but the study finds no evidence that models can generate playable puzzles.

  • Problem

    How well LLMs generalize to wordplay puzzles remains underexplored, despite their strong performance on a variety of language tasks.

    Source: Benchmarking AI through Games · p. 1

  • Method

    The authors introduce PUZZLEQA and use few-shot experiments to evaluate LLMs on puzzle solving and generation.

    Source: Introduction · p. 1, Free Response Experiments · p. 2

  • Results

    GPT-3.5 solves 50.2% of free-response problems, the best publicly available model achieves 33%, and the study finds no evidence that LLMs can generate new puzzles for human players.

    Source: Conclusion · p. 4, Results · p. 4

  • Takeaways & Limitations

    Generating NPR Sunday Puzzle-style games remains beyond the capabilities of current LLMs, even when they can solve puzzles.

    Source: Conclusion · p. 4

    Because training data for GPT-3, GPT-3.5, and LLaMA is not public, the authors cannot measure whether those models were trained on PUZZLEQA problems.

    Source: Limitations and Future Work · p. 4

Abstract

from arXiv · show

We explore the ability of large language models to solve and generate puzzles from the NPR Sunday Puzzle game show using PUZZLEQA, a dataset comprising 15 years of on-air puzzles. We evaluate four large language models using PUZZLEQA, in both multiple choice and free response formats, and explore two prompt engineering techniques to improve free response performance: chain-of-thought reasoning and prompt summarization. We find that state-of-the-art large language models can solve many PUZZLEQA puzzles: the best model, GPT-3.5, achieves 50.2% loose accuracy. However, in our few-shot puzzle generation experiment, we find no evidence that models can generate puzzles: GPT-3.5 generates puzzles with answers that do not conform to the generated rules. Puzzle generation remains a challenging task for future work.

Introduction

This paper tests whether large language models can solve and generate NPR Sunday Puzzles, which combine information retrieval, wordplay, and pattern recognition. It introduces PUZZLEQA and finds that GPT-3.5 solves many puzzles but does not generate playable games.

  • LLMs have achieved strong performance on novel tasks through few-shot learning and prompt engineering, motivating their evaluation on puzzles.Source: Introduction · p. 1
  • GPT-3.5 solves 50.2% of PUZZLEQA puzzles but cannot generate playable games.Source: Introduction · p. 1
  • The study asks whether few-shot learning and prompt engineering can help LLMs solve NPR Sunday Puzzle questions.Source: Introduction · p. 1

Benchmarking AI through Games

The paper positions wordplay puzzles as an underexplored way to assess LLM generalization, presents PUZZLEQA, and evaluates several models. The benchmark comprises 558 NPR Sunday Puzzles from 2007–2021, and the experiments use four LLMs.

Benchmarking AI through Games

  • Wordplay puzzles offer an underexplored test of LLM generalization beyond their performance on language tasks such as question-answering.Source: Benchmarking AI through Games · p. 1
  • NPR Sunday Puzzles combine information retrieval, wordplay, and pattern recognition, while requiring less specialized knowledge than Jeopardy! and Quiz Bowl.Source: Benchmarking AI through Games · p. 1

Dataset

  • PUZZLEQA contains 558 NPR Sunday Puzzle games from 2007–2021.Source: Dataset · p. 1

Model Selection

  • The experiments compare publicly available GPT-J and LLaMA with proprietary GPT-3 Davinci and GPT-3.5, using temperature 0.1 as optimal.Source: Model Selection · p. 2

Multiple Choice Experiments

The multiple-choice task supplies three answers from the same puzzle alongside the correct answer. GPT-3 and GPT-3.5 solve 78% of problems, while the answer-only baseline checks for bias in the options.

Free Response Experiments

The experiments compare few-shot free-response prompting strategies and find that small-scale prompt-engineering gains do not carry over to the full dataset. GPT-3.5 performs best on the full PUZZLEQA set, while its results are sensitive to prompt wording.

Evaluation Metrics

  • Free-response evaluation uses exact matching and loose matching because correct answers can have different phrasings.Exact matching requires the gold solution verbatim; loose matching accepts containment after removing nonalphabetic characters and lowercasing.Source: Evaluation Metrics · p. 2

Exploring Prompt Engineering Techniques

  • The prompt-engineering subset contains 110 randomly sampled questions, with 10 questions from each of 11 categories.The two techniques explored are summarization and chain-of-thought reasoning.Source: Exploring Prompt Engineering Techniques · p. 2

Summarization

  • Summarization tests whether making informally described puzzle rules more consistent helps the model solve them.GPT-3.5 summarizes each puzzle description into a more consistent format.Source: Summarization · p. 2, Figure 3 · p. 3

Chain-of-thought Reasoning

  • Chain-of-thought prompting asks the model to explain its reasoning before answering, and generated explanations can be used as few-shot examples.The authors automate explanation generation for rule-question-answer triplets; one example explains changing “blouse” to “blue” by dropping two letters.Source: Chain-of-thought Reasoning · p. 3, Figure 4 · p. 3

Prompt Engineering Results

Free Response Results

  • The full-dataset comparison evaluates the selected prompt-engineering technique against a baseline few-shot model.The technique was selected based on the small-scale experiment.Source: Free Response Results · p. 3
  • GPT-3.5 solves 50.2% of full-dataset puzzles with the baseline prompt, while small-scale prompt-engineering gains do not replicate across PUZZLEQA.Rephrasing the chain-of-thought prompt from “reasoning and answer” to “answer and reasoning” substantially lowers performance.Source: Free Response Results · p. 3

Discussion

  • Chain-of-thought prompting produces 87 “N/A” responses versus 3 baseline refusals, and its explanations may offer a window into model decisions.The authors report that most manually examined explanations are consistent with the answer, while noting that some puzzles are faulty because of web-scraping errors.Source: Discussion · p. 3

Game Generation

The authors test few-shot puzzle generation with GPT-3.5, using example puzzle triplets and evaluating consistency and conformity. Although the model answers many generated questions consistently, few outputs follow their explanations, and the conforming games are trivial.

Game Generation

  • The generation experiment uses GPT-3.5, the model with the highest free-response performance, to test whether LLMs can create puzzles for humans.The authors introduce generation after finding that prompt-engineering gains did not hold across the full dataset.Source: Game Generation · p. 3

Prompt Design

  • Each generation prompt contains five example puzzles as explanation-question-answer triplets, drawn from a balanced 110-question subset across 22 prompts.The model is then asked to generate a puzzle.Source: Prompt Design · p. 3, Figure 6 · p. 4

Evaluation Metrics

  • Consistency checks whether GPT-3.5 can solve its own generated puzzle, while conformity checks whether consistent answers follow the explanation’s rules.Conformity is assessed manually.Source: Evaluation Metrics · p. 4

Results

  • Of 22 GPT-3.5-generated games, 17 are answered consistently, but only 6 conform to their explanations, and the conforming games are trivial.The authors find no evidence that LLMs can generate new puzzles for human players.Source: Results · p. 4, Table 1 · p. 4

Limitations and Future Work

The authors note limits in measuring performance and dataset coverage, and that year-to-year accuracy varies without a clear trend. They also cannot rule out training overlap for most tested puzzles.

Limitations and Future Work

  • Training-data overlap cannot be ruled out because GPT-3, GPT-3.5, and LLaMA training data are not public; a 2023 subset (n=116) performed on par with the full dataset.The 2023 questions postdate the stated training-data cutoffs for those models.Source: Limitations and Future Work · p. 4
  • Exact accuracy varies by puzzle air date, but performance shows no clear trend across years.Source: Figure 7 · p. 4, Limitations and Future Work · p. 4
  • Web scraping missed some questions, loose accuracy may miss valid answers, and future work could assess difficulty using whether human players answered correctly.Source: Limitations and Future Work · p. 4

Conclusion

On PUZZLEQA, GPT-3.5 solves some puzzles, but the authors find no evidence that current LLMs can generate playable NPR-style puzzles. Prompt engineering did not improve performance, though chain-of-thought may aid explainability.

Conclusion

  • GPT-3.5 solves 50.2% of free-response problems, while the best publicly available model achieves 33%.The authors note that GPT-3.5 training data are not public.Source: Conclusion · p. 4
  • The explored prompt-engineering techniques failed to improve performance, although the authors argue chain-of-thought reasoning remains helpful for explainability.Source: Conclusion · p. 4
  • The authors find no evidence that current LLMs can generate NPR-style games, even when they can solve them.Source: Conclusion · p. 4
  • Puzzle types and topics may vary over time, and the authors identify in-depth content analysis as future work.They suggest exploring fine-tuning on PUZZLEQA rather than few-shot learning.Source: Conclusion · pp. 4–5
Loading 2306.12255v1…