PaperlayerView exact revision on arXiv

Source-linked AI summary

A fine-grained comparison of pragmatic language understanding in humans and language models

Jennifer Hu, Sammy Floyd, Olessia Jouravlev, Evelina Fedorenko, Edward Gibson

arXiv:2212.06801v2cs.CLcs.AI

TL;DR

Pragmatic understanding is essential to communication, yet building language models that capture human pragmatic behavior remains a long-standing challenge. This paper compares humans and models across seven pragmatic phenomena using expert-curated English materials and zero-shot prompting. The largest models show high accuracy, human-like error patterns, and preliminary alignment in linguistic cues, while struggling with phenomena involving social expectations.

  • Problem

    Building models that capture human pragmatic behavior remains a long-standing NLP challenge, and prior evaluations provided limited insight into distinct ways models fail.

    Source: §1 · p. 1

  • Method

    The authors compare humans and language models on expert-curated English materials spanning seven pragmatic phenomena, using zero-shot prompting and analyses of choices, errors, and linguistic cues.

    Source: §1 · p. 2

  • Results

    The largest models achieve high accuracy and match human error patterns, favoring literal interpretations over heuristic distractors when incorrect; preliminary evidence also suggests similar sensitivity to linguistic cues.

    Source: §1 · p. 2, §6 · p. 9

  • Takeaways & Limitations

    The findings suggest that some pragmatic behaviors can emerge without explicitly constructed representations of agents’ mental states.

    Source: §1 · p. 2

    The evaluation uses only English materials, and some tasks reflect Western cultural norms; the authors caution against extrapolating these results to arbitrary inputs or downstream behavior.

    Source: Limitations · p. 10

Abstract

from arXiv · show

Pragmatics and non-literal language understanding are essential to human communication, and present a long-standing challenge for artificial language models. We perform a fine-grained comparison of language models and humans on seven pragmatic phenomena, using zero-shot prompting on an expert-curated set of English materials. We ask whether models (1) select pragmatic interpretations of speaker utterances, (2) make similar error patterns as humans, and (3) use similar linguistic cues as humans to solve the tasks. We find that the largest models achieve high accuracy and match human error patterns: within incorrect responses, models favor literal interpretations over heuristic-based distractors. We also find preliminary evidence that models and humans are sensitive to similar linguistic cues. Our results suggest that pragmatic behaviors can emerge in models without explicitly constructed representations of mental states. However, models tend to struggle with phenomena relying on social expectation violations.

1 Introduction

Pragmatic understanding—the ability to go beyond literal semantics—is essential to communication, yet building models that capture human pragmatic behavior remains a long-standing NLP challenge. This study compares humans and language models across seven phenomena, examining pragmatic choices, error patterns, and linguistic cues; the largest models achieve high accuracy and show human-like errors, but struggle with social expectations.

1 Introduction

  • Earlier empirical evaluations often reduced responses to a pragmatic/non-pragmatic distinction, obscuring whether errors favored literal readings or socially inappropriate non-literal ones.Distinguishing these error patterns may suggest model-improvement directions and foreshadow user-facing problems.Source: §1 · p. 1
  • Comparing models with humans also offers a way to investigate whether pragmatic competence can arise without explicit mental-state representations.Human comprehension may involve mentalizing or lower-cost heuristics, and the balance between them remains open.Source: §1 · p. 1
  • The study compares humans and language models on seven pragmatic phenomena using expert-curated English multiple-choice materials and zero-shot prompting.It asks whether models select pragmatic interpretations, make human-like errors, and use similar linguistic cues.Source: §1 · p. 2
  • The largest models achieve high accuracy and mirror human response patterns; when wrong, they favor literal answers over heuristic-based distractors.Models also show preliminary sensitivity to linguistic cues similar to humans, while struggling with humor, irony, and conversational maxims.Source: §1 · p. 2

2 Related work

Prior work has studied language models’ recognition of non-literal meaning, implicatures, and pragmatic interpretation, but this paper distinguishes itself through broader phenomena coverage and detailed human comparisons.

2 Related work

  • Prior evaluations have tested scalar implicature and figurative language, while Zheng et al. (2021) evaluated five Gricean implicature types after task training.This study includes Gricean implicatures among seven phenomena and evaluates pretrained models without fine-tuning on its tasks.Source: §2 · p. 2
  • Ruis et al. (2022) use prompting for pragmatic interpretation, but test implicatures through yes/no consistency rather than selecting among multiple interpretations.Their materials focus on indirect question answering, whereas this study covers broader phenomena and utterance types and supports detailed error and human comparisons.Source: §2 · p. 2
  • Social-commonsense benchmarks interpret actions and events, whereas this study focuses on speaker utterances; theory-of-mind benchmarks primarily test false-belief reasoning.Prior language-model results on theory-of-mind tests vary, with reported successes and failures.Source: §2 · p. 2

3 Evaluation materials

The evaluation uses expert-curated English materials covering seven pragmatic phenomena, with multiple-choice answers representing distinct interpretation strategies. The tasks range from contextual speaker intentions to humor and discourse coherence.

3.1 Overview of stimuli

  • Most items combine a short story, a character’s utterance, and a question about intended meaning; Humor and Coherence are exceptions.The materials are multiple-choice and cover seven phenomena.Source: §3.1 · p. 2
  • Answer options distinguish the target pragmatic interpretation, a plausible literal reading, and incorrect non-literal distractors based on heuristics such as lexical similarity.The target is called “Correct” as a normative evaluation, not a claim that individual human variation is captured.Source: §3.1 · p. 3
  • Each task has 20–40 manually curated items intended to cover varied non-literal phenomena and elicit individual differences among humans.The materials were not specifically designed to require Theory of Mind; the authors’ “Correct” label denotes a normative answer and does not capture variation in human tendencies.Source: §3.1 · p. 3

3.2 Tested phenomena

  • The Deceits task asks why a speaker uses a white lie, while IndirectSpeech asks what an indirect request conveys; both depend on story context for interpreting intent.The text notes that intentions behind polite deceits are rarely explicit, while indirect requests may be associated with descriptions of the requested action being completed.Source: §3.2 · p. 4
  • Irony items ask what a character intends with an ironic statement, while Maxims items ask why a character flouts a conversational maxim.Irony can reverse an utterance’s semantic content; maxim flouting may be recognizable from utterance length, syntax, or contextual mismatch, though intent recovery is uncertain.Source: §3.2 · p. 4
  • Metaphor items ask what a speaker intends, requiring access to world knowledge and abstract relations between domains; Humor items ask for the funniest punchline.Some humor items require complex world knowledge, such as recognizing that dividing a pie into four rather than eight pieces does not change its total amount.Source: §3.2 · pp. 4–5
  • Coherence items test whether sentence pairs form a coherent story, with the task assuming naturalistic training text is primarily coherent.The task concerns discourse relationships across sentences rather than a speaker utterance.Source: §3.2 · p. 5

4 Experiments

The experiments use zero-shot prompts to compare model performance across seven tasks while keeping inputs close to those presented to human participants. Models span several families and are compared with a human baseline.

4.1 Evaluation paradigm

  • Zero-shot prompting evaluates knowledge emerging through training rather than in-context task adaptation, while enabling a common approach across tasks and inputs close to the human stimuli.The prompt uses task instructions and a query; models are scored by whether the correct answer token receives the highest probability, with answer order randomized across five item versions.Source: §4.1 · p. 5

4.2 Models

  • Seven models across four families include GPT-2, instruction-finetuned Tk-Instruct and Flan-T5 variants, and two InstructGPT-based models.Tk-Instruct variants have 3B and 11B parameters; Flan-T5 variants have 250M and 3B parameters; text-ada-001 has 350M parameters, while text-davinci-002’s size is unknown.Source: §4.2 · pp. 5–6
  • Model accuracy is shown for each task, with 95% confidence intervals and task-specific random baselines; the human comparison uses data from 374 participants.The models’ accuracy is also examined against parameter count, excluding text-davinci-002 because its parameter count is unknown.Source: Figure 1 · p. 6, Figure 2 · p. 6, §4.2 · p. 6

5 Results

The largest models generally select pragmatic interpretations above chance, and their response patterns resemble humans, though they struggle with tasks involving social expectation violations. Context removal, scrambling, and item-level comparisons provide evidence about which cues models and humans use.

5.1 Do models choose the target pragmatic interpretation?

  • The largest models perform above chance on all tasks except Tk-Instruct (11B) on Maxims, sometimes near human-level; Humor, Irony, and Maxims are hardest.The authors suggest these tasks may challenge models because they involve violations of listeners’ social expectations.Source: §5.1 · p. 6
  • Accuracy jumps after 1B parameters, but size does not explain all variation: Flan-T5 (XL) outperforms equally sized Tk-Instruct (3B).Models below 1B achieve similar accuracy.Source: §5.1 · p. 6

5.2 Do models and humans make similar types of errors?

  • Across tasks, humans mainly choose Correct, while text-davinci-002 shows a similar pattern but selects Literal more often; larger models generally favor Correct and Literal over distractors.Next-largest models prefer Literal, while GPT-2 and InstructGPT-3 (ada) perform at chance.Source: Figure 3 · p. 7, §5.2 · p. 7
  • In Metaphor, text-davinci-002 and Flan-T5 (XL) favor a figurative distractor over Literal, while text-davinci-002 selects the funny distractor in Humor more than other distractors.For Humor, it selects the human-preferred punchline only 55% of the time.Source: §5.2 · p. 7

5.3.1 The role of context

  • Removing story context most degrades Irony, followed by Deceits and Maxims, for both humans and models; IndirectSpeech and Metaphor show smaller declines.Humor and Coherence were excluded because the context manipulation was incompatible with those tasks.Source: Figure 4 · p. 8, §5.3.1 · p. 8, §5.3.2 · p. 8

5.3.2 Scrambling

  • Models maintain reasonable performance on most tasks after sentence- or word-order scrambling, with Metaphor the notable exception.The authors expect word-level scrambling may disrupt human performance, but leave human performance on these items for future work.Source: §5.3.2 · p. 8

5.3.3 Item-level alignment

  • Larger models align better with human item-level accuracy, with strongest correlations for IndirectSpeech, Irony, Maxims, and Metaphor, suggesting similar cue sensitivity on those tasks.The comparison correlates by-item human accuracy with each model’s probability assigned to the correct answer.Source: §5.3.3 · pp. 8–9

6 Discussion

The discussion reports that large models match human response patterns on pragmatic tasks while considering how such behavior might arise without explicit mental-state representations. The experiments do not establish which mechanisms support that behavior.

6 Discussion

  • Flan-T5 (XL) and text-davinci-002 achieve high accuracy and tend to favor literal interpretations over heuristic-based distractors when incorrect, matching human error patterns.Model and human accuracy also correlate at the item level for several tasks and degrade similarly when context is removed.Source: §6 · p. 9
  • The experiments do not disentangle how different training procedures contribute to pragmatic behavior, so the authors avoid strong claims about how models learn pragmatics.The tested models reflect different learning processes, including linguistic prediction, instruction fine-tuning, and human feedback.Source: §6 · p. 9
  • The tested models lack explicitly constructed mental-state representations, leaving open whether their pragmatic behavior relies on low-level linguistic cues or an emergent heuristic form of Theory of Mind.The discussion presents these as two possibilities, not conclusions established by the experiments.Source: §6 · p. 9
  • Because the stimuli were not designed specifically to require Theory of Mind, the experiments cannot distinguish the proposed hypotheses; targeted tests and broader model comparisons are future directions.The authors suggest evaluating a wider range of training objectives or architectures and manipulating Theory of Mind alongside language comprehension.Source: §6 · p. 9

Limitations

The authors identify limits in item-set size, language and cultural scope, model scale, and the ability to predict behavior on arbitrary inputs. These boundaries constrain how broadly the results should be applied.

Limitations

  • The manually crafted evaluation has relatively few items, which can limit statistical power and introduce lexical-item bias, although the authors consider these concerns manageable.They cite expert validation, direct human comparisons, and sufficient signal to distinguish tested models as reasons for that judgment.Source: Limitations · p. 10
  • The evaluation uses English materials, and some tasks reflect Western communication norms, limiting its coverage of pragmatic variation across languages and cultures.The authors identify evaluation beyond English as an important direction for future work.Source: Limitations · p. 10
  • Except for OpenAI API models, the study tests only models with at most 11B parameters because of computational limits; intermediate-scale models could behave differently.The authors specifically note that models between 11B parameters and text-davinci-002's size might show qualitatively different behavior.Source: Limitations · p. 10
  • The authors caution against expecting pragmatic behavior on arbitrary downstream inputs, and note that text-davinci-002's training protocol is largely unknown.They state that model responses to arbitrary inputs are impossible to predict; OpenAI API results may also be difficult to reproduce.Source: Limitations · p. 10

Ethics statement

The ethics statement discusses potential harms from biased or misleading LLM outputs. Appendix A supplies example prompts for seven tasks, illustrating their distinct response formats and pragmatic targets.

Ethics statement

  • LLM applications could reinforce discrimination and socioeconomic inequities, including through hiring decisions or legal rulings.The statement connects these risks to stereotypes in language models trained on large internet-text collections.Source: Ethics statement · p. 10

A.1 Deceits

  • The deceit prompt asks why Henry says he was reading after switching off the television and picking up a textbook.Answer choices include avoiding trouble for not studying and avoiding offense to his mother.Source: §A.1 · p. 15

A.2 IndirectSpeech

  • The indirect-speech example asks what Nate's wife conveys by asking whether he is going out while pointing to a full garbage bag.One option is that she wants him to take the garbage out.Source: §A.2 · p. 15

A.3 Irony

  • The irony example asks what a father means by calling a car ride pleasant while his children argue.The options include that he does not want to listen to their arguments.Source: §A.3 · p. 15

A.4 Maxims

  • The maxims example asks why Jane changes the subject to coffee after Leslie asks about a man she saw Jane with.An option interprets the reply as avoiding the topic Leslie raised.Source: §A.4 · p. 15

A.5 Metaphor

  • The metaphor example asks what Bob means by saying investors are squirrels collecting nuts.One interpretation is that investors buy stocks hoping for future profit.Source: §A.5 · p. 15

A.6 Humor

  • The humor task presents a joke with five possible punch lines and asks readers to choose the ending that makes it funny.The pastry-shop example turns on Martha asking for eight pieces because five people will share dessert.Source: §A.6 · p. 16

A.7 Coherence

  • The coherence task asks whether two sentences form a coherent story, using binary coherent or incoherent choices.Its example pairs brushing against a table with deciding to study harder.Source: §A.7 · p. 16

B Timestamps of OpenAI model queries

Table 3 records timestamps for requests sent to the OpenAI API.

B Timestamps of OpenAI model queries

C.1 Details of human experiments

The no-context human experiment recruited 30 participants for 105 trials, and the reported accuracy comparison covers humans and three top-performing models on original and no-context items.

C.1 Details of human experiments

  • The study had institutional review board approval at the authors’ home institution.The protocol number was 2010000243.Source: §C.1 · p. 16
  • 30 participants recruited from English-speaking countries completed 105 no-context trials after passing an English proficiency task.Participants were recruited through Mechanical Turk and pre-screened with sentence completions for basic coherence and grammaticality.Source: §C.1 · p. 16
  • Trials were grouped by phenomenon, with blocks and items presented in random order.Source: §C.1 · p. 16

C.2 Raw accuracy scores

  • Figure 6 reports accuracy for humans and the three best-performing models on original and no-context test items.Source: §C.2 · p. 16

D Sentence- and word-level scrambling

The scrambling analysis compares model accuracy across original, sentence-level, and word-level conditions. It also contrasts accuracy on original and no-context items for humans and the three best-performing models.

D Sentence- and word-level scrambling

  • Figure 7 presents the three best-performing models’ accuracy on each task across original, sentence-level, and word-level scrambling conditions.Its error bars denote 95% confidence intervals, and the dashed line marks the random baseline.Source: D Sentence- and word-level scrambling · p. 16, Figure 7 · p. 18
  • Figure 6 compares the proportion of correct answers from humans and models on original and no-context items.Shaded bars indicate original items and empty bars indicate no-context items.Source: Figure 6 · p. 17

D.1 Sentence-level scrambled prompt

  • The sentence-level prompt retains the short story and asks why the character responds, with four answer options.The example presents Dan’s statement about the dog knocking over the vase alongside the intact scenario.Source: §D.1 · p. 17

D.2 Word-level scrambled prompt

  • The word-level condition scrambles the scenario’s word order while retaining the task instructions and question.The example’s answer options include three interpretations of Dan’s response.Source: §D.2 · p. 17

Related papers

Loading 2212.06801v2…