PaperlayerView exact revision on arXiv

Source-linked AI summary

Pragmatics in the Era of Large Language Models: A Survey on Datasets, Evaluation, Opportunities and Challenges

Bolei Ma, Yuting Li, Wei Zhou, Ziwei Gong, Yang Janet Liu, Katja Jasinskaja, Annemarie Friedrich, Julia Hirschberg, Frauke Kreuter, Barbara Plank

arXiv:2502.12378v3cs.CL

TL;DR

Pragmatic competence in NLP models remains limited, and LLMs are not yet sufficiently evaluated for contextually appropriate communication. This survey reviews pragmatic-evaluation resources and finds a need for more nuanced, contextually rich benchmarks. Its scope is predominantly NLP literature, leaving some linguistic resources and methods potentially uncovered.

  • Problem

    LLM pragmatic competence remains insufficiently evaluated, and current practices still lack a comprehensive, fine-grained framework for pragmatic evaluation.

    Source: §1 · p. 1, §6.2 · p. 8

  • Method

    The survey categorizes pragmatic-evaluation datasets by phenomena and analyzes their task designs, data collection methods, and evaluation approaches.

    Source: §1 · p. 2, §7 · p. 9

  • Results

    The survey identifies trends, challenges, and gaps and underscores the need for more nuanced, contextually rich evaluation benchmarks.

    Source: §7 · p. 9

  • Takeaways & Limitations

    The survey aims to guide researchers and practitioners toward pragmatic reasoning research and more human-like communication in NLP.

    Source: §7 · p. 9

    The survey focuses mainly on NLP literature and text-based resources, so it may omit linguistic resources and evaluation methods.

    Source: Limitations · p. 10

Abstract

from arXiv · show

Understanding pragmatics-the use of language in context-is crucial for developing NLP systems capable of interpreting nuanced language use. Despite recent advances in language technologies, including large language models, evaluating their ability to handle pragmatic phenomena such as implicatures and references remains challenging. To advance pragmatic abilities in models, it is essential to understand current evaluation trends and identify existing limitations. In this survey, we provide a comprehensive review of resources designed for evaluating pragmatic capabilities in NLP, categorizing datasets by the pragmatic phenomena they address. We analyze task designs, data collection methods, evaluation approaches, and their relevance to real-world applications. By examining these resources in the context of modern language models, we highlight emerging trends, challenges, and gaps in existing benchmarks. Our survey aims to clarify the landscape of pragmatic evaluation and guide the development of more comprehensive and targeted benchmarks, ultimately contributing to more nuanced and context-aware NLP models.

1 Introduction

Pragmatics concerns how context shapes meaning beyond the literal words, an ability that remains limited in NLP models and insufficiently evaluated in LLMs. The survey reviews pragmatic-evaluation resources and their tasks, data collection, and evaluation approaches.

1 Introduction

  • Pragmatics examines how context shapes meaning and how speakers convey implied meanings, emotions, and intentions.Foundational accounts of implicature, speech acts, and contextual inference inform computational approaches.Source: §1 · p. 1
  • Traditional and transformer-based NLP models remain limited in understanding pragmatic meaning beyond literal word definitions.The paper traces approaches from rule-based systems through statistical models to deep learning.Source: §1 · p. 1
  • LLM pragmatic competence is insufficiently evaluated despite the importance of contextually appropriate responses for trustworthy human-computer interaction.The need is especially salient for real-world use cases requiring sophisticated pragmatic reasoning.Source: §1 · p. 1
  • The survey reviews pragmatic-evaluation resources, categorizes datasets by phenomena, and analyzes task design, data collection, evaluation, gaps, and recommendations.It covers core pragmatic concepts, evaluations across NLP tasks, dataset construction, and evaluation metrics and techniques.Source: §1 · p. 2

2 Core Concepts in Pragmatics

The section introduces pragmatics as the study of context-sensitive interpretation, then surveys linguistic phenomena and how NLP evaluations address them. The reviewed phenomena span contextual reference and inference, communicative action, discourse, and social interaction.

2 Core Concepts in Pragmatics

  • Pragmatics concerns interrelated phenomena explaining how context influences language interpretation.The section introduces core concepts before summarizing phenomena examined in surveyed work.Source: §2 · p. 2

2.1 Pragmatic Phenomena in Linguistics

  • Linguistic context includes surrounding language and temporal, spatial, and social factors; deixis relies on situational details such as speaker location and utterance time.Deictic forms include personal, spatial, and temporal expressions.Source: §2.1 · p. 2
  • Implicatures add unstated meaning through contextual assumptions and conversational maxims, while presuppositions are implicit assumptions required for an utterance to be understood.The examples distinguish implied content from assumptions treated as already established.Source: §2.1 · p. 2
  • Speech acts both convey information and perform actions, while discourse coherence concerns how sentences and larger text units connect.Social pragmatics also considers how culture, power, gender, and interpersonal dynamics shape language use.Source: §2.1 · p. 2

2.2 Pragmatic Phenomena in the Survey

  • The survey groups overlapping pragmatic phenomena by the key aspects addressed in each paper and maps phenomena to surveyed works.Its coverage includes context and deixis, where tasks test interpretation and reference resolution in situational or linguistic context.Source: §2.2 · p. 3
  • Implicature evaluations test inference beyond literal meaning, including conversational, scalar, and manner implicatures and work on indirectness and presupposition.The surveyed applications also include social-media implicatures and pragmatic inference for bias detection.Source: §2.2 · p. 3
  • Speech-act studies test intent recognition in requests, commands, promises, debates, and collaborative or multiparty dialogue settings.Some studies examine how speakers generate messages and listeners infer intentions in collaborative tasks.Source: §2.2 · pp. 3–4
  • Surveyed work also covers discourse relations and coherence, alongside social norms, biases, power dynamics, and shared cultural knowledge.The review connects these areas with discourse markers, dialogue structure, and socially shaped language use.Source: §2.2 · p. 4

3 Task Types

Pragmatic evaluation uses a broad range of task formats, from structured selection and inference to dialogue, sentiment analysis, and visually grounded communication. The task choice determines which communicative function or reasoning process is assessed.

Task overview

  • The surveyed tasks range from multiple-choice and open-ended question answering to dialogue modeling and multimodal challenges.The review organizes them by general NLP task classifications.Source: §3 · p. 4

Question-based tasks

  • Multiple-choice questions test selection of an utterance interpretation, while question-answering tasks assess reasoning about unspoken intent, including in multi-turn dialogue.Some QA datasets include literal answers to probe reasoning.Source: §3 · p. 4

Natural Language Inference

  • Pragmatic NLI tasks use premise-hypothesis relations to test whether models select pragmatically related or unrelated sentences.Evaluations vary particles or words, or draw sentence pairs from dialogue.Source: §3 · pp. 4–5

Sentiment and additional tasks

  • Pragmatic sentiment analysis adds communicative intent, discourse roles, and social dynamics to polarity, with tasks on emotion, sarcasm, hatefulness, and autism-related medical applications.Other specialized tasks include speech-based evaluation, bias detection, and cloze tests for discourse markers.Source: §3 · p. 5

Image Captioning

  • Image-captioning evaluations use visual content to study reasoning about alternatives and context, including contrastive captions and speaker-listener dynamics.Some datasets use negation-based descriptions to examine pragmatic reasoning.Source: §3 · p. 5

Reference Games

  • Reference games ask a speaker to describe a target among shared candidates so a listener can identify it, testing context-dependent interpretation and collaboration.Variants include sequential language use and visual games focused on referential terms.Source: §3 · p. 5

4 Dataset Construction

The survey distinguishes bottom-up dataset construction, which collects data before annotation, from top-down construction, which starts with theory-motivated labels and expands seed examples. It reviews source types, annotation practices, and generated-data approaches.

4 Dataset Construction

  • Dataset construction is divided into bottom-up collection followed by annotation and top-down expansion from theory-motivated seed labels.The distinction helps characterize dataset scope, source data, limitations, and special features.Source: §4 · pp. 5–6

4.1 Bottom-up Approach

  • Bottom-up datasets draw on databases, data collected directly from people, or existing datasets, with existing datasets forming the majority.Sources include interviews and media, open-domain or specialized domains, human game data, and datasets extended or annotated for pragmatic phenomena.Source: §4.1 · p. 6
  • Bottom-up annotation commonly uses crowd-sourced or expert labels, and some workflows combine LLM assistance with human review.In one QA dataset effort, direct LLM generation of implicature inferences was found unreliable; another used human raters to review generated stimuli.Source: §4.1 · p. 6

4.2 Top-down Approach

  • Top-down construction uses linguistic-theory-motivated labels that can be applied automatically, including scalar pairs expanded into generated sentences or premise-hypothesis pairs.Manual edits can ensure sentence independence and grammatical correctness.Source: §4.2 · p. 6

5 Evaluation

Pragmatic evaluation commonly uses automatic metrics, but label-based and text-generation measures can miss the broader, nuanced nature of pragmatic phenomena. The section highlights human assessment and recommends combining evaluation methods with holistic tests across multiple phenomena.

Evaluation on Model Performance

  • Automatic metrics dominate, but label-based classification and generation scores often miss broader contextual and interconnected pragmatic abilities.Binary labels may capture only a narrow aspect, while metrics such as ROUGE, BLEU, and BERTScore can fall short across tasks.Source: §5 · p. 7

Beyond Automatic Evaluation

  • Human annotators assess pragmatic qualities such as politeness, uncertainty, and empathy that automatic metrics do not easily measure.Examples include 1-to-3 ratings of politeness and uncertainty and crowd-sourced judgments of empathy.Source: §5 · p. 7
  • A hybrid framework evaluates both general text generation and nuanced aspects of human communication, providing a more complete performance picture.Source: §5 · p. 7
  • The survey identifies integrated automatic and manual evaluation, alongside holistic tests of multiple phenomena, as promising directions.Suggested combinations include emotion detection, politeness, and social reasoning in complex real-world scenarios.Source: §5 · p. 7

6 Opportunities and Challenges

The survey identifies gaps in pragmatic evaluation and outlines ways to make benchmarks more generalizable and fine-grained. It also considers how pragmatics may inform LLM alignment and support linguistic research.

6.1 Gaps and Chances in Generalizability

  • Only 11 of 57 surveyed resources include non-English languages, while datasets also remain mostly unimodal and focus on isolated phenomena.The survey argues these gaps limit cross-linguistic and cross-cultural generalizability and make holistic pragmatic competence difficult to assess.Source: §6.1 · pp. 7–8
  • Human-label subjectivity and variation between annotators make demographic factors important to consider in data collection and LLM evaluation.The survey identifies this consideration as a gap in current work.Source: §6.1 · pp. 7–8
  • The survey recommends multilingual, multimodal datasets and tasks that test interactions among pragmatic phenomena in holistic, real-world settings.It also suggests combining synthetic LLM-generated data with human validation.Source: §6.1 · p. 8

6.2 Fine-Grained Evaluation of Pragmatics

  • Despite many benchmarks for specific pragmatic abilities, fine-grained evaluation needs a comprehensive framework and dynamic, context-sensitive measures, including human-in-the-loop assessment.The survey notes that psycholinguistic and psychometric metrics may help assess LLM responses.Source: §6.2 · p. 8
  • Most studies classify or generate outputs without examining how models reach responses; qualitative analysis and human feedback could probe failures in subtle social inference.The survey also proposes LLM-generated controlled datasets with human validation, while noting that rich pragmatic data are difficult to construct at scale.Source: §6.2 · p. 8

6.3 Pragmatics Aids LLM Alignment

  • Pragmatic constraints and examples may improve instruction-following, while current systems remain weak at indirect requests, social-meaning shifts, ambiguity, and implicit meaning.The survey also describes degraded communication in multi-agent interactions and relates alignment to contextual appropriateness and social coherence.Source: §6.3 · p. 9

6.4 LLMs Enhance Research in Pragmatics

  • LLMs may generate context-rich experimental materials and preliminary annotation labels, but reliable use calls for human review, collaboration, and rigorous evaluation.The survey recommends collaboration among computational linguists, cognitive scientists, and NLP practitioners, alongside qualitative studies.Source: §6.4 · p. 9

7 Conclusion

The survey maps resources for evaluating pragmatic understanding in NLP and identifies the need for more nuanced, contextually rich benchmarks as LLMs evolve.

7 Conclusion

  • The survey categorizes datasets by pragmatic phenomena and analyzes their task designs, data collection methods, and evaluation with models.It highlights trends and challenges in pragmatic evaluation.Source: §7 · p. 9
  • The authors identify a need for more nuanced and contextually rich evaluation benchmarks, especially as LLMs continue to evolve.Source: §7 · p. 9
  • The survey aims to guide researchers and practitioners advancing pragmatic reasoning in NLP and promote more human-like communication and interdisciplinary collaboration.Source: §7 · p. 9

Limitations

The survey's scope leaves out some potentially informative resources and perspectives: it centers NLP tasks, English-language literature, and predominantly textual data.

Survey Sources

  • The survey predominantly covers NLP literature, not resources from linguistic studies, which may omit resources or evaluation methods relevant to data collection and evaluation.The authors suggest future surveys review resources for pragmatic language understanding in linguistics.Source: Limitations · p. 10

NLP Tasks

  • The survey excludes intrinsic pragmatic tasks such as discourse modeling, though these may offer useful evaluation insights.Source: Limitations · p. 10

Multilinguality

  • The review is limited to English-language publications, so pragmatic datasets created in other languages may be missing.The authors note that this may limit inclusion of phenomena and methodologies unique to particular languages.Source: Limitations · p. 10

Multimodality

  • Most covered data is textual, and some resources may be missed because the survey focuses mainly on NLP venues.The authors encourage future work on pragmatic evaluation across diverse modalities.Source: Limitations · p. 10

Ethical Considerations

The paper frames LLM pragmatics through observable outputs, not assumptions about human-like cognition, beliefs, or intentions.

Ethical Considerations

  • Discussing pragmatic abilities in LLMs can raise anthropomorphism concerns, so the paper clarifies that it does not imply cognition or consciousness.Source: Ethical Considerations · p. 10
  • The authors advocate using human pragmatic features to build benchmarks intended to improve benchmark design and support user-friendly, effective human-machine interactions.Source: Ethical Considerations · p. 10
  • The evaluation examines pragmatic aspects in LLM textual outputs rather than attributing beliefs or intentions to the models.The authors say this perspective avoids over-personalization.Source: Ethical Considerations · p. 10

A Overview of Surveyed Works

The survey searched for pragmatic evaluation work in NLP, applied eligibility criteria, and added relevant studies identified through manual inspection.

A Overview of Surveyed Works

  • The survey searched ACL Anthology papers published primarily by 31.12.2024 using the keywords “pragmatics” and “datasets.”Source: A Overview of Surveyed Works · p. 17
  • ACL Anthology was the primary source because the authors considered it the main publication platform for the *CL community and a relevant source for pragmatic NLP evaluation work.Source: A Overview of Surveyed Works · p. 17
  • Included work introduced datasets for pragmatic evaluation, discussed language-model assessment methods, or empirically evaluated NLP models on related tasks.Source: A Overview of Surveyed Works · p. 17
  • The authors manually inspected literature to include relevant work missed by keyword variation and added some recent papers from outside the ACL Anthology, producing a final set of 58 papers.Source: A Overview of Surveyed Works · p. 17

B Pragmatic Phenomena and their Corresponding Tasks

The section maps pragmatic phenomena to NLP task types and finds no one-to-one correspondence between them. This flexibility supports diverse evaluation formats and motivates finer-grained tasks.

B Pragmatic Phenomena and their Corresponding Tasks

Related papers

Loading 2502.12378v3…