Are Natural Language Inference Models IMPPRESsive? Learning IMPlicature and PRESupposition

1 Introduction

• Pragmatic inferences differ from classical entailments: implicatures can be denied, while presuppositions persist under entailment-canceling operators.Negation and questions are examples of such operators.Source: §1 · p. 1
• IMPPRES is a 25.5K-pair, linguist-template-generated diagnostic dataset targeting implicatures and presuppositions with lexical variety and controls.The introduction frames it as a test of specific pragmatic inference types.Source: §1 · pp. 1–2
• BERT shows evidence of reasoning about some implicatures and presuppositions despite few relevant MultiNLI examples, but still mishandles some lexical-trigger meanings.The paper notes BERT treats some word pairs, including or and and, as synonyms.Source: §1 · p. 2

2 Background: Pragmatic Inference

• Pragmatic inference relates sentences through utterance context and interlocutors’ conversational goals, unlike entailment between isolated sentence meanings.The paper treats implicatures and presuppositions as pragmatic inference types.Source: §2 · p. 2

2.1 Implicature

• Scalar implicatures arise from lexical scales such as some/all, suggesting that a stronger alternative was false; unlike entailments, they can be explicitly denied.The example “Jo ate some of the cake” suggests she did not eat all of it, but that suggestion can be denied.Source: §2.1 · p. 2

2.2 Presupposition

• Presuppositions are taken-for-granted facts associated with triggers and generally project through negation, questions, modals, and conditionals.Recognizing a premise presupposition as entailed requires accommodation: inferring the presupposed fact as new information.Source: §2.2 · pp. 2–3

3 Related Work

• Existing NLI datasets cover many tasks and genres, while commonsense reasoning has also been used to frame NLI inference.The cited landscape includes training, evaluation, and failure-analysis datasets.Source: §3 · p. 3
• IMPPRES explicitly probes whether NLI models trained for commonsense reasoning treat implicatures and presuppositions as entailments without targeted training.The paper presents it as the first NLI dataset for this question.Source: §3 · p. 3
• Prior pragmatic resources focus on narrower trigger classes and gradient judgments, which are not straightforward to use for categorical NLI evaluation.The paper leaves direct comparisons between these approaches for future work.Source: §3 · pp. 3–4

4 Annotating MultiNLI for Pragmatics

• Three expert annotators assessed 200 randomly selected MultiNLI matched development pairs for inference relation, subtype, and trigger embedding under entailment-canceling operators.The annotators had 17 combined years of formal semantics and pragmatics training.Source: §4 · p. 4
• Only 8 presupposition pairs and 3 implicature pairs had agreement from at least two annotators; just one matched an IMPPRES inference type.That example involved the presupposition of possessed definites.Source: §4 · p. 4
• MultiNLI offers at best loose pragmatic evidence from world knowledge and discourse, with almost no explicit evidence for lexically triggered inferences tested here.Most agreed examples were conversational implicatures or existence presuppositions, generally not linked to particular lexical triggers.Source: §4 · p. 4

5 Methods

• IMPPRES generates sentence pairs from templates and a vocabulary of over 3,000 annotated lexical items to control grammatical, syntactic, and semantic well-formedness.Generated sentences share a template’s structure while varying in sampled constituents.Source: §5 · pp. 4–5
• Generated sentences can describe unlikely scenarios or low-frequency word combinations, so IMPPRES is intended for evaluation rather than model training.The authors note that simple regularities in generated data may be exploited by supervised classifiers.Source: §5 · p. 5
• The study evaluates MultiNLI-trained BOW, InferSent, and BERT-Large models, using GloVe representations for BOW and InferSent.BOW sums premise and hypothesis word embeddings; InferSent encodes each sentence with a bidirectional LSTM.Source: §5 · p. 5
• BERT-Large reached 86.6% MultiNLI development accuracy, compared with 49.6% for BOW and 67.6% for InferSent.The authors report these as development-set accuracies.Source: §5 · p. 5

6.1 Scalar Implicature Datasets

• The scalar-implicature datasets cover six lexical-scale types, with target pairs for implicature and negated implicature and contradiction controls.Each type uses 100 paradigms of 12 unique sentence pairs; labels represent either logical or pragmatic interpretations.Source: §6.1 · pp. 5–6

6.2 Implicatures Results & Discussion

• BERT performed at ceiling on implicature controls, while BOW and InferSent showed highly variable performance.Control success was necessary for interpreting target-condition results; only BERT passed all controls.Source: §6.2 · p. 6
• For connectives, scalar adjectives, and verbs, BERT’s results fit neither predicted behavior; modal results were split across the three labels.The authors conclude that BERT lacks sufficient knowledge of these words’ meanings.Source: §6.2 · pp. 6–7
• Numeral results were inconsistent: neither an “exactly” nor an “at least” interpretation held across the board.Source: §6.2 · p. 6
• 55% vs. 36% of determiner results were consistent with pragmatic vs. logical reasoning, respectively, and 9% with neither; pragmatic behavior varied by condition.BERT’s determiner results showed pragmatic reasoning more often overall, with effects of sentence order and negation.Source: §6.2 · p. 7
• Negation generally lowered pragmatic reasoning, and results were more pragmatic when the implicature trigger appeared in the premise.The some–all condition showed more pragmatic behavior than all–some for negated implicatures.Source: §6.2 · p. 7

7.1 Presupposition Datasets

• Each of eight presupposition datasets contains 100 paradigms of 19 sentence pairs, including 15 targets and four controls.Targets test trigger entailment, contradiction or neutrality after altering the presupposition, and behavior under entailment-canceling operators; controls test the operators’ effects on entailment.Source: §7.1 · p. 7

7.2 Presupposition Results & Discussion

• BERT exceeds chance on every control and identifies contradiction under negation 100% of the time, while BOW and InferSent perform at or below chance.On the other controls, BERT identifies neutrality the majority of the time, indicating it mostly learns that negation, modals, questions, and conditionals cancel classical entailments.Source: §7.2 · p. 8, Figure 5 · p. 8
• BERT always accommodates cleft and possessed-definite existence presuppositions and the presupposition of only, but almost never accommodates numeracy presuppositions.BOW and InferSent accommodate nearly all trigger types above chance, with cleft uniqueness entailment accuracy above 90%, though results are less reliable for most triggers.Source: §7.2 · p. 8, Figure 6 · p. 8

• Projection results are counted only when a model recognizes operator cancellation in the matching control and the unembedded presupposition as an entailment.This filtering guards against apparent projection caused by, for example, systematically ignoring negation.Source: There are exactly two flowers that… · p. 8
• BERT identifies the presupposition as entailed at ≥80% accuracy and BOW at ≥90% after filtering, across entailment-canceling operators.BERT alone reliably identifies contradiction when the presupposition is negated; no model reliably exceeds chance when the presupposition is altered to be neutral.Source: There are exactly two flowers that… · pp. 8–9, Figure 7 · p. 9
• BOW may learn some projection because presupposition behavior is often insensitive to a trigger’s position in the sentence.The authors note edge cases where this heuristic is insufficient, which IMPPRES is not designed to test.Source: There are exactly two flowers that… · p. 9
• Across unembedded triggers, all evaluated models accommodate a wide variety of presuppositions, though BERT rejects numeracy cases.BERT and BOW also appear to learn the projective behavior of some presuppositions.Source: There are exactly two flowers that… · p. 9

• BERT shows strong evidence of scalar implicatures for some and all, while other scales are not diagnosable because models often treat scalar pairs as synonymous.For presuppositions, BERT and, to some extent, BOW perform well on only, cleft and possessive existence, and questions, but models often miss the basic meanings of other triggers.Source: §8 · p. 9
• Because MultiNLI contains few examples of the tested inference types, positive results may draw on pragmatic information in BERT’s pretraining.The authors present this as a possibility: pretraining text may contain recoverable patterns about pragmatic assumptions.Source: §8 · p. 9
• The authors describe IMPPRES as an initial step toward investigating semantic versus pragmatic inference and as a reusable test set for NLI models.They also report releasing IMPPRES data, evaluated models, MultiNLI annotations, and processing scripts.Source: §8 · p. 9

• Table 4 presents example sentence pairs from six scalar-implicature datasets in the “Implicature (+ to −)” condition.Source: Table 4 · p. 14
• Table 5 presents example sentence pairs from eight presupposition datasets in the “Plain Trigger / Presupposition” condition.Source: Table 5 · p. 14
• Figures 8–13 report scalar-implicature results by target condition for adjectives, determiners, modals, numerals, and verbs.Source: Figure 8 · p. 15, Figure 9 · p. 15, Figure 10 · p. 16