A negation detection assessment of GPTs: analysis with the xNot360 dataset

1 Introduction

• State-of-the-art transformer models perform worse on negated instances, especially when negation is crucial for comprehension.Prior work also reports that common NLU corpora contain relatively few negations and that many predictions remain correct when negations are ignored.Source: §1 · p. 1
• RLHF aligns generative-model outputs with user satisfaction, but the authors caution that this may not ensure logical consistency.They note that the approach faces challenges if users are not concerned with logical consistency.Source: Figure 1 · p. 1, §1 · p. 2
• The authors argue that benchmarking GPTs on negation can inform research on logical reliability in high-stakes domains such as law and healthcare.They frame this need against the ambiguity and adaptability of natural language communication.Source: §1 · p. 2
• The study creates xNot360, a 360-sample benchmark designed to assess negation detection across contexts and domains.It includes sentences with negations and corresponding non-negated counterparts, aiming to capture negation's subtleties and complexities.Source: §1 · p. 2
• The authors benchmark GPT-2, GPT-3, GPT-3.5, and GPT-4 on xNot360 using zero-shot prediction.They focus on GPT models because of their documentation, development history, and testing on other benchmarks.Source: §1 · p. 2

• GPT-4 exceeded predecessor models and average law-student MBE performance, but exam success may not establish real-world legal effectiveness.The UBE combines MBE recall and application, MEE essays, and MPT performance tasks, which may not expose complex reasoning limits.Source: §2 · pp. 2–3, Figure 2 · p. 3
• The authors caution that GPT models’ impressive UBE performance should not be taken as assurance of effectiveness in complex, real-world legal practice.Source: §2 · p. 3
• GPT-4’s USMLE performance, including near-perfect calibration [19], does not confirm logical ability; exam focus and accuracy or calibration metrics may not capture reasoning.The cited discussion also flags hallucinations and errors in high-stakes medical applications.Source: Figure 3 · p. 3, §2 · p. 3
• ChatGPT and GPT-4 performed well on familiar reasoning benchmarks but dropped considerably on newly released and out-of-distribution datasets [16].In a limited LogiQA error sample, GPT-4 errors included logical errors, scope errors, and unresolved semantic ambiguity [16].Source: §2 · p. 3
• Because broad reasoning benchmarks do not isolate negation, the authors introduce xNot360 for evaluating GPT models’ negation detection [16].They call for robust logic-specific evaluation methods and metrics to inform targeted improvements and more reliable high-stakes applications.Source: §2 · pp. 3–4

3.1 xNot360 Dataset

• Conditional sentences can be mistaken for negations; the dataset uses classical logic and a truth-table example to distinguish their logical forms.The example contrasts A → B (¬A ∨ B) with ¬A → ¬B (A ∨ ¬B).Source: §3.1 · p. 4, Table 1 · p. 4
• xNot360 contains 360 samples: 180 positive and 180 negative sentence pairs, with sentence lengths from 5 to 20 words.Sentence lengths approximate a Gaussian distribution, providing varied sample complexity.Source: §3.1 · p. 4

3.2 Experimental Settings

• GPT-2, GPT-3, GPT-3.5, and GPT-4 receive sentence pairs in a zero-shot prompt asking whether sentence two negates sentence one.Source: §3.2 · p. 4
• GPT-2 uses HuggingFace’s zero-shot-classification pipeline, while GPT-3, GPT-3.5, and GPT-4 are accessed through OpenAI API endpoints.Evaluation uses accuracy, precision, recall, F1-score, confusion matrices, and analysis of sentence-pair errors.Source: §3.2 · p. 4

4.1 Model Performance

• GPT-4 led xNot360 with 0.7833 accuracy and 0.7706 F1-score, while GPT-3.5 recorded 0.4306 accuracy and 0.2705 F1-score.The performance chart also highlights variation across accuracy, F1-score, precision, and recall.Source: Table 2 · p. 4, §4.1 · p. 4
• The performance chart shows GPT-4 leading major metrics, GPT-3.5 dipping, and GPT-2 pairing the lowest accuracy with the highest recall.GPT-3 is described as having moderate performance.Source: §4.1 · pp. 4, 6, Figure 5 · p. 5

4.2.1 GPT-2

• GPT-2 predicted every label as 1, while GPT-3 still favored label 1 and GPT-3.5 favored label 0 and scored below a random guess.GPT-4's predictions were more balanced than its predecessors, and it achieved 0.7833 accuracy and 0.7706 F1-score.Source: §4.2.1 · p. 6
• The authors call for continued investigation and refinement of negation detection, especially in healthcare, science, and law, where logical reliability is critical.Source: §4.2.1 · p. 6

4.3 Error Patterns

• Most shared errors relied on a negation cue in one part of a conditional statement to judge the whole sentence; some pairs also challenged human judgment.The example “Not saving money consistently will help you build wealth” can mislead readers who treat “Not+sentence” as a reliable negation template.Source: §4.3 · p. 6
• The authors argue that logical rules help ensure xNot360 tests models’ ability to understand and handle negation, while noting that translating natural language into logic is not always straightforward.They point to practical improvements from applying negation rules in legal reasoning.Source: §4.3 · p. 6

5.1 Evaluating other generative models

• The authors plan to evaluate generative models beyond GPT using a similar approach to broaden understanding of their negation capabilities and limitations.They anticipate generative models similar to GPT will become increasingly prevalent.Source: §5.1 · p. 6

5.2 Expanding the dataset with more negation types

• The planned dataset expansion adds affixal, standard, imperative, quantifier, and negative-indefinite constructions for a broader evaluation of negation handling.Examples include “incurable” for affixal negation and “Few judges ruled in favor of the defendant” for a negative quantifier.Source: §5.2 · pp. 6–7

6 Discussions and Conclusions

• The study evaluated GPT-2, GPT-3, GPT-3.5, and GPT-4 with zero-shot prediction on the custom xNot360 dataset.The dataset was designed to assess negation handling across contexts and sentence structures.Source: §6 · p. 7
• GPT-4 showed the highest negation-detection proficiency, GPT-3.5 a marked dip, and overall competence remained modest.The authors suggest negation detection pushes the models’ natural-language understanding capabilities.Source: §6 · p. 7
• The findings highlight logical reliability as significant in high-stakes domains including healthcare, science, and law.The authors frame their work as highlighting limitations of current state-of-the-art pre-trained language models in handling negation.Source: §6 · p. 7
• Future work could refine evaluation methods or create benchmarks for logical reasoning, while exploring alignment and training methods prioritizing logical consistency.The authors specify maintaining other aspects of model performance as a condition for such training methods.Source: §6 · pp. 7–8
• The authors offer xNot360 as a resource to the research community to support further advances in negation understanding.Source: §6 · p. 8