PaperlayerView exact revision on arXiv

Source-linked AI summary

Thunder-NUBench: A Benchmark for LLMs' Sentence-Level Negation Understanding

Yeonkyoung So, Gyuseong Lee, Sungmok Jung, Joonhak Lee, JiA Kang, Sangho Kim, Jaejin Lee

arXiv:2506.14397v4cs.CL

TL;DR

Negation remains challenging for LLMs, while existing benchmarks often treat it as part of broader tasks. Thunder-NUBench evaluates sentence-level understanding by contrasting standard negation with local negation, contradiction, and paraphrase; experiments find models still struggle to distinguish these variants despite supervised fine-tuning and in-context learning.

  • Problem

    Existing benchmarks often treat negation as a minor aspect of broader tasks, and dedicated resources for evaluating generative models’ negation understanding are lacking.

    Source: §1 · p. 1, §2 · p. 3

  • Method

    Thunder-NUBench evaluates sentence-level negation with manually created sentence-negation pairs and a multiple-choice task contrasting standard negation with local negation, contradiction, and paraphrase.

    Source: §1 · p. 2

  • Results

    Supervised fine-tuning and in-context learning can reduce specific errors, but models still struggle to distinguish standard negation from closely related semantic variants.

    Source: §6 · p. 9

  • Takeaways & Limitations

    Thunder-NUBench provides a diagnostic tool for analyzing LLMs’ negation-understanding limitations and a benchmark for future research.

    Source: §6 · p. 9

    Thunder-NUBench is constructed exclusively in English, so its findings may not generalize to multilingual or cross-lingual settings.

    Source: Limitations · p. 10

Abstract

from arXiv · show

Negation is a fundamental linguistic phenomenon that poses ongoing challenges for Large Language Models (LLMs), particularly in tasks requiring deep semantic understanding. Current benchmarks often treat negation as a minor detail within broader tasks, such as natural language inference. Consequently, there is a lack of benchmarks specifically designed to evaluate comprehension of negation. In this work, we introduce Thunder-NUBench, a novel benchmark explicitly created to assess sentence-level understanding of negation in LLMs. Thunder-NUBench goes beyond merely identifying surface-level cues by contrasting standard negation with structurally diverse alternatives, such as local negation, contradiction, and paraphrase. This benchmark includes manually curated sentence-negation pairs and a multiple-choice dataset, allowing for a comprehensive evaluation of models' understanding of negation.

1 Introduction

Negation matters for language understanding, yet models struggle with its semantics and dedicated evaluation resources are lacking. Thunder-NUBench addresses this gap with a sentence-level benchmark that distinguishes standard negation from closely related alternatives.

1 Introduction

  • Approximately 25% of sentences in English texts contain some form of negation, which is relevant to sentiment analysis, question answering, knowledge base completion, and NLI (Sarabi and Blanco, 2016; Hossain et al., 2020; Horn and Wansing, 2025; Khandelwal and Sawant, 2020; Hosseini et al., 2021; Singh et al., 2023).Source: §1 · p. 1
  • Existing work often treats negation as a minor part of broader tasks or focuses on narrow syntactic detection, while PLMs and LLMs can rely on superficial cues.The paper identifies a lack of dedicated benchmarks for evaluating negation understanding.Source: §1 · p. 1
  • Thunder-NUBench asks models to select a sentence’s standard negation from four options, using local negation, contradiction, and paraphrase as distractors.The distractors are designed to test understanding of semantic scope and logical oppositions, not just recognition of surface cues.Source: §1 · p. 2
  • The benchmark pairs manually created sentence-negation examples with a multiple-choice evaluation task and systematic evaluations across decoder-based LLMs.The evaluations examine variation across model families, scales, and training methods.Source: §1 · p. 2

2 Related Work

Prior work spans syntactic detection, broader NLU benchmarks, and studies of generative models, but the paper identifies a gap in robust evaluation of sentence-level negation understanding. Thunder-NUBench directly tests whether models distinguish standard negation from related alternatives.

2 Related Work

  • Rule-based and traditional machine-learning approaches were limited across domains, while later neural and transformer methods largely focused on syntactic span detection.The paper describes CNNs, BiLSTMs, and pretrained transformers as improving negation detection while leaving deeper semantic understanding challenging.Source: §2 · p. 2
  • Existing NLU benchmarks may contain few negations or ones immaterial to task performance, allowing high accuracy even when models ignore negation.Source: §2 · p. 2
  • Negation-sensitive NLI benchmarks report performance declines when negation affects outcomes, with findings suggesting reliance on superficial patterns rather than semantics.Source: §2 · p. 2
  • Distributional models can represent negated and affirmative expressions similarly despite opposing meanings, complicating their treatment of negation.The paper gives “not good” and “good” as an example of expressions that often occur in similar contexts.Source: §2 · p. 3
  • Studies of generative language models report positive bias and difficulty producing or interpreting negated statements.Source: §2 · p. 3
  • Thunder-NUBench targets sentence-level negation directly, unlike NLI or QA benchmarks that evaluate it only within broader inference tasks.The paper presents this as a more direct examination of whether models distinguish standard negation from closely related distractors.Source: §2 · p. 3

3 Scope and Categorization of Negation

The section sets semantic boundaries for negation and distinguishes whole-sentence standard negation from local negation and non-negated contradiction. Its typology treats negation by scope, form, and target. (p14, p27, p37, p23, p20)

3.1 Typology of Negation

  • The typology classifies negation by scope, form, and target: clausal or subclausal, morphological or syntactic, and verbal or non-verbal.The paper presents this framework to clarify semantic boundaries and enable more systematic evaluation of language models' handling of negation.Source: §3 · p. 3, §3.1 · p. 4

3.2 Negation and Contradiction

  • Contradiction means two propositions cannot both be true, but it can arise without negation, as with incompatible claims about a person's birthplace.The section examines whether LLMs distinguish negation from non-negated contradictions.Source: §3.2 · p. 4

3.3 Standard Negation

  • Standard negation reverses the truth value of the main predicate, thereby reversing the proposition of the declarative sentence.The paper treats it as a truth-functional operation mapping the main predicate to its complement set.Source: §3.3 · p. 4
  • For complex sentences, standard negation is applied pairwise over logical structure until atomic propositions remain, including conditionals treated as Neg(P) or Q.This recursive account is intended to reverse the truth value of the whole sentence.Source: §3.3 · p. 5

3.4 Local Negation

  • Local negation targets a verb phrase outside the main clause, across relative, participle, adverbial, and compound sentence structures.In compound sentences, negating only some main clauses counts as local negation; sentence-level standard negation requires all main clauses to be negated.Source: §3.4 · p. 5

4 Thunder-NUBench Dataset

Thunder-NUBench is built through preprocessing, dataset generation, and review. It combines sentence-negation pairs with a multiple-choice evaluation set that contrasts standard negation with local negation, contradiction, and paraphrase.

4 Thunder-NUBench Dataset

  • The benchmark is built in three stages: preprocessing, generation, and review.The workflow is intended to maintain quality control and consistency through continuously updated generation and review guidelines.Source: §4 · pp. 6–7
  • Preprocessing extracts sentences from HoVer and Wikipedia summaries, then corrects grammar and merges or splits text into well-formed single-sentence units.The corpora were selected for factual content and complex sentence structures suitable for negation examples.Source: §4 · p. 6
  • Generation produces manually crafted sentence-negation pairs and multiple-choice items with standard negation, local negation, contradiction, and paraphrase options.The distractor categories are designed to test semantic understanding rather than reliance on superficial cues.Source: §4 · p. 6
  • Humans create standard and local negation options because LLMs often produce incorrect or subclausal negations; contradiction and paraphrase options begin as prompted generations and are refined in review.A separate author cross-checks each instance, with disagreements addressed in regular meetings.Source: §4 · pp. 6–7
  • The final dataset includes training sentence-negation pairs, a multiple-choice evaluation set, and 50 few-shot demonstrations selected to avoid test overlap and balance local-negation types.The demonstration examples use unique Wikipedia page indices and match the overall dataset's local-negation-type distribution.Source: §4 · p. 7

5 Experiments

Experiments compare decoder LLMs across evaluation formats, prompt styles, and training setups; performance generally improves with model scale and additional examples or training, but gains and errors vary by setting.

5.1 Evaluation Setup

  • Evaluation spans completion-based scoring and shuffled option selection, definition-based and detailed instructions, zero-/few-shot settings, and supervised fine-tuning on benchmark sentence-negation pairs.Few-shot settings use 1, 5, and 10 examples.Source: §5.1 · p. 7

5.2 Overall Performance and Key Findings

  • Performance tends to improve with model size, with larger gains in option-selection than completion-based evaluation.Figure 3 shows zero-shot accuracy for instruction-tuned Qwen3 models under the detailed prompt setting.Source: §5.2 · p. 7, Figure 3 · p. 8
  • Instruction tuning generally improves both evaluation formats and yields larger gains with detailed prompts, though some Qwen3 pretrained models match or exceed tuned versions.Figure 4 presents accuracy distributions for pretrained and instruction-tuned models across evaluation settings and prompt instructions.Source: §5.2 · p. 8, Figure 4 · p. 8
  • More in-context demonstrations generally improve performance, but gains diminish as examples accumulate and additional demonstrations do not always help API models under detailed prompts.The possible explanation offered is that detailed instructions already guide API-based large models strongly, while demonstrations may add noise or encourage unnecessary pattern following.Source: §5.2 · p. 8
  • Supervised fine-tuning generally improves zero-shot performance, with more pronounced gains in completion-based evaluation than option-selection, and does not degrade general language abilities.General-benchmark evaluations are cited as evidence that SFT does not degrade general language abilities.Source: §5.2 · p. 8

5.3 Error Analysis

  • For Llama-3.1-8B-Instruct, completion-based errors generally decline from zero-shot to 5-shot and further after SFT; local-negation choices dominate, with compound sentences most confused but showing the largest relative SFT improvement.Table 6 presents the error distribution and confusion analysis across evaluation settings.Source: §5.3 · p. 9, Table 6 · p. 9
  • In option selection, local negation still dominates errors, paraphrases are more often confused than in completion-based evaluation, and SFT sometimes raises error rates over few-shot baselines.Option selection also has lower confusion rates for adverbial-clause negation, and its improvements do not consistently follow the expected progression across training configurations.Source: §5.3 · p. 9

6 Conclusion

Thunder-NUBench evaluates sentence-level negation beyond surface cues by contrasting standard negation with related semantic variants. The experiments find that in-context learning and supervised fine-tuning can reduce specific errors, but models still struggle to distinguish standard negation from close alternatives.

6 Conclusion

  • Thunder-NUBench assesses sentence-level negation beyond surface cues by distinguishing standard negation, local negation, contradiction, and paraphrase; learning methods help reduce specific errors, but close distinctions remain difficult.The benchmark is presented as a diagnostic tool for model limitations and a benchmark for future research across model families and settings.Source: §6 · p. 9

Limitations

Thunder-NUBench is limited to English and formal encyclopedic sources, focuses on standard negation, and may retain bias despite cross-checking.

Limitations

  • The benchmark is exclusively English, so its findings may not generalize to multilingual or cross-lingual settings.Negation’s syntactic and semantic expressions may vary across languages.Source: Limitations · p. 10
  • HoVer and Wikipedia-summary sources are formal and encyclopedic, potentially underrepresenting real-world sentence structures and informal language.Subjective review decisions may also leave residual bias despite independent cross-checking.Source: Limitations · p. 10
  • The benchmark focuses on standard negation and its distinction from local negation, contradiction, and paraphrase, not double negation or negative polarity items.Those other phenomena are not directly addressed.Source: Limitations · p. 10

Ethical Considerations

This section documents Thunder-NUBench’s negation categories, example-review and dataset-release practices, and related datasets. It also details rules for producing standard negations, including antonym choice, sentence structure, clause- or predicate-level limits, and common errors with corrections.

Ethical Considerations

  • The authors report no crowdsourcing, manual review for quality and safety, and release Thunder-NUBench under CC BY-NC-SA 4.0; related resources include HoVer and the Wikipedia Summary Dataset.HoVer has 18,171 training, 4,000 validation, and 4,000 test examples; the Wikipedia Summary Dataset contains approximately 430,000 articles and only a training set.Source: Ethical Considerations · p. 10, G HoVer Dataset · p. 18, H Wikipedia Summary Dataset · p. 18

A Typology of Contradiction

  • Contradiction extends beyond negation to seven categories, including antonymy, numeric mismatch, structural reversals, lexical incompatibilities, and world-knowledge conflicts.The typology follows De Marneffe et al. (2008).Source: A Typology of Contradiction · p. 15

B Copular Verbs

  • Standard negation targets the entire predicate, including a copula and its complement; prior paraphrasing is allowed only when it preserves meaning, tense, and structure.Complementary-antonym forms such as “is a nonexpert” count as verbal negation when they reverse the main predicate’s truth value.Source: B Copular Verbs · p. 15, J Detailed Principles and Examples of… · p. 20

C Negation of Implications

  • For conditionals, the correct negation is P and Neg(Q); the benchmark tests whether models avoid intuitive but invalid alternatives, including conditionals expressed with other connectives.The authors apply the same conditional principle to expressions such as “when,” “as long as,” and “unless.”Source: C Negation of Implications · pp. 15–16

E Local Negation Constructions Excluded

  • The benchmark excludes infinitive, appositive, and prepositional-phrase negation because these constructions alter phrase-level or peripheral content rather than negate the verb.These exclusions align with the benchmark’s focus on verbal negation.Source: E Local Negation Constructions Excluded · p. 17

F Double Negation

  • Double-negation constructions were excluded as paraphrase candidates because antonyms may not be complementary and litotes can add emphasis, making semantic equivalence ambiguous.Although some double negatives can convey affirmative meanings, the authors keep the dataset focused on standard negation.Source: F Double Negation · pp. 17–18

I Human Review Protocol

  • Independent, layered reviews and consensus adjudication resolve disputed cases, while relatively low post-checking edit rates indicate most instances met the guidelines.The authors report edit-based quality-control statistics rather than label-based interannotator agreement; edits and escalations are resolved by consensus.Source: I Human Review Protocol · pp. 18–19

D Compound Sentences and Coordinating Conjunction

J Detailed Principles and Examples of the Thunder-NUBench

  • Negate each atomic proposition when a coordinated sentence decomposes into separate propositions, but negate the predicate as a whole when it expresses one collective action or state.Collective predicates such as “share clothes” and “be the same” are treated as single units.Source: J Detailed Principles and Examples of… · p. 21
  • Standard negation should use complementary antonyms, not gradable or relational antonyms, because only complementary pairs provide a clear binary reversal of truth value.Complementary examples include alive/dead and true/false; gradable pairs such as hot/cold and relational pairs such as buy/sell do not qualify.Source: J Detailed Principles and Examples of… · pp. 21–22
  • A standard negation preserves the original sentence’s elements, prioritizes logical correctness over naturalness, and removes clauses that contradict the negated main clause.Minimal rephrasing for fluency is allowed, but a clause that directly contradicts the negation must be removed.Source: J Detailed Principles and Examples of… · p. 22
  • When a predicate such as “fuse” denotes combining all listed elements, standard negation preserves “and” rather than changing it to “or.”The passage identifies changing the conjunction as an incorrect negation of a sentence about musical styles and lyrics.Source: J Detailed Principles and Examples of… · p. 22
  • With clauses joined by “while” to express contrast, negation applies only to the main clause; the contrasting clause remains affirmative.The example corrects a negation that incorrectly changes both clauses.Source: J Detailed Principles and Examples of… · p. 22

K.1 Sentence-Negation Pair Dataset

Thunder-NUBench combines sentence-negation pairs with a multiple-choice evaluation set, built from English text and reviewed by the authors. The prompts define standard negation and distinguish it from local negation, contradiction, and paraphrase.

K.1 Sentence-Negation Pair Dataset

  • Sentence-negation pairs begin with randomly sampled HoVer supported facts, which are grammar-corrected and merged or split into single sentences.GPT-4 is used for sentence merging, while GPT-3.5-turbo is used for grammar correction.Source: §K.1 · p. 22

K.2 Multiple Choice Dataset

  • The multiple-choice set draws from Wikipedia summaries, filters out sentences shorter than 30 words, and adds GPT-4o-generated contradictions and paraphrases with human review.Because conditionals are rare in the source, the authors generate them with GPT-4o-mini and manually filter or lightly edit them; 20 of 100 conditionals are manually collected and 80 are converted from sampled summaries.Source: §K.2 · p. 23, §L.2 · p. 24

L Thunder-NUBench Dataset Structure

  • Thunder-NUBench contains a sentence-negation pair subset for supervised fine-tuning and a multiple-choice subset for evaluation, both built on English text and author-reviewed.The pair subset matches affirmative sentences with their logically negated forms; the evaluation subset offers four transformations per original sentence.Source: L Thunder-NUBench Dataset Structure · p. 24, §L.1 · p. 24, §L.2 · p. 24

L.1 Sentence-Negation Pair Dataset

  • The pair subset records each original sentence as a premise and its logically negated form as a hypothesis.Its fields also include an index.Source: §L.1 · p. 24

L.2 Multiple-Choice Dataset

  • Each evaluation item offers standard negation, local negation, a non-negated contradiction, and a meaning-preserving paraphrase.Standard negation is the correct answer; the dataset also records the local-negation type and the affected phrase or clause.Source: §L.2 · p. 24

M Prompt Templates Used in Evaluation

  • Evaluation uses definition and detailed instruction styles, with detailed prompts specifying how to negate main-clause predicates while preserving other content and handling logical structure.The prompts also support completion-based and option-selection evaluation formats.Source: M Prompt Templates Used in Evaluation · pp. 24–25

N Evaluation Setup

The evaluation spans pretrained and instruction-tuned models across several model families and API providers. It tests zero-shot and few-shot prompting, alongside zero-shot evaluation after supervised fine-tuning on the sentence-negation pairs.

N Evaluation Setup

N.1 Models

  • The model set spans pretrained and instruction-tuned systems from Gemma, Qwen, LLaMA, and Mistral, plus OpenAI and Anthropic API models.API models are evaluated only with option selection because token-level log probabilities are unavailable for completion-based evaluation.Source: §N.1 · pp. 25–26

N.2 Zero-shot and few-shot evaluation

  • Models were tested zero-shot and with 1, 5, or 10 demonstrations, averaging few-shot results over five random seeds.Demonstrations come from the demonstration set, and reported performance is measured on the test set.Source: §N.2 · p. 26

N.3 Supervised fine-tuning

  • SFT uses the sentence-negation pair dataset with LoRA for three epochs and is limited to models below 10B parameters.After SFT, models are evaluated zero-shot to isolate task-specific training from in-context examples; API models are not fine-tuned.Source: §N.3 · p. 26

O Total Model Performance on Thunder-NUBench

  • Thunder-NUBench results are reported separately for definition and detailed prompts, across zero-shot, few-shot, and zero-shot SFT conditions.The appendix directs readers to separate tables for openweight and API models under both instruction styles.Source: O Total Model Performance on… · p. 26

P General Benchmark Performance after SFT

The authors tested whether Thunder-NUBench fine-tuning affects broader natural-language understanding performance. General-benchmark performance remained broadly stable after SFT.

P General Benchmark Performance after SFT

Q Total Error Analysis of Model Predictions on the Thunder-NUBench

The error analysis examines incorrect-choice distributions and local-negation confusion rates across model sizes, training paradigms, and evaluation conditions. For API models, it separately tracks outputs that violate the required answer format.

Q Total Error Analysis of Model Predictions on the Thunder-NUBench

  • The analysis compares error patterns across model sizes and pretrained versus instruction-tuned models under zero-shot and few-shot conditions.Tables report error rates, incorrect-choice distributions, and confusion rates across local-negation subcategories, with SFT results also included.Source: Q Total Error Analysis of Model… · p. 27
  • Few-shot error analysis uses a fixed seed of 1234 to keep specific error patterns interpretable.Source: Q Total Error Analysis of Model… · p. 27
  • API outputs that fail the A/B/C/D format are counted separately, excluded from wrong-choice and confusion statistics, and treated as an indicator of format-following ability.Source: Q Total Error Analysis of Model… · p. 27

R Human Evaluation

Human evaluation found that distinguishing standard negation from related alternatives can challenge readers, though human accuracy exceeded zero-shot performance of most models. Eleven English-proficient adults evaluated a stratified sample of Thunder-NUBench items.

R Human Evaluation

  • The human evaluation received IRB approval, and participants gave informed consent before taking part.Responses were collected anonymously, and only aggregate evaluation scores were to be reported.Source: R Human Evaluation · p. 27
  • Eleven adults in their 20s–30s participated: six highly proficient English users at CEFR C level and five native or dominant English speakers.Participants were required to be native or highly proficient English users, verified during recruitment.Source: R Human Evaluation · p. 27
  • The evaluation used 50 stratified multiple-choice items, including 12 each from four local-negation clause types and two without local negation.Participants selected the standard-negation option under detailed instructions, without example items.Source: R Human Evaluation · p. 27
  • Human accuracy ranged from 0.58 to 0.96 (mean 0.78), exceeding zero-shot performance of most models in the paper.The authors interpret the range as evidence that distinguishing standard negation from closely related alternatives can challenge human readers.Source: R Human Evaluation · pp. 27–28, Table 13 · p. 27
Loading 2506.14397v4…