PaperlayerView exact revision on arXiv

Source-linked AI summary

Not all Negation Cues are Equal: Affixal Negations Yield Better Negation Understanding

Tian Tan, Eduardo Blanco

arXiv:2609.13685v1cs.CLcs.AI

TL;DR

Prior work has focused mainly on a few frequent single-word cues, while negation remains challenging for LMs and LLMs. The paper constructs NegCue, a corpus of over 1.8M samples with equal numbers of affixal, single-word, and multi-word cues, and studies pre-training encoder-only LMs and smaller LLMs. Across five benchmarks, further pre-training improves both LMs and LLMs, with affixal negation yielding larger gains and single-word negation more modest gains.

  • Problem

    Prior work has focused mainly on a few frequent single-word cues, while negation remains challenging for LMs and LLMs.

    Source: §1 · p. 1

  • Method

    The paper constructs NegCue, a corpus of over 1.8M samples with equal numbers of affixal, single-word, and multi-word cues, and studies pre-training encoder-only LMs and smaller LLMs.

    Source: §1 · pp. 1–2

  • Results

    Across five benchmarks, further pre-training improves both LMs and LLMs, with affixal negation yielding larger gains and single-word negation more modest gains.

    Source: §5 · p. 5, §7 · p. 9

  • Takeaways & Limitations

    The results indicate that negation types contribute unevenly to model improvements, with single-word cues producing the smallest gains in the reported comparisons.

    Source: §7 · p. 9

    The analysis is limited to English corpora; extending it to languages with different negation systems remains future work.

    Source: Limitations · p. 10

Abstract

from arXiv · show

Negation remains a longstanding challenge for both language models (LMs) and large language models (LLMs). Prior work mainly focuses on a small set of high-frequency single-word negation cues, such as not and never, with limited exploration of broader negation types and modern LLMs. To address this gap, we construct NegCue, a large-scale dataset containing over 1.8M samples spanning single-word, multi-word, and affixal negation with more than 200 unique cues. We further pre-train both encoder-only LMs and LLMs on NegCue to investigate how different negation types affect negation understanding. Experiments on five downstream benchmarks show that negation types contribute unevenly to performance gains under the same training scale. In particular, affixal negation yields the largest improvements, while the gains from the commonly studied single-word negation remain modest. Moreover, our results demonstrate that further pre-training improves negation understanding for both LMs and LLMs.

1 Introduction

Negation remains difficult for language models, while prior work has studied only a limited set of cues and approaches. This paper investigates training with diverse negation cues using a self-supervised task and a large corpus.

1 Introduction

  • Models continue to struggle with negation, including in question answering and information retrieval.Prior work also found that BERT often disregards negation cues during standard language modeling.Source: §1 · p. 1
  • The study extends cue-focused pre-training to affixal, single-word, and multi-word negation for encoder-only LMs and smaller LLMs.Earlier approaches either generate affirmative interpretations, increasing input size and inference compute, or pre-train encoder-only LMs with selected cues.Source: §1 · p. 1
  • NegCue is a corpus of over 1.8M samples, with equal numbers of affixal, single-word, and multi-word cues; NSPP and NegCue are evaluated on BERT, RoBERTa, and smaller Llama and Qwen variants.Source: §1 · p. 2
  • Across five benchmarks, pre-training with NegCue improves performance, with affixal and multi-word cues generally yielding larger gains than single-word cues.The paper also analyzes which cues are most beneficial during pre-training and inference.Source: §1 · p. 2

2 Related Work

Prior research documents challenges in processing negation and has largely focused on a few frequent cues. Existing approaches include generating affirmative interpretations and self-supervised pre-training; this paper extends cue coverage and evaluates across tasks and model types.

2 Related Work

  • Negation occurs in multiple forms, including affixal, single-word, and multi-word cues, beyond frequent forms such as not and never.Prior estimates report negation in over 28% of book sentences, 27% of conversational sentences, and over 13% of biomedical-paper sentences (Hossain et al., 2020b).Source: §2 · p. 2
  • Negation challenges models across language understanding, machine translation, question answering, and information retrieval.The cited work includes evidence of BERT disregarding negation and task-specific difficulty for encoder-only LMs and LLMs.Source: §2 · p. 2
  • Prior methods either append generated affirmative interpretations, which adds inference compute, or use self-supervised pre-training with a handful of cues.The cited approaches include Hossain and Blanco (2022) and Rezaei and Blanco (2024) for affirmative interpretations, and Hosseini et al. (2021) and Truong et al. (2022a) for pre-training.Source: §2 · p. 2
  • NSPP predicts whether the next sentence contains negation from the preceding sentence, using discourse relations as predictive cues.Rezaei and Blanco (2025) proposed the task for encoder-only LMs; this paper extends its study to all cue types and LMs and LLMs across five benchmarks.Source: §2 · p. 2

3 Improving Negation Understanding

The paper uses Next Sentence Polarity Prediction (NSPP) to train models on NegCue, a corpus with balanced representation of three negation types and over 200 cues. The section describes the task, corpus construction, and cue distributions.

3.1 Next Sentence Polarity Prediction (NSPP)

  • NSPP predicts whether the next sentence contains negation from the preceding sentence, using a binary label.The authors motivate the task with discourse continuity: preceding context may provide cues about the occurrence and type of negation in the next sentence.Source: §3.1 · p. 3
  • A logistic-regression probe trained on about 6K samples predicted next-sentence negation and captured negation-type patterns above random chance.The probe used only the preceding sentence as input.Source: §3.1 · p. 3

3.2 NegCue: A Large-Scale Negation Corpus

  • Each cue type makes up one-third of NegCue, but its top three cues comprise 71% of multi-word, 65% of single-word, and 24% of affixal cues.NegCue has 11 multi-word cues and 167 affixal cues; the most frequent affixal cue, unable, accounts for 10.29%.Source: Figure 2 · p. 3, §3.2 · p. 4
  • The authors combine cue resources to identify 214 cues, then classify them using affixal membership, token counts, longest matches, and manual review.Longest-match prioritization distinguishes a multi-word cue such as “no longer” from the single-word cue “no.”Source: §3.2 · p. 4
  • NegCue uses consecutive Wikipedia sentences, selecting negated examples with one listed cue and non-question targets, then balancing affirmative and negated samples.The final corpus balances total samples across cue types while retaining natural cue frequencies within each type.Source: §3.2 · p. 4
  • The training split contains 1,673,226 samples, balanced across labels and cue types, with 557,742 samples per cue type.The validation and test splits contain 92,952 and 92,964 samples, respectively.Source: §3.2 · p. 4

4 Experiments

The experiments compare off-the-shelf models with models further pre-trained on NegCue across five negation-specific benchmarks. Encoder-only LMs are fine-tuned for evaluation, while LLMs use zero-shot prompting.

4 Experiments

  • Experiments test BERT-large, RoBERTa-large, Llama3.2-1B/3B, and Qwen2-0.5B on five negation-specific benchmarks.Both off-the-shelf and further pre-trained models are evaluated.Source: §4 · pp. 4–5

4.1 NSPP Pre-training with NegCue

  • Further pre-training uses either the full NegCue corpus or subsets containing one negation cue type, enabling comparisons across training data types.Encoder-only LMs share pre-training settings; LLMs use QLoRA with 4-bit quantization.Source: §4.1 · p. 5

4.2 Evaluation on Downstream Tasks

  • Encoder-only LMs are evaluated after supervised fine-tuning, whereas LLMs are evaluated with zero-shot prompting.CondaQA also compares cue-type and cue-frequency subsets; LMs use the benchmark training split and LLMs use zero-shot in-context learning.Source: §4.2 · p. 5
  • NeQA is converted into binary question–answer-option samples for LM fine-tuning, while LLMs are evaluated zero-shot.NeQA has no official training-validation split, so the authors construct these splits from dataset metadata.Source: §4.2 · p. 5
  • NevIR is reformulated as choosing the relevant document from two candidates, and NMoNLI and ScoNe-NLI are evaluated with zero-shot LLMs and SNLI-trained LMs.Direct fine-tuning on the NLI benchmark training sets produces test accuracies exceeding 99%, making meaningful comparisons difficult.Source: §4.2 · p. 5

5 Results and Discussion

Further pre-training with NegCue benefits encoder-only LMs and smaller LLMs across benchmarks, but gains vary by negation type and cue frequency.

5 Results and Discussion

  • NegCue NSPP pre-training consistently improves encoder-only LMs and smaller LLMs across all five benchmarks.CondaQA and the four other NLU benchmarks are reported in Tables 2 and 4.Source: §5 · p. 5, Table 2 · p. 6, Table 4 · p. 8

5.1 Results on Large-Scale CondaQA

  • On CondaQA, Accuracy (All) gains are BERT: +3.3–7.1%, RoBERTa: +20.5–25.8%, and Llama3.2-1B/3B: +0.3–3.0% / +0.7–1.1%, depending on subset.These are improvements over the off-the-shelf counterparts.Source: §5.1 · p. 6
  • Cue-frequency gains generally peak in Q2 and Q3, while performance gradually decreases from Q1 to Q4; NSPP pre-training generally helps across groups.Q1 contains the top 25% most frequent cues.Source: Table 3 · p. 7, §5.1 · p. 7
  • Across LMs and LLMs, affixal and multi-word pre-training yields larger improvements than single-word pre-training, with affixal cues strongest.The table caption also reports that both model families benefit, especially from affixal cues.Source: §5.1 · pp. 6–7, Table 2 · p. 6

5.2 Results on other NLU Benchmarks

  • Across other benchmarks, the best 3B model reaches 73% on NeQA versus the prior 71%; NevIR gains are significant (LMs 1.3–4.9%, LLMs 7.3–19.1%), and nearly all NLI models improve.RoBERTa-large pre-trained on multi-word cues improves by more than 200% on the NLI benchmarks.Source: §5.2 · pp. 7–8, Table 4 · p. 8

6 Ablation Studies

Ablations indicate that negation-specific learning drives the observed gains, and affixal negation remains strongest when cue types are tested under balanced conditions.

6 Ablation Studies

  • The ablations test whether gains come from negation-specific learning rather than generic continued pre-training, and balance cue counts to isolate negation type.The balanced corpora use equal samples per cue type.Source: §6 · p. 8
  • Continued pre-training with affirmative samples and NSP substantially degrades performance, often falling below off-the-shelf models and all NegCue variants.The comparison holds the other experimental settings identical to NSPP pre-training.Source: §6 · pp. 8–9, Table 5 · p. 9
  • Under the controlled setting, affixal-corpus pre-training consistently performs best on CondaQA and NeQA across models.The comparison uses NSPP pre-training on balanced affixal, single-word, and multi-word corpora.Source: Table 6 · p. 9, §6 · p. 9

7 Conclusion

NegCue broadens negation coverage beyond a few frequent cues, and pre-training on it improves performance across five benchmarks. Affixal negation yields larger gains than commonly studied single-word negation.

7 Conclusion

  • NegCue covers over 200 cues across affixal, single-word, and multi-word negation, addressing prior focus on a few frequent forms.The corpus contains over 1.8M samples.Source: §7 · p. 9
  • Further pre-training encoder-only LMs and LLMs on NegCue improves performance across five benchmarks.Source: §7 · p. 9
  • Affixal negation yields substantially larger gains, while commonly studied single-word negation yields only modest improvements.Source: §7 · p. 9

Limitations

The appendix examines the English-only scope of the work, tests what NSPP can learn from preceding sentences, and documents NegCue construction and manual quality checks.

Limitations

  • NegCue's analysis is limited to English; extending it to languages with different negation systems remains future work.Source: Limitations · p. 10
  • Natural cue-frequency imbalances may limit pre-training effectiveness by concentrating exposure on frequent cues and limiting exposure to rare ones.Source: Limitations · p. 10

A.1 Polarity Prediction

  • Manual probing with frozen S1 embeddings achieves nearly 58% accuracy on polarity prediction, above the 50% random baseline.Source: §A.1 · p. 13, Table 7 · p. 13

A.2 Negation Type Prediction

A Validating the Learnability of NSPP

  • Together, the probes suggest S1 contains learnable signals about whether the following sentence is negated and its likely negation type.Source: §A.2 · p. 14

B.1 Construction Details

  • NegCue combines single-word, multi-word, affixal, and All Cues corpora built from Wikipedia sentence pairs and split into train, validation, and test sets.Source: §B.1 · p. 14

B.2 Manual Inspection

  • Two annotators confirmed all 214 cues were correctly categorized; manual sample checks found 99.8% label correctness, 100% cue-type accuracy, and 98.4% sentence validity.Source: §B.2 · pp. 14–15
  • Manual inspection found one affirmative sample with an unlisted negation cue, eight incomplete sentences, and three samples with cues outside the inventory.Source: §B.2 · p. 15

C Additional Results for Qwen

Qwen2-0.5B benefits from NegCue further pre-training across benchmarks, with gains varying by negation type and task. Multi-word and affixal variants generally help more than single-word variants.

C Additional Results for Qwen

D NSPP Further Pre-training Details

The paper details encoder and LLM NSPP pre-training configurations, stopping criteria, and training costs. LLM training loss decreases and converges across the reported settings.

D NSPP Further Pre-training Details

E Benchmark Details

The paper describes a varied benchmark suite and task-specific evaluation protocols for encoder-only LMs and LLMs. The setup includes cue-based analysis for CondaQA and adaptations for evaluating models on NeQA and NevIR.

E Benchmark Details

  • The benchmark-statistics table summarizes training, validation, and test-set sizes across the evaluation datasets.CondaQA also includes group consistency as an evaluation metric.Source: E Benchmark Details · p. 17, Table 10 · p. 18

F.3 Evaluation on NevIR

  • NevIR’s original similarity-ranking task had not been evaluated on language models, so the study reformulates samples as query-and-candidate-document instances.Encoder-only LMs are fine-tuned on the reformulated task; LLMs receive instruction-style prompts.Source: E Benchmark Details · p. 18, §F.3 · p. 19

E Benchmark Details

  • The suite covers contextual QA, commonsense QA, retrieval, and NLI, including benchmarks designed to test negation scope.NMoNLI and ScoNe-NLI are binary entailment tasks; ScoNe-NLI varies negation scope across six contrast splits.Source: E Benchmark Details · pp. 17–18

F Evaluation on Downstream Tasks

  • Encoder-only LMs are fine-tuned on downstream tasks, while LLMs are evaluated zero-shot with deterministic decoding.The evaluation setup also reports hyperparameters for LM fine-tuning.Source: F Evaluation on Downstream Tasks · p. 18

F.1 Evaluation on CondaQA

  • CondaQA evaluation breaks out accuracy and group consistency by negation-cue type and cue-frequency quartile.The test set is partitioned into affixal, single-word, and multi-word cues, and four frequency quartiles.Source: §F.1 · p. 18

F.2 Evaluation on NeQA

  • NeQA uses the same 100 test examples as prior work, with training and validation drawn from the remaining data to prevent test leakage.For encoder-only LMs, each question-option pair is converted into a binary classification example.Source: §F.2 · p. 19

F.4 Evaluation on NLI Benchmarks

In-domain fine-tuning on the NLI benchmarks leaves little room for comparison, so the study also evaluates models fine-tuned on SNLI. On NaN-NLI, RoBERTa further-pretrained on NegCue outperforms the reported baseline across the reported metrics.

F.4 Evaluation on NLI Benchmarks

  • In-domain fine-tuning saturated performance, including 100% accuracy across all six ScoNe-NLI splits for RoBERTa-large, so the authors instead fine-tuned on SNLI.Both negation-focused NLI benchmarks derive from SNLI; models were fine-tuned for one epoch before test evaluation.Source: §F.4 · pp. 19–20, Table 12 · p. 20

G Additional Evaluation on NaN-NLI

  • For the additional NaN-NLI evaluation, encoder-only models were continued-pretrained on NegCue, fine-tuned on MNLI, and assessed under Binary and Strict settings.The comparison is restricted to encoder models to match the original study’s evaluation scope.Source: G Additional Evaluation on NaN-NLI · p. 20
  • Overall F1 rose from 0.670 to 0.709 and Strict accuracy from 0.250 to 0.271 for RoBERTa-NegCue-MNLI versus RoBERTa-MNLI on NaN-NLI.The comparison is reported for the Binary setting and Strict accuracy, respectively.Source: G Additional Evaluation on NaN-NLI · p. 20, Table 13 · p. 21
Loading 2609.13685v1…