PaperlayerView exact revision on arXiv

Source-linked AI summary

WikiContradiction: Detecting Self-Contradiction Articles on Wikipedia

Cheng Hsu, Cheng-Te Li, Diego Saez-Trumper, Yi-Zhan Hsu

arXiv:2111.08543v1cs.CLcs.IRcs.LG

TL;DR

Noisy, low-quality Wikipedia articles matter to fact-checking, while conventional contradiction detection compares sentence pairs rather than reasoning over an article and locating its conflicts. The paper introduces WikiContradiction and PCNN for article classification and sentence-pair localization, reporting that PCNN outperforms typical document-classification models on the dataset.

  • Problem

    Wikipedia’s use in fact-checking makes noisy, low-quality articles a concern, while conventional contradiction detection compares sentence pairs rather than reasoning over an article and locating its conflicting pairs.

    Source: §I · p. 1

  • Method

    PCNN encodes article sentences, pretrains pairwise contradiction learning on SNLI and MNLI, then uses top-K high-probability sentence-pair embeddings for article classification.

    Source: §I · p. 2, B. Pairwise Contradiction Learning · p. 5

  • Results

    On WikiContradiction, PCNN outperforms typical document-classification models; pairwise-learning pretraining contributes most, and case studies show it identifies the most contradictory sentence pairs.

    Source: §I · p. 2

  • Takeaways & Limitations

    The authors present WikiContradiction as a training resource for improving Wikipedia article quality and contributing to fact-checking and claim verification.

    Source: §VII · p. 9

    PCNN models sentences only within a paragraph, so it may fail to detect contradictions between paragraphs in long articles.

    Source: §VI · p. 8

Abstract

from arXiv · show

While Wikipedia has been utilized for fact-checking and claim verification to debunk misinformation and disinformation, it is essential to either improve article quality and rule out noisy articles. Self-contradiction is one of the low-quality article types in Wikipedia. In this work, we propose a task of detecting self-contradiction articles in Wikipedia. Based on the "self-contradictory" template, we create a novel dataset for the self-contradiction detection task. Conventional contradiction detection focuses on comparing pairs of sentences or claims, but self-contradiction detection needs to further reason the semantics of an article and simultaneously learn the contradiction-aware comparison from all pairs of sentences. Therefore, we present the first model, Pairwise Contradiction Neural Network (PCNN), to not only effectively identify self-contradiction articles, but also highlight the most contradiction pairs of contradiction sentences. The main idea of PCNN is two-fold. First, to mitigate the effect of data scarcity on self-contradiction articles, we pre-train the module of pairwise contradiction learning using SNLI and MNLI benchmarks. Second, we select top-K sentence pairs with the highest contradiction probability values and model their correlation to determine whether the corresponding article belongs to self-contradiction. Experiments conducted on the proposed WikiContradiction dataset exhibit that PCNN can generate promising performance and comprehensively highlight the sentence pairs the contradiction locates.

I. INTRODUCTION

The paper frames self-contradiction detection as an article-level Wikipedia quality task and introduces WikiContradiction and PCNN to classify articles while locating conflicting sentence pairs.

I. INTRODUCTION

  • The task targets low-quality Wikipedia articles, which matter because Wikipedia is used for fact-checking and claim verification.The introduction motivates identifying articles that contain incorrect information.Source: §I · p. 1
  • Examples include conflicting birthplaces and disagreement over whether a suit was genuine or a copy.The paper notes that contradictory parts can occur in different sentences within an article.Source: §I · p. 1, Table I · p. 1
  • Self-contradiction detection must classify an article and identify its conflicting sentence pairs, combining article semantics with pairwise comparisons.This differs from conventional contradiction detection, which classifies pairs of sentences or claims.Source: §I · pp. 1–2
  • PCNN encodes article sentences, learns pairwise contradiction probabilities, then uses the top-K sentence pairs’ embeddings to classify the article.The pairwise-learning module is pretrained on SNLI and MNLI and fine-tuned on WikiContradiction.Source: §I · p. 2
  • Experiments on WikiContradiction report that PCNN outperforms typical document-classification models, pairwise-learning pretraining contributes most, and case studies show it identifies contradictory sentence pairs.These are the paper’s three stated findings.Source: §I · p. 2
  • The authors create WikiContradiction, a dataset for detecting self-contradiction in Wikipedia articles based on the “self-contradictory” template.The paper describes it as the first dataset for this task on Wikipedia.Source: §I · pp. 1–2

II. RELATED WORK

Prior contradiction-detection work classifies sentence or claim pairs, using varied linguistic and learned features across domains. This work distinguishes itself by addressing article-level self-contradiction and highlighting conflicting input parts.

II. RELATED WORK

  • Prior methods classify pairs of sentences or claims using approaches such as linguistic features, sentiment, textual similarity, and neural models across several domains.Examples include scientific claims, social media, rumorous claims, and commercial or item reviews.Source: §II · p. 2
  • Unlike pairwise contradiction studies, this work addresses self-contradiction within an article and identifies which sentence pairs conflict.Source: §II · p. 2, Table II · p. 3
  • The comparison covers six aspects, including contradiction type, input, features, classifier, and explainability; this work uses learned features and highlights conflicting input parts.Source: §II · p. 2, Table II · p. 3

III. WIKICONTRADICTION DATASET

WikiContradiction uses Wikipedia’s self-contradictory template and later revisions to construct balanced self-contradiction and non-contradiction examples. The template labels rely on expert editors, but may miss articles that have not been reviewed.

III. WIKICONTRADICTION DATASET

  • Template labels are treated as high-quality manual annotations because experienced editors add them, although coverage may be incomplete when articles go unreviewed.The authors report that 87% of users adding a template had over 1,000 Wikipedia edits.Source: §III · p. 3
  • The dataset uses articles with a self-contradictory template and newer revisions where editors removed the template to indicate the contradiction was resolved.The template may appear at the article top or in the section containing the contradiction.Source: §III · pp. 3–4, Figure 1 · p. 3
  • From English Wikipedia revisions through March 2020, the authors found 2,321 templated articles; 1,105 had a later version with the template removed, while 1,216 were discarded.Source: §III · p. 4
  • The resulting dataset contains 1,105 positive and 1,105 negative articles, with all versions of an article kept within either the training or testing split.This split prevents classification-label leakage across versions.Source: §III · p. 4

IV. THE PROPOSED MODEL: PCNN

PCNN treats self-contradiction detection as article classification while ranking sentence pairs by their predicted contradiction probability. It combines sentence representations, pairwise contradiction learning, and correlation modeling of the highest-scoring pairs.

IV. THE PROPOSED MODEL: PCNN

  • A self-contradictory article contains at least two sentences whose meanings or referenced facts contradict, and the task predicts the article label while ranking explainable sentence pairs.Source: §IV · p. 4
  • PCNN generates sentence representations, learns pairwise contradiction probabilities, fine-tunes that learning on WikiContradiction, then uses self-attention over the most suspicious pairs for article classification.The selected pairs are those with the highest contradiction probabilities.Source: §IV · p. 4

A. Sentence Representation

PCNN fine-tunes Sentence-BERT to represent article sentences, then compares sentence pairs using a contradiction-learning network. It focuses pairwise comparisons within paragraphs and pre-trains that network on SNLI and MNLI before WikiContradiction fine-tuning.

A. Sentence Representation

B. Pairwise Contradiction Learning

  • Pairwise comparisons are restricted to sentences within a paragraph, based on paragraph coherence and the computational cost of enumerating every article-level pair.The authors state that they evaluate paragraph-level versus article-level comparison in experiments.Source: B. Pairwise Contradiction Learning · p. 4
  • The PCL network combines transformed sentence vectors with their elementwise difference to estimate contradiction probability; it is pre-trained on contradiction-labeled SNLI and MNLI examples, then fine-tuned on WikiContradiction.Source: B. Pairwise Contradiction Learning · pp. 4–5

C. Article Representation & Classification

PCNN represents an article using its most contradiction-prone sentence pairs, then models how those pairs relate to produce an article-level self-contradiction probability.

C. Article Representation & Classification

  • PCNN selects the sentence-pair embeddings with the highest contradiction probabilities to represent an article, because contradictory sentences tend to be limited.The selected pairs are the top K by probability.Source: C. Article Representation &… · p. 5
  • A self-attention layer models correlations among selected pairs and estimates each pair’s contribution to the article embedding.The attention weights are obtained by applying softmax to the dot product of two transformations of each pair embedding.Source: C. Article Representation &… · p. 5
  • A one-hidden-layer feed-forward network and sigmoid convert the article embedding into a self-contradiction probability, optimized with cross-entropy loss.Training uses Adam with a learning rate of 2e-5 and a linear warm-up over 10% of the training data.Source: C. Article Representation &… · p. 5

V. EXPERIMENTS

The experiments evaluate PCNN against four baselines across balanced and imbalanced settings, varying training proportions and reporting average classification metrics.

A. Evaluation Settings

  • Experiments use balanced data and positive-to-negative ratios of 10:90, 30:70, and 50:50, with ten random sets per setting.The default train-test split is 80:20; the training percentage is also varied from 20% to 80%.Source: A. Evaluation Settings · p. 5
  • Evaluation reports average Precision, Recall, F1, and Accuracy scores; PCNN’s default top-K setting is K = 10.The experiments use PyTorch on Nvidia GeForce GTX 1080 Ti GPUs.Source: A. Evaluation Settings · p. 5
  • PCNN is compared with Random, LSTM, HAN, and BERT, with competing-model hyperparameters following their respective studies.The competing methods use 32-dimensional word embeddings and 64-dimensional intermediate embeddings.Source: A. Evaluation Settings · p. 5

B. Experimental Results

PCNN consistently outperforms competing methods in the balanced main evaluation, while analyses examine ranking, component contributions, top-K selection, imbalance, and remaining difficulty.

Main Results

  • PCNN consistently outperforms competing methods across training percentages and evaluation metrics under the balanced setting.The reported comparison is based on all testing sentences.Source: B. Experimental Results · p. 5, Table III · p. 6
  • Even with 80% of the data used for training, PCNN’s scores remain below 0.65, and the authors describe detection as challenging.This performance boundary is reported after the preceding discussion of approaches to improve results.Source: B. Experimental Results · p. 6

Ranking Results

  • PCNN again leads on Precision and Recall when ranking the top-k predicted self-contradiction articles, although Recall scores are low.The ranking evaluation uses k = 10, 20, and 30 articles with the highest prediction probabilities.Source: B. Experimental Results · p. 6, Table IV · p. 6

Ablation Study

  • Pairwise contradiction learning provides the largest performance improvement; Sentence-BERT and top-K selection also help, while self-attention contributes less.Paragraph-level pairwise learning does not improve performance, but dividing articles into paragraphs can reduce learning time through parallel processing.Source: B. Experimental Results · p. 6, Figure 3 · p. 6

Top-K Sentence Pairs

  • PCNN performs better with K around 10–20; smaller K can miss contradiction evidence, while larger K can include non-contradictory pairs.The study varies K across 1, 5, 10, 20, 30, 40, and 50 and reports Precision, Recall, and F1.Source: B. Experimental Results · p. 7, Figure 4 · p. 7

Imbalance Analysis

  • Under greater class imbalance, including P:N = 10%:90%, performance drops across methods, but PCNN still outperforms competitors and maintains similar Recall.Precision is quite low at P:N = 10%:90%; the study reports comparisons across positive-to-negative ratios.Source: B. Experimental Results · p. 7, Table V · p. 7

C. Case Study

The case study examines correctly and incorrectly classified articles, finding that PCNN’s highlighted sentence pairs can reasonably explain its predictions.

C. Case Study

  • When both ground truth and prediction indicate self-contradiction, PCNN can identify the sentence pair that contradicts.The examples include conflicting birthplaces and differing claims about a suit’s authenticity.Source: C. Case Study · p. 7
  • Some false negatives arise when PCNN treats semantically related sentences as consistent rather than contradictory.Examples include statements about a Cambodian ambassador and embassy, and about birth statistics and origins.Source: C. Case Study · p. 7
  • In false positives, PCNN highlights sentence pairs it regards as contradictory, and the authors say those pairs reasonably explain its predictions.The paper also concludes that PCNN can identify inconsistent sentence parts in non-self-contradictory articles.Source: C. Case Study · p. 7

VI. DISCUSSION

The discussion proposes broader pre-training data and article-pair extensions, while identifying annotation subjectivity and long-document handling as current limitations.

VI. DISCUSSION

  • Table VI pairs ground-truth and PCNN classifications and manually highlights possible contradiction spans in the sentences identified by PCNN.Source: Table VI · p. 8
  • PCNN pre-training could be extended with more contradiction-labeled datasets to cover more diverse aspects of contradiction.The authors suggest this broader coverage may benefit fine-tuning for self-contradiction detection.Source: §VI · p. 8
  • PCNN could be extended from sentence-level to article-level contradiction by changing sentence pairs to article pairs in its encoders.The authors also describe detecting incongruence between a news headline and its body by combining them as an article input.Source: §VI · p. 8
  • Annotation subjectivity may affect how PCNN models contradiction and its detection performance, but the model does not address that issue.The authors suggest mitigating subjectivity in future extensions.Source: §VI · p. 8
  • PCNN models sentences only within paragraphs, so contradictions between paragraphs in long documents may go undetected.A long-document transformer is suggested for processing long sequences of sentences and paragraphs; the authors also describe using PCNN to test contradictions between Wikipedia articles.Source: §VI · pp. 8–9

VII. CONCLUSIONS

The conclusion presents WikiContradiction and PCNN for detecting self-contradictory Wikipedia articles, reports promising empirical performance and case-study explainability, and identifies possible uses and future improvements.

VII. CONCLUSIONS

  • The paper introduces WikiContradiction and PCNN, whose pairwise contradiction learning is pre-trained on SNLI and MNLI and fine-tuned on the new dataset.Source: §VII · p. 9
  • Experiments show promising PCNN performance, and the case study demonstrates model explainability.Source: §VII · p. 9
  • The authors identify WikiContradiction as a potential training resource for improving Wikipedia quality and point to sentence representations, pairwise reasoning, and finer-grained sentence-pair selection for future work.Source: §VII · p. 9
Loading 2111.08543v1…