PaperlayerView exact revision on arXiv

Source-linked AI summary

Hierarchical Graph Convolutional Network Approach for Detecting Low-Quality Documents

Jaeyoung Lee, Joonwon Jang, Misuk Kim

arXiv:2610.21350v1cs.AI

TL;DR

Low-quality documents can be internally inconsistent or incongruent with their headlines, a problem that the paper addresses with an HGCN using document metadata. Across inconsistency and incongruence datasets, the model outperformed most comparative models, while the authors note difficulty with sparse-information and specialized-domain documents.

  • Problem

    Low-quality documents may lack internal consistency or be unrelated to their headlines, and such documents fatigue readers.

    Source: §1 · p. 1

  • Method

    The paper proposes an HGCN that uses document metadata to detect within-document inconsistency and headline-body incongruence.

    Source: §1 · p. 2

  • Results

    91.20% on the inconsistency dataset and 92.00% on the incongruence dataset were reported, outperforming most comparative models.

    Source: §5.3 · p. 7, §6 · p. 9

  • Takeaways & Limitations

    The results support the HGCN's applicability to detecting both document inconsistency and headline-body incongruence.

    Source: §6 · p. 9

    The model struggled with low-quality documents when metadata such as subheadings or image captions was absent or brief, and with complex specialized-domain content.

    Source: §5.5 · p. 9

Abstract

from arXiv · show

Consistency within a document is a crucial feature indicative of its quality. Recently, within the vast amount of information produced across various media, there exists a significant number of low-quality documents that either lack internal consistency or contain content utterly unrelated to their headlines. Such low-quality documents induce fatigue in readers and undermine the credibility of the media source that provided them. Consequently, research to automatically detect these low-quality documents based on natural language processing is imperative. In this study, we introduce a hierarchical graph convolutional network (HGCN) that can detect internal inconsistencies within a document and incongruences between the title and body. Moreover, we constructed the Inconsistency Dataset, leveraging published news data and its meta-data, to train our model to detect document inconsistencies. Experimental results demonstrated that the HGCN achieved superior performance with an accuracy of 91.20% on our constructedInconsistency Dataset, outperforming other comparative models. Additionally, on the publicly available incongruent-related dataset, the proposed methodology demonstrated a performance of 92.00%, validating its general applicability. Finally, an ablation study further confirmed the significant impact of meta-data utilization on performance enhancement. We anticipate that our model can be universally applied to detect and filter low-quality documents in the real world.

1. Introduction

The paper addresses low-quality documents with internal inconsistency or headline-body incongruence, which can fatigue readers. It proposes a metadata-using HGCN and constructs a news dataset to detect both types.

1. Introduction

  • Low-quality documents may be internally inconsistent or incongruent with their headlines, fatiguing readers.The paper describes documents that initially contain relevant content but later include inconsistent advertising material.Source: §1 · p. 1
  • Earlier approaches and datasets often targeted only headline-body incongruence or within-document consistency, limiting coverage of varied low-quality documents.The authors argue for a model that detects multiple types of low-quality documents.Source: §1 · pp. 1–2
  • The proposed metadata-using HGCN targets both internal consistency and headline-body incongruence, and the Inconsistency Dataset contains 216,512 real-news documents.The dataset uses title, subheading, body text, and image caption; inconsistency is modeled as sentence intrusion.Source: §1 · p. 2
  • HGCN achieved significant performance on document inconsistency and headline-body incongruence detection.The authors present this as evidence of the model's ability to detect both types of low-quality documents.Source: §1 · p. 2

2. Related Works

Related work treats coherence, consistency, and congruence as distinct document-text tasks. Prior methods and datasets address these tasks in different ways, while the paper positions its approach as covering both inconsistency and incongruence.

2. Related Works

  • The paper distinguishes coherence, consistency, and congruence as separate document-text tasks and targets the latter two.Coherence concerns sentence-order flow, consistency concerns intrusion from another document, and congruence concerns relations among text types.Source: §2 · p. 2

2.1. Inconsistency Detection

  • Prior inconsistency work modeled sentence intrusion, including graph-based detection, but Jung et al. (2022) used only article bodies and omitted titles, subheadings, and captions.Shen et al. (2021) used Wikipedia and CNN articles; Jung et al. (2022) swapped sentences between same-category news articles.Source: §2.1 · p. 2

2.2. Incongruent Detection

  • Incongruence research ranges from linguistic and statistical features to neural attention and graph models, but these approaches focus on incongruence rather than inconsistency.The cited work compares text types such as titles, subheadings, and bodies.Source: §2.2 · pp. 2–3

2.3. Related Dataset

  • Dataset construction methods insert or swap article bodies, paragraphs, or sentences; prior approaches have limitations including length bias and dependence on particular selection signals.Yoon et al. (2021) swapped paragraphs to reduce length bias; Jang et al. (2022) and Jung et al. (2022) used class codes to select similar news.Source: §2.3 · p. 3

3. Dataset

The dataset uses Yonhap News articles and multiple metadata fields, then creates inconsistent examples through automated sentence swapping between selected articles.

3.1. Data Description

  • The source data comprises 22 months of Yonhap News articles with titles, subheadings, body text, image captions, and additional metadata.Articles with insufficient text or specified unsuitable formats were excluded, and preprocessing removed text noise and personal information.Source: §3.1 · pp. 3–4

3.2. Dataset Generation

  • The automated generation process selects same-class articles with similar body lengths, avoids overlapping send dates, and swaps sentences at a 0.5 shuffle ratio.The model receives title, subheading, body text, and image caption for binary inconsistency classification.Source: §3.2 · p. 4, Figure 1 · p. 4

4. Proposed Method

HGCN represents document components at multiple levels, builds headline, body, and document vectors, then propagates information over a graph for inconsistency classification.

4. Proposed Method

  • HGCN organizes document components as a hierarchy graph and trains a classifier to detect document inconsistency.The model places edges between related components and minimizes a loss based on consistency labels and predicted inconsistency probabilities.Source: §4 · p. 4, §4.5 · p. 6

4.1. Sentence Embedding

  • BERT's final-layer [CLS] vector embeds the title, subheading, image caption, and each body sentence; body length is set to k=21.The title, subheading, and image caption are each encoded as one vector, while the body is encoded sentence by sentence.Source: §4.1 · p. 4

4.2. Headline Representation

  • Title and subheading vectors are concatenated and passed through a two-layer MLP with ReLU and down-projection to create a dimension-matched headline.The down-projection makes the headline representation dimensionally consistent with the title and subheading vectors.Source: §4.2 · p. 5

4.3. Body Representation

  • Eight-head self-attention captures body-sentence relations; headline–body congruence and internal-consistency scores are combined to weight the body representation.The weight α controls the contribution of congruence and consistency information.Source: §4.3 · p. 5

4.4. Document Representation

  • The document representation is formed by concatenating the headline and body vectors, then passing them through a two-layer MLP.The authors describe this structure as capturing hierarchical document features.Source: §4.4 · p. 6

4.5. Hierarchical Graph Convolutional Model

  • Four graph-convolution layers propagate information over an undirected graph, with skip connections intended to prevent over-smoothing before binary classification.The graph layer count was set to four through hyperparameter search.Source: §4.5 · p. 6

5. Experimental Results

The experiments evaluate HGCN and comparative models on the Inconsistency Dataset and a dataset for headline–body incongruence. HGCN achieves the highest reported performance on the Inconsistency Dataset and also performs well on the incongruence dataset, though below Jang et al.’s model there.

5.1. Experimental Setup

  • The experiments used 216,512 samples split into 194,860 training, 10,826 validation, and 10,826 test examples.Table 1 presents characteristics of the Inconsistency Dataset.Source: §5.1 · p. 6, Table 1 · p. 6

5.2. Comparative Models

  • Comparative models included TF-IDF+SVM, GRU variants, 3HAN, AHDE, and the Headline-Body Attention Model, with document fields concatenated where needed for fair comparison.The comparisons also included models designed to detect headline–body incongruence.Source: §5.2 · pp. 6–7

5.3. Experimental Results of Low-quality Document Detection

  • 91.20% was HGCN’s highest performance on the Inconsistency Dataset, ahead of the compared models, including the Headline-Body Attention Model at 83.12%.The comparison uses accuracy because the dataset’s labels are balanced.Source: §5.3 · p. 7, Table 2 · p. 7
  • 92.00% was HGCN’s performance on the incongruence dataset, 1.68% below Jang et al.’s model.That dataset contains headline–body incongruence but not inherent document inconsistencies.Source: §5.3 · pp. 7–8, Table 2 · p. 7

5.4. Ablation Study

The ablations examine GCN depth, headline-attention weighting, and document metadata. Results favor four GCN layers, an alpha value of 0.3, and using both subheadings and image captions.

5.4.1. Effectiveness of Number of GCN Layers

  • Performance rose with GCN depth but declined beyond four layers, so subsequent experiments used four layers.The authors added a two-layer MLP and skip connections to strengthen node representations, but these did not completely solve over-smoothing.Source: §5.4.1 · p. 8, Table 3 · p. 8

5.4.2. Effectiveness of Headline Attention

  • The best performance was observed at alpha = 0.3, the headline-attention reflection ratio used in subsequent experiments.Alpha controls the reflection ratio of the headline representation.Source: §5.4.2 · p. 8, Table 4 · p. 8

5.4.3. Significance of Meta-data Information

  • Using neither subheadings nor image captions yielded 80.78%; excluding only image captions yielded 86.10%, and excluding only subheadings yielded 89.02%.The results indicate that both metadata fields contribute to the model’s node representations.Source: §5.4.3 · p. 8, Table 5 · p. 8

5.5. Qualitative Analysis

The qualitative analysis identifies several cases in which HGCN struggles: documents with little metadata or text, specialized subject matter, and subtly swapped content that appears related to the title.

5.5. Qualitative Analysis

  • HGCN struggles with low-quality documents when subtitles or image captions are absent or the text is brief.The paper notes these cases may provide limited information.Source: §5.5 · pp. 8–9
  • Complex documents in specialized domains such as medicine, law, and accounting are difficult for the model to detect.The paper notes these domains can challenge even human readers without expertise.Source: §5.5 · p. 9
  • Subtle content swaps can evade detection when title and body appear to concern the same issue.The paper says these cases call for stronger contextual understanding of sequences, numerics, dates, and domain knowledge.Source: §5.5 · p. 9

6. Conclusions

The conclusion reports that HGCN detects both within-document inconsistency and headline–body incongruence, with strong performance on two datasets. It also outlines future extensions, including a two-phase approach using large language models.

6. Conclusions

  • 91.20% and 92.00% were HGCN’s respective performances on inconsistency- and incongruence-related datasets, outperforming most comparative models.The Inconsistency Dataset contains 216,512 Yonhap News articles with titles, subheadings, body text, and image captions.Source: §6 · p. 9
  • HGCN uses metadata to identify inconsistency within documents and incongruence between headlines and bodies.Comparative experiments found that the utilized metadata significantly impacts performance.Source: §6 · p. 9
  • A proposed two-phase approach would apply large language models to samples not accurately classified by the resource-efficient model.The paper presents this as a potential way to enhance detection performance.Source: §6 · p. 9
  • The authors aim to expand the model structure and use more metadata to detect a broader range of low-quality document types.The current stated focus is inconsistency and incongruence issues.Source: §6 · p. 9

A. Examples of Qualitative Analysis

The appendix illustrates three types of difficult cases: documents lacking metadata, specialized-domain documents, and documents that subtly swap content while retaining a shared topic.

Documents Lacking Metadata Information

Documents Related to Specialized Domains

Documents Involving the Same Issue

  • The model struggled with a document that swapped content between articles about different rallies on the same pension-reform issue.Table A.3 presents examples involving the same issue; distinguishing them may require sequence and event-date context.Source: A. Examples of Qualitative Analysis · p. 11, Table A.3 · p. 14

Related papers

Loading 2610.21350v1…