PaperlayerView exact revision on arXiv

Source-linked AI summary

Does Linguistic Structure Enrichment Enhance Coherence Assessment? Not With Current Architectures

Victor Mazzotti, Luiz Pereira, Marina Bitencourt dos Santos, Helena Maia, Carlos Caetano, Nádia Felix, Sandra Avila

arXiv:2609.10893v1cs.CLcs.AI

TL;DR

The paper asks whether adding syntactic and rhetorical structure improves coherence assessment for fluent but potentially incoherent language-model text. It tests RST and POS enrichment and reports that plain text performs best with current architectures, while coherence assessment also shows promise as a disinformation proxy. The authors therefore point toward architecture-aware ways of incorporating linguistic features.

  • Problem

    The paper addresses the challenge of detecting contradictions, topic shifts, and logical flaws in fluent language-model text, particularly where coherence resources are scarce.

  • Method

    The study enriches text with RST and POS information, modifies the tokenizer and language model, and trains coherence classifiers across enriched and plain-text pipelines.

  • Results

    Plain-text models achieve the best accuracy over RST- and POS-enriched texts, while coherence classification can serve as a proxy for disinformation detection.

  • Takeaways & Limitations

    Simply adding RST and POS structure to text does not improve coherence assessment with current architectures, motivating embedding or specialized-network approaches.

  • Takeaways & Limitations

    The approach is limited by RST parser coverage to six languages and by the scarcity and inconsistency of expert RST annotations.

Abstract

from arXiv · show

Recent advances in large language models have transformed human-computer interaction. Despite their fluency, these models often produce texts that are grammatically correct but semantically incoherent, containing contradictions or disruptions in logical flow. This work investigates whether enriching text with syntactic and rhetorical information can improve incoherence prediction. Our experiments and analysis show that plain texts achieved higher accuracy because the added information was structurally and syntactically incompatible with the language model's architecture. Additionally, to demonstrate the practical importance of coherence assessment, we performed zero-shot experiments on a Brazilian disinformation dataset, suggesting that textual coherence can serve as a proxy for detecting misleading content. Code and models are available at https://github.com/ittozzamV/cohereclassifier.

1 Introduction

The paper examines whether linguistic enrichment with RST and POS information improves incoherence detection in generated narratives. It also evaluates whether textual coherence can act as an indirect signal of disinformation.

  • Textual incoherence includes contradictions, abrupt topic shifts, and logical flaws, making coherence assessment important for sensitive applications and under-resourced languages such as Portuguese.
  • The study combines Rhetorical Structure Theory and Part-Of-Speech information with machine learning to test whether enriched inputs improve incoherence detection.
  • The authors evaluate whether textual incoherence can serve as an indirect indicator of disinformation using the Brazilian FakeTrueBR dataset.
  • The proposed contributions include testing RST/POS-derived symbols for incoherence classification and assessing coherence classification as a practical proxy for disinformation detection.

2 Background and Related Work

The background introduces RST and POS as sources of rhetorical and syntactic information, then situates the work among two strategies for incorporating linguistic knowledge into classifiers. The paper targets the less-studied strategy of augmenting Transformer inputs with linguistic information.

  • Rhetorical Structure Theory: RST represents rhetorical relations between non-overlapping text segments called EDUs, distinguishing salient nuclei from supporting satellites.
  • Part-Of-Speech Tagging: POS tagging assigns grammatical class labels to tokens, providing syntactic information for analyzing grammatical structure and meaning.
  • Coherence Classification: Coherence-classification research incorporates linguistic knowledge either through architectural or embedding changes, or by processing inputs and outputs to include linguistic information.
  • Proposed Method: The proposed pipeline enriches text with linguistic information before tokenizer and language-model modification, encoding, and coherence-classifier training.
  • Coherence Classification: This work investigates augmented inputs in Transformer-based coherence classification, addressing mixed prior evidence about whether linguistic information improves performance.

3 Coherence Classifier

The coherence classifier compares plain text with versions enriched by RST or POS information. Its pipeline extracts linguistic features, inserts them into text, adapts the tokenizer and model, encodes the text, and trains a classifier.

  • 3 Coherence Classifier: The five-step methodology extracts POS/RST information, enriches text, modifies the tokenizer and language model, encodes text, and trains a coherence classifier.The Plain pipeline includes only text encoding and classifier training.
  • 3.2 Application of Text Enrichment: RST information is extracted with DMRST, while spaCy supplies word-level grammatical-class annotations for POS enrichment.The two enrichment methods both add preprocessed information to plain text but differ in how it is added.
  • 3.2 Application of Text Enrichment: RST enrichment adds special tokens for rhetorical relations, nuclearity, and EDU boundaries, using 35 tokens to represent the supported combinations.The tokens are inserted around EDU divisions according to parser output.
  • 3.2 Application of Text Enrichment: POS enrichment adds underscore-prefixed tokens representing the 17 grammatical classes returned by the parser.The tags follow the multilingual Universal POS tag set.
  • 3.3 Changing Tokenizer and Model: The tokenizer treats enrichment markers as unsplittable special tokens, and added embedding vectors are initialized from average existing weights to support model convergence.These changes accommodate the expanded vocabulary required by the enrichment methods.
  • 3.3 Changing Tokenizer and Model: Weighted binary cross-entropy gives greater weight to the minority incoherent class because the GCDC corpus is imbalanced.Equation 1 defines the class weights, labels, predicted positive-class probability, and dataset size.

4 Experiments

The experiments train coherence classifiers on the English GCDC corpus and evaluate transfer to Brazilian Portuguese FakeTrueBR texts. The study uses balanced accuracy and Brier Score Loss, with FakeTrueBR reserved for zero-shot evaluation.

  • 4.1 GCDC: GCDC contains English forum posts, business reviews, and emails annotated at three coherence levels, which the study converts into coherent versus incoherent labels.High-coherence texts become coherent, while low-coherence texts become incoherent; medium coherence is excluded.
  • 4.2 FakeTrueBR: FakeTrueBR contains about 3500 Brazilian Portuguese texts from news and other digital media and is used exclusively for zero-shot evaluation without training or validation.The transfer hypothesis links coherence-based detection with contradictions and informal language in disinformation texts.
  • 4 Experiments: The study evaluates balanced accuracy and Brier Score Loss, reporting mean and standard deviation across five runs by corpus and subset.Brier Score Loss measures alignment between predicted probabilities and observed outcomes.

5 Results

On GCDC, Plain achieved the strongest balanced accuracy, while RST generally ranked second; Brier Score Loss instead favored POS. In zero-shot FakeTrueBR evaluation, Plain again performed best and classified about three-quarters of texts correctly.

  • 5.1 GCDC: Plain consistently outperformed RST and POS in GCDC balanced accuracy, with RST ranking second.With 10 learning-rate cycles, Plain exceeded 70.0% balanced accuracy; reducing the learning rate produced the highest average across pipelines.
  • 5.1 GCDC: Yelp produced the weakest subset performance, while RST’s overall advantage over POS did not hold for Clinton and Yahoo.The authors hypothesize that annotation conventions for colloquial and norm-following writing affect Yelp results.
  • 5.1 GCDC: POS achieved the best overall Brier Score Loss, whereas Plain performed worst despite its strongest balanced accuracy.The score ranges from 0, best, to 1, worst, and reflects prediction confidence.
  • 5.2 FakeTrueBR: In zero-shot FakeTrueBR evaluation, Plain performed best, followed by RST, and classified around 3/4 of texts correctly.The models were trained on GCDC and applied to Brazilian Portuguese without task-specific training.

6 Discussion

The discussion attributes enriched pipelines’ weaker coherence performance to mismatches between linguistic representations and the Transformer architecture. Zero-shot results on FakeTrueBR nevertheless indicate a connection between textual incoherence assessment and misleading-content detection.

  • Plain text outperformed RST- and POS-enriched pipelines because the added structures were incompatible with the Transformer architecture.The authors link POS errors to weak alignment with coherence evaluation and RST errors to structural incompatibility.
  • RST enrichment underperformed POS enrichment on the Yahoo and Clinton GCDC subsets because the RST pipeline performed poorly, not because POS produced a gain.The cited comparison is reported for the Yahoo and Clinton subsets in Table 6.
  • Transformer processing flattens RST graphs, causing their structural relations to be retrieved as noise during training.The authors frame this explanation through the Transformer's non-isomorphic mapping in the sequence-vector space.
  • In FakeTrueBR zero-shot evaluation, the Plain pipeline matched its coherence-assessment pattern, while RST and POS performed worse than their coherence counterparts.The POS-enriched pipeline was the worse of the two enriched variants.
  • The Plain pipeline reached 3/4 accuracy on FakeTrueBR despite fine-tuning on another task and language, suggesting a connection between incoherence and misleading content.The authors caution that syntactically well-written misleading texts can be misclassified as real news under POS enrichment.

7 Conclusion and Future Work

The paper concludes that directly adding RST or POS symbols does not improve coherence assessment with the evaluated models. It proposes architectural or representation changes for future enrichment methods and reports promising zero-shot disinformation results while calling for further investigation.

  • Plain text achieved the best accuracy compared with RST- and POS-enriched texts in coherence classification.The conclusion presents this as the paper’s main empirical finding.
  • Directly enriching text with RST or POS information does not serve coherence assessment well in the evaluated setting.The authors restrict this conclusion to RST and POS enrichment.
  • Future work will test additional embedding or alternative network architectures, including graph- and convolution-based approaches, for incorporating linguistic features.The paper specifically proposes architectures that can better account for hierarchical structures.
  • PDTB enrichment is proposed as a potentially better-aligned alternative because its discourse framework is linear and structurally aligned with Transformers.The authors plan to implement and evaluate this enrichment.
  • FakeTrueBR zero-shot experiments produced promising results, but the correlation between coherence and disinformation requires further study to distinguish misleading texts from poorly structured ones.The authors also propose repeating these experiments in the same language domain.

8 Ethics Statements and Limitations

The paper reports ethical use of public data without human subjects and identifies limitations involving annotation resources, coherence’s distinction from truthfulness, and environmental costs. These boundaries constrain how broadly the findings should be interpreted.

  • The study uses publicly available datasets, involves no human-subject data, and is intended solely for academic research.The authors state that the work follows NLP research ethics principles.
  • RST annotation is limited by the DMRST parser’s six supported languages, scarce annotated datasets, and potential inconsistency among expert annotators.The paper attributes inconsistency to differences in interpretation and linguistic nuance.
  • Coherent text can still be biased, misleading, or factually incorrect, so coherence does not by itself establish truthfulness.The authors illustrate this distinction with xenophobic but structurally coherent GCDC examples.
  • The reported experiment consumed an estimated 1,155.00 kWh, corresponding to 113.58 kgCO2e and 21,464.91 L of water across three GPUs.These estimates include the stated Brazil-based footprint calculation.
  • The authors used DeepL Translate, ChatGPT, and Gemini for language refinement and editing, while retaining responsibility for the final text.The tools were used during paper preparation rather than as the evaluated research method.

Appendix

The appendix formalizes why decoder-only Transformers cannot represent the full continuous output space, establishing non-surjectivity and non-isomorphism. It connects this architectural constraint to the flattening of rhetorical structure and the underperformance of enriched inputs.

  • Transformer formulation: The architecture is represented as a composition of Transformer blocks that apply self-attention, activation, and normalization to sequence embeddings.Each block is modeled as a parametrized function from R^d×n to R^d×n, and the full Transformer composes L such blocks.
  • Non-surjectivity theorem: The appendix proves that a continuous decoder-only Transformer is not surjective, so its image is only a proper subset of R^d×n.The proof constrains the image to a lower-dimensional manifold, which has measure zero in the codomain.
  • Non-surjectivity theorem: Layer normalization maps each sequence vector onto a manifold of dimension d − 2, making the full image dimension nd − 2n rather than nd.The full image is formed as the Cartesian product of the per-vector manifolds.
  • Discrete-to-continuous setting: The Transformer is also non-surjective when modeled as a discrete-to-continuous map because a finite vocabulary cannot cover the uncountable simplex.The appendix therefore characterizes the operator as non-isomorphic in both continuous and discrete-to-continuous settings.
Loading 2609.10893v1…