PaperlayerView exact revision on arXiv

Source-linked AI summary

Interpretive Variation and Discourse Structure in Detecting Enthymemes in Controversial Political Discourse

Martial Pastor, Nelleke Oostdijk

arXiv:2610.30987v1cs.CL

TL;DR

Enthymeme detection depends on readers reconstructing unstated argument components, and judgments can vary. This paper evaluates models against vote distributions and trains a model to predict them; although verdict F1 remains near 0.67, distribution-level properties improve with more data. The study also examines how discourse structure relates to model behavior and reader disagreement.

  • Problem

    Because enthymeme detection involves interpreting implicit argument structure, readers can disagree, while majority-label evaluation collapses their judgments into one verdict.

    Source: §1 · p. 2, §5.1 · p. 11

  • Method

    The paper evaluates predictions against annotator-vote distributions, introduces Betabin to predict vote-count distributions, and analyzes discourse features and model representations.

    Source: §1 · p. 2

  • Results

    Across training sizes, hard-label F1 is flat while distribution-level metrics change; Betabin improves calibration with more data, and certain absence is mostly learned while certain presence mostly is not.

    Source: §5.4 · p. 16, §6.4 · p. 24

  • Takeaways & Limitations

    Evaluating only the majority verdict leaves unmeasured how readers divide, while vote-distribution evaluation can distinguish contested tweets from consensus judgments.

    Source: §9 · p. 29

    The dataset is small, limiting insight into generalization even within the same topic distribution, and observed phenomena may be tied to the specific topic.

    Source: §10 · p. 30

Abstract

from arXiv · show

Enthymemes are arguments that leave a premise or conclusion unstated, and they are a common persuasion device on social media. Because readers must fill in the missing part themselves, they often disagree on whether a text contains one. This paper asks how to detect and evaluate enthymemes when such disagreement is part of the task. On tweets about vaccination and immigration, each labeled by five annotators, F1 stalls around 0.67 for every fine-tuned model we train, regardless of data size, added features or label type. We therefore ask how these models should be evaluated instead, and propose metrics that compare predictions with the full distribution of annotator votes, together with a model that predicts this distribution directly and captures disagreement better at no cost in F1. We then ask what role discourse structure plays. Models learn to rely on explicit argumentative structure: they reliably recognize tweets without an enthymeme but rarely commit on tweets that contain one. Both the models and a discourse parser miss arguments whose links are left implicit and must be inferred by the reader. Finally, we discuss what these blind spots mean for content moderation systems trained on majority labels.

1 Introduction

The paper treats enthymeme detection as an interpretive task: readers reconstruct unstated argumentative components, so their judgments can vary. It evaluates this variation and examines how discourse structure relates to model success, failure, and disagreement.

1 Introduction

  • The study asks how to evaluate enthymeme detection when readers legitimately disagree about whether an implicit argumentative component is present.It retains variation across readers rather than treating the majority verdict as the only target.Source: §1 · pp. 1–2
  • The paper studies discourse structure through injected RST features, descriptive analyses of model behavior, and probes of encoder representations.Together these analyses examine which discourse profiles models rely on, miss, and share with tweets that divide readers.Source: §1 · p. 2
  • The authors organize disagreement metrics by the properties of vote distributions they assess and add a per-vote measure conditioned on annotator behavior.The added metric tests whether a model’s errors fall on votes unusual for the annotator who cast them.Source: §1 · p. 2
  • Betabin predicts a distribution over vote counts, separating a tweet’s leaning from expected reader division while matching the text-only baseline on macro-F1.Its predictions improve on every measure of vote-distribution shape and remain interpretable in vote units.Source: §1 · p. 2
  • Models learn certain absence of an enthymeme more readily than certain presence, and reported-speech and evaluation configurations mark tweets on which readers divide.Injected RST features are redundant; probing finds fine-tuning promotes explicit stance and attribution configurations, but not implicit marking, and sheds trope content.Source: §1 · p. 3

2 Related Work

Prior work has studied enthymeme reconstruction and detection, discourse structure in implicit argumentation, and learning from annotator disagreement. The paper situates its contribution at their intersection: natural short-text detection evaluated with multiple judgments.

2.1 Enthymeme Detection

  • Earlier enthymeme work emphasized recovering missing components; detection studies used feature-based or neural models, often on texts made enthymematic by deletion rather than naturally occurring tweets.Those studies did not examine detection under label variation because they lacked resources retaining multiple annotations per item.Source: §2.1 · p. 3

2.2 Discourse Structure and Implicit Argumentation

  • Discourse structure has received little attention as a signal for detecting implicit argumentation, though Green’s ArgRST work characterized contexts licensing omitted argument elements.The paper identifies no prior study of discourse relations for detecting implicit premises or conclusions in short text alongside reader variation.Source: §2.2 · p. 3

2.3 Perspectivist NLP and Variation

  • Perspectivist NLP treats annotator disagreement as potentially informative, and prior work develops methods for learning from and evaluating distributions of judgments.For enthymemes, semantic interpretation of possible argument structure can lead readers to different judgments, making variation informative about interpretation.Source: §2.3 · p. 4

3 Task, Dataset, and Annotation

The study frames enthymeme detection as a binary task on naturally occurring vaccination and immigration tweets, retaining five annotator votes per item. Its annotation framework treats an implicit component as in scope only when it is controversial.

3.1 Enthymemes

  • An enthymeme is an argument with a missing premise or conclusion, reconstructed here using Walton’s argumentation schemes and a completeness check based on entailment.The scheme’s inferential license is reified as a conditional and added to the premise set.Source: §3.1 · pp. 4–5
  • The annotation guidelines count an unstated component only when it expresses a controversial stance; non-stance implicit content is labeled none.The dataset’s controversial-component criterion is restricted to vaccination and immigration.Source: §3.1 · p. 5

3.2 Task

  • The study merges implicit-premise and implicit-conclusion labels into a binary enthymeme-versus-none task while retaining each tweet’s five annotator votes.The source dataset contains 1,482 English tweets: 1,169 about vaccination and 313 about immigration.Source: §3.2 · p. 5, §3.3 · p. 5

3.3 Dataset and Annotation

  • The dataset’s majority labels comprise 980 none tweets, 439 implicit-premise tweets, and 63 implicit-conclusion tweets.The tweets were drawn from a trope dataset collected through vaccine and immigration keyword queries from late 2019 to late 2023.Source: §3.3 · p. 5

3.4 Trope and RST Annotations

  • The study uses parser-generated RST features for discourse form and gold trope labels for recurring, culturally situated content cues.RST represents coherence in a tree of clause-like elementary discourse units linked by relations; the nine trope categories cover human situations such as “distrust experts.”Source: §3.4 · p. 5

4 Modeling Enthymeme Detection

Across the tested transformer configurations, macro-F1 stays around 0.67, while the vote-distribution model captures disagreement better without losing F1. The section compares feature representations, training targets, and model designs, then examines the performance ceiling with learning curves.

4.1 RST and Trope Feature Representations

  • The 24-dimensional RST encoding uses indicators and ratios to represent relations, nuclearity, ordering, coordination, and tree shape without encoding text length.Trope labels form a separate nine-dimensional binary vector; the feature-only combination reaches 0.608 macro-F1.Source: §4.1 · p. 6

4.3 Fine-Tuned Transformers with Hard vs. Soft Labels

  • Soft-label training targets the five annotators’ vote fraction, while hard-label training targets the majority verdict; soft training gains around one point across seed-paired comparisons.The hard model’s 0.5-threshold score is 0.627 versus 0.671 for soft training, largely reflecting calibration; tuning recovers 0.662 for hard training.Source: §4.3 · p. 7, §4.6.3 · p. 9

4.5 Betabin: Modeling the Votes

  • Betabin predicts a Beta distribution over annotator response probabilities and fits the observed vote count, modeling both a tweet’s leaning and expected reader division.Its vote-distribution gains are not visible in hard-label F1, where it is at parity with the text-only baseline.Source: §4.5 · pp. 7–8, §4.6.3 · p. 9

4.6.1 Results of Features Alone

  • RST features alone reach 0.583 out-of-fold and 0.607 on test, while trope flags reach 0.538 and 0.544, respectively.RST features over-call enthymemes, attaining 0.804 enthymeme recall at the cost of 0.505 none recall.Source: §4.6.1 · pp. 8–9

4.2 Classical Approaches

  • On test, TF-IDF alone reaches 0.554 macro-F1, RST lifts it to 0.605, and the full classical feature combination reaches 0.624.RST chiefly raises enthymeme recall in the full combination, from 0.451 to 0.667; Logistic Regression outperformed linear SVM in every feature combination.Source: §4.2 · p. 6, §4.6.2 · p. 9

4.6.3 Results of Transformer Approaches

  • 0.671 ± 0.02 macro-F1 is reached by the text-only soft-label baseline on test; Betabin scores 0.675 ± 0.02, a seed-paired parity result, and no tested feature injection beats the baseline.Transformer scores cluster between 0.65 and 0.69; the full-size test-set results are in Table 1.Source: Table 1 · p. 8, §4.6.3 · p. 9

4.4 Feature Injection Architectures

  • Gated RST, trope, and combined feature injection do not beat the text-only baseline; the RST system is consistent with structure being a stable but redundant signal.Concatenation performed poorly on short texts, while the gated design lets the model selectively use or ignore features.Source: §4.4 · p. 7, §4.6.3 · p. 9

4.7 Learning Curves and the Performance Ceiling

  • All three learning curves stay within a band of under two points from 300 to 1,334 examples, and tuned hard- and soft-label training are indistinguishable on macro-F1 at each size.Which 300 tweets a model sees matters more than increasing the pool fourfold.Source: §4.7 · p. 10, Figure 1 · p. 10

5 Evaluation Under Annotator Disagreement

Because readers can legitimately interpret enthymemes differently, this section evaluates models against annotator disagreement rather than relying only on a majority-vote verdict. It also proposes a per-vote metric that compares model surprise with annotator-specific voting behavior.

5 Evaluation Under Annotator Disagreement

  • The section evaluates disagreement as a measurable part of enthymeme detection, keeping the five annotators’ judgments rather than reducing them to one verdict.It proposes a metric for whether model errors fall on votes unusual for the annotator who cast them.Source: §5 · p. 11

5.1 Why Gold-Label F1 Falls Short

  • Gold-label F1 discards genuine interpretive variation, and the text-only baseline’s narrow probabilities obscure differences between consensus and contested tweets.Its probabilities range from 0.07 to 0.62, with unanimous enthymemes barely above 0.5; F1 cannot distinguish confidence or consensus from contestation.Source: §5.1 · pp. 11–12

5.2.1 Existing Metrics and What They Measure

  • The soft metrics assess where disagreement occurs, prediction distance from vote fractions, and calibration; depth scores alone can reward predicting near 0.5 on contested items.Ambiguity Spearman assesses whether uncertainty tracks vote entropy; soft-ECE checks whether probabilities match the observed vote distribution.Source: §5.2.1 · p. 12

5.2.2 Results Under Soft Metrics

  • Under soft metrics, the classical system beats gated RST on most vote-shape measures despite lower F1, while Betabin ranks first among transformers.The reported transformer F1 range is .667–.675, compared with .624 for the classical system and .627 for the hard-label model at threshold 0.5.Source: §5.2.2 · pp. 12–13

5.3 An Annotator-Conditioned Per-Vote Metric

  • The per-vote metric correlates model surprise with how unexpected each vote is under its annotator’s behavior in the context of the other four votes.A routine dissent receives lower deviation than the same dissent from an annotator who rarely casts one; the model still makes one tweet-level prediction without annotator identities.Source: §5.3 · pp. 13–14

5.2.2 Results Under Soft Metrics

  • Betabin improves vote-distribution measures while matching the text-only baseline on macro-F1, including 13% lower Manhattan distance and soft-ECE .017 vs. .114.It also raises consensus separation from .188 to .286 and ambiguity Spearman from .172 to .282.Source: §5.2.2 · p. 13, Table 2 · p. 15

5.3.1 Results under the Annotator-Conditioned Per-Vote Metric

  • On the per-vote metric, Betabin reaches .239, while gated RST scores .232 and the text-only soft model .205.Gated RST is second on this metric despite ranking weakest among transformers on vote-distribution shape measures.Source: §5.3.1 · pp. 14, 16, Table 2 · p. 15

5.4 Summary and Scope of the Variation Metric Results

  • Across training-pool sizes, F1 is flat from 300 examples onward while distribution metrics change; different training regimes improve different vote-distribution properties.Betabin’s calibration improves continuously, while the gated RST system is flat or declining on nearly every panel.Source: Figure 3 · p. 15, §5.4 · p. 16

6 Discourse-Structure-Oriented Success and Error Analysis

The analysis profiles where models succeed and fail against discourse structure, comparing easy consensus cases with gains and errors under vote-distribution metrics. Betabin is restrained on contested tweets and commits on unanimous nones, while explicit argumentative structure helps explain both patterns.

6.1.1 Discourse Profile of Easy Nones

  • Easy nones have flatter trees with fewer argumentative satellites, attribution nodes, and embedded joint nodes than their band-matched reference.Their spine ratio is .52 versus .32 (δ +.48).Source: §6.1.1 · p. 17, §6.1 · p. 17, Table 3 · p. 17

6.1 Success Cases on Consensus Tweets

  • All four systems classify easy nones with flat, largely non-argumentative trees, while easy enthymemes expose more argumentative satellites near the root.Easy cases are more common under annotator consensus (.65 of unanimous nones and .59 of unanimous enthymemes) than on even splits (.26).Source: §6.1 · p. 16, §6.1.1 · p. 17, §6.1.2 · p. 18

6.1.2 Discourse Profile of Easy Tweets Containing an Enthymeme

  • Easy enthymemes have more root-proximate argumentative satellites, and most argument components are overtly related in the full-tweet tree.Co-occurring evaluation or contrast with cause, explanation, or condition is 2.4 times more frequent than in the remaining potential-enthymeme tweets.Source: §6.1.2 · p. 18

6.2.1 Discourse Profile of Where Contested Enthymemes

  • The 15 contested success tweets share more attribution and evaluation, a reported-speech-plus-judgment pattern associated with divided readings.All 15 move toward .5 under betabin; they account for the full out-of-fold ambiguity-Spearman gap (.172 to .209).Source: §6.2.1 · p. 19

6.2.2 Discourse Profile of Depth, Calibration, Consistency Unanimous Nones

  • The depth, calibration, and consistency success sets largely contain unanimous nones; betabin commits on them while the text-only model remains near threshold.The calibration set has no enriched discourse dimension, while the consistency set shows marginal topic–comment enrichment linked to rhetorical questions and comments.Source: §6.2.2 · pp. 19–20, Figure 5 · p. 20

6.2 Success Analysis by Variation Metric Family

  • Betabin moves contested tweets toward .5 and commits on unanimous nones, while the text-only model stays near threshold on the latter.The two behaviors align with reported speech plus evaluation in contested cases and weaker argumentative structure among unanimous nones.Source: §6.2.2 · pp. 19–21, §6.2.1 · p. 19

6.3 Error Analysis by Variation Metric Family

The error analysis finds that models misread implicit relations and rely on visible argumentative structure as a proxy for enthymeme labels. This creates failures in both directions: argued nones can look uncertain, while enthymemes with unstated links remain uncommitted.

6.3.1 Discourse Profile of Where Failures

  • The 120 tweets whose contestedness all four systems misrank include implicit junctions, contested items without reported speech, and consensus enthymemes that models never fully commit to.Unmarked junctions can conceal evidence relations and reader-supplied warrants; the parser and models instead treat them as continuation.Source: §6.3.1 · pp. 21–22

6.3 Error Analysis by Variation Metric Family

  • The where metric ranks uncertainty rather than probability placement, and the failure analysis profiles the 120 tweets all four systems misrank.The vote-band breakdown is 68 consensus nones, 18 contested tweets, and 34 consensus enthymemes.Source: §6.3.1 · p. 21, §6.3 · p. 21

6.3.2 Discourse Profile of Depth, Calibration, Consistency Failures

  • Uncommitted consensus nones often contain explicit argument structure, whereas uncommitted consensus enthymemes have fewer argumentative satellites and implicitly marked junctions.The sets are nested (consistency ⊂ calibration ⊂ depth); 81% of consensus enthymemes are uncommitted, versus 36% of consensus nones.Source: §6.3.2 · pp. 22, 24

6.4 Summary of the Success and Error Analysis

The success and error analyses show an asymmetry: discourse structure helps models recognize the certain absence of an enthymeme more than its certain presence. Vote-distribution metrics reveal this behavior and reader variation that majority-label F1 misses.

6.4 Summary of the Success and Error Analysis

  • Certain absence is mostly learned, while certain presence mostly is not; variation metrics expose this asymmetry without reducing F1.Betabin’s advantage includes restraint on divided votes and commitment at the unanimous poles.Source: §6.4 · p. 24
  • Models use surface argumentative structure as a proxy: errors arise when discourse form and the enthymeme label diverge, especially at unmarked junctions.The parser also reads these implicit relations as continuation rather than evidence.Source: §6.4 · p. 24
  • The enthymeme-side failure spans 81% of its consensus band, and learning curves show no improvement on these tweets with more data.Source: §6.4 · p. 24

7 Discourse Structure Inside the Model

The probing analysis compares discourse configurations in frozen pretrained and fine-tuned encoders. Fine-tuning promotes explicit stance and attribution patterns but does not strengthen implicit marking, while trope content is shed.

7.1 Probing Setup

  • The probing compares frozen pretrained and fine-tuned DeBERTa representations using logistic-regression probes and out-of-fold macro-F1.The fine-tuning shift is the probe score after fine-tuning minus the pretrained score; shifts around +.04 or higher count as promotion against the random-signature floor.Source: §7.1 · p. 25, §7.2 · p. 25

7.2 Probing Results

  • Fine-tuning promotes explicit stance and attribution configurations but leaves implicit marking and continuation unpromoted, while shedding trope content.Any-trope presence shifts by −.055, the largest demotion and the only one beyond the random-signature floor; effect sizes are moderate and the probe battery is a screening.Source: §7.2 · pp. 25–26

8 Discussion

The discussion argues that F1 has limited room to improve on this data and can obscure meaningful differences in how models treat agreement and disagreement. Distributional metrics expose three possible distortions in moderation pipelines, while leaving the failure to detect consensus enthymemes unresolved.

8 Discussion

  • Majority-label moderation pipelines collapse judgments that may be constitutive of interpretation, while missed implicit premises may become taken-for-granted community knowledge.The discussion presents this sedimentation and polarization pathway as a possibility, not a demonstrated effect of the systems studied.Source: §8 · pp. 26–27

8.1 Closing the Case for F1 in Semantic Interpretation Tasks

  • On this data, better F1 could mitigate some distortion by catching enthymemes with visible argumentative satellites and clearing flat single-relation nones.The discussion also cautions that a deployed detector may invite producers to adapt surface form to the paratactic profile it misses.Source: §8.1 · p. 27
  • 0.65–0.69 macro-F1 across transformer configurations and flat learning curves from 300 examples give little reason to expect further F1 gains on this data.The paratactic-profile failure spans 81% of consensus enthymemes and shows no sign of improving with more data.Source: §8.1 · p. 27

8.2 Variation Metrics and the Discourse Ecosystem

  • A verdict-based pipeline can treat complete arguments as near-controversial when models hold consensus nones at the threshold; calibrated distributional output separates confident rejection from indecision.Betabin makes this separation without losing F1.Source: §8.2 · pp. 27–28
  • F1 can penalize predictions near .51 on contested tweets and reward a single settled reading, whereas the where metric can flag them for rerouting or withholding automatic action.The discussion identifies reported speech paired with an evaluation as a discourse profile associated with annotator division.Source: §8.2 · p. 28
  • Variation metrics cannot fix the consensus-enthymeme blind spot: no model commits to these cases, though distributions can show that threshold-derived verdicts lack model commitment.The discussion frames this as making a declared blind spot visible rather than falsely clearing the content.Source: §8.2 · p. 28

9 Conclusion

The conclusion reports a macro-F1 ceiling alongside gains in evaluating vote distributions and a discourse-based account of model failures. It argues that judgments and their variation matter because readers supply missing argument components that are not present in the text.

9 Conclusion

  • 0.65–0.69 macro-F1 across fine-tuned configurations, flat learning curves from 300 examples, and 37% reader disagreement mark a detection ceiling on this task.Betabin predicts vote-count distributions and matches the text-only baseline on macro-F1.Source: §9 · p. 28
  • Vote-distribution metrics separate systems tied on F1, and the measured distributional properties improve with data where verdict performance does not.Betabin moves contested tweets toward .51 and commits on unanimous nones held at .47 by the text-only model.Source: §9 · p. 28
  • The per-vote metric conditions each vote on its annotator’s behavior, offering a first step toward distinguishing inconsistent annotation from genuine disagreement.A dissenting vote can be routine for one annotator and unusual for another.Source: §9 · pp. 28–29
  • Fine-tuning promotes explicit stance and attribution configurations, while models use explicit argumentative structure as a proxy that fails when form and label dissociate.The failures include threshold-level predictions for supported judgments and no commitment on enthymemes without argumentative marking.Source: §9 · p. 29
  • The failure to commit affects 81% of consensus enthymemes, whose implicit junctions the parser labels elaboration or joint rather than evidence or cause.The learning curves show no indication that this failure recedes with more data.Source: §9 · p. 29
  • On the same tweet surface, five annotators can agree on an enthymeme whose warrant is absent from the text, while both parser and detector fail to supply it.The conclusion locates the limit at a missing component readers must infer.Source: §9 · p. 29
  • Reported speech paired with an evaluation, often ironic or rhetorical, forms a discourse profile for divided judgments that Betabin learned to associate with split votes.Probing also finds that fine-tuning promotes this configuration in the encoder.Source: §9 · p. 29
  • The authors recommend collecting more information per tweet—including more annotators, individual votes, and reconstructions of missing components—and improving parsers for implicit relations with manually corrected trees.They say more data of the same kind is unlikely to help because learning curves are flat.Source: §9 · pp. 29–30

10 Limitations

The authors identify dataset size as limiting insight into observed patterns and model generalization, and note that the phenomena may be tied to the specific topic. They also caution that generated RST annotations may affect the findings.

10 Limitations

  • The dataset’s size limits insight into observed patterns, particularly model generalization, even within the same type and topic distribution.The authors note that the observed phenomena may also be closely tied to the specific topic.Source: §10 · p. 30
  • Only generated RST annotations were used, and better parser labeling of less-recognized relations could have yielded different results.Source: §10 · p. 30

A RST Feature Encoding

The RST feature encoding represents automatically generated discourse trees with 24 indicators or ratios, selected for predictive value across training folds.

A RST Feature Encoding

  • The encoding contains 24 dimensions, each an indicator or ratio computed over an automatically generated RST tree.Source: Table A.1 · p. 35
  • Candidate features came from hypothesized argumentative-satellite patterns distinguishing explicit from implicit arguments and discourse configurations associated with annotator disagreement.Source: Table A.1 · p. 35
  • Features were retained if predictive across training folds and assigned nonzero L1-logistic-regression weight in at least three of five folds.Source: Table A.1 · p. 35

B RST Trees of the Discourse Profiles

This appendix presents RST trees exemplifying discourse profiles on which models divide, with corrections, argument reconstructions, and annotator vote counts to contextualize each example.

B RST Trees of the Discourse Profiles

Loading 2610.30987v1…