PaperlayerView exact revision on arXiv

Source-linked AI summary

Leveraging Speech Acts for Low-Data and Cross-Domain Conversation Derailment Forecasting

Angela Yifei Yuan, Christine De Kock, Christopher Leckie

arXiv:2608.25359v1cs.CL

TL;DR

Derailment forecasting must operate across low-data and cross-domain settings where annotations are limited and lexical variation is substantial. The paper integrates speech-act information with textual semantics as an auxiliary training signal, and reports consistent improvements across three datasets, especially for low-data and cross-domain forecasting. LLM-based speech-act extraction may introduce biases and unreliable handling of indirect intent, so the authors recommend human supervision rather than fully autonomous decisions.

  • Problem

    Derailment forecasting remains underexplored in low-data and out-of-domain environments, where annotation is expensive and new or niche communities have limited labelled data.

  • Method

    The paper integrates speech-act information with textual semantics through hierarchical models that use speech-act detection as an auxiliary training task.

  • Results

    The approach shows consistent improvements over strong baselines across three datasets, particularly in low-data and cross-domain settings.

  • Takeaways & Limitations

    Speech acts provide a pragmatic signal for more robust dynamic moderation systems across diverse and data-scarce real-world settings.

  • Takeaways & Limitations

    LLM-based speech-act extraction can reflect cultural, dialectal, and community biases and may fail to recognize indirect intents, motivating human supervision rather than fully autonomous decisions.

Abstract

from arXiv · show

Conversational derailment forecasting aims to predict when online discussions will escalate into hostility, enabling proactive moderation. Existing approaches often struggle in low-data settings and to generalize across domains. This poses a challenge for new platforms and smaller communities where annotated data is limited. We propose modeling pragmatic representations of conversations to reduce lexical noise and improve generalizability. Specifically, speech act information is used as an auxiliary learning signal alongside textual semantics. Experimental results show improved performance across three datasets, particularly in low-data and cross-domain settings.

1 Introduction

Conversational derailment forecasting seeks to anticipate hostile breakdowns for proactive moderation, but lexical variation and limited annotations hinder low-data and cross-domain performance. The paper introduces speech acts as transferable pragmatic signals and evaluates them through auxiliary learning across datasets and settings.

  • Derailment forecasting predicts hostile breakdowns before they occur, supporting proactive intervention on online platforms.
  • Existing systems often struggle with lexical shifts across communities or require large datasets to learn generalizable features.
  • Low-data forecasting is needed because annotation is expensive and new platforms or smaller communities often have limited labelled data.
  • The proposed approach models speech acts as low-dimensional pragmatic representations intended to mitigate lexical noise and improve generalizability.
  • Across three datasets, incorporating speech-act information improves generalizability over state-of-the-art baselines, particularly in low-data and cross-dataset settings.
  • The paper examines whether pragmatic features provide greater benefits when transferring to highly divergent datasets.
  • It introduces average temporal aggregation to evaluate dynamic forecasting through per-timestep behavior.

2 Related Work

Derailment forecasting anticipates whether civil conversations will become hostile, unlike toxicity detection, which classifies abusive utterances after they occur. Prior methods largely emphasize supervised textual modeling, while low-data and cross-domain transfer remain underexplored; speech acts offer a pragmatic alternative but create integration challenges.

  • Derailment forecasting anticipates whether an initially civil conversation will degrade into hostility, enabling intervention before discussions get out of control.
  • Prior approaches range from handcrafted linguistic features and conversational metadata to pretrained language models for supervised forecasting.
  • Existing work largely overlooks low-data and cross-domain generalizability, despite expensive annotation and limited labels in new or niche communities.
  • Speech Act Theory treats language as action, with speech-act verbs representing speaker intents and mental states.
  • Mapping text into a low-dimensional action space can reduce domain-specific lexical noise and capture latent pragmatic functions.
  • Using pragmatic properties for real-time forecasting requires balancing speech-act and textual features while avoiding inference latency from extraction.

3 Proposed Approach

The approach extracts speech-act information from dialogue and integrates it with textual semantics through hierarchical forecasting models. Speech acts supervise training as an auxiliary signal, while the two architectures differ in how directly forecasting relies on that signal.

  • The proposed pipeline extracts speech-act information from raw dialogues and uses it in derailment forecasting models.
  • The speech-act taxonomy contains 50 verbs plus an “other” label, with descriptions of meanings and, where applicable, psychological states.
  • Zero-shot inference prompts gpt-oss-120b to assign sentence-level multi-label speech acts, marking an utterance when any constituent sentence expresses an act.
  • Prompting targets intended speech acts as perceived by listeners, addressing differences between literal phrasing and underlying intention.
  • Human validation found moderate agreement across 51 granular classes and almost perfect agreement under a coarser labeling scheme.
  • Speech-act detection regularizes the model during training, so external speech-act extraction is unnecessary at inference time and adds no extraction latency.The auxiliary signal is used only during training.
  • Both models use a PLM utterance encoder, a Transformer conversation encoder, and separate speech-act and derailment classification heads.
  • Hparallel predicts from conversation-level representations alongside speech-act detection, whereas Hsequential feeds speech-act probabilities through a GRU before forecasting.

4 Experimental setup

The experiments assess dynamic derailment forecasting across diverse platforms, training-data regimes, and cross-dataset conditions. Evaluation uses temporal aggregation, robust classification metrics, learning-curve summaries, and comparisons with established forecasting models.

  • The study covers Wikipedia editorial negotiations, Reddit ChangeMyView debates, and GitHub technical discussions to evaluate cross-domain generalizability.
  • WIKI contains 4,188 paired derailed and civil conversations from Wikipedia talk pages.
  • CMV contains 19,578 paired conversations from Reddit ChangeMyView, labeled by moderator removals for rude or hostile behavior.
  • GITHUB includes 202 derailed and 696 civil dialogues from GitHub issues and pull requests, with a four-to-one civil-to-toxic sampling ratio.
  • All datasets retain responses before derailment, truncate civil training dialogues, and crop GitHub conversations to their 10 most recent utterances.
  • The models use 51 speech-act classes, with supplementary experiments testing robustness under an 18-class taxonomy.
  • Dynamic evaluation predicts at each conversation prefix, producing one probability per timestep before aggregating predictions at the conversation level.
  • Mean aggregation remains high only when derailment probabilities are early and consistently elevated, reducing sensitivity to isolated false-positive spikes.

5 Results and Discussion

Across in-domain and cross-dataset evaluations, Hparallel generally performs best, especially with limited training data, while speech-act-informed models improve transfer across domain shifts. Performance depends on update strategy and dataset divergence, with pragmatic signals most useful when domains differ substantially.

  • Aggregate Performance: Hparallel consistently achieves the best Dynamicmean AULC across datasets, with PLM and Hablation typically ranking second.Dynamicmean AULC summarizes learning-curve performance in a single metric.
  • Per-Timestep Performance: Hparallel consistently outperforms other models in Dynamicmean but ranks only first or second in Dynamicmax, favoring timestep-wise prediction updates.Dynamicmax fixes predictions upon any derailment alarm, whereas Dynamicmean reflects repeatedly updated predictions.
  • Performance Across Data Regimes: Hparallel demonstrates strong performance with 300-2500 training samples, but standard PLM marginally outperforms it at 11802 samples.The results indicate that explicit speech-act guidance is most beneficial when training resources are limited.
  • Per-Timestep Performance: All models improve as dialogues approach derailment or conversation end, while their relative ranking remains largely consistent.The analysis evaluates performance at individual dialogue timesteps rather than only after temporal aggregation.
  • Cross-Dataset Generalization: SA-informed models Hparallel and Hsequential yield superior cross-dataset performance, although GITHUB → CMV remains uniformly difficult.For this transfer, Macro-F1 peaks at only 36.19, while Hablation leads Macro-F1 and Hparallel leads AUPRC.
  • Cross-Dataset Generalization: CRAFT has the lowest transferability among baselines, while its lightweight, domain-specific pretraining still yields comparable performance in some settings.CRAFT is unavailable for GITHUB because its required pretraining corpus is missing.
  • Dataset Deviation and Generalization: Pragmatic features provide the greatest transfer benefit for highly divergent datasets, while GITHUB has the highest overall semantic and lexical deviation.GITHUB is marginally more similar to WIKI than to CMV despite being the most divergent corpus overall.
  • Dataset Deviation and Generalization: When transferring from WIKI or CMV to GITHUB, Hsequential ranks first and Hparallel second; training on GITHUB produces uniformly lower transferability.Hparallel trained on GITHUB with 300-500 samples reaches 66.56% AUPRC AULC on WIKI, close to WIKI’s 66.94% best in-domain result.

6 Conclusion

The paper models speech acts as pragmatic signals alongside textual semantics to improve derailment forecasting in low-data and cross-domain settings. Across three datasets, the proposed architectures show consistent improvements over strong baselines, supporting more robust dynamic moderation in data-scarce environments.

  • The paper models speech acts as a pragmatic signal alongside textual semantics for low-data and cross-domain derailment forecasting.
  • Two architectures integrate speech-act information as an auxiliary learning signal.
  • Experiments across three datasets demonstrate consistent improvements over strong baselines.

Limitations

The paper’s limitations concern speech-act taxonomy coverage and the reliability and scope of LLM-based speech-act extraction. The expanded taxonomy addresses earlier semantic gaps, but extraction remains sensitive to linguistic and community variation.

  • LLM-based speech-act extraction may introduce biases related to culture, dialect, and community norms.
  • Indirect intents remain difficult for LLMs to recognize reliably, potentially producing sub-optimal downstream signals.
  • The original taxonomy’s restriction to root-node children leaves semantic gaps and omits nuances carried by lower-level speech-act verbs.
  • The expanded taxonomy preserves the original structure while enabling speech acts to be merged or collapsed across granularity levels.
  • The taxonomy construction combines literature-based sourcing, semantic definitions, and filtering of overlapping verbs to reduce redundancy and labeling ambiguity.

A.2 Speech Act Extraction Alignment

Speech-act extraction agreement is evaluated against manual labels using Cohen’s Kappa on a sampled set of sentences. The evaluation follows prior work while adapting the agreement statistic to a two-rater setting.

  • Cohen’s Kappa replaces Fleiss’ Kappa because the evaluation compares two raters: manual labels and LLM-extracted speech acts.
  • The evaluation manually labeled 85 sentences from 30 randomly selected responses spanning the three datasets.
  • Table 3 reports agreement scores, z-scores, and p-values for manual and LLM-extracted speech acts.

B.2 Model Implementation and Information

The implementation uses large pretrained models, auxiliary speech-act objectives, and dataset-specific training procedures. Results indicate strong cross-dataset performance for speech-act-informed models, while the pipeline has substantial extraction costs and an open optimization limitation.

  • Computational requirements: A complete CMV training run requires approximately 3 hours for Hparallel, 2 hours for Hsequential and Hablation, 1 hour for RoBERTa-large, and under 10 minutes for CRAFT.
  • Computational requirements: Speech-act extraction requires approximately 7 seconds per response and 180 GPU hours overall, motivating its use during training rather than real-time inference.
  • Model objectives: The models use Cross-Entropy for derailment forecasting and Binary Cross-Entropy or Asymmetric Loss for speech-act detection.
  • Training: Hparallel uses an intermediate speech-act phase followed by multi-task training, while Hsequential optimizes both tasks simultaneously.
  • Results: Hparallel and Hsequential achieve the highest rank in the majority of cross-dataset Dynamicmax cases, although they are not always best in-domain.
  • Limitations: Future work could optimize directly for Dynamicmax, whose single threshold-crossing prediction better matches single-warning moderation workflows.

C.2 Model Performance Across Data Regime

Speech-act-informed models are especially effective when training data is scarce and can improve transfer across divergent datasets. Hparallel is strong in low-data settings, while Hsequential can be more robust against source-domain overfitting.

  • Low-data performance: With just 300 samples, Hparallel achieves higher performance than other models reach with 500 or 1000 samples.This reflects a smaller performance decline as training data decreases.
  • Low-data performance: Hparallel performs strongly in low-data regimes across WIKI, GITHUB, and CMV, although text-based models can outperform it with more data.
  • Cross-dataset performance: Speech-act signals generally provide greater benefits in cross-dataset settings when training data is limited.
  • Cross-dataset performance: Hsequential is less prone to source-domain overfitting and demonstrates robust generalizability to GITHUB.
  • Evaluation: The learning curves in Figures 5 and 6 show Dynamicmean AUPRC performance on WIKI and GITHUB across training subset sizes.

C.5 Robustness to Model Architecture: GraphNLI

A supplementary GraphNLI experiment tests whether speech-act supervision remains effective beyond the proposed hierarchical architecture. The results consistently support gains in both in-domain and cross-domain forecasting, suggesting compatibility with alternative backbone models.

  • Experimental setup: The GraphNLI experiment evaluates robustness to model architecture using the original graph-based context-aware implementation with task-specific adjustments.Training epochs are increased for smaller training sets, following the schedule used for PLM/H-ablation training.
  • Experimental setup: GraphNLI-SA adds a speech-act classification head and uses the same auxiliary objective and two-phased training procedure as Hparallel.GraphNLI uses distilroberta-base, while its full PLM is trained during intermediate training to learn speech-act classification.
  • Results: SA integration consistently improves GraphNLI performance on both in-domain and cross-domain tasks.This result is reported in the GraphNLI ablation study and is consistent with the paper’s previous findings.
  • Interpretation: Consistent gains across GraphNLI and the hierarchical framework suggest that improvements arise from additional SA supervision rather than a specific hierarchical architecture.The findings indicate that the approach is compatible with alternative backbone models.

C.6 Robustness to SA Extraction

The paper tests whether forecasting remains robust when speech-act extraction uses a coarser taxonomy rather than the main fine-grained scheme. Under the 300-training-sample setting, in-domain results remain highly comparable, while cross-dataset AUPRC is relatively consistent and Macro-F1 varies more.

  • Taxonomy design: The fine-grained 51-act taxonomy is hierarchically merged into 17 speech-act classes plus an “other” label.The merged classes follow the taxonomy of Compagno et al. (2018), with categories organized by shared taxonomy nodes.
  • Experimental setup: The coarse-taxonomy robustness experiment uses 300 training samples to focus on low-data performance across datasets.The setting reflects possible variation in speech-act distributions and interaction patterns between datasets.
  • Results: Hparallel remains robust to the specific speech-act extraction used, with highly comparable in-domain performance.The largest in-domain discrepancy is 0.56% AUPRC and 1.59% Macro-F1 on GITHUB.
  • Results: Cross-dataset AUPRC shows similar consistency, with a maximum deviation of 2.83% for CMV → GITHUB.The lower-performing model still outperforms the best baseline by 7.2% in that comparison.
  • Results: Cross-dataset Macro-F1 is less stable, reaching a maximum difference of 6.91% for GITHUB → CMV.The passage suggests this variance could be driven by reliance on source-tuned classification thresholds.
  • Additional analyses: The supplementary figures track AUPRC over dialogue progression for CMV, WIKI, and GITHUB under varying training-data sizes.Each figure reports curves averaged across five random seeds and identifies Hparallel as strongest across time steps, especially in low-data settings.
Loading 2608.25359v1…