PaperlayerView exact revision on arXiv

Source-linked AI summary

Argument Structure Prediction in Online Conversations: A Comparative Study of Modeling Paradigms and Task Architectures

Siddharth Bhargava, Sara Tonelli, Patricia Martín-Rodilla, Javier Parapar

arXiv:2609.39225v1cs.CLcs.IR

TL;DR

Dialogical ASP is challenging because noisy, implicit interactions complicate formalization and prediction. The paper adapts IAT corpora into bipolar structures and compares fine-tuned and prompted models across architectures. Relation identification remains the principal bottleneck, particularly for implicit or context-dependent content.

  • Problem

    Noisy interactions, implicit argumentative content, and complex discourse dynamics make formal modeling and automated ASP especially challenging in dialogical settings.

    Source: §1 · p. 1

  • Method

    The study adapts IAT-annotated corpora into bipolar structures, compares fine-tuned and prompt-based ASP configurations, and releases its data-processing and modeling pipelines.

    Source: §1 · p. 2, §3 · p. 3, §7 · p. 10

  • Results

    Identifying whether argument units are linked remains ASP’s key bottleneck, amplified by implicit or context-dependent content that requires broader discourse context and deeper reasoning.

    Source: §7 · p. 10

  • Takeaways & Limitations

    The findings indicate a need for explicit contextual grounding when designing ASP systems for complex dialogical interactions.

    Source: §7 · p. 10

    Evaluated approaches generalize poorly across distinct dialogical and argumentative characteristics, and some datasets may have appeared in LLM pretraining corpora.

    Source: §7 · p. 10

Abstract

from arXiv · show

Argument structure prediction (ASP) constructs complete argument structures from discourse by identifying argumentative units and their relations. While recent work has explored diverse approaches---including unified neural models, multi-step pipelines, and prompt-based large language models (LLMs)---their relative trade-offs remain under-explored, particularly in dialogical settings. We present a systematic evaluation of ASP under strict schema constraints, comparing supervised fine-tuning and prompt-based LLMs across single- and multi-step task architectures, generating complete argument structures from dialogical input end-to-end. We benchmark them on three diverse dialogical corpora adapted from Inference Anchoring Theory into bipolar argument structures. Under a shared evaluation framework, we assess predictive performance, cross-domain generalization, schema compliance, and computational efficiency. Our results show that ASP remains a challenging task, with identifying argumentative relations emerging as the primary bottleneck, largely due to the implicit and context-dependent nature of dialogical argumentation. To facilitate future research, we release our data processing pipeline and end-to-end modeling framework for computational ASP on dialogical corpora.

1. Introduction

Argument structure prediction (ASP) represents argumentative units and their support or attack relations, but dialogical settings intensify its formalization and prediction challenges. This study compares modeling paradigms and task architectures under a shared schema and evaluation framework.

1. Introduction

  • Noisy interactions, implicit content, and complex discourse dynamics make ASP’s theory-driven formalization and data-driven prediction especially challenging in dialogical settings.Developing systems also requires formal representations grounded in argumentation theory and discourse characteristics, alongside annotated corpora and supervised learning.Source: §1 · p. 1
  • The study compares single-step and multi-step ASP architectures alongside supervised fine-tuning and prompting with self-refinement.The configurations share an argument schema and enable assessment of trade-offs and feasibility for large-scale dialogical argumentation.Source: §1 · p. 2
  • Three AIFdb dialogical corpora annotated with Inference Anchoring Theory are converted from inferential and conflicting relations into bipolar support and attack structures.Their interaction modes, topics, and dialogue goals differ, enabling cross-domain evaluation.Source: §1 · p. 2
  • The work contributes IAT-corpus adaptation, an open-source end-to-end ASP pipeline, and a shared evaluation framework for performance, generalization, schema compliance, and efficiency.The framework compares single-step and multi-step architectures and supervised fine-tuning and prompt-based generative approaches.Source: §1 · p. 2

2. Related Work

Prior ASP work varies in modularization and modeling paradigm, but direct comparisons under shared constraints are limited. This study addresses that gap by comparing fine-tuning and prompting across single- and multi-step architectures.

2. Related Work

  • Multi-step ASP decomposes prediction into sequential subtasks, while single-step approaches jointly predict units and relations in one training framework.Multi-step methods favor interpretability and specialization; single-step methods enable shared representations and resource efficiency but face structural-validity challenges and error propagation.Source: §2 · p. 2
  • Earlier feature-engineered approaches offered interpretability but required substantial manual effort and lacked generalization; neural models learned contextual representations but depend on annotated data.Neural models also often struggle with out-of-distribution robustness.Source: §2 · pp. 2–3
  • LLM-based ASP approaches use pretrained knowledge and contextual reasoning, but may lack grounding in argumentation theory and rely on superficial data patterns.The cited work raises concerns about limited interpretability and robust reasoning.Source: §2 · p. 3
  • Prior comparisons typically treat modeling paradigm and task architecture as separate design choices; this study evaluates them jointly across diverse dialogical settings.The evaluation covers performance, cross-domain generalization, efficiency, and structural validity.Source: §2 · p. 3

3. Data

The study adapts selected IAT-annotated dialogical corpora into bipolar argument structures for computational ASP. Its data pipeline defines argument units and relations, while the modeled structures exclude rephrasing relations.

3. Data

  • The study adapts IAT-annotated AIFdb corpora into bipolar structures containing arguments, support relations, and attack relations.It selects corpora that fit a common processing framework and focuses on IAT annotations that can be adapted to this representation.Source: §3 · p. 3
  • Inferential relations map to support and conflict relations to attack; argument units are locutions linked by at least one such relation.Each unit is a verbatim span expressing a coherent argumentative intent, with a unique identifier, text, and brief role description; units must contain at least three words or form a semantically coherent span.Source: §3 · pp. 3–4
  • Relations are directed support or attack links between units, with source–target pairs chronologically ordered so the source occurs after the target.The study’s representation is compatible with the IAT framework.Source: §3 · p. 4
  • The pipeline can optionally preserve rephrasing relations, speaker, and dialogical-act information, but this study does not model rephrasing.The authors leave investigation of rephrasing relations for future work.Source: §3 · p. 4
  • The selected corpora are QT30, US2016reddit, and RIP1; QT30 is split 80:20 into training and test sets, stratified by attack-relation frequency.Fine-tuning uses cross-validation without a separate validation set.Source: §3 · p. 4
  • Conversations with fewer than three units or two relations are removed, and conversation length is capped at 1,500 tokens for computational reasons.QT30 and US2016reddit contain persuasive political discourse, while RIP1 is collaborative and non-political; US2016reddit has shorter units and a higher attack proportion than the offline corpora.Source: §3 · p. 4

4. Models

The study evaluates fine-tuned and prompted models using both single-step and multi-step architectures. Fine-tuning uses joint or separate task models, while prompted models generate schema-constrained structures in one pass or in stages.

4. Models

  • The study evaluates supervised fine-tuning and prompted LLMs under single-step and multi-step ASP architectures.The multi-step configurations produce intermediate outputs validated against the schema.Source: §4 · p. 5

4.1. Supervised fine-tuning

  • Single-step fine-tuning uses a biLSTM-ER model to jointly predict BIO-tagged argumentative spans, support or attack relations, and relational distance.The relational-distance estimate ranges from −8 to +8 sentences.Source: §4.1 · p. 5
  • Multi-step fine-tuning trains separate models for argument-unit extraction and relation-type classification.The classifier evaluates predicted unit pairs in chronological order, with a 150-token context window and negative-sample augmentation.Source: §4.1 · p. 5

4.2. Prompting with self-refinement

  • Prompting uses three open-source LLMs, with schema-constrained prompts for full-structure generation or staged unit extraction and relation prediction.Invalid outputs trigger one retry with validation errors and corrective instructions; in-context examples vary across zero-, three-, and five-shot settings.Source: §4.2 · pp. 5–6

5. Experimental Setup

The experiments compare fine-tuned and prompted models in single- and multi-step settings, using QT30 for fine-tuning and deterministic, schema-validated inference for prompted LLMs.

5. Experimental Setup

  • All experiments ran on a single 48GB NVIDIA Ampere A40 GPU.Fine-tuned models used QT30 training data, Optuna hyperparameter search, and five-fold cross-validation with early stopping.Source: §5 · p. 6
  • Fine-tuned single- and multi-step models were trained on QT30, with weighted losses addressing label imbalance across subtasks.Optuna selected learning rates across eight trials per configuration; models were retrained with five-fold cross-validation and early stopping.Source: §5 · p. 6
  • Fine-tuned predictions were post-processed differently by architecture before producing schema-compliant units and support or attack relations.Single-step processing corrected BIO spans and relational distance conflicts; multi-step processing filtered malformed spans and constructed chronologically valid unit pairs.Source: §5 · p. 6
  • Prompted LLM inference used deterministic decoding, structured JSON generation, schema validation, and at most two tries per conversation.The decoding settings included temperature 0, top_p 0.0, and random seed 42.Source: §5 · p. 6

6. Results

Prompt-based single-step configurations performed best among those tested, but generalization to unseen corpora or dialogical settings remained weak; relation pairing and context were major sources of error.

6.1. Task Performance and Generalization

  • On QT30-test, single-step models achieved stronger RTC than multi-step models while performing similarly on AUE; prompted models with more examples outperformed fine-tuned models.Zero-shot and Gemma-based configurations performed poorest; Qwen2.5-14B-Instruct performed best, while Gemma3-12B-it had poor schema compliance and malformed generations.Source: §6.1 · p. 7
  • Across US2016reddit and RIP1, RTC performance dropped more than AUE; US2016reddit performed higher than RIP1, likely reflecting its closer alignment with QT30.Single-step architectures continued to outperform multi-step models on RTC across the cross-domain corpora.Source: §6.1 · pp. 7–8
  • Prompt-based single-step configurations performed best overall, but none generalized well to unseen corpora or different dialogical settings.The reported comparison covers the tested configurations and the evaluated corpora and settings.Source: §6.1 · p. 8

6.2. Computational Efficiency and Schema Compliance

  • Prompting generally outperformed fine-tuning but incurred higher computational cost and more frequent schema violations; retries improved compliance while adding runtime and token use.Minor violations were post-processed, while major schema violations triggered one retry with error reports and corrective feedback.Source: §6.2 · p. 8
  • Multi-step settings benefited more from retries, but retries often nearly doubled token usage while providing limited overall improvements.Retries were most beneficial in low-context settings such as zero-shot, where they yielded higher gains per additional token.Source: §6.2 · p. 8

6.3. Error Analysis

  • Argument-unit errors were driven mainly by inconsistent segmentation; performance fell sharply at exact overlap, highlighting difficulty identifying dialogical span boundaries.At 50% partial overlap, unit alignment with gold spans improved and recall increased.Source: §6.3 · p. 9
  • Relation recall suffered when partially overlapping units were paired incorrectly, and models often missed relations or failed to resolve implicit and long-distance links.Precision remained relatively stable because only correctly paired relations proceeded to evaluation; long discussions often produced predictions between immediate neighbors instead.Source: §6.3 · p. 9
  • Among correctly paired relations, most configurations classified support well and attack near-perfectly despite class imbalance, making relation pairing the primary bottleneck.The authors suggest separating explicit source-target linking from subsequent relation-type classification.Source: §6.3 · pp. 9–10

7. Conclusion

The paper evaluates dialogical ASP under a shared schema and releases its data-processing and modeling pipelines. It finds relation identification to be the main bottleneck, especially when argumentative content is implicit or context-dependent.

7. Conclusion

  • The study adapts IAT-annotated corpora into bipolar argument structures and compares supervised fine-tuning and prompt-based LLMs across single- and multi-step architectures.Its shared-schema evaluation examines performance, generalization, schema compliance, and computational efficiency; the data-processing and modeling pipelines are released.Source: §7 · p. 10
  • Relation identification remains the main ASP bottleneck, particularly for implicit or context-dependent dialogical content.The paper says accurate inference requires broader discourse context and deeper reasoning.Source: §7 · p. 10
  • The evaluated approaches show limited cross-domain generalization, with performance degrading on datasets with distinct dialogical and argumentative characteristics.The authors also note that some datasets may have appeared in the evaluated LLMs’ pre-training corpora.Source: §7 · p. 10
  • The authors identify source–target relation prediction as a key challenge and suggest retrieval-augmented context or structural knowledge as future directions.They also call for broader modeling strategies and more diverse dialogical corpora.Source: §7 · p. 10
  • Robust dialogical ASP could support conflict resolution, collaborative reasoning and decision-making, and argument-based summarization and explanations.Source: §7 · p. 10

Declaration on Generative AI

The authors report using ChatGPT for language editing and state that they reviewed the content and take responsibility for it.

Declaration on Generative AI

  • The authors used ChatGPT for grammar, formal tone, and spelling checks, then reviewed and edited the content and take responsibility for it.Source: Declaration on Generative AI · p. 10

A. Online Resources

The paper lists online repositories for its system, data-processing pipeline, and IAT2BAS dataset.

A. Online Resources

B. Prompts used for ASP

The paper provides templates for its three ASP prompts in Figures 3–5 and directs readers to GitHub for details on their use.

B. Prompts used for ASP

  • Templates for the three ASP prompts appear in Figures 3, 4, and 5, with usage details available in the GitHub repository.Source: B. Prompts used for ASP · p. 14

C. Quantitative Error Analysis Details

The error analysis finds that LLMs tend to miss relations, while unit-prediction patterns vary across corpora. It distinguishes unit errors from relation errors, including endpoint mismatches linked to unit alignment.

C. Quantitative Error Analysis Details

  • The analysis distinguishes false-positive units from missed ground-truth units, which are often more implicit and require human-level reasoning and contextual understanding.Table 5 reports six structural error types across datasets and configurations at a 50% overlap threshold, using seed 42 and best configurations.Source: C. Quantitative Error Analysis Details · p. 14, Table 5 · p. 18
  • Relation errors comprise endpoint mismatches, false-positive and false-negative links between true-positive units, and correctly identified links with an incorrect support or attack label.Endpoint mismatches arise when a relation is linked to a false-positive or false-negative unit.Source: C. Quantitative Error Analysis Details · p. 14, Figure 6 · p. 18
  • For QT30 and RIP1, LLMs tend to overpredict units, whereas US2016reddit shows the reverse pattern, potentially reflecting more argumentative, implicit conversation.The QT30 and RIP1 pattern includes higher false-positive and relatively lower false-negative unit counts.Source: C. Quantitative Error Analysis Details · pp. 14–15
  • Across all three corpora, LLMs predict significantly more false-negative than false-positive relations; endpoint mismatches also depend on unit errors.The authors report that superior unit alignment allows fewer endpoint-mismatched relations.Source: C. Quantitative Error Analysis Details · pp. 15–16
Loading 2609.39225v1…