Improving Synthetic Data Generation for Argument Mining via Adversarial Reinforcement Learning

1 Introduction

• Scarce, costly AM annotation limits model development, while synthetic samples must be linguistically natural, structurally valid, and diverse.Existing AM benchmarks are typically small; the AAEC training set has 289 essays.Source: §1 · p. 1
• The framework jointly trains a generator and discriminator, using discriminator feedback and rule-based constraints as generator signals.The generator uses GRPO and is encouraged to produce samples that are realistic and difficult for the discriminator to distinguish.Source: §1 · p. 2
• Across three AM datasets, Qwen3-8B-generated augmentation improves strong baselines in full-data and low-resource settings, with gains surpassing GPT-4o-generated data.Further analysis attributes the gains to improved linguistic naturalness and greater structural diversity.Source: §1 · p. 2

2.1 Argument Mining

• AM includes component segmentation and classification and relation identification; staged approaches can propagate errors, motivating joint and end-to-end modeling.AM has also been formulated as seq2seq generation, but annotated-corpus scarcity remains tied to costly annotation.Source: §2.1 · p. 2

2.2 Synthetic Data Generation

• Structured-output synthesis must preserve both text quality and label correctness, motivating task-specific constraints and structure-preserving mechanisms.Source: §2.2 · p. 2
• GPT-4o-based AM synthesis relies on costly prompting, while existing RL methods target general generation rather than consistency between argument text and annotations.This work explores reinforcement learning specifically for AM data synthesis.Source: §2.2 · p. 3

3.1 Overview

• Generator optimization and discriminator refinement alternate over several rounds, allowing both models to improve through adversarial interaction.Source: §3.1 · p. 3

3.2 Generator Optimization Stage

• The generator is initialized by supervised fine-tuning to produce complete text–annotation pairs that follow the AM schema while differing from the source sample.Source: §3.2 · p. 3
• Discriminator feedback rewards hard-to-identify samples, while a ROUGE-L source-similarity penalty discourages near-copies; GRPO increases the probability of higher-reward outputs.The similarity penalty is inactive below a threshold and bounded between −1 and 0; the objective encourages realistic, diverse, structurally valid samples.Source: §3.2 · p. 4

3.3 Discriminator Refinement Stage

• The discriminator classifies a shuffled set containing one annotated sample and K synthetic samples, using the annotated sample’s position as supervision.Its training is intended to improve detection of textual and structural quality issues.Source: §3.3 · p. 4
• After each generator update, newly generated challenging samples are added to discriminator training to refine its ability to detect subtle inconsistencies and annotation errors.Source: §3.3 · p. 4

3.4 Adversarial Evolution

• Each evolution round uses discriminator feedback plus the source-similarity penalty to update the generator, then uses its new samples to refine the discriminator.The alternating updates progressively evolve both models.Source: §3.4 · pp. 4–5

4.1 Datasets

• Experiments use AbstRCT, AAEC, and CDCP with either 100% or 5% of the original training data.Results are averaged over five random seeds.Source: §4.1 · p. 5, §4.2 · p. 5

4.2 Implementation Details

• Evaluation uses F1span, F1aci, and F1ari to measure component-span matching, component-type classification, and relation mapping, respectively.The experiments use Qwen3-8B as the generator and discriminator, with three adversarial rounds and K=3.Source: §4.2 · p. 5

4.3 AM Models / 4.4 Compared Methods

• The study tests parsing-based ST, generative UniASA, and Directly-SFT, comparing augmentation against EDA, GPT-4o-based methods, QOS/DOS, and SFTSyn.QOS+DOS combines 25% QOS with 75% DOS; Qwen3-8B is used for Directly-SFT and the LLM comparisons.Source: §4.3 · p. 5, §4.4 · p. 5

4.5 Main Results

• The framework improves all three AM models across datasets and data settings and generally outperforms other synthesis methods, including QOS+DOS.Table 1 reports statistically significant improvements on Avg. (p < 0.05); under 5% data, gains over QOS+DOS are reported across all three datasets.Source: §4.5 · pp. 5–7, Table 1 · p. 6

4.6 Ablation Study

• Removing discriminator refinement or the similarity penalty degrades performance, consistent with their roles in improving reward signals and discouraging repetitive generations.The authors attribute the first drop to weaker discrimination of later, more challenging samples.Source: §4.6 · p. 7

4.7 Analysis and Discussion

• More evolution rounds improve average performance but eventually plateau, while increasing K improves all metrics; gains generally shrink as more original training data is available.The authors set r=3 to balance performance and training cost; F1ari on CDCP is an exception to the data-setting trend, which they attribute to sparse relation labels.Source: §4.7 · pp. 7–8, Figure 2 · p. 7, Figure 3 · p. 8
• Synthetic samples have a broader, more dispersed structural distribution, and the method outperforms POPri and SEAL on all three datasets in the 5% setting.The distribution analysis uses graph embeddings and t-SNE based on argument structure, not textual semantics.Source: §4.7 · pp. 8–9, Figure 4 · p. 8, Table 4 · p. 9
• Adding a structure-aware reward improves overall performance on AbstRCT and AAEC but slightly reduces it on CDCP.The authors describe structural supervision as complementary to adversarial feedback, with effects that depend on dataset characteristics.Source: §4.7 · p. 9, Table 5 · p. 9

• The framework targets the scarcity of high-quality data in Argument Mining.Source: §5 · p. 9
• A generator produces text-annotation pairs, and a discriminator supplies quality feedback to guide iterative refinement.Source: §5 · p. 9
• Experiments on three datasets show that the framework significantly improves existing AM models’ performance and effectively alleviates data scarcity.Source: §5 · p. 9

• Ensuring generated-data quality requires multiple optimization rounds for both models, adding computational overhead.Source: Limitations · p. 10
• Effectiveness depends partly on the generator and discriminator’s inherent capabilities; unstable long-text generation can affect final performance.Source: Limitations · p. 10
• The method still relies on existing high-quality human-annotated data; generating such data from scratch remains future work.Source: Limitations · p. 10

Ethics Statement

• The paper filters generated samples using format and structural validity checks, but recommends human review for practical or high-risk uses because automatic checks cannot guarantee semantic correctness.Source: Ethics Statement · p. 10

A Task Formulation

• End-to-end AM predicts a document’s argument-component spans, their type labels, and the relations between components.Source: A Task Formulation · p. 12

B Algorithm

• The section introduces pseudo-code for the proposed framework.Source: B Algorithm · p. 12

• The experiments use AbstRCT, AAEC, and CDCP, English-language corpora drawn from biomedical abstracts, persuasive essays, and public comments, respectively.Source: §C.1 · pp. 12–13
• AbstRCT contains 500 abstracts with 3,279 components and 2,060 relations, using three component types and three relation types.Source: §C.1 · p. 13
• AAEC contains 402 essays with 6,089 components and 3,832 relations; this work maps Claim:For and Claim:Against to Claim because LLMs cannot distinguish their subtle difference.Source: §C.1 · p. 13
• The 5% and 100% settings differ only in training-split size; validation and test sets remain unchanged.Source: Table 6 · p. 13
• CDCP contains 731 comments with 4,779 components and 1,353 relations, with five component types and two relation types.Source: §C.1 · p. 13

• The comparison includes EDA synonym replacement, FTGA generation followed by automatic annotation, and JTLS joint generation of text and AM annotations.Source: §C.2 · p. 13
• QOS paraphrases context and components while preserving the original argument structure, whereas DOS targets new samples with different topics, expressions, and argumentation patterns.Source: §C.2 · pp. 13–14
• Figure 5 presents RST tree-depth distributions for different argument data under the full-data setting.Source: Figure 5 · p. 14
• SFTSyn fine-tunes an LLM to map original labeled AM samples to synthetic labeled samples.Source: §C.2 · p. 14
• POPri ranks candidate samples to form preference pairs and optimizes its generator with DPO, while SEAL refines self-edits using a proxy reward based on structural validity, similarity, and novelty.For AM, both methods are adapted to generate complete text–annotation pairs.Source: §C.2 · p. 14
• POPri and SEAL use Qwen3-8B, the same generator SFT initialization, data splits, and synthetic-data budget as the proposed method.Source: §C.2 · p. 14

D.1 Robustness Analysis under Different Model Configurations

• With Qwen3-4B as generator and Qwen3-8B as discriminator, the framework is comparable to or better than QOS+DOS on most metrics; reducing generator size to Qwen3-1.7B causes a noticeable drop.Source: §D.1 · p. 15
• Replacing the discriminator with a smaller model causes only marginal degradation when the generator is unchanged, while Longformer-Base-4096 as discriminator reduces overall performance across all three datasets.Source: §D.1 · p. 15

D.2 Discourse Diversity Analysis of Synthetic Data

• The discourse analysis parses generated texts into RST trees and compares tree-depth distributions to characterize discourse organization and rhetorical relations.Source: §D.2 · p. 15
• Across all three datasets, framework-generated data have deeper RST trees, while QOS’s depth distribution is relatively close to the original data.Source: §D.2 · p. 15

D.3 Adversarial Training Stability and Mode Collapse Analysis

• Discriminator accuracy drops after generator updates and recovers after refinement, while generator dynamics remain stable; no obvious mode collapse appears in the examined rounds.The authors attribute the repeated accuracy drops to newly generated samples posing fresh challenges; similarity penalties further suppress excessive reuse of original samples.Source: §D.3 · pp. 15–16, Table 8 · p. 16

D.4 Computational Efficiency Analysis

• The method uses more training time and GPU memory than SFTSyn and the Longformer-discriminator variant, but achieves better downstream performance on all three datasets.Generated synthetic data can be reused across AM models without additional downstream inference overhead.Source: §D.4 · p. 16, Table 9 · p. 16

D.5 Case Study

• The case study’s synthetic sample remains topic-consistent but changes content organization, argument components, and logical relations.The analysis considers text fluency, span alignment, component labels, and whether relations match textual logic and semantics.Source: §D.5 · pp. 16–17, Figure 6 · p. 18

E Prompt

• The paper supplies prompts for generator initialization and discriminator initialization to support clarity and reproducibility.These prompts appear in Figures 7 and 8, respectively.Source: E Prompt · p. 17, Figure 7 · p. 19, Figure 8 · p. 20