PaperlayerView exact revision on arXiv

Source-linked AI summary

Do Social Patterns Hold in Synthetic Data? Analyzing Cyberbullying Dynamics in LLM-Generated and Authentic Dialogues

Arefeh Kazemi, Hamza Qadeer, Sinan Asci, Joachim Wagner, Brian Davis

arXiv:2609.17549v1cs.CLcs.CY

TL;DR

The paper asks whether LLM-generated cyberbullying conversations reproduce authentic social dynamics rather than only supporting downstream tasks. It compares authentic and synthetic dialogues with a multidimensional framework and human evaluation, finding preserved high-level structure but model-dependent distortions in finer-grained behavior.

  • Problem

    It is unclear whether synthetic cyberbullying conversations faithfully reproduce authentic social dynamics beyond supporting downstream task performance.

  • Method

    The study compares authentic and GPT-, Grok-, and LLaMA-generated dialogues across interactional, linguistic, affective, behavioral, temporal, and human-evaluated dimensions.

  • Results

    Synthetic data preserves high-level role participation, directional power asymmetry, and broad behavioral-marker distributions, but distorts finer-grained behavior and temporal dynamics in model-dependent ways.

  • Takeaways & Limitations

    Synthetic CB data is useful for modeling global interactional structure but remains an imperfect proxy when behavioral realism and social dynamics are essential.

  • Takeaways & Limitations

    Human evaluation shows that structural similarity can coexist with limited perceived realism, including outputs that lack recognizable cyberbullying interactions.

Abstract

from arXiv · show

Cyberbullying (CB) is a complex social phenomenon characterized by repeated aggression, power imbalance, and multi-party interaction. Although large language models (LLMs) are increasingly used to generate synthetic CB conversations for data augmentation and benchmarking, it remains unclear whether such data faithfully reproduces the social dynamics of authentic interactions beyond supporting downstream task performance. We present a comprehensive framework for evaluating the social realism of LLM-generated CB conversations. We compare authentic and synthetic dialogues generated by GPT, Grok, and LLaMA across interactional structure (turn-taking, power dynamics, and repair behavior), linguistic and stylistic realism (pronoun usage and humor), affective and behavioral markers (CB types, profanity, and toxicity), and temporal escalation dynamics. We further complement automatic analyses with a human evaluation of cyberbullying presence, scenario relevance, role plausibility, and social realism. Our results show that LLM-generated data consistently preserves high-level interactional structure, including role participation patterns, directional power asymmetry, and broad distributions of behavioral markers. However, all models systematically distort finer-grained social phenomena, including behavioral magnitude, role-specific allocation, categorical distributions, and temporal dynamics. These distortions are strongly model-dependent: GPT suppresses harmful content, Grok amplifies aggressive behaviors, and LLaMA provides the most balanced approximation while smoothing role distinctions. Our findings show that synthetic CB data is useful for modeling global interactional structure but remains an imperfect substitute for authentic conversations when behavioral realism and social dynamics are essential.

1 Introduction

Synthetic CB data is increasingly attractive because it can scale NLP development where authentic data are costly and ethically difficult to collect. This paper therefore evaluates whether generated dialogues preserve social patterns beyond task performance.

  • Synthetic data can support augmentation, few-shot learning, and dataset generation, especially where labeled data are scarce or expensive.The introduction frames synthetic data as a scalable and flexible alternative that can sometimes approach human-annotated-data performance.
  • Authentic cyberbullying data are difficult to collect because conversations involving minors raise ethical and legal constraints, while annotation is costly and may expose annotators to distressing material.
  • The study asks whether LLM-generated CB dialogues preserve authentic social patterns rather than merely supporting downstream task performance.
  • Its framework examines interactional structure, linguistic and stylistic realism, and affective and behavioral markers.The proposed dimensions include turn-taking, power dynamics, repair behavior, pronoun usage, hedging, emphatic devices, CB types, toxicity, and profanity.

2 Related Work

Related work moves from message-level CB detection and task-centric LLM evaluation toward socially grounded analysis of conversational realism. This study extends that direction by comparing authentic and synthetic role-based CB dialogues across multiple behavioral dimensions.

  • Earlier CB detection commonly treated messages independently, which limited its ability to capture context-dependent and interactional behavior.
  • LLMs have been used for synthetic data creation in augmentation, knowledge distillation, self-training, and few-shot learning.
  • SynBullying generates multi-turn, role-based CB conversations, providing a structured alternative to isolated-message datasets.
  • LLM evaluation has broadened beyond accuracy toward holistic assessment of robustness, fairness, calibration, efficiency, and behavior-aware performance.
  • Despite this shift, limited work has tested whether LLM-generated data preserves social and interactional realism in complex domains such as cyberbullying.

3 Data

The study compares authentic and synthetic role-based CB conversations built around shared scenarios and participant roles. The synthetic data use three LLMs and preserve ordered, role-labeled multi-party exchanges for analysis.

  • The authentic comparison data come from an English CB dataset built through controlled teen role-play sessions with Victim, Bully, Bully Supporter, and Victim Supporter roles.
  • The annotation scheme covers threats, insults, discrimination, harassment, exclusion, defamation, sexual talk, encouragement to harassment, and other categories.
  • The synthetic counterpart, SynBullying, contains role-based dialogues generated with GPT-4o, Llama-3.3-70B-Instruct, and Grok-2.
  • The study annotates harmful messages with CB types and excludes the Defense category from its scheme.
  • Synthetic prompts incorporate the same role-play scenarios, producing ordered messages explicitly associated with participant roles.
  • Table 1 summarizes conversation and message distributions across authentic WhatsApp and synthetic datasets by scenario.

4 A Metric-Based Framework for Analyzing Social Dynamics in CB Dialogues

The framework operationalizes the question of social-pattern preservation through complementary metrics spanning interactional structure, linguistic and stylistic realism, and affective and behavioral signals. Each metric is paired with a definition, motivation, and computation procedure.

  • The framework measures interactional structure, linguistic and stylistic realism, and affective and behavioral signals in CB conversations.
  • The framework is designed to answer whether social patterns hold in synthetic CB data.
  • Each metric includes a definition, theoretical motivation, justification, and precise computation procedure.

4.1 Interactional Structure

The framework evaluates interactional structure through role-based participation, power-related speech acts, and repair-related language. These measures capture how conversational control, dominance, conflict mitigation, and repair are distributed across roles.

  • 4.1.1 Turn-taking Patterns: Turn-taking is operationalized as each role’s share of labeled messages, yielding a normalized participation distribution.For each conversation, the role-specific message count is divided by the total number of messages.
  • 4.1.2 Power Dynamics: Power dynamics are measured through the proportion of each role’s messages labeled as commands or accusations.These speech acts represent linguistic attempts to control, intimidate, or socially position others.
  • 4.1.3 Apology and Defensive Language: Repair behavior is measured through role-specific proportions of apology and defensive acts that can mitigate or respond to hostility.The operationalization counts messages containing apology or defensive acts for each role.

4.2 Linguistic and Stylistic Realism

Linguistic and stylistic realism is assessed through role-conditioned pronoun usage and humor frequency. These measures target perspective-taking, interpersonal orientation, mocking, sarcasm, and social alignment.

  • 4.2.1 Pronoun Usage: Pronoun distributions distinguish role-specific perspective and orientation, with second-person forms linked to targeting and first-person forms to self-reference or defense.Relative frequencies are computed at the token level for first- and second-person pronouns.
  • 4.2.2 Humor: Humor frequency is measured by counting messages classified as humorous for each participant role.The paper treats humor as a social strategy that may express mocking, sarcasm, or group bonding among aggressors.

4.3 Affective and Behavioral Markers

Affective and behavioral realism is evaluated through cyberbullying categories, profanity, toxicity, and toxicity trajectories over normalized conversation time. Together, these metrics assess both harmful-content composition and temporal escalation patterns.

  • 4.3.1 CB Types: CB-type distributions measure the frequency of harmful categories such as insults, threats, and defamation across datasets.The categories use annotated CB-type labels from the original SynBullying dataset.
  • 4.3.2 Profanity: Profanity is detected with a curated pattern-matching lexicon that handles censored and elongated forms, then normalized per 100 tokens.Custom tokenization preserves variants such as “f*k” and “sh*t.”
  • 4.3.3 Toxicity: Toxicity is assigned at the message level by ToxicBERT and averaged within each role.The classifier produces scores in the interval [0, 1].
  • 4.3.4 Escalation Dynamics: Escalation dynamics are modeled by binning normalized message positions and computing mean toxicity for each dataset and temporal bin.The resulting trajectory enables comparisons of toxicity patterns over normalized conversation time.
  • 4.3 Affective and Behavioral Markers: The framework combines CB-type distributions, profanity rates, toxicity scores, and temporal toxicity trajectories to evaluate behavioral realism.This multi-level operationalization tests whether synthetic data preserves deeper interactional and behavioral patterns beyond surface features.

5 Results

Synthetic dialogues broadly preserve role participation and directional power asymmetry, but distort the magnitude and role allocation of power and repair behaviors relative to authentic conversations.

  • Turn-taking Patterns: Grok most closely matches authentic role-message contributions, followed by LLaMA, while GPT shows larger but still modest deviations.Grok records Cramér’s V = 0.042 and JSD = 0.0010; LLaMA records 0.061 and 0.0020; GPT records 0.155 and 0.0140.
  • Power Dynamics: Bullies consistently produce more power-related expressions than victims, preserving directional asymmetry across authentic and synthetic datasets.In authentic data, bullies produce approximately twice the victim rate: 9.05% versus 4.84%.
  • Power Dynamics: Synthetic data preserves overall power imbalance but changes its magnitude and role-specific allocation, especially among supporting participants.GPT and Grok widen bully–victim gaps, LLaMA narrows them, and authentic victim supporters’ leading participation is only partially reproduced.
  • Apology and Defensive Language: Authentic repair language is concentrated among victims and victim supporters, whereas synthetic datasets only partially preserve this role asymmetry and distort its magnitude.Authentic defensive-language rates are 49.57% for victims and 24.68% for victim supporters, versus 1.43% and 2.08% for bully and bully-support roles.

5.2 Linguistic and Stylistic Realism

LLMs reproduce broad role-linked linguistic and stylistic patterns, but model-specific deviations weaken the fidelity of pronoun usage and humor distributions.

  • Pronoun Usage: Role-conditioned pronoun usage is broadly captured, with bullies using more second-person pronouns and victims more first-person pronouns, though model consistency varies.Grok shows the largest divergence, with V = 0.157 and JSD = 0.0389.
  • Humor: Authentic humor is concentrated in bully-aligned roles and nearly absent from victims, a pattern only partially preserved by synthetic datasets.Authentic humor rates are 13.33% for bullies, 16.32% for bully supporters, 5.41% for victim supporters, and 0.57% for victims.

5.3 Affective and Behavioral Markers

Synthetic data retains broad behavioral structure, including role-linked aggression and some temporal variation, but models diverge in categorical distributions, behavioral magnitude, and escalation trajectories.

  • Cyberbullying Types: Authentic cyberbullying types are dominated by Insult_General at 41.91%, while synthetic models distort secondary and rare categories in model-dependent ways.GPT most closely matches the distribution with JSD = 0.0317; Grok diverges more substantially with JSD = 0.0739.
  • Profanity: Profanity remains concentrated in bully-aligned roles, but GPT suppresses it, Grok amplifies it, and LLaMA partially flattens role differences.GPT has the closest distributional alignment, Grok preserves role ordering despite magnitude distortion, and LLaMA shows redistribution across roles.
  • Toxicity: No single model simultaneously reproduces the authentic structure and scale of toxicity across roles.Each model captures different aspects of the behavioral pattern rather than all properties together.
  • Escalation Dynamics: Authentic toxicity follows a nonmonotonic, multi-peaked pattern that synthetic models only partially reproduce.Authentic toxicity is higher in early bins, fluctuates midconversation, and decreases toward the final bins.
  • Escalation Dynamics: LLaMA best matches toxicity magnitude and global temporal trend, while GPT best follows trajectory variation and Grok matches normalized distribution but reverses trajectory direction.LLaMA has L2 distance 0.2360, correlation r = 0.4934, and slope difference -0.0003; GPT has r = 0.5456; Grok has JSD = 0.0067 but r = -0.3294.

6 Human Evaluation

The human evaluation tests whether authentic and synthetic CB snippets are recognizable, scenario-relevant, role-plausible, and socially realistic. Synthetic datasets generally preserve scenario coherence, while model-specific differences emerge in CB presence, role plausibility, and social realism.

  • Evaluation Design: The evaluation uses 33 conversations divided into 99 blinded, role-labeled snippets sampled from early, middle, and late stages.The set contains 9 authentic and 24 synthetic conversations, with two synthetic conversations per scenario and model.
  • Inter-Annotator Agreement: Three annotators independently assess each snippet, with agreement interpreted using Fleiss’ κ and prevalence-adjusted PABAK-OS.PABAK-OS values range from 0.787 to 0.917, indicating substantial to near-perfect agreement despite low Likert-scale κ values.
  • Human Evaluation Results: Scenario relevance is high across all datasets, ranging from 4.57 for GPT to 4.94 for Grok.LLaMA scores 4.74 and authentic data 4.73 on the five-point Likert scale.
  • Human Evaluation Results: Grok receives the highest role-plausibility score at 4.71, while GPT scores lowest at 3.94 and authentic data scores 4.51.LLaMA scores 4.40, suggesting weaker role behavior for GPT than for the other evaluated datasets.
  • Human Evaluation Results: LLaMA most closely matches authentic social realism, scoring 4.36 versus 4.38 for authentic conversations, while GPT and Grok score 4.11 and 4.04.The lower scores for GPT and Grok are associated with suppression and amplification effects, respectively.

7 Discussion

Synthetic CB dialogues preserve coarse interactional structure but distort behavioral magnitude, role allocation, categorical distributions, and temporal dynamics. These distortions vary by model, making synthetic data useful for global structure but unreliable as a faithful substitute for nuanced social behavior.

  • Preservation of Social Structure: LLMs reproduce role-based turn-taking, directional bully–victim power asymmetry, and broad behavioral-marker ordering, supporting preservation of coarse CB interaction structure.Human evaluation also identifies Grok and LLaMA as containing CB in 100% of evaluated cases.
  • Distortions in Fine-Grained Dynamics: Models systematically distort behavioral magnitude, role-specific allocation, and temporal dynamics, even when preserving who tends to do more than whom.No model simultaneously preserves the ordering, shape, and magnitude of behavioral trajectories across a conversation.
  • Model-Specific Biases: Model suitability depends on the behavioral dimension: GPT suppresses harmful content, Grok amplifies aggression, and LLaMA offers a more balanced approximation while smoothing role distinctions.No single model consistently outperforms the others across all aspects of CB interaction modeling.
  • Model-Specific Biases: Grok preserves strong role separation and CB coverage but exaggerates behavioral intensity, collapses repair acts, and overrepresents discrimination-related CB types.Its high role plausibility and 100% CB coverage therefore coexist with less natural interactional dynamics.
  • Implications for NLP Research: Synthetic CB data is useful for pretraining, augmentation, and exploratory analysis of coarse interactional structure when authentic data is scarce or sensitive.Relevant coarse patterns include role asymmetries, conversational flow, and broad harmful-versus-non-harmful behavior distributions.
  • Implications for NLP Research: Synthetic data is unreliable for fine-grained signals such as behavioral intensity, rare categories, repair, humor, and escalation, and should not be treated as a faithful proxy for real-world CB behavior.Quantitative structural similarity can coexist with divergence in perceived realism or behavioral validity.

8 Conclusion

Synthetic CB data preserves high-level interactional structure but systematically distorts finer-grained behaviors and temporal dynamics. Its usefulness therefore depends on whether an application prioritizes global structure or behavioral realism.

  • Synthetic data reliably preserves role participation, directional power asymmetry, and broad distributions of toxicity, profanity, and pronoun usage.
  • Across datasets, synthetic dialogues distort behavioral magnitude, role-specific allocation, temporal evolution, repair behavior, humor, and specific CB-type distributions.
  • GPT suppresses harmful content, Grok amplifies aggressive and identity-related behaviors, and LLaMA offers the most balanced approximation while smoothing role distinctions.
  • Synthetic CB data suits modeling global role dynamics and general interactional structure but is less reliable for behavioral intensity, rare categories, and temporally grounded dynamics.
  • Because fine-grained realism remains limited, synthetic data should not directly substitute for authentic conversations when high ecological validity is required.
  • Future work should improve temporal modeling, role-specific behavior, context-sensitive language, and evaluations combining computational and human-centered perspectives.
Loading 2609.17549v1…