Source-linked AI summary
Using LLMs to Mimic the Conversational Dynamics of Reddit Communities
Vedaant Jain, Yoshee Jain, Ishq Gupta, Aditi Shrivastava, Koustuv Saha, Eshwar Chandrasekharan
TL;DR
Online communities need scalable ways to support constructive interaction, while prior LLM simulations often lack user-specific context. This paper tests whether Reddit users’ comment histories can ground two prompting approaches for mimicking their comments. Gemini captures comment formality and structure relatively well but struggles with emotional nuance, especially by understating joy and overstating anger.
Problem
The paper addresses whether LLMs can mimic specific Reddit users’ communication patterns from partial histories, where earlier approaches lacked sufficient user-specific context.
Method
The study uses Reddit comment histories and evaluates predict and masked fill-in-the-blank prompts for generating comments conditioned on conversational context.
Results
Gemini captures formality and sentence structure relatively accurately, but emotional imitation is uneven, with lower joy and higher anger than the emulated comments.
Takeaways & Limitations
The findings support further investigation of LLM-generated responses for influencing online conversations, while identifying emotional intensity as a key area for improvement.
Takeaways & Limitations
The study calls for larger, more diverse multi-subreddit datasets and more rigorous analytical techniques to evaluate LLM performance and limitations.
Abstract
from arXiv · showhide
Online communities face a constant battle against toxicity and misinformation. While human moderators struggle to keep pace with the volume of content, LLMs offer a promising solution for automatically generating constructive responses and shaping online interactions. This paper preliminarily investigates if LLMs can mimic the communication styles of Reddit users using their comment history as context. We evaluate two prompting approaches: predicting a target comment and filling in masked comments. We find that LLMs outperform expectations at replicating comment structure and formality, but struggle to accurately capture nuanced emotions, e.g. understating joy and overstating anger. These findings highlight a promising direction for LLMs in guiding online conversations towards prosociality influencing emergent communication patterns and norms within the community. The results of our study inspire future work with more rigorous methods of evaluation to explore the LLMs' effectiveness across diverse online communities to better understand their broader societal impact.
1 INTRODUCTION AND BACKGROUND
The paper examines whether LLMs can mimic Reddit users when grounded in their prior interactions, addressing the lack of user-specific context in earlier approaches. This capability is framed as a possible aid for constructive moderation and synthetic data generation.
- Earlier LLM approaches generated content from general user descriptions or simulated conversational roles without sufficient user-specific context for communication nuances.
- The study asks whether LLMs can reproduce individual Reddit users’ semantic structure and sentiment from partial interaction histories.
- The authors hypothesize that prior user activity can help LLMs model individual characteristics and produce more accurate, nuanced simulations.
- The proposed application is automated response generation that could guide conversations toward neutrality or prosociality while augmenting human moderation.
2 METHODOLOGY
The study uses r/science data and two LLM prompting settings to generate comments conditioned on thread context, post content, and the target user’s comment history. The settings differ in whether the target comment is predicted from preceding context alone or reconstructed with surrounding comments.
- The dataset comes from the highly moderated r/science subreddit, chosen to provide a controlled setting for evaluating LLM performance.
- The data include 12,000 posts, comments, author and timestamp metadata, and hierarchical thread structure, with sampling focused on authors having at least eight prior comments.
- Experiments use Gemini 1.0-pro with temperature 1.0 across the two comment-generation settings.
- Predict Setting: The predict setting generates a target comment from ancestral comments, the original post, and selected prior comments by the target author.
- Masked Fill-in Setting: The masked fill-in setting generates a missing comment using both ancestral and successor comments, alongside post content and user history.
3 EVALUATION
The evaluation compares generated comments with Reddit ground truth across stylistic and content-based features. It represents feature values as distributions, compares their differences, and tests whether mean differences indicate systematic bias.
- Generated responses are compared with Reddit ground-truth comments using style similarity and content similarity criteria.
- Style measures formality, question-versus-statement classification, and syntactic similarity, while content measures semantic parallelism, emotion, and sentiment.
- Distribution Comparison: For each feature, the analysis defines LLM and ground-truth distributions and assumes approximate normality after visual checks of histograms and quantile-quantile plots.
- Distribution Comparison: The difference distribution is defined as X_fi^diff = X_fi^Truth − X_fi^LLM to compare generated and ground-truth feature values.
- Hypothesis Testing: t-tests assess whether the mean difference is zero or whether generated estimates are systematically lower or higher than ground truth.
- Feature-wise accuracy is summarized with grouped averages across features and filtered averages restricted to features significantly present in ground-truth comments.
4 RESULTS
Gemini replicated comment structure and formality relatively well across both prompting tasks, but its emotional imitation was uneven, particularly for joy and anger. The results indicate stronger stylistic than emotional fidelity.
- Emotion: 0.817 grouped average accuracy and 0.593 filtered accuracy show that masked fill-in prompts replicated significant emotions with moderate success.The filtered average considers only emotions classified as significantly present in the ground-truth comment.
- Emotion: 0.865 Sadness, 0.991 Love, 0.873 Fear, and 0.982 Surprise were among the more accurately replicated emotions for masked fill-in prompts.
- Emotion: Regardless of prompt style, Gemini tended to understate joy and overstate anger relative to the emulated comments.The same directional pattern appeared for both masked fill-in and predict prompts.
- Formality: 0.743 filtered-average accuracy for fill and 0.741 for predict indicate relatively accurate formality imitation, despite a shift toward more formal language.The authors note that this formality shift could make generated comments recognizable as LLM-produced in generally informal Reddit settings.
- Sentence structure: 0.912 filtered-average accuracy for fill-in-the-blank and 0.909 for predict indicate strong sentence-structure imitation, although outputs skewed toward statements rather than questions.
- Overall interpretation: The authors conclude that Gemini’s structural and formal accuracy does not compensate for its inconsistent emotional intensity when comments are intended to fit r/Science conversations.In particular, more anger and less joy may hinder steering discussion in the intended direction.
5 CONCLUSION
The preliminary analysis shows that Gemini can reproduce stylistic features of Reddit communication, especially formality and comment structure, but struggles with nuanced emotional expression. These limitations motivate larger, more diverse studies and more rigorous evaluation methods.
- Gemini captures users’ formality and comment structure, suggesting that generated responses could integrate into existing conversations.The conclusion identifies stylistic imitation as a promising basis for synthetic comment generation in online moderation.
- The model understates joy and overstates anger across prompting styles, limiting its ability to reproduce subtle emotional undertones.In r/science, inaccurate emotional replication could escalate tensions instead of supporting neutral or prosocial discussion.
- Larger-scale studies should use diverse datasets from multiple subreddits and a wider range of state-of-the-art LLMs.The authors identify broader data and model coverage as priorities for more rigorous evaluation.
- More robust analytical techniques, such as ANOVA, are needed to examine factors influencing LLM performance in online moderation.The authors present this multifaceted evaluation as important for understanding capabilities and limitations before deployment.