Source-linked AI summary
Revising Context, Shifting Simulated Stance: Auditing LLM-Based Stance Simulation in Online Discussions
Xinnong Zhang, Wanting Shan, Hanjia Lyu, Zhongyu Wei, Jiebo Luo
TL;DR
LLM-based stance simulation may reflect context-sensitive model behavior rather than precise, stable user beliefs, raising a need for systematic auditing. The paper applies controlled text-only and multimodal context revisions, reruns stance simulation, and evaluates directional shifts and transition patterns. It finds effective and robust stance transitions across topics and models, while also identifying distinct revision mechanisms and important robustness trade-offs.
Problem
It remains unclear whether LLM-based simulations capture precise user-specific beliefs or are highly sensitive to semantically independent changes in conversational context.
Method
The paper audits stance simulation by applying controlled text-only and multimodal revisions to original conversations, then rerunning the same simulator under each revised context.
Results
The study finds robust and effective stance transitions across three topics and different simulation models, with distinct mechanisms for text-only and multimodal revisions.
Takeaways & Limitations
Counterfactual context revision provides a framework for understanding the context sensitivity, opportunities, and limitations of LLM-based social simulation.
Takeaways & Limitations
The evaluation uses Reddit data and three LLMs, so broader topics and more diverse, higher-quality platform data would strengthen the findings.
Abstract
from arXiv · showhide
Large language models are increasingly used to simulate social media users and infer how individuals may respond to online discussions. However, it remains unclear whether these simulations reflect precise user-specific beliefs or whether they are highly sensitive to semantically independent changes in conversational contexts. In this work, we study counterfactual context revision as a framework for auditing LLM-based stance simulation. Given an original online conversation, we first infer a target user's stance toward a specific topic. We then apply controlled revision strategies to the conversational context and simulate the user's stance again under the revised context. We compare text-only revision strategies with a multimodal one that incorporates meme-based context and evaluate two main effectiveness metrics, i.e., average directional stance shift and stance transition rate. The results reveal effective and robust stance transitions in both text-only and multimodal strategies across different polarization-preference mechanisms. Our study contributes an evaluation framework for understanding the context sensitivity of LLM-based stance simulation. More broadly, it highlights both the promise and risk of using LLMs to simulate online opinion dynamics.
1 Introduction
The paper audits whether LLM-based stance simulations reflect stable user beliefs or sensitivity to conversational context by applying controlled text-only and multimodal revisions. It finds that these revisions can shift simulated stances through distinct mechanisms, motivating a framework for understanding both the opportunities and limitations of social simulation.
- Motivation: LLM-based social simulation is used to model online behavior, infer opinions, and predict responses at scale.These applications complement surveys, annotation studies, and observational analyses by processing rich conversational context.
- Research question: The central question is whether simulated stances reflect user-specific preferences, conversational evidence, model assumptions, or superficial prompt cues.The issue concerns stance toward a specific target, rather than sentiment classification alone.
- Approach: The study compares text-only and multimodal context revisions to examine how altering the surrounding conversation changes simulated stance.The multimodal strategy incorporates visual signals such as memes, images, screenshots, and reaction images.
- Findings: Experiments across three stance topics and different simulation models find robust stance simulation alongside effective stance changes from added contextual information.Both text-only add strategies and multimodal meme strategies can change simulated stance.
- Findings: Revision mechanisms differ by modality, with add showing a significant depolarized trend and meme showing a strong polarized trend.These findings motivate mechanism-level analysis of how revision strategies influence simulated stance.
- Contribution: Counterfactual context revision is introduced as an auditing framework for evaluating the opportunities and limitations of LLM-based social simulation.The framework systematically compares revision strategies and analyzes why they influence simulated user stance.
2 Study Design and Experimental Setup
The study audits stance simulation by comparing a target user’s stance before and after controlled conversational revisions. It evaluates directional movement, transition patterns, and robustness across targets, communities, prompts, settings, and simulator models.
- Study design: The study frames stance simulation as counterfactual context revision: infer a target user’s stance, revise surrounding conversation, and simulate the same stance again.Only the surrounding conversational context is changed across revised conditions.
- Data: The dataset comprises Reddit discussions about DeepSeek, Claude, and Llama collected across diverse subreddits and posts.The final dataset contains 1,821 conversation instances across the three LLM families.
- Task formulation: The simulator infers positive, neutral, or negative stance toward a target topic from the conversation context and the last other-user message.Baseline validity compares inferred stance after masking the target user’s last message with the observed stance.
- Revision and simulation: Controlled revisions modify the last other-user message while preserving the topic, coherence, and stance-target identity, using text-only and meme-based multimodal conditions.The same simulator instruction, target identifier, and stance target are used across conditions.
- Evaluation: Effectiveness is measured by average directional stance shift and stance change rate, capturing overall movement and specific transitions.Positive shift values indicate movement toward support, while the change-rate analysis distinguishes supportive changes from opposing backfire changes.
- Robustness: Robustness is tested across stance targets, subreddits, discussion topics, simulator models, prompt variations, and decoding settings.The aim is to distinguish consistent revision effects from effects limited to a particular model or condition.
3 Revision Strategies
The paper uses revision strategies that vary how the last conversational message is changed, from wording-preserving reformulation to added arguments and meme-based multimodal context. These strategies test whether stance simulation responds to language, explanation, added content, or visual framing.
- Paraphrase: Paraphrase reformulates the last message with similar contextual language while preserving the underlying argumentation content.It tests sensitivity to wording differences without intentionally adding arguments or interpretations.
- Explain: Explain expands reasoning and clarification across the conversation without altering the message itself.The strategy addresses misunderstanding or ambiguity and examines whether clearer reasoning changes stance interpretation.
- Add: Add extends the conversational content with extra arguments or perspectives addressing problems expressed by the target user.It aims to encourage a more positive interpretation while retaining factual consistency with the initial conversation.
- Meme: Meme is a multimodal strategy in which a revision model generates meme text and a multimodal model produces the final meme from that text and a template.The procedure uses conversation history to tailor the meme content to the discussion context.
4 Results
The results establish a baseline for stance simulation, then show that controlled text-only and meme-based revisions produce distinct stance shifts, with multimodal effects supported during both revision generation and stance inference.
- 4.1 Original Conversations Provide a Baseline for Stance Simulation: Macro F1 was 78.10 and accuracy was 77.64 for original stance simulation, supporting the inferred stance as a baseline proxy.Macro F1 ranged from 76.05 for Llama to 78.25 for Claude across three target topics.
- 4.2 Context Revisions Produce Distinct Patterns of Simulated Stance Shift: The analysis compares revised-context predictions with original-context predictions using average directional stance shift and stance transition rate.The revision strategies modify the conversational context before simulating the same target user's stance again.
- 4.2 Context Revisions Produce Distinct Patterns of Simulated Stance Shift: +44.8% was the largest average positive shift among text-only strategies for add, while explain was smaller and paraphrase had limited effect.Strategies changing evidential or interpretive structure appeared more influential than surface-wording changes.
- 4.2 Context Revisions Produce Distinct Patterns of Simulated Stance Shift: +49.3% was the meme-based average directional shift, compared with +44.8% for add and -4% for paraphrase.The reported magnitude depended on the target and conversation type.
- 4.2 Context Revisions Produce Distinct Patterns of Simulated Stance Shift: Add reduced opposition toward neutral or supportive predictions, whereas meme moved initially neutral predictions toward support at an average rate of 17.6%.The average negative reduction rate for add was 4.7%, indicating different transition mechanisms.
- 4.2 Context Revisions Produce Distinct Patterns of Simulated Stance Shift: Meme revisions produced stronger shifts but higher backfire rates, so effectiveness must be considered with robustness rather than shift magnitude alone.A portion of originally positive or neutral predictions became more opposing after meme revision.
- 4.3 Meme Templates Provide Multimodal Effects: The r_meme variant generally outperformed r_white_meme despite identical textual content, indicating that meme-template visuals supplied additional inference cues.The visual information remained beneficial after revision text generation and contributed complementary context during interpretation.
- 4.4 Revision Mechanisms Are Different Across Strategies: Add reduced bipolar tone polarization toward moderate values, whereas meme intensified polarization and shifted tone toward more extreme values.Directional analysis found a strong depolarizing trend for add, while meme shifted tone almost randomly.
5 Related Work
Related work positions LLMs as tools for stance detection and broader computational social science applications, including simulation of users and opinion dynamics.
- 5 Related Work: LLMs are used across computational social science to simulate online users, detect stance, and support opinion-dynamics research.The cited work includes prompting or adapting LLMs for stance detection and using LLMs as zero-shot tools across the CSS pipeline.
6 Conclusion
The study audits LLM-based stance simulation through revision strategies and finds robust, effective stance-transition improvements alongside distinct revision mechanisms.
- The study audits LLM-based stance simulation through counterfactual revision strategies and extensive analysis.
- Table 3 isolates meme-template contributions during revision generation and stance inference through four ablation variants.
- Table 4 reports combined neutral and positive stance-transition rates across meme-based strategy variants.
- Table 5 compares tone-shift depolarization rates across revision types and models.
- The findings reveal robust and effective stance-transition improvements through several strategies, with distinct revision mechanisms.
Limitations
The study’s evidence is bounded by its focus on Claude, DeepSeek, and Llama discussions collected exclusively from Reddit.
- The study examines discussions about Claude, DeepSeek, and Llama, with all data collected from Reddit.The authors state that broader topic selections and higher-quality data from diverse platforms would strengthen the results.
Ethical Considerations
The study treats context revisions as controlled interventions for auditing simulated stance rather than attempts to change real users or public opinion, while applying privacy and safety precautions.
- The revision strategies are controlled interventions for auditing model behavior, not methods for manipulating real users or changing public opinion.Stance shifts are interpreted as changes in LLM simulated stance rather than evidence of actual human opinion change.
- The study removes or anonymizes user identifiers and avoids revised contexts containing harassment, private information, or harmful misinformation.Qualitative examples are paraphrased or masked when necessary to protect user identity.
A Additional Details of Data Collection and Preprocessing
The dataset captures Reddit discussions about major LLM families and related organizations, preserving conversational structure while applying relevance and context filters for stance analysis.
- Data collection: The study motivates data collection with the role of online discussions in shaping public perception, technology adoption, trust, and acceptance.AI-system discussions provide rapidly changing information, strong engagement, competing narratives, and evidence ranging from technical evaluations to memes.
- Data collection: Conversations were searched across 443 unique technology-related subreddits covering model capabilities, releases, safety, pricing, accessibility, reputation, and comparisons.
- Preprocessing: Each instance retains the Reddit title, body, comment structure, timestamps, subreddit, and conversational path to the target comment.Threads with deleted comments interrupting the middle of a conversation are removed, and target users must speak at least twice.
- Stance-target filtering: The target stance is defined at the level of an LLM family or associated organization, with explicit model-name mentions required in the target user’s comments.Manual validation found 45 of 50 keyword-containing conversations relevant to the target model, versus 6 of 50 without the keywords.
- Preprocessing: Cases with ambiguous targets or insufficient contextual information for stance inference are removed during preprocessing.
- Evaluation: Table 6 reports the average directional stance shift of strategies across all target topics and stance models.
B Implementation Details
The study uses multiple stance and revision models, and tests whether revision effects remain consistent across semantic topics and model settings. Topic-level analysis supports the reported contrast between depolarizing add revisions and polarization-amplifying meme revisions.
- Models: Stance simulation uses GPT-5.2, Qwen3.5, and Claude Sonnet 4.6, with GPT-5.2 results reported mainly in the paper and full results in Appendix D.1.Gemini and Claude Haiku are used for text revision, while Gemini and GPT-Image-2 generate multimodal meme revisions.
- Topic-level analysis: Topic-level analysis applies BERTopic with Sentence-Transformers embeddings, UMAP reduction, and HDBSCAN clustering to identify semantically similar discussion themes.The analysis examines whether polarization patterns are driven solely by differences in discussion content.
- Cross-topic results: Across topics, add shows a depolarizing tendency while meme amplifies polarization, indicating that the pattern remains consistent across discussion contexts.The trend is reported for the AI Model Evaluation, AI-Assisted Software Development Practices, AI Model Training Costs and Efficiency, and AI Language Models and Consciousness Debate topics.
- Cross-model results: Across GPT-5.2, Sonnet-4.6, and Qwen3.5, meme remains among the most effective strategies, whereas add varies more by simulator model.The reported variation may depend on simulator priors, instruction following, or sensitivity to conversational framing.
D.2 Results on Claude-Based Revision Strategy
Claude-based revision results show similar trends across text-only strategies, while the topic-level figures examine whether tone distributions remain consistent across revised contexts. The section also reports meme-template directional stance shifts across target topics.
- Claude-based revisions: Claude-based revision results show similar trends across the text-only revision strategies.The corresponding results use GPT-5.2 for stance inference and are presented in Figure 10.
- Topic-level tone analysis: Figures 5–8 compare tone-score distributions for original, add-revised, and meme-revised comments across four target topics.The topics cover AI model evaluation, AI-assisted software development practices, AI model training costs and efficiency, and AI language models and consciousness debate.
- Meme-template analysis: Table 7 reports the average directional stance shift of different meme templates across all target topics.The supplied table caption identifies the metric and scope but does not provide template-specific values.
E Sensitivity Analysis
Sensitivity analyses examine prompting, temperature, meme-template, and stance-inference configurations. The supplied materials document the evaluated prompts and pipeline roles, while the robustness passages report consistency across meme templates and stance models.
- Text-only sensitivity: Text-only revisions are tested at temperatures 0.5 and 1.0 and with two paraphrased prompts against a default temperature-0 configuration.The analysis evaluates whether prompting style or decoding temperature substantially changes stance interpretations.
- Meme-template sensitivity: Multimodal sensitivity analysis evaluates average directional stance shift across five meme templates, with high consistency observed within each topic.The templates are collected from top-ranking ImgFlip templates.
- Add revision: The add strategy revises only the last other-user message by introducing new, reasonable arguments while preserving factual accuracy and conversational consistency.The prompt prohibits modifying earlier messages, contradicting earlier statements, or introducing unverifiable facts.
- Meme generation: The meme strategy generates a reply intended to change the target user’s stance, converts it into concise meme text, and places text according to the template layout.The output uses JSON keys for positions such as top_text, bottom_text, panels, caption, left, or right, including only positions present in the template.
- Stance inference: The observed-stance prompt focuses exclusively on the target user’s explicit or strongly implied sentiment toward the target model.This provides a stance-inference variant based on the target user’s own statements rather than other users’ opinions.
- Stance inference: The inferred-stance prompt predicts positive, neutral, or negative stance after the conversation using the target user’s statements and contextual influence, defaulting to neutral when evidence is insufficient.It instructs the simulator not to treat other users’ stances as the target user’s stance.