Source-linked AI summary
Towards Simulating Social Media Users with LLMs: Evaluating the Operational Validity of Conditioned Comment Prediction
Nils Schwager, Simon Münker, Alistair Plum, Achim Rettinger
TL;DR
LLM-based social simulation needs stronger validation of whether models reproduce specific users’ behavior. This paper introduces CCP and benchmarks multilingual 8B models under alternative conditioning and SFT settings, finding that SFT can align form while weakening semantic grounding in lower-resource languages, whereas explicit biographies add little after fine-tuning.
Problem
LLM social simulation lacks extensive validation of operational validity against authentic individual behavior.
Method
The paper benchmarks CCP across English, German, and Luxembourgish using 8B models, explicit and history-based conditioning, and SFT.
Results
Fine-tuning makes explicit biographies nearly redundant: History-Only reaches embedding distance 0.399 versus 0.397 for Biography+History, while low-resource settings show weakened semantic grounding.
Takeaways & Limitations
Authentic behavioral histories are prioritized over descriptive personas for fine-tuned user simulation, while SFT should be treated mainly as formatting control in German and Luxembourgish.
Takeaways & Limitations
With 8B models, SFT was difficult for German and Luxembourgish, failing to improve German semantic grounding and degrading it in Luxembourgish.
Abstract
from arXiv · showhide
The transition of Large Language Models (LLMs) from exploratory tools to active "silicon subjects" in social science lacks extensive validation of operational validity. This study introduces Conditioned Comment Prediction (CCP), a task in which a model predicts how a user would comment on a given stimulus by comparing generated outputs with authentic digital traces. This framework enables a rigorous evaluation of current LLM capabilities with respect to the simulation of social media user behavior. We evaluated open-weight 8B models (Llama3.1, Qwen3, Ministral) in English, German, and Luxembourgish language scenarios. By systematically comparing prompting strategies (explicit vs. implicit) and the impact of Supervised Fine-Tuning (SFT), we identify a critical form vs. content decoupling in low-resource settings: while SFT aligns the surface structure of the text output (length and syntax), it degrades semantic grounding. Furthermore, we demonstrate that explicit conditioning (generated biographies) becomes redundant under fine-tuning, as models successfully perform latent inference directly from behavioral histories. Our findings challenge current "naive prompting" paradigms and offer operational guidelines prioritizing authentic behavioral traces over descriptive personas for high-fidelity simulation.
1 Introduction
The paper addresses the weak validation of LLM-based social-media simulation by introducing a behavioral-grounded prediction task and evaluating conditioning, multilinguality, and fine-tuning. It argues that authentic behavioral traces and operational validity are central to assessing user simulation.
- LLM-based social simulation assumes models can predict how specific individuals respond to new stimuli.
- Existing persona prompting is largely heuristic and can produce plausible comments without reproducing authentic user-specific patterns.
- Conditioned Comment Prediction isolates whether models can predict a user’s reply to a stimulus from the available conditioning context.
- The study benchmarks open-weight 8B models across English, German, and Luxembourgish using prompting strategies, SFT, lexical metrics, and semantic distance.
- The paper investigates whether SFT universally improves prediction fidelity across linguistic resource tiers.
- Its contributions include multilingual benchmarking, comparison of explicit and implicit conditioning, and operational guidelines for more valid simulation research.
2 Background
The background identifies a validation gap in LLM-based social simulation and contrasts persona descriptions with behavioral histories as competing conditioning paradigms. The paper frames CCP as a direct test of alignment with authentic individual behavior.
- LLM social simulations often rely on plausibility judgments or aggregate statistics that may miss individual-level behavioral fidelity.
- Operational validity evaluates generated responses against actual user behavior rather than abstract plausibility.
- Explicit conditioning represents personas through natural-language descriptions of demographics, ideology, communication style, and behavior.
- Implicit conditioning supplies prior behavioral traces and requires the model to infer the user’s underlying behavioral signature.
- The paper directly compares explicit, implicit, and combined conditioning because their relative fidelity remains underexplored.
3 Methods for Conditioned Comment Prediction
The method evaluates conditioned reply prediction by comparing generated comments with authentic responses across users, languages, conditioning strategies, and model-training conditions. Histories, biographies, controls, and standardized evaluation metrics operationalize user-level fidelity.
- Task and conditioning: CCP compares a predicted reply with the authentic response to test individual-level behavioral alignment.
- Task and conditioning: The study varies explicit profiles, implicit behavioral examples, and their combination to distinguish persona following from persona inference.
- Task and conditioning: Implicit conditioning provides up to 30 prior stimulus–response pairs without an explicit user description.
- Task and conditioning: The generated biography summarizes basics, language, worldview, and behavior from authentic comments for explicit conditioning.
- Task and conditioning: The combined condition tests whether biographies and histories are complementary, redundant, or interfering.
- Task and conditioning: The control condition removes personalization to separate user alignment from generic stimulus-conditioned comment generation.
- Models and data: The benchmark uses Llama3.1, Qwen3, and Ministral instruction-tuned 8B models, including identically configured SFT across models.
- Models and data: Datasets cover English, German, and Luxembourgish social-media or news-comment interactions, with user-level splits and standardized preprocessing.
4 Experiments
The experiments evaluate multilingual comment prediction, fine-tuning, conditioning strategies, and history length. Results show resource-sensitive performance, surface–semantic divergence in Luxembourgish, and latent inference from behavioral histories after fine-tuning.
- 4.1 Prediction Fidelity (RQ1 & RQ2): Base-model alignment follows a linguistic-resource hierarchy, with English outperforming German and Luxembourgish.English achieves BLEU 0.053 and embedding distance 0.420, while Luxembourgish reaches BLEU ≈0.003.
- 4.1 Prediction Fidelity (RQ1 & RQ2): SFT improves English lexical alignment from BLEU 0.053 to 0.083 and reduces embedding distance from 0.420 to 0.397.
- 4.1 Prediction Fidelity (RQ1 & RQ2): In German, SFT raises BLEU from 0.065 to 0.095 while semantic alignment remains approximately 0.50.The pattern indicates improved lexical performance without deeper semantic improvement.
- 4.1 Prediction Fidelity (RQ1 & RQ2): In Luxembourgish, SFT improves surface metrics but worsens Llama3.1 embedding distance from 0.579 to 0.605.It also constrains Ministral’s length ratio from approximately 2.98 to 1.07, while semantic fidelity declines.
- 4.1 Prediction Fidelity (RQ1 & RQ2): Llama3.1 is the most stable model, maintaining realistic length ratios of 1.11 in English and 1.29 in Luxembourgish.Ministral’s Luxembourgish base-model ratio is 2.98, making it structurally misaligned without fine-tuning.
- 4.2 Ablation Study: Implicit vs. Explicit Conditioning: Without user context, fine-tuning achieves ROUGE-1 0.207 but embedding distance 0.418, compared with 0.399 for History conditioning.Topic and general comment style support lexical prediction, whereas user-specific context improves individual style, stance, and semantic intent.
- 4.2 Ablation Study: Implicit vs. Explicit Conditioning: Biography-Only base models produce a length ratio of 4.907, whereas fine-tuning reduces it to 0.935.History examples provide structural grounding for brevity and informality; SFT teaches persona descriptions to map to the reply format.
- 4.2 Ablation Study: Implicit vs. Explicit Conditioning: After fine-tuning, Biography+History offers little semantic advantage over History-Only, with embedding distances of 0.397 and 0.399 respectively.This supports latent inference from behavioral traces and makes an additional profiling step yield diminishing returns.
5 Recommendations and Future Work
The paper recommends grounding simulation in behavioral traces rather than relying on explicit personas, while treating SFT as primarily a formatting tool in lower-resource settings. It also identifies evaluation, scale, comparability, misuse, and privacy boundaries that motivate future work.
- Recommendations: Base models using Biography-Only conditioning exhibit structural failure and extreme verbosity, whereas generic behavioral demonstrations can anchor comment format and length.The reported length ratio is approximately 4.9 for Biography-Only.
- Recommendations: Authentic digital traces provide both structural compliance and stronger semantic and lexical alignment than generic histories or synthetic personas.The paper recommends prioritizing raw behavioral logs when available.
- Limitations and Future Work: SFT is not universal: for German and Luxembourgish 8B models, it corrected output length but failed to improve or degraded semantic grounding.The paper therefore frames SFT mainly as formatting control in these settings.
- Recommendations: English fine-tuning converges different base models and conditioning strategies toward a similar performance ceiling of BLEU ≈0.08.The paper describes this as an equalizing effect in high-resource English.
- Future Work: Future evaluations should test multi-turn stability, prompt information thresholds, non-verbal actions, and richer environmental inputs.These extensions target stability, minimum data requirements, and generalization of user simulation.
- Future Work: The low-resource form-content decoupling remains unresolved because it is unclear whether it reflects SFT or the 8B model scale.Future work should test larger models, including models at least 70B parameters.
- Limitations: Cross-language comparisons may be confounded because predictive signal and output variety differ across datasets.Weaker German and Luxembourgish fidelity may therefore reflect dataset unpredictability rather than only linguistic model limitations.
- Limitations: Automated metrics cannot fully capture nuanced persona failures such as tonal drift or subtle hallucinations.The evaluation uses BLEU, ROUGE, and Embedding Distance exclusively.
A Prompts
The appendix specifies separate profiling and reply prompts for generating explicit biographies and simulating comments from behavioral context. The profiler structures inferred user information across four dimensions, while no-biography conditions instruct consistency with prior writing.
- A.1 User Profiler Prompt: The profiler writes in second-person active voice and reports direct observations without reasoning or citations.Its prescribed forms include “You are...” and “You write...”.
- A.1 User Profiler Prompt: Profiles are requested at 500–1000 tokens and should become shorter when information is sparse or the user is unremarkable.The prompt explicitly conditions profile length on available information.
- A.1 User Profiler Prompt: The Basics section captures inferable demographics and account type, including whether the account is personal, parody, activist, professional, or organizational.It also asks whether the voice is authentic or performative.
- A.1 User Profiler Prompt: The Language section records languages, code-switching, formality, dialect markers, nativeness, and distinctive style.These fields describe how the user communicates.
- A.1 User Profiler Prompt: The Worldview section records beliefs, issue positions, ideological patterns, affiliations, opponents, and consistency or contradiction.It focuses on the user’s expressed perspectives and alignments.
- A.1 User Profiler Prompt: The Behavior section records engagement frequency, argumentation style, tone, audience, and intended objectives.It describes how the user operates in interactions.
- A.1 User Profiler Prompt: Four examples provide few-shot guidance covering a Baden-Württemberg professional, a vague or disengaged user, and a Luxembourgish language advocate.The appendix notes that diverse examples are included.
- A.1 User Profiler Prompt: The biography profiler is instructed to write clearly and confidently for roleplay simulation.It should produce a useful profile from the supplied content.
B Data, Code, and Model Availability
The technical pipeline and source code are available on GitHub, while fine-tuned models and datasets are restricted to scientific use and shared upon request.
- Availability: The technical pipeline and source code are available on GitHub.The repository is identified as Conditioned-Comment-Prediction.
- Availability: Fine-tuned models and datasets are shared only upon request to mitigate potential misuse while preserving reproducibility.The policy is presented as balancing open science and responsible research practices.
C.1 Multilingual Joint Training
Joint multilingual training is comparable to monolingual training overall for the evaluated 8B models, but it does not improve Luxembourgish prediction. Thus, mixing languages may simplify deployment without resolving low-resource data scarcity.
- Performance Parity: Mixed and monolingual training produce effectively indistinguishable BLEU and embedding-distance results.The differences are not statistically significant.
- Deployment Implication: A single mixed model remains a viable deployment strategy because multilingual training does not produce the reported performance penalty.The stated efficiency benefit is serving one model instead of three.
- Absence of Cross-Lingual Synergy: Joint training does not provide positive transfer to Luxembourgish: LB BLEU is 0.008 in Mix versus 0.009 in Mono.The paper interprets this as evidence against cross-lingual remediation of data scarcity.
C.2 Impact of Model Size
Model size constrains semantic fidelity: SFT improves output structure in small Qwen3 models but does not overcome their limited semantic reasoning.
- C.2 Impact of Model Size: Table 5 compares Qwen3 models from 0.6B to 8B parameters on English comment-prediction performance.Results are reported as means with standard deviations across five independent generation runs on 650 users.
- C.2 Impact of Model Size: SFT reduced the 1.7B Base model’s length ratio from 3.585 to 1.319, regulating structural output.This structural improvement did not compensate for the model’s limited semantic reasoning.
- C.2 Impact of Model Size: BLEU plateaued at ≈0.058 for the 0.6B and 1.7B models after SFT.The plateau indicates that small models gained little lexical fidelity beyond this level.
- C.2 Impact of Model Size: Embedding distance remained at approximately 0.420, indicating that SFT did not substantially improve semantic alignment in these small models.The models mimicked speech format without capturing deeper semantic patterns.
D Extended Tables
The extended tables document how fine-tuning comparisons were evaluated across lexical and semantic metrics, languages, models, and embedding systems.
- D Extended Tables: Table 6 compares Supervised Fine-Tuning with base-model prompting on lexical prediction metrics for RQ1 and RQ2.The evaluation covers 8B models in English, German, and Luxembourgish, using combined Biography+History conditioning.
- D Extended Tables: Table 7 compares Supervised Fine-Tuning with base-model prompting using embedding distance on a 0–2 scale.It reports results using Qwen3-Embedding-8B, embeddinggemma-300m, and LuxEmbedder.