Source-linked AI summary
Thinking Less to Simulate Better: Intuitive Prompting Improves LLM Agents Simulating Individual Social Media Reactions, Including Unfamiliar Content
Ljubisa Bojic, Tijana Stanic, Joerg Matthes, Agariadne Dwinggo Samala, Bojana Dinic, Jue Wang
TL;DR
Simulated users could support platform-policy testing, but validation has focused more on matching human behaviour than on profile consistency. This study profiles eight Serbian participants and tests four models across five prompt conditions; attitudinal profiles outperform demographic backstories, while intuitive instructions improve fidelity and individuation.
Problem
Validation has largely assessed agreement with human behaviour, with less attention to whether agents react in line with their supplied profiles, despite the stakes of simulating public opinion.
Source: §1 · p. 2Method
The study profiled eight Serbian adults, recorded reactions to 68 posts, and tested four language models under five prompt conditions.
Source: §2.1 · p. 5, §4 · p. 15Results
Attitudinal profiles outperformed demographic backstories (367 questionnaire-only versus 252 demographics-only correct item-level cells, p < .001); intuitive instructions improved fidelity and individuation.
Source: §3.1 · p. 9, §4.1 · p. 16Takeaways & Limitations
Synthetic-respondent designs need attitudinal profile content, and prompt framing should be treated as a design variable.
Source: §4.1 · p. 16Eight participants support claims within a rich-profile design, not population-level estimates; condition differences rely on item-level comparisons because four-model tests lack conventional power.
Source: §4.3 · p. 18
Abstract
from arXiv · showhide
Platform policies are increasingly tested on artificial users, making agent fidelity important. Yet convincing fake profiles could also manipulate perceived public opinion before elections. Validation has concentrated on agreement with human behaviour and has paid little attention to whether an agent behaves in line with the profile it was given. The present study profiled eight Serbian participants through a questionnaire, a deep interview, and a written self-presentation, recorded their reactions to sixty-eight social media posts, and asked four language models to predict those reactions under five prompt conditions varying profile content and instruction style. Attitudinal content improved prediction over demographic backstories by a wide margin. Agents matched their stated profiles more closely than participants matched their own survey answers, and consistency proved unrelated to fidelity once profile information was present. Instructing models to respond intuitively and immediately rather than analytically gave the highest fidelity of any condition and cut the compression of individual differences from seven times the human level to three. The advantage held on posts about topics the questionnaire never raised, where that condition reached the highest fidelity of any setup and beat a crowd baseline by a wide margin, which suggests that agents prompted this way could serve as general-purpose simulated users rather than specialists on the topics they were profiled for. Results may bear implications for the development of language models, because intuition-based setups appear better suited to some tasks than reasoning-based ones.
1. Introduction
LLM-based simulated users could support platform-policy testing but also create plausible fake profiles capable of distorting perceived public opinion. The introduction frames individual-level fidelity and profile consistency as underexamined questions amid concerns about compressed differences and how prompts shape reactions.
1. Introduction
- Simulated users could help test platform policies, but plausible fake profiles could also shift perceived opinion before elections.The paper treats fidelity to individuals as a governance concern, not only a technical one.Source: §1 · p. 2
- Prior work reports stronger model performance on aggregates than on particular people, and persona prompts can push outputs toward caricatures.The literature also distinguishes language competence from fidelity to a specific reader’s reaction.Source: §1 · p. 3
- Compressed individual differences can distort simulated populations, potentially obscuring disagreement and minority positions in policy tests.Prior work cited in the introduction links compressed variance to flipped regression signs and manufactured significance in synthetic data.Source: §1 · p. 3
- Most validation measures agreement with human behaviour, while whether agents follow their supplied profiles has received less attention.The paper treats internal consistency as a diagnostic, since people themselves often act inconsistently with stated attitudes.Source: §1 · p. 4
- Because social-media reactions are described as rapid judgments, the study tests whether immediate rather than analytical instructions better match them.The introduction also motivates depth across eight people as a trade-off against breadth across a national population.Source: §1 · pp. 4–5
2. Methodology
The study combines in-depth profiles and recorded reactions from eight Serbian adults with predictions from four language models under five prompt conditions. It compares predictions with human reactions, profile-implied reactions, and differences between individuals, while documenting its procedures and data safeguards.
2. Methodology
- The factorial computational experiment tests four models on eight profiles and 68 posts under five conditions, measuring fidelity, profile-consistency, and individuation against the same human data.The design yielded 20 configurations and 10,880 predicted reactions.Source: §2 · p. 5, §2.4 · p. 6
2.1 Participants
- Eight Serbian adults were recruited for variation in political self-placement and institutional trust, rather than demographic representativeness.The design prioritizes depth per person for individual-level prediction rather than population estimates.Source: §2.1 · p. 5
2.2 Profile construction
- Profiles combined a Serbian-language questionnaire on demographics, attitudes, trust, values, and other characteristics with a deep interview and free-text self-presentation.Interview transcripts and self-presentations entered prompts in full rather than in summarized form.Source: §2.2 · p. 5
2.3 Stimulus set
- The stimulus set comprised 68 Serbian posts, split evenly between questionnaire-covered and unraised topics, and participants recorded like-or-dislike reactions to each.The panel produced 544 human reactions; 51.1 per cent were likes.Source: §2.3 · p. 6
2.4 Prompt conditions
- Five conditions varied both profile content and instruction style, from demographics alone to rich interview-based profiles with inference or intuitive-response instructions.The intuitive prompt asked models to respond immediately rather than analytically and to accept reactions at odds with the stated profile.Source: §2.4 · p. 6
2.5 Outcome measures
- Fidelity used Matthews correlation as the primary measure; profile-consistency scored reactions against survey-implied patterns, and individuation measured across-person variation in like-rates.Neutral survey responses were excluded from consistency scoring; compression ratios above one indicate that agents treat people more alike than humans do.Source: §2.5 · pp. 6–7
2.6 Baselines and statistical procedures
- Five baselines included simple response rules, participant and crowd priors, logistic regression, and random forest; baselines used leave-one-participant-out cross-validation.Condition comparisons used paired agent-post observations, model-level Wilcoxon tests, and Pearson correlations for consistency and fidelity.Source: §2.6 · p. 7
2.7 Ethics and reproducibility
- Participants consented to the study and profile use; names were replaced, transcripts checked for identifying details, and code, prompts, post key, and converted outputs were made available.The materials support varying rule thresholds and repeating the analysis.Source: §2.7 · p. 7
3. Results
Across eight participants, intuitive prompting produced the strongest fidelity and reduced the compression of individual differences. The results also distinguish fidelity to observed reactions from consistency with a stated profile, and show transfer to posts on unprofiled topics.
3.1 Fidelity across prompt conditions
- Immediate intuitive framing produced the highest mean fidelity and beat every alternative in item-level comparisons, though model-level tests lacked power across four models.Questionnaire beat demographics (367 versus 252 cells, p < .001); interview material added 0.027 mean fidelity (238 versus 208 cells, p = .170).Source: Table 1 · p. 9, §3.1 · pp. 9–10, Figure 2 · p. 9
3.2 Agents against their own profiles
- Eleven of twenty configurations exceeded human profile-consistency; questionnaire-only configurations reached 0.904, above participants’ 71.3% strict and 79.1% partial-credit scores.Questionnaire-only agents often retrieve numeric attributes, while narrative material invites judgment and brings consistency nearer the human level.Source: §3.2 · p. 10, Figure 3 · p. 10
3.3 Individuation
- Intuitive prompting reduced compression of between-person like-rate variation to 3.1 times the human value, with no model assigning all participants identical rates.The estimate rests on eight participants; the intuitive-versus-interview difference was 0.004 with a 95% interval from −0.010 to 0.019.Source: §3.3 · pp. 10–11, Figure 4 · p. 11
3.4 Consistency and fidelity as separate properties
- Across all configurations, consistency and fidelity correlated positively (r = .490), but among those with attitudinal profiles the association was negative and nonsignificant (r = −.364).The results support using profile-consistency as a diagnostic alongside fidelity, not as a standalone target.Source: §3.4 · pp. 11–12, Figure 5 · p. 12
3.5 Generalisation to unprofiled topics
- On unprofiled posts, fidelity ranged from 0.324 to 0.399 against a crowd baseline of 0.265; intuitive framing led at 0.399, while questionnaire-only was lowest at 0.324.Interview material’s gain grew from 0.027 in-domain to 0.050 out-of-domain; intuitive beat questionnaire-only 161–120 cells (p = .017), but its contrast with demographics was nonsignificant (p = .203).Source: Table 2 · p. 12, §3.5 · pp. 12–14
3.6 Comparison against baselines
- On the 33 contested posts, the crowd prior fell to −0.008 while every configuration stayed positive; questionnaire and intuitive conditions reached 0.350 and 0.337.Random forest and logistic regression scored 0.302 and 0.183, below every configuration with attitudinal information; majority vote scored 0.739 versus 0.756 for the best single configuration.Source: §3.6 · pp. 14–15, Table 3 · p. 14
3.7 Statistical considerations
- Item-level contrasts use paired agent-post observations, while model-level tests use only four models; Holm correction preserves the reported item-level conclusions.Cohen’s kappa tracked Matthews correlation rankings, though Matthews correlation remained the primary measure.Source: §3.7 · p. 15
4. Conclusion
The study finds that intuitive prompting improves fidelity and preserves more individual variation, while attitudinal profiles outperform demographic descriptions. Its conclusions also identify the scope of transfer and several limits on generalization.
4.1 Answers to the hypotheses
- Mean fidelity rose from 0.300 with demographic descriptions to 0.406 with attitudinal profiles, with 367 item-level gains against 252 losses (p < .001).Every configuration receiving attitudinal information exceeded the random-forest baseline of 0.302 and logistic-regression baseline of 0.183.Source: §4.1 · p. 16
- 0.473 mean fidelity under instructions that suppress deliberation was the highest of any condition, while compression of individual differences fell from seven times the human level to three.The authors identify this as the study’s main constructive result.Source: §4.1 · p. 16
- Profile consistency and predictive fidelity were distinct once profile information was present: greater consistency no longer implied greater fidelity, while participants matched their stated attitudes in 71.3% of cells.Eleven of twenty configurations exceeded participants’ self-consistency rate; the three questionnaire-only configurations reached 0.904.Source: §4.1 · p. 16
- On unprofiled topics, intuitive framing reached 0.399, the best result, exceeding the crowd baseline of 0.265 and the questionnaire-only condition’s 0.324.The advantage on unfamiliar content was qualified: transfer across configurations ranged from 0.324 to 0.399, and the tested topics were selected by the research team.Source: §4.1 · p. 16, §4.3 · p. 18
4.2 Implications
- The authors connect the findings to both research utility and misuse: simulated populations could respond to arbitrary content, while convincing fake accounts could shift apparent opinion.They also note that reduced compression narrows a signature detection methods might exploit; intuitive prompting’s components remain unresolved.Source: §4.2 · p. 17, §4.3 · p. 19
- Prompt framing improved fidelity without changing the model or profile, and the gain from intuitive instructions exceeded that from adding a deep interview to a questionnaire.The authors argue that matching the manner of the task to quick human judgments may matter as much as matching profile content.Source: §4.2 · p. 17
- Narrative profile material added little within surveyed topics but improved transfer beyond them; the authors recommend it when agents must respond to novel content.The interview raised unprofiled-topic fidelity from 0.324 to 0.374, and intuitive framing reached 0.399.Source: §4.1 · p. 16, §4.2 · p. 18
4.3 Limitations and future research
- Generalization is constrained by eight Serbian participants, one set of unprofiled topics, and prompt conditions that combined multiple changes; each configuration was run once.The authors also report shared errors on 8.3% of items and caution that simulation remains unreliable as a replacement for asking people directly.Source: §4.3 · p. 18, §4.2 · p. 18
Ethical Approval
The study received institutional ethics approval, obtained participant consent, and reported no competing interests.
Ethical Approval
- The University of Belgrade ethics committee approved the study on 12 May 2026, and procedures followed institutional and Helsinki standards.The approval was from the Institute for Philosophy and Social Theory.Source: Ethical Approval · p. 19
- All participants provided informed consent after receiving information about the study, risks, benefits, confidentiality, anonymity, and voluntary participation.Consent was indicated by clicking a consent button.Source: Ethical Approval · p. 19
- The authors declared no competing interests.Source: Ethical Approval · p. 19
- Author contributions covered conceptualization, methodology, analysis, investigation, resources, data curation, writing, and visualization.Source: Ethical Approval · p. 19