PaperlayerView exact revision on arXiv

Source-linked AI summary

Wiki-Talkie: Multilingual Benchmarking of Persona-Based Agents on Real-World Discussions

Dennis Fucci, Andrea Bacciu, Dong Liu, Weronika Łajewska, Saab Mansour

arXiv:2610.08513v1cs.CLcs.AI

TL;DR

Existing persona-based conversation datasets often use fictional personas or cover few languages, leaving limited evidence for evaluating agents against real multilingual discussions. Wiki-Talkie pairs Wikipedia Talk conversations with real-user-derived personas and evaluates next-turn generation across conditioning strategies. Comment histories consistently outperform explicit persona information, with small cross-lingual differences.

  • Problem

    Existing persona-based conversation datasets rely on fictional personas or have limited language coverage, leaving no resource for evaluating real-persona behavioral fidelity in multilingual cooperative discussions.

    Source: §1 · p. 1

  • Method

    Wiki-Talkie pairs five-language Wikipedia Talk discussions with anonymized profiles and behavioral traits, then evaluates next-turn generation for population-level behavioral fidelity.

    Source: §1 · pp. 1–2

  • Results

    Comment histories consistently outperform explicit persona information as a conditioning strategy, while cross-lingual differences are small.

    Source: §1 · p. 2

  • Takeaways & Limitations

    Population-level evaluation compares community behavior without requiring fidelity to a specific individual persona, which the paper identifies as raising impersonation and consent concerns.

    Source: §1 · p. 2

    The dataset covers five high-resource European languages, so findings may not generalize to communities with different writing systems, discourse conventions, or editorial practices.

    Source: Limitations · p. 9

Abstract

from arXiv · show

LLMs are increasingly deployed as autonomous agents in social environments, making it critical to study their ability to faithfully simulate human interactions. Central to this is grounding agents in realistic user personas, yet existing datasets rely on fictional personas and are limited to a handful of languages, lacking the empirical grounding necessary to evaluate behavioral fidelity across diverse populations. We introduce Wiki-Talkie, a multilingual dataset of real-world conversations from Wikipedia Talk pages across five languages spanning two language families: Germanic (German, English) and Romance (Spanish, French, Italian), paired with personas derived from real user communities and encompassing sociodemographic attributes, self-descriptions, and behaviorally grounded interaction traits. Using Wiki-Talkie, we evaluate agent interactional behavior on a next-turn generation task across various persona conditioning strategies. Our evaluation assesses whether agents collectively reproduce the distributional behavioral patterns observed in human discussions. Results show that user's comment history exemplifying interaction behavior consistently outperforms explicit persona information. In addition, models systematically underproduce negative or extreme sentiments, while over producing references and suggestions, revealing biases toward agreeableness and positivity. Crucially, these patterns hold robustly across languages, with small cross-lingual differences.

1 Introduction

The paper addresses the lack of multilingual conversational datasets grounded in real users, which limits evaluation of whether agents reproduce real populations’ opinions and interaction dynamics. It introduces Wiki-Talkie and evaluates population-level behavior, finding that comment histories condition agents more effectively than explicit persona information, with small cross-lingual differences.

1 Introduction

  • Existing persona datasets rely on fictional personas, while real-persona resources cover only Chinese or English, leaving no multilingual resource for real-world cooperative discussions.Such a resource is needed to evaluate behavioral fidelity against actual human communities, including conflict, misalignment, and evolving viewpoints.Source: §1 · p. 1
  • Wiki-Talkie provides real-world Wikipedia Talk discussions in five languages, with personas derived from user pages and comment histories and identities anonymized.Profiles are regenerated neutrally rather than copied verbatim to maintain research utility while protecting privacy.Source: §1 · pp. 1–2
  • The evaluation measures population fidelity through next-turn generation, comparing human and agent sentiment and dialogue-act distributions alongside instance-level text similarity.It contrasts behavioral traits extracted from comment histories with explicit profile descriptions across five languages.Source: §1 · p. 2
  • Comment history consistently outperforms explicit persona information as a conditioning strategy, and cross-lingual differences are small.The results suggest showing how a persona behaves is more effective than telling what a persona is.Source: §1 · p. 2

2 Related Work

Prior persona datasets often use simulated or synthetic conversations, while naturally occurring interaction datasets remain concentrated in a few languages and lack linked real-user personas. Existing work also leaves open whether social-behavior biases persist in naturalistic, multilingual cooperative discussions.

2 Related Work

  • No existing resource combines naturally occurring multilingual dialogue, real user-derived personas linked to conversations, and cooperative but contentious interaction.PersonaChat and its extensions use fictional personas, while naturally occurring interaction datasets remain largely limited to English and Chinese.Source: §2 · p. 2
  • Prior LLM social-behavior findings report agreement and positivity biases, but are mainly based on English-only settings, synthetic surveys, or opinion elicitation.Whether these biases persist in multilingual, naturalistic cooperative discussion remains open.Source: §2 · pp. 2–3

Datasets Derived from Wikipedia Talks

  • Wikipedia Talk datasets have supported studies of discourse, conflict, and toxicity, but are mostly English-centric and lack persistent personas for multilingual evaluation.Prior work had not used Talk pages to analyze LLM behavior in socially grounded collaborative settings.Source: Datasets Derived from Wikipedia Talks · p. 3

3 Wiki-Talkie

Wiki-Talkie combines multilingual Wikipedia discussions, article context, and two kinds of user personas: self-reported profiles and behavior inferred from comment histories. Its construction extracts and revises discussions, retrieves the contemporaneous article, and derives and selects user attributes and behavioral traits.

3 Wiki-Talkie

  • Wiki-Talkie pairs multilingual Wikipedia Talk threads with article context and personas drawn from user profiles and comment-history behavior.It spans five languages across Germanic and Romance families; profiles include attributes such as gender, profession, interests, and contribution topics.Source: §3 · p. 3, Figure 1 · p. 3
  • Threads average 3.2–3.6 turns and 2.1–2.3 unique users across languages, while German contributes the largest corpus and turn lengths vary from 47.74 words in German to 79.73 in Spanish.The high standard deviations indicate substantial variation, from brief clarifications to extended debates.Source: §3 · pp. 3–4

3.1 Dataset Construction

  • Thread extraction parses headings, signatures, and reply indentation, then uses an extractor model to revise turn boundaries and attributions or flag irrecoverable cases.The associated article is retrieved at the revision immediately before the thread’s first comment, and usernames are anonymized with deterministic hashing.Source: §3.1 · p. 4
  • User-page content is filtered for substantive self-description and summarized into profiles, with extracted attributes used for subgroup analysis and user selection.Users with generated profiles are capped at 10,000 per language; selection prioritizes contribution topics, then gender and profession, and retains threads whose participants all belong to the selected subset.Source: §3.1 · p. 4
  • Behavioral traits are inferred from each user’s comments across up to 50 sampled threads and characterize eight dimensions, including disagreement strategy and editing behavior.Traits use closed response options only when evidence is sufficient; a manual audit of profile and trait quality is reported in Appendix I.Source: §3.1 · p. 4

4 Experimental Setting

The experiments use next-turn generation with ground-truth prior comments, comparing five persona-conditioning strategies across languages and models. Evaluation combines instance-level text similarity with population-level sentiment and dialogue-act distributions.

4 Experimental Setting

  • Agents generate a target user’s next comment from at least one preceding ground-truth comment, enabling controlled comparisons across languages and conditioning strategies.The setup spans over 100 experimental conditions: five languages, four models, and five conditioning methods.Source: §4 · pp. 4–5
  • Generation uses up to 5,000 sampled threads per language, with users required to have at least six corpus comments and up to 50% of sampled comments authored by females.This subsample was selected to manage generation costs.Source: §4 · p. 5
  • Five conditioning methods range from base role-play and self-reported profiles to behavioral traits, traits with model reasoning, and 5–20 prior comments from the same user.The history condition provides indirect conditioning through prior stylistic and behavioral patterns.Source: §4 · p. 5
  • Experiments use four state-of-the-art models spanning proprietary and open-weight systems, while ROUGE-L and BERTScore provide complementary instance-level similarity measures.These text-similarity metrics are coarse reference points because generation is open-ended.Source: §4 · p. 5
  • Population-level evaluation compares sentiment and dialogue-act distributions using Wasserstein Distance and Jensen-Shannon Divergence, alongside text similarity averaged over output-reference pairs.Figure 2 organizes ROUGE-L and rescaled BERTScore across five conditioning methods, averaging by language in one panel and by model in the other.Source: §4 · pp. 5–6, Figure 2 · p. 6

5 Results

Across text similarity and interaction behavior, concrete demonstrations—especially comment history and behavior paired with reasoning—align agents more closely with human patterns than abstract persona descriptions. Alignment remains limited by systematic behavioral differences and the evaluation’s five high-resource European languages.

5.1 Text Similarity

  • History yields the highest text similarity across models and languages, reaching ROUGE-L 0.154 for Model A in es and rBERT 0.105 for Model A in de.behavior+r is second, with ROUGE-L 0.138 for Model A in es and rBERT 0.085 for Model A in de; profile generally degrades similarity relative to base.Source: §5.1 · p. 6

5.2 Interaction Behavior

  • History was the strongest method, achieving the lowest JSD overall (0.085 for Model A in en); behavior+r ranked second, consistently improving over base (Model B: 0.272 → 0.241; Model D: 0.199 → 0.166).Behavior was weakest, increasing JSD across most models and languages, except Model B, where it improved over base (0.272 → 0.245). Cross-lingual differences remained marginal.Source: §5.2 · p. 7

5.1 Text Similarity

  • Cross-lingual differences in text similarity are small, but the evaluation spans only five high-resource European languages.The languages span two language families and differ in Wikipedia community size and editorial norms.Source: §5.1 · pp. 6–7

5.2 Interaction Behavior

  • Across text similarity and interaction behavior, history and behavior+r outperform abstract descriptions or sociodemographic information in aligning agent outputs with human patterns.This is the dominant pattern across models and languages.Source: §5.1 · p. 6, §5.2 · p. 7
  • Longer profile prompts do not consistently improve similarity, and acts or sentiment JSD mostly worsen with longer profiles, arguing against a pure context-budget explanation.The authors interpret information type, rather than quantity, as the operative factor.Source: §5.2 · p. 7
  • Agents over-produce REQUEST and COMMIT but under-produce NEGATIVE and REPORT; Figure 4 reports proportion differences averaged across languages.Examples include REQUEST δ = −0.177 for Model C in de on behavior, COMMIT δ = −0.108 for Model D in fr on behavior, and REPORT δ = 0.088 for Model C in fr on behavior.Source: §5.2 · pp. 7–8, Figure 4 · p. 8
  • Explicit gender conditioning through profile neither amplifies nor reduces gender disparities in dialogue-act profiles compared with base or history.The analysis covers two models and English and French under history, profile, and base conditions.Source: §5.2 · p. 8
  • Sentiment alignment also favors history, with minimum WD 0.097 for Model C in en; agents show fewer negative comments and rarely produce sentiment extremes.These patterns hold across models, conditioning methods, and languages.Source: §5.2 · p. 8, Figure 5 · p. 8

6 Conclusions

Wiki-Talkie is a multilingual dataset of Wikipedia discussions used to evaluate persona-conditioned agents’ population-level interaction behavior. Comment-history examples outperform abstract persona descriptions, while agents show agreeableness-related behavioral biases and small cross-lingual differences.

6 Conclusions

  • Wiki-Talkie pairs Wikipedia discussions in five languages with self-reported profiles, comment-derived behavioral traits, and article context.The evaluation uses four LLMs and targets population fidelity, complemented by instance-level text similarity metrics.Source: §6 · p. 8
  • Concrete comment examples consistently outperform abstract persona descriptions across languages, while agents are more action-oriented and less confrontational than humans.Sociodemographic information can degrade performance; cross-lingual differences remain small, with English showing slightly greater alignment.Source: §6 · p. 8
  • Agents are more action-oriented and less confrontational than humans, indicating a bias toward agreeableness and avoidance of extremes.Source: §6 · p. 8
  • The experiments focus on next-turn generation, while Wiki-Talkie is designed to support multi-turn generation, summarization, factual-grounding assessment, and quality evaluation.The authors encourage the community to explore these richer settings.Source: §6 · p. 9

Limitations

The paper’s findings are bounded by dataset coverage and quality, the conditioning methods tested, and the evaluation target. Several comparisons also retain an information-density confound, while persona descriptors depend on one extractor and prompting design.

Limitations

  • Residual noise includes occasional misattributed comments, imperfect comment boundaries, and formatting artifacts, though the authors expect little effect on evaluation or analysis.Source: Limitations · p. 9
  • Wiki-Talkie covers five high-resource European languages, so findings may not generalize to communities with different writing systems, discourse conventions, or editorial practices.The dataset remains gender-skewed despite subsampling, which may affect the robustness of gender-based analyses.Source: Limitations · p. 9
  • Persona richness varies across users and languages, and English is better represented, making its conditioning experiments potentially more reliable than those for other languages.The profiles and behavioral traits come from one anonymized extractor model and prompting design; conclusions apply to these generated descriptors, not all possible persona representations.Source: Limitations · p. 9
  • Experiments test prompting-based persona conditioning, leaving fine-tuning, retrieval-augmented conditioning, and embedding-based representations for future work.Source: Limitations · p. 9
  • Conditioning methods are not matched for information density: history supplies 5–20 raw comments, while profile and behavior use compressed summaries.A fully matched comparison remains future work; the history-length analysis was not performed because lower-comment buckets were too sparse.Source: Limitations · p. 9
  • The primary evaluation measures population fidelity rather than whether each response faithfully reflects its specific persona, leaving individual-level consistency for future work.Source: Limitations · pp. 9–10

Ethic Statement

The ethics statement treats Wikipedia-derived discussions as human-derived data and describes safeguards for privacy, access, and misuse. It also acknowledges residual identification risks and the ethical tension created by reuse without individual consent.

Privacy, Data Protection, and Contextual Integrity

  • The dataset is treated as human-derived data requiring safeguards beyond copyright compliance, guided by data minimization, proportionality, contextual integrity, and harm reduction.Source: Privacy, Data Protection, and… · p. 10
  • Profiles are anonymized, neutral re-elaborations rather than verbatim user pages, omitting direct identifiers but potentially diverging from source content or introducing erroneous attributes.The authors manually inspected a random sample of outputs.Source: Privacy, Data Protection, and… · p. 10

Re-Identification Risk and Access Control

  • Publicly archived discussion text leaves a content-based re-identification risk, and aggregation may lower the cost of searching for original authors.Access requests, an audit trail, opt-out removal, and non-de-anonymization requests are among the stated mitigations.Source: Re-Identification Risk and Access… · p. 10
  • Individual consent was not sought, and the authors acknowledge that licensing permissibility does not resolve the tension with contributors’ original communicative intent.Source: Re-Identification Risk and Access… · p. 10

Experimental Ethics and Persona Conditioning

A.1.1 Distributions

Across the five languages, Wiki-Talkie’s conversation distributions show broadly consistent patterns, with short turns and threads and a long tail of more active users. The appendix also describes user and persona coverage, preprocessing, examples, and prompt structure.

A.1.1 Distributions

  • Conversation statistics—words per turn, turns per user, turns per thread, and users per thread—show remarkably consistent patterns across the five languages.Source: §A.1.1 · p. 14, Figure 6 · p. 15
  • Turn lengths are heavily right-skewed in every language, with most turns under 200 words; threads containing turns over the uniform 1,000-word cap were excluded.Source: §A.1.1 · p. 15
  • Most users contribute fewer than 50 turns, while Italian and Spanish have narrower spreads consistent with their smaller thread collections.Source: §A.1.1 · p. 15
  • Threads typically contain 2–5 turns and exactly two users; threads with more participants are rare.Source: §A.1.1 · pp. 15–16

A.1.2 Users and Personas

  • User pages are right-skewed, while constructed profiles cluster around 50–100 words; English descriptions are richest, and behavioral-trait missingness varies modestly across languages.Agreement and disagreement-related traits have the largest non-deducible or malformed shares; several other traits remain at or below 0.10.Source: §A.1.2 · p. 16

A.2 Anonymization and Normalization

  • Preprocessing replaces usernames with salted hashes and normalizes thread formatting while preserving templates and external links.Source: §A.2 · p. 16

B Examples

  • The appendix provides one representative example per language, pairing a discussion thread with participant persona profiles.Behavioral traits and article text are omitted from these examples due to space constraints.Source: B Examples · p. 16

C Prompts

  • Study prompts use a system-and-user structure, with the user prompt supplying task instructions, definitions, exemplars where applicable, and inference-time inputs.Source: C Prompts · p. 16

D Data Sample

Wiki-Talkie’s sampled data spans five language splits, with differences in split size and gender balance. Generation inputs were truncated to fit each model’s context window, with history reduced when necessary.

D Data Sample

  • English and French splits contain roughly 9–10k comments each, while Spanish and Italian are about half their size.German yielded fewer turns despite starting from the same 5,000 initial threads.Source: D Data Sample · pp. 16, 18
  • All languages average around three previous comments per turn, but user distributions are male-skewed, most strongly in Italian and Spanish.Gender balancing was impossible in those two splits because the pool of female editors was limited.Source: D Data Sample · p. 18

E Generation Details

  • When prompts exceeded a model’s context window, the article body was progressively shortened; history was reduced until it fit or fell below five threads.Prompts that still could not fit after these truncation strategies were discarded.Source: E Generation Details · p. 18

F Validation of Acts Annotation

The authors adapted SEWD annotations to Wiki-Talkie’s target-turn format and tested an LLM annotator across different context windows. The two-comment setting achieved the highest Micro-F1, near the reported human agreement figure, with a caveat that the evaluation sets differ.

F Validation of Acts Annotation

  • The validation pipeline converted SEWD into target comments paired with preceding context, then manually filtered unreliable mappings, leaving 735 usable threads.SEWD labels from two trained annotators served as the gold standard.Source: F Validation of Acts Annotation · pp. 18–19
  • Micro-F1 reached 0.77 with two preceding comments, close to the reported human inter-annotator agreement of 0.79.The results are not directly comparable: the validation set was converted and manually filtered, while the cited human result used cross-validation over all SEWD.Source: F Validation of Acts Annotation · p. 19
  • PARTIAL, REFERENCE, NEGATIVE, and INFORM were the most challenging labels, each with F1 below 0.70.In more than half of 20 inspected disagreements per class, the author judged the LLM label more appropriate, possibly reflecting differences in label operationalization.Source: F Validation of Acts Annotation · p. 19

G Full Results

Agent comments were substantially longer than human comments, although history conditioning produced the shortest agent outputs. Agent comments also had lower communicative-intent density than human comments across conditions.

G Full Results

  • Human comments averaged approximately 48–78 words by language, while agent comments were longer; history conditioning produced the shortest agent comments.For Model A, history outputs averaged approximately 88 words, the closest approximation to human length.Source: G Full Results · p. 20
  • Act density ratios stayed below 1.0 across conditions, while history raised the ratio above 0.7 on average except for Model B.Behavior conditioning yielded the lowest ratios; the authors suggest its prescribed traits are dense in interactional information.Source: G Full Results · p. 20

H Information Density vs. Information Type

The profile-length analysis found no consistent metric improvement as profiles grew, supporting an information-type rather than information-quantity interpretation. A manual audit found generally grounded representations, while also identifying scope and privacy considerations.

H Information Density vs. Information Type

  • History length was not analyzed because most users had 16–20 prior comments; profile length served as a more interpretable proxy for input richness.Profiles were stratified into five word-count buckets, with buckets under 20 samples omitted and sparse longest buckets treated cautiously.Source: H Information Density vs. Information… · pp. 21–22
  • Across all 20 language–model pairs, profile length produced no consistent metric gains, providing evidence against an information-density explanation and supporting an information-type interpretation.The analysis does not fully isolate every possible confound.Source: H Information Density vs. Information… · pp. 22–23

I Qualitative Audit of Profiles and Behavioral Traits

  • The audit examined 50 users across profile faithfulness, trait-reasoning coherence, and privacy; all 50 profiles contained relevant information supported by their source pages.The authors characterize identified inaccuracies as isolated rather than substantial distortions.Source: I Qualitative Audit of Profiles and… · p. 23
  • When gender was not self-declared, it was typically inferred from gender-marked terms or given names; uptake of userbox information varied without a clear cross-language pattern.Source: I Qualitative Audit of Profiles and… · p. 23
  • Only 7 of 400 trait-reasoning pairs were reported as having labels traceable to their reasoning, although most rationales cited concrete comment-history excerpts.Source: I Qualitative Audit of Profiles and… · p. 23
  • No real legal names, contact details, or other directly identifying information were leaked, though the audit found a self-disclosed middle name and a reproduced username.The username was treated as a separate named entity in a German profile.Source: I Qualitative Audit of Profiles and… · p. 23
  • The findings concern the representations produced by this extractor and prompting design, not all possible persona representations; summarization may also lose stylistic examples from raw history.Source: I Qualitative Audit of Profiles and… · p. 24

Related papers

Loading 2610.08513v1…