PaperlayerView exact revision on arXiv

Source-linked AI summary

Back in Style: A Sociolinguistic Approach to Authoring and Measuring Persona Fidelity in User Simulation

Lex Konnelly, Elena Khasanova, Riqiang Wang, Matthias Lee, Harsh Saini, Parsa Kavehzadeh

arXiv:2610.10988v1cs.CL

TL;DR

User simulators can misrepresent agent capability across diverse populations, while persona fidelity is commonly assessed with costly, calibrated LLM judges. This paper specifies personas through observable stylistic rates and applies deterministic stylometric and lexicon-based diagnostics. The style-based schema improves adherence and distinguishability over a descriptive baseline for six of ten tested models, while the study’s model coverage leaves alignment effects unresolved.

  • Problem

    User-simulation fidelity matters because low fidelity can misrepresent agent capability across diverse populations, yet deterministic, spec-mapped measures are rarely primary fidelity signals and LLM judges are costly and require calibration.

    Source: §2.2 · p. 3

  • Method

    The paper authors personas as concrete stylistic rates and applies authorship-verification stylometry and lexicon-based content analysis as deterministic fidelity diagnostics.

    Source: §1 · pp. 1–2

  • Results

    For six of ten models, the sociolinguistically grounded definition schema improved stylistic adherence and inter-persona distinguishability over the descriptive baseline.

    Source: §5 · p. 6

  • Takeaways & Limitations

    The metrics are most useful for error-attribution analysis that localizes where fidelity breaks down, a first step toward interventions for more faithful simulations.

    Source: §5 · p. 6

    All tested models were instruction-tuned assistants, so the study cannot determine whether that alignment limits achievable persona fidelity; the planned user-aligned model was too unstable to include.

    Source: Limitations · pp. 6–7

Abstract

from arXiv · show

As agentic systems gain commercial popularity, user simulators increasingly serve as measurement instrument for their evaluation. However, the fidelity of simulated users in comparison to real human users is generally low, and typically assessed by costly, subjective LLM judges. In this pilot study, we ask whether fidelity can instead be measured deterministically by treating a user persona sociolinguistically: as a social type that emerges from observable linguistic style, rather than one predicted by labels or descriptions a model must extrapolate into behaviour. We author personas as concrete stylistic rates, which lets us transfer two established, model-free instruments -- authorship-verification stylometry and lexicon-based content analysis -- as fidelity diagnostics. We A/B-test the sociolinguistic schema against a flat descriptive baseline across five task-oriented customer-service agents. Results show that the sociolinguistic schema improves both stylistic adherence and stylometric distinguishability for most of the tested models, with a caveat that persona style fidelity does not necessarily equal persona "naturalness". We argue that a sociolinguistic approach to persona design is a promising path towards more diverse and representative user personas, and that these metrics are most valuable in an error-attribution analysis, localizing where fidelity breaks down. This is a first step towards interventions that move user simulations closer to faithful renderings of diverse and variable linguistic outputs.

1 Introduction

Because simulated users can misrepresent agent capability across diverse populations, the paper asks whether persona fidelity can be measured deterministically from observable linguistic style. It tests a sociolinguistic schema and model-free metrics against descriptive personas across task-oriented agents.

1 Introduction

  • Low-fidelity user simulation can misrepresent real-world agent capability, particularly across diverse user populations.The paper therefore treats simulator fidelity as a core priority in agentic evaluation.Source: §1 · p. 1
  • Sociolinguistic personas specify observable stylistic rates, such as hedging, politeness, fillers, and emoji use, rather than asking models to extrapolate from labels.The approach treats persona identity as emerging from recurrent linguistic choices and discursive stance-taking.Source: §1 · pp. 1–2
  • The study tests whether authorship-verification stylometry and lexicon-based content analysis can measure fidelity without an LLM judge or human annotator.The authors also ask whether the metrics can localize which components of a persona’s linguistic structure are rendered faithfully.Source: §1 · pp. 1–2
  • The pilot compares a style-based schema with a descriptive baseline across five customer-service agents, text and voice modes, and ten simulator models.The models span providers, capabilities, and open- and closed-source alternatives.Source: §1 · pp. 1–2
  • The sociolinguistic schema improves stylistic faithfulness and inter-persona distinguishability for most tested models, with the strongest effects for gpt-5-mini and effects also holding in voice.Gemini-3.1-pro-preview and Gemini-3.5-flash showed little schema effect; qualitative review also found that style fidelity does not necessarily equal naturalness.Source: §1 · p. 2
  • A schema×model factorial tests whether fidelity gaps reflect underspecified personas or simulator steerability, and whether the schema effect replicates in voice.The paper’s contributions include a sociolinguistic persona schema and two deterministic, spec-mapped fidelity diagnostics.Source: §1 · p. 2

2 Related work

Prior persona simulations commonly use descriptions or trait controls that do not specify the observable quantity a fidelity metric should measure. This paper frames persona as a social type realized in language and situates deterministic, style-based fidelity measures against reliance on LLM judges.

2.1 Defining a user persona for simulation

  • Common simulation approaches use profile or attribute lists, or control a single trait, but neither explicitly specifies the quantity a fidelity metric should measure.Role-steering work also reports collapse toward a default assistant voice under sustained interaction.Source: §2.1 · p. 2
  • In the sociolinguistic account, personas emerge as durable styles from recurrent linguistic choices and stance-taking, rather than residing behind the words.A social type is relational, formed through how speakers position themselves toward talk and others, and is not reducible to demographic identity.Source: §2.1 · p. 2

2.2 Evaluating persona adherence

  • Task-oriented simulation fidelity is commonly judged by LLMs, while deterministic, model-free measures rarely serve as the primary fidelity signal.The paper identifies deterministic, spec-mapped stylometric measures as a gap for multi-attribute personas in customer-service simulation across text and voice.Source: §2.2 · pp. 2–3
  • The illustrated personas contrast self-blame, hedges, filler, and an apologetic emoticon with company-blame, a negative epithet, and a bare imperative.The contrast makes stance observable through surface linguistic forms.Source: §2.2 · p. 3, Figure 1 · p. 3

3 Methodology

The methodology holds the evaluated agent fixed and measures only simulated-user turns. It defines observable stylistic personas, applies complementary deterministic metrics, and tests schema and model effects across text and voice settings.

3.1 Evaluation system

  • The evaluation pipeline has a user simulator interact with an agent harness, but this study holds the agent fixed and evaluates only simulated-user turns in the transcript.The simulator receives a persona schema, task scenario, and scenario-specific details before a multi-turn dialogue.Source: §3.1 · p. 3

3.2 Sociolinguistic personas

  • The definition schema assigns eight personas ordinal rate targets for eight linguistic features, with modality-specific targets and instructions to counter default assistant behavior.The same personas are described in the archetype baseline through adjectival traits, communication style, motivations, and example phrases.Source: §3.2 · p. 3

3.3.1 Stylometric distinguishability (authorship-verification ROC–AUC)

  • Stylometric distinguishability uses character n-gram TF-IDF and cosine similarity to predict whether conversation pairs share a persona, reporting ROC-AUC.AUC near 0.5 indicates indistinguishable personas and 1.0 indicates perfect separability; the fixed, feature-agnostic transform is paired with bootstrap intervals and a label-permutation control.Source: §3.3.1 · p. 4

3.3.2 Stylistic adherence (Kendall’s τb)

  • Stylistic adherence tests whether observed feature rates rise with specified persona levels using Kendall’s τb between ordinal specifications and observed rates.It complements distinguishability, which measures whether personas differ but not whether they differ as specified; the metrics are read jointly to localize fidelity breakdowns.Source: §3.3.2 · p. 4

3.4 Experimental design

  • A factorial design crosses schema and simulator model, then tests the schema comparison in voice; experiments use eight personas, five agents across five service verticals, and five reused scenarios.The text arm also varies simulator model, while voice tests whether the schema effect replicates in a realtime pipeline.Source: §3.4 · pp. 4–5

4 Results

The definition schema improves stylometric distinguishability across all five agents and both modalities, while adherence gains occur for most models but not Gemini or Gemma. Distinguishability gains survive a verbosity control, and adherence gains concentrate in fine-grained style markers.

4.1 Stylometric distinguishability

  • Model capability did not reliably predict distinguishability: Claude Opus-5 had the highest definition score, but Haiku-4-5 and GPT-5-mini exceeded it in schema delta, while Gemini and Gemma performed poorly.All three Gemini configurations and Gemma scored below chance in both schema arms.Source: §4.1 · p. 5
  • Text distinguishability AUC improves for all five agents, from near chance under the archetype baseline to moderate under the definition schema, with the pattern holding across text and voice.Distinguishability measures whether personas exhibit distinct stylistic variations from one another.Source: Table 1 · p. 5, §4.1 · p. 5
  • Truncating every sample to a common 150-character budget leaves the definition schema’s overall distinguishability advantage unchanged.Source: §4.1 · p. 5

4.2 Stylistic adherence

  • The definition schema yields stronger adherence for most models in text and voice, with Gemini-family models and Gemma essentially flat; voice averages six realizable non-orthographic features.Source: §4.2 · p. 5, Table 2 · p. 6
  • Adherence gains concentrate in fine-grained markers: CMC acronyms reach +0.838 for GPT-5-mini, all-caps +0.73 for Haiku, and voice hedging and fillers gain +0.47 and +0.42.Features already common in a helpful register, such as politeness, barely move.Source: §4.2 · p. 6

5 Discussion and conclusion

The definition schema improved both stylistic adherence and inter-persona distinguishability over descriptive archetypes for six of ten models. The metrics also help identify formulaic realizations and overlapping persona definitions, making them useful for diagnosing and refining fidelity.

5 Discussion and conclusion

  • The definition schema improved both spec adherence and inter-persona distinguishability over the descriptive baseline for six of ten models.The authors report that the metrics also localized schema effects to specific stylistic features.Source: §5 · p. 6
  • Rate-based adherence does not ensure natural-sounding style: some models rendered politeness and disfluency formulaically.The authors suggest lexical-diversity measures or conditioning on dialog acts as partial remedies, while noting that deterministic scoring still struggles to capture persona authenticity.Source: §5 · p. 6
  • Low pairwise separability can reveal persona definitions that converge in observable style and should be pulled apart to increase diversity.The authors describe overlap between a chatty “social” persona and a rambling “storyteller,” and between an anxious persona and a technophobic one.Source: §5 · p. 6
  • The metrics are most useful for error attribution, turning fidelity into a reproducible, feature-level signal that localizes where it breaks down.The authors position this as a first step toward interventions for more faithful renderings of diverse and variable linguistic outputs.Source: §5 · p. 6

Limitations

The study’s evidence is bounded by its pilot scale, model coverage, operationalization of style, and one-shot persona prompting. Persona distinguishability also depends substantially on conversation length.

Limitations

  • The pilot covers five customer-service agents on one platform, with eight personas and five conversations per persona per arm; overlapping per-agent intervals make the result rest on cross-agent consistency.The authors note that no single agent shows a significant effect on its own.Source: Limitations · p. 6
  • All tested models are instruction-tuned and aligned as “helpful assistants,” so the study cannot determine whether that alignment limits persona fidelity.A planned user-aligned USERLM-8B model was left for future work because it was too unstable as a next-turn predictor.Source: Limitations · pp. 6–7
  • The style measures cover eight engineered features and a character-ngram fingerprint, not task success, semantic appropriateness, or stylistic dimensions outside that feature set.The authors propose testing broader markers and isolating effects of specific markers and instructions in follow-up work.Source: Limitations · p. 7
  • Persona instructions appear only once at the start, with no in-dialogue reinforcement, so the findings characterize fidelity under one-shot prompting.The authors reserve testing live, in-conversation persona steering for future research.Source: Limitations · p. 7
  • Transcript length correlates with definition-arm AUC (r = 0.79) and schema effect (r = 0.73), so distinguishability reflects both schema and elicited speech.The authors plan to incorporate length normalization to better isolate the schema effect.Source: Limitations · p. 7

Ethics Statement

The ethics statement reports no human-subjects data and describes licensed assets and API use. The appendix contrasts a label-based archetype with a sociolinguistic definition using stance and modality-specific style rates.

Ethics Statement

  • The study uses LLM-simulated users and hand-authored lexicons, with no human-subjects data; assets follow stated licenses and provider APIs.Preference-alignment annotation was performed in-house, without external annotation services.Source: Ethics Statement · p. 7

A Example persona schemas: archetype vs. definition

  • The example contrasts verbatim simulator inputs: an archetype built from labels, prose, and example phrases, and a definition built from stance and modality-specific ordinal style rates.The rates are compiled into bands in the definition schema.Source: A Example persona schemas: archetype… · p. 8

A.1 Archetype schema (label-based baseline)

  • The archetype describes The Anxious Inquirer through hesitation, reassurance-seeking, communication-style settings, motivations, and example questions.Its example questions ask for confirmation and reassurance before acting.Source: §A.1 · p. 8

A.2 Definition schema (sociolinguistic, rate-based)

  • The definition specifies the persona with stance and identity fields, modality-specific linguistic and voice rates, behavioral contracts, and backstory.Its linguistic settings include high hedging and politeness-marker rates, while voice settings include pitch, pace, and disfluency tendencies.Source: §A.2 · pp. 8–9

B Ablation test without behavioural contracts block

In an exploratory gpt-5-mini ablation, removing behavioral contracts lowered every fidelity metric, though scores remained above the archetype baseline. The authors attribute a role to both stylistic rates and contracts.

B Ablation test without behavioural contracts block

  • The ablation removes the behavioral-contracts block while holding the rest of the definition schema fixed, testing gpt-5-mini on all five agents.The comparison uses conversations clean in both variants; the archetype baseline is included for reference.Source: B Ablation test without behavioural… · p. 9
  • The authors limit the finding to assistant-aligned models tested and plan to test whether user-roleplay-aligned models also require the additional instruction.They report that rate specifications alone still lift metric scores substantially above the archetype baseline.Source: B Ablation test without behavioural… · p. 9
  • Removing behavioral contracts lowered every metric score, but scores without the block remained significantly above the archetype baseline (one-sided Wilcoxon p = 0.031).The authors say contracts help realize stylistic features and counter regression to the helpful-assistant voice.Source: Table 3 · p. 9, B Ablation test without behavioural… · p. 9

C Per-persona distinguishability

The per-persona decomposition uses within- and between-persona similarity to characterize separability. Tables report this decomposition for definition-arm personas across three model groupings.

C Per-persona distinguishability

  • Per-persona separability is the gap between same-persona similarity and similarity to other personas; a large positive gap indicates a sharply drawn persona.Because conversations share a customer-service register and have uniformly high absolute similarities, persona-specific signal is carried by the gap.Source: C Per-persona distinguishability · p. 9
  • Table 4 reports definition-arm per-persona separability for OpenAI models, averaged over five agents.The decomposition reports same, diff, and gap for each persona.Source: C Per-persona distinguishability · p. 9, Table 4 · p. 10
  • Tables 5 and 6 report the same definition-arm per-persona separability decomposition for Gemini and Gemma models, and for Claude models, respectively.Both tables report means over five agents.Source: C Per-persona distinguishability · p. 9, Table 5 · p. 10, Table 6 · p. 10

D Per-feature adherence

The per-feature adherence tables report Kendall’s τb by feature for different simulator model groups. CMC acronyms and all-caps are text-only markers and are N/A for voice.

D Per-feature adherence

  • Tables 7–9 report per-feature adherence (τb, per-persona) for OpenAI, Gemini and Gemma, and Claude models, with means over five agents.The reported values cover eight authored features and distinguish archetype, definition, and their gap.Source: D Per-feature adherence · p. 10, Table 7 · p. 11, Table 8 · p. 11
  • CMC acronyms and all-caps are text-only orthographic markers, so the tables report them as N/A for the voice channel.The voice-channel adherence average covers six features that can be realized there.Source: D Per-feature adherence · p. 10

E Human Preference Analysis

Human preferences generally tracked the automated metrics at their extremes, favoring the definition schema for the strongest metric gains and showing little preference when gains were near zero. Persona-level results included exceptions and should be treated cautiously because the sample was small and inconsistent.

E Human Preference Analysis

  • Two annotators rated 10 randomly sampled transcript pairs per model, choosing archetype, definition, or no preference under otherwise identical settings.Source: E Human Preference Analysis · p. 11
  • For claude-opus-5 (∆AUC = +0.105) and claude-sonnet-5 (∆AUC = +0.095), annotators mostly preferred the definition schema.claude-opus-5 had the study’s largest adherence gain.Source: E Human Preference Analysis · p. 11
  • gemini-3.5-flash (low) drew “no preference” in 85% of cases, while gemini-3.5-flash (high) drew definition-schema preference in 62% despite ∆AUC = +0.001.Source: E Human Preference Analysis · p. 11
  • tech_savvy_expert and busy_multitasker were mostly rated “no preference”; frustrated_complainer leaned toward archetype despite modest automated gains for definition, with small, inconsistent samples.The authors caution that these persona-level findings should be taken with a grain of salt.Source: E Human Preference Analysis · p. 11
Loading 2610.10988v1…