PaperlayerView exact revision on arXiv

Source-linked AI summary

Faithful Persona-based Conversational Dataset Generation with Large Language Models

Pegah Jandaghi, XiangHai Sheng, Xinyi Bai, Jay Pujara, Hakim Sidahmed

arXiv:2312.10007v1cs.CLcs.LG

TL;DR

Existing persona-based datasets have limitations including a lack of guaranteed faithfulness. This paper uses an LLM-based framework to generate and refine persona-based conversational data. Across iterations, Synthetic-Persona-Chat’s losing rate in the Turing test falls by 48% and below 10%.

  • Problem

    Existing persona-based datasets have limitations including small size, static content, and no guarantee that conversations faithfully align with user personas.

    Source: §1 · pp. 1–2

  • Method

    An unsupervised LLM framework generates, expands, and updates personas and uses conversation quality criteria, including faithfulness, to reduce human labor.

    Source: §1 · p. 2

  • Results

    48% reduction in Synthetic-Persona-Chat’s losing rate from Iteration 1 to Iteration 3, falling below 10%.

    Source: §4.3 · p. 7

  • Takeaways & Limitations

    Synthetic-Persona-Chat is intended to support persona-aware agents, and the Generator-Critic approach may generalize to other specialized dataset-generation tasks.

    Source: §6 · p. 8

    The framework requires computational resources and LLM access, assumes fluent speakers and consistent conversation flow without interruptions, and has difficulty incorporating less tangible traits.

    Source: Limitations · pp. 8–9

Abstract

from arXiv · show

High-quality conversational datasets are essential for developing AI models that can communicate with users. One way to foster deeper interactions between a chatbot and its user is through personas, aspects of the user's character that provide insights into their personality, motivations, and behaviors. Training Natural Language Processing (NLP) models on a diverse and comprehensive persona-based dataset can lead to conversational models that create a deeper connection with the user, and maintain their engagement. In this paper, we leverage the power of Large Language Models (LLMs) to create a large, high-quality conversational dataset from a seed dataset. We propose a Generator-Critic architecture framework to expand the initial dataset, while improving the quality of its conversations. The Generator is an LLM prompted to output conversations. The Critic consists of a mixture of expert LLMs that control the quality of the generated conversations. These experts select the best generated conversations, which we then use to improve the Generator. We release Synthetic-Persona-Chat, consisting of 20k conversations seeded from Persona-Chat. We evaluate the quality of Synthetic-Persona-Chat and our generation framework on different dimensions through extensive experiments, and observe that the losing rate of Synthetic-Persona-Chat against Persona-Chat during Turing test decreases from 17.2% to 8.8% over three iterations.

1 Introduction

Existing persona-based conversational datasets are limited by small size, static content, quality concerns, and lack of guaranteed faithfulness. The paper introduces an LLM framework for generating customized persona-based datasets and releases Synthetic-Persona-Chat.

1 Introduction

  • Existing persona-based datasets can be small, static, labor-intensive to create and update, and may contain irrelevant or contradictory utterances without guaranteed faithfulness.The proposed framework uses unsupervised LLMs to reduce human labor and applies quality criteria including faithfulness.Source: §1 · pp. 1–2

3. Conversation Generation

  • Synthetic-Persona-Chat contains 5k user personas and 20k faithful dialogues, and the framework can be reused for application-specific datasets.The paper identifies generating and extending specialized personas and evolving datasets under different objectives as framework contributions.Source: §3 · p. 2

2 Definitions

  • Faithfulness requires that conversation utterances do not reduce the inference probability of either speaker’s persona attributes relative to their prior probabilities.The figure contrasts a conversation about loving steak with a vegetarian profile against one that does not contradict or weaken the profile.Source: §2 · p. 3, Figure 1 · p. 3

3 Method

The method generates persona-based conversations from an initial dataset with minimal human input, using user generation, user pairing, and conversation generation.

3 Method

  • The method begins with an initial dataset and proceeds through user generation, user pairing, and conversation generation.Figure 2 is captioned as the Dataset Augmentation Pipeline.Source: §3 · p. 3, Figure 2 · p. 3
  • The three components expand seed persona attributes into user profiles, pair profiles as interlocutors, and iteratively generate conversations between profile pairs.These components are presented as the stages of the dataset-generation process.Source: §3 · p. 3

3.1 User Generation

  • User Generation is divided into two subcomponents, which this passage introduces without specifying their details.The section proceeds to describe the components below.Source: §3.1 · p. 3

1. Persona Expansion

The framework expands seed persona attributes, builds consistent user profiles, pairs similar users, and iteratively generates conversations using critic-selected quality policies.

3.1.1 Persona Expansion

  • Query induction expands seed-category queries to cover unobserved categories, then uses category-specific prompts to generate new persona attributes until the set reaches 5k.The prompts combine sampled attributes with queries; BERT embeddings are used to avoid adding attributes too close to existing ones.Source: §3.1.1 · p. 4

3.1.2 User Profile Construction

  • User profiles are built from expanded attributes, admitting only attributes that are consistent with existing ones and not redundant, until each profile has five attributes.An NLI model checks consistency, while persona-attribute clustering is used to assess redundancy.Source: §3.1.2 · pp. 4–5

3.2 User Pairing

  • User pairs are selected when their profiles share more than two persona categories, with similarity measured by the number of shared categories.BERT represents the profiles for the similarity calculation.Source: §3.2 · p. 5

3.3 Conversation Generation

  • The Generator creates candidate conversations for paired profiles from a few examples, and the Critic evaluates them against predefined conversation-quality policies.Selected candidates are added to the dataset for the next generation iteration.Source: §3.3 · p. 5
  • The Critic uses separate experts for general conversation quality, persona faithfulness, and toxicity, filtering unfaithful and toxic conversations before selection.General-quality experts compare candidates pairwise using FED-derived policies; the faithfulness expert checks consistency with user profiles, and the toxicity expert detects harmful traits.Source: §3.3 · pp. 5–6
  • Among the conversations that pass those filters, the Critic selects the best using a majority vote among the general conversation-quality experts.The selected instances become data for the Generator’s next iteration.Source: §3.3 · p. 6

4 Evaluation

The evaluation examines expanded personas, next-utterance prediction, and human judgments of Synthetic-Persona-Chat (SPC) against Persona-Chat (PC). It reports gains in persona coverage, prediction performance, and judged human-likeness across generation iterations.

4.1 Evaluation of the Expanded Personas

  • Persona expansion increases SPC’s attribute count by 119%, categories by 71% over PC, and new-attribute length by 107%.For Wikipedia personas, the corresponding increases are 108% in attribute count, 140% in categories, and 45% in attribute length.Source: Table 1 · p. 6, §4.1 · p. 6

4.2 Next Utterance Prediction

  • The Transformer (Ranker) trained on SPC achieves hit@1 of 64.24 on SPC test, 350% higher than PC (14.24).The table caption identifies Persona-Chat test results as the unparenthesized values and Synthetic-Persona-Chat test results as the values in parentheses.Source: Table 2 · p. 7, §4.2 · p. 7
  • Persona conditioning raises the IR Baseline by 31% on SPC versus 97% on PC; training on SPC and testing on PC yields a 104% improvement.The authors attribute PC’s larger persona-conditioned retrieval gain partly to participants copying persona words; SPC reflects personas more implicitly, making word-based retrieval more challenging.Source: §4.2 · p. 7
  • With personas as input, generative models show a 3% performance drop when trained on SPC versus 20% on PC, and perplexity rises 9% versus 41%.The authors interpret the smaller drops and perplexity increase as evidence of higher conversation-persona alignment in SPC.Source: §4.2 · p. 7
  • Results using no, one, or both users’ personas suggest greater bidirectionality in SPC.The passage refers readers to Appendix B.1 for details.Source: §4.2 · p. 7

4.3 Human Evaluation

  • SPC’s Turing-test losing rate falls 48% from iteration 1 to iteration 3 and below 10%, while faithfulness stays above 75% with at most 3% variation.The table covers 200 generated conversations per iteration; 91% of SPC conversations were judged as human-like as human-generated conversations, and only 8.04% of conversations for new personas were judged artificial.Source: §4.3 · p. 7, Table 3 · p. 8
  • After two iterations, the smaller-model multi-iteration approach outperforms the larger 540-billion-parameter LLM’s single-iteration approach on the Turing test.The 540-billion-parameter model has a 5% advantage over the smaller model’s first iteration.Source: §4.3 · p. 8

5 Related Work

Related work covers LLM-based data creation and augmentation, conversational dataset generation, and persona-based conversation research. It also notes challenges in ensuring and evaluating synthetic conversational data quality.

5 Related Work

6 Conclusion and Future Work

The framework produces Synthetic-Persona-Chat, a 20k-conversation dataset, and the authors describe the Generator-Critic approach as reusable beyond persona-based data. They identify constraints in the model, conversation assumptions, and persona traits, and discuss potential misuse and rater protections.

6 Conclusion and Future Work

  • Synthetic-Persona-Chat contains 20k conversations, and the Generator-Critic approach may extend to other specialized dataset-generation tasks.The authors also identify domain-specific, multisession conversations as a possible future direction.Source: §6 · p. 8

Limitations

  • Dataset quality is bounded by the LLM, while realistic generation assumes fluent speakers, perfectly consistent flow, and no unexpected events.The framework also requires computational resources and access to an LLM; improving the critics is left for future work.Source: Limitations · p. 8
  • The method has difficulty incorporating less tangible traits, such as humor, and attributes that require multiple conversation sessions to emerge.Source: Limitations · p. 9

Ethics Statement

  • The authors note that objective-driven dataset generation could support harmful datasets and malicious models, while also describing rater protections and toxic-conversation filtering.Human raters participated voluntarily, were informed of their rights, and received above-minimum compensation; potentially toxic conversations were filtered from the final dataset.Source: Ethics Statement · p. 9

A Dataset Generation Framework

The framework expands seed personas, constructs user profiles, and generates conversations with an LLM-based mixture-of-experts critic. The appendix describes the generation settings and persona checks, and reports that the general-quality experts rank conversation pairs with over 80% accuracy.

A Dataset Generation Framework

A.1 User Generation

  • Persona expansion clusters BERT-based attribute representations by average cosine similarity, joining clusters above a 0.1 threshold before bootstrapping queries for 100 steps.The process samples three attributes per cluster to create initial queries, then samples five queries at each bootstrapping step.Source: §A.1 · p. 12
  • Candidate profile attributes are rejected as redundant above 0.9 cosine similarity or as inconsistent when an NLI check detects a contradiction.The consistency check compares LLM-generated negations of profile attributes against the candidate attribute and its negation.Source: §A.1 · p. 13

A.2 Conversation Generation

  • The general-quality critic experts are more than 80% accurate at distinguishing the better conversation in pairs from the FED baseline.The experts assess specific policies through prompts to an LLM.Source: §A.2 · p. 13
  • The critic also includes toxicity and persona-faithfulness experts, with curated faithful and unfaithful examples supplied in the faithfulness prompt.Source: §A.2 · p. 13

B Synthetic-Persona-Chat

Synthetic-Persona-Chat contains 20k conversations and is evaluated against Persona-Chat using automatic measures of toxicity, diversity, task performance, and conversation quality. Across these evaluations, SPC outperforms PC on the reported quality dimensions, with some qualifications across metrics and tasks.

B Synthetic-Persona-Chat

  • 20k conversations averaging 11.8 turns per user make up SPC, which is compared with PC using automatic and human assessments.Source: B Synthetic-Persona-Chat · p. 13

B.1 Automatic Evaluation

  • PC has more than 50 strongly toxic samples, whereas SPC has at most three; medium-profanity and toxicity conversations are four times less frequent in SPC.Conversations marked strongly toxic were removed from the released SPC dataset.Source: §B.1 · pp. 13–14, Table 9 · p. 17
  • SPC has similar topic diversity to PC at the most general level, but slightly lower diversity at deeper levels.The comparison uses conversations with similar personas in PC.Source: §B.1 · p. 14, Table 11 · p. 18
  • For next-utterance prediction, persona-conditioned models trained on PC improve most with self-personas, a pattern not observed for SPC.The authors interpret this difference as indicating greater bidirectionality in SPC conversations than in PC conversations.Source: §B.1 · pp. 14, 18, Table 12 · p. 19
  • Profile-extraction methods perform better on SPC in three of four scenarios, and both datasets score better when the target user differs from the speaker.The task ranks persona attributes using user utterances, with attributes from other profiles included as distractors.Source: §B.1 · p. 18, Table 13 · p. 19
  • At least 23 points of average improvement appear on each evaluated criterion with GPT-Score, and SPC consistently outperforms PC across evaluated dimensions.Source: §B.1 · p. 19

B.2 Human Evaluation

The human evaluation uses a Turing test to compare SPC with PC and a separate faithfulness study that asks annotators to infer persona attributes from conversations. The supplied passages describe the study designs and measures, but do not report their outcome values.

B.2 Human Evaluation

  • Both human studies are conducted at the end of each dataset-generation iteration.Source: §B.2 · p. 19
  • In the Turing test, annotators identify which of two conversations for a user pair was synthetically generated, with SPC losing rates and Fleiss’ Kappa reported.The sample contains 200 user pairs from PC; the reported agreement is described as fair to moderate.Source: §B.2 · p. 19, Table 16 · p. 22
  • The faithfulness study asks annotators to select inferred persona attributes from eight options, four real and four distractor attributes.Distractors include negated, randomly selected, and contradicting personas; annotator precision is used as a proxy for conversation faithfulness.Source: §B.2 · pp. 19, 22

C Ablation Studies

The ablation studies examine the persona expansion module and the experts in the Critic’s mixture.

C Ablation Studies

  • The ablations assess the importance of individual framework components.They examine the persona expansion module and the impact of each expert in the mixture forming the Critic.Source: C Ablation Studies · p. 22
  • The study begins with the persona expansion module before reviewing the impact of each expert in the mixture.Source: C Ablation Studies · p. 22

C.1 Persona Expansion

The ablation indicates that query-based persona expansion increases the number and diversity of personas, with attribute-length effects differing between datasets.

C.1 Persona Expansion

  • Expansions without the inducted query set (Q) were labeled Wikipedia-0 and PC-0, and each dataset was run for 100 iterations.Source: §C.1 · p. 22
  • PC-0 and Wikipedia-0 had 6% and 49% fewer persona clusters, respectively, suggesting query-based expansion maintains persona-set diversity.Source: §C.1 · p. 22
  • PC-0’s average persona attribute length was 11.38 tokens, 28% less than SPC, while Wikipedia-0 and Wikipedia+ lengths were similar.Source: §C.1 · p. 22
  • 4,477 PC-0 attributes were 20% fewer than PC, while 4,742 Wikipedia-0 attributes were 50% fewer than Wikipedia+.Source: §C.1 · p. 22, Table 17 · p. 23

C.2 Conversation Quality

The ablations find that expert removal generally worsens conversation-quality evaluations and faithfulness judgments, while effects on next-utterance prediction vary by model type.

C.2 Conversation Quality

  • Removing experts worsened most conversation-quality criteria: 3 of 4 in LLM-Eval, 4 of 6 in GPT-Score, and 3 of 5 in G-Eval.Source: §C.2 · p. 24

C.3 Faithfulness

  • Without the faithfulness expert, Turing-test precision fell from 78.0% to 66.0% and recall from 36.0% to 23.0% across 200 conversations.The authors attribute the precision result to eliminating contradictory persona information and the recall result to greater persona reflection when the expert is present.Source: §C.3 · p. 24
  • The LLM-based faithfulness evaluation covered the full dataset and used a different LLM for annotation than for generation.The annotator was gpt-3.5-turbo.Source: §C.3 · p. 24

C.4 Next Utterance Prediction

  • −12% hit@1, −9% BLEU, and −10% ROUGE were observed for generative models when FED was absent.The comparison evaluated next-utterance models trained on SPC against models trained on datasets created without selected experts.Source: §C.4 · p. 24, Table 19 · p. 25
  • The IR Baseline peaked at 39% hit@1 without FED, whereas the Transformer Ranker peaked at 13.9% hit@1 with all experts present.Source: §C.4 · p. 24, Table 19 · p. 25
Loading 2312.10007v1…