PaperlayerView exact revision on arXiv

Source-linked AI summary

Enhancing Persona Consistency for LLMs' Role-Playing using Persona-Aware Contrastive Learning

Ke Ji, Yixin Lian, Linxu Li, Jingsheng Gao, Weiyuan Li, Bin Dai

arXiv:2503.17662v2cs.CL

TL;DR

LLMs can struggle to provide personalized role-playing, while role-play alignment faces costly annotation and diverse, implicit supervision. PCL uses persona self-questioning and contrastive self-play; its evaluations report stronger results than vanilla LLMs with comparable general knowledge.

  • Problem

    Role-playing requires personalized, consistent interactions, but collecting high-quality annotations is costly and role-enactment supervision is implicit and diverse.

    Source: §1 · pp. 1–2

  • Method

    PCL uses annotation-free persona self-questioning based on dialogue context and role characteristics, followed by contrastive training with and without role characteristics.

    Source: §1 · p. 2

  • Results

    PCL-equipped LLMs significantly outperform vanilla models under GPT-4 and CharEval evaluations while maintaining comparable general knowledge.

    Source: §1 · p. 2

  • Takeaways & Limitations

    The paper reports that PCL improves role alignment in evaluations while retaining competitive knowledge performance.

    Source: §6 · p. 9

Abstract

from arXiv · show

In recent years, large language models (LLMs) have achieved breakthrough progress in many dialogue generation tasks. However, their lack of emotion and fine-grained role awareness limits the model's ability to provide personalized and diverse interactions further. Current methods face high costs in collecting high-quality annotated data for scenarios such as role-playing, and traditional human alignment methods are difficult to deploy due to the inherent diversity of model behavior in role-playing scenarios. Inspired by the alignment of models for safety behaviors through RLHF (Reinforcement Learning from Human Feedback), in this paper, we revisit model role-playing behavior from the perspective of persona alignment and propose a novel annotation-free framework named \textbf{\underline{P}}ersona-Aware \textbf{\underline{C}}ontrastive \textbf{\underline{L}}earning (PCL) to align LLMs' behavior during role-playing, enhancing the model's role consistency. Specifically, we first design a role chain method to encourage the model to self-question based on the role characteristics and dialogue context to adjust personality consistency. Then, we further enhance the model's role-playing strategy through iterative contrastive learning between the use of role characteristics and not. Experiments on both black-box and white-box LLMs show that LLMs equipped with PCL significantly outperform vanilla LLMs under automatic evaluation methods (CharEval \& GPT-4) and human expert evaluation.

1 Introduction

Role-playing asks LLMs to sustain a specific persona, but costly annotations, implicit and diverse supervision, and potential losses in generalization complicate alignment. The paper proposes annotation-free Persona-Aware Contrastive Learning (PCL) and reports stronger role-playing evaluations than vanilla LLMs while maintaining comparable general knowledge.

Motivation

  • LLM role-playing requires consistent alignment with a given role profile, while earlier specialized designs lack versatility and scalability.The paper also identifies costly high-quality data collection, implicit and diverse supervision, and possible harms to generalization from role-specific fine-tuning as key challenges.Source: §1 · pp. 1–2
  • Earlier character-specific methods used customized architectures or pre-training, but these approaches can limit scalability or generalization.The paper notes that extensive character preference annotation requires deep character knowledge and can vary across annotators.Source: §1 · pp. 1–2

Contributions

  • PCL combines persona-based self-questioning with self-play contrastive training to align role-playing behavior without external annotation.The model reflects on dialogue context and role characteristics, then contrasts role-plays with and without those characteristics.Source: §1 · p. 2
  • PCL-equipped black- and white-box LLMs significantly outperform vanilla models under GPT-4 and CharEval evaluations while maintaining comparable general knowledge.The paper also reports human expert evaluation of role-playing quality.Source: §1 · p. 2

2 Preliminaries

The preliminaries define role-playing as generating a character-consistent response from a profile and dialogue context, and describe how the task is assessed across dialogue turns.

Role-Playing Conversational Agent

  • The role-playing conversational agent takes a character profile and current dialogue context as inputs and aims to generate a profile-consistent response.The context is represented as a sequence of dialogue utterances.Source: §2 · p. 2

Role-Playing Task

  • Role-playing tasks provide a character name or brief description and assess consistent self-awareness and nuanced character-specific knowledge across multiple dialogue rounds.Source: §2 · pp. 2–3

3 Related Work

Related work covers human-preference alignment, reasoning prompts in dialogue, and LLM role-playing, highlighting annotation challenges and gaps in role-playing proficiency.

3.1 Human Alignment

  • Preference-based alignment methods rely on human-annotated preferences, while role-specific annotation is costly and can vary across annotators.RLAIF replaces human labeling with off-the-shelf LLMs, but the resulting preference noise is described as unstable.Source: §3.1 · p. 3

3.2 Chain-of-Thought

  • Prior dialogue work explored chain-of-thought prompting, and this paper investigates its use in persona dialogue to enhance role consistency.Source: §3.2 · p. 3

3.3 Role-Playing

  • Role-playing can support vividness, interactivity, personalization, and complex tasks, but open-source LLMs still trail closed-source models such as GPT-4 in role-playing proficiency.Source: §3.3 · p. 3

4 Method

PCL first prompts persona-aware self-questioning, then contrasts role-plays with and without the character profile to align consistent behavior through self-play. For less capable models that cannot reliably follow the prompt, persona-aware training uses GPT-generated outputs.

4.1 Overview

  • PCL contrasts persona-conditioned and persona-free role-plays, then iteratively uses self-play to align the model’s consistent persona behavior.After each epoch, the model generates outputs for the next round without requiring additional external annotation.Source: §4.1 · p. 3, §4.4 · pp. 4–5

4.2 Chain of Persona Design

  • Chain of persona (COP) prompts the model to complete persona-aware self-reflections before producing its final role-playing response.The output includes self-questions and answers as intermediate steps, followed by the final response.Source: §4.2 · pp. 3–4

4.3 Persona-Aware Training

  • Less capable models that cannot reliably follow the COP prompt can be trained using 1,000 dialogue-and-profile samples with GPT-sampled, COP-formatted outputs.The persona-aware training objective is described as similar to vanilla supervised fine-tuning.Source: §4.3 · p. 4

4.4 Contrastive Self-Play Alignment

  • Contrastive self-play alignment forms preference pairs from role-plays with and without the character profile, then applies Direct Preference Optimization.The model regenerates outputs after each epoch for the next round of training.Source: §4.4 · pp. 4–5

5 Experiments

Experiments assess PCL on CharacterEval across general and unseen-role transfer settings, using varied LLM baselines and role-play evaluations. PCL improves role-playing across open-source and proprietary models, with additional analyses of knowledge retention, chain length, and method components.

5.1.1 Dataset and Experiments Setting

  • CharacterEval is a multi-turn role-playing benchmark; its general setting places all 77 character profiles in both train and test sets, while transfer uses 60 profiles for training and the remaining 17 for test.Source: §5.1.1 · p. 5

5.1.2 LLM baselines

  • PCL complements existing role-play methods rather than replacing them; evaluated baselines include ICL and vanilla SFT across proprietary and open-source models.The evaluated models include GPT-3.5, GPT-4, Qwen-7B, Baichuan2-7B, and CharacterGLM-6B.Source: §5.1.2 · pp. 5–6

5.2 Evaluation metrics; 5.3 Hyperparameters of PCL

  • Evaluation uses CharacterRM for role-play quality, adds conversational and character-consistency measures, and fixes PCL hyperparameters across experiments.Conversational ability includes fluency, coherency, and consistency.Source: §5.2 · p. 6, §5.3 · p. 7

5.4 Main Results

  • PCL improves role-playing in both open-source and proprietary LLMs, with small models reaching performance comparable to GPT-4.Baichuan2-7B and Qwen-7B surpass GPT-3.5 on character consistency, and conversational ability improves across all three metrics. GPT-3.5+PCL* gains the most attractiveness among baseline groups, while Baichuan2-7B+PCL remains competitive with GPT-4 on character consistency and attractiveness.Source: Table 3 · p. 7, §5.4 · pp. 7–8

5.5 PCL Preserves General Knowledge

  • A separate experiment evaluates general-knowledge retention under role-playing alignment, motivated by avoiding substantial loss of general knowledge.The passage identifies Table 4 as the evaluation across various tasks but does not state its results.Source: §5.5 · p. 8

5.6 Transferability Potential of Role-Playing

  • On unseen roles, PCL consistently improves role-playing ability for Qwen-7B and Baichuan-7B, indicating potential for zero-shot role transfer.The transfer test uses character profiles absent from training.Source: §5.6 · p. 8, Table 5 · p. 8

5.8 Ablation Study

  • The ablation finds CSPA adds more improvement when built on COP, while removing COP greatly reduces CSPA’s effect.The results are presented in Table 7.Source: Table 7 · p. 8, §5.8 · p. 8

5.7 Impact of Chain Length

  • For Qwen-7B, performance rises with chain length but declines beyond five, with redundant self-questioning offered as a possible explanation.The authors suggest redundant iterations may add noise and reduce output diversity.Source: §5.7 · p. 8, Table 6 · p. 8

6 Conclusion

PCL uses persona self-questioning without external annotated data, and the paper reports role-alignment effectiveness across expert and automated evaluations while maintaining competitive knowledge performance.

6 Conclusion

  • PCL prompts multiple rounds of persona-based self-questioning before generating dialogue, without external annotated data.The method aims to align responses with the character persona.Source: §6 · p. 8
  • Expert and automated evaluations validate role-alignment effectiveness while knowledge evaluation performance remains competitive.Source: §6 · p. 9

A Ethics Statement

  • The study uses publicly available datasets about fictional characters and includes no sensitive private information.Source: A Ethics Statement · p. 13

B Benchmark for Evaluating LLMs’ Knowledge

The benchmark appendix describes datasets used to broadly evaluate LLM knowledge, covering scientific, medical, open-domain, trivia, and reasoning questions.

B Benchmark for Evaluating LLMs’ Knowledge

C The Impact of Using Different COP Template

The appendix describes a template-variation experiment for Baichuan2-7B + PCL, with performance metrics reported for the tested templates.

C The Impact of Using Different COP Template

D Impact of external warmup data

The appendix tests whether PCL depends on external warmup data and reports that it still outperforms the baseline without externally generated warmup data, with a slight performance reduction.

D Impact of external warmup data

E Role-Playing Response Quality

Role-playing response quality is assessed through human expert preferences and GPT-4 pairwise judgments, with safeguards intended to improve evaluation consistency and fairness.

Evaluation overview

Human evaluation

  • Ten experts each judged 50 randomly selected samples, yielding 500 pairwise preferences between model responses.Evaluators received role-evaluation criteria and consistency training; response positions were randomized.Source: E Role-Playing Response Quality · p. 14

Automated evaluation

Loading 2503.17662v2…