PaperlayerView exact revision on arXiv

Source-linked AI summary

Large Language Models are Superpositions of All Characters: Attaining Arbitrary Role-play via Self-Alignment

Keming Lu, Bowen Yu, Chang Zhou, Jingren Zhou

arXiv:2401.12474v1cs.CLcs.LG

TL;DR

General-purpose LLMs can lack qualities needed for engaging, extensive role-play conversations. DITTO self-aligns an instruction-following model using character knowledge and simulated dialogue, and across four model sizes it outperforms open-source role-play models with performance comparable to GPT-4-turbo.

  • Problem

    General-purpose LLMs can lack experiential events and emotions, limiting their ability to support engaging, extensive conversations.

    Source: §1 · p. 1

  • Method

    DITTO uses character knowledge and self-simulated dialogue to fine-tune an instruction-following LLM, without distilling outputs from a more potent role-play model.

    Source: §1 · p. 2, §3.2 · p. 4

  • Results

    Across four sizes, DITTO outperforms existing open-source role-play models and reaches performance comparable to GPT-4-turbo.

    Source: §7 · p. 10

  • Takeaways & Limitations

    Cross-supervision results indicate role-specific knowledge is bounded by seed-model capabilities, while role-play styles can generalize from weaker supervision.

    Source: §1 · p. 3, §6.2 · p. 10

    The best Qwen-72B-Chat DITTO model remains behind GPT-4 and GPT-4-Turbo, and its self-generated training data contains noticeable noise.

    Source: Limitations · p. 10

Abstract

from arXiv · show

Considerable efforts have been invested in augmenting the role-playing proficiency of open-source large language models (LLMs) by emulating proprietary counterparts. Nevertheless, we posit that LLMs inherently harbor role-play capabilities, owing to the extensive knowledge of characters and potential dialogues ingrained in their vast training corpora. Thus, in this study, we introduce Ditto, a self-alignment method for role-play. Ditto capitalizes on character knowledge, encouraging an instruction-following LLM to simulate role-play dialogues as a variant of reading comprehension. This method creates a role-play training set comprising 4,000 characters, surpassing the scale of currently available datasets by tenfold regarding the number of roles. Subsequently, we fine-tune the LLM using this self-generated dataset to augment its role-playing capabilities. Upon evaluating our meticulously constructed and reproducible role-play benchmark and the roleplay subset of MT-Bench, Ditto, in various parameter scales, consistently maintains a consistent role identity and provides accurate role-specific knowledge in multi-turn role-play conversations. Notably, it outperforms all open-source role-play baselines, showcasing performance levels comparable to advanced proprietary chatbots. Furthermore, we present the first comprehensive cross-supervision alignment experiment in the role-play domain, revealing that the intrinsic capabilities of LLMs confine the knowledge within role-play. Meanwhile, the role-play styles can be easily acquired with the guidance of smaller models. We open-source related resources at https://github.com/OFA-Sys/Ditto.

1 Introduction

DITTO treats role-play as an ability that can be elicited from knowledge already encoded in LLMs, rather than requiring outputs from a stronger role-play model. The paper also introduces an efficient, reproducible evaluation and reports strong results against open-source role-play systems.

Motivation

  • General-purpose LLMs can lack experiential events and emotions, limiting their ability to support engaging, extensive conversations.Prior imitation approaches can mimic GPT-4’s style yet fall short in factuality and may increase hallucination.Source: §1 · pp. 1–2

Method and contribution

  • DITTO uses character profiles and simulated dialogues to elicit intrinsic role knowledge, and WIKIROLE covers 4,000 characters—ten times the role count of publicly available role-play datasets.The method does not require distilling outputs from more potent role-play models.Source: §1 · p. 2

Results

  • On evaluation, DITTO based on Qwen-72B-Chat achieves 90% role identity consistency and its general performance is on par with GPT-3.5-Turbo, but slightly falls short of GPT-4 and Qwen-Max on accurate role-related knowledge.Qwen-Max achieves the highest scores on the current role-play benchmark and the role-play subset of MT-Bench.Source: §1 · p. 2

Contributions

  • The paper proposes objective role-play evaluation of identity consistency, role-related knowledge, and cognitive boundaries, alongside cross-supervision analysis of role-play capabilities.The evaluation is designed to be reproducible, explainable, and efficient compared with manual annotations.Source: §1 · pp. 2–3

2 Related Works

Prior role-play work used prompting, human annotation, or outputs from proprietary models to create character dialogue datasets. DITTO instead builds its training data through self-alignment and separates character knowledge from conversational style.

Role-play

  • DITTO builds role-play training data through self-alignment rather than imitating proprietary LLMs, separating character knowledge from conversation style and scaling to 4,000 roles.The authors describe this as a multilingual dataset with 16 times the role count of previous works.Source: §2 · p. 3
  • Earlier role-play datasets used prompting or proprietary-model-generated dialogue, with examples ranging from 77 to 250 characters.The cited work includes datasets built from novels and scripts, human-annotated data, and generated dialogues.Source: §2 · p. 3

Self-alignment

  • Prior work notes that imitation can reproduce GPT’s style while weakening factuality and fostering hallucination; self-alignment instead stimulates role styles and knowledge acquired during pretraining.The passage also discusses self-generated training examples and avoiding closed-model APIs in other settings.Source: §2 · p. 3

3 Methods

DITTO defines role-play around consistent self-awareness and role-specific knowledge across multi-turn conversations. It constructs supervision by collecting character profiles, simulating dialogue as reading comprehension, and fine-tuning on the resulting dataset.

3.1 Problem Definition

  • Role-play is assessed by whether a model maintains self-awareness and demonstrates nuanced role-specific knowledge across multi-turn conversations, given a character name or concise description.Source: §3.1 · p. 3

3.2 Character Knowledge Collection

  • DITTO collects character names, descriptions, and key properties from Wikidata, using the corresponding Wikipedia article as the character profile.The study focuses on Chinese and English characters, while the authors note the sources can support extension to more languages.Source: §3.2 · p. 4

3.3 Dialogue Simulation

  • Dialogue simulation generates role-specific and contrastive queries, then appends each query to a verbalized character profile so the LLM responds by extracting relevant context and emulating the character.Contrastive queries target information outside a character’s knowledge scope; the authors frame response generation as reading comprehension to reduce hallucinations relative to directly prompting GPT-4 for role-play data.Source: §3.3 · pp. 4–5

3.4 Supervised Finetuning

  • DITTO fine-tunes the LLM on self-generated dialogue data, removing injected profile knowledge so the model retains only a brief character introduction and must draw on context or inherent knowledge.Source: §3.4 · p. 5

4 Evaluation

The evaluation replaces costly, variable manual annotation with an objective role-play assessment. It measures role identity consistency, role-related knowledge accuracy, and rejection of questions beyond a character’s cognitive boundary.

4 Evaluation

  • The evaluation proposes objective assessment because manual role-play annotation is costly and variable, limiting comprehensive comparisons and reproducibility.It uses an LLM-judge recipe for three core metrics.Source: §4 · p. 5
  • The evaluation presents a brief character profile to test whether models can draw on inherent role-play knowledge.The profile example identifies Edward III as king of England.Source: §4 · p. 5

4.1 Metric Design

  • Role identity is assessed by asking a judger to select the portrayed character from four options in a multiple-choice evaluation.The selection tests whether the model maintains character consistency across a multi-turn conversation.Source: §4.1 · pp. 5–6
  • Role-related knowledge is assessed by giving a judging LLM golden knowledge and checking whether the response is consistent with that evidence.The golden knowledge comes from the dialogue-simulating scheme.Source: §4.1 · p. 6
  • Cognitive-boundary awareness is measured by whether the model rejects questions that fall outside a character’s knowledge boundary.Questions are manually annotated for each character’s boundary, and rejection accuracy is calculated.Source: §4.1 · p. 6

5 Experiments

DITTO is evaluated across Qwen-Chat scales, with results showing stronger role-play performance at larger scales and benefits from knowledge injection and more accurate query simulation.

5.1 Experimental Setup

  • Experiments use Qwen-Chat seeds at 1.8B, 7B, 14B, and 72B, with GPT-4-Turbo judging; a GPT-4-Turbo-generated held-out test set has 100 nonoverlapping roles and 498 chat turns.Source: §5.1 · pp. 6–7

5.2 Main Results

  • Role-play specialists exceed general baselines on consistency and rejection, but CharacterGLM and Xingchen knowledge scores only match 7B general baselines; Qwen-Max leads the benchmark.Source: §5.2 · p. 7
  • 0.9 on consistency and 6.64 on knowledge: DITTO-Qwen-72B surpasses all baselines on consistency and role-play experts on knowledge, with rejection on par with GPT-4.Source: §5.2 · p. 7, Table 2 · p. 8

5.3 Analysis

  • Human ratings of 400 queries show simulation accuracy rises as seed scale grows from 1.8B to 72B, with half role-specific and half contrastive queries sampled.Source: §5.3 · p. 8, Figure 4 · p. 8
  • On Qwen-1.8B-Chat, knowledge injection improves all three dialogue-simulation metrics, especially knowledge and rejection.Source: §5.3 · p. 9, Table 3 · p. 9

6 Dissecting Role-play by Cross Supervision

Cross-supervision experiments separate role identity from role-specific knowledge: identity can improve through imitation despite weaker supervision, while knowledge remains constrained by the seed model’s capabilities. Knowledge-related performance generalizes as seed models scale, unlike role identity.

6.1 Cross-supervision Setting

  • The cross-supervision setup varies the supervision LLM across four Qwen-Chat models from 1.8B to 72B and fine-tunes all four seed models on each simulation.Supervision models generate queries and responses using DITTO’s simulation recipe; supervision performance evaluates those models, while imitation performance evaluates fine-tuned seed models.Source: §6.1 · p. 9

6.2 Discussion

  • Role identity improves through imitation even with worse supervision, while role-specific knowledge and rejection can degrade after imitation.The authors interpret identity as easier to learn and more robust to supervision quality than role-specific knowledge and rejection behaviors.Source: §6.2 · p. 10
  • In strong-to-weak settings, more capable supervision yields only marginal knowledge gains for weaker seed models, indicating that inherent seed-model capabilities bound role-specific knowledge.The authors conclude that a strong foundation model matters more than SFT data as the central bottleneck for commendable role-play performance.Source: §6.2 · p. 10
  • Knowledge and rejection show weak-to-strong generalization as seed models scale, but role identity does not.With weak supervision, Qwen-72B-Chat can reach on-par performance with self-aligned Qwen-7B-Chat on knowledge.Source: §6.2 · p. 10

7 Conclusion

DITTO enables instruction-following LLMs to gain role-play proficiency through self-alignment without distilling proprietary models. Its results distinguish role-specific knowledge, which is constrained by the model’s capabilities, from conversational style, which can be acquired with smaller-model guidance.

7 Conclusion

  • DITTO achieves role-play proficiency through self-alignment without distilling proprietary counterparts such as GPT-4.The authors report that it outperforms existing open-source role-play models across four model sizes, from 1.8B to 72B, and reaches performance comparable to proprietary LLMs such as GPT-4-turbo.Source: §7 · p. 10
  • Role-specific knowledge is inherently constrained by the language model’s capabilities, while conversational style displays weak-to-strong generalization.The authors state that conversational style can be acquired with guidance from a smaller-sized model.Source: §7 · p. 10
  • The authors present the decomposition of role-play into role-specific knowledge and conversational style as a basis for reconsidering role-play alignment capabilities.Source: §7 · p. 10

Limitations

The strongest DITTO model remains behind advanced chatbots, and its self-generated dialogue data contains noticeable noise.

Limitations

  • The best DITTO model, based on Qwen-72B-Chat, is still outperformed by GPT-4 and GPT-4-Turbo.Source: Limitations · p. 10
  • Even Qwen-72B-Chat DITTO training data contains noticeable noise.Source: Limitations · p. 10
  • The authors expect manual cleaning of the self-generated dialogue simulations to further improve DITTO’s performance.Source: Limitations · p. 10

Ethics Statements

The ethics statement warns that DITTO-aligned role-play models may have minimal safety alignment. The appendices describe character collection queries and prompts for generating contrastive role-play questions.

Ethics Statements

  • DITTO-aligned role-play models may have minimal safety alignment and may generate toxic or harmful content under induction.The authors say these models are for research purposes and should receive careful safety alignment in future work.Source: Ethics Statements · p. 10

A Wikidata Queries

B.1 Query Simulation

  • The simulation prompt requests questions Character A can answer but Character B cannot, constrained by their era and character settings.Each question is accompanied by an explanation of why A can answer it and B cannot.Source: §B.1 · p. 13
  • The prompt asks for casual, second-person questions in English and specifies a JSON list of question strings.Source: §B.1 · p. 13

C Hyperparameters

The section specifies the models’ training and inference settings, including how query and response simulations were generated.

C Hyperparameters

  • Training used 300 steps, a global batch size of 128, sequence length 8,192, and learning rates of 2e-6 to 2e-7.Prompts and ChatML roles were masked during training.Source: C Hyperparameters · p. 13
  • Inference used topP 0.8, length penalty 1.1, sequence length 8,192, and a maximum of 2,048 new tokens.Source: C Hyperparameters · p. 13
  • Query simulation generated three rounds and randomly selected one format-valid query; response simulation generated one response.Smaller LLMs sometimes failed to follow the output format.Source: C Hyperparameters · p. 13

D API Configurations

The section describes API inference configurations for general and role-play expertise baselines.

D API Configurations

  • OpenChat-3.5, Mistral-7B-Instruct-v0.2, and Mixtral-8x7B-Instruct-v0.1 were run through Together API with default hyperparameters and an 8,192 maximum length.Source: D API Configurations · p. 13
  • Claude 2.1 and Wenxin 4.0 APIs used default parameters, while OpenAI API baselines used temperature 0.7 and maximum length 8,192.The OpenAI baselines were GPT-3.5-Turbo, GPT-4, and GPT-4-Turbo.Source: D API Configurations · p. 13
  • CharacterGLM was called through Zhipu AI’s official API and Xingchen through its official Python SDK, both with user_name and user_info set to “user”.Source: D API Configurations · p. 13

Related papers

Loading 2401.12474v1…