PaperlayerView exact revision on arXiv

Source-linked AI summary

ChatHaruhi: Reviving Anime Character in Reality via Large Language Model

Cheng Li, Ziang Leng, Chenxi Yan, Junyi Shen, Hao Wang, Weishi MI, Yaying Fei, Xiaoyang Feng, Song Yan, HaoSheng Wang, Linkang Zhan, Yaokai Jia, Pingyu Wu, Haozhen Sun

arXiv:2308.09597v1cs.CLcs.HC

TL;DR

Although role-playing chatbots have drawn interest, reliably mimicking specific fictional characters remains a challenge. ChatHaruhi uses script-derived character memories, prompts, and simulated dialogues, and reports improved role-playing in automatic and human evaluations. The authors also demonstrate the feasibility of fine-tuning multiple roles into a single 7B model.

  • Problem

    Despite interest in role-playing, better techniques are needed to mimic specific fictional characters; prompt-only methods depend on model memories and can hallucinate, while minor characters may lack training dialogues.

    Source: §1 · p. 1

  • Method

    ChatHaruhi retrieves relevant script exchanges, combines them with character-setting prompts, and generates simulated dialogues for dataset construction and fine-tuning.

    Source: §1 · pp. 2–3

  • Results

    The authors report improved role-playing over the same base model in automatic and human evaluations, and demonstrate feasibility of fine-tuning multiple roles into one 7B model.

    Source: §1 · p. 3, §7 · p. 11

  • Takeaways & Limitations

    The released ChatHaruhi-54K dataset, models, and code are intended to support further research on role-playing language models.

    Source: §7 · p. 11

    The first-version Alpaca-generated questions are influenced by the examples and tend to relate to original scripts; the authors hope to filter for real user questions in later testing.

    Source: §5.2 · p. 9

Abstract

from arXiv · show

Role-playing chatbots built on large language models have drawn interest, but better techniques are needed to enable mimicking specific fictional characters. We propose an algorithm that controls language models via an improved prompt and memories of the character extracted from scripts. We construct ChatHaruhi, a dataset covering 32 Chinese / English TV / anime characters with over 54k simulated dialogues. Both automatic and human evaluations show our approach improves role-playing ability over baselines. Code and data are available at https://github.com/LC1332/Chat-Haruhi-Suzumiya .

1 Introduction12

Although role-playing chatbots have attracted interest, mimicking specific fictional characters remains difficult. ChatHaruhi combines character memories extracted from scripts with improved prompts and simulated dialogues, and reports better role-playing than baselines in automatic and human evaluations.

1 Introduction12

  • Prompt-only role-play relies heavily on a model’s existing memories, gives vague instructions about character knowledge, and can leave style dependent on the base model.The authors also report that fine-tuning can increase hallucinations and that minor characters may lack enough dialogue for training.Source: §1 · p. 1
  • The project treats character background, consistent personality, and linguistic habits as distinct components of realistic role-play.Understanding the story setting may require external knowledge bases.Source: §1 · p. 2
  • The system retrieves relevant classic plots from a character memory database and combines them with prompts about the character’s setting to guide responses.It also generates character-fitting dialogues to provide fine-tuning data, including for characters with fewer original dialogues.Source: §1 · p. 2
  • ChatHaruhi covers 32 Chinese and English TV and anime characters, combining more than 22,000 collected exchanges with over 31,000 simulated dialogues.The combined collection forms the ChatHaruhi-54K dataset.Source: §1 · pp. 2–3
  • The proposed algorithm improves role-playing performance over the same base language model in automatic and human evaluations.Automatic evaluation compares responses to classic plot points; human raters assess character-setting alignment and response quality.Source: §1 · p. 3
  • The authors make the project’s data and code available to support further research.They also describe ongoing work to modularize the code for easier use.Source: §1 · p. 3

2 Related Work

Related work covers in-context learning and automatic dialogue generation. ChatHaruhi uses in-context dialogue generation for a specific persona, conditioning exchanges on that character’s background, personality, and prior conversations.

2 Related Work

  • Prior in-context learning research addresses question phrasing, example selection, contextual objectives, and instruction following, while also reporting vulnerability and over-sensitivity.The cited work includes Zhao et al. (2021), Holtzman et al. (2021), Liu et al. (2021), Lu et al. (2022), Rubin et al. (2022), Chen et al. (2022), Mishra et al. (2022), Efrat et al. (2021), Wei et al. (2022), and Sanh et al. (2022).Source: §2 · p. 3
  • ChatHaruhi uses in-context learning to generate user questions and persona-specific responses from a character’s background and prior memories.Generated question-answer pairs are used to analyze and learn persona-agent behavior.Source: §2 · pp. 3–4
  • Automatic dialogue-generation precedents include Alpaca’s self-instruction, Vicuña’s ChatGPT dialogues, Baize’s self-conversation pipeline, and CAMEL’s inception prompting.The cited works are Taori et al. (2023), Ahn et al. (2023), Xu et al. (2023), and Li et al. (2023).Source: §2 · p. 4
  • Unlike general conversational-agent work, ChatHaruhi generates dialogues tailored to a specific character using substantial background, personality, and prior-conversation prompts.The resulting exchanges are intended as data for learning that persona’s behavior.Source: §2 · p. 4

3 ChatBot Design

ChatHaruhi conditions character responses on a character prompt, relevant script memories, and dialogue history. The design addresses reliance on a model’s own knowledge and supports conversational continuity.

3 ChatBot Design

  • The design combines character settings, prior dialogue, retrieved script memories, and conversation history to generate answers, since model knowledge alone may fail for unfamiliar characters.Sentence embeddings retrieve query-relevant memories from a larger bank when a character has a larger worldview.Source: §3 · p. 4

3.1 System Prompt

  • The prompt emphasizes role-playing, reuse of classic lines, and supplementary personality details, which the authors found can affect the model’s output.It addresses tendencies to avoid repeating context and to retain the model’s own language preferences despite character memories.Source: §3.1 · p. 5

3.2 Dialogues from each Character

  • Character memories use story-form script excerpts that preserve plot context, including narration, other characters’ dialogue, or actions around at least one dialogue pair.The relaxed excerpt format can also support preparing more script data.Source: §3.2 · p. 5

3.3 Original Dialogue Searching

  • Sentence embeddings retrieve the M excerpts closest to a query by cosine similarity, with the number adjusted to fit the token budget.For OpenAI turbo-3.5, the implementation limits D to 1,500 tokens; the authors identify replacing the embedding model as future work.Source: §3.3 · pp. 5–6

3.4 Chat Memory

  • The system records user queries and chatbot responses as dialogue history, but limits context to 1,200 tokens and does not address character long-term memory.The authors estimate this history can accommodate about 6–10 dialogue rounds.Source: §3.4 · p. 6

4 Character Dataset Building

ChatHaruhi builds character-specific script collections from varied media and extracts them with source-specific methods, forming the original-dialogue basis for a 32-character dataset.

4.1 Characters

  • The dataset covers 32 characters and distinguishes Qiao Feng from Xiao Feng to examine whether they respond differently to the same questions.The characters span Chinese and English works, including TV, anime, novels, films, crosstalk, and game wikis.Source: §4.1 · pp. 6–7, Figure 5 · p. 7
  • The collection includes characters from crosstalk, game wikis, and several Chinese works, including Wang Duoyu, who the chatbot struggles to keep from revealing a plot secret.The authors suggest constructed reasoning chains or constitutional additions as possible directions for improving that case.Source: §4.1 · p. 7

4.2 Original Script Extraction

  • Original character dialogues require source-specific extraction because they do not naturally appear in the chatbot’s desired story format.The story data preserves dialogue pairs alongside narration, other characters’ lines, or actions.Source: §4.2 · p. 7

4.2.1 Quotes Data

  • For quote-based data, the authors manually format some online scripts, use existing crosstalk exchanges, and organize quotes and plots gathered from wikis and websites.Yu Qian’s crosstalk exchanges are split at locally maximal utterance lengths; Big Bang Theory data uses the novel-extraction finite-state machine.Source: §4.2.1 · pp. 7–8

4.2.2 Extract from TV Series

  • For TV dialogue, the authors use speech recognition or subtitles, identify speakers with ECAPA-TDNN embeddings, then manually correct errors and split scripts.Because this manual work is time-consuming, they process only Haruhi Suzumiya and Li Yunlong this way.Source: §4.2.2 · p. 8

4.2.3 Extract from Novel

  • For novels, Kor extracts character-action-dialogue information, after which a finite-state machine segments dialogue with limited length, few characters, and little non-dialogue text.The extraction prompt records spoken text as dialogue and summarizes actions when a sentence contains no dialogue.Source: §4.2.3 · p. 8
  • The reported extraction statistics are presented in a table, but the supplied passage does not state its values.The original-script collection is also described as covering movies, novels, scripts, and TV shows.Source: §4.2.3 · p. 8, Figure 6 · p. 8

5 Dialogue Synthesizing

Dialogue synthesis converts character-specific prompts and script context into training dialogues, while question augmentation expands the prompts available for characters with limited data.

5.1 Generate Dialogue from Question

  • To create training data from story-form scripts, the system uses the exchanges before a character’s turn as the initial question and generates a dialogue from it.Generated continuations can contain multiple sentences close to 50% of the time; when the question overlaps the original text, the prompt design tends to elicit the character’s original lines.Source: §5.1 · pp. 8–9

5.2 Question Generating

  • For characters with little data, the authors adapt Alpaca-like question generation, keeping generated questions and using the dialogue-generation method to produce training dialogues.They use ChatGPT, GPT-4, and Claude; the questions tend to relate to the example scripts, and filtering for real-user questions is left for later versions.Source: §5.2 · p. 9
  • The dataset contains 54,726 dialogues: 22,752 collected originals and 31,974 simulated questions with corresponding dialogues.Individual dialogues do not necessarily consist of a single question-answer pair.Source: §5.2 · p. 9

6 Experiments

The experiments report qualitative evidence that classic dialogue examples and improved prompts support character style, while quantitative experiments and the user study remain unfinished.

6.2 Metrics for User Study

  • The authors argue that role-playing evaluation should assess role consistency separately from dialogue quality, but the user study is still in progress.They note that a role-consistent answer may have lower dialogue quality, motivating separate human-rating dimensions.Source: §6 · p. 9, §6.2 · p. 9

6.1 Metrics for Automatic Evaluation

  • Automatic evaluation tests response plausibility by comparing generated replies with character lines from classic narratives using sentence-embedding cosine similarity.The evaluation uses 30 stories for each of 27 roles; five Genshin Impact roles are excluded because their dialogues are limited and lack continuity.Source: §6.1 · p. 9

6.3 Language Model Tuning

  • The authors fine-tune ChatGLM2-6B on original dialogues, the full 54K dataset, or original character utterances, using three epochs on four A100 GPUs.The report says versions A and B will be released with it, while version C is planned for a future update.Source: §6.3 · p. 10

6.4 Qualitative Results

  • With classic dialogues and an improved prompt, ChatGPT can adopt specific characters’ speaking styles, and fine-tuning a 7B model helps it internalize the full prompt.The qualitative comparison considers prompted ChatGPT and ChatGLM2 variants, including a fine-tuned ChatGLM2 model.Source: §6.4 · pp. 10–11

6.5 Quantitative Results

  • Quantitative experiments are still in progress and are reserved for a future version of the report.The user study is also not yet complete.Source: §6.5 · p. 11, §6.6 · p. 11

7 Conclusion, Discussion & Future Work

The report presents a character role-playing system using prompts and classic-scene examples, generates a 54K-dialogue dataset, and demonstrates the feasibility of fine-tuning multiple roles into one 7B local model.

7 Conclusion, Discussion & Future Work

  • The system uses character-specific prompts and classic-scene passages to mimic distinctive conversational styles, and demonstrates multi-role fine-tuning in a single 7B local model.It also generates a dataset of 54K simulated dialogues.Source: §7 · p. 11
  • The project releases model A trained on 23K original transcripts, model B trained on the full 54K dataset, demos, and the complete dataset.The first character was Haruhi Suzumiya; the project and dataset were initially named ChatHaruhi and Haruhi-54K.Source: §7 · p. 11
  • Future work will refine the interface for easier reuse and supplement the quantitative evaluations.Source: §7 · p. 11

8.1 Contributors

The contributors describe a collaborative, open-source effort spanning project design, training and data generation, script collection, backend and tool development, deployment, and character artwork.

8.1 Contributors

  • Cheng Li proposed the project and built most functionality; Ziang Leng designed training, data generation, and backend architecture, while Chenxi Yan maintained the backend.Source: §8.1 · p. 11
  • Junyi Shen implemented training code; Hao Wang collected My Own Swordsman scripts; Weishi MI and Yaying Fei contributed to augmented-data generation.Source: §8.1 · p. 11
  • Other contributions included script conversion and collection, voiceprint recognition and speech synthesis, Genshin Impact data, frontend and GPU work, training deployment, and character artwork.These tasks are attributed across Song Yan, HaoSheng Wang, Linkang Zhan, Yaokai Jia, Pingyu Wu, and Haozhen Sun.Source: §8.1 · pp. 11–12
  • The authors invite project suggestions and requests to add references through the project issue tracker.Source: §8.1 · p. 12

A Appendix-A: ChatHaruhi 2.0 Design

The appendix proposes refactoring ChatHaruhi's demo-oriented design to support research with multiple chatbots, with a simpler interface based on prompts and a vector database.

A Appendix-A: ChatHaruhi 2.0 Design

Loading 2308.09597v1…