PaperlayerView exact revision on arXiv

Source-linked AI summary

CharacterGLM: Customizing Chinese Conversational AI Characters with Large Language Models

Jinfeng Zhou, Zhuang Chen, Dazhen Wan, Bosi Wen, Yi Song, Jifan Yu, Yongkang Huang, Libiao Peng, Jiaming Yang, Xiyao Xiao, Sahand Sabour, Xiaohan Zhang, Wenjing Hou, Yijia Zhang, Yuxiao Dong, Jie Tang, Minlie Huang

arXiv:2311.16832v1cs.CLcs.AI

TL;DR

The paper addresses limits in LLMs’ support for social connection and emotional support, and introduces CharacterDial and CharacterGLM for customizable character dialogue. CharacterGLM-66B is reported to marginally outperform GPT-4 on the Overall metric, while long-term memory and self-awareness remain challenges.

  • Problem

    The paper identifies social connection and emotional support as needs that existing LLMs remain incompetent at accomplishing.

    Source: §1 · p. 1

  • Method

    CharacterGLM builds on ChatGLM to generate CharacterDial by conditioning dialogue on character profiles and refining models with deployment interaction data.

    Source: §1 · p. 3, §3.2 · p. 7

  • Results

    CharacterGLM-66B marginally outperforms GPT-4 by approximately 1.4% on the Overall metric and is on par with GPT-4 on general generation performance.

    Source: §4.1.2 · p. 7

  • Takeaways & Limitations

    The authors release CharacterGLM-6B and a portion of training data to support research on character-based dialogue systems.

    Source: §5 · p. 14

    The paper identifies long-term memory and growth, and maintaining AI characters’ self-awareness, as challenges for future work.

    Source: §5 · p. 14

Abstract

from arXiv · show

In this paper, we present CharacterGLM, a series of models built upon ChatGLM, with model sizes ranging from 6B to 66B parameters. Our CharacterGLM is designed for generating Character-based Dialogues (CharacterDial), which aims to equip a conversational AI system with character customization for satisfying people's inherent social desires and emotional needs. On top of CharacterGLM, we can customize various AI characters or social agents by configuring their attributes (identities, interests, viewpoints, experiences, achievements, social relationships, etc.) and behaviors (linguistic features, emotional expressions, interaction patterns, etc.). Our model outperforms most mainstream close-source large langauge models, including the GPT series, especially in terms of consistency, human-likeness, and engagement according to manual evaluations. We will release our 6B version of CharacterGLM and a subset of training data to facilitate further research development in the direction of character-based dialogue generation.

1 Introduction

Although LLMs are capable general assistants, the paper identifies social connection and emotional support as unmet needs. It introduces CharacterDial and CharacterGLM to create customizable characters for more sustained, character-specific conversations.

1 Introduction

  • LLMs’ short-turn training data is contrasted with the paper’s goal of supporting long-term social connections and emotional support.The introduction frames conversational engagement with desired social entities as one way people may fulfill needs for love and belonging.Source: §1 · pp. 1–2
  • CharacterDial lets users configure profiles for preferred characters, whose worldviews, values, and common facts shape tailored conversations.The profiles can represent fictional or real personalities.Source: §1 · p. 2
  • Tuning-free role-play struggles to align profiles with dialogue, while persona-based dialogue omits complex attributes and behaviors; both challenges are especially relevant in long conversations.The paper also notes that Character.AI’s foundational model is proprietary, limiting accessible research resources.Source: §1 · pp. 2–3, Figure 2 · p. 3
  • CharacterGLM is proposed as a model series for customizing AI characters to produce consistent, human-like, and engaging conversations.The models build on ChatGLM and range from 6B to 66B parameters.Source: §1 · p. 3
  • The authors introduce CharacterDial as a task and crowdsource a Chinese corpus spanning character categories and dialogue topics.A released subset contains 1,034 dialogue sessions spanning 250 characters; the paper says the 6B model and part of the data will be released.Source: §1 · p. 3

2 Design Principle of CharacterGLM

CharacterGLM’s design treats characters as combinations of enduring attributes and dynamic behaviors, then evaluates dialogue through consistency, human-likeness, and engagement.

2 Design Principle of CharacterGLM

  • The design separates response content, shaped mainly by attributes, from tone and style, shaped mainly by behaviors.The three evaluation dimensions are consistency, human-likeness, and engagement.Source: §2 · p. 4

2.1 Cores of AI Characters: Attributes & Behaviors

  • Character profiles cover seven attribute groups, including identity, interests, viewpoints, experiences, achievements, social relationships, and other skills.These attributes supply background information that can influence character reactions and interactions.Source: §2.1 · p. 4
  • Behaviors include linguistic features, emotional expression, interaction patterns, and personality traits such as gentleness or coldness.Examples include an elderly character using more formal language and a teenager using current slang.Source: §2.1 · p. 4

2.2 Features of AI Characters: Consistency, Human-likeness, and Engagement

  • Consistency means stable attributes and behaviors, human-likeness means natural human-like interaction, and engagement measures user interest, interaction, and emotional connection.The paper links consistency to users feeling they interact with the same individual.Source: §2.2 · pp. 4–5

3 Implementation of CharacterGLM

CharacterGLM’s implementation combines character-based dialogue collection with model training. The collection process draws on human role-playing, LLM synthesis, and literary dialogue extraction.

3 Implementation of CharacterGLM

  • The implementation covers character-based dialogue collection and training for dialogue generation.The paper considers four categories: celebrities, daily life, games & videos, and virtual love.Source: §3 · p. 5, §3.1 · p. 5

3.1 Character-Based Dialogue Collection

  • In human role-playing, workers create character profiles and conduct dialogues as characters or users, sometimes using character relationships to shape the interaction.The released data subset comes from this source.Source: §3.1 · p. 5
  • GPT-4 generates profiles and multi-turn dialogues through a few-shot pipeline, after which workers rephrase Chinese outputs to sound more colloquial.Prompts use pluggable placeholders to balance categories, relationships, and gender distribution.Source: §3.1 · p. 5
  • Workers manually extract two-party dialogues from literary sources because automated extraction faces context, speaker, profile, and non-verbal-cue challenges.Workers also summarize the two characters’ profiles.Source: §3.1 · p. 5

Step I: Collect CharacterDial Data

CharacterDial data collection combines curated dialogue sources and iterative user interactions. CharacterGLM training turns character profiles into prompts for supervised fine-tuning and uses deployment interactions for self-refinement.

Step I: Collect CharacterDial Data

  • Workers interact with deployed prototype characters and revise unsatisfactory responses, producing data for model self-refinement.Users customize characters and engage in multi-turn dialogue during this process.Source: Step I: Collect CharacterDial Data · p. 6
  • Quality inspectors review all collected data and require marked low-quality parts to be repaired to meet quality requirements..Source: Step I: Collect CharacterDial Data · p. 6

3.2 Training LLMs for Character-Based Dialogue Generation

  • Character profiles are written as natural-language prompts, with summarization, paraphrasing, and stylization used to diversify prompts.Claude-2 is used to synthesize diverse prompts.Source: §3.2 · p. 7
  • ChatGLM backbones from 6B to 66B parameters are fine-tuned on concatenated character prompts and dialogue, with training data growing linearly as prompts are augmented.The figure summarizes the broader workflow from profile collection through fine-tuning and deployment-data self-refinement.Source: Figure 4 · p. 6, §3.2 · p. 7
  • After deployment, human-prototype interaction data is added to supervised fine-tuning to support continuous model self-refinement.The collection process involves users customizing characters and interacting with the prototype over multiple turns.Source: Step I: Collect CharacterDial Data · p. 6, §3.2 · p. 7

4 Experiments

Across point-wise and pair-wise evaluations, CharacterGLM-66B performed strongly against GPT-4, GPT-3.5, and MiniMax, with results varying by evaluation dimension, character category, and dialogue topic. The experiments also examined turn-level errors, long-term interaction, and response-length preferences.

4.1.1 Experimental Setup

  • The point-wise evaluation used 10 annotators rating models on six dimensions and Overall, while the pair-wise study sampled 24 characters across four categories and three dialogue scenes.Pair-wise annotators compared responses in interactions of at least 20 turns.Source: §4.1.1 · p. 7, §4.2.1 · p. 10

4.1.2 Performance Analysis

  • CharacterGLM-66B topped the “Overall” point-wise metric, outperforming GPT-4 by approximately 1.4%.The study used 1–5 annotator ratings across six sub-dimensions and an Overall metric.Source: §4.1.2 · p. 7, §4.1.1 · p. 7, Table 4 · p. 8
  • CharacterGLM-66B outperformed MiniMax by 6% on Engagement and showed strong performance on character consistency and human-likeness.Its attribute consistency was suboptimal, while behavior consistency was good; the paper defines overall consistency as the mean of these two components.Source: §4.1.2 · pp. 8–9, Figure 5 · p. 9
  • CharacterGLM-66B also performed strongly on general Quality, Safety, and Correctness, matching GPT-4 on Correctness.The point-wise evaluation used API-accessed Chinese-capable models, including Claude-2 via an unofficial API encapsulation.Source: §4.1.1 · p. 7, §4.1.2 · p. 8

4.1.3 Fine-grained Error Analysis

  • CharacterGLM-66B’s turn-level “Overall” score led MiniMax by approximately 4%, despite not leading most individual error dimensions.The authors also report strong results on Less-informativeness and a suboptimal score on Repetition, with moderate Proactivity.Source: §4.1.3 · pp. 9–10, Table 5 · p. 9

4.2.2 Performance Analysis

  • By dialogue topic, CharacterGLM-66B matched MiniMax in chit-chat and led it by 7% in interviews; against GPT-4, it had a 15% love-scene advantage and a 1% overall disadvantage.It outperformed GPT-3.5 across all dialogue topics, including by 8% in love scenes and 3% in interviews.Source: §4.2.2 · pp. 11–12
  • CharacterGLM-66B began conversations slightly behind MiniMax but improved in later stages, while longer responses were generally preferred and its shorter replies remained comparable.MiniMax generated longer responses on average.Source: §4.2.2 · pp. 12–13

5 Conclusion and Future Work

CharacterGLM-66B is reported as competitive with some proprietary models, and CharacterGLM-6B and a portion of the training data have been released. The paper identifies long-term memory, self-awareness, character-to-character interaction, and deeper cognitive processes as future challenges.

Conclusion and Future Work

  • Table 12’s preference comparison of CharacterGLM-66B and MiniMax across different turn intervals reports that CharacterGLM shows remarkable advantages in modeling long-term dialogue.Source: Table 12 · p. 14
  • When one party responded longer in the same context, longer responses obtained a general preference in comparisons of CharacterGLM-66B and MiniMax.Table 13 compares their response lengths.Source: §5 · p. 14, Table 13 · p. 14
  • The authors released CharacterGLM-6B and a portion of the training data to support further research on character-based dialogue systems.Source: §5 · p. 14
  • Long-term character relationships require memory of interactions, statements, and actions beyond finite context windows, alongside character growth.Source: §5 · p. 14
  • AI characters need self-awareness of their distinct personalities and knowledge boundaries to support more engaging and trustworthy interactions.Source: §5 · p. 14
  • Character-to-character interaction could let AI characters learn from users and their own society, enriching information for their development.The authors describe this as a future direction for exploring a “character society.”Source: §5 · pp. 14–15
  • Character dialogue remains limited by surface-level text-pattern mimicry; deeper understanding of intentions, emotions, and social behavior is a proposed future direction.Source: §5 · p. 15

Related papers

Loading 2311.16832v1…