PaperlayerView exact revision on arXiv

Source-linked AI summary

CogDual: Enhancing Dual Cognition of LLMs via Reinforcement Learning with Implicit Rule-Based Rewards

Cheng Liu, Yifei Lu, Fanghua Ye, Jian Li, Xingyu Chen, Feiliang Ren, Zhaopeng Tu, Xiaolong Li

arXiv:2507.17147v1cs.CL

TL;DR

Role-playing agents often imitate character behavior without modeling the cognitive processes behind it. CogDual combines situational and self-awareness before responding and uses two reinforcement-learning rewards; it achieves the best performance among comparable methods on CoSER and generalizes on Cross-MR and LifeChoice.

  • Problem

    Role-playing approaches often overlook the situational and self-awareness processes behind character behavior, beyond assessing whether responses align with a character profile.

    Source: §1 · p. 2

  • Method

    CogDual uses cognize-then-respond reasoning that combines situational and self-awareness, then applies reinforcement learning with ICLG and LSA rewards.

    Source: §1 · p. 2

  • Results

    CogDual achieves the best performance among comparable methods on CoSER and strong generalization on Cross-MR and LifeChoice, with up to a 9.24% average improvement over baseline on CoSER.

    Source: §1 · p. 2, §6 · p. 9

  • Takeaways & Limitations

    The authors conclude that dual cognition supports more contextually grounded and psychologically coherent role-playing responses, with reinforcement learning improving on the supervised fine-tuning baseline.

    Source: §6 · p. 9

    The experiments use English datasets, and the reinforcement-learning approach has not been evaluated on larger-scale models such as Llama3.1-70B-Instruct.

    Source: Limitations · p. 9

Abstract

from arXiv · show

Role-Playing Language Agents (RPLAs) have emerged as a significant application direction for Large Language Models (LLMs). Existing approaches typically rely on prompt engineering or supervised fine-tuning to enable models to imitate character behaviors in specific scenarios, but often neglect the underlying \emph{cognitive} mechanisms driving these behaviors. Inspired by cognitive psychology, we introduce \textbf{CogDual}, a novel RPLA adopting a \textit{cognize-then-respond } reasoning paradigm. By jointly modeling external situational awareness and internal self-awareness, CogDual generates responses with improved character consistency and contextual alignment. To further optimize the performance, we employ reinforcement learning with two general-purpose reward schemes designed for open-domain text generation. Extensive experiments on the CoSER benchmark, as well as Cross-MR and LifeChoice, demonstrate that CogDual consistently outperforms existing baselines and generalizes effectively across diverse role-playing tasks.

1 Introduction

Existing role-playing approaches emphasize character-aligned behavior but often overlook the situational and self-awareness processes behind it. CogDual pairs those forms of cognition before response generation and applies reinforcement learning with two general-purpose rewards.

1 Introduction

  • Prior approaches assess profile alignment and related character behaviors, but the introduction identifies situational and self-awareness processes as an overlooked dimension of role-playing agents.These processes are presented as more than surface imitation of linguistic patterns or behavior.Source: §1 · pp. 1–2
  • The approach emphasizes a comprehensive character cognition process, distinguishing it from fragmented self-questioning or isolated mental-state simulation in Ji et al. (2025) and Xu et al. (2025).The paper describes its dual reasoning as aligning psychological dynamics with narrative context.Source: §1 · p. 2
  • CogDual formalizes a cognize-then-respond paradigm that combines external Situational Awareness with internal Self-Awareness before generating a response.The paper frames this dual-cognition design as a psychologically plausible model of human-like behavior generation.Source: §1 · p. 2
  • Up to a 9.24% average improvement over baseline is reported for reinforcement learning with two general-purpose reward schemes on CoSER.The schemes are ICLG, which quantifies how intermediate cognitive steps improve response likelihood, and LSA, which measures semantic similarity to gold-standard references.Source: §1 · p. 2
  • CogDual outperforms all baseline methods on Cross-MR and LifeChoice, which the authors present as evidence of cross-task transferability.The benchmarks are identified as Cross-MR (Yuan et al., 2024) and LifeChoice (Xu et al., 2024).Source: §1 · p. 2

2 Related Work

Related work traces role-playing agents from character understanding and imitation toward evaluating behavioral coherence, while cognitive modeling and reasoning research motivate explicit, cognitively grounded reasoning for role-play.

2.1 Role-Playing Language Agents

  • RPLA research has moved from character prediction and imitation toward evaluation of internal coherence through motivation recognition, persona-driven decisions, and psychological assessment.The cited evaluative frameworks include Yuan et al. (2024), Xu et al. (2024), and Wang et al. (2024a).Source: §2.1 · p. 2

2.2 LLM-Based Cognitive Modeling

  • Cognitive modeling research covers traits such as self-awareness, emotion understanding, intent recognition, and deliberative reasoning, while Wang et al. (2025a) highlight limits in LLM psychological states and intrinsic motivations.The paper adopts a cognitive psychology perspective to define and examine cognition in role-play settings.Source: §2.2 · pp. 2–3

2.3 Reasoning Techniques in LLMs

  • Reasoning-augmented models can exhibit stylistic drift between reasoning traces and character-based generation, motivating a cognitively grounded template for role-play reasoning.Feng et al. (2025b) highlight this limitation in role-playing scenarios.Source: §2.3 · p. 3

3 Methodology

CogDual places explicit dual cognition between perception and response, modeling situational and self-awareness before generating character behavior. It trains this process with supervised fine-tuning and GRPO using two implicit rewards.

3.1 Cognition-Driven Reasoning Paradigm

  • CogDual models cognition as an explicit step between perception and response, producing structured representations of the environment, other characters, and the agent’s own state.The response is conditioned on this cognition and the dialogue context.Source: §3.1 · p. 3, §3.2 · p. 3

3.2 Preliminaries

  • At each turn, the character model takes its profile, scene, character set, and dialogue history as input, then generates cognition followed by a response.The initial dialogue history is empty.Source: §3.2 · p. 3

3.3 Dual Cognition of RPLAs

  • Situational Awareness interprets the scene and others through environmental cues, behavior, emotion, and intention analysis.Environmental cues include spatial layout, temporal shifts, scene dynamics, and dialogue interruptions.Source: §3.3 · pp. 3–4
  • Self-Awareness combines emotional-state recognition, context-driven intentions, relevant memories, and internal strategy for planning and anticipating outcomes.Internal strategy integrates memory, emotion, and intention.Source: §3.3 · p. 4

3.4 Dual Cognition Behavior Learning

  • Dual cognition behavior learning uses cognitive-based Chain-of-Thought prompting and a two-stage training framework; this section details the latter.The supervised-training dataset contains dual-cognition trajectories.Source: §3.4 · pp. 4–5

3.4.1 Stage 1: Supervised Fine-tuning for CogDual Initialization

  • Stage 1 initializes CogDual with supervised fine-tuning that minimizes next-token negative log-likelihood over the dual-cognition training data.The input context comprises the character profile, global scene, character set, and dialogue history.Source: §3.4.1 · p. 5

3.4.2 Stage 2: Reinforcement Learning with Two Implicit Rule-Based Rewards

  • Stage 2 uses GRPO with ICLG and LSA rewards derived from model outputs and a frozen reference policy, without additional annotation cost.ICLG rewards cognition that improves reference-response likelihood; LSA rewards semantic similarity to the reference, allowing wording variation.Source: §3.4.2 · p. 5
  • GRPO combines the two rewards with fixed weights and updates the policy using a surrogate loss with an importance ratio and KL penalty.The KL penalty strength is controlled by β.Source: §3.4.2 · p. 6

4 Experimental Setup

The evaluation uses CoSER as its main benchmark and Cross-MR and LifeChoice to assess generalization, across several base models and against prompting and fine-tuning baselines. Simulated conversations are judged on four dimensions by a GPT-4o critic.

4 Experimental Setup

  • The main benchmark is CoSER, with Cross-MR and LifeChoice used to assess generalization.Source: §4 · p. 6

4.1 Base Models

  • Experiments use LLaMA3.1-8B-Instruct, Qwen2.5-7B-Instruct, and LLaMA3.1-70B-Instruct, plus prompting evaluations on GPT-3.5-Turbo, GPT-4o, and o1-preview.Source: §4.1 · pp. 6–7

4.2 Baselines

  • Baselines include Chain-of-Thought prompting and vanilla SFT using either the complete CoSER dataset or LongCoT data.LongCoT is constructed from the same source as CogDual, with data size matched to CogDual’s initialization data.Source: §4.2 · p. 7

4.3 Evaluation Metrics

  • Simulated conversations are evaluated by a GPT-4o critic on storyline consistency, anthropomorphism, character fidelity, and storyline quality.The dimensions assess narrative alignment, human-like behavior, character representation, and coherence and fluency.Source: §4.3 · p. 7

5 Experimental Results and Analyses

CogDual improves role-playing performance on CoSER and transfers to Cross-MR and LifeChoice, with results also examining its cognitive components, reward settings, and evaluation robustness. Across these analyses, the full dual-cognition model and reinforcement learning with implicit rewards show benefits, alongside trade-offs across tasks and metrics.

5.2 Generalization to Other Benchmarks

  • CogDual-equipped LLMs outperform all baselines on Cross-MR and LifeChoice and perform comparably to o1-Preview; CogDual-RL also beats CogDual-SFT on both benchmarks.Table 2 compares accuracy on the two benchmarks.Source: Table 2 · p. 7, §5.2 · p. 8

5.1 Main Results

  • After two-stage training, CogDual boosts Storyline Quality by 11.65% for Qwen2.5-7B-Instruct and Character Fidelity by 21.84% for Llama3.1-8B-Instruct, with a 13.12% average increase.The comparison is against baselines.Source: §5.1 · p. 7
  • CogDual clearly outperforms GPT-4o-distilled Long-CoT baselines even when both use the same amount of SFT data.The authors describe this as a practical solution for test-time scaling in smaller models.Source: §5.1 · p. 7
  • RL models exceed SFT-only models by 2.85 Storyline Consistency points and 3.50 Character Fidelity points on average.The authors attribute narrative-causal coherence to ICLG and closer persona alignment to LSA; Qwen2.5-7B-Instruct with RL also surpasses o1-preview on multiple metrics.Source: §5.1 · p. 7

5.4 Ablation II: Effects of the Two Implicit Reward Mechanisms

  • The hybrid ICLG=0.7, LSA=0.3 setting yields the highest average while improving or maintaining all in-domain metrics; pure LSA leads in Anthropomorphism, while pure ICLG leads in Storyline Consistency and Quality.Table 3 compares reward-weight combinations for CogDual-SFT and CogDual-RL trained on LLaMA3.1-8B-Instruct.Source: Table 3 · p. 8, §5.4 · pp. 8–9

5.3 Ablation Study I: Effect of Dual Cognition Components

  • The full dual-cognition model scores highest or near-highest on primary role-play metrics, while removing situational awareness leads on generalization benchmarks but trails the full model by 1.8 Storyline Quality points and 1.3 average-score points.The authors suggest the generalization tasks favor self-focused reasoning, but note situational awareness’s role in coherent, context-aware multi-turn interaction.Source: §5.3 · p. 8

5.5 Evaluator Robustness and Human Evaluation

  • CogDual-RL retains the top rank under DeepSeek-v3 and Gemini-2.0-Flash judges, and human evaluation reports stronger performance on subjective metrics.The human evaluation compares methods on Llama-3.1-Instruct-8B and finds CogDual better captures character complexity and nuanced emotions.Source: §5.5 · p. 9

6 Conclusion

CogDual combines dual cognition with a cognize-then-respond paradigm, then uses reinforcement learning with ICLG and LSA rewards to improve on supervised fine-tuning. The paper reports its best performance among comparable methods on CoSER and strong generalization on Cross-MR and LifeChoice.

6 Conclusion

  • CogDual uses a cognize-then-respond reasoning paradigm to produce contextually grounded and psychologically coherent responses through dual cognition.The conclusion presents CogDual as a role-playing language agent.Source: §6 · p. 9
  • Reinforcement learning with the ICLG and LSA reward schemes further improves CogDual over the supervised fine-tuning baseline.Source: §6 · p. 9
  • CogDual achieves the best performance among comparable methods on CoSER and shows strong generalization on Cross-MR and LifeChoice.Source: §6 · p. 9

Limitations

CogDual’s evaluation has three stated boundaries: model scale, language coverage, and how character-specific memory is accessed.

Limitations

  • The reinforcement-learning approach was not evaluated on larger-scale models such as Llama3.1-70B-Instruct because of computational constraints.The authors note that such models may further benefit from the reward design.Source: Limitations · p. 9
  • Experiments use only English datasets, leaving adaptability to non-English role-playing contexts such as Chinese unexplored.Source: Limitations · p. 9
  • Self-awareness extracts previously mentioned memory fragments from context without explicit retrieval of character-specific memory, which may omit relevant information.Source: Limitations · p. 9

Ethics Statement

The ethics statement describes the study’s aim, data provenance, and precautions against harm and misuse.

Ethics Statement

  • The research aims to equip role-playing agents with cognitive capabilities for contextually grounded and psychologically coherent responses.Source: Ethics Statement · p. 9
  • All research data come from publicly available sources, which the authors associate with transparency and reproducibility.Source: Ethics Statement · p. 9
  • The authors report measures to avoid harm to individuals or groups and a commitment to avoiding deception or information misuse.Source: Ethics Statement · p. 9

A Details about Experiments on CoSER

The experiments build supervised dual-cognition data from CoSER, then train and evaluate role-playing models using staged procedures and varied contexts.

A Details about Experiments on CoSER

  • Trajectory sampling alternates first- and third-person reasoning prompts, while quality checks verify the required format and cognitive fields and use GPT-4o to filter narrative- or character-misaligned reasoning.The checks include alignment with character background and consistency of dialogue.Source: A Details about Experiments on CoSER · p. 13
  • Training data vary instruction templates and randomly include or omit character profiles, plot summaries, and motivations to broaden contextual configurations.Source: A Details about Experiments on CoSER · p. 13
  • CoSER supplies interactions from 771 novels and 17,966 characters; filtering scenes for 400 sampled characters yields 38,724 cognitively relevant instances and 17,762 supervised examples.Each retained instance is used to sample four cognitive trajectories, from which high-quality examples are selected.Source: A Details about Experiments on CoSER · p. 13
  • The RL stage samples 10,000 prompts from the broader cognitively filtered dataset, including contexts outside the supervised set, to support training stability and generalization.Source: A Details about Experiments on CoSER · p. 13
  • SFT runs for two epochs; RL samples 16 response trajectories per prompt and weights the two rewards 7:3, with configurations detailed in Tables 4 and 5.The stated settings include batch sizes of 64 for SFT and 8 for RL, and a 1e-5 SFT learning rate.Source: A Details about Experiments on CoSER · p. 13, Table 4 · p. 14, Table 5 · p. 14
  • CoSER testing uses the final 10% of data from 100 original novels and 100 unseen books, with role-play performance assessed across multiple dimensions.Source: A Details about Experiments on CoSER · pp. 13–14

B The Effect of Evaluator Choice on Model Generalization

CogDual outperforms baselines across the tested evaluators, and its RL version outperforms its SFT version. The additional evaluators address a possible GPT-4o evaluation bias.

B The Effect of Evaluator Choice on Model Generalization

C The Details of Human Evaluation

The appendix describes human evaluation and reference prompts, then illustrates CogDual’s character- and context-aligned responses through case studies. The supplied passages do not report the human-evaluation win-rate values.

C The Details of Human Evaluation

  • Five character-familiar evaluators selected the best of five models’ responses for 50 CoSER test samples across four dimensions, with win rates reported in Table 8.The dimensions were Storyline Consistency, Anthropomorphism, Character Fidelity, and Storyline Quality; the provided caption gives no win-rate values.Source: C The Details of Human Evaluation · p. 14, Table 8 · p. 15

D A Reference Prompt for the CoT Data Construction

E A Reference Prompt for Constructing CogDual Training Data

F A Reference Prompt for Cognitive-Based Chain-of-Thought

  • CB-CoT guides an LLM through in-context definitions of dual cognition and requests outputs in CogDual’s structured format.The prompt design is shown in Table 13.Source: F A Reference Prompt for… · pp. 14–15

G A Reference Prompt for Semantic Matching

H Case Study

  • Two representative CogDual reasoning cases are selected from the test set to examine the model’s cognitive reasoning process.Source: H Case Study · p. 15

H.1 Case 1: Catherine Leigh Dollanganger in Flowers in the Attic

  • In the Flowers in the Attic example, Catherine responds to Bart’s distress with empathy and practical advice, drawing on situational cues, personal memories, and responsibility.The passage characterizes the resulting response as contextually appropriate and consistent with her persona and the narrative.Source: §H.1 · p. 15, A Reference Prompt for the CoT Data… · p. 16

H.2 Case 2: Nicholas of Morimondo in The Name of the Rose

The case studies portray CogDual responses as shaped by environmental and social cues alongside characters’ memories, emotions, and motivations. A configuration comparison contrasts the full model with vanilla and single-awareness variants.

H.2 Case 2: Nicholas of Morimondo in The Name of the Rose

  • Nicholas’s measured reply protects sensitive abbey details while acknowledging the political importance of the librarian position, following reasoning about the crypt, William’s intentions, and Nicholas’s memories.The passage links the reply to his pride in the abbey’s legacy and caution.Source: §H.2 · p. 16

H.3 Case 3: An example of an extracted conversation and its multi-agent simulation

  • In the Dragon Reborn simulation, characters’ internal struggles and distinctive reasoning are tied to their outward actions, producing interactions described as believable and narrative-faithful.The passage also characterizes the role-play as contextually appropriate and psychologically plausible compared with standard baselines.Source: §H.3 · p. 17

H.4 Case 4: Role Behavior Under Different Cognitive Configurations

  • The vanilla response is generic, while situational-only and self-awareness-only variants have distinct blind spots; the full CogDual response integrates both to be emotionally attuned and tactful.The full model’s response is also described as faithful to the character’s goals and memories.Source: §H.4 · p. 17, Table 26 · p. 31

A Reference Prompt for Generating Dual Cognitive Reasoning before Character Responses

  • The reasoning prompt moves from situation judgments and interpretations of others’ behavior to self-awareness, then a strategy before the character’s next action.Inputs include character profiles, scene context, historical dialogue, and the character’s psychological or motivational state.Source: A Reference Prompt for Generating Dual… · p. 18

1. Situational Awareness Analysis

2. Self-Awareness Analysis

Related papers

Loading 2507.17147v1…