Source-linked AI summary
Beyond One World: Benchmarking Super Heros in Role-Playing Across Multiversal Contexts
Perapard Ngokpol, Kun Kerdthaisong, Pasin Buakhaw, Pitikorn Khlaisamniang, Supasate Vorathammathorn, Piyalitt Ittichaiwong, Nutchanon Yongsatianchot
TL;DR
Version-specific role-play and alignment between models’ reasoning and decisions remain underexplored. Beyond One World benchmarks these challenges across superhero versions, finding that cross-version consistency and alignment between thinking and acting remain difficult.
Problem
Role-play research has limited evidence on consistent version-specific character portrayal across contexts and on alignment between reasoning traces and decisions.
Source: §1 · p. 1Method
Beyond One World evaluates 30 heroes and 90 versions through Canon Events and Moral Dilemmas, scoring canonical accuracy and reasoning fidelity while separating thinking from acting.
Source: §1 · p. 2Results
Models often show divergent strengths in thinking and acting, while cross-version generalization remains difficult and chain-of-thought helps weaker models but can hurt stronger models’ canonical accuracy.
Source: §5.4 · p. 6Takeaways & Limitations
The findings point to a need for integrated reasoning–persona modeling, potentially combining structured world knowledge with dynamic narrative alignment.
Source: §5.4 · p. 7Current LLMs fall short of fully capturing version-specific portrayals, especially under timeline shifts and moral dilemmas.
Source: §5.4 · p. 7
Abstract
from arXiv · showhide
Large language models (LLMs) are increasingly used as role-playing agents, yet their capacity to faithfully and consistently portray version-specific characters -- for example, superheroes across comic and cinematic universes -- remains underexplored. Superhero canons such as Marvel and DC provide a rich testbed: decades of storytelling yield multiple incarnations of the same character with distinct histories, values, and moral codes. To study this problem, we introduce Beyond One World, a benchmark for character-grounded roleplay spanning 30 iconic heroes and 90 canon-specific versions. The benchmark comprises two tasks: (i) Canon Events, which probes factual recall of pivotal life stages, and (ii) Moral Dilemmas, which confronts models with ethically charged scenarios. We score responses for canonical accuracy and reasoning fidelity under a framework that separates internal deliberation ("thinking") from outward decisions ("acting"). We further propose Think-Act Matching, a metric that quantifies alignment between reasons and actions and serves as a proxy for model trustworthiness. Experiments across reasoning- and non-reasoning-oriented models yield three findings: (1) chain-of-thought prompting improves narrative coherence in weaker models but can reduce canonical accuracy in stronger ones; (2) cross-version generalization within a character remains a major obstacle; and (3) models often excel at either thinking or acting, but rarely both. Beyond One World exposes critical gaps in multiversal consistency and reasoning alignment, offering a challenging evaluation for role-playing LLMs.
1 Introduction
Version-specific role-play and alignment between a model’s reasoning and decisions remain underexplored. Beyond One World evaluates these challenges across superhero versions and separates internal deliberation from outward decisions.
1 Introduction
- Existing role-play benchmarks often target single-character consistency or factual recall, leaving version-specific portrayal across evolving timelines and reasoning–decision alignment underexplored.The challenge includes distinguishing younger, more idealistic incarnations from later, more cynical ones.Source: §1 · pp. 1–2
- Marvel and DC’s documented, canonically distinct histories provide ground truth for evaluating role-play across timelines and parallel worlds.The narrative record includes multiple versions of the same hero with evolving histories, moral codes, psychological profiles, motivations, and relationships.Source: §1 · p. 2
- Beyond One World benchmarks character-grounded role-play across 30 heroes and 90 canon-specific versions using Canon Events and Moral Dilemmas.Canon Events probes pivotal life stages; Moral Dilemmas presents ethically charged scenarios.Source: §1 · p. 2
- The benchmark uses an evaluation framework that separates internal “thinking” from outward “acting” for reasoning-oriented and standard models.It assesses character-consistent reasoning and decisions.Source: §1 · p. 2
2 Related Work
Prior work covers persona conditioning, character-alignment corpora, social personalization, and point-in-time consistency. Beyond One World focuses on multiple versions of the same character across temporal phases and universes.
2.1 From Persona-Chat to Modern RPLAs
- PersonaChat showed that short textual profiles can improve response consistency, while later work added goals, memories, and affect as conditioning signals.These approaches contributed to contemporary role-playing language agents.Source: §2.1 · p. 2
2.2 Character-Alignment Corpora
- Character-alignment corpora support fine-grained imitation through scene metadata and evolving relations or through broad dialogue metrics across literary figures.The Harry Potter Dialogue corpus pairs about 1k scenes with metadata; CharacterEval provides 1,785 dialogues and thirteen metrics.Source: §2.2 · p. 2
2.3 Evaluating Sociality and Personalization
- SocialBench evaluates persona behavior across scenarios, while PersoBench measures profile-tailored responses and finds personalization gaps despite fluent generation.SocialBench also reports that strong solo performance does not necessarily translate to robust group behavior.Source: §2.3 · pp. 2–3
2.4 Temporal Consistency on Characters
- TimeChara evaluates point-in-time hallucinations and reports that GPT-4o can leak future facts or inconsistent traits; expert decomposition mitigates but does not eliminate the issue.Beyond One World extends this line by testing multiple versions of the same character across phases and timelines.Source: §2.4 · p. 3
3 Dataset Creation
The benchmark curates Marvel and DC heroes into timeline variants and evaluates factual recall alongside ethical decision-making. Its dilemmas are character-grounded scenarios spanning archetypal moral conflicts.
3 Dataset Creation
- The collection contains 30 manually curated Marvel and DC heroes, each represented in childhood, pre-hero, and hero phases, for 90 versions.Arcs in which a chosen figure adopts a villainous identity were excluded to avoid role ambiguity.Source: §3 · p. 3
3.1 Canon Events
- Canon Events uses 1,346 hand-curated multiple-choice questions about pivotal lore moments across childhood, pre-hero, and active-hero phases.Each timeline variant receives three childhood questions, three pre-hero questions, and nine hero-phase questions.Source: §3.1 · p. 3
3.2 Dilemma Situations
- The Moral Dilemma task contains 1,080 scenarios, with three scenarios per character version spanning four conflict types, manually filtered for canon fidelity and clarity.Claude Sonnet 3.7 was used to synthesize the scenarios.Source: §3.2 · p. 3
3.2.1 Dilemma Generation Methodology
- Dilemma generation grounds scenarios in each character’s lore and moral framework, then creates three sequential variations per character-conflict pairing to reduce redundancy.Each output includes a situation, two competing choices, and explicit consequences for each choice.Source: §3.2.1 · pp. 3–4
3.2.2 Moral Conflict Taxonomy
- The taxonomy includes sacrifice of a few for many, heroic ideals versus morally questionable means, and duty versus personal relationships or well-being.The taxonomy frames dilemmas as tensions in moral reasoning across diverse ethical frameworks.Source: §3.2.2 · p. 4
4. Ends Justify the Means vs. Moral
The moral-code-versus-outcome dilemma tests whether beneficial outcomes override a hero’s established principles. The benchmark’s task summary reports phase-specific Canon Event counts and balanced dilemma situations.
4. Ends Justify the Means vs. Moral
- Moral Code vs. Outcome pits adherence to established principles against compromising them for objectively beneficial results.The passage frames this as a conflict between utilitarian outcomes and deontological constraints, with Batman’s refusal to kill as an illustration.Source: §4 · p. 4
- Task 1 contains 270 Childhood, 270 Pre-Hero, and 806 Hero questions, while Task 2 consists of balanced dilemma situations.These are the task counts reported in the benchmark summary.Source: Table 2 · p. 4
4 Experimental Setups
The experiments use task-specific prompts and cross-character tests to evaluate role-play across character versions. Their reasoning-analysis framework separates deliberation from action and scores both character fidelity and alignment between them.
4.1 Main Experiment
- Experiments evaluate character role-play through multiple-choice question answering with structured prompting strategies.The main experiment covers the Canon and Dilemma tasks.Source: §4.1 · p. 4
4.1.1 Structured Prompting Design
- Prompts provide minimal character context, identify the character and source material, and insert a task-formatted question.The base template asks the model to act and think as the named character from the specified lore.Source: §4.1.1 · pp. 4–5
4.1.2 Cross-Character Evaluation
- Cross-character evaluation tests whether models distinguish same-named characters from different fictional universes.The protocol uses variants of a character with distinct source materials.Source: §4.1.2 · p. 5
Cross-Character Evaluation
- The example protocol tests each Spider-Man variant on questions designed for the other variants.It gives CID4 from the Marvel Cinematic Universe answering questions designed for CID5 and CID6.Source: Cross-Character Evaluation · p. 5, §4.1.2 · p. 5
4.2 Reasoning analysis framework
- Responses to Canon and Dilemma tasks are segmented into thinking and acting, then judged using character attributes.The framework treats thinking as internal deliberation and acting as decision-making behavior or physical appearance; Think-Act Matching uses the maximum cosine similarity between paired embeddings as an alignment proxy.Source: §4.2 · p. 5, Figure 2 · p. 5
5 Results and Discussion
Results show that cross-character transfer remains difficult, while chain-of-thought prompting has mixed effects on canonical accuracy and reasoning–action alignment. Models also diverge in their thinking and acting scores, and the discussion identifies gaps in multiversal consistency and role-play trustworthiness.
5.3 Thinking vs. Acting
- Under CoT, r1 shows the largest increase in reasoning–action alignment despite reduced canonical accuracy.The paper interprets this as a way reasoning traces can improve trustworthiness without strictly improving correctness.Source: §5.3 · p. 6
5.2 Cross-Character Generalization
- Sonnet3.5 achieves 0.69 on Dilemma Cross and 0.65 on Canon Cross, while cross-character accuracy is substantially below within-character evaluation.The results indicate difficulty distinguishing overlapping but divergent timelines.Source: Figure 3 · p. 6, §5.2 · p. 6
5.4 Discussion
- CoT can improve coherence in weaker models but harm stronger models through over-generation or departures from canon.The discussion relates this mixed effect to prior findings on CoT in knowledge-intensive tasks (Feng et al., 2025).Source: §5.4 · p. 6
5.1 Overall Performance on Canon Events
- For Canon Events, CoT raises 4o-mini accuracy by +0.02 and F1 by +0.024, but reduces accuracy for Sonnet3.5 and Sonnet3.7.For r1, reasoning–action cosine similarity rises by +0.075 even as raw accuracy falls.Source: §5.1 · pp. 5–6, Table 3 · p. 7
5.3 Thinking vs. Acting
- Gemini2 has stronger thinking (3.67) but weaker action alignment, while Sonnet3.7 scores highest in acting (3.65) with moderate reasoning (3.03).The results illustrate that internal deliberation and persona-faithful action do not necessarily coincide.Source: §5.3 · p. 6, Figure 4 · p. 7
5.4 Discussion
- Current models fall short on version-specific portrayals, especially under timeline shifts and moral dilemmas, while thinking and acting can remain misaligned.The authors suggest exploring integrated reasoning–persona modeling, potentially combining structured world knowledge with dynamic narrative alignment.Source: §5.4 · pp. 6–7
A Our Benchmark
The benchmark divides evaluation into Canon Events and Dilemma Situations, with separate prompts for canonical questions and dilemmas. The supplied captions identify inference-configuration tables for answering and generation.
A Our Benchmark
- Canon Events cover Childhood, Pre-Hero, and Hero questions, while Dilemma Situations cover four ethical contrasts, with 270 questions for each contrast.The four contrasts are Save one vs. the greater good, Hero or Villain, Duty vs. Desire, and Moral Code vs. Outcome.Source: A Our Benchmark · p. 10
- Figure 5 distinguishes the three Canon Event phases from the balanced Dilemma Situation task.Blue denotes Childhood, Pre-Hero, and Hero question types; red denotes Dilemma Situation.Source: Figure 5 · p. 10
B Model Inference Configs
- Tables 4 and 5 are identified as inference configurations for answering and generation, respectively.Source: Table 4 · p. 10, Table 5 · p. 10
C.2 Task prompt
- The canonical-question prompt asks the model to play and think as a named character from specified lore and answer a multiple-choice question.Source: §C.2 · p. 10
- The dilemma prompt supplies a character, lore, scenario, two choices and consequences, and a dilemma-type question.Source: §C.2 · p. 11
C.3 Reasoning Analysis
The scoring prompt evaluates character role-play on separate thinking and acting dimensions, each scored from 0 to 5.
C.3 Reasoning Analysis
- Thinking and acting receive separate 0–5 scores for how closely the response matches the reference character.Thinking concerns the response’s reasoning; acting concerns whether the response behaves like the character.Source: §C.3 · p. 11
- The prompt supplies the character’s name, source, and attributes alongside the response being evaluated.Source: §C.3 · p. 11
- The requested output is a thinking-score, acting-score pair, exemplified as “3,2.”Source: §C.3 · p. 11
D Full Results
The passage describes a curated selection of Marvel and DC heroes, filtered for documented backstories and consistent heroic intent.
D Full Results
- The selection spans Marvel and DC universes, including cinematic and animated universes and comics-related databases.Source: §E.1 · p. 11
- Chosen heroes were required to have well-documented backstories, consistently purehearted intentions, and no anti-hero categorization.Source: §E.1 · p. 11
- The selection criteria were applied to keep the chosen heroes coherent with the research scope and objectives.Source: §E.1 · p. 11
E.2 Hero Attributes
Character profiles use five attributes, combining general character information with psychological traits; the Thor MCU profile illustrates this scheme.
E.2 Hero Attributes
- The five profile attributes are age, power, MBTI, race, and Enneagram.Source: §E.2 · p. 11
- Age, power, and race come from universe-specific wiki pages, while MBTI and Enneagram are drawn from Personality Database user assessments.Source: §E.2 · p. 11
- Thor’s MCU profile gives his age as around 1,500 years and uses user-contributed assessments for his MBTI and Enneagram traits.The age is described as revealed in Avengers: Infinity War.Source: §E.2 · p. 11, Figure 6 · p. 11
E.3 Question Constructions
The benchmark’s question-construction process uses structured annotation to capture character knowledge and decision-making. It separates canon-event recall questions from dilemmas that test decisions consistent with a character’s personality, morals, and values.
E.3 Question Constructions
- A custom annotation platform combines GPT-4o-mini assistance with expert oversight, with experts responsible for final validation, refinement, and approval.The platform uses structured templates for the benchmark scenarios.Source: §E.3 · pp. 11–12, Figure 9 · p. 14
E.3.1 Generation Structure
- Canon Events test factual recall of established character-history events through multiple-choice questions, while Dilemmas assess nuanced, character-consistent decisions.The canon-event questions target established lore; dilemma scenarios focus on personality, morals, and values.Source: §E.3.1 · pp. 12–13
- Experts define each canon event’s character and universe, timing, location, factual description, and question; GPT-4o-mini generates the distractor answers.Experts draft the question and provide the canonically correct answer.Source: §E.3.1 · pp. 12–13
- Dilemma scenarios specify the character and universe, time, location, context, and dilemma type.Examples of dilemma types include save versus sacrifice, hero or villain, and duty versus desire.Source: §E.3.1 · p. 13