PaperlayerView exact revision on arXiv

Source-linked AI summary

Video2Roleplay: A Multimodal Dataset and Framework for Video-Guided Role-playing Agents

Xueqiao Zhang, Chao Zhang, Jingtao Xu, Yifan Zhu, Xin Shi, Yi Yang, Yawei Luo

arXiv:2509.15233v1cs.MMcs.CLcs.CV

TL;DR

Role-playing agents mainly rely on static modalities, while effective integration of video’s dynamic information remains underexplored and long videos can introduce redundancy and memory costs. The paper introduces dynamic role-playing and a large-scale dataset, combining adaptive temporal sampling with dynamic profiles and static dialogue and summary contexts. Extensive experimental results and analyses demonstrate the framework’s effectiveness and superior performance in role-playing agents.

  • Problem

    Role-playing agents mainly rely on static modalities, while effective integration of video’s dynamic information remains underexplored and long videos can introduce redundancy and memory costs.

    Source: §1 · pp. 1–2

  • Method

    The paper introduces dynamic role-playing and a large-scale dataset, combining adaptive temporal sampling with dynamic profiles and static dialogue and summary contexts.

    Source: §1 · p. 2

  • Results

    Extensive experimental results and analyses demonstrate the framework’s effectiveness and superior performance in role-playing agents.

    Source: §1 · p. 2, §6 · p. 9

  • Takeaways & Limitations

    The combined profiles support responses consistent with character identity and narrative context.

    Source: §1 · p. 2

    Limited computational resources prevented experiments with a larger base model, denser frame sampling, or full-parameter fine-tuning; the study uses LoRA fine-tuning.

    Source: Limitations · p. 9

Abstract

from arXiv · show

Role-playing agents (RPAs) have attracted growing interest for their ability to simulate immersive and interactive characters. However, existing approaches primarily focus on static role profiles, overlooking the dynamic perceptual abilities inherent to humans. To bridge this gap, we introduce the concept of dynamic role profiles by incorporating video modality into RPAs. To support this, we construct Role-playing-Video60k, a large-scale, high-quality dataset comprising 60k videos and 700k corresponding dialogues. Based on this dataset, we develop a comprehensive RPA framework that combines adaptive temporal sampling with both dynamic and static role profile representations. Specifically, the dynamic profile is created by adaptively sampling video frames and feeding them to the LLM in temporal order, while the static profile consists of (1) character dialogues from training videos during fine-tuning, and (2) a summary context from the input video during inference. This joint integration enables RPAs to generate greater responses. Furthermore, we propose a robust evaluation method covering eight metrics. Experimental results demonstrate the effectiveness of our framework, highlighting the importance of dynamic role profiles in developing RPAs.

1 Introduction

Existing role-playing agents rely mainly on static modalities, while integrating video effectively remains challenging, especially for long videos with redundant information. Video2Roleplay introduces dynamic role profiles, a 60k-video and 700k-dialogue dataset, and a framework combining adaptive sampling with dynamic and static profiles.

1 Introduction

  • The paper identifies limited video-modality exploration and the challenge of integrating dynamic video information with static modalities, compounded by redundancy and memory costs in long videos.These gaps motivate dynamic role-playing profiles and methods for efficiently representing video.Source: §1 · p. 2
  • Video2Roleplay introduces video-based dynamic role profiles and a framework that combines adaptive temporal sampling with dynamic and static representations.The framework uses sampled frames in temporal order, training-video dialogues for fine-tuning, and an input-video summary for inference.Source: §1 · p. 2
  • The authors report that extensive experiments demonstrate superior performance for their framework.The introduction also describes the framework as comprehensive and evaluates it through a series of experiments and metrics.Source: §1 · p. 2
  • Role-playing-Video60k contains 60k videos and 700k dialogues across varied categories, durations, and scenarios.The dataset is designed to support the development of dynamic role profiles for role-playing agents.Source: §1 · p. 2

2 Related Work

Related work covers static role-playing agents and datasets, video-understanding systems, and multimodal language models. The paper positions its work at the intersection of these areas by introducing video into role-playing agents.

2.1 Static Role Playing

  • Static role-playing work includes dialogue datasets and persona frameworks, such as ChatHaruhi (Li et al., 2023a), RoleLLM (Wang et al., 2024a), and CoSER (Wang et al., 2025b).The surveyed work also includes multimodal role-playing agents and benchmarks, including MMrole (Dai et al., 2024) and RoleMRC (Lu et al., 2025).Source: §2.1 · pp. 2–3

2.2 Video Understanding

2.3 Multimodal Large Language Model

3 Dataset Curation

The dataset draws on social-media videos and pairs them with captions and dialogues. Its curation covers three video types, staged captioning, and generated question-answer pairs that are filtered for quality.

3 Dataset Curation

  • The dataset draws daily-life videos, lifestyle vlogs, and personal documentaries from Xiaohongshu, Douyin, Weibo, and Bilibili, with captions and related dialogues.It is intended as a resource for role-playing-agent development.Source: §3 · p. 4

3.1 Video Type

  • The dataset groups videos into Live, Vlog, and Documentary types, capturing motion, daily life, and personal experiences, respectively.Live clips emphasize motion around a specific moment; documentaries often include scene transitions and life events.Source: §3.1 · p. 4

3.2 Video Caption

  • Captioning uses 64 uniformly sampled video segments, generates a description for each representative frame, then summarizes descriptions in video order.Captions support question-answer generation during fine-tuning, while input-video captions provide role context at inference.Source: §3.2 · p. 4, Figure 3 · p. 4

3.3 Dialogue Generation and Filtering

  • An LLM generates question-answer pairs from video captions and prompts, and filtering removes unanswerable or erroneous outputs and stock refusals.The prompt includes the role definition, detailed video description, social-media comment examples, and generation instructions.Source: §3.3 · pp. 4–5

4 Methodology

The framework combines adaptive video sampling with dynamic frame-sequence profiles and static role information from training dialogues and video summaries.

4.1 Adaptive Temporal Sampling

  • Sampling varies by video length: short lives are sampled densely, vlogs every five seconds, and long documentaries by key scene events, up to 128 frames.For long videos, candidate frames are sampled at one per second, filtered by frame difference and group variation, then merged when adjacent frames exceed a CLIP-similarity threshold.Source: §4.1 · pp. 5–6
  • For long videos, the method selects representative frames within uniform groups and merges adjacent frames whose CLIP similarity exceeds a threshold.Candidates are first selected using frame differences; the final key-frame set is formed by iterative adjacent-frame merging.Source: §4.1 · p. 6

4.2 Dynamic Role Profile Representation

  • The dynamic role profile presents sampled video frames as ordered <image> tokens, preserving the video's original temporal sequence.Frames are transformed and stacked into a tensor.Source: §4.2 · p. 6

4.3 Static Role Profile Representation

  • Supervised fine-tuning learns static role information from dialogues centered on the roles and scenes in the videos.Figure 5 shows the fine-tuning data format, with <video> marking where the video is inserted.Source: §4.3 · p. 6, Figure 5 · p. 6
  • At inference, a summary agent captions successive video segments and uses chain-of-thought to condense them into context for in-context role-playing.The resulting video summary guides the language model's role-playing response.Source: §4.3 · p. 6

5 Experiment

The experiments compare the framework with general and role-playing language-model baselines, evaluate its components and resource use, and report its role-playing performance and fine-tuning trade-offs.

5.1 Experimental Settings

  • Evaluation uses a randomly split 57k/3k training-inference dataset and 328 manually selected test questions, with three rounds of GPT-4o and o3-mini judgments averaged per sample.The evaluator API temperature is set to 0.0.Source: §5.1 · p. 6

5.2 Evaluation Metric

  • The study uses eight evaluation metrics and a 21-question human study comparing the model with Gemini 2.5 Pro Preview 0325.Most baselines lack direct video input, so video-text relevance is evaluated only in the ablation study.Source: §5.2 · pp. 6–7

5.3 Baseline

  • The comparison includes 16 general-purpose advanced LLMs and three role-playing expertise models.The paper reports framework results alongside both baseline types.Source: §5.3 · p. 7, Table 2 · p. 7

5.4 Comparative Studies

  • The framework performs comparably across metrics to substantially larger baselines and achieves state of the art on human-likeness.The authors report that fine-tuned RPAs tend to produce shorter, more concise responses than untrained RPAs.Source: Table 2 · p. 7, §5.4 · p. 8

5.5 Analysis

  • The 60k-video, 700k-conversation dataset with video SFT outperforms the base model across metrics, while video input improves almost all metrics over dialogue-only or image-and-dialogue controls.Video-text relevance improves with video, but its score remains low.Source: §5.5 · p. 8, Table 3 · p. 8
  • Replacing summary context with full descriptions of sampled frames does not improve any metric; the summary-context model performs better.The authors characterize the CoT-guided summary as more concise while capturing key points from the longer descriptions.Source: §5.5 · p. 8, Table 3 · p. 8
  • On a single case using two RTX A6000 GPUs, inference time grows approximately linearly from 32 to 64 frames, while FlashAttention reduces attention memory from O(N^2) to O(N).Below 32 frames, inference time remains nearly constant and does not significantly exceed single-image-and-text input.Source: §5.5 · p. 9, Table 4 · p. 9
  • SFT improves role-playing but reduces performance on some general benchmarks, without causing a collapse in generalization or significant degradation in multimodal understanding.The authors describe the alignment tax as relatively minor compared with the role-playing gains.Source: §5.5 · p. 9

6 Conclusion

The conclusion presents video-based dynamic role-playing, a varied video dataset, and a framework combining adaptive sampling with dynamic and static profiles. The authors report framework effectiveness and identify social applications and digital humans as future directions.

6 Conclusion

  • The paper introduces dynamic role-playing through video and a varied, large-scale video dataset, alongside a framework combining adaptive sampling with dynamic and static role profiles.The dataset covers varied video types, lengths, and roles.Source: §6 · p. 9
  • The authors report that experiments and analyses demonstrate the framework's effectiveness.They identify social applications and connections with digital humans as potential future directions for improving user interaction.Source: §6 · p. 9

Limitations

The authors could not test larger base models or denser frame sampling because of computational-resource limitations.

Limitations

  • Computational resources prevented testing a larger-scale base model or more densely sampled video frames.The authors identify model size and frame-acquisition density as areas for further exploration.Source: Limitations · p. 9

Ethics Statements

The authors caution that the model may have minimal safety alignment and could generate toxic or harmful content under induction.

Ethics Statements

  • The model may have minimal safety alignment and could generate toxic or harmful content under induction.The authors say the dataset and LLM are for research purposes and call for careful future safety alignment.Source: Ethics Statements · p. 9

A.1 Baseline Model URL List

The appendix provides URLs for model APIs used in the research.

A.1 Baseline Model URL List

  • Figure 6 lists URLs for the model APIs involved in the research.The supplied passage identifies the list but does not enumerate the URLs.Source: §A.1 · p. 13, Figure 6 · p. 13

A.2 Dataset

The appendix describes dataset annotation and dialogue-generation procedures, reports a user study, and gives implementation details. It also documents challenges encountered during video annotation.

A.2 Dataset

  • Figure 7 summarizes the dataset’s video-type distribution by duration.The appendix presents the distribution as a statistical analysis of video types based on duration.Source: §A.2 · p. 13, Figure 7 · p. 13
  • For live, vlog, and documentary videos, caption-generation max_new_token settings were 1024, 2048, and 4096; six SOTA LLMs generated dialogues at temperature 1.0.The listed models are Qwen-Max, Deepseek-R1, GPT-4.1, GPT-4o, Claude-3-7-Sonnet-Thinking, and Gemini-2.5-Pro-Exp.Source: §A.2 · p. 13
  • Annotation challenges included repetitive content, limited diversity for static-scene videos, and truncated captions for some videos with frequent scene changes.The authors manually reviewed the first two issues and raised token limits for affected videos with truncation.Source: §A.2 · pp. 13–14
  • Dialogue generation used video captions and prompts to create candidates, followed by regex extraction, prompt-based selection for scene relevance, and formatting cleanup.The generation and filtering prompts are shown in Figures 9 and 10.Source: §A.2 · pp. 14–15

A.3 User Study

  • 84 participants (57%) preferred the authors’ responses, compared with 52 (35%) for a closed-source model; 11 (8%) were unsure.Participants chose which response was closer to a real social-media blogger’s response.Source: §A.3 · p. 15

A.4 Implementation Details

  • Training used AdamW with a learning rate of 4e-5, weight decay of 5e-2, warm-up ratio of 3e-2, and one epoch.The optimizer is identified as AdamW.Source: §A.4 · p. 15

Related papers

Loading 2509.15233v1…