PaperlayerView exact revision on arXiv

Source-linked AI summary

FlowAct-R2: Beyond Talking Avatar via Streaming Multimodal References and Proactive Agent Planning

Ziyao Huang, Zhengkun Rong, Shiyang Qin, Shuang Liang, Wentao Hu, Yuxuan Luo, Yuan Zhang, Mingyuan Gao

arXiv:2609.35728v1cs.CV

TL;DR

Open-ended live interaction calls for digital humans that maintain context, organize activity proactively, and respond to unexpected input while coordinating multimodal behavior. FlowAct-R2 pairs streaming multimodal video generation with an agent that prepares session assets and schedules responses online; it reports real-time 720p generation and higher GSB scores than Vidu-S1 on video quality and real-time interaction.

  • Problem

    Many existing digital-human systems are confined to a single scenario, while open-ended interaction calls for coherent long-term context, proactive activity, and coordinated multimodal behavior.

    Source: §1 · pp. 1–2

  • Method

    FlowAct-R2 combines a streaming multimodal reference generator with an agent that prepares personas, agendas, and reusable skills in advance, then schedules behavior using audience input and interaction history.

    Source: §1 · p. 2, §2.2.1 · p. 3, §2.2.2 · p. 3

  • Results

    FlowAct-R2 generates real-time 720p video across four live settings and achieves GSB scores of +54.76% for video quality and +40.48% for real-time interaction against Vidu-S1.

    Source: §1 · p. 2, §3.1 · p. 4, §3.2 · p. 5

  • Takeaways & Limitations

    The system supports expressive and task-oriented behavior across video chatting, live shopping, entertainment streaming, and live vlogging, beyond conventional question answering.

    Source: §1 · p. 2, §4 · p. 7

Abstract

from arXiv · show

We present FlowAct-R2, a framework for interactive humanoid video generation that combines continuous multimodal control with proactive agent planning. Our method consists of two coupled components. First, a Streaming Multimodal Reference Diffusion Transformer adapts the pretrained Seedance 2.0 Mini reference-to-video backbone to accept rolling action prompts, streaming audio, and dynamically updated image, audio, and video references. Video-driven rotary positional embeddings align reference chunks with the generation timeline, while reference-plus-image conditioning and partially noised historical motion frames preserve appearance and avoid accumulated drift. Second, a Proactive Interaction Agent separates pre-online planning from online scheduling and response: it prepares a persona, a long-horizon agenda, and reusable multimodal skills in advance, then autonomously schedules behaviors, responds to audience input, and handles interruptions during a live session. FlowAct-R2 supports real-time 720p generation and hour-scale streaming across entertainment streaming, live shopping, video chatting, and live vlogging.

1 Introduction

Existing digital humans often serve one scenario, while open-ended live interaction calls for coherent long-term context, proactive activity, and coordinated multimodal behavior. FlowAct-R2 pairs continuous multimodal video generation with an agent that plans, schedules, and adapts behavior during a session.

1 Introduction

  • Many digital-human systems remain confined to dialogue, performance, or commerce, while open-ended interaction calls for coherent context, proactive activity, and coordinated speech, motion, object, and scene actions.The generator must handle changing references without losing identity or temporal continuity, while the agent prepares activities and responds to interruptions without disrupting the session.Source: §1 · pp. 1–2
  • FlowAct-R2 pairs a streaming multimodal reference generator with a proactive interaction agent to address these coupled challenges.The generator accepts rolling prompts and changing image, audio, and video references; the agent separates pre-online planning from online scheduling and response.Source: §1 · p. 2
  • The system supports real-time text and audio interaction across video chatting, live shopping, entertainment streaming, and interactive gaming, with expressive and task-oriented actions beyond question answering.It also supports hour-level ultra-long video generation at 720p.Source: §1 · p. 2

2 Method Overview

FlowAct-R2 couples a streaming video generator with an agent that prepares long-horizon session assets in advance and schedules behavior online. The generator accepts changing multimodal conditions, while the agent coordinates audience input and ongoing activity.

2 Method Overview

  • The system couples a Streaming Multimodal Reference Diffusion Transformer that produces video with a Proactive Interaction Agent that determines behavior and its schedule.Offline preparation supplies persona, agenda, and skill assets; online decisions combine them with audience events and interaction history.Source: §2 · p. 2

2.1 Streaming Multimodal Reference Diffusion Transformer.

  • The generator adapts the pretrained Seedance 2.0 Mini reference-to-video backbone for autoregressive streaming with rolling prompts, driving audio, and switchable image, audio, or video references.Diffusion Forcing assigns independent noise levels to sequence tokens while retaining bidirectional spatiotemporal modeling.Source: §2.1 · p. 2
  • Video-driven RoPE aligns incoming reference chunks with the generation timeline, while reference-plus-image conditioning anchors identity and appearance.Partially noised historical motion frames improve robustness to imperfect histories and reduce error accumulation; audio-driven training synchronizes speech, lips, expressions, and head poses, with coarse-to-fine inference restoring detail.Source: §2.1 · pp. 2–3

2.2.1 Offline planning.

  • The agent moves long-horizon preparation outside the latency-sensitive interaction loop by deriving a persona, organizing an agenda, and compiling reusable multimodal skills.These assets are prepared from multimodal character material and support behavior consistent with the persona and planned activities.Source: §2.2 · p. 3, §2.2.1 · p. 3

2.2.2 Online scheduling and response.

  • Online, the agent combines prepared assets with audience events and interaction history to decide what behavior to perform and when, including whether to initiate, overlay, defer, interrupt, or resume it.Idle, listening, speaking, and skill-execution periods supply rolling prompts, audio, and reference updates to the generator; history informs later decisions.Source: §2.2.2 · p. 3

3 Results

FlowAct-R2 demonstrates real-time 720p generation across four live settings and reports higher human-evaluation GSB scores than Vidu-S1. Qualitative comparisons show varied conversational behavior and more faithful product-reference switching than Vidu-S2 in the described examples.

3.1 Streaming and application demonstrations.

  • FlowAct-R2 generates real-time 720p video in demonstrations spanning entertainment, shopping, video chat, and live vlogging.Examples include singing and dancing, product presentation and try-on, expressive turn-taking, and audience-driven vlog branching.Source: §3.1 · p. 4, Figure 3 · p. 4, Figure 4 · p. 5

3.2 Comparison with Vidu-S1.

  • GSB scores against Vidu-S1 are +54.76% for video quality and +40.48% for real-time interaction.The comparison also describes improvements in visual clarity, motion naturalness and expressiveness, action responsiveness, and contextual appropriateness of dialogue.Source: §3.2 · p. 5
  • During listening and response, FlowAct-R2 varies gaze, head pose, and facial expression, while Vidu-S1 remains close to a fixed frontal pose, smile, and hand gesture.The comparison covers the same audience request and subsequent response.Source: §3.2 · p. 5, Figure 5 · p. 6

3.3 Qualitative comparison with Vidu-S2.

  • Across the described product switches, FlowAct-R2 retains packaging details and proportions that Vidu-S2 blends or alters, including a red layout, a green pouch pattern, and a shallow octagonal box.In the separate cylinder example, Vidu-S2 replaces the pale patterned wrap with multicolored stripes.Source: §3.3 · p. 6, Figure 6 · p. 7

4 Conclusion

FlowAct-R2 combines streaming multimodal reference generation, proactive planning, and behavior scheduling to support expressive and task-oriented activity across live scenarios. The conclusion frames this combination as moving digital humans beyond reactive talking avatars toward persistent visual agents that plan and act through continuous video.

4 Conclusion

  • FlowAct-R2 combines rolling multimodal references, offline preparation of personas, agendas, and skills, and online behavior scheduling across four live scenarios.The conclusion characterizes the result as persistent visual agents that plan and act through continuous video, beyond reactive talking avatars.Source: §4 · p. 7

Related papers

Loading 2609.35728v1…