PaperlayerView exact revision on arXiv

Source-linked AI summary

Towards Interactive Intelligence for Digital Humans

Yiyi Cai, Xuangeng Chu, Xiwei Gao, Sitong Gong, Yifei Huang, Caixin Kang, Kunhang Li, Haiyang Liu, Ruicong Liu, Yun Liu, Dianwen Ng, Zixiong Su, Erwin Wu, Yuhan Wu, Dingkun Yan, Tianyu Yan, Chang Zeng, Bo Zheng, You Zhou

arXiv:2512.13674v2cs.CVcs.CLcs.GRcs.HC

TL;DR

Digital humans often imitate surface behavior without interaction logic, despite the need for responsive, coherent multimodal engagement. Mio combines five specialized modules and scores 76.8 on the Interactive Intelligence Score, 7.8 points above a composite of state-of-the-art baselines.

  • Problem

    Existing digital humans often reproduce surface behavior without interaction logic, while end-to-end systems face challenges in narrative consistency, responsiveness, and natural multimodal behavior.

    Source: §1 · p. 2

  • Method

    Mio is an end-to-end framework that organizes interactive digital humans into Thinker, Talker, Face Animator, Body Animator, and Renderer modules.

    Source: §1 · p. 2

  • Results

    Mio scored 76.8 on the Interactive Intelligence Score, a +7.8-point improvement over a composite of state-of-the-art baselines.

    Source: §9 · p. 34

  • Takeaways & Limitations

    The paper frames digital humans as autonomous agents capable of personality-aligned expression, adaptive interaction, and self-evolution.

    Source: §9 · p. 34

    Kodama-Tokenizer trails XY-Tokenizer on LibriTTS (0.81 vs 0.83), XCodec2.0 on Seed-TTS-Eval English (0.81 vs 0.89), and XY-Tokenizer on Chinese (0.84 vs 0.87).

    Source: §7.2.1 · p. 22

Abstract

from arXiv · show

We introduce Interactive Intelligence, a novel paradigm of digital human that is capable of personality-aligned expression, adaptive interaction, and self-evolution. To realize this, we present Mio (Multimodal Interactive Omni-Avatar), an end-to-end framework composed of five specialized modules: Thinker, Talker, Face Animator, Body Animator, and Renderer. This unified architecture integrates cognitive reasoning with real-time multimodal embodiment to enable fluid, consistent interaction. Furthermore, we establish a new benchmark to rigorously evaluate the capabilities of interactive intelligence. Extensive experiments demonstrate that our framework achieves superior performance compared to state-of-the-art methods across all evaluated dimensions. Together, these contributions move digital humans beyond superficial imitation toward intelligent interaction.

1 Introduction

Existing digital humans often imitate surface behavior without interaction logic, while real-time multimodal systems face challenges in consistency, responsiveness, and natural animation. Mio combines five specialized modules, and its experiments report advantages across speech, facial motion, body motion, persona fidelity, and visual identity.

1 Introduction

  • Existing digital humans are often imitative rather than autonomous, while real-time interaction is challenged by narrative and persona drift, stiff listening faces, motion-generation limits, and inconsistent multi-view identity.Traditional CG can require prohibitive production time, while generative workflows are limited to offline generation.Source: §1 · p. 2
  • Mio organizes interactive digital humans into Thinker, Talker, Face Animator, Body Animator, and Renderer modules for cognitive reasoning and multimodal embodiment.The Thinker supports narrative consistency and personality fidelity; the Face Animator generates facial dynamics during silence; the Body Animator generates real-time motion; the Renderer targets multi-view consistency.Source: §1 · p. 2
  • Mio reports advantages over baselines in speech generation, listening naturalness, body-motion quality and latency, persona fidelity, and multi-view identity preservation.More than 90% of users preferred its facial-animation results over DualTalk; body motion reached FID 0.057 on HumanML3D.Source: §1 · p. 2
  • The authors make the codebase, pretrained models, and proposed evaluation benchmark publicly available.Source: §1 · p. 3

2 Talker

The Talker turns Thinker text into expressive speech using compact, semantically and acoustically disentangled tokens. Kodama-Codec supplies the speech interface for an autoregressive Kodama-TTS system, trained on a large curated speech dataset.

2 Talker

  • The Talker converts Thinker text into natural, high-fidelity speech, prioritizing context understanding and generation efficiency.Source: §2 · p. 3

2.1.1 Challenges and Motivation

  • Speech tokenizers face trade-offs among compression, reconstruction quality, semantic–acoustic disentanglement, and streaming latency; Kodama targets lower rate and bitrate with separated information streams.Source: §2.1.1 · p. 3

2.1.2 Method

  • Kodama-Tokenizer uses semantic and acoustic encoders, RVQ codebooks, and a vocoder, with semantic and pitch reconstruction losses supporting disentanglement and prosody reconstruction.Source: §2.1.2 · p. 4
  • 1920× compression at 12.5 Hz with 8 codebooks yields a bitrate of 1 kbps; the tokenizer encodes semantic and acoustic streams separately for waveform decoding.Source: §2.1.2 · p. 4

2.2.1 Challenges and Motivation

  • Kodama-TTS models joint text and audio-token sequences autoregressively, using the original codec decoder to preserve encoder–decoder alignment without a separately trained diffusion refinement stage.Source: §2.2.1 · p. 5

2.2.2 Method

  • A dual-transformer setup predicts audio tokens from text and acoustic context; optional acoustic exemplars support in-context voice cloning, and the codec decoder renders generated tokens.Source: §2.2.2 · p. 5

2.3 Dataset

  • The curated dataset contains approximately 500k hours of multilingual speech spanning audiobooks, podcasts, and online video content.Source: §2.3 · p. 6

3 Facial Animator

UniLS generates coordinated 3D facial motion for both participants, combining speech-aligned speaking behavior with context-modulated listening behavior. Its two-stage training addresses listening stiffness, and qualitative comparisons report more expressive listening and better-aligned speaking motions.

3.1 Task Description

  • Dual-track audio drives both speakers’ 3D facial sequences, with speaking motion aligned to the speaker’s own audio and listening motion shaped by intrinsic behavior and partner audio.The outputs include FLAME-based expression parameters, head and jaw pose, and eye-gaze dynamics.Source: §3.1 · pp. 6–7
  • Listening stiffness arises because a speaker’s own audio strongly cues speaking motion, while the listener’s motion is only weakly related to speech.Directly mapping dual-track audio to both faces can yield low-variance, nearly static listener expressions.Source: §3.1 · p. 7

3.2 Approach: UniLS

  • UniLS first learns audio-free facial motion priors from unpaired video, then finetunes on paired dialogue using separate cross-attention to each speaker’s audio.The newly added cross-attention layers are trained from scratch while the pretrained backbone is finetuned with LoRA, retaining the learned internal motion priors.Source: §3.2 · pp. 7–8
  • Training uses 675.5 hours of conversational data and 546.5 hours of multi-scenario video, with 622.5 conversational hours used for training.The conversational data includes both speaking and listening motions, extracted as per-frame FLAME parameters.Source: §3.2 · p. 8

3.3 Result

  • UniLS produces more expressive, temporally diverse listening motions than ARTalk* and DualTalk, and shows excellent lip synchronization with realistic speaking expressions and head motion.The listening comparison highlights more natural head dynamics, mouth shapes, and micro-expressions; the speaking comparison notes alignment with phonetic articulation.Source: §3.3 · p. 9, Figure 5 · p. 9, Figure 6 · p. 9

4 Body Animator

FloodDiffusion turns time-varying Thinker instructions into physically plausible body motion through streaming diffusion, with a lower-triangular schedule enabling bounded-latency generation. The section reports online, editable animation with 0.057 FID and illustrates prompt-timing changes and stopping motion with a rest prompt.

4.1 Task Description

  • The task is causal streaming synthesis of physically plausible motion from Thinker instructions, with mid-gesture edits, multi-rate controls, and long-horizon coherence as challenges.The motion representation has 263 dimensions, and the example frame budget is 33 ms at 30 FPS.Source: §4.1 · p. 10

4.2 Approach: FloodDiffusion

  • FloodDiffusion uses diffusion forcing to support flexible, low-latency generation by assigning different noise levels to different frames.It is designed to avoid the first-token latency of autoregressive models and the limited long-term history of chunk-based diffusion.Source: §4.2 · p. 10

4.2.1 Preliminaries

  • Vectorized time schedules assign distinct noise levels to different frames at the same global denoising time.The scalar schedules α_t and β_t are extended across the sequence.Source: §4.2.1 · p. 11

4.2.2 Motion Latent Space via Causal VAE

  • A strictly causal VAE compresses 263-dimensional motion into a compact continuous 4 × T/4 latent representation while preserving high-frequency motion details.The latent space uses temporal downsampling by four and four channels, reducing downstream diffusion computation.Source: §4.2.2 · p. 11

4.2.3 Tailored Diffusion Forcing

  • The lower-triangular schedule separates fixed past, actively denoised frames, and future noise, so finalized frames can be sent to the Renderer; bidirectional attention within the active window is essential.Restricting that attention to be causal degrades performance, with FID reported as changing from 0.057 to 3.37.Source: §4.2.3 · p. 11

4.2.4 Time-Varying Text Conditioning

  • Frame-wise text conditioning applies the prompt active at each frame, and overlapping windows integrate updated prompts into future frames for smooth transitions.The system uses a biased attention mask to pair each motion frame with its active prompt.Source: §4.2.4 · p. 12

4.2.5 Streaming Inference Algorithm

  • Inference advances the active window by a fixed step, updating only its frames and committing exiting frames to the causal VAE decoder for the Renderer.The pipeline is reported to have constant computational cost per step and strictly bounded latency of approximately n_s frames.Source: §4.2.5 · p. 12

4.3 Results

  • 0.057 FID and strict frame-rate adherence support fully online, instruction-editable body animation; figures show prompt timing changes motion and a “stand” prompt stops it.The Body Animator combines a causal VAE with a tailored diffusion-forcing scheduler.Source: §4.3 · pp. 12–13, Figure 9 · p. 13, Figure 10 · p. 13

5 DiT-based Rendering

AvatarDiT renders identity-consistent human video from FLAME and SMPL parameters with camera control, using staged training to combine facial control and multi-view coherence. Qualitative results report consistent identity, appearance, and fine-grained expression across viewpoints and sequences.

5.1 Task Description

  • The renderer maps facial and body motion parameters, a reference identity image, and camera configurations to video while targeting identity preservation, parameter-faithful motion, temporal stability, and multi-view consistency.Its inputs include FLAME facial parameters and SMPL body parameters.Source: §5.1 · pp. 13–14

5.1.1 Challenges

  • Key challenges include disentangling identity from motion, bridging compact parameter spaces to image features, preserving cross-view coherence, and learning across datasets with mismatched supervision.High-quality FLAME labels and multi-view imagery are seldom available together.Source: §5.1.1 · p. 14

5.2 Approach: AvatarDiT

  • AvatarDiT combines a video Diffusion Transformer, multimodal conditioning encoders, and control modules, trained in face-control, multi-view-control, and joint fine-tuning stages.The stages respectively learn parameter-driven facial control, cross-view coherence, and unified generation.Source: §5.2 · pp. 14–15

5.2.1 Parameter-based face control

  • FLAME parameters provide more adjustable facial-motion control than RGB-derived motion embeddings, and joint optimization lets the framework use meshes, parameters, or both.The paper reports comparable face control to Wan Animate’s RGB-based approach and generalization to out-of-distribution inputs.Source: §5.2.1 · p. 15

5.2.2 SMPL-driven Multi-view control

  • SMPL renderings paired with camera poses provide 3D motion and view controls, while cross-view references and camera-conditioned DiT modulation support consistency across views.Training samples one to five reference frames from different views.Source: §5.2.2 · p. 16

5.2.3 Dataset for Joint Fine-tuning

  • The joint stage uses an identity-disjoint set of short sequences with synchronized multi-view images and frame-aligned FLAME labels to reconcile facial control with view geometry.The face-control and multi-view stages use Seamless Interaction and MVHumanNet, respectively.Source: §5.2.3 · p. 17, §5.2.2 · p. 17

5.3 Results

  • The renderer maintains coherent facial identity, body shape, and appearance across viewpoints, while parameter-based facial control supports fine-grained expressions and temporal stability.The text contrasts this with prior methods’ reported identity drift or view-dependent artifacts as camera parameters change.Source: §5.3 · pp. 17–18, Figure 14 · p. 18

6 Thinker

Mio’s Thinker maps multimodal inputs and narrative memory into unified instructions for its embodiment modules. It combines story-time-aware retrieval with multimodal reward decomposition and competitive self-training to support narrative consistency and persona alignment.

6.1 Task Description

  • The Thinker maps multimodal inputs and internal short- and long-term state into a unified plan of dialogue, emotion, and gestures for the embodiment modules.Its stated objectives are personality coherence, narrative causality, and user engagement.Source: §6.1 · p. 18

6.2 Approach: NPC-Centric Embodied LLM

  • Hierarchical memory pairs a conversational context buffer with a story-time-tagged Diegetic Knowledge Graph, whose retrieval gate excludes events later than Mio’s current narrative time.The system retrieves graph nodes for entities and edges for thematic relationships.Source: §6.2 · pp. 19–20

6.2.1 Multimodal reward model

  • A frozen LLM converts multimodal user reactions into turn-level rewards that decompose a sparse global satisfaction signal without manual turn-by-turn annotation.Reaction descriptions are derived from visual, prosodic, and textual features, and the decomposed rewards serve as learning feedback.Source: §6.2.1 · p. 20

6.2.2 Data-free self training

  • Data-free self-training uses GRPO in a competitive loop: a scenario-generative policy probes weaknesses while an actor policy produces dialogue, emotional state, and gestures, trained with synthetic preference pairs.The reward combines multimodal feedback with a persona-based reward derived from positive and negative response pairs.Source: §6.2.2 · pp. 20–21

7 Benchmark

The benchmark evaluates speech reconstruction and zero-shot TTS with objective and user-rated measures, alongside metrics for the Thinker, facial animation, body motion, and rendering. Kodama-Tokenizer leads on reported reconstruction quality and intelligibility, while Kodama-TTS shows especially strong Japanese generation.

7.1 Evaluation Metrics

  • Evaluation covers speech quality, intelligibility and speaker identity; literary-simulation behavior and engagement; facial-motion dynamics; HumanML3D/BABEL motion; and renderer identity, view consistency and video quality.The Thinker uses CharacterBox, adversarial frame-breaking tests and user studies; Talker uses objective metrics and user-rated MOS.Source: §7.1 · pp. 21–22

7.2.1 Speech Reconstruction Performance

  • Speaker similarity is a weaker reconstruction dimension: SIM is 0.81 versus XY-Tokenizer’s 0.83 on LibriTTS and trails named baselines on English and Chinese Seed-TTS-Eval.The authors characterize this as a trade-off favoring reconstruction clarity and naturalness in noisier settings.Source: §7.2.1 · p. 22
  • PESQ-NB reaches 3.37 on JSUT versus XY-Tokenizer’s 2.28, while STOI exceeds 0.91 across test sets.The reported PESQ-NB scores are 3.26 on LibriTTS and 3.07 on Seed-TTS-Eval-ZH.Source: §7.2.1 · p. 22, Table 1 · p. 23

7.2.2 Zero-Shot TTS Performance

  • 32.82% CER on Japanese zero-shot TTS is below MOSS-TTSD’s 317.53% and Higgs’s 92.44%; Kodama-TTS also leads Japanese naturalness with an N-MOS of 4.2.The study reports Higgs at 1.55 in Naturalness; Kodama-TTS’s Japanese Similarity score is 3.95.Source: Table 2 · p. 23, §7.2.2 · pp. 23–24

7.3 Facial Animator Evaluation

The facial animator is evaluated for speaking accuracy and listening naturalness, then assessed through a pairwise user-preference study. The reported results show improvements in both motion measures and participant preference.

Quantitative Results

  • The animator improves speaking lip-sync and speech-style alignment metrics, and listening-motion distribution measures, supporting unified speaking and listening generation.The authors report more diverse listening expressions and head movements rather than a neutral or static pose.Source: §7.3 · p. 24

User Study

  • The pairwise protocol showed randomized side-by-side videos, with reaction naturalness omitted for baselines that do not generate listening behaviors.Source: §7.3 · p. 24
  • More than 90% of 25 participants preferred the animator’s listening reactions over DualTalk; the method was preferred over all baselines across assessed aspects.The pairwise study assessed lip synchronization, facial-expression naturalness, listening-reaction naturalness and head-pose naturalness.Source: §7.3 · p. 24

7.4 Body Animator Evaluation

FloodDiffusion is evaluated for motion quality and streaming responsiveness, with ablations and a user study examining its design and perceived smoothness. The reported benchmark results place its HumanML3D quality near the best offline method while outperforming streaming baselines on BABEL metrics.

Baselines

  • FloodDiffusion combines diffusion refinement with causal low-latency processing, addressing PRIMAL’s chunk-wait latency and MotionStreamer’s stated long-term consistency challenge.Source: §7.4 · p. 25

Quantitative Results

  • FID is 0.057 on HumanML3D, better than MotionStreamer’s 0.092 and PRIMAL’s 0.511, and on par with offline MoMask’s 0.045.On BABEL, FloodDiffusion has the lowest Peak Jerk and Area Under Jerk among streaming baselines.Source: Table 5 · p. 25, §7.4 · p. 25

Ablation Studies

  • Removing bidirectional attention degrades FID to 3.377, while replacing the lower-triangular schedule with a random schedule yields FID 3.883.The authors relate the attention ablation to active-window consistency and the schedule ablation to learning the streaming task.Source: §7.4 · p. 25

User Study

  • In a 100-participant Bradley-Terry study, users preferred FloodDiffusion over PRIMAL and MotionStreamer for transition smoothness by +0.152 score and overall preference.Source: §7.4 · p. 25

7.5 DiT Renderer Evaluation

The renderer evaluation examines facial control, cross-view identity consistency and perceptual quality through ablations, baseline comparisons and a user study. AvatarDiT reports gains in these dimensions, with trade-offs tied to control signals and model priors.

7.5.2 Comparison with Baseline

  • AvatarDiT has the highest CLIP similarity (0.8693) and lowest LPIPS (0.2231) for multi-view identity consistency; the authors also report identity drift or view-dependent artifacts in baselines.They attribute stronger cross-view preservation to camera-modulated DiT blocks and multi-view training.Source: Table 6 · p. 26, §7.5.2 · p. 27

7.5.1 Ablation study

  • Without motion-embedding guidance, the FLAME adapter fails to respond to parameter variations; removing FLAME mesh RGB or parameters also reduces synchronization consistency and mouth realism.The authors describe joint motion-encoder and adapter optimization as transferring motion priors into parameter space.Source: §7.5.1 · p. 26
  • Varying global head pose and jaw/mouth parameters independently tests the disentangled, interpretable FLAME control space.Source: §7.5.1 · p. 26
  • Camera parameters serve as view-control signals to address posterior-view misalignment.Source: §7.5.1 · p. 26

7.5.2 Comparison with Baseline

  • Against WanAnimate, FLAME-driven control raises PSNR from 21.40 to 27.04 and SSIM from 0.8655 to 0.9202 (↑0.0547).The authors attribute the comparison to excluding identity and shape factors that otherwise entangle with expression; combining SMPL+FLAME leaves residual shape leakage from SMPL-side priors.Source: Table 7 · p. 27, §7.5.2 · p. 27
  • AvatarDiT obtains the best FID and FVD while achieving SSIM/PSNR comparable to other methods, and adds explicit facial-motion control.CHAMP has the best SSIM and PSNR but cannot control facial motion; its backbone limits overall generation fidelity.Source: Table 7 · p. 27, §7.5.2 · p. 27

7.5.3 Qualitative User Study

  • AvatarDiT scores highest among generative systems in all four user-rated criteria, with +0.6 Facial Expression Accuracy and +0.5 Multi-view Consistency over WAN-Animate.The study used 15 participants and 7-point Likert ratings; ratings approached the ground-truth reference level.Source: §7.5.3 · pp. 27–28, Figure 16 · p. 28

7.6 Thinker Evaluation

The Thinker evaluation tests persona fidelity, robustness to out-of-domain prompts, and narrative timeline coherence. Results favor the complete system, with self-training and diegetic memory contributing to different measured capabilities.

7.6 Thinker Evaluation

  • BA: 4.250 vs. 3.683 and PT: 4.267 vs. 3.150; Full Thinker exceeds GPT-4o across every CharacterBox metric.The comparison supports the reported advantage of the specialized self-training pipeline over prompt-only conditions.Source: Table 8 · p. 29, §7.6 · p. 29
  • CharacterBox evaluates open-ended character trajectories across seven metrics, with four characters repeated 15 times under five system conditions.The metrics include knowledge and behavioral accuracy, personality traits, emotional expression, immersion, adaptability, and behavioral coherence.Source: §7.6 · pp. 28–29, Table 8 · p. 29

Custom Tests for Robustness and Coherence

7.7 The Interactive Intelligence Score (IIS)

The Interactive Intelligence Score (IIS) combines five normalized performance dimensions into one system-level measure. Mio scores above the composite of previous best baselines on this measure.

7.7.1 Definition

  • IIS averages five normalized dimensions—cognitive, acoustic, facial, somatic, and visual—on a 0–100 scale.Source: §7.7 · p. 30, §7.7.1 · p. 30
  • Cognitive Resonance combines CharacterBox, timeline coherence, and refusal robustness; Facial Synchrony scores speaking lip error and listening-motion deviation.Source: §7.7.1 · pp. 30–31
  • Acoustic Fidelity averages normalized speech measures across English, Chinese, and Japanese, including intelligibility, speaker similarity, quality, and pronunciation.Source: §7.7.1 · pp. 30–31
  • Somatic Fluidity combines motion quality and transition smoothness, while Visual Integrity aggregates semantic similarity, structural similarity, and perceptual distortion.Source: §7.7.1 · p. 31

7.7.2 Comparison to Previous Best

  • 76.0 IIS, a +8.4-point improvement over the aggregated previous state-of-the-art baselines.Source: §7.7.2 · p. 31

8 Related work

The related work situates Mio at the intersection of speech, facial and body animation, rendering, and agentic reasoning. It presents these areas as complementary components of an integrated system for expressive, embodied, consistent NPCs.

8.1 Talker: Speech Modeling and Thinking–Talking Architectures

  • Speech research spans Thinker–Talker systems, multilingual interaction, high-fidelity zero-shot TTS, style transfer, and efficient audio generation.Source: §8.1 · pp. 31–32

8.2 Face Animator: Audio-Driven Talking Head Generation

  • Talking-head work progresses from audio-visual alignment to controllable pose, emotion, identity, and photorealistic free-view expression.Source: §8.2 · p. 32

8.3 Body Animator: Text- and Audio-Driven Human Motion

  • Text- and speech-driven motion methods use diffusion, language modeling, and masked motion models; HumanML3D is a standard benchmark.Source: §8.3 · p. 32

8.4 Renderer: Diffusion Transformers and Controllable Generative Models

  • Prior rendering approaches often lack multi-view or 3D consistency; AvatarDiT uses FLAME and SMPL controls with camera-aware modulation for viewpoint-consistent identity.Source: §8.4 · pp. 32–33

8.5 Thinker: Agentic Memory, Planning, and Reasoning

  • Prompting and curated fine-tuning can improve role-play but face persona drift or annotation costs; Mio’s Thinker uses data-free self-training within a five-pillar system.Source: §8.5 · p. 33

9 Conclusion

The conclusion presents Interactive Intelligence as a shift from imitative digital humans toward autonomous agents with personality-aligned expression, adaptive interaction, and self-evolution. Mio integrates reasoning and real-time embodiment, and the authors report an improved aggregate IIS score.

9 Conclusion

  • Mio combines the Thinker with Talker, Face Animator, Body Animator, and Renderer to integrate cognitive reasoning with real-time embodiment.The conclusion characterizes the result as narrative depth alongside physical fluidity.Source: §9 · p. 34
  • 76.8 IIS, a +7.8-point improvement over a composite of state-of-the-art baselines.Source: §9 · p. 34
  • The authors make the codebase, pretrained models, and evaluation benchmark publicly available to support further exploration.Source: §9 · p. 34
Loading 2512.13674v2…