PaperlayerView exact revision on arXiv

Source-linked AI summary

Towards Seamless Interaction: Causal Turn-Level Modeling of Interactive 3D Conversational Head Dynamics

Junjie Chen, Fei Wang, Zhihao Huang, Qing Zhou, Kun Li, Dan Guo, Linfeng Zhang, Xun Yang

arXiv:2512.15340v2cs.CV

TL;DR

Existing work often treats talking and listening heads separately rather than modeling intertwined dyadic dynamics. TIMAR uses turn-level causal modeling over interleaved audio-visual context to generate continuous 3D head motion. It reports lower FD/P-FD than DualTalk on test and out-of-distribution data under progressive-context streaming inference.

  • Problem

    Talking- and listening-head generation is often treated as separate processes rather than modeling the intertwined dynamics of dyadic communication.

    Source: Listening-Head Generation · p. 4

  • Method

    TIMAR models interactive 3D head generation as turn-level causal autoregressive diffusion, interleaving speakers’ audio-visual tokens and decoding continuous 3D head motion.

    Source: §1 · pp. 2–3

  • Results

    About 15–30% lower FD/P-FD on test and 5–10% lower FD/P-FD on OOD than DualTalk under progressive-context streaming inference.

    Source: TIMAR achieves consistent gains on… · p. 11

  • Takeaways & Limitations

    Explicit causal and interleaved modeling leads to more humanlike conversational dynamics, with reported improvements in realism and synchronization over DualTalk.

    Source: §5 · p. 14

    Preprocessing requires T to be divisible by the chunk duration c.

    Source: §3.2 · p. 5

Abstract

from arXiv · show

Human conversation involves continuous exchanges of speech and nonverbal cues such as head nods, gaze shifts, and facial expressions that convey attention and emotion. Modeling these bidirectional dynamics in 3D is essential for building expressive avatars and interactive robots. However, existing frameworks often treat talking and listening as independent processes or rely on non-causal full-sequence modeling, hindering temporal coherence across turns. We present TIMAR (Turn-level Interleaved Masked AutoRegression), a causal framework for 3D conversational head generation that models dialogue as interleaved audio-visual contexts. It fuses multimodal information within each turn and applies turn-level causal attention to accumulate conversational history, while a lightweight diffusion head predicts continuous 3D head dynamics that captures both coordination and expressive variability. Experiments on the DualTalk benchmark show that TIMAR reduces Fr\'echet Distance and MSE by 15-30% on the test set, and achieves similar gains on out-of-distribution data. The source code has been released at https://github.com/CoderChen01/towards-seamless-interaction.

1 Introduction

Prior head-generation systems separate speaking and listening, while full-sequence models are less suited to turn-by-turn streaming. TIMAR unifies both through interleaved audio-visual context, causal turn-level modeling, and diffusion-based 3D motion prediction.

1 Introduction

  • Most talking- and listening-head frameworks model the two directions independently, while DualTalk’s full-sequence bidirectional attention is less suited to causal or streaming generation.Talking-head systems use the speaker’s audio; listening-head systems react to the interlocutor, but natural conversation intertwines both.Source: §1 · pp. 1–2
  • TIMAR models conversation as causally linked turns, conditioning each turn’s facial motion on both speakers’ preceding speech and visual cues.This turn-level formulation is intended to reflect continuous multimodal feedback and conversational flow.Source: §1 · p. 2
  • TIMAR interleaves both speakers’ speech and 3D head tokens, using bidirectional fusion within turns and causal attention across turns.The fused context accumulates conversational history for coherent modeling.Source: §1 · pp. 2–3
  • A compact diffusion-based decoder models 3D head motion as a continuous probabilistic process, capturing natural variability while maintaining coherence across turns.The introduction presents this as the framework’s lightweight generative decoding component.Source: §1 · p. 3
  • Compared with DualTalk’s full-sequence modeling, TIMAR’s causal formulation enables natural, streaming-capable generation that mirrors conversational timing and feedback, with consistent evaluation improvements in realism, synchronization, and responsiveness across in-distribution and unseen scenarios.Source: §1 · p. 3

2 Related Work

Prior work advances speech-driven talking heads and reactive listening heads, including 3D representations, but generally treats the two as separate processes. Interactive 3D methods unify speaking and listening, yet full-sequence processing remains less suited to causal streaming.

2.1 3D Talking- and Listening-Head Generation

  • Talking- and listening-head systems have advanced visual realism, synchronization, and socially meaningful feedback, but generally model speaking and responding as separate processes.Talking-head methods range from RGB video generation to 3DMM and direct 3D motion; listening-head work models nonverbal responses such as nodding, gaze shifts, and micro-expressions.Source: Listening-Head Generation · pp. 3–4, Talking-Head Generation · p. 3

2.2 Modeling Interactive 3D Conversational Heads

  • Interactive 3D modeling addresses bidirectional coordination, but DualTalk’s full-sequence bidirectional processing is designed for offline synthesis and is less suited to causal streaming.Some 2D methods model mutual coordination but lack explicit 3D geometry control.Source: §2.2 · p. 4

3 The TIMAR Framework

TIMAR generates 3D conversational head motion by interleaving both speakers’ speech and head-motion tokens, fusing context within and across turns, and predicting masked agent motion with diffusion. Its training and sampling procedures use this turn-level causal context to support coherent generation across dialogue turns.

3 The TIMAR Framework

  • TIMAR interleaves both speakers’ speech and head-motion tokens, fuses turn-level context, and uses a diffusion head to generate masked agent head motion.The workflow applies bidirectional fusion within turns and causal fusion across turns; at sampling, it caches history tokens and denoises each new turn.Source: §3 · p. 4, Figure 2 · p. 6

3.1 Interleaved Audio-Visual Context

  • Speech and 3D head motion are aligned in a shared token space, divided into fixed-length chunks, and interleaved by turn to form the multimodal context.The speech tokenizer resamples speech features to the motion frame rate, while each chunk is encoded independently to avoid exposing future information.Source: §3.1 · pp. 4–5, Interleaving · p. 5

3.2 Turn-Level Causal Multimodal Fusion

3.3 Lightweight Diffusion Head

  • The Lightweight Diffusion Head predicts masked agent-head parameters through conditional denoising in continuous space, modeling the stochasticity and multimodality of natural 3D facial motion.During sampling, it starts from noise and iteratively denoises the parameters conditioned on fused context.Source: §3.3 · p. 7

3.4 TIMAR Training Recipe

3.5 TIMAR Sampling

4 Experiments

Experiments compare TIMAR with DualTalk and other baselines under streaming and nonstreaming protocols, using test and out-of-distribution data. TIMAR shows benchmark, user-preference, and robustness gains, while ablations assess its diffusion head and causal design.

4 Experiments

TIMAR achieves consistent gains on both streaming and full-sequence

5 Conclusion

TIMAR uses turn-level autoregressive diffusion with interleaved multimodal tokens and causal attention to model cross-speaker dependencies while preserving temporal coherence. Experiments report consistent realism and synchronization improvements over DualTalk, and ablations support the diffusion head and causal design.

5 Conclusion

  • TIMAR combines turn-level autoregressive diffusion with interleaved multimodal tokens and causal attention to model cross-speaker dependencies and preserve temporal coherence.The paper reports consistent improvements over DualTalk in realism and synchronization, with ablations validating the diffusion head and causal design.Source: §5 · p. 14

Supplementary Material for “TIMAR: Causal

The task is to generate an agent’s 3D head motion from both participants’ speech and visual behavior. TIMAR frames this as causal, turn-by-turn prediction from accumulated dialogue context.

A Problem Statement

  • The target is the agent’s 3D head motion, conditioned on both participants’ speech and motion in an interpretable 3DMM representation.The representation includes expression, jaw, and pose parameters and is described as temporally stable and directly controllable.Source: A Problem Statement · p. 20
  • Unlike full-sequence bidirectional modeling such as DualTalk, turn-level causal generation predicts current-turn motion without relying on future information.The passage describes full-sequence formulations as offline and unsuitable for streaming or autoregressive response.Source: Turn-Level Causal Formulation · pp. 20–21
  • Each turn uses the user’s speech and motion and the agent’s speech to predict the agent’s head motion at that turn.The dialogue is divided into N fixed-length turns, with context accumulated through the current turn.Source: Turn-Level Causal Formulation · p. 21
  • The causal factorization prevents future leakage and supports turn-by-turn streaming while preserving interleaved interaction so rhythm, affect, and timing can propagate across turns.These principles form the foundation of TIMAR.Source: Turn-Level Causal Formulation · p. 21

B.1 Network Details

TIMAR encodes speech and head motion into a shared token space, fuses interleaved turn-level context with causal attention, and uses a diffusion head to predict head motion.

B.1 Network Details

  • A speech tokenizer converts audio chunks into shared-space tokens using wav2vec 2.0 features, interpolation to the motion frame rate, and learned projection.Figure 2 depicts the speech tokenizer architecture.Source: Speech Tokenizer · p. 21, Figure 2 · p. 22
  • The head-motion encoder maps each FLAME-based motion segment into the shared token space using a two-layer MLP with ReLU activations.Figure 1 depicts the 3D head motion encoder architecture.Source: 3D Head Motion Encoder · p. 21, Figure 1 · p. 22
  • Turn-level fusion interleaves four modality streams and uses separator tokens to mark modality boundaries and turn transitions.Turn-Level Causal Attention permits within-turn bidirectional interaction while restricting cross-turn attention to past tokens.Source: 3D Head Motion Encoder · p. 22, Figure 3 · p. 23
  • The diffusion head conditions denoising on noisy head parameters and contextual features, using residual modulation blocks to produce 3D head parameters.Context-derived shift and scale modulate normalized activations; Figure 4 shows the head architecture.Source: 3D Head Motion Encoder · pp. 22–23, Figure 4 · p. 23

B.2 Implementation Details

The implementation uses a 56-dimensional FLAME representation, separates diffusion loss across head-parameter subsets, and trains the model on aligned one-second turns within eight-second clips.

B.2 Implementation Details

  • The 56-dimensional FLAME representation comprises 50 expression coefficients, three jaw parameters, and three head-pose parameters.Diffusion loss is computed separately for expression, jaw, and pose, then aggregated.Source: Loss Formulation · pp. 23–24
  • Table 1 summarizes DualTalk dataset scale, official train, test, and OOD splits, and conversation-round distribution.Its caption reports 50 hours and over 1000 unique identities.Source: Table 1 · p. 24
  • The default model uses 16 Transformer layers and attention heads, with token and encoder dimensions of 1024 and three diffusion modulation blocks.The diffusion latent dimension is also 1024; experiments use PyTorch and load pretrained components through Transformers.Source: Software Framework · p. 23, Default Configuration · p. 24
  • Training uses eight-second clips with one-second aligned turns, 25-fps head motion, and 16-kHz speech, masking 70% of agent head tokens.The configuration also specifies AdamW, batch size 32, 400 epochs, a 1 × 10−4 learning rate, and classifier-free guidance dropout of 0.1.Source: Training Configuration · p. 24

C.1 Datasets

DualTalk provides processed, synchronized audio-visual recordings of multi-round face-to-face conversations, with official training, test, and out-of-distribution splits.

C.1 Datasets

  • The corpus draws on open-domain interview and dialogue videos selected for frontal visibility of both speakers and high-quality audio.The official preprocessing pipeline performs speaker separation, tracking, and 3D reconstruction to produce aligned speech and head parameters.Source: §C.1 · pp. 24–25
  • The processed corpus contains about 50 hours, over 1000 identities, and 5763 samples, split into 4853 training, 533 test, and 377 OOD samples.The OOD set contains unseen speakers and conversation scenarios.Source: §C.1 · p. 25
  • Most dialogues contain one to three alternating speaker turns, reflecting short-turn interaction patterns.Table 1 summarizes dataset scale and conversation-round distribution.Source: §C.1 · p. 25

C.2 Metrics

The section defines metrics for distributional similarity, paired interaction, frame-level accuracy, motion diversity, and temporal responsiveness. It also specifies how motion features and statistics are computed for evaluation.

C.2 Metrics

  • Fréchet Distance compares generated and real motion distributions in deep feature space, while Paired Fréchet Distance evaluates inter-speaker coherence and synchronization using concatenated agent and user embeddings.Lower FD indicates closer distributional alignment.Source: Fréchet Distance (FD) · p. 25, Paired Fréchet Distance (P-FD) · p. 25
  • MSE measures frame-level reconstruction accuracy between predicted and ground-truth 3D head parameters, with lower values indicating higher fidelity to the reference.Source: Mean Squared Error (MSE) · p. 25
  • SID measures generated-motion diversity using k-means clustering with k = 40 and cluster-assignment entropy; higher SID indicates greater variety and less repetition.Source: SI for Diversity (SID) · p. 25
  • Residual Pearson Correlation Coefficient compares generated and real user–agent motion-correlation patterns, with lower values indicating more accurate interactive timing and responsiveness.It measures the L1 distance between Pearson correlation patterns for each speaker pair.Source: Residual Pearson Correlation… · p. 26
  • Metrics are computed separately for expression, jaw, and pose features; Fréchet statistics use entire test sequences, and SID follows DualTalk’s clustering configuration.These metrics provide complementary views of realism, synchrony, and diversity.Source: Implementation Notes · p. 26

D User Study Details

The user study compares TIMAR and DualTalk through blinded ratings of synchronized dialogue clips, while a separate scalability study examines diffusion-head capacity. On the test dataset, performance improves consistently as capacity increases.

D User Study Details

  • Ten participants rated TIMAR and DualTalk on 50 clips—25 test and 25 out-of-distribution—using four criteria after viewing synchronized agent, reference, and user videos.Method identities were blinded, and the ground-truth agent served only as a visual reference.Source: D User Study Details · pp. 26–27
  • The study reports TIMAR’s preference rate over DualTalk from averaged four-criterion scores, counting ties as half-votes and using 95% bootstrap intervals over 500 pairs.The 500 pairs comprise each participant’s evaluation of all 50 clips.Source: D User Study Details · p. 27, Table 2 · p. 27

E Diffusion Head Scalability Study

  • Increasing diffusion-head hidden dimension and residual-block count consistently improves test-set performance across the reported metrics.The study varies hidden dimension dm and residual-block count K; the table marks FD, P-FD, MSE, and rPCC as lower-is-better and SID as higher-is-better.Source: E Diffusion Head Scalability Study · p. 27, Table 3 · p. 27
Loading 2512.15340v2…