PaperlayerView exact revision on arXiv

Source-linked AI summary

EvolvingAvatar: Interactive 3D Head Generation That Adapts as Conversations Unfold

Junjie Chen, Fei Wang, Kun Li, Yiqi Nie, Xun Yang, Yanbin Hao, Linfeng Zhang, Meng Wang

arXiv:2609.35616v1cs.CV

TL;DR

Interactive 3D head generation must coordinate speaking and listening motion as conversations evolve, while existing generators keep their parameters fixed and leave conversational context unused as a learning signal. EvolvingAvatar adapts from user face video and dyadic audio through test-time training without target motion labels, and InterHead-Bench unifies single- and dual-view conversation videos. Experiments report improved motion statistics and human ratings under distribution shifts, while recorded conversations do not capture users’ responses to generated motion.

  • Problem

    Existing generators use incoming observations as context but keep parameters fixed, leaving conversational patterns unused as a signal for learning coordinated interactive motion.

    Source: §1 · pp. 1–2

  • Method

    EvolvingAvatar uses Dyadic Context Prediction to adapt a causal generator from user video and dyadic audio without target motion labels, retaining persistent conversational updates and transient jaw adaptation.

    Source: §1 · p. 2

  • Results

    Experiments show improved motion statistics and human ratings under distribution shifts, while matched ablations support continued learning within conversations.

    Source: §7 · p. 10

  • Takeaways & Limitations

    The work extends interactive 3D head generation from conditioning on conversational context to learning from it throughout each conversation.

    Source: §7 · p. 10

    Recorded conversations do not capture how users respond to generated motion.

    Source: §7 · p. 10

Abstract

from arXiv · show

Interactive 3D head generation requires coordinated speaking and listening motion that responds to an evolving conversation. Existing generators use incoming observations as context but keep their parameters fixed, leaving conversational patterns unused as a learning signal. We introduce EvolvingAvatar, a causal generator that uses test-time training to adapt to user face video and dyadic audio during interaction. Its dyadic context prediction objective provides a self-supervised learning signal from audiovisual context without target motion labels at test time. Persistent fast weights accumulate these updates within each conversation to guide motion generation, while transient jaw adaptation responds to current audiovisual context. Predicted speech activity controls how persistent adaptation guides motion. We also introduce InterHead-Bench, a unified 455.95-hour benchmark built from single-view and dual-view conversation videos. Experiments show improved conversational motion statistics over strong baselines. On the hardest out-of-distribution split, generation improves as conversations unfold, reducing mismatch with recorded user-avatar expression statistics by up to 11.1% from the first interval.

1 INTRODUCTION

Interactive head generation must coordinate speaking, listening, and transitions in response to an evolving conversation, but existing generators leave their parameters fixed during interaction. EvolvingAvatar adapts from user video and dyadic audio without target motion labels, and the paper introduces a unified benchmark for evaluating this setting.

1 INTRODUCTION

  • Nods signal understanding, partners coordinate response timing, and people may mirror a partner’s nonverbal behavior, motivating self-supervision from repeated conversational observations.User behavior and both participants’ speech provide a basis for learning that may help guide motion.Source: §1 · p. 1
  • Existing generators use avatar audio, dyadic audio, or dyadic audio with estimated user motion, but keep parameters fixed during inference.A dialogue model can supply avatar speech from user audiovisual input.Source: §1 · p. 1, Figure 1 · p. 2
  • The challenge is to learn from conversational context without target motion labels, which existing generators do not use to update their parameters during interaction.Interactive motion must coordinate speech articulation, listening responses, and transitions between them.Source: §1 · pp. 1–2
  • EvolvingAvatar uses Dyadic Context Prediction for test-time learning from user video and dyadic audio, with persistent conversational updates and transient jaw adaptation.Predicted activity controls how persistent adaptation guides speaking and listening motion.Source: §1 · p. 2
  • Experiments improve conversational motion statistics over baselines, and the 455.95-hour InterHead-Bench evaluates speaking and listening across single-view and dual-view conversations.The benchmark combines aligned multimodal annotations with parameter-space and mesh-space metrics and human ratings.Source: §1 · p. 2

2 RELATED WORK

Prior interactive head generators expand from avatar audio to user and dyadic context, but retain fixed parameters at deployment. Test-time training offers a way to adapt the context-to-motion mapping as a conversation unfolds.

2 RELATED WORK

  • Prior work spans avatar-audio speech animation, user-context listening responses, and dyadic generation, yet existing generators retain fixed parameters at deployment.Dyadic approaches include image-space coordination, audiovisual head generation, dual-audio 3D synthesis, and methods incorporating user motion.Source: §2 · p. 2
  • Conversation-specific behavior and turn-taking dynamics motivate adapting the mapping from context to motion as each interaction reveals distinct patterns.These patterns include user behavior, avatar speaking style, and turn-taking dynamics.Source: §2 · p. 3
  • Test-time training updates fast parameters from unlabeled deployment observations, and EvolvingAvatar uses the unfolding dialogue as a self-supervised adaptation signal.This brings conversation-specific adaptation to causal head generation without target motion at deployment.Source: §2 · p. 3

3 PRELIMINARIES

The paper represents head motion as aligned FLAME-parameter trajectories and formulates generation from conversational observations arriving over time. Its setting uses causal user face frames rather than a precomputed user-motion trajectory, while test-time training updates fast weights from unlabeled observations.

3 PRELIMINARIES

  • Each frame’s 106-D FLAME motion vector contains 100 expression coefficients, three neck-pose values, and three jaw-pose values.The representation excludes shape, global root pose, and eye pose; neck and jaw rotations use axis-angle coordinates.Source: §3 · p. 3
  • The task generates avatar head motion from conversational context, including user audio, user visual evidence, and avatar audio available through time.The setting observes causally arrived user face frames rather than a precomputed user FLAME trajectory.Source: §3 · p. 3
  • Test-time training updates fast weights from an arrived observation using a self-supervised inner objective, without requiring a task label.The observation then informs the fast state used for generation.Source: §3 · p. 3

4 EVOLVINGAVATAR

EVOLVINGAVATAR generates motion causally from user video and dyadic audio, adapting during each conversation without motion labels. Its structured codec and adaptive generator combine persistent conversational learning with transient articulation response.

4.1 REGION-STRUCTURED CAUSAL FLAME CODEC

  • The codec represents expression, neck pose, jaw pose, and shared coordination in structured latents, then decodes them through region-specific heads and a shared causal backbone.Training uses a region-balanced reconstruction loss on motion and temporal differences, with an information-rate constraint; the frozen codec’s posterior mean becomes the generator target.Source: §4.1 · p. 4

4.2 ADAPTIVE GENERATOR

  • At deployment, user face frames and both participants’ audio update fast weights before generation; dyadic context prediction learns context regularities without motion labels.Persistent fast weights retain context across intervals, and generation uses only current and earlier observations.Source: §4 · p. 3, §4.2 · pp. 4–5
  • Transient TTT adapts the jaw route to current speech, while predicted speaking/listening activity gates the persistent adaptation added to motion context.Activity changes only the added adaptation term; it does not enter fast-weight updates or the context passed to later blocks.Source: §4.2 · p. 5
  • Training processes context segment A before supervised segment B, applies DCP updates throughout, and computes motion losses only on B.Clips without A train generation from learned initial fast weights; motion is generated by flow-matching heads and decoded with codec-defined regional routing.Source: §4.2 · pp. 5–6
  • Persistent main weights guide current and later motion but reset between conversations, whereas jaw updates last one interval.Both branches update per complete interval and reuse adapted conditions throughout generation.Source: §4.2 · p. 6

5 INTERHEAD-BENCH

INTERHEAD-BENCH unifies single-view and dual-view conversation recordings in a shared construction and evaluation framework. Its splits span 455.95 hours and support motion, conversational-behavior, and perceptual comparisons.

Dataset construction framework

  • DIALOG3D-FACTORY standardizes dual-view streams with separate participant audio and single-view recordings with mixed audio into aligned video, speech, motion, transcript, and interaction annotations.Its workflow tracks and segments faces, recovers participant audio, annotates motion and interaction, then validates and assembles accepted segments.Source: §5 · p. 6

Dataset composition and statistics

  • The benchmark contains 455.95 hours from 4,365 source participant IDs, with OOD-Hard testing transfer to a different recording domain.Train, Dev, ID, and OOD use dual-view Seamless Interaction recordings; OOD-Hard uses single-view RealTalk recordings.Source: §5 · p. 6, Figure 4 · p. 7

Baselines

  • Four baselines are adapted to predict the same 106-D Avatar FLAME motion, with generation-time conditioning inputs retained for interpreting comparisons.DiffPoseTalk and ARTalk use Avatar audio; UniLS adds User audio, and DualTalk also uses precomputed user FLAME motion.Source: §5 · pp. 6–7, Table 1 · p. 7

Quantitative and perceptual evaluation

  • Evaluation combines parameter-space and mesh-space motion metrics with a blind forced-choice study of synchronization, naturalness, turn-taking, responsiveness, and conversational realism.Metrics include framewise error, motion and joint-statistic comparisons, diversity, amplitude errors, and lip/full-mesh errors.Source: §5 · p. 7

6 EXPERIMENTAL RESULTS

EvolvingAvatar improves motion-statistics agreement across speaking and listening states, with particular gains on challenging distribution shifts, while human judgments favor its conversational behavior. Continued test-time updates and retained conversational context improve expression statistics, and component ablations show complementary, region-dependent effects.

6 EXPERIMENTAL RESULTS

  • On OOD-Hard, speaking/listening FDD is 19.09/18.13 for EvolvingAvatar versus 20.91/19.84 for ARTalk; expression and neck FD are lowest in both states on every split.On OOD-Hard, jaw FD also leads; DualTalk has lower MSE and LVE/MHD but lower motion coverage, and UniLS retains lower FDD on ID/OOD.Source: §6 · p. 7, Table 2 · p. 8, Table 3 · p. 8
  • On OOD-Hard, expression P-FD falls 7.1% from the first to last interval, while all four baselines worsen on OOD.On ID, full TTT lowers expression P-FD by 20.4% against No write, 4.2% against Freeze-1, and 8.6% against No carry; neck P-FD instead increases by 8.9% against No write.Source: §6 · p. 8, Figure 5 · p. 9, Table 4 · p. 9
  • On OOD-Hard, overall-realism preference reaches 90% against DiffPoseTalk and ARTalk and 86% against DualTalk.Preference over UniLS rises from 58% on ID to 66% on OOD and 72% on OOD-Hard, but all three uncertainty intervals include equal preference.Source: §6 · pp. 8–9, Figure 6 · p. 9
  • Regional coding lowers ID expression and neck P-FD by 1.9% and 0.8% versus unified coding, while activity conditioning lowers all six FD/P-FD scores.Jaw FM is associated with 5.1% lower jaw P-FD; RGB improves expression and neck statistics over audio alone on OOD-Hard but worsens expression statistics on ID/OOD.Source: §6 · p. 9, Table 5 · p. 9

7 CONCLUSION

EvolvingAvatar learns from audiovisual context throughout each conversation, and the benchmark results support improved motion statistics and human ratings under distribution shifts. The paper also clarifies what its complementary motion metrics capture and what they cannot establish.

7 CONCLUSION

  • Dyadic Context Prediction enables test-time learning without target motion labels; persistent adaptation retains conversation patterns, while transient jaw adaptation responds to current articulation.InterHead-Bench unifies single-view and dual-view conversation videos for evaluating speaking and listening across distributions.Source: §7 · pp. 9–10
  • Recorded conversations do not capture how users respond to generated motion; future work could study continual learning in live interactions.The paper proposes clinician-guided studies to assess perceived support alongside motion quality over longer conversations.Source: §7 · p. 10

A METRIC DEFINITIONS AND PROPERTIES

  • For evaluation, predicted, reference, and aligned user motions use T × 106 parameters; user FLAME motion is context only for P-FD and rPCC, and scores average equally across directed records.Speaking includes avatar speech and overlap; listening includes avatar listening and silence, while all-frame scores use every valid frame.Source: A METRIC DEFINITIONS AND PROPERTIES · p. 16

A.1 PARAMETER-SPACE METRICS

  • Parameter-space metrics cover aligned coefficient error, marginal and joint motion statistics, user–avatar correlation, occupancy, and neck and jaw dispersion, with distinct limits on temporal and causal interpretation.FD discards frame order and higher-order structure; P-FD compares same-frame user–avatar cross-covariance but cannot establish response timing or causal influence.Source: §A.1 · p. 16

A.2 MESH-SPACE METRICS

  • Mesh-space metrics assess expression and jaw geometry, while parameter-space metrics evaluate neck motion.LVE emphasizes the largest lip-vertex displacement and is outlier-sensitive; MHD averages error across matching vertices and frames, potentially diluting local errors.Source: §A.2 · p. 17

A.3 INTERPRETING THE METRICS TOGETHER

  • Paired error measures agreement with one recorded response, although conversations can admit several appropriate responses and a sampled response can have higher expected squared reference error than the conditional mean.The evaluation combines paired fidelity, motion statistics, amplitude, and perceptual evaluation; neither the mean nor sampling guarantees a plausible response.Source: §A.3 · p. 17

B ADDITIONAL VISUAL COMPARISONS

Across the visual comparisons, EvolvingAvatar more closely follows reference expressions, neck poses, and mouth openings during speaking and listening. The examples show that the appropriate facial response varies with the exchange, not just the activity label.

B ADDITIONAL VISUAL COMPARISONS

  • Across the displayed cases, EvolvingAvatar better follows reference expressions, neck poses, and mouth openings; several baselines show excessive neck deviations, exaggerated openings, or weaker expressions.Rows show the user video, Avatar GT, EvolvingAvatar, and four baselines; columns align time points across methods.Source: B ADDITIONAL VISUAL COMPARISONS · p. 18
  • EvolvingAvatar follows changing articulation from listening openings to speaking openings, and from speaking articulation to a listening smile, while retaining smiles with small openings.These patterns are illustrated in Figures 9, 14, and 17.Source: B ADDITIONAL VISUAL COMPARISONS · p. 18, Figure 9 · p. 19, Figure 14 · p. 22
  • In examples with smiles during listening, EvolvingAvatar preserves the expression, sometimes with parted lips, whereas some baselines understate the smile or mouth opening.Other examples show smaller listening openings, underscoring that the desired response depends on the ongoing exchange.Source: B ADDITIONAL VISUAL COMPARISONS · p. 18, Figure 8 · p. 19, Figure 13 · p. 21
  • EvolvingAvatar better retains the relation between neck orientation and facial expression across speaking and listening, while several baselines introduce stronger downward or sideways rotations.The references include both a slight downward listening pose with a sideways speaking lean and nearly upright poses with changing mouth shapes.Source: B ADDITIONAL VISUAL COMPARISONS · p. 18, Figure 10 · p. 20, Figure 12 · p. 21

C ALL-FRAME QUANTITATIVE RESULTS

All-frame parameter-space results favor EvolvingAvatar over the baselines in expression, neck, and jaw FD across splits. The mesh-space advantage is strongest on OOD-Hard, while the metrics also show differences between motion statistics and agreement with one recorded response.

C ALL-FRAME QUANTITATIVE RESULTS

  • EvolvingAvatar outperforms all four baselines in expression, neck, and jaw FD across splits on all valid frames, including overlap and silence.All-frame P-FD also favors EvolvingAvatar except for OOD-Hard expression, where DualTalk is slightly lower (13.49 versus 13.52).Source: C ALL-FRAME QUANTITATIVE RESULTS · p. 26
  • EvolvingAvatar outperforms all baselines in all-frame mesh FDD on OOD-Hard; UniLS leads FDD on ID/OOD, while DualTalk has lower LVE and MHD across splits.Mesh metrics assess expression and jaw geometry; parameter metrics also assess neck motion.Source: C ALL-FRAME QUANTITATIVE RESULTS · p. 26
  • Motion statistics and agreement with one recorded response assess different aspects of quality, with human judgments adding evidence about perceived conversational quality.The paper describes parameter-space and mesh-space evidence as complementary.Source: C ALL-FRAME QUANTITATIVE RESULTS · p. 26

D CONTEXT SIGNAL COMPARISON

The context comparison tests audio, RGB, and pre-extracted FLAME inputs while retaining the adaptive generator design. Results show complementary strengths across representations, alongside implementation details for the causal codec, audiovisual backbone, and adaptation modules.

D CONTEXT SIGNAL COMPARISON

  • Across context interfaces, the audio model improves FD and P-FD over UniLS, the FLAME model improves FD over DualTalk, and all settings lower FDD relative to DualTalk.DualTalk retains lower expression/neck MSE, LVE, and MHD; same-checkpoint comparisons separately find continued TTT updates and retained context improve ID expression statistics.Source: D CONTEXT SIGNAL COMPARISON · p. 27
  • User FLAME lowers jaw FD and P-FD against RGB and audio in every split and state, while RGB gains vary by split and motion component.FLAME has neck FD/P-FD advantages over RGB on ID and OOD-Hard, but RGB is stronger on OOD; audio beats RGB on expression FD and P-FD on ID and OOD, while RGB gains several neck, LVE, and MHD comparisons on OOD-Hard.Source: D CONTEXT SIGNAL COMPARISON · pp. 27–28

E.1 CAUSAL MOTION CODEC

  • The codec groups motion into 200 ms intervals of five frames, standardizes FLAME parameters, and mixes each interval into one token before causal convolutions.Its regional decoder paths share convolution weights but retain separate bounded histories; phase embeddings and linear heads expand each token back to five frames.Source: E METHOD IMPLEMENTATION AND STREAMING… · p. 29, §E.1 · p. 29
  • Codec reconstruction uses normalized motion residuals and their first and second differences, while a KL rate constraint targets 69.3 nats per valid interval.The codec is frozen after training and supplies posterior-mean targets for generator training and latent decoding at inference.Source: §E.1 · p. 29, E METHOD IMPLEMENTATION AND STREAMING… · p. 29

E.2 AUDIOVISUAL FEATURES AND MOTION BACKBONE

  • The audiovisual backbone pools five frames into four visual and two audio tokens, with separate audio-role caches and phase-aware queries.Visual tokens alternate audio cross-attention and visual self-attention, and only the motion token enters the velocity decoder.Source: §E.2 · pp. 29–30

E.3 FAST WEIGHTS AND TRANSIENT JAW ADAPTATION

  • Persistent fast weights update from dyadic context prediction and carry across intervals, while transient jaw updates use current audio-visual context and are discarded.Persistent adaptation affects speaking and listening through predicted activity; jaw adaptation adds its residual without an activity gate.Source: §E.3 · p. 30

E.4 GENERATOR TRAINING AND OPTIMIZATION

Generator training builds context from a causal prefix and supervises a suffix, with motion, jaw, and activity losses. Streaming processes aligned inputs in intervals, updates persistent weights once per complete interval, and resets conversation-local state between conversations.

E.4 GENERATOR TRAINING AND OPTIMIZATION

  • Training clips use a causal prefix for context and a supervised suffix for motion losses, reserving at least two suffix intervals when available.The prefix is empty with probability 0.2; otherwise its length is sampled from 20 intervals to the largest feasible prefix, with the lower bound reduced for short clips.Source: §E.4 · p. 30
  • Training combines flow losses with endpoint alignment, decoded-jaw supervision, and activity cross-entropy, each auxiliary loss weighted 0.1.Codec targets and DINOv2 stay frozen; development data does not select checkpoints.Source: §E.4 · p. 30

E.5 STREAMING EXECUTION

  • Streaming consumes each arrived frame and aligned audio, emits five motion frames after the fifth arrival, and incurs a four-frame buffering delay of 160 ms.Both flows use four left-Euler steps at times 0, 1/4, 1/2, and 3/4.Source: §E.5 · p. 30
  • The first main evaluation updates persistent fast weights, later evaluations read them, and a partial final interval emits valid frames without updating them.Transient jaw adaptation is computed and discarded within the interval, and all conversation-local state resets between conversations.Source: §E.5 · p. 31
  • No-write disables both adaptation routes, freeze-one retains the first main update, and current-only clears main deltas after each interval.All controls retain frontend and decoder histories, inputs, solver steps, and matched initial noise.Source: §E.5 · p. 31

F DATASET CONSTRUCTION DETAILS

The dataset pipeline aligns single-view and dual-view conversations on a shared timeline, extracts paired audiovisual and motion annotations, and validates participant correspondence. Baseline comparisons use shared held-out records, while a blind A/B study collects ratings of conversational motion.

F DATASET CONSTRUCTION DETAILS

  • Single-view and dual-view recordings are resampled to 25 fps video and mono 16 kHz audio while preserving temporal origin and aligning crops, speech, motion, activity, and utterances.Dual-view inputs must already be synchronized; the shared timeline preserves when one participant acts and the other responds.Source: F DATASET CONSTRUCTION DETAILS · p. 32
  • Intervals are retained only when both participants have valid landmarks, and single-view speech channels are matched to faces by audiovisual distance without imposing alternating turns.Short bounding-box gaps are interpolated, unreliable boundaries trimmed, and short intervals discarded.Source: F DATASET CONSTRUCTION DETAILS · p. 32
  • FLAME annotations hold identity shape fixed within each participant segment while expression, neck pose, and jaw pose vary over time.The resulting annotations use the same 106-D motion representation for single-view and dual-view videos.Source: F DATASET CONSTRUCTION DETAILS · p. 32
  • Activity labels preserve sole-speaker, overlap, and mutual-silence states, while transcripts cover sole-speaker intervals and overlap remains labeled without transcripts.Samples failing correspondence and identity checks are excluded to target preprocessing errors that could resemble conversational behavior.Source: F DATASET CONSTRUCTION DETAILS · pp. 32–33
  • OOD excludes training participant and source interaction IDs, while RealTalk OOD-Hard uses official single-view intervals with fixed participant pairs.Seamless uses session-level splits; durations count physical segments before expansion into two directions, and metadata IDs do not establish globally unique people.Source: F DATASET CONSTRUCTION DETAILS · p. 33

G BASELINE TRAINING AND COMPARISON PROTOCOL; H PERCEPTUAL EVALUATION DETAILS

Related papers

Loading 2609.35616v1…