PaperlayerView exact revision on arXiv

Source-linked AI summary

Learning to Listen: Modeling Non-Deterministic Dyadic Facial Motion

Evonne Ng, Hanbyul Joo, Liwen Hu, Hao Li, Trevor Darrell, Angjoo Kanazawa, Shiry Ginosar

arXiv:2204.08451v1cs.CV

TL;DR

The paper addresses the challenge of generating realistic, nondeterministic listener motion from multimodal speaker inputs in dyadic conversation. It combines motion-audio cross-attention with a motion VQ-VAE and evaluates the resulting 3D motion using broad quantitative and human studies, reporting stronger overall performance than competing methods. A 72-hour in-the-wild dataset supports this research, while the current setting excludes gaze and full conversational back-and-forth.

  • Problem

    Listener responses are nondeterministic and speakers communicate through both speech and nonverbal motion, while existing conversational data is often limited to laboratory or small-scale recordings.

  • Method

    The method fuses speaker speech audio and facial motion with cross-attention, then uses a motion VQ-VAE and autoregressive prediction to sample multiple synchronous listener trajectories.

  • Results

    The proposed method achieves the best overall balance across realism, diversity, and synchrony metrics among the compared methods, supported by qualitative and human evaluations.

  • Takeaways & Limitations

    The framework models multiple realistic 3D listener responses synchronized with a speaker and provides a 72-hour in-the-wild dyadic-conversation dataset for further study.

  • Takeaways & Limitations

    The study uses teleconferencing data without eye contact and models only listener responses to a speaker, leaving in-person gaze and full dyadic back-and-forth for future work.

Abstract

from arXiv · show

We present a framework for modeling interactional communication in dyadic conversations: given multimodal inputs of a speaker, we autoregressively output multiple possibilities of corresponding listener motion. We combine the motion and speech audio of the speaker using a motion-audio cross attention transformer. Furthermore, we enable non-deterministic prediction by learning a discrete latent representation of realistic listener motion with a novel motion-encoding VQ-VAE. Our method organically captures the multimodal and non-deterministic nature of nonverbal dyadic interactions. Moreover, it produces realistic 3D listener facial motion synchronous with the speaker (see video). We demonstrate that our method outperforms baselines qualitatively and quantitatively via a rich suite of experiments. To facilitate this line of research, we introduce a novel and large in-the-wild dataset of dyadic conversations. Code, data, and videos available at https://evonneng.github.io/learning2listen/.

1. Introduction

The paper models dyadic listener motion as a multimodal, nondeterministic response to a speaker, addressing the difficulty of capturing realistic conversational dynamics. It combines speaker motion and speech with learned discrete motion representations and introduces a large in-the-wild dataset for evaluation.

  • Motivation: Coordinated speaker-listener motion supports successful conversation, rapport, and meaningful interaction with virtual agents.The paper frames nonverbal communication as central to how people signal mutual communication.
  • Motivation: Listener responses are nondeterministic, while speakers communicate through both speech and facial or body motion.These properties make realistic modeling of dyadic interaction difficult for rule-based, hand-designed, and database-driven methods.
  • Approach: The model combines speaker speech audio and facial motion through motion-audio cross-attention to synthesize multiple synchronous listener responses autoregressively.Different generated modes represent possible listener reactions to the same multimodal speaker input.
  • Approach: A sequence-encoding VQ-VAE learns a discrete listener-motion space from which multinomial sampling produces diverse, perceptually plausible responses without long-horizon motion drift.The discrete codebook supports multiple modes and autoregressive prediction for speaker sequences of any length.
  • Dataset and evaluation: The authors collect 72 hours of in-the-wild dyadic conversations with frontal views and detailed 3D annotations.The dataset is intended to support machine-learning research on dyadic communication beyond small or laboratory datasets.
  • Dataset and evaluation: Evaluation compares synthesized motion with ground truth, baselines, and ablations using metrics for realism, diversity, and speaker-listener synchrony, plus human judgments.The study evaluates raw 3D mesh output and uses pixel-rendered video only for viewing.

2. Related Work

Related work spans rule-based conversational agents, data-driven interactional motion, and conditional motion synthesis. The paper distinguishes its approach by learning a 3D discretized facial-motion space and using multimodal transformer-based generation.

  • Conversational agents: Earlier conversational-avatar systems manually encode interactional motion with rules, limiting gesture variety or relying on simplifying assumptions.The cited approaches are described as poorly suited to in-the-wild data.
  • Data-driven interactional motion: Prior data-driven methods predict 2D conversational motion using predefined dictionaries built from clustered motion frequencies or facial keypoints.The paper instead reasons in 3D and learns a discretized latent space for facial motion.
  • Data-driven interactional motion: Other work investigates 3D interactional dynamics through full-body motion and turn taking, while recent facial-motion methods add listener text or speech inputs.The paper positions its setting around 3D facial motion conditioned on speaker-side inputs.
  • Conditional motion synthesis: Conditional motion synthesis has used convolutional autoencoders, adversarial losses, flow models, sampling methods, and transformers to generate diverse motion.Recent transformer methods are associated with modeling diverse motion with long-range dependencies.

3. Method

The method predicts listener facial motion autoregressively from speaker audio and motion while representing listener responses with a discrete VQ-VAE codebook. Sampling codebook indices enables multiple realistic, synchronous motion trajectories.

  • Problem definition: The task is to autoregressively predict corresponding 3D listener facial motion from a speaker’s facial motion and audio.
  • Listener motion prediction: The predictor models a multinomial distribution over the next listener codebook index, then samples and decodes that index into continuous future motion.
  • Speaker input: A motion-audio cross-modal transformer fuses speaker audio and facial motion, allowing temporal re-alignment between modalities.
  • Quantized listener motion: A transformer-based sequence-encoding VQ-VAE maps listener motion segments to a discrete codebook and reconstructs approximate motion through a decoder.
  • Benefits: Discrete latent codes support multiple output modes, reduce autoregressive drift by staying on the realistic motion manifold, and preserve high-frequency movements.
  • Speaker input: Speaker audio and motion are kept continuous because speaker-motion quantization did not improve results, while audio quantization significantly deteriorated them.

4. In-the-wild Conversational Dataset

The paper introduces a 72-hour in-the-wild dyadic conversation dataset from split-screen online interviews, with extracted facial motion and audio for modeling communication.

  • Dataset: The dataset contains 72 hours of in-the-wild conversations recorded through split-screen online interviews.The setup captures speakers and listeners in frontal view across diverse interviewees, hosts, and backgrounds.
  • Data extraction: Facial motion is recovered as 3D head pose and expression coefficients using DECA and the FLAME 3DMM.The representation includes 50 expression coefficients, jaw rotation, and 3D head rotation.
  • Data extraction: Speaker-only audio is isolated with sound-source separation and paired with facial motion for model training.The extracted expressions, poses, and audio serve as pseudo-ground truth for the codebook and prediction model.
  • Dataset: The dataset is publicly released to support machine-learning research on dyadic communication.Its in-the-wild scale is intended to enable investigation of conversational interaction beyond laboratory or small motion-capture settings.

5. Experiments

The experiments evaluate listener facial motion through realism, diversity, and synchrony metrics, comparisons with baselines, ablations, and human judgments. The proposed method achieves the strongest overall balance and benefits from quantized motion representations and cross-modal fusion.

  • Experimental setup: The evaluation measures listener motion realism, diversity, and synchrony using multiple complementary metrics.Metrics are applied to expression and rotation, with comparisons against raw ground-truth listener motion.
  • Evaluation metrics: Frechet distance measures realism, variation and SI measure diversity, and paired FD, PCC, and TLCC measure listener-speaker synchrony.PCC evaluates covariance in smile or nod motion, while TLCC identifies the lag of peak synchronization.
  • Quantitative results: The proposed method outperforms competing methods across metrics and balances realism, diversity, and synchrony better than individual baselines.Median is competitive on L2 but less diverse and realistic, while nearest-neighbor and random methods lack expression synchrony; mirroring produces excessive synchrony and variation.
  • Quantitative results: The VQ-VAE produces a realistic motion manifold: Ours Random Walk significantly outperforms Random Expression while remaining competitive with Random.Codebook walks generate realistic motion, although they are not necessarily synchronized with the speaker.
  • Synchrony analysis: Ground truth and the proposed method both show an average listener response time of approximately 17 frames, or 0.5 seconds.This response time is used as the optimal delay for the Mirror Delay baseline.
  • Model ablations: Removing the VQ-VAE produces unrealistic, overly smoothed sequences, while cross-attention improves performance over concatenating audio and motion or using either modality alone.The ablations support the importance of both the codebook and cross-modal fusion.
  • Qualitative results: Human evaluators preferred Ours over NN motion 75.3% of the time and over the audio-motion concatenation ablation 71.1% of the time.Evaluators also preferred Ours over avatars rendered from ground-truth listeners 50.1% of the time.

6. Discussion

The method models synchronous, non-deterministic listener facial motion from multimodal speaker inputs using cross-attention and VQ-VAE representations. Its scope is bounded by teleconferencing artifacts and modeling only the listener’s response to the speaker.

  • Discussion: The framework combines speaker motion and audio with cross-attention and autoregressively produces convincing 3D listener facial motion correlated with the speaker.It is trained on an in-the-wild dyadic-conversation dataset.
  • Discussion: Teleconferencing data limits the setting through absent eye contact and time delays introduced by remote connections.The authors identify in-person conversations with gaze as a future direction.
  • Discussion: Modeling only listener motion in response to a speaker leaves the full dyadic back-and-forth cycle for future work.The paper also directs readers to its appendix for misuse concerns and additional analyses.

A. Related Works

Prior work establishes synchronous motion as a central feature of human communication and has developed computational measures for detecting it. These studies span manual analysis, physiological signals, and 2D or 3D visual keypoints.

  • A. Related Works: Human synchrony is rooted in early interaction, including infant-caregiver affective synchrony and synchronous behavior interpreted as independent agency.The passage frames synchronous interaction as central to human communication.
  • A. Related Works: Interactional synchrony describes listener motion flowing in rhythm with the speaker’s speech and motion.Early dyadic-conversation studies used manual analysis of videotaped data.
  • A. Related Works: Computational studies detect interactional motion using EMG correlation and facial 2D or full-body 3D keypoints.One reported finding is that smiles typically synchronize within 1 second.

B. Method

The method processes multimodal speaker signals with a cross-modal transformer, quantizes listener motion with a VQ-VAE, and predicts listener-motion tokens autoregressively. Its implementation aligns audio, speaker motion, and listener representations across temporal scales.

  • B. Method: The VQ-VAE uses three convolutional and max-pooling stages before a Transformer bottleneck to encode listener-motion sequences.A length-32 sequence is reduced to length 4, with hidden size 512, 8 heads, and 12 layers.
  • B. Method: The cross-modal transformer projects audio and speaker motion separately, uses audio as queries over motion keys and values, then downsamples temporally.The output is matched to the temporal extent of the quantized listener input.
  • B. Method: Audio is converted to a 16,000-Hz mel spectrogram, downsampled to the motion sequence length, and independently projected before cross-modal fusion.The raw audio sequence is initially four times longer than the motion sequence.
  • B. Method: Downsampling the speaker representation improved results empirically by avoiding overly long conditioning and temporal mismatch with the shorter listener sequence.Without convolutional downsampling, the network would need to learn alignment between different-length sequences on its own.
  • B. Method: The predictor embeds listener indices, concatenates them with speaker features, and uses masked attention during training to support autoregressive generation.At test time, it progressively reduces masking and takes only the first output index.

C. In-the-wild Conversational Dataset

The dataset is built from split-screen interview videos and processed to isolate relevant two-person conversational segments and speaker-listener roles. The authors report no non-public PII and no human subjects under the cited regulatory definition.

  • C. In-the-wild Conversational Dataset: The authors report that the dataset contains no non-public PII and no human subjects under 45 CFR 46.This statement is attributed to guidance from the local IRB.
  • C. In-the-wild Conversational Dataset: The dataset-processing pipeline extracts frames at 30 fps and removes irrelevant video segments before annotating speaker and listener splits.The source videos may switch between split-screen, single-person views, and inserted pictures.
  • C. In-the-wild Conversational Dataset: Relevant segments are identified with facial detection by retaining footage containing two conversing faces and removing unsuitable views or false positives.The process addresses view switches and extraneous detections in interview videos.
  • C. In-the-wild Conversational Dataset: Speaker-listener roles are detected by combining active speaker identification with sound-source separation in a voting-based approach.Using either method independently produced noisy speaker predictions for different reasons.

D. Evaluation

The evaluation compares the proposed method with baselines and ablations across realism, diversity, synchronization, and listener-generalization settings. Results show strong performance across person-specific, listener-agnostic, motion-only, and multimodal evaluations, while also identifying a data-coverage limitation for some listeners.

  • Person-specific evaluation: Across person-specific results, Ours outperforms all baselines and ablations for the evaluated listeners.The reported trends remain consistent across Persons A, B, C, and D.
  • Evaluation scope: Two listeners are excluded from person-specific modeling because guest-host appearances provide insufficient data, although those data portions are released.This limits person-specific training coverage for those listeners rather than excluding the data entirely.
  • Listener-agnostic evaluation: In listener-agnostic evaluation, Ours-agnostic ranks first on FD and PFD, while person-specific modeling better captures individual listening styles.Ours-agnostic achieves FD 30.01 and PFD 31.36; lower is better for both metrics.
  • Cross-dataset evaluation: On the LFI dataset with motion-only input, the proposed motion model outperforms LFI on FD, PFD, and variation.The reported values are Ours m: FD 1.88, PFD 2.12, variation 1.82; LFI: FD 2.97, PFD 3.10, variation 1.01.
  • Evaluation protocol: The evaluation uses metrics covering realism, diversity, synchronization, and listener-speaker dynamics, alongside qualitative human judgments.The metrics include FD, P-FD, variation, SI, and time-lagged cross-correlation analyses.
  • Multiple-output evaluation: When sampling multiple listener trajectories, Ours reaches lower average minimum L2 distance in fewer samples than the a+m ablation.The analysis samples up to 200 trajectories from the same 256-length test sequences and compares their closest match to ground truth.

E. Risks and Potential Misuse

The framework is intended for human-machine interaction, but its photorealistic outputs can be mistaken for real people. This creates potential misuse risks, including fabricated conversations or unauthorized meeting attendance.

  • Potential misuse: Photorealistic synthesized videos can be confused with recordings of real people and potentially used to fabricate conversations or impersonate meeting participants.The authors advocate safe use and watermarking synthesized content.
Loading 2204.08451v1…