PaperlayerView exact revision on arXiv

Source-linked AI summary

EMODY Flow: Emotion-Aware Audio-Driven Full-Body Motion Generation

Harsh Kumar Agarwal, Xavier Alameda-Pineda, Olivier Perrotin

arXiv:2609.16011v1cs.GRcs.CVcs.LGcs.MMcs.ROcs.SDeess.AS

TL;DR

Omni-modal LLMs understand multiple modalities but respond only with language, leaving a gap in generating embodied responses aligned with speech and emotion. EMODY Flow reuses audio representations from frozen Qwen-3 Omni and adds an auxiliary emotion classifier; it achieves state-of-the-art BEAT2 gesture metrics while addressing suppressed emotion conditioning.

  • Problem

    Omni-modal LLMs respond only with language, leaving a gap in generating embodied responses that coordinate full-body motion with speech and emotion.

    Source: §1 · p. 1

  • Method

    EMODY Flow uses Qwen-3 Omni’s internal Mimi audio embeddings to condition parallel body and face flow generators, with a training-time auxiliary classifier to make generated motion emotion-identifiable.

    Source: §1 · p. 1, §3.4 · p. 4

  • Results

    FGD 0.302, BC 0.853, and Diversity 24.62 on BEAT2 improve over prior state of the art by 26%, 5%, and 62%, respectively.

    Source: §1 · p. 2, §5 · p. 7

  • Takeaways & Limitations

    The auxiliary classifier resolves suppressed low-dimensional conditioning and may extend to other discrete control axes such as speaker persona, occupation, formality, or interaction context.

    Source: §5 · p. 7

    Body and face streams are sampled independently, and their coordination is not measured.

    Source: §5 · p. 7

Abstract

from arXiv · show

Embodied conversational agents require synchronized full-body motion (body gestures and facial expressions) that aligns with speech and emotional state. Omni-modal large language models excel at multimodal understanding but produce only linguistic outputs, leaving a critical gap in embodied response generation. We identify and address a failure of emotion conditioning: like other conditional generators that under-use weak conditioning signals, a flow-matching model given both a rich audio embedding and a discrete emotion label suppresses the emotion, generating near-identical motion regardless of the specified emotion. We present EMODY Flow, a lightweight (around 35M parameters) flow-matching framework that attaches to a frozen Qwen-3 Omni model and reuses its internal Mimi audio-codecs to condition two parallel DiT generators - one for SMPL-X body pose, one for FLAME facial expressions. A training-time auxiliary emotion classifier restores emotion sensitivity by forcing generated motion to be emotion-identifiable. EMODY Flow sets a new state of the art on BEAT2 gesture quality, with FGD 0.302, Beat Correlation 0.853, and Diversity 24.62 - improving over the best prior results by 26%, 5%, and 62% respectively - and transfers to zero-shot facial animation on TFHP without domain-specific fine-tuning. Beyond these quantitative gains, the classifier yields clearly emotion-separated motion, which we demonstrate qualitatively through a multidimensional-scaling analysis of the generated gestures.

1 Introduction

EMODY Flow addresses the gap between omni-modal LLMs’ multimodal understanding and their language-only responses by generating audio-driven, emotion-aware full-body motion. It reuses Qwen-3 Omni’s internal audio representations and adds an auxiliary emotion classifier to counter suppressed emotion conditioning.

1 Introduction

  • A flow model given rich audio embeddings and an explicit emotion label can produce near-identical, emotionally neutral motion regardless of the target emotion.The authors characterize this as under-use of a weak conditioning signal.Source: §1 · p. 1
  • EMODY Flow attaches two parallel DiT-based flow-matching generators to frozen Qwen-3 Omni, using its internal Mimi audio embeddings for SMPL-X body pose and FLAME facial expressions.The framework has about 35M parameters per module and requires no separately trained audio encoder.Source: §1 · p. 1
  • A training-time auxiliary emotion classifier restores sensitivity to the emotion label by making generated motion emotion-identifiable.The classifier loss complements the audio-conditioned flow objective.Source: §1 · pp. 1–2
  • On BEAT2, the emotion-agnostic base model achieves FGD 0.302, BC 0.853, and Diversity 24.62, respectively 26%, 5%, and 62% better than prior state of the art.Source: §1 · p. 2
  • Temporal frame-shuffling regularization and a lip-sync auxiliary loss qualitatively reduce motion jitter and facial desynchronization observed with standard flow-matching training.These are practical training techniques reported by the authors.Source: §1 · p. 2

2 Related Work

Prior co-speech motion systems vary in their inputs, representations, and emotion-control designs. EMODY Flow instead combines continuous motion flow matching with audio representations computed inside an omni-modal LLM.

Co-speech gesture generation

  • GestureLSM is the closest comparison: it uses flow matching in a learned VQ latent space, while EMODY Flow operates directly on continuous SMPL-X/FLAME parameters and avoids VQ quantization loss.GestureLSM tokenizes the body into spatial regions and models their interactions with spatial-temporal attention.Source: §2 · p. 2
  • Interactive systems use richer conversational inputs, but none reuse audio representations computed inside an omni-modal LLM to drive a single lightweight, emotion-controllable motion module.The cited approaches variously compose specialized models or fine-tune large backbones on aligned corpora.Source: §2 · p. 2

Emotion-conditioned motion

  • Prior emotion-controllable methods engineer emotion into motion generation with dedicated disentanglement modules or emotion encoders, whereas EMODY Flow tests joint conditioning with audio and a discrete emotion label.The paper reports that the joint-conditioning flow model under-uses the low-dimensional emotion label, motivating an auxiliary-classifier loss.Source: §2 · p. 2

Flow matching and speech representations

  • Flow matching transports noise to continuous motion through an ODE in a few integration steps, while Mimi provides 512-dimensional continuous codec vectors at 12.5 codes per second.The authors use Mimi embeddings computed inside Qwen-3 Omni’s Talker component for motion synthesis.Source: §2 · p. 2

3 Method

EMODY Flow reuses Qwen-3 Omni’s Mimi audio representations to drive parallel body and face flow generators, with training objectives designed to improve lip synchronization, reduce jitter, and make generated motion reflect emotion.

3.1 System Overview

  • The parallel body and face generators use Mimi embeddings from Qwen-3 Omni’s Talker, the same representations that produce the spoken response.The models output SMPL-X body pose and translation, and FLAME facial expressions.Source: §3.1 · pp. 2–3

3.2 Flow-Matching Generators

  • Each generator learns a velocity field along a linear interpolation from Gaussian noise to a ground-truth motion window, then generates motion by integrating the learned ODE.Inference uses an adaptive RK45 solver.Source: §3.2 · p. 3

3.3 Training Objectives

  • Face training adds an equally weighted lip-blendshape loss to improve synchronization, while shuffling 20% of audio frames during body-model training visibly reduces jitter.Post-hoc Gaussian smoothing further removes residual noise while preserving motion structure.Source: §3.3 · p. 3

3.4 Emotion Conditioning: Failure and Solution

  • Because reconstruction loss supplies no signal to use the emotion label when rich audio embeddings already predict plausible motion, the authors add a classifier loss to encourage emotion-identifiable outputs.The generator is fine-tuned against a frozen classifier after staged generator and classifier training.Source: §3.4 · p. 4
  • After auxiliary-classifier training, body samples gravitate toward target-emotion centroids, while facial samples remain entangled and emotion separation is weak.For body motion, FGD rises from 0.302 to 0.362 as a trade-off for greater expressivity.Source: Figure 3 · p. 5, §3.4 · p. 5

4 Experiments

Experiments evaluate EMODY Flow on BEAT2 gesture generation and zero-shot TFHP facial animation, and examine whether generated gestures reflect their target emotions. The results show strong gesture quality, a trade-off for emotion control, and distinct qualitative motion across emotions.

4.1 Experimental Setup

  • The evaluation uses English BEAT2 with 60 hours of SMPL-X motion, 25 speakers, 1,762 sequences, and eight emotion classes; TFHP facial evaluation is zero-shot and uses neutral emotion.Gesture metrics follow the EMAGE protocol; non-EMODY table rows are quoted from prior work, while EMODY rows are computed under the stated protocols.Source: §4.1 · p. 6
  • For classifier training, the flow network’s predicted velocity forms a differentiable one-step motion reconstruction, allowing the classifier’s cross-entropy gradient to flow back into the generator.The setup trains body and face generators independently using precomputed Mimi embeddings from dataset audio.Source: §4.1 · p. 6

4.2 Main Results

  • FGD 0.302 beats GestureLSM’s 0.409 by 26%; BC 0.853 is best overall, and Diversity 24.62 is 62% above the prior 15.17.Table 1 reports the BEAT2 co-speech gesture comparison.Source: Table 1 · p. 6, §4.2 · p. 6
  • In zero-shot TFHP evaluation, the base model has LVE 11.89 and MOD 2.40; the classifier variant worsens these to 12.11 and 2.47 but improves FDD from 28.10 to 23.95.Table 2 covers talking-head synthesis on TFHP, with lower values better for all metrics and an asterisk marking zero-shot results.Source: Table 2 · p. 6, §4.2 · pp. 6–7
  • With classifier training, FGD rises to 0.362 and BC is 0.852, both second best; the classifier is discarded at inference, leaving emotion as a label input.The base model serves quality-critical rendering, while the classifier variant serves emotion-controllable interaction.Source: §4.2 · p. 6
  • The Diversity and BC gains reflect variation across audio, and additionally across emotion for the classifier variant; the authors attribute reduced high-frequency artifacts to frame shuffling and smoothing.They report that post-processing smoothing slightly worsens FGD (0.362→0.371) and BC (0.852→0.831) while improving visual smoothness.Source: §4.2 · p. 6

4.3 Qualitative Analysis of Emotion Generation

  • Using the same audio for four emotion conditions, generated motion shows relaxed neutral posture, expansive happy gestures, tense forward-leaning anger, and contracted, low-energy sadness.These distinctions are reported as consistent across test sequences and complement the quantitative emotion separation in the MDS projections.Source: §4.3 · p. 7

5 Conclusion

EMODY Flow reuses Mimi audio embeddings from a frozen Qwen-3 Omni model to generate full-body motion, achieving strong BEAT2 gesture results. The paper identifies suppressed emotion conditioning as a central issue, proposes an auxiliary classifier solution, and notes limits in emotion labels, facial control, duration, coordination, and evaluation.

5 Conclusion

  • EMODY Flow reuses Talker-computed Mimi embeddings in a lightweight framework attached to frozen Qwen-3 Omni, achieving BEAT2 FGD 0.302, BC 0.853, and Diversity 24.62.The conclusion presents these as state-of-the-art gesture results.Source: §5 · p. 7
  • The auxiliary classifier addresses emotion suppression by supplying a gradient from the target conditioning dimension; the authors propose the recipe may extend to other low-dimensional controls.Suggested control axes include speaker persona, occupation, formality, and interaction context.Source: §5 · p. 7
  • The authors state that discrete labels oversimplify continuous emotion, facial emotion conditioning is weak, generation spans 10 seconds, body-face coordination is unmeasured, and evaluation relies on automatic metrics.Autoregressive extension beyond the 10-second window introduces transition artifacts.Source: §5 · p. 7

Ethics and Privacy Statement

The work aims to support more natural human–AI interaction through expressive avatar animation while noting risks of misuse and limitations in generalizing emotion conditioning.

Ethics and Privacy Statement

  • Experiments use publicly available BEAT2 data collected with participant consent, and the research collected no new human-subject data.Source: Ethics and Privacy Statement · p. 7
  • The primary societal benefit is more natural human–AI interaction through expressive avatar animation.Source: Ethics and Privacy Statement · p. 7
  • Realistic synchronized motion and facial expressions from audio raise deepfake and deceptive-avatar concerns; the authors encourage disclosure of synthetic agents.Source: Ethics and Privacy Statement · p. 7
  • The emotion classifier uses discrete BEAT2 labels that may reflect cultural and demographic biases, limiting generalization to speakers outside the training distribution.Source: Ethics and Privacy Statement · p. 7

Neutral

Figure 4 presents generated motion conditioned on four emotion classes, with visible differences in posture, gesture amplitude, and facial configuration.

Neutral

  • Figure 4 shows representative generated motion frames conditioned on four emotion classes, with visible differences in posture, gesture amplitude, and facial configuration.Source: Figure 4 · p. 8

A Implementation Details

The appendix supplies implementation details for reproducibility and describes the shared DiT configuration and task-specific objectives of the two generators.

A Implementation Details

A.1 Model Configuration

  • The body and face generators share DiT hyperparameters but differ in output dimension and use separate task-specific objectives.The face model adds lip-sync loss, while the body model uses temporal regularization.Source: §A.1 · p. 9
  • Each DiT block emits nine AdaLN modulation parameters for self-attention, cross-attention, and the MLP, rather than the six used in vanilla DiT.Source: §A.1 · p. 9

A.2 Training and Inference

Training uses a three-stage generator–classifier schedule and feeds the generators Mimi audio embeddings; the appendix also specifies classifier architecture and compute settings.

A.2 Training and Inference

  • Stage 1 trains the generator with flow matching, stage 2 trains a classifier for 150 epochs, and stage 3 trains the restored generator against the frozen classifier.Stage-2 generator updates are discarded; body and face models use different stage-1 and stage-3 epoch counts.Source: §A.2 · p. 9, Table 5 · p. 10
  • The body classifier weight is λclf=0.005, while face classification loss was unstable across the swept weights and did not yield clean emotion separation.The face-model sweep tested {0.01, 0.1, 1, 10}.Source: §A.2 · p. 9
  • Mimi audio is supplied as eight residual vector-quantization streams, yielding 512-dimensional embeddings at approximately 12.5 codes per second at 24 kHz.BEAT2 audio is resampled from 16 kHz to 24 kHz during preprocessing.Source: §A.2 · p. 9
  • Training runs on one NVIDIA A6000 at approximately 75 seconds per epoch, and generating the full test set takes about six minutes.Source: §A.2 · p. 10

A.3 Auxiliary Emotion Classifier

  • The classifier is a 1D ResNet with three residual blocks, temporal downsampling, a 128-dimensional embedding, and an eight-logit output head.Cross-entropy uses inverse-frequency class weights.Source: §A.3 · p. 10

A.4 Rejected Emotion-Conditioning Fixes

Three attempted fixes—audio dropout, an audio bottleneck, and flattened pitch and loudness—did not improve emotion correlation or produce emotion-separated motion, motivating an explicit classifier signal.

A.4 Rejected Emotion-Conditioning Fixes

  • Audio dropout, which left the emotion label as the only cue on 10% of training examples, did not improve emotion correlation.The approach dropped the audio embedding with 10% probability during training.Source: §A.4 · p. 10
  • An audio-embedding bottleneck (512→64→512) limiting audio information did not improve emotion correlation.Source: §A.4 · p. 10
  • Flattening pitch and loudness in the input audio did not produce emotion-separated motion, motivating an explicit classifier signal.None of the three tried fixes produced emotion-separated motion.Source: §A.4 · p. 10

B Dataset Class Distribution

The BEAT2 training split is imbalanced, so classifier loss uses inverse-frequency class weighting. At inference, audio and an emotion label condition window-wise motion generation followed by smoothing and rendering.

B Dataset Class Distribution

C Inference Procedure

  • At inference, audio is encoded as Mimi embeddings and motion is generated for 10-second windows with a 300-frame stride.Talker embeddings may be used instead of audio at inference; window embeddings are zero-padded to 300 frames and generation starts from sampled Gaussian noise.Source: C Inference Procedure · p. 10
  • For each window, the model integrates the conditioned ODE from t=0 to t=1 with dopri5 to obtain the generated motion.The ODE is conditioned on the window embeddings and emotion label.Source: C Inference Procedure · p. 10
  • Generated windows are concatenated, smoothed with a Gaussian filter (σ=1.4), and rendered through a standard SMPL-X/FLAME pipeline.Source: C Inference Procedure · p. 10
Loading 2609.16011v1…