Source-linked AI summary
Ex-Omni: Enabling 3D Facial Animation Generation for Omni-modal Large Language Models
Haoyu Zhang, Zhipeng Li, Yiwen Guo, Tianshu Yu
TL;DR
OLLMs have not broadly supported jointly generated speech and 3D facial animation, although synchronized facial motion matters for natural interaction and is difficult to derive from token-level semantics. Ex-Omni addresses this gap with a two-stage speech-and-face generation design, gated semantic fusion, and the InstructS2SF-1200K dataset. Experiments report competitive speech abilities, better synchronization, and lower face-generation latency than cascaded pipelines.
Problem
Existing open-source OLLMs largely omit expressive 3D facial animation, while mapping sparse LLM semantics to dense, temporally smooth facial motion is difficult.
Method
Ex-Omni separates semantic reasoning from temporal synthesis with a blendshape-aware speech unit generator, Blendshape Decoder, TQGF, and InstructS2SF-1200K.
Results
Ex-Omni preserves competitive speech understanding and generation while achieving better SyncNet-based audio-visual synchronization and lower face-generation latency than cascaded pipelines.
Takeaways & Limitations
Native joint speech-and-face generation provides synchronized 3D facial animation within an OLLM while retaining competitive speech-task capability.
Takeaways & Limitations
Speech modeling capacity remains limited by the small speech-generation backbones used in Ex-Omni and Qwen2.5-Omni.
Abstract
from arXiv · showhide
Omni-modal large language models (OLLMs) aim to unify multimodal understanding and generation, yet extending them to jointly produce speech and 3D facial animation remains largely unexplored despite its importance for natural human-computer interaction. A key challenge is the mismatch between the discrete semantic reasoning of LLMs and the dense temporal dynamics required for 3D facial motion. We propose Expressive Omni (Ex-Omni), an open-source model that augments OLLMs with native speech-accompanied 3D facial animation. Ex-Omni decouples semantic reasoning from temporal generation through a blendshape-aware speech unit generator and a blendshape decoder, where speech units provide temporal scaffolding and hidden speech representations carry facially relevant cues. We further introduce a unified token-as-query gated fusion (TQGF) mechanism for controlled semantic injection, as well as InstructS2SF-1200K, a dataset consisting of 1200K samples for pre-training. Extensive experiments show that Ex-Omni maintains competitive speech understanding and generation ability while achieving better audio-visual synchronization and lower face-generation latency than cascaded pipelines.
1 Introduction
Ex-Omni extends omni-modal language models with native speech-aligned 3D facial animation, addressing the mismatch between token-level semantics and dense temporal motion. Its two-stage design, gated fusion, and large-scale dataset support synchronized and efficient generation while preserving competitive speech-task performance.
- Motivation and contribution: 3D facial animation remains underexplored in open-source OLLMs despite its role in conveying non-verbal cues during natural interaction.Existing systems primarily produce linguistic, acoustic, or pixel-level visual outputs.
- Data and results: InstructS2SF-1200K contains 1000K TTSF samples and 200K dialogue-based S2SF samples for speech-blendshape co-pretraining.The dataset is designed for both synthesis and dialogue-oriented training stages.
- Data and results: Ex-Omni preserves competitive speech-task performance while providing better synchronization and lower face-generation latency than cascaded pipelines.The reported comparison concerns native speech-aligned 3D facial animation generation.
- Motivation and contribution: Ex-Omni enables unified instruction following and generation across text, speech, and speech-accompanied 3D facial animation.It is described as among the first open-source OLLMs to natively support speech-aligned 3D facial animation generation.
- Method: Ex-Omni decouples semantic reasoning from temporal generation using a blendshape-aware speech unit generator and Blendshape Decoder.Discrete speech units provide temporal scaffolding for generating ARKit-52 blendshape coefficients non-autoregressively.
- Method: TQGF selectively regulates how and when semantic information enters speech and facial generation, simplifying optimization and improving temporal alignment.The mechanism uses token-as-query gated fusion to bridge high-level semantics and temporal motion generation.
2 Related Work
Related work spans unified omni-modal language models, 2D and 3D facial animation generation, and end-to-end speech language models. These lines of research establish the surrounding capabilities and design strategies relevant to Ex-Omni.
- Omni-modal Large Language Models: Omni-modal language models unify understanding and generation across modalities including text, speech, and vision.Examples include Mini-Omni, OpenOmni, Qwen2.5-Omni, and Ming-Omni, which use different architectures and training strategies.
- Facial Animation Generation: Facial animation research has progressed from mature 2D methods toward 3D generation using mesh or parameterized representations.Recent 3D methods focus on improving realism through these representations.
- Speech Language Models: Speech language models improve end-to-end speech understanding and generation while addressing latency, information bottlenecks, and error accumulation.Examples use full-duplex generation, semantic-perceptual decoupling, speech segmentation, streaming reasoning, and large-scale pre-training.
3 Method
Ex-Omni uses an LLM-centered pipeline that separates semantic reasoning from temporal generation to produce speech and speech-accompanied 3D facial animation. Its speech units provide temporal structure, while gated semantic fusion supplies facially relevant information for non-autoregressive blendshape prediction.
- Overview: The model generates discrete speech units and 3D facial animation parameters within one unified framework.The architecture represents facial motion with blendshape coefficients and decodes speech units into waveform speech.
- Unified Speech-Text Representation: Ex-Omni maps speech and text into a shared LLM embedding space, then uses the LLM for instruction understanding and high-level semantic reasoning.Speech is encoded and projected, while text tokens are embedded before concatenation into a unified input.
- Joint Speech and 3D Facial Animation Generation: TQGF uses token sequences as queries and LLM-derived semantic representations as gated cross-attention context.This asymmetric design assigns temporal responsibility to target tokens while selectively injecting semantic cues.
- Joint Speech and 3D Facial Animation Generation: The speech unit generator conditions autoregressive speech-unit prediction on generated text tokens and semantic hidden states.The resulting speech units provide inputs for facial animation generation.
- Joint Speech and 3D Facial Animation Generation: The blendshape decoder resamples speech-unit features to video-frame length and predicts all 52 facial coefficients in parallel.Frame-level queries are fused with semantically rich contextual representations using TQGF before temporal refinement.
- Training: Training proceeds through speech-to-text pretraining, speech-blendshape co-pretraining, and dialogue-based speech-blendshape co-pretraining.The final stage jointly optimizes the LLM, speech projector, unit generator, and blendshape decoder on ASR, TTSF, and S2SF data while keeping speech encoder and decoder frozen.
- Training Objectives: Text and speech units use autoregressive objectives, whereas facial animation uses frame-wise regression over valid temporal ranges.The autoregressive target is either a text token or speech unit depending on the supervision, and the facial loss compares predicted and ground-truth blendshapes.
4 Data Construction
Ex-Omni’s training corpus combines external ASR data with large-scale synthetic speech-face data constructed for two co-pretraining stages. The dataset uses TTSF and dialogue-based S2SF samples with teacher-generated blendshape supervision.
- Corpus Overview: The full training corpus combines an external ASR corpus with the proposed InstructS2SF-1200K dataset.The corpus is organized around the three-stage training schedule.
- Stage I: Stage I uses about 720K ASR samples, primarily from Emilia, supplemented by LibriSpeech and high-confidence WenetSpeech data.The WenetSpeech subset is filtered to confidence scores above 0.95.
- Stage II: Stage II constructs 1000K Text-to-Speech-Face samples by synthesizing speech from text and generating facial motion with Audio2Face-3D.The synthetic speech uses Qwen3-TTS with a single unified speaker identity, addressing the scarcity of large motion-capture facial-animation datasets.
- Stage III: Stage III constructs 200K dialogue-based Speech-to-Speech-Face samples from InstructS2S-200K using reconstructed speech and Audio2Face-3D blendshape generation.The pipeline conditions Audio2Face-3D on a 5D compound emotion vector estimated from audio.
- Stage III: The Stage III pseudo-labels mainly emphasize mouth articulation and lip-speech synchronization because most generated QA responses are close to neutral.Teacher-generated blendshape sequences serve as weak supervision, and 10K ASR plus 10K TTSF replay samples preserve earlier capabilities.
5 Experiments
Ex-Omni is evaluated for facial-animation quality, speech understanding and generation, synchronization, ablations, latency, and qualitative behavior against cascaded and representative OLLM baselines.
- Evaluation Setup: Evaluation uses Sync-C and Sync-D from SyncNet for lip-speech synchronization, with higher Sync-C and lower Sync-D indicating better alignment.S2F and T2F evaluations use AlpacaEval and CommonEval; T2F changes only the input modality from speech to text.
- 3D Facial Animation Results: Ex-Omni achieves better audio-visual synchronization than cascaded baselines and benefits from native joint speech-face generation within one framework.Cascaded pipelines with different OLLM backbones show similar performance, indicating that downstream task-specific models largely determine cascaded S2F quality.
- Speech QA Results: 65.53 is Ex-Omni’s overall VoiceBench score, ranking second behind Qwen2.5-Omni-7B while surpassing it on WildVoice, BBH, and IFEval.Ex-Omni uses only 200K dialogue-based S2SF samples in Stage III, whereas performance remains weaker on MMSU and OBQA.
- Speech Generation Results: Long generated responses, especially beyond roughly 60 seconds, can raise WER because speech may be truncated or become inaccurate in later segments.The authors attribute this behavior mainly to limited capacity in the small speech-generation backbones used by Ex-Omni and Qwen2.5-Omni.
- Ablation Study: The full TQGF model achieves the best ablation results, including Sync-D 9.233 and Sync-C 5.385, while removing TQGF degrades synchronization.Speech units provide temporal scaffolding, but speech tokens alone do not recover all fine-grained semantic-temporal cues; concatenation performs worst on synchronization at 10.209 / 4.043.
- Latency Analysis: 0.012s is Ex-Omni’s face-generation latency, compared with 0.105–0.117s for cascaded pipelines, while both share Overall RTF 2.158 and Avg Speech TTFT 0.029s.The latency evaluation uses 100 randomly sampled CommonEval instances and compares Overall RTF, Avg Speech TTFT, and Avg Face Latency.
6 Conclusion
Ex-Omni extends OLLMs with native speech-accompanied 3D facial animation and separates semantic reasoning from temporal synthesis. Its experiments show competitive speech abilities alongside improved synchronization and lower face-generation latency than cascaded pipelines.
- Ex-Omni extends OLLMs with native speech-accompanied 3D facial animation generation.
- Ex-Omni addresses the mismatch between token-level semantics and fine-grained facial motion through discrete speech-unit scaffolding and token-as-query gated fusion.
- InstructS2SF-1200K provides large-scale speech-face supervision for synthesis and dialogue-oriented scenarios.
- Ex-Omni preserves competitive speech understanding and generation while achieving better SyncNet-based audio-visual synchronization and lower face-generation latency than cascaded pipelines.