Source-linked AI summary
HumanOmni: A Large Vision-Speech Language Model for Human-Centric Video Understanding
Jiaxing Zhao, Qize Yang, Yixing Peng, Detao Bai, Shimin Yao, Boyuan Sun, Xiang Chen, Shenghao Fu, Weixuan chen, Xihan Wei, Liefeng Bo
TL;DR
HumanOmni addresses the lack of large-scale fine-grained human-centric data and targeted architectures for multimodal video understanding. It combines specialized visual branches, instruction-driven feature fusion, and joint audio-video processing, achieving strong performance across human-centric tasks and comparable results on automatic speech recognition.
Problem
Large-scale human-centric datasets with fine-grained annotations are lacking, while existing multimodal methods show performance gaps on dynamic emotion recognition.
Method
HumanOmni uses a dataset with over 2.4M human-centric video clips and over 14M instructions, three specialized visual branches, instruction-driven feature fusion, and joint vision-speech processing.
Results
HumanOmni achieves state-of-the-art performance across various human-centric tasks, surpassing open-source, specialized, video-language, audio-language, and omni models in reported evaluations.
Takeaways & Limitations
Joint video-audio understanding and instruction-adaptive fusion support comprehensive human-centric scene understanding across emotion recognition and facial expression description tasks.
Abstract
from arXiv · showhide
In human-centric scenes, the ability to simultaneously understand visual and auditory information is crucial. While recent omni models can process multiple modalities, they generally lack effectiveness in human-centric scenes due to the absence of large-scale, specialized datasets and non-targeted architectures. In this work, we developed HumanOmni, the industry's first human-centric Omni-multimodal large language model. We constructed a dataset containing over 2.4 million human-centric video clips with detailed captions and more than 14 million instructions, facilitating the understanding of diverse human-centric scenes. HumanOmni includes three specialized branches for understanding different types of scenes. It adaptively fuses features from these branches based on user instructions, significantly enhancing visual understanding in scenes centered around individuals. Moreover, HumanOmni integrates audio features to ensure a comprehensive understanding of environments and individuals. Our experiments validate HumanOmni's advanced capabilities in handling human-centric scenes across a variety of tasks, including emotion recognition, facial expression description, and action understanding. Our model will be open-sourced to facilitate further development and collaboration within both academia and industry.
1 Introduction
HumanOmni addresses the limited effectiveness of existing multimodal models in human-centric scenes by combining vision and speech understanding with specialized data and architecture. It introduces a large human-centric dataset, instruction-driven visual branching, and broad multimodal evaluation.
- Existing vision-language models generally lack audio processing, while omni models often lack targeted training and specialized architecture for human-centric scenes.
- HumanOmni processes vision and speech simultaneously for human-centric video understanding.
- The dataset contains over 2.4M human-centric video clips, over 14M visual-pretraining instructions, and 50K manually annotated clips with more than 100K fine-tuning instructions.
- Three branches separately handle face-related, body-related, and interaction-related scenes, while instruction-driven fusion dynamically adjusts their feature weights.
- HumanOmni achieves state-of-the-art performance across human-centric tasks and remains comparable to leading models on audio-only automatic speech recognition.
2 Our Model
HumanOmni combines multimodal encoding with three specialized visual branches and instruction-driven fusion. Audio, visual, and text representations are projected into a shared space, while fusion weights adapt to the user’s request.
- HumanOmni processes textual, auditory, and visual inputs through a unified multimodal pipeline.
- Three visual branches target face-related, body-related, and interaction-related features because different human-centric tasks depend on different visual cues.
- The face branch uses a detail-sensitive projector, whereas body and interaction branches use spatial-temporal projectors for continuous actions and interactions.
- Instruction-driven fusion weights branch features according to the user’s request, emphasizing facial features for emotion recognition and interaction features for interaction scenes.
- BERT encodes instructions and two MLPs transform the [CLS] representation into weights for combining the three visual projectors.
- The instruction-driven fusion approach improves flexibility and adaptability while reducing the resource burden of directly concatenating all branch features.
- Whisper-large-v3 encodes audio, MLP2xGeLU maps it into the text domain, and audio, visual, and text tokens are passed to the LLM decoder.
3 Human-Centric Data Collection
The data pipeline builds a fine-grained human-centric video dataset by filtering temporal quality and redundancy, generating verified captions and annotations, and converting them into branch-specific instructions. The resulting collection supports face, body, and interaction pretraining plus manually annotated multimodal tasks.
- The dataset addresses the lack of large-scale human-centric video data with fine-grained annotations.
- Video Collection: Scene detection and segmentation divide videos into smoother clips, while keyframe analysis removes clips with minimal temporal variation and low-resolution clips are excluded.
- Video Collection: Semantic similarity filtering removes redundant clips after brief descriptions are generated with Qwen2-VL-72B.
- Video Collection: Detailed captions are generated and cross-checked with multiple multimodal models, while person and face bounding boxes align visual content with annotations.
- Instruction Data: Qwen2.5-72B converts captions from 2.4M collected clips into structured instruction pairs for the face, body, and interaction branches.
- Instruction Data: The face, body, and interaction branches receive 4.12M, 5.75M, and 4.8M instruction pairs, respectively.
- Annotation: The pretraining instructions use double-checked captions and filtered clips, while manual annotations cover emotion, facial expression, and speaker-specific speech tasks.
4 Training
HumanOmni is trained through staged visual, auditory, and cross-modal learning, using specialized visual branches and instruction-driven fusion. Its audio training draws on diverse speech, captioning, and sound-event data before joint multimodal optimization.
- Training strategy: Training proceeds in three stages: visual capability construction, standalone audio capability development, and cross-modal interaction training.The stages respectively build spatio-temporal visual representations, speech understanding, and integrated audio-visual processing.
- Visual capability construction: Three visual branches separately process face-related, body-related, and interaction-related scenes, with branch-specific instruction data used during pretraining.Only projector parameters are updated during branch pretraining to keep the remaining model parameters aligned for later integration.
- Visual capability construction: An instruction-driven fusion module combines the three branches during fine-tuning while training the projectors, language model, and fusion-weight MLPs.The visual encoder and BERT parameters remain frozen during this phase.
- Auditory capability development: Approximately 18,000 hours of audio data train the audio projector across automatic captioning, speech recognition, and sound-event classification tasks.The audio collection includes WavCaps and multiple ASR datasets covering diverse speech and language settings.
- Cross-modal interaction training: Cross-modal training jointly fine-tunes the decoder, projectors, and fusion MLPs on synthesized visual-auditory interaction data.Special tokens distinguish modalities, while missing modalities are replaced with default tokens to support visual-only and audio-visual inputs.
5 Experiments
Experiments evaluate HumanOmni on emotion recognition, facial expression description, human action understanding, speech recognition, and modality effects. Across these evaluations, the model shows strong performance in human-centric multimodal tasks and leads existing Omni models on speech benchmarks, while audio-visual input is especially beneficial for emotion recognition.
- Evaluation scope: HumanOmni is evaluated on emotion recognition, facial expression description, action understanding, audio-only speech recognition, and modality effects.The evaluation spans multiple human-centric tasks and compares different input configurations.
- Emotion recognition: HumanOmni significantly outperforms video-language, audio-language, Omni, and specialized methods on dynamic emotion recognition.The authors attribute the task’s relevance to combining video and audio information.
- Facial expression description: HumanOmni with combined video and audio input surpasses open-source models and FaceTrack-MM on facial expression description.The evaluation uses the DFEC benchmark and its recommended metrics.
- Action understanding: HumanOmni significantly outperforms nearly all mainstream methods of the same parameter size on selected human-related MVBench tasks.The exceptions are a few methods that use the full MVBench dataset.
- Speech recognition: HumanOmni leads current Omni models across four speech-recognition benchmarks, though audio-visual methods remain behind proprietary speech-recognition approaches.The benchmarks include LibriSpeech, WenetSpeech, and Fleurs among the reported evaluation datasets.
- Modality effects: For emotion recognition, audio-only or video-only configurations perform notably worse than the combined video-and-audio configuration.The modality comparison also covers facial expression description and action understanding.
6 Conclusion
HumanOmni combines a large human-centric dataset, specialized visual branches, instruction-guided feature fusion, and joint audio-video processing. Extensive experiments report effectiveness across complex human-centered interactions, and the authors plan to open-source the model and code.
- Conclusion: HumanOmni uses over 2.4 million human-centric video clips and more than 14 million captions and instructions for diverse scene understanding.Its dataset is paired with face-, body-, and interaction-related branches for different human-centric scene categories.
- Conclusion: Instruction-guided adaptive fusion combines features from the three specialized branches to improve robustness across scenarios.The model also supports joint audio and video input for more comprehensive scene understanding.
- Conclusion: Experiments across multiple human-centric tasks demonstrate HumanOmni’s effectiveness in understanding complex human-centered interactions.The authors state that the code and model will be open-sourced for further research and community development.