PaperlayerView exact revision on arXiv

Source-linked AI summary

Baichuan-Omni-1.5 Technical Report

Yadong Li, Jun Liu, Tao Zhang, Tao Zhang, Song Chen, Tianpeng Li, Zehuan Li, Lijun Liu, Lingfeng Ming, Guosheng Dong, Da Pan, Chong Li, Yuanbo Fang, Dongdong Kuang, Mingrui Wang, Chenglin Zhu, Youwei Zhang, Hongyu Guo, Fengyu Zhang, Yuran Wang, Bowen Ding, Wei Song, Xu Li, Yuqi Huo, Zheng Liang, Shusen Zhang, Xin Wu, Shuai Zhao, Linchu Xiong, Yozhen Wu, Jiahui Ye, Wenhao Lu, Bowen Li, Yan Zhang, Yaqi Zhou, Xin Chen, Lei Su, Hongda Zhang, Fuzhong Chen, Xuezhen Dong, Na Nie, Zhiying Wu, Bin Xiao, Ting Li, Shunya Dang, Ping Zhang, Yijia Sun, Jincheng Wu, Jinjie Yang, Xionghai Lin, Zhi Ma, Kegeng Wu, Jia li, Aiyuan Yang, Hui Liu, Jianqiang Zhang, Xiaoxi Chen, Guangwei Ai, Wentao Zhang, Yicong Chen, Xiaoqin Huang, Kun Li, Wenjing Luo, Yifei Duan, Lingling Zhu, Ran Xiao, Zhe Su, Jiani Pu, Dian Wang, Xu Jia, Tianyu Zhang, Mengyu Ai, Mang Wang, Yujing Qiao, Lei Zhang, Yanjun Shen, Fan Yang, Miao Zhen, Yijie Zhou, Mingyang Chen, Fei Li, Chenzheng Zhu, Keer Lu, Yaqi Zhao, Hao Liang, Youquan Li, Yanzhao Qin, Linzhuang Sun, Jianhua Xu, Haoze Sun, Mingan Lin, Zenan Zhou, Weipeng Chen

arXiv:2501.15368v1cs.CLcs.SDeess.AS

TL;DR

Baichuan-Omni-1.5 targets the limited modality coverage and interaction efficiency of open-source MLLMs. It combines broad multimodal data, a semantic-acoustic audio tokenizer, and multistage training to support unified understanding and end-to-end audio generation, reporting strong general and medical results while retaining clear areas for improvement.

  • Problem

    Open-source MLLMs have typically focused on visual-text integration, while separate ASR and TTS modules increase latency and complexity for real-time multimodal interaction.

  • Method

    The model unifies visual and audio branches with a pretrained LLM, using cross-modal pretraining, an 8-layer RVQ audio tokenizer, and two-stage audio-text integration.

  • Results

    Baichuan-Omni-1.5 demonstrates strong text, image, audio, and video understanding, high-quality controllable audio generation, and leading medical image-understanding results.

  • Takeaways & Limitations

    The reported results support a unified omni-modal system capable of cross-modal interaction and dual text-and-audio outputs across general and medical settings.

  • Takeaways & Limitations

    The authors identify room to improve text understanding, longer-video understanding, and audio understanding and generation for natural environmental sounds.

Abstract

from arXiv · show

We introduce Baichuan-Omni-1.5, an omni-modal model that not only has omni-modal understanding capabilities but also provides end-to-end audio generation capabilities. To achieve fluent and high-quality interaction across modalities without compromising the capabilities of any modality, we prioritized optimizing three key aspects. First, we establish a comprehensive data cleaning and synthesis pipeline for multimodal data, obtaining about 500B high-quality data (text, audio, and vision). Second, an audio-tokenizer (Baichuan-Audio-Tokenizer) has been designed to capture both semantic and acoustic information from audio, enabling seamless integration and enhanced compatibility with MLLM. Lastly, we designed a multi-stage training strategy that progressively integrates multimodal alignment and multitask fine-tuning, ensuring effective synergy across all modalities. Baichuan-Omni-1.5 leads contemporary models (including GPT4o-mini and MiniCPM-o 2.6) in terms of comprehensive omni-modal capabilities. Notably, it achieves results comparable to leading models such as Qwen2-VL-72B across various multimodal medical benchmarks.

1 Introduction

Baichuan-Omni-1.5 addresses the limited modality coverage and interaction costs of open-source MLLMs with unified multimodal understanding and speech generation. It reports strong general, vision-language, speech, and medical capabilities.

  • Motivation: Open-source MLLMs have typically centered on visual-text integration, while separate ASR and TTS modules increase latency and complexity for real-time interaction.Recent end-to-end approaches use staged training to incorporate different modalities.
  • Contributions: Baichuan-Omni-1.5 processes text, image, audio, and video inputs and produces high-quality text and speech outputs without compromising modality capabilities.The architecture supports multimodal interaction across pure and combined inputs.
  • Results: 73.3 average across ten image-understanding benchmarks surpasses GPT-4o-mini by an average of 6 points.This is reported as the model’s vision-language capability.
  • Results: An 8-layer RVQ audio tokenizer operating at 12.5 Hz captures semantic and acoustic information for controllable bilingual real-time conversations.The authors also introduce OpenAudioBench for end-to-end audio understanding and generation evaluation.
  • Results: 83.8% on OpenMM-Medical using a 7B LLM surpasses Qwen2-VL-72B’s 80.7% score.The model is also reported to achieve state-of-the-art performance on GMAI-MMBench.

2 Related works

Related work describes the progression from multimodal language models toward omni models that combine text, vision, and audio. Medical MLLMs increasingly integrate diverse biomedical data for broader medical understanding.

  • Multimodal models: MLLMs connect visual and textual information to support multimodal tasks and machine interaction with the world.The related work situates open-source MLLMs within broader advances in language modeling.
  • Omni models: Omni models fuse text, vision, and audio streams to reason over richer contexts and support cross-modal tasks.The cited work links multimodal fusion with single-modality and cross-modal capabilities.
  • Medical applications: Medical MLLMs combine visual and textual biomedical data to provide more comprehensive medical insights.Examples include systems supporting multiple biomedical modalities and medical-context adaptation.
  • Medical applications: Med-PaLM fine-tunes PaLM-E with millions of samples for medical contexts.This example illustrates domain-specific adaptation in medical MLLM research.

3 Baichuan-Omni-1.5

This section introduces the paper’s overview of Baichuan-Omni-1.5, covering its data, architecture, and multistage multimodal training strategy.

  • The paper provides an overview of Baichuan-Omni-1.5 through its high-quality data, model architecture, and multi-stage multimodal training strategy.

3.1 High-Quality Multimodal Pretrain Data

Baichuan-Omni-1.5 is trained on broad cross-modal data spanning text, images, video, audio, and their interactions. The pipeline combines public, synthesized, interleaved, and cleaned data sources.

  • Dataset scope: The pretraining collection includes text, image-text, video-text, audio-text, and interleaved image-audio-text and video-audio-text data.The dataset is illustrated as an extensive omni-modal collection.
  • Image data: Image pretraining data is organized into interleaved image-text, caption, and question-answer data from diverse open-source sources.The pipeline uses sampling techniques to construct different data ratios.
  • Image data: Image synthesis adds interleaved image-text, OCR, chart, and detailed caption data from books, papers, and a dedicated caption model.These sources are intended to improve data diversity and provide specialized, knowledge-intensive content.
  • Video data: Video pretraining covers classification, action recognition, temporal localization, captioning, and question answering from public resources.Video captions also include GPT-4o-generated descriptions for YouTube videos.
  • Audio data: Audio data spans ASR, audio question answering, speech translation, audio-text interleaving, TTS, interleaved TTS, and pure audio.Interleaved data alternates text and audio at punctuation boundaries, while pure audio supports independent audio processing.
  • Text data: The cleaned corpus contains 150.7 million pure-text entries after redundancy removal and noise filtering.
  • Cross-modal data: Cross-modal interaction data is synthesized in image-audio-text and video-audio-text formats by converting selected text segments into audio.The source image-text data includes caption and interleaved data.

3.2 Model Architecture

Baichuan-Omni-1.5 unifies visual and audio branches with a pretrained LLM to accept multimodal inputs and generate text or speech. Its audio branch tokenizes speech and decodes audio tokens into waveforms.

  • Unified Architecture: The unified architecture combines visual and audio branches with a pretrained LLM backbone for multimodal input and text or audio output.The visual branch processes images and videos into visual tokens, while the audio branch supports end-to-end speech input and output.
  • Audio Branch: The audio tokenizer converts audio signals into discrete tokens that are supplied to both the LLM and audio decoder.It uses Whisper Large Encoder features, residual convolutional downsampling, and an 8-layer residual vector quantizer at 12.5 Hz.
  • Audio Branch: The audio decoder uses flow matching to generate target Mel spectrograms from audio tokens, followed by HiFi-GAN waveform synthesis.The decoder is trained on 24 kHz audio data.

3.3 Omni-Modal Training Strategies

Baichuan-Omni-1.5 progressively integrates vision, audio, and cross-modal interaction through staged pretraining and supervised fine-tuning. The strategy introduces audio after visual alignment and uses modality-specific updates to reduce conflicts and preserve existing capabilities.

  • Image-Text Pretrain: 300 billion image-text samples are used to extend the LLM’s ability to process and understand visual input.Training first aligns image representations with text, then jointly trains the visual encoder, LLM, and projector.
  • Image-Audio-Text Pretrain: 887k hours of speech-text data extend the visually pretrained LLM to understand audio end to end with an audio tokenizer, embedding layer, and independent audio head.The audio LLM alternates aligned text and audio tokens, using a special token to switch modalities.
  • Image-Audio-Text Pretrain: Interleaving audio and text data, together with two-stage training, is used to mitigate speech-text conflicts while preserving textual knowledge.The first stage updates only the audio embedding layer and audio head; the second trains most parameters while excluding the visual encoder and audio tokenizer.
  • Training Pipeline: The training pipeline incrementally incorporates vision and audio into the LLM while addressing modality conflicts.It proceeds from image-text pretraining to audio integration, cross-modal interaction training, and omni-modal supervised fine-tuning.
  • Cross-Modal Interaction: Cross-modal interaction training uses image-audio-text and video-audio-text data and extends the maximum sequence length to 64k for long streams.Video frames are sampled at 1 frame per second, with at most 32 frames per video.

3.4 Multimodal Supervised Fine-Tuning

The omni-modal supervised fine-tuning phase improves instruction following across text, audio, images, and video using approximately 17 million multimodal data pairs. Its datasets cover broad visual, medical, video, and audio capabilities, with synthesized speech quality checked by ASR.

  • SFT Overview: Approximately 17 million data pairs support omni-modal supervised fine-tuning across text, audio, image-text, video-text, and image-audio tasks.The data combine open-source, synthetic, and in-house annotated sources.
  • Image Data: Image SFT covers natural scenes, structured documents, charts, medical imagery, multiple languages, and both single-image and multi-image tasks.The dataset includes real photographs and synthetically generated visuals.
  • Image Data: Image task categories include general question answering, OCR, graphical reasoning, mathematics, spatiotemporal reasoning, captioning, and medical tasks.The categories are organized by task-specific competencies and draw on numerous public datasets.
  • Image Data: Medical image data combine PubMed references with specialized dermatology, pathology, and ophthalmology datasets.Examples include HAM10000, PathVQA, and AIROGS.
  • Video Data: Video SFT data cover general video understanding, action recognition, and temporal understanding for complex real-world scenarios.The collected video data are systematically classified by task type.
  • Audio Data: Audio instructions are synthesized from textual instructions using 10,000 voice tones, then validated by comparing ASR outputs with the original text.Errors are recycled into TTS or ASR training datasets to create challenging conversational examples.

4 Experiment

Experiments evaluate Baichuan-Omni-1.5 across text, image, video, audio, medical, and omni-modal benchmarks against proprietary and open-source baselines. The model shows strong performance across modalities, including leading results in video, audio, and medical evaluations.

  • Evaluation setup: Baichuan-Omni-1.5 is evaluated against proprietary, open-source language, vision-language, and omni-modal models across six benchmark categories.The evaluation covers text, image, video, audio, medical, and omni-modal tasks.
  • Pure language tasks: 72.2% on MMLU exceeds Llama3-Instruct’s 67.1%, while Baichuan-Omni-1.5 also surpasses MiniCPM-o 2.6 on CMMLU, C-Eval, and AGIEval.The results indicate that multimodal training did not substantially degrade pure-text understanding.
  • Video understanding tasks: 63.8% average on four general video VQA benchmarks exceeds VITA-1.5’s 56.3% and MiniCPM-o 2.6’s 59.8%.Baichuan-Omni-1.5 also achieves state-of-the-art performance on ActivityNet-QA and MSVD-QA among the compared open-source and omni-modal models.
  • Audio understanding tasks: 50 on Reasoning QA and 7.79 on AlpacaEval are achieved in speech-to-text evaluation, while speech-to-speech results exceed GLM-4-Voice by 14.4 and 2.05.The audio evaluation reports both speech-to-text and speech-to-speech settings.
  • Medical tasks: 83.8% on OpenMM-Medical using a 7B LLM surpasses MiniCPM-o 2.6’s 73.6%, while Baichuan-Omni-1.5 also leads GMAI-MMBench.OpenMM-Medical contains 88,996 images sourced from 42 publicly available medical image datasets.

5 Conclusion

Baichuan-Omni-1.5 combines multimodal understanding, parallel omni-modal processing, dual-modal outputs, medical performance, and controllable audio generation. The authors nevertheless identify substantial room to improve the foundational capabilities of individual modalities.

  • Conclusion: Baichuan-Omni-1.5 uses high-quality multimodal data and multistage omni-modal pre-training and fine-tuning to process video, image, text, and audio.The reported features include robust pure-text and multimodal understanding.
  • Conclusion: The model supports end-to-end parallel processing of omni-modal inputs with text and audio outputs, alongside medical performance and controllable audio generation.These are identified as key features of Baichuan-Omni-1.5.
  • Limitations and future work: Substantial improvement remains needed in text understanding, longer-video comprehension, and audio understanding and generation of environmental sounds.The authors specifically mention flowing water, bird songs, and collision noises as future audio targets.

6 Contributors

The listed contributors are Jianhua Xu, Haoze Sun, and Mingan Lin.

  • Contributors: Jianhua Xu is listed as a contributor.
  • Contributors: Haoze Sun is listed as a contributor.
  • Contributors: Mingan Lin is listed as a contributor.
Loading 2501.15368v1…