Baichuan-Omni-1.5 Technical Report

Motivation

• Open-source MLLMs often focus on vision and text; separate ASR/TTS modules add latency and complexity, while end-to-end approaches can suffer modality conflicts that hurt omni-modal performance.The passage cites separate-module approaches [44] and end-to-end approaches VITA-1.5 [45] and Mini-Omni2 [159].Source: §1 · p. 2

Architecture

• During audio generation, the decoder alternates text and audio token prediction, then an Audio Decoder converts audio tokens into the final audio.The architecture accepts pure text or audio, as well as video or image combined with text or audio.Source: Figure 2 · p. 2

Omni-modal Interaction

• Baichuan-Omni-1.5 processes text, image, audio, and video inputs and delivers text and speech outputs with high-quality cross-modal interaction without compromising any modality.Source: §1 · p. 2

Vision-Language Capability

• 73.3 average across ten image-understanding benchmarks surpasses GPT-4o-mini by an average of 6 points.Source: §1 · p. 2

Unified Speech Capabilities

• Its 8-layer RVQ Baichuan-Audio-Tokenizer runs at 12.5 Hz, balancing semantic and acoustic information for controllable real-time Chinese and English conversations.The authors also open-sourced OpenAudioBench to evaluate end-to-end audio capabilities.Source: §1 · p. 3

Medical Image Understanding

• 83.8% on OpenMM-Medical with a 7B LLM surpasses Qwen2-VL-72B’s 80.7%, and the model achieves state-of-the-art performance on GMAI-MMBench and OpenMM-Medical.OpenMM-Medical integrates existing datasets.Source: §1 · p. 3

2.1 Multimodal Large Language Models (MLLMs)

• Open-source visual and audio MLLMs advance cross-modal understanding and speech tasks, but most still lag proprietary models such as GPT-4o in comprehensive multimodal interaction; Baichuan-Omni-1.5 targets this gap.The report describes its focus as text, audio, and vision perception, processing, understanding, and generation.Source: §2.1 · p. 3

2.2 Omni Models with MLLMs

• Omni models expand multimodal interaction: EMOVA [18], VITA [44], VITA 1.5 [45], Mini-Omni [158], and Mini-Omni2 [159] introduce emotional dialogue, interruption, generation, and real-time voice capabilities.These approaches use non-wake-word interaction, audio interruption, complex-scenario analysis, and command interruption to improve dialogue control and data utilization.Source: §2.2 · p. 3

2.3 Medicine with MLLMs

• Medical MLLMs combine biomedical modalities and image-text data, use few-shot visual question answering, and adapt general-purpose models for medical precision and clinical applicability.Med-PaLM fine-tunes PaLM-E with millions of samples for medical contexts.Source: §2.3 · pp. 3–4

• The section overviews Baichuan-Omni-1.5’s high-quality data.Source: §3 · p. 4
• The section also covers Baichuan-Omni-1.5’s model architecture.Source: §3 · p. 4
• Its overview further addresses the multi-stage multimodal training strategy.Source: §3 · p. 4

3.1 High-Quality Multimodal Pretrain Data

• The pretraining collection spans text, image-text, video-text, audio-text, and interleaved image-audio-text and video-audio-text data.The authors describe the dataset as comprehensive and cross-modal, with examples illustrated in Figure 3.Source: Figure 3 · p. 4, §3.1 · p. 4
• Image training data combines open-source sources with parsed books and papers, specialized OCR and chart data, detailed captions, and synthesized Chinese captions and interleaved data.The pipeline uses sampling techniques to construct different data ratios, and a dedicated caption model generates captions such as OCR hints.Source: §3.1 · pp. 4–5
• Video training covers captioning and question answering, drawing on public datasets and GPT-4o-generated captions for YouTube videos.The sources include ShareGPT4Video, Koala, and WebVid for captions, and ActivityNet-QA, VideoChatGPT-Plus, ShareGemini, and NExTVideo for QA.Source: §3.1 · p. 5
• Audio data covers understanding tasks, including ASR, audio question answering, and translation, as well as TTS, interleaved text-audio, and pure-audio generation data.Aligned interleaved text-audio supports audio-token generation under text supervision, paired data supports speech tasks, and pure audio supports independent audio processing.Source: §3.1 · p. 5
• The text corpus draws on web pages, books, academic papers, and code, with processing to improve diversity and quality and filter redundancy and noise.The resulting corpus contains 150.7 million entries of pure text data.Source: §3.1 · pp. 5–6
• 100B tokens of cross-modal interaction data combine images or videos with audio and text.For image-text data, a random quarter of sentences is converted to audio using 44 voice types; video audio is extracted directly from the source videos.Source: §3.1 · p. 6

3.2 Model Architecture

• The model unifies visual and audio branches with a pretrained LLM backbone, supporting text, audio, and visual input and end-to-end text and audio output.Source: §3.2 · p. 6

3.2.1 The Visual Branch

• The visual branch uses Qwen2-VL’s NaViT encoder for arbitrary-resolution images and videos, then a two-layer MLP projector compresses visual features by 2×2.The resulting visual tokens enter the LLM alongside text tokens.Source: §3.2.1 · p. 6

3.2.2 The Audio Branch

• The audio branch combines Baichuan-Audio-Tokenizer with a flow-matching decoder to convert speech into discrete tokens and audio tokens into waveforms.The decoder is refined for audio quality and perceptual fidelity using a flow-matching model.Source: §3.2.2 · pp. 6–7
• The tokenizer uses an 8-layer RVQ [31] at 12.5 Hz, with Whisper Large Encoder features and multi-objective training [85] [115].Its tokens support Mel reconstruction and transcript prediction; pretrained LLM parameters are fixed for semantic alignment, and interleaved text-audio data improves contextual modeling. The flow-matching U-Net has one down-sampling block, 12 intermediate blocks, and one up-sampling block; it generates target Mel spectrograms from 24 kHz audio for HiFi-GAN waveform conversion [69] [37].Source: §3.2.2 · p. 7

3.3.1 Image-Text Pretrain

• Two-stage image-text pretraining uses 300 billion samples, progressing from projector-only alignment to joint training of the LLM, visual encoder, and projector.The second stage includes interleaved and caption data, including OCR and chart data, alongside pure text comprising 40% of the data to maintain language capabilities.Source: §3.3.1 · p. 7

3.3.2 Image-Audio-Text Pretrain

• Audio pretraining adds 887k hours of speech-text data and interleaves audio and text in two stages to integrate audio while preserving the LLM’s textual knowledge.Audio tokens become embeddings, and the model alternates aligned text and audio tokens; Stage I trains only the audio embedding layer and head, while Stage II trains all parameters except the visual encoder and audio tokenizer, using 20% audio, 40% image, and 40% pure text. The independent audio head has three depth-transformer layers and eight classification heads.Source: §3.3.2 · pp. 7–8

3.3.3 Omni-Modal Pretrain

• Cross-modal pretraining trains all parameters on image-audio-text and video-audio-text data, extending the maximum sequence length to 64k for long voice and video streams.Videos are sampled at 1 frame per second, with at most 32 frames resized to 560×1120 pixels; this stage uses a 4e −6 learning rate. Subsequent supervised fine-tuning uses omni-modal data, first freezing the audio head for understanding, then training the audio head and embedding layer with audio-generation data.Source: §3.3.3 · p. 8, Figure 5 · p. 8

3.4 Multimodal Supervised Fine-Tuning

• Approximately 17 million SFT data pairs span text, audio, image-text, video-text, and image-audio tasks.Table 4 summarizes the categories and quantities of the SFT data.Source: §3.4 · p. 9, Table 4 · p. 9

3.4.1 Image Data

• Image SFT draws on millions of public examples across visual domains, languages, and single- and multi-image tasks, with filtering, answer regeneration, and manual validation.The data mix includes real photographs and synthetic visuals, with proportions allocated to cover competencies.Source: §3.4.1 · p. 9
• Image SFT also covers general image QA, chart reasoning, mathematics, spatial relationships, captioning, and specialized medical imagery.These task groups include CoT-rich math data, synthetic and scientific captioning data, and datasets for dermatology, pathology, and ophthalmology.Source: §3.4.1 · pp. 9–10
• OCR data constitute 26.51% of image data, a proportion tuned through repeated adjustment because it significantly affects overall performance.The OCR collection supports recognizing and understanding image text and responding based on it.Source: §3.4.1 · p. 10

3.4.2 Video Data

• Video SFT comprises 100K high-quality examples for classification, action recognition, and temporal localization, with GPT-4o-assisted classification and balancing by scene, difficulty, accuracy, and quality.Source: §3.4.2 · p. 10

3.4.3 Audio Data

• Audio SFT synthesizes conversational examples from text instructions using 10,000 voice tones, then uses ASR checks to validate audio and route errors into TTS or ASR training.Text responses are segmented at natural pauses, and cases where audio conversion changes meaning or creates ambiguity are reviewed and adjusted.Source: §3.4.3 · p. 10

4.1 Performance in Pure Language Tasks

• 72.2% on MMLU exceeds Llama3-Instruct’s 67.1%, while Baichuan-Omni-1.5 also outperforms MiniCPM-o 2.6 across the reported general and Chinese text benchmarks.The paper attributes its maintained pure-text ability to training-strategy adjustments and balanced multimodal data that retain a proportion of pure-text data; evaluations use zero-shot measurements.Source: Table 6 · p. 11, §4.1 · pp. 11–12

4.2 Performance in Image Understanding Tasks

• Baichuan-Omni-1.5 outperforms VITA-1.5 and MiniCPM-o 2.6 on most image benchmarks, including six of ten benchmarks against MiniCPM-o 2.6.It also surpasses GPT4o on MMBench-EN/CN and OCRBench, and is comparable to Qwen2-VL-7B across image-understanding benchmarks. Evaluations use VLMEvalKit in zero-shot settings.Source: Table 7 · p. 12, §4.2 · p. 12, Table 8 · p. 13

4.3 Performance in Video Understanding Tasks

• 63.8% average across four general video VQA benchmarks exceeds VITA-1.5’s 56.3% and MiniCPM-o 2.6’s 59.8%.Baichuan-Omni-1.5 also achieves state-of-the-art Accuracy and Score on ActivityNet-QA and MSVD-QA across the compared open-source general and omni-modal models, and exceeds GPT-4o-mini’s 62.1% ActivityNet-QA Accuracy.Source: §4.3 · p. 13, Table 9 · p. 14, Table 10 · p. 15

4.4 Performance in Audio Understanding Tasks

• In s→t, Baichuan-Omni-1.5 scores 50 on Reasoning QA and 7.79 on AlpacaEval; in s→s, it leads GLM-4-Voice by 14.4 and 2.05, respectively.The s→s setting evaluates text merged from interleaved text-and-audio output, while s→t evaluates generated text. GPT-4o evaluates answer correctness or scores across the audio benchmarks.Source: §4.4 · p. 14, Table 11 · p. 15

4.5 Performance in Omni Tasks

• 47.9 versus 42.9 and 46.9 versus 37.7 are Baichuan-Omni-1.5’s results with audio transcripts versus original audio in the two OmniBench image-input settings.Across models, transcript inputs score better than original audio; Baichuan-Omni-1.5 beats MiniCPM-o 2.6 in three of four settings, including 42.9 versus 40.5, 37.7 versus 30.8, and 46.9 versus 46.3. OmniBench evaluates four combinations of original or captioned images and original audio or transcripts.Source: §4.5 · pp. 14–15, Table 12 · p. 16

4.6 Performance in Medical Tasks

• 83.8% on OpenMM-Medical and 49.9% on GMAI-MMBench validation are Baichuan-Omni-1.5’s highest reported scores, leading the compared models on both benchmarks.MiniCPM-o 2.6 scores 73.6% on OpenMM-Medical, while GPT4o-mini scores 46.3% on GMAI-MMBench validation. OpenMM-Medical contains 88,996 multiple-choice VQA images from 42 public medical image datasets.Source: Table 13 · p. 16, §4.6 · p. 16

5 Conclusion

• The model delivers strong text and multimodal understanding, parallel omni-modal input processing with text-and-audio outputs, medical performance, and controllable audio generation.The report attributes these capabilities to high-quality multimodal data and multistage omni-modal pre-training and fine-tuning.Source: §5 · p. 17
• The report identifies room to improve text understanding, support longer video-frame understanding, and recognize environmental sounds as well as human voices.Examples of environmental sounds include flowing water, bird songs, and collision noises.Source: §5 · p. 17
• Future research will refine these areas toward more sophisticated, versatile models that comprehend and interact with complex environments, contributing to the broader goal of AGI.The report anticipates that continued advances in these domains will contribute to achieving Artificial General Intelligence.Source: §5 · p. 17