PaperlayerView exact revision on arXiv

Source-linked AI summary

Mini-Omni2: Towards Open-source GPT-4o with Vision, Speech and Duplex Capabilities

Zhifei Xie, Changqiao Wu

arXiv:2410.11190v3eess.AScs.AIcs.CVcs.LGcs.SD

TL;DR

Open-source models often cover only some GPT-4o capabilities, while integrating modalities into end-to-end understanding and output remains challenging. Mini-Omni2 combines pretrained visual and audio encoders with a three-phase training process and command-based interruption. The authors report consistency with the original model on basic image-captioning and speech-recognition tasks, while noting that broader text and vision experiments remain forthcoming.

  • Problem

    Open-source approaches often address only selected GPT-4o capabilities, while end-to-end multimodal understanding and output remain challenging amid multimodal data, architecture, and training complexity.

    Source: §1 · p. 2

  • Method

    Mini-Omni2 combines pretrained visual and audio encoders with a three-phase modality-expansion and alignment process, then trains multimodal output and command-based audio interruption.

    Source: §1 · pp. 2–3

  • Results

    The authors present Mini-Omni2 as a unified text, speech, and vision model with streaming audio output and duplex interaction; basic image-captioning and speech-recognition performance is consistent with the original model.

    Source: §1 · p. 3, §6 · p. 10

  • Takeaways & Limitations

    The authors plan to open-source the models and datasets, positioning Mini-Omni2 as a research resource.

    Source: §6 · p. 10

    Evaluation is limited to initial data results, application cases, and basic capability assessments; more comprehensive text and vision experiments remain forthcoming.

    Source: §4 · p. 8

Abstract

from arXiv · show

GPT-4o, an all-encompassing model, represents a milestone in the development of large multi-modal language models. It can understand visual, auditory, and textual modalities, directly output audio, and support flexible duplex interaction. Models from the open-source community often achieve some functionalities of GPT-4o, such as visual understanding and voice chat. Nevertheless, training a unified model that incorporates all modalities is challenging due to the complexities of multi-modal data, intricate model architectures, and training processes. In this paper, we introduce Mini-Omni2, a visual-audio assistant capable of providing real-time, end-to-end voice responses to visoin and audio queries. By integrating pretrained visual and auditory encoders, Mini-Omni2 maintains performance in individual modalities. We propose a three-stage training process to align modalities, allowing the language model to handle multi-modal inputs and outputs after training on a limited dataset. For interaction, we introduce a command-based interruption mechanism, enabling more flexible interaction with users. To the best of our knowledge, Mini-Omni2 is one of the closest reproductions of GPT-4o, which have similar form of functionality, and we hope it can offer valuable insights for subsequent research.

1 Introduction

Open-source systems often address only parts of GPT-4o’s multimodal interaction, while end-to-end multimodal understanding and output remain challenging. Mini-Omni2 combines visual, audio, and text inputs with streaming speech output and command-based interruption, using a three-phase training process.

1 Introduction

  • Mini-Omni2 uses pretrained visual and speech encoders with a language model, concatenating encoder features and text embeddings as input and generating text and audio in parallel.The paper identifies CLIP as the visual encoder and Whisper’s encoder as the audio feature extractor; it says the model does not use a token-in-token-out paradigm.Source: §1 · p. 2
  • Initial evaluation reports consistency with the original model on basic image-captioning and speech-recognition tasks, alongside response-speed tests and case studies.The introduction describes these as initial assessments of multimodal interaction capabilities.Source: §1 · p. 3
  • Mini-Omni2 is presented as an open-source model spanning vision, speech, text, and auditory interruption, with end-to-end functionality similar to GPT-4o.The authors qualify the comparison as their assessment of similarity.Source: §1 · p. 3
  • The three-phase pipeline first aligns responses to multimodal inputs, then extends model outputs to speech using delayed parallel generation for real-time output.The phases cover modality expansion, alignment, and joint training, using speech-recognition and image-caption data before multimodal question answering and audio-output training.Source: §1 · pp. 2–3
  • The command-based interruption method uses streaming tokens and constructed training data so semantic cues can control the audio output stream.The model encodes incoming audio in real time with SNAC; demonstration data uses “stop omni” and frame-level irq and n-irq tokens.Source: §1 · p. 3

2 Related Work

Related work covers vision-language architectures, audio language modeling, and multimodal voice interaction. Mini-Omni2 continues prior work on parallel speech-text output while aiming to combine multimodal input with duplex interaction.

Large Vision Language Models

  • Vision-language models commonly combine a vision encoder, adapter, and language model; this paper uses a classical approach similar to Llava [Liu et al., 2024].The section also identifies CLIP [Radford et al., 2021] as the vision encoder used in this work.Source: §2 · pp. 3–4

Audio Language Modeling

Multi-modal Interaction Model

  • Prior interaction models include parallel text-audio generation and duplex systems that take speaking and listening signals as input; this work continues Mini-Omni toward multimodal input and low-latency parallel speech-text output.Mini-Omni [Xie and Wu, 2024] introduced parallel text and audio generation; LSLM [Ma et al., 2024] and Moshi [Défossez et al., 2024] explored duplex interaction.Source: §2 · p. 4

3 Mini-Omni2

Mini-Omni2 combines visual and audio encoders with a language model to handle multimodal inputs and text-audio outputs. Its modeling and three-stage training approach culminate in command-based interruption for duplex interaction.

3.1 Architecture

  • Mini-Omni2 uses CLIP’s ViT-B/32 visual encoder and Whisper-small audio encoder, with Qwen2-0.5B as its foundational language model; a single-layer LlamaMLP serves as the vision adapter.For multi-layer codebook generation, it adds 7 × 4160 sub-LM-heads, resulting in a vocabulary size of 181,120.Source: §3.1 · pp. 4, 6

3.2 Multimodal Languague Modeling

  • Inputs can mix one to three modalities, while the model represents text, discrete speech tokens, and visual features within a unified autoregressive formulation.For visual-audio or visual-text input, visual features precede Whisper features and a response token; single-modality inputs default to image captioning, speech-to-text QA, or text-to-text QA.Source: §3.2 · p. 6
  • Text-Instruct Delay Parallel Decoding generates text and speech tokens simultaneously for real-time output, and text-only responses are used to transfer text-based capabilities to audio tasks.The model generates eight tokens per step, including text, with a one-step delay between layers; the approach supports eight multimodal tasks.Source: §3.2 · pp. 6–7

3.3 Training Strategies

  • Training proceeds through encoder adaptation, modality alignment using text responses, and post-training that adds audio responses and interruption tasks.Stage 1 trains only the connecting linear layer; Stage 2 freezes adapters while training the language model, and Stage 3 revisits earlier tasks with audio outputs.Source: §3.3 · pp. 7–8

3.4 Duplex Interaction

  • The command-based duplex task trains the model to stop speaking when it detects “Stop Omni,” using irq and n-irq tokens to signal whether the user intends to interrupt.The command phrases are mixed with speech and other noise; at inference, irq stops generation so the model can listen to a new question.Source: §3.4 · p. 8

4 Data and Evaluation

The section describes Mini-Omni2’s training data and construction, then reports initial capability assessments; fuller text and vision experiments are left for a later update.

4 Data and Evaluation

  • The evaluation presents initial results and practical cases, with more comprehensive text and vision experiments slated for a later update.The current section showcases only a few application cases and basic capability assessments.Source: §4 · p. 8

4.1 Datasets

  • Training draws on 1.5 million Open-Orca question-answer pairs, 400,000 ALLaVA-4V image samples, speech datasets, synthetic spoken QA, and VoiceAssistant-400K.Speech recognition data includes LibriTTS, VCTK, and Multilingual LibriSpeech; spoken QA derives from Moss-002-sft.Source: §4.1 · p. 8

4.2 Training Parameters

  • Training used eight A100 GPUs, the Qwen2-0.5B base model, a global batch size of 192, and one epoch per stage.Learning rates ranged from 2e-5 to 1e-3 for adapter training and 2e-6 to 2e-4 for language-model training; final fine-tuning used 2e-6 to 2e-5.Source: §4.2 · pp. 8–9

4.3 Data Construction

  • Spoken-dialogue data uses randomly selected voice prompts and synthetic speech, while interruption examples insert “Stop Omni” and label the following tail as irq.The interruption data first stream-encodes and decodes noise; material before the tail is labeled n-irq.Source: §4.3 · p. 9

4.4 Experimental Results

  • Mini-Omni2’s ASR accuracy declines slightly after adding vision compared with Mini-Omni, yet it outperforms Whisper on LibriSpeech-other.The authors suggest the decline may reflect a reduced proportion of speech data.Source: Table 2 · p. 9, §4.4 · pp. 9–10

4.5 Case Study

  • The case-study subsection presents Mini-Omni2 use cases without reporting further evaluation outcomes in the supplied text.Figure 6 is identified as showing use cases.Source: §4.5 · p. 10, Figure 6 · p. 10

5 Limitations

The authors identify model and data scaling, audio-output diversity, and semantic interruption as areas for further improvement.

5 Limitations

  • The authors propose exploring larger model and data scales, greater audio style diversity, and a richer semantic interruption mechanism.Mini-Omni2 targets small models with limited resources; the authors believe more data and compute can greatly enhance capabilities.Source: §5 · p. 10

6 Conclusion

Mini-Omni2 combines text, speech, vision, streaming audio output, and duplex interaction through pretrained encoders and a three-phase training process.

6 Conclusion

  • Mini-Omni2 is presented as a unified model for text, speech, vision, end-to-end streaming audio output, and duplex interaction.The authors describe it as one of the closest functional reproductions of GPT-4o, to their knowledge.Source: §6 · p. 10
  • Pretrained vision and speech encoders are aligned with the language model to extend its modalities.Source: §6 · p. 10
  • The proposed training process uses three phases for modality alignment and expansion, alongside a method and data construction for duplex interaction.The authors hope the work can serve as a reference for future research.Source: §6 · p. 10
Loading 2410.11190v3…