PaperlayerView exact revision on arXiv

Source-linked AI summary

Omni-Interactive Universal Embedder

Wei-Yao Wang, Kazuya Tateishi, Shuyang Cui, Christian Simon, Takashi Shibuya, Shusuke Takahashi, Yuki Mitsufuji

arXiv:2608.27044v1cs.AIcs.CV

TL;DR

Existing embedders mainly support text and image interactions, limiting precise user guidance across multimodal content. OmniUE unifies text, video, and audio representations while accepting textual, visual, and audio interactions, and it consistently outperforms baselines across evaluated scenarios, including a 5.1-point average gain for its 3B model on MMEB-v2-video.

  • Problem

    Existing embedders mainly focus on text and image interactions, while text or visual guidance can be insufficient for precise intent in complex video and audio scenarios.

  • Method

    OmniUE combines multimodal query and interaction streams with visual and audio segmenters and multi-layer token aggregation to produce unified, user-conditioned embeddings.

  • Results

    OmniUE consistently outperforms baselines across four MMEB-v2-video scenarios, with OmniUE-3B surpassing existing 2B/3B models by 5.1 points on average.

  • Takeaways & Limitations

    OmniCHOIR provides a more rigorous and realistic testbed for evaluating multimodal interaction, compositionality, and reasoning in omni-interactive embeddings.

  • Takeaways & Limitations

    OmniUE depends on pretrained models such as Qwen2.5-omni and SAM-Audio because training all modules from scratch is infeasible.

Abstract

from arXiv · show

Multimodal representation learning has been shifting from traditional two-tower architectures to large language model (LLM)-based embedders due to their strong instruction-following capabilities. Despite this progress, existing approaches primarily focus on language and image modalities, which also remain the dominant modalities for user-conditioned interactions in current embedders. In this paper, we propose the first Omni-Interactive Universal Embedder (OmniUE), which not only learns a unified embedding space across text, video, and audio by leveraging intermediate-layer representations from dedicated learnable tokens, but also supports omni-interactive querying, enabling users to provide inputs in the form of text, visual regions of interest, and audio spans. Within OmniUE, visual and audio segmenters process diverse user interactions and integrate them with an omni-LLM to produce user-conditioned any-to-any embeddings via context aggregation. To evaluate OmniUE's omni-interactive capabilities, we introduce OmniCHOIR, benchmarking models for omni-interactive compositional audio retrieval based on the given text, video, and audio as well as unimodal or multimodal interaction prompts. OmniUE consistently surpasses state-of-the-art baselines across diverse modalities, with average improvements of 10.5% on textual-interactive video benchmarks (MMEB-v2-video), 1.1% on audio tasks (MAEB), 83.7% on visual-interactive benchmarks (SCaR), and 24.1% on our omni-interactive OmniCHOIR benchmark. We believe that jointly advancing omni-modal representation learning and omni-interactive querying paves the way toward universal embedders.

1 Introduction

The introduction motivates moving beyond text-dominant multimodal embedders toward omni-interactive systems that capture fine-grained intent across text, visual, and audio inputs. It presents OmniUE and the OmniCHOIR benchmark as complementary solutions for unified embedding and rigorous omni-modal evaluation.

  • Motivation: MLLM-based embedders extend two-tower models by supporting arbitrary modality combinations and instruction-guided embedding extraction.Their progress is attributed to large-scale training corpora, multimodal flexibility, and instruction-following abilities.
  • Motivation: Text remains the dominant interaction medium, but text alone cannot precisely express complex intent and visual prompts can fail for ambiguous video-audio scenes.These limitations motivate interaction through video and audio, including cases with off-screen sources, overlapping sounds, or ambiguous audiovisual correspondences.
  • OmniCHOIR: OmniCHOIR is introduced as a text-video-audio-to-audio retrieval benchmark for evaluating textual, visual, and audio interactions across unimodal and multimodal settings.It targets fine-grained compositional and reasoning challenges involving distractor sounds, shared backgrounds, and sounds from related categories, which existing benchmarks miss.
  • OmniUE: OmniUE is introduced as the first omni-interactive omni-modal universal embedder, combining text, visual, and audio prompts with arbitrary multimodal inputs to produce unified embeddings.Its design uses visual and audio segmenters together with multilayer context aggregation; the query and interactive-prompt streams encode inputs and user-specified signals.

2 Related Works

Prior work has established instruction-following multimodal embedders and interactive evaluation, while audio-language alignment and omni-interactive benchmarking address remaining modality and interaction gaps. OmniCHOIR targets this gap with text-video-audio-to-audio retrieval under unimodal and multimodal interactions.

  • LLM-Based Multimodal Embedders: MLLM-based embedders [26] [27] handle diverse user queries effectively and outperform traditional dual-encoder paradigms [11] [21].
  • LLM-Based Multimodal Embedders: Audio-language representation learning includes CLAP [16], which aligns audio and text in a shared embedding space.
  • Interactive Embedding Benchmarks: Interactive evaluation has shifted from unimodal and bimodal retrieval benchmarks toward large-scale, instruction-driven multimodal benchmarks assessing diverse scenarios.
  • Interactive Embedding Benchmarks: OmniCHOIR introduces an omni-interactive text-video-audio-to-audio retrieval benchmark covering unimodal and multimodal interactions.

3 OmniUE

OmniUE unifies text, video, and audio inputs with visual and audio interaction prompts in a shared embedding framework. It combines modality-specific segmenters, learnable tokens, and adaptive multi-layer context aggregation to produce user-conditioned embeddings efficiently.

  • 3 OmniUE: OmniUE supports text-only, video-only, audio-only, joint multimodal inputs, and visual, audio, or textual interaction prompts through user-conditioned hidden representations.Visual and audio queries and interaction prompts are processed by dedicated segmenters, encoders, and connectors before integration with the omni-LLM.
  • Interactive prompts: Visual and audio segmenter streams capture fine-grained entity-level and temporal-span information that holistic modality representations cannot represent.The streams support points, bounding boxes, masks, text, and temporal spans as interaction inputs.
  • Interactive prompts: Pretrained SAM-3 and SAM-Audio [49] encode visual regions and separated sounds into interaction-aware features within a unified interaction framework.SAM-3 segments entities from user-specified regions of interest, while SAM-Audio separates sounds from mixed audio conditioned on optional video, text, or audio spans.
  • Interaction connectors: OmniUE compresses visual and audio feature maps with Conv2D and Conv1D segmenter connectors, then uses self-attention and MLP layers to produce LLM-compatible interaction embeddings.The connectors reduce spatial resolution for visual features and project audio features into the unified sequence space, avoiding excessively long flattened sequences.
  • Context aggregation: OmniUE concatenates holistic, user-conditioned, and learnable-token representations, extracts token states every four layers, and adaptively weights layer-token features for the final embedding.Normalized importance weights emphasize informative levels, while mean pooling and softmax aggregation combine local and global representations efficiently.

4 OmniCHOIR: Omni-Interactive TVA2A Retrieval Benchmark

OmniCHOIR is introduced as an omni-interactive text-video-audio-to-audio retrieval benchmark that evaluates matching target sounds and background context from multimodal interaction prompts. It builds challenging, held-out mixtures from SAM-Audio-Bench to assess zero-shot generalization beyond existing text- or text-image-focused benchmarks.

  • Motivation: OmniCHOIR evaluates omni-interactive text-video-audio-to-audio retrieval, addressing benchmarks that mainly assess textual interaction or limited text-image interaction.The benchmark focuses on TVA2A retrieval because audio mixtures superpose multiple sources.
  • Problem Definition: Given video, mixed audio, and any combination of audio, visual, and textual prompts, models retrieve the most relevant audio from 16 candidates.Textual instructions are always included, and relevance is computed by cosine similarity between the query and candidate embeddings.
  • Dataset Construction: OmniCHOIR is constructed from 819 ten-second SAM-Audio-Bench samples containing 24-fps video, audio, multimodal prompts, and human-annotated visual masklets.The source benchmark spans speech cleaning, speaker separation, music removal, 37 instrument classes, and environmental sounds across six public datasets.
  • Dataset Construction: All source samples are held out from OmniUE training, enabling evaluation of zero-shot generalization on multimodal audio retrieval.The video and audio data come from AudioSet, VGGSound [7], MUSIC, MUSIC-AVQA, AVSpeech [17], and CondensedMovies [4].
  • Dataset Construction: After sound separation and manual validation, the benchmark retains 479 instances and generates one ground-truth plus five negative background categories using Qwen3-Omni-30B [63] and ESC-50.A dynamic mixer combines target and background sounds at randomized intervals, varying background volume from -15 dB to -5 dB relative to the target.

5 Experiments

Experiments evaluate OmniUE across textual-interactive video and audio, visual-interactive, and omni-interactive retrieval benchmarks. OmniUE consistently outperforms baselines, with especially large gains on video and SCaR, while OmniCHOIR stress-tests seven uni- and multimodal interaction conditions.

  • Implementation Details: OmniUE-3B and OmniUE-7B use Qwen2.5-Omni backbones with sam-audio-base and SAM-3 segmenters, LoRA adaptation, trainable prompt connectors, and approximately 3.5M multimodal training samples.The LLM LoRA rank is 32 with alpha 64; audio and visual prompt connectors are trained from scratch.
  • Video benchmarks (MMEB-v2-video): OmniUE consistently outperforms all baselines across four MMEB-v2-video meta-task scenarios, improving by 5.1 points over existing 2B/3B models and 4.3 points over 7B baselines.MMEB-v2-video covers 18 textual-interactive video tasks spanning classification, question answering, retrieval, and moment retrieval.
  • Audio benchmarks (MAEB): OmniUE outperforms both two-tower and LLM-based embedders across MAEB’s 30 textual-interactive audio tasks and achieves the best performance in 7 of 8 meta-task categories against LCO-Emb-7B.Compared with LCO-Emb-7B, gains include +15.1 on Audio Retrieval and +9.5 on Re-ranking; OmniUE-3B performs on par with LCO-Emb-3B on average.
  • Visual-interactive benchmarks (SCaR): OmniUE consistently outperforms visual-interactive baselines across all five SCaR datasets, improving by 20.9 and 27.4 points over the strongest 3B and 7B baselines, respectively.SCaR evaluates text-image-to-text retrieval using an image, text instruction, and bounding box.
  • Omni-interactive benchmark (OmniCHOIR): OmniCHOIR stress-tests OmniUE under seven uni- and multimodal interaction conditions spanning text, spans, masks, and their combinations.The benchmark evaluates text, span, mask, text+span, text+mask, span+mask, and text+span+mask conditions, using Recall@1.

6 Conclusion

OmniUE is presented as the first omni-interactive universal embedder, jointly encoding omni-modal inputs and interactions as holistic context and fine-grained guidance. It extends interaction beyond text and images by using visual and audio segmenters with multi-token layer aggregation to produce user-conditioned query embeddings.

  • 6 Conclusion: OmniUE jointly encodes omni-modal inputs and omni-modal interactions as holistic context and fine-grained guidance.
  • 6 Conclusion: Unlike existing approaches limited to text and image interactions, OmniUE supports broader omni-modal interaction.
  • 6 Conclusion: Using SAM-3 and SAM-Audio segmenters with multi-token layer aggregation, OmniUE produces user-conditioned query embeddings.

A Limitation and Broader Impacts

OmniUE depends on pretrained components because training all modules from scratch is infeasible, while its omni-interactive embeddings support interaction-aware modeling and similarity-based evaluation. Unlike holistic similarity methods, it jointly models multimodal inputs with user-provided interactions.

  • Limitations: OmniUE’s main limitation is dependence on pretrained models such as Qwen2.5-omni and SAM-Audio because training all modules from scratch is infeasible.The paper also cites Xiao et al. on a positive relationship between embedding and generative performance, suggesting future gains from stronger generative models.
  • Broader Impacts: OmniUE can enable interaction-aware models [13] and serve as a similarity-based evaluation model, extending approaches such as ImageBind [20].Unlike conventional holistic similarity, it jointly models multimodal inputs and user-provided interactions to support interaction-aware similarity.

C Implementation Details · C.1 Parameter Setting

OmniUE-3B and OmniUE-7B use the same temperature and learning rate, train on 8×H100 80GB hardware, and retain default configurations from their component models. The iterative sampler follows SAM-Audio’s ODE solver, with 8 steps matching 16-step performance while reducing GPU memory use.

  • C.1 Parameter Setting: OmniUE-3B and OmniUE-7B are trained with τ=0.02 and a learning rate of 2e-5.
  • C.1 Parameter Setting: Training uses 8×H100 80GB GPUs for approximately 177 hours for OmniUE-3B and 195 hours for OmniUE-7B.
  • C.1 Parameter Setting: All remaining configurations follow the default settings of Qwen2.5-omni, SAM-Audio, and SAM-3.
  • C.1 Parameter Setting: The iterative sampling procedure uses the ODE solver from the original SAM-Audio paper.
  • C.1 Parameter Setting: 8 sampling steps perform on par with SAM-Audio’s original 16-step configuration.
  • C.1 Parameter Setting: Using 8 rather than 16 sampling steps reduces GPU memory use without degrading observed performance.

C.2 Training Datasets … D.2 Prompt Template for Video-Audio Captioning

OmniUE is trained on approximately 3.5M multimodal samples, including synthesized captions that jointly describe video and audio. OmniCHOIR spans diverse domains, while its captioning prompt enforces scene-grounded gold and distractor sounds in valid JSON.

  • C.2 Training Datasets: 3.5M multimodal samples comprise OmniUE’s training data, with Table 6 organizing sources by modality.The dataset overview marks synthesized text captions with an asterisk.
  • C.2 Training Datasets: Qwen3-Omni-30B-A3B-Instruct [63] synthesizes captions from paired video and audio, jointly capturing visual and auditory information.For VGGSound, captions are taken from AudioSetCaps.
  • D OmniCHOIR Details; D.1 OmniCHOIR Examples: OmniCHOIR examples span diverse domains and are provided in the supplementary index.html.The examples are presented to improve readability.
  • D.2 Prompt Template for Video-Audio Captioning: The video-audio captioning prompt asks an expert scene analyst to infer one visible-scene background sound and generate semantically different plausible distractors.It requires strict adherence to visible or strongly implied entities and actions, using the original caption as the primary sound.
  • D.2 Prompt Template for Video-Audio Captioning: The prompt requires the model to return only valid JSON without extra text.The response schema contains a gold_background field and a negative_backgrounds list.
  • D.2 Prompt Template for Video-Audio Captioning: The negative_backgrounds output specifies five exact categories from the provided list.The template repeats the exact-category placeholder five times.

E Additional Experiments · E.1 Ablation and Parameter Study

Ablations show that OmniUE’s interaction streams, multi-token inputs, and layer aggregation provide complementary gains, while parameter studies identify robust design choices across modality-specific benchmarks. Additional module comparisons further favor the selected segmenters, connectors, and aggregation strategy.

  • E.1 Ablation and Parameter Study: The studies use OmniUE-3B with batch size 512 and 3000 training steps, evaluating OmniCHOIR with text conditions across 15 variant designs and choices.Each training configuration takes around 48 hours.
  • E.1 Ablation and Parameter Study: Removing any proposed component consistently degrades performance across all interaction benchmarks, with visual and audio streams especially important for SCaR and OmniCHOIR.The interaction streams provide targeted gains in their corresponding visual and audio scenarios.
  • E.1 Ablation and Parameter Study: Multi-token inputs and layer aggregation yield consistent gains across all benchmarks, while interaction streams provide complementary modality-specific improvements.The ablation isolates these architectural refinements from the omni-interaction formulation.
  • E.1 Ablation and Parameter Study: The best overall parameter setting aggregates hidden states every l = 4 layers, while sparse aggregation such as l = 8 performs worse.More intermediate hidden states generally improve performance, but l = 4 balances information richness and redundancy.
  • E.1 Ablation and Parameter Study: Using K = 1 learnable token significantly degrades performance, whereas K = 2 consistently improves results for heterogeneous multimodal interactions.Two and eight attention heads perform comparably; two heads are selected for the performance–efficiency trade-off.
  • E.1 Ablation and Parameter Study: Varying visual-interactive tokens primarily affects visual benchmarks, whereas varying audio-interactive tokens predominantly affects audio benchmarks.LoRA-rank, visual-token, and audio-token ablations further validate the selected design choices.
  • E.1 Ablation and Parameter Study: Among module alternatives, base SAM-Audio achieves the best overall performance among available sizes, while SAM-3 outperforms SAM2.1 as a visual segmenter.The comparison also evaluates connector self-attention and aggregation strategies.

E.2 Analysis on Latency and Memory · E.3 Robustness of OmniUE to Interaction Prompts · E.4 Additional Modality Directions with Textual Interactions

OmniUE adds modest inference overhead while improving text-conditioned performance, remains testable under imperfect interaction prompts, and outperforms applicable state-of-the-art methods on additional cross-modal retrieval benchmarks.

  • E.2 Analysis on Latency and Memory: 14.3% performance gain under text-conditioned settings accompanies only marginal inference overhead for OmniUE-7B’s omni-interactive capabilities.The analysis reports average end-to-end latency and GPU memory across seven unimodal and multimodal query modes on OmniCHOIR.
  • E.2 Analysis on Latency and Memory: The latency and memory study evaluates OmniUE-7B and LCO-Emb-7B across all seven operational modes on OmniCHOIR.Measurements cover average end-to-end inference latency in seconds and GPU memory consumption in gigabytes.
  • E.2 Analysis on Latency and Memory: Replacing segmentation masks with bounding boxes slightly reduces performance but significantly lowers latency, enabling a throughput–fidelity trade-off.This substitution is presented as a practical inference recipe for real-world applications.
  • E.3 Robustness of OmniUE to Interaction Prompts: Robustness experiments test imperfect interaction inputs using randomly sampled visual masks, randomly sampled audio spans, and complete removal of the text prompt.These conditions probe OmniUE’s sensitivity to noisy or missing interaction prompts.
  • E.3 Robustness of OmniUE to Interaction Prompts: Table 11 reports OmniUE’s performance under noisy masks, noisy spans, and altered text interaction prompts.The table organizes the robustness analysis around the three imperfect-input conditions.
  • E.4 Additional Modality Directions with Textual Interactions: OmniUE-7B outperforms applicable state-of-the-art methods on V2A/A2V retrieval on VGGSound and VA2T/T2VA retrieval on VALOR-32K.Evaluation uses Recall@1, comparing against WAVE-7B on VGGSound and VALOR on VALOR-32K.
  • E.4 Additional Modality Directions with Textual Interactions: Additional modality-direction experiments cover V2A, A2V, VA2T, and T2VA retrieval using standard Recall@1 evaluation.V2A and A2V are evaluated on VGGSound, while VA2T and T2VA are evaluated on VALOR-32K.
Loading 2608.27044v1…