Source-linked AI summary
Syn-Omni: Structured Specialization and Progressive Collaboration for Omnimodal Embeddings
Youngtaek Oh, Qiyu Wu, Hiromi Wakaki, Junmo Kim, Yuki Mitsufuji
TL;DR
Existing omnimodal embedding methods often use fully shared adaptation, limiting separation between universal and modality-specific representations. Syn-Omni combines shared and modality-specialized LoRA paths with progressively enabled cross-modal routing, and outperforms baselines across 81 diverse tasks.
Problem
Fully shared adaptation provides limited structural separation between modality-shared and modality-specific representations.
Source: §1 · p. 1Method
Syn-Omni combines shared and modality-specific LoRA paths with routing that moves experts from modality priors toward cross-modal interaction.
Source: §1 · p. 2, §3.3 · p. 4Results
Takeaways & Limitations
The results demonstrate the effectiveness of structured specialization and progressive cross-modal collaboration for unified omnimodal representation learning.
Source: §1 · p. 2Dynamic expert routing adds 17–18% latency and less than 1% GPU memory, with the cost most visible in latency-sensitive settings.
Source: Limitations · p. 9
Abstract
from arXiv · showhide
Omnimodal embeddings naturally involve both shared representations and modality-specific features across heterogeneous inputs. However, existing omnimodal embedding methods often rely on a single shared parameter space over mixed-modality data, limiting structural separation between universal and modality-specific representations. To address this, we propose Syn-Omni, a unified framework for structured omnimodal adaptation with modality specialization and controlled cross-modal collaboration. Specifically, we introduce Orthogonal Modality-Expert LoRA (OME-LoRA), which decomposes adaptation into a shared LoRA path for universal semantics and modality-expert LoRA paths for modality-aware specialization. Furthermore, Progressive Synergy Routing (PSR) enables experts to first establish modality-specific priors, then gradually interact with other modality-experts for cross-modal synergy. Evaluated across 81 diverse tasks spanning image, video, audio, and audiovisual modalities, Syn-Omni consistently outperforms omnimodal baselines, demonstrating the effectiveness of structured specialization and cross-modal progressive collaboration.
1 Introduction
Existing omnimodal embedding methods often rely on fully shared adaptation, which offers limited structural separation between shared and modality-specific representations. Syn-Omni combines shared and modality-specialized paths with progressive cross-modal collaboration and outperforms unified baselines across diverse settings.
1 Introduction
- Syn-Omni consistently improves Uni-Omni across modality scenarios, with gains on composed multimodal tasks such as audiovisual understanding.Figure 1 compares modality-specific models, Uni-Omni, and Syn-Omni.Source: Figure 1 · p. 1, §1 · p. 2
- Fully shared adaptation offers limited separation of modality-specific characteristics, while isolated modality models restrict cross-modal interaction and generalization.The introduction identifies a need to support unified semantics, modality-aware specialization, and collaboration together.Source: §1 · pp. 1–2
- Syn-Omni pairs shared and modality-specific LoRA paths with Progressive Synergy Routing, which lets experts establish modality priors before gradually interacting.This design aims to combine universal semantics with modality-aware characteristics.Source: §1 · p. 2
- Across 81 image, video, audio, and audiovisual tasks, Syn-Omni outperforms omnimodal embedding baselines with more balanced performance across modality settings.The tasks include classification, retrieval, and QA.Source: §1 · p. 2
2 Related Work
Related work repurposes multimodal language models as contrastively fine-tuned embedding models and explores structured adaptation. Syn-Omni adapts this separation to modality experts and pairs it with progressive collaboration.
2 Related Work
- Contrastive fine-tuning with LoRA repurposed multimodal language models as embedding models, extending from text-image tasks toward video, audio-text, and any-to-any retrieval.The cited work also includes explicit calibration strategies for cross-modal alignment.Source: §2 · p. 2
- MoELoRA (Luo et al., 2024) and MoDULA (Ma et al., 2024b) instantiate task-level LoRA experts, while another line imposes orthogonality within adapter subspaces.Related work also reports that modality-specific experts can require explicit control for stable training.Source: §2 · p. 2
- Syn-Omni separates experts along the modality axis and regularizes their relation to the shared expert; Figure 2 depicts the paths and progressive routing.Its framework includes OME-LoRA and Progressive Synergy Routing.Source: §2 · p. 2, Figure 2 · p. 3
3 Syn-Omni Framework
Syn-Omni combines shared and modality-specialized adaptation paths with progressively enabled collaboration among modality experts. Its framework applies this design to contrastive omnimodal embedding learning and trains it with structural regularization.
3 Syn-Omni Framework
- Syn-Omni pairs OME-LoRA’s shared and modality-expert paths with PSR’s progressive cross-modal collaboration.The shared path captures universal semantics, while expert paths model modality-specific characteristics.Source: §3 · p. 3
3.1 Omnimodal Embedding Learning
- Uni-Omni embeds non-empty combinations of text, image, video, and audio inputs and uses contrastive optimization with LoRA adapters.The pretrained backbone weights are frozen, and matched pairs are aligned in a unified embedding space using InfoNCE.Source: §3.1 · p. 3
3.2 Orthogonal Modality-Expert LoRA
- OME-LoRA activates a shared LoRA path for every input and selectively engages modality-expert paths, while an orthogonal penalty reduces redundancy between shared and expert paths.The penalty applies to input-side projections and leaves output projections unconstrained.Source: §3.2 · pp. 3–4
3.3 Progressive Synergy Routing
- PSR begins with hard routing to establish modality priors, then blends toward soft collaboration; SML preserves absent-expert activations through a relative margin.The routing schedule has warmup, blending, and synergy phases.Source: §3.3 · pp. 4–5, Figure 3 · p. 5
3.4 Training Strategy
- Syn-Omni trains OME-LoRA throughout the backbone end to end, combining contrastive learning with orthogonality and Synergy Margin Loss regularization.Both regularization terms are averaged across layers containing OME-LoRA modules.Source: §3.4 · p. 5
4 Experiment
Syn-Omni improves overall omnimodal embedding performance across image, video, audio, and audiovisual tasks. Ablations and analyses examine how its shared and modality-specific paths, progressive routing, and synergy loss contribute to specialization and cross-modal interaction.
4.1 Main Results
- 49.4% at 3B and 51.6% at 7B are Syn-Omni’s best overall scores, with statistically significant improvements over Uni-Omni across 81 tasks.Syn-Omni improves most on audio and audiovisual tasks at 3B (+2.5% and +3.0%), with more moderate margins on image and video.Source: Table 1 · p. 6, §4.1 · p. 6
4 Experiment
- Evaluation covers 81 tasks across four modalities and compares Syn-Omni with recent Omni-MLLM embedding models under a unified protocol.The framework uses 1.12M training samples and 3B- and 7B-scale variants built on pretrained Qwen-2.5-Omni.Source: §4 · pp. 5–6
4.2 Ablation Study
- Adding a shared-expert structure improves over Uni-Omni, while PSR with SML further improves results and makes the orthogonal penalty beneficial.The orthogonal penalty adds a 49.0 → 49.4 gain when multiple expert outputs are jointly integrated.Source: §4.2 · p. 7, Table 2 · p. 7
- At an identical 44.9M parameter count, OME-LoRA improves over shared-only adaptation, and adding PSR raises performance to 49.4%.Removing the shared path degrades performance; expert-only adaptation scores 47.0 versus 47.7 for the shared-only baseline at 29.9M parameters.Source: §4.2 · p. 7, Table 3 · p. 7
4.3 Empirical Analysis
- Without progressive blending, pathways show high cosine similarities; Syn-Omni’s lower similarities indicate clearer modality specialization.The analysis uses final-layer signals averaged across samples and submodules on text-video-audio inputs.Source: Figure 5 · p. 8, §4.3 · p. 8
4.2 Ablation Study
- 49.4% is the best overall score for progressive blending with SML; SML improves both standard-soft and progressive-blending configurations over BCE.Performance remains relatively stable near λortho = 0.20, λSML = 0.20, and γ = 0.2, while larger margin values cause a noticeable drop.Source: Table 4 · p. 7, §4.2 · p. 8, Figure 4 · p. 8
4.3 Empirical Analysis
- Under video-audio inputs, present-modality experts retain dominant weights while absent-modality experts gain non-zero activations; BCE suppresses the latter.The analysis attributes the synergistic behavior to SML.Source: Figure 6 · p. 8, §4.3 · p. 8
5 Conclusion
Syn-Omni combines shared and modality-specialized parameters, then progressively shifts from isolated specialization toward cross-modal collaboration. Across 81 diverse tasks, it outperforms existing baselines with robust, balanced performance across four modalities.
5 Conclusion
- Across 81 diverse tasks, Syn-Omni outperforms existing baselines with robust and well-balanced performance across image, video, audio, and audiovisual modalities.The framework decouples modality-shared and specialized parameters with OME-LoRA and uses PSR to shift from isolated specialization toward dynamic cross-modal collaboration.Source: §5 · p. 9
Limitations
Dynamic routing in OME-LoRA adds inference overhead, particularly relevant in latency-sensitive settings such as real-time query encoding.
Limitations
- Dynamic expert pathways add 17–18% latency and less than 1% GPU memory; the shared pathway remains mergeable, unlike modality-expert weights.The authors identify latency-sensitive settings such as real-time query encoding as where the cost is most visible.Source: Limitations · p. 9
Ethical Considerations
The authors flag potential bias and misuse risks, and describe a training mixture and joint-training setup spanning four modality groups.
Ethical Considerations
- Unified omnimodal embeddings may inherit training-data biases that skew cross-modal retrieval and could be misused for unauthorized surveillance or identity linkage.The authors call for strict deployment oversight.Source: Ethical Considerations · p. 9
A.1 Training Datasets
- The training mixture covers image, video, audio, and audiovisual groups, including composed modality queries and/or targets.Datasets exceeding their specified target sizes were randomly subsampled to match those sizes.Source: §A.1 · p. 15
- Uni-Omni and Syn-Omni were jointly trained on the full mixture, while modality-specific models used only datasets from their respective modality groups..Source: §A.1 · p. 15
A.2 Evaluation Datasets
The evaluation suite covers image, video, audio, and audiovisual tasks, extending existing frameworks and adding bidirectional video-text retrieval.
A.2 Evaluation Datasets
- The image and video evaluation tasks use the MMEB-v1 and MMEB-v2 suites, respectively, with video-to-text retrieval added to complement text-to-video.The benchmark also expands to additional audio and audiovisual tasks.Source: §A.2 · p. 15
- Figure 7 maps query-to-target modality compositions using T, I, V, and A for text, image, video, and audio, and gives total sample sizes per modality group.Source: Figure 7 · p. 15
- Evaluation statistics distinguish corpus size by task: class count for classification, choice count for QA, and database size for retrieval.The collection spans common-domain settings, including generic audio and music; English is the primary language across multimodal tasks.Source: §A.2 · p. 15
B.1 Training Details & Hyperparameters
Training uses Qwen-2.5-Omni with LoRA parameters in its LLM layers. Syn-Omni combines contrastive learning with orthogonality and synergy-margin objectives.
B.1 Training Details & Hyperparameters
- Training uses the pretrained Qwen-2.5-Omni backbone, with LoRA inserted into seven target modules across all 36 LLM layers.The visual and audio encoders are frozen.Source: §B.1 · pp. 15–16
- Syn-Omni uses a shared rank-8 LoRA path and rank-4 modality experts, for total effective rank 24; the router adds less than 8% parameter overhead.The standard Uni-Omni baseline and modality-specific models use rank 16; LoRA scaling is twice the rank and dropout is 0.1.Source: §B.1 · p. 16, Table 5 · p. 16
- Syn-Omni is trained for one epoch with contrastive loss, an orthogonal penalty, and synergy margin loss; training uses batch size 768 and no hard negatives.The orthogonal penalty and synergy margin loss each have weight 0.2; the PSR warmup and transition ratios are 0.1 and 0.5.Source: §B.1 · p. 16
B.2 Evaluation Pipeline
The evaluation pipeline extends an existing framework to audio and audiovisual inputs, enabling any-to-any query-to-target retrieval. Layer-wise analysis finds that synergy margin loss preserves activations for absent-modality experts, unlike BCE routing.
B.2 Evaluation Pipeline
- The extended VLM2Vec-V2 evaluation framework supports image, video, audio, and audiovisual inputs for any-to-any query-to-target retrieval.Previous methods are evaluated under the unified setting using their official checkpoints.Source: §B.2 · p. 16
- At inference, Syn-Omni merges and removes its shared path but retains expert paths and router-generated soft weights for dynamic routing.Standard LoRA adapters are instead merged into the pretrained weights and no longer used separately.Source: §B.2 · p. 16
C.1 Routing Behavior across Layers
- Synergy margin loss preserves meaningful activations for absent-modality experts across layers, while BCE routing approaches deterministic hard routing for present modalities.The comparison uses converged models with audiovisual, video-only, and audio-only inputs from VALOR32k.Source: §C.1 · p. 17, Figure 8 · p. 17
C.2 Modality-Specific Models
Modality-specific models show strong native-modality performance and some cross-modal transfer, while Syn-Omni consistently improves over Uni-Omni and modality-specific models in their native modalities.
C.2 Modality-Specific Models
- Single-modality models perform strongly on their respective modalities, with the audio-specific model performing well on audio tasks and the audiovisual-specific model a noted exception.The authors report that exposure to image, video, and audio data in unified training is helpful for audiovisual tasks.Source: §C.2 · p. 17
- The image-specific model performs solidly on video tasks, and the video-specific model has reasonable image and audiovisual results, trends the authors connect to cross-modal synergy.The authors cite these transfer patterns as motivation for OME-LoRA paired with PSR.Source: §C.2 · pp. 17–18
- Syn-Omni consistently improves performance over Uni-Omni and the respective modality-specific models in their native modalities.Uni-Omni is comparable to or slightly below the modality-specific models in those native modalities.Source: §C.2 · p. 18
C.3 Runtime Analysis
Runtime analysis reports a modest latency and memory increase for Syn-Omni at 3B scale. The appendix also provides per-dataset results, dataset overviews, and an account of AI tool use.
C.3 Runtime Analysis
- For the 3B variant, mean latency rises from 300.3 ms to 354.6 ms, while peak GPU memory changes from 9612 MB to 9676 MB.The benchmark uses VALOR-32k audiovisual inputs on one NVIDIA RTX A6000 GPU at batch size 1; Syn-Omni retains expert paths and router parameters at inference.Source: §C.3 · p. 18
C.4 Per-dataset Evaluation
- Per-dataset performances are reported separately for image, video, audio, and audiovisual tasks.Tables 16–19 give individual modality results, with their method order matching Table 1.Source: §C.4 · p. 18, Table 16 · p. 25, Table 17 · p. 26
D AI Tool Usage
- ChatGPT was used to improve manuscript fluency and grammar; the authors attribute the core ideas, methods, experimental designs, and analyses to their original work..Source: D AI Tool Usage · p. 18
E Artifact
- Training dataset overviews cover image-text, video-centric, audio-centric, and audiovisual-centric data; the authors report following original licenses and research-use conditions when specified.All datasets were obtained from public benchmarks and repositories; derivatives and checkpoints target non-commercial research.Source: E Artifact · pp. 18, 22, Table 8 · p. 19, Table 9 · p. 20
- Evaluation dataset tables organize image, video, audio, and audiovisual datasets by task type and query-to-target composition.The tables are numbered 12–15, respectively.Source: Table 12 · p. 23, Table 13 · p. 24, Table 14 · p. 24