OmniFusion: Simultaneous Multilingual Multimodal Translations via Modular Fusion

1 Introduction

• Visual context can disambiguate caption translation, while using images or slides in speech translation remains underexplored..Source: §1 · p. 1
• Cascaded ASR–MT pipelines remain strong in speech translation, but add latency—especially in SimulST—and cannot directly use multimodal context.MMFMs generally have weaker multilingual translation quality than specialized translation LLMs.Source: §1 · p. 1
• Prior systems often address speech, speech–image, and text–image translation separately, and often leave MMFM internal cross-modal representations underused.OmniFusion's design is motivated by the hypothesis that MMFM hidden states can provide useful cross-modal signals with limited training.Source: §1 · p. 1
• OmniFusion connects a frozen MMFM to a translation-specialized LLM for speech-to-text, speech-and-image-to-text, and text-and-image-to-text translation.Its contributions also report improved SimulST latency–quality trade-off over the same cascade, fewer major and critical offline ST errors, and competitive CoMMuTE performance.Source: §1 · pp. 1–2

2.1 Architecture

• Audio and image inputs are encoded by the MMFM and used as additional context for the translation model.The approach connects a frozen MMFM to a translation-specialized LLM through a lightweight fusion module.Source: §2.1 · p. 2, §2 · p. 2

2.1.1 Probing MMFM Representations

• Adjacent-layer cosine similarity serves as a diagnostic for selecting candidate MMFM layers, not as proof of the semantic information they encode.The first, middle, and last layers are selected to cover three representation regions, and downstream ablations validate their usefulness.Source: §2.1.1 · pp. 2–3, Figure 2 · p. 3

2.2 Gated Layer Fusion

• For each multimodal token, the model concatenates first, middle, and last layer states and learns per-layer weights to form a fused representation.A bias-free linear layer produces logits normalized by softmax; the resulting representation is projected into the translation model’s embedding space with a three-layer MLP.Source: §2.2 · p. 3
• Projected multimodal embeddings are prepended to source-token embeddings and treated as additional input tokens for translation.The projected embeddings use the translation model’s token-embedding dimension.Source: §2.2 · p. 3

2.3.1 Multi-task Learning

• Multitask training combines speech translation, speech–image translation, and text–image translation to expose the model to different modality combinations.The tasks are described as encouraging multimodal alignment.Source: §2.3.1 · p. 3

2.3.2 Self-Cascading (ASR and OCR)

• Self-cascading trains the model to produce intermediate ASR or OCR text before translation, with the examples applied stochastically.The MMFM is frozen while the translation LLM is fine-tuned with LoRA; adding LoRA to the MMFM degrades performance due to catastrophic forgetting.Source: §2.3.2 · pp. 3–4

3.1 Training Data

• Training uses Europarl-ST and CoVoST for speech translation, M3AV and its Seed-X-PPO-7B-translated augmentation for image-aware speech translation, and Multi30k for text-and-image translation.M3AV-Aug supports speech-image translation and self-cascade training; the authors report high translation quality despite using unfiltered synthetic data.Source: §3.1 · p. 4

3.2 Models

• The system pairs Qwen2.5-Omni 7B for multimodal input with Seed-X-PPO-7B for translation, and preliminary experiments suggest the approach extends to other pairings.The selected checkpoint has the lowest CoVoST dev loss after 20k training steps.Source: §3.2 · p. 4

3.3 Evaluation: Data & Metrics

• Evaluation uses MCIF and FLEURS for speech translation and simultaneous speech translation, and CoMMuTE for text-and-image translation.M3AV is excluded from testing because of training overlap and its monolingual nature; latency uses Average Lagging, with XCOMET-XL for speech translation and COMET4 for text-and-image translation.Source: §3.3 · p. 4

4.1 Simultaneous Speech Translation

• OmniFusion achieves better latency and quality than the fine-tuned cascade, while image input brings further quality gains; its audio-only setting has comparable quality at lower latency.StreamUni-Phi4 has lower computation-aware latency, and OmniFusion scores substantially higher than both external SimulST baselines in the evaluated settings.Source: §4.1 · pp. 5–6, Figure 3 · p. 5

4.2 Fusion Architecture and Layer Selection

• The three-layer gated model performs best across evaluated language pairs; using all MMFM layers degrades performance, and attention-based fusion is less robust, particularly for zero-shot source languages.OmniFusion also outperforms SeamlessM4T and Tower-Spire in all directions described.Source: §4.2 · p. 6, Table 1 · p. 6

4.3 Impact of Training Tasks

• Training on all tasks gives the strongest direct-inference results for en→de and en→it and remains competitive for en→zh; under self-cascade inference, the full model performs best across all language pairs.Removing OCR gives the weakest performance across the three language pairs, while removing both OCR and self-cascading performs better than removing OCR alone.Source: §4.3 · pp. 6–7, Table 2 · p. 6

4.4 Offline Speech Translation

• The best cascaded system has the highest aggregate XCOMET-XL score, while OmniFusion with gated fusion, images, and self-cascade nearly matches it and produces fewer severe errors.Directly fine-tuning the MMFM performs substantially worse than OmniFusion; image input provides only marginal gains for the cascade but clearer gains for OmniFusion.Source: §4.4 · p. 7, Table 3 · p. 7

4.5 Text-Image Translation

• On CoMMuTE, Mid Fusion slightly outperforms Gated Fusion, while OmniFusion matches or surpasses strong multimodal baselines on most language pairs.Middle-layer fusion suits text-image translation, where images disambiguate text; gated fusion is better suited to speech translation, where audio is primary.Source: Table 4 · p. 7, §4.5 · p. 8

4.6 Inference-Time Gate Behavior

• The gate assigns low weight to the last MMFM layer, uses early representations more for speech, and relies more on middle-layer representations for image-only and speech-plus-image inputs.Its layer weighting adapts to the available modality rather than selecting one fixed layer for every task.Source: §4.6 · p. 8, Figure 4 · p. 8

Multimodal MT

• Multimodal caption translation includes models trained from scratch and pretrained MT models augmented for images [Elliott et al., 2017; Vijayan et al., 2024; Costa-Jussà et al., 2022; Futeral et al., 2025].Fewer studies jointly use speech and visual context [Sinhamahapatra and Niehues, 2025; Anwar et al., 2023; Choi et al., 2024]; OmniFusion uses a lightweight fusion interface to leverage both.Source: §5 · p. 8

Simultaneous ST

• Simultaneous speech translation adapts offline models with fixed or learned policies, including wait-k, Local Agreement, attention-based heuristics, and LLM-based policies [Ma et al., 2019; Elbayad et al., 2020; Liu et al., 2020; Papi et al., 2023; Koshkin et al., 2024b,a; Guo et al., 2025; Fu et al., 2025].OmniFusion uses Local Agreement for evaluation but is compatible with other SimulST policies.Source: §5 · p. 8

Modality Fusion

• Multimodal LLMs commonly merge separately encoded modalities in the LLM, while other work studies earlier audio-visual interaction, projectors, and discrete tokens [Abdin et al., 2024; Xu et al., 2025; Team et al., 2025; Ye et al., 2025; Verdini et al., 2024; Gaido et al., 2024b; Zhan et al., 2024; Li et al., 2025].OmniFusion instead studies which internal MMFM layers transfer to the translation LLM.Source: §5 · p. 8

• Early and middle MMFM layers transfer better than late layers, and selected-layer fusion improves over all-layer fusion in ST.Source: §6 · p. 8
• OmniFusion improves the SimulST latency–quality trade-off under Local Agreement decoding while remaining competitive in offline ST and CoMMuTE.Source: §6 · p. 9
• The best fusion strategy depends on the task: gated fusion is more robust for ST, while middle-layer fusion can suffice when images mainly provide disambiguation context.Source: §6 · p. 9

• Training data is primarily English-centric, which may limit performance for other source languages.Zero-shot experiments on non-English source languages suggest encouraging transfer, but broader source-language training and evaluation are needed.Source: Limitations · p. 9
• Fixed task-specific prompts across the three supported translation settings mean OmniFusion does not exhibit general instruction-following capabilities.Whether the fusion strategy extends to more flexible, instruction-driven multimodal behavior remains unclear.Source: Limitations · p. 9
• The manual CoMMuTE analysis is small-scale, and broader human evaluation is needed to assess visually grounded translation behavior.The analysis shows reasonable performance on targeted visual disambiguation.Source: Limitations · p. 9

• The manual evaluation uses the first 50 English-to-German source sentences, each paired with two images, for 150 examples.A single annotator inspects each source sentence and image, with hypotheses and disambiguation labels provided to reduce potential bias.Source: §A.1 · p. 12
• On 50 source sentences, OmniFusion correctly disambiguates 18 examples, compared with TowerVision’s 23.The subset contains 150 examples, with each source sentence paired with two images.Source: §A.1 · p. 12
• The benchmark examples are described as highly challenging and synthetic, and OmniFusion scores above a text-only baseline that would score zero.Source: §A.1 · p. 12
• Although OmniFusion underperforms TowerVision on this image-focused subset, it also supports audio and was not trained on the same scale of data.The authors characterize OmniFusion as retaining broader multimodal applicability while requiring substantially less modality-specific training data.Source: §A.1 · p. 12

• Forcing the first MMFM layer degrades quality, the middle layer produces hallucinated transcripts and ASR-like outputs, and the last layer emits only the eos token.The table reports XCOMET-XL scores on the MCIF en →de test set for forced first-, middle-, and last-layer controls.Source: Table 5 · p. 12, §A.2 · p. 12
• These single-layer failure modes suggest that the gate balances complementary information from multiple MMFM depths rather than selecting one best layer.Source: §A.2 · p. 12
• The ablations and analysis support the need for both selected MMFM layers with gated fusion and OCR and self-cascading training objectives.The authors state that these architectural and training choices make multimodal representations useful for translation.Source: §A.2 · p. 12

A.3 Attention Fusion

• Attention-based fusion combines token-wise representations from selected MMFM layers; the example uses first, middle, and last layers, and the formulation extends to arbitrary selected-layer counts.Source: §A.3 · p. 13

A.4 Prompt Building

• Prompts direct the model to use images for translation context and OCR spelling, and to resolve formality, gender, keywords, and other disambiguating details.Source: §A.4 · p. 13

A.5 Layer Analysis, Zero-shot ability and APE - AudioTower

• Fine-tuned Qwen2 Audio remains below the cascaded setup across languages; Tower-based APE struggles on hard zero-shot directions such as fr→kr, and off-the-shelf end-to-end ST lags.Source: §A.5 · p. 15
• Middle-layer AudioTower performs strongest overall, surpassing the cascaded baseline in several languages; first and middle layers transfer to unseen languages, unlike the last-layer collapse.Source: §A.5 · p. 15
• Speech-Aware APE delivers the best overall results by combining cascaded hypotheses with raw audio, and remains comparable to AudioTower-Mid with empty hypotheses.Source: §A.5 · pp. 15–16

A.6 Latency Reproduction

• Latency reproduction feeds fixed-duration audio chunks iteratively, applies Local Agreement, and records committed output and consumed audio; cascades wait for ASR stabilization before MT agreement.Source: §A.6 · p. 16

A.7 Decoding

• Decoding uses max_new_tokens 256 and beam size 5, with inference run on one NVIDIA RTX A6000 GPU.Source: §A.7 · p. 16