From Fragmentation to Fusion: A Survey and Roadmap for Omnimodal Embeddings

1 Introduction

• Embedding models turn inputs such as sentences, images, audio, graphs, and time series into vectors whose geometry supports semantic search, clustering, classification, and cross-modal alignment.The paper describes this shared representational foundation as central to retrieval and recommendation systems as well as cross-modal applications.Source: §1 · p. 4

1.1 The Fragmentation Problem: Silos of Research, Surveys, and Evaluation

• Research, surveys, and evaluation remain fragmented by modality, with cross-boundary works accounting for under 5% of both analyzed samples.The analysis covers 1,352 formally published embedding papers from 2023–2025 and a decade of arXiv preprints from 2015–2025.Source: §1 · p. 4, §1.1 · p. 5

1.2 Why Fragmentation Matters

• Embedding research matters beyond organizational tidiness because real-world concepts and current AI systems span modalities, while embedding fusion trails multimodal generation.The paper points to multimodal retrieval, grounded generation, robot memory, and cross-modal retrieval as existing but pair-limited examples.Source: §1.2 · pp. 5–6

1.3 Four Convergent Trends

• Across modalities, contrastive objectives, Transformers, pretrain–finetune workflows, and representations of whole semantic units have become shared trends.The shift toward whole-unit representations supports broad retrieval and cross-modal alignment, while local features remain useful for fine-grained grounding and dense correspondence.Source: §1.3 · pp. 6–7

1.4 Our Position and Roadmap

• The paper’s roadmap moves from a modality-agnostic foundation and convergent principles to transformation, alignment, and ultimately joint embedding, assessed through coverage, comparability, and compositionality.Its final parts address modality-specific properties, cross-pollination, and unified evaluation; the paper presents itself as the first unified cross-cutting treatment across major modalities.Source: §1.4 · pp. 7–8

2.1 The Anatomy of an Embedding Model

• The anatomy separates structural components from modeling decisions, with intended use guiding the data unit, encoder, aggregation, and information retained.The five components are orthogonal axes: changing the encoder need not change the objective or representational object.Source: Figure 2 · p. 9, §2.1 · pp. 9–10

2.2 Data and Representational Objects

• The data unit is the atom of the input, while the representational object is the semantic atom exposed to downstream consumers.The same encoder can expose global vectors, local vectors, discrete codes, distributions, or magnitude-preserving vectors, each retaining different information.Source: §2.2 · pp. 10–11

2.3 Our Analytical Lens: Intersection, Difference, and Reuse

• The analytical lens distinguishes shared principles, modality-specific challenges, and methods transferred across modalities; a finding may belong to more than one category.For example, audio’s superposition of factors is a modality-specific challenge, while disentanglement methods it prompts may be reused elsewhere.Source: §2.3 · p. 11
• Convergence has distinct meanings: architecture reuse, objective reuse, coordinate alignment, and parameter sharing, and one does not entail another.A shared trunk need not produce aligned retrieval vectors, and aligned outputs need not use shared encoder weights.Source: §2.3 · p. 11, Figure 3 · p. 12

3 Training Objectives: The Lingua Franca

• Training objectives converge across modalities: contrastive, reconstructive, and generative–hybrid losses share mechanisms, while modality-specific data shapes their targets.The chapter also describes pretraining followed by finetuning as a common workflow across modalities.Source: §3 · p. 13

3.1 Contrastive Objectives

• InfoNCE keeps the contrastive mechanism fixed, but each modality’s positive pair declares a different invariance and can fail when that invariance is abused.Hard-negative mining and alignment–uniformity analysis provide shared refinements; negative-free approaches instead use architectural asymmetry or batch statistics to prevent collapse.Source: §3.1 · pp. 13, 15, Table 3 · p. 14

3.2 Reconstructive Objectives

• Reconstructive objectives mask and predict, but their usefulness depends on whether the target carries meaning beyond its context.For smooth time series, reconstruction can favor interpolation and low-order structure that classical decomposition already captures; discrete tokens, cluster IDs, and object pixels provide more meaningful targets.Source: §3.2 · pp. 15–16

3.3 Generative and Hybrid Objectives

• Generative objectives specify what is predicted, while representations may be distributional, continuous, or discrete; hybrid approaches jointly train representation and generation.Hybrid approaches include shared sequence-to-sequence formats and models combining contrastive embedding losses with autoregressive generation objectives.Source: §3.3 · pp. 16–17

3.4 The Pretrain–Finetune Recipe

• Pretraining followed by finetuning recurs across modalities, but the data and signals filling each stage vary, and the pipeline has boundary cases.Web-scale paired data can make contrastive training the pretraining stage and replace finetuning with zero-shot transfer; graph pretraining can also hurt when mismatched to data structure.Source: §3.4 · pp. 17–18

4.1 The Transformer as a Universal Backbone

• The Transformer’s weakly biased, all-to-all self-attention can process any modality tokenized as a set, but demands more data than architectures with stronger priors.Absent positional encodings and causal masks, token permutations produce correspondingly permuted outputs; pooled readouts yield permutation-invariant embeddings.Source: §4.1 · p. 19
• Modality-specific tokenizers make the Transformer interface modality-agnostic, while positional schemes or other injected structure restore order, topology, and identity where needed.The same frozen encoder with modality-specific tokenizers and lightweight heads processes 12 modalities without paired cross-modal data.Source: §4.1 · pp. 19–20, Figure 4 · p. 20, Table 4 · p. 21
• A shared backbone can process many modalities, but parameter sharing alone does not align their outputs for cross-modal retrieval.The chapter distinguishes shared parameters from learned cross-modal representations; retrieval requires an added alignment objective.Source: §4.1 · pp. 20–21

4.2 The Rival Universal Encoder: State Space Models and Linear Attention

• State space models offer linear-cost computation and constant-memory inference, but fixed-size sequential states sacrifice random access and contextual recall needed for fine-grained embedding tasks.Their efficiency is decisive when sequence length is the binding constraint, such as raw audio, video, or genomic sequences too long for attention.Source: §4.2 · p. 22

4.3 Modality-Specific Architectures as Inductive Biases

• Strong modality-specific priors improve data efficiency and low-data performance but constrain expressiveness; Transformers favor generality and scalability when data and compute are ample.Small-data, high-noise time-series settings are among the cases where non-Transformer approaches remain competitive.Source: §4.3 · pp. 22–23
• Dual encoders support scalable approximate retrieval, whereas cross-encoders provide higher-accuracy joint scoring; pooling choices can also trade efficiency for dense correspondence.Cross-encoders are used as rerankers over a dual-encoder shortlist, while local vision or audio features can be retained instead of globally pooled.Source: §4.3 · p. 23

5.1 Units and Representational Objects

• The input data unit and the representational object are distinct: the former is what the model receives, while the latter is what downstream users retrieve, compare, or combine.Choosing an output object commits to which information must survive and which may be discarded.Source: §5.1 · p. 24
• Input units trade detail against sequence length, and their boundaries may be manufactured or task-dependent rather than naturally given.For time series, a sampled value is uninterpretable without its variable, unit, instrument, scale, and timestamp.Source: §5.1 · p. 24
• Global vectors suit retrieval and classification but discard location and correspondence, whereas local vectors preserve tessellation at greater storage and comparison cost.Other output forms include discrete codes, probability distributions, and magnitude-preserving vectors; no representational object is universally best.Source: §5.1 · pp. 24–25

5.2 The Granularity Shift—and Why It Is Partially Reversible

• Across modalities, embeddings have generally moved from measurable atoms toward whole semantic units, gaining invariance, fixed-size indexing, and efficient comparison.The pattern spans pooled text tokens, image patches, audio frames, graph neighborhoods, time-series samples, and table cells.Source: §5.2 · p. 25
• The shift is reversible: pooling can discard task-relevant location, counts, ordering, uncertainty, identity, or relations, so systems may expose a lattice of readouts.Global vectors favor latency and storage; local or set-valued outputs favor correspondence and attribution.Source: §5.2 · pp. 25–26

6 Path 1: Unification by Transformation

• Transformation reuses mature single-modality systems to broaden coverage, but alone guarantees neither comparability nor compositionality.Its three forms render data as images, serialize it as text, or tokenize it for language-model interfaces.Source: §6 · p. 27

6.1 Rendering as a Unifying Canvas

• Rendering works when a 2D image preserves task-relevant semantics, especially for layout-bearing documents, but can flatten graph topology or time-series magnitude.A shared visual shape does not guarantee shared semantics; rendering is part of the representation function.Source: §6.1 · pp. 27, 29

6.2 Serialization as a Unifying Language

• Serialization leverages the text stack, but linear strings can discard native structure, including graph topology, table geometry, multivariate channels, and time-series magnitude.The loss is greatest when formats leave structure implicit; tables and graphs suffer more than univariate forecasting.Source: §6.2 · pp. 29–30

6.3 Discrete Tokens: Leveraging the LM Interface

• Discrete tokenization lets language models consume continuous data through learned or fixed vocabularies, but manufactured tokens lack guaranteed cross-corpus conventions.A tokenizer can carry analog detail that serialization flattens, yet transfer depends on whether token meanings remain stable across domains.Source: §6.3 · pp. 30–31
• Transformation is a pragmatic shortcut that provides coverage at the expense of native geometry, motivating alignment and fusion to preserve or recover structure.It excels when the modality’s intrinsic semantics is visual layout; elsewhere, its lossy projections expose what joint embeddings must preserve.Source: §6.3 · p. 31

7 Path 2: Cross-Modal Alignment

• Alignment earns comparability by stitching independently encoded modalities into a shared retrieval space, but its separate inputs prevent compositionality.The modality gap also leaves different modalities occupying separate regions of that space.Source: §7 · p. 31, §7.3 · p. 35

7.1 The Alignment Paradigm

• Dual encoders trained on paired and unpaired examples provide a reusable alignment recipe, whose web-scale contrastive supervision enabled broad cross-modal retrieval.The recipe has been transplanted across video, audio, 3D, medicine, molecules, and time series while preserving its paired-data, two-tower essentials.Source: §7.1 · pp. 32–33

7.2 The Hub Choice: Which Modality Anchors the Space?

• To avoid combinatorial pairwise alignment, systems anchor modalities to a hub, reference-item coordinates, or a frozen tower.ImageBind aligns each modality to an image hub, while language-centric hubs benefit from readily available verbal descriptions but may omit what was never verbalized.Source: §7.2 · p. 33

7.3 The Limits of Alignment: The Modality Gap

• The modality gap persists from initialization and contrastive-objective geometry, can bias mixed-modality retrieval, and matters differently across tasks.It can be a ranking pathology in mixed-modality search yet carry functional information for unimodal zero-shot classification.Source: §7.3 · pp. 34–35

8 Path 3: Multimodal Joint Embedding

• Joint embedding is the advocated third path: modalities interact inside the model to produce one representation encoding their interaction.Unlike transformation, which flattens modalities, fusion aims to represent interaction rather than only proximity between separately encoded parts.Source: §8 · pp. 35–36

8.1 The Fusion Paradigm

• Intra-item fusion uses cross-attention to condition each modality’s representation on another’s content, while retaining per-item encoding and offline indexability.Cross-attention is quadratic in interacting token counts, making fused encoders more expensive than separately encoded parts.Source: §8.1 · pp. 36–37

8.2 Joint Embedding as a General and Ultimate Framework

• Joint vectors should capture cross-modal interaction and beat the strongest unimodal encoder and additive baselines, a criterion the paper calls compositional gain (∆comp).Fusion weakly dominates alignment only in the capacity-unconstrained limit; whether its extra capacity pays off under a given compute and data budget remains empirical.Source: §8.2 · p. 39
• Multimodal inputs trigger fusion, whereas unimodal inputs can use the same parameters as a within-modality encoder, though specialist parity is not automatic at finite capacity.E5-V transfers from text-only training to image and composed-image retrieval; Qwen3-VL-Embedding and MM-Embed provide counterexamples to automatic specialist parity.Source: §8.2 · pp. 39–40
• The section frames the three paths as a progression: transformation homogenizes inputs, alignment offers shared metric spaces, and joint embedding targets deep interaction.Joint embedding is argued to retain modality-specific structure inside the fused representation rather than discard it through transformation.Source: §8.2 · p. 40
• Fusion is costly and can fail through modality collapse, gradient interference, or modality competition, so building a joint embedding that fuses without collapsing remains open.Contrastive losses preserve shared information but do not necessarily remove modality-specific nuisance or protect modality-specific signal.Source: §8.2 · p. 40

8.3 Desiderata for a Multimodal Joint Embedding Model

• A good joint model should preserve modality-specific interfaces and parameters, and let each modality deepen before cross-modal interaction.The section also recommends evaluating compositional gain and actively enforcing unimodal parity rather than assuming it.Source: §8.3 · p. 41
• The remaining criteria are to define policies for absent streams and expose flexible readouts, since deployed tasks may use different modality subsets and output granularities.The five criteria together cover where modality-specific structure should reside and what fusion should deliver: composition, parity, and robustness.Source: §8.3 · pp. 41–42

Modality-specific challenges

• A joint embedding must preserve each modality’s defining structure rather than treating modality-specific properties as obstacles to be engineered away.The section frames these properties as the information each modality carries and proposes cross-modal reuse as a route to mechanisms that respect them.Source: §9 · pp. 44, 48

Text and code

• Text embeddings must resolve polysemy while capturing compositional meaning, and code embeddings must represent behavioral equivalence and hierarchical structure beyond surface form.Compiler intermediate representations offer a language- and machine-independent view of code behavior.Source: §9 · p. 44

Vision

• Vision embeddings must balance invariance against equivariance while preserving spatial, multi-scale composition; video also requires retaining motion when retrieval is motion-sensitive.The section notes that pixels and patch boundaries do not supply stable recurring units across images.Source: §9 · p. 44

Audio

• Audio embeddings must specify which superposed factor they represent, because tasks may require preserving one factor while suppressing another.The relevant factors can include content, speaker, prosody, language, room, and microphone; the appropriate factor depends on the task and evaluation protocol.Source: §9 · pp. 45–46, Table 7 · p. 46

Graph

• Graph embeddings must preserve task-relevant relations and be insensitive to arbitrary node naming, with relational structure explicitly encoded in the model.Graph specificity may involve arity, geometry, invariance, dynamics, or topology rather than vectors alone.Source: §9 · pp. 46–47

Tables

• Table embeddings must choose between document and feature-matrix readings, which require different sensitivities to row and column order.Cell meaning also depends on value, headers and types, row identity, and positional relations.Source: §9 · p. 47

Time series

• Time-series embeddings must preserve continuous time and magnitude while accommodating irregular sampling and domain-specific recurring motifs.Instance normalization can discard amplitude semantics; NuTime instead attaches each window’s mean and standard deviation to its normalized shape.Source: §9 · p. 47

Cross-modal regularities

• Frequency-domain representations recur across continuous-signal families, but text, code, and tables lack a data-native spectrum in the account presented.The section identifies a transform-based intersection across audio, image, graph, and time series while distinguishing symbolic data.Source: §9 · pp. 47–48

Cross-pollination blueprint

• Cross-pollination is conditional rather than a free transplant: a transfer is expected to work when the target modality satisfies the source method’s preconditions.The opportunity map distinguishes documented uses from proposed experiments and states expected benefits when those conditions hold.Source: §10 · pp. 48, 50, 51, Table 8 · p. 49

Text’s instruction tuning → audio and time series

• Instruction-conditioned embeddings could transfer text’s task-specific approach to audio and time series, where one model could produce task-dependent representations.For audio, the proposed instructions include embedding for speaker identity while discarding content, or for acoustic scene while ignoring speech.Source: §10 · p. 48

Audio’s codec discretization → code and time series

• Learned semantic codebooks could transfer audio codec discretization to code execution traces and time series, provided the target has stable recurring units.Where stable units do not form, magnitude-preserving numeric embedding is presented as the more faithful bridge for time series.Source: §10 · pp. 48–50

Graph’s structural modeling → text and code

• Graph structural modeling offers a relational layer for text and code, motivating study of when explicit relations improve embeddings beyond content alone.The proposed discipline is to name the represented unit and relational objective before comparing architectures.Source: §10 · p. 50

Time series’ continuous-time indexing → video and dynamic graphs

• Continuous-time indexing from time-series research could support variable-speed video and temporal knowledge-graph embeddings without treating timestamps only as discrete identifiers.The section identifies continuous-time dynamic graphs as an existing transfer example and proposes continuous-time functions for temporal knowledge graphs.Source: §10 · p. 50

Factor accounting → vision and text

• Audio’s factor-accounting approach could make factor preservation and removal testable in vision and text through factor tables and removal metrics.The proposed examples include appearance versus identity in vision and content versus style in text.Source: §10 · p. 50

• The 3C lens treats coverage, comparability, and compositionality as separate, progressively stronger evidence claims; a mechanism’s ability to support a capability does not demonstrate it.Source: Table 9 · p. 52, Table 10 · p. 52
• Current benchmark islands unevenly cover modalities; none measures compositionality across all modalities under one protocol.Source: §11 · p. 52, Table 11 · p. 53
• A unified suite should span unimodal, cross-modal retrieval, and multimodal fusion tasks under a frozen-encoder weak-head protocol, with standardized probe budgets and decoder-independent time-series retrieval.Source: §11 · p. 54
• ∆comp operationalizes compositionality as joint-input utility over best proper-subset and weighted score-fusion or vector-addition baselines; C3 credit requires positive gain.Source: §11 · p. 54
• Building such a benchmark is constrained by costly, hard-to-release fusion labels, unimodal shortcuts and contamination, plus expensive evaluation and rapidly aging static snapshots.Source: §11 · p. 55
• Aggregating releasable datasets risks missing fusion tasks, while MTEB’s licensing curation, down-sampling, and maintenance offer a workable precedent.Source: §11 · p. 55
• Most broad-input systems establish coverage and comparability, while nearly all explicitly multi-input systems have only partial compositionality evidence; the sole strong result covers two modalities.Source: Table 12 · p. 56, §11 · pp. 56–57, Table 13 · p. 57
• Until a unified, decoder-independent benchmark tests compositional gain across modalities, joint embedding’s unification and superiority over alignment remain hypotheses, not empirical facts.Source: §11 · pp. 55, 57

12 Conclusion and Future Outlook

• Joint embedding lets modalities interact inside one model, generalizing alignment while specializing for unimodal inputs, provided training mitigates modality interference.Transformation loses structural information, while alignment retains persistent modality gaps from the contrastive cone effect and dual-encoder bias (Liang et al., 2022).Source: §12 · p. 58
• A path to this goal must respect modality-specific structure, enable conditional cross-pollination, and establish a unified, decoder-independent omnimodal benchmark.The benchmark is described as a prerequisite for turning the convergence thesis from a hypothesis into a measured fact.Source: §12 · p. 58
• A genuine joint embedding would serve as a universal interface where any modality’s data can be retrieved, compared, and combined in a common geometry.The paper argues this could help close the gap between omni representation and omni generation, and may be a stepping-stone toward more general machine intelligence.Source: §12 · p. 58

A Per-Modality Cheat Sheets

• The cheat sheets summarize each modality’s data and granularity, suitable backbones, objectives, embedding interfaces, developmental arc, milestones, benchmarks, and cross-modal relevance.They are designed to provide quick orientation without interrupting the paper’s concept-first main thread.Source: A Per-Modality Cheat Sheets · p. 59

A.1 Text

• Text embeddings shift from word vectors toward sentence- and document-level interfaces, with contrastive finetuning and instruction conditioning shaping their development.Multi-vector readouts retain token-level information for grounding, while instruction-conditioned interfaces expose task-aware vectors.Source: §A.1 · pp. 59–60

A.2 Code

• Code embedding research progressively targets deeper structure—from surface tokens and syntax to semantics and behavior—with LLVM IR offering language- and machine-independent units.Structure-bearing inputs and masks, rather than the Transformer backbone itself, supply inductive bias.Source: §A.2 · pp. 60–62

A.3 Table

• Tables straddle semi-structured documents and tabular ML, making task-dependent cell, row, and column representations and 2D structure central to their embeddings.Structure-aware encoders preserve grid relationships that flat serialization can hide; the two readings serve retrieval and QA versus prediction.Source: §A.3 · pp. 62–63

A.4 Image and Video

• Visual embeddings span pixels through events, with global vectors as the canonical interface and dense or local features retained for grounding, correspondence, and motion.ViTs became the default image descriptor, while video adds a temporal axis to image appearance models.Source: §A.4 · pp. 64–65
• Image and video embeddings progress from engineered visual codes through learned and self-supervised representations to multimodal and dense representations, while video adds temporal and hierarchical units.The cross-modal pointers identify rendered text, CLIP/ImageBind alignment, spatial multi-scale representations, diffusion latents, and continuous-time indexing as links or differences with other modalities.Source: §A.4 · pp. 65, 67

A.5 Audio

• Audio embeddings span speaker vectors, masked prediction, codec tokens, and audio-language alignment; their factor-superposed content makes the appropriate representation grain depend on the factor queried.Audio exposes different objects—including frame sequences, utterance-level vectors, discrete codes, and dense frame sets—and its four eras were tracked through SUPERB.Source: §A.5 · pp. 67–68

A.6 Graph

• Graph embeddings develop through overlapping threads in relational scoring, structure-aware encoders, reusable self-supervision, and foundation models, alongside cross-modal alignment; this is not a necessary progression toward one graph vector.The represented object may be a node, edge, triple, path, subgraph, whole graph, or time slice, and the appropriate granularity depends on the question.Source: §A.6 · pp. 69–70

A.7 Time Series

• Time-series interfaces expose magnitude-preserving global vectors when level matters, fixed-size patch embeddings otherwise, or history-conditioned states for event streams.No broadly adopted frozen-encoder MTEB analog exists; representation quality is probed on UCR and UEA, while Monash and GIFT-Eval measure end-to-end forecasting.Source: §A.7 · pp. 71–72
• Time-series modeling spans four eras that have not cleanly superseded one another, from features and kernels through contrastive learning and patching to foundation models and LLM bridges.Time-series units and boundaries are task-dependent, while approaches include recurrent, convolutional, patch-Transformer, and continuous-time encoders and objectives ranging from contrastive learning to masked reconstruction and forecasting.Source: §A.7 · pp. 70–71

A.8 Other Modalities

• Point-cloud work ranges from permutation-invariant PointNet set aggregation to masked or generative point encoders, then transformation and language-alignment routes.Evaluation uses trained heads for classification and part segmentation, or zero-shot recognition in text-aligned spaces; no broadly adopted MTEB analog exists.Source: §A.8 · pp. 72–73
• Biomolecular embeddings span molecular graphs, SMILES strings, 3D conformations, and protein sequences, with contrastive alignment to language or assay descriptions.Evaluation relies on task heads for molecular property prediction and protein structure or function prediction, without an embedding-native benchmark on the horizon yet.Source: §A.8 · p. 73

B Quantitative Analysis of the Literature

• Bibliometric evidence indicates that embedding research is growing rather than receding and remains fragmented across communities, while early cross-modal convergence is measurable through publication venue and terminology.The analysis uses an arXiv title-term baseline for 2015–2025 and a manually curated census of formally published conference and journal papers for 2023–2025.Source: B Quantitative Analysis of the… · p. 74

B.1 An arXiv title-term analysis (2015–2025)

• The arXiv title analysis retained 10,952 of 11,806 candidates after excluding spurious matches and position-embedding phrases.The search covered six arXiv categories from 2015 to 2025.Source: §B.1 · p. 74
• Multimodal embedding titles rose from 15 in 2023 to 29 in 2024 and 62 in 2025, a 2.1× increase from 2024 to 2025 against 13% total growth.Cross-modal embedding titles never exceeded eight per year across the 2015–2025 window.Source: §B.1 · p. 74, Figure 15 · p. 74

B.2 A curated census of conference papers (2023–2025)

• Researchers manually collected and classified embedding and representation-learning papers across nine modality buckets from major peer-reviewed venues spanning NLP, vision, speech, machine learning, and information retrieval-related communities.Source: §B.2 · p. 75
• Cross-modal alignment was the only bucket to roughly double twice, rising from 7 to 22 to 37 papers, while most buckets were flat or declining.Text embeddings clustered in NLP venues, audio in speech venues, and alignment papers were concentrated at CVPR.Source: §B.2 · p. 75, Figure 16 · p. 75

B.3 Limitations

• The phrase-based counts are lower bounds, title matching misses work that does not name embeddings, and publication patterns cannot establish representation quality or downstream reasoning gains.The authors frame the views as measures of attention, venue silos, and which literature areas are growing fastest.Source: §B.3 · p. 75