Source-linked AI summary
Does a Technique for Building Multimodal Representation Matter? -- Comparative Analysis
Maciej Pawłowski, Anna Wróblewska, Sylwia Sysko-Romańczuk
TL;DR
The paper addresses the lack of comparative evidence for choosing among multimodal representation techniques. It compares late fusion, early fusion, and sketch methods across classification datasets, finding that technique choice should reflect modality relevance, task type, and memory constraints.
Problem
No universal multimodal fusion technique had been established, leaving the relative behavior and selection criteria of existing methods unclear.
Method
The study compares late fusion, early fusion, and sketch representations in classification experiments on Amazon Reviews, MovieLens25M, and MovieLens1M.
Results
Multimodal performance improves when the input modalities are meaningful for the task, while movie posters can reduce results and sketch performs worse in typical classification problems.
Takeaways & Limitations
Choosing a multimodal technique requires considering each modality's task impact, the ML task type, and memory requirements.
Takeaways & Limitations
The work does not entirely solve the problem of choosing the proper technique for building a multimodal representation.
Abstract
from arXiv · showhide
Creating a meaningful representation by fusing single modalities (e.g., text, images, or audio) is the core concept of multimodal learning. Although several techniques for building multimodal representations have been proven successful, they have not been compared yet. Therefore it has been ambiguous which technique can be expected to yield the best results in a given scenario and what factors should be considered while choosing such a technique. This paper explores the most common techniques for building multimodal data representations -- the late fusion, the early fusion, and the sketch, and compares them in classification tasks. Experiments are conducted on three datasets: Amazon Reviews, MovieLens25M, and MovieLens1M datasets. In general, our results confirm that multimodal representations are able to boost the performance of unimodal models from 0.919 to 0.969 of accuracy on Amazon Reviews and 0.907 to 0.918 of AUC on MovieLens25M. However, experiments on both MovieLens datasets indicate the importance of the meaningful input data to the given task. In this article, we show that the choice of the technique for building multimodal representation is crucial to obtain the highest possible model's performance, that comes with the proper modalities combination. Such choice relies on: the influence that each modality has on the analyzed machine learning (ML) problem; the type of the ML task; the memory constraints while training and predicting phase.
1 Introduction
Multimodal learning combines complementary modalities, but no universal fusion technique has been established. This paper compares late fusion, early fusion, and sketch representations to identify selection criteria across classification scenarios.
- Motivation: Multimodal learning combines text, images, audio, numerical, or behavioral data, and prior work reports performance gains from integrating complementary modalities.Applications include speech recognition, information retrieval, medicine, and human behavior analysis.
- Research gap: Existing multimodal fusion approaches are often task-specific or too complex, limiting their interpretability and flexibility.
- Selection criteria: The study evaluates technique choice according to modality influence on the task, ML task type, and memory constraints during training and prediction.
- Contribution: The paper aims to compare existing techniques because understanding their differences is necessary for developing a more universal multimodal approach.
- Approach: The paper compares late fusion, early fusion, and sketch representations in classification tasks using Amazon Reviews, MovieLens25M, and MovieLens1M.Late fusion combines independently learned modalities before the decision; early fusion combines modalities before training; sketch transforms them into a common space.
2 Related Work
Related work characterizes multimodal representations, fusion strategies, and hashing-based sketches, emphasizing complementary information, robustness, memory efficiency, and unresolved trade-offs. The paper positions its comparison of early fusion, late fusion, and sketch methods against this background.
- Types of modalities: The study covers tabular, graph, signal, and sequence modalities, including labels, text, images, and movie-viewing relations.
- Multimodal evidence: Prior studies report that combining modalities can outperform unimodal models, while the examined fusion techniques had not previously been compared.
- Representation requirements: A good multimodal representation should preserve intermodal similarity and remain usable when some modalities are absent.
- Fusion techniques: Early fusion concatenates modality vectors before training one model, whereas late fusion trains independent modality models and combines their outputs.
- Deep learning limitations: Deep learning fusion models can require substantial resources because they contain many weights, including parameters for modalities that contribute little information.They may also be inflexible in dynamic data environments.
- Hashing approaches: Hashing methods reduce dimensionality while preserving similarities, offering memory efficiency and robustness, but may lose information during compression.
- Sketch representation: Sketch representations transform modalities into a common, sparse space and can add modalities on the fly; the paper modifies this approach into binary features linked to hyperplanes.The binarized sketch is intended to preserve more information about individual observations while using less memory.
3 Datasets
The study evaluates multimodal representations across three datasets and three classification tasks, using dataset-specific modalities, splits, and evaluation considerations. The datasets differ in scale, modality completeness, and class balance.
- The experiments cover multiclass product classification, multilabel genre classification, and binary gender classification across Amazon Reviews, MovieLens25M, and MovieLens1M.
- Amazon Reviews and MovieLens25M use 0.6/0.2/0.2 train-validation-test splits, whereas MovieLens1M uses an 0.8/0.2 split with k-fold cross-validation on training data.
- Amazon Reviews contains textual product descriptions and titles plus product images, with approximately 25% of descriptions missing and 3% of titles empty.
- MovieLens25M provides movie plots, posters, and viewing-relationship graphs for 60,763 processed movies, with missing modalities and highly imbalanced genres.
- MovieLens1M uses movie plots and posters to predict user gender; it has no missing modalities, but males comprise approximately 71.7% of users.
4 Research Design and Experiments
The experiments compare late fusion, early fusion, and sketch-based multimodal representations across three classification settings. Each dataset uses architectures and evaluation metrics suited to its task and modalities.
- The study compares late fusion, early fusion, and classical or binarized sketch representations in three classification experiments.
- Amazon Reviews: Amazon Reviews evaluates multimodal neural-network fusion for balanced multiclass product classification using accuracy.
- Amazon Reviews: In late fusion, modalities are modeled independently and combined before decision-making, while early fusion concatenates modality embeddings at the input level.
- Amazon Reviews: Early fusion concatenates modality embeddings, whereas sketch models transform embeddings into a shared sketch space before flattening and concatenation.
- MovieLens25M: MovieLens25M evaluates sketch and early-fusion representations for imbalanced multilabel genre classification using micro-AUC and micro-mAP.
- MovieLens1M: MovieLens1M predicts gender from users’ watched movies, representing each user as a sum of movie sketches before normalization, flattening, and classification.
5 Results
Results show that multimodal performance depends on both the fusion technique and whether each modality provides meaningful information for the task. Late fusion is favored when one modality dominates, early fusion when modalities interact, while sketch trades classification performance for lower storage use.
- Amazon Reviews: Multimodal models improve over unimodal models across Amazon Reviews categories, especially where unimodal models have similar accuracy, such as Wallet.Categories dominated by one strong modality, such as Watch, gain less from multimodality.
- Amazon Reviews: Missing approximately 25% of Sweater descriptions does not prevent the description model from significantly outperforming the image model in that category.The result indicates that a modality with missing data may still provide unique information.
- Amazon Reviews: Trimodal models usually do not significantly outperform title-and-image bimodal models, while requiring twice the training time.The added modality may therefore be too costly for its slight improvement in practical settings.
- Amazon Reviews: Late fusion achieves the best results in nearly all Amazon Reviews scenarios, particularly when one modality is more informative than the others.Its independent modality models can exploit the dominant signal while incorporating additional modalities.
- Fusion-technique comparison: Early fusion is recommended when several modalities affect performance similarly because their interactions may reveal hidden information.This contrasts with late fusion, which is favored when one modality is dominant.
- Fusion-technique comparison: Sketch performs worse in typical classification tasks but uses less storage, and its performance almost always increases as more modalities are added.The representation is therefore a memory-efficient option whose classification potential depends on modality count.
6 Conclusions
The study compares three multimodal fusion techniques and identifies selection criteria for choosing among them, while acknowledging that the technique-selection problem remains incomplete. It also calls for a benchmark to validate conclusions across multimodal datasets.
- The comparison covers late fusion, early fusion, and sketch, identifying their respective strengths and weaknesses.
- The paper does not entirely solve how to choose the proper technique for building a multimodal representation.
- Three criteria guide technique selection: each modality’s task impact, the ML task type, and memory usage.These criteria support recommendations for multimodal representation building.
- A benchmark for multimodal datasets is proposed as the next step to confirm conclusions from this and related work.