Source-linked AI summary
Efficient Large-Scale Multi-Modal Classification
D. Kiela, E. Grave, A. Joulin, T. Mikolov
TL;DR
The paper addresses efficient classification of large-scale data combining discrete text with continuous visual information. It compares fusion strategies and discretizes continuous features to simplify and accelerate fusion. Multi-modal methods improve over text-only classification, while discretized fusion retains gains with lower computational cost and improved interpretability.
Problem
The paper asks how to fuse discrete and continuous modalities accurately and efficiently for large-scale multi-modal classification.
Method
The paper compares fusion methods and converts continuous visual features into discrete tokens used within a standard FastText setup.
Results
Multi-modal models outperform standard FastText and continuous-only methods, while bilinear-gated fusion has the highest accuracy and simpler models offer higher speed.
Takeaways & Limitations
Discretized multi-modal features provide a faster, lower-storage alternative that preserves multi-modal benefits and enables interpretation of visual features.
Takeaways & Limitations
Continuous models require expensive matrix-vector multiplication and large floating-point matrices, especially as feature dimensionality grows.
Abstract
from arXiv · showhide
While the incipient internet was largely text-based, the modern digital world is becoming increasingly multi-modal. Here, we examine multi-modal classification where one modality is discrete, e.g. text, and the other is continuous, e.g. visual representations transferred from a convolutional neural network. In particular, we focus on scenarios where we have to be able to classify large quantities of data quickly. We investigate various methods for performing multi-modal fusion and analyze their trade-offs in terms of classification accuracy and computational efficiency. Our findings indicate that the inclusion of continuous information improves performance over text-only on a range of multi-modal classification tasks, even with simple fusion methods. In addition, we experiment with discretizing the continuous features in order to speed up and simplify the fusion process even further. Our results show that fusion with discretized features outperforms text-only classification, at a fraction of the computational cost of full multi-modal fusion, with the additional benefit of improved interpretability.
Related work
Prior work has studied text classification, multi-modal fusion, and cross-modal applications, while this paper focuses on integrating continuous information into efficient text-classification architectures.
- Text classification: Text classification supports tasks including document retrieval, categorization, sentiment analysis, and topic classification.
- Fusion strategies: Multi-modal fusion research integrates inputs from multiple modalities across tasks such as speech recognition, sensor management, and face recognition.
- Multi-modal NLP: Multi-modal NLP has applied non-textual information to image annotation, captioning, image-text mapping, and visual question answering.
- Multi-modal deep learning: This work extends an efficient neural architecture for text classification to include continuous information and explores methods for combining multi-modal features.
Evaluation
The evaluation uses three datasets spanning medium-sized and very large-scale multi-modal classification settings, with experiments designed to examine accuracy and speed trade-offs.
- Evaluation: Three datasets are used because large-scale multi-modal classification datasets are scarce and accuracy/speed trade-offs require sufficiently large data.
- Food101: Food101 contains web pages with textual recipe descriptions for 101 food labels, each paired with a single image.
- MM-IMDB: MM-IMDB contains movie plot outlines and posters, with the task of multilabel movie-genre prediction.
- FlickrTag and FlickrTag-1: FlickrTag pairs Flickr photographs, usually with short user captions, and predicts user-provided tags.
- FlickrTag and FlickrTag-1: FlickrTag-1 is the first shard, representing one-tenth of the full FlickrTag dataset, used for fusion and speed-versus-accuracy studies.
Approach
The approach extends efficient FastText text classification with continuous or discretized visual features and compares fusion models under accuracy and computational-efficiency trade-offs.
- Model and inputs: The model starts from FastText and adds 2048-dimensional ResNet features, or 512-dimensional ResNet-34 features for FlickrTag datasets.The transferred features come from pre-softmax layers of ImageNet-trained ResNet networks.
- Training: All models minimize negative log likelihood over classes, with network output o, multi-modal input x_n, and label y_n.Hyperparameters are selected on validation data using grid search and softmax loss.
- Continuous fusion: The study compares additive, max-pooling, gated, and bilinear fusion models in increasing order of complexity.Gated fusion uses a sigmoid-based modality attention mechanism, while bilinear fusion models cross-modal associations.
- Discretized features: Continuous models require expensive matrix-vector multiplication and substantial floating-point storage, motivating discretization of continuous features into special tokens.Discretized features are processed in the standard FastText setup and require less storage.
- Discretized features: Product quantization divides vectors into equal-sized subvectors, assigns each to its nearest k-means centroid, and represents the assignments as indexed tokens.A 100-dimensional vector can be divided into ten 10-dimensional subvectors, producing tokens such as (1, N(s1)) through (10, N(s10)).
- Discretized features: Random sample product quantization repeats product quantization over random permutations to retain overlapping semantic information that hard subvector boundaries may lose.Both PQ and RSPQ treat discretized features as reweighted special tokens and run standard FastText.
- Efficiency trade-offs: O(HK) complexity applies to additive, max-pooling, and gated classifiers, whereas bilinear fusion requires O(H^2K) computation.The simpler models produce hidden layers of the same size as standard FastText; max-pooling and gating require one extra operation.
Results
Across the evaluated tasks, multi-modal models consistently outperform text-only and continuous-only baselines. Bilinear-gated fusion maximizes accuracy, while discretized features provide a much faster alternative with interpretability benefits.
- Multi-modal models always outperform standard FastText and the continuous-only approach, regardless of fusion type.
- The bilinear-gated model achieves the highest accuracy across all three tasks, although it is also the most complicated and least efficient.
- For speed-sensitive settings, additive and max-pooling models provide reasonably high accuracy, while discretized models further simplify and accelerate fusion.
- Around one minute is required to train discretized methods on FlickrTag-1, compared with around one hour for bilinear models.
- A 2.7% accuracy increase from FastText to RSPQ on full FlickrTag corresponds to 16778 additional correctly classified test documents.
- Nearest-neighbor clusters for quantized features expose interpretable visual concepts, including donuts, crème brûlée, and Japanese food.
Conclusion & Outlook
The paper compares fusion methods for large-scale multi-modal classification and finds that discretized features improve speed while retaining multimodal benefits and interpretability.
- The bilinear-gated model achieves the highest accuracy across the three tasks.
- Simpler additive and max-pooling models provide reasonably high accuracy at higher speed.
- Discretized multi-modal features speed up the model further while preserving the benefits of multimodal information.
- Discretized features also make it possible to interpret what visual information the classifier uses for its decisions.
- The authors present the method as a useful baseline for further work in multi-modal classification.