PaperlayerView exact revision on arXiv

Source-linked AI summary

Efficient Large-Scale Multi-Modal Classification

D. Kiela, E. Grave, A. Joulin, T. Mikolov

arXiv:1802.02892v1cs.CLcs.AIcs.CV

TL;DR

The paper addresses efficient classification of large-scale data combining discrete text with continuous visual information. It compares fusion strategies and discretizes continuous features to simplify and accelerate fusion. Multi-modal methods improve over text-only classification, while discretized fusion retains gains with lower computational cost and improved interpretability.

Abstract

from arXiv · show

While the incipient internet was largely text-based, the modern digital world is becoming increasingly multi-modal. Here, we examine multi-modal classification where one modality is discrete, e.g. text, and the other is continuous, e.g. visual representations transferred from a convolutional neural network. In particular, we focus on scenarios where we have to be able to classify large quantities of data quickly. We investigate various methods for performing multi-modal fusion and analyze their trade-offs in terms of classification accuracy and computational efficiency. Our findings indicate that the inclusion of continuous information improves performance over text-only on a range of multi-modal classification tasks, even with simple fusion methods. In addition, we experiment with discretizing the continuous features in order to speed up and simplify the fusion process even further. Our results show that fusion with discretized features outperforms text-only classification, at a fraction of the computational cost of full multi-modal fusion, with the additional benefit of improved interpretability.

Related work

Prior work has studied text classification, multi-modal fusion, and cross-modal applications, while this paper focuses on integrating continuous information into efficient text-classification architectures.

Text classification

  • Text classification supports tasks including document retrieval, categorization, sentiment analysis, and topic classification.Source: Related work · p. 1

Fusion strategies

  • Multi-modal fusion research integrates inputs from multiple modalities across tasks such as speech recognition, sensor management, and face recognition.Source: Related work · p. 2

Multi-modal NLP

  • Multi-modal NLP has applied non-textual information to image annotation, captioning, image-text mapping, and visual question answering.Source: Related work · p. 2

Multi-modal deep learning

  • This work extends an efficient neural architecture for text classification to include continuous information and explores methods for combining multi-modal features.Source: Related work · p. 2

Evaluation

The evaluation uses three datasets spanning medium-sized and very large-scale multi-modal classification settings, with experiments designed to examine accuracy and speed trade-offs.

Evaluation

  • Three datasets are used because large-scale multi-modal classification datasets are scarce and accuracy/speed trade-offs require sufficiently large data.Source: Evaluation · p. 2

Food101

  • Food101 contains web pages with textual recipe descriptions for 101 food labels, each paired with a single image.Source: Food101 · p. 2

MM-IMDB

  • MM-IMDB contains movie plot outlines and posters, with the task of multilabel movie-genre prediction.Source: MM-IMDB · p. 2

FlickrTag and FlickrTag-1

Approach

The approach extends efficient FastText text classification with continuous or discretized visual features and compares fusion models under accuracy and computational-efficiency trade-offs.

Model and inputs

  • The model starts from FastText and adds 2048-dimensional ResNet features, or 512-dimensional ResNet-34 features for FlickrTag datasets.The transferred features come from pre-softmax layers of ImageNet-trained ResNet networks.Source: Approach · p. 2

Training

  • All models minimize negative log likelihood over classes, with network output o, multi-modal input x_n, and label y_n.Hyperparameters are selected on validation data using grid search and softmax loss.Source: Approach · p. 3, Hyperparameters and training · p. 4

Continuous fusion

  • The study compares additive, max-pooling, gated, and bilinear fusion models in increasing order of complexity.Gated fusion uses a sigmoid-based modality attention mechanism, while bilinear fusion models cross-modal associations.Source: Continuous Multi-Modal Models · p. 3

Discretized features

  • Continuous models require expensive matrix-vector multiplication and substantial floating-point storage, motivating discretization of continuous features into special tokens.Discretized features are processed in the standard FastText setup and require less storage.Source: Discretized Multi-Modal Models · p. 3
  • Product quantization divides vectors into equal-sized subvectors, assigns each to its nearest k-means centroid, and represents the assignments as indexed tokens.A 100-dimensional vector can be divided into ten 10-dimensional subvectors, producing tokens such as (1, N(s1)) through (10, N(s10)).Source: Discretized Multi-Modal Models · pp. 3–4
  • Random sample product quantization repeats product quantization over random permutations to retain overlapping semantic information that hard subvector boundaries may lose.Both PQ and RSPQ treat discretized features as reweighted special tokens and run standard FastText.Source: Discretized Multi-Modal Models · p. 4

Efficiency trade-offs

  • O(HK) complexity applies to additive, max-pooling, and gated classifiers, whereas bilinear fusion requires O(H^2K) computation.The simpler models produce hidden layers of the same size as standard FastText; max-pooling and gating require one extra operation.Source: Model complexity · p. 4

Results

Across the evaluated tasks, multi-modal models consistently outperform text-only and continuous-only baselines. Bilinear-gated fusion maximizes accuracy, while discretized features provide a much faster alternative with interpretability benefits.

  • Multi-modal models always outperform standard FastText and the continuous-only approach, regardless of fusion type.Source: Results · p. 4
  • The bilinear-gated model achieves the highest accuracy across all three tasks, although it is also the most complicated and least efficient.Source: Results · p. 5
  • For speed-sensitive settings, additive and max-pooling models provide reasonably high accuracy, while discretized models further simplify and accelerate fusion.Source: Speed · p. 5, Results · p. 5
  • Around one minute is required to train discretized methods on FlickrTag-1, compared with around one hour for bilinear models.Source: Speed · p. 5
  • A 2.7% accuracy increase from FastText to RSPQ on full FlickrTag corresponds to 16778 additional correctly classified test documents.Source: Speed · p. 5
  • Nearest-neighbor clusters for quantized features expose interpretable visual concepts, including donuts, crème brûlée, and Japanese food.Source: Interpretability · p. 5, Table 5 · p. 5

Conclusion & Outlook

The paper compares fusion methods for large-scale multi-modal classification and finds that discretized features improve speed while retaining multimodal benefits and interpretability.

Related papers

Loading 1802.02892v1…