PaperlayerView exact revision on arXiv

Source-linked AI summary

Quantum-inspired Multimodal Fusion for Video Sentiment Analysis

Qiuchi Li, Dimitris Gkoumas, Christina Lioma, Massimo Melucci

arXiv:2103.10572v2cs.MM

TL;DR

The paper addresses multimodal sentiment analysis, where text alone is not enough to judge sentiment. It presents a quantum-theoretic fusion framework and reports performance comparable to state-of-the-art models.

  • Problem

    Text alone is not enough to judge sentiment in multimodal sentiment analysis.

    Source: §1 · p. 2

  • Method

    The paper presents a quantum-theoretic framework for multimodal fusion.

    Source: §1 · p. 5

  • Results

    The model achieves comparable accuracy to state-of-the-art models.

    Source: §1 · p. 6, §9 · p. 48

  • Takeaways & Limitations

    The paper concludes that its quantum-inspired framework obtains performance comparable to state-of-the-art models.

    Source: §9 · p. 48

    The authors note that extracted visual and acoustic features are not high quality and identify cleaner feature extraction as a promising direction.

    Source: §9 · p. 48

Abstract

from arXiv · show

We tackle the crucial challenge of fusing different modalities of features for multimodal sentiment analysis. Mainly based on neural networks, existing approaches largely model multimodal interactions in an implicit and hard-to-understand manner. We address this limitation with inspirations from quantum theory, which contains principled methods for modeling complicated interactions and correlations. In our quantum-inspired framework, the word interaction within a single modality and the interaction across modalities are formulated with superposition and entanglement respectively at different stages. The complex-valued neural network implementation of the framework achieves comparable results to state-of-the-art systems on two benchmarking video sentiment analysis datasets. In the meantime, we produce the unimodal and bimodal sentiment directly from the model to interpret the entangled decision.

1. Introduction

The paper presents a quantum-inspired framework for multimodal sentiment fusion, using superposition to model within-modality word interactions and entanglement to fuse modalities, with sentiment predicted through quantum measurement. It reports accuracy comparable to state-of-the-art systems on CMU-MOSI and CMU-MOSEI and illustrates interpretability through classical explanations of the quantum process and unimodal and bimodal judgments.

1. Introduction

  • Because sentiment can differ across visual-acoustic contexts and modalities can yield different individual tendencies, multimodal sentiment requires a joint decision.The introduction argues that textual clues alone are insufficient and that visual and acoustic signals are also essential.Source: §1 · pp. 2–3
  • The framework offers a principled quantum-inspired approach to multimodal fusion in response to implicit interaction modeling and low interpretability in neural approaches.The paper identifies the unclear modeling of both intra-modal and inter-modal interactions as part of the interpretability problem.Source: §1 · pp. 3–4
  • Quantum superposition captures interactions between words at the feature level, extending its use from text analysis to multimodal word representations.The superposition is instrumented using the multimodal word representation.Source: §1 · p. 5
  • Quantum entanglement fuses unimodal sentiment decisions to represent an overall judgment that cannot be reduced to separate modality-based decisions.The implementation uses an explicit tensor product of unimodal sentiment representations.Source: §1 · pp. 5–6
  • The model predicts final sentiment by measuring multimodal sentence states on sentiment-related aspects and passing their probabilities to a fully connected layer.The paper describes each sentiment-related aspect as an entangled state.Source: §1 · p. 6
  • The model achieves accuracy comparable to state-of-the-art systems on CMU-MOSI and CMU-MOSEI.Source: §1 · p. 6
  • The paper illustrates interpretability by explaining the quantum process from a classical point of view.Source: §1 · p. 6
  • The model presents unimodal and bimodal sentiment judgments implicitly entailed in its entangled sentiment predictions.Source: §1 · p. 6

2. Preliminaries on Quantum Theory

This section introduces quantum-theory concepts underlying the multimodal fusion framework, including superposition, mixtures, measurement, composite systems, entanglement, and subsystem representations. It also presents the reduced density matrix for interpreting how unimodal judgments compose the final sentiment and generating predictions from unimodal and bimodal features.

2. Preliminaries on Quantum Theory

  • Quantum theory describes microscopic events as subspaces in a vector space with projective geometry and uses complex Hilbert spaces and Dirac ket/bra notation.Source: §2 · p. 7

2.1. Superposition

  • Superposition represents a state as a normalized linear combination of mutually orthogonal basis states, with squared amplitude moduli corresponding to measurement probabilities.Source: §2.1 · pp. 7–8

2.2. Mixture

  • A probabilistic ensemble of pure states is represented by a positive semidefinite, unit-trace density matrix, which can represent pure states as well.Source: §2.2 · pp. 8–9

2.3. Measurement

  • Measurement resolves uncertainty into an eigenvalue of a self-adjoint observable, with Born’s rule assigning outcome probabilities from the premeasurement density matrix.Source: §2.3 · p. 9

2.4. Composite Quantum System

  • Composite systems are defined on the tensor product of their component Hilbert spaces, with dimensions equal to the product of the component dimensions.Source: §2.4 · pp. 10–11

2.4.1. Entanglement

  • Entanglement links measurement outcomes across subsystems and describes a composite state that cannot be factored into subsystem states.Source: §2.4.1 · p. 11

2.4.2. Reduced Density Matrix

  • A reduced density matrix represents subsystem A so that measurements on A match measurements M⊗I on the composite state while subsystem B remains unchanged.Source: §2.4.2 · p. 12
  • The reduced density matrix supports interpreting how unimodal judgments compose the final sentiment and generating predictions from any unimodal or bimodal features.Source: §2.4.2 · p. 13

3. Related Works

Prior multimodal sentiment methods often leave interactions implicit or difficult to interpret, while prior quantum-inspired methods applied quantum formalisms to multimodal tasks and text modeling. This work extends these ideas into an end-to-end multimodal framework that explicitly models word-level and cross-modal interactions and exposes unimodal and bimodal decisions.

3.1. Multimodal Sentiment Analysis

  • Decision-level fusion first makes unimodal sentiment decisions, then merges them via averaging, majority voting [31], weighted sum, or a learnable model; it ignores low-level cross-modal interactions, adversely affecting performance.Source: §3.1 · p. 14
  • Feature-level recurrent methods lack separate intra- and inter-modal components, while seq2seq methods correlate cross-modal word representations but largely ignore word-order information.The paper contrasts these approaches with its explicit treatment of both interaction types.Source: §3.1 · p. 16
  • Tensor-based methods mostly combine sentence representations, missing fine-grained word-level cross-modal interactions; across these model families, key operations remain hard to interpret.The paper characterizes their implementations as black-box-like structures without mappings to concrete concepts.Source: §3.1 · p. 16
  • Quantum notions explicitly model word interactions at the feature level and cross-modal interactions at the decision level, while enabling unimodal and bimodal decisions to interpret the trimodal model’s entangled pattern.The framework is described as a quantum-like multi-system state preparation and measurement process.Source: §3.1 · p. 17

3.2. Quantum-inspired Models for Multimodal Tasks

  • Preliminary studies address multimodal tasks with quantum inspiration; Wang et al. build a multimodal image retrieval system whose text–visual correlations use simple statistical methods and that works no better than simple image-and-text concatenation.Source: §3.2 · pp. 17–18
  • The multimodal framework performs word-level fusion end to end and is the first to implement quantum processes in a multimodal context using complex values.The work contributes in both theory and implementation.Source: §3.2 · p. 18

3.3. Quantum-inspired Models for Text Analysis

  • Quantum-inspired text models use superposition and entanglement to represent interactions among linguistic units, including words, sentences, concepts, and documents.Prior approaches also addressed cross-feature interactions within unified textual representations.Source: §3.3 · p. 20
  • Prior quantum-inspired text models were mostly unsupervised, did not support learning from large datasets, and were less competitive than state-of-the-art neural models.The paper identifies this as a limitation of earlier approaches despite their improvements over strong baselines.Source: §3.3 · p. 20
  • The framework extends text-focused quantum methods to multimodal data, treating each modality as a particle and using entanglement to formulate inter-modal interactions.The extension addresses the need to model interactions across modalities in multimodal data.Source: §3.3 · pp. 20–21

4. Problem Formulation and Notation

The formulation treats multimodal sentiment analysis as mapping video segments to sentiment labels. Each segment has textual, visual, and acoustic features aligned by word and padded to a common sequence length.

4. Problem Formulation and Notation

  • A dataset contains N video segments, each associated with text, visual, and acoustic features and a sentiment label.The passage identifies the three modalities and associates each segment with a label.Source: §4 · p. 21
  • The three modalities are aligned by words and segments are zero-padded to produce time-series data of the same length L.Source: §4 · p. 21
  • Each video segment Xi is mapped to its associated sentiment yi.The task is to establish a mapping f from segments to sentiment.Source: §4 · p. 21

5. Quantum-inspired Multimodal Fusion Framework

This section describes a quantum-inspired framework for representing multimodal sentences and deriving sentiment decisions through measurement, including unimodal and bimodal decisions inferred from the trimodal model. It also notes that further details of the sentiment-measurement process are deferred to Section 6.5.

5.1. Multimodal Hilbert Space

  • Textual, visual, and acoustic feature spaces combine into a multimodal Hilbert space whose basis states represent combinations of unimodal feature dimensions.The framework is described as general and adaptable to other modalities.Source: §5.1 · p. 22

5.2. Word State

  • Each word is represented as a product of its textual, visual, and acoustic states, enabling tensor-based fusion and context-dependent multimodal representations.Visual and acoustic representations are word-independent, while the textual representation depends on the word; this allows a word’s multimodal representation to vary across contexts.Source: §5.2 · pp. 22–24

5.3. Sentence State

  • A sentence is a weighted mixture of word states represented by a density matrix, which supports word-level mixture and superposition but is separable across modalities.Because it is a probability measure over the multimodal space, the density matrix can represent different word combinations under different contexts; inter-modal interactions are instead implemented in sentiment decisions.Source: §5.3 · pp. 24–26

5.4. Sentiment Measurement

  • Sentence sentiment is produced by measuring the sentence state against sentiment-related aspects, with each aspect an entangled state across the three modalities.Measurement probabilities over the aspects characterize sentence sentiment; reduced density matrices yield statistically equivalent observables for bimodal and unimodal systems, allowing their decisions to be inferred from trimodal decisions.Source: §5.4 · pp. 26–28
  • Sentiment measurement applies the observable to each sentence representation, and the process details are introduced in Section 6.5.Source: §5.4 · p. 28

6. Complex-valued Network for Multimodal Sentiment Analysis

This section describes the complex-valued network for quantum-inspired multimodal fusion, including how it represents and combines modality features and measures sentiment, and how its learned trimodal model supports interpretation of modality subsets and word combinations.

6.1. Complex-valued Multimodal Word Embedding

  • The text representation E(w) is generated via a deep neural network Dt, while visual and acoustic moduli are word-independent and non-trainable.The values π, π/2, and 0 map to -1, 0, and 1, indicating negative, neutral, and positive sentiment.Source: §6.1 · pp. 30–31

6.2. Mixture

  • Global weighting and local mixture address word importance and sentiment-bearing contexts by weighting words across modalities and mixing words within each context.Sliding windows of varying lengths produce local n-gram density matrices, with context word weights softmax-normalized so each mixture yields a legal density matrix.Source: §6.2 · pp. 31–32

6.3. Measurement

  • A shared observable is applied to local-context density matrices to generate probability sets and identify contexts discriminative for sentiment classification.Source: §6.3 · p. 33
  • Measurement learns K sentiment-related eigenstates without requiring orthogonality, then row-wise max-pools local-context likelihoods into a sentence representation for Dout.The eigenstates are not hard-coded as orthogonal because sentiment aspects may be dependent and enforcing orthogonality during training is computationally costly.Source: §6.3 · pp. 33–34

6.4. Network Learning

  • Network learning initializes text from word embeddings and sentiment-related arguments, pretrains visual/acoustic arguments on unimodal data, and trains moduli and arguments separately with unit-norm moduli.Intermediate complex-valued layers use real and imaginary parts separately for backpropagation.Source: §6.4 · p. 34

6.5. Network Interpretation

  • The learned trimodal network directly yields unimodal/bimodal predictions and word- or context-level sentiment judgments without retraining, exposing the original model’s behavior.The authors identify this property and present exact subset prediction results in Section 8.3.Source: §6.5 · pp. 35–36

7. Experimental Setup

The experiments use CMU-MOSI and CMU-MOSEI, with word-aligned multimodal features and sentiment labels, and compare against early- and late-fusion LSTM baselines.

7. Experimental Setup

  • CMU-MOSI contains 2,199 short monologue clips, while CMU-MOSEI contains 22,777 YouTube video clips.Both datasets use human sentiment ratings from -3 (strongly negative) to 3 (strongly positive).Source: §7 · p. 37
  • Visual and acoustic features are aligned with words, and each dataset sample receives a human-annotated sentiment score.CMU-MOSI acoustic and visual features are sampled at 12.5 Hz and 15 Hz, respectively; CMU-MOSEI unaligned sequences use 20 Hz acoustic and 15 Hz visual sampling.Source: §7 · p. 37
  • The authors reconstructed original words by cross-checking GloVe vectors and merged CMU-MOSEI versions to recover missing words.The MulT repository version contained word vectors rather than original words; the SDK version supplied words but omitted some important words in many samples.Source: §7 · p. 38
  • For the final datasets, each word has a 35-dim feature and a 74-dim acoustic feature, and sentence lengths are normalized; no normalization target is stated.Source: §7 · p. 38
  • The comparisons include EF-LSTM, which concatenates modalities at each timestamp, and LF-LSTM, which encodes each modality separately before concatenating sentence states.Both baselines use LSTMs to construct sentence-level multimodal representations for sentiment prediction.Source: §7 · pp. 38–39

3. Multi-Attention Recurrent Network (MARN) [4]. This model

This section describes comparison systems for multimodal sentiment analysis and the shared evaluation setup used to compare them.

3. Multi-Attention Recurrent Network (MARN) [4]. This model

  • MARN combines inter-modal dynamics at each timestamp with unimodal LSTMs to form a multimodal sentence representation for sentiment classification.MFN instead builds a multimodal gated memory whose cell is updated as hidden states evolve.Source: §3 · p. 39
  • TFN forms a multimodal sentence representation from the tensor product of unimodal vectors, while LMF applies low-rank weights to produce a low-dimensional vector.The TFN product includes interactions among subsets of modalities; LMF uses tensor decomposition to express the product through weighted unimodal features.Source: §3 · p. 39
  • MulT uses inter-modal attention to enrich unimodal representations, which are concatenated for sentiment prediction.The baselines collectively cover early- and late-fusion LSTMs, dynamic fusion, tensor-based approaches, and a sequence-to-sequence model; MFM is excluded as a generic structure.Source: §3 · p. 40
  • For fair comparison, models use uniform experimental settings; all except LMF share a sentiment-prediction network based on multimodal sentence representations, with two fully connected layers using ReLU activation.Source: §3 · p. 40
  • Evaluation uses 7-class and binary accuracy, F1, mean absolute error, and correlation with human annotation.Word embeddings are trainable and initialized with GloVe; models using pretrained contextualized embeddings are excluded to keep external-corpus use identical.Source: §3 · p. 41

8. Results and Discussion

The section reports QMF’s competitive performance on CMU-MOSI and CMU-MOSEI and uses ablations to assess its components and design choices. It also examines how unimodal and bimodal predictions contribute to the entangled multimodal decision, with text most influential and visual and acoustic features providing complementary information.

8.1. Performance on Multimodal Sentiment Analysis

  • On CMU-MOSI, QMF achieves the best MAE and Correlation and ranks second in binary accuracy and F1; on CMU-MOSEI, it underperforms the best model by a relative difference of less than 2.5% across all metrics.Source: §8.1 · p. 42
  • Advanced LSTM-, tensor-, and seq2seq-based models perform similarly and outperform EF-LSTM and LF-LSTM, with smaller gaps that the authors attribute to trainable word embeddings.They suggest trainable embeddings can let even simple networks perform acceptably, whereas fixed embeddings make neural-structure complexity more consequential.Source: §8.1 · p. 43

8.2. Ablation Study

  • On CMU-MOSEI, all QMF ablations show a notable performance drop, supporting the use of complex-valued components, argument initialization, local mixture, and max-pooling.The study changes one component at a time from the best settings to assess its effect.Source: §8.2 · pp. 43, 46, Table 4 · p. 46
  • The QMF-real discrepancy indicates that complex values contribute through combining features in the modulus and argument parts, not merely by doubling parameters.This combination leads to a positive performance gain for the full model.Source: §8.2 · p. 46

8.3. Interpretation of Multimodal Decision

  • On CMU-MOSEI, QMF predicts sentiment best with all three modalities; unimodal text predictions are reasonably accurate, while visual and acoustic predictions are weak.Combining text with visual or acoustic features improves judgments over the unimodal predictions.Source: Table 5 · p. 47, §8.3 · p. 47
  • The interpretation indicates that text carries most sentiment information, while visual and acoustic modalities are less relevant alone but provide complementary information that boosts predictions.Source: §8.3 · pp. 47–48
  • The process reduces the visual and acoustic modalities’ abilities to predict sentiment independently.Source: §8.3 · p. 48
  • QMF is comparable to SOTA baselines on both CMU-MOSEI and CMU-MOSI.Source: §8.3 · p. 48
  • Ablations show that complex-valued representations, mixtures of local contexts, global-mixture measurement results, and the initialization method each positively influence model effectiveness.Source: §8.3 · p. 48
  • The authors present CMU-MOSEI unimodal and bimodal predictions to examine how unimodal decisions compose an entangled multimodal decision.Source: §8.3 · p. 48

9. Conclusion and Future Work

The paper presents a quantum-inspired framework for multimodal sentiment analysis that models intra-modal and inter-modal interactions at different levels. It reports performance comparable to state-of-the-art models and identifies feature quality and measurement design as areas for future work.

9. Conclusion and Future Work

  • The framework explicitly models intra-modal interactions at the feature level and inter-modal interactions at the decision level using an end-to-end complex-valued neural network.The framework is trained in a supervised way.Source: §9 · p. 48
  • The model achieves performance comparable to state-of-the-art models and provides an interpretation approach for multimodal interactions from quantum and classical perspectives.The authors also report producing unimodal and bimodal sentiment directly from the model to interpret the entangled decision.Source: §9 · p. 48
  • The quality of extracted visual and acoustic features is not high; extracting clean, sentiment-sensitive non-textual features is proposed as future work.The authors say this could allow the value of models in the field to be appropriately judged.Source: §9 · p. 48
  • The measurement component's inconsistency with quantum theory remains to be handled.The authors suggest incorporating a legal observable with an orthogonal eigenbasis and concrete meanings for observed values into the end-to-end network.Source: §9 · p. 49
Loading 2103.10572v2…