PaperlayerView exact revision on arXiv

Source-linked AI summary

Tailor Versatile Multi-modal Learning for Multi-label Emotion Recognition

Yi Zhang, Mingyuan Chen, Jundong Shen, Chongjun Wang

arXiv:2201.05834v1cs.CVcs.MM

TL;DR

MMER methods often use shared modality spaces and identical label representations, leaving modality diversity and relationships among modalities and labels underused. TAILOR refines common and private modality representations, fuses them progressively, and uses label semantics to tailor label representations. On CMU-MOSEI, it surpasses state-of-the-art methods in aligned and unaligned settings, except on recall.

  • Problem

    Prior MMER methods often project modalities into a shared space and learn identical representations for labels, leaving modality diversity, label semantics, and modality-label relationships underused.

    Source: Introduction · pp. 1–2

  • Method

    TAILOR adversarially refines common and private modality representations, progressively fuses them, and uses label semantics to guide tailored representations for each label.

    Source: Introduction · p. 2

  • Results

    TAILOR significantly surpasses state-of-the-art methods on all evaluation metrics except recall (R), with experiments on CMU-MOSEI in aligned and unaligned settings.

    Source: Introduction · p. 2, Experimental Results and Analysis · p. 6

  • Takeaways & Limitations

    Experiments and analysis in aligned and unaligned settings verify TAILOR’s effectiveness and generalization.

    Source: Conclusion · p. 7

Abstract

from arXiv · show

Multi-modal Multi-label Emotion Recognition (MMER) aims to identify various human emotions from heterogeneous visual, audio and text modalities. Previous methods mainly focus on projecting multiple modalities into a common latent space and learning an identical representation for all labels, which neglects the diversity of each modality and fails to capture richer semantic information for each label from different perspectives. Besides, associated relationships of modalities and labels have not been fully exploited. In this paper, we propose versaTile multi-modAl learning for multI-labeL emOtion Recognition (TAILOR), aiming to refine multi-modal representations and enhance discriminative capacity of each label. Specifically, we design an adversarial multi-modal refinement module to sufficiently explore the commonality among different modalities and strengthen the diversity of each modality. To further exploit label-modal dependence, we devise a BERT-like cross-modal encoder to gradually fuse private and common modality representations in a granularity descent way, as well as a label-guided decoder to adaptively generate a tailored representation for each label with the guidance of label semantics. In addition, we conduct experiments on the benchmark MMER dataset CMU-MOSEI in both aligned and unaligned settings, which demonstrate the superiority of TAILOR over the state-of-the-arts. Code is available at https://github.com/kniter1/TAILOR.

Introduction

Existing multi-modal and multi-label approaches leave modality diversity and label-specific characteristics insufficiently represented. TAILOR addresses these challenges by refining modality representations, fusing them progressively, and using label semantics to guide tailored label representations.

Introduction

  • The multi-modal learning challenge is to integrate shared information while preserving modality diversity and interactively aligning modality distributions for fusion.Prior methods that project modalities into a shared space can neglect distinctive emotional characteristics revealed from different perspectives.Source: Introduction · p. 1
  • Multi-label learning must encode each label’s inherent and discriminative characteristics in both feature and label spaces.Approaches that learn label-specific representations independently may ignore relationships among labels.Source: Introduction · pp. 1–2
  • TAILOR uses an adversarial network constrained by common semantics and orthogonality to extract intra-modal commonality and diversity, then fuses modalities with a granularity-based BERT-like encoder.The paper describes this approach as placing less emphasis on pre-training.Source: Introduction · p. 2
  • TAILOR uses self-attention to exploit high-order label correlations and a Transformer decoder to align fused multi-modal representations with label semantics for tailored label representations.The decoder is intended to construct each label’s representation under the guidance of its semantics.Source: Introduction · p. 2
  • TAILOR adversarially models commonality and diversity across modalities and enhances the discriminative capacity of label representations.It progressively fuses private and common representations and incorporates label semantics; experiments on CMU-MOSEI report strong performance in aligned and unaligned settings.Source: Introduction · p. 2

Related Work

Prior work explores multi-modal fusion through aggregation and cross-modal adaptation, while multi-label research studies label correlations and label-specific features. MMER work also examines relationships between modalities, features, and labels.

Related Work

  • Multi-modal fusion research includes aggregation using concatenation, tensor, or attention, and alignment-based adaptation; joint embeddings can neglect modality specificity.Other approaches combine modality-specific streams or preserve common information alongside modality-specific patterns.Source: Related Work · p. 2
  • Multi-label methods model label correlations or learn label-specific representations using discriminative characteristics such as visual or textual attention.The cited work presents both as strategies for multi-label learning.Source: Related Work · p. 2
  • MMER research has modeled modality-to-label and feature-to-label dependence alongside label correlations.Source: Related Work · p. 2

Methodology

TAILOR formulates MMER as assigning relevant emotion-label sets from visual, audio, and text features. Its methodology context-enriches each modality, separates common and private representations, and uses label semantics to generate tailored representations for classification.

Methodology

  • MMER learns to assign a set of relevant emotion labels to an unseen video from its visual, audio, and text features.The feature spaces are sequences, and each video has a corresponding set of relevant labels.Source: Methodology · p. 2

Uni-modal Extractor

  • Separate Transformer encoders enrich visual, audio, and text features with sequence-level context before producing modality embeddings.The resulting embeddings share dimension d and sequence length τ.Source: Uni-modal Extractor · pp. 2–3

Adversarial Multi-modal Refinement

  • A shared generator aligns common representations, while modality-specific projections form private representations; adversarial training confuses the discriminator on common features and distinguishes private modality sources.The discriminator estimates the source modality of a representation.Source: Adversarial Multi-modal Refinement · p. 3
  • An orthogonal loss penalizes redundancy in the common and private representations, and the final objective combines classification, adversarial, orthogonal, and common-semantic losses.The overall loss weights the additional terms with α, β, and γ.Source: Adversarial Multi-modal Refinement · pp. 4–5
  • The Cross-Modal Encoder sums two modality representations with modality-token and positional embeddings, then feeds the result to a Transformer encoder.Source: Figure 3 · p. 4
  • Label semantics guide multi-head attention over joint multimodal representations to produce tailored per-label representations for sigmoid classification with binary cross-entropy.The tailored representations are generated by a feed-forward network with residual connections and layer normalization.Source: Adversarial Multi-modal Refinement · p. 5

Experiment

The experiments evaluate TAILOR on CMU-MOSEI in aligned and unaligned settings, then examine its components, adversarial representations, and learned label correlations.

Experimental Setup

  • CMU-MOSEI contains 22,856 segments from 1,000 speakers, with six emotion labels and visual, audio, and text features evaluated in aligned and unaligned settings.The experiments use 35-dimensional visual, 74-dimensional audio, and 300-dimensional text features.Source: Experimental Setup · p. 5
  • The evaluation uses Accuracy, Micro-F1, Precision, and Recall, and Table 2 compares TAILOR with state-of-the-art methods on aligned and unaligned sequences.The table caption says the best performance for each criterion is bolded; supplied passages do not give its values.Source: Experimental Setup · p. 5, Table 2 · p. 6

Experimental Results and Analysis

  • Ablations show the weakest variant removes adversarial multi-modal learning, while joint commonality and diversity, fusion order, modality tokens, and label-specific learning affect performance.The label-specific variants incorporating label correlations and label-modal dependence outperform the variant without label-specific learning.Source: Experimental Results and Analysis · p. 6
  • Over epochs, private adversarial and overall losses decrease nearly monotonically and converge smoothly, while the common adversarial loss first fluctuates and then stabilizes.The authors report that neither common nor private adversarial learning can improve at the endpoint.Source: Experimental Results and Analysis · p. 7, Figure 4 · p. 7
  • Adversarial training aligns common representations across modalities, while the orthogonality constraint separates the common subspace from private representations.Without adversarial training, common and private distributions sometimes overlap; the visualized distributions are from aligned CMU-MOSEI.Source: Experimental Results and Analysis · p. 7, Figure 6 · p. 7
  • Across attention heads, angry correlates with fear, disgust, and sad, while surprise is often highly correlated with happy.The displayed label-correlation visualization covers four attention heads.Source: Experimental Results and Analysis · p. 7, Figure 7 · p. 7

Conclusion

TAILOR combines adversarial refinement with gradual multimodal fusion and label-guided representations. Experiments in aligned and unaligned settings support its effectiveness and generalization.

Conclusion

  • TAILOR consists of a uni-modal extractor, adversarial multi-modal refinement, and label-modal alignment.The modules refine private and common representations, gradually fuse modalities, and use label semantics to guide label-specific representations.Source: Conclusion · p. 7
  • Gradual fusion and label-guided representation construction are the stated roles of TAILOR’s modules.The label semantics guide construction of a tailored representation for each label.Source: Conclusion · p. 7
  • Experiments and analysis in aligned and unaligned settings verify TAILOR’s effectiveness and generalization.Source: Conclusion · p. 7
Loading 2201.05834v1…