PaperlayerView exact revision on arXiv

Source-linked AI summary

Routing with Self-Attention for Multimodal Capsule Networks

Kevin Duarte, Brian Chen, Nina Shvetsova, Andrew Rouditchenko, Samuel Thomas, Alexander Liu, David Harwath, James Glass, Hilde Kuehne, Mubarak Shah

arXiv:2112.00775v1cs.CV

TL;DR

Capsule networks had mainly been used in small-scale, fully supervised settings with clean data, and conventional routing is computationally expensive. This paper introduces self-attention routing for multimodal capsules and reports improved comparisons with routing methods alongside competitive performance across four datasets and two zero-shot tasks.

  • Problem

    Capsule networks had mainly been used in small-scale, fully supervised settings with clean data, while conventional routing is computationally expensive.

    Source: §1 · p. 2, §3.1 · p. 3

  • Method

    The network uses self-attention to route multimodal capsules, select relevant representations, and map them into a joint embedding space.

    Source: §1 · p. 2

  • Results

    The network improves performance over existing routing mechanisms and achieves competitive performance across four datasets in two zero-shot tasks.

    Source: §1 · p. 2, §5 · p. 9

  • Takeaways & Limitations

    The evaluation demonstrates the approach's effectiveness on zero-shot video retrieval and action localization.

    Source: §1 · p. 2, §5 · p. 9

    Capsule applications had mainly remained small-scale and fully supervised on clean data.

    Source: §1 · p. 2, §3.1 · p. 3

Abstract

from arXiv · show

The task of multimodal learning has seen a growing interest recently as it allows for training neural architectures based on different modalities such as vision, text, and audio. One challenge in training such models is that they need to jointly learn semantic concepts and their relationships across different input representations. Capsule networks have been shown to perform well in context of capturing the relation between low-level input features and higher-level concepts. However, capsules have so far mainly been used only in small-scale fully supervised settings due to the resource demand of conventional routing algorithms. We present a new multimodal capsule network that allows us to leverage the strength of capsules in the context of a multimodal learning framework on large amounts of video data. To adapt the capsules to large-scale input data, we propose a novel routing by self-attention mechanism that selects relevant capsules which are then used to generate a final joint multimodal feature representation. This allows not only for robust training with noisy video data, but also to scale up the size of the capsule network compared to traditional routing methods while still being computationally efficient. We evaluate the proposed architecture by pretraining it on a large-scale multimodal video dataset and applying it on four datasets in two challenging downstream tasks. Results show that the proposed multimodal capsule network is not only able to improve results compared to other routing techniques, but also achieves competitive performance on the task of multimodal learning.

1. Introduction

The paper adapts capsule networks for large-scale multimodal learning by replacing costly routing with self-attention. It evaluates the resulting network on four datasets across two zero-shot downstream tasks.

1. Introduction

  • Capsules can capture relationships between low-level feature activations and higher-level concepts, but prior use was mainly fully supervised with clean data.Learning multimodal joint embeddings also requires relating semantic concepts across distinct representations.Source: §1 · pp. 1–2
  • Self-attention routing selects relevant capsules from lower-level representations to form higher-level capsules and activations.The mechanism uses key-query-value tuples and is described as more scalable than standard dynamic and EM routing.Source: §1 · p. 2
  • The network improves on existing routing mechanisms and achieves competitive performance across four datasets in two zero-shot tasks.The authors also report state-of-the-art results on challenging benchmark tasks.Source: §1 · p. 2
  • The network maps selected capsules into a joint multimodal embedding space using a contrastive loss.It is pretrained on HowTo100M, which contains 1.2 million YouTube instruction videos, then evaluated on retrieval and action localization.Source: §1 · p. 2
  • The authors present capsule architectures for large-scale multimodal data without human annotation and propose a novel self-attention routing mechanism.They frame this as a first use of capsule architectures for large-scale multimodal data without human annotation.Source: §1 · p. 2

2. Related Work

Prior multimodal work commonly used contrastive learning, sometimes augmented with explicit clustering, while capsule research addressed hierarchical grouping at additional routing cost. This work uses self-attention routing to group concepts within the network.

Multimodal Learning

  • Multimodal methods commonly pretrain on large datasets and fine-tune, while some apply models trained on noisy data directly to downstream tasks.Contrastive losses are used in direct-training approaches; some methods add clustering, usually with an additional Sinkhorn or K-means stage.Source: §2 · p. 2

Capsule Networks

  • Capsules route lower-level representations into higher-level capsules, but dynamic and EM routing can be computationally expensive.Prior work explored alternatives including attention-based routing, self-routing, KDE-based, spectral, and subspace methods.Source: §2 · pp. 2–3
  • Unlike Efficient-CapsNet and SCAEs, the proposed routing separates key-query routing coefficients from votes and derives activations with a linear layer and softmax.The authors report greater computational efficiency and higher performance than Set Transformer in their targeted setup.Source: §2 · p. 3

3. Multimodal Learning with Capsule Networks

The network maps video, audio, and text features into capsules, routes them with self-attention, and projects activation-weighted capsules into a joint embedding space. Pairwise contrastive losses bring embeddings from the same clip together and separate embeddings from different clips.

3.1. Multimodal Capsule Architecture

  • Capsules represent each modality’s features as pose vectors and existence probabilities, with learned layers producing a set of primary capsules.The poses encode entity properties, while activations represent the probability that an entity exists in the input.Source: §3.1 · p. 3
  • Self-attention replaces iterative routing-by-agreement to find agreement between high-dimensional capsule representations and pass information to higher-level capsules.Capsule poses are weighted by their activations before being used in routing.Source: §3.1 · p. 3
  • Query and key representations determine routing coefficients, while value representations serve as votes for higher-level capsule properties.The resulting higher-level capsule activations are used to select relevant capsules, whose weighted features are concatenated and linearly projected into the joint representation.Source: §3.1 · p. 4, Figure 2 · p. 5
  • Higher-level capsules appear to capture concepts at different granularities, from general categories such as “outdoors” to specific activities such as “paper folding.”The authors report this pattern from analyzing capsule existence probabilities across inputs.Source: §3.1 · p. 4

3.2. Contrastive Multimodal Learning

  • The training objective sums pairwise Masked Margin Softmax losses for video-audio, text-audio, and video-text embeddings.For each modality pair, same-clip embeddings are made more similar and different-clip embeddings less similar; negatives include samples from the same and other videos.Source: §3.2 · pp. 4–5

4. Experimental Evaluation

The evaluation tests zero-shot text-to-video retrieval and temporal action localization to assess how well semantic concepts are identified and grouped across modalities. It compares the proposed approach with prior methods and examines implementation choices, routing, capsule counts, baselines, and qualitative behavior.

4.1. Implementation Details

  • The system uses fixed visual, audio, and text feature extractors and trains on 1.2 million HowTo100M videos for 20 epochs with four V100 GPUs.Unless otherwise stated, it uses C=128 capsules, dimensions d1=32 and d2=256, and a 4096-dimensional joint representation; video-audio-text tuples follow dataset transcription timestamps.Source: §4.1 · pp. 5–6

4.2. Text-To-Video Retrieval

  • Retrieval performs strongly against prior approaches on YouCook2 and MSR-VTT; adding audio boosts YouCook2, while modality mismatch can reduce performance (Table 1).The authors note that dropping the misaligned domain can improve MSR-VTT performance.Source: §4.2 · p. 6, Table 1 · p. 6

4.3. Temporal Action Localization

  • With R152+RX101 features, localization performs strongly across datasets and metrics; S3D-G recall improves on CrossTask, while results beat supervised and weakly supervised baselines on CrossTask and Mining YouTube, respectively (Table 2).The reported comparisons include MIL-NCE [36] and UniVL [32] for CrossTask recall, the supervised baseline on CrossTask, and the weakly supervised approach [27] on Mining YouTube.Source: §4.3 · pp. 6–7, Table 2 · p. 6

4.4. Ablations

  • At C=64 and d1=d2=16, self-attention and dynamic routing are least unstable, and the proposed method beats dynamic routing with fewer computational resources (Table 3).Routing generally outperforms the corresponding no-routing baselines, while EM routing suffers greatly from training instability.Source: §4.4 · p. 7, Table 3 · p. 7
  • Increasing capsule count generally improves performance, but gains diminish at C≥128; runtime remains below iterative routing methods even at large counts (Table 4).The capsule-count experiments use d1=32 and d2=256.Source: §4.4 · p. 7, Table 4 · p. 7
  • The capsule network outperforms fully connected and self-attention baselines; self-attention beats the fully connected baseline on MSR-VTT but remains below the proposed method (Table 5).The baseline self-attention model groups input activations into 128 equal-length vectors before applying a multi-head self-attention layer.Source: §4.4 · p. 7, Table 5 · p. 7

4.5. Qualitative Analysis

  • Routing retrieves visually varied videos, including for a specific bun-condiment query, while capsule activations represent distinct concepts such as games, cooking, vegetables, and cars across datasets.The activation examples examine videos with high activation for individual capsules.Source: §4.5 · p. 8, Figure 3 · p. 8, Figure 4 · p. 8

5. Conclusion

The paper presents a multimodal capsule network that maps entities from different modalities into a joint embedding space, using scalable self-attention routing for noisy video data. It reports improved performance over previous routing methods and evaluation across two zero-shot tasks and four datasets.

  • The network models entities within modalities and maps them into a joint embedding space.Source: §5 · p. 9
  • A scalable self-attention-based capsule routing mechanism is presented for learning from large amounts of noisy video data.Source: §5 · p. 9
  • The evaluation spans two downstream zero-shot tasks across four datasets, and the method outperforms previous routing methods on this task.Source: §5 · p. 9

A. Additional Ablations

The architecture shares weights across modalities after extracting the initial capsules. The authors report fewer learned parameters and improved representations, with improved performance on data more closely related to training, such as YouCook2.

B. Self-attention Architectural Details

Self-attention routing uses linear projections to form query, key, and value matrices from input capsules. Its output is normalized and processed through linear layers, dropout, and a residual connection to produce output capsule features.

C. Additional Qualitative Results

Qualitative retrieval examples show visually varied matches and comparisons among routing, no-routing, and MIL-NCE. Capsule activations appear consistent across datasets for represented concepts, while concepts absent from YouCook2 yield seemingly random highest-activation videos.

Related papers

Loading 2112.00775v1…