Learning Modality-Invariant Representations for Speech and Images

• The paper aims to map speech audio and images into a shared semantic embedding space that captures content independently of modality.This goal is motivated by humans associating different sensory inputs with the same concept.Source: §1 · p. 1
• Inputs are mapped to the mean and log variance of diagonal Gaussian posterior distributions, from which semantic embeddings are sampled.Source: §1 · p. 1
• The similarity loss makes embeddings from paired audio and images more similar than embeddings from mismatched pairs.Source: §1 · p. 1
• A VAE-inspired KL regularizer drives posterior distributions toward a unit Gaussian to reduce modality information while preserving semantic information.The authors report that modality information tends to be filtered out before semantic information as regularization increases.Source: §1 · p. 1

• Unsupervised multimodal learning uses co-occurring sensory inputs instead of costly labels and supports cross-modality transfer learning and information retrieval.Cross-modality retrieval searches for related instances in another modality, while transfer learning uses labels from one modality to predict another.Source: §2 · pp. 1–2
• Prior methods include teacher-student audio-visual transfer, shared semantic spaces for image-caption retrieval, and adversarial modality-invariant representations [4].Source: §2 · p. 2
• The proposed approach changes Harwath et al.’s deterministic embeddings into samples from diagonal Gaussian distributions and adds posterior regularization.Source: §2 · p. 2
• Like Saito et al. [4], the method moves embeddings toward a unit Gaussian, but the authors report deceiving a discriminator without gradient reversal.The paper identifies modality-invariant speech embeddings as an underexplored contribution.Source: §2 · p. 2

Similarity objective

• Paired images and audio are optimized so co-occurring inputs have higher representation similarity than non-co-occurring inputs.The similarity objective increases the margin between positive and negative pairs.Source: §3 · p. 2

Probabilistic embeddings

• Instead of deterministic embeddings, each encoder predicts the mean and log variance of a diagonal Gaussian, then samples semantic embeddings from that posterior.This makes the visual and audio encoders nondeterministic.Source: §3 · p. 2

Information-gain regularization

• The information-gain loss averages KL divergence between predicted embedding posteriors and the unit-Gaussian prior.This regularizes the information encoded in semantic embeddings.Source: §3 · p. 2

Model structure

• Figure 1 illustrates the high-level process by which the visual and audio inputs are encoded and processed.The supplied figure materials identify the model structure and its illustration in Figure 1.Source: §3 · p. 2, Figure 1 · p. 3

Total objective

• The total objective combines similarity loss, information-gain regularization, and weight decay with tunable coefficients.LWD is the sum of Frobenius norms of weight matrices and convolutional kernels.Source: §3 · p. 3

MNIST

• MNIST provides 60K training and 10K test images of 28x28 8-bit grayscale handwritten digits.Pixel values are preprocessed to lie between 0 and 1.Source: §4 · p. 3

TIDIGITS

• TIDIGITS provides 6,456 training, 1,076 test, and 1,076 validation utterances after retaining single-number recordings from men, women, and children.The “oh” and “zero” labels are combined into one class for cluster analysis.Source: §4 · p. 3

Audio preprocessing

• Audio is represented with 80-dimensional log mel-filterbank features and standardized one-second spectrogram inputs.Spectrograms use 25 ms windows, 10 ms frame shifts, and padding or center cropping to 100 frames.Source: §4 · p. 3

Model configuration

• Convolutional networks predict posterior parameterizations for both image and audio inputs, using a 128-dimensional embedding space.The encoders output mean and log variance vectors for the posterior distributions.Source: §5 · p. 3

Experimental comparison

• The experiment compares embedding spaces with λIG = 0 against spaces with λIG > 0 to assess information-gain regularization.Both conditions minimize the stated total loss on MNIST and TIDIGITS.Source: §5 · p. 3

Sampling and batches

• Each batch uses 128 distinct image-audio pairs and 16 sampled embeddings per input, producing 2,048 embedding pairs.Positive pairings match corresponding sampled embeddings.Source: §5 · p. 3

Negative sampling

• Negative samples may come from any digit class because excluding same-class negatives would require ground-truth class knowledge within the batch.The authors describe this restriction as equivalent to giving the network an oracle for batch labels.Source: §5 · p. 3

Training

• The model trains for 100 epochs, with each epoch covering the larger MNIST training set once and requiring about 35 minutes on a TitanX GPU.The reported training duration applies to the described experimental setup.Source: §5 · p. 4

Evaluation

• The evaluation samples 16 embeddings per unseen test input, clusters them with K-means using k = 10, and measures cluster purity from ground-truth digit labels.Modality information is evaluated separately using an SVM trained on 2,152 embeddings, with 1,600 for training and the remainder for testing.Source: §6 · p. 4

Visualization

• The study also uses shared two-dimensional t-SNE projections for audio and image embeddings within each hyperparameter setting to visualize their relative organization.The projections use 200 samples per modality and color points by class label.Source: Figure 2 · p. 4, §6 · p. 5

Results

• At λIG = 0, cluster purity is 0.525 and modality classification reaches 100% accuracy, indicating separate modality-specific clusters rather than modality-invariant embeddings.The analysis attributes the low purity partly to two clusters per digit, one for each modality, when K-means uses k = 10.Source: §6 · p. 5
• The KL-regularization term improves modality invariance, with the best setting producing SVM modality-classification accuracy of 55.4%, near the 50% random baseline.The comparison is reported in Table 1 and is substantially closer to modality-invariant behavior than the LSim-only space.Source: Table 1 · p. 4, §6 · p. 5
• λIG ≈ 2.15 · 10^-2 is the empirically observed cutoff where further regularization begins reducing cluster purity by restricting semantic information.Figure 2 qualitatively shows under-regularized, well-regularized, and over-regularized embedding spaces.Source: §6 · p. 5, Figure 3 · p. 5

Conclusions

• Sampling encodings and regularizing their posterior distributions yields a more modality-invariant semantic space for speech and images.The approach deceives a modality-classifying SVM without gradient reversal or an adversarial training setup.Source: §7 · p. 5
• The authors suspect LIG may be useful in other generative-adversarial approaches to domain- or modality-invariant embeddings.This is presented as a suspected broader application rather than an established result.Source: §7 · p. 5

Future work

• Future work includes combining VAE and GAN techniques and replacing dot-product similarity with symmetric divergence between variational distributions.The proposed divergence-based loss is intended to provide a more probabilistically grounded formulation with potentially broader implications.Source: §7 · p. 5