Disentangling by Partitioning: A Representation Learning Framework for Multimodal Sensory Data

• Disentangled representations identify underlying explanatory factors in sensory data and support tasks including zero-shot learning, novelty detection, and transfer learning [25] [8] [16].Source: §1 · p. 1
• VAEs combine probabilistic generative models with neural networks to learn latent representations directly from raw images, speech, and videos [23] [36] [17] [6].Source: §1 · p. 1
• Multimodal synchronization supervises disentanglement, but prior work mainly retained shared factors and discarded modality-dependent information [11] [20] [27] [37].The paper targets representations that preserve both shared and modality-dependent factors in separate latent variables.Source: §1 · p. 2
• PVAE partitions multimodal explanatory factors into modality-invariant semantic and modality-dependent style variables, achieving over 99% classification accuracy on both modalities.The model is evaluated on two multimodal spoken/written digit datasets using quantitative and qualitative analyses.Source: §1 · p. 2

2.1 Proposed framework: partitioned variational autoencoder (PVAE)

• PVAE partitions explanatory factors into one shared semantic variable and modality-specific style variables, conditioning each modality only on its relevant latent subset.The generative process draws shared semantics and modality-specific styles separately before generating each observed modality.Source: §2.1 · p. 2, §2.1.1 · p. 2

2.1.2 Multimodal and unimodal inference networks

• PVAE combines partitioned generative networks with multimodal and unimodal inference networks, treating them as probabilistic decoders and encoders.Multimodal inference estimates all latent variables, whereas unimodal inference recovers only the shared semantic variable and the corresponding modality style variable.Source: §2.2 · p. 3, §2.1.2 · p. 3

2.2.1 Variational lower bound

• The variational lower bound balances reconstruction and KL terms while penalizing one semantic KL divergence instead of M modality-specific terms when shared factors are encoded in zs.This gives the decoders access to shared information while reducing the KL penalty associated with representing it.Source: §2.2.1 · pp. 3–4

2.2.2 Coherence between multimodal and unimodal inference

• Multimodal-unimodal coherence minimizes divergence between multimodal and unimodal posteriors for the shared semantic and corresponding modality-specific style variables.The objective targets consistent latent inference when only one modality is available.Source: §2.2.2 · p. 4

2.2.3 Cross-modality semantic contrastiveness

• Cross-modality semantic contrastiveness makes same-sample semantic posteriors similar and different-sample posteriors dissimilar using a hinge loss.The implementation uses posterior-mean similarity with a radial basis kernel and margin t = 0.5.Source: §2.2.3 · p. 4

2.2 Training objectives

• The three objectives are combined as a weighted training criterion, with negative samples randomly drawn from the training set.The combined objective includes the variational lower bound, multimodal-unimodal coherence, and cross-modality semantic contrastiveness.Source: §2.2 · p. 3, §2.2.3 · p. 4

Multimodal representation learning

• Prior multimodal methods commonly learn modality-invariant semantic representations, discarding speaker or image-style information that is useful for modality-specific tasks.The paper positions PVAE as a unified framework that retains both shared semantics and modality-dependent residual factors.Source: §3 · p. 4
• PVAE differs from prior work by jointly learning modality-invariant semantic information and modality-dependent residual factors within a partitioned framework.This design targets tasks requiring information present in only one modality, including image generation or speaker verification.Source: §3 · pp. 4–5

Disentangled representation learning with VAEs

• Dimension-wise VAE methods encourage factorial latent distributions, but exchangeability among latent dimensions prevents interpretation without manually inspecting generated samples.These methods associate different explanatory factors with dimensions of one multidimensional latent variable.Source: §3 · p. 5
• Variable-wise approaches encode distinct data aspects in separate latent variables and use fixed graphical relationships to make representations interpretable.PVAE instead ties one latent variable to different modalities within a sample, rather than to dataset instances or sequence segments.Source: §3 · p. 5

Multimodal variational autoencoders

• JMVAE supports bidirectional cross-modality generation but assumes every modality depends on the same latent explanatory factors, so its representations are not disentangled.Its image-and-attribute-label setting also has a relatively deterministic mapping between modalities.Source: §3 · p. 5

4.1 Datasets, network architectures, and training

• The experiments paired spoken and written digits, using TIDIGIT and SecuVoice speech with MNIST images and randomly matched same-identity examples.Speech was represented with 80-dimensional Mel-scale filter bank coefficients, while images were 28×28 pixel matrices.Source: §4.1 · p. 5

4.2 Baseline Models

• The comparison included JMVAE-kl with one shared latent variable and speech-only and image-only VAEs using architectures matched to PVAE modules.JMVAE-kl used the same overall network architecture as PVAE but conditioned both generative networks on a single latent variable.Source: §4.2 · p. 6

4.3 Existence of modality-dependent factors

• JMVAE-kl decoding showed that representations inferred from one modality could vary that modality while producing the same-identity output in the other, supporting modality-dependent factors.The result implies that style in one modality is independent of style in the other and cannot be reliably inferred from the other modality.Source: §4.3 · p. 6, Figure 2 · p. 6

4.4 Evaluating disentanglement performance

• Over 99% accuracy was reached in both modalities on both datasets when PVAE used cross-modality semantic contrastiveness.Weighted average purity from 10-class k-means clustering measured digit-identity information in the latent variables.Source: §4.4 · p. 6, Table 1 · p. 7
• PVAE factorized semantic and style information: zs remained consistent across styles, while za and zi encoded modality-specific residual style information.Recombining inferred latent variables produced samples with row-consistent styles and identical same-digit columns in both written and spoken digits.Source: §4.4 · pp. 6–7

4.5 Controlled cross-modal generation

• PVAE supported controlled cross-modal conversion by combining a semantic variable from one modality with a style variable from a reference in the target modality.Both spoken-to-written and written-to-spoken Spanish digit generation preserved digit identity while referencing the specified style.Source: §4.5 · p. 7, Figure 5 · p. 8

• PVAE separates modality-invariant semantic information from modality-dependent style information into different latent variables.Source: §5 · p. 8
• Experimental results show that PVAE discovers latent semantic information without explicitly assigning one-hot labels.Source: §5 · p. 8
• The framework is presented as general and applicable to parallel audio-visual speech or instrument-playing audio-video data.The authors also plan to study samples containing multiple concepts that may be observed in only partial modalities.Source: §5 · p. 8

• The variational lower bound applies Jensen’s inequality and expands the result using factorization assumptions of PVAE’s generative and inference networks.Source: A. Derivation of PVAE variational… · p. 11
• The derivation reorganizes the expanded expression, marginalizes latent variables absent from expectations, and rewrites the final terms with KL divergence.Source: A. Derivation of PVAE variational… · p. 11

Generative networks

• The audio decoder concatenates semantic zs and audio-style za at each LSTM step to predict frame distributions, while the image decoder combines zs and image-style zi through fully connected and transposed-convolutional layers.The audio LSTM has 512 cells; the image decoder upsamples through two transposed convolutions.Source: B. PVAE neural network architectures · pp. 11–12
• Figures 6 and 7 depict the PVAE audio and image generative networks, respectively.Source: Figure 6 · p. 12, Figure 7 · p. 12

Inference networks

• Multimodal inference concatenates audio and image pre-encoder outputs to estimate distributions over zi, za, and zs, whereas unimodal inference estimates semantic and modality-specific style variables from one modality.Source: B. PVAE neural network architectures · p. 12
• The audio pre-encoder is a 512-cell LSTM, while the image pre-encoder uses two stride-2 convolutional layers followed by a 512-unit fully connected layer.Source: B. PVAE neural network architectures · p. 12
• The pre-encoder parameters are not shared between multimodal and unimodal inference networks.Source: B. PVAE neural network architectures · p. 13

Semantic latent variables

• Ten disconnected clusters appear in semantic zs embeddings inferred separately from speech and image samples, matching the k-means analysis.Source: C. Visualization of PVAE latent spaces · p. 13, Figure 11 · p. 13, Figure 12 · p. 13

Cross-modal semantic alignment

• Pooled speech and image semantic embeddings form ten digit-identity clusters regardless of the modality used for inference.Source: C. Visualization of PVAE latent spaces · p. 13, Figure 13 · p. 14

Style latent variables

• Audio and image style variables za and zi show no visible digit-identity clusters in their respective visualizations.Source: C. Visualization of PVAE latent spaces · p. 14, Figure 14 · p. 14, Figure 15 · p. 14

Interpretation

• The visualization results, together with k-means analysis, suggest that PVAE separates digit identity and style into different latent variables.Source: C. Visualization of PVAE latent spaces · p. 14

D. Additional examples of within-modality style transformation

• The additional examples extend style-transfer evaluation from selected digit combinations to all ten digits using semantic and style variables inferred from examples.This broader evaluation is intended to further verify the effectiveness of the learned disentangled representations.Source: D. Additional examples of… · p. 14
• Written-digit transformations preserve digit identities within columns while maintaining tilting, width, and stroke boldness within rows.The demonstrations condition columns on image-inferred semantic variables and rows on image-style variables.Source: D. Additional examples of… · pp. 14–15, Figure 16 · p. 15
• Spanish spoken-digit transformations preserve digit identities within columns while maintaining volume, pitch, duration, and starting-time offset within rows.The semantic variables are inferred from top-row audio samples, while audio-style variables are inferred from left-column samples.Source: D. Additional examples of… · p. 15, Figure 17 · p. 16, Figure 18 · p. 17

E. Additional examples of controlled cross-modality generation

• The experiments generate written Spanish digits from spoken digits with specified writing styles and spoken Spanish digits from written digits with specified speaking styles.The additional examples apply these controlled transformations to all digit combinations from which semantic and style variables are inferred.Source: E. Additional examples of controlled… · p. 18
• All four cross-modality demonstrations show successful conditional generation between written digits and spoken digits in Spanish and English.Figures 19–22 use models trained on SecuVoice-MNIST and TIDIGIT-MNIST and support PVAE’s ability to synthesize novel samples with disentangled representations.Source: E. Additional examples of controlled… · p. 18, Figure 19 · p. 19, Figure 20 · p. 20