PaperlayerView exact revision on arXiv

Source-linked AI summary

MUNITE: Unified Multimodal Latent Inference for Any-to-Any Multimodal Generation

Kyeongmin Yeo, Minhyuk Sung

arXiv:2610.09866v1cs.LG

TL;DR

Flexible any-to-any generation needs a way to handle arbitrary observed modality subsets, while encoding and latent generation are often treated separately. MUNITE unifies them as conditional inference over a complete-observation latent and learns from incomplete examples through self-distillation. Across evaluated benchmarks, it improves joint-generation coherence while maintaining competitive single-target quality.

  • Problem

    Flexible any-to-any generation calls for specialized models that support arbitrary combinations of observed and generated modalities, while encoding and latent generation remain separable modeling roles.

    Source: §1 · p. 1

  • Method

    MUNITE uses one conditional flow model for the complete-observation latent across evidence levels, trains partial-observation conditionals by self-distillation, and decodes modalities independently from a shared latent.

    Source: §1 · pp. 1–2

  • Results

    Across the evaluated benchmarks, MUNITE improves coherence among jointly generated modalities while maintaining competitive single-target generation quality.

    Source: §1 · p. 2, §5 · p. 9

  • Takeaways & Limitations

    A shared latent sample captures cross-target variation while modality-specific decoders model remaining uncertainty, supporting coherent joint generation across the evaluated settings.

    Source: §1 · p. 2, §5 · p. 9

    The gradient-equivalence result assumes a fixed representation and exact teacher, whereas practice uses the current model as teacher and numerically integrates its conditional flow.

    Source: §2.3 · p. 5

Abstract

from arXiv · show

We introduce MUNITE, a latent-variable framework for flexible any-to-any multimodal generation that treats encoding and latent generation as the same inference problem under different amounts of observed evidence. Given any subset of modalities, MUNITE models the conditional distribution over the latent representation associated with the complete observation. Full observation recovers deterministic encoding, no observation recovers the latent marginal, and intermediate subsets define conditional latent inference, all within a single conditional flow model. A shared latent sample captures variation that must remain consistent across generated targets, while modality-specific generative decoders model the remaining uncertainty independently. To learn these conditional distributions from incomplete training examples, we extend conditional flow matching through self-distillation: predictions conditioned on richer available observations supervise the same model conditioned on smaller subsets at the same intermediate latent state. When the richer-evidence trajectory follows the exact conditional flow, this provides the same expected learning signal as full-target denoising. Across PolyMNIST-D-Q, FFHQ64, and image-text-audio, MUNITE achieves competitive or better generation quality and source-target alignment, with higher joint-generation coherence. In particular, it attains the highest coherence in all one-to-many and unconditional image-text-audio comparisons, showing the effectiveness of unified latent inference across diverse multimodal settings.

1 INTRODUCTION

MUNITE treats encoding, partial-observation inference, and unconditional latent generation as one problem: inferring the complete-observation latent from varying evidence. It learns these conditional distributions with self-distillation and reports stronger joint-generation coherence across its evaluated benchmarks.

1 INTRODUCTION

  • MUNITE targets specialized any-to-any generation for applications that need arbitrary combinations of observed and generated modalities without requiring a large language model backbone.The motivation notes that some applications need little or no text processing.Source: §1 · p. 1
  • One conditional model maps any observed modality subset to the latent distribution of complete observations, spanning deterministic encoding, intermediate inference, and unconditional generation.A shared latent captures variation shared across targets, while modality-specific decoders model remaining uncertainty independently.Source: §1 · pp. 1–2, Figure 1 · p. 2
  • For incomplete examples, richer-observation predictions supervise smaller-subset predictions at the same flow state, matching full-target denoising's expected gradient under the stated assumptions.This lets partially observed examples train the desired conditional distribution without a direct complete-observation latent target.Source: §1 · p. 2
  • MUNITE achieves the best unconditional fidelity and coherence on PolyMNIST-D-Q and FFHQ64, and the highest coherence in every one-to-many and unconditional image–text–audio comparison.It remains competitive on one-to-one and many-to-one generation.Source: §1 · p. 2

2 METHOD

MUNITE uses one conditional latent-inference model across observation subsets, paired with modality-specific decoders. Reconstruction, conditional flow matching with self-distillation, and contrastive alignment train the shared representation and its conditional distributions.

2.1 SHARED-LATENT GENERATIVE FRAMEWORK

  • For any observed subset, MUNITE models the conditional distribution over latents of complete observations; full and absent observation recover encoding and the latent marginal, respectively.A shared latent carries cross-modal dependence, and modality-specific decoders model the remaining uncertainty.Source: §2.1 · pp. 2–3

2.2 LEARNING THE SHARED REPRESENTATION

  • Reconstruction trains decoders through the shared representation, while target-detached self-reconstruction stops the target stream's reconstruction gradients from carrying target-private information into the latent.Full-observation latents train each decoder; smaller conditioning subsets further stabilize shared-representation learning.Source: §2.2 · p. 3

2.3 CONDITIONAL FLOW MATCHING WITH PARTIAL OBSERVATIONS

  • Self-distillation gives the same expected gradient as full-target denoising when the richer-evidence teacher reproduces the conditional time marginals, with the equivalence stated for a fixed representation and exact teacher.In practice, the current model serves as teacher and its conditional flow is integrated numerically.Source: §2.3 · pp. 4–5

2.4 CONTRASTIVE ALIGNMENT ACROSS OBSERVATION SUBSETS

  • A directional InfoNCE objective aligns predictions from complementary observation subsets, and shared backbone parameters make this alignment stabilize the model's internal representation.The two directions are averaged; these one-step partial-observation predictions are not samples of the target latent distribution.Source: §2.4 · p. 5

2.5 MODEL ARCHITECTURE

  • A shared Transformer keeps modality streams separate while latent registers aggregate information from observed streams and project their final states to clean-latent predictions.For reconstruction targets, key/value paths are detached so gradients stop at the target stream while its information remains in the forward pass.Source: §2.5 · pp. 5–6, Figure 3 · p. 5

2.6 TRAINING AND INFERENCE

  • At inference, one latent drawn by integrating the conditional flow is shared by all target decoders, as required by the factorization.Reconstruction updates the inference model and sampled target decoder; distillation and contrastive losses update only the inference model.Source: §2.6 · p. 6

2.7 RELATION TO MULTIMODAL LATENT-VARIABLE MODELS

  • Unlike JNF's composed modality-specific posteriors, MUNITE directly models the same full-observation latent conditional on any subset and can learn these conditionals from partial examples.The richer-evidence prediction serves as the self-distillation target. Sampling the shared missing variation once yields the conditional joint over multiple targets, unlike target-wise sufficiency alone.Source: §2.7 · p. 6

3 RELATED WORK

Prior multimodal generation work includes masked prediction, coupled joint flows, and shared-latent models. MUNITE is positioned as directly modeling conditional distributions of a full-observation latent across arbitrary observed subsets.

Related work

  • The discussion focuses on multimodal generation: shared-latent VAEs use modality-specific encoders and decoders, while MUNITE models a coherence-sufficient full-observation latent conditional on arbitrary observed subsets using conditional transport rather than diagonal-Gaussian subset posteriors.Source: §3 · p. 7

Any-to-any generation

  • Masked models predict target-modality tokens from observed-modality tokens, while joint flow models couple modality trajectories and CoDi couples modality-specific denoisers through cross-attention.The paper's controlled baselines implement continuous and discrete joint-flow approaches.Source: §3 · p. 7

Shared-latent generative models

  • MUNITE differs from shared-latent approaches by directly modeling the conditional distribution of a coherence-sufficient full-observation latent under arbitrary subsets, using conditional transport instead of diagonal-Gaussian subset posteriors.The cited approaches include MUNI and FlowBind (Yeo et al., 2026; Cha et al., 2026).Source: §3 · p. 7

4 EXPERIMENTS

MUNITE is evaluated across four generation routes and unconditional co-generation on PolyMNIST-D-Q, FFHQ64, and image–text–audio. It improves joint-generation coherence across these benchmarks while retaining competitive single-target quality.

4.1 EXPERIMENTAL SETUP

  • The benchmarks span categorical attributes, dense visual modalities, and heterogeneous text, image, and audio, with training examples that are fully or partially observed.The study evaluates one-to-one, many-to-one, one-to-many, many-to-many, and unconditional generation.Source: §4 · p. 7, §4.1 · p. 7

4.2 POLYMNIST-D-Q

  • Under quadrant conditioning, digit coherence rises from 0.1074 with MUNI to 0.9904 with MUNITE; under digit conditioning, quadrant coherence rises from 0.2528 to 1.0000.MUNITE also attains the lowest FD and highest unconditional coherence for both labels.Source: Table 1 · p. 8, §4.2 · p. 8

4.3 FFHQ64

  • On FFHQ64, conditioning on age and gender, MUNITE raises RGB–segmentation coherence from 0.6038 to 0.8888 and lowers normal error from 0.1106 to 0.0500 versus MUNI.DFM has slightly lower normal error (0.0492 versus 0.0500), while MUNITE has higher segmentation coherence (0.8888 versus 0.8142).Source: §4.3 · p. 9

4.4 IMAGE–TEXT–AUDIO

  • MUNITE achieves the highest coherence in all three one-to-many directions and all three unconditional image–text–audio comparisons, while remaining competitive on one-to-one and many-to-one routes.The evaluation covers text, image, and audio modalities.Source: Table 2 · p. 9, §4.4 · p. 9

5 CONCLUSION AND FUTURE WORK

MUNITE unifies encoding and any-to-any generation as inference over a full-observation latent, using self-distillation to learn from incomplete data. The paper proposes multimodal world modeling as a next step.

5 CONCLUSION AND FUTURE WORK

  • MUNITE unifies encoding and any-to-any multimodal generation as inference over a full-observation latent, with self-distillation extending learning to incomplete examples.A shared latent carries cross-modal dependencies, while modality-specific decoders model remaining uncertainty.Source: §5 · p. 9
  • A proposed next step is to learn dynamics in MUNITE’s latent space for state estimation and future prediction under varying multimodal observations.Modality-specific decoders could render predicted states into desired modalities.Source: §5 · p. 9

ETHICS STATEMENT

The ethics statement cautions that face and audiovisual generation may reproduce dataset biases or enable deceptive synthetic media. The appendix describes full-observation latents as encoder-induced distributions and reports that query dependence is practically negligible.

ETHICS STATEMENT

  • Face and audiovisual outputs may reproduce dataset biases or be misused for deceptive synthetic media; FFHQ64 age and gender labels are annotations, not validated personal attributes.Generated examples should be treated as synthetic outputs rather than representations of real individuals or verified attributes.Source: ETHICS STATEMENT · p. 10

A FULL-OBSERVATION LATENTS AND CONDITIONAL GENERATION

  • For a fixed representation W = E(XM), conditional latent distributions are pushforwards of data conditionals through the full-observation encoder; full observation gives a point mass and empty conditioning gives the latent marginal.Source: §A.1 · p. 14

A.1 THE DISTRIBUTION INDUCED BY FULL OBSERVATIONS

  • The full-observation latent can depend on a Gaussian query, but measured variance ratios of 2.6 × 10−5, 7.9 × 10−5, and 6.5 × 10−4 support practical determinism across the three benchmarks.The ratios correspond to PolyMNIST-D-Q, FFHQ64, and image–text–audio, respectively.Source: §A.1 · p. 14

A.2 DERIVATION OF THE CONDITIONAL JOINT

The conditional joint is obtained by integrating over the latent conditional, with one shared latent sample for all target decoders. The appendix explains why predictive sufficiency alone does not ensure coherent joint decoding.

A.2 DERIVATION OF THE CONDITIONAL JOINT

  • The conditional-joint derivation requires one shared latent sample across target decoders; separate samples discard dependencies that remain after observing the inputs.The unconditional co-generation case is obtained when the input set is empty.Source: §A.2 · p. 14

A.3 PREDICTIVE SUFFICIENCY AND JOINT COHERENCE

  • Predictive sufficiency constrains dependence within observed modalities and preserves information about each missing modality, but does not require missing targets to be conditionally independent.Thus, factorized decoding can miss the joint conditional even when the latent satisfies the stated predictive-sufficiency conditions.Source: §A.3 · p. 14
  • In the appendix example, independently decoded targets coincide only with probability 1/2 although the data targets always coincide; MUNITE instead requires coherence sufficiency of the full-observation latent.Source: §A.3 · p. 15

B DERIVATION OF PARTIAL-DATA SELF-DISTILLATION

The derivation shows how richer-evidence predictions can supervise smaller-subset predictions at a correctly distributed intermediate state, yielding the same expected gradient under stated assumptions. It also explains how fully observed examples and stratified subset averaging fit this objective.

B DERIVATION OF PARTIAL-DATA SELF-DISTILLATION

  • The equivalence assumes a fixed representation and teacher, matched time and subset distributions, and partial examples formed by data-independent modality removal.Synthetic masking in PolyMNIST-D-Q and FFHQ64 matches this setting; image–text–audio availability follows its source corpus.Source: B DERIVATION OF PARTIAL-DATA… · p. 15

B.1 THE TEACHER STATE

  • An exact teacher’s conditional interpolation velocity gives the partial example a correctly distributed evaluation state without revealing its complete latent.Only the state marginals need to match; individual teacher trajectories may differ from straight interpolation paths.Source: §B.1 · p. 15

B.2 PROOF OF PROPOSITION 1

  • The distillation and denoising objectives have the same expected gradient because their difference is a parameter-independent residual after conditional-expectation orthogonality.Their common function-space minimizer is the conditional expectation of the latent given the noisy state and smaller observed subset.Source: §B.2 · p. 15

B.3 FULL OBSERVATIONS AND SUBSET AVERAGING

  • With full observations, the teacher target is the encoded complete observation and the state is the analytic interpolant; averaging one subset of each size preserves the uniform-size objective.The stratified subset average equals the objective under uniformly sampled subset sizes by linearity of expectation.Source: §B.3 · pp. 15–16

C.1 ARCHITECTURE

MUNITE uses a Transformer-based latent model and modality-specific decoders, trained through reconstruction, distillation, and contrastive objectives. The experiments use several multimodal benchmarks and compare against latent and flow-based baselines.

C.1 ARCHITECTURE

  • The architecture combines a Transformer latent model with modality-specific decoders and benchmark-specific latent-token dimensions.Images use 8×8 patches; categorical modalities use learnable embeddings, while image–text–audio inputs are precomputed 768-dimensional feature vectors mapped to eight tokens. The latent dimensions are 4 × 8, 64 × 8, and 1 × 768 for PolyMNIST-D-Q, FFHQ64, and image–text–audio, respectively.Source: §C.1 · p. 16, Table 3 · p. 16

C.2 TRAINING PROCEDURE

  • Training samples reconstruction targets and conditioning subsets from available modalities, with an additional cross-modal reconstruction sampling rule for image–text–audio.The objective combines reconstruction, distillation, and contrastive losses, weighted 1, 2, and 0.1, respectively.Source: §C.2 · p. 16, §C.3 · p. 17

C.3 BASELINES

  • Baselines include MUNI’s product-of-experts latent aggregation and CFM and DFM models configured with MUNITE’s Transformer width and attention heads.CFM uses continuous patches or vectors and straight Gaussian flow; DFM uses VQ-VAE codes and absorbing-mask corruption. Their additional blocks approximately match MUNITE’s trainable parameter count.Source: §C.3 · pp. 16–17

D.1 DATASETS

  • The benchmarks cover PolyMNIST-D-Q, FFHQ64, and image–text–audio, with different modality availability and dataset construction across settings.PolyMNIST-D-Q provides image views and digit/quadrant labels; FFHQ64 includes RGB, segmentation, normals, age, and gender. Image–text–audio training combines paired and triplet observations whose available modalities follow their source corpora.Source: §D.1 · pp. 17–18, Table 4 · p. 17, Table 5 · p. 18

D.2 MODEL SIZES AND TRAINING BUDGETS

Across the supplied evaluation results, MUNITE’s shared latent is associated with coherent jointly generated views, while single-target scores are not uniformly better. The section also specifies benchmark evaluation procedures, model budgets, and evaluation-set constraints.

D.2 MODEL SIZES AND TRAINING BUDGETS

  • CFM and DFM approximately match MUNITE’s trainable parameter count, while MUNI and FlowBind retain the larger architectures of their released implementations.Training uses Adam and the reported method-specific budgets; EMA is used for MUNI, MUNITE, CFM, and DFM, but not FlowBind.Source: §D.2 · p. 18, Table 8 · p. 19

D.3 INFERENCE BUDGETS

  • Inference comparisons account for model evaluations and decoding, and CFM receives as many network evaluations as MUNITE’s latent flow and decoder combined.FlowBind averages inferred source latents when given multiple sources and has no unconditional sampler; image and audio features are rendered with fixed decoders.Source: §D.3 · pp. 18–19, Table 9 · p. 19

D.4 VERIFIERS

  • The frozen verifiers used to score PolyMNIST-D-Q and FFHQ64 are trained on all benchmark tuples, including held-out examples.The verifiers use SegFormer backbones fine-tuned for each benchmark’s classification, segmentation, and normal-estimation tasks.Source: §D.4 · p. 19

E.2 FFHQ64

  • FFHQ64 evaluation uses 5,000 held-out tuples and 5,000 unconditional samples, scoring RGB fidelity with InceptionV3-feature FID and generated maps against verifier predictions.For age-and-gender-conditioned generation, dense-map scores compare generated targets with one another because the inputs provide no reference maps.Source: §E.2 · p. 20

E.3 IMAGE–TEXT–AUDIO

  • The 975 joint-evaluation triplets pair captions and real audio with FLUX.1-generated images, so their three modalities are not co-recorded.Evaluation reports fidelity and source–target alignment metrics, including CLIP for text–image and CLAP for text–audio; external-model scores use released pretrained models.Source: §E.3 · pp. 20–21

E.1 POLYMNIST-D-Q

  • With separately drawn latents, one-to-many agreement falls to chance while input-label accuracies remain 0.995 and 1.000 for digit and quadrant inputs.MUNITE’s agreement stays close to the uniform missing-label reference; the authors attribute agreement among views to the shared latent sample.Source: Table 10 · p. 20, §E.1 · p. 20

E.3.1 SINGLE-TARGET GENERATION

  • MUNITE scores best on three of six one-to-one fidelity measures, with higher text–audio but lower text–image alignment than MUNI in one-to-one and many-to-one generation.Because its single outputs are not uniformly better, the reported coherence gains are not explained by single-output quality.Source: Table 11 · p. 21, §E.3.1 · p. 21

E.3.2 COMPONENT ABLATIONS

The ablations show that one-to-many coherence remains strong without any single auxiliary component, while unconditional coherence and single-target fidelity are more sensitive to component removal. The qualitative examples complement the quantitative results, especially by illustrating whether jointly generated outputs agree on unspecified content.

E.3.2 COMPONENT ABLATIONS

  • Removing target detaching or contrastive loss degrades fidelity on all six one-to-one routes, with the contrastive-loss removal most visible in image-to-audio FAD and audio-to-text CIDEr.Full-only distillation has mixed fidelity effects: it slightly improves four image-related routes but degrades text-to-audio FAD and audio-to-text CIDEr.Source: §E.3.2 · p. 22
  • Every ablation variant stays within two points of full-model one-to-many coherence and above MUNI on all three routes; without target detaching, unconditional CLAP and AIS coherence drop 5.6 and 9.6 points.Restricting distillation to fully observed examples lowers unconditional CLAP and AIS coherence by 2.8 and 4.6 points; this removes partial-data supervision of latent conditionals.Source: §E.3.2 · pp. 21–22, Table 14 · p. 23

F QUALITATIVE RESULTS

  • The qualitative examples are selected to illustrate each route and complement quantitative results by showing whether joint outputs agree on content left unspecified by the input.Examples for routes whose only output is audio are available on the project page.Source: F QUALITATIVE RESULTS · p. 23

F.1 POLYMNIST-D-Q

The qualitative examples show how MUNITE handles agreement across generated modalities, from conditioned labels and shared attributes to image, text, and audio describing compatible content. They also contrast its outputs with competing methods on several routes.

F.1 POLYMNIST-D-Q

  • MUNI and MUNITE render conditioned labels in every single-target example, whereas CFM often produces illegible digits and DFM often omits or changes the digit.Figure 4 shows the input labels and the view generated for each column.Source: §F.1 · p. 23, Figure 4 · p. 24
  • All four unconditional MUNITE samples agree across the three views and two labels, while MUNI samples sometimes disagree on digit identity or quadrant.Each group in Figure 5 is one joint sample of the three views and both labels.Source: §F.1 · p. 23, Figure 5 · p. 24
  • MUNITE keeps the missing attribute consistent across one-to-many views, placing a given digit in the same quadrant or drawing the same digit for a given quadrant.MUNI’s views can each match the input while differing on the missing attribute.Source: §F.1 · p. 23, Figure 6 · p. 25

F.2 FFHQ64

  • MUNITE’s RGB faces follow conditioned age and gender; its unconditional labels agree with the faces, and its segmentation and normals align with the RGB image.In many-to-many generation, its maps follow head pose, hairline, and facial layout in every example; MUNI’s maps sometimes describe a different face.Source: Figure 7 · p. 26, MUNITE · p. 26, Figure 8 · p. 27

F.3 IMAGE–TEXT–AUDIO

  • MUNITE’s one-to-one and many-to-one outputs follow input content; for faucet image, water, and speech inputs, its caption mentions both water and speaker, while MUNI mentions only water.For the roaring-lions input, MUNITE generates a lion where FlowBind shows a tiger; its caption for a stop-sign image reads the sign.Source: §F.3 · p. 28, Figure 10 · p. 29, Figure 11 · p. 30
  • Unconditional MUNITE samples decode one latent into text, image, and audio that describe one scene, including details not all outputs mention.For a painting of a woman at a table, the image shows cups and a plate and the audio contains dishes and cutlery, while the caption leaves that detail out.Source: §F.3 · p. 28, Figure 12 · p. 31
  • In one-to-many generation, independently decoded outputs from one latent share details beyond the input, as captions and images both depict a crowd near ducks and audio adds thunder and rain to a lightning scene.The examples are the ducks-and-speech input and the horse-under-lightning input.Source: §F.3 · p. 28, Figure 13 · p. 31
Loading 2610.09866v1…