PaperlayerView exact revision on arXiv

Source-linked AI summary

MODUS: Decoder-Only Any-to-Any Modeling of Diverse Modalities

Mingqiao Ye, Zhaochong An, Zhitong Gao, Xian Liu, François Fleuret, Chuan Li, Amir Zadeh, Serge Belongie, Afshin Dehghan, Jesse Allardice, David Mizrahi, Oğuzhan Fatih Kar, Roman Bachmann, Amir Zamir

arXiv:2607.25948v1cs.CVcs.AIcs.LG

TL;DR

Existing any-to-any models are usually trained from scratch, limiting their ability to leverage pretrained decoder-only priors. Modus extends a pretrained decoder-only model to unified any-to-any generation across diverse modalities, achieving strong out-of-the-box performance and robust generalization across modality combinations.

  • Problem

    Existing any-to-any systems are typically trained from scratch with encoder–decoder or diffusion architectures, limiting their use of pretrained decoder-only priors and scalability.

  • Method

    Modus uses a unified decoder-only architecture and training recipe to condition on or generate arbitrary modalities without modality-specific heads, losses, or task pipelines.

  • Results

    Across diverse benchmarks, Modus demonstrates strong out-of-the-box performance and robust generalization across modality combinations.

  • Takeaways & Limitations

    A single model supports flexible any-to-any generation, chained generation, cross-modal self-verification, and visual representation composition without additional architectural complexity.

  • Takeaways & Limitations

    Chained generation increases inference cost by requiring one generation per intermediate modality, adding decoding steps and latency.

Abstract

from arXiv · show

Any-to-any models predict any modality from any combination of others within a single network, a formulation used in multimodal vision and vision-language models, and increasingly in scientific domains such as ecology and astronomy. Existing any-to-any models are typically trained from scratch using encoder-decoder or diffusion architectures, impacting their performance and preventing them from using strong pre-trained decoder-only models as a prior. In this work, we investigate decoder-only any-to-any multimodal modeling, which treats all modalities symmetrically and supports arbitrary modalities as inputs and outputs without modality-specific heads, losses, or task pipelines. Because every modality is both an input and an output of the same model, the resulting model, named Modus, can support a range of applications, such as chained generation through intermediate modalities or cross-modal self-verification by scoring the model's own outputs with another generated modality. Modus demonstrates strong out-of-the-box performance and is competitive with specialist and multitask baselines using a single model across various benchmarks. All materials are open-sourced at https://modus-multimodal.epfl.ch/.

1. Introduction

MODUS introduces a decoder-only any-to-any model that treats modalities symmetrically, extending a pretrained decoder-only foundation model across diverse modality pairs without modality-specific heads, losses, or task pipelines. It supports 15 modalities trained on a 29M-sample corpus and enables chained generation, cross-modal self-verification, and visual representation composition.

  • Motivation: Existing any-to-any models are typically trained from scratch with encoder–decoder or diffusion architectures, preventing them from leveraging priors in large pretrained decoder-only models.This increases training costs and limits scalability while requiring joint learning of modality structures and cross-modal alignment.
  • Motivation: Decoder-only multimodal systems remain limited by privileged text and RGB modalities, modality-specific components, task-dependent losses, or text-centric pivots.These constraints hinder arbitrary modality-to-modality generation within a single symmetric framework.
  • MODUS: MODUS realizes any-to-any generation in one decoder-only model, treating all modalities symmetrically without modality-specific heads, losses, or task pipelines.It allows arbitrary modality pairs to serve as conditioning inputs or generation targets within a single model.
  • Data and capabilities: 15 modalities spanning geometry, structure, semantics, and learned representations are trained using the 29M-sample MODUS-DATASET, which is openly released with two model checkpoints.The corpus aligns annotations on a shared image base and supports arbitrary input–target modality pairs, including depth →canny edge and canny →surface normal.
  • Training: MODUS extends a pretrained decoder-only foundation model using uniform timestep sampling and a staged curriculum, inheriting pretrained priors and scalability.The recipe yields strong out-of-the-box performance competitive with task-specific specialists at a fraction of the compute.
  • Data and capabilities: MODUS’s unified design enables chained generation, cross-modal self-verification, and visual representation composition.These capabilities are studied in Section 4.

2. Related Work

Prior multimodal decoder-only models enable multimodal understanding but often retain text-only outputs, while any-to-any systems pursue flexible cross-modal mappings with modality-agnostic objectives. MODUS is situated within this shift toward shared architectures spanning diverse modalities.

  • Decoder-only multimodal models: Decoder-only transformers have become dominant because of scalability, unified training, and strong zero-shot generalization.The passage attributes this paradigm to Hurst et al. (2024), Touvron et al. (2023), and Team et al. (2023).
  • Decoder-only multimodal models: Early multimodal decoder-only models couple visual encoders with large language decoders for multimodal understanding, but their outputs remain text-only.Examples include LLaVA, MiniGPT-4, InternVL, and Qwen-VL.
  • Any-to-any modeling: Any-to-any models support flexible mappings between multiple modalities within a shared architecture.The passage describes this design as task-agnostic modeling across modalities using modality-agnostic objectives.
  • Any-to-any modeling: Unified-IO and 4M use encoder–decoder systems, while OneDiffusion uses unified diffusion to demonstrate task-agnostic multimodal modeling.These systems show that cross-modal modeling can be achieved with modality-agnostic objectives.
  • MODUS framework: MODUS supports a diverse and extensible set of modalities spanning 1D sequential and 2D spatial modality families.The figure lists text, visual grounding boxes, object detection, feature tokens, RGB images, depth, surface normals, segmentation, and canny edges.

3. Method

MODUS is a unified decoder-only any-to-any model that represents diverse modalities in one sequence and processes them through shared autoregressive context. Its method combines modality-specific tokenization and expert routing with stabilized flow-matching training, progressive initialization, and aligned multimodal data.

  • Unified Tokenization: 1D modalities use discrete tokenizers and next-token prediction, while 2D modalities combine semantic features with continuous reconstruction latents and flow-matching denoising.The 1D Expert handles text, grounding, and features; the 2D Expert handles RGB images, depth, normals, segmentation, and edges.
  • Architecture: MODUS uses a decoder-only Mixture-of-Transformers architecture in which arbitrary modalities serve as inputs or targets without modality-specific heads, task pipelines, or separate generators.It adapts pretrained BAGEL-7B and unifies training objectives and token representations within one sequence-modeling framework.
  • Token Routing: MODUS routes tokens through modality-appropriate experts while preserving shared causal cross-modality attention, allowing generated tokens from either expert to condition subsequent decoding.ViT semantic tokens use the 1D Expert with intra-modality bidirectional attention, whereas VAE reconstruction tokens use the 2D Expert and flow matching.
  • Training and Data: Training begins from pretrained BAGEL-7B and proceeds through progressive stages that expand from 1D cross-modal alignment to long-context, multi-condition, and chained generation.MODUS-DATASET provides aligned tuples containing one image and annotations for caption, grounding, detection, depth, normals, segmentation, edges, and feature-map tokens.
  • Training Stabilization: Uniform timestep sampling mitigates modality confusion by balancing exposure across the flow trajectory, because early timesteps determine the target modality while later timesteps refine visual quality.Logit-normal sampling undersamples early steps and oversamples intermediate ones, causing drift between modalities and mixed outputs.
  • Multimodal Inference: The unified representation enables chained generation by reusing outputs as later conditioning inputs and supports cross-modal self-verification with auxiliary modalities.These capabilities operate without retraining or architectural changes because all modalities share the same tokenized representation and model.

4. Experiments

Experiments show that MODUS performs flexible zero-shot any-to-any generation across diverse modalities and supports compositional chaining and self-verification within one decoder-only model. Chaining improves structural consistency and can benefit geometrically aligned predictions, while combined visual representations and timestep sampling strengthen performance.

  • Zero-shot multimodal generation: MODUS generates arbitrary target modalities from arbitrary inputs in one architecture, with zero-shot grounding, depth, and surface-normal results on unseen data.The model’s unified generative behavior is visualized across modality pairs, including text-specified region localization and NYUv2 geometric prediction.
  • System-level comparison: MODUS supports broader modality transformations and multi-condition mappings than RGB- or text-centric benchmarks, including canny→depth, surface normal→depth, and RGB+canny→depth.Quantitative results for these non-standard transformations are reported in Section C.4.
  • Chained generation: Chaining lets MODUS feed generated modalities back as conditioning, and routing through spatially aligned canny edges improves surface-normal prediction.Intermediate modalities can provide scene structure or geometry without retraining or architectural changes; the study compares canny edges, depth, and DINOv2 global features.
  • Caption-to-Any consistency: Chained generation produces stronger cross-modal structural consistency than independent caption-to-target generation by fixing spatial layout through an intermediate modality.For example, Text→Edge→RGB constrains the layout before final image generation, at the cost of additional decoding steps and modest added latency.
  • Cross-modal verification: 0.84 GenEval is achieved after self-verification selects among four sampled images, improving from 0.81 without an external verifier.MODUS scores its own outputs using self-predicted grounding confidence or VQA answer likelihood for requested objects.
  • Visual representation ablation: ViT–VAE conditioning achieves the best performance, combining high-level semantic features with low-level reconstruction details for depth and surface-normal estimation.ViT-only conditioning preserves coarse semantic identity but distorts fine-grained geometry, motivating complementary feature composition.

5. Conclusion

MODUS is a unified decoder-only model for any-to-any multimodal generation across diverse modalities. Unified tokenization and staged training enable flexible generation and multimodal capabilities without modality-specific heads or task pipelines.

  • Conclusion: MODUS extends decoder-only modeling beyond text and RGB images to support diverse modalities within one unified any-to-any generation model.It uses unified tokenization rather than modality-specific heads or task pipelines.
  • Conclusion: Uniform timestep sampling and staged training provide stable, scalable multimodal training while efficiently adding modalities and multi-condition settings.Uniform timestep sampling avoids modality mixing.
  • Conclusion: MODUS supports flexible any-to-any generation, chained generation, cross-modal self-verification, and visual representation composition without additional architectural complexity.These capabilities follow from using the same model across modalities as inputs and outputs.

Impact Statement … B. Additional Ablations

MODUS is positioned as a unified any-to-any research tool whose general-purpose design may extend beyond the demonstrated scenarios. The related work traces decoder-only models from vision–language systems toward unified image understanding and generation, while identifying limited heterogeneous output support as an open gap.

  • Impact Statement: MODUS advances unified multimodal representation learning and generation as a general-purpose any-to-any research tool.Its diverse modality transformations may enable applications beyond those examined, motivating thoughtful consideration when extending it further.
  • A. Additional Related Work: Decoder-only vision–language systems connect visual encoders to pretrained GPT-style decoders for image captioning, VQA, and multimodal interaction.Examples include LLaVA, Qwen-VL, and DeepSeek-VL, using encoders such as CLIP or SigLIP.
  • A. Additional Related Work: Autoregressive image generators apply next-token prediction to VQ-based image tokens, while newer tokenizers improve perceptual fidelity and flexibility.The passage notes that these GPT-style generators remain challenged in capturing fine details compared with diffusion models.
  • A. Additional Related Work: Chameleon and Show-O unified image understanding and generation in decoder-only backbones, followed by systems that improve visual representations with pretrained encoders or hybrid tokenizers.EMU-3 and BLIP-3o enable stronger semantic reasoning and higher visual representation quality.
  • A. Additional Related Work: Janus and Janus-Pro decouple visual understanding from generation, while JanusFlow adds rectified flow to improve realism within a unified decoder-only architecture.Semantic encoders support recognition, and VQ-based tokenizers serve the generation branch.
  • A. Additional Related Work: Specialist models continue advancing individual visual domains, including depth estimation, surface normals, semantic understanding, and spatial localization.The passage names Depth Anything, Marigold, DepthFM, Lotus, Omnidata, GeoWizard, Grounding DINO, and Grounded-SAM.

B.1. Modality Mixing and Timestep Sampling · B.2. Training Stages

Uniform timestep sampling stabilizes early target-modality selection, whereas logit-normal sampling can cause modality confusion. MODUS uses a three-stage curriculum that introduces new 1D modalities before jointly training them with stronger-prior 2D modalities.

  • B.1. Modality Mixing and Timestep Sampling: Uniform sampling keeps depth and surface-normal predictions stable across timesteps, with reasonable target modalities appearing even during early denoising steps.Later timesteps primarily refine structural details.
  • B.1. Modality Mixing and Timestep Sampling: Balanced early-timestep coverage lets the model commit reliably to the correct target modality early in the trajectory.Uniform sampling frequently exposes the regime where modality selection is determined.
  • B.1. Modality Mixing and Timestep Sampling: Logit-normal sampling concentrates probability on cleaner middle timesteps, undersampling early noisy steps and producing unstable or mixed-modality outputs.The model may fail to recover the correct modality as denoising progresses.
  • B.1. Modality Mixing and Timestep Sampling: Sufficient early-timestep coverage is therefore important for reliable target-modality selection, while logit-normal sampling may need regularization or curriculum strategies.The visualizations show this contrast for both depth and surface-normal prediction.
  • B.2. Training Stages: Loss curves indicate stable convergence across representative 1D and 2D modalities under the unified training regime.Initializing from BAGEL accelerates convergence across modalities.
  • B.2. Training Stages: MODUS training follows a three-stage curriculum that first introduces only new 1D modalities, including grounding and DINOv2 feature tokens.These modalities are learned over a longer period without strong priors.
  • B.2. Training Stages: The second stage adds 2D modalities and jointly trains them with the new 1D modalities to improve training efficiency.The 2D modalities provide stronger priors.

B.3. Training from Scratch … C.3. Image–Text Capability Preservation

MODUS benefits from unified initialization and unified-sequence I/O, while extending its any-to-any capabilities across base models and evaluations. It preserves strong multimodal performance, including depth, grounding, and image–text capabilities, when trained with an appropriate data mixture.

  • B.3. Training from Scratch: Training from scratch converges more slowly, especially for text–image modalities, and lowers performance on most tasks.Unified MLLM initialization provides multimodal priors that improve optimization stability; DINOv2 feature prediction shows the smallest training-regime difference.
  • B.3. Training from Scratch: The three-stage training scheme gradually incorporates 1D and 2D modalities before enabling multi-conditioned generation.The final stage supports multi-conditioned generation within the unified training procedure.
  • B.4. Decoupled vs Unified I/O: Unified I/O substantially improves geometric tasks over per-modality heads, especially NYUv2 surface normal estimation.On MMMU, the decoupled and unified variants score 50.4 vs 51.1; DIODE depth scores 0.322 vs 0.285, and NYUv2 surface normal estimation scores 50.75 vs 19.92.
  • B.5. Generalization to a Different Base Model: Applying the MODUS pipeline to Janus-Flow-1.3B preserves most MMMU performance while adding depth and surface normal estimation capabilities.The extension scores 27.1 vs 29.3 on MMMU, 0.304 on DIODE, and 36.53 on NYUv2 surface normal estimation, at approximately 13× less compute than original Janus-Flow training.
  • C.1. Depth Estimation: MODUS delivers competitive zero-shot depth estimation across NYUv2, ScanNet, and DIODE despite jointly training with many modalities.Its performance is reported as on par with multitask encoder–decoder and diffusion-based baselines and comparable to specialized single-task depth models.
  • C.2. Referring Object Grounding: MODUS achieves comparable or superior zero-shot referring-expression comprehension across RefCOCO, RefCOCO+, and RefCOCOg evaluation splits.Unlike narrower specialist and decoder-only systems, it maintains strong grounding while supporting a broader any-to-any generation setting.
  • C.3. Image–Text Capability Preservation: Mixing LLaVA-OneVision data into MODUS-DATASET training mitigates degradation of BAGEL’s original image–text capabilities.Without the mix, POPE falls 87.23 →73.67, VizWiz 59.41 →8.34, and MME-S 2377 →1561; the full setup recovers most losses.
  • C.3. Image–Text Capability Preservation: The full MODUS setup keeps VizWiz within 1.2 points of BAGEL, POPE within 0.04 of baseline, and MME scores within 3% of BAGEL.Document and high-resolution benchmarks, including DocVQA and ChartQA, retain some residual drop.

C.4. Multi-Condition and Direct Any-to-Any Generation … D.3. Self-Verification

MODUS supports multi-condition and direct any-to-any generation across non-standard modalities, chained inference, and self-verification within one unified model. These capabilities extend to efficient generation, coherent intermediate representations, and improved text-to-image selection.

  • C.4. Multi-Condition and Direct Any-to-Any Generation: Multi-conditioning improves surface-normal estimation: RGB+Depth→Surface Normal reaches 19.58 and Edge+Depth→Surface Normal reaches 19.72, versus 20.02 for standard RGB→Surface Normal.Edge+Depth approaches the RGB-based baseline without RGB input, indicating complementary multimodal signals.
  • C.4. Multi-Condition and Direct Any-to-Any Generation: Direct transfers are weaker but achievable within MODUS: Edge→Depth scores 0.302 versus 0.285 for RGB→Depth, while Depth→Surface Normal scores 29.51 versus 20.02 for RGB→Surface Normal.The same pretrained model performs these mappings without task-specific pathways.
  • C.5. Contamination Check: Teacher-data checks found no DIODE or NYUv2 training overlap for DepthAnything V2 and Marigold, while the GLaMM pipeline remains zero-shot on COCO.COCO was not used during MODUS-DATASET construction.
  • C.6. GenEval Per-Category Breakdown and Self-Verification: MODUS matches or slightly improves BAGEL on simple GenEval categories but loses on harder compositional categories; Best-of-4 self-verification raises overall GenEval from 0.81 to 0.84.Verification re-ranks candidates using MODUS’s own grounding and VQA capabilities.
  • C.7. Inference Efficiency: Independent vs Chained: 10-step chained generation reaches the accuracy of 50-step independent generation at less than half the latency: 4.28 s/img versus 9.07 s/img.This benchmark uses 512×512 resolution, one GH200 GPU, and 654 NYUv2 images.
  • D.1. Independent Any-to-Any Generation: MODUS generates every other target modality from an arbitrary input within one architecture, while independent outputs are unconstrained by mutual consistency.The chained variant instead conditions each output on previously generated modalities to enforce cross-modal consistency.
  • D.2. Chained Generation: Chained text-to-image and image-to-surface-normal visualizations show coherent intermediate Canny edges, depth maps, and surface normals that remain consistent with the source input.Final outputs follow the intermediate representations across these chained tasks.
  • D.3. Self-Verification: Self-verification generates several image candidates and selects one using MODUS grounding logits, whose confidence values often correlate with requested-object presence.The procedure improves image quality and alignment with the input prompt through task-internal test-time search.

D.4. Visual Representation Composition · E. Implementation Details · E.1. Dataset

MODUS combines semantic and reconstruction features to generate plausible 2D outputs while preserving geometric fidelity, and it is trained on a 29M-sample multimodal corpus spanning extensible modality families. Its unified tokenization and training recipe supports adding modalities without architectural changes.

  • D.4. Visual Representation Composition: MODUS represents each 2D modality with complementary ViT-derived semantic and VAE-based reconstruction features.The semantic branch supports global understanding, while the reconstruction branch preserves local geometry and fine details.
  • D.4. Visual Representation Composition: ViT-only conditioning produces semantically consistent depth and surface-normal maps but can alter object shapes, instance counts, and fine geometric details.Across five depth and five surface-normal samples per image, these distortions recur systematically.
  • D.4. Visual Representation Composition: Combining ViT semantic and VAE low-level features restores geometric fidelity, preserving scene structure, object boundaries, and local surface details.The comparison highlights that semantic features alone do not sufficiently constrain spatial layout.
  • E.1. Dataset: 29M samples comprise MODUS-DATASET, which extends the BLIP-3o image–caption corpus with aligned annotations across diverse modalities.High-quality pseudo-labelers provide supervision at scale, particularly where human annotations are scarce.
  • E.1. Dataset: The dataset includes instance segmentation, edge maps, and global or spatially local feature maps generated by Grounded-SAM, SAM, Canny, DINOv2, CLIP, and ImageBind.Localization supervision combines GLaMM ground-truth boxes with ViTDet detection using an EVA-02 backbone.
  • E. Implementation Details: The implementation uses a unified tokenization and training recipe across all supported modalities.This common recipe enables modalities from different families and generation mechanisms to be handled within one framework.
  • E.1. Dataset: MODUS’s modality set is representative rather than fixed, allowing multiple instantiations within each family and new modalities without architectural changes.Examples include classical or SAM-based edges and global or spatially local feature maps.

E.2. Training

Modus training samples sequences with multiple conditioning modalities and one target modality, using modality-aware attention patterns. Sequence packing is applied to keep training sequence lengths efficient and stable.

  • Training procedure: Training sequences contain several conditioning modalities and one target modality.
  • Attention pattern: Attention between modalities is causal, while within-modality attention is causal for 1D modalities and bidirectional for 2D ViT and VAE tokens.
  • Training efficiency: Sequence packing maintains efficient and stable sequence lengths during training.

F. Limitation Discussion

MODUS remains limited in same-modality iterative editing and explicit reasoning because relevant datasets and post-training objectives are not yet incorporated. These gaps point to broader opportunities in dataset coverage and task formulation within a unified decoder framework.

  • Same-modality editing: MODUS does not yet support iterative editing within the same modality, such as refining depth or editing segmentation.This capability requires curated sequential-edit or multi-turn refinement datasets, which are scarce in existing multimodal corpora.
  • Reasoning: MODUS is trained purely through pre-training and therefore does not explicitly target complex reasoning tasks.Dedicated post-training objectives and datasets, including multimodal chain-of-thought, could address this limitation.
  • Future improvement: Addressing these limitations will require further investigation into dataset coverage and task formulation.The authors identify lightweight post-training and synthetic multimodal editing datasets as natural directions for improvement.
  • Future improvement: The results suggest strong potential for supporting a broader range of tasks within a single unified decoder framework.This potential depends on resolving the current limitations in editing support and reasoning capabilities.

G. MODUS-DATASET Examples

MODUS-DATASET examples visualize pseudo labels spanning the model’s supported modalities. The full dataset and two MODUS checkpoints are released alongside these examples.

  • MODUS-DATASET examples: The release includes the full MODUS-DATASET and two checkpoints: Modus-15modality-14B-A7B and Modus-15modality-77B-A13B.
  • MODUS-DATASET examples: Figure 16 aligns each image with annotations across RGB, caption, depth, surface normal, edges, masks, segmentation, detection, grounding, and multimodal features.Features include DINOv2, CLIP, and ImageBind representations in both global and local forms.
Loading 2607.25948v1…