4M: Massively Multimodal Masked Modeling

• The motivation is that vision models trained only on RGB with a single objective have not shown the broad multitask behavior seen with language modeling.The introduction frames vision as requiring diverse sensory inputs and a wide range of tasks.Source: §1 · p. 1
• 4M trains one Transformer encoder-decoder with multimodal masked modeling, unifying modalities as discrete tokens and predicting randomized subsets.The approach targets scalability across data, architecture, and training objective while maintaining compatibility with multiple modalities.Source: §1 · p. 2
• Pseudo-labeling aligned multimodal data from image datasets allows training without requiring those datasets to have multimodal or multitask annotations.CC12M is cited as an example binding dataset.Source: §1 · pp. 2–3
• 4M performs many key vision tasks out of the box, transfers to unseen downstream tasks and modalities, and supports steerable multimodal generation and editing.The authors also report extensive ablation analyses of factors affecting performance.Source: §1 · p. 3

2 Method Description

• The method combines tokenization of modalities, one compatible Transformer encoder-decoder, and multimodal masked pre-training over randomized token subsets.This design is intended to support cross-modal predictive coding while remaining simple and computationally efficient.Source: §2 · p. 3

2.1 Modalities & data

• Pre-training covers RGB, captions, depth, surface normals, semantic maps, bounding boxes, and CLIP features, with pseudo-labeling used to assemble aligned multimodal data.Modality-specific tokenizers map these formats to discrete tokens; captions and boxes use WordPiece, while image-like and CLIP modalities use learned VQ-VAEs.Source: §2.1 · pp. 3–4
• Chained generation feeds each completed modality into generation of the next, producing multiple outputs that are self-consistent.The illustrated sequence generates RGB from partial RGB and bounding-box inputs, then generates a caption.Source: Figure 3 · p. 5

2.2 Multimodal Transformer

• The encoder embeds modality tokens with modality and positional information and can also accept RGB pixels through a learnable patch-wise projection.The decoder handles dense image-like and sequence-like outputs with shared cross-attention access to encoded tokens and modality-separated attention masks.Source: §2.2 · p. 5

2.3 Multimodal masking strategy

• Input and target masking each sample a small token subset, limiting decoder cost while supporting pre-training across many modalities.Token counts are sampled per modality using a symmetric Dirichlet distribution; dense modalities use random token sampling and sequences use span masking.Source: §2.3 · pp. 5–6

• 4M transfers exceptionally well across downstream tasks, outperforming baselines on detection, segmentation, and depth estimation, but specialized models such as DeiT III outperform it on ImageNet-1K.Preliminary experiments with a 4M-XL model with 2.7B parameters showed overfitting to the pseudo labeled CC12M dataset, yielding limited additional improvement on downstream tasks.Source: §3 · p. 6
• 4M’s in-painting and any-to-any prediction support multimodal editing, including geometry-grounded generation and semantic edits.The figure also illustrates weighting input modalities to control their influence on generation.Source: Figure 4 · p. 7
• Manipulating bounding-box placement or a semantic class illustrates how 4M predicts scene-consistent image changes.The potted-plant examples vary box placement to produce geometrically and physically plausible arrangements.Source: Figure 5 · p. 7

4 Generative Capabilities & Probing the Learned Representation

• 4M combines arbitrary cross-modal conditioning with masking-based in-painting and out-painting for semantic editing, geometric grounding, and weighted multi-condition generation.Conditions can be hand-specified or extracted from an image using 4M itself.Source: §4 · pp. 7–8
• 4M can generate every pre-training modality by iteratively decoding tokens, either unconditionally or conditioned on other modalities.Source: §4 · p. 7
• The reported image-generation results use a 4M-XL model and include specialized super-resolution and use-case fine-tuning to improve fidelity.Source: §4 · pp. 7–8
• Manipulating part of an input while holding the rest fixed reveals 4M predictions of geometrically and physically plausible arrangements that account for scene semantics.Source: §4 · p. 8

5 Ablations

• The ablations vary dataset size, training length, model size, pre-training inputs and targets, and masking choices, measuring transfer validation cross-entropy on tasks including RGB-to-target and cross-modality transfers.The transfer set includes COCO detection, ADE20K segmentation, and Taskonomy and Hypersim tasks.Source: §5 · p. 8

5.1 Reference model

• The reference model uses 12 encoder and 12 decoder layers, trains on CC12M for 100B tokens, and samples 12 input and 12 target tokens using a symmetric Dirichlet distribution with α = 0.2.Source: §5.1 · p. 9

5.2 Input modalities and target tasks

• Pre-training on all target modalities consistently gives the best average loss across the tested input settings, while multimodal pre-training helps new-input transfers but lowers performance on transfers using RGB alone.The paper identifies all target modalities as the preferred configuration when downstream uses are unknown or varied.Source: §5.2 · p. 9, Table 2 · p. 9

5.3 Multimodal masking strategy

• Uniform sampling over the modality-proportion simplex performs best on average, but only by a small margin; 128–256 visible input tokens perform well, and decoding a small random target subset performs well while reducing computation.Source: §5.3 · p. 10, Figure 6 · p. 10

5.4 How well does 4M scale?

• 4M scales with dataset size, training length, and model size across experiments ranging from 1/64 of CC12M to models with 705M parameters.Training length is measured as total input and target tokens seen.Source: §5.4 · p. 10

6 Related Work

• Many vision scaling efforts train specialized models for a single task and modality, while multi-task or multimodal approaches often require modality-specific modeling choices.Source: §6 · p. 10
• Image-based unification struggles with text and neural feature maps; tokenization-based methods address this, and 4M uses modality-specific tokenizers to cover visual modalities beyond images.4M builds on MultiMAE's multimodal masking and extends it beyond image-like modalities.Source: §6 · pp. 10–11
• Many generative models focus on text-to-image or offer limited conditioning, whereas 4M can condition on any subset of training modalities and generate those modalities.Some alternatives require a separate model for each new modality.Source: §6 · p. 11

7 Conclusion and Limitations

• 4M combines out-of-the-box vision tasks, downstream transfer, and multimodal generative editing in a single generalist framework.Its in-painting and any-to-any generation capabilities support these editing tasks.Source: §7 · p. 11
• 4M can benefit from more modalities, better tokenizers, and larger, higher-quality training datasets.Smaller tokenized image regions or higher resolutions may improve image quality but are expected to cost more computation; web-scraped data may contain low-quality images and mismatched captions.Source: §7 · p. 11

A.1 Token super-resolution

• 4M-SR fine-tunes 4M-L to map low-resolution tokens to high-resolution tokens, adding image detail while doubling resolution.It uses 14×14 tokens for 224×224 images and 28×28 tokens for 448×448 images.Source: §A.1 · p. 20, Figure 7 · p. 20

A.2 Generation-specific specializations of 4M

• Generation-specific fine-tuning skews token sampling toward fully unmasked captions to optimize 4M for text-to-image generation.The specialization continues pre-training for 50B tokens, or 10% of the pre-training length.Source: §A.2 · pp. 20–21

A.3 Generation procedure details

• The generation schedule sets each step’s target modality, decoding scheme, token count, temperature, and sampling values.Image-like modalities use MaskGIT or random-order autoregressive decoding; sequence modalities use left-to-right autoregressive decoding.Source: §A.3 · p. 21
• 4M can chain generated modalities as conditions for subsequent generation, and intermediate modalities such as CLIP tokens can improve caption-to-RGB fidelity.Classifier-free guidance combines conditioned and unconditioned logits; multimodal guidance weights multiple conditions, including negatively, for steerable edits.Source: §A.3 · pp. 21–22

A.4 Additional visualizations

• 4M predicts surface normals, depth, semantic segmentation, object detections, and captions from RGB inputs.The examples illustrate common RGB-to-X vision tasks.Source: §A.4 · p. 22, Figure 8 · p. 22
• 4M generates any modality from one or more input modalities, and chained generation can improve image fidelity with intermediate modalities such as CLIP.Examples include generating outputs from a single starting modality and conditioning image generation on caption-derived intermediates.Source: Figure 9 · p. 22, Figure 8 · p. 22, §A.4 · p. 23
• Masked image regions can be filled conditionally or unconditionally, forming a basis for multimodal editing.Conditions can include the masked image and a caption or other modalities.Source: §A.4 · p. 24, Figure 12 · p. 24
• 4M generates image variations grounded in extracted properties such as surface normals, while flexibly using combinations of input modalities.Unlike ControlNet, 4M can generate the conditioning modalities itself and performs this with one model trained in one run.Source: §A.4 · p. 24, Figure 13 · p. 25
• Multimodal guidance uses weak or strong condition weights to control alignment and negative weights to steer away from concepts.Examples vary the weight of a normal or segmentation map while keeping caption weight fixed, or combine a caption and bounding box with a masked image.Source: §A.4 · p. 26, Figure 14 · p. 26, Figure 15 · p. 27
• Representation probes manipulate one input aspect while holding the rest fixed, revealing how 4M responds to changed bounding boxes or segmentation classes.The examples show generations changing with box placement and remaining plausible when a segment’s class is changed.Source: §A.4 · p. 29, Figure 18 · p. 29, Figure 19 · p. 29

B.1 Pseudo labeled multimodal training dataset details

• CC12M’s aligned RGB images and captions were augmented with pseudo-labels from off-the-shelf models for the remaining tasks.DPT-Hybrid produced normals and depth; Mask2Former supplied segmentation labels, ViTDet supplied boxes, and CLIP supplied feature maps.Source: §B.1 · p. 30

B.2 Tokenization of captions & bounding boxes

• Bounding boxes are serialized as ordered object sequences with separate coordinate tokens, allowing masked spans without ambiguous coordinate meanings.Objects are ordered by distance to the origin; each of the four coordinate types has 1,000 tokens.Source: §B.2 · p. 30
• A joint WordPiece vocabulary of 30K tokens covers captions, 4,000 special box-coordinate tokens, and shared COCO class-label tokens.COCO class labels use the same tokens in captions and boxes.Source: §B.2 · p. 30

B.3 Tokenization of dense modalities

• Dense modalities use VQ-VAE variants; RGB, normals, and depth use diffusion decoders, while segmentation and CLIP feature maps use ViT-based encoders and decoders.Segmentation reconstruction uses cross-entropy and CLIP reconstruction uses smooth L1 loss.Source: §B.3 · pp. 30–31
• The RGB, normal, and depth tokenizers train their encoder and diffusion decoder end-to-end, then receive multi-resolution fine-tuning for resolutions up to 448 × 448.All tokenizers receive one CC12M fine-tuning epoch except RGB, which receives five.Source: §B.3 · pp. 31–32

C.1 Additional 4M architecture details

• The encoder and decoder share modality embeddings, and 4M treats raw RGB pixels as a separate input-only modality alongside tokenized RGB.Sequence-modality decoder input and output embeddings are also shared.Source: §C.1 · p. 32
• For variable-length sequences, input masking increases until the sequence fits its token budget, while over-budget targets are truncated starting at a sentinel token.The input masking probability is sampled uniformly; dense modalities can respect their token budgets strictly, unlike variable-length sequences.Source: §C.1 · p. 32

C.2 Training details

• 4M-B trained for 1.5 days on 64 A100 GPUs, 4M-L for 3 days on 128, and 4M-XL for 8 days on 128.For 4M-XL, activation checkpointing, optimizer-state and gradient sharding, and PyTorch FSDP reduced GPU memory consumption.Source: §C.2 · pp. 32–33
• The model-size variants follow T5’s naming scheme, and their reported parameter counts exclude embedding layers.The table lists encoder and decoder depths, model dimension, and attention-head counts for each size.Source: Table 4 · p. 33
• The training configuration for the 4M models used in transfer experiments and generation results is reported in Table 5.The supplied passage identifies the table’s scope but does not provide its configuration values.Source: Table 5 · p. 33

D Transfer Experiments Details

• Transfer settings were adjusted to keep any single transfer below 20% of pre-training compute.Adjustments included lower COCO fine-tuning resolution and fewer intermediate ImageNet-21K fine-tuning epochs.Source: D Transfer Experiments Details · p. 33

D.1 Architectural differences with the baselines

• Although 4M and its baselines have nearly identical parameter counts and FLOPs, their architectures differ in biases, feedforward activations, [CLS] tokens, and positional embeddings.The stated differences include SwiGLU versus GELU, and fixed versus learnable positional embeddings in specified baselines.Source: §D.1 · p. 33

D.2 Image classification on ImageNet-1K

• ImageNet-1K evaluation uses intermediate fine-tuning on ImageNet-21K before full fine-tuning, except for DeiT III, which was originally trained on ImageNet-21K.The intermediate stage is reported to improve overall performance for models not originally trained on ImageNet-21K.Source: §D.2 · p. 34

D.3 Object detection and instance segmentation on COCO

• COCO detection and instance segmentation use a simplified Cascade Mask-RCNN head and 512 × 512 resolution, over 4× cheaper to train but lower-performing than 1024 × 1024.The setup retains global attention rather than using window attention.Source: §D.3 · p. 34

D.4 Semantic segmentation on ADE20K

• ADE20K semantic segmentation uses a depth-4 ConvNext prediction head on the encoder.The settings follow MultiMAE.Source: §D.4 · p. 35

D.5 Depth estimation on NYUv2

• NYUv2 depth estimation replaces MultiMAE’s DPT head with a depth-2 ConvNeXt head, which the authors report performs comparably on that baseline.The replacement removes the need to extract and rescale intermediate-layer features.Source: §D.5 · p. 35

E.1 Benchmark tasks

• The benchmark includes RGB-input detection, segmentation, and transfers to geometric and semantic targets, plus sampled single- and two-input cross-modal tasks.The sampled tasks span Taskonomy-20K and Hypersim; transfer from unseen modalities requires training new input embeddings.Source: §E.1 · pp. 36–37
• Validation loss is measured 50 times per transfer, with the best value reported to account for different overfitting speeds.Models use a constant learning rate and evaluation uses an exponential moving average of weights.Source: §E.1 · p. 36
• For each task and modality, transfer uses a new tokenizer trained on the task’s training set, and unseen input modalities receive staged embedding and model training.The Transformer is initially frozen while new input and output embeddings are trained, then the full model is unfrozen.Source: §E.1 · p. 36
• Transfer comparisons report validation cross-entropy rather than task-specific metrics, isolating token prediction because each setting uses the same tokenizer.The authors also cite cross-entropy’s uniformity across tasks with otherwise incomparable metrics.Source: §E.1 · p. 36

E.2 Reference model

• The reference model uses 12 encoder and 12 decoder layers, all modalities as inputs and targets, 128 sampled input and target tokens, and 100B tokens seen.It is trained on CC12M at 224 × 224 resolution without cropping or color augmentation; the input and target modality samplers use a symmetric Dirichlet distribution with α = 0.2.Source: §E.2 · p. 37, Table 14 · p. 38
• Transfer results are averaged over ten reference models trained with different seeds, and the reported table also includes randomly initialized models as a lower bound.Results within two standard deviations of the lowest losses are highlighted to estimate significance.Source: §E.2 · pp. 37–38, Table 15 · p. 38

E.3 Input modalities and target tasks

• Transfer preferences vary: no target-task setting wins every transfer, all-target pretraining gives the best average loss, and all-modality inputs help unseen-modality transfers but cost RGB-only performance.When the optimal downstream setting is unknown, the table caption identifies all-input, all-target pretraining as the most versatile choice.Source: §E.3 · pp. 38–39, Table 16 · p. 39

E.4 Multimodal masking strategy

• Higher Dirichlet α generally favors RGB-task transfers, while lower input α favors transfers to novel input modalities.Lower α tends to concentrate sampled tokens in fewer modalities; higher α tends toward equal token counts across modalities.Source: §E.4 · p. 39, Table 17 · p. 39
• Decoding a small random subset of targets improves fixed-duration performance while reducing compute, and mixing RGB-only with all-modality inputs offers a compromise when deployment needs are unknown.The mixture does not beat either individual masking strategy on any reported transfer.Source: §E.4 · p. 40, Table 19 · p. 40, Table 20 · p. 41

E.5 How well does 4M scale?

• 4M scales with larger CC12M subsets, longer training schedules, and model sizes from Tiny to Large.The duration ablation counts both input and target tokens as tokens seen.Source: §E.5 · p. 41

E.6 Architectural design choices

• With 24 layers, balanced and encoder-heavy allocations slightly edge decoder-heavy; per-modality loss outperforms per-token loss, and SwiGLU helps while query-key normalization hurts transfer.The authors use SwiGLU and omit query-key normalization in the final model.Source: §E.6 · p. 42

E.7 Training design choices

• Repeatedly sampling buffered images with new random masks can improve training efficiency without significant performance loss.Samples are replaced with fresh ones after the specified repeat count.Source: §E.7 · p. 43, Table 28 · p. 43

E.8 Self-baselines

• The RGB→RGB self-baseline benefits from higher masking and no input-target overlap, whereas RGB→CLIP benefits from lower masking and spatial overlap.The ablation also includes an unmasked standard RGB→X setting.Source: §E.8 · p. 43, Table 29 · p. 43

E.9 Comparison to the final models

• The final 4M models significantly outperform reference models on all benchmark tasks; the masking-strategy mixture explains much of the RGB-input task gap.The final models combine insights from the ablations.Source: §E.9 · p. 44, Table 30 · p. 44

F.1 Out of the box (zero-shot) performance

• Without fine-tuning, 4M matches or outperforms strong baselines on surface normals, depth, and segmentation, and is competitive with its pseudo-labelers.The evaluation uses DIODE for normals and depth and COCO for segmentation at 224 × 224 resolution.Source: §F.1 · p. 44, Table 31 · p. 44

F.2 Text-to-image performance

• In the controlled text-to-image setting, 4M matches the Muse-like specialist on FID and CLIP scores, but has a larger gap to Stable Diffusion 2.1 on COCO.The authors note that Stable Diffusion 2.1 used orders of magnitude more data and compute; 4M can also be conditioned on any pre-training modality and perform vision tasks out of the box.Source: §F.2 · p. 45, Table 32 · p. 45

G.1 Computational costs

• Training took 1.5 days for 4M-B on 64 A100 GPUs, 3 days for 4M-L on 128, and 8 days for 4M-XL on 128.The ablation reference model took 12 hours on 32 A100 GPUs, and transfer evaluation across 35 tasks took about 3 days on 8.Source: §G.1 · p. 46

G.2 Social impact

• Because CC12M may retain societal biases despite curation, the authors advise caution when using these models generatively.They note that training data may include samples gathered in different ways from the internet.Source: §G.2 · p. 46