MaMMUT: A Simple Architecture for Joint Learning for MultiModal Tasks

1 Introduction

• Combining contrastive and generative objectives has been challenging, with prior approaches requiring multiple components or training stages and special recipes for video.Decoder-only language models offer simpler architectures and parameter reduction.Source: §1 · p. 1
• MaMMUT combines contrastive, localization-aware, and autoregressive captioning pretraining with a single language decoder and image encoder.Its two-pass strategy uses cross-attention and causal masking for caption generation, then disables both for contrastive learning.Source: §1 · p. 2
• MaMMUT is presented as a compact model that shares weights across diverse tasks, unlike prior models that often cover fewer task categories.The introduction contrasts its task coverage with examples including PaLI, GIT, FLAVA, Florence, ALBEF, BLIP, and Flamingo.Source: §1 · p. 2
• Experiments report above-state-of-the-art results on image-text and text-image retrieval, VideoQA, and open-vocabulary detection, outperforming much larger models.VQA and Video Captioning results are described as very competitive.Source: §1 · p. 3
• The model extends to video-language tasks and open-vocabulary detection; its video extension requires no additional pretraining.Contributions include image/text retrieval, VQA, VideoQA, Video Captioning, and open-vocabulary detection.Source: §1 · pp. 2–3

2 Related Work

• Generative vision-language pretraining applies language-modeling losses to image and text inputs for tasks such as VQA, captioning, and classification.GIT is cited as using a simple decoder, but addressing text-generative tasks only.Source: §2 · p. 3
• Contrastive pretraining learns cross-modal embeddings from noisy image-language pairs, while two-tower models can require downstream modifications.CLIP and Align are cited as popularizing this approach.Source: §2 · p. 3
• Prior combined approaches use two-tower models with cross-attention or cross-modal masking; ALBEF and CoCa distinguish unimodal from multimodal text modeling.CoCa adds a second cross-attending decoder for text generation, whereas MaMMUT uses a fully shared text model.Source: §2 · p. 3
• Video-language approaches include video-first models and image-language models adapted to video, with some adaptations processing frames individually.MaMMUT instead directly processes spatiotemporal information and is described as outperforming more sophisticated, better-trained video models after light fine-tuning.Source: §2 · pp. 3–4
• Open-vocabulary detection targets novel object categories, a task often overlooked by vision-language models focused on predefined classes.The cited work presents this detection capability as an extension of MaMMUT.Source: §2 · p. 4

3 Method

• MaMMUT’s architecture also supports video and open-vocabulary detection extensions without adding encoders or repeatedly applying the encoder.The paper describes these as straightforward extensions of the same architecture.Source: §3 · p. 4, §3.1 · p. 6

3.1 MaMMUT Architecture

• A single vision encoder and text decoder support contrastive and autoregressive tasks, with the same decoder weights shared across both passes.Contrastive learning uses bidirectional masking without image cross-attention; caption generation uses causal masking with cross-attention.Source: §3 · p. 4, §3.1 · p. 5

3.2 Pretraining Losses

• Focal contrastive loss adds sensitivity to objects, which the authors say equips the model for downstream detection.The authors prefer this modification over classical contrastive loss in their model.Source: §3.2 · p. 6
• The combined training objective sums captioning and focal contrastive losses, with loss-balancing hyperparameters.The objective is shared with the full model.Source: §3.2 · p. 7
• MaMMUT is pretrained end-to-end from scratch using web alt-text data only, with about 16% overhead over a pure contrastive learner.The paper presents this as a relatively light training-efficiency overhead.Source: §3.2 · p. 7
• Cropped Positional Embedding trains on randomly cropped regions of upsampled embeddings to better match detection’s region-level recognition.This adapts full-image pretraining positional embeddings to downstream detection.Source: §3.2 · p. 7
• MaMMUT reports state-of-the-art zero-shot image-to-text and text-to-image retrieval on COCO and Flickr30K at comparable model capacity.The table caption gives no metric values, so no numerical margins are stated.Source: Table 1 · p. 8

4 MaMMUT for Video Tasks

• Unlike frame-by-frame approaches, MaMMUT applies its video model once; processing each frame individually can constrain frame count through memory and runtime limits.The paper also notes evidence that too many effective tokens can deteriorate performance.Source: §4 · p. 8
• Video tubes and sparse-stride 2D patches are concatenated and passed through the shared ViT backbone in a lightweight adaptation of TubeViT (Piergiovanni et al., 2023a).The adaptation uses both learned and fixed positional embeddings, with a weighted connection to the newly added fixed embeddings.Source: §4 · p. 8
• The video adaptation requires no additional video-data pretraining; experiments fine-tune the image-text model directly on a video dataset.Source: §4 · p. 8

5.1 Zero-Shot Image Retrieval Results

• MaMMUT outperforms state of the art on image-to-text and text-to-image retrieval by 2–4 points on recall@1.The evaluation uses zero-shot retrieval benchmarks MS COCO and Flickr.Source: §5.1 · p. 8

5.2 Visual Question Answering Results

• On VQAv2, MaMMUT is competitive among open-ended generators despite modest capacity, decoding from a 256K English vocabulary.Most prior approaches use classification, while some open-ended methods restrict inference to 3K candidate answers.Source: §5.2 · pp. 8–9, Table 2 · p. 9
• On VQAv2 test-std, yes/no questions perform best, while questions about how many objects are present are most challenging.Source: Table 3 · p. 9

5.3 Video Question Answering Results

• MaMMUT surpasses VideoQA state of the art on MSRVTT-QA and MSVD-QA in open-ended generation without video-text pretraining.The model is 2.5 times smaller than GIT2 and 40 times smaller than Flamingo.Source: §5.3 · pp. 9–10, Table 4 · p. 10

5.4 Video Captioning Results

• MaMMUT outperforms prior approaches by large margins on MSVD captioning and all but GIT/GIT2 on MSRVTT, using image-language pretraining alone.The model is directly fine-tuned on each dataset.Source: Table 4 · p. 10, §5.4 · p. 10

5.5 Open-vocabulary Detection Results

• 31.0 APr exceeds OWL-ViT by 5.4 points on open-vocabulary detection.The model is not optimized for detection AP, unlike detection-only OWL-ViT.Source: §5.5 · p. 10

Ablation setup

• The ablation setup uses a ViT-Base image encoder, a 128M language decoder, a 4K batch, and 500K iterations unless noted otherwise.Source: §6 · p. 10

Open-Vocabulary Detection

• MaMMUT performs well on novel-object detection, with the table caption highlighting its Average Precision for rare objects.The detector is trained on base categories and evaluated on novel categories at inference.Source: Table 5 · p. 11

Cross-task benefits

• Joint contrastive and text-generative pretraining improves VQA by 8 points over contrastive-only and 1.8 over generative-only, improves text-to-image retrieval, and enables retrieval tasks unsupported by generative pretraining.It remains very competitive on image-to-text retrieval, where joint training is not as beneficial.Source: §6 · p. 11

Cross-attention Design

• Denser cross-attention benefits VQA, while one or few layers are sufficient for retrieval.The prose gives four layers as an example for VQA and two for retrieval.Source: §6 · p. 11, Table 7 · p. 11

Balancing contrastive-vs-generative losses

• The smaller-model, fewer-step loss-balancing experiment finds competing objectives and selects equal loss weights to favor VQA while limiting retrieval trade-off.The caption identifies the compared retrieval measures as zero-shot R@1 on COCO and Flickr30K and VQA accuracy.Source: §6 · p. 11, Table 8 · p. 12

Scaling Image Encoder

• Increasing image-encoder capacity consistently improves image-text retrieval.The scaling ablation uses a 16K batch; its final row uses a 1B-parameter text encoder rather than 128M elsewhere.Source: §6 · p. 12, Table 9 · p. 12

Video Model Ablations

• Among the video adaptations tested, video tubes are most important, followed by fixed embeddings and gated connections.The authors attribute the video tubes' importance to learning spatio-temporal video information; the ablation uses the smaller ViT-L model.Source: §6 · p. 12

• The authors identify potential uses in understanding visual and language inputs and responding to questions more accurately.Source: §7 · p. 12
• Text-generative components may produce off-topic, stereotypical, unwanted, or other outputs, which the authors say need further investigation.Source: §7 · p. 12
• MaMMUT was trained on noisy image-language pairs and used in the paper only for evaluation and visualization.Source: §7 · p. 12

• MaMMUT uses a vision encoder and text decoder with two-pass learning to jointly train retrieval and text-generative tasks with fully shared weights.Source: §8 · p. 12
• The model covers image-text and text-image retrieval, open-vocabulary detection, VQA, VideoQA, and Video Captioning, with very strong performance relative to the state of the art.Source: §8 · p. 12
• The video-adaptation experiments compare design choices for adapting the model to video.Source: Table 10 · p. 12

Language pre-training effects

• Language-only pretraining provides small but consistent improvements on image-text retrieval benchmarks.The table caption specifies zero-shot retrieval R@1 on COCO and Flickr30K.Source: §A.1 · p. 17, Table 11 · p. 17

Projections and attention pooling

• Additional projections and attention pooling are not needed for joint learning, supporting full sharing of the text decoder between tasks.The experiment reports Flickr30K retrieval and uses 100K training iterations.Source: §A.1 · p. 17, Table 12 · p. 18

Bi-directional masking

• Bi-directional masking benefits contrastive learning more than causal masking because retrieval text is fully available before its final representation is generated.The comparison is evaluated on the Flickr dataset.Source: §A.1 · p. 17, Table 13 · p. 18

A.2 Total Train Compute Usage

• MaMMUT uses 3.4x less training compute than PaLI, 5.5x less than CoCa, 10.3x less than Flamingo, and 41.2x less than GIT-2.The estimates compare total training compute across the models.Source: §A.2 · p. 18
• Training-compute estimates use the approximation technique in Brown et al. (2020), with some estimates taken from prior papers and others based on paper details.CoCa is given as an example of a model estimated from paper details.Source: §A.2 · p. 18
• MaMMUT is trained from scratch, whereas PaLI and Flamingo are examples of methods relying on pretrained image encoders.Those encoders are pretrained using contrastive learning or image classification.Source: §A.2 · p. 18