MixGen: A New Multi-Modal Data Augmentation

1. Introduction

• Large-scale vision-language training can require extensive compute; CLIP’s ImageNet example used 400M image-text pairs for 12 days on 256 V100 GPUs.The passage also notes that many large-scale datasets are not publicly accessible and that reproduction or further improvement is challenging for researchers with limited compute.Source: §1 · p. 1
• Augmentation is difficult for paired vision-language data because changing an image may require corresponding changes to color or positional words in its text.Prior work generally uses mild image augmentation or leaves text augmentation to masked language modeling; some co-augmentation methods target specific downstream tasks rather than generic pretraining.Source: §1 · p. 1
• MixGen creates image-text pairs by interpolating images and concatenating text, with semantic relationships expected to remain matched in most cases.The figure depicts the same operation on two existing image-text pairs to produce a new pair.Source: §1 · p. 1, Figure 1 · p. 2
• +6.2% on COCO fine-tuned image-text retrieval and +5.3% on Flicker30K zero-shot retrieval accompany ALBEF improvements across five downstream tasks.The other reported gains are +0.9% on RefCOCO+, +0.9% on NLVR2, +0.3% on VQA2.0, and +0.4% on SNLI-VE.Source: §1 · p. 2

2. MixGen

• MixGen creates a new pair by linearly interpolating the raw pixels of two images and concatenating their text sequences.The random combination is intended to preserve semantic relationships in most cases and increase training diversity.Source: §2 · p. 2, Figure 2 · p. 3

2.1. MixGen variants

• The five design variants test sampled rather than fixed image-mixing weights, image-only or text-only mixing, and alternative ways to select text tokens.The alternatives include sampling λ from a Beta distribution, choosing one image or text sequence, selecting tokens proportionally, or retaining half of concatenated tokens.Source: §2.1 · pp. 2–3
• The default MixGen variant achieves the overall best performance and consistently outperforms the other variants across four downstream tasks.The study compares five straightforward variants of image mixup and text augmentation.Source: §2.1 · pp. 2–3

2.2. Input-level and embedding-level MixGen

• Input-level MixGen consistently performs better than embedding-level MixGen and avoids modifying the network architecture or model forward pass.Input-level augmentation acts on raw inputs; embedding-level augmentation mixes image and text features extracted by encoders.Source: §2.2 · p. 3

3.1. Pre-training

• Experiments add MixGen to CLIP, ViLT, ALBEF, and TCL without modifying their original training settings; the available pretraining set has 3.3M images and 4.4M pairs.The paper calls this dataset setting 3M, compared with the 4M setting of prior work using 5.1M pairs.Source: §3.1 · pp. 3–4

3.2. Image-text retrieval

• MixGen boosts retrieval across four architectures; with 3M ALBEF, it raises COCO fine-tuned RSUM by 6.2% and Flickr30K zero-shot RSUM by 5.3%.The results also report retrieval gains for ViLT, CLIP, ALBEF, and TCL under a 1M-image setting.Source: §3.2 · pp. 4–5, Table 1 · p. 4, Table 2 · p. 4
• ALBEF with MixGen trained on 1M, 2M, and 3M samples matches baseline ALBEF trained on 2M, 3M, and 4M samples, respectively.MixGen is reported to help at every tested data level, with larger improvement in the low-data regime.Source: §3.2 · p. 5, Figure 3 · p. 5

3.3. Evaluation on VQA, VR, VE and VG

• 0.28% on VQA test-std, 0.89% on NLVR2 test-P, and 0.36% on SNLI-VE test are ALBEF's absolute gains with MixGen under 3M.The 3M MixGen model also outperforms original 4M ALBEF on VQA and NLVR2.Source: Table 4 · p. 5, §3.3 · p. 6
• Under 3M training, ALBEF with MixGen surpasses its baseline on both RefCOCO+ test sets and outperforms 14M ALBEF on TestB.The comparison uses ALBEF_itm as the baseline because it performs better than ALBEF_itc.Source: Table 5 · p. 5, §3.3 · p. 6

3.4. Visualizations of MixGen

• Retrieval visualizations show MixGen usually ranks the matching image in the top three, while Grad-CAM examples show more precise grounding and attention to relevant VQA cues.The visualizations also include failures, including a time-telling VQA example where both models fail.Source: §3.4 · p. 6, Figure 5 · p. 6, Figure 4 · p. 6

Design variants

• Default MixGen achieves the best overall performance among the five variants, outperforming the others across tasks except VQA.The variants use the same ALBEF 1M pretraining setting and are evaluated across five downstream datasets.Source: §4 · p. 7

Input-level vs embedding-level MixGen

• Input-level MixGen consistently outperforms embedding-level MixGen, although both perform better than the baseline.Source: §4 · p. 7

Optimal ratio of MixGen samples

• With batch size B=512, M=128 (B/4) generated pairs leads to the best performance.The ablation varies how many training samples are replaced with newly generated pairs.Source: §4 · p. 7

Limitations

• The study compares only selected related augmentation configurations, leaving other popular unimodal techniques for future work.The authors cite the unlimited combination of possible configurations as the reason for limiting comparisons.Source: §5 · p. 7
• Computational limits restrict pretraining comparisons to the widely adopted 4M set, leaving larger datasets such as LAION-400M unexplored.Source: §5 · p. 7
• Evaluation covers four models and five vision-language tasks, leaving generalization to other architectures and tasks untested.Examples proposed for future investigation include generation, video-text retrieval, text-to-image generation, detection, and segmentation.Source: §5 · p. 7

Ablation findings

• Input-level MixGen performs consistently better than embedding-level MixGen, and M=128 (B/4) is optimal for batch size B=512.Source: Figure 6 · p. 8

Vision-language pre-training

• Although CV and NLP have many augmentation techniques, joint multimodal augmentation is rarely studied, and existing joint approaches target specific downstream tasks,.Source: §6 · p. 8

Relation to mixup

• Mixup generates samples by combining inputs and one-hot labels; an NLP adaptation replaces image pixels with text embeddings [12].Source: §6 · p. 8
• Unlike mixup, MixGen uses data from different modalities without one-hot labels and concatenates text sequences rather than mixing them.The authors state that concatenation best preserves text information.Source: §6 · p. 8

• MixGen consistently improves four recent vision-language models across five downstream tasks and is reported to improve their data efficiency.The authors present the results as empirical evidence for joint multimodal data augmentation.Source: §7 · p. 8