Vilio: State-of-the-art Visio-Linguistic Models applied to Hateful Memes

• Hateful memes require joint image-text understanding because either modality alone may not convey the message [2].Source: §1 · p. 1
• Vilio provides a code base containing 12 vision+language models organized like the Hugging Face Transformers library.Source: §1 · p. 1
• The implementation applies performance-enhancing modifications to state-of-the-art vision+language models.Source: §1 · p. 1
• Vilio evaluates different models on the Hateful Memes Dataset and supplies code for future research on it [2].Source: §1 · p. 1

• 8500 training, 500 development, and 1000 test images include text- and vision-confounders that require both modalities for classification.Swapping only the text or image can change a meme from hateful to non-hateful or vice versa.Source: §2 · p. 1
• Figure 1 illustrates a confounder by neutralizing a hateful meme after replacing E.T. with an unrelated image.Source: §2 · p. 1, Figure 1 · p. 2
• AUROC is the competition’s key metric because it evaluates how well models rank memes by hatefulness rather than the absolute probability values.Source: §2 · p. 2

3 Approach

• Vilio’s pipeline has three stages: Preparation, Modelling, and Ensembling.Source: §3 · p. 2, Figure 2 · p. 3

3.1 Preparation

• Preparation extracts diverse image features with detectron2, varies retained RoIs from 36 to 100, and combines them with OCR-extracted meme text.Features use models pretrained on VisualGenome with and without attributes.Source: §3.1 · p. 2

3.2 Modelling

• The modelling stage fine-tunes pretrained vision-language models with binary cross-entropy, Adam optimization, warm-up, gradient clipping, and model-specific training schedules.The implementation uses a batch size of 8 and limited hyperparameter optimization because architecture changes are treated as more impactful.Source: §3.2.1 · p. 2, §3.2 · p. 2

3.2.2 ERNIE-ViL

• ERNIE-ViL uses a second 10–100-box feature set as fake ground truth because the dataset lacks ground-truth boxes, alongside VCR-pretrained weights [8] [20].Source: §3.2.2 · p. 3

3.2.3 UNITER & OSCAR

• UNITER and OSCAR receive transformer-library updates, while OSCAR adds task-specific Hateful Memes pretraining and uses an LXMERT classifier with GeLU activation [7].Without pretraining, RoBERTa and other language-transformer variants cannot match performance.Source: §3.2.3 · p. 3

3.2.4 VisualBERT

• VisualBERT separates visual and language token types, improving AUROC by an absolute 1.2%, with additional gains from multi-sample dropout and learned layer averaging.Source: §3.2.4 · p. 3

3.3 Ensembling

• Ensembling averages predictions across 3–5 seeds and combines models using Simple, Rank, Power, or Simplex averaging, with Simplex weights learned on development predictions.Source: §3.3 · p. 3

4.1 Performance & Limits

• Vilio’s ensemble closes the gap between baseline models and reported human performance on Hateful Memes, as summarized in Table 1.Source: §4.1 · p. 3, Table 1 · p. 4
• The reported human comparison may be optimistic because the tested humans were not experts, and actual human performance might approach 100%.Source: §4.1 · p. 3
• 75.40% ensemble accuracy with absolute labels trails 84.70% human accuracy on the test set [2].The authors attribute this difference to optimizing ranking, while humans perform better at binary predictions than probabilities.Source: §4.1 · p. 4
• The ensemble contains 19 trained models, and ERNIE-ViL instability motivates averaging five seeds and reducing compute in future work.Source: §4.1 · p. 4

4.2 Future Work

• Future work includes hateful GIF classification and integrating OCR into trainable models to exploit caption font and size variation.Source: §4.2 · p. 4
• The paper also proposes training single- or dual-stream transformers from scratch and testing whether an ERNIE-VisualBERT hybrid can outperform dual-stream ERNIE-ViL.Source: §4.2 · p. 4