Source-linked AI summary
AI Wizards at EXIST 2026: Hierarchical Soft-Label Learning for Multimodal Sexism Identification in Memes
Matteo Fasulo, Antonio Gravina, Luca Tedeschini, Luca Babboni
TL;DR
Memes make multimodal sexism identification difficult because meaning can be implicit, ironic, ambiguous, and jointly constructed by text and images. This paper uses hierarchical conditional soft-label learning over fixed vision-language embeddings, ranking first on Task 2.3 and fourth on Tasks 2.1 and 2.2.
Problem
Memes require multimodal sexism detection that captures implicit, ironic, or ambiguous meanings arising from text-image interaction.
Method
The system hierarchically predicts annotator distributions using fixed Gemini Embedding 2 representations, gated MLPs, KL divergence, uncertainty weighting, and conditional decoding.
Results
The system ranked 1st on Task 2.3 and 4th on Tasks 2.1 and 2.2 on the official Soft-Soft leaderboards.
Takeaways & Limitations
Hierarchical soft-label learning provides a compact approach for soft-label sexism detection across all three EXIST 2026 subtasks.
Takeaways & Limitations
Reliance on proprietary Gemini Embedding 2 constrains full reproducibility of the pipeline.
Abstract
from arXiv · showhide
We present the AI Wizards submission to EXIST 2026 for multimodal sexism identification in memes. The task is composed of three, increasingly harder subtasks. We model them hierarchically as conditional soft-label prediction over empirical annotator distributions. Our system maps fixed Gemini Embedding 2 vision-language representations through a lightweight Gated MLP trained with KL divergence and homoscedastic uncertainty weighting. Our submissions ranked first on Task 2.3 and fourth on Tasks 2.1 and 2.2 on the official Soft-Soft leaderboards. The code is available at https://github.com/NLP-AI-Wizards/EXIST-2026
1. Introduction
The paper motivates robust multimodal sexism detection by the harms and ambiguity of sexist memes, and presents a hierarchical soft-label system for EXIST 2026’s three nested subtasks. Its approach combines fixed vision-language embeddings, gated MLPs, KL-based uncertainty-weighted learning, and taxonomy-aware decoding.
- Motivation: Sexist online content reinforces women’s marginalization, amplifies patriarchal structures, and causes psychological harms that suppress participation in public discourse.These harms motivate robust automated detection as a critical intervention for digital safety.
- Challenge: Memes require multimodal modeling because text-image interactions construct meanings that may be ironic, implicit, ambiguous, or culturally dependent.Humor and irony can also provide plausible deniability, complicating detection.
- Task formulation: EXIST 2026 defines three hierarchically nested subtasks: binary sexism identification, source-intention detection, and five-type multi-label categorization.Under the LeWiDi paradigm, systems predict the full empirical annotator distribution rather than a single majority label.
- Approach: The system addresses all subtasks through hierarchical conditional multi-task learning over pre-computed vision-language embeddings while directly optimizing soft-label evaluation.This formulation preserves semantic dependencies among the subtasks.
- Architecture and training: The method uses Gemini Embedding 2 features, compact gated MLP blocks, KL-divergence learning with homoscedastic uncertainty weighting, and conditional taxonomy-aware decoding.Conditional loss masking and probabilistic decoding enforce the task hierarchy without architectural overhead.
2. Related Work
Related work traces sexism detection from text-only systems toward multimodal benchmarks and video, while adapting vision-language models and soft-label methods to address cross-modal toxicity and annotator disagreement. Existing research has explored task-specific multimodal fusion and hierarchical classification, but few systems combine hierarchy with soft-label prediction.
- Related Work: Multimodal benchmarks showed that unimodal systems fail when toxicity arises from text-image interaction, extending sexism detection beyond earlier text-only platforms.The EXIST lab followed this progression from text in 2021–2023 to memes in 2024 and video in 2025.
- Related Work: CLIP and SigLIP learn joint image-text representations, whereas Hate-CLIPper adds bilinear fusion to specialize contrastive alignment for hate speech detection.The contrastive models’ general-purpose objectives are not tailored to hateful content detection.
- Related Work: LeWiDi treats inter-annotator disagreement as signal by evaluating predictions of full annotator distributions rather than majority-vote labels.Prior hierarchical approaches include cascaded architectures and conditional decoding, but few combine hierarchy with soft-label prediction.
3. Dataset and Problem Formulation
EXIST 2026 defines three hierarchical subtasks for multimodal sexist-content analysis and provides Soft-Soft empirical annotator distributions over 3,984 training and 1,053 test memes. Auxiliary physiological signals were assessed, but insufficient linear EEG separability motivated prioritizing semantic representations over physiological fusion.
- 3. Dataset and Problem Formulation: Three hierarchically organized subtasks address multimodal sexist content in EXIST 2026.The supplied passage identifies this hierarchy but does not provide the complete definition of all three subtasks.
- 3. Dataset and Problem Formulation: Task 2.1 classifies whether a meme depicts or endorses sexist content versus containing no sexist content.This is a binary YES/NO classification task.
- 3. Dataset and Problem Formulation: 3,984 training and 1,053 test memes comprise the English- and Spanish-language dataset.For the Soft-Soft track, each target is the empirical annotator distribution.
- 3. Dataset and Problem Formulation: Tasks 2.2 and 2.3 are semantically defined only when sexism probability is non-zero, motivating hierarchical conditional prediction.The Soft-Soft targets represent empirical annotator distributions.
- 3.1. Physiological Data and Modality Selection: EEG provides 16 channels, 5 frequency bands, and 80 bandpower features, alongside gaze at 200 Hz and heart-rate recordings.These are the auxiliary physiological modalities supplied by EXIST 2026.
- 3.1. Physiological Data and Modality Selection: EEG was prioritized because it has the highest feature dimensionality and the most established link to implicit cognitive and evaluative processing.Gaze and heart rate were not subjected to the same statistical testing procedure, and their potential utility remains untested.
- 3.1. Physiological Data and Modality Selection: 21 PCA components explaining ∼95% of total variance preceded MANOVA; Tasks 2.1 and 2.3 showed no significant linear separability.Task 2.2 was statistically significant (𝑝perm = 0.0010) but had negligible effect (Wilks’ 𝜆= 0.9946, partial 𝜂2 < 0.01).
- 3.1. Physiological Data and Modality Selection: Linear analysis found insufficient signal in the provided physiological representations, so semantic representations were prioritized over physiological fusion.Non-linear biosignal integration and deeper biosignal modeling remain future-work directions; gaze and heart rate also remain untested under the same procedure.
4. Methods
The method combines frozen multimodal embeddings with a lightweight Gated MLP, soft-label training, and hierarchical conditional constraints. Homoscedastic uncertainty weighting and masked downstream losses adapt optimization across heterogeneous subtasks, while an ablation indicates explicit architectural gating is unnecessary.
- Method principles: The system extracts multimodal semantics with a frozen Gemini Embedding 2 encoder and enforces hierarchical constraints through conditional training and decoding.Its design targets limited-data settings, annotator disagreement distributions, and hierarchical task structure without fine-tuning massive models.
- Architecture: A lightweight Gated MLP applies one SwiGLU block with expansion factor 2 and dropout 0.2 to each fixed 768-dimensional meme embedding.The resulting representation feeds three independent linear classification heads.
- Architecture: 3.5M trainable parameters comprise the Gated MLP and classification heads, whose gating suppresses benign confounders and amplifies hostile cross-modal contradictions.The backbone produces a shared representation h for the three subtasks.
- Soft-label objective: KL divergence trains each head against empirical annotator distributions, with binary KL independently applied per category and averaged for Task 2.3.This objective penalizes probability mass assigned to categories collectively rejected by annotators.
- Multi-task optimization: Homoscedastic uncertainty weighting learns task variances, while masking Tasks 2.2 and 2.3 gradients to instances with downstream annotations.The weighting supports self-paced learning, and the log σ_i term prevents complete task abandonment.
- Architectural ablation: Comparable performance of explicit stopped-gradient structural conditioning suggests conditional loss masking and probabilistic decoding alone capture the hierarchical constraints.The architectural ablation scales downstream features by predicted sexism probability while preventing downstream losses from corrupting the Task 2.1 detector.
5. Training and Inference
Training uses stratified data partitioning, specified optimization and scheduling choices, mixed precision, and validation-loss early stopping. Inference applies threshold-based hierarchical decoding, while submitted runs vary model seeds or use detached soft gating.
- Training: 80/10/10% stratified partitions split the training data by Task 2.1 majority label.The split proportions are 80%, 10%, and 10%.
- Training: AdamW training uses learning rate 10^-4, weight decay 10^-2, batch size 8, OneCycleLR, bf16 precision, and validation-loss early stopping.OneCycleLR uses 30% warm-up and cosine annealing.
- Training: Three runs comprise two base Gated MLP models with different random seeds and one detached soft-gating variant.Runs 1–2 are the base models, while Run 3 uses detached soft gating.
- Inference: Hard decoding thresholds ^p(1) at 0.5, assigning downstream labels null below the threshold, and thresholds Task 2.2 ^p(2) at 0.5 otherwise.The hard-label predictions were submitted to the Hard-Hard track without threshold tuning.
- Inference: For Task 2.3, categories with ^p(3)_c ≥0.5 form the active set; empty sets select the maximum-probability category to ensure at least one category.No task-specific tuning was performed.
6. Results
Results show seed-sensitive local performance but strong official Soft-Soft rankings, especially first place on Task 2.3. The system favors soft-label prediction, while probabilistic outputs still support meaningful hard-label performance without threshold tuning.
- Local results: Local results show meaningful seed variance, with Run 2 best on Task 2.3 and Run 1 leading Task 2.2.The gating variant performed comparably to the base architecture, indicating that conditional losses and probabilistic decoding capture hierarchical constraints without architectural modification.
- Official results: 1st on Task 2.3 and 4th on Tasks 2.1 and 2.2 were the system’s official Soft-Soft leaderboard rankings.These were the best official Soft-Soft results reported per task.
- Official results: Soft-label prediction and discrete decision quality are related but distinct objectives, with hard-label performance likely benefiting from task-specific threshold optimization.The Hard-Hard results were obtained without threshold tuning or hard-label-specific calibration.
- Official results: The system is best matched to soft-label prediction and performs strongest on the most challenging subtask, Task 2.3.Probabilistic outputs retain enough discriminative structure for non-trivial hard-label results even with naive thresholding.
7. Limitations
The methodology has three principal limitations, including reliance on proprietary Gemini Embedding 2 and fixed-threshold hard-label decoding. Its modular architecture nevertheless permits substitution with open-weight alternatives.
- Reproducibility: Gemini Embedding 2 is proprietary, constraining full reproducibility of the proposed pipeline.The modular architecture allows straightforward substitution with open-weight alternatives in future work.
- Decoding: The hard-label decoding procedure uses a fixed decision threshold of 0.5.Consequently, the passage cautions against interpreting Hard-Hard results as an upper bound, though the supplied text is truncated.
8. Conclusion
The paper presents an efficient hierarchical multi-task system for soft-label sexism detection in memes, combining frozen multimodal representations with compact gated prediction modules and annotator-distribution training. It achieves strong official Soft-Soft leaderboard results, ranking first on Task 2.3 and fourth on Tasks 2.1 and 2.2.
- The system performs hierarchical multi-task soft-label sexism detection in memes.
- 3.5M trainable parameters enable efficiency through frozen Gemini Embedding 2 representations and compact gated MLP blocks.Training uses KL divergence against annotator distributions and learned homoscedastic uncertainty weighting.
- 1st on Task 2.3 and 4th on Tasks 2.1 and 2.2 were achieved on the official Soft-Soft leaderboards.
Declaration on Generative AI
The authors used OpenAI-ChatGPT for grammar and spelling checks and for paraphrasing and rewording during the work’s preparation. They reviewed and edited the resulting content and retain full responsibility for the publication.
- Declaration on Generative AI: The authors used OpenAI-ChatGPT for grammar and spelling checks and for paraphrasing and rewording, then reviewed and edited the content.They state that they assume full responsibility for the publication’s content.