PaperlayerView exact revision on arXiv

Source-linked AI summary

Through the Eyes of the Beholder: Biometric and Demographic Conditioning for Multimodal Sexism Detection

Ana-Maria Luisa Mocanu, Sebastian Mocanu, Ciprian-Octavian Truică, Elena-Simona Apostol

arXiv:2609.15608v1cs.CLcs.AIcs.CVcs.LG

TL;DR

Meme sexism can be subtle and subjective, with irony and visual meaning complicating detection. The paper combines annotator context with multimodal inputs in a human-centered system; it ranks 29th of 114 on Subtask 2.2 under soft evaluation and stays above baseline on Subtasks 2.1 and 2.2.

  • Problem

    Meme sexism often relies on irony and hidden visual meaning, and judgments vary with viewers’ personal biases and backgrounds.

    Source: §1 · p. 1

  • Method

    A FiLM-conditioned cross-attention model fuses multimodal inputs and annotator context, with soft-label learning and a feature-based SVM ensemble.

    Source: §1 · p. 2, §4 · p. 6

  • Results

    The system ranks 29th of 114 on Subtask 2.2 under soft evaluation and stays above the organizers’ baseline on Subtasks 2.1 and 2.2.

    Source: §6 · p. 13

  • Takeaways & Limitations

    The results support treating human-centered conditioning as a useful part of an integrated system, not as an independently decisive component.

    Source: §5.5 · p. 12

    Cross-lingual augmentation lifts Spanish performance but degrades English performance.

    Source: §5.6 · p. 13

Abstract

from arXiv · show

Detecting sexism on the internet is a fundamentally subjective task; our team, VANGUARD, addresses this challenge in the EXIST 2026 Task 2 by proposing a human-centered multimodal framework that analyses and incorporates the psychological and demographic characteristics of human annotators into the detection pipeline. We fuse five input modalities through a cross-attention architecture with Feature-wise Linear Modulation conditioning. Meme text is extracted and visually described with Gemma 4, then augmented by automatic translation between English and Spanish with NLLB-200. Text and image representations are produced by LoRAadapted XLM-RoBERTa and CLIP encoders and fused with sensor features encoded by a pretrained autoencoder. To model annotator subjectivity, we frame Subtask 2.1 as a label distribution learning problem, optimizing a Kullback-Leibler divergence loss over the full annotator label distribution. At inference time, predictions are produced by soft-voting between the deep multimodal network and a complementary SVM trained on stylometric and physiological features. Our best submission ranks 29th out of 114 on Subtask 2.2 (source intention) under soft evaluation, and the normalized ICM scores remain above the baseline on Subtasks 2.1 and 2.2, indicating that annotator-centered conditioning contributes a usable signal. We release our full pipeline and analysis to support reproducible human-centered modeling.

1. Introduction

The paper frames meme sexism detection as subjective: irony and hidden visual meaning make judgments difficult, and annotator backgrounds shape perception. Its human-centered approach incorporates annotator context and disagreement into multimodal detection.

1. Introduction

  • The paper studies the EXIST 2026 task, where memes can conceal sexism through irony and hidden visual meaning, making interpretation subjective.Source: §1 · pp. 1–2
  • Annotator disagreement motivates modeling the full distribution of opinions rather than relying only on a majority-vote binary label.The approach uses a soft-label KL divergence objective for Subtask 2.1.Source: §1 · p. 2
  • The system combines biometric and demographic conditioning, cross-lingual augmentation, VLM-based text and visual enrichment, label distribution learning, cross-attention fusion, and a neural–SVM ensemble.The neural model uses annotator eye-tracking, heart-rate variability, and demographic embeddings; EEG bandpower features are used by the SVM.Source: §1 · p. 2
  • No component, including biometric conditioning, survives Holm–Bonferroni correction across 63 comparisons, so the paper reports this as a cautionary result.Architectural variants were retrained over five random seeds.Source: §1 · p. 2
  • Gemma 4 extracts embedded meme text and generates structured visual descriptions, while English–Spanish translation expands the training data.The translations are intended to encourage language-invariant learning.Source: §1 · p. 2
  • LoRA-adapted XLM-RoBERTa and CLIP representations are fused by cross-attention, with sensor embeddings injected through FiLM conditioning.Source: §1 · p. 2

2. Related Work

Related work traces sexism detection from text features to transformers and describes the shift toward multimodal memes and annotator data. The paper situates its approach among multimodal fusion, soft-label learning, and physiological-signal research.

2. Related Work

  • The EXIST task series expanded from text-only tweets to multimodal memes and videos, and its 2026 dataset adds annotator physiological signals and demographic profiles.Source: §2 · p. 2
  • Meme meaning depends on text–image interplay, while prior multimodal models have struggled when offensiveness arises from individually benign modalities.Source: §2 · p. 3
  • Prior work models annotator disagreement with soft-label distributions; this paper also weights instances by annotator entropy and adds supervised contrastive learning.Source: §2 · p. 3
  • The paper combines physiological signals—previously used in affective computing—with a pretrained sensor autoencoder and FiLM modulation in a cross-attention architecture.Source: §2 · p. 3

3. Data Analysis

The data analysis describes the EXIST 2026 dataset and examines annotator signals, meme text, and image properties. Physiological responses differ across meme classes, while the text and image analyses characterize the data’s distributions and formats.

3. Data Analysis

  • The dataset contains 3,984 memes viewed by 16 subjects, with recorded eye movements, heart rate, and EEG.Source: §3 · p. 3

3.1. Demographic and Physiological Analysis

  • Majority voting sets the binary target at 50% agreement, while a retained soft label represents annotator disagreement for KL-divergence training.Source: §3.1 · p. 3
  • Demographic traits significantly influence labeling behavior (p < 0.001); physiological features were averaged across users, and significant features were retained at p < 0.05.Source: §3.1 · p. 3
  • Sexist memes show higher reaction times, fixation counts, and heart-rate variability (HR Std.) than non-sexist memes.Source: §3.1 · p. 3, Figure 1 · p. 4
  • Fixations and Saccades are extremely correlated (ρ ≈ 1.0), indicating redundancy, but all significant features were retained for the network to weight internally.Source: §3.1 · p. 3, Figure 2 · p. 4

3.2. Text

  • The dataset is nearly balanced between English (2,005 records) and Spanish (1,979); text averages 124.3 characters and 21.6 words, with 75% of memes at 26 words or fewer.Source: §3.2 · p. 5

3.3. Image

  • Image aspect ratios have a median of 1 and mean of 1.03; square resizing causes minimal information loss, though tall storytelling memes may benefit from multiple sub-images.Average grayscale brightness is 138.31 (σ = 43.55), centered in the mid-tones.Source: §3.3 · p. 5, Figure 3 · p. 5

4. Methodologies

The pipeline enriches memes with extracted text, visual descriptions, and translated copies, then combines content and annotator context in a multimodal model with an SVM ensemble. Its method includes sensor pretraining, cross-attention and FiLM fusion, and supervised contrastive learning.

4. Methodologies

  • The full pipeline enriches memes, trains a five-stream multimodal network with auxiliary contrastive learning, and combines its predictions with a feature-based SVM at inference.Figure 4 summarizes the stages from meme enrichment through soft-voting.Source: §4 · p. 6

4.1. Text Enhancer

  • Gemma 4 extracts embedded text and generates visual descriptions using strict sampling settings intended to capture literal meaning and avoid hallucinations.A fallback pass keeps strict settings for transcription but relaxes visual-description sampling when JSON parsing fails.Source: §4.1 · p. 6

4.2. Data Augmentation

  • NLLB-200 translates cleaned text and visual descriptions between English and Spanish, doubling training data while keeping augmented copies in their source meme's split.Splits are assigned by parent ID on the original de-duplicated dataset to prevent leakage.Source: §4.2 · p. 6

4.3. Sensor Auto Encoder

  • Four low-dimensional, noisy physiological features are encoded by an unsupervised pretrained autoencoder before multimodal training, then transferred and fine-tuned end-to-end.The encoder maps standardized four-dimensional sensor inputs to 32-dimensional representations.Source: §4.3 · pp. 6–7

4.4. Cross-Attention Fusion

  • Cross-attention grounds the visual description in image patches, while FiLM uses a 56-dimensional demographic-and-sensor context vector to modulate the fused representation.The modulated representation is concatenated with the text [CLS] embedding into a shared 512-dimensional state for task-specific heads.Source: §4.4 · pp. 7–8

4.5. Contrastive Learning

  • Supervised contrastive learning supplements the KL objective by pulling together memes with the same consensus label and separating those with different labels.It operates on a 128-dimensional projection of the final fused state.Source: §4.5 · p. 8

4.6. Ensembling

  • At inference, soft-voting combines the multimodal network with an RBF-kernel SVM trained on 17 stylometric, demographic-ratio, and physiological features.The blending weight and decision threshold are grid-searched on validation data to maximize macro F1.Source: §4.6 · pp. 8–9

5. Experiments

The experiments compare official EXIST 2026 results with organizer baselines, then test augmentation, ensembling, and architectural components. Results are strongest for source intention, while ablations and limitations qualify what can be attributed to individual components and languages.

5.1. Experimental Setup

  • Experiments use Gemma 4 for text augmentation and visual interpretation, with LoRA-adapted XLM-RoBERTa-base and CLIP ViT-B/32 encoders.Training used one NVIDIA GPU, PyTorch 2.x, and AdamW with differential learning rates.Source: §5.1 · p. 9

5.3. Augmentation and Ensemble Ablation

  • The non-augmented deep model achieved the best validation ICM-Norm (0.5210) and F1 Yes (0.7651), and the best All-split test ICM-Norm (0.4861).Translation augmentation improved Spanish but degraded English on validation; on test, the augmented ensemble achieved the best English ICM-Norm (0.5496) and F1 (0.7336), while non-augmented variants were stronger on Spanish.Source: §5.3 · pp. 9–11, Table 6 · p. 11

5.2. Results

  • 29th of 114 systems on soft-evaluation Subtask 2.2; Subtasks 2.1 and 2.2 exceed the majority-class baseline under both protocols, but hard Subtask 2.3 matches it.The reported hard ICM-Norm comparisons are 0.4861 vs. 0.2947 on Subtask 2.1 and 0.3612 vs. 0.1369 on Subtask 2.2; Subtask 2.3 scores 0.0703, equal to its baseline.Source: §5.2 · p. 9, Table 4 · p. 10, Table 5 · p. 10

5.4. Architectural Component Ablation

  • Across 63 ablation tests, no individual component effect survived Holm-Bonferroni correction, so isolated architectural benefits were not statistically detectable at five seeds.The no-image configuration had higher Subtask 2.1 ICM-Norm variance (standard deviation 0.089 vs. ≤0.023); Subtask 2.3 stayed near 0.1174±0.0018 across most configurations, while single-task training reached 0.1346±0.0246.Source: §5.4 · pp. 11–12, Table 7 · p. 11, Table 8 · p. 12

5.5. Discussions

  • The authors frame annotator conditioning as useful within an integrated system, not an independently decisive module; they also report modest validation overfitting and a wider English–Spanish gap on test.They associate soft-label KL training with reproducing annotator disagreement, matching the soft evaluation protocol.Source: §5.5 · p. 12

5.6. Limitations

  • Averaging physiological and demographic signals across annotators discards individual variation, limiting viewer-specific modeling of perceived sexism.Other stated boundaries include highly correlated eye-tracking features, EEG entering only through the SVM, and no isolated component effect detectable at five seeds after correction.Source: §5.6 · p. 13
  • Fine-grained categorization remains constrained by class imbalance and multi-task interference; cross-lingual augmentation lifts Spanish performance while degrading English.Tall storytelling memes are square-padded, compressing content, and the deep model and SVM are optimized separately and combined by grid search.Source: §5.6 · p. 13

6. Conclusions

The paper concludes that its human-centered multimodal system combines annotator and meme information, with its strongest reported result on source-intention classification. It also identifies unresolved component effects and directions for future work.

6. Conclusions

  • The system combines textual, visual, demographic, and physiological modalities through FiLM-conditioned cross-attention and uses soft-label KL divergence to model annotator subjectivity.Source: §6 · p. 13
  • 29th of 114 systems under soft evaluation on Subtask 2.2; the system also stays above the organizers’ baseline on Subtasks 2.1 and 2.2.Source: §6 · p. 13
  • Physiological responses were associated with perceived sexism, but no individual component showed a statistically detectable benefit at the tested seed count.Source: §6 · p. 13
  • Future work will explore individual-level sensor modeling, text-image attention, larger seed counts, lower-noise evaluation, and a universal model later specialized by task.Source: §6 · pp. 13–14

Declaration on Generative AI

The authors disclose using Claude Opus 4.6 to improve the paper’s clarity and style, then reviewing and editing the text and taking responsibility for its content.

Declaration on Generative AI

Related papers

Loading 2609.15608v1…