Source-linked AI summary
MexHat: A Dataset for Hate Speech Detection in Mexican Spanish Videos
Itzel Tlelo-Coyotecatl, Hugo Jair Escalante
TL;DR
Spanish-language hate-speech resources remain scarce, although detection requires contextual interpretation of multimodal content. MexHat is a Mexican-Spanish video dataset evaluated across three-way and fine-grained labels. Its baselines show strong initial performance from text, while distinguishing hate speech remains challenging.
Problem
Spanish has almost no presence in the hate-speech dataset literature described, despite the need for culturally representative resources for context-dependent cues.
Source: §1 · p. 2Method
MexHat collects YouTube videos, annotates clips for three-way and fine-grained hate-speech tasks, and evaluates unimodal, bimodal, and multimodal representations.
Source: §2.1 · p. 3, §2.2 · p. 3, §4 · p. 13Results
Text alone or combined with other modalities provides sufficient information for strong initial performance, while hate speech remains difficult to distinguish accurately.
Source: §4 · p. 13Takeaways & Limitations
The evaluations indicate that hate speech in the Mexican setting is bound to culture and context for interpretation.
Source: §4 · p. 13Inherent class imbalance motivates future evaluations using adapted penalizations to boost performance.
Source: §4 · p. 13
Abstract
from arXiv · showhide
Ensuring online safety through content monitoring had raised Hate Speech Detection as a crucial task to be addressed. By essence the task demands the capture of contextual cues, which are essential for a precise understanding of the content's intent. Although automated detection approaches for the task have advanced significantly, the scarcity of non-English resources persists, limiting the ability of models to adapt to the subtle, context-dependent, and culturally related nature of multimodal content. In this paper, we introduce MexHat, a video dataset designed to capture the linguistic and cultural cues for the hate-speech detection task in a Mexican Spanish context. Our dataset comprises around 1k video clips annotated across two tasks: a three-way class evaluation (no negative content, offensive content and hate-speech content), and a fine-grained class evaluation including three hate-speech sub-categories. The dataset statistics and the baseline results highlight the inherent challenges associated with the task. Disclaimer: This paper contains sensitive content that may be disturbing to some readers.
1 Introduction
The introduction frames hate-speech detection as an online-safety need requiring contextual, multimodal interpretation. It motivates MexHat through the scarcity of Spanish resources and introduces a Mexican-Spanish video dataset, two classification tasks, and baselines.
1 Introduction
- Video hate-speech systems have moved from text-only, to audio-text, to joint text-audio-visual fusion, leaving interdependent cues challenging to model.Language interpretation can also depend on tone and body language, alongside linguistic expressions.Source: §1 · pp. 1–2
- Spanish-language resources remain scarce despite the need for culturally representative data to capture context-dependent hateful cues.The introduction notes Portuguese, Tagalog, English, and Chinese resources, but describes Spanish as having almost no presence.Source: §1 · p. 2
- MexHat provides a little more than 1k Mexican-Spanish video clips with multimodal features, including transcriptions, audio, and video.The authors describe it as the only available resource for this language to their knowledge.Source: §1 · p. 2
- The dataset supports three-way labels for no negative, offensive, and hate-speech content, plus fine-grained hate-speech subcategories to explore cultural and contextual cues.The two task granularities reflect the language- and culture-dependent interpretation targeted by the dataset.Source: §1 · p. 2
- The paper introduces comprehensive baselines for both tasks as a reference for content moderation that reflects the task’s subjectivity and challenges.The introduction also presents a culturally-aware annotation framework.Source: §1 · p. 2
2 Building the MexHat Dataset
MexHat is built from YouTube video clips gathered through term-based searches, seed videos, and channels or playlists, then annotated for three-way and fine-grained hate-speech labels.
2.1 Data Collection
- The collection used 56 inclusive-language terms and 29 Mexico-context offensive terms, then added similar videos from relevant seeds and publicly available channel playlists.Videos were segmented into one-minute clips, producing around 8,000 clips before filtering.Source: §2.1 · p. 3
2.2 Annotation Process
- Two annotators labeled each clip; hate-speech clips were further grouped as G1 ethnicity, nationality, or race; G2 sexism or LGBTQ+ attacks; or G3 other forms.Sixteen annotators participated across three campaigns; final labels prioritized hate speech over offensive content and no negative content when annotators disagreed.Source: §2.2 · p. 5, §2.3 · p. 5
2.3 Annotator Agreement
- Three-class κ was 0.416 and fine-grained κ was 0.514; about 466 videos (34% of the final dataset) had differing votes, most commonly NN–OF.Disagreements between offensive and hate-speech labels were less evident but remained difficult to resolve.Source: Figure 3 · p. 5, §2.3 · pp. 5–6
2.4 Text Modality between Classes
- NN word clouds contain a broader variety of general terms, while OF features “wey” and “favor” and HS features “ja” and “pinche.”The paper notes that “wey” and “pinche” can refer to someone explicitly and may be interpreted as derogatory depending on context.Source: §2.4 · p. 6, Figure 4 · p. 6
2.5 Examples of Videos
- The HS example is a home-renovation sketch labeled as possible classism, placing it in the hs3 “others” group.The section also presents examples from NN and OF.Source: §2.5 · pp. 6–7, Figure 5 · p. 7
2.6 Feature Extraction
- Whisper transcriptions were encoded with BETO, I3D supplied flow and RGB features, and VGGish supplied audio features; LSTM processing and pooling standardized temporal length to 50.Text features were projected from 768 to 384 dimensions, and video features to 1,024 dimensions.Source: §2.6 · p. 8
2.7 The MexHat Dataset
- The dataset provides train, test, and validation partitions; test-set t-SNE plots show only slight class separation for both unimodal and multimodal representations.Figure 6 uses green for NN, yellow for OF, red for HS, and “+” for concatenation.Source: §2.7 · p. 8, Figure 6 · p. 9
3 Experiments & Results
The experiments evaluate two hate-speech classification tasks using unimodal, bimodal, and multimodal representations. Results are weak overall, with Text and Text+Audio stronger in both tasks and class imbalance remaining a challenge.
3.1 Tasks
- The benchmark evaluates three-way no-negative, offensive, and hate-speech classification, plus fine-grained recognition of three hate-speech groups.Source: §3.1 · p. 10
3.2 Baselines
- Baseline representations span individual Text, Video, or Audio modalities, their pairwise concatenations, and a concatenation of all three; accuracy and macro-f1 are reported.Source: §3.2 · p. 10
3.3 Learning Models
- The MLP classifies each temporally distributed feature vector independently and assigns the video the majority-vote class; the Transformer encoder feeds concatenated three-modality vectors to an MLP.Source: §3.3 · p. 10
3.4 Results
- For three-way classification, accuracies were below 0.5 except for T and A+T; only A+T achieved a macro-f1 score slightly above 0.4.The class-level results showed no-negative content was easiest to identify, followed by offensive content; few configurations identified hate speech.Source: §3.4 · p. 10, Table 3 · p. 11
- Fine-grained performance dropped sharply: row 4 (T +A) achieved the best macro-f1 score at 0.247, while no configuration identified hs1.Some configurations showed attempts to identify hs2, while visual information in V and V + A seemed to help identify hs3.Source: §3.4 · pp. 10–11, Table 4 · p. 11
- Across both tasks, Text alone and Text+Audio were the stronger representations, but low scores and class imbalance kept hate-speech detection difficult.Source: §3.4 · p. 11
3.5 Classification Errors
- Vector-level evaluation examines class decisions within a video; for one gold hate-speech instance, A+V+T predicted correctly after Text and A+T mostly assigned other classes.The authors suggest that examining specific cues at a more detailed level could help mitigate video-level misclassification.Source: §3.5 · pp. 11–12
4 Conclusions
The paper introduces MexHat, a Mexican-Spanish multimodal video dataset with two hate-speech classification tasks and baseline evaluations. The results show that Text-based representations perform relatively well, while accurate hate-speech detection remains difficult and class imbalance motivates adapted evaluation penalties.
4 Conclusions
- MexHat’s pipeline covers video collection, annotation, feature extraction, and baseline evaluation for three-way and fine-grained hate-speech classification.The authors conclude that interpreting hate speech in the Mexican scenario depends on culture and context, with a thin line between harmless humor and offensive intentions.Source: §4 · pp. 12–13
- Text alone or combined with other modalities provided sufficient information for strong initial performance, while accurately distinguishing hate speech remained challenging.Source: §4 · p. 13
- Misclassified instances suggest that understanding speaker intention requires integrating linguistic cues with visual body-language cues.Source: §4 · p. 13
- Vector-level evaluation offers a feasible way to locate classification errors, and more precise capture of specific evidence could benefit the global decision.Source: §4 · p. 13
- Because the dataset has inherent class imbalance, the authors identify adapted evaluation penalties as future work to boost performance.Source: §4 · p. 13