PaperlayerView exact revision on arXiv

Source-linked AI summary

Wazobia Eval: A Benchmark for Nigerian Pidgin Emotion Understanding, Sarcasm Detection, and Cultural Reasoning

Stephanie Okoye

arXiv:2608.21369v1cs.CLcs.AI

TL;DR

Nigerian Pidgin remains underrepresented in language-model evaluation, especially for culturally grounded emotion, sarcasm, and reasoning. Wazobia Eval introduces a manually annotated benchmark with a 16-category taxonomy and dedicated evaluation tracks; preliminary pilots indicate that context-dependent understanding remains challenging for contemporary models.

  • Problem

    Existing evaluation resources give limited attention to Nigerian Pidgin's emotional, cultural, and pragmatic dimensions, which are not fully captured by translation or generic sentiment tasks.

  • Method

    Wazobia Eval combines a manually annotated dataset of more than 550 Nigerian Pidgin examples, a 16-category emotion taxonomy, and tracks for emotion understanding, sarcasm detection, and cultural reasoning.

  • Results

    Preliminary pilot evaluations suggest that culturally grounded Nigerian Pidgin understanding remains challenging, particularly when interpretation depends on context, sarcasm, or culturally specific emotional concepts.

  • Takeaways & Limitations

    The benchmark provides evaluation infrastructure that moves assessment beyond translation and lexical recognition toward deeper Nigerian Pidgin language understanding.

  • Takeaways & Limitations

    Formal inter-annotator agreement analysis has not yet been completed for the full manually annotated dataset.

Abstract

from arXiv · show

Nigerian Pidgin is one of Africa's most widely spoken languages, yet remains severely underrepresented in language model evaluation. Existing benchmarks primarily focus on translation, transcription, or generic sentiment analysis, leaving critical aspects of culturally grounded language understanding unmeasured. We introduce Wazobia Eval, a benchmark for evaluating Nigerian Pidgin emotion understanding, sarcasm detection, and cultural reasoning. The benchmark is built on a manually annotated dataset containing over 550 examples and a 16-category emotion taxonomy designed to capture culturally specific emotional registers that are not represented in conventional sentiment frameworks. Wazobia Eval provides standardized evaluation protocols and benchmark tasks for assessing model performance on nuanced Nigerian language understanding. We present the benchmark design, annotation methodology, taxonomy development process, and preliminary pilot evaluation results. Our goal is to provide foundational evaluation infrastructure for Nigerian language AI and establish a reproducible benchmark for future research. The dataset is publicly available at https://huggingface.co/WAZOBIALABS.

1. Introduction

Wazobia Eval addresses the underrepresentation of Nigerian Pidgin in language-model evaluation by targeting culturally grounded emotion, sarcasm, and reasoning capabilities. It combines a manually annotated dataset, a 16-category emotion taxonomy, and standardized evaluation infrastructure.

  • Nigerian Pidgin is widely spoken, but existing evaluation research has concentrated on speech recognition, translation, and basic sentiment analysis.
  • Culturally specific meanings, social norms, and shared experiences can make Nigerian Pidgin emotions poorly represented by positive, negative, and neutral labels.
  • Nigerian Pidgin communication also involves sarcasm, indirect speech, social signaling, commerce, religion, and culturally grounded reasoning that multilingual benchmarks rarely represent.
  • Wazobia Eval evaluates Nigerian Pidgin emotion understanding, sarcasm detection, and cultural reasoning using more than 550 examples and a 16-category emotion taxonomy.
  • The benchmark provides reproducible infrastructure for measuring capabilities, comparing systems, identifying weaknesses, and tracking progress in Nigerian Pidgin understanding.
  • Its contributions are a culturally grounded taxonomy, a multi-task benchmark dataset, and a standardized protocol for systematic model comparison.

2. Related Work

Related work established standardized multilingual and reasoning benchmarks, but their emphasis on English, high-resource languages, or translated tasks leaves African communicative practices insufficiently evaluated. Wazobia Eval extends African NLP benchmarking by focusing on Nigerian Pidgin emotion, sarcasm, and contextual reasoning.

  • GLUE and SuperGLUE standardized multi-task language-understanding comparisons, while MMLU broadened evaluation toward reasoning and knowledge assessment.
  • Most widely used benchmarks remain focused on English and other high-resource languages, limiting assessment in underrepresented linguistic contexts.
  • Existing emotion resources have advanced NLP research but may not transfer fully to African contexts because their linguistic and cultural assumptions differ.
  • Wazobia Eval addresses this limitation with a Nigerian Pidgin-specific, culturally grounded emotion taxonomy.
  • African NLP initiatives such as Masakhane and AfriSenti expanded resources and highlighted the importance of culturally relevant evaluation.
  • Wazobia Eval contributes an open benchmark emphasizing emotional interpretation, sarcasm recognition, and contextual reasoning rather than translation alone.

3. Nigerian Pidgin Emotion Taxonomy

The Nigerian Pidgin emotion taxonomy was developed to represent culturally meaningful distinctions that conventional polarity labels miss. Its 16 categories prioritize communicative intent, social meaning, and culturally grounded emotional states for practical model evaluation.

  • Conventional positive, negative, and neutral labels do not adequately represent many emotionally meaningful distinctions in everyday Nigerian Pidgin communication.
  • The taxonomy contains sixteen categories developed from Nigerian Pidgin examples to reflect how emotions are expressed and interpreted in Nigerian contexts.
  • Unlike conventional sentiment frameworks, the taxonomy emphasizes communicative intent, social meaning, and culturally grounded emotional states rather than simple polarity.
  • Categories emerged through iterative annotation and review when existing emotion labels failed to capture utterance meanings.
  • The taxonomy is a practical evaluation framework for testing whether models distinguish emotionally significant Nigerian Pidgin expressions, not an exhaustive theory of emotion.
  • The categories include conventional states such as joy, pride, anger, suspicion, shock, craving, celebration, contempt, betrayal, sarcasm, and neutral.
  • Culturally specific categories include prayer gratitude, hustle fatigue, hustle energy, market energy, and forming.
  • These retained categories make distinctions important for real-world understanding visible and support future expansion to other Nigerian languages and dialects.

4. Dataset Construction

The dataset was built from naturally occurring Nigerian Pidgin communication across everyday domains and manually annotated with a culturally specific taxonomy. Its sarcasm subset and context-sensitive examples target pragmatic interpretation beyond lexical matching, while the release is intended as an initial benchmark resource.

  • Data were collected from naturally occurring Nigerian Pidgin expressions and conversations representing everyday communication contexts.
  • Examples cover social interaction, greetings, commerce, health communication, emotional expression, and culturally specific conversational exchanges.
  • The dataset preserves authentic Nigerian communicative patterns, pragmatic meaning, local expressions, discourse markers, and culturally embedded language use rather than translated English.
  • Each example was manually annotated with a custom sixteen-category Nigerian emotion taxonomy, prioritizing contextual meaning over lexical cues.
  • 4.3 Sarcasm Dataset Construction: The sarcasm subset pairs sarcastic expressions with semantically similar sincere counterparts to test literal-versus-intended meaning.
  • Identical utterances can receive different labels because speaker intent, conversational context, and interpersonal dynamics alter their emotional meaning.
  • The benchmark therefore tests contextual understanding beyond keyword matching and supports future tracks for emotion disambiguation and pragmatic reasoning.
  • The current release contains more than 550 annotated examples, 16 emotion categories, 28 sarcasm pairs, and health, social, commerce, and gender-related coverage.

5. Wazobia Eval Benchmark Design

Wazobia Eval evaluates Nigerian Pidgin beyond literal translation and surface-level sentiment through emotion understanding, sarcasm detection, and cultural reasoning.

  • Wazobia Eval evaluates emotional understanding, pragmatic reasoning, and culturally grounded interpretation in Nigerian Pidgin.
  • The benchmark contains three primary tracks: Emotion Classification, Sarcasm Detection, and Cultural Reasoning.Together, these tasks assess linguistic competence and contextual understanding.

WAZOBIA EVAL

Wazobia Eval is designed to measure Nigerian Pidgin understanding rather than translation, combining culturally grounded interpretation with pragmatic and emotional evaluation.

  • Emotion Classification: Emotion Classification predicts one label from a sixteen-category taxonomy for each Nigerian Pidgin utterance.The primary metric is Macro-F1, which gives equal importance to all emotion categories.
  • Sarcasm Detection: Sarcasm Detection determines whether an utterance is sarcastic or non-sarcastic.The task evaluates whether models distinguish intended meaning from literal meaning.
  • Cultural Reasoning: Cultural Reasoning requires short explanations of culturally grounded meanings and intentions embedded in Nigerian Pidgin expressions.Outputs are assessed through qualitative human evaluation focused on contextual correctness rather than lexical similarity.
  • Evaluation Conditions: All models receive identical instructions, prompts, benchmark examples, and scoring methodology for comparability.
  • The benchmark measures Nigerian Pidgin understanding beyond literal translation and surface-level sentiment classification.It prioritizes cultural competence, pragmatic understanding, and emotionally grounded interpretation.

6. Evaluation Protocol

The evaluation protocol is model-agnostic and standardized, using common prompts, benchmark examples, scoring metrics, and task-specific assessment procedures.

  • Wazobia Eval supports evaluation of proprietary and open-source language models.
  • The pilot evaluated GPT-5.5 under standardized prompting conditions, while future releases will compare multiple frontier and open-source systems.
  • Prompt Design: The benchmark uses identical instructions and examples to reduce prompt-induced differences and isolate model understanding.
  • Evaluation Dataset: The initial benchmark uses a manually annotated Nigerian Pidgin dataset containing more than 550 examples across three task types.
  • Evaluation Dataset: The evaluation set contains 253 benchmark-ready examples distributed across sixteen emotion categories.
  • Pilot Validation: The pilot workflow was validated with a balanced subset representing each emotion category before large-scale evaluation.
  • Future Expansion: The dataset is intended to expand through additional annotation, quality review, larger splits, and new ambiguity-focused tracks.
  • Evaluation Metrics: Macro-F1 is the primary metric, with Accuracy, Precision, Recall, and F1-based measures supporting reproducible comparison.Macro-F1 assigns equal weight to emotion categories with varying frequencies.

7. Pilot Benchmark Results

The pilot evaluation found limited Nigerian Pidgin emotion-classification performance and recurring errors involving cultural context, pragmatic intent, and emotionally related categories.

  • Pilot Result: 43.75% accuracy was achieved by GPT-5.5, correctly classifying 7 of 16 benchmark examples.
  • Pilot Result: The small pilot provides initial evidence that culturally grounded Nigerian Pidgin emotion understanding remains challenging for contemporary language models.
  • Implication: These observations support evaluating culturally grounded understanding rather than translation or surface-level sentiment recognition.
  • Observed Challenges: The model frequently confused emotionally related categories such as joy, celebration, and pride despite recognizing positive sentiment.
  • Observed Challenges: Culturally grounded categories including forming, contempt, and sarcasm were difficult because they depend on context, speaker intent, and pragmatic interpretation.
  • Observed Challenges: Context-dependent expressions such as “You don try well well” can indicate celebration, sincere praise, sarcasm, or contempt.
  • Observed Challenges: Some market_energy examples were interpreted as craving, indicating confusion between culturally specific categories and general emotional concepts.The passage suggests that clearer label definitions and benchmark guidance may improve evaluation consistency.

8. Discussion

Pilot findings indicate that Nigerian Pidgin understanding remains difficult because culturally specific, context-dependent, and pragmatically grounded meanings are not captured by lexical recognition alone. The benchmark motivates expanded evaluation of contextual emotion and culturally grounded language understanding.

  • Culturally grounded Nigerian Pidgin understanding remains difficult for contemporary language models, especially when interpretation depends on context, sarcasm, or culturally specific emotional concepts.
  • Many errors arise from cultural context, pragmatic intent, sarcasm, social signaling, and emotionally ambiguous expressions rather than vocabulary knowledge.
  • Hustle_fatigue, hustle_energy, market_energy, forming, and prayer_gratitude capture Nigerian discourse patterns rarely represented in conventional sentiment benchmarks.
  • Context-dependent emotional meanings expose weaknesses in lexical-matching approaches because identical Nigerian Pidgin forms can support multiple interpretations.
  • Contextual Emotion Disambiguation is proposed as a future track requiring models to infer emotional meaning from utterances together with context.
  • Future benchmark expansions may add Nigerian languages, multilingual tracks, contextual disambiguation, safety assessments, and human-model comparisons.

9. Limitations

The current benchmark is an initial, relatively small Nigerian Pidgin resource whose labels and text-only format limit the strength and breadth of conclusions. The authors identify data scale, linguistic coverage, annotation validation, contextual ambiguity, and pilot-result status as boundaries.

  • More than 550 annotated examples make Wazobia Eval an initial release, but additional data collection is needed for larger, more comprehensive evaluations.
  • The benchmark primarily covers Nigerian Pidgin and does not capture Nigeria’s or Africa’s full linguistic diversity.
  • Formal inter-annotator agreement analysis has not yet been completed for the full manually annotated dataset.
  • Text-only data cannot fully capture meaning derived from context, tone, speaker relationships, and situational factors, leaving some examples open to multiple interpretations.
  • Pilot results are preliminary and intended mainly to validate benchmark workflows, so larger evaluation sets and controlled procedures are needed for stronger performance conclusions.
  • Despite these limitations, Wazobia Eval provides an initial foundation that can be expanded through future research and community contributions.

10. Conclusion

Wazobia Eval establishes a culturally grounded benchmark for Nigerian Pidgin emotion, sarcasm, and cultural reasoning, using a 16-category taxonomy and more than 550 manually annotated examples. Preliminary evaluations show that context-sensitive understanding remains challenging, motivating broader African-language evaluation infrastructure and future expansion.

  • Wazobia Eval evaluates emotion understanding, sarcasm detection, and cultural reasoning in Nigerian Pidgin.
  • The benchmark uses sixteen culturally grounded emotion categories, more than 550 manually annotated examples, and tracks for emotional understanding, pragmatic interpretation, and culturally informed reasoning.
  • Preliminary pilot evaluations suggest that contemporary language models struggle with Nigerian Pidgin when interpretation depends on context, sarcasm, or culturally specific emotional concepts.
  • Reliable benchmarks can measure progress and identify capability gaps as language technologies become increasingly important in education, healthcare, government, and commerce.
  • Future work will expand coverage, add contextual disambiguation, include additional African languages, establish inter-annotator agreement studies, and support community participation.
  • The dataset and benchmark are openly available to support more inclusive, culturally aware, and representative language technologies.
Loading 2608.21369v1…