Source-linked AI summary
Evaluation of Conversational Agents: Understanding Culture, Context and Environment in Emotion Detection
Martha Teiko Teye, Yaw Marfo Missah, Emmanuel Ahene, Twum Frimpong, Auxane Boch
TL;DR
The paper addresses cultural, ethical, and contextual gaps in emotion detection for human–AI interactions, including sarcasm. It combines speech and image data with CNNs and AFME, reporting improved predictions and sarcasm detection while noting limitations in African emotion datasets.
Problem
Emotion detection research must account for cultural and ethical factors, while sarcasm can depend on mismatches between speech, facial expression, tone, and context.
Method
The approach combines CNN models for facial and speech emotion datasets with data balancing, preprocessing, augmentation, three CNN blocks, and the Audio Frame Mean Expression algorithm.
Results
The AFME-enhanced approach improved predictions on test data and detected sarcasm, while experiments reported misclassifications in precision and recall.
Takeaways & Limitations
Combining speech and image information with AFME provides a paper-supported route toward more context-sensitive emotion and sarcasm detection.
Takeaways & Limitations
Access to readily available, well-structured African emotion datasets remains limited, and standalone results were insufficient for accurately predicting emotions on black faces.
Abstract
from arXiv · showhide
Valuable decisions and highly prioritized analysis now depend on applications such as facial biometrics, social media photo tagging, and human robots interactions. However, the ability to successfully deploy such applications is based on their efficiencies on tested use cases taking into consideration possible edge cases. Over the years, lots of generalized solutions have been implemented to mimic human emotions including sarcasm. However, factors such as geographical location or cultural difference have not been explored fully amidst its relevance in resolving ethical issues and improving conversational AI (Artificial Intelligence). In this paper, we seek to address the potential challenges in the usage of conversational AI within Black African society. We develop an emotion prediction model with accuracies ranging between 85% and 96%. Our model combines both speech and image data to detect the seven basic emotions with a focus on also identifying sarcasm. It uses 3-layers of the Convolutional Neural Network in addition to a new Audio-Frame Mean Expression (AFME) algorithm and focuses on model pre-processing and post-processing stages. In the end, our proposed solution contributes to maintaining the credibility of an emotion recognition system in conversational AIs.
I. INTRODUCTION
The introduction frames emotion detection in conversational AI as an ethical and cultural challenge, especially for Black African communities. It motivates models that account for social context, demographic bias, micro-expressions, and multimodal cues.
- Conversational AI seeks to emulate human interaction, making psychological, social, and ethical considerations central to human-AI engagement.
- Black people remain underrepresented in facial-recognition technologies, while African societies face risks from both technological exclusion and unmitigated deployment.
- AI applications should fit cultural and ethical values while remaining compatible with social and technical norms.
- Sarcasm detection requires combining facial expression, speech tone, gestures, and contextual cues because verbal content may conflict with visual expression.
- Bias mitigation can occur during pre-processing, in-processing, and post-processing, including re-weighting datasets representing underrepresented groups.
- Prior emotion-recognition systems often focus on static images or speech, with limited attention to diversity, micro-expressions, and real-time detection.
III. MODEL DESIGN
The model design combines facial-expression and speech-emotion datasets, balances localized African data, and uses CNN and LSTM training with a new mathematical model for emotion prediction.
- The study combines facial-expression data with randomly collected African-descent social-media data and performs preprocessing to balance representation.
- The model is trained with Long Short-Term Memory and Convolutional Neural Networks using varied hyperparameter values.
- A new mathematical model is incorporated to provide better accuracies across the described emotions.
A. ENVIRONMENT SETUP
The study uses Python-based notebook tooling and draws on facial-expression and speech-emotion datasets, supplemented by locally generated Ghanaian images.
- The implementation uses Python 3 in Jupyter Notebook with NumPy, Pandas, and Matplotlib.
- Training relies on the FER 2013 facial-emotion dataset and Speech Emotion Recognition datasets.
- The speech data sources include CREMA-D, RAVDESS, SAVEE, and TESS.
- The dataset adds 1000 locally generated Ghanaian images spanning seven emotion categories.
C. DATA CONVERSION AND PRE-PROCESSING
Pre-processing converts image and audio inputs into representations suitable for multimodal emotion modeling. Images are standardized and facial features localized, while audio is transformed across amplitude, frequency, and time.
- FER images are already 48 x 48 grayscale pixels, while locally collected RGB images are converted to grayscale.
- The grayscale conversion uses Grayscale = (0.3 x Red) + (0.59 x Green) + (0.11 x Blue) to support balancing and bias reduction.
- Dlib localizes 68 facial landmarks, after which OpenCV selects prominent brow, eye, and jaw features for training.
- A central motion region helps capture larger-area macro expressions, while mouth and philtrum landmarks vary across emotions.
- Audio processing converts files to WAV, analyzes frequency-time structure, transcribes speech, and applies natural-language processing for classification.
- Audio is represented through amplitude, frequency, and time, with Fourier transformation applied to the signal.
D. PREDICTION MODEL
The prediction model uses CNNs to classify image and audio emotion data, with three CNN blocks selected to improve accuracy while reducing overfitting.
- Two separate CNN models classify the image and audio datasets into different emotions.
- Three CNN blocks improved accuracy significantly while addressing rapid overfitting of the test dataset.A single block achieved 72% accuracy but overfit the test data.
E. LIBRARIES USED IN TRANING THE AUDIO DATA
The audio pipeline represents speech through acoustic features that capture signal crossings, pitch energy, spectral variation, power, and time-frequency structure.
- The audio model uses ZCR, chroma vector, chroma deviation, MFCC, RMS, and MelSpectrogram features.These features represent signal changes, pitch-related energy, spectral variation, power spectrum, amplitude, and time-frequency content.
IV. RESULTS AND DISCUSSIONS
The results examine training metrics across epochs and report challenges involving video micro-expressions and facial appearance conditions.
- After 30 epochs, accuracy steadily increased and peaked between the 46th and 50th epochs.The loss dropped significantly during the first 10 epochs and then continued a constant linear decline.
- Figure 9 reports metrics after 70 epochs, while Table I presents metrics after the machine-learning model on test data.
- Micro-expressions were difficult to detect in non-static video frames.The stated contributing conditions included facial complexion against the background and facial accessories such as makeup and eyeglasses.
- 82.59 per cent accuracy was achieved after increasing the trained features and number of epochs.The reported loss value also decreased to 47.21% per cent.
D. MITIGATING FALSE POSITIVE AND TRUE
The approach combines CNN outputs from facial and speech emotion datasets with AFME to validate predictions and address ambiguous or misclassified emotions, including sarcasm detection.
- CNN models were applied to facial-expression and speech-emotion datasets, supplemented with 20% localized data for video-frame and micro-expression tests.
- AFME validates and improves neural-network results by comparing emotion outputs from speech and image processing.
- Fear, disgust, and surprise were especially misclassified, with some sadness predicted as neutral, fear, or disgust.These misclassifications were linked to dataset characteristics and included edge cases between contrasting emotions.
- The algorithm reports sarcasm when the highest-scoring emotion sets from speech and images do not match.
F. ANALYSIS AFTER AUDIO-FRAME MEAN
The AFME-enhanced approach combines audio and video-frame emotion evidence, improving classification outcomes and detecting sarcasm despite some misclassifications.
- F. ANALYSIS AFTER AUDIO-FRAME MEAN: The model analyzes captured video micro-frames alongside spoken audio to compare visual and vocal emotion signals.In one example, visual frames indicated happiness while extracted audio was classified as anger; AFME compared these signals using Plutchick’s emotion wheel.
- F. ANALYSIS AFTER AUDIO-FRAME MEAN: The reported evaluation compares post-AFME performance metrics with results from other models.Table 2 reports performance metrics after AFME, while Table 3 compares final results with other models.
- F. ANALYSIS AFTER AUDIO-FRAME MEAN: AFME improved predictions on test data and detected sarcasm, although precision and recall still showed a small number of misclassifications.The evaluation used recall, precision, and F1 scores based on true and false prediction counts.
G. REASON FOR CONSIDERING BOTH SPEECH (AUDIO) AND IMAGE FRAMES ASSOCIATED WITH SPEECH
The paper considers both speech and image frames because emotion and sarcasm can be expressed through interacting verbal, facial, prosodic, and non-verbal cues.
- G. REASON FOR CONSIDERING BOTH SPEECH (AUDIO) AND IMAGE FRAMES ASSOCIATED WITH SPEECH: Sarcasm may pair text with a contrasting facial expression, making visual and contextual cues relevant to emotion detection.Tone and gestures can indicate whether an emotion carries humor or irony.
- G. REASON FOR CONSIDERING BOTH SPEECH (AUDIO) AND IMAGE FRAMES ASSOCIATED WITH SPEECH: Speech segmentation can capture words, phrases, and non-verbal content from real-life conversations.A support vector machine approach for prosodic phrases achieved 52% accuracy, while non-verbal sounds contributed to emotion detection.
- G. REASON FOR CONSIDERING BOTH SPEECH (AUDIO) AND IMAGE FRAMES ASSOCIATED WITH SPEECH: Facial expressions and body language provide emotion cues when speech is minimal or absent.The paper describes non-verbal communication as a major component of human communication.
- G. REASON FOR CONSIDERING BOTH SPEECH (AUDIO) AND IMAGE FRAMES ASSOCIATED WITH SPEECH: Text-based sentiment and tone can also depend on contextual markers such as emojis, whose use may sometimes be misleading.The paper contrasts messages such as “okay” and “okay :)” while noting possible wrongful emoji interpretations.
H. RECOMMENDATION ON STRUCTURE OF DESIGNING CONVERSATIONAL AGENTS
The paper recommends conversational agents that integrate emotional, environmental, social, and relational capabilities while addressing dataset and bias constraints.
- H. RECOMMENDATION ON STRUCTURE OF DESIGNING CONVERSATIONAL AGENTS: Conversational agents should perceive emotions and surroundings, respond through high-level dialogue, and recognize other agents’ emotion models.These capabilities are presented as design criteria for competent conversational agents.
- H. RECOMMENDATION ON STRUCTURE OF DESIGNING CONVERSATIONAL AGENTS: Agents should create and maintain social relationships, learn social qualities, and build trust through natural cues.The recommendation links social competence with relationship maintenance and trust.
- H. RECOMMENDATION ON STRUCTURE OF DESIGNING CONVERSATIONAL AGENTS: Limited access to well-structured African emotion data remains a constraint requiring dataset restructuring and balancing.The paper identifies access to African data as a limitation and reports balancing datasets for black demographics.
- H. RECOMMENDATION ON STRUCTURE OF DESIGNING CONVERSATIONAL AGENTS: Future work should compare IBM 360 bias-mitigation techniques and examine facial and vocal features across groups.The proposed analyses are intended to identify additional dataset bias and add group-related features to model detection.