Source-linked AI summary
Task-Centric Multimodal Learning for Cognitive Impairment Detection
Sankalp Vaish, Rishank Singh, Sumukh Veeramalla, Caroline Summerour, John A. Batsis, Xiaohui Liang
TL;DR
Early cognitive-impairment detection is difficult outside clinical settings, and existing work often treats interaction context as secondary. This paper analyzes video, audio, and text from structured Alexa interactions; multimodal modeling reduces the reported error counts relative to unimodal models.
Problem
Existing cognitive-impairment studies often prioritize features or fusion while treating interaction context as secondary, leaving task effects on behavioral signals insufficiently examined.
Source: §I · p. 1Method
The study evaluates video, audio, and text signals from three structured Alexa interaction tasks using separate task models and unimodal and multimodal comparisons.
Source: §I · pp. 1–2, B. Modeling · p. 3, §IV · p. 3Results
The multimodal model reported 21 Type I errors and 16 Type II errors, compared with higher unimodal error counts.
Source: §I · p. 2Takeaways & Limitations
Natural conversational behavior contains meaningful signals for early cognitive-impairment detection, with visual, acoustic, and linguistic cues providing complementary indicators.
Source: §VI · p. 6
Abstract
from arXiv · showhide
Dementia affects millions of older adults worldwide, yet early detection remains difficult outside clinical settings. Traditional models primarily rely on text or voice-based responses, capturing only a limited view of human cognition despite the inherently multimodal nature of human expression. This work proposes a contactless, multimodal screening framework that leverages video, audio, and text signals collected during structured voice-based interactions with Amazon Alexa. By incorporating these modalities, our approach captures a more holistic representation of human behavior encompassing speech patterns, vocal characteristics, facial expressions, and linguistic structure, which may enable more accurate detection of cognitive decline. Different participants completed three conversational tasks designed to probe spontaneous speech, comprehension, and recall. We pursued three objectives: (1) evaluate which of the three Alexa tasks provides the most informative cognitive signals, (2) compare the effectiveness of unimodal models, and (3) assess the benefits of multimodal fusion models. Task-level analysis shows that the video model achieves its best performance on Command-Reading, with an accuracy of 0.65 (AUC 0.69), while the audio model performs best on Question-Answering, reaching an accuracy of 0.67 (AUC 0.73). In contrast, the multimodal model demonstrates consistently stable performance across tasks, achieving an accuracy of 0.76 (AUC 0.75) on Command-Reading and 0.76 (AUC 0.73) on Question-Answering. These findings demonstrate that multimodal behavioral cues captured through naturalistic interactions can reliably distinguish MCI from healthy aging, highlighting the limitations of traditional diagnostic paradigms and supporting the shift toward multimodal, real-world cognitive assessment frameworks.
I. INTRODUCTION
The introduction argues that interaction context is underexamined in cognitive-impairment detection and proposes analyzing task-dependent behavioral signals across three Alexa tasks. It combines video, audio, and text features, with multimodal evaluation intended to capture complementary cues.
I. INTRODUCTION
- Existing studies often prioritize feature representations or fusion strategies while treating interaction context as secondary, leaving task effects on behavioral signals insufficiently understood.The paper distinguishes Command-Reading, Command-Generation, and Question-Answering as paradigms that elicit different cognitive and linguistic processes.Source: §I · p. 1
- The framework evaluates facial behavior, audio biomarkers, and textual linguistic features independently and together to assess multimodal fusion.The introduction describes the signals as capturing facial expressions, speech patterns, and linguistic structure.Source: §I · pp. 1–2
- The multimodal model reduced both reported error types, with 21 Type I errors and 16 Type II errors, compared with higher unimodal error counts.The video model had 41 Type I and 22 Type II errors; the audio model had 14 Type I and 46 Type II errors.Source: §I · p. 2
- The video model reached 0.65 accuracy (AUC 0.69) on Task 1 and 0.59 accuracy (AUC 0.59) on Task 3, while audio peaked on Task 3 at 0.67 accuracy (AUC 0.73).The dataset uses three Alexa-based tasks to capture varied cognitive behaviors across natural conversation, semi-structured interaction, and memory-based question answering.Source: §I · p. 2
II. RELATED WORK
Related work motivates contactless behavioral screening while identifying gaps in multimodal integration and task-level evaluation. This paper extends voice-assistant-based approaches by adding video and evaluating multiple interaction types and sessions.
II. RELATED WORK
- Earlier facial-cue studies typically treated video as a standalone modality, while speech-based methods largely overlooked additional behavioral channels.Behavioral signals from everyday interactions are presented as an unobtrusive alternative to costly imaging methods for home-based or longitudinal monitoring.Source: §II · p. 2
- Prior voice-assistant work demonstrated longitudinal detection feasibility using speech and linguistic data, but omitted video-derived behavioral cues.The paper adds facial expression and Action Unit analysis and combines video, audio, and text in one pipeline.Source: §II · p. 2
- The study evaluates multiple structured interaction types and sessions to provide a more comprehensive, ecologically valid evaluation for home-based screening.Its stated advances include facial expression and Action Unit analysis alongside a unified video, audio, and text pipeline.Source: §II · p. 2
III. METHODOLOGY
The preprocessing pipeline extracts standardized behavioral features from synchronized Alexa audio-video recordings and automatic transcripts. It derives video, audio, and text representations for subsequent modeling.
A. Preprocessing
- The pipeline maps raw recordings to modality-specific feature embeddings for modeling architectures.The dataset contains synchronized audio-video recordings and Rev.ai transcripts.Source: A. Preprocessing · pp. 2–3
- Video processing isolates the older adult participant, estimates frame-level emotions, aggregates emotion descriptors, and extracts facial Action Units.Emotion descriptors include counts, distributions, variability, and shifts; Action Units represent specific facial-muscle activations.Source: A. Preprocessing · p. 3
- Audio processing extracts acoustic features and prosodic or speech biomarkers from participant speech segments.Features include MFCC statistics, spectral measures, pitch, intensity, jitter, shimmer, speaking rate, pauses, and articulation rate.Source: A. Preprocessing · p. 3
- Transcript processing derives linguistic and semantic features, then uses sentence embeddings and PCA to retain the top 10 components.Features include lexical richness, syntactic complexity, semantic coherence, fillers, hesitations, and discourse markers.Source: A. Preprocessing · p. 3
B. Modeling
The modeling framework compares task-specific models and unimodal performance with a multimodal model. It uses transformer encoders with multi-head self-attention to model temporal dynamics in extracted features.
B. Modeling
- A transformer encoder with multi-head self-attention models temporal dynamics in the extracted features.The framework spans traditional machine-learning classifiers and multimodal transformer architectures.Source: B. Modeling · p. 3
- Separate models are trained on each of the three tasks to identify which task provides the most informative signals for dementia detection.The passage frames task comparisons as a way to isolate cognitive and behavioral cues.Source: B. Modeling · p. 3
- The framework compares unimodal models for each task with a multimodal model integrating information across tasks and modalities.Source: B. Modeling · p. 3
IV. DATASET
The dataset comprises 35 older adults completing three Alexa tasks designed to elicit different forms of verbal engagement. Sessions were repeated over 18 months, with MoCA scores used to classify cognitive status.
IV. DATASET
- 35 older adults participated in structured interactions with Amazon Alexa.Each session included all three tasks and lasted approximately 35–75 minutes; the sessions were repeated seven times over 18 months.Source: §IV · p. 3
- The three tasks probed different behaviors: following predefined commands, generating commands from prompts, and answering questions aloud.Command-Reading targeted clarity and recall; Command-Generation targeted language generation, flexibility, and planning; Question-Answering targeted comprehension, recall, and spontaneous speech.Source: §IV · p. 3
- MoCA scores served as the cognitive-status labels: 18–25 indicated MCI, while scores of 26 or above indicated healthy status.Figure 2 provides an overview of session duration and composition and lists participant IDs and session numbers.Source: §IV · p. 3, Figure 2 · p. 3
A. Emotion Analysis
Emotion patterns differ between the described MCI and healthy participants, and the examples show that emotional expression varies across tasks. The MCI profile is more negative and constrained, while the healthy participant shows more persistent positive affect and broader variation.
A. Emotion Analysis
- Healthy participants show more happy and neutral emotion counts, while MCI participants show more sadness, disgust, and surprise.The passage describes these group patterns in the emotion-count distribution across participants.Source: A. Emotion Analysis · p. 3, Figure 3 · p. 4
- For the MCI participant, sadness is prevalent across tasks and increases markedly in task 3, while happiness appears infrequently.The passage suggests the task-3 increase may correspond to searching for or formulating an answer, potentially reflecting cognitive effort or task-related frustration.Source: A. Emotion Analysis · p. 4, Figure 4 · p. 4
- The healthy participant expresses positive emotions, especially happiness, more consistently across tasks and shows richer variation in task 3 than the MCI participant.The passage notes that positive expression persists even during more cognitively demanding tasks.Source: A. Emotion Analysis · p. 4, Figure 5 · p. 4
- As task complexity increases, the MCI participant shows reduced emotional expressiveness and variability, whereas the healthy participant maintains richer emotional expression.The comparison describes the two participants’ responses across tasks.Source: A. Emotion Analysis · p. 4
B. Audio Analysis
The audio analysis describes group differences across prosodic, temporal, and voice-quality features, alongside task-related patterns in individual participants. The MCI example shows changing acoustic behavior across tasks, while the healthy control’s distributions remain more consistent.
B. Audio Analysis
- MCI participants show lower pitch variability, reduced speaking and articulation rates, longer pauses, and less stable phonation than healthy controls.The reported group patterns span prosodic, temporal, and voice-quality features.Source: B. Audio Analysis · p. 4, Figure 6 · p. 4
- For the MCI participant, task 1 has elevated pitch measures, task 2 partially normalizes pitch but retains temporal irregularity, and task 3 brings increased pitch variability and more pronounced pauses.The passage suggests the task-3 pattern may reflect fatigue or diminished cognitive control as tasks progress.Source: B. Audio Analysis · pp. 4–5, Figure 7 · p. 4
- The healthy control shows compact, consistent acoustic distributions across tasks, with task 3 adding some pause-frequency and pitch-range variation.The passage describes the task-3 shifts as remaining within healthy adaptive modulation rather than cognitive strain.Source: B. Audio Analysis · p. 5, Figure 8 · p. 5
- Task 3 elicits longer utterances in both groups, but the healthy participant’s utterances are more sustained and consistent while the MCI participant shows greater duration variability.The passage also reports greater jitter and shimmer variability for the MCI participant, particularly in tasks 2 and 3, versus controlled acoustic behavior for the healthy participant.Source: B. Audio Analysis · p. 5
C. Model Analysis
Across tasks, the Video+Audio+Text model achieves the most balanced classification, while task-level performance varies by modality. Video signals are stronger on tasks 1 and 2, audio on task 3, and the combined model remains comparatively consistent across tasks.
C. Model Analysis
- Tasks 2 and 3 provide more informative signals overall, while task 1 shows lower performance and greater variability.Video performs particularly well on tasks 1 and 2, audio performs more strongly on task 3, and the combined model is comparatively stable across tasks.Source: Figure 9 · p. 5, C. Model Analysis · p. 5
- The combined Video+Audio+Text model achieves the most balanced and reliable detection, correctly identifying 37 MCI participants with 16 false negatives and 21 false positives, while mitigating dominant-class bias.Source: C. Model Analysis · p. 6
- The unimodal models show dominant-class bias, with the Video model producing substantial false negatives and the Audio model reducing them while introducing many false positives for MCI.The passage reports 41 healthy participants misclassified as MCI by Video and 14 false negatives for Audio.Source: C. Model Analysis · p. 5
VI. DISCUSSION
The discussion identifies meaningful cognitive signals across facial, audio, and linguistic behavior in natural conversations. It also reports task-dependent differences and outlines dataset expansion, longitudinal modeling, and improved fusion as future directions.
VI. DISCUSSION
- Facial emotion patterns and Action Unit dynamics distinguish healthy and MCI participants, with MCI participants showing more negative affect and less emotional variability.Audio and linguistic features also indicate impairment through disruptions in prosody, speech fluency, and semantic coherence.Source: §VI · p. 6
- Video signals are more informative in question-generation or spontaneous-response tasks, whereas predefined question answering is more informative for audio.Source: §VI · p. 6
- Future work includes broader participant and demographic coverage, longitudinal cross-session Transformer modeling, improved multimodal fusion, and LLM-based reasoning.The stated aims include more interpretable and clinically meaningful reasoning over behavioral signals.Source: §VI · p. 6
VII. CONCLUSION
The conclusion presents a multimodal pipeline for distinguishing healthy individuals from those with MCI using facial, audio, and linguistic signals. It reports improved predictive performance from combining modalities and describes potential remote-monitoring applications.
VII. CONCLUSION
- Combining facial, audio, and linguistic modalities achieves accuracy = 0.75 and AUC = 0.80, outperforming any unimodal system.Source: §VII · p. 6
- Command-Reading and Command-Generation reveal richer facial behavioral markers, while acoustic and linguistic cues provide complementary information in Command-Generation and Question-Answering.Source: §VII · p. 6
- The framework is described as suitable for embedding in telehealth, smart-home, video-calling, and digital-wellness settings for remote monitoring and early alerts.Source: §VII · p. 6