Source-linked AI summary
MemoCare: An Interactive Multimodal Mobile System for Automated Cognitive Screening
Duy-Cat Can, Mau Minh Phuc Le, Tuan-Khoa Hoang, Hai-Dang Nguyen, Trung-Hieu Do, Dang Minh Ly, Minh-Duc Nguyen, Nghia TT Hoang, Linh-Trung Nguyen, Huy-Hieu Pham, Huong Ha, Binh T. Nguyen, Oliver Y. Chén
TL;DR
Cognitive screening combines response types that require task-specific interpretation and integration. MemoCare routes speech, location, touch, and drawing through a mobile multimodal workflow; predefined software tests passed, and clinicians rated the workflow 4/5 pooled median.
Problem
Cognitive screening combines heterogeneous interactions that must be captured, interpreted according to task-specific semantics, and integrated into a coherent scoring workflow.
Source: §1 · p. 2Method
Results
151/151 software cases passed; ShuffleNetV2 x1.5 achieved 91.33% mean BA and 78.87% exact three-criterion accuracy on the locked test; clinician inspection pooled median was 4/5.
Source: §4.1 · p. 5, §4.2 · p. 6Takeaways & Limitations
The modular design supports replacing individual processing components without changing the mobile flow, and the demo exposes the interaction and scoring loop to attendees.
Source: §2 · p. 2, §6 · p. 6MemoCare is presented as a research prototype, not a validated diagnostic system.
Source: §1 · p. 2
Abstract
from arXiv · showhide
MemoCare is an interactive mobile system for automated multimodal cognitive screening. A React Native application combines spoken responses, temporal and spatial orientation, touchscreen actions, and visuoconstruction in complete English and Vietnamese workflows. Speech is transcribed by Google Speech-to-Text and scored locally with deterministic task-specific natural language processing rules; GPS coordinates are resolved by the MemoCare spatial module before answer matching; touch tasks are scored from interaction events; and the drawing task uses a three-model convolutional neural network consensus with separate visual interpretation. Software tests pass 151/151 predefined cases across speech/language, spatial-answer, and touch-interaction scoring, while spatial regression passes 48/48 four-country coordinate-resolution cases. For the drawing module, validation-selected ShuffleNetV2 x1.5 achieved 91.33% mean balanced accuracy and 78.87% exact three-criterion accuracy on a locked 71-image test set. Four clinician co-authors additionally inspected the end-to-end workflow, yielding a pooled median rating of 4/5 across eight criteria, with item-level medians ranging from 3 to 4.5. At MMM, attendees can directly try a shortened multimodal screening workflow and inspect automatic item-level and total scoring.
1 Introduction
Cognitive screening combines heterogeneous interactions, so mobile automation must capture different response types, interpret each using task-specific semantics, and integrate the results. MemoCare contributes a compact interactive multimodal workflow, presented as a research prototype rather than a validated diagnostic system.
1 Introduction
- MMSE includes speech, orientation, memory, calculation, commanded actions, language, and visuoconstruction, requiring mobile systems to interpret heterogeneous responses by task and integrate their scores.Displaying a questionnaire alone does not capture the different response types or their task-specific semantics.Source: §1 · p. 2
- MemoCare contributes modality-aware scoring for speech, location, touch, and drawing through task-specific processing and a unified scoring interface.The work also presents a mobile prototype with technical verification and formative inspection by four clinician co-authors.Source: §1 · p. 2
- MemoCare is a research prototype, not a validated diagnostic system.Source: §1 · p. 2
2 MemoCare System
MemoCare presents a mobile workflow for multiple cognitive-screening tasks, routing responses to task-specific modules and showing item-level and total scores. It supports repeat, skip, pause, and resume controls, with English and Vietnamese versions using the same app flow.
2 MemoCare System
- MemoCare routes temporal and spatial orientation, memory, attention and calculation, language, touch commands, and visuoconstruction through task-specific interactions, then displays item-level and total scores.Some interactions are adapted to mobile input while preserving the cognitive operation.Source: §2 · p. 2
- The React Native app captures speech, touch and drawings, and GPS location; Google Speech-to-Text transcribes speech, while AWS-hosted inference analyzes drawings.The demo prototype stores assessment state and results locally and requires no user accounts.Source: §2 · p. 2
- The workflow issues reminders for unusable or absent responses, supports repeat, skip, pause, and resume, and can continue after repeated non-response.Sequential tasks retain earlier information when later scoring depends on it.Source: §2 · p. 2
- English and Vietnamese versions share the same app flow, with language-specific scripts, normalization rules, and spatial vocabularies adapting each workflow.Source: §2 · p. 2
3 Multimodal Processing and Scoring
MemoCare routes speech, location, touch, and drawing responses to modality-specific processing. Its modules apply task-specific rules, resolve spatial answers, verify touch events, and analyze drawings against defined criteria.
3.1 Speech and Language Scoring
- Speech is transcribed by Google STT, normalized, and scored with deterministic task-specific rules for parsing, sequential calculation, recall, ordered phrases, and sentence completeness.Task-state information supports sequential responses; hesitations and repetitions are tolerated, and some rules prioritize task-relevant words without inferring negation or correction from potentially incomplete transcripts.Source: §3.1 · p. 3
3.2 Location-Aware Spatial Reasoning
- The spatial module retrieves nearby administrative regions from indexed boundaries, resolves the device location, and matches the transcript against accepted current and historical place names.It uses 89,043 administrative boundary records across Vietnam, Switzerland, China, and the United States; matching allows aliases and renamed regions.Source: §3.2 · pp. 3–4
3.3 Touch Interaction
- Touch tasks are scored from recorded events, checking action type, target, event order, or whether a touch falls within a predefined image region.Touch processing is performed locally without a camera or additional sensor.Source: §3.3 · p. 4
3.4 Drawing Representation and Data
- The drawing task captures a touchscreen reproduction and predicts three structural criteria, C1–C3; the binary item result is positive only when all three are satisfied.The manually drawn dataset contains 329 training, 66 validation, and 71 locked-test drawings; balanced accuracy averages sensitivity and specificity, while exact accuracy requires all three criteria to be correct for one drawing. A rule-based generator produced 45,724 labeled examples covering valid and invalid structures and visual variations.Source: §3.4 · p. 4
3.5 Vision Models and Deployment
- Eight lightweight CNN models predict C1–C3 after staged synthetic-data training and fine-tuning on manually drawn examples.The models use ImageNet-pretrained weights from ShuffleNetV2, MobileNetV3, EfficientNet-B0, EfficientNet-Lite, and MnasNet families.Source: §3.5 · p. 4
- Validation data select probability calibration and decision thresholds; a stable AWS inference endpoint keeps the mobile client independent of the selected vision backbone.A separate criterion-aware U-Net provides visual interpretation but does not affect the binary drawing score.Source: §3.5 · p. 5
4 Technical Verification and Expert Review
Technical verification passed the predefined software cases, and the selected drawing model achieved strong results on the locked test set. Four clinicians inspected the end-to-end workflow, with a pooled median rating of 4/5 across eight criteria.
4.1 Technical Verification
- 151/151 software cases passed; ShuffleNetV2 x1.5 achieved 91.33% mean BA and 78.87% exact three-criterion accuracy on the locked test set.The 151 cases cover speech/language, spatial-answer, and touch/action scoring; the locked test set contains 71 images and was not used for model selection, calibration, or threshold tuning.Source: §4.1 · p. 5
- 95.17% mean BA was achieved by a post-hoc three-model consensus on the locked test set, making this result descriptive rather than confirmatory.The consensus combined MobileNetV3-Large, EfficientNet-B0, and EfficientNet-Lite1 after reviewing completed evaluation results.Source: §4.1 · p. 5
4.2 Clinical Expert Inspection
- Four clinicians inspected the application end to end with mock responses; the pooled median was 4/5, with item-level medians ranging from 3 to 4.5.The inspection characterizes the implemented workflow rather than patient-level diagnostic performance.Source: §4.2 · p. 6
5 Live Demonstration and Requirements
The live demonstration lets MMM attendees explore a shortened multimodal screening workflow or the complete workflow, from interaction through modality-specific processing to item and total scores.
5 Live Demonstration and Requirements
- Attendees can use an Android phone or tablet to try speech, GPS-based spatial scoring, touch tasks, drawing analysis, and item-level and total results.They may skip among representative tasks; the complete workflow is also available.Source: §5 · p. 6
- The demonstration exposes the transition from each interaction to its modality-specific processing and item result.Source: §5 · p. 6
- The demo requires a microphone, location access, and Wi-Fi; Google STT and remote drawing inference require internet access.No additional sensors or special space are required.Source: §5 · p. 6
6 Conclusion
MemoCare combines four interaction modalities with task-specific processing and scoring in a modular mobile screening workflow. Technical verification and clinician inspection cover the system, while patient-level clinical validation remains future work.
6 Conclusion
- MemoCare integrates speech, spatial context, touch, and drawing through deterministic language scoring, historically aware spatial resolution, event-based touch verification, and conditional CNN consensus with separate visual interpretation.Source: §6 · p. 6
- Technical verification covers modality-specific components, and clinical experts inspect the end-to-end workflow.Source: §6 · p. 6
- The modular design keeps the interaction workflow independent of particular processing implementations, supporting replacement of individual components.Source: §6 · p. 6
- The live demo exposes the complete multimodal interaction and scoring loop; formal patient-level clinical validation remains future work.Source: §6 · p. 6
A Demonstration Video and Setup
The demonstration materials show MemoCare’s mobile workflow, attendee interactions, and setup requirements, including the network access needed for remote services.
A Demonstration Video and Setup
- A demonstration video of up to 3 minutes shows the implemented mobile workflow, major modalities, automated scoring, and conference-demo setup.Source: A Demonstration Video and Setup · p. 8
- The live demo presents task delivery, modality-specific scoring, visual interpretation, and item-level and total results on an Android phone or tablet.Attendees operate the device, try spoken, location, touchscreen, and drawing tasks, and may skip tasks or follow a shortened 3–5-minute path.Source: A Demonstration Video and Setup · p. 8
- Setup requires an Android phone or tablet, table space, power, and Wi-Fi permitting Google STT and remote drawing inference.The device uses its built-in microphone, touchscreen, and GPS; no extra sensors or special physical space are required.Source: A Demonstration Video and Setup · p. 8