Source-linked AI summary
AI-Powered Symptom Assessment and User Experience: A Case Study of Simtomi and Simtomi-Care
Jinha Lee, Chan Hyung Lee, Hyunsung Lee, Seunghwan Kim, Ban Hyung Lee, Minjun Shin, Hojin Shin, Jungdo Park
TL;DR
Digital symptom checkers are widely used, but concerns remain about reliability, constrained input, and clinical workflow integration. This case study examines Simtomi through Korean interviews and a U.S. survey. Korean physicians valued questioning aligned with consultation, while U.S. professionals emphasized multilingual access and workflow integration.
Problem
Digital symptom checkers are widely used, but concerns include inconsistent accuracy, constrained input formats, overlooked context, and limited follow-up guidance.
Source: §I · p. 1Method
The study examines the multilingual Simtomi app and provider platform through interviews with Korean users and physicians and a U.S. survey analyzed using sentence-level semantic clustering.
Source: §I · p. 2, C. Study Phase 1: South Korea · p. 3Results
Korean physicians valued questioning aligned with consultation and clinically phrased summaries, while U.S. professionals emphasized multilingual access and provider workflow integration.
Source: A. Results of Study Phase 1 · p. 4, A. Discussion · pp. 5–6Takeaways & Limitations
The case provides a cross-context account of usability, trust, and usefulness for a symptom assessment system designed to serve patients and clinicians.
Source: §I · p. 2Because U.S. participants did not evaluate Simtomi in a real clinical environment, the findings should not be read as a direct cross-country comparison of system performance or acceptance.
Source: B. Limitations and Future Work · p. 6
Abstract
from arXiv · showhide
Digital symptom checkers are widely used for quick guidance on health concerns, yet many systems still face challenges in collecting accurate information, supporting communication, or integrating with clinical workflows. To explore how these tools function in real use, we examine the case of the Simtomi system, which pairs a multilingual symptom assessment application with a provider-facing platform. Empirical studies were conducted in two countries. In South Korea, based on participants' firsthand experience, we found that the system improved how patients communicated their symptoms and helped clinicians review cases more efficiently through structured summaries aligned with diagnostic reasoning. In the United States, responses from prospective users and healthcare professionals highlighted the value of multilingual support, structured questioning, and the system's potential to assist clinical coordination. These findings offer a grounded account of how AI-based symptom assessment tools can operate across different healthcare contexts and provide broader insight into usability, trust, and usefulness in digital health.
I. INTRODUCTION
Digital symptom checkers offer rapid guidance, but concerns remain about reliability, constrained input, and user trust. This study examines Simtomi across South Korea and the United States to assess usability and usefulness for patients and healthcare professionals.
I. INTRODUCTION
- Systematic reviews report inconsistent symptom-checker performance and continuing accuracy and reliability concerns,,,.The introduction also notes that technical performance alone is insufficient for patient-facing tools.Source: §I · p. 1
- Many checkers constrain input, miss context, and offer limited follow-up, while interface design and emotional reassurance shape perceived credibility and acceptance,,,,.The introduction identifies cultural responsiveness alongside emotional reassurance as relevant to acceptance.Source: §I · p. 1
- Simtomi pairs multilingual patient symptom entry and guided questions with clinician-facing structured summaries intended for EHR integration.The study asks how this system may facilitate access and communication and how users and professionals assess its usability and usefulness.Source: §I · p. 2
- The two-phase case study combines interviews with Korean users and physicians with a U.S. survey analyzed through sentence-level semantic clustering [14].The authors frame the cross-setting approach as a way to compare perspectives across cultural and institutional contexts,,.Source: §I · p. 2
A. Simtomi System
Simtomi links a patient mobile app with Simtomi-Care, a provider platform that turns symptom information into structured clinical summaries and supports workflow integration.
B. Simtomi System
- Simtomi comprises a patient-facing mobile app and Simtomi-Care, a provider-facing platform for clinical environments.The two modules accept patient information through the app or directly at the point of care.Source: A. Simtomi System · p. 2
- The app accepts preferred-language symptom reports, guides users through AI-driven questions, and suggests probable conditions and care destinations using location-based services.A translated pharmacy support module also helps patients communicate simple symptoms and request medications across languages.Source: A. Simtomi System · p. 2
- Simtomi-Care converts symptom descriptions into medically precise terminology and delivers structured summaries to clinicians for review and EHR integration.Providers can also use differential diagnosis guidance and multilingual communication tools.Source: A. Simtomi System · p. 2
- The system supports 14 languages and reports clinical workflow use in South Korea, Vietnam, and South Africa.The passage also reports ISO 13485, ISO 27001, and ISO 17100 certifications.Source: A. Simtomi System · p. 2
B. Technical Implementation of Simtomi
Simtomi’s technical implementation combines multilingual complaint matching, adaptive diagnostic questioning, and LLM-generated EHR-compatible summaries, with clinical and technical contributors involved across stages.
B. Technical Implementation of Simtomi
- A multilingual corpus maps free-text symptom descriptions or associative terms to standardized chief complaints, supporting 14 languages.The associative expressions were constructed and validated by a professional multilingual translation team.Source: B. Technical Implementation of Simtomi · p. 2
- Adaptive questioning uses symptom nodes, age, and gender to generate follow-up questions and refine candidate-condition probabilities, with pathways limited to fewer than 20 items.Questions drew on textbooks and guidelines and underwent review by 30 departmental specialists, followed by iterative testing and feedback.Source: B. Technical Implementation of Simtomi · p. 2
- In a preliminary internal evaluation excluding pediatrics, correct diagnoses appeared in 62% of Top-1 outputs, 76% of Top-3 outputs, and approximately 88% of Top-5 outputs.The passage says formal external validation using standardized clinical vignettes remains a priority.Source: B. Technical Implementation of Simtomi · p. 2
- LLMs convert patient interaction data into structured EHR-compatible summaries, such as rendering “pain in the right upper side of the belly” as “RUQ pain.”The development team collaborated across technical and clinical domains and applied AI techniques across symptom entry, questioning, inference, summarization, and translation.Source: B. Technical Implementation of Simtomi · p. 2
- Developers aligned summarized outputs with original patient input and refined mappings through repeated testing to reduce possible hallucinations and inaccuracies.Medical terms were fine-tuned across languages with professional translators and prompt-engineering strategies.Source: B. Technical Implementation of Simtomi · p. 3
C. Study Phase 1: South Korea
The South Korean phase gathered firsthand perspectives from Simtomi users and physicians through semi-structured interviews, then analyzed the transcripts using iterative thematic coding.
C. Study Phase 1: South Korea
- The study interviewed 11 consumers with personal Simtomi experience and four physicians who used the integrated system in clinical practice.Participants were recruited through professional networks, local clinics, and user communities; interviews lasted 20–30 minutes.Source: C. Study Phase 1: South Korea · p. 3
- User interviews focused on the app, while physician interviews covered both the app and Simtomi-Care and asked about accuracy relative to medical diagnoses.The protocols addressed five evaluation areas and distinguished patient and provider perspectives on usefulness.Source: C. Study Phase 1: South Korea · p. 3
- Researchers transcribed interviews verbatim and used iterative open and axial coding, refining categories through team discussion and consensus.Separate codebooks surfaced role-specific themes while preserving comparison between users and professionals.Source: C. Study Phase 1: South Korea · p. 3
D. Study Phase 2: United States
Study Phase 2 examined perceived usefulness and experience in the U.S. through prospective-user responses and healthcare-professional interviews. Users viewed a standardized walkthrough; professionals tested the app, while their reflections on Simtomi-Care followed a demonstration because it was not deployed.
D. Study Phase 2: United States
- The U.S. study addressed perceived usefulness and experience in a context of varied healthcare access and system fragmentation.The study also considered whether Simtomi’s functions could address access gaps and connect digital technologies with healthcare services.Source: D. Study Phase 2: United States · p. 3
- The study recruited 201 U.S. general users through CloudResearch, who watched a three-minute walkthrough before responding to open-ended questions.The walkthrough standardized exposure to the app’s navigation and features.Source: D. Study Phase 2: United States · p. 3
- Researchers analyzed 5,743 sentences from 1,205 responses using pre-trained general-domain SBERT (all-MiniLM-L6-v2), cosine similarity, and k-means clustering, selecting k primarily by Silhouette performance.Each sentence was encoded as a 384-dimensional vector. BART-generated theme labels used centroid-nearest responses.Source: D. Study Phase 2: United States · pp. 3–4
- Seven U.S. healthcare professionals tested the patient-facing app and saw a Simtomi-Care demonstration; the provider module was not deployed there.Interviewees included physicians, nurse practitioners, and a nurse, with an average of 16 years of clinical experience.Source: D. Study Phase 2: United States · p. 4
A. Results of Study Phase 1
In South Korea, firsthand users described Simtomi as accessible and useful for care decisions and communication, while physicians valued its clinically aligned questioning and structured summaries. Both groups connected trust and usefulness to how the system translated patient input into clinically relevant information.
A. Results of Study Phase 1
- Users found the app approachable and easy to navigate; its prompts encouraged reflection, while reports helped with care decisions and clearer physician communication.Participants also described greater confidence and a stronger role in directing their care.Source: A. Results of Study Phase 1 · p. 4
- Physicians said Simtomi’s stepwise questioning mirrored consultation reasoning, and its summaries used familiar clinical language that could support documentation.They also described suggested diagnoses as reasonable and relevant to routine practice.Source: A. Results of Study Phase 1 · p. 4
- Physicians associated trust with clinically accurate language and diagnostic logic, and reported that streamlined presentations could shorten the first consultation by about two minutes.They also valued diagnostic breadth and structured summaries that could translate into medical documentation.Source: A. Results of Study Phase 1 · p. 4
B. Results of Study Phase 2
U.S. user responses emphasized intuitive design, accessible symptom input, clinical trust, and both care guidance and communication support. Healthcare professionals valued usability and potential workflow efficiencies, while noting refinements and EHR integration as relevant conditions.
B. Results of Study Phase 2
- Manual spot-checks found over 90% conceptual fit between assigned themes and original responses, supporting face validity despite moderate theme cohesion.Mean intra-theme similarity was 0.470, and mean theme-assignment confidence was 0.431.Source: B. Results of Study Phase 2 · p. 5
- U.S. users described intuitive design and stepwise usability, grounded trust in clinical framing or physician origins, and valued both care directions and clearer doctor communication.Some users said they would still double-check with a healthcare provider about serious issues.Source: B. Results of Study Phase 2 · p. 5
- Healthcare professionals valued clear navigation, plain-language prompts, clinical relevance, and prepopulated histories that could reduce repetitive documentation.They also noted occasional autocomplete gaps and lengthy question pathways; they tied realized efficiency to integration with existing EHR systems.Source: B. Results of Study Phase 2 · p. 5
A. Discussion
Across both countries, participants valued Simtomi’s clear interface, multilingual support, and guided interaction, while professionals also valued its clinical credibility and workflow integration. Their priorities and concerns varied by healthcare context, including customization needs and barriers related to literacy, age, and cultural differences.
Discussion
- Participants across both countries described the interface as clean, simple, and visually pleasing, while usability also depended on accessible information, guided interaction, and fit with patient and professional practices.Korean users highlighted actionable information, multilingual support, and symptom prompting; U.S. users highlighted autocomplete, body charts, and step-by-step questions.Source: A. Discussion · p. 5
- Professionals in both countries valued multilingual support and viewed the system as credible, but U.S. providers also raised data-security and missed-diagnosis concerns and emphasized institutional integration and liability.Participants also identified usability barriers for some rural patients and older adults, as well as difficulty capturing cultural nuances through standardized questions.Source: A. Discussion · p. 5
- Professionals in both countries valued Simtomi-Care’s workflow integration, while Korean hospitals sought customized questionnaires and summaries suited to their institutional practices.The passage links differing preferences to healthcare structures and physician roles, and proposes specialty-specific models or hospital-managed questionnaires as future directions.Source: A. Discussion · p. 6
- Pharmacy translation was used four to five times more frequently than hospital consultations, highlighting the importance of accounting for different community care-seeking pathways.The passage notes that pharmacies can involve lower financial barriers, shorter interactions, and simpler administrative procedures than hospitals.Source: A. Discussion · p. 6
B. Limitations and Future Work
The study’s qualitative samples were small, and U.S. participants did not evaluate the system in a real clinical environment. The authors call for comparable cross-setting studies and longitudinal research with diverse patient groups.
Limitations and Future Work
- Small qualitative samples and written U.S. survey responses limit the breadth and depth of perspectives, and U.S. participants’ lack of real-clinical evaluation precludes direct cross-country comparison of performance or acceptance.The authors recommend comparable studies across healthcare settings and longitudinal research with diverse patient groups.Source: B. Limitations and Future Work · p. 6