Source-linked AI summary
"Brilliant AI Doctor" in Rural China: Tensions and Challenges in AI-Powered CDSS Deployment
Dakuo Wang, Liuping Wang, Zhan Zhang, Ding Wang, Haiyi Zhu, Yvonne Gao, Xiangmin Fan, Feng Tian
TL;DR
AI-CDSS adoption in practice remains understudied, especially in developing countries. Through observations and interviews with 22 clinicians at six rural Chinese clinics, this paper examines Brilliant Doctor and finds tensions involving context, workflow, technical usability, transparency, and trustworthiness, while participants remained positive about AI as a doctor’s assistant.
Problem
AI-CDSS usage and clinicians’ post-adoption experiences remain insufficiently understood, particularly in developing countries.
Method
The paper uses observations and interviews with 22 clinicians from six rural Chinese clinics to study Brilliant Doctor in practice.
Results
The study identifies tensions involving rural-context and workflow misalignment, technical limitations, usability barriers, and AI-CDSS trustworthiness.
Takeaways & Limitations
The findings support designing AI-CDSS as a clinician-AI collaboration in which the system acts as a doctor’s assistant.
Takeaways & Limitations
The qualitative study is limited to one rural area in Beijing, China, and did not interview patients.
Abstract
from arXiv · showhide
Artificial intelligence (AI) technology has been increasingly used in the implementation of advanced Clinical Decision Support Systems (CDSS). Research demonstrated the potential usefulness of AI-powered CDSS (AI-CDSS) in clinical decision making scenarios. However, post-adoption user perception and experience remain understudied, especially in developing countries. Through observations and interviews with 22 clinicians from 6 rural clinics in China, this paper reports the various tensions between the design of an AI-CDSS system ("Brilliant Doctor") and the rural clinical context, such as the misalignment with local context and workflow, the technical limitations and usability barriers, as well as issues related to transparency and trustworthiness of AI-CDSS. Despite these tensions, all participants expressed positive attitudes toward the future of AI-CDSS, especially acting as "a doctor's AI assistant" to realize a Human-AI Collaboration future in clinical settings. Finally we draw on our findings to discuss implications for designing AI-CDSS interventions for rural clinical contexts in developing countries.
1 INTRODUCTION
AI-CDSS can support clinical decision-making, but its practical adoption remains understudied, particularly in developing-country settings. This study examines clinicians’ experiences with Brilliant Doctor in rural Chinese clinics and draws design implications for similar contexts.
- Research gap: AI-CDSS usage in real-world practice remains poorly understood, limiting knowledge about user acceptance, system uptake, and implementation barriers.The paper emphasizes that technical performance alone does not explain adoption, because AI introduces concerns including opacity and possible job displacement.
- Research gap: Research on AI-CDSS has focused mainly on developed countries, while developing countries face greater implementation challenges and limited evidence about local social and contextual factors.These challenges include weaker health-information infrastructures and distinct clinical environments.
- Study focus: The study investigates a recently deployed Brilliant Doctor system through fieldwork in rural Chinese clinics, where many residents have limited income and healthcare access.The setting is intended to inform design and implementation for similar rural contexts in other developing countries.
- Contributions: The paper contributes an empirical account of clinicians’ perceptions, usage challenges, and the social, cultural, and contextual factors shaping AI-CDSS use.It also develops design implications for increasing uptake and utilization in rural areas.
- Study context: The study is unaffiliated with both the system’s developer and the government agency that deployed it.The researchers learned about the deployment through news coverage before contacting the clinics.
2 BACKGROUND AND RELATED WORK
Prior work establishes CDSS’s potential while showing that AI-CDSS adoption in practice remains poorly understood. The paper situates its study at the intersection of AI-specific barriers, human professional concerns, and developing-country implementation conditions.
- CDSS and AI-CDSS: CDSS supports diagnosis, treatment suggestions, and safety alerts, while newer AI methods shift systems from rule-based reasoning toward data-driven approaches.Neural networks, knowledge graphs, and large clinical datasets have accelerated AI-CDSS development.
- Implementation challenges: Few AI-CDSS systems have been deployed clinically, leaving clinicians’ perceptions, usage patterns, and deployment barriers insufficiently understood.The paper frames empirical understanding as necessary for improving future system design.
- Implementation challenges: Real-world CDSS adoption can be hindered by false or frequent alarms, poor interfaces, workflow interference, time pressure, and inadequate training.AI-CDSS may introduce additional barriers because its technologies are increasingly complex.
- Human-AI relationship: AI-CDSS black-box behavior can impede clinicians’ understanding and trust, while perceived threats to professional autonomy can reduce willingness to use such systems.Both concerns motivate research on cooperative Human-AI systems in healthcare.
- Developing-country context: Developing countries face infrastructure, computer-literacy, cost, and contextual challenges, yet social and cultural factors in AI-CDSS implementation remain underexamined.Limited EHR adoption can restrict the patient data on which CDSS depends.
3 METHOD
The paper combines rural fieldwork, clinician interviews, qualitative coding, and system description to examine how Brilliant Doctor fits into everyday clinical work. The studied system integrates AI recommendations with an existing EHR and offers multiple interaction and information-retrieval features.
- Research site: The study examined Brilliant Doctor after six months of deployment across six rural clinics in Pinggu county, Beijing, where 18 clinics and two higher-tier facilities serve the region.The system was introduced to all clinics in the county in early 2019.
- Participants: The participants were 22 clinicians with 8–32 years of experience, including physicians, surgeons, and four Traditional Chinese medicine practitioners.Patients were not interviewed because clinicians were the system’s directly targeted users.
- Data collection: Researchers spent 10 days observing clinic work and conducted 40-minute-to-1-hour semi-structured interviews with all 22 participants.Observations covered work activities, artifacts, and system use; interviews addressed responsibilities, experience, perceptions, and barriers.
- Data analysis: Observation and interview materials were analyzed through open coding in Chinese, followed by iterative coding in Nvivo and identification of themes and representative quotations.Photographs from fieldwork were also analyzed for contextual information.
- System: Brilliant Doctor is integrated beside the EHR and generates real-time diagnostic options with confidence scores from patient medical context.The EHR stores symptoms, history, allergies, diagnosis, prescriptions, and treatment plans, while the AI window presents recommendations and highlighted features.
- System interaction: Clinicians can use a Type-In interaction that parses EHR entries or manually answer symptom, history, screening, diet, and emotional-state questions.The manual approach is designed to replicate textbook clinical reasoning, while the Type-In approach preserves ordinary EHR entry.
- Additional features: Beyond diagnosis, the system retrieves similar cases and provides an offline medical-information search engine for diseases, medicines, and treatments.These tools were intended to support clinicians’ learning and information lookup in clinics disconnected from the Internet.
4 FINDINGS
The findings section organizes the study’s evidence around contextual adoption challenges, system-level technical and usability barriers, and clinicians’ perceived benefits and future views of AI-CDSS.
- Rural clinical context: The paper first examines how rural clinical conditions create challenges for adopting AI-CDSS.This frames adoption as shaped by the surrounding clinical context rather than by system features alone.
- Technical and usability barriers: It then reports technical limitations and usability barriers encountered with Brilliant Doctor in practice.The findings treat these issues as distinct from the broader rural-context challenges.
- Benefits and future directions: Finally, the section discusses perceived benefits, clinicians’ views about AI-enabled CDSS’s future, and design considerations for improvement.The stated focus includes supporting future AI-CDSS design in rural clinical settings.
4.1 The Contextual Challenges to Adopting AI-CDSS in Rural Clinics
Rural clinics’ high patient volume, compressed multitasking workflows, limited resources, and disconnected information systems conflicted with Brilliant Doctor’s textbook, stepwise design. These contextual and technical mismatches reduced clinicians’ use of AI-CDSS and weakened the relevance of its recommendations.
- Workflow and workload: Rural clinicians handled high patient volume and frequent interruptions, leaving little time for the segmented workflow assumed by Brilliant Doctor.One clinic reported 75 patients per doctor daily, while another averaged 150 visits per day for two clinicians.
- Workflow and workload: Clinicians multitasked across diagnosis steps rather than following the textbook sequence, completing consultations in roughly 2 to 4 minutes.They entered symptoms while questioning patients, recorded diagnoses while deciding them, and used templates to accelerate routine cases.
- Resource constraints: Staff shortages limited the patient information captured for AI-CDSS, reducing the system’s ability to produce accurate and comprehensive diagnoses.Nurses were often unavailable for initial checks and medical-history collection, making required inputs difficult to obtain.
- Usability barriers: The system’s click-through interaction required more time than rural clinicians could spare, so users preferred typing and rarely used AI-CDSS in practice.The click-through design supported individual workflow steps, including screening questions and automatic treatment entry, but conflicted with compressed consultations.
- Local-context mismatch: AI-CDSS recommendations often mismatched rural clinics’ available tests, medicines, and specialties, so clinicians ignored or bypassed them.Clinics lacked equipment such as CT scanners, had limited medication stocks, and some systems lacked TCM codes and medicines.
- Technical limitations: Poor interoperability with laboratory, pharmacy, and departmental systems caused missing inputs, inaccurate suggestions, and manual workarounds.Clinicians manually maintained medicine-stock notes, while laboratory results and pharmacy availability were not seamlessly synchronized with AI-CDSS.
4.2 Challenges related to Usability, Technical Limitations, and Trustworthiness of AI
Participants reported that Brilliant Doctor created usability and technical barriers, while limited accuracy, opaque reasoning, inadequate training, and accountability concerns constrained trust and adoption in practice.
- Usability Barriers: The system’s pop-up window obstructed EHR fields and controls, forcing clinicians to minimize it or move it aside during routine work.Participants specifically reported that the window could occupy one third of the screen and block buttons such as “Save” or “Exit.”
- Usability Barriers: Grouping many functions and alerts in scrollable pop-up boxes made features difficult to discover and caused clinicians to skip excessive alerts.Some participants did not know that features such as Similar Cases existed, while dense alert lists were not read through.
- Technical Limitations: Recommendations were not consistently accurate or personalized because the system lacked complete patient information and could not capture subtle clinical cues.Participants cited low confidence scores, protocolized symptom-based recommendations, and inability to interpret facial expressions or other contextual hints.
- Technical Limitations: Outdated or overly narrow medical information further reduced usefulness, including inaccurate medication instructions and weak coverage of surgical problems.Participants reported discrepancies in medicine dosage descriptions and limited support for wounds, fractures, and other surgical issues.
- Trustworthiness and Professional Autonomy: Clinicians viewed AI-CDSS as requiring verification because its black-box reasoning, limited training, and possible effects on professional autonomy weakened trust.Accountability for patient outcomes remained with clinicians, who also criticized time-consuming click-through questions and feared overreliance or deskilling.
- Trustworthiness and Professional Autonomy: The system’s technical limitations and interface demands reinforced clinicians’ view that AI should assist rather than replace human clinical judgment.Participants linked responsibility for final decisions to clinicians and emphasized that diagnosis depends on expertise, patient interaction, and professional judgment.
4.3 Perceived Usefulness of AI-CDSS
Despite criticizing Brilliant Doctor’s functionality and user experience, participants identified practical benefits across diagnosis, information search, on-the-job learning, and prevention of treatment risks.
- Overall Usefulness: 18/20 participants believed AI-CDSS already supported their work, particularly through diagnostic assistance, information search, training, and adverse-event prevention.These four scenarios were identified as recurring ways the system contributed to clinical work despite adoption challenges.
- Supporting Diagnostic Process: Diagnostic suggestions helped clinicians consider alternatives, investigate discrepancies, and verify their own diagnoses.Participants described the system as a reminder and reassurance mechanism, especially for uncommon or complex diseases.
- Medical Information Search: The embedded search engine broadened clinicians’ access to medicine and disease information while reducing reliance on manually maintained reference materials.Participants used it frequently for changing medication information such as side effects and usage instructions.
- On-the-Job Training Opportunity: Similar Case supported on-the-job learning by exposing clinicians to cases from top-tier research hospitals and enabling self-education.Participants considered this especially useful for rarely encountered cases and for expanding clinicians’ knowledge pools.
- On-the-Job Training Opportunity: AI-CDSS could support junior clinicians by providing recommendations for unfamiliar conditions while they gained diagnostic knowledge and experience.Participants specifically described recent residents as likely beneficiaries when they lacked prior experience with rare symptoms.
- Preventing Adverse Events: Automated alerts helped clinicians identify prescription errors, drug allergies, incompatible medicines, and risks such as CT scans for pregnant patients.Participants framed these alerts as safeguards against serious mistreatment and medication-related adverse events.
5 DISCUSSION
The discussion identifies tensions between Brilliant Doctor and rural clinical practice, while arguing for cooperative human-AI collaboration with clear division of labor. It also notes that the qualitative study’s Beijing setting and clinician-only sample constrain generalizability and stakeholder coverage.
- Adoption was hindered by technical limitations, workflow and system-integration gaps, usability barriers, and concerns about AI trustworthiness and professional autonomy.
- Brilliant Doctor was not used as extensively as predicted, largely because its design did not account for local context and rural work practices.
- The system’s click-through interaction required at least 10 minutes for comprehensive data collection, conflicting with clinicians’ limited time per patient.
- Clinicians sometimes dismissed recommendations because the system lacked patient-specific contextual factors, including insurance coverage and medication cost.
- Participants preferred AI as a doctor assistant rather than a replacement or replica of clinicians’ diagnostic work, supporting collaboration with a clear division of labor.
- The qualitative study focused on one rural area in Beijing and did not interview patients, limiting generalizability and excluding potentially affected patient perspectives.
6 CONCLUSION
The conclusion reports an observational and interview study of clinicians using a recently deployed AI-CDSS in rural China. It identifies contextual, technical, usability, and trust-related adoption challenges and derives design implications from them.
- The study observed and interviewed 22 clinicians at 6 rural clinics in China about their use of a recently deployed AI-CDSS system.
- The study found misalignment with local context and workflow, technical limitations, user-experience issues, and concerns about system trustworthiness.
- The authors use these findings to discuss design implications for AI-CDSS deployment in rural clinical contexts.
APPENDIX
Figure 4 presents English translations of screenshots showing the EHR interface on the left and the AI-CDSS interface on the right.
- The figure places the EHR user interface on the left and the AI-CDSS user interface on the right.