PaperlayerView exact revision on arXiv

Source-linked AI summary

A Disease Diagnosis and Treatment Recommendation System Based on Big Data Mining and Cloud Computing

Jianguo Chen, Kenli Li, Huigui Rong, Kashif Bilal, Nan Yang, Keqin Li

arXiv:1810.07762v1cs.LGstat.ML

TL;DR

The paper addresses inaccurate disease classification across multiple treatment stages and limited sharing of medical expertise. It proposes DDTRS, combining DPCA clustering, Apriori association analysis, and Apache Spark parallelization. The system is reported to provide effective clustering and high-quality, low-latency recommendations.

  • Problem

    Traditional classification may struggle with diseases involving multiple treatment stages, varied symptoms, and multi-pathogenesis, while medical knowledge sharing across hospitals and departments is limited.

  • Method

    DDTRS uses DPCA for disease-symptom clustering, Apriori for Disease-Diagnosis and Disease-Treatment rules, and Apache Spark for parallel processing.

  • Results

    The system effectively clusters disease symptoms and provides high-quality treatment recommendations with low latency response.

  • Takeaways & Limitations

    DDTRS supports diagnosis and treatment recommendations for doctors and patients using historical inspection and treatment records.

  • Takeaways & Limitations

    The original DPCA density calculation has low robustness and may identify inaccurate density peaks in multi-density or non-uniform data.

Abstract

from arXiv · show

It is crucial to provide compatible treatment schemes for a disease according to various symptoms at different stages. However, most classification methods might be ineffective in accurately classifying a disease that holds the characteristics of multiple treatment stages, various symptoms, and multi-pathogenesis. Moreover, there are limited exchanges and cooperative actions in disease diagnoses and treatments between different departments and hospitals. Thus, when new diseases occur with atypical symptoms, inexperienced doctors might have difficulty in identifying them promptly and accurately. Therefore, to maximize the utilization of the advanced medical technology of developed hospitals and the rich medical knowledge of experienced doctors, a Disease Diagnosis and Treatment Recommendation System (DDTRS) is proposed in this paper. First, to effectively identify disease symptoms more accurately, a Density-Peaked Clustering Analysis (DPCA) algorithm is introduced for disease-symptom clustering. In addition, association analyses on Disease-Diagnosis (D-D) rules and Disease-Treatment (D-T) rules are conducted by the Apriori algorithm separately. The appropriate diagnosis and treatment schemes are recommended for patients and inexperienced doctors, even if they are in a limited therapeutic environment. Moreover, to reach the goals of high performance and low latency response, we implement a parallel solution for DDTRS using the Apache Spark cloud platform. Extensive experimental results demonstrate that the proposed DDTRS realizes disease-symptom clustering effectively and derives disease treatment recommendations intelligently and accurately.

1. Introduction

The paper proposes DDTRS to address inaccurate disease classification, limited medical knowledge sharing, and the need for efficient recommendations across hospitals and departments.

  • Motivation: DDTRS analyzes historical inspection and treatment data to recommend disease diagnoses and treatment plans according to current disease stages.The system is intended to support both medical doctors and patients.
  • Contributions: DPCA clusters disease symptoms from large-scale historical inspection data, particularly for diseases with multiple treatment stages and multi-pathogenesis.
  • Contributions: Apriori analysis derives Disease-Diagnosis and Disease-Treatment association rules to share experienced doctors’ and developed hospitals’ knowledge.
  • Contributions: Interactive interfaces let doctors and patients inspect reports, receive recommendations for different treatment stages, and update inspection results.
  • Contributions: Apache Spark parallelizes DDTRS, with medical data stored in HDFS and processing implemented through the RDD programming model.

2. Related Work

Related work covers healthcare big-data analytics, clustering, association analysis, recommendation systems, and cloud-computing platforms for medical applications.

  • Healthcare analytics: Healthcare research applies big-data analytics to electronic health records for pattern discovery, association analysis, visualization, querying, privacy, and predictive analytics.
  • Clustering analysis: Existing clustering research includes partitioning, hierarchical, density-based, grid-based, model-based, and density-peak-based methods.
  • Recommendation and association analysis: Prior recommendation studies use user models, user-project matrices, random walks, group recommendation, graph-based taxonomies, and context-aware approaches.
  • Cloud computing: Big-data and cloud-computing research includes medical-data processing systems and recommendation methods designed for cloud environments.

3. Disease Diagnosis and Treatment Recommendation System

DDTRS combines disease-symptom clustering with diagnosis and treatment recommendation, using patient inspection reports as the input for stage-specific plans.

  • System architecture: DDTRS has a disease-symptom clustering module and a disease diagnosis and treatment recommendation module.
  • System architecture: DPCA classifies disease symptoms from patient inspection reports to identify disease-symptom clusters.
  • Workflow: Patients submit inspection reports to obtain a disease-symptom cluster and corresponding diagnosis and treatment recommendations.

3.1. Standardization Process of Medical Data

The system standardizes heterogeneous inspection data from departments and hospitals, extracts feature variables, and organizes diagnosis and treatment records for association analysis.

  • Medical-data standardization: Inspection datasets from different departments and hospitals require standardization because hospitals may use incompatible inspection standards.
  • Medical-data types: The inspection data include numeric, text, and image formats collected from diverse inspection tasks.
  • Numeric inspection data: Numeric inspection reports use patient information and inspection items as feature variables for disease-symptom clustering.
  • Text and imaging data: Imaging tasks such as CT, MRI, X-ray, sonography, and Doppler ultrasound express symptom results as images or text.
  • Diagnosis and treatment data: Disease diagnosis-scheme data contain detailed doctor-recorded diagnoses, including diverse schemes for different treatment stages of the same disease.
  • Diagnosis and treatment data: Association items link disease-symptom clusters with diagnosis schemes and treatment schemes for later association analysis.
  • Diagnosis and treatment data: Treatment schemes include injections, intravenous infusions, surgery, needle therapy, cupping therapy, and physical therapy, with plans adjusted to inspection results.

3.2. DPCA-based Disease-Symptom Clustering

DPCA clusters disease symptoms by identifying density peaks and assigning remaining points to higher-density neighboring clusters. Its GKF-based density calculation improves robustness for multi-density and non-uniform data, while the workflow separately processes inspection tasks.

  • Clustering procedure: DPCA clusters disease symptoms by identifying centers with high local density and delta distance, then assigning remaining points to nearby higher-density clusters.Low-density points with high delta distances are treated as outliers.
  • Data preparation: Each inspection-task dataset is clustered separately because the datasets have different dimensions.The numeric inspection data are grouped by task name before DPCA clustering.
  • Limitations of original DPCA: The original cutoff-based density measure is sensitive to cutoff distance, discretizes density values, and can produce inaccurate peaks under multi-density or non-uniform distributions.These limitations make accurate detection of local density peaks and cluster centers difficult.
  • GKF-based improvement: The improved DPCA uses a Gauss Kernel Function to calculate continuous local densities, reducing cutoff sensitivity and improving robustness, including for uniformly distributed data.The kernel width parameter σ is set to the cutoff distance dc in this work.
  • Complexity: DPCA performs cluster assignment in one step with time complexity O(Mn^2) and space complexity O(n^2), which the paper characterizes as low relative to iterative clustering.M denotes the number of inspection tasks and n the average number of records per task.

3.3. Disease Diagnosis and Treatment Recommendation

DDTRS clusters disease symptoms, mines diagnosis–treatment associations, and delivers stage-specific recommendations through doctor and patient interfaces. The system uses Apriori rules over clustered records, while recommendation quality requires clinical feedback and patient-facing safeguards.

  • DDTRS uses disease-symptom clusters to accumulate doctors’ diagnosis and treatment knowledge, then recommends disease treatment plans to doctors and patients.
  • Apriori mines D-D and D-T associations between disease-symptom clusters, diagnosis schemes, and treatment schemes.The extracted strong rules support diagnosis and treatment recommendations.
  • Association records combine inspection reports, diagnosis schemes, and corresponding treatment schemes from multiple patient visits, grouped by disease.
  • D-T rules retain treatment associations meeting minimum support 2 and minimum confidence 50%.
  • Interactive doctor and patient applications expose inspection reports and corresponding recommendations, which can be updated as inspection results change.
  • Recommendation evaluation depends on doctor feedback because diverse pathologies and patient conditions make a common treatment-quality standard difficult.
  • Patient recommendations may cause anxiety or be difficult to understand, so doctors can control message access and patient evaluations are excluded.

4. Parallel Solution of DDTRS

DDTRS parallelizes disease-symptom clustering and treatment-scheme association analysis on Apache Spark to improve recommendation efficiency and response latency. The clustering pipeline uses DPCA-derived density measures, while association analysis exploits distributed RDD computation to reduce complexity.

  • System objectives: DDTRS targets efficient disease-symptom clustering and low-latency treatment recommendations after patient symptom data are submitted.The section identifies clustering efficiency and recommendation response latency as its critical performance issues.
  • Spark deployment: Historical inspection datasets are stored in HDFS and processed as distributed RDD objects on the Spark cloud platform.RDD operations execute across computing nodes, enabling parallel handling of inspection-task subsets.
  • RDD execution: RDD dependencies form a task-scheduling DAG for the parallel disease-symptom clustering process.The clustering workflow contains both narrow and wide dependencies among generated RDD objects.
  • Disease-symptom clustering: DPCA computes local density ρi and delta distance δi in parallel, selecting records exceeding ερ and εδ as disease-symptom cluster centers.Remaining records are assigned to their nearest cluster independently across inspection tasks.
  • Treatment association: Treatment-scheme association analysis parallelizes Apriori computation over treatment records and disease-symptom clusters, reducing its time complexity through distributed allocation and parallel processing.The association workflow uses RDD dependencies between treatment schemes and disease-symptom clusters.

5. Experiments and Applications

The experiments evaluate DDTRS through clustering accuracy, recommendation quality, application cases, and overall system performance.

  • Evaluation scope: DDTRS is evaluated for clustering accuracy and recommendation quality.These evaluations are presented in Section 5.2.
  • Applications: Two disease treatment-recommendation cases are analyzed as applications of DDTRS.The case analyses are presented in Section 5.3.
  • Performance assessment: The performance of DDTRS is assessed separately from its accuracy, quality, and case analyses.The performance assessment is presented in Section 5.4.

5.1. Experimental Setup

DDTRS uses a client/server architecture in cooperating hospitals, with data analysis deployed on a 30-node Apache Spark cloud environment at NSCC.

  • System architecture: DDTRS uses data-collection terminals in cooperating hospitals and a data-analysis terminal for processing and feedback.The system gathers historical and current medical datasets and returns clustering and recommendation results to doctors.
  • Hardware environment: The Spark experimental platform contains 30 computing nodes, each equipped with eight Intel Xeon Nehalem EX CPU cores and 32GB memory.The passage specifies the node count, processor configuration, and memory capacity.
  • Association-analysis setup: The parallel Apriori process takes disease-symptom clusters and treatment-scheme records as inputs and returns frequent itemsets after support and confidence filtering.Rules are retained when their support and confidence meet the preset minimum thresholds.
  • Software environment: The computing nodes run Ubuntu 15.10 with Apache Spark 1.6.0 and communicate through a high-speed Gigabit network.The setup specifies the operating system, Spark version, and inter-node connection.

5.2. Accuracy Evaluation

The evaluation compares disease-symptom clustering methods using Cluster Accuracy and examines DPCA robustness to cutoff-distance changes. DPCA is especially effective for diseases with multiple stages, varied symptoms, or multi-pathogenesis, while its density distributions remain stable across tested cutoff distances.

  • Accuracy of Disease-Symptom Clustering: Cluster Accuracy measures the ratio of correctly classified or clustered instances to predefined class labels, with higher values indicating purer clusters.The metric uses majority labels within each class or cluster.
  • Accuracy of Disease-Symptom Clustering: 88.35% is the anemia accuracy achieved by DPCA, exceeding K-Means at 72.36%, RF at 66.21%, and C4.5 at 57.92%.For influenza and Diabetes Mellitus, traditional classification methods perform better than clustering methods.
  • Accuracy of Disease-Symptom Clustering: DPCA is suited to diseases with multiple treatment stages, various symptoms, or multi-pathogenesis, whereas traditional classification performs better for simpler diseases.The reported comparison includes influenza and Diabetes Mellitus as simpler cases and anemia as a more complex case.
  • Robustness of the DPCA Algorithm: Using GKF, increasing cutoff distance from 0.57 to 0.88 leaves the density distribution basically stable, with the same three candidate centers and four outliers.The comparison contrasts GKF with the greater cutoff-distance sensitivity observed for Eq. (2).
  • Quality Evaluation of Treatment Recommendation: The treatment-quality evaluation uses effectiveness, chronergy, non-harmful side-effects, economy, and patient satisfaction, aggregated from weighted doctor scores.Each indicator is scored from 1 to 5, and the combined quality is represented by a radar-graph area.
  • Quality Evaluation of Treatment Recommendation: 34.34 is the average overall quality score across four treatment schemes, while economy is the lowest-rated indicator at 3.24 (64.80%) on average.Overall scores range from 28.44 to 37.27; effectiveness averages 4.33 (86.60%).

5.3. Application of Disease Treatment Recommendation

DDTRS applies disease-symptom clustering to inspection reports and then recommends diagnosis and treatment plans from the resulting disease cluster. In two cases, it identifies Liver Cancer and ACAS and supplies corresponding clinical recommendations.

  • Liver Cancer Case: A tumor-marker inspection report is clustered into a Liver Cancer disease-symptom cluster.The cluster center is characterized by mean symptom values from similar historical inspection data.
  • Liver Cancer Case: For Liver Cancer, DDTRS recommends surgical resection as the treatment scheme and provides a medication plan.The recommendation is described as the best choice because it can remove the tumor tissue completely.
  • ACAS Case: A blood-serum inspection report from a pregnant 33-year-old volunteer is clustered into Anti-Cardiolipin Antibody Syndrome rather than the inexperienced-doctor alternatives SLE or RA.The cited case describes the ACAS cluster and subsequent diagnosis and treatment recommendations.
  • ACAS Case: ACAS is described as clinical signs caused by an anti-cardiolipin antibody, associated with thrombosis, thrombocytopenia, spontaneous abortion, and intrauterine fetal death.The diagnosis description connects the antibody with SLE and other autoimmune diseases.
  • ACAS Case: The recommended ACAS treatment combines heparin sodium injections with medication.The medication plan is provided separately in Table 7.

5.4. Performance Evaluation

The Spark implementation improves DDTRS performance as computing nodes increase from 5 to 30. Clustering and association-analysis workloads both show substantial execution-time reductions, with larger datasets benefiting more from parallel optimization.

  • Performance of Disease-Symptom Clustering: At five nodes, clustering takes 1976.56 s for hemorrhagic disorder, 1621.25 s for anemia, 764.37 s for diabetes mellitus, and 528.52 s for influenza.The datasets range from 5GB to 20GB, respectively.
  • Performance of Disease-Symptom Clustering: Influenza clustering falls from 528.52 s to 103.68 s, while diabetes-mellitus clustering falls from 764.37 s to 145.52 s as nodes increase from 5 to 30.The paper attributes the improvement to data-parallel and task-parallel schemes.
  • Performance of Disease-Symptom Clustering: The clustering benefit is more noticeable for larger datasets or when the number of slave nodes increases to a finite extent.The reported comparison links dataset size and node count with the observed parallel-optimization benefit.
  • Disease-Treatment Association Analysis and Recommendation: At five nodes, association-analysis execution times are 262.45 s for influenza, 240.71 s for proteinuria, 213.36 s for chronic gastritis, and 192.12 s for gout.These are the starting values before scaling the Spark cluster.
  • Disease-Treatment Association Analysis and Recommendation: Scaling from 5 to 30 nodes yields speedup ratios of 6.02, 5.70, 5.20, and 4.78, with an average execution time of approximately 41.76 s.The paper characterizes the resulting speedup as almost linear.

6. Conclusions

The DDTRS recommends treatments from patient inspection reports by clustering symptoms and identifying disease-treatment links. Results indicate high-quality recommendations with low-latency response, while effectiveness evaluation and security remain future work.

  • DDTRS clusters patients’ inspection reports to identify disease symptoms and applies association analysis to discover disease-treatment links.These links support treatment recommendations based on patients’ inspection reports.
  • The system provides diagnosis and treatment recommendations for patients and doctors using the identified disease-treatment relationships.The paper demonstrates recommendation use through qualitative patient and doctor applications.
  • The system achieves high-quality recommendations with low latency response according to experimental and application results.
  • Future work will evaluate treatment-scheme effectiveness by tracking changes in each patient’s inspection parameters.The authors also identify weighted recommendations and system security as future concerns.
Loading 1810.07762v1…