Design of Machine Learning Algorithms and Internal Validation of a Kidney Risk Prediction Model for Type 2 Diabetes Mellitus

Introduction

The prevalence and incidence of type 2 diabetes (T2D), representing more than 90% of all diabetes cases, are rapidly increasing worldwide.¹ Diabetic kidney disease (DKD) is a critical microvascular complication of diabetes mellitus with a high prevalence, mortality rate, and substantial financial burden.² Approximately one-third of T2D patients can progress to DKD,³ subsequently leading to end-stage kidney disease (ESKD) and death.⁴ Identifying risk factors and early diagnosis of DKD are crucial for the prevention and treatment of ESKD.

Chronic kidney disease leading to ESKD in T2D includes DKD, nondiabetic kidney disease (NDKD), or a combination of DKD and NDKD. Given the different treatment options, renal biopsy histopathology remains the gold standard for distinguishing between DKD and NDKD.⁵ However, renal biopsy is cautiously used in clinical diagnosis. In contrast to histopathological biopsy, diagnosing DKD based on clinical, physiological, and biochemical indicators can shorten the diagnostic time, reduce patient discomfort, and minimize medical risks. Therefore, many risk factors involved in the initiation and progression of DKD, such as increasing age, family history, hyperglycemia, hypertension, dyslipidemia, dietary patterns, and lifestyles have been reported.⁶ Physiological indicators, such as urinary albumin-to-creatine ratio and estimated glomerular filtration rate (eGFR), have been included as screening indicators for the occurrence and grading of DKD in clinical diagnostic guidelines for DKD.

Although researchers have studied early predictive markers of DKD, including proteomics and genomics, these new biomarkers are challenging to popularize in clinical setting due to their low sensitivity and specificity in evaluating early DKD. Even the effectiveness of microalbuminuria as a traditional DKD marker and the best opportunity for intervention has recently been questioned. In fact, some biochemical indicators of T2D patients have been considered to provide guidance for the early diagnosis of DKD. The varying presentation of DKD poses challenges for clinicians in terms of accurate detection and selecting appropriate individualized interventions in high-risk individuals.⁷ Therefore, it is necessary to build a simple and practical DKD risk prediction model to help clinicians identify patients’ risks early, provide early treatment, and prevent T2D patients from progressing to DKD, rather than blindly pursuing new indicators.

In recent years, machine learning (ML) algorithms have made significant progress in disease prediction and prognosis forecasting. Previous studies have successfully applied ML algorithms for the prediction of ESRD in DKD participants.⁸ As a novel statistical modeling method, based on conventional laboratory test data, we will apply multiple ML algorithms to develop a real-world DKD artificial intelligence early warning model. Our research will provide effective guidance for early diagnosis and risk prediction of DKD.

Methods

Study Populations and Design

T2D hospitalized patients in this study were enrolled from January 2021 to July 2022 at the First Hospital of Jilin University and the Third Hospital of Jilin University. Only patients with a clinical diagnosis of T2D and a definite pathological diagnosis were included. This study does not include ESKD patients (defined as those receiving basic dialysis, kidney transplantation, or with eGFR < 15 mL/min/1.73 m²), severe cardiovascular disease (CVD), nervous system disease, acute infectious disease, hepatitis virus infection, or carrying a history of such infection, diabetic ketoacidosis, any type of cancer, or identified autoimmune diseases. T2D was diagnosed according to the American Diabetes Association standard.⁹ Nephropathological Classification of DKD is based on the Society of Renal Pathology (RPS) criteria from 2010.¹⁰ A schematic diagram of patient selection is shown in Figure 1. Our research complies with the Declaration of Helsinki and was approved by the Ethics Committee of the First Hospital of Jilin University (No. 2023–333) and the Ethics Committee of the Third Hospital of Jilin University (No. 2023033021).

Figure 1 Flowchart showing the selection of patients and construction of the predictive model.

Blood Sampling and Laboratory Testing

Blood samples from patients were collected into metal-free tubes after an overnight fast. After centrifugation at 1500 g for 10 min, serum was aliquoted into metal-free Eppendorf tubes and stored at −80°C. A total of 50 hematological and biochemical parameters were tested at baseline, including fasting plasma glucose, hemoglobin A1c (HbA1c), glycated albumin (GA), and aspartate aminotransferase (AST). Detailed indicators are shown in Table 1. These indicators were examined using the AU5800 (Sunto-gun Bakman, Japan, 454–32 Higashino, Nagaizumi-cho, Sunto-gun, Japan), XN-9000 (Sysmex, Hyogo Prefecture, Japan), UniCel DXI800 (Chaska Bakman, USA), and HLC-723G8 glycosylated hemoglobin analyzer (Tosoh, Japan). All laboratory data have passed external quality evaluation and certification by the Jilin Clinical Trial Center. These instruments underwent strict quality control inspection before use. Demographic data of these patients, including age and sex, were extracted from their medical records. The requirement for written informed consent was waived by the ethics committee because the data used in the research were anonymous.

Table 1 Clinical Characteristics of the Study Participants

Data Cleaning and Normalization

To improve the data quality and ensure data accuracy, consistency, and availability, we performed data cleaning and standardization. The patient’s ID number was used as the unique identifier.¹

Data elimination and reporting: we removed data points with a missed detection rate exceeding 30%. Abnormal values, defined as those greater than three times the interquartile range, were also removed. For variables with missing data, we utilize appropriate filling methods such as median substitution, mean substitution, or complex substitution based on the data distribution characteristics.²

eGFR reporting: we assessed the renal function periodically using the CKD-EPI method.¹¹

Statistical Analysis

The frequency difference of categorical variables was estimated using either a block test or Fisher’s precision test. For the comparison between continuous variables with normal distribution, we used t-test and expressed them as the mean ± standard deviation. Variables that did not follow a normal distribution were compared using the Mann–Whitney U-test or Kruskal–Wallis test, with results expressed as the median and interquartile range (IQR). Categorical variables were presented as frequency with proportion and analyzed using the chi-square test. To explore correlations among the parameters, Spearman correlation analysis was conducted.

Feature selection includes two steps as follows. First, we calculated the Pearson correlation coefficient (PCC) among all features. If the PCC value between a pair of features exceeded 0.8, one of the features was randomly removed. The remaining features underwent an F-test, and 50% of them were further selected to establish the model.

Model Development

The Deepwise & Beckman Coulter DxAI platform was used for feature selection and model building. Five commonly used ML algorithms, namely Extreme Gradient Boosting (XGBoost), Random Forest (RF), Support Vector Machine (SVM), Gradient Boosting Machine (GBM), and Logistic Regression (LR), were used to select key features and construct risk prediction models.

A 5-fold cross-validation approach was applied, where the entire dataset was divided into five subsets. In each iteration, four of these subsets were used as a training set to train the model, while the remaining one served as a validation set to evaluate the model. This process was repeated five times, and the results were averaged.

The maximum likelihood model was constructed using the training set, and the diagnostic performance of the optimal likelihood model was evaluated. A comparison was made between the best-performing ML model and each major risk factor (Figure 1).

Assessment of Model Performance

To assess the performance of the diagnostic model, receiver operating characteristic (ROC) curves were employed. We calculated the 95% CI and the area under the curve (AUC). A model with an AUC greater than 0.75 was considered to demonstrate good performance. A P-value of less than 0.05 was considered statistically significant. The calibration curve was used to evaluate the goodness-of-fit of the model. A P-value exceeding 0.05 was considered indicative of a well-fitting model. Moreover, decision curve analysis (DCA) was used to evaluate the clinical efficacy of the model. The data analysis software used in this study includes SPSS 24.0 (IBM, Armonk, NY), Stata 15.0 (Stata Corp LLC, Texas, USA), and GraphPad Prism 8.0 (GraphPad Software Corp, San Diego, USA).

Results

Baseline Characteristics of the Participants

In total, data from 166 T2D patients with DKD and 68 T2D patients without DKD were collected. All routine laboratory indices, age, and sex of the patients were selected as modeling characteristics. The Kolmogorov–Smirnov Normality Test results showed that the data for laboratory indicators were not normally distributed (except for P_Age = 0.168, P_Zn = 0.582, P_UIBC = 0.062, all other P-values were <0.01). Some routine laboratory indicators had missing data, and median interpolation was used to address it. The baseline characteristics of the study participants are presented in Table 1.

Construction of ML Model and Evaluation

The Pearson correlation coefficient (PCC) value for certain feature pairs exceeding 0.8, and as a result, the F-test was applied, leading to the removal of 50% of those characteristics. These features included DBil, IBil, BUN, Cystatin C, Cre, and LY%.

In the training set, the result of the internal 5-fold cross-validation for the five algorithms indicated that the RF algorithm method showed superior performance during internal validation. The RF algorithm identified five major factors: eGFR, GA, Uric acid, HbA1c, and Zn. The model’s performance was assessed using various metrics, including AUC, negative predictive value, positive predictive value, accuracy (ACC), recall, sensitivity (SEN), specificity (SPE), and precision (Table 2) When compared to all evaluation techniques, the RF algorithm outperformed others by achieving a higher AUC.

Table 2 5-Fold Cross-Validation Effectiveness of Five ML Algorithms

Model Implementation in Clinical Settings

To understand the contribution of each predictor to the ML models, we computed the feature variables’ importance in the RF algorithm model for each outcome. All risk factors (eGFR, GA, Uric acid, HbA1c, and Zn) were assigned scores based on their weightings, with the highest score being 38.01.

We encouraged doctors and professionals to test our optimal predictive ML model during the early stages of diagnostic testing for educational purposes. To facilitate this, we have developed a web-based risk calculation tool for predicting DKD. The Deepwise & Beckman Coulter DxAI platform was used for feature selection and model building. The system can be accessed by users for easy evaluation of DKD risk via the following link: DKD Risk Prediction Tool. (https://dxonline.deepwise.com/prediction/index.html?baseUrl=%2Fapi%2F&id=34208&topicName=undefined&from=share&platformType=wisdom)

Internal Validation of the Model

The RF model demonstrated a high level of prediction accuracy, with the AUC value of 0.960 (95% CI: 0.936–0.984) during internal validation (Figure 2). The result of the Hosmer-Lemeshow goodness-of-fit test indicated that the P-value for each model in both the training and validation sets exceeded 0.05, indicating a good model fit (Figure 3). The DCA model has a significant net gain in both the training and validation sets (Figure 4)

Figure 2 Receiver operating characteristic curve. (A) Training set. (B) Validation set.

Figure 3 Calibration curve. (A) Training set. (B) Validation set.

Figure 4 Decision curve analysis in the training set and validation set.

External Validation of the Model

To validate the ML model established in this study, data from patients at the Third Hospital of Jilin University were collected as an external test set. Among all the results, the RF algorithm demonstrated the best predictive performance and successfully passed external validation. In this analysis, the model’s accuracy was assessed using the AUC (0.9316), sensitivity (0.9000) and overall accuracy (0.9000). An external dataset was used to ensure consistent analysis with the internal validation (Figure 5).

Figure 5 Receiver operating characteristic curve in the external validation set.

Comparison of RF and the Five Features Screened

We compared the five features screened with the highest feature importance in the RF model and the RF algorithm method. Among these, the diagnostic effectiveness of the RF model (AUC = 0.960) was significantly higher than each of the five features. The performance of the model was evaluated by AUC, Cut-off, sensitivity, specificity, Youden’s index (YI), standard error and 95% CI (Table 3).

Table 3 Comparison of RF and the Five Features Screened in the Validation Set

Discussion

Despite the significance of DKD, few studies have established a differential diagnosis model for DKD, particularly when combined with laboratory parameters. The importance of developing a simple and practical diagnostic model lies in its potential to enable accurate and rapid clinical diagnosis of DKD, rather than blindly adding more indicators. Our study aimed to construct a risk prediction model for patients with DKD using artificial intelligence (AI), a method whose performance has been extensively tested and validated.

Furthermore, this model could play a crucial role in the prevention and management of T2D, especially in individuals with diabetes at an increased risk of developing DKD. To be more specific, if high-risk T2D patients exhibit abnormalities in the parameters outlined in the diagnostic model during routine physical examinations, this could serve as an early alert to the possibility of progressing to DKD, thus enabling timely intervention.

The results of the ML model can be influenced by the characteristics of the participants. Due to the imbalance between the groups studied, there may be qualitative differences, such as gender, and age, between the two groups. Significant correlations between the factors under investigation may lead to deviations in the results. In this study, the correlation of indicators was analyzed using Spearman correlation. Highly correlated metrics were excluded to ensure that other causes or confounding factors did not affect the study results.

The efficient predictive model established in this study offers several advantages. First, the simple model has been uploaded to a website, making it easy to understand and use. The system developed as part of the research is based on an RF model on a server, allowing for high-speed analysis of patient data. Users can easily access this data through the Internet using PCs and smartphones.

Second, the model comprehensively assesses representative laboratory test indicators to distinguish between DKD and T2D, including hematological and biochemical parameters, with a special focus on the novel indicator GA to improve diagnostic accuracy. To reduce the workload of measuring and inputting patient data, the number of variables in our system model has been limited. AI and communication technology have been used to implement this system.

Nevertheless, the performance of the diagnosis model was excellent, highlighting its potential utility as an adjunct test for a wide range of DKD patients. In this study, the diagnostic model, which combines eGFR, GA, Uric acid, HbA1c, and Zn to assess kidney function damage in T2D, was established using ML algorithms. A progressive decline in eGFR is a clear indication of DKD and is part of the diagnostic criteria.¹² Poor and continuous control of blood glucose is one of the most significant risk factors for DKD. GA is the final product of non-enzymatic glycosylation resulting from the reaction between serum albumin and glucose. GA is associated with blood glucose concentration and albumin half-life, reflecting the average blood glucose status of patients in the first 2–3 weeks.¹³ The GA concentration can be determined based on the quantitative measurement of glycosylated serum albumin and the percentage of serum albumin. HbA1c is currently recognized as a key marker for the development of long-term diabetic complications in patients with diabetes. Additionally, HbA1c is an important index for preventing the occurrence and progression of diabetic complications by achieving good blood glucose control.¹⁴ Previous studies have indicated that the concentration of HbA1c, unrelated to blood sugar, may be affected by various conditions, such as iron deficiency, red blood cell lifespan, hemolytic anemia, renal anemia, hemoglobin mutations, chronic kidney disease, and more.^15–17 GA detects glycemic fluctuations more effectively than HbA1c in the context of DKD.

Moreover, the measurement of GA serves as an additional tool for the clinical diagnosis of DKD, facilitating an index for evaluating the severity of kidney function disorder to guide treatment.^17–19 GA contributes to the progression of DKD by promoting the production of acidic molecules by membrane cells and epithelial cells after being absorbed by cells. GA also accelerates insulin resistance by increasing the production of intracellular reactive oxygen species, which inhibits glucose uptake by muscle and fat cells.²⁰ Moreover, we performed the study to obtain data from participants to establish a novel model, that highlights the possibility for clinical assessment of blood glucose control and the identification of kidney function damage in the diagnosis and screening of DKD.

Uric acid is the end product of purine base metabolism, generated in the exchange process between nucleotides and adenosine triphosphate (ATP). Increasing epidemiological evidence indicates that hyperuricemia, acting as an oxidant, serves as an independent risk factor for DKD. It is closely associated with the onset and progression of DKD. Small proof-of-concept clinical trials have suggested that reducing uric acid levels with allopurinol may lead to a decrease in the rate of eGFR decline, though whether it benefits DKD progression remains controversial.^21–26

The potential mechanisms behind uric acid-induced kidney damage may involve mediating insulin resistance, inactivation through oxidative reactions that lower nitric oxide (NO) levels, pancreatic β-cell dysfunction, oxidative stress, stimulation of the renin-angiotensin system activation, the inflammatory response, and more.^27–30

Patients in this study received treatment with hypoglycemic, and anti-inflammatory drugs, as well as traditional Chinese medicine and other medications. The results do not entirely rule out the potential effects of some of these drugs on renal function. However, the predictive model was constructed using data from the initial laboratory study at admission, a point at which most patients had not yet started taking medication. Therefore, the impact of drugs on renal function can be effectively disregarded.

The model’s performance was evaluated through ROC curve analysis and DCA, focusing on discrimination, calibration, and clinical application value. The model demonstrated good results across all parameters. Therefore, we believe that this model can be used in routine clinical practice to assist doctors in identifying renal dysfunction in T2D patients. The significance of zinc in preventing and slowing the progression of DKD has been highly regarded in experimental studies. This underscores the importance of paying attention to this trace element and understanding how it can protect against kidney damage caused by diabetes.

Zinc has a negative correlation with the risk of DKD, and is considered a vital factor in reducing the incidence of diabetic kidney damage. This can be attributed to its ability to alleviate oxidative stress and inflammation.^31,32

Studies have confirmed that Zn plays a protective role in DKD by mediating inflammation, regulating insulin receptors, addressing glycoxidative stress, countering pancreatic β-cell impairment, promoting healthy lipid profiles, and activating important proteins related to cell signaling, which contribute to maintaining glucose homeostasis.^33–35 We observed a notable reduction in serum Zn concentration in DKD compared to T2D subjects (p = 0.056), which is consistent with previous studies.^36,37 Zn appears to have a potential role in assessing glycemic control and managing renal impairment in the course of diabetes. Zn antagonizes diabetes-related risk factors and complications, such as insulin accumulation in the pancreas, improving blood glucose control, and stabilizing insulin hexamers.³⁸

The study showed that abnormal Zn metabolism may be associated with diabetes complications.³⁹ Achieving and maintaining appropriate serum Zn levels may be a valuable goal for alleviating symptoms related to T2D complications and slowing the progression of renal damage in patients at high risk of end-stage renal disease. In summary, analyzing the beneficial effects of Zn in our study will contribute to a better understanding of potential treatment strategies for diabetes and DKD.

According to the ROC analysis, the AUC for the training set was 0.960 (95% CI: 0.936–0.984), and for the validation set, it was 0.9316 (95% CI: 0.8747–0.9885), respectively. When comparing the model’s results with the five features screened with the highest feature importance in the RF model, it shows good results across all parameters. Therefore, this model can be used in daily clinical practice to assist doctors judge renal function damage in DKD patients.

This study still has some limitations. First of all, the statistical significance of the data may be affected by the relatively small sample size of the patients included in this study. Second, the data used in this study was derived from the initial laboratory examination conducted after each patient’s admission to the hospital, rather than from laboratory results at the onset of symptoms. Obtaining data at the onset of symptoms can be challenging unless patients seek immediate medical attention after symptom onset. Finally, although this study suggests that the diagnostic model could serve as a potential indicator for predicting and preventing the progression of DKD, it remains unclear what the appropriate cutoff values are and the prognostic value of the indicators included in this model. Further longitudinal studies may provide additional clarification in the future.

Conclusions

In conclusion, the online DKD risk prediction model constructed using the RF algorithm was selected based on its strong performance in the internal validation. In contrast to previous risk prediction models, our model emphasized the significance of the novel indicator GA. The RF algorithm exhibited a high level of accuracy in predicting the progression of kidney disease, surpassing the predictive ability of the major risk factors associated with the occurrence of DKD. Therefore, it may serve as a valuable alternative or complementary tool for enhancing the diagnosis of DKD. To further validate the clinical applicability of our model, large-scale studies involving diverse patient cohorts across multiple sites should be conducted in the future.