Comparative Analysis of Logistic Regression, Gradient Boosted Trees, SVM, and Random Forest Algorithms for Prediction of Acute Kidney Injury Requiring Dialysis After Cardiac Surgery

Introduction

In recent years, the field of medical research has seen exponential growth in the use of machine-learning algorithms to improve clinical decision-making and patient outcomes.^1–3 Amongst the countless applications, predicting postoperative complications after cardiac surgery has emerged as a crucial area of research. Acute kidney injury (AKI) is a common and serious complication that has been shown to have a significant impact on patient morbidity, mortality, and healthcare costs. In particular, cases requiring dialysis due to AKI represent even greater challenges and require rapid and accurate prognostic tools to identify patients at risk.

Generally, clinical risk assessment scores are used to assess the likelihood of postoperative complications, including AKI.^4,5 Many scoring systems fall short of accurately predicting outcomes when dealing with intricate interactions among multiple clinical variables. This is where machine learning algorithms can be utilized to their fullest potential, as they possess the ability to detect intricate patterns in vast data sets, resulting in more accurate predictions. Recent studies have highlighted the efficacy of machine learning approaches in predicting postoperative complications and outcomes. For instance, Zhang et al developed a machine learning model that exhibited superior performance compared to traditional risk scores in forecasting postoperative acute kidney injury.⁶ Similarly, Smith et al applied machine learning techniques to enhance the prediction of various postoperative complications, demonstrating improved accuracy and reliability.⁷ Furthermore, Qing Qian et al demonstrating the potential of machine learning approaches in improving risk prediction compared to traditional scoring systems.⁸

This study deals with a comprehensive comparative analysis of four well-known machine learning algorithms, namely Logistic Regression, Gradient Boosted Trees, Support Vector Machine, and Random Forest. Each of the algorithms used for this study represents a unique approach to predictive modelling. Logistic Regression is a simple yet interpretable algorithm well-suited for binary classification tasks, while Gradient Boosted Trees iteratively build decision trees, adeptly handling complex non-linear relationships and performing notably well in medical applications. Support Vector Machines, on the other hand, excel at finding the optimal class-separating hyperplane, thus performing robustly on complex datasets with numerous features. Lastly, Random Forests construct an ensemble of decision trees and combine their predictions, effectively reducing overfitting while enhancing generalization performance.

While several studies have explored machine learning techniques for predicting postoperative complications, including AKI, our research offers a comprehensive comparative analysis of four distinct algorithms: Logistic Regression, Gradient Boosted Trees, Support Vector Machine, and Random Forest. This multi-algorithm approach allows for a thorough evaluation of their respective strengths and weaknesses in the context of AKI prediction after cardiac surgery.

Xue et al studied a machine learning algorithm to obtain better predictive power for Cardiac surgery-associated AKI outcomes.⁹ This study compared four machine learning algorithms (random forest (RF), logistic regression with LASSO regularization, extreme gradient boosting (Xgboost), and support vector machine (SVM)) for the entire dataset to identify the most important clinical variables for AKI prediction outcomes. Similarly, Jiang et al evaluated whether machine learning algorithms could significantly improve the risk prediction of postoperative AKI.¹⁰ This study utilized conventional logistic regression (LR) alongside five ML algorithms (decision tree, random forest, gradient boosting classifier (GBC), Gaussian Naive Bayes and multilayer perceptron).

Even though previous studies show it was conducted with a broader range of algorithms, our study focuses specifically on the Malaysian population, providing insights into the applicability and performance of these models in a different context. Socioeconomic status and the prevalence of comorbidities may be different from other studies that may play the factors that could influence the effectiveness of AKI prediction models in the Malaysian population.

This research study aims to determine the best-performing algorithm for predicting AKI that requires dialysis after cardiac surgery. By leveraging diverse patient characteristics including demographic, clinical, and laboratory variables, we intend to develop reliable predictive models that can aid physicians in identifying high-risk patients and providing timely interventions. Furthermore, this study provides valuable insight into the advantages and disadvantages of each technique in the context of AKI prediction. The ultimate objective is to contribute to the growing body of knowledge at the intersection of machine learning and healthcare, allowing for more accurate risk assessment and enhanced patient care in cardiac surgery.

Materials and Methods

Data

This study was based on “Validating Cleveland Clinic Score to Predict Acute Kidney Injury Requiring Dialysis After Cardiac Surgery” which included all data collection (at a national level) of patients who underwent cardiac surgery at a tertiary cardiothoracic center in Malaysia between 2011 and 2015. The approach of this data collection process was described in a prior paper and published elsewhere.¹¹ The electronic health records (EHR), which held detailed patient information, including clinical, surgical, and laboratory data, served as the primary data source for this investigation. The dataset comprised 1741 patients, of which 179 patients were diagnosed with AKI. The Cleveland Clinic Score, a commonly used predictive model for AKI requiring dialysis, served as the foundation for the variables included in this study. The variables were gender, insulin use, history of chronic obstructive pulmonary disease (COPD), prior surgeries, left ventricular ejection fraction (LVEF), intra-aortic balloon pump (IABP) use, congestive heart failure (CHF) diagnosis, type of surgery, emergency status, creatinine levels, and the need for renal replacement therapy as the primary endpoint. Additionally, other variables such as age, hypertension status, discharge status, and length of stay were incorporated due to their potential significance in predicting the outcome.

Preprocessing

The dataset underwent meticulous preprocessing to ensure quality and consistency. Patients with insufficient or missing data were excluded from the analysis. We applied SMOTE (Synthetic Minority Over-sampling Technique) in RapidMiner to handle class imbalance in our dataset. This technique involves increasing the number of instances in the minority class by duplicating or generating synthetic samples. Furthermore, it can improve the model’s ability to recognize patterns in the underrepresented class, leading to better overall classification performance and a more robust and fair model evaluation.

Feature Selection

Feature selection was applied to identify the most relevant predictors for predicting AKI requiring dialysis. The primary focus was on assessing data quality, with a critical emphasis on removing predictors that contributed little to the analysis. Hence, in our study, the predictors were identified based on their significant contributions during univariate analysis and by detecting discernible patterns within the data. Predictors of interest included those closely related to the target column (Correlation), those with a high degree of uniqueness (ID-ness), those exhibiting stability (Stability), those with missing values (Missing), and those potentially containing free-text information (Text-ness). Generally, predictors with low Missing, Stability, and ID-ness values were favored, with some cases warranting the retention of text columns.

Analysis

In order to depict various methods of predictive modelling, the four algorithms under consideration; Logistic Regression, Gradient Boosted Trees, Support Vector Machine, and Random Forest were picked. As a linear technique for binary classification, Logistic Regression served as the foundational model. Logistic Regression has been developed for binary classification tasks, making it an important choice because of its simplicity and interpretability. Meanwhile, Gradient Boosted Trees were chosen because of their resistance to complicated interactions and capacity for dealing with non-linearity. This ensemble method iteratively builds successive decision trees, with each tree correcting the shortcomings of the previous one, known for handling complex non-linear relationships, excelling in capturing intricate patterns within data, and achieving particularly notable success in medical applications.

Support Vector Machine was chosen as an example of kernel-based techniques due to its ability to recognize high-dimensional patterns. It is a powerful classification algorithm that emphasizes finding the most efficient class-separator hyperplane and helps to improve performance on complex datasets with large number of features. Finally, random forest is a cluster learning method that builds multiple decision trees and combines their results for improved accuracy and generalization. By using decision trees and pooling their predictions, random forests effectively reduce overfitting, increasing the ability of the model to accurately predict the unobserved data.

The dataset was divided into training and validation sets, ensuring an 80:20 split, to facilitate impartial model evaluation. To improve model performance, the hyperparameters of each algorithm were automatically tuned by RapidMiner’s AutoModel feature during the model training process. This automated approach effectively searches the hyperparameter space, leveraging sophisticated optimization techniques to identify the optimal or near-optimal hyperparameter configurations. These configurations are specifically tailored to maximize the performance metrics of the respective machine learning models. The models were subsequently developed using the training set, and the validation set was used to assess them using the proper metrics, including accuracy, AUC, precision, F-measure, sensitivity, and specificity.

RapidMiner, a powerful data science platform, was used to implement our machine learning workflows.¹² Custom processes were created to handle data preprocessing, feature selection, and model training. RapidMiner’s built-in operators were leveraged for algorithm implementation. Figure 1 provides a comprehensive overview of the key stages involved, namely data preprocessing, model development, and model evaluation, all of which were carried out using the RapidMiner software.

Figure 1 Overview of data preprocessing, model development and model evaluation using RapidMiner. Abbreviation: ML, machine learning.

Ethics

This study was conducted in accordance with the ethical principles outlined in the Declaration of Helsinki. This study did not require ethical approval as it is an article discussion from a methodological perspective, utilizing secondary data. The requirement for informed consent was waived since all identifiable data was removed, ensuring anonymity and confidentiality. However, this article has received approval for publication from the Director General, Ministry of Health, Malaysia.

Results

In this analysis, we began with a dataset comprising 1741 patients, of which only 179 patients were diagnosed with AKI. This presented an inherent class imbalance issue, as the AKI cases were significantly underrepresented compared to non-AKI cases. To mitigate this imbalance and ensure our dataset’s suitability for modeling, we performed a resampling technique known as upsampling. This technique involved increasing the number of AKI patients to 1250 while keeping the number of non-AKI cases at 1250, resulting in a balanced representation of both classes.

Based on univariate statistical analyses, we initially identified a set of variables that showed significant associations with AKI, particularly with emphasis on p-values. The selected variables included age, insulin use, hypertension status, LVEF, IABP use, type of surgery, emergency status, creatinine levels, discharge status, and length of stay. We performed an additional selection step as part of our model refining approach to optimize model stability and performance by detecting discernible patterns within the data as mentioned in the methodology section. As a result, we omitted two variables, IABP and emergency status, because their stability metrics exceeded 90%.

We constructed an analysis of four machine learning models using Logistic Regression, Random Forest, Gradient Boosted Trees, and Support Vector Machine for predicting AKI requiring dialysis after cardiac surgery. The final model incorporated a comprehensive set of variables, including age, insulin use, hypertension status, left ventricular ejection fraction (LVEF), type of surgery, creatinine levels, discharge status, and length of hospital stay to the final model that consist variable age, insulin use, hypertension status, LVEF, type of surgery, creatinine levels, discharge status, and length of stay. Table 1 summarizes the performance metrics achieved by each algorithm in the validation dataset. The algorithms were evaluated using various performance metrics including accuracy, AUC, precision, F-measure, sensitivity, and specificity.

Table 1 Performance of Four Different Machine Learning Algorithms

The results in Figure 2 show variations in the performance of the various models. Gradient Boosted Trees achieved the best accuracy (88.66%) and AUC (94.61%), indicating its effectiveness at predicting AKI that requires dialysis. It also had the highest sensitivity (91.30%), indicating its ability to appropriately identify patients at risk. Random Forest exhibited high AUC (94.78%) and precision (88.73%) values.

Figure 2 Machine learning performance. Abbreviation: AUC, area under the curve.

In contrast, the Support Vector Machine showed relatively lower accuracy (79.02%) and precision (70.81%), as well as a higher sensitivity (98.57%) but lower specificity (59.55%). The overall performance of Logistic Regression was consistent, with balanced sensitivity (87.70%) and specificity (87.05%).

In addition to the performance metrics, we analyzed each model’s Receiver Operating Characteristic (ROC) curves to assess their discriminative power. The ROC curves graphically illustrate in Figure 3, the trade-off between sensitivity and specificity at various decision thresholds.

Figure 3 ROC curve for estimating the discrimination power between the machine learning models.

From the ROC curve comparisons, it is evident that Gradient Boosted Trees and Random Forest outperformed the other models, showing a higher AUC value, which indicates their superior ability to discriminate between patients at risk of AKI requiring dialysis and those who are not. Logistic Regression demonstrated a balanced ROC curve, while Random Forest exhibited relatively lower discriminative power. These results indicate that Gradient Boosted Trees and Random Forest may be promising models for predicting AKI requiring dialysis after cardiac surgery.

Discussion

The results from the different machine learning models, such as Logistic Regression, Random Forest, Gradient Boosted Trees, and Support Vector Machine, offer important insights into how well these models perform at identifying patients who are at risk of AKI and need dialysis after cardiac surgery.

The accuracy rates of 87.39% for both the Logistic Regression and Random Forest models show that they are both capable of classifying instances accurately. Random Forest surpassed Logistic Regression in terms of AUC with an AUC of 94.78%, indicating that it is more effective at differentiating between positive and negative situations. Gradient Boosted Trees, on the other hand, had a competitive AUC of 94.61% and the method with the best accuracy, at 88.66%.

In comparison to the other models, Random Forest showed the best precision (88.73%) and a lower false positive rate. The precision ratings of all models are nevertheless reasonably near to one another, showing a reasonable trade-off between recall and precision. Gradient Boosted Trees scored the highest overall balance between precision and recall, with a score of 88.93%, according to the F-measure, which combines precision and recall into a single metric.

Support Vector Machine showed the highest sensitivity (recall) at 98.57%, indicating that it successfully identified true positive situations with a low false negative rate. However, it’s essential to consider the specificity as well. The Support Vector Machine had the highest false positive rate with the lowest specificity (59.55%). This shows that the Support Vector Machine might have a propensity to categorize some negative cases as positive.

Gradient Boosted Trees outperform the other models in terms of performance, with strong F-measure, balanced precision, and high sensitivity. Random Forest was a competitive option for this classification assignment because it had the greatest AUC and also did well in terms of precision and F-measure. Its dependability for this work was demonstrated by the constant performance of the Logistic Regression model across a range of parameters. Support Vector Machine performed exceptionally well in terms of sensitivity but had shortcomings in terms of specificity, which could be problematic in situations where false positives are expensive.

Our results are consistent with earlier studies that have shown the capability of machine learning algorithms in foretelling AKI following cardiac surgery.^13–15 However, by highlighting how the choice of the most suitable model depends on the precise goals and limitations of the classification problem, our analysis adds a fresh dimension. Gradient Boosted Trees or Random Forest stands out as strong contenders when looking for a balance between sensitivity and precision. These models fared better than the more conventional Logistic Regression, showing that cutting-edge machine-learning techniques can greatly improve forecast accuracy. However, Logistic Regression continues to be a solid option for people who value simplicity and interpretability.

Despite the fact that our study provided insightful findings, it is important to recognize its shortcomings. The generalizability of our findings to larger patient populations may be constrained because our analysis was based on a retrospective dataset from a single cardiac surgery unit. The dataset also included a few missing values and other source-related biases. Future studies should investigate techniques to deal with data restrictions as well as the validation of these models in various clinical situations.

Future studies in this field might concentrate on improving the predictive models by including more clinical factors, like biomarkers or genetic data, to increase prediction precision. It is necessary to conduct prospective studies in clinical settings to evaluate the models’ practicality and validate their effectiveness. Additionally, investigating interpretable machine-learning methods can help to improve our comprehension of the causes of AKI following heart surgery.

Conclusion

In conclusion, the precise goals of the classification task and the respective weights of precision, recall, and other performance indicators should serve as the basis for choosing the best machine-learning model. It could be necessary to conduct further validation and fine-tuning studies to guarantee reliable model performance in real-world scenarios.