Comparison of Two Predictive Models of Sepsis in Critically Ill Patients Based on the Combined Use of Inflammatory Markers

Introduction

Sepsis, associated with a dysregulated host response to infection, is an important global health problem and a major cause of death around the world.¹ It is estimated that about 48.9 million patients are diagnosed with sepsis, and 11.0 million patients died due to sepsis every year, accounting for approximately 20% of the global deaths.^2,3 Although mortality in sepsis remains high, it is treatable. Timely targeted interventions, including antibiotics administration, removal of the source of infection, full fluid resuscitation, and other supportive treatments can reduce mortality and improve prognosis.^4–7 Therefore, it is important to identify sepsis early and conduct adequate and timely interventions.

Sepsis is induced by infection, involving complex inflammatory responses during its progress, which is accompanied by pro-inflammatory and anti-inflammatory mechanisms.^8,9 Conventional inflammatory markers, such as procalcitonin (PCT), interleukin-6 (IL-6), white blood cell (WBC), and C-reactive protein (CRP), may be served as potential markers to help clinicians identify sepsis early.^10–14 Neutrophil-to-lymphocyte ratio (NLR), an easily accessible biomarker that can be calculated from components of the differential white cell count (dividing neutrophil by lymphocyte count), has shown good predictive and prognostic performance in various diseases, and some studies have focused on the performance of NLR in the early diagnosis of sepsis.^15–19 However, to date, none of these markers has gained unanimous independent acceptance for identification of sepsis. Although some previous studies have combined different markers to predict sepsis, the combination of WBC, IL-6, PCT, CRP and NLR to develop predictive models of sepsis in critically ill patients has not been explored.

Nomogram model, which can reduce statistical predictive models into a single numerical estimate of the probability of an event, are widely used to predict diagnosis, staging and prognosis in various situations.²⁰ Therefore, we conducted this study to develop and validate a nomogram model and a simple risk scoring model for prediction of sepsis in critically ill patients. Furthermore, we compared the predictive performance of the two models. The aim was to find a model with strong clinical practicability, which can assist clinicians in identifying potential sepsis at an early stage.

Materials and Methods

Data Source and Patient Selection

This retrospective study was conducted in the intensive care unit (ICU) of the First Medical Center of the Chinese People’s Liberation Army General Hospital, a 3000-bed tertiary teaching hospital in Beijing, China. The medical records of adult critically ill patients admitted to ICU from August 2017 to December 2020 were analyzed. We excluded patients who: 1) Were younger than 18 years old; 2) Were pregnant; 3) Discharged early within 48 hours after ICU admission; 4) Were readmitted to the ICU; 5) Were immunosuppressed; 6) Had hematological malignancy; or 7) Were undergoing cardiopulmonary resuscitation before ICU admission.

For this study, sepsis was diagnosed according to the latest Sepsis-3 definitions.¹ All sepsis patients received standard-of-care managements in our ICU following the guidelines of 2016 Surviving Sepsis Campaign.²¹ The study complied with the Declaration of Helsinki and has been approved by the Ethical Committee of our hospital (S2017-054-02). Considering that this was a retrospective observational study and the personal identifying information was not included, the Institutional Review Committee granted a waiver of informed consent for this study.

Data Collection

Variables included demographics (age, gender, body mass index (BMI)), vital signs at ICU admission (temperature, systolic blood pressure (SBP), respiratory rate (RR) and heart rate (HR)), Sequential Organ Failure Assessment (SOFA) scores, comorbidities, and the results of the first laboratory test after ICU admission (blood routine tests, blood biochemical tests, arterial blood gas tests, CRP, PCT, IL-6) were collected from hospital electronic medical system.

Laboratory Tests

Blood samples were obtained on admission to the ICU. NLR and WBC could be obtained from routine blood tests, and the routine blood tests were performed by using an automated hematology analyzer (Sysmex-XN2000, Kobe, Japan). The CRP was examined by using the CRP quantitative analyzer (PA990, Shenzhen, China). The PCT was measured by using the electrochemical luminescence immunoassay system (Roche, cobas 602), and the IL-6 was measured by the Siemens- IMMULT2K.

Statistical Analysis

For the continuous variables, the Kolmogorov–Smirnov test was performed to test the normal distribution. Continuous variables were expressed as median (interquartile range) or mean (standard deviation), as appropriate, and compared by Mann–Whitney U-test and Student t-test for non-normally distributed data and for normally distributed data, respectively. Categorical variables were expressed as frequencies (percentages) and the differences between two groups were compared by using chi-square test.

To examine the association between sepsis and the inflammatory markers (WBC, IL-6, PCT, CRP and NLR), both univariate logistic regression analyses and multivariable logistic regression analyses using forward stepwise regression were performed with results were expressed as odds ratios (ORs) and 95% confidence intervals (CIs). A P-value less than or equal to 0.05 was considered statistically significant.

Development and Validation of the Nomogram

Patients were randomly divided into a training cohort (70%) and a validation cohort (30%). Based on the results of the logistic regression, the training cohort was used to construct a nomogram by the “rms” package of R software. The receiver operating characteristic curves (ROC) were constructed and the area under the curve (AUC) was calculated to evaluate the discrimination performance of the nomogram model. A calibration curve was performed with 1000 bootstrap resampling to compare the predictive probability of the sepsis with the observed incidence of sepsis. The validation cohort was used to assess the generality and stability of the nomogram model. The probability of sepsis was calculated for each patient in the validation cohort according to the established nomogram model. Then, an ROC curve constructed, and finally, the AUC was calculated.

Development and Validation of the Simple Risk Scoring Model

We also used the training cohort (70%) and validation cohort (30%) of the nomogram model to develop and validate a simple risk scoring model for early detection of sepsis. To get a quick score for each variable and finally calculate the total score, all continuous variables included in the model were transformed into dichotomous variables. Considering that either an elevated or a decreased in WBC may indicate infection, it was transformed into a dichotomous variable according to whether its value was within the normal range (4~10*10⁹/L). Other continuous variables (IL-6, CRP, PCT, and NLR) were transformed into dichotomous variables based on their optimal cut-off values for the identification of sepsis alone in the training cohort. For the training cohort, a multivariable logistic regression analysis was performed and the OR for each variable was used to construct the simple risk scoring model.^22,23 Then, the ROC curve was constructed and the AUC was calculated to evaluate the discrimination performance of this scoring model. The optimal cut-off value of the score for predicting sepsis was determined using the Youden index. Finally, this simple risk scoring model was used to calculate risk scores for each patient in the validation cohort, and the AUC was derived based on the risk scores.

The predictive performance between two models and SOFA score were compared by AUC test, which could be achieved by MedCalc (version 19.0.4, Ostend, Belgium). Statistical analyses in this study were performed by SPSS (version 17.0, Chicago, USA), R software (version 4.1.0, Vienna, Austria) and MedCalc (version 19.0.4, Ostend, Belgium).

Results

Patient Selection and Characteristics

During the study period, 2514 patients were admitted in ICU. According to our inclusion and exclusion criteria, 2074 patients were included in study. Of these, 1451 patients were randomly assigned to the training cohort and 623 to the validation cohort (Figure 1). The baseline characteristics of the patients in two cohorts were similar, as shown in Table 1. The incidence of sepsis was 688/1451 (47.4%) in the training cohort and 284/623 (45.6%) in the validation cohort. No significant difference was detected (p=0.444).

Table 1 Baseline Characteristics of the Training Cohort and Validation Cohort

Figure 1 Flowchart of the enrolled patients. According to our inclusion and exclusion criteria, 2074 patients were included in study. Of these, 1451 patients were randomly assigned to the training cohort and 623 to the validation cohort.

Results of Logistic Regression Analyses in the Training Cohort

After univariate logistic and multivariable logistic regression analyses, all of these five markers (WBC, CRP, IL-6, PCT, and NLR) were the independently associated with sepsis. The optimal cut-off values for CRP, IL-6, PCT and NLR identifying sepsis alone were 1.3 mg/dl, 90 pg/mL, 0.5 ng/mL and 15, respectively. Then, they were transformed into dichotomous variables according to the cut-off values. Furthermore, another multivariable logistic regression analysis was conducted based on dichotomous variables. The adjusted ORs for WBC, CRP, IL-6, PCT and NLR were 2.13 (95% CI: 1.63–2.78), 2.32 (95% CI: 1.76–3.05), 2.82 (95% CI: 2.17–3.67), 4.74 (95% CI: 3.52–6.39) and 4.66 (95% CI: 3.56–6.10), respectively (Table 2).

Table 2 Univariate and Multivariable Logistic Regression Analyses of Inflammatory Markers Related to Sepsis in the Training Cohort

Performance of the Nomogram

According to the results of logistic regression analyses, the first laboratory test results of WBC, CRP, IL-6, PCT and NLR after ICU admission were used to construct nomogram (Figure 2). In the training cohort, the nomogram showed a good discrimination for sepsis, with an AUC 0.854 (95% CI: 0.835–0.872). At the optimal cut-off value, the sensitivity and specificity were 82.0% and 73.7%, respectively (Figure 3A). In the validation cohort, the AUC was 0.857 (95% CI: 0.827–0.883). There was no significant difference between two cohorts in terms of the discrimination (P = 0.879; Figure 3A). Furthermore, we compared the discrimination between nomogram and SOFA scores, and nomogram showed a better discrimination (P < 0.0001; Figure 3C). The calibration curve presented a good agreement between the nomogram prediction and actual observed for incidence of sepsis (Figure 4).

Figure 2 Nomogram predicting the probability of sepsis in critically ill patients of training cohort. When using it, drawing a vertical line from each variable to the points axis for the score, then the points for all the inflammatory markers (WBC, CRP, IL-6, PCT and NLR) were added, finally, a line from the total points axis was drawn to correspond the risk of sepsis at the bottom.

Figure 3 Receiver operating characteristic curve analyses of models for predicting sepsis. (A) Nomogram model; (B) the simple risk scoring model; (C) Nomogram model, simple risk scoring model and SOFA scores.

Figure 4 Calibration curves for nomogram model in the training cohort (A) and validation cohort (B). In the calibration curve, the X-axis represents the predicted probability of sepsis, and the Y-axis represents the actual sepsis incidence rate. The 45° diagonal dotted line represents ideal predictions. The solid line represents the performance of the nomogram model, of which a closer fit to the diagonal dotted line represents a better prediction.

Performance of the Simple Risk Scoring Model

Based on the adjusted ORs for WBC, CRP, IL-6, PCT and NLR, we developed a simple risk scoring model, with the total scores ranging from 0 to 16.5 points (Table 2). The model showed good discrimination for sepsis in both the training cohort and the validation cohort, with an AUC 0.842 (95% CI: 0.822–0.861) and 0.847 (95% CI: 0.816–0.874). No significant difference was detected between the two cohorts in terms of the discrimination (p = 0.7941; Figure 3B). In the training cohort, the optimal cut-off value was 7.5 points, with a sensitivity 77.03% and specificity 75.75%. Then, we compared the simple risk scoring model with nomogram and SOFA scores for sepsis discrimination. Results indicated the simple risk scoring model was comparable with the nomogram in the discrimination of sepsis (p = 0.1298; Figure 3C), and the simple risk scoring model performed better than SOFA scores in the discrimination of sepsis (p < 0.0001; Figure 3C).

Discussion

Sepsis is a complex disorder with high morbidity and mortality, and remains a global health priority.²⁴ Early identification of sepsis can facilitate timely clinical interventions and may improve prognosis.²⁵ Our study indicated that inflammatory markers, including WBC, CRP, IL-6, PCT and NLR, were the independently associated with sepsis in critically ill patients admitted to ICU. Furthermore, a nomogram and a simple risk scoring model were constructed. The prediction performance of the two models on sepsis was comparable to each other and provided better prediction than that of SOFA score.

To our knowledge, early detection of sepsis is necessary, so that specific goal-directed therapy bundles to minimize complications may be initiated; hence, many studies have focused on early sepsis detection. Some have used biomarkers, some have built scoring models, and in recent years, an increasing number of studies have focused on machine learning.^22,26,27 Microbial cultures are still the gold standard for diagnosing sepsis, but the results are generally available after several days. In addition, in patients receiving antimicrobial treatment, the results sometimes provide a false negative.

Sepsis is a systemic inflammatory response due to infection, resulting in organ dysfunction. Inflammation plays a crucial role in the process of sepsis. Our study combined inflammatory markers, including WBC, CRP, IL-6, PCT and NLR, to predict sepsis. WBC is the most common inflammatory marker in clinic, and WBC count greater than 12,000 or less than 4000/microliters is one of the criteria for the diagnosis of systemic inflammatory response syndrome (SIRS).²⁸ It is well established that CRP is an acute inflammatory marker that acts as part of the acute-phase reaction in sepsis, and its concentration increases during infection.^29,30 CRP has a half-life of 19 hours. It is produced within 4–6 hours after inflammation onset and peaks at 36–48 hours.^31,32 IL-6 is an important proinflammatory cytokine produced early in inflammation and plays a role in the complex pathophysiology process of sepsis.^33,34 IL-6 levels begin to rise within 1 hour of the infectious stimulus and peaked at 2 hours.³⁵ PCT has attracted much attention due to its high accuracy in sepsis diagnosis and prognosis.³⁶ PCT is secreted by many cell types during systemic inflammation of infectious origin, and the level is generally low in nonbacterial origin systemic inflammation.³⁷ The half-life of PCT is 24 hours. Its level begins to rise within 2–4 hours after systemic inflammation and peaks at 12–48 hours, making it valuable in the early detection of sepsis.³⁸ In our study, the weight of PCT is the greatest among five inflammatory markers in both nomogram model and the simple risk scoring model. In the simple risk scoring model, it would be assigned 4.5 points when greater than the cut-off value. During the progress of sepsis, apoptosis-induced lymphopenia is a prominent feature of sepsis, and neutrophils are recruited to control infection.^39–42 A dramatical increase in NLR can be observed in sepsis. These markers are readily available and routinely measured at admission to our ICU. However, using any of above-mentioned markers alone to identify sepsis may lack sensitivity and specificity. Therefore, we developed and validated a nomogram model including all the above markers to predict sepsis in critically ill patients.

Although some previous studies have constructed nomogram models in predicting sepsis, there has been no study combining the results of WBC, CRP, IL-6, PCT and NLR measured at ICU admission to predict sepsis.^43–46 To our knowledge, these five inflammatory markers are routinely monitored in the ICU for patients with suspected infection and the results can be obtained quickly. Our nomogram model showed good prediction performance as indicated by the AUC value, which is better than that of SOFA score (P < 0.0001). In addition, our nomogram model has fewer items and is more convenient to use compared with the SOFA score. The calibration curves presented a good agreement between the nomogram prediction and actual observed for incidence of sepsis. Nomogram is a visualized, reliable and intuitive graphical tool for predicting and quantifying risk of individual experiencing a clinical event based on relevant factors; however, sometimes bedside applications are not so convenient. Therefore, we developed a simple risk scoring model based on dichotomous variables and its prediction performance was comparable to that of the nomogram, which can somewhat help clinicians quickly determine the risk of sepsis at an early stage.

This is the first study to construct a nomograph predictive model of sepsis in ICU using five commonly available inflammatory markers. Furthermore, a simple risk scoring model with good prediction performance is constructed. This study also has some limitations. First, this is a single-center retrospective study that still requires external validation. The model needs other centers to further evaluate. Secondly, the study mainly focused on ICU patients, which may not be suitable for emergency patients. Thirdly, to make the model simple and practical, we did not include other laboratory tests other than inflammatory markers, which may potentially have reduced the performance of the model.

Conclusion

This study used five commonly available inflammatory markers, including WBC, CRP, IL-6, PCT and NLR, to develop a nomogram model and a simple risk scoring model to predict sepsis in critically ill patients. Although the prediction performance of the two models is comparable, the simple risk scoring model may be more practical for clinicians to identify potential sepsis in critically ill patients at an early stage and strategize treatments.