Validation and Improvement of the Saga Fall Risk Model: A Multicenter Retrospective Observational Study

Introduction

Falls in hospitals and long-term care facilities have become a significant issue in healthcare and caregiving,^1–3 making fall prevention highly desirable. Multiple fall prediction models have been developed to prevent falls.^4–6 These models have often been designed for specific settings, including community-dwelling individuals (outpatients),⁷ older inpatients,⁸ acute care hospitals,⁹ tertiary hospitals,¹⁰ rehabilitation wards,^11,12 patients with liver cirrhosis,¹³ patients with chronic obstructive pulmonary disease,¹⁴ and long-term care facilities.^15,16 These prediction models are typically intended for specific populations and settings. Therefore, a single fall prediction model that can target a wide range of settings would be highly convenient and valuable for healthcare and caregiving.

Although numerous fall prediction models have been developed, many facilities in Japan use assessment tools developed without established evidence.¹⁷ Therefore, using data from adult inpatients aged ≥20 years in a single acute care hospital in a regional city in our previous study, we developed a fall prediction model, the Saga Fall Risk Model 2 (SFRM2), which can be easily applied upon admission.¹⁸

The SFRM2 comprises eight easily assessable items at admission (age, sex, emergency admission, admission to neurosurgery, use of hypnotics, history of falls, independence of eating, and Bedriddenness ranks). Notably, it includes Bedriddenness rank, a widely used official activity of daily living (ADL) scale in healthcare and long-term care in Japan.^19,20 The Bedriddenness rank can be assessed in four steps, enabling a rapid evaluation. It demonstrates excellent inter-rater reliability and criterion-related validity, making it a convenient and objective ADL scale.^21,22 In a prospective validation study conducted at two acute and chronic care hospitals, the overall area under the curve (AUC) of SFRM2 was 0.793. However, it varied between hospitals, with 0.822 in the acute care hospital and 0.642 in the chronic care hospital, indicating differences in the SFRM2 accuracy between hospitals.²³ Therefore, this study aimed to externally validate SFRM2 using data from various hospitals with diverse backgrounds, demonstrate its accuracy, and adjust the coefficients of the prediction model to improve its precision.

Materials and Methods

Study Design, Setting, and Participants

This retrospective observational study was conducted in multiple regions of Japan in chronic care, acute care, and tertiary hospitals. This study included eight hospitals, the characteristics of which are shown in Table S1. The study included all patients aged ≥20 years admitted from April 1, 2018, to March 31, 2021.

Data and Definitions

All data, including admission and discharge dates; age; sex; admitting department; emergency admission; Bedriddenness ranks and Cognitive function scores by the Ministry of Health, Labour, and Welfare; and ADL score, including independence of eating, prescription and name of hypnotic medications upon admission, history of falls, presence of post-stroke sequelae (I60.0–I64, I69), Parkinson’s syndrome (G20, G21), use of mobility aids, visual impairment, transportation by ambulance, referral letter, surgery operations during hospitalization, rehabilitation, and in-hospital falls, were extracted from the medical charts or health records of each hospital. The Bedriddenness rank, a public ADL scale developed by the Ministry of Health, Labour, and Welfare in Japan, has five major classes that are further divided into nine detailed categories. The Cognitive function score was classified into six major classes and subdivided into eight detailed categories.^19,20 In this study, Bedriddenness ranks were categorized into five major classes (normal, J: independence/autonomy, A: housebound, B: chair-bound, and C: bed-bound), and Cognitive function scores were classified into six major classes (normal, 1, 2, 3, 4, and M). Hypnotic medications were defined as benzodiazepines and non-benzodiazepines, excluding melatonin receptor agonists and orexin receptor antagonists.¹⁸ In-hospital fall was the outcome of this study, which was defined as any unexpected fall from any height or position, regardless of injury, including falls from stairs, chairs, and beds that occurred during walking, while sitting, or when in a supine position. Fall-related data were collected from incident or accident reports documenting fall-related incidents. In hospitals where information on previous falls was unavailable, patients with a history of femoral neck fracture were defined as having a history of falls.^24,25 Missing data included in the analysis were treated as independent categories.

Statistical Analysis

For all patients and the two groups of patients who experienced or did not experience falls during hospitalization, continuous and categorical variables are presented as median values (interquartile range) and absolute numbers (percentages), respectively. The fall prediction model used in this study was the SFRM2 logistic regression model developed previously. The formula for SFRM2 is as follows:

Formula of Model 2 (SFRM2)

Using the eight items of the SFRM2, a logistic regression analysis using the forced entry method was conducted with in-hospital falls as the outcome, encompassing the entire study population. Additionally, individual scores for SFRM2 were calculated for each patient, and the AUC of SFRM2, 95% confidence interval (95% CI), and shrinkage coefficient were determined.

Furthermore, we performed random sampling from the entire study population at a 2:1 ratio, designating two-thirds as the test set for model readjustment and one-third as the validation set for the readjusted model. The outcome in the test set was in-hospital falls, and logistic regression analysis with a forced entry method was conducted for the eight variables of the SFRM2. A new model formula, SFRM2.1, was developed based on the obtained regression coefficients. Using SFRM2.1, we calculated the AUC and 95% CI for both the test and validation sets. For the SFRM2 scores of the entire population and the SFRM2.1 scores of the validation set, we computed fall probabilities, sensitivity, specificity, positive predictive value (PPV), and negative predictive value (NPV) for various cutoff values derived using a sensitivity of 90%, the Youden index, or a specificity of 90%. We conducted subgroup analyses, distinguishing patients aged ≥65 years from other groups and calculated the AUC for SFRM2 and SFRM2.1. All analyses were conducted using SPSS Statistics version 25 by IBM, with a significance level set at p <0.05.

Sample Size

We determined the sample size of 1,327 patients based on the effect size of 0.20 (predicted AUC of 0.70, null hypothesis AUC of 0.50), a fall rate of 3.5%, alpha error of 0.05, and beta error of 0.20, as estimated by the AUC of 0.793.²³

Ethical Considerations

This study conformed to the Declaration of Helsinki and the ethical guidelines for medical and health research involving human subjects issued by the Ministry of Health, Labour, and Welfare and the Ministry of Education, Culture, Sports, Science, and Technology in Japan. This study was approved by the Saga University Clinical Research Review Board (no. 2021–07-R-07). The study was registered with the University Hospital Medical Information Network (UMIN) at www.umin.ac.jp (ID: UMIN000045420). Consent was obtained from all patients using the hospital’s comprehensive agreement method, and their anonymity was protected. We disclosed the research information on the hospital’s website and allowed patients to opt out of the study.

Results

Patients’ Background and Incidence of Fall Events

During the study period, 162,177 patients were admitted. We excluded 11,899 patients aged <20 years and 25,757 patients with data entry errors, including missing data related to emergency admissions, admitting departments, history of falls, Bedriddenness ranks, and information on falls, resulting in an analytical cohort of 124,521 patients (Figure 1). Table 1 shows the patients’ characteristics. Among these patients, 2,986 experienced falls, accounting for an incidence rate of 2.4%. The median age (interquartile range) was 71 years (range, 59–79 years), with 66,283 (53.2%) men, and the median length of hospital stay (interquartile range) was 9 days (range, 4–17 days). The incidence rate of falls was calculated to be 1.71 per 1,000 patient-days.

Table 1 Characteristics of Patients

Figure 1 Data flow diagram.

Multivariable Analysis

Multivariable logistic regression analysis showed a significant relationship between falls and the eight items comprising the SFRM2, ie, age, men, emergency admission, admission to neurosurgery, use of hypnotics, history of falls, requiring eating assistance, and Bedriddenness ranks (Table 2). Notably, the odds ratios were 2.1 (95% CI 1.95–2.35, p < 0.001) for having a history of falls, 2.6 (95% CI 1.86–3.58, p < 0.001) for Bedriddenness rank of A, and 3.2 (95% CI 2.31–4.49, p < 0.001) for Bedriddenness rank of B.

Table 2 Results of Multivariate Logistic Regression Analysis

Performance of Predictive Models

The AUC of SFRM2 as a measure of predictive model performance was 0.687 (95% CI, 0.678–0.697) (Figure 2). The observed incidence of falls was consistent with the predicted incidence calculated using the predictive model, with a shrinkage coefficient of 0.996 (Figure 3). The sensitivity, specificity, PPV, and NPV of SFRM2 are shown in Table 3. The cutoff values for achieving a sensitivity of 90%, the Youden index, and a specificity of 90% were − 3.20, − 2.85, and − 2.16, respectively. The corresponding PPV and NPV values for each cut-off point were 3.0% and 99.1%, 3.9% and 98.8%, and 6.3% and 98.1%, respectively. Additionally, we assessed the validation results using cutoff values from a previous study (Table 3).

Table 3 Validation of the Predictive Model with the Cutoff Points Determined in the Present and Previous Studies

Figure 2 Receiver operating characteristic curves and areas under the curves.

Figure 3 The predicted and observed rates of falls in 10 groups divided into 10 deciles by score using the predictive model.

Development and Validation of the New Model SFRM2.1

The test set comprised 81,790 patients, of which 1,944 (2.4%) experienced falls (Figure 4). The median age (interquartile range) of the patients was 71 (range, 59–79) years, and 53.3% were men. Similarly, the validation set included 42,731 patients, of which 1,042 (2.4%) patients experienced falls, and the median age (interquartile range) was 71 (range, 59–79), with 53.0% being men (Table S2). Using data from the test set, we conducted a logistic regression analysis using a forced-entry method with the eight factors that comprise SFRM2 (Table 4). Based on the obtained regression coefficients, we developed a new model, SFRM2.1 (Supplementary Material 1). The AUC of SFRM2.1 in the test set was 0.733 (95% CI: 0.723–0.743) and 0.745 (95% CI: 0.731–0.758) in the validation set (Figure 5). In the validation set, the cutoff values for achieving a sensitivity of 90%, the Youden index, and a specificity of 90% were determined to be − 4.35, − 4.02, and − 2.98, respectively. The corresponding PPV and NPV values for each cutoff point were 3.7% and 99.4%, 4.4% and 99.2%, and 7.6% and 98.2%, respectively (Table 5).

Table 4 Results of Multivariate Logistic Regression Analysis in the Test Set

Table 5 Validation of the Predictive Model (SFRM2.1) with the Cutoff Points Determined in the Validation Set and Previous Study

Figure 4 The data flow diagram of the test and validation sets.

Figure 5 The area under the curves of Saga Fall Risk Model 2.1 in the test and validation sets.

Subgroup Analyses

The AUC for SFRM2 in subgroups aged <65 years and ≥65 years were 0.707 (0.685–0.729) and 0.651 (0.640–0.662), respectively. For SFRM2.1, the AUC values were 0.782 (0.749–0.815) and 0.705 (0.688–0.721) for the respective subgroups (Figures 6 and 7).

Figure 6 Receiver operating characteristic curves and areas under the curves (AUC) for the groups aged <65 years and ≥65 years for SFRM2.

Figure 7 Receiver operating characteristic curves and areas under the curves (AUC) for the groups aged <65 years and ≥65 years in the validation set for SFRM2.1.

Discussion

The SFRM2, comprising eight factors including Bedriddenness ranks, demonstrated good discrimination and calibration even in the external validation using data from a diverse set of eight hospitals. Although SFRM2 proved to be valuable in a heterogeneous population encompassing both acute and chronic care settings, its AUC in previous studies (0.787–0.793)^18,23 and that reported by Hendrich et al (0.71–0.80)²⁶ were somewhat higher. However, SFRM2.1, which was created by adjusting the model coefficients, showed a higher AUC in the internal validation.

The AUC of SFRM2 was 0.687 for the entire study population, which was lower than that reported previously. This discrepancy in AUC is attributable to substantial variations in patient backgrounds, as this study included patients not only from mid-sized acute care hospitals, such as hospitals where SFRM was initially developed but also from tertiary, chronic care, and general hospitals with several specialized departments. One notable factor contributing to the variance in AUC is the Bedriddenness rank, which is included in the SFRM. In our previous study, the proportion of patients with normal Bedriddenness ranks was 48%,¹⁸ whereas in the present study, it significantly decreased to 3.9%. Conversely, the proportion of patients with a Bedriddenness rank of J increased from 9.9% in the previous study¹⁸ to 46.9% in this study. These differences indicate substantial disparities in patient backgrounds between the previous and current studies. ADL can fluctuate greatly depending on a patient’s clinical condition after admission.^27,28 Additionally, poor initial ADL have been associated with adverse outcomes, such as in-hospital mortality, delirium, and nosocomial infections,²⁹ implying that patients with initially poor ADL may experience more significant changes in ADL during their hospital stay. As the SFRM is a prediction model for in-hospital falls based solely on admission data, the higher prevalence of patients with initially poor ADL and their subsequent ADL fluctuations in this population may have led to the decreased predictive accuracy for in-hospital falls compared to that in the previous study.

An increased AUC was observed in the internal validation of SFRM2.1, in which the same eight items as those in SFRM2 were used with adjusted coefficients. We hypothesized that the differences in population characteristics in this study contributed to the decrease in the AUC of SFRM2. This study included patients from acute care, chronic care, and tertiary hospitals, thereby representing diverse healthcare settings. Despite the heterogeneity of this patient population, the AUC in the internal validation was improved by simply adjusting the model coefficients without altering the evaluation parameters. Furthermore, subgroup analyses of the patients aged ≥65 years demonstrated comparable results. Although external validation was not conducted, this suggests the potential for enhancing predictive accuracy by repeatedly adjusting the model coefficients while keeping the evaluation parameters constant when analyzing new data, even in heterogeneous populations. In other words, while maintaining the simplicity of the SFRM evaluation, a more generalized fall prediction model can be developed by utilizing larger datasets and fine-tuning the coefficients without changing the evaluation criteria. Therefore, we advocate for the optimal utilization of SFRM2.1 across diverse clinical settings.

History of falls and Bedriddenness ranks consistently showed significant associations with in-hospital falls.^18,23,30,31 Furthermore, in the multivariate analysis of the present study, odds ratios higher than 2.0 were observed for both history of falls and Bedriddenness ranks A and B, suggesting their potential significance as predictive factors for falls. Data on fall history collected from each hospital included items with vague definitions of the location and timing of falls, and we were unable to establish a clear definition. However, similar results were obtained in previous studies that limited the timeframe for falls to a certain period before admission,^23,32,33 suggesting that further clarification of the definition may enhance the accuracy of predicting falls. Moreover, Bedriddenness ranks are a unique public ADL scale widely used in Japanese healthcare and long-term care settings. However, its association with falls has not been examined extensively. Because poor ADL is related to falls.^34,35 the Bedriddenness rank is a relevant factor for predicting falls. The well-known Barthel Index, a widely used ADL assessment tool, is time-consuming,³⁶ whereas the Bedriddenness rank is a convenient ADL scale encompassing various activities, including independence of transferring and eating.^21,22 Therefore, the Bedriddenness rank, which exhibited a strong association with falls in this study, can potentially serve as a valuable predictive factor for falls. Furthermore, this study showed a significant association between falls and male sex. Similarly, a previous study involving older inpatients in acute care hospitals in Japan found a significant association between falls and male sex.³⁷ However, findings across studies have been inconsistent, with some studies reporting no sex differences and others indicating a significant association between falls and female sex.³⁸ The relationship between falls and sex remains inconclusive, highlighting the need for further research.

Limitations

There are several limitations in this study. First, this was a retrospective study. There is a possibility of inadequate data accuracy, even if patients were independent in ADL before admission, which may have been classified as having Bedriddenness ranks other than normal, such as Bedriddenness ranks J or A, owing to the need for monitoring. Additionally, because data from multiple facilities were used, the data quality may have been heterogenous across different hospitals. Furthermore, the excluded data of 25,757 patients with missing information on items included in the SFRM2, such as the Bedriddenness rank, may have potentially impacted the results, as a significant proportion of these individuals may have had normal Bedriddenness ranks. Finally, interventions for fall prevention were not assessed.

Conclusion

The SFRM2, that includes the history of falls and the Bedriddenness rank, which are particularly useful in predicting falls, showed good ability to predict falls in the validation in diverse populations with significant background variations. Furthermore, it is possible to improve the accuracy by adjusting the coefficients while keeping the model parameters fixed. Validation with big data encompassing more diverse populations is necessary to further generalize the model.