Limited Generalizability of Retrospective Single-Center Cohort Study in Comparison to Multicenter Cohort Study on Prognosis of Hepatocellular Carcinoma

Introduction

Of the various observational study designs, retrospective cohort studies allow relatively quick, cost-effective, and practicable analyses of the associations between multiple exposures and the corresponding outcomes.¹ These outcomes are established on the basis of existing data for a representative patient population under broad inclusion criteria, thereby providing more generalizable results. In the clinical setting, retrospective cohort studies are especially important in hepatocellular carcinoma (HCC), a disease with heterogeneous tumor characteristics and treatment options depending on staging and underlying liver function.^2–4 Due to its nature, randomized controlled trials (RCT) are less likely to reveal the variable clinical course of HCC and may not reflect real-world treatment outcomes. This highlights the need for observational studies that can objectively portray overall survival (OS) in actual clinical practice.⁵ However, retrospective cohort studies have several limitations, of which external validity and selection bias are considered of major concern.⁶

Retrospective cohort studies are often conducted at a multicenter level to overcome this problem,⁷ but the rationale for this approach is mostly based on evidence acquired from RCTs.⁸ In previous RCTs, single-center trials have shown larger intervention effects than multicenter trials,^9,10 or, in other cases, positive results of single-center trials have been contradicted by subsequent multicenter trials.^8,11,12 However, the differences in outcomes seen in RCTs have not yet been demonstrated in retrospective cohort studies despite their frequency and significant role in clinical practice. Due to the differences between RCTs and retrospective cohort studies in study design and patient population,¹³ it is unclear whether retrospective single-center cohort studies have the same drawbacks as single-center RCTs. If retrospective single-center cohort studies were capable of providing comparable results to those of multicenter cohort studies, researchers might be spared the time and effort needed to achieve uniformity of data among different institutions while obtaining a similar degree of external validity. Furthermore, demonstration of similar treatment outcomes between single-center and multicenter cohorts would conceivably enhance acceptance of the quality of single-center research and provide guidance for applying evidence achieved from these studies in clinical practice.

We thus hypothesized that a well-conducted single-center study could fully reflect the heterogeneous disease course and tumor features of HCC and potentially establish survival outcomes comparable to that of a multicenter cohort.

Methods

Study Design and Patient Selection

We conducted a retrospective analysis of de-identified patients newly diagnosed with HCC using data from a nationwide multicenter cohort and a single-center cohort in South Korea between January 2008 and December 2018. Diagnosis of HCC was made histologically or radiologically according to the criteria of the American Association for the Study of Liver Disease (AASLD), European Association for the Study of the Liver (EASL), and the Korean Liver Cancer Association (KLCA).^2–4,14 The Korean Primary Liver Cancer Registry (KPLCR) was selected as the multicenter cohort, and the Asan Medical Center (AMC) HCC registry, developed using the well-established Research Electronic Data Capture (REDCap) Cloud platform at South Korea’s largest cancer institute and hospital (https://eng.amc.seoul.kr), was selected as the single-center cohort.^15,16 The KPLCR is a database containing approximately 15% of the patients newly-diagnosed with HCC registered in the South Korean Central Cancer Registry, from which patients are randomly selected each year using the probability-proportional-to-size method and stratification by region (54 hospitals, with a variety of levels of care).¹⁷ Eligible patients were male and female patients aged 18 years or over, and patients for whom no information was available regarding the number of tumors, treatment modality, and age at diagnosis were excluded. This study was approved by the Institutional Review Board of the Asan Medical Center (IRB no.:2022–1274), which waived a requirement for informed consent owing to the retrospective nature of the study.

Variables

Baseline characteristics of the study population included age, sex, body mass index, underlying hypertension or diabetes mellitus, and presence of viral hepatitis, which was defined as any of the following: positive hepatitis B surface antigen or hepatitis C antibody, positive viral titer, or previous history of antiviral therapy. Baseline liver function was assessed by Child Pugh score and Model for End-stage Liver Disease (MELD) score. Tumors were staged according to the Barcelona Clinic Liver Cancer (BCLC) strategy and the modified Union for International Cancer Control (mUICC) system.^18,19 Index date was set as the date of diagnosis.

Treatment Modalities

Initial treatments used in the two cohorts consisted of the following: surgical resection, liver transplantation, local ablation therapy (LAT), transarterial chemoembolization/radioembolization (TACE/TARE), radiotherapy, systemic therapy, and best supportive care. These treatment modalities were further categorized as curative treatment (surgical resection, liver transplantation, and LAT), non-curative treatment (TACE/TARE, radiotherapy, and systemic therapy), and best supportive care. TARE and radiotherapy were excluded from the treatment options in the subcohort analysis as they are currently not standardized as primary treatment options in the BCLC recommendations. In principle, the medical, surgical, and interventional procedures for HCC carried out by Korean clinicians were based on the Korean Liver Cancer Association’s own practice guidelines internationally recommended for use without modification.^4,20–22

Outcomes

The primary outcome of this study was OS. Death certificate data were accessed from the national statistical data collected by the Ministry of Government Administration and Home Affairs in South Korea, and patients who were recorded as alive without a specified follow-up date were in all cases labelled with the last evaluation date of a patient diagnosed in the same year. OS according to sex, liver function, mUICC staging, and type of initial treatment were obtained, and OS of the entire cohorts were additionally analyzed using propensity score (PS) matching to balance the distribution of confounding variables.

Although the BCLC staging system is designed to guide the choice of treatment for each stage in accordance with AASLD and EASL practice guidelines, primary treatment of HCC in clinical practice varies widely among patients of the same stage due to differences in underlying liver function and tumor features.^5,23 Therefore, patients with preserved liver function (Child-Pugh class A) who received the BCLC-recommended treatment options for each stage (BCLC stage 0 or A, single tumor: surgical resection, BCLC stage A with 3 or less nodules each up to 3 cm: LAT, BCLC stage B: TACE, BCLC stage C: systemic therapy),¹⁸ and patients with any degree of liver function who received a liver transplant according to the Milan criteria were further grouped together for the subcohort analysis. OS of these subcohorts were then compared to evaluate whether there were differences between the two cohorts even in patients treated according to the same criteria.^2,3

Statistical Analysis

With regard to baseline characteristics, differences in the distribution of categorical variables were analyzed by the Chi-square test and differences between continuous variables were analyzed by Student’s t-test or the Wilcoxon rank-sum test. Multivariable Cox proportional hazard models with 95% confidence intervals (CI) were used to assess OS, and survival curves were estimated using the Kaplan-Meier method and Log rank test. Because of the retrospective nature of the study, missing data were handled in one or other of two ways: either by analysis with missing data substituted, using the multiple imputation technique, or analysis with missing data classified as a category. Multiple imputation by Markov Chain Monte Carlo methods was used to fill-out incomplete baseline variables, on the assumption that data were missing at random,²⁴ while interaction analysis was used to evaluate whether the effect of the registry was different within subgroups (sex, liver function, mUICC staging, type of initial treatment). PS matching was performed by matching patients 1:1 using the nearest neighbor method with a 0.05 caliper in order to adjust for differences in baseline variables.²⁵ PS were determined by taking into account the following variables: sex, age, body mass index, Child-Pugh class, BCLC staging, mUICC staging, and type of initial treatment. All statistical analyses were conducted using SAS version 9.4 (SAS Institute Inc, Cary, NC, USA). Two-sided P-values ≤0.05 were considered statistically significant.

Results

Study Population

Between January 2008 and December 2018, a total of 16,781 patients newly diagnosed with HCC were registered in the KPLCR database (multicenter cohort), and 15,707 patients were recorded in the AMC HCC registry (single-center cohort). After applying the exclusion criteria, a total of 32,098 patients (16,443 patients in the multicenter cohort and 15,655 patients in the single-center cohort) were included in the study (Figure 1).

Figure 1 Patient flowchart of the study population. Abbreviations: AMC, Asan Medical Center; HCC, hepatocellular carcinoma; KPLCR, Korean Primary Liver Cancer Registry.

Baseline characteristics of the two cohorts with missing data excluded from analysis are presented in Table 1. The mean ages at diagnosis were 57.7 years (standard deviation [SD], 10.4) and 61.1 years (SD, 11.5) in the single-center and multicenter cohorts, respectively. The single-center cohort had a higher proportion of early-stage patients than the multicenter cohort according to BCLC. Consequently, the use of curative treatment modalities was higher in the single-center cohort, and the use of best supportive care lower. After PS matching, on the other hand, the two cohorts were generally well-balanced. Results with missing data classified as a category are presented in Supplementary Table 1, and the baseline characteristics of the propensity score-matched populations are given in Supplementary Table 2.

Table 1 Baseline Characteristics of the Study Populationsa

Distribution of Liver Function by Initial Treatment

The distribution of liver function according to Child-Pugh class was identified for each initial treatment modality to evaluate differences in distribution between the two cohorts (Supplementary Table 3). There was no significant difference among the patients who received LAT, whereas the multicenter cohort had a significantly higher proportion of patients with Child-Pugh class A liver function than the single-center cohort among those who received liver transplants (44.4% vs 30.0%), radiotherapy (58.2% vs 39.7%), systemic therapy (61.8% vs 55.1%), and best supportive care (40.5% vs 29.6%) (Ps<0.001 for all comparisons).

Survival Outcomes of the Entire Cohorts and PS-Matched Cohorts

The median follow-up duration of single-center and multicenter cohort was 36.2 (interquartile range [IQR]=9.7–66.9) and 30.0 (IQR=6.1–60.0) months, respectively. The single-center cohort had a significantly higher OS than the multicenter cohort, with median survival times of 73.6 (95% CI=69.6–77.5) and 34.0 (95% CI=33.0–35.0) months, respectively (Figure 2, P<0.001 by Log rank test), and this was confirmed by PS matching with adjustment for prognostic variables (Supplementary Figure 1, P<0.001). This finding was also consistent regardless of sex, liver function according to Child-Pugh class, and mUICC staging (Supplementary Figures 2–4, Ps<0.001 for all comparisons). In univariate analysis, the multicenter cohort was associated with a significantly higher risk of mortality compared to the single-center cohort (hazards ratio [HR]=1.55, 95% CI=1.50–1.59, P<0.001). Multivariable analysis also showed significantly higher risk of death in the multicenter cohort after adjustment for cancer variables and patient demographics (adjusted hazards ratio [aHR]=1.16, 95% CI=1.13–1.20, P<0.001) (Table 2).

Table 2 Cox Regression Analysis of Factors Associated with Mortality in the Entire Cohortsa

Figure 2 Kaplan–Meier estimates of overall survival in the two cohorts.

Comparisons of OS in the entire cohorts according to first-line treatment yielded variable results (Table 3 and Supplementary Figures 5 and 6). Multivariable analysis with multiple imputation revealed a higher risk of mortality in the multicenter cohort in patients who received surgical resection (aHR=1.32, 95% CI=1.22–1.44, P<0.001), LAT (aHR=1.50, 95% CI=1.32–1.71, P<0.001), TACE/TARE (aHR=1.24, 95% CI=1.19–1.29, P<0.001), and liver transplantation (aHR=2.10, 95% CI=1.30–3.38, P=0.002). Overall, there was a higher risk of death among patients in the multicenter cohort who received curative treatment (aHR=1.48, 95% CI=1.39–1.59, P<0.001) or non-curative treatment (aHR=1.22, 95% CI=1.17–1.27, P<0.001), and death was significantly lower in patients who received systemic therapy (aHR=0.83, 95% CI=0.74–0.92, P=0.001) and best supportive care (aHR=0.85, 95% CI=0.79–0.91, P<0.001). OS following radiotherapy as an initial option, however, did not differ significantly between the two cohorts (aHR=1.14, 95% CI=0.91–1.42, P=0.25). The results of multivariable analysis with missing data classified as a category are presented in Supplementary Tables 4 and 5, which gave similar outcomes.

Table 3 Cox Regression Analysis of Risk of Mortality by Initial Treatment in the Entire Cohortsa

Subcohort Analysis of Patients Treated According to BCLC Guidelines

Subcohort analysis was conducted to further compare survival outcomes between two subcohorts (n=2797 and n=5151 for multicenter and single-center subsets, respectively) comprised of patients with preserved liver function (Child-Pugh class A) who received treatment according to the BCLC strategy, and patients with any level of liver function who received liver transplants according to the Milan criteria.

OS did not differ between the two subcohorts in patients who received surgical resection (P=0.17 by Log rank test; Figure 3A), liver transplants (P=0.38; Figure 3B), and systemic therapy (median survival time, 5.5 [IQR=5.0–6.1] and 5.1 [IQR=4.5–6.0] months, respectively, P=0.23; Figure 3E). These findings were confirmed in multivariable analysis: risk of mortality among patients with preserved liver function who received surgical resection (aHR=1.07, 95% CI=0.93–1.23, P=0.33) or systemic therapy (aHR=0.94, 95% CI=0.81–1.10, P=0.44) did not differ between the two cohorts, and for patients who received liver transplants within the Milan criteria (aHR=1.30, 95% CI=0.65–2.60, P=0.45) (Table 4).

Table 4 Cox Regression Analysis of Risk of Mortality by Initial Treatment in BCLC-Guided Subcohortsa

Figure 3 Kaplan–Meier estimates of overall survival of patients who received (A) surgical resection, (B) liver transplants, (C) LAT, (D) TACE, and (E) systemic therapy according to the treatment indications*. Abbreviations: BCLC, Barcelona Clinic Liver Cancer; LAT, local ablation therapy; TACE, transarterial chemoembolization. Notes: *Treatment indications: patients of BCLC stage 0 or A, single tumor: surgical resection, BCLC stage A with 3 or less nodules each up to 3 cm: LAT, BCLC stage B: TACE, BCLC stage C:systemic therapy, and patients with any degree of liver function who meet the Milan’s criteria: liver transplantation.

Among patients with preserved liver function who received either LAT (P=0.02 by Log rank test; Figure 3C) or TACE (P<0.001; Figure 3D) in accordance with the BCLC treatment strategy, the multicenter subcohort was associated with a higher risk of death than the single-center subcohort. These differences were also demonstrated in both univariate (HR=1.42, 95% CI=1.07–1.90, P=0.02; and HR=1.74, 95% CI=1.50–2.02, P<0.001, respectively) and multivariable analyses (aHR=1.44, 95% CI=1.08–1.92, P=0.01; and aHR=1.72, 95% CI=1.48–2.00, P<0.001, respectively). Similar outcomes were obtained in multivariable analysis with missing data classified as individual categories (Supplementary Table 6).

Subcohort Analysis of Patients with Child-Pugh Class B Liver Function

In addition to BCLC-guided subcohort analysis, OS according to initial treatment was compared among Child-Pugh class B patients (n=2183 and n=2103 for multicenter and single-center subsets, respectively) and the results are presented in Supplementary Table 7.

While multivariable analysis did not show significant differences in OS between the two subcohorts among patients who received surgical resection (aHR=1.14, 95% CI=0.83–1.55, P=0.42), liver transplantation (aHR=2.03, 95% CI=0.94–4.39, P=0.07), and LAT (aHR=1.26, 95% CI=0.97–1.64, P=0.09), the multicenter cohort patients treated with TACE had a higher risk of death (aHR=1.10, 95% CI=1.01–1.20, P=0.03), and patients who received systemic therapy had a significantly lower risk of death (aHR=0.72, 95% CI=0.60–0.86, P<0.001); this effect resembled the outcomes of systemic therapy in the cohorts as a whole.

Subgroup Analysis

Interaction analysis performed to evaluate the effect of type of registry in the different subgroups showed that the multicenter cohort was associated with a significantly higher risk of mortality in both sexes, for all degrees of liver function, as well as for all stages of the mUICC system (Ps<0.001 for all subgroups) (Table 5 and Supplementary Table 8). In terms of initial treatment modality, subgroups of the multicenter cohort who received curative or non-curative treatment had higher risks of mortality, but OS was higher in the subgroup that received best supportive care (Ps<0.001 for all).

Table 5 Subgroup Analysisa

Discussion

In this outcome-comparison study, we found that the single-center cohort (AMC group) was generally associated with significantly higher OS than the multicenter cohort (KPLCR group); moreover, these results were consistent after PS matching and across treatment modalities except for systemic therapy and best supportive care (Supplementary Table 9).

These findings are noteworthy because, to the best of our knowledge, this is the first study to compare the OS of all-staged HCC patients in a retrospective cohort setting using two large cohorts, comprised of a nationwide multicenter cohort and a single-center cohort. The retrospective design reflects real-life clinical practice in HCC patients with heterogeneous tumor features and variable prognoses, whereas this may be limited in RCTs as they involve highly-selected patient populations enrolled under strict eligibility criteria.⁵ The differences observed between the two cohorts in OS are consistent with the findings of past studies that have compared the treatment outcomes of single-center and multicenter RCTs. These earlier studies showed that single-center RCTs produced larger treatment effects than multicenter RCTs,^9,10,26 and a review article has also highlighted the limited external validity of single-center RCTs by noting many instances in intensive care medicine in which the positive treatment outcomes found in single-center studies were not confirmed in multicenter RCTs.⁸ However, the validity of retrospective studies of single-center cohorts has not been examined despite its clinical significance.

The higher OS observed above for systemic therapy and best supportive care in the multicenter cohort compared to the single-center cohort may be attributed to a center effect: in a previous study, patients who visited tertiary hospitals tended to receive more chemotherapy than patients who visited hospitals of secondary or primary levels.²⁷ In the tertiary hospital chosen as the single center in our investigation, a greater proportion of patients with unpreserved liver function received systemic treatment or best supportive care than in the multicenter series. As the survival of HCC patients is primarily dependent on baseline liver function,⁵ one might anticipate that clinical outcomes would be less favorable in the single-center cohort in patients with on average poorer liver function receiving systemic therapy and best supportive care.

On the other hand, the association of the single-center cohort with better survival outcomes for both surgical and loco-regional treatment modalities is likely to be related to the use of relatively homogeneous indications and the provision of standardized interventions by teams of high expertise in high volume single-centers.^10,26,28 In addition, treatment outcomes obtained at different centers with varying treatment strategies and levels of experience, especially for difficult-to-treat cases, may not directly reflect the setting of any particular center-favorable outcomes in large centers, and therefore may be overshadowed by the inclusion of a number of small volume centers with higher mortality in the multicenter series.^29,30 This may apply especially to HCC, as patients of the same stage can be treated differently due to individual tumor features as well as the variety of available or feasible treatment modalities, specific indications, and levels of skill among the different healthcare centers.⁵

Due to this heterogeneity, we established subcohorts to additionally compare the survival outcomes of treatments administered strictly according to the BCLC algorithm and the Milan criteria. These gave variable results; while there were no differences in OS between the two subcohorts for surgically and systemically-treated patients with favorable liver function as well as transplant patients with any level of liver function, the multicenter cohort was associated with a significantly higher risk of mortality in patients who were locally treated with TACE or LAT as a standard option. The absence of a difference between patients who received liver transplants may be explained by the evidence that postoperative survival is not associated with transplant center volume, but is more likely attributable to other factors including donor age and patient characteristics such as age and MELD score.³¹ Similarly, there was no significant difference in OS following surgical resection among Child-Pugh class A patients, as in studies that found no association between center type or volume and OS after surgical treatment of various cancers.^32–34 Surgical resection in most cases results in complete removal of the neoplasm,³⁵ making it an effective choice of curative treatment in patients who satisfy the indications. Also, advances in surgical techniques and perioperative management may have decreased the gap in treatment outcomes between centers, at least for cases with preserved function.³⁶ Survival outcomes of systemic therapy also did not differ between the two subcohorts with good hepatic function as opposed to other malignancies.^37,38 The lack of difference in survival outcomes for systemic therapy was probably related to the period when the study was performed: until 2018, sorafenib was the only approved treatment option for advanced HCC and it had only a modest survival benefit.³⁹ As numerous anticancer drugs for HCC have been approved since 2018,^40–42 we believe that further studies are required to examine this interpretation.

The survival outcomes of TACE and LAT were, however, significantly different in the Child-Pugh class A subset: the multicenter cohort was associated with a higher risk of mortality than the single-center cohort, similar to the outcomes observed between the entire cohorts. This finding may be attributable to the specialized nature of these modalities and hence the influence that the interventional radiologists’ skill and experience have upon the risk of recurrence as well as the post-procedural morbidity and mortality.^43–45 Previous studies have shown that differences in skill have a greater impact on the efficacy of non-pharmacologic interventions than pharmacologic ones, as the level of expertise of care providers plays a more significant role in the former.^46–48 This may also explain why we detected significant differences in OS between the two cohorts in patients who received TACE or LAT, but not in those who received systemic therapy.

This study has potential limitations, which are mostly inherent in the retrospective nature of the study and the nature of the corresponding data sources. The variables reported, especially in the nationwide data, lacked some details such as family history of cancer, smoking status, and specific grade of performance. Additionally, data on disease recurrence and specific cause of death were unavailable and as a result, the impact of disease recurrence and subsequent treatment on OS could not be assessed. Because recurrence or progression is common in HCC, progression-free survival might provide additional information regarding comparative treatment outcomes.⁴⁹ Completeness of the datasets was another issue, but we treated unavailable data in two ways to deal with that issue. We included the results of analyses performed with missing data classified both as a category and with the missing data substituted by multiple imputation, and we showed that the results obtained with the two methods did not differ significantly. Another possible limitation may be selection bias. The single-center cohort included a significantly higher proportion of early-stage patients according to BCLC staging (BCLC stage 0 or A) than the multicenter cohort. Consequently, the frequency of curative treatment as initial modality was higher, and the frequency of best supportive care lower in the single-center cohort than in the multicenter cohort. However, adjustment for these confounding variables, using both multivariable analysis and PS matching, yielded similar outcomes, which supports the consistency of our study findings. Additionally, while there is an overlap of approximately 14–15% of patients between the KPLCR cohort and the AMC HCC registry, this should not significantly impact our findings as patients in the KPLCR cohort were sampled from 54 diverse hospitals. This sampling strategy employed a probability-proportional-to-size method along with regional stratification, ensuring a balanced representation across different healthcare settings¹⁷ and minimizing the potential bias that might arise from the overlap with the AMC registry. Lastly, the single-center data in our series were recruited from the highest-volume hospital in South Korea, and this could have led to the superior outcomes in terms of several modalities compared to the multicenter data. In general, however, the amount of retrospective HCC data from a low-volume single-center would not be sufficient to provide less bias and adequate statistical power, and so would undermine the purpose of this study.

In conclusion, comparison of OS between the multicenter and single-center cohorts of patients with HCC revealed significant differences according to primary treatment modality possibly due to heterogeneity related to volume-specific center effect and variability in treatment strategies. The prognostic discrepancies between the two retrospective cohorts suggest that retrospective single-center studies should be interpreted with caution, particularly when evaluating HCC treatment outcomes beyond the BCLC criteria, and should involve careful consideration of center volume and patient population. In short, good generalizability of treatment outcomes may still require collaboration between multiple centers.