Consistency of CSCO AI with Multidisciplinary Clinical Decision-Making Teams in Breast Cancer: A Retrospective Study

Introduction

Breast cancer stands as the most prevalent malignancy afflicting women globally. According to the latest data from GLOBOCAN2020, compiled by the International Center for Research on Cancer, breast cancer has now surged ahead of lung cancer to claim the top spot in global cancer incidence for 2020, registering approximately 2.3 million new cases. It also ranks as the fifth leading cause of cancer-related deaths worldwide.¹ In fact, breast cancer’s incidence ranks first in 159 countries, with mortality rates topping the charts in 110 nations. Notably, China contributes 420,000 new breast cancer cases, constituting 9.1% of all new cancer cases.² Thus, enhancing breast cancer diagnosis, treatment, and patient survival has become an urgent mission for Chinese experts and scholars.

Medical advancements have diversified breast cancer treatment options. A combination of interventions encompassing surgery, chemotherapy, radiotherapy, targeted therapy, endocrine therapy, and immunotherapy has notably enhanced prognosis for breast cancer patients.³ The health and quality of life of breast cancer survivors are also receiving increasing attention.^4,5 Given this diversity, single approaches struggle to cater to the complex needs of patients. Collaborative efforts across multiple disciplines prove pivotal in synthesizing diverse treatment perspectives, yielding standardized treatment plans. The “single disease, multi-discipline” diagnostic and treatment approach enhances scientific decision-making and underscores the importance of Multidisciplinary Teams (MDTs) in the evolving landscape of breast cancer treatment.⁶

Clinical guidelines, being standards formulated by leading experts grounded in evidence-based medical data, offer clinicians a benchmark for standardized treatment choices. However, China’s vast geographical expanse results in divergent medical resources, leading to disparities in breast cancer diagnosis and treatment standards. In this era of rapid information updates, clinicians have access to guidance from sources like National Comprehensive Cancer Network (NCCN)⁷ and Committee of the Chinese Society of Clinical Oncology (CSCO)⁸ guidelines, yet the diversity in breast cancer treatment compounds the challenge of standardization. The heavy workload on clinicians, the proliferation of innovative anti-tumor drugs, and the annual update of clinical guidelines all contribute to the difficulty frontline doctors face in keeping pace with the latest treatment norms. This challenge is even more pronounced in grassroots hospitals, where timely access to high-level diagnosis and treatment insights remains elusive.

The integration of evidence-based medicine’s latest evidence into clinical practice, bridging regional disparities, and ensuring equitable access to optimal care are pressing concerns. Here, the rapid progress of artificial intelligence (AI) and its medical applications offer a ray of hope. Clinical decision support systems (CDSS), like Watson for Oncology (WFO), have emerged as tools to tackle these issues.⁹ CDSS is a computer program that collects and represents knowledge in a way, which allows a computer to simulate human reasoning in order to generate recommendations.¹⁰ AI systems in healthcare typically focus on acquiring knowledge from unstructured data, such as text or large structured data sets. After storing, indexing, and organizing this knowledge, they apply it to specific cases using computational reasoning methods, generate and evaluate hypotheses, and in turn provide an assessment for clinicians.¹¹

WFO, an AI-powered tumor treatment decision system, delivers recommendations aligned with MDT perspectives based on patient data.¹² Yet, its effective implementation in China necessitates localization and clinical research evaluation, considering factors like regional differences in guidelines and medication access. Presently, WFO’s availability in mainland China is limited due to its collaboration with MD Anderson Center.

The Chinese Society of Clinical Oncology Artificial Intelligence System (CSCO AI) is grounded in the Breast Cancer CSCO. This clinical decision support system leverages Chinese breast cancer data, guidelines, and expert insights to offer doctors reference-worthy treatment plans.¹³ Drawing from NCCN guidelines and the St.Gallen Consensus, CSCO issues guidelines tailored to China’s specific context, ensuring standardized diagnosis and treatment approaches.

CSCO AI’s launch has led to its adoption across nearly 29 provinces, over 300 hospitals, and by more than 5000 clinicians in China, positively impacting over 100,000 patients.¹⁴ Yet, the clinical application of CSCO AI remains relatively unexplored. There exists a dearth of evidence-based medical data assessing CSCO AI’s clinical application and its influence on decision-making. Given this context, our study seeks to compare the adherence rate of CSCO AI’s recommended protocol against the MDT team’s protocol at the Second Affiliated Hospital of Anhui Medical University. Our goal is to evaluate CSCO AI’s normative efficacy and the specific influence of the clinical practice model built on CSCO AI on clinical decision-making.

Method

Ethical Approval

This retrospective comparative, single-center, non-interventional clinical study focuses on breast cancer patients treated at the Second Affiliated Hospital of Anhui Medical University between January 2017 and December 2022. Eligible patients were women with unilateral breast cancer, including ductal, lobular, or mixed invasive breast cancer, age≥18 years, and I to III stages. The exclusion criteria were as follows: (1) incomplete clinical data; (2) men; (3) carcinoma in situ; (4) stage IV breast cancer; (5) cases of specific breast cancer variants; (6) recurrence after initial treatment;(7) age younger than 18 years or older than 89 years; and (8)participation in any clinical trial. The medical records of 537 patients were meticulously retrieved from the hospital’s clinical records system by trained breast surgeons X.W. and L.Y. Treatment approaches for clinical use were also discerned. Figure 1 illustrates the study’s flowchart. Informed consent was obtained from all participants, and the research received approval from the institutional research ethics committee of the Second Affiliated Hospital of Anhui Medical University (Approval No.YX2023-099). The study was conducted in accordance with the principles of the Declaration of Helsinki.

Figure 1 The flow diagram of the study.

CSCO AI Decision Generation

CSCO AI, founded upon CSCO guidelines and databases, operates as an artificial intelligence clinical decision support system. It furnishes treatment recommendations for medical practitioners based on individual patient conditions. Additionally, CSCO AI provides information pertinent to each distinct protocol, encompassing evidence-based medical data, guidelines, dosages, contraindications, and adverse effects. CSCO AI separates entity relationships through the construction of knowledge graph to form a knowledge network structure similar to human understanding. For complex professional medical experience and knowledge, it is classified and summarized under the guidance of experts to make it structured and systematic, and form a knowledge set of tumor diagnosis and treatment. CSCO AI uses an iterative process for training. First, it is taught to analyze breast cancer training cases to provide evidence-based treatment recommendations. After CSCO AI’s recommendations are reviewed and scored by experts, the feedback is incorporated into the system and used by CSCO AI to refine its analysis process. For text-based data, text analysis is required under the specification of medical standard terminology to obtain all conceptual data sets for tumor diagnosis and treatment. Then the interaction between data and clinical workers is completed through semantic analysis, inference and auxiliary diagnosis and treatment system. During use, CSCO AI can also update the system based on patients’ clinical data.

The treatment recommendations are categorized as either “recommended” or “not recommended”. Patient condition data from clinical records were input into the CSCO AI system by two trained senior oncology researchers J.L. and L.Z. to generate AI-based suggestions. Neither researcher possessed information about the patient’s clinical treatment or the advice offered by CSCO AI.

Determination of CSCO AI Consistency

Clinical cases from 2017 to 2022 underwent analysis through CSCO AI. These cases were then retrospectively compared to the treatment options recommended by CSCO AI and the actual treatment choices. Consistency was gauged based on our hospital’s treatment protocols and the classifications outlined in the CSCO AI treatment recommendation catalog. An agreement was considered “consistent” if the actual treatment plan matched CSCO AI’s recommendation. Any discrepancy was deemed “inconsistent”. Notably, instances arose where the actual clinical protocols were not present in the CSCO AI treatment database; these were also labeled as “inconsistent”. Cases exhibiting inconsistency in the initial analysis were subjected to a second assessment by the MDT team at the Second Affiliated Hospital of Anhui Medical University. This aimed to gauge the level of agreement between treatment suggestions from different sources. The second analysis was conducted blindly, with the MDT being unaware of both the actual patient treatment options and CSCO AI’s recommendations. It’s important to mention that cases deemed consistent in the initial analysis were not re-evaluated in the second round due to MDT’s existing heavy commitments.

Statistical Analysis

Patient characteristics, including age, menstrual status, surgical procedures, axillary lymph node surgery, TNM stage, and molecular classification, were scrutinized. Data processing and analysis were carried out using SPSS 26.0 software. Statistics was performed by W.X. and C.X. Single-factor analysis employed the chi-square test, while multivariate analysis employed logistic regression. A significance level of P<0.05 indicated statistical significance.

Result

Baseline Feature

A total of 537 eligible breast cancer patients were included in this study. Supplementary Table 1 presents the clinicopathological attributes of the entire breast cancer patient cohort treated at our hospital from January 2017 to December 2022. The average age at treatment initiation was 51.6±9.6 years. Variation was observed in molecular typing among the subjects, with Luminal B comprising the largest subgroup (54.7%). In the overall sample, a majority of patients fell within the early-stage categories, with stage I and stage II constituting 42.3% and 52.9% respectively. Overall side-effects during and after oncological therapies were shown in Supplementary Table 2.

Consistency Analysis

In comparing the actual treatment choices for breast cancer patients to CSCO AI recommendations, an overall treatment agreement rate of 80.4% was noted. Subgroup analyses were conducted for treatment conformity based on stage and receptor status (Table 1). Subpopulation analysis concerning TNM staging and molecular subtypes revealed considerable consistency. Regarding TNM stage, patients with stages I and III exhibited significantly higher agreement rates compared to stage II patients, and this difference was statistically significant (85.02% vs 76.06%, 88.46% vs 76.06%, P=0.023). Consistency exhibited variations across different tumor types. Specifically, the concordance for Invasive Ductal and Lobular Carcinoma (88.46%) surpassed that of Invasive Ductal Carcinoma and Infiltrating Lobular Carcinoma (85.02% and 76.06% respectively). Furthermore, the guideline adherence rates varied by molecular subtype among breast cancer patients. Triple-Negative Breast Cancer (TNBC) demonstrated the highest compliance rate (87.50%), followed by Luminal A type (88.66%) and HER2 Over-expression type (85.25%) breast cancers. Luminal B (HER2 Positive) and Luminal B (HER2 Negative) exhibited comparatively lower rates (74.36% and 76.92% respectively) (Figure 2). This difference was statistically significant (P=0.009). No substantial variations were observed in guideline compliance rates based on different ages, ECOG (Eastern Cooperative Oncology Group) status, or menstrual status. Similarly, the method of breast surgery and axillary surgery had negligible impact on the agreement rate.

Table 1 The Concordance Between CSCO AI and Clinical Treatment Plan

Figure 2 (A)The consistency of tumor type between CSCO AI decisions and actual treatment options in breast cancer; (B)The consistency of TNM stage between CSCO AI decisions and actual treatment options in breast cancer; (C)The consistency of molecular subtype between CSCO AI decisions and actual treatment options in breast cancer.

When comparing MDT-recommended treatment options with CSCO AI decisions, an overall agreement rate of 92.4% was observed (Table 2). Notably, given that this study employed the latest 2021 version of CSCO guidelines, the MDT’s recommendations were reassessed for cases from 2019 and earlier to ascertain any modifications in light of updated medical knowledge and guidelines.

Table 2 The Concordance After the Initial Analysis and Second Review

Consistency Analysis of Logistic Regression

Logistic regression analysis was used to explore the independent risk factors affecting the guideline compliance rate (Table 3). Univariate analysis showed that ECOG and molecular subtype were independent risk factors for consistency in breast cancer patients (Figure 3A). The multivariate analysis unveiled statistically significant distinctions in age, menstrual status and molecular subtype consistency among breast cancer patients (Figure 3B).

Table 3 Logistic Regression Model of Concordance Between CSCO AI and Clinical Treatment Plan

Figure 3 Logistic regression analysis is used to explore independent risk factors that affect guideline compliance rate; (A) Univariate analysis of independent risk factors for consistency of breast cancer patients; (B) Multivariate analysis of independent risk factors for consistency of breast cancer patients.

Discussion

Based on the 2020 Global Cancer burden data unveiled by the World Health Organization’s International Agency for Research on Cancer, cancer retains its status as the primary global disease burden.¹⁵ Despite the presence of guidelines such as NCCN, CSCO, and others to address the treatment of malignant tumors, achieving standardization in treatment remains a challenge. The propagation of the latest treatment options continues to be complex due to the substantial clinical workload faced by medical practitioners, the proliferation of groundbreaking anti-tumor medications, and the swift updates to clinical protocols. Notably, artificial intelligence has recently exhibited remarkable potential in clinical contexts, particularly in aiding treatment decisions.¹³ Within this landscape, CSCO AI emerges as a pivotal algorithm rooted in Chinese patient data.¹⁴ Utilizing the foundations of CSCO guidelines and databases, the resultant artificial intelligence clinical decision support system, powered by CSCO AI, furnishes doctors with treatment recommendations tailored to individual patient circumstances.

Breast cancer exhibits distinct attributes among Chinese women compared to their Western counterparts. Firstly, the average age of breast cancer diagnosis in Chinese women ranges from 45 to 55 years.¹⁶ Analyzing GLOBOCAN 2020 data stratified by age, breast cancer’s age-standardized incidence and mortality top the list among all cancers affecting women aged 30 to 54 years.¹⁵ This scenario prompts Chinese experts to place heightened emphasis on the long-term quality of life for breast cancer patients and safeguarding fertility in younger patients. Secondly, China grapples with a low rate of early breast cancer detection. Comparative studies between breast cancer characteristics in China and the United States reveal that China experiences a higher prevalence of advanced tumors and a lower estrogen receptor positivity rate.¹² In addition, China has witnessed a steady annual rise in breast cancer incidence. Data reflecting China’s cancer burden underscores an ascending trend in breast cancer incidence and mortality among Chinese women from 2000 to 2015.¹⁷ Factors contributing to this increase could encompass lifestyle and environmental shifts coupled with enhanced early screening initiatives.¹⁸ Hence, a broader inclusion of Asian women in clinical trials, epidemiological studies, and translational research endeavors can illuminate the varied face of breast cancer across races, shedding light on intricate interactions between environmental and inherent factors in its pathogenesis. Such insights could pave the way for enhanced strategies in breast cancer prevention, diagnosis, and management tailored to the tapestry of ethnic diversity.

The study encompassed 537 breast cancer patients from China. Initially, the degree of alignment between CSCO AI recommendations and actual treatment protocols was assessed, revealing an overarching concurrence rate of 80.4%. Upon delving into subgroup analysis, it emerged that the alignment rate was higher for patients in stage I and III in contrast to those in stage II (85.02% and 88.46% vs 76.06%, p < 0.05). Notably, the molecular classification of breast cancer played a role in impacting the compliance rate of CSCO AI. Specifically, the conformity rates were as follows: Luminal A (88.66%), Luminal B (HER2 Positive) (74.36%), Luminal B (HER2 Negative) (76.92%), Triple Negative Breast Cancer (87.5%), and HER2 Over-expression (85.25%). The observed distinctions in these rates were statistically significant. Furthermore, an additional review was conducted to evaluate the agreement between CSCO AI decisions and expert review comments. This second review elevated the overall concordance rate to 92.4%.

Although CSCO AI has certain limitations in the treatment of breast cancer, its advantages and development prospects cannot be ignored. First, clinicians face heavy clinical workloads that limit their time to learn. CSCO AI can help physicians improve the efficiency of learning and the accuracy of clinical decisions. Second, doctors or patients may not be able to choose the most appropriate standardized protocol due to preference. CSCO AI is characterized by objective neutrality and provides a detailed list of treatment options based on evidence. Third, the unbalanced allocation of domestic medical resources makes it difficult for grassroots patients to obtain standardized treatment. The emergence of CSCO AI makes it possible for patients in grassroots hospitals to receive the same standardized and personalized treatment plans as those in first-tier cities. Thus, when doctors, MDT, and CSCO AI coordinate closely, they can compensate for their respective inadequacies to achieve optimal care.

In China, the clinical research on CSCO AI was spearheaded by Professor Jiang Zefei’s team.⁸ Findings disclosed in July 2018 underscored that CSCO AI achieved a 100% recognition rate when applied to the medical records of 200 patients with advanced breast cancer who underwent screening. A subsequent clinical study, concluded in September 2018, scrutinized the degree of alignment between CSCO AI and CSCO Breast Cancer Guidelines Grade I–II recommendations across a cohort of 300 breast cancer patients. These investigations indicated that the overall compliance rate could surpass 95%. Notably, this study marks the inaugural attempt to assess the congruence between the performance of CSCO AI and the tangible treatment regimens administered in clinical settings. However, it’s important to acknowledge that this approach of assessing decision consistency, while commonplace in evaluating CSCO AI’s performance, bears inherent limitations. It fails to address the aspect of whether these decisions conform to standardized practices. As such, it is imperative to undertake additional research that delves into the quality evaluation of treatment decision-making. This study specifically delved into the harmony between CSCO AI’s decision viewpoints and those of experts to gauge the caliber of CSCO AI’s decision-making process.

The study boasts several significant strengths. Firstly, the inclusion of a substantial number of breast cancer cases drawn from authentic clinical scenarios in China contributes to a robust sample size. This enables an in-depth exploration of the specific factors driving shifts in treatment decisions. Secondly, this research is bifurcated into two distinct research phases: the evaluation of decision consistency and the assessment of decision quality. The initial phase scrutinizes the alignment between CSCO AI recommendations and the tangible clinical treatment strategies. Building upon this foundation, a subsequent prospective study is undertaken, centering on an expert secondary review protocol. This dual approach thoroughly examines the harmony with CSCO AI’s suggestions at both the clinical treatment protocol and expert opinion levels. This comprehensive assessment affords a thorough understanding of CSCO AI’s capabilities and underscores its potential value for clinical applications.

Nonetheless, this study does possess certain limitations. Firstly, the findings of this study solely pertain to the alignment between CSCO AI recommendations and physician-driven decision-making. The degree to which clinical decisions are standardized and tailored to the specific circumstances of each patient remains inadequately captured. Secondly, owing to the constraints of the study center, data was exclusively sourced from breast cancer patients within the most recent five years. As the five-year mortality and recurrence rates for breast cancer are generally lower, a comparison of prognoses between the consistent and inconsistent groups was not feasible. Extended follow-up periods are warranted to comprehensively gauge the ramifications of CSCO AI decisions on patient outcomes. Lastly, the focus of this study centers on the stage of decision consistency research, which marks the initial phase of CSCO AI investigation. Ongoing efforts encompass additional clinical studies, spanning patient benefits, the augmentation of CSCO AI’s alignment with physician decision-making, and the safety of clinical deployments. To achieve the objective of accurately assessing the clinical utility of CSCO AI, further research endeavors are indispensable, given the intricate nature of research subjects and goals pertinent to the application of CSCO AI.

Conclusions

In conclusion, treatment recommendations provided by CSCO AI and MDT for breast cancer patients tend to align closely. The consistency is notably affected by molecular typing. Presently, CSCO AI cannot entirely substitute clinicians. However, it serves as a valuable tool to assist doctors in their decision-making processes.