A Cross-Sectional Study on Inequity and Unmet Needs in Conducting Systematic Reviews (SRMA) and Meta-Analysis Among Medical Students and Junior Doctors

Introduction

With the rapidly evolving knowledge of healthcare practice, there is a need to critically appraise, synthesize and summarize currently available evidence for optimal decision-making and patient care.¹ Evidence-based practices use systematic reviews to seize the best evidence of clinical effectiveness, which is built by analyzing multiple studies meeting the inclusion criteria objectively rather than individual studies.¹

Systematic reviews of randomized controlled trials have been considered top of the evidence-based medicine study hierarchy and may often include meta-analyses. Systematic reviews allow a transparent, rigorous, and replicable analysis of study results, thus reducing bias and providing reliable findings to facilitate decision-making.² Meta-analyses use statistical techniques to summarize the results of multiple related studies.^2,3 The results of a meta-analysis study provide a more exact assessment of the influence of a therapy or risk factor for illness. Moreover, meta-analyses contribute to a pooled analysis of data and are, therefore, a useful tool in evidence-based medicine.⁴ Traditionally, review articles have been written by experts in the fields which appear reasonable and appropriate. Experts often may lack the objectivity desirable in preparing and critiquing a review leading to subjective bias.⁵ Hence, a systematic approach of conducting a review with meta-analysis provides more objective findings.⁵ Systematic reviews and meta-analyses help clinicians remain aligned with the latest evidence, inform routine decision-making in clinical practice, and help formulate clinical practice guidelines and policies.^2,3

The General Medical Council (United Kingdom) states that the learning outcomes for medical graduates should include critically appraising evidence and using research skills to influence patient care.⁶ The research experience of medical students positively impacts the career profile and graduate employability of graduating medical students and inculcates a culture of evidence-based medicine in clinical practice.⁷ Similarly, other national governing bodies such as the Accreditation Council for Graduate Medical Education (ACGME) and the Association of American colleges advocate mandatory research requirements across different stages of their study.^8,9

In contrast, as a medical undergraduate in developing countries, research has been given “poor importance” within the curriculum.¹⁰

If opportunities, supervision, or time to conduct clinical projects are lacking, medical students can seek other research opportunities such as critically appraising literature for review articles while still being immersed in research culture and developing information literacy and critical thinking skills.⁷ This situation has been especially true during the COVID-19 pandemic, when medical students conducting laboratory and clinical-based research, as part of an intercalated Bachelor of Science (BSc), Master of Science (MSc), or Doctor of Philosophy (PhD), had to halt their projects.¹¹ As a result, many research projects have shifted to online formats, and more researchers were required to work rapidly, remotely and responsively.¹²

COVID-19 pandemic has greatly impacted medical education, and many institutes have transitioned from traditional physical face-to-face classes to online mode.¹³ Utilization of webinars in various aspects of medical education (eg, choosing academic medicine programs and specialty training pathways) has been recognized in cross-sectional studies,^14,15 but the usage of e-learning techniques to provide high-quality education needs further rigorous evaluation to identify its benefits and limitations.¹⁶ According to an international report, budgets for education, particularly in poorer nations, are not adjusting proportionately to the problems posed by COVID-19. Despite the need for more investment, two-thirds of low- and lower-middle-income nations have cut their public education spending since the outbreak.¹⁷ The COVID-19 pandemic has shed light on the gaps in internet access, fueling inequality among students’ education worldwide. Only 38.5% of the students in higher educational institutions in Africa have access to online learning during the pandemic.¹⁸ Low- and middle-income countries struggle to provide quality education because students do not have adequate and reliable internet access.^19,20

This unprecedented situation calls for creative solutions and planning to move students’ research forward, with systematic reviews and meta-analyses advocated as potential options.²¹ The aim of this cross-sectional study was to ascertain students’ attitudes on their current knowledge, confidence and preparedness of appraising and conducting systematic reviews.

Materials and Methods

A free webinar was organized in May 2021, which was open internationally to medical students and junior doctors. For our study, junior doctors were defined as doctors practicing for less than 5 years after graduation. The webinar was designed and delivered by the senior author (AK). The webinar covered the basics of the publishing process, tips to improve ones’ curriculum vitae (CV), approaching a mentor for a project, and the various aspects of academic medicine along with personal experiences of the host. The prospects of a career in research were discussed as a routine researcher alongside full-time clinical practice, clinician-scientist, or a 50:50 (academic:clinical practice) split.

We aimed to motivate medical students and junior doctors and enhance their interest in research and quality improvement projects. Ethical approval was not required for this research because this cross-sectional study is an evaluation of a teaching intervention. Participation in the study was voluntary, and written informed consent was gained. The registrants reserved the right to be excluded from the study analysis any time until a week after the webinar date. Furthermore, the registrants were given the opportunity to send in questions regarding the research training and academic medicine which the host answered in a dedicated question and answer component of the webinar. A questionnaire was included in the registration form to ascertain key insights into the knowledge and experience of medical students and junior doctors (Table 1). We used Google Forms (Google, USA) for circulating the online questionnaire, which stores data in the Google Sheets (Google, USA) encrypted database. This questionnaire included 15 questions to ascertain social demographics, knowledge, experience and confidence of medical students and junior doctors in preparing and conducting systematic reviews and meta-analyses.

Table 1 Pre-Webinar Questionnaire

Statistical Analysis

Delegates’ perceptions of the knowledge, experience, and confidence in preparing and conducting a systematic review and meta-analysis were ascertained using a Likert scale (1 = low, 5 = high) in this questionnaire. Associations were examined between knowledge, interest, and confidence in conducting and preparing systematic review and meta-analysis, along with other variables, using Chi-square and crosstabs analysis. Significance was determined at P < 0.05. Statistical analysis was calculated using IBM SPSS Statistics for Windows, V.26 (IBM Corp). The anonymity of the respondents was maintained while doing the analysis.

Results

Demographics

Of the 2106 responses received, 2004 responses were used for analysis (95.2%). This included registrants from 104 countries after removing duplicates, erroneous submissions, and those who did not want to be a part of the study. According to the World Bank country report 2021, countries of current education of registrants were classified economically.²² Majority of the delegates were from lower middle-income countries (LMIC, 1186, 59.2%) followed by high-income (HIC, 385, 19.2%), upper middle-income (UMIC, 375, 18.7%) and low-income economies (LIC, 58, 2.9%). Of the included responses, 57.1% (1145) identified as female, 42.8% (858) as male, and one respondent “preferred not to say” and hence their data were not analyzed when crosstabs analysis was done with gender as one of the variables. The majority of attendees included medical students (1342, 67%) compared to 662 junior doctors (33%).

Questionnaire Responses

Out of the total respondents included in analysis, 54.1% (1084) and 81.1% (920) did not know about the status of systematic reviews (with homogeneity) of randomized controlled trials in “Oxford Centre for Evidence-based Medicine: level of evidence”²³ and about the PRISMA checklist²⁴ for preparing and conducting systematic review and meta-analysis, respectively.

A majority of the included respondents (83%, 1664) had never previously attended any formal training (eg, webinars or courses) on preparing and conducting systematic reviews and meta-analyses. Furthermore, 92.7% (1858) of the delegates either strongly agreed (1495, 74.6%) or agreed (363, 18.1%) with the statement-“Formal training is needed among medical students and junior doctors in medical schools and hospitals to prepare and conduct systematic reviews and meta-analyses”.

Of the included respondents, 91.8% (1839) were either extremely interested (1444, 72.1%) or interested (395, 19.7%) in learning about preparing and conducting systematic reviews and meta-analyses, respectively. However, the majority of the respondents felt extremely unknowledgeable (513, 25.6%) or unknowledgeable (540, 26.9%) about the process of preparing systematic reviews, respectively. Furthermore, 43% (861) of the respondents felt extremely unconfident (427, 21.3%) or unconfident (434, 21.7%) in preparing and conducting systematic reviews and meta-analysis, respectively.

This lack of familiarity and confidence could be explained by the majority respondents being medical students (67%).

The most used resources to learn about preparing and conducting systematic reviews were websites (386, 19.3%) followed by university help (298, 14.9%), published journal articles (230, 11.5%), and friends and colleagues (223, 11.1%). However, 25.8% (518) did not use any tools to learn about preparing and conducting systematic reviews and meta-analysis. Of note, the majority of the attendees (1896, 94.6%) did not have any experience with publishing systematic reviews or meta-analyses. The majority of the delegates (1452, 72.5%) of the respondents felt that their medical school or hospital had given them extremely minimal advice (976, 48.7%) or minimal advice (476, 23.8%) on teaching them about preparing and conducting systematic reviews and meta-analyses. Only 3.8% (76) of the respondents felt that they received extensive advice.

Disparities in Various Cohorts

The proportion of delegates knowing about the PRISMA checklist was significantly higher in HIC and UMIC (203/760, 26.7%) compared to LMIC and LIC (175/1244, 14%) both numerically and in percentage (P<0.001). The proportion of delegates knowing about the position of systematic reviews (with homogeneity) of randomized controlled trials in “Oxford center for evidence-based medicine: level of evidence”, was significantly higher (P<0.001) in HIC and UMIC (458/760, 60.3%) compared to LMIC and LIC (462/1244, 37.1%). Only 12.4% (63/508) of the first- and second-year students had knowledge of the PRISMA checklist, which was significantly inferior to the higher levels of practice, ie, third-, fourth- and fifth-year students and junior doctors (P<0.001) (22.7%, 22%, 19.1% and 20.1% respectively). Moreover, 28.1% (143/508) of the first- and second-year students had knowledge of the status of systematic reviews (with homogeneity) of randomized controlled trials in “Oxford center for evidence-based medicine: level of evidence”, which was also significantly inferior to students of higher levels of practice (P<0.001). Among those who had attended formal training, the proportion was significantly higher in those belonging to HIC and UMIC combined (154/760, 20.3%) than LMIC and LIC combined (186/1244, 15%), respectively (P<0.001). Only 13.6% of the junior doctors reported receiving any formal training.

A significantly higher proportion (P<0.001) of LIC and LMIC respondents felt that their institutes gave them either extremely minimal or minimal advice (924/1244, 74.3%) compared to data from HIC and UMIC respondents combined (528/760, 69.5%).

Discussion

This article demonstrates a clear need to address gaps on educating medical students and junior doctors on performing systematic reviews and meta-analyses. This has been challenging with the COVID-19 pandemic and more evident in countries of lower income.

Since their first application in the 1970s, systematic reviews allow evidence to be thoroughly mapped and reviewed in an unbiased manner in order to review the evidence quality objectively.²⁵ This is highly beneficial for medical students and junior doctors as it removes ethical committee input, which can be daunting and time-consuming for those new to research.²⁶ Similarly, meta-analysis facilitates the ability to incorporate and review critically the results of similar studies to enhance the impact of an intervention.²⁷ Grasping the concepts of these highly beneficial research methodologies enables future doctors to apply their role as a “scholar and scientist” as per the GMC outcomes⁶ and simultaneously enhances communication and teamwork.²⁶ The COVID-19 pandemic highlighted the need to provide innovative ways to continue education through online platforms.¹⁰ This modality can be highly effective to help achieve learning outcomes with appropriate student engagement.²⁸

Despite the advancement in online teaching, students and junior doctors from less-developed countries are at an inherent disadvantage compared to those from developed countries because of limited resources and opportunities with delayed or no access to new software or technologies.²⁸ Key barriers faced with e-learning in developing countries include lack of sufficient quality internet access and technical support.²⁸ Gismilla et al’s study found that 24% of the medical students were reluctant to engage in e-learning due being unaccustomed to the format and 42.4% had concerns about how to prepare for online teaching.²⁸ This highlights the need to further support and enhance online teaching as a valuable educational tool.²⁶ Amgad et al report 50% of the medical students felt restricted to partake in research due to financial circumstances.²⁹ These disparities are clear in our study which further reflect on the inequities and unmet needs of students from poorly developed regions, and more emphasis needs to be put for their upliftment. Formal training methodology shall be focused to provide a more coherent and scientifically backed dissemination of knowledge to conduct systematic reviews, rather than other ways which may not be standardized enough to provide accurate information.

With increased interest in systematic reviews and journals regularly promoting readers about conducting systematic reviews and the need for reviewers,³⁰ it becomes important to teach the upcoming doctors about systematic reviews and meta-analysis. This process will contribute to the development of robust data collection, analysis, and critical appraisal skills. Furthermore, students that got involved in research as a medical student “had positive behavior changes in practice” (62.3%; 95% CI: 46.7–77.9%).³¹

Abushouk et al’s cross-sectional study highlighted students did have a positive mentality towards engaging in research; however, only 23.8% actually engaged in research.³² This was due to reduced mentorship, funding, time and inadequate knowledge.³² Siemens et al’s cross-sectional study found only 25% of the medical students from self-reported surveys felt they had sufficient education on appraising literature critically.³³

Evidence within the literature about medical student’s attitudes towards research are in keeping with our study. Less than 20% of the registrants knew about the PRISMA checklist, and less than half of the respondents knew about the status of systematic reviews (with homogeneity) of randomized controlled trials in “Oxford center for evidence-based medicine: level of evidence”. This is an unmet need of medical students and junior doctors, which needs to be addressed, and medical curriculum assessment may be needed.

Further large-scale studies are needed to analyze the factors leading to more students attending formal training in HIC and UMIC compared to LIC and LMIC combined. According to an international report, in 2018–2019, HIC families spent the equivalent of $8501 per child or youth on education, while LIC families spent an average of $48 annually.¹⁴ A local study in an HIC found that the major challenges to online medical education were related to communication, student assessment, usage of technology tools, online experience, and pandemic-related anxiety or stress.³⁴ In LMIC, the challenges were more often related to lack of logistical knowledge to participate and conduct online classes, lack of conducive home learning environment due to family conflicts, family commitments or learning space constraints, power interruptions, and limited access to devices and internet.^19,20 A major advantage of online mode of study is auto-transcribing and recording feature available in some video-conferencing software which saves time than transcribing notes for the session manually.³⁵ Recordings can also be reviewed at one’s own convenience later on, thereby enhancing flexibility, which in turn can promote asynchronous learning within the curriculum. This aids in instilling a life-long learning approach by promoting self-directed learning.³⁶ More studies on the effect of transition of physical to online classes due to the COVID-19 pandemic on medical education are needed to assess these differences.

The Preferred Reporting Items for Systematic reviews and Meta-Analyses (PRISMA) statement was published in 2009, aimed at facilitating authors’ reporting of systematic reviews and meta-analyses.³⁷ The revised version, PRISMA 2020, provides updated reporting guidelines for systematic reviews, reflecting the advances in methods to identify, select, appraise, and synthesize studies. These statements allow healthcare providers and policymakers to judge the applicability of the findings, enable readers to evaluate the suitability of the methods or intervention, and help decision-makers formulate practices or policies, or guidelines.³⁷ The proportion of respondents from LIC and LMIC having knowledge about the PRISMA checklist was significantly lower than registrants from UMIC and HIC. This difference may highlight the disparity in the level of education in various economic settings, and further work is needed to analyze the deep-rooted impact of having less financial resources as a nation on medical education. E-learning interventions aiming to provide affordable access to high-quality medical education may be used, but more rigorous evaluations are needed for the assessment of their benefits and pitfalls.¹⁶ The fact that first- and second-year students were least aware of the PRISMA checklist and the status of systematic reviews level of evidence hierarchy was somewhat expected as these students have recently started their medical career and may have minimal research-oriented knowledge in these early years of medical education. This is reflected in a large cross-sectional study by Muhandriamge et al who evaluated 704 responses that explored medical student’s attitudes towards research in a developing country, and found that understanding research metrics was greater in older students and those at later stages of medical school.³⁸ The COVID-19 pandemic may have affected their routine medical curriculum. The current usage of technology especially in LMIC needs to be further improved with an emphasis on faculty development and development of students’ skills.³⁹

Since the majority of the respondents either used no additional tools or collected information from non-standardized sources (eg, websites, friends and colleagues) to learn about systematic reviews, this may lead to the spread of misinformation and usage of incorrect techniques because of lack of quality control and protocol implementation. This further stresses the importance of studies needed for the assessment of choices and reasoning behind so few people opting for formal training. The pandemic may further impact equity, diversity, and inclusion with the challenges of amplifying cognitive factors related to implicit bias and lack of opportunities for students from disadvantaged backgrounds.⁴⁰

Strategies to overcome these barriers include making projects more accessible through funding, employing a subscription service to update students on research opportunities, active encouragement through platforms such as an undergraduate research showcase and universal teaching amongst faculty on research methodologies.³³

Internal auditing and further regional studies may be needed to identify the potential pitfalls in the current curriculum to promote research interests as the majority felt that their institutions helped them minimally. Corresponding changes may be needed in the current medical curriculums as more than 90% of the respondents are interested in preparing systematic reviews, but the majority feel less knowledgeable. The pandemic also had economic repercussions with decreased funding of institutions and poor impact on mental health and well-being.⁴⁰ As respondents were majorly medical students rather than junior doctors, the importance of providing and expanding systematic reviews and meta-analyses courses for students in their early medical career is emphasized.

To the authors’ knowledge, despite this study being the largest cross-sectional study related to this topic, further large-scale studies are needed to understand the attitude and rationale of working on research projects and the opportunities available to medical students and junior doctors that may lead to medical curriculum changes. Since participation in this study was entirely voluntary, the number of registrants from various countries and levels of practice were unequally distributed. This limitation may be influenced by more endorsement of the event in authors’ and their colleagues’ native countries. Further large-scale studies are needed especially focusing on LIC as their representation was the least in this study. Since the university name was not collected in the questionnaire, we cannot ascertain whether the disparities in data are specifically a country-wide phenomenon or the scenario is different in different universities (eg, students from high ranked universities having better knowledge than others even in a LIC). Future cross-sectional studies will be performed to include these questions.

Conclusions

This study highlights the inequities and unmet needs of medical students and junior doctors in performing systematic reviews and meta-analyses. With the shift towards online research during the COVID-19 pandemic, the majority of the registrants felt dissatisfaction with the level of advice and help provided by their medical institution. Clear disparities are found in country income and the level of education. Incorporating compulsory research modules within the curriculum could aid in standardizing access.²² Future large-scale studies are needed to understand the rationale of working on online research projects and the opportunities available to medical students and junior doctors that may lead to medical curriculum changes. Well-designed systemic review and meta-analysis courses that are openly available could overcome the gaps in disparity and serve as additional training for interested students.