Quality Assessment of Systematic Reviews Performed As An Undergraduate thesis in A Peruvian University: A Systematic Review

Introduction

Systematic reviews (SR) consist of straightforward and schematic summaries of information available oriented towards answering specific clinical questions.¹ They represent high level of evidence due to the multiple sources of information included. Additionally, systematic reviews count on transparent and comprehensible descriptions from the drafting process to critically collect, select, evaluate, and summarize all available evidence regarding new treatment, diagnosis, and prognosis.¹ For this, they use explicit and systematic methods selected to minimize bias in order to provide more reliable results from where to extract conclusions and make critical clinical decisions.² At the Ricardo Palma University of Lima-Peru, medical students, within the educational curriculum, take a course called “Thesis Preparation Workshop” during their fifth year of medical school, intended mainly for planning and monitoring the Development of their Thesis Project. In the last year of the undergraduate degree, students receive personalized advice for the execution and delivery of their thesis.

The quality is evaluated through A Measurement Tool to Assess Systematic Reviews (AMSTAR) tool, which was developed to evaluate SR of randomized trials, which allows for a more detailed evaluation of SR.^2,3 Although a more extensive validation has not been carried out for other study designs, this instrument follows critical steps in comprehensively undertaking SR.

These days, in our faculty, we are witnessing a notable increase in the production of systemic reviews submitted as undergraduate degree theses. However, it is worth noting that despite their undeniable potential, they still face a moderate reception in the academic field when presented for this purpose.⁴ It is crucial to comprehensively evaluate these systematic reviews to obtain a situation analysis that leads to future improvements in our research processes using this methodology. Accordingly, our study aimed to evaluate the quality of systematic reviews submitted as undergraduate thesis at the Medical School of Ricardo Palma University.

Methods

General Design

We conducted a systematic review following the criteria established in the PRISMA (Preferred Reporting Items for Systematic Reviews and Meta-Analysis) statement. Additionally, the protocol was registered in PROSPERO (CRD42023435277).

Eligibility Criteria

We included undergraduate theses written as systemic reviews from the Medical School at Ricardo Palma University. There was no language restriction, and clinical experiments, observational studies, and dissertations were excluded.

Information Sources

A comprehensive systematic search was carried out, which began in the Institutional Repository of Ricardo Palma University and the National Theses Registry (RENATI) of Peru, to find submitted theses. Subsequently, using the keywords derived from the theses found, we searched its published versions in known databases such as EMBASE, MEDLINE, SCOPUS, and Scielo, from its inception to present day (July 10, 2023).

Search Strategy

For the search we used the following terms: Systematic Review and Medical School in the Institutional Repository and RENATI. The details are presented in (Annex 1).

Selection of Studies

Two researchers (ALCM and AACA) independently reviewed and selected the studies. Those duplicated or presented insufficient data for the qualitative synthesis were excluded. If there was any discrepancy between the authors (ACLM and AACA), a third researcher (RPR) made the final decision.

Data Extraction Process

Two researchers (CGR and ERT) independently carried out the data review and extraction of the studies. In case there was any discrepancy among them, we counted on the assistance of a third author (RPR) to make the final decision. The extracted data was registered in a base designed based on the objectives, including information such as primary author, year of publication, search period, total number of studies and patients included, population characteristics, and instrument used to evaluate the quality of the studies selected.

Quality Assessment

To evaluate the quality of the systematic review, we used the instrument AMSTAR-2. (Annex 2) For the systematic review of observational studies, we used AMSTAR-2 modified according to that reported by Santos-Marques et al.² AMSTAR-2 gives us the following categories as a result:

• High: No or one non-critical weakness. The SR provides an accurate and comprehensive summary of the results of the available studies.
• Moderate: No critical weakness and more than one non-critical weakness. The SR has weaknesses but no critical flaws and may provide an accurate summary of the results of the available studies.
• Low: Up to one critical flaw, with or without non-critical weaknesses. The SR may not provide an accurate and comprehensive summary of the available studies.
• Critically Low: More than one critical flaw with or without non-critical weaknesses. The SR is not reliable.

Quality was reevaluated after its scientific publication.

We chose this instrument given that it allows us to evaluate Quality objectively in comparison to other instruments, reducing subjectivity of the evaluation (Joanna Briggs, CAPS).

Synthesis of the Results

Given the review’s focus, we opted to exclusively conduct a qualitative information synthesis.

Results

Selection of Studies

We found a total of 1488 theses. We finally included 11 theses available^5–15 (Figure 1).

Figure 1 Flow chart.

Characteristics of Studies

Three SRs only included clinical trials, and eight were observational studies.^5–15 (Table 1)

Table 1 General Characteristics of the Studies Included in the Review

The systematic reviews with meta-analysis of randomized clinical trials of observational studies included in the review were published between the years 2020 and 2023, representing the latest theses from graduates of the Medical School from Ricardo Palma University.^5–15 Only four theses did not present a registration in PROSPERO.^6–8,14 The years that had the most publications were 2022 and 2023.

The reviews included a median of six studies, and clinical trials included a total of 18 studies.^7,9,10 Most studies included over 1000 patients, only one had a total of 420 participants,⁷ and another review did not report the number of participants.⁸ Only two reviews used the GRADE (Grade of Recommendation, Assessment, Development, and Evaluation) methodology to evaluate and stratify the evidence quality (Table 1).

Only two SRs included a meta-analysis.^7,15 Seven theses were published in a scientific journal, of these two are published in a local journal.

Quality Assessment

Only Two systematic reviews^7,15 presented low quality. On the other hand, the remaining studies presented critically low quality.^5,6,8–14 (Table 2)

Table 2 Results of the Quality Evaluation of the Instruments AMSTAR II and Modified AMSTAR II for Observational Studies

Clinical Trials

All the reviews of clinical trials had a negative response on items 3, 11, 14 and 16.^7,9,10 Only items 1 and 2 had an affirmative response in all reviews.^5–15 Most items had at least two negative responses in three systematic reviews of clinical trials. Only items 7 and 9 had an affirmative response in at least one systematic review.^5–15

Observational Studies

In the systematic review of observational studies, only items 5, 6, 9, and 11 had affirmative responses in all the reviews.^5–15 Items 3, 12, and 13 had negative responses in all the reviews, and items 4 and 8 had a partial yes in all the reviews.^5–15 Most reviews had at least one negative response in items 1, 2, 7, 10, 14, 15 and 16.

Reevaluation of the Quality of Theses Prior to the Scientific Publication

We did not observe modifications in any critical component (item 2,4,7,9,11,13,15) of AMSTAR-2 during the reevaluation of the systematic reviews following its publication as a scientific article, therefore there were no changes in the pre-publication and post-publication AMSTAR 2 scores. We only noted text language, structure, and style modifications.

Discussion

With the growing number of theses produced as systematic reviews in the Medical School of Ricardo Palma University, it is essential to evaluate the quality of its production and report positive feedback for students and professors, in order to establish a situational analysis that allows future improvement of the production of these with this methodology. According to our results, 90.9% of theses included presented critically low quality. This differs from the results of Santos-Marques et al,² where the quality of systematic reviews of COVID-19 was critically low in only 27% of the studies evaluated. Likewise, in a study carried out in Korea, 41% of the systematic reviews evaluated were of low quality. Since our last search, no studies have reported the quality of systematic reviews presented as undergraduate theses, thus presenting the first evidence in our field.¹⁶

It is essential to mention that during our study, we observed that the AMSTAR-2 instrument had very restrictive characteristics in its different items and little flexibility for different situations at the time of evaluation of systematic reviews that were already previously reported by other authors,¹⁷ which would explain the low and critically low quality of the theses evaluated. However, the tool allows us an objective assessment, in comparison to other instruments. Additionally, this instrument did not present a valid version for systematic reviews of observational studies, only counting on the adjustments made to the instrument, limiting the possibility of a more comprehensive evaluation of these types of studies.¹⁸ Taking item 4 (“Did the review authors use a comprehensive literature search strategy?”) as an example, it is scarcely applicable to observational studies. To obtain a “yes”, one should have conducted a search in clinical trial registry databases. However, for observational studies, this requirement would be nonsensical. In a systematic review of this type of study, the highest attainable rating for this critical item would be a ‘partial Yes’, requiring modification if we aim to conduct a comprehensive quality review.¹⁸ Thus, we would be reducing the quality of all systematic reviews of observational studies to low or critically low since it compromises a critical item, which is unacceptable since this is only due to a limitation in the instrument’s instructions.¹⁸

The most important strength was the ready availability of the full-text undergraduate theses in the institutional portals of the university and the national registry of theses. On the other hand, among the main limitations, our study did not count on a valid instrument to evaluate the quality of systematic reviews of observational studies and had little flexibility when evaluating the systematic reviews of clinical trials. Likewise, another relevant limitation was the low quantity of theses produced as systematic reviews. We recommend reevaluating the instrument and proposing validation with new items and instructions for systematic reviews of observational studies and clinical trials. It is essential to replicate this study in the future to reevaluate the quality of theses with a new instrument. Although students take thesis preparation courses, there is no deeper focus on learning how to prepare systematic reviews; however, training in systematic reviews is currently being implemented for students and helping increase the rigor of evaluation by the advisor professors at the time of its evaluation.

Conclusions

The systematic reviews presented as undergraduate theses in Medical School of Ricardo Palma University demonstrated low and critically low quality. It is imperative to reevaluate the evaluation instruments for these reviews, considering the inclusion of new items and instructions, culminating in a validation process. We recommend focusing its efforts on strengthening the training of its students and professors in systematic reviews and intensifying the rigor in evaluating these projects.