Back to Journals » Advances in Medical Education and Practice » Volume 9
A national survey: use of the National Board of Medical Examiners® basic science subject exams and Customized Assessment Services exams in US medical schools
Received 23 March 2018
Accepted for publication 17 May 2018
Published 27 August 2018 Volume 2018:9 Pages 599—604
DOI https://doi.org/10.2147/AMEP.S169076
Checked for plagiarism Yes
Review by Single anonymous peer review
Peer reviewer comments 3
Editor who approved publication: Dr Md Anwarul Azim Majumder
William S Wright,1 Kirk Baston2
1Department of Biomedical Sciences, University of South Carolina School of Medicine Greenville, Greenville, SC, USA; 2Department of Pathology, University of South Carolina School of Medicine Greenville, Greenville, SC, USA
Purpose: The National Board of Medical Examiners® (NBME®) offers basic science subject exams and exams developed using the Customized Assessment Services (CAS) program. The scope and utilization of these exams by US medical schools has not been reported. Therefore, this survey aimed to measure US medical school usage of NBME® basic science subject exams and exams administered using the CAS program.
Methods: In May 2016 a survey was sent to 139 US medical schools with provisional or full accreditation by the Liaison Committee on Medical Education®. Schools were asked if they administered NBME® basic science subject exams or exams developed using the NBME® CAS program. Follow-up questions were asked if the school responded “Yes” to administering the exams.
Results: Responses were received from 46–48 schools (33%–35% response rate). Of the schools responding, 7–13 schools (15%–28%) administer an NBME® basic science subject exam and 14 schools (30%) administer an exam developed using the CAS program. Of the schools that administer an NBME® basic science exam, 7–12 (83%–100%) use the exam for summative assessment for a course and 6–11 (77%–100%) use the exam results for curricular revision. Twelve schools (86%) that administer an NBME® exam developed using the CAS program use the exam for summative assessment for a course. Time of administration within the curriculum for NBME® basic science subject exams varied between 1 month prior to >6 months prior to students taking the United States Medical Licensing Examination® Step 1.
Conclusion: According to our data, most US medical schools responding to the survey do not administer NBME® basic science subject exams or exams developed using the NBME® CAS program. Schools that do administer NBME® basic science subject exams use the exams for summative assessment for a course and curricular revision, while exams administered using the CAS program are used predominately for summative assessment for a course.
Keywords: NBME, Customized Assessment Services, subject exams, survey, assessment
Introduction
US medical schools strive to prepare all students to become licensed physicians. Part of the process for becoming a licensed physician is completion of the United States Medical Licensing Examination® (USMLE®) Step 1 examination which measures basic science knowledge.1 Although this single external exam is a top factor for ranking applicants for the National Resident Match Program®,2 each program is unique in its delivery and assessment of basic science content.
As we have previously reported, a National Board of Medical Examiners® (NBME®) comprehensive basic science exam (CBSE) is used by US medical schools to identify students performing below expectations;3 however, the NBME® offers basic science subject exams and clinical science subject exams “for the purpose of assessing the educational achievement of individuals in specific subject areas.”4 The program was developed in the 1960s to measure basic and clinical science.5 In an effort to allow programs to develop exams consistent with changing medical school curricula, the NBME® created the Customized Assessment Services (CAS) program in 2007 which allows faculty to create integrated basic science or clinical science subject exams.5
There are many studies reporting the utility of the NBME® clinical science subject exams in undergraduate medical education.6–11 In addition, Haist et al5 reported that in 2015 “greater than 550,000 assessments were provided through the subject examination program, NBME self-assessment services, and CAS.” It was also reported that the number of exams administered using the NBME® CAS has steadily increased since its inception.5 Although the number of exams provided has been reported, the scope and utilization in US medical schools for NBME® basic science subject exams and exams developed using the NBME® CAS program is limited.
The purpose of this study was to determine the extent to which NBME® basic science subject exams and exams developed using the NBME® CAS are used 1) in preclinical medical education, 2) as part of summative or formative assessments, and 3) for curricular revision.
Methods
Survey methods
Survey methods have previously been described.3 Briefly, 139 Liaison Committee on Medical Education® schools with provisional (4 schools) or full (135 schools) accreditation as of February 29, 2016, were identified. An e-mail message and link to the anonymous survey, hosted at SurveyMonkey®, was sent to the Associate Dean for Curriculum, or equivalent on May 3, 2016. A follow-up reminder was sent 1 week later and the survey ended on May 16, 2016. The aim of the study was presented in the e-mail message.
Ethical approval
A review by the University of South Carolina institutional review board was conducted and it was determined that this was not human subject research.
Results
There were 46–48 responses (33%–35% response rate) regarding the use of NBME® subject exams for basic science content. The first question regarding NBME® subject exams was answered by 48 schools; however, all schools did not answer the remaining questions regarding NBME® subject exams. There were 46 responses (33% response rate) regarding the use of exams developed using the NBME® CAS program.
Data presented cover only schools that administer the NBME® basic science subject exams or exams administered using the CAS program
Table 1 provides the data regarding the number of schools that offer NBME® basic science subject exams. In addition, Table 1 provides the number of schools that use the NBME® basic science subject exams for curriculum revision, formative assessment for a course, and summative assessment for a course. Of the schools that responded, the histology and cell biology subject exam is administered by the lowest number of schools (15%) and the pathology subject exam is administered by the highest number of schools (28%) (Table 1). Of the schools that offer the NBME® basic science-specific subject exams, most use the exams for curricular revision (range 77%–100%) and summative assessments for a course (range 83%–100%).
Most pharmacology, pathology, and behavioral sciences subject exams are administered 3 months or less prior to students taking the USMLE® Step 1 while most biochemistry, gross anatomy and embryology, histology and cell biology, neuroscience, and physiology exams are administered >6 months prior to students taking the USMLE® Step 1 (Table 2).
The number of schools administering exams using the NBME® CAS program is presented in Table 3. Of the schools that administer exams using the NBME® CAS program, most use the data for summative assessment for a module (Table 3).
Figure 1 presents the percent of schools that administer the NBME® CBSE which also administer an NBME® subject exam. The pathology subject exam is administered by the highest percentage (34%) of schools that offer the NBME® CBSE, while the histology and cell biology subject exam is administered by the lowest percentage (17%) of schools that offer the NBME® CBSE. We have previously reported the number of schools that administer the NBME® CBSE.3
Figure 2 presents the percent of schools that administer an NBME® subject exam that also administer the NBME® CBSE. All schools that administer behavioral sciences subject exam also administer the NBME® CBSE (highest percentage), while only 86% of schools that administer the histology and cell biology subject exam also administered the NBME® CBSE (lowest percentage).
Discussion
To our knowledge, this is the first report describing US medical school use of NBME® basic science subject exams and exams administered using the CAS program for preclinical medical education. On average, less than a quarter of the schools responding to the survey administer an NBME® basic science subject exam. Schools that do offer an NBME® basic science subject exam use the exams predominately for summative assessment for a course and to help revise curriculum (77%–100% depending on exam) (Table 1). In addition, only 30% of schools that responded to the survey administer exams using the CAS program, with most schools using the exams administered for summative assessment for a course (Table 3), while only 36% of schools that administer exams using the CAS program use the data to help revise curriculum (Table 3). Of the schools that administer an exam through the CAS program, 29% (4 out of 14 schools) also administer an exam using one or more of the NBME® basic science exams. Most schools that administer an NBME® basic science subject exam also administer the NBME® CBSE (Figure 2).
The finding that <30% of respondents administer NBME® basic science subject exams and only 30% administer exams created using the NBME® CAS program for preclinical assessment of medical student basic science knowledge is surprising given the fact that the NBME® basic science subject exams offer high-quality questions over a broad range of basic science content with the “majority of subject test items being used previously on the USMLE.”5 Although there is continued growth in assessments developed using the CAS program,5 offering exams using NBME® basic science exam or exams administered using the CAS program would provide exposure of USMLE® board style questions to students throughout their preclinical training. Use of externally validated exams may be an alternative for programs as it has been reported that internally developed exams are low quality, with the exception being for exam items developed by formally trained question writers.12 Additionally, it has been reported that multiple preclinical NBME® basic science subject exams correlate with performance on the NBME® surgery subject exam, and when a multiple regression analysis was performed the NBME® pathology exam remained the strongest single predictor of NBME® surgery exam performance.13
The fact that most schools use the NBME® basic science subjects for curriculum revision is not surprising. These exams provide targeted feedback of student performance in specific basic science content areas. However, what is surprising is the fact that only 77% of the schools administering the pathology NBME® basic science subject exam use the results for curricular revision. It is not clear why more schools do not use the pathology basic science subject exam for curricular revision.
The CAS program was introduced by the NBME® in 2007 with assessments administered through the program continuing to increase.5 However, the reported scope and utilization by US medical schools is limited. We report 30% of respondents use the CAS program. This program allows schools with integrated curricula to develop assessments tailored to the integrated courses provided. Interestingly, fewer schools (36%) use exams administered through the CAS program for curricular revision. The reason for this difference compared to NBME® basic science exams is unclear. Exams developed and administered using the NBME® CAS program are mostly used for summative assessments for a course (86%). Our findings report that only 21% of schools administer an exam using the CAS program for formative assessment for a course, with 2 schools stating that they use exams administered using the CAS program for both formative and summative assessments for a course. Use of exams developed using the NBME® CAS program for formative assessments for a course may be of interest as it has recently been reported that these exams correlate with performance on the USMLE® Step 1 and is a tool used to identify students at risk for low performance on the USMLE® Step 1.14
Further research into the performance of students attending schools that offer externally developed exams compared to schools that administer only internally developed exams would be of interest. In addition, determining why more schools do not offer externally developed exams is needed as this provides additional test item exposure to students preparing to take external board exams (ie, USMLE® Step 1). Additional areas of future research are to determine student and faculty perceptions of externally developed exams for courses as compared to internally developed exams.
A strength of this survey is the ability to gather data regarding the use of external exams in preclinical medical education using a low-cost method. A limitation of the survey is the survey response rate. A second limitation of the survey is that follow-up questions regarding outcomes data for the schools that use the external exam for curricular change were not included in the survey.
Conclusion
In conclusion, we report that most US medical schools responding to our survey do not offer NBME® basic science subject exams (15%–28% of schools offering an exam depending on the exam) or exams administered using the NBME® CAS program (30% of schools offering an exam) for preclinical medical education. However, of the schools responding most schools that offer an NBME® basic science subject exam use the exam for summative assessment for a course and for curricular revision, whereas of the schools that administer an exam using the NBME® CAS program most use the exam for summative assessment for a course while fewer use it for curricular revision.
Acknowledgment
The authors would like to thank Michael Wiederman, PhD for his help with SurveyMonkey® and expertise with the development and reporting of the survey.
Author contributions
William S Wright is responsible for data acquisition, data analysis, and preparation of manuscript. Both authors contributed toward conceptualization of project, survey development, interpretation of findings, revision of manuscript, and agree to be accountable for all aspects of the work. Both authors have reviewed and approved the final draft of the manuscript.
Disclosure
The authors report no conflicts of interest in this work.
References
USMLE. Step 1: Content Description and General Information. A Joint Program of the Federation of State Medical Boards of the United States, Inc., and the National Board of Medical Examiners®. United States Medical Licensing Examination. 2016:1–7. Available from: https://www.usmle.org/step-1. Accessed February 9, 2017. | ||
National Resident Matching Program. Data Release and Research Committee: Results of the 2016 NRMP Program Director Survey. Washington, DC: National resident matching program; 2016. | ||
Wright WS, Baston K. Use of the National Board of Medical Examiners® comprehensive basic science exam: survey results of US medical schools. Adv Med Educ Pract. 2017;8:377–383. | ||
NBME®. Guide to the Subject Examination Program. Philadelphia: NBME; 2017. | ||
Haist SA, Butler AP, Paniagua MA. Testing and evaluation: the present and future of the assessment of medical professionals. Adv Physiol Educ. 2017;41(1):149–153. | ||
Monteiro KA, George P, Dollase R, Dumenco L. Predicting United States medical licensure examination step 2 clinical knowledge scores from previous academic indicators. Adv Med Educ Pract. 2017;8:385–391. | ||
Alamiri NN, Maliska CM 3rd, Chancellor-McIntosh H, Sclabas G. Comparing surgical clerkship performance of medical and physician assistant students. J Surg Educ. 2017;74(3):466–470. | ||
Kimbrough TN, Heh V, Wijesooriya NR, Ryan MS. Family-centered rounds and medical student performance on the NBME pediatrics subject (shelf) examination: a retrospective cohort study. Med Educ Online. 2016;21(1):30919. | ||
Casey PM, Palmer BA, Thompson GB, et al. Predictors of medical school clerkship performance: a multispecialty longitudinal analysis of standardized examination scores and clinical assessments. BMC Med Educ. 2016;16:128. | ||
Ouyang W, Cuddy MM, Swanson DB. US medical student performance on the NBME subject examination in internal medicine: do clerkship sequence and clerkship length matter? J Gen Intern Med. 2015;30(9):1307–1312. | ||
Ryan MS, Bishop S, Browning J, et al. Are scores from NBME subject examinations valid measures of knowledge acquired during clinical clerkships? Acad Med. 2017;92(6):847–852. | ||
Jozefowicz RF, Koeppen BM, Case S, Galbraith R, Swanson D, Glew RH. The quality of in-house medical school examinations. Acad Med. 2002;77(2):156–161. | ||
Kozar RA, Kao LS, Miller CC, Schenarts KD. Preclinical predictors of surgery NBME exam performance. J Surg Res. 2007;140(2):204–207. | ||
Brenner JM, Bird JB, Willey JM. Formative assessment in an integrated curriculum: identifying at-risk students for poor performance on USMLE step 1 using NBME custom exam questions. Acad Med. 2017;92(11S Association of American Medical Colleges Learn Serve Lead: Proceedings of the 56th Annual Research in Medical Education Sessions):S21–S25. |
© 2018 The Author(s). This work is published and licensed by Dove Medical Press Limited. The full terms of this license are available at https://www.dovepress.com/terms.php and incorporate the Creative Commons Attribution - Non Commercial (unported, v3.0) License. By accessing the work you hereby accept the Terms. Non-commercial uses of the work are permitted without any further permission from Dove Medical Press Limited, provided the work is properly attributed. For permission for commercial use of this work, please see paragraphs 4.2 and 5 of our Terms.