Utilizing Post-Clerkship Surveys to Optimize Emergency Medicine Program Outcomes in the Match

Introduction

The recent two-year period (2022 and 2023) has presented new challenges for Emergency Medicine. According to the National Resident Matching Program (NRMP) website, The Match, as the NRMP is commonly known, is a private non-profit organization founded in 1952. Its main objective is to streamline the placement process of US medical school students into residency training programs at teaching hospitals throughout the United States. The 2022 cycle of The Match left 219 Emergency Medicine residency positions unfilled.¹ In the following year, the 2023 cycle of The Match produced even more unfavorable outcomes with 554 vacant Emergency Medicine residency positions after The Match and a fifth of emergency medicine programs remaining unfilled.² Data is still being gathered as to why this has occurred. Our modest Emergency Medicine program, situated in Lubbock, Texas—with a population of 250,000—successfully occupied all available positions during both the 2022 and 2023 cycles of The Match. Eager to gain insights, we delved into comprehending the potential advantages and disadvantages that our program has offered to medical students on rotation.

Moreover, our program was able to fill more quickly in the 2023 cycle of The Match when compared to the preceding year, suggesting that our program was able to attract more competitive candidates.

The literature indicates that the Emergency Medicine Clerkship is a vital tool for attracting students to a particular program;³ however, a notable gap remains in the exploration of how post-clerkship surveys might influence students’ perception of emergency medicine programs. This void becomes particularly pertinent when considering the diverse challenges in filling residency positions,^1,2 prompting the employment of various approaches to investigate this process. Previous studies have investigated the effectiveness of clerkship instruction,⁴ the impact of clerkship on students’ career choices,⁵ and the potential of the Emergency Medicine Clerkship to shape students’ perception of the field.³ Nonetheless, limited research has been conducted on strategies implemented by new programs located in smaller cities surrounded by rural communities with ethnically diverse populations to recruit students for residency positions. Therefore, the purpose of this study was to examine the potential efficacy of a post-clerkship survey in enhancing outcomes from The Match in smaller emergency medicine residency programs.

Methods

Lubbock, TX is not commonly associated with scenic views, nightlife, or outdoor activities - all major factors that can help attract prospective Emergency Medicine (EM) residents. In addition, our program, being relatively new, having graduated its first class of residents in June 2023, which raises further concerns for many potential EM residents. With these obstacles in mind, we undertook extensive measures to enhance our recruitment process for medical students into our residency program. To achieve this objective, we initially gathered data through a Post-Clerkship Survey (Appendices A and B). Subsequently, we thoroughly analyzed the collected data (Appendices C and D) to identify areas of improvement for our program and clerkship, and to refine our approach to highlighting our positive attributes. For instance, as part of the post-survey modifications, we introduced a presentation that showcases the unique and attractive attributes of living in Lubbock, TX. Additionally, to provide prospective applicants with a firsthand experience of the city’s charm, we incorporated a social night out in the area. This initiative allows potential candidates to get a glimpse of what it’s like to live in Lubbock, TX and helps them make more informed decisions about their future. Based on the feedback from the post-clerkship survey, we were able to demonstrate the effectiveness of our innovative approach to Standard Letter of Evaluation (SLOE) transparency, highlighting its positive impact on the program. Building on this success, we are now planning to introduce a curriculum specifically tailored for medical students, taking into account the most recent survey data.

We distributed the post-clerkship survey to all medical students who successfully completed an EM clerkship at our program and received a SLOE from us. At our program, following the EM clerkship standards of practice, SLOEs were provided to fourth-year medical students interested in Emergency Medicine upon completion of the clerkship. It is worth noting that we strictly maintained anonymity throughout the process, ensuring that individual students and their survey responses remained unidentified. The clerkship, offered as a 4th-year elective rotation at TTUHSC Emergency Medicine, accommodates a varying number of students each year. In the 2021–22 academic year, 21 4th-year medical students, eight of whom were home students, rotated with us. In comparison, 17 EM-bound students, including five home students, rotated with us in the 2022–23 academic year. Located exclusively within University Medical Center, the region’s only Level 1 Trauma Center and Burn Center, our program and clerkship benefit from the unique resources offered by the institution, including a dedicated pediatric emergency department. With an annual patient volume of around 75,000, the Emergency Department serves a significant number of patients in the region.

To gather feedback and identify areas for improvement, the clerkship director developed a post-clerkship survey (Appendices A and B) that was reviewed by the program director, assistant program director, and program coordinator. The survey was designed using a 3rd party platform and is anonymous. The survey consisted of a mix of multiple-choice and free-text questions. The multiple-choice section prompted students to rank our program based on factors related to the residency, clerkship, and the city of Lubbock, TX. Within this section, students were asked to reflect on their experiences with different aspects of our program and how these experiences influenced their personal ranking compared to other emergency medicine residency programs they might have encountered. They had three options to provide their feedback: “Ranked Program Higher”, “Ranked Program Lower”, or “No Change”. Importantly, students were not obligated to disclose their specific ranking lists or any other methods they used to compare emergency medicine residency programs. The free text questions asked students to list three positive attributes of the program/clerkship, three areas of improvement, and to provide further feedback in an open comment box. For anonymity purposes, the comments have been excluded from the survey results to maintain the confidentiality of the participants and to encourage them to continue expressing their honest opinions in the future. These comments mainly supplemented the responses given to the multiple-choice questions, providing explanations for the respondents’ answers.

The results and data were presented at the annual resident retreat as well as a monthly core faculty meeting. During those sessions, feedback was obtained, and appropriate changes were made to the clerkship based on the data and feedback collected. An example of the initial survey can be found in Appendix A.

Following the 2021–22 post-clerkship survey (Appendix A) and review, several changes to the medical student clerkship program were implemented based on the feedback received. In the subsequent version (Appendix B), all faculty and residents’ feedback was incorporated to implement necessary changes and additions. Additionally, the assistance of our Academic Chief Resident was enlisted in this endeavor. These changes were aimed at enhancing the overall experience of the clerkship program and improving student satisfaction. Implemented changes included:

• The 1-hour clerkship orientation presentation now placed more emphasis on Lubbock, TX, highlighting its positive aspects such as the cost of living, proximity to outdoor sports like watersports, snow sports, and wilderness sports, and showcasing Lubbock, TX, events. This change was made to give students a better understanding of Lubbock, TX, and its offerings.
• Students are now sent to the monthly “First Friday Art Trail”, a culturally rich experience in downtown Lubbock, TX, that showcases multiple artists and vendors in the area. This provides students with an opportunity to explore Lubbock’s vibrant art scene and various cultural offerings.
• The make-up of the people who interview candidates and represent the program at various functions was changed to showcase more diversity. The program made this change to reinforce its dedication to fostering diversity and inclusion, considering that the geographical area it serves, west Texas, primarily comprises non-Hispanic white and Hispanic populations. These modifications were prompted by some student feedback expressing dissatisfaction and align with the program’s overarching strategic plan to enhance diversity.
• Additional questions were added to the post-clerkship survey, including motivation to rotate with the program. This change was made to gather more comprehensive feedback from students and improve the program’s performance.
• Residents and faculty were given guidance on appropriate conversations/discussions to have around students while on shift. This was done to ensure a professional and respectful environment for all parties involved.
• The decision was made to continue the program’s SLOE transparency.

After implementing these changes, an evaluation was conducted using data obtained from the 2022–23 post clerkship survey (Appendix B) to assess the impact of these modifications. Overall, Figure 1 provides a summary of the sequence of events involved in the utilization of post-clerkship surveys during both the 2021–22 and the 2022–2023 cycles.

Figure 1 Timeline of events outlined in relationship to the 2022 and 2023 cycles of The Match.

Results

The findings from the post-clerkship survey conducted in the 2021–22 academic year are presented in Appendix C. The survey yielded a response rate of 52%, with an average completion time of 17 minutes. A comprehensive account of the results can be found in Appendix C. The overall assessment of the survey results and accompanying comments were positive in nature. With regards to the multiple-choice questions, there were multiple areas where most respondents ranked our program higher as outlined in Table 1.

Table 1 Survey Domains with Respondents Ranking Our Program Higher in the 2021–22 Academic Year Post-Clerkship Survey by Percent Ranked Higher

The 2021–22 survey also revealed that the majority of the respondents assigned a lower rank to the program in one domain, “Living in Lubbock, TX”, where 55% of the participants ranked our program lower. The second domain with the highest number of respondents ranking our program lower was “current diversity in the program”, with 18% of participants indicating dissatisfaction with the program’s current level of diversity. Furthermore, the survey results demonstrated that the majority of the participants reported no change in their rankings of the program with respect to several domains, including the boarding of patients (91% no change), political viewpoints in Lubbock, TX (73% no change), and current diversity in the program (64% no change).

In the 2021–22 cycle of The Match the program was able to fill all residency positions. Moreover, the program was successful in matching two away rotators from the same clerkship cycle.

The post-clerkship survey results for the 2022–23 clerkship year are presented in Appendix D. The response rate increased to 65% compared to the previous year. Despite the inclusion of additional questions, the average completion time for the survey decreased to 8 minutes. Once again, the overall results revealed a positive outlook from the respondents. With regards to the multiple-choice questions, there were several domains where the majority of respondents ranked our program higher, consistent with the previous year’s survey. These domains are outlined in Figure 2.

Figure 2 Survey domains with respondents ranking our program higher in the 2022–23 academic year post-clerkship survey by percent ranked higher and change from the previous academic year post-clerkship survey.

No question received a majority negative or “ranked the program lower” response in the 2022–23 post-clerkship survey. It is worth noting that the question related to living in Lubbock, TX, which had previously received the highest percentage of respondents ranking the program lower, showed a 10-point decrease in dissatisfaction, with only 45% of the respondents assigning a lower rank to the program on this matter. Only one question had a tie between “no change” and “ranked program higher”, and this was EM’s relationship with trauma, with both categories receiving selections from 45% of respondents.

The categories that showed a majority of “no change” responses included:

• Current diversity in the program (64% no change, 18% ranked the program lower, no significant change from previous)
• Political viewpoints in Lubbock, TX (55% no change, decrease by 18 points)
• Boarding of patients (82% no change, new question)
• Social events (73% no change, new question)
• Pre-interview zoom social (82% no change, new question)
• Rotation orientation/SIM lab (64% no change, new question)

The changes we made in the program not only showed positive effects in the survey, but also in our results from this cycle of The Match. In 2023 we were able to match 4 of our rotators, compared to 2 the previous year. We were also able to fill in the 2023 cycle of The Match, and we did not go as far down our rank list in the 2023 cycle of The Match when compared to 2022.

Discussion

Numerous studies have highlighted the positive impact of medical students’ clerkship experiences on shaping their perceptions and fostering potential interest in specific medical specialties.^6–9 This influence extends beyond mere exposure, as seen in our research findings. Notably, factors such as faculty engagement, resident interactions, and program location emerge as pivotal elements, echoing their significance across other medical specialties,^8,10 supporting the results from our survey.

In the context of our investigation, the Emergency Medicine Clerkship emerges as a particularly potent instrument for cultivating students’ fascination with the field of emergency medicine.³ Although the use of post-clerkship surveys is not a novel concept in previous research, literature review did not yield any publications specifically addressing the assessment of student perceptions about a program and its distinguishing features through post-clerkship surveys. Avegno et al⁴ investigated the efficacy of their clerkship’s instruction by conducting pre- and post-clerkship surveys, but no responses regarding opinions of the program were obtained. Cevik et al⁵ employed a survey to appraise the effect of a clerkship on students’ career choices. Nagurka et al³ similarly utilized surveys to assess how a clerkship could shape students’ perceptions of Emergency Medicine as a specialty, but not the program itself. So, as of date, no further research has been identified through literature review specifically detailing the use of pre- and post-clerkship surveys aiming to improve a student’s receptibility to a specific emergency medicine residency program.

Upon reviewing the outcomes of the initial 2021–22 post-clerkship survey (Appendix C), our primary concern was the fixed aspect that we could not modify, which is our location in Lubbock, TX. The survey enabled us to obtain targeted feedback from students, and the dissemination of the survey results to faculty and residents facilitated gathering suggestions on how to accentuate the numerous advantages of residing in our city. Therefore, we extensively modified the clerkship orientation PowerPoint presentation to feature Lubbock, TX, and several specific domains deemed important by students. Additionally, residents were encouraged to extend invitations to students for events they planned on attending in and around Lubbock, TX. Furthermore, students from every clerkship were taken to a monthly local event featuring regional artists, performers, and cuisine to enhance their understanding of living and working in Lubbock, TX. The survey findings revealed a mixed impact of these initiatives. Nonetheless, three students reported higher program ranking due to these efforts, and therefore, these activities will persist in the future. As discussed above, these modifications to the clerkship orientation showed some improvement by decreasing the percent of respondents who ranked the program lower based on the “Living in Lubbock, TX” domain. This addition of promoting aspects of social life in Lubbock, TX is being maintained to provide a point of comparison in subsequent iterations of the survey, which we anticipate will yield more data to ascertain the true impact of this modification on students’ program ranking perceptions. The revised clerkship orientation PowerPoint presentation now encompasses the following sections:

• “Living in Lubbock, TX”: A comprehensive overview of our population, transportation options to and from Lubbock, TX, and the plethora of activities ranging from arts to outdoor recreation available in the area.
• “Cost of Living”: Statistical data obtained from the Lubbock, TX, Economic Development Alliance was employed to demonstrate how Lubbock, TX, compares favorably to the national average in terms of cost of living.
• “Arts”: Three slides were dedicated to spotlighting the diverse arts and cultural events in our city, such as the symphony, Broadway musical series, and other cultural occasions.
• “The University”: As our program is affiliated with a Division 1 University, two slides were utilized to showcase the myriad of sporting events and entertainment opportunities available in the area due to the university.
• “Travel Time and Care Region”: A slide displaying a map of our region was employed to highlight our city’s proximity to various outdoor activities, including multiple state and national parks. Additionally, the map was utilized to emphasize the scope of our care region, particularly for Trauma and Burn patients.

It was noted in the survey conducted in 2022–23 that the majority of the respondents reported “no change” when asked how the orientation presentation effected how they ranked the program. This is potentially due to the lapse of at least six months since the presentation was delivered at the commencement of their rotation. Nonetheless, the presentation was deemed effective, as three students attributed a higher ranking to our program following its delivery. Thus, the presentation will be retained, with requisite modifications, to sustain its efficacy in the future.

The implementation of SLOE transparency was a novel approach that was first introduced during the 2021–22 clerkship year. We recognized the potential risk of receiving negative feedback on the clerkship and/or program, as we are committed to using the full range of the SLOE ranking, including the “Lower 1/3” ranking. To address this concern, we informed students at the beginning of the rotation about the different types of data we collect to write our SLOEs, how we write them, and our desire to be as transparent as possible. We further informed students that we would be disclosing their “overall ranking” on the SLOE, but no other information, such as the narrative portion, would be shared. The positive impact of this approach, as indicated above, on the students’ ranking of our program was evident in the 2021–22 survey results, which was further reinforced by the 2022–23 survey results. Thus, we intend to continue this practice and explore the effects of SLOE transparency in greater detail in future studies.

The survey domains of “The residents and attendings at the program”, “Exposure to a wide range of pathology”, “Working in a county hospital”, “Exposure to trauma and critically ill patients”, and “The residency leadership” all maintained a high level of “Ranked higher” responses from respondents of the post-clerkship surveys. This reinforces the point that the most important aspects of our program, and therefore what we should highlight most, are our faculty, residents, and hospital. Thus, it becomes crucial for us to prioritize highlighting these aspects prominently. We actively engage in presenting and discussing the significance of these positive factors with our faculty and current residents during the annual resident retreat. The primary objective is to underscore the profound impact these strengths have on shaping the overall perception of our program and nurturing a positive and enriching learning environment for rotating medical students.

Our experience demonstrates that this tool has become invaluable for identifying critical areas that require improvement, continuation, or removal in our residency and clerkship, and our results from the past two cycles of The Match have validated its efficacy. It’s crucial to emphasize that the modifications introduced to our program’s recruitment process were solely based on the concerns raised by students in the conducted surveys. However, it’s important to acknowledge that the current state of the survey might not fully encompass all potential issues or positive aspects of our program that have yet to be brought to light. In continuation, it is important to discuss limitations of this study. First, only post-clerkship surveys from two academic years were utilized in the study, with only the 2022–23 post-clerkship survey containing data from respondents who received the revised clerkship orientation PowerPoint. Another limitation is that this study was only performed at a single institution and geographic location, thus allowing only a small sample size. Further data will need to be collected across a greater sample size of programs from varying geographies for validation.

Lastly, it should be noted that the COVID-19 pandemic was another factor that has been shown to have influenced The Match, during which in-person interviews and other traditional recruitment strategies were not available. The negative effects of the COVID-19 pandemic on medical residents and medical students have been demonstrated in previous studies in multiple settings.^11–13 Further, the COVID-19 pandemic has been shown to have an effect on medical students’ choice of specialty interest.¹⁴ Although we could not quantify the extent of the effect of COVID-19 on our specific program, it is possible that the pandemic altered students’ perception of the proportion of critically ill patients and how a county hospital such as ours was more likely to be at the front line of pandemics.

Conclusion

Our new EM program in rural West Texas faces several challenges when it comes to matching a full complement of residents in the current environment. Through use of data obtained through a post-clerkship survey, we have been able to adapt our clerkship and our program to this challenging environment and were able to fill each of the past 2 years. Program residents and faculty, our hospital, and SLOE transparency were found to be nearly unanimous positive factors in how students chose to rank us, while our location in Lubbock, TX, continues to be the program’s biggest drawback. Overall, we will continue to gather this data each year and continue to make changes to our program and clerkship based on it. More time is needed to see if the success we saw in this initial phase will continue.