What Feedback Content Do Clinical Teachers Address During OSTEs?

Introduction

During the last decades, an increasing number of residency programs have adopted a competency based curriculum and the Entrustable Professional Activities (EPAs) as a robust assessment framework of competencies in several countries.¹ The EPAs are a unit of translation of competencies into clinical practice:² they are work activities or tasks that supervisors entrust trainees with once they achieve adequate competencies.³ Direct observation or reviews of performance followed by formative feedback have become the cornerstone of assessment of competency-based education programs, especially since the EPAs have been implemented.⁴ Formative feedback, defined as an information given to the learner with the intention of adjusting his or her thinking or behavior for the purpose of improving learning,⁵ is indeed the most widely used approach to stimulate learning and development at all levels of clinical expertise development.⁶ It has been identified as a major enabler to effective clinical supervision in the workplace, especially if it is regular, credible, based on a trusting relationship, stimulates reflection, and has a learning goal-orientation.^7,8

Several factors influence the quality of the feedback content: clinical teachers should have time, feel comfortable with the issues observed and addressed, be thinking of the theoretical or conceptual frameworks related to such issues to help students gain a deeper understanding of factors influencing their own reasoning or behaviours, support students to challenge their assumptions and values and develop a broad range of possible responses and interventions.^9,10 They should also be aware of learners’ needs as well as educational standards linked to the learners’ level of training.⁹ The focus of feedback may vary between patient care (management of patient care) and / or educational needs (educational resources).

Côté et al explored feedback content in terms of clinical teachers’ educational goals and the use of conceptual frameworks based on hypothetical clinical situations.^9,11,12 Gonzalo et al analysed the issues addressed in the feedback and the timing of feedback given by experienced clinical teachers after bedside rounds.¹³ Some authors explored supervisors’ verbal or written feedback given in the workplace according to the Can MEDS roles.^14–16

Faculty Development programs have been developed in order to train clinical teachers to supervise and give feedback in effective and constructive ways.¹⁷ Several approaches can be used such as workshops, short courses, seminar series or workplace activities. Instructional methods may include small-group discussion, interactive exercises, peer observation and coaching, and experiential learning such as OSTEs (objective structured teaching encounters).^18–20 OSTE is a training tool where participants practice clinical teaching skills with a simulated learner. Although OSTEs are not real workplace- based assessments, they are still designed to reflect the real world and attempt to replicate the challenges of giving feedback on various issues in everyday clinical practice. OSTEs can be used to evaluate feedback skills and represent an opportunity to explore what content is addressed.

The aims of this study were several: 1) to analyse the content of feedback and the way content was addressed during objective structured teaching sessions (OSTE) as part of a Faculty Development program; 2) to evaluate whether the feedback content varied according to the clinical teachers’ training phase, the medical discipline of the clinical teacher or the type of task observed; finally 3) to evaluate if there was some alignment between the feedback content addressed by clinical teachers and the content identified as essential by experts.

Method

Design and Setting

We conducted a multi-method (quantitative and qualitative) study as part of a longitudinal faculty development program conducted in five different departments (ie paediatrics, obstetrics / gynaecology, psychiatry, internal medicine, and family medicine) at the Geneva University Hospitals (Switzerland).

Participants

Eighty nine clinical teachers from the five different departments were trained to supervise and give feedback to residents on several clinical topics in a six-month training program between 2015 and 2017. Participants’ mean duration of clinical teaching practice was 2.11 years (SD 2.27) and mean duration of clinical practice was 9.78 years (SD 4.61); 16% had attended a training on feedback skills, 6% on supervision skills and 2% on reflective practice skills.

Pre and Post Training OSTE and Training

OSTEs

Before and after the training, clinical teachers completed four OSTE stations during which senior medical students participated as simulated residents. For three stations, clinical teachers were asked to look at short videotaped clinical interactions between a simulated resident and patient and give an immediate eight-minute feedback to the resident. Feedback was given on tasks focused on communication skills, inter-professional collaboration a physical exam or a technical procedure (Table 1). All of them included some elements of clinical expertise. The OSTE stations were developed by experts in each medical discipline and were meant to mirror common clinical issues encountered in practice. These experts were attending physicians involved in undergraduate and postgraduate teaching activities who participated in the design and implementation of the Faculty development program. Simulated residents were trained to explicitly express (if prompted) two specific learning needs in regard to the task completed. All feedback sessions were videotaped for both teaching and research purposes.

Table 1 Topics per Task and Medical Discipline

Faculty Development Training

The training itself consisted of two two-hour workshops related to feedback and clinical supervision. Each workshop included an activity soliciting participants’ perceptions of their role as clinical teachers, an interactive presentation of teaching concepts and strategies, analysis of videos modelling teaching strategies and role-plays for skills practice. Both workshops were facilitated by experienced clinical teachers. Participants attended either three one-hour personalized coaching sessions on their pre-intervention OSTE performance or a two-hour reflective practice workshop during which they watched and reflected on the pre-intervention OSTE as part of a randomized controlled study.²¹ This study showed that both formats were effective in improving feedback skills such as exploring students’ learning needs, stimulating their self- assessment, taking into account their self-assessment in the feedback content, limiting the items addressed, and checking students’ understanding.²² However, participants receiving personalized coaching outperformed the guided self-reflection group in terms of both feedback delivery and formulation of learning objectives.

Analysis of the Feedback Content

Development of a Generic Feedback Content Grid

Because the OSTES addressed various tasks implying different skills and roles and included different clinical topics inside each task according to the medical specialty (Table 2), we developed both deductively and inductively a generic grid to capture dimensions that could be comparable between OSTEs and specialties. We initially tested a global communication rating scale developed for objective structured clinical examinations. This grid, used for summative high stake exams in Switzerland, has shown high internal consistency and validity.²³ It consists of four transversal dimensions: A) structure, B) patient perspectives, C) verbal expression, and D) nonverbal expression. RL and NJP first read several OSTE transcripts focused on communication skills. They found that feedback content related to communication could be easily categorized using the general structure of this scale. These dimensions were integrated into a larger “targeted content” category. As RL and NJP proceeded in the reading and analysis of the transcripts, they found other dimensions of content that were added to the grid after discussion with the research team to cover all the skills and feedback content addressed within the communication task. The additional categories were: E) Other clinical content, which included other CanMEDS competencies such as “Clinical expertise” (history taking, physical exam, clinical reasoning), and “Professionalism”; F) Teaching strategies; and G) Self-management/other (management of one’s emotions, taking care of self). The targeted content was related to task-specific frames of references when existing (Table 3).^24–26 While the targeted content grid for Interprofessional collaboration included the same dimensions as the communication one, the targeted content grid for the physical exam and technical procedure was slightly modified to incorporate a dimension about the precision/accuracy of the procedure and two different dimensions related to communication (communication addressing the clinical facts and communication addressing patient comfort and emotions) in addition to the structure dimension.

Table 2 Categorisation of Elaboration Occurrences According to Tasks

Table 3 Generic Rating Grid Used to Assess the Feedback Content per Task

Feedback content was also coded as positive when the clinical teachers valued or reinforced the student’s performance or negative when they questioned student performance or made suggestions for improvement.

Categorisation of Teaching Strategies Used to Address the Different Dimensions of Feedback Content

In order to analyse whether clinical teachers’ use of teaching strategies differed according to the feedback content addressed (targeted content, other CanMEDS content, etc.), we created a grid inspired by the educational goals by Côté et al⁹ and derived from our data, aiming at categorizing the types of teaching strategies used to address the content of the feedback: restitution of observations (eg, “I noticed that when you explained the procedure, the patient reacted negatively”), implicit questioning (eg, “Do you think that the patient was happy about the way you explained the technical procedure?”), suggestions (eg, “One thing that might help in this situation is to ask the patient how he/she feels before continuing to explain”), active involvement of the resident (eg, “What do you think you could have done when the patient reacted negatively?”), and elaboration. Elaboration was defined as a “micro teaching” phase during which clinical teachers elaborated in a directive or interactive way on the importance of reinforcing or changing the residents’ specific behaviour addressed during the feedback sessions (generalization beyond a simple suggestion: “One thing you must know is that whenever the patient seems to express emotions during an explanation, you must pause and explore what happens; otherwise there is a high risk that he/she will not listen to you anymore”). RL and NJP coded the occurrences related to both the content and the type of teaching strategies. Inter-rater reliability (intraclass correlation coefficient calculated based on 10% of double coding) was 0.91.²⁷

Elaboration occurrences were further analysed since they may reflect participants’ teaching priorities and areas of comfort. All the investigators read about 20 elaboration occurrences and documented the type of frameworks used. We ended up with four main categories: teaching through sharing experience and tips targeted to the task, teaching physician roles/professionalism, teaching clinical reasoning, teaching theoretical concepts, the remaining occurrences being classified as “other”. Two investigators per task coded the elaboration occurrences independently. Differences of coding were solved through discussion or intervention of a third person.

Identification by Experts of Strengths and Weaknesses of Simulated Residents During Tasks

We asked three experts (one who created the scenarios of the videotaped patient-simulated resident encounters of the OSTEs and two other experienced clinical educators involved in the Faculty Program) to identify the main strengths and weaknesses of the simulated residents’ performance for all the tasks of their medical discipline. They identified 2–4 strengths and 2–4 weaknesses per task (Appendix 1). We selected the items all three experts agreed on and compared them with those addressed by the participants during the feedback session to check the degree of alignment between the content addressed and the content identified as essential by experts.

Analysis

The distribution of content, feedback approaches and focused teaching using frequency measures were descriptively analysed. ANOVA test was applied to evaluate whether the feedback content was influenced by the intervention phase (pre and post), the type of task observed and the medical discipline. Post-hoc analysis for significant categories resulting from ANOVA analysis was conducted using Tukey HSD. Results were summarized using confidence intervals (α = 5%).

We descriptively assessed the percentage of alignment between feedback content items addressed by the participants and the items identified by experts as the main strengths and weaknesses of residents’ performance.

Results

We analysed the content of 317 feedback sessions involving 89 participants: internal medicine (93 sessions), family medicine (70 sessions), gynaecology-obstetrics (69 sessions), paediatrics (56 sessions), and psychiatry (29 sessions). A total of 5388 occurrences were coded.

Type and Distribution of Feedback Content Addressed

Around two -thirds of the feedbacks given focused on the targeted content of the task and around two-thirds of the content addressed items needing improvement (negative feedback) (Table 4). Clinical reasoning in “other clinical content” was the main clinical skill addressed after the targeted content. Learning strategies and self-management were rarely discussed, but when addressed, learning strategies were treated in a positive way.

Table 4 Distribution of the Feedback Content

Type of Feedback Approaches Used According to the Content

Targeted content was mainly addressed through restitution of observations and suggestions while participants used restitution of observations, suggestions and elaboration in the same proportions for the “other clinical content” (Figure 1). Learning strategies were addressed through suggestions and interactive participation. Self-management was mainly addressed through feedback and to some extent through suggestions. Implicit questioning was rare.

Figure 1 Distribution of the type of teaching approaches used in the feedback phase (%).

Elaboration (micro teaching) mainly consisted of sharing experiences and tips about the task itself (Table 2). Physicians’ roles, attitudes, and values (notably professionalism) were addressed more often during feedback focused on communication than other technical tasks. We also found that participants addressed clinical reasoning more often during feedback on communication, interprofessional and technical procedure tasks than on physical exam tasks. Examples of elaboration occurrences are given in Appendix 1.

Factors Influencing the Feedback Content

Feedback content distribution varied according to the observed task and the medical discipline. More targeted content was addressed in physical exam and technical procedure than in communication and interprofessional asks (95% CIs [0.41,1.05] and [0.46,1.16]). Paediatrician participants included more targeted content items in the feedback than participants from internal medicine or psychiatry (95% CIs [0.24,1.04] and [0.35,1.43]). No difference in feedback content was observed according to the phase of the training (pre or post): F(1313)=.29, p=.59 for targeted content; F(1243)=.46, p=.50 for other content; F(1114)=2.4, p=.12 for teaching strategies; and F(1111)=.02, p=.88 for other and self-management.

Percentage of Alignment Between Skills/Items Identified by OSTE Developers as Essential and Those Addressed by Participants During the Feedback Session

The percentage of alignment between skills/items perceived as essential by experts and those addressed by participants during the feedback session was diverse but generally low. It seemed to vary according to the medical discipline and the type of task observed. However, the small sample of participants per medical discipline did not allow further statistical analyses (Table 5).

Table 5 Percentage of Alignment Between Skills/Items Identified by OSTE Developers/Experts as Important (Either Positive or Negative) and Those Addressed by Participants During the Feedback Session

Discussion

This study aimed to analyse the content of the feedback given by clinical teachers during OSTE stations before and after a faculty development training on supervision and feedback skills.

Our results show that about two-thirds of the feedback given by clinical teachers focused on the targeted content of the task and that the feedback was mainly focused on areas of improvement. Targeted content was addressed mainly through the restitution of observations and suggestions, while elaboration (micro teaching) was equivalently used for the other clinical content. Micro-teaching itself essentially consisted of experience and tips sharing. The phase of the training did not influence the content addressed. However, the topic of the observed task and the clinical teachers’ medical discipline influenced the feedback content to some extent. Finally, there was a poor alignment between the content addressed by clinical teachers and the one identified as essential by experts.

Clinical teachers addressed targeted content in a large proportion. The percentage (70%) is higher than in a prior study where alignment of feedback regarding intrinsic CanMEDS roles in case presentations made by simulated residents varied between 48% and 56%.¹⁴ The fact that the OSTE stations were explicitly labelled as “communication”, “interprofessional collaboration”, “physical exam” or “technical procedure” may have helped clinical teachers select the feedback content. This is important since previous studies have revealed how certain roles such as interprofessional collaboration and communication are implicitly addressed during feedback in the workplace.²⁸ Introducing in OSTEs content related to roles often neglected in workplace teaching may be a powerful and innovative way of heightening teachers’ interest in and attention to such roles. However, more specific training might be necessary since clinical teachers often lack frames of reference and concepts to address such issues during clinical supervision in the workplace.

Feedback essentially focused on negative aspects of simulated residents’ performance. The resident’s performance during OSTEs was designed to be average to low. The purpose was to decrease clinical teachers’ cognitive load and help them focus more on the feedback skills than on identifying and analysing the problems faced by the resident. Although this may explain why balance between positive and negative feedback was poor, it might also suggest that clinical teachers may feel less tension around balancing the positive with the negative or critical aspects of feedback than earlier described.²⁹

The small alignment between the content addressed by clinical teachers and the one identified as essential by experts is striking. Clinical teachers’ feeling of comfort with the issues observed is one of the several factors that influence the feedback content.³⁰ A prior study showed that alignment between communication skills identified as key by communication experts and selected as key by clinical teachers involved in training programs on how to teach communication skills in the workplace ranged between 30% and 60% - items identified by clinical teachers varied substantially independently of the communication situations.³¹ Although communication is considered to be inherently subjective and dependent on what it means to patients and physicians in a specific context,³² our findings suggest that this statement may apply to other types of skills or clinical expertise and reflect clinical teachers’ areas of comfort or expertise. Variability in feedback focus and difficulty in identifying key deficiencies in workplace-based assessment have been reported earlier.^29,33 Such variabilities may reflect differences in clinical teachers’ teaching scripts resulting from the interaction between teachers’ knowledge of the topic, the context, the teaching strategies and the students. They are considered to be highly personalized and idiosyncratic.^34–36 In the time of competency-based education and entrustment of residents through observation of daily tasks, it is commonly accepted that competencies are complex and multifaceted. Therefore, variability between assessors in the workplace may be considered as less dissenting than complementary if we consider that different clinical teachers observe different aspects of competence.^37,38 However, there is need to ensure that clinical teachers identify and address elements of competence according to residents’ needs and context priorities and not based on their personal preferences.

Limitations

There are several limitations. We could not use of a validated grid to analyse feedback content since the content of each task and the key behaviours displayed by simulated residents varied according to clinical teachers’ medical discipline. Although such variety may mimic the challenge of giving feedback on various issues in everyday clinical practice, it makes comparisons difficult since clinical situations are context and case specific. As mentioned above, the fact that simulated residents were poorly or moderately performant during OSTEs may have prompted participants to address more negative than positive feedback. We analysed and categorized the content but did not assess the quality of the feedback content. This would have implied including clinical teachers as well as simulated residents’ perspectives. As we did not explore participants’ reasons for choosing the feedback content they addressed with simulated residents, we can only provide assumptions for choices made. Finally, as feedback is considered a conversation in which both the clinical teacher and the student collaboratively reflect on his/her performance and how to improve it,^39,40 the content may also be influenced by the students’ statements and reflections. Although simulated residents were trained to provide standardised responses to the clinical teachers, we cannot formally discard students’ influence on the feedback content. Finally, we did not explore the influence of clinical teachers’ and students’ gender on the feedback content.

Conclusion

The study shows that feedback content given during OSTEs is highly variable but mainly addresses the specific roles of a given task, including interprofessional collaboration and communication. Although OSTEs are designed to mimic authentic clinical teaching encounters and may represent an interesting approach to help clinical teachers consider all roles and address them when giving feedback, they do not replace clinical teaching sessions in real-life clinical settings. More studies should analyse what content clinical teachers address during feedbacks in the workplace and explore which elements of the context, the learner or the task itself most influence the content of the feedback chosen by clinical teachers. Indeed, clinical teachers are often unaware of the different dimensions of their teaching script and how they influence the content of both supervision and feedback sessions. A better understanding of what factors are in interplay during clinical teaching may help them chose the content according to residents’ needs and context priorities more than based on their personal preferences.