Implementation of Digital Faculty Portfolio in Medical College Using an Intervention Mapping Approach

Introduction

For any medical college to succeed in its mission, faculty members must be actively engaged on multiple fronts: cutting-edge scientific research, effective pedagogy, efficient administration, and interpersonal communication. On each of these fronts, faculty must simultaneously lead individual courses and mentor individual students, all while integrating their work into the broader curriculum of the institution. This requires coordination among faculty and the freedom of each faculty member to design courses and balance competing goals. Given the complexity of faculty work, leaders and administrators have the difficult task of evaluating faculty performance in a multi-dimensional space.¹ The Joint Information Systems Committee (JISC) of the UK defines an e-portfolio as a digital collection of artefacts that learners create to showcase their formal and informal learning, experiences, and achievements.²

Teaching portfolios are more comprehensive than student evaluations alone and are less affected by student bias,^3–5 utilizing multimedia technology; modern teaching portfolios can incorporate even more detail, including presentation slides and teaching videos.^6,7

Teaching portfolios have been used for decades to evaluate present and potential faculty fields.^8,9 They provide greater detail than student evaluations and are less prone to bias.⁴ Today, digital media gives us the power to create multimedia “digital teaching portfolios” (DTPs) with far greater scope and flexibility than traditional print portfolios.^10,11

The Digital Faculty Portfolio (DFP) extends beyond the traditional teaching portfolio by incorporating additional professional responsibilities such as research, administration, mentoring, committee involvement, and community outreach. Unlike a teaching portfolio focusing solely on classroom duties, the DFP allows faculty members to track and manage all their actions, providing a broader evaluation method for employers. As a result, it offers a more comprehensive perspective than a teaching portfolio. In this study, we experimented with a novel tool for evaluating and coordinating faculty work: The Digital Faculty Portfolio or DFP. The DFP is based on a well-established Teaching Portfolio, a tool that combines student evaluations with teaching materials, narrative reflections, and evidence of pedagogical effectiveness.⁹

In prior research on digital teaching portfolios, researchers have found that the main limitation of a digital format is the technical details of implementation and ongoing user support. As valuable as this format may be in concept, it can only succeed if it is well-designed and if users (who may have little or no experience with platforms of this kind) have access to technical support.^6,11 In this study, we expanded the concept of a traditional DTP by implementing a Digital Faculty Portfolio (DFP), which goes beyond teaching by including all faculty responsibilities inside and outside the classroom. The activities were divided into seven sections: teaching and assessment (traditional DTP), research, academic management including committee services and leadership roles, faculty enhancement, community services, faculty goals and achievements, and self-reflection. Therefore, the current study will reveal insights into the specific technical details most conducive to success in a DFP program.

Once individual faculty members have completed their DFPs, the department can develop a collective DFP that summarizes its curriculum, approach to teaching, and areas of excellence and expertise. These can then be compared with other departments within the same institution or with corresponding departments at different institutions, and the department can gain the same benefits of integrated curriculum and niche development that faculty members gain within the department.

Therefore, the aims of the DFP research study are threefold: first, to test the DFP concept and determine whether faculty find it useful; second, to provide a platform for integrating faculty activities, including both teaching and extracurricular activities; and third, to identify key technical details that need to be addressed before a larger-scale DFP platform can be created.

Materials and Methods

This study was conducted at the College of Medicine, King Saud bin Abdulaziz University for Health Sciences (KSAU-HS), Riyadh, Saudi Arabia. In this DFP study, we used the Intervention Mapping Model (IM)¹² to quantify the need for a DFP and measure its effects as reported by users. Intervention Mapping is a planning protocol originally developed for public health applications but widely applicable to any setting where an intervention must be planned, coordinated, implemented, and evaluated using theory and evidence. Implementation Mapping has the advantage of providing an overview of the entire implementation process, from assessing needs to determining whether those needs have been filled. The following is the description of the model process:

1. Needs assessment: determine what problems exist and what changes need to be made.
2. Program outcomes and objectives: define specific goals.
3. Program design: make plans to achieve goals and design intervention systems (eg, software)
4. Program production: create tools and test functionality.
5. Program implementation plan: determine logistics, rollout, and onboarding.
6. Evaluation plan: determine the effectiveness of the implementation.

In the present study, we followed the three sequential stages: (1) assessment of the educational needs of the full-time faculty members, (2) design of ‘The DFP program, and (3) validation and refinement of the program.

Design and Implementation

Figure 1 shows the design and implementation flow chart.

Figure 1 The DFP flow chart of design and implementation phases.

Database Search

At this stage, we conducted a literature review on PubMed and Google Scholar to learn how teacher education institutes handle the implementation of technology into their curricula. What new national accreditation requirements require education institutes to prepare future teachers to incorporate technology into their curricula? What are the tools, environments, and ongoing professional development that university faculty and students require to integrate technology into teacher education curricula?¹³ Our prime aim was to determine the DFP domains that should be covered and to select a model that would assist us in designing a DFP that integrates empirical and theoretical research and involves key stakeholders in the planning process to develop better and more effective interventions. However, knowing exactly how and when to use theory, evidence, and community involvement during planning can be difficult. This difficulty is addressed by Intervention Mapping, a framework for theory- and evidence-based health promotion program design that offers a systematic and step-by-step method of preparing interventions. With more than 1000 articles using the framework, intervention mapping has been utilized to create health promotion interventions and implementation techniques worldwide in community and clinical settings. The use of intervention mapping for planning implementation strategies, adapting evidence-based programs to new settings, and novel methodologies for identifying behavioural and environmental variables are some of the new areas in intervention mapping (IM) application that we use in our planning.¹⁴

Based on the analysis of various approaches, expert opinion, and literature, we concluded that the domains of a faculty member’s DFP should include elements indicating their professional growth: general knowledge, scientific and technological activity, instructional approaches, and key aspects that the teacher uses and expands. As a result, we identify three domains: performance, motivation, and experience. As a result, three domains are developed: performance, motivation, and expertise.

Pre-Implementation Survey (Needs Assessment)

We solicited responses from teachers in 2019 about how they access, use, and incorporate technology into their teaching. A short survey for the needs assessment was created and circulated by Email to all faculty members at the College of Medicine. The survey questions covered three topics: performance, motivation, and experience. In the first category, performance, questions address the respondent’s work responsibilities, including specific responsibilities (eg, curriculum design) and general attributes (eg, work ethic). In the second category, motivation, questions address the respondent’s affective disposition concerning their work responsibilities and their perception of the general disposition of their colleagues. These questions provide information on the respondent’s motivation to succeed. In the final category, experience, questions address the respondent’s prior exposure to DFP or similar systems. These questions were used to prepare a DFP system that would adequately support our users’ needs. A team of experts from both basic and clinical sciences validated the survey. The reliability of each domain of DFP was calculated. The Cronbach’s Alpha for the performance domain found was 0.88, motivation was 0.87, and experience was 0.79.

Respondents were asked to use a 5-point Likert scale to rate their agreement with a statement for each question. We then calculated the percentage of respondents at each rating level. We also summed the 4- and 5-point responses (“agree” and “strongly agree”) to quantify the total rate of agreement among respondents.

The sample size was estimated using the Pieface. Considering a significance level of 5%, the power of test 80 with an estimated mean difference between groups 20 and estimated SD= (upper limit – lower limit /4) 100–60/4=10, the sample size was 61. All full-time faculty members (male and female) at the College of Medicine in Riyadh (n=54) and Jeddah (n=28) were included in the process. We used a convenience sampling technique. Invitation letters were sent by Email to invite all male and female full-time faculty in the College of Medicine to participate in the online questionnaire.

A digital faculty portfolio using intervention mapping integrated with the existing Student Academic Management System (SAMS) was developed. Changes in the program were made after the experts’ feedback and IT members’ cooperation to support the system. Their support also helps to ensure that faculty have technical and conceptual support throughout faculty training workshops to ensure maximum understanding and participation. Faculty were provided access to log in to the DFP system on SAMS, and the faculty teaching load information was available in each DFP. Continuous evaluation and monitoring of the system was conducted regularly by recording the utilization of the system by faculty and the level of completeness of DFP.

Program Design

The implementation team used this feedback to understand better the faculty training infrastructure that must be in place for the faculty development program’s successful integration of the digital portfolio initiative. According to faculty data, more training is needed for the IT skills required to successfully apply the digital portfolio model. These results and additional supporting information demonstrated the necessity for faculty to receive training that would enable them to interact with the portfolio model within the framework of their instructional development. Additionally, it was agreed that faculty training would occur in groups to foster peer support and establish the foundation for long-term sustainability. Focusing on skill development in the context of integrating and implementing the digital portfolio model across courses was the goal when study owners set out to offer teacher training. Changes to the program were created in response to input from education experts, a digital portfolio specialist, and system supporters to give a holistic approach to organizing, planning, and communicating faculty training. Their assistance enables faculty to receive technological and conceptual support during the training sessions, ensuring that knowledge and engagement are maximized. This diversity created a sense of participation and responsibility across programs. The facilitation team’s inclusion of various stakeholders also made it possible for training to be tailored to each program’s specific requirements and objectives within the larger digital portfolio project. A digital faculty portfolio using intervention mapping integrated with the existing Student Academic Management System (SAMS) was then developed.

Faculty Training and Implementation of DFP

All faculty members received invitations to the faculty digital training workshops through their department chairs or relevant deans due to the observations and experiences of different stakeholders. A total of three-week, half-day workshops were conducted in the College of Medicine computer lab by the researcher, digital portfolio expert, and IT expert, who were one of the project owners with the support of the IT department. The facilitators emphasized a web page and artefact development techniques in all faculty development workshops. All full-time faculty members (male and female) at the College of Medicine (N=82) were included in the process. Faculty were informed of workshop goals and provided access to log in to the DFP system on SAMS, and the faculty teaching load information was made available in each DFP. A stepwise user guide was also prepared and distributed to all the faculty members (Figure 2). Continuous evaluation and monitoring of the system were conducted regularly by recording the system’s utilisation by faculty and the level of completeness of DFP.

Figure 2 User’s Guide.

Post-Implementation Survey

The participants in the faculty development opportunity were once again surveyed following the three-week workshop schedule to assess their involvement and success in the program. The evaluation included a quantitative survey and an open-ended call for user comments. In the quantitative study, we circulated an 8-question survey and collected responses by email. We also asked for comments and suggestions.

Statistical Analysis

Data were analyzed using the IBM Statistical Package for the Social Sciences version 24 (BM SPSS^® Statistics for Windows; IBM Corp, Armonk, NY, USA). Descriptive statistics, including frequencies and percentages, were used to summarize the responses generated from the study, as they were all categorical variables. Pearson correlation test was done to see the relationship between different domains of DFP. A p-value less than 0.05 level was considered significant.

When analyzing the qualitative data, we used theme analysis to draw attention to similarities and patterns in the remarks made by the participants about their experiences. In the initial stage, essential phrases were identified and labelled using open coding. Line-by-line checks were made to ensure that each data item was accurately labelled. These identifiers were later employed to create categories and themes. Due to the author’s review of the codes, subthemes and themes started to develop.

For qualitative data analysis, we used theme analysis to identify similarities and patterns in the participants’ comments about their experiences. First, significant phrases were identified, and relevant labels were assigned through open coding. Then, each line was carefully reviewed to ensure accurate labelling of each item, and these labels were used to establish categories and themes that were further translated into codes, subthemes, and overarching themes. Finally, the authors thoroughly reviewed the codes, subthemes, and themes to establish a better understanding.

Results

This report focuses on stages 1 (needs assessment) and 6 (evaluation) since these stages produced quantitative results suitable to the format.

Needs Assessment

All 82 faculty members participated in the surveys. Most participants understood the different domains of DFP (Table 1).

Table 1 Participants’ Responses on Need Assessment Survey Performance (Green), Motivation (Blue), Experience (Orange)

In the performance section, we found that users are generally confident in their work ethic and performance quality. For example, the statement “Compared to my colleagues, I am a relatively hard worker” had an agreement rate of 77%. Likewise, there was an 80% agreement rate with the statement, “I have a good understanding of my role in the department”. Moreover, faculty expressed a desire for more avenues for conveying their work for evaluation (81%) and for more incentives (80%) (See Table 1).

However, this section also revealed a lack of coordination across the curriculum, as 39% of respondents agreed or strongly agreed with the statement, “My classes are not integrated into a larger curriculum”. While this is a minority, it is a sizable minority, given the importance of an integrated curriculum. A similar question, inverted form, received almost the same response rate: only 58% of respondents agreed with the statement, “I have a good understanding of what others teach in my department”. Again, this is more than half, but it implies that a significant minority of respondents are unsure of their role within the broader curriculum.

Finally, this section revealed a widespread desire for improved evaluation metrics. Only half of the respondents (49%) agreed with the statement, “The department gives incentives for exceptional work”. A full 67% of respondents agreed with the statement, “I perform many services outside of teaching and research…that are not recognized by my department”. (See Table 1)

In the motivation domain, most of the faculty reported that teaching is a highly important part of their professional life (92%), they feel they excel in teaching (88%) and other aspects of work (86%) and are eager to improve teaching skills through classroom experience or professional development opportunities (83%). There are concerns about community services and research exposure; only 47% reported that departments are committed to research, and their job is to reach out to community services (53%) (Table 1).

The objective of the third section of the needs assessment survey was to know whether faculty members have technical skills, prior knowledge about portfolios, and willingness to use DFP in their teaching. Interestingly, most of the faculty members (81%) showed their eagerness for DFP and the positive effect of DFP (75%). On the other hand, a participant expressed their concerns about more time spent on creating DFP (39%), technical difficulty (23%), and its worth in teaching (23%). A few participants (31%) were not aware of the effectiveness of digital portfolios in teaching.

Additionally, we calculated the correlation between the domains of DFP and found a high association between the domains of motivation and evaluation (r= 0.310, P=0.05) and performance and motivation (r=0.508, P=0.01). There was no association between experience and other domains identified (See Table 2).

Table 2 Association Between Different Domains of DFP

Evaluation

The post-implementation evaluation survey consisted of only eight questions, which can be easily reported in full. Eighty-seven percent of respondents agreed or strongly agreed with the following statements: “DFP assesses my work adequately”; “DFP is user-friendly”; “I can see how DFP was relevant to my needs and responsibilities”; “The activities I mentioned in my DFP had a positive impact on me and my college”. Slightly fewer respondents (80%) agreed or strongly agreed with the following statements: “I can see the personal benefits of the DFP”; “The process of creating a DFP has enhanced my professional competency or skills”. Only 73% of respondents reported wanting to continue using/updating their DFP. Surprisingly, only 60% of users responded favourably to the statement “The DFP was useful for me”. Although a large majority reported that the DFP was relevant and adequately assessed their work, only a relatively slim majority reported finding the exercise useful (See Figure 3).

Figure 3 Evaluation responses.

In the open-ended comments section, respondents offered both encouragement and constructive criticism. After reviewing all survey responses, we identified three main themes: The benefit of using the DFP, technical issues in the platform’s design, and content suggestions for expansions to the content and scope of the DFP. Table 3 summarizes the comments on these themes.

Table 3 Thematic Analysis of Participants’ Comment

Discussion

This study aimed to develop a platform that integrates faculty activities, encompassing both teaching and extracurricular engagements, tailored to the specific needs of faculty members. It’s a unique initiative in the Arab world, which aims to identify the educational requirements of the DFP to enhance teaching abilities. This study showed that the creation of digital portfolios by faculty members positively impacts them in various ways, including a better understanding of their teaching process, reflective processes, learning about technology, and a reevaluation of their teaching methods. Research findings can be a tool to persuade teachers of the value and effectiveness of using new approaches, even though instructors do not always welcome their adoption with excitement. It is quite promising that the teachers who utilized DFP have found it simpler, easier to maintain, and more effective. This is supported by the opinions of 87% of teachers who support the usage of electronic portfolios.

The study demonstrated the potential advantages of digital portfolios for teachers and their students. Through examination of teachers’ responses, it became clear that teachers benefited from developing their digital portfolios because they were involved in a process of self-reflection and creation that expanded from a shared understanding that each person had similarly gone through this learning process. Literature suggests that in-service teachers’ use of digital portfolios and reflective practice were found to have favourable effects.¹⁵ Turner and Simon (2013) examined their master’s program, encouraging teachers to develop digital portfolios. They uncovered that creating digital portfolios urged participants to engage in reflective practice during and after the process.¹⁶

The needs assessment in the current study revealed two possible benefits of a DFP: improving cross-curricular integration and evaluation. As indicated in the “performance” section, only a few respondents feel that their teaching is integrated into a broader curriculum. Curriculum integration is one of the oldest ideas in the university pedagogy.^17,18 While it has some limitations,¹⁹ it remains a key objective for many departments worldwide. By encouraging faculty to publicize the details of their teaching, the DFP program can help address this gap at the College of Medicine.

Although it is documented in the literature that one of the advantages of utilizing a digital portfolio as a continuing, available archive of activities for professional development maintained by the teacher is the ability to show growth over time.²⁰ However, our needs assessment survey on the domain of the “performance” section revealed that respondents reported needing improved evaluation processes. The majority reported relatively low satisfaction levels with the departmental review processes they are subject to. This highlights an additional potential use for the DFP program that should be considered going forward. Using the DFP system for learning can be anticipated to improve the quality of teaching and, as a result, impact student learning and academic accomplishment. These benefits go beyond those for supporting professional learning and growth.²¹

In the implementation phase, we held training sessions to carry out the project’s goal through the model stages. The faculty workshops, in general, were successful in helping faculty members understand the digital portfolio framework and, more specifically, how to include prospects of artefact development in their teaching. The participants expressed gratitude for the chance and, more specifically, for the practice component of the workshops. According to Sprague, Kopfman, and Dorsey (1998), meeting the demands of faculty members is challenging because preconceived beliefs frequently undermine trainer goals.²² The faculty’s objectives and programmatic engagement in the training were a huge blessing for us. Facilitating faculty workshops was more enjoyable with active participation and a focus on common, institutionally supported initiatives.

While utilizing DFP, participants found numerous challenges that impeded the potential impact on teachers’ competencies regarding technical implementation and practical integration. The same issues were raised in earlier studies, which found that implementing the portfolio goal was complicated by several issues, concerns, and expectations.²³ The other most frequently cited challenges were the time spent developing a digital portfolio, system glitches, and data uploading. Other authors who have written about electronic portfolios have noted the same drawbacks.^5,24,25

Additionally, a significant percentage of faculty ratings of the impact of portfolios are extremely favourable. They view this tool as an effective technique to improve the quality of their work and have direct control over teachers’ motivation to teach and to use technology for professional growth. They emphasized the elements that could improve the impact and quality of DFP, such as adding new activities like counselling and administrative jobs. A study was conducted to determine that an academic faculty’s e-portfolio is a part of the quality management systems of the institution’s overall educational activity, and it defines the model as a prospective measure of the achievements of their activity and automates the creation of the internal rankings of the institution.²⁶ Based on this experience, we recommend creating cross-campus faculty training to promote the development of technological skills and technology integration to support education.^26,27

Limitations of the Current DFP Program

The needs assessment identified a role for the DFP program, and the evaluation survey was designed to determine if the current program fulfils that role. As the literature and our needs assessment survey indicate, technical support plays a vital role in successfully implementing the DFP. However, it’s important to note that the sample size and methodology of the study have limitations. Therefore, to enhance outcomes for future studies, a qualitative approach could be employed to meet the needs of all stakeholders, including students.

Evidently, the current program has a significant deficiency: users have reported difficulties with the attachment upload process. In future project extensions, we are determined to rectify this technical issue by addressing it as the core functionality of the DFP platform.

Conclusion

The DFP program has effectively addressed a need among the faculty community, which has led to new opportunities for communication and coordination and broadened the possible scope of the evaluation process, both of which are important aims identified in the needs assessment phase. The College of Medicine’s DFP program has successfully provided information for KPI calculation, tracking strategic plan initiatives, and supporting faculty evaluation and promotion processes. It also provides long-term benefits like a full-faculty database, an organized evaluation mechanism, and a transparent method for promotion. In the future, the DFP program is expected to generalize, include joint appointments and other colleges, calculate a faculty workload, and generate an automated annual report for faculty promotion and evaluation. This initiative would improve the faculty’s performance and motivation and encourage a sense of community at the College of Medicine.