The Feasibility of Conducting Safe Objective Structured Clinical Exams (OSCEs) During the COVID-19 Era

Introduction

Objective structured clinical examination (OSCE) is a tool used to assess students or residents on certain practical topics.¹ The goal of conducting OSCEs is to objectively assess the learner’s ability to apply learned clinical and practical skills in structured simulated cases that resemble real ones.^2,3 An OSCE typically assesses students’ ability to take diagnose patients, prognosticate disease states, take a thorough medication history, perform patient counseling, or deliver a pharmacotherapy or non-pharmacotherapy plan to a simulated patient or mannequins while being evaluated by an expert in the field of intended OSCE topic during pre-specified timeframe.⁴ In Saudi Arabia, OSCEs are part of high-stake clinical pharmacy credentialing at the Saudi Commission for Health Specialties. Internationally. It has a crucial role for entry-to-practice licensing examinations at the United States Medical Licensing Examination, and the Canadian Pharmacist Qualifying Examination.^5–7 A new trend towards using OSCE for pharmacy education was observed over the past few years evaluate clinical competency for pharmacy students.³ The most challenging aspect of conducting OSCEs appropriately is the lack of logistical support and poor planning. This type of assessment has been conducted physically since 1979 in medical schools which later was adopted by pharmacy schools as well.^2,3 Owing to the rapid spread of COVID-19 in early 2020 and concerns about the safety of the public, the Saudi Ministry of Higher Education announced that classes would be postponed until further notice. On March 11, 2020, the World Health Organization (WHO) classified the COVID-19 outbreak as a pandemic.⁸ This urged the country to use machine learning applications (Tawakkalna app and Tabaud app) in addition to other strict public health measures to control the spread of COVID-19.^9,10 Conducting OSCEs is challenging during the COVID-19 era due to concerns about the spread of the virus through contact with contaminated surfaces and infected individuals. Many schools have either abandoned OSCE entirely or tried to conduct OSCE virtually during COVID-19 pandemic.^11–19 Despite the obvious concerns for COVID-19 transmission, our college has been leading pharmacy education in Saudi Arabia with high standards in education quality. Our goal is to maintain the same assessment modalities as much as possible to avoid compromising our outcomes.

The College of Pharmacy (COP) at King Saud bin Abdulaziz University for Health Sciences (KSAU-HS) is a frontier college of pharmacy in Saudi Arabia and has received full program accreditation by the National Center for Academic Accreditation and Evaluation for its Doctor of Pharmacy program. Patient assessment is an OSCE-dependent course which is delivered by the faculty of the pharmacy practice department to second-year pharmacy students. The patient assessment course is developed to primarily focus on the identification, interpretation, and assessment of drug-related problems in various disease states. It is a skill-based course that focuses heavily on critical thinking, problem solving, and hands-on assessments in a case-based approach. The course consists of didactic lectures, which are supported by topic discussion and practical skills lab to enrich and widen students’ comprehension. Mock patient interviews and roleplaying are among the teaching strategies utilized to further consolidate the course objectives. The assessment of this course includes continuous and end-of-course examinations that utilize multiple assessment modalities to ensure the appropriate assessment of course learning outcomes. OSCEs accounted for 35% of the course’s overall grade. OSCE stations were developed to assess students’ practical skills in history taking and physical examination and provide appropriate patient-case evaluations. Given that OSCEs are integral aspects of the patient assessment course, we worked on a protocol that would enable the students to participate in these assessments while simultaneously providing a safe environment for the students and examiners without compromising the integrity of the exam. This study aims to report students’ satisfaction on their OSCE experience in addition to describing an experience of the college of pharmacy in conducting physical OSCE during COVID-19 era.

Materials and Methods

This was a descriptive study by comparing the 2019–2020 cohort to the 2020–2021 cohort in addition to a descriptive framework for the feasibility of conducting physical OSCEs in the college of pharmacy at King Saudi Bin Abdulaziz University for Health Sciences in Riyadh, Saudi Arabia. This study was exempted from undergoing an institutional review board approval at our institution due to the availability of an access to a non-identifiable data and our descriptive nature for the experience.

During the conduction of patient assessment course, OSCEs are part of the end-of-course assessment. Students are provided with an opportunity to have two practical sessions the week before their OSCEs to practice their skills on the topics covered during didactic and practical sessions, which include basic pharmacy practice skills such as the proper use of inhalers and glucometers, measuring blood pressure, and appropriate patient-case evaluation by utilizing the correct risk assessment tools for each disease state.

The College of Pharmacy is equipped with a high-fidelity mannequin (SimMan 3G PLUS^®), three training non-high-fidelity mannequins, fully-equipped patient assessment lab with stethoscopes, arm simulator for practicing blood pressure measurement, multiple types of glucometers, insulin vials, alcohol swabs, and other necessary supplies. This course is taught by practicing clinical pharmacists with the support of lecturers, administrative staff, and lab technicians. OSCEs are conducted in the KSAU-HS simulation center, which is fully equipped with the necessary materials, in coordination with the COP to provide the logistics and manpower support needed for examinations. All OSCE stations were prepared by the COP’s faculty using a validated OSCE form. Prior to the COVID-19 pandemic, the course had ten OSCE stations that tested different skills that students were trained on throughout the course. Thirty-seven personnel were required, including OSCE examiners, simulated patients, administrative staff members, and course coordinators. Students were divided into subgroups for each OSCE round, with rest and waiting areas available.

The measures included utilizing applications (apps; Tawakkalna and Tabaud) that use artificial intelligence to predict those who had recently been exposed to personnel infected with COVID-19.^9,10 Additionally, students were divided into smaller groups in multiple holding areas and social distancing was strictly implemented. Students were asked to leave the building as soon as they finished their OSCE round. Each OSCE room was ventilated appropriately and the examiner, student, and simulated patient followed proper hand hygiene and social distancing norms (if needed). The entrance to the simulation center was guarded by a security officer/administrative staff member who asked students whether they had displayed any symptoms of COVID-19 and checked their status on Tawakkalna and Tabaud. Students were instructed to exit the simulation center from a different route to minimize human traffic near the building doors.

We also examined students’ satisfaction and compared the responses of the 2019–2020 cohort and the 2020–2021 cohort. A validated psychometric questionnaire was used to assess students’ satisfaction with the patient assessment course using three different thematic questions: initial course assessment (three questions); course performance by teaching faculty members and course coordinators (20 questions), which contained three questions pertaining to the practical part of the patient assessment course (including the OSCE); and one question to assess their overall satisfaction based on a Likert scale. The authors utilized this questionnaire the COP Quality Assurance and Academic Accreditation Unit confirmed its validity and reliability. Data were analyzed using Microsoft Excel – 97.

Results

Comparison of Students’ Satisfaction

All 184 (96 and 88 in 2019–2020 and 2020–2021, respectively) students who participated in the patient assessment course during the 2019–2020 and 2020–2021 academic years responded to our student satisfaction survey. Students’ overall satisfaction with the patient assessment course improved when using virtual education in comparison to classic education strategies (physically attending the course). This effect was primarily driven by female students, whose reported satisfaction increased from 3.20 to 3.85; which reflects a significant difference when comparing the two batches; p < 0.05. However, male students reported a relatively lower satisfaction during the 2020–2021 academic year. Male students reported a higher satisfaction than female students (3.33 vs 3.20; p <0.05) during the 2019–2020 academic year. However, female students reported a high degree of satisfaction than male students during the 2020–2021 academic year. Moreover, with regard to the skill-oriented questions in the questionnaire, the female students in both cohorts reported a higher level of satisfaction than male students.

OSCEs in the COVID-19 Era

The COP formed a COVID-19 task force with the goal of implementing COVID-19-related policies and procedures made by the KSAU-HS COVID-19 Crisis Management Committee in collaboration with the KSAU-HS COVID-19 Infection Prevention and Control Advisory Sub-Committee to ensure adherence with best practices and facilitate the flow of education and exams. The COVID-19 task force was responsible for following exposure plans made by the university to help reduce the risk of COVID-19 transmission in the COP and to assess any suspected COVID-19 cases (Figure 1, Figure 2 and Table 1). In addition, the COP continuously communicated with the infection control department and informed it about any COVID-19 positive cases in the college to help keep track of the number of cases as it may instruct the college to close temporarily for the safety of students and employees. One of the most important tools used to ensure the safety of the students was the Tawakkalna app. It was developed by the Saudi Data and Artificial Intelligence Authority in a bid to control the spread of COVID-19. During the curfew and when the conditions gradually return to normal, the app served as a tool to track and record the health status of the residents of Saudi Arabia. It is mandatory for the residents to use this app, and it is linked to individuals’ ID and health records.⁹ The app indicates whether a user is healthy, exposed, or infected via implementing validated machine learning principles. When users flagged as “exposed”: it means they came in close contact within 6 feet nearby a person labeled as “infected” per their electronical status using their phone device data without disclosing any personal health information. The national regulation requires any personnel who gets flagged as “exposed” to have COVID-19 swab done within 24 hours in the nearest health institution. The COP is almost 2 minutes away from our affiliated tertiary hospital with dedicated clinic for suspected COVID-19 cases which provided free of charge to all citizens. Each student was required to pass through the visual screening point to check their temperature and show their health status on the app. Students were also required to strictly follow social distancing measures by following the signs and bringing their own pencils, calculators, and hand sanitizers. In addition, the Tabaud app was also used. Tabaud notifies the user whether there is a confirmed case of COVID-19 nearby. Users can voluntarily share their COVID-19 test results with those they came into contact with within the past 14 days, and they will be notified if they have been exposed to an infected individual.¹⁰

Table 1 Matrix of Objective Structured Clinical Examination Before and During COVID-19

Figure 1 The process of conducting patient assessment course before COVID-19.

Figure 2 The process of conducting patient assessment during before COVID-19.

While overseeing the students’ entrance to the building for the exam, three cases were referred to the COVID-19 committee because of their suspected exposure, which was noticed by the security officer while checking the Tawakkalna app. These students were redirected to an isolation room to evaluate their cases and to ensure the safety of the others. Based on the alert on the Tawakkalna app, the students were found to have different degrees of exposure varying from household to non-household exposure. As per the protocol, the students were required to be isolated for 14 days, and all their exams were to be rescheduled. The students were informed that a makeup exam would be scheduled for them after the end of their home isolation period. The students left the building after a clear path was provided to ensure the safety of other students and employees.

Discussion

Recently, many healthcare-oriented schools have suggested using a virtual platform to conduct OSCEs in order to minimize the risk of the spread of COVID-19. Other schools suggested omitting or postponing all practical sessions for the same reason. While these decisions may have been the only choice in some countries due to their respective lockdown policies, some healthcare-oriented schools made these decisions internally without any government-enforced restrictions.^11–18 This study presented the feasibility of conducting OSCEs during the COVID-19 era.

Given how courses that incorporate OSCEs have been conducted, it was expected that the exam would be affected by the COVID-19 pandemic. The Weill Cornell College of Medicine shared their experience of implementing a Web-OSCE using online teleconferencing platforms. They used Zoom to conduct their OSCE and used WhatsApp as a backup strategy in the case of connectivity issues.¹⁹ Other schools used different strategies to conduct high-stake OSCEs virtually using different technological modalities. However, we believe that the virtual implementation of high-stake OSCEs may be insufficient for assessing students’ competency in the assigned practical skills.^2,3 We thus conducted in-person OSCEs in a highly controlled environment in order to deliver the best experience to our students during this crisis. Despite reducing the overall number of OSCEs in 2020 as compared to 2019, we utilized more human resources and materials to facilitate this experience. Conducting safe and effective OSCEs necessitates paying extra attention to operational aspects such as the building’s entrance and exit, in addition to ensuring the provision of holding areas with proper ventilation and the implementation of social distancing policies. Additionally, all examiners, simulated patients, administrators, and students were instructed to wear masks at all times. No vaccine was available when this activity was conducted. However, it would be a safer option to henceforth ensure that all participants are vaccinated to reduce the risk of the transmission of COVID-19.^20,21 This strict process ensured that the exam was conducted safely. We believe that following the strict protocol and the instructions provided by the Saudi government, which included the use of the Tawakkalna app, facilitated the process of conducting OSCEs safely. Overall, the success of this activity was evident through the students’ grades and responses to the satisfaction survey. Additionally, the students were cooperative and willing accommodate the change to ensure their safety and to successfully pass the exam. In order to conduct the OSCEs assessments in a safe and effective manner, the COP conducted all OSCEs on the same day. Moreover, the number of personnel was decreased to 24 as simulated patients were replaced with written scenarios to further reduce interpersonal contact. The COP decided to reduce the number of OSCEs to seven in order to reduce interpersonal contact and the risk of transmission without compromising the quality of the course. The COP implemented strict precautionary measures that were to be followed prior to, during, and after the students’ OSCEs. This strict process may be costly and could be replicated in other countries with similar strict public health measures.

The following describes the measures and resources needed to conduct safe OSCEs:

• Minimize the number of OSCEs so as to evaluate only the essential skills and reduce interpersonal contact during the examinations.
• Follow a strict policy on hand washing/sanitization techniques before, during and after students leave the premises.
• Follow a 2:1 ratio for students to faculty/admin staff/simulated patients.
• Utilize apps that track individuals’ exposure to COVID-19 (if possible) prior to having them on campus.
• Utilize appropriately ventilated areas that allow for social distancing as OSCE rooms and holding areas.
• Ensure the provision of sufficient thermometers to minimize the waiting time for students during their entrance.
• Utilize different controlled doors for entrance and exit with clear instructions on their next station to avoid crowding.
• Ensure participants remain masked throughout the process
• Prohibit food and drinks in the holding areas
• Devise a back-up plan to be implemented when students are found to have close contact with a confirmed infected individual.

Moreover, the students reported a higher overall satisfaction in 2020–2021 than in 2019–2020. Interestingly, female students’ increased satisfaction was the driving force for the combined effect, despite their lower average grades in this year’s OSCEs as compared to their male counterparts. This effect could be a result of the fact that the process was effectively organized. The male students reported a similar level of satisfaction across both years. However, female students were more satisfied with the course in 2020–2021 compared to 2019–2020. Similarly, regarding skill-oriented questions and overall course satisfaction in the validated survey, female students in 2020–2021 were more satisfied with the course than both their male counterparts and the 2019–2020 female cohort.

Many limitations may arise when using our strategy for conducting OSCEs during the COVID-19 era, the most serious of which would be an individual contracting COVID-19 during the process. Additionally, our process included the utilization of considerable human resources and materials that would be financially challenging for certain universities to acquire. In such cases, offering a virtual OSCE would be a better fit. Nevertheless, this process would require immense oversight from the course coordinators and college administration to ensure the effective implementation of the steps and addressal of any unexpected issues that emerge prior to, during, or after the OSCEs. The COP utilized a COVID-19 taskforce that included two infectious disease experts to mitigate and solve any clinical issues that emerged during these exams, and it received the complete support of the deanship and university administration. A clear communication from the university was in-place as a contingency plan for any suspected or confirmed case within campus. The delegation of responsibility and close observation allowed for better communication and coordination throughout the examination period. More data would be beneficial to explore other operational challenges from different universities in conducting OSCEs during the COVID-19 pandemic.

Conclusion

Students satisfaction increased despite the overall difference in the education modality. This study reveals the feasibility of conducting physical OSCEs for pharmacy schools during the COVID-19 crisis in countries with strict public health measures such as Saudi Arabia. There were no reported cases of COVID-19 transmission. Other universities may adopt a similar approach so as to provide an optimal educational experience while ensuring the safety of their staff and faculty. However, more coordination and administrative support are needed to ensure the implementation of appropriate OSCEs during the COVID-19 era.