Objective structured practical examination (OSPE) for formative assessment of medical undergraduates in Biochemistry

INTRODUCTION

Medical education is experiencing a broad re-evaluation with a major focus on the assessment of medical students. Assessment is the most important determinant of what students learn. It also serves as a tool for manipulating the whole education process. Assessment should be focused on the student’s potential for incorporation, application, and use of knowledge, and all the components should be considered in a scoring pattern.

The conventional practical examination (CPE) involves writing detailed procedures for the given experiments, followed by unobserved performance on the topic in the form of a viva voce.^1,2 This assessment method has several problems, especially in terms of outcome. It does not allow the examiner to assess the student’s skill, and scoring may be subjected to the examiner’s bias. Although marking should depend only on the student’s competence, variability in examiners and experiments selected affect grading in CPE. Moreover, the scores reflect the overall performance rather than individual competencies.³ Further, the subjectivity involved in CPE also affects the correlation between marks awarded by different examiners and the performance of the same candidate.⁴

In objective structured practical examination (OSPE), the process and the product are tested, emphasising individual competency. This prevents student and faculty variability, thus improving the validity of the examination.⁵

We felt that a structured approach to assessing practical skills in biochemistry is needed to provide strategies that faculty can use to enhance skill performance and increase training efficiency while assessing several objectives that must be implemented for conducting the practical assessment in biochemistry. To fulfill the desired targets, newer assessment methods, including OSPE, have been sought.

We evaluated the feasibility and acceptability of OSPE as a formative assessment tool for practical skills in biochemistry and compared the performance of medical undergraduates in OSPE and CPE. We also explored student and faculty perceptions regarding the use of OSPE as a learning and assessment tool in biochemistry.

METHODS

This prospective study was done in the Department of Biochemistry at Vardhman Mahavir Medical College and Safdarjung Hospital, New Delhi, India. OSPE was used as a formative assessment tool for Phase 1 MBBS students in Biochemistry of the 2020–21 batch. It included 170 students and 6 faculty members. The study was conducted after approval from the Institutional Ethics Committee.

Feedback questionnaires for the faculty and students were prepared, peer-reviewed and validated by experts. The questionnaire for students was prepared on a 5-point Likert scale and had open-ended questions. The statements were framed in cognitive, psychomotor and affective domains. Two open-ended questions were included to note their perceptions about OSPE. A separate validated questionnaire with a similar pattern was also designed for feedback from the faculty. Cronbach’s alpha of the questions administered had internal consistency, with a score of 0.83 for students and 0.89 for the faculty.

RESULTS

An analysis of mean scores obtained by students using the two methods suggested that students scored significantly higher marks in OSPE compared to CPE in all four assessments (Table I).

Student performance using the two methods was compared using the Bland–Altman plot (Fig. 1). The limits of agreement were defined as within 1.96 SD of the mean difference. We found an interval of –4.119 to 13.319 for competency-1 (glucose), –4.465 to 8.677 for competency-2 (urine), –5.304 to 11.681 for competency-3 (creatinine) and –5.607 to 13.63 for competency-4 (proteins).

Close

FIG. 1.

Bland–Altman plots for comparing students’ performance on objective structured practical examination (OSPE) and conventional practical examination (CPE) in glucose, urine, creatinine and protein, respectively

Export to PPT

The visual examination of the plots indicates global agreement between the two assessment methods.

Using the Likert scale, feedback from students and faculty about various aspects of OSPE was analysed. The median satisfaction score reported by the undergraduates was 4.2 (range 3–5). The mean score of each item in the questionnaire ranged from 3.4 (item 8) to 4.6 (item 11). Item 8 stated that OSPE is less stressful than the conventional method of assessment, while item 11 stated that feedback given after OSPE was helpful in clearing all doubts of students. The median satisfaction score of faculty about OSPE using the Likert scale was 5 (range 3–5).

Over 90% of students felt that questions asked in OSPE were relevant to assessing their skills and knowledge. Though 66.4% agreed that adequate time was provided for stations, 24.7% of students also stated that more time could have been given for procedure stations to complete the task. Only 66.5% of students responded that OSPE is less stressful than CPE, whereas more than 70% of students said that OSPE was easier to score marks, was more objective, and they were satisfied with the conduct of OSPE. About 81.2% of students also opined that such types of examinations should be included in other disciplines as well.

All the faculty involved in conducting OSPE felt that it was objective and uniform and helped to assess students’ practical skills better than CPE. They felt both OSPE and CPE should be used. OSPE was helpful for all levels of learners and helped them to score better. They felt that it should also be used routinely for practical assessments as it can serve as a learning tool through feedback given to students, which will help them improve their practical skills. They also agreed that the design of OSPE suited the expectations of the new CBME curriculum.

Themes were also identified for responses to open-ended questions regarding students’ and faculty’s perceptions of OSPE and its shortcomings over CPE (Table III).

DISCUSSION

Assessment is a method that decides the adequacy and effect of exercises considering their goals. In customary strategies for assessment, there are numerous inadequacies. Aside from the performance of students, different components, such as trial factors, instrument conditions and examiner factors, have a role in scoring. Furthermore, singular aptitudes are not assessed and an outcome is observed for scoring. The CPE in biochemistry does not evaluate the psychomotor performance and communication skills of students. Thus, most students are assessed for the cognitive domain and not for the psychomotor or affective domains.⁷ Moreover, the examination process is time-consuming, where the students are required to perform the practical in the first part followed by a viva voce. Students often complain about the subjectivity of the examination as well as variability in the questions asked in performance exercises which leads to variation in the scores.

To overcome these drawbacks, new models of assessment have been proposed. An earlier innovation in this regard is the objective structured clinical examination (OSCE) later extended to the practical examination (OSPE) described in 1975 and in greater detail in 1979 by Harden and his group from Dundee.^8,9 This method, with some modifications has stood the test of time and has largely overcome the problems of CPE mentioned earlier.

OSPE is useful for any subject and its advantage is that both the assessment procedure and the examinee are assessed. It helps to achieve the learning objectives developed for the competencies as laid down in the CBME. The domain assessed is the objective of the station. At the procedure station, the students’ psychomotor skills are assessed. At the response station, cognitive skills are tested. The examiner uses a checklist to record the performance. Standards to check the competencies are decided, and peer-agreed checklists are used for marking and evaluation.¹⁰ Students also take more interest and keep themselves alert during the entire assessment procedure. It also integrates teaching and evaluation through the feedback given to students at the end.¹¹ It also has a better discrimination index in spite of its drawbacks such as organizational training of more faculty members who are familiar with competency-based assessments, the requirement of more materials and manpower for its execution, standardization of OSPE stations and checklists in terms of external and internal validity is time demanding and proper planning. Though the organization of OSPE requires teamwork and logistics, at the same time, a large number of students can be tested with standard settings in a short period of time.

OSPE is becoming more popular among medical colleges in India. Electronic objectively structured practical examination (eOSPE) was introduced by Dutta et al.,¹² during the Covid-19 pandemic to facilitate the formative assessment. Cherian evolved a more practical method of administering OSPE.¹³ With advances in computer and software technology, it has now been named computer-assisted OSPE (COSPE).

Since OSPE has been endorsed for the functional evaluation of preclinical and paraclinical subjects, we endeavoured to test its feasibility and agreeableness as a formative assessment tool in biochemistry by contrasting it with CPE and furthermore by obtaining students’ and faculty perceptions for the same.

In our study, 170 undergraduate medical students’ practical skills in the analysis of abnormal constituents of urine, serum creatinine, serum proteins and blood glucose estimation were assessed with OSPE and CPE. The mean score of the students was significantly higher (p<0.0001) in OSPE as compared to CPE in all four practical competencies. A similar study has been reported by Mokkapati et al.¹⁰

Better performance in OSPE could be because the scoring, based on established standards of competence and a peer-agreed checklist, is objective. Examiner variability is reduced, which also affects the scoring in CPE. Studies by Rahman et al.¹⁴ and Menezes et al.¹⁵ also emphasized OSPE as a better assessment technique over traditional methods for measuring the practical skills of MBBS students in Physiology and Forensic Medicine, respectively.

Our study further supports the findings of these earlier studies, as favourable responses were obtained from the students regarding OSPE. Students in our study felt that OSPE assessed their relevant practical skills, and it covered the appropriate knowledge consistent with the learning objectives. A positive approach of students towards OSPE was observed. Interaction between students and teachers also increased, and students felt confident in performing practical skills after covering their areas of weakness. Feedback from students reflected that OSPE improved their practical skills, satisfaction with assessment, and rendered confidence in performing skills. The students appreciated the feedback provided at the end of OSPE and felt it to be an important factor in improving their learning.

Though most students were satisfied with the time given at individual stations, some students found it difficult to manage time at procedure stations and thus demanded more time at these stations, probably due to the lack of practice. This component of our study can be compared with the study done by Manjula et al.¹⁶

The lowest satisfaction index of 69.7 by students, for the question that OSPE is less stressful, indicates that examination is a well-known source of stress and anxiety and OSPE can be stressful, especially due to first-time exposure of students to such kind of examination at our institute. This component of our study can also be compared with the study by Manjula et al.¹⁶ Feedback from the faculty gave insight into their satisfaction and motivation to adopt OSPE as an assessment tool. In addition, the scoring of students at the procedure stations emphasized the need for a more comprehensive teaching– learning effort. Also, there is a need to formulate multiple OSPE stations with valid structured and standardized checklists to assess the range of psychomotor and cognitive domains, along with continuous faculty development programmes to acquaint them with innovations in medical education and technology.