Standard setting of objective structured practical examination by modified Angoff method: A pilot study

Abstract

Background. The undergraduate curriculum at our institution is divided system-wise into four blocks, each block ending with theory and objective structured practical examination (OSPE). The OSPE in Physiology consists of 12 stations, and a conventional minimum score to qualify is 50%. We aimed to incorporate standard setting using the modified Angoff method in OSPE to differentiate the competent from the non-competent student and to explore the possibility of introducing standard setting in Physiology OSPE at our institution. Methods. Experts rated the OSPE using the modified Angoff method to obtain the standard set cut-off in two of the four blocks. We assessed the OSPE marks of 110 first year medical students. Chi-square test was used to compare the number of students who scored less than standard set cut-off and conventional cut-off; correlation coefficient was used to assess the relation between OSPE and theory marks in both blocks. Feedback was obtained from the experts.
Results. The standard set was 62% and 67% for blocks II and III, respectively. The use of standard set cut-off resulted in 16.3% (n=18) and 22.7% (n=25) students being declared unsuccessful in blocks II and III, respectively. Comparison between the number, who scored less than standard set and conventional cut-off was statistically significant (p=0.001). The correlation coefficient was 0.65 (p=0.003) and 0.52 (p<0.001) in blocks II and III, respectively. The experts welcomed the idea of standard setting.
Conclusion. Standard setting helped in differentiating the competent from the non-competent student, indicating that standard setting enhances the quality of OSPE as an assessment tool.