Performance of medical students in final professional examination: Can in-course continuous assessments predict students at risk?

Introduction

Assessment drives students’ learning; it measures the level of students’ understanding. It is used to analyse the extent of knowledge acquired and their learning skills.^[1] There are different methods of assessment that include continuous or in-course assessment, formative assessments and final or summative assessments. While formative assessment provides feedback that assists students in the preparation of topics, continuous assessments are associated with a more distributed learning effort throughout the course and are thought to promote deeper learning, greater motivation, and consequently improved understanding of course material.^[2] They attempt to ensure that students do not wait for the end of the semester or term to make efforts to study.^[3]

Continuous assessments contribute to entry criteria for final examination placements and determine their impact on the overall performance of the student. The weight of the in-course continuous assessments (ICAs) from the final examination varies from 10% to 60%.^[1],[4],[5] The ICAs at International Medical University, Seremban, Malaysia are 3 sets of composite marks derived from course works, summative theory paper and clinical examination conducted 6- monthly in semesters 6, 7 and 8 in which students are graded as pass/ fail. The theory paper includes short answer questions and 1 best answer questions. The clinical examination includes end-of-posting practical examination in the form of single long case examination. These are conducted 6-monthly in semesters 6, 7 and 8; they are graded as pass/fail for each student. This contributes 30% to the final examination. The final professional examination including modified essay questions (MEQs), 18-question objective structured practical examination (OSPE) and a 16-station objective structured clinical examination (OSCE) which are graded as pass/fail. MEQs are short clinical scenarios that include series of questions with a structured format for scoring. The organization of knowledge, reasoning and problem solving are assessed. The OSPE includes 18 stations in which the students perform tasks such as ability to obtain/interpret data, solve the problem and communicate thus showing competency of skills and/or attitudes. The 16-station OSCE samples across a wide range of clinical competencies with 5 minutes of short OSCE and 15 minutes of long OSCE. The distribution of weightage for the assessments that contributes to 100% of the final score includes: ICA 30%, final professional examination MEQs 20%, OSPE 10% and OSCE 40%. Failure is defined as <50% of marks in the assessment component. We aimed to investigate the correlation between ICA and final examination results and whether the ICA predicts students at risk of failure that will facilitate early remedial measures.

Methods

ICA and final professional examination results of 3 cohorts of medical students (n=245) were retrieved from the examination unit of the International Medical University. Failure in ICAs to predict failure in each component of final professional examination was tested using chi-square test and presented as odds ratio (OR) with 95% confidence intervals (CIs).

Results

Failure in all ICAs was statistically significantly associated with failure in MEQ and OSPE, with OR of 6.9-25.1 [Table - 1]. In view of the small sample size, the confidence intervals of the ORs were rather wide. In contrast, there was no statistically significant relationship between failure in ICAs and long and short OSCE, with the exception of ICA for semester 6 and short OSCE. MEQ and OSPE assess mainly theoretical knowledge and problemsolving, while long and short OSCE assess clinical skills. The analysis shows that failure in ICA strongly predicts failure in MEQ/OSPE but not OSCE. It is likely the ICA, MEQ and OSPE assess fairly similar domain, i.e. theoretical knowledge.

Table 1: Association between performance in in-course assessment and final professional examination component

Discussion

The aim of any medical educator is to identify students requiring remediation at the earliest possible stage. This exploratory study investigated the potential predictive ability of ICAs in identifying ‘at-risk’ students early and at different stages of their studies that can allow attention, and possible remediation. Through the ICA system of assessment, students are continuously under assessment and in an ongoing process of learning, cognisance of which can be taken to predict students at risk of failure in their final examinations.^[6]-^[7] According to Yoloye, continuous assessment aims at getting the truest possible picture of each student’s ability and at the same time helps each student to develop his or her abilities to the fullest.^[8] The method or process of continuous assessment takes into account in a systematic manner the performance of students during a given period of time in medical school. Apart from all these, continuous assessment has the characteristics of being comprehensive by making use of many evaluation instruments, and cumulative by taking into consideration all past records to compute the final grades of the students.^[8]

We observed that the ICA scores in theory (MEQs and OSPE) and clinical examination (OSCE) correlated well with the final examination marks of students of all 3 semesters (6-8) with OR of 3.8-14.3 (all analyses p<0.001) and OR of 2.4-6.9 (p<0.05). This supports the good predictive validity of our ICA examinations. The correlation was stronger with MEQ and OSPE compared to OSCE. This can be attributed to external factors such as stress among students exposed to external examiners who have little or no contact with the students, resulting in decreased performance.

Although we did find ICA scores to be a valuable predictor of high-risk students, we do not expect to consider this as the only means of detecting all potential strugglers. Nevertheless, at this stage a workable system in the form of ICA scores for early detection and remediation of high-risk students may be valuable. When academic remediation is instituted at this stage it is may help to overcome the stigma of failure.^[9] Timely intervention for poor performance helps students deal with adverse learning and behaviour patterns promptly before it adversely affects their clinical performance. It is has also been observed that weak medical students become doctors, highlighting the importance of early identification of underperforming students.^[10]

Remediation measures for failing students have been developed in many medical schools. These measures are tailored to the nature of the problems students’ encounter and the availability of staff and their interest. Most of these measures involve 3 steps : (i) identification of the problem; (ii) implementing remedial measures; and (iii) assessing the effectiveness of remedial measures.^[11],[12] Most medical schools have remediation processes, developed usually on the basis of, for example, staff availability and interest. However, assessing the effectiveness of the measures is viewed with an opinion different from the literature. A study on asse ssment- focused revision and reassessment among poor performers in OSCE identified no improvement in subsequent performances; however, the data are derived from 1 medical school.^[11] The finding is different from few studies when remedial measures evaluation was done for theory or clinical examination and was found to be effective.^[13],[14] A recent systematic review by Cleland et al. found that the maj ority of remedial interventions are targeted at learners in the latter years of medical school.^[10] The remedial measures undertaken at our university include activities such as case presentations, small group discussions, tutorials and feedback, when the students have failed their professional examinations at the end of semester 9, which is close to the time of completing their course. We believe that early remedial interventions have the potential to improve the performance of many struggling students. Identifying struggling students with ICAs will help initiate early remedial measures, giving students a sense of control over their learning and performances. While remediation serves to improve strategic learning, resource management, internal motivation, there must be a place to tackle non-academic problems such as language, communication and inter-professional skills of the students. This will enable early remedial action to be in place by targeted individual academic and pastoral support.^[15]

Limitations

Our study was done at a single medical school and the results may not be generalizable to other medical schools with different curricula and student profiles. We are mindful that besides continuous assessment, there are independent variables such as learning styles, and pre-academic entry qualifications that we did not observe which may influence the final results. With regard to the predictive ability of pre-admission grades, there are mixed results with some studies showing low discriminatory power with individual mathematics, biology, or other science grades being more appropriate as potential predictors than comprehensive high school grades.^[16] This is contradictory to our findings that students with better grades in their pre-university examinations had better performance in their final examinations, regardless of the science subjects that they took at the pre-university level.^[17]

Conclusions

We have shown that there is definite correlation in the performance of medical students in ICA scores and final summative examination scores. More importantly, it allows medical educators to identify ‘at-risk’ students so that remediation measures can be undertaken. The earlier the students are ‘flagged’, we are more likely to obtain a positive outcome. We believe that we can do this during the early semesters of their clinical years and we are relatively confident of identifying ‘at-risk’ students 2 years in advance of the final exit examination. We believe that this correlation between ICA scores and the final examination scores is a true reflection of future student performance at our university. This predictor will assist medical educators to identify ' at-risk’ students at key points in the programme for effective remediation measures to be instituted. We propose that ICAs be developed to identify underperforming students so that medical schools initiate early remedial measures for students who fail and advise students to seek alternative careers despite adequate remedial interventions.

Acknowledgements

We sincerely thank International Medical University and the examination unit, Clinical School for their valuable assistance in keying in all the data in the SPSS format.

Conflicts of interest. None declared