Evaluation of a mentored student project programme using reflective summaries

Introduction

Undergraduate research experience has become increasingly relevant for today’s medical students, considering the professional requirements of their challenging future. Many medical schools worldwide have included a research component in the undergraduate medical curriculum. Research skills and attributes are cardinal for medical students in their future clinical practice, for appropriate decision-making and patient safety.^[1],[2] Engaging in research has helped students improve their problemsolving skills and have enabled them to become more confident to undertake research in their future career.^[3] Students’ research experience has been reported to enhance their research skills,^[4] relationship with mentors,^[4] their interest for enquiry-driven learning and production of publications and presentations in academic meetings. Shaw et al.^[5] reported enhanced research preparedness and confidence in their ability to undertake research-oriented tasks in fourth year undergraduate students enrolled in a research programme. In a study undertaken as part of the Undergraduate Learning in Science Project, Ryder^[6] reported an improvement in time management, communication and personal organization skills of students. Some medical schools have initiated the Summer Program for International Research Internship and Training and Local Research and Training Program, to facilitate, reinforce and support student research through opportunities for collaborative research and incentives for conducting research.^[7]

Questionnaire surveys have been used mostly to evaluate research programmes.^[8] There is a paucity of published findings on evaluation of research programmes using reflective summaries or reflections by programme stakeholders. The Kirkpatrick programme evaluation model,^[9] which has been widely used to evaluate educational programmes, focuses on four levels of programme outcomes: (i) learner satisfaction regarding the programme; (ii) attributes of learning (knowledge, skills and attitude); (iii) learner behaviour; and (iv) results of the programme in a larger context.

Reflective ability is a relevant attribute for healthcare professionals to achieve professional excellence.^[10] Reflection is the process of thinking about prior experiences so that an increased realization of one’s own thoughts and behaviour is achieved.^[11] In the medical education context, reflection is a means towards effective learning.^[11] Reflective journals have been used by some educators to facilitate reflection in problem-based learning.^[12] Reflection journal writing is reported to enhance students’ critical reviewing processes of their learning experiences and help them transform that learning into practice.^[13] We describe the evaluation of a mentored student project (MSP) programme at Melaka Manipal Medical College (MMMC), using students’ reflective summaries.

Methods

Educational context

Melaka Manipal Medical College, Manipal Campus, Manipal Academy of Higher Education, caters to the needs of international students, mainly from Malaysia and Sri Lanka, by offering a twin-mode MBBS programme. Students with a Grade Point Average score above 3 are admitted and they spend 2.5 years of preclinical training at Manipal, India, and the remaining clinical training at Melaka, Malaysia. The MSP programme is among the best practices that have been implemented since 2007. The primary objectives of this programme are to inculcate among undergraduate medical students the relevant knowledge, skills and attitude pertaining to research so that they are sufficiently trained for future research endeavours. In MSP, students undertake a short-term (6 months) group (n=5) research project under the guidance of their mentor. Each MSP group is required to get the approval of the institutional research committee for their protocols before they proceed with data collection and analysis, abstract writing, poster presentation of their research work and finally individual reflective summary writing of their research experience. Students are provided timely orientation on the relevance and objectives of MSP, research methodology, abstract and reflective summary writing and poster presentation by members of the coordination committee. Our study was approved by the Institutional Ethics Committee.

Applying the eclectic model as an approach to teach reflection

Students were taught reflective summary writing using the Koole et al. eclectic model^[14] as the framework. Phase 1 of this model involves a review of MSP experience by identifying the important components of the programme and to describe one’s own thoughts and feelings. Phase 2 is critical analysis wherein students write about the knowledge, skills they gained, the change in attitude through MSP which they experienced and describe how those changes happened. Phase 3 is to draw conclusions from the MSP experience and how they plan to apply the learning to future research experiences.

It has been stated that research-related outcomes have not been given sufficient attention in undergraduate medical education.^[15],[16] Educators emphasize that research outcomes should be identified and defined clearly in medical curricula.^[17] Thus, the short-term outcomes of the MSP programme identified were (i) improved knowledge, skills and attitude regarding research in students; (ii) improved grades in performance appraisal for faculty members; and (iii) publications and conference presentations of research findings.^[18] We report the findings of the MSP programme evaluation using reflective summaries of a batch of students. The findings related to the first programme outcome only are described in this article.

Analysis of reflective summaries

All reflective summaries were graded by the two authors individually using a rubric.^[19] A consensus was reached to grade the reflective summaries as A, B and C. To provide in-depth information about programme outcomes, data from 41 reflective summaries (21 female and 20 male students) were analysed qualitatively using the thematic analysis approach. The learning outcomes at the third (learner satisfaction regarding the programme) and the fourth (attributes of learning: knowledge, skills and attitude) levels of the Kirkpatrick evaluation model were determined from the summaries. Initially, both researchers coded the reflective summaries individually and identified a set of 7 and 9 themes. Then by consensus the themes were reduced to 4 themes. It took us 10–15 minutes to analyse each reflective summary.

Results

Learner satisfaction regarding MSP

In general, students were satisfied with their MSP experience, which they reflected as a very useful experience through which they gained various generic skills in addition to research skills [Table - 1].

Table 1: Students’ responses (n=41) regarding their satisfaction about the mentored student project (MSP)

Knowledge and skills gained through MSP and a change in attitude towards research

Four themes [Table - 2] related to the first programme outcome emerged from the analysis of reflective summaries.

Table 2: Summary of students’ reflections pertaining to the themes on programme outcomes

Teamwork skills

Students commented that the teamwork through MSP helped them to deal with lack of self-motivation towards projects, to deal with depression, to manage language barriers encountered with data collection of some projects and to identify solutions for problems.

Communication skills

Students felt that MSP provided them an opportunity to improve their communication skills, as they had to communicate with the study sample, which included students, patients, the general public and faculty.

Change in attitude towards research

Students opined that MSP helped them to attain a positive attitude towards research as they realized that research work demands commitment, hard work, honesty and patience to deal with different people and contexts. They realized the importance of deadlines and commented that deadlines were essential without which they would have procrastinated the work. They also felt that their MSP experience was a learning phase that could not be obtained from textbooks. They commented that they wished to do research in future and would like to serve the society through research, as MSP had improved their confidence to conduct further research.

Improved mentor–student relationship

Students responded that through MSP their relationship with their mentor improved, as mentors constantly motivated them to complete tasks on time, monitored progress of the study, provided assurance that they were on the right track, provided constructive feedback and gave quick solutions to problems.

Students also commented about a few strengths of MSP as well as a few challenges they faced during the programme.

Strengths

Students felt that MSP broadened their knowledge and trained them for out-of-the-box thinking, taught them not to procrastinate work, helped them in capacity building, improved their practical knowledge in terms of dealing with different situations, helped them to realize their learning gaps, taught them discipline and to do tasks in an organized way. They also felt that the programme itself was well organized and commented that poster presentation was the best experience in MSP.

Challenges

Students reported that they did face a few challenges while preparing the documents required for the approval of their projects by the ethics committee, less cooperation from some group members, initial anxiety in balancing studies and research, resistance to work in groups and apprehension due to a lack of previous experience in conducting research.

Discussion

Nurturing generic skills in medical students is an important task associated with curriculum development in medical schools. These skills enable students to perform duties meticulously and judiciously in their future workplace. Educators argue that training in generic skills should be embedded in the curriculum rather than making it a standalone activity and opportunities should be created in the curriculum to support these generic skills. The MSP programme of our institution intends to nurture generic skills among students through research.

Proficiency in scientific research is an additional role expected from teachers in education of health professionals,^[20],[21] besides the 12 roles mentioned by Harden and Crosby.^[20] Consequently, medical schools have incorporated curricular components intended to nurture research skills during undergraduate training. We have described students’ reflections on the MSP programme initiated to inculcate research skills in undergraduate medical students and how the reflections were used to evaluate the programme outcomes using the Kirkpatrick evaluation model.

Reflection is reported to be an effective teaching–learning tool that could be implemented at any level of education to augment personal development and improve learning.^[23] It has been used as a path to help students achieve a particular learning outcome, for example in problem-solving,^[12],[14] to improve communication skills,^[24],[25] and to develop professionalism.^[26] The reflection, which could happen before, during and after a learning experience, is a guiding factor for lifelong learning. It is also an attribute of self-regulated learning, which in turn helps learners to develop a deeper approach to learning and thereby improved academic performance.^[11] Studies have described how to teach students reflection, compared different methods that engaged students in reflection and the role of feedback in enhancing reflection. Studies have reported that learners engage in reflection through writing exercises. All these means of reflection create an increased self-awareness and self- improvement. This study adds to the already existing information on undergraduate research by providing a basis for the use of reflective summaries in evaluation of research programmes.

In a previous questionnaire survey conducted by us in 2011–12 on MSP programme evaluation, we reported that the MSP experience culminated in an improvement in students’ knowledge and skills pertaining to research and paved the way for a more positive attitude towards research.^[18] In the present study too, the analysis of reflective summaries from a different batch of students revealed that students were satisfied and convinced about the positive attributes of the programme. The themes that emerged from the analysis of reflective summaries were in alignment with the programme objectives and outcomes.^[18]

Durairajanayagam et al. reported that participants of the summer internship course at Cleveland clinic felt that teamwork was the most important attribute of the programme wherein they were given opportunities to engage in research.^[27] In our study too, all reflective summaries had a mention about an improvement in teamwork skills through MSP. Improved relations with mentors constitute a relevant outcome that has been achieved through MSP, as effective mentoring could augment the scope of students’ confidence to engage in research.^[28] Improvement in communication skills and a positive change in attitude have been reported previously with the MSP programme,^[18] which was reaffirmed by students of a different batch.

In our previous study,^[18] some of the challenging factors in undertaking MSP reported were: time constraints, difficulty in finding topics, inadequate guidance from mentors, uncooperative group members, misunderstanding among group members and the absence of sound knowledge regarding research methodology. The present study sample provided more positive responses, emphasizing the various generic skills gained by them during MSP and also echoing their willingness to undertake research in future. The support they received from mentors resonated in all summaries, which could be attributed to the increased importance mentors assign for MSP and the commitment shown by mentors to their mentees. This positive change in attitude of mentors could be due to the mentor orientation provided by the programme coordinators, wherein mentors are oriented on the relevance of MSP, followed by the provision of elaborate and well-defined guidelines on MSP.

Strengths and limitations of the study

A major strength of our study is that reflective summaries allowed the MSP coordination committee to get a broader view of the challenges of the programme, which would not have been possible by a questionnaire survey. Based on the feedback from students, the committee incorporated a few changes to improve the programme. The duration of the programme has been increased with the topic selection starting from year 1 itself. To ensure uniform participation of all students, MSP assessment has been introduced, wherein students are assessed by their mentors on their active participation in research, teamwork and communication skills. Mentors are also oriented regarding the programme requirements and also on their role in MSP. A major limitation is that the study findings cannot be generalized as only a limited number of reflective summaries were analysed.

Conclusions

This study reaffirms the feasibility of research in a packed undergraduate medical curriculum and also provides an example of how reflective summaries could be used for evaluation of a research programme. The study is an examplar of an easy, feasible and cost-effective method of evaluation of a research programme. By using the qualitative approach for data analysis, the programme coordinators could obtain a broader perspective about the strengths and barriers of the programme and could incorporate a few changes for improvement.

Conflicts of interest. Nil