Translate this page into:
Evaluation of a mentored student project programme using reflective summaries
2 Department of Physiology, Melaka Manipal Medical College, Manipal Academy of Higher Education, Manipal Campus, Manipal 576104, Karnataka, India
Reem Rachel Abraham
Department of Physiology, Melaka Manipal Medical College, Manipal Academy of Higher Education, Manipal Campus, Manipal 576104, Karnataka
|How to cite this article:|
Devi V, Abraham RR. Evaluation of a mentored student project programme using reflective summaries. Natl Med J India 2020;33:102-106
AbstractBackground. Undergraduate research experience has become increasingly relevant for today’s medical students, considering the professional requirements of their challenging future.
Methods. In the mentored student project (MSP) programme at Melaka Manipal Medical College, students undertake a short-term group research project under the guidance of their mentor. After data collection and analysis, students are required to write an abstract, present a poster and also write individual reflective summaries of their research experience. We evaluated the MSP programme using reflective summaries of a batch of undergraduate medical students. Data from 41 reflective summaries were analysed using the thematic analysis approach. The learning outcomes at the third and fourth levels of the Kirkpatrick evaluation model were determined from the summaries.
Results. Students’ reflective summaries indicated that they were satisfied with the MSP experience. In all the summaries, there was a mention of an improvement in teamwork skills through MSP. Improved relations with mentors were another relevant outcome. Improvement in communication skills and a positive change related to research attitude were also reported by students.
Conclusions. Reflective summaries as a means to evaluate the MSP programme was found to be an easy, feasible and cost-effective method. The qualitative approach adopted for data analysis enabled the programme coordinators to assess the strengths and barriers of the programme.
Undergraduate research experience has become increasingly relevant for today’s medical students, considering the professional requirements of their challenging future. Many medical schools worldwide have included a research component in the undergraduate medical curriculum. Research skills and attributes are cardinal for medical students in their future clinical practice, for appropriate decision-making and patient safety., Engaging in research has helped students improve their problemsolving skills and have enabled them to become more confident to undertake research in their future career. Students’ research experience has been reported to enhance their research skills, relationship with mentors, their interest for enquiry-driven learning and production of publications and presentations in academic meetings. Shaw et al. reported enhanced research preparedness and confidence in their ability to undertake research-oriented tasks in fourth year undergraduate students enrolled in a research programme. In a study undertaken as part of the Undergraduate Learning in Science Project, Ryder reported an improvement in time management, communication and personal organization skills of students. Some medical schools have initiated the Summer Program for International Research Internship and Training and Local Research and Training Program, to facilitate, reinforce and support student research through opportunities for collaborative research and incentives for conducting research.
Questionnaire surveys have been used mostly to evaluate research programmes. There is a paucity of published findings on evaluation of research programmes using reflective summaries or reflections by programme stakeholders. The Kirkpatrick programme evaluation model, which has been widely used to evaluate educational programmes, focuses on four levels of programme outcomes: (i) learner satisfaction regarding the programme; (ii) attributes of learning (knowledge, skills and attitude); (iii) learner behaviour; and (iv) results of the programme in a larger context.
Reflective ability is a relevant attribute for healthcare professionals to achieve professional excellence. Reflection is the process of thinking about prior experiences so that an increased realization of one’s own thoughts and behaviour is achieved. In the medical education context, reflection is a means towards effective learning. Reflective journals have been used by some educators to facilitate reflection in problem-based learning. Reflection journal writing is reported to enhance students’ critical reviewing processes of their learning experiences and help them transform that learning into practice. We describe the evaluation of a mentored student project (MSP) programme at Melaka Manipal Medical College (MMMC), using students’ reflective summaries.
Melaka Manipal Medical College, Manipal Campus, Manipal Academy of Higher Education, caters to the needs of international students, mainly from Malaysia and Sri Lanka, by offering a twin-mode MBBS programme. Students with a Grade Point Average score above 3 are admitted and they spend 2.5 years of preclinical training at Manipal, India, and the remaining clinical training at Melaka, Malaysia. The MSP programme is among the best practices that have been implemented since 2007. The primary objectives of this programme are to inculcate among undergraduate medical students the relevant knowledge, skills and attitude pertaining to research so that they are sufficiently trained for future research endeavours. In MSP, students undertake a short-term (6 months) group (n=5) research project under the guidance of their mentor. Each MSP group is required to get the approval of the institutional research committee for their protocols before they proceed with data collection and analysis, abstract writing, poster presentation of their research work and finally individual reflective summary writing of their research experience. Students are provided timely orientation on the relevance and objectives of MSP, research methodology, abstract and reflective summary writing and poster presentation by members of the coordination committee. Our study was approved by the Institutional Ethics Committee.
Applying the eclectic model as an approach to teach reflection
Students were taught reflective summary writing using the Koole et al. eclectic model as the framework. Phase 1 of this model involves a review of MSP experience by identifying the important components of the programme and to describe one’s own thoughts and feelings. Phase 2 is critical analysis wherein students write about the knowledge, skills they gained, the change in attitude through MSP which they experienced and describe how those changes happened. Phase 3 is to draw conclusions from the MSP experience and how they plan to apply the learning to future research experiences.
It has been stated that research-related outcomes have not been given sufficient attention in undergraduate medical education., Educators emphasize that research outcomes should be identified and defined clearly in medical curricula. Thus, the short-term outcomes of the MSP programme identified were (i) improved knowledge, skills and attitude regarding research in students; (ii) improved grades in performance appraisal for faculty members; and (iii) publications and conference presentations of research findings. We report the findings of the MSP programme evaluation using reflective summaries of a batch of students. The findings related to the first programme outcome only are described in this article.
Analysis of reflective summaries
All reflective summaries were graded by the two authors individually using a rubric. A consensus was reached to grade the reflective summaries as A, B and C. To provide in-depth information about programme outcomes, data from 41 reflective summaries (21 female and 20 male students) were analysed qualitatively using the thematic analysis approach. The learning outcomes at the third (learner satisfaction regarding the programme) and the fourth (attributes of learning: knowledge, skills and attitude) levels of the Kirkpatrick evaluation model were determined from the summaries. Initially, both researchers coded the reflective summaries individually and identified a set of 7 and 9 themes. Then by consensus the themes were reduced to 4 themes. It took us 10–15 minutes to analyse each reflective summary.
Learner satisfaction regarding MSP
In general, students were satisfied with their MSP experience, which they reflected as a very useful experience through which they gained various generic skills in addition to research skills [Table - 1].
Knowledge and skills gained through MSP and a change in attitude towards research
Four themes [Table - 2] related to the first programme outcome emerged from the analysis of reflective summaries.
Students commented that the teamwork through MSP helped them to deal with lack of self-motivation towards projects, to deal with depression, to manage language barriers encountered with data collection of some projects and to identify solutions for problems.
Students felt that MSP provided them an opportunity to improve their communication skills, as they had to communicate with the study sample, which included students, patients, the general public and faculty.
Change in attitude towards research
Students opined that MSP helped them to attain a positive attitude towards research as they realized that research work demands commitment, hard work, honesty and patience to deal with different people and contexts. They realized the importance of deadlines and commented that deadlines were essential without which they would have procrastinated the work. They also felt that their MSP experience was a learning phase that could not be obtained from textbooks. They commented that they wished to do research in future and would like to serve the society through research, as MSP had improved their confidence to conduct further research.
Improved mentor–student relationship
Students responded that through MSP their relationship with their mentor improved, as mentors constantly motivated them to complete tasks on time, monitored progress of the study, provided assurance that they were on the right track, provided constructive feedback and gave quick solutions to problems.
Students also commented about a few strengths of MSP as well as a few challenges they faced during the programme.
Students felt that MSP broadened their knowledge and trained them for out-of-the-box thinking, taught them not to procrastinate work, helped them in capacity building, improved their practical knowledge in terms of dealing with different situations, helped them to realize their learning gaps, taught them discipline and to do tasks in an organized way. They also felt that the programme itself was well organized and commented that poster presentation was the best experience in MSP.
Students reported that they did face a few challenges while preparing the documents required for the approval of their projects by the ethics committee, less cooperation from some group members, initial anxiety in balancing studies and research, resistance to work in groups and apprehension due to a lack of previous experience in conducting research.
Nurturing generic skills in medical students is an important task associated with curriculum development in medical schools. These skills enable students to perform duties meticulously and judiciously in their future workplace. Educators argue that training in generic skills should be embedded in the curriculum rather than making it a standalone activity and opportunities should be created in the curriculum to support these generic skills. The MSP programme of our institution intends to nurture generic skills among students through research.
Proficiency in scientific research is an additional role expected from teachers in education of health professionals,, besides the 12 roles mentioned by Harden and Crosby. Consequently, medical schools have incorporated curricular components intended to nurture research skills during undergraduate training. We have described students’ reflections on the MSP programme initiated to inculcate research skills in undergraduate medical students and how the reflections were used to evaluate the programme outcomes using the Kirkpatrick evaluation model.
Reflection is reported to be an effective teaching–learning tool that could be implemented at any level of education to augment personal development and improve learning. It has been used as a path to help students achieve a particular learning outcome, for example in problem-solving,, to improve communication skills,, and to develop professionalism. The reflection, which could happen before, during and after a learning experience, is a guiding factor for lifelong learning. It is also an attribute of self-regulated learning, which in turn helps learners to develop a deeper approach to learning and thereby improved academic performance. Studies have described how to teach students reflection, compared different methods that engaged students in reflection and the role of feedback in enhancing reflection. Studies have reported that learners engage in reflection through writing exercises. All these means of reflection create an increased self-awareness and self- improvement. This study adds to the already existing information on undergraduate research by providing a basis for the use of reflective summaries in evaluation of research programmes.
In a previous questionnaire survey conducted by us in 2011–12 on MSP programme evaluation, we reported that the MSP experience culminated in an improvement in students’ knowledge and skills pertaining to research and paved the way for a more positive attitude towards research. In the present study too, the analysis of reflective summaries from a different batch of students revealed that students were satisfied and convinced about the positive attributes of the programme. The themes that emerged from the analysis of reflective summaries were in alignment with the programme objectives and outcomes.
Durairajanayagam et al. reported that participants of the summer internship course at Cleveland clinic felt that teamwork was the most important attribute of the programme wherein they were given opportunities to engage in research. In our study too, all reflective summaries had a mention about an improvement in teamwork skills through MSP. Improved relations with mentors constitute a relevant outcome that has been achieved through MSP, as effective mentoring could augment the scope of students’ confidence to engage in research. Improvement in communication skills and a positive change in attitude have been reported previously with the MSP programme, which was reaffirmed by students of a different batch.
In our previous study, some of the challenging factors in undertaking MSP reported were: time constraints, difficulty in finding topics, inadequate guidance from mentors, uncooperative group members, misunderstanding among group members and the absence of sound knowledge regarding research methodology. The present study sample provided more positive responses, emphasizing the various generic skills gained by them during MSP and also echoing their willingness to undertake research in future. The support they received from mentors resonated in all summaries, which could be attributed to the increased importance mentors assign for MSP and the commitment shown by mentors to their mentees. This positive change in attitude of mentors could be due to the mentor orientation provided by the programme coordinators, wherein mentors are oriented on the relevance of MSP, followed by the provision of elaborate and well-defined guidelines on MSP.
Strengths and limitations of the study
A major strength of our study is that reflective summaries allowed the MSP coordination committee to get a broader view of the challenges of the programme, which would not have been possible by a questionnaire survey. Based on the feedback from students, the committee incorporated a few changes to improve the programme. The duration of the programme has been increased with the topic selection starting from year 1 itself. To ensure uniform participation of all students, MSP assessment has been introduced, wherein students are assessed by their mentors on their active participation in research, teamwork and communication skills. Mentors are also oriented regarding the programme requirements and also on their role in MSP. A major limitation is that the study findings cannot be generalized as only a limited number of reflective summaries were analysed.
This study reaffirms the feasibility of research in a packed undergraduate medical curriculum and also provides an example of how reflective summaries could be used for evaluation of a research programme. The study is an examplar of an easy, feasible and cost-effective method of evaluation of a research programme. By using the qualitative approach for data analysis, the programme coordinators could obtain a broader perspective about the strengths and barriers of the programme and could incorporate a few changes for improvement.
Conflicts of interest. Nil
Frenk J, Chen L, Bhutta ZA, Cohen J, Crisp N, Evans T, et al. Health professionals for a new century: Transforming education to strengthen health systems in an interdependent world. Lancet 2010;376:1923–58.[Google Scholar]
Stigler FL, Duvivier RJ, Weggemans M, Salzer HJ. Health professionals for the 21st century: A students’ view. Lancet 2010;376:1877–8.[Google Scholar]
Lopatto D. Undergraduate research as a catalyst for liberal learning. Peer Rev 2006;8:22–5.[Google Scholar]
Lopatto D. Survey of undergraduate research experiences (SURE): First findings. Cell Biol Educ 2004;3:270–7.[Google Scholar]
Shaw K, Holbrook A, Bourke S. Student experience of final-year undergraduate research projects: An exploration of ‘research preparedness’. Stud High Educ 2011;38:711–27.[Google Scholar]
Ryder J. What can students learn from final year research projects? Biosci Educ 2004;4:1–8.[Google Scholar]
Alamodi AA, Abu-Zaid A, Anwer LA, Khan TA, Shareef MA, Shamia AA, et al. Undergraduate research: An innovative student-centered committee from the Kingdom of Saudi Arabia. Med Teach 2014;36 Suppl 1:S36–42.[Google Scholar]
Möller R, Shoshan M. Medical students’ research productivity and career preferences; a 2-year prospective follow-up study. BMC Med Educ 2017;17:51.[Google Scholar]
Kirkpatrick DL. Evaluating training programs: The four levels. 2nd ed. San Francisco, CA:Berrett-Koehler; 1998.[Google Scholar]
General Medical Council: Tomorrow’s doctors. London:GMC; 2009. Available at www.gmc-uk.org/TomorrowsDoctors_2009.pdf_39260971.pdf . (accessed on 28 Oct 2010).[Google Scholar]
Sandars J. The use of reflection in medical education: AMEE guide no 44. Med Teach 2009;31:685–95.[Google Scholar]
Mann K, Gordon J, MacLeod A. Reflection and reflective practice in health professions education: A systematic review. Adv Health Sci Educ Theory Pract 2009;14:595–621.[Google Scholar]
Gleaves A, Walker C, Grey J. Using digital and paper diaries for assessment and learning purposes in higher education: A case of critical reflection or constrained compliance? Assess Eval Higher Educ 2008;33:219–31.[Google Scholar]
Koole S, Dornan T, Aper L, Scherpbier A, Valcke M, Cohen-Schotanus J, et al. Does reflection have an effect upon case-solving abilities of undergraduate medical students? BMC Med Educ 2012;12:75.[Google Scholar]
Ills of the system. Nature 2015;527:7.[Google Scholar]
van Schravendijk C, März R, Garcia-Seoane J. Exploring the integration of the biomedical research component in undergraduate medical education. Med Teach 2013;35:e1243–51.[Google Scholar]
Laidlaw A, Aiton J, Struthers J, Guild S. Developing research skills in medical students: AMEE guide no 69. Med Teach 2012;34:e754–71.[Google Scholar]
Devi V, Ramnarayan K, Abraham RR, Pallath V, Kamath A, Kodidela S, et al. Short-term outcomes of a program developed to inculcate research essentials in undergraduate medical students. J Postgrad Med 2015;61:163–8.[Google Scholar]
Devi V, Abraham RR, Kamath U. Teaching and assessing reflecting skills among undergraduate medical students experiencing research. J Clin Diagn Res 2017; 11: JC01–5.[Google Scholar]
Abu-Zaid A, Alkattan K. Integration of scientific research training into undergraduate medical education: A reminder call. Med Educ Online 2013;18:22832.[Google Scholar]
Abu-Zaid A, Alnajjar A. Female second-year undergraduate medical students’ attitudes towards research at the college of medicine, Alfaisal University: A Saudi Arabian perspective. Perspect Med Educ 2014;3:50–5.[Google Scholar]
Harden RM, Crosby J. AMEE Guide No 20: The good teacher is more than a lecturer––The twelve roles of the teacher. Med Teach 2009;22:334–47.[Google Scholar]
Kanthan R, Senger JL. An appraisal of students’ awareness of “self-reflection” in a first-year pathology course of undergraduate medical/dental education. BMC Med Educ 2011;11:67.[Google Scholar]
Hill K, Hamilton J. Using videoed simulated clinical interaction to promote communication skills and reflective practice for overseas-born medical students. Commun Med 2013;10:1–11.[Google Scholar]
Watling CJ, Brown JB. Education research: Communication skills for neurology residents: Structured teaching and reflective practice. Neurology 2007;69: E20–E26.[Google Scholar]
Ballon BC, Skinner W. ‘Attitude is a little thing that makes a big difference’: Reflection techniques for addiction psychiatry training. Acad Psychiatry 2008; 32:218–24.[Google Scholar]
Durairajanayagam D, Kashou AH, Tatagari S, Vitale J, Cirenza C, Agarwal A, et al. Cleveland clinic’s summer research program in reproductive medicine: An inside look at the class of 2014. Med Educ Online 2015;20:29517.[Google Scholar]
Taraban R, Logue E. Academic factors that affect undergraduate research experiences. J EducPsychol 2012;104:499–514.[Google Scholar]