Translate this page into:
Health research methodology workshop: Evaluation with the Kirkpatrick model
2 Saudi Board of Preventive Medicine, Ministry of Health, Riyadh, Kingdom of Saudi Arabia
Vinoth Gnana Chellaiyan
Department of Community Medicine, Chettinad Hospital and Research Institute, Kelambakkam, Chennai, Tamil Nadu
|How to cite this article:|
Chellaiyan VG, Suliankatchi RA. Health research methodology workshop: Evaluation with the Kirkpatrick model. Natl Med J India 2019;32:100-102
AbstractBackground. Workshops on research methodology impart skills of research among medical students. Both qualitative and quantitative evaluation of an academic programme is essential to enhance the effectiveness and quality improvement. We assessed the gain in learning and effect of a workshop on research methodology among medical students.
Methods. We did a quasi-experimental, single-group study at a tertiary care hospital and research institute in southern India. It included 33 students enrolled in various residency positions of the institute. The Kirkpatrick 4-level model was used to assess the effectiveness of the workshop on research methodology. Paired t-test was used to compare pre- and postworkshop scores.
Results. Twenty-five students rated the academic sessions as excellent. The score before the workshop ranged from 0 to 17 with mean (SD) 9.27 (4.2). The post-workshop score had a minimum to maximum score of 10-26 with mean (SD) of 16.18 (3.7) (p<0.005). The effect size d cohen (confidence interval [CI]) was 1.743 (0.942-2.545). The mean (SD) of absolute and relative gain was 10.8 (3.8) and 1.41 (0.07), respectively. 66.7% medical students showed a 30% rise in their post-workshop scores, the cut-off for effectiveness of the workshop.
Conclusion. The evaluation of a workshop on research methodology provided insights into the outcomes and modifications required for their future improvement.
Evidence-based medicine (EBM) requires the use of the available evidence in making decisions about individual patients.1,2 It is important that research brings benefits to patients and to society in healthcare. However, many medical graduates have opined that research skills have not been given importance in the undergraduate curriculum.3-5 Few studies have established the effectiveness of educational interventions in research methodology and the importance of EBM curriculum based on group activities and workshops. , Both qualitative and quantitative evaluation of an academic programme is essential to enhance its effectiveness and quality. We evaluated the outcome of an educational intervention in research methodology among undergraduate medical students.
Our study was done at Chettinad Hospital and Research Institute, a tertiary care hospital and research institute in Chennai, India. We used a quasi-experimental, single group, pre- and post-test study design.
We included all 33 residents enrolled in various postgraduate courses at our institution in the academic year 2016. The workshop sessions were free and no tuition fee was charged for attending the workshop. We used the Kirkpatrick 4-level model8 to assess the workshop on research methodology. The levels included reaction, learning, behaviour and results, and the model has been validated for the evaluation of educational interventions including research methodology. ,
Level 1. Satisfaction levels of the participants were measured in a 4-point Likert scale--excellent, good, average and poor. Scores were given to assess the satisfaction of participants as: excellent 4; good 3; average 2; and poor 1. Five parameters were used to assess quality: quality of sessions, hands on experience (practice problems at the end of every session), ability of teaching faculty to explain concepts, teaching materials (handouts and shared power point slides) and time allocation for every session.
Level 2. Pre-workshop knowledge on research methodology was assessed using a questionnaire before the start of the academic sessions. The academic sessions were conducted by experts from the Department of Community Medicine of Chettinad Hospital and Research Institute. The workshop included 19 sessions. Each daily session lasted an hour. The materials for the academic session were prepared from the following resources: WHO health research methodology, ′Principles of epidemiology in public health practice′ by the Centers for Disease Control and Prevention, Epidemiology- Lancet Series, Ethical guidelines of biomedical research on human participants by the Indian Council of Medical Research, and ′Research methods in community medicine: Surveys, epidemiological research, programme evaluation, clinical trials′ by Abramsons and some online material.,,,,,,, The topics covered were: study designs (5 sessions), sampling methods (1), basics of biostatistics (1), estimation of sample size (2), critical appraisal of research studies (3), ethical considerations in research (1), review of literature (1), tests of significance (3), questionnaire development for descriptive studies (1) and research protocol development (1).
At the end of each session, a group discussion was held and handouts relevant to that session were provided.
The pre-workshop assessment questionnaire included 30 items. The items covered various sections: epidemiology, biostatistics, ethical considerations in research and critical appraisal of research studies. The assessment was done using multiple-choice questions and a few open-ended questions. After the last academic session, a post-workshop assessment was done. The post-test questionnaire consisted of 30 items similar to those in the pre-workshop assessment.
Research knowledge acquisition (learning gain) was measured from pre- and post-workshop assessment scores. Total scores and section scores were calculated. For measuring effect size, dcohen was calculated (d=Mean scorepost – mean scorepre/standard deviation). A pre-defined target of 30% was taken as the cut-off for the workshop to be considered as effective. Absolute gain (post-workshop score– pre-workshop score) and relative gain (post-workshop score– pre-workshop score/pre-workshop score) were calculated.
Level 3. Application of gained research knowledge in practice was assessed in terms of proposal writing, sample size estimation and manuscript writing. Proposal writing assessment was done at the end of the workshop. Sample size calculation exercises for various types of study designs were administered. Research manuscript writing assessment of content was done with guidelines and checklists for critical appraisal of different study designs.
Level 4. Overall impact was measured in terms of research studies started and manuscripts made ready for publication within 3 months of the research workshop. This level was assessed by enumeration of all research studies and assessment of their quality by the research committee of the institution.
Data entry and analysis
Data were entered and analysed with SPSS IBM version 21.0. Proportions were calculated for categorical variables and mean and standard deviation (SD) used for scores. A comparison of means between pre- and post-workshop scores was made with paired t-test. A value of p<0.05 was considered statistically significant.
Thirty-three medical students participated in the study, 17 were men [Table - 1]. Twenty-eight (85%) students attended >90% of academic sessions. None of the students had exposure to any previous health research workshop.
Levels 1 and 2
Overall, 76% of the students rated the academic sessions as excellent, 18% said the sessions were good and 6% said the sessions were average.
Assessment scores and effect size
The performance as shown by mean scores improved after the workshop with the scores improving from 0– 17 to 10– 26 ([Figure - 1] and [Table - 2]). The effect size dcohen (confidence interval) was 1.74 (0.94-2.55). The mean (SD) of absolute and relative gain was 10.8 (3.8) and 1.41 (0.07). Twenty-two (66.7%) students showed a 30% rise in their post-workshop scores.
|Figure 1:Box plot showing total pre- and post-research workshop scores|
On comparing the outcome, i.e. the total scores with paired test applied, there was a statistically significant improvement in the total scores [Table - 3].
Levels 3 and 4
Twenty-six (79%) participants were able to write the proposal correctly as per research protocol guidelines. Twenty-three (70%) were able to estimate sample size for various studies. Twenty-one (64%) participants were able to write the manuscript as per the guidelines of critical appraisal.
Within 3 months of the research methodology workshop, 30 (91%) participants started research studies; 24 studies were started of which 14 were case reports, 7 were cross-sectional studies and 3 interventional studies. About 25% of study participants prepared manuscripts of research studies to be sent for publication.
The majority of participants rated the workshop as excellent. However, 6% of participants rated it as average. This Likert rating provides an insight to work further on difficult areas of understanding and enhance the quality of workshop in the future. All our students were interested in conducting research studies and practising EBM. Such a positive attitude and high level of satisfaction have been reported in previous studies. ,
The post-workshop scores improved significantly (p=0.001). The workshop on research methodology was effective in imparting knowledge to the participants as shown by a 30% rise in the post- workshop assessment scores. The majority of participants were able to write a complete research proposal and estimate the sample size correctly. Hence, our workshop was able to impart skills as was reported by similar evaluation studies. , A few studies from India too have shown improvement in post-workshop scores. ,, Pawar et al.′s study showed only a moderate improvement of scores.
The impact of our workshop in terms of students starting new research studies and writing manuscripts was considerable. It was higher than the 57% of participants starting research studies in another study.  Thus, such workshops could play an important role in acquiring knowledge and skills as well as developing a ′research culture′--an imminent need. ,,, The Medical Council of India has proposed reforms in the graduate and postgraduate medical education curriculum envisaging promotion of research.27 Similarly, to promote research among undergraduate students, the Indian Council of Medical Research conducts short-term studentship projects every year. Our study emphasizes the need to include research methodology in the medical education curriculum. The restriction of levels 3 and 4 evaluation within the study duration was a limitation of the study.
The performance of students who attended the workshop on research methodology was significantly improved and there was evidence of application of the knowledge gained into practice.
Conflicts of interest. None declared
Sackett DL, Rosenberg WM, Gray JA, Haynes RB, Richardson WS. Evidence based medicine: What it is and what it isn't. BMJ 1996; 312: 71-2.BMJ 1996; 312: 71-2.'>[Google Scholar]
Sackett DL, Richardson WS, Rosenberg W, Haynes RB. Evidence-based medicine: How to practice and teach. 2nd ed. Edinburgh:Churchill-Livingstone; 2000.[Google Scholar]
Rosenthal F, Ogden F. Changes in medical education: The beliefs of medical students. Med Educ 1998; 32: 127-32.[Google Scholar]
Aslam F, Qayyum MA, Mahmud H, Qasim R, Haque IU. Attitudes and practices of postgraduate medical trainees towards research--a snapshot from Faisalabad. J Pak Med Assoc 2004; 54: 534-6.[Google Scholar]
Siemens DR, Punnen S, Wong J, Kanji N. A survey on the attitudes towards research in medical school. BMC Med Educ 2010; 10: 4.[Google Scholar]
Mostafa SR, Khashab SK, Fouaad AS, Abdel Baky MA, Waly AM. Engaging undergraduate medical students in health research: Students' perceptions and attitudes, and evaluation of a training workshop on research methodology. J Egypt Public Health Assoc 2006; 81: 99-118.J Egypt Public Health Assoc 2006; 81: 99-118.'>[Google Scholar]
Taheri H, Mirmohamadsadeghi M, Adibi I, Ashorion V, Sadeghizade A, Adibi P. Evidence-based medicine (EBM) for undergraduate medical students. Ann Acad Med Singapore 2008; 37: 764-8.[Google Scholar]
Donald L, James D, Kirkpatrick WK. Kirkpatrick's four level evaluation model. Newnan, GA:Kirkpatick Partners; 2016. Available at www.kirkpatrickpartners.com/ OurPhilosophy/TheKirkpatrickModel/tabid/302/Default.aspx (accessed on 21 Jul 2016).. Newnan, GA:Kirkpatick Partners; 2016. Available at www.kirkpatrickpartners.com/ OurPhilosophy/TheKirkpatrickModel/tabid/302/Default.aspx (accessed on 21 Jul 2016).'>[Google Scholar]
Abdulghani HM, Shaik SA, Khamis N, Al-Drees AA, Irshad M, Khalil MS, et al. Research methodology workshops evaluation using the Kirkpatrick's model: Translating theory into practice. Med Teach 2014; 36 (Suppl 1):S24-S29.Med Teach 2014; 36 (Suppl 1):S24-S29.'>[Google Scholar]
Bates R. A critical analysis of evaluation practice: The Kirkpatrick model and the principle of beneficence. Eval Program Plan 2004; 27: 341-7.[Google Scholar]
World Health Organization. Health research methodology. Geneva:WHO; 2001. Available at www.wpro.who.int/publications/docs/Health_research_edited.pdf (accessed on 21 Jul 2016).[Google Scholar]
Centers for Disease Control and Prevention. Principles of epidemiology in public health practice. Atlanta, GA:Centers for Disease Control and Prevention; 2012. Available at www.cdc.gov/ophss/csels/dsepd/ss1978/ss1978.pdf (accessed on 21 Jul 2016).[Google Scholar]
Lancet. Epidemiology. Lancet Series; 2002. Available at www.thelancet.com/series/ epidemiology-2002 (accessed on 21 Jul 2016).[Google Scholar]
Indian Council of Medical Research. Ethical guidelines for biomedical research on human participants. New Delhi:Indian Council of Medical Research; 2006. Available at www.icmr.nic.in/ethical_guidelines.pdf (accessed on 21 Jul 2016).[Google Scholar]
Abramson JH, Abramson ZH. Research methods in community medicine: Surveys, epidemiological research, programme evaluation, clinical trials. 6th ed. Hoboken:John Wiley and Sons; 2008.[Google Scholar]
Dean AG, Sullivan KM, Soe MM. OpenEpi: Open Source Epidemiologic Statistics for Public Health, Version. Updated; 6 April 2013. Available at www.OpenEpi.com (accessed on 19 Jul 2016).[Google Scholar]
Consort Checklist. Consort Group; 2010. Available at www.consort-statement.org/ Media/Default/Downloads/CENT%20checklist.pdf (accessed on 12 Jun 2016).[Google Scholar]
Strobe Checklist. Geneva:Strobe Statement; 2007. Available at www.strobe- statement.org/?id=available-checklists (accessed on 12 Jun 2016).[Google Scholar]
Nel D, Burman RJ, Hoffman R, Randera-Rees S. The attitudes of medical students to research. S Afr Med J 2013; 104: 33-6.[Google Scholar]
Prabhu GR, Rao SB, Sridhar MS, Reddy AK, Chandrasekharan PA, Shankara Reddy SD, et al. Evaluation of research methodology workshop for postgraduates in a medical college, Tirupati. J Evol Med Dent Sci 2015; 4: 15868-72.[Google Scholar]
Thakre SB, Thakre SS, Golawar SH, Ughade SN, Thakre AD. Awareness about biomedical research among postgraduate students at a tertiary care hospital, Central India: Pretest-posttest design. Prospect Med Res 2014; 2: 8-14.[Google Scholar]
Khan H, Khwaja MR. Impact of a workshop on the knowledge and attitude of medical students regarding health research. J Coll Phys Surg Pak 2007; 17: 59.[Google Scholar]
Pawar DB, Gawde SR, Marathe PA. Awareness about medical research among resident doctors in a tertiary care hospital: A cross-sectional survey. Perspect Clin Res 2012; 3: 57-61.[Google Scholar]
Scaria V. Whisking research into medical curriculum. Calicut Med J 2004; 2: e1.[Google Scholar]
What is the purpose of medical research? Lancet 2013; 381: 347.[Google Scholar]
Lansang MA, Dennis R. Building capacity in health research in the developing world. Bull World Health Organ 2004; 82: 764-70.[Google Scholar]
Medical Council of India. Reforms in Undergraduate and Postgraduate Medical Education. Vision 2015. New Delhi:The Medical Council of India; 2011:34. Available at www.mciindia.org/tools/announcement/MCI_booklet.pdf (accessed on 6 Jul 2016).[Google Scholar]
ICMR. Short Term Studentships. New Delhi:Indian Council of Medical Research; 2015. Available at www.icmr.in/short.htm (accessed on 20 Jul 2016).[Google Scholar]