Health research methodology workshop: Evaluation with the Kirkpatrick model

Introduction

Evidence-based medicine (EBM) requires the use of the available evidence in making decisions about individual patients.1,2 It is important that research brings benefits to patients and to society in healthcare. However, many medical graduates have opined that research skills have not been given importance in the undergraduate curriculum.3-5 Few studies have established the effectiveness of educational interventions in research methodology and the importance of EBM curriculum based on group activities and workshops. ^[6],[7] Both qualitative and quantitative evaluation of an academic programme is essential to enhance its effectiveness and quality. We evaluated the outcome of an educational intervention in research methodology among undergraduate medical students.

Methods

Our study was done at Chettinad Hospital and Research Institute, a tertiary care hospital and research institute in Chennai, India. We used a quasi-experimental, single group, pre- and post-test study design.

We included all 33 residents enrolled in various postgraduate courses at our institution in the academic year 2016. The workshop sessions were free and no tuition fee was charged for attending the workshop. We used the Kirkpatrick 4-level model8 to assess the workshop on research methodology. The levels included reaction, learning, behaviour and results, and the model has been validated for the evaluation of educational interventions including research methodology. ^[9],[10]

Data collection

Level 1. Satisfaction levels of the participants were measured in a 4-point Likert scale--excellent, good, average and poor. Scores were given to assess the satisfaction of participants as: excellent 4; good 3; average 2; and poor 1. Five parameters were used to assess quality: quality of sessions, hands on experience (practice problems at the end of every session), ability of teaching faculty to explain concepts, teaching materials (handouts and shared power point slides) and time allocation for every session.

Level 2. Pre-workshop knowledge on research methodology was assessed using a questionnaire before the start of the academic sessions. The academic sessions were conducted by experts from the Department of Community Medicine of Chettinad Hospital and Research Institute. The workshop included 19 sessions. Each daily session lasted an hour. The materials for the academic session were prepared from the following resources: WHO health research methodology, ′Principles of epidemiology in public health practice′ by the Centers for Disease Control and Prevention, Epidemiology- Lancet Series, Ethical guidelines of biomedical research on human participants by the Indian Council of Medical Research, and ′Research methods in community medicine: Surveys, epidemiological research, programme evaluation, clinical trials′ by Abramsons and some online material.^{[11],[12],[13],[14],[15],[16],[17],[18]} The topics covered were: study designs (5 sessions), sampling methods (1), basics of biostatistics (1), estimation of sample size (2), critical appraisal of research studies (3), ethical considerations in research (1), review of literature (1), tests of significance (3), questionnaire development for descriptive studies (1) and research protocol development (1).

At the end of each session, a group discussion was held and handouts relevant to that session were provided.

The pre-workshop assessment questionnaire included 30 items. The items covered various sections: epidemiology, biostatistics, ethical considerations in research and critical appraisal of research studies. The assessment was done using multiple-choice questions and a few open-ended questions. After the last academic session, a post-workshop assessment was done. The post-test questionnaire consisted of 30 items similar to those in the pre-workshop assessment.

Research knowledge acquisition (learning gain) was measured from pre- and post-workshop assessment scores. Total scores and section scores were calculated. For measuring effect size, dcohen was calculated (d=Mean scorepost – mean scorepre/standard deviation). A pre-defined target of 30% was taken as the cut-off for the workshop to be considered as effective. Absolute gain (post-workshop score– pre-workshop score) and relative gain (post-workshop score– pre-workshop score/pre-workshop score) were calculated.

Level 3. Application of gained research knowledge in practice was assessed in terms of proposal writing, sample size estimation and manuscript writing. Proposal writing assessment was done at the end of the workshop. Sample size calculation exercises for various types of study designs were administered. Research manuscript writing assessment of content was done with guidelines and checklists for critical appraisal of different study designs.

Level 4. Overall impact was measured in terms of research studies started and manuscripts made ready for publication within 3 months of the research workshop. This level was assessed by enumeration of all research studies and assessment of their quality by the research committee of the institution.

Data entry and analysis

Data were entered and analysed with SPSS IBM version 21.0. Proportions were calculated for categorical variables and mean and standard deviation (SD) used for scores. A comparison of means between pre- and post-workshop scores was made with paired t-test. A value of p<0.05 was considered statistically significant.

Results

Thirty-three medical students participated in the study, 17 were men [Table - 1]. Twenty-eight (85%) students attended >90% of academic sessions. None of the students had exposure to any previous health research workshop.

Table 1: Disciplines of the students who attended the research workshop (n=33)

Levels 1 and 2

Overall, 76% of the students rated the academic sessions as excellent, 18% said the sessions were good and 6% said the sessions were average.

Assessment scores and effect size

The performance as shown by mean scores improved after the workshop with the scores improving from 0– 17 to 10– 26 ([Figure - 1] and [Table - 2]). The effect size dcohen (confidence interval) was 1.74 (0.94-2.55). The mean (SD) of absolute and relative gain was 10.8 (3.8) and 1.41 (0.07). Twenty-two (66.7%) students showed a 30% rise in their post-workshop scores.

Figure 1:Box plot showing total pre- and post-research workshop scores

Table 2: Overall satisfaction of the workshop participants (n=33)

On comparing the outcome, i.e. the total scores with paired test applied, there was a statistically significant improvement in the total scores [Table - 3].

Table 3: Mean (SD) scores of workshop participants before and after the academic sessions (n=33)

Levels 3 and 4

Twenty-six (79%) participants were able to write the proposal correctly as per research protocol guidelines. Twenty-three (70%) were able to estimate sample size for various studies. Twenty-one (64%) participants were able to write the manuscript as per the guidelines of critical appraisal.

Within 3 months of the research methodology workshop, 30 (91%) participants started research studies; 24 studies were started of which 14 were case reports, 7 were cross-sectional studies and 3 interventional studies. About 25% of study participants prepared manuscripts of research studies to be sent for publication.

Discussion

The majority of participants rated the workshop as excellent. However, 6% of participants rated it as average. This Likert rating provides an insight to work further on difficult areas of understanding and enhance the quality of workshop in the future. All our students were interested in conducting research studies and practising EBM. Such a positive attitude and high level of satisfaction have been reported in previous studies. ^[9],[19]

The post-workshop scores improved significantly (p=0.001). The workshop on research methodology was effective in imparting knowledge to the participants as shown by a 30% rise in the post- workshop assessment scores. The majority of participants were able to write a complete research proposal and estimate the sample size correctly. Hence, our workshop was able to impart skills as was reported by similar evaluation studies. ^[6],[9] A few studies from India too have shown improvement in post-workshop scores. ^{[20],[21],[22]} Pawar et al.′s study showed only a moderate improvement of scores.^[23]

The impact of our workshop in terms of students starting new research studies and writing manuscripts was considerable. It was higher than the 57% of participants starting research studies in another study. ^[9] Thus, such workshops could play an important role in acquiring knowledge and skills as well as developing a ′research culture′--an imminent need. ^{[19],[24],[25],[26]} The Medical Council of India has proposed reforms in the graduate and postgraduate medical education curriculum envisaging promotion of research.27 Similarly, to promote research among undergraduate students, the Indian Council of Medical Research conducts short-term studentship projects every year.^[28] Our study emphasizes the need to include research methodology in the medical education curriculum. The restriction of levels 3 and 4 evaluation within the study duration was a limitation of the study.

Conclusion

The performance of students who attended the workshop on research methodology was significantly improved and there was evidence of application of the knowledge gained into practice.

Conflicts of interest. None declared