Translate this page into:
Psychometric analysis and construct validation of Health Professional Education in Patient Safety Survey in the Indian context
To cite: Wundavalli L, Hussain M, Gupta A, Arya SK. Psychometric analysis and construct validation of the Health Professional Education in Patient Safety Survey in the Indian context. Natl Med J India 2021;34:266–70.
Abstract
Background
Improving patient safety (PS) is critical to optimizing healthcare delivery. There is a need to develop curricula or incorporate PS concepts in health professionals’ (HPs) education, in both theoretical and practical training. Consequently, there is a need to measure the perception of HPs regarding various PS competencies imparted to them during their training. The Health Professional Education in Patient Safety Survey (H-PEPSS) is a tool that measures HPs’ self-reported PS competence and was designed to reflect six sociocultural areas central to PS. The tool has been validated in Canada but not in India. We did a confirmatory factor analysis (CFA) and psychometric validation of the H-PEPSS in the Indian context.
Methods
The sample comprised 240 HPs. We used the maximum likelihood estimation method on AMOS V20 (SPSS Inc.) to carry out a CFA of the tool. We used the normed fit index, Tucker–Lewis index, comparative fit index, standard root mean square residual, root mean square residual and root mean square error of approximation to evaluate the model fit. Internal consistency and reliability of the six factors of the model were examined using Cronbach’s alpha. Convergent validity of the model was examined using average variance extracted and composite reliability. Discriminant validity was examined using the Fornell and Larcker criterion and the heterotrait–monotrait method.
Results
The results indicate a good fit. The H-PEPSS was found to be reliable and valid for assessing PS competencies among HPs. Comparison of the results with the results of the Canadian setting confirmed external validity.
Conclusion
The 16-item H-PEPSS has good psychometric properties for use in the Indian context. The 23-item HPEPSS was found to be reliable and valid for assessing PS.
INTRODUCTION
Patient safety (PS) is an important dimension of quality of care alongside accessibility, acceptability, effectiveness, efficiency and people-centredness. In 2002, the WHO Member States agreed on a PS resolution during the World Health Assembly. In 2015, during the 68th WHO Regional Committee for SouthEast Asia, all Member States of the Region, including India, endorsed the ‘Regional Strategy for Patient Safety in the WHO South-East Asia Region (2016–2025)’ aiming to support the development of a national quality of care and PS strategies, policies and plans and committed to translate six objectives of the regional strategy into actionable strategies at the country level. In this context, a multistakeholder PS Expert Group constituted by the Ministry of Health and Family Welfare, Government of India, developed the National Patient Safety Implementation Framework (NPSIF).1
The strategic objective 3 of the NPSIF emphasizes a ‘catch them young’ approach, where PS principles are incorporated in medical and nursing education, and then the accumulated skills are sustained through continual medical/nursing education and on-the-job training.
The competency-based undergraduate curriculum for the Indian Medical Graduate 2018 states that at the time of graduation, the clinician should demonstrate knowledge of policies that influence PS. He/she should also be a leader and member of the healthcare team and system who shall participate appropriately and effectively, inter alia, in measures that will advance PS within the healthcare system.2
The Australian Patient Safety Education Framework (2005)3 and the WHO4 proposed the following topics in the PS curriculum: communicating effectively; identifying, preventing and managing adverse events and near misses; using evidence and information; working safely; being ethical; continuing learning and other specific issues such as preventing wrong site, wrong procedure and wrong patient treatment and medication safety. The WHO also incorporated infection control in its curriculum.4 In 2009, Canada released a framework entitled ‘The Safety Competencies: Enhancing patient safety across the health professions’ that proposed six PS competencies (Fig. 1).5
- Safety Competencies Framework into Health Professions Education Program (modified from reference 5)
The Safety Competencies Framework is the theoretical framework which underpins the Health Professional Education in Patient Safety Survey (H-PEPSS).6 The H-PEPSS comprising 23 items was developed by Ginsburg et al.6 from the six PS competencies framework and designed to measure health professionals’ (HPs) self-reported PS competence around the time of entry to practice. The H-PEPSS focuses primarily on the sociocultural aspects of PS including culture, teamwork, communication, managing risk and understanding human factors. The safety competencies were designed to be relevant across health disciplines. The six domains of the safety competencies are: (i) contribute to a culture of PS; (ii) work in teams of PS; (iii) communicate effectively for PS; (iv) manage safety risks; (v) optimize human and environmental factors and (vi) recognize, respond to and disclose adverse events.
In India, there are no scales that measure the perception of students and residents of their training with respect to PS like the H-PEPSS which measures the perception of students and residents with respect to their training in PS in the Canadian setting. This study aimed to do a confirmatory factor analysis (CFA) and psychometric validation of the H-PEPSS in the Indian context.
METHODS
The study was done at a public sector healthcare institute in New Delhi, India. The bed complement of the hospital attached to the institute is 2362. During 2016–2017, the hospital catered to about 4.14 million outpatients and 0.23 million inpatients and performed over 0.17 million surgical procedures. The average bed occupancy was close to 86%, the average hospital stay was 9 days and the net death rate was <2%. Patient care is mostly provided by resident doctors at various levels of training under the overall supervision of the relevant specialty’s faculty. In the context of a high patient load and hospital bed utilization, it is necessary that PS aspects are addressed rigorously.
Sample size
The study population comprised all the resident doctors of the clinical departments in the hospital attached to the institute. The literature recommends a sample size of 200 to perform a CFA.7 The recommended ratio of sample size to the number of items is 10 to 1 to perform a CFA.8 The H-PEPSS has 23 items. Therefore, the minimum sample size required was 230. We used a non-probability convenient sampling.
Tool
The H-PEPSS6 comprises 23 items. All items are answered using a 5-point Likert scale. For each item, participants were required to respond separately about their confidence in what they learnt in the classroom setting versus the clinical setting. Residents with at least 1-year experience were included and medical students were excluded.
Statistical analysis
The data were checked for the presence of univariate outliers, skewness, kurtosis and any multivariate outliers. Kaiser– Meyer–Olkin (KMO) test and Bartlett’s test of sphericity were carried out to assess the adequacy of the sample. Harman’s single-factor test was done to rule out common method bias. Amos V. 20 (SPSS Inc.) was used to carry out a CFA of the sample. The maximum likelihood estimation method was used. Four CFAs were carried out. The first and second CFA tested the 6-factor 23-item model of PS competency in the classroom setting and in clinical settings, respectively. The third and fourth CFA tested the 6-factor 16-item model of PS competency in the classroom setting and in clinical settings, respectively. The normed fit index (NFI), Tucker–Lewis index (TLI), comparative fit index (CFI), root mean square residual (RMR), standard RMR (SRMR) and root mean square error of approximation (RMSEA) were used to evaluate the model fit. Internal consistency and reliability of the six factors of the model was examined using Cronbach’s alpha. Convergent validity of the model was examined using average variance extracted (AVE) and composite reliability (CR). Discriminant validity was assessed using the Fornell and Larcker criterion and the heterotrait–monotrait (HTMT) method. External validity was assessed by comparing the study results with the Canadian sample results.
Ethical considerations
Permission was obtained from the institution to conduct the survey for statistical validation of the tool. The survey was conducted in accordance with the Indian Council of Medical Research ethical guidelines for biomedical and health research on human participants.9 Written informed consent was obtained from all the participants. The responses were anonymized. No personal details of any kind were sought or recorded from any participant.
RESULTS
Of the 240 filled-in questionnaires that were received, 11 responses were incomplete and were excluded from the study. The data were checked for the presence of univariate outliers, skewness, kurtosis and any multivariate outliers, and 22 samples were again excluded based on this. KMO test and Bartlett’s test of sphericity were carried out to assess the adequacy of the sample. Harman’s single-factor test was done to rule out common method bias. Four CFAs were carried out on the final sample comprising 207 respondents. The results of Cronbach’s alpha, AVE and CR are given in Tables I and II. The results of the fit indices are given in Tables III and IV. The results of the HTMT method to demonstrate discriminant validity are given in Table V. The path diagrams for the 6-factor 16-item model are shown in Figs 2 and 3.
| Factor | 23-item model Cronbach’s alpha | 16-item model | ||
|---|---|---|---|---|
| Cronbach’s alpha | AVE | CR | ||
| Teamwork | >0.870 | >0.747 | >0.522 | >0.84 |
| Communication | >0.834 | >0.834 | >0.634 | >0.90 |
| Safety risk management | >0.868 | >0.868 | >0.688 | >0.92 |
| Human and environmental factors | >0.827 | >0.799 | >0.675 | >0.88 |
| Recognizing adverse events | >0.853 | >0.864 | >0.763 | >0.92 |
| Safety culture | >0.812 | >0.781 | >0.566 | >0.86 |
| Total | >0.921 | >0.894 | >– | >– |
AVE: average variance extracted CR: composite reliability
| Factor | 23-item model Cronbach’s alpha | 16-item model | Canadian model (16 items) | ||||
|---|---|---|---|---|---|---|---|
| Cronbach’s alpha | AVE | CR | Cronbach’s alpha | AVE | CR | ||
| Teamwork | >0.845 | >0.731 | >0.49 | >0.74 | >0.848 | >0.564 | >0.885 |
| Communication | >0.815 | >0.815 | >0.78 | >0.82 | >0.816 | >0.731 | >0.891 |
| Safety risk management | >0.848 | >0.848 | >0.81 | >0.85 | >0.850 | >0.769 | >0.909 |
| Human and environmental factors | >0.833 | >0.762 | >0.78 | >0.66 | >0.833 | >0.75 | >0.900 |
| Recognizing adverse events | >0.866 | >0.821 | >0.84 | >0.82 | >0.868 | >0.716 | >0.910 |
| Safety culture | >0.803 | >0.755 | >0.51 | >0.76 | >0.807 | >0.631 | >0.872 |
| Total | >0.926 | >0.899 | >– | >– | >– | >– | >– |
AVE: average variance extracted CR: composite reliability
| Model | Cmin/df | RMR | NFI | TLI | CFI | RMSEA | SRMR |
|---|---|---|---|---|---|---|---|
| 23-item model | >2.55 | >0.075 | >0.822 | >0.861 | >0.882 | >0.085 (0.076–0.094) | >0.06 |
| 16-item model | >1.75 | >0.068 | >0.917 | >0.949 | >0.962 | >0.059 (0.043–0.074) | >0.05 |
Cmin /df chi square/defree of freedom ratioRMR root mean square residual NFI normed fit index TLI Tucker–Lewis index CFI comparative fit index RMSEA root mean square error of approximation SRMR standard root mean square residual
| Model | Cmin/df | RMR | NFI | TLI | CFI | RMSEA | SRMR |
|---|---|---|---|---|---|---|---|
| 23-item model | >1.574 | >0.038 | >0.872 | >0.939 | >0.806 | >0.0539 (0.042–0.063) | >0.048 |
| 16-item model | >1.549 | >0.330 | >0.914 | >0.956 | >0.967 | >0.0520 (0.034–0.068) | >0.042 |
Cmin /df chi square/defree of freedom ratioRMR root mean square residual NFI normed fit index TLI Tucker–Lewis index CFI comparative fit index RMSEA root mean square error of approximation SRMR standard root mean square residual
| Factor | Culture | Recognizing adverse event(s) | Human environment | Risk | Communication |
|---|---|---|---|---|---|
| Recognizing adverse event(s) | >0.21 | >– | >– | >– | >– |
| Human environment | >0.29 | >0.71 | >– | >– | >– |
| Risk | >0.83 | >0.80 | >0.97 | >– | >– |
| Communication | >0.49 | >0.81 | >0.74 | >1.04 | >– |
| Team | >0.54 | >0.74 | >0.71 | >0.79 | >1.17 |
- Path diagram of the 6-factor 16-item confirmatory factor analysis model in clinical settings
- Path diagram of the 6-factor 16-item confirmatory factor analysis model in classrooms
DISCUSSION
In the context of an increasingly complex health system, all HPs are required to be educated and trained in PS concepts and principles if they are expected to maintain a safe environment for patients and manage them. In this regard, the H-PEPSS is an important tool to measure the subjective perception of residents and students with respect to their confidence in PS issues after obtaining training in the classroom and in clinical settings.
In the study by Ginsburg et al.,6 the initial 6-factor 23-item model was reduced to a 6-factor 16-item model. Seven items were removed on theoretical grounds such as redundancy or the item being distal to the remaining items in the construct. We did a CFA of both the models in the two settings.
A comparison of the results of this study with that of Ginsburg’s study suggests a generalizability of the PS competencies raised in the H-PEPSS. The internal consistency reliability of the factors was above 0.70 for all the factors in both the settings. An alpha value of 0.60 and 0.70 or above is the criterion for demonstrating internal consistency of new scales and established scales, respectively.10
The relative chi-square values were between 1 and 3 in our study, which are indicative of a satisfactory model fit.11 The extent to which the model is supported by data was evaluated with the help of fit indices. The closer the values of the fit indices are to 1.00, the better is the fit. The fit measures are classified as absolute fit measures, relative fit measures, fit measures based on non-central chi-square distribution and information theoretic fit measures. Absolute fit measures judge the fit of a model per se without reference to other models. Some of the absolute fit measures are the goodness-of-fit index, RMR, SRMR and adjusted goodness of fit index (AGFI). The relative fit measures such as NFI, NNFI and CFI measure the fit of the study model relative to a standard. Fit measures based on the non-central chi-square distribution such as RMSEA state that a model can only be approximately correct.12 Usually, RMSEA around 0.05 is considered as a sign of good fit.12 Alternately, RMSEA results between 0.08 and 0.10 provide a mediocre fit and values close to 0.06 provide a good model fit.13 In our study, the RMSEA values of the 16-item 6-factor model in the classroom and clinical settings were below 0.06, suggesting a good model fit. The lower limit of the confidence intervals should be close to 0 and the upper limit should be <0.08 in order to indicate a good fit.14 A SRMR value of 0 indicates a perfect fit and values <0.05 indicate a good model fit.15 Values were below 0.05 in our study, which are indicative of a good model fit. Models with TLI and CFI values >0.95 are indicative of a good model fit.13 In our study, the TLI and CFI values of the 16-item 6-factor model in the classroom and clinical settings were above 0.95, suggesting a good model fit.
The AVE value for five factors was >0.50, which indicated that the convergent validity of H-PEPSS was met. Overall, H-PEPSS produced good reliability with a CR score equaling 0.77.
In our study, the square root of the AVE exceeded the intercorrelations of the factor with other factors, which ensured discriminant validity as per the Fornell and Larcker criterion.16 As per Henseler et al., HTMT criterion is defined as the mean value of the item correlations across constructs (i.e. the heterotrait correlations) relative to the geometric mean. A threshold value of 0.90 is suggested if constructs are conceptually similar and 0.85 if the constructs are conceptually more distinct.17 However, in our study, the values were more than 0.90 for the construct pairs: teamwork/communication and managing risk/communication (Table V).
Since the 23-item model was already reduced to a 16-item model in the study by Ginsburg et al., a second sample was not selected for assessing the model fit of the 16-item model in this study.
The major limitation of this tool is that it is a subjective self-report measure that highlights the sociocultural aspects of PS. Other measures such as objective structured clinical examinations are also required for a holistic assessment of residents.
Conclusion
The competency-based undergraduate curriculum for the Indian Medical Graduate 2018 states that at the time of graduation, the clinician should demonstrate knowledge of policies that influence PS. He/she should also be a leader and member of the healthcare team and system who shall participate appropriately and effectively, inter alia, in measures that will advance PS within the healthcare system.2 The H-PEPSS may serve as a vital validated instrument for formative assessment of PS and for remediation of PS topics in the Indian setting. It may serve as a starting tool to gauge the responses of doctors and other HPs before initiating a major curriculum reform in HP education as envisaged in the NPSIF. The results may be used to test a correlation between residents’ confidence in managing PS issues and the rate of adverse events in the hospital. However, the tool highlights only the sociocultural aspects of PS.
ACKNOWLEDGEMENT
We are grateful to Professor Liane Ginsburg, School of Health Policy and Management, York University, for giving permission to use the Health Professional Education in Patient Safety Survey (H-PEPSS).
Financial support and sponsorship
Nil.
Conflicts of interest
None declared.
References
- National Patient Safety Implementation Framework (2018-2025) 2018. Ministry of Health and Family Welfare, Government of India. Available at https://mohfw.gov.in/sites/default/files/national%20patient%20safety%20implimentation_for%20web.pdf (accessed on 21 Feb 2019)
- [Google Scholar]
- Medical Council of India. 2018. :19-20. Available at https://old.mciindia.org/UG-Curriculum/UG-Curriculum-Vol-I.pdf (accessed on 21 Mar 2019)
- [Google Scholar]
- Australian Commission for Safety and Quality in Health Care. 2005. Available at www.safetyandquality.gov.au/wp-content/uploads/2012/06/National-Patient-Safety-Education-Framework-2005.pdf (accessed on 21 Feb 2019)
- [Google Scholar]
- 2011. Available at https://apps.who.int/iris/bitstream/handle/10665/44641/9789241501958_eng.pdf;jsessionid=42A46DA48E799CA5FA6991E8B967867A?sequence=1 (accessed on 21 Feb 2019)
- [Google Scholar]
- 2008. Ottawa: Canadian Patient Safety Institute; Available at www.patientsafetyinstitute.ca/en/toolsResources/safetyCompetencies/Documents/Safety%20Competencies.pdf (accessed on 21 Feb 2019)
- [Google Scholar]
- The H-PEPSS: An instrument to measure health professionals' perceptions of patient safety competence at entry into practice. BMJ Qual Saf. 2012;21:676-84.
- [CrossRef] [PubMed] [Google Scholar]
- Factor analysis: Statnotes. 2008. North Carolina State University Public Administration Program. Available at www.2.chass.ncsu.edu/garson/pa765/factor.htm (accessed on 10 Jun 2018)
- [Google Scholar]
- National Ethical Guidelines for Biomedical and Health Research Involving Human Participants. New Delhi: Indian Council of Medical Research; 2017. Available at icmr.nic.in/sites/default/files/guidelines/ICMR_Ethical_ Guidelines_2017.pdf (accessed on 2 Jun 2017)
- [CrossRef] [PubMed] [Google Scholar]
- Structural equation modelling. 2008. Statnotes: Topics in Multivariate Analysis. Available at http://faculty.chass.ncsu.edu/garson/PA-765/structur.htm (accessed on 10 Jun 2018)
- [Google Scholar]
- Introduction to structural equation modeling using IBM SPSS statistics and EQS. London:Sage; 2016
- [CrossRef] [Google Scholar]
- A beginners guide to structural equation modeling (3rd ed). New York: Routledge Taylor Francis Group; 2010.
- [Google Scholar]
- Structural Equation Modeling with AMOS (2nd ed). New York: Routledge Academic; 2009.
- [Google Scholar]
- Validating the organizational climate measure: Links to managerial practices, productivity and innovation. J Organ Behav. 2005;26:379-408.
- [CrossRef] [Google Scholar]
- Evaluating structural equation models with unobservable variables and measurement error. J Mark Res. 1981;18:39-50.
- [CrossRef] [Google Scholar]
- A new criterion for assessing discriminant validity in variance-based structural equation modelling. J Acad Mark Sci. 2015;43:115-35.
- [CrossRef] [Google Scholar]
