Article Title: "The Association between Computer Literacy and Training on Clinical Productivity and User Satisfaction in Using the Electronic Medical Record"
Authors: May Alasmary, Ashraf El Metwally and Mowafa Housch
Published: 24 June 2014; Journal of Medical Systems
Main Results of the Article
This study looks to explore whether or not computer literacy and training have an effect on clinical productivity and satisfaction of Electronic Medical Records (EMRs). The setting is one hospital in Saudi Arabia, where the hospital had recently implemented an EMR system. The subjects being studied were nurses and physicians who were current users of the system. The study involved the use of a 40 question survey. Interviews were also conducted to validate the results of the survey and to offer insights into the statistical results.
According the research paper, the following results were noted:
• Majority of participants were generally satisfied with the system
• Satisfaction scores were higher among physicians
• Physicians were more satisfied with training
• Most nurses and physicians agreed that the system increased perceived clinical productivity
• A statistically significant weak positive correlation exists between age and satisfaction
• Years of experience could not predict system satisfaction
• Correlation between system productivity and system satisfaction was statistically significant.
• Statistically significant medium positive correlation exists between computer literacy and satisfaction
• Statistically significant differences between mean satisfaction of systems; with age (older is more satisfied), performance, comparison with paper systems and computer literacy.
The conclusions noted were that the EMR systems seems to be effective and highly appreciated by its users. Increasing productivity and EMR user satisfaction could be an ultimate goal of any health care organization. The study identified that high computer literacy has a positive impact on user satisfaction.
Critique of Methodology Used and the Author’s Interpretation of Results
First, there is no formal hypothesis in this research paper. The researcher identifies the following research question: “Do computer literacy, training have an impact on clinical productivity and satisfaction of Electronic Medical Record?” This question does not predict directionality and the terms are not clearly defined. From the survey, one might infer how the researcher measured computer literacy and/or “perceived clinical productivity” but is not clearly defined in the research paper.
The research paper goes on to state the following study objectives:
• To investigate end users (physicians and nurses), satisfaction levels of the newly implemented EMR
• To investigate clinical productivity of the new EMR
• To investigate the association between computer literacy and EMR user satisfaction perceptions
• To investigate the association between training and users satisfaction
The researcher should have postulated the directionality of the variables in the form of a hypothesis. In addition, the researcher concludes on findings which were not indicated in the research question or objectives. As such, the research paper outlines a process that appears more exploratory in nature.
The sampling method was a convenience sample of all physicians and nurses at the hospital who utilize the system. A total of 123 completed the survey although 12 results were removed as those individuals had participated in a pilot survey. As such, there was no random assignment or randomized method performed. An ANOVA, which is used to analyze some of the data, assumes that the dependent variables follow a normal distribution and that variance matrices must be equal for all treatment groups. There was no check of these assumptions. In this case, more nurses took the survey then physicians which could skew the outcome of the data. Additionally, since all participants came from the same hospital and were using the same system; this data may not be generalizable to other populations even within the same region.
The researcher documents multiple statistical techniques used: descriptive statistics, t-tests, regression analysis and one-way ANOVA. The fact that there was no formal hypothesis seems to have led to a hodgepodge of analysis to find correlations and meaning from the data. The danger in doing this is that it can lead to spurious correlations – correlations that just don’t mean anything or just aren’t real. There is no way to know if there are other factors that could have driven satisfaction. In addition, conducting different types of tests increases the experiment wise error rate. The experiment should be designed and the statistical method should be chosen to reduce the experiment wise error rate and in this case it was not.
According to the article, “One-way ANOVA showed that there was statistically significant difference between means satisfaction of system with age (p=.011; older staff [showed] higher levels of satisfaction), performance satisfaction (p=0.26) and the comparison with paper systems (p=0.008) and computer literacy (p<0.01). “
The above sentence really doesn’t explain anything but it appears that the researcher may have run an ANOVA where the DV was satisfaction of the system and the IVs included age, performance satisfaction, comparison of paper systems and computer literacy. The researcher did not define his/her terms so it is difficult to tell whether or not he conducted the ANOVA correctly. The DV should be quantitative and as such the researcher could have converted satisfaction scores into some type of ratio scale. The IVs should be qualitative in nature. The Likert scales used in the survey are nonmetric and could be used or could be converted to (Hi, Med, Lo) type variables. The IV “age” would need to be converted to a qualitative score as well, such as (old, young).
Assuming all the above was handled appropriately, the bolded quote above continues to have flaws. First, it identifies performance satisfaction as significant at p=.26. At an alpha level of .05 or even .10, this would not be considered significant. My best guess is that the researcher discovered that the overall model showed some significant effect and he/she is simply listing the p-values of all the variables
in the model. The researcher should have looked to the difference in means for the effect “satisfaction” and determined which variable(s) were the most significant in driving satisfaction with the system.
Does the analysis answer the research question posed?
Research Question: Do computer literacy, training have an impact on clinical productivity and satisfaction of Electronic Medical Record?
1. None of the results compared computer literacy to clinical productivity.
2. None of the results compared training to clinical productivity.
3. There was a result that compared productivity to system satisfaction (but that wasn’t the question)
4. The researcher’s conclusion on training’s impact on satisfaction is discussed below.
5. The researcher’s conclusion on computer literacy’s impact on satisfaction is discussed below.
As it relates to whether or not training had an impact on satisfaction, the researcher failed to find a statistically significant correlation between training and satisfaction. Training did not appear to be considered in the one-way ANOVA as an independent variable but the lack of detail in the paper makes this unclear. As it relates to whether or not computer literacy has an impact on satisfaction, the analysis may answer the question for that specific hospital but could not be generalized to other populations. Even within their own hospital system, the search to find correlations as opposed to setting forth a formal hypothesis and seeking to test it could have created a situation where there may be a correlation but it may be a spurious one. I would argue that the researcher has not answered the question.
Recommendation
In revising the research, the first thing I would do is to create the formal hypothesis. The research question as stated by the researcher was “Do computer literacy, training have an impact on clinical productivity and satisfaction of Electronic Medical Record?” An example might be:
H1: Computer literacy (IV) and training (IV) have a positive effect on user satisfaction of an EMR (DV)
H2: Computer literacy (IV) and training (IV) have a positive effect on perceived clinical productivity (DV).
With two DVs that are likely to be correlated, the researcher should conduct a MANOVA. Also, with two IVs, the researcher should investigate possible interaction between literacy and training.
Next, I would clearly define each of the variables and the manner in which they would be measured and evaluated in the statistical analysis. I’d also select the most appropriate statistical technique (MANOVA) that would assist in lowering the experiment wise error rate. I would refrain from exploratory methods to find correlations as those could lead to spurious correlations.
Finally, rather than a convenience sample, I would attempt to construct a random sample and would use more than one hospital location in the region or country (depending on the population of interest).
Math81
Sample-Paper-Critique.docx
0
