Article Analysis 2

ESEARCH IN NURSING EDUCATION HAS

EVOLVED DURING THE PAST 50 YEARS.

From 1957 to 1982, nursing research predominately focused on nurse preparation (Stevens & Cassidy, 1999). In the 1980s, coincid- ing with the establishment of the National Center for Nursing Research in 1985 and its transition to the National Institute of Nursing Research in 1993, the focus began to shift to clinical issues. The result has been that little, if any, funding is directed toward educational research.

Despite the shift in focus, some researchers in the 1980s contin- ued to aspire to understand the nature of research in nursing edu- cation. Tanner and Lindeman (1987), using a Delphi survey tech- nique, concurred that “research in nursing education can and should meet criteria for scientific merit applied to other areas of sci- entific investigation” (p. 50). Furthermore, while the proportional emphasis of nursing research in the United States is no longer on education, nursing education studies continue to increase in fre- quency. From 70 published studies during the period 1976 to 1982 (an average of 12 per year), the total grew to 423 from 1988 to 1991 (105 per year); 1,963 articles were published for the period 1993 to 1997 (393 per year) (Stevens & Cassidy, 1999). An evaluation of the nursing education research literature published between January 1991 and December 2000 (Yonge et al., 2005) showed the most common topics to be continuing education and patient education, with more than 100 articles focused on each of these two topics.

Today, nursing education research falls short of where it could be. Writing in a 2009 Nursing Outlook editorial, Broome noted the limited data available to demonstrate the most effective and efficient

methods to produce nurses capable of caring for patients in our complex and fragmented health care system. Broome identified a need to build the evidence base for nursing education.

The Institute of Medicine (IOM) Future of Nursing report (2011) included research priorities for transforming nursing education: “Nursing education needs to be transformed in a number of ways to prepare nursing graduates to work collaboratively and effectively with other health professionals in a complex and evolving health care sys- tem in a variety of settings” (p. 164). To achieve IOM goals, the methodological rigor of nursing education research must be improved so that studies are replicable, fundable, and provide direction for improving the education of the nursing workforce.

Funding Shortage Current research dollars are insufficient to support professional education studies, both in nursing and in other disciplines and professions (Benner, Sutphen, Leonard, & Day, 2010; Broome, 2009). For a study of published medical edu- cation research, Reed, Kern, Levine, and Wright (2005) surveyed first authors of studies published in 13 prominent peer-reviewed journals in 2002-2003. When asked to estimate the cost of their studies, author responses ranged from $4,000 to $25,000, with a median of $10,000. However, Reed et al. also calculated expendi- tures for these studies based on costs of author efforts, research assistants, statisticians, equipment, data entry assistants, secretar- ial support, postage, and other resources. The calculated costs of studies were higher, ranging from $11,531 to $63,808, with a median of $24,471.

It is highly likely that lack of sufficient funding for nursing edu-

R

Methodological Quality and Scientific Impact of Quantitative Nursing Education Research

Over 18 Months CAROLYN B. YUCHA, BARBARA ST. PIERRE SCHNEIDER,

TISH SMYER, SUSAN KOWALSKI, AND EVA STOWERS

ABSTRACT The methodological quality of nursing education research has not been rigorously studied. The purpose of this study was to evalu-

ate the methodological quality and scientific impact of nursing education research reports. The methodological quality of 133 quantitative nursing

education research articles published between July 2006 and December 2007 was evaluated using the Medical Education Research Study Quality

Instrument (MERSQI). The mean (± SD) MERSQI score was 9.8 ± 2.2. It correlated (p < .05) with several scientific impact indicators: citation counts

from Scopus (r = .223), Google Scholar (r = .224), and journal impact factor (r = .216); it was not associated with Web of Science citation count, fund-

ing, or h Index. The similarities between this study’s MERSQI ratings for nursing literature and those reported for the medical literature, coupled with

the association with citation counts, suggest that the MERSQI is an appropriate instrument to evaluate the quality of nursing education research.

N U R S I N G E D U C AT I O N P E R S P E C T I V E S / N OV E M B E R / D E C E M B E R 2 0 1 1

362 Nursing Education Perspectives

R E S E A R C H

N o v e m b e r / D e c e m b e r Vol.32 No.6 363

S C I E N T I F I C I M PAC T / N U R S I N G E D U C AT I O N R E S E A R C H

cation studies makes it difficult to conduct high quality studies. Because nursing programs are typically the most expensive under- graduate programs located in higher education settings, many nursing-related studies are conducted within single sites on small samples. If nursing educational researchers could improve their methodological rigor, funding agencies might be more likely to fund these studies. One strategy to improve rigor is to evaluate the use of a scientific quality instrument to assess nursing education research.

Improving Methodological Rigor via Evaluation A num- ber of medical researchers have published on the quality of medical education literature during the last five years. Reed, Price, et al. (2005) present a well-organized table comparing five previously published guides for appraising reports of medical education inter- vention studies. These include seven categories of variables and questions one might consider: purpose, rationale, objectives, design, interventions, evaluation, and educational significance.

Cook, Beckman, and Bordage (2007) reviewed articles reporting experimental studies in medical education published in six well- respected medical journals in 2003-2004. Of 185 articles meeting inclusion criteria, they randomly selected 110 for full review. Cook et al. found that the reporting of experimental studies in medical education was generally incomplete. For example, only 45 percent contained a literature review, 55 percent presented a theoretical framework, and 76 percent included a statement of study purpose. A mere 16 percent provided an explicit statement of study design. Lastly, only 47 percent of the studies operationally defined the inde- pendent variable(s), while only 32 percent operationally defined the dependent variable(s).

In an effort to evaluate the relationship between quality and funding, Reed et al. (2007) developed the Medical Education Research Study Quality Instrument (MERSQI) to measure the methodological quality of educational research. The MERSQI includes 10 items reflecting six domains of study quality: study design, sampling, type of data, validity, data analysis, and out- comes. As designed, it is not limited to intervention studies only, but is appropriate for all quantitative studies.

The development of the MERSQI and its testing for reliability and validity have been well described by Reed et al. (2007). In brief, a literature review was conducted to elicit factors that reflect research quality. Items were defined and modified during repeated pilot testing using studies not included in the validation study. The MERSQI was then applied to 210 medical education research stud- ies published in 13 peer-reviewed journals from 2002 to 2003. Principal components analysis was done to select items to be retained; Cronbach’s alpha was 0.6, demonstrating internal consis- tency, intraclass correlation coefficients, and acceptable interrater

(range 0.72-0.98) and intrarater (0.78-0.998) reliability. Criterion validity for the MERSQI was demonstrated via the

association with global assessment of methodological quality by two independent experts, a three-year citation rate, and impact factor of the publishing journal (Reed et al., 2007). The MERSQI score was strongly and significantly correlated with the expert global quality rating, the three-year citation rate (0.8 increase in score per 10 cita- tions), and journal impact factor (1.0 increase in score per six-unit increase in impact factor). The scores were also associated with the total previous medical education peer-reviewed publications by the first author (1.46 increase in MERSQI for each 20 publications). Of the 210 studies, 71 percent had no funding, 14 percent had less than $20,000 funding, and 15 percent had $20,000 or more. The MERSQI scores were also associated with study funding of $20,000 or more (1.29 increase in MERSQI for $20,000 or more in funding). The greater the funding level, the more likely the study was multi- institutional and/or used a two-group randomized controlled design.

More recently, Reed et al. (2008) showed that MERSQI scores predict editorial decisions, at least among those manuscripts sub- mitted for publication in the annual medical education issue of the Journal of General Internal Medicine. Of 100 manuscripts, the total MERSQI was 9.6 ± 2.6 (range 5 to 15.5). Papers with one point higher total MERSQI scores (e.g., score of 10.0 versus 9.0) were associated with editorial decisions to: a) send manuscripts for peer review versus reject without review, b) invite revisions after review versus reject after review, and c) accept rather than reject the man- uscript. MERSQI scores of accepted manuscripts were significant- ly higher than scores of rejected manuscripts (10.7 ± 2.5 versus 9.0 ± 2.4, p = 0.003). In summary, the MERSQI score was associated with: a) expert quality ratings, b) three-year citation rate, c) journal impact factor, d) number of previous medical education peer- reviewed publications by the first author, e) amount of study fund- ing, and f) editorial decisions.

While the MERSQI was found to be a reliable and valid instru- ment for measuring methodological quality in medical education research, methodological quality in nursing education research has not been as rigorously evaluated. To evaluate the methodological quality of nursing education research and ensure a solid evidence base for nursing education, the authors examined the relationships between MERSQI scores and h Index, citation counts, and journal impact factor. The relationships between the MERSQI and the fund- ing sources of the studies and the country of data collection were also examined.

Method D ES IG N The cross-sectional design of this study was chosen so that methodological quality and scientific impact of recent nursing education research reports could be evaluated. Because this study was a review of published literature, it did not

364 Nursing Education Perspectives

S C I E N T I F I C I M PAC T / N U R S I N G E D U C AT I O N R E S E A R C H

involve human subjects. Therefore, upon review, the University of Nevada, Las Vegas Institutional Review Board for the Protection of Human Subjects deemed the study excluded from review.

S A MPLE The time period July 2006 to December 2007 was selected so that recent reports could be evaluated and a two-year postpublication citation rate per report could be determined. A minimum of 100 peer-reviewed reports meeting established inclu- sion criteria were evaluated; this sample size was selected based on the work of others in this area (Cook et al., 2007; Reed, Beckman, & Wright, 2009).

Article inclusion criteria were as follows: a) available in English; b) included original quantitative research (used descrip- tive statistics to present all or a portion of findings or inferential statistics to analyze all or a portion of results); c) focused on nurs- ing students as subjects; and d) featured a descriptive, experimen- tal, quasi-experimental, or observational (including case-control, cohort, cross-sectional) design. Article exclusion criteria were: a) solely qualitative research, b) meta-analysis, c) systematic review, or d) literature review.

IN S TR U MEN T The instrument selected for the study, the Medical Education Research Study Quality Instrument (MERSQI), measures the methodological quality of a published educational research report. To the researchers’ knowledge, the proposed study is the first to assess nursing education research reports using the MERSQI; its reliability, validity, and items are described above.

MERSQI domains and items within each domain of the instru- ment are shown in Table 1. Each item within a domain is assigned a value; the maximum of each domain score is 3, producing a max- imum possible score of 18 and potential range of 5 to 18. Total scores were calculated as the percentage of total achievable points to account for “not applicable” (NA) responses. For example, the response rate of retrospective studies of student records was deemed NA; validity of evaluation instruments was deemed NA when the instruments used were physiological in nature. If the eval- uation consisted of a standardized test such as the NCLEX-RN® or course grades, validity measures were considered present, even if no information was provided about validity by the authors.

VA R IA BLES The main variable was the methodological quality of published research reports. Two variables addressed the impact of the research reports: citation count and journal impact factor. In addition, study funding and country of data collection were explored for their association with methodological quality.

Citation Counts To the researchers’ knowledge, this study is the first to address the scientific impact of nursing education research reports by examining citation rate. Because of this novel approach, the research team librarian obtained citation counts 36 months post-publication via three available databases: Web of Science, Scopus, and Google Scholar.

The Web of Science database, with backfiles to 1900, has tradi- tionally been used to measure the impact of journal articles. Web of Science, the online version of Science Citation Index, Social Sciences Citation Index, and Arts and Humanities Citation Index, is arguably the leading database for providing this information in sci- ence and medical literature and includes more than 12,000 journals.

Scopus, a newer database, indexes more journals than Web of Science; it includes approximately 18,000 peer-reviewed journals with most of its content dating from 1996 to the present.

Increasingly, researchers are using Google Scholar, not only to locate research reports, but also for citation rates. Although there are flaws in its citation count, including a problem with duplicate records, Google Scholar indexes most current peer-reviewed jour- nals and other open-access materials. Therefore, it includes more journals than either Web of Science or Scopus and may yield higher citation rates for nursing education research reports than the two other databases. Google Scholar also differs from the others in that its citation count indicates the number of unique online sources that currently cite the article, providing a real-time citation count rather than a history of citation as with Scopus or Web of Science.

Journal Impact Factor Using journal impact factor to assess the scientific impact of nursing education research reports is also a novel approach. Journal impact factor for each research report was gathered from Journal Citation Reports. For the 2006 and 2007 arti- cles, 2008 and 2009 data were used, respectively. A two-year time frame was selected to allow sufficient time for citation. The research assistant located the journal impact factor and entered the value directly into a designated Microsoft Excel spreadsheet. Studies without reported impact factors were excluded from this analysis.

h Index The relationship between the h Index of first authors and methodological quality of the corresponding studies was examined; one might expect studies with higher MERSQI scores to be conduct- ed by authors with a higher h Index. This index, a measure of an author’s scientific productivity and scientific impact, was developed by Hirsch (2005). An author has index h if h of his or her number of papers have at least h citations each and the other papers have fewer than h citations each (Hirsch).

The h Index is easily found in the Scopus and Web of Science databases or can be calculated manually. This study used the h Index from Scopus, which is available for papers published since 1995. In essence, all of an author’s papers are listed in decreasing order of the number of citations for that paper, and each paper is given a rank in ascending order of the number of citations. The h Index is equal to the article rank at the point where article rank equals the number of citations. Thus, an author with a long publica- tion record would tend to have a higher h Index.

Other Factors The two other factors examined for their associ- ation with methodological quality were the country in which data

N o v e m b e r / D e c e m b e r Vol.32 No.6 365

Table 1. Medical Education Research Study Quality Instrument (MERSQI) DOMAIN MERSQI ITEM N PERCENT*

STUDY DESIGN

Single-group cross-sectional or single-group posttest only 74 55.6

Single-group pretest and posttest 25 18.8

Nonrandomized, two or more groups 29 21.8

Randomized controlled trial 5 3.8

SAMPLING NO. OF INSTITUTIONS STUDIED

1 100 82.7

2 4 3.0

>2 19 14.3

RESPONSE RATE PERCENTAGE

Not applicable 11

<50% or not reported 35 28.0

50-74% 28 23.0

>75% 59 48.4

TYPE OF DATA

Assessment by study participant (knowledge self-report) 86 64.7

Objective measurement (knowledge test) 47 35.3

VALIDITY OF EVALUATION INTERNAL STRUCTURE

INSTRUMENT Not applicable 12

Not reported 63 52.1

Reported 58 47.9

CONTENT VALIDITY

Not applicable 12

Not reported 78 64.5

Reported 43 35.5

RELATIONSHIPS TO OTHER VARIABLES

Not applicable 12

Not reported 101 83.5

Reported 20 16.5

DATA ANALYSIS APPROPRIATENESS OF ANALYSIS

Inappropriate for study design or type of data 7 5.3

Appropriate for study design & type of data 126 94.7

COMPLEXITY OF ANALYSIS

Descriptive analysis only 42 31.6

Beyond descriptive analysis 91 68.4

OUTCOMES

Satisfaction, attitudes, perceptions, opinions, general facts 84 63.1

Knowledge, skills 40 30.1

Behaviors 7 5.3

Patient/health care outcomes 2 1.5 *Ratings of “not applicable” are not included in percentages.

were collected and funding source. Funding source was designated as: a) internal funding, b) external funding, c) internal and external funding, or d) not stated.

PR O CED U R E The librarian searched the CINAHL database for articles that involved nursing students as subjects and were pub- lished in the second half of 2006 through 2007. The search was done by using the CINAHL heading “students, nursing,” and lim- iting the results to publication type “research” and “peer- reviewed” journals. Ulrich’s Periodicals Directory was checked to

verify that the journals are refereed. In cases where the directory did not have that information, the journal web pages were checked to confirm that research articles were subject to peer review.

The librarian then reviewed the abstracts and eliminated reports that met the exclusion criteria. Those reports meeting the inclusion criteria were distributed to the four nursing members of the research team. If the librarian was unsure if a report met the inclusion criteria, she conferred with the principal investigator, who made the determination.

366 Nursing Education Perspectives

Four nurse faculty members of the research team read and scored the reports using the MERSQI. Initially, all four faculty and the MERSQI consultant, Dr. Darcy Reed, evaluated four articles. Once the scoring guidelines were understood and agreed upon, the rest of the reports were divided into two alphabetical sets (A-M and N-Z). Each set was scored by two nursing faculty; if they differed on scores, the final score was derived by consensus. The research assistant entered the MERSQI scores into a spreadsheet, and two faculty scorers checked every tenth article for accuracy.

DATA ANALYSIS Data were analyzed using the Statistical Program for the Social Sciences (SPSS version 17.0) software. Alpha was set at .05. Descriptive analyses were followed by corre- lational analyses and comparisons of means to test the following null hypotheses: 1. There is no association between MERSQI score and citation rate(s); 2. There is no association between MERSQI score and h Index; 3. There is no association between MERSQI score and journal impact factor. In addition, ANOVA was used to compare MERSQI scores by funding category. Lastly, Student’s t- test was used to compare studies done within and outside the United States. All data are expressed as means ± SD.

Results REPORT DEMOGRAPHICS One hundred thirty-three articles, published between July 1, 2006 and December 31, 2007, met the inclusion criteria. Of these articles, 58 (43.6 percent) were conducted in North and South America (United States, 50; Canada, 4; Brazil, 3; Mexico, 1); 33 (24.8 percent) in Europe (United Kingdom, 27 [20.3 percent of total]); 17 (12.8 percent) in Australia/New Zealand; 14 (10.5 percent) in Asia; 10 (7.5 percent) in the Middle East; and 1 (0.8 percent) in Africa. Of these studies, 119 (89.5 percent) studied undergraduate nursing students, 10 (7.5 percent) studied graduate nursing students, and 4 (3 percent) did not identify the level of the students. Twenty-four reports (18.0 percent) also included other health professional students.

MERSQI SCORE The MERSQI score for the 133 studies was 9.8 ± 2.2 (range 6.0 to 14.5). Table 1 shows the breakdown in item scoring. The majority (55.6 per- cent) of studies were cross-sectional in design or posttest only and involved only one institution. Most studies (71.4 percent) had high response rates (> 50 percent), but involved self-report by participants (64.7 percent), rather than objective data (35.3 percent). The validity of the instruments used, internal structure, content validity, and relationships to other variables, was not generally reported (52.1, 64.5, and 83.5 per- cent, respectively). Statistical analyses were appropri- ate (94.7 percent) and the majority (68.4 percent) included inferential procedures. Most of the study out-

comes were related to student satisfaction and attitude (63.1 percent) or knowledge/skills (30.1 percent); very few outcomes were behav- ioral in nature (5.3 percent) or related to patient outcomes (1.5 per- cent). Cronbach’s alpha for the overall MERSQI was .547.

MERSQI SCORE AND CITATION COUNTS AND h INDEX

Table 2 shows the citation counts and h Index. As expected, the Google Scholar citation count was higher than the Scopus count, which was higher than that from Web of Science; all three citation measures were strongly correlated with one another. The MERSQI score correlated significantly, but weakly, with citation counts from Scopus and Google Scholar. The MERSQI score was not signifi- cantly associated with the Web of Science citation count, nor with the h Index of the first author. However the first author’s h Index was significantly correlated with the Web of Science citation count.

MERSQI SCORE AND JOURNAL IMPACT FACTOR Of the 133 articles, 78 (59 percent) were published in journals with an impact factor (0.996 ± .488, range 0.171 to 2.696). There was a small (r = .216) but significant correlation (p = .029) between the MERSQI score of reports published in these journals and their impact fac- tors. Articles published in journals with an impact factor had sig- nificantly higher MERSQI scores than those published in journals without impact factors (F(1, 131) = 4.75, p = .031), with mean MERSQI scores of 10.1 ± 2.1 and 9.3 ± 2.3, respectively.

MERSQI SCORE AND FUNDING AND COUNTRY Most (n = 99, 74.4 percent) of these studies made no mention of funding; 15 (11.3 percent) had received internal funding; 18 (13.5 percent) had received external funding; and 1 (.8 percent) had received both internal and external funding. MERSQI scores for those studies with no funding reported were 9.7 ± 2.2; those with internal fund-

S C I E N T I F I C I M PAC T / N U R S I N G E D U C AT I O N R E S E A R C H

Table 2. Citation Counts and Correlations with MERSQI Score ITEM MEAN±SD MERSQI WEB OF SCOPUS GOOGLE

N RANGE SCORE SCIENCE SCHOLAR

MERSQI 9.8±2.2 133 6.0–14.5

Web of Science 4.80±6.00 81 0–39 .138

Scopus 5.74±6.65 125 0–50 .223** .948**

Google Scholar 9.05±9.91 127 0–77 .224** .915** .940**

h Index 40.1±3.36 127 0–19 .023 .197* .127 .108

< .05 (one-tailed); **p < .001 (one-tailed); using Pearson correlation

N o v e m b e r / D e c e m b e r Vol.32 No.6 367

S C I E N T I F I C I M PAC T / N U R S I N G E D U C AT I O N R E S E A R C H

ing were 9.5 ± 2.2; and those with external funding were 10.5 ± 2.1 (these were not statistically different). Studies conducted in the United States had significantly higher MERSQI scores (10.3 ± 2.5 vs. 9.5 ± 1.9, F(1,131), p = .027), with these studies scoring signif- icantly more points for design (x2 = 9.437, p = .024) and validity of instruments (x2 = 10.820, p = .013).

Discussion This study was the first investigation to evaluate the methodological quality of nursing education research reports using an instrument designed specifically for (medical) educational reports. The major findings of the current study were that nursing education research reports scored 9.8 ± 2.2 on the MERSQI and their MERSQI score correlated with citation counts in Scopus and Google Scholar as well as journal impact factor, but was not associ- ated with Web of Science citation count or the first authors’ h Index.

Reed and colleagues (2007) reported a Cronbach’s alpha of .6 for the medical education literature in contrast to the current study’s finding of .547 for the nursing education literature, suggesting that this tool is indeed reliable across these two professions. The MER- SQI score of the nursing education research reports was 0.17 points lower than that reported for the medical literature (Reed et al.). This difference may be due to the method used to select articles for review. Reed and colleagues selected articles, published between September 2002 and December 2003, from widely cited (well- known) medical journals, including JAMA, New England Journal of Medicine, Academic Medicine, Medical Education, Teaching and Learning in Medicine, Medical Teacher, and seven journals repre- senting core medical specialty areas. The nursing MERSQI mean was also 0.9 points lower than the manuscripts accepted for publi- cation in the Journal of General Internal Medicine (Reed et al., 2008). This difference is not unexpected; this journal had a 2009 impact factor of 2.65, in contrast to the mean of 0.996 for this study’s 78 journals with an impact factor. Because this study was the first nursing education research investigation to use the MERSQI, all articles indexed in CINAHL were selected to ensure inclusiveness.

Comparison of the distribution of data for the individual items comprising the MERSQI (Table 1) with those reported by Reed et al. (2007) showed surprisingly similar trends within each domain. For example, Reed et al. reported the design distribution of their articles as follows: single-group cross-sectional or posttest, 66.7 percent; pretest and posttest, 15.7 percent; nonrandomized two or more groups, 14.8 percent; and randomized control trial, 2.8 per- cent. Therefore, despite the 0.17-point difference in MERSQI scores between the medical and nursing reports, the methodolog- ical quality is similar and extends Reed’s medical educational research finding to health-related educational research.

Regarding the current study’s hypotheses, the MERSQI scores of this study’s reports correlated with Scopus and Google Scholar citation

counts, but not the citation count from Web of Science, rejecting null hypothesis No. 1. If one assumes that articles of higher scientific qual- ity are cited more frequently, these findings suggest that the MERSQI is indeed a valid instrument to evaluate scientific quality of nursing education literature. Null hypothesis No. 3 was also rejected because the MERSQI score was weakly correlated for the journals that have been appraised for impact factor. The low correlation may be related to the small range of the MERSQI scores and impact factors.

In addition, this study is the first investigation using the MERSQI to examine the relation between the MERSQI score and h Index (null hypothesis No. 2). The hypothesis was accepted, indicat- ing a lack of an association. The h Index for this group of authors was relatively low, with 42 percent having an h Index of 2 or less. This result suggests that many authors may be new to publishing or do not publish extensively. To advance the science of nursing edu- cation, nursing needs to develop and fund nurse researchers to sup- port their programs of research in this area.

Reed and her colleagues (2007) examined categories of funding amounts: $0, $1 to $19,999, and more than $20,000. Funding of $20,000 or more was associated with a 1.29-point increase in the MERSQI score; in contrast, the MERSQI score did not differ by the current study’s funding categories of a) unfunded or b) funded by internal or external souses. The investigations reviewed in the cur- rent study are very likely to have been funded at very low levels, accounting for this difference in findings.

The distribution of scores in all domains suggests that the methodological quality of nursing (and medical) education articles can be increased by a number of strategies. First, objective meas- ures of outcomes need to be used. Instruments used should have internal and content validity and correlate with other instruments designed to measure similar or related concepts. This science must be advanced from the measurement of attitudes and knowledge to the measurement of behaviors and patient outcomes. Finally, more complex experimental designs should be used. To increase the gen- eralizability of the findings, more collaborative research across uni- versities throughout the world needs to be done.

Regarding collaborations, this study showed a difference in methodological quality between studies conducted within and out- side the United States. This finding may be related to the education and experience of nurse researchers in the United States. In their review of the nursing education research literature from 1991 to 2000, Yonge et al. (2005) found that most (58 percent) of the quan- titative, qualitative, or mixed-methods research originated in North America; 83 percent of these were generated in the United States and 17 percent were generated in Canada. This predominance of North American or US studies is similar to the current study’s find- ings. However, higher percentages in studies conducted in Australia/New Zealand (12.8 vs. 6.7 [New Zealand not reported]),

368 Nursing Education Perspectives

S C I E N T I F I C I M PAC T / N U R S I N G E D U C AT I O N R E S E A R C H

Asia (10.5 vs. 2.8), and the Middle East (7.5 vs. not reported) were observed in the current investigation (total 9.5 percent vs. 30.8 percent), suggesting that quantitative nursing education research studies are increasing outside the United States. This rise repre- sents an exciting opportunity to explore the development of studies that involve collaborations across continents and address global nursing education issues.

Limitations of the Study This study is the first of its kind in nursing education literature. As such, it was limited to evaluation of quantitative research articles published three years ago within an 18-month time period. A wide diversity of journals indexed in CINAHL was included, but qualitative research was not consid- ered. Had the focus been on the premier nursing journals, this study’s findings might be very different. In this study, Cronbach’s alpha for the MERSQI was low (.547, compared to the .60 report- ed for the medical literature, Reed et al., 2007); ideally a new instrument should show a Cronbach’s alpha of .70. Although MERSQI scores of reports in journals with an impact factor were higher than those reports from journals without an impact factor, the difference was not statistically significant. Research pub- lished today may have greater methodological rigor than the sam- ple used in this study.

Conclusion This evaluation suggests that the MERSQI is appli- cable to the assessment of nursing education research reports. The use of the MERSQI is likely to improve the quality of nursing edu-

cation research by: a) providing a guideline for nursing education researchers as they develop their studies; b) providing a template for the preparation of nursing education research reports; c) provid- ing a template for the evaluation of research reports; d) allowing for the evaluation of the quality of nursing education research reports across journals, countries of origin, years of publication, and fund- ing levels; and e) providing justification for increased funding for nursing education research.

About the Authors The authors are faculty at the School of Nursing, University of Nevada, Las Vegas. Carolyn B. Yucha, PhD, RN, FAAN, is dean and professor. Barbara St. Pierre Schneider, DNSc, RN, is associate professor, Tish Smyer, DNSc, RN, is professor, and Susan Kowalski, PhD, RN, is faculty emeri- tus. Eva Stowers, MA, is health sciences librarian. The authors would like to thank: Darcy Reed, MD, MPH, assistant professor, Mayo Clinic College of Medicine, for her helpful discussions regarding the Medical Education Research Study Quality Instrument (MERSQI); Kirsten Speck and Isabelle Stoate, gradu- ate students, for their valuable assistance with data entry; and Jennifer Orozco, for data verification. This study was funded, in part, by the National League for Nursing Ruth Corcoran Grant for Nursing Education Research. For more information, contact Dr. Yucha at [email protected].

Key Words Citation Rate – Impact Factor – Methodological Quality

– Nursing Education Research

References

Benner, P., Sutphen, M., Leonard, V., & Day, L. (2010).

Educating nurses: A call for radical transformation. San

Francisco: Jossey-Bass.

Broome, M. E. (2009). Building the science for nurs-

ing education: Vision or improbable dream

[Editorial]. Nursing Outlook, 57, 177-179.

doi: 10.1016/j.outlook.2009.05.005

Cook, D. A., Beckman, T. J., & Bordage, G. (2007). A

systematic review of titles and abstracts of experi-

mental studies in medical education: Many informa-

tive elements missing. Medical Education, 41, 1074-

1081.

doi: 10.1111/j.1365-2923.2007.02861.x

Hirsch, J. E. (2005). An index to quantify an individ-

ual’s scientific research output. Proceedings of the

National Academy of Sciences of the United States of

America, 102, 16569-16572.

doi: 10.1073/pnas.0507655102

Institute of Medicine. (2011). The future of nursing:

Leading change, advancing health. Retrieved from

http://iom.edu/Reports/2010/The-Future-of-

Nursing-Leading-Change-Advancing-Health.aspx.

Reed, D., Price, E. G., Windish, D. M., Wright,

S. M., Gozu, A., Hsu, E. B., … Bass, E. B. (2005).

Challenges in systematic reviews of educational

intervention studies. Annals of Internal Medicine,

142(12 II), 1080-1089.

Reed, D. A., Beckman, T. J., & Wright, S. M. (2009). An

assessment of the methodologic quality of medical

education research studies published in American

Journal of Surgery. American Journal of Surgery, 198,

442-444.

doi: 10.1016/j.amjsurg.2009.01.024

Reed, D. A., Beckman, T. J., Wright, S. M., Levine, R.

B., Kern, D. E., & Cook, D. A. (2008). Predictive

validity evidence for medical education research

study quality instrument scores: Quality of submis-

sions to JGIM’s medical education special issue.

Journal of General Internal Medicine, 23, 903-907. doi:

10.1007/s11606-008-0664-3

Reed, D. A., Cook, D. A., Beckman, T. J., Levine, R. B.,

Kern, D. E., & Wright, S. M. (2007). Association

between funding and quality of published medical

education research. Journal of the American Medical

Association, 298, 1002-1009.

doi: 10.1001/jama.298.9.1002

Reed, D. A., Kern, D. E., Levine, R. B., & Wright,

S. M. (2005). Costs and funding for published

medical education research. Journal of the American

Medical Association, 294, 1052-1057. doi:

10.1001/jama.294.9.1052

Stevens, K. A., & Cassidy, V. R. (1999). Evidence-based

teaching: Current research in nursing

education. Boston: Jones and Bartlett.

Tanner, C. A., & Lindeman, C. A. (1987).

Research in nursing education: Assumptions and

priorities. Journal of Nursing Education, 26, 50-59.

Yonge, O. J., Anderson, M., Profetto-McGrath, J.,

Olson, J. K., Skillen, D. L., Boman, J., … Day, R.

(2005). An inventory of nursing education research.

International Journal of Nursing Education Scholarship,

2, i-11. doi: 10.2202/1548-923x.1095

NLN

Copyright of Nursing Education Perspectives is the property of National League for Nursing and its content

may not be copied or emailed to multiple sites or posted to a listserv without the copyright holder's express

written permission. However, users may print, download, or email articles for individual use.