Assessing Research Components and Considerations

EVIDENCE BASED PRACTICE

MA I N L ES S O N O VER VI EW A

This module takes a closer look at the two basic types of research

introduced in the previous module: Qualitative Research and

Quantitative Research. The module also begins the process of

determining whether a study is well designed and implemented,

and whether the findings are solid enough to merit incorporation

into practice.

A

A: Example of a Good Study

The researchers want to see if a new drug for high blood pressure

works to keep the blood pressure (BP) at acceptable low

levels. The research design would require a random assignment of

the study participants into either an experimental group or a

control group; the subjects would have their BPs measured in

advance; and the experimental group would receive the

medication, followed by additional measures of BP for both

groups. The [valid] research question would be: does the drug

cause a reduction in blood pressure that would not be seen in a

group without the drug? The sample size would be 50 in each

group, and subjects would all be matched on age, weight, history

of blood pressure spikes, and no other medical problems (control

for error). The data would be interval-scaled, allowing parametric

statistics to be used. The research question asks if there would be

significant differences in BP measures between the two groups

due to the new drug. The statistic would thus be a statistical

analysis of variance (ANOVA), since it is the highest level of

statistical analysis that fits the research question, the data type,

and the expected causal relationship between the drug and patient

BP levels.

The following explains how the above is an example of how a

Good Research Study responds to satisfactorily meeting the

following four questions.

• Does the study address a valid clinical question? Yes –

studying a drug to see if its use results in acceptable BP levels

is a valid clinical question.

• Do the study group participants replicate the overall population

under study? Yes – the study participants are selected

according to a number of applicable factors.

• Does the study randomly assign participants into the

experimental or control groups? Yes – once qualified, the

participants are randomly assigned to the experimental and

control groups.

• Does the study’s structure aim at validly applicable statistics?

Yes – the study results would be an ANOVA.

MA I N L ES S O N O VER VI EW B

Basically, the way to determine if a study is “bad” is to measure it

against the criteria of a “good” study. This sounds simple, but often

times studies are labeled as good, valid, when they really do not

measure up. It is the role of reviewers to make sure all of the “i’s”

are dotted and the “t’s” are crossed. Decisions based on unsound

studies can have potentially large negative financial implications,

and can result in injuries and death.

B

A: Example of a Bad Study

Bad—Same research question, but the researchers do not screen

their subjects for weight, age, and other medical conditions. They

do not use a control group, so they can’t tell if any changes in BP

after the drug is given are due to the drug, and not to other

possible factors. They use a single sample of 5 people, which is

not big enough to tell anything. Instead of actual BP measures,

they just list the results as “up” or “down,” but try to use an ANOVA

(which cannot be used with categorical data such as up or down).

The statistic does not fit the data or the sample, and all results are

questionable due to bad design and poor statistical selection.

The following explains why the above is an example of a Bad

Research Study.

1. Does the study address a valid clinical question? Yes – studying a drug to see if its use results in acceptable BP levels

is a valid clinical question.

2. Do the study group participants replicate the overall population under study? NO – the study group participants are not

screened to a valid set of criteria.

3. Does the study randomly assign participants into the experimental or control groups? NO – the study does not use a

control group.

4. Does the study’s structure aim at validly applicable statistics? NO – the study does not result in valid statistical data.

Therefore, the study is considered bad since three of four markers

of a valid, or good, study are missing (questions 2, 3, and 4).

Summary of Lesson

The purpose of this tutorial was to familiarize you with the basic

concepts in the methods, measurement, and the review of healthcare

evidence-based research studies. If structured and conducted

correctly, evidence-based research studies can lead to evidence-

based practice, and consequently, more effective healthcare for the

patient.

Main Module 1 Readings

1. Introduction

The term evidence-based practice has become pervasive in the

health care industry. This concept signals a major shift in how health

care providers deliver their services, and in how health care

institutions function. It is no longer acceptable just to perform in a

given way because “that’s how it’s always been done.”

Research to promote evidence-based practice is becoming more and

more a part of the regular work of health care leaders. However, as

with any research, it is important to be able to tell the difference

between good, solid research, and flawed research with questionable

conclusions. Since changing practice can be difficult at best, it is

essential that changes be grounded in solid evidence.

2. Evidence-Based Practice

What is evidence-based practice? This approach can be defined as

the continuous use of current, best evidence-based research in

decisions regarding patient care. Such research involves having a

clinical question that needs addressing; the search for information

and critical appraisal of that information as it relates to the clinical

question; integration of the question’s basic concepts/components

with existing clinical expertise; and understanding the projected

impacts any change can have on patients.

This approach also requires the review and integration of the results

of more than one study into the critical appraisal, so that the reliability

and generalizability of the studies’ results are stronger than any one

study can be.

3. Types of Research

A health care facility leader deals with two major types of research:

quantitative and qualitative.

When addressing clinical issues, research is traditionally performed

using a quantitative design. This may involve timed studies, where

the experimental variables are measured at different points in time on

the same study sample; it may also involve comparison of an

experimental group against a control group; or it may involve the

impacts of several independent variables on a single dependent

variable.

There are many different types of quantitative studies, but they all

require the following: the appropriate selection of a random sample of

subjects that replicates the overall population under study; the use of

a statistical analysis appropriate to the design; and a design that

effectively controls the variables under study.

4. Qualitative Research

The other type of standard research is the qualitative study.

Qualitative studies tend to focus on the experiences of subjects and

on gaining a stronger understanding of those experiences.

Researchers observe subjects in a given setting, watch for behavioral

themes, and develop formative and summative observations and

conclusions. An example of qualitative research would be a case

study, where a particular patient, process, or event is studied and

analyzed, and logical conclusions drawn from the data gathered.

Another example could be a root-cause analysis for determining flaws

and error causes in a system.

A significant amount of exploratory research is done on a qualitative

basis. Qualitative studies are often followed up by a range of

quantitative studies to derive specific answers to more narrowly

focused research questions.

5. Research Findings

How are evidence-based research findings used in a health care

setting? One of the most powerful ways to use research is as a

foundation toward improving practice outcomes. Hence, one finds the

concept of evidence-based practice; that is, practice based on

evidence research. For example, a physician may be using a

standard mix of medications to control infections. However, current

research results indicate that a particular single medication is more

effective at reducing infections than the mix traditionally used.

Since a major goal of health care leaders is to reduce hospital-

acquired infections, it would be important to disseminate this

information to physicians, with the goal of changing their practice to

achieve better outcomes.

Another example is the research on causes of stomach ulcers.

Traditionally and historically, the assumption was that high levels of

stomach acid eroded the stomach lining, producing bleeding ulcers.

The treatments at the time included medications to dilute stomach

acids, dietary changes such as drinking more milk to coat the

stomach lining, and lifestyle changes to reduce the stress that was

presumed to cause increases. People were shocked to see research

revealing H Pylori, a bacterium in the stomachs of ulcer sufferers, as

the real cause of ulcers, and that the treatment was a course of

antibiotics.

6. Health Care Leader Plan

What factors would a health care leader incorporate in a plan that

would make the process of evidence-based research leading to

improved medical care successful? It is critical to base change on

valid, reliable research findings. The strongest findings typically come

from studies that have been replicated by subsequent researchers.

There are examples across the scientific world of seemingly earth-

shaking results from a single study that could not be reproduced by

other researchers. One example is the cold fusion debacles of recent

decades, wherein various researchers claimed to be able to make

nuclear fusion occur at room temperatures. In each case, study

results rocked the world of physics and many researchers rushed to

duplicate the study. However, not a single duplication attempt was

successful. Failure to replicate findings means that, at best, they are

not generalizable outside the study sample, and, at worst, the

research methodology was flawed in some way.

Other issues to consider include: the correlation between the

research question and the study design; the type of data collected

and its impact on the statistics produced; and the ability, or inability,

to control the variables within the study as well as extraneous

variables that may have an impact on results.

7. Implications of a Successful Practice

Finally, what are the implications of successfully implementing

evidence-based practice changes in the health care environment?

One of the fundamental implications the leader must consider is the

financial implications of the change. Some may be favorable, as

when a generic antibiotic is shown to be as effective as a brand name

antibiotic, but 70% cheaper.

Other practice changes are more expensive, as when drug-coated

heart stents first made their appearance for application to patients

with coronary artery disease and the cost of the procedure went up

by thousands of dollars versus the nondrug-coated stents.

Considering the organization’s stakeholders when making change

can have a powerful effect on the viability and smoothness of the

change. For example, key stakeholders such as physicians, high

level staff, or even outside vendors can make a change effort difficult,

or even unsuccessful, if they align to resist the change. The

experienced change agent must understand the “political” climate and

stakeholders thoroughly before initiating a change process.

Finally, change theory repeatedly demonstrates that change is most

difficult when the people affected by the change are satisfied with the

status quo. One of the elements that new, valid research can

demonstrate is better outcomes than the status quo can achieve. This

can help to create a readiness to change that will facilitate the entire

process. But be prepared; logic does not always ensure an easy

transition.

8. Conclusion

The implementation of change precipitated by research findings from

evidence-based practice studies is an increasing responsibility of the

health care leadership role. Monitors of quality of patient care are

becoming publicly available through the Center for Medicare and

Medicaid Studies (CMS) and its national Web site. Consequently, the

general public is now able to see how different health care

organizations perform on national indicators of quality.

In order to meet the thresholds required by payers, changes in

practice must be implemented, driven by effective research. Financial

reimbursement may be tied to successful changes in practice,

especially where patient outcomes improve. Therefore a key element

of successful implementation of practice change is that it be based on

valid research.

A critical skill set for the health care leader to develop is the ability to

distinguish excellent research studies from those that may contain

errors that affect the validity of the results. As we continue in the

course, you will learn techniques for assessing research studies. You

will also explore issues that can complicate implementation of

changes in practice.

9. References

Melnyk, B., & Fineout-Overholt, E. (2005). Evidence Based Practice

in Nursing and Healthcare: A guide to best practice. New York:

Lippincott Williams & Wilkins.

Keys to Assess a Research Study (Panel)

There are a number of elements that comprise valid research. The

following is an examination of the key elements that the health care

administrator/manager would want to look at as basic to the makeup

of a valid research study. That is, what are the questions that need to

be answered?

KEY ELEMENTS

1. Literature Review:

a. Is the research question appropriately derived from the

literature review?

b. Does it make sense to ask the question after reading the

review of the literature?

c. Is the research question clearly stated and the variables

included in it?

2. Variables:

a. What are the independent and dependent variables?

b. Do they make sense from the perspective of the research

question?

c. Is the data measurement of each of the variables the correct

type of data needed by the selected statistic?

3. Research Design:

a. Is the sample randomly selected from the population, or are

subjects picked on the basis of a criterion?

b. What is the sample size?

c. What errors could occur in the design, and are they

controlled for?

d. Does the study have validity; does it answer the research

question on the expected relationship between the Independent

variable and the dependent variable?

e. Does the design call for an experimental and a control

group?

4. Statistic:

a. Is the statistic the correct one based on the research

question, the expected relationship between the independent

variable and the dependent variable, and the type of data taken

from subjects?

b. Is the statistic significant at the accepted level?

More Information (Alphabetical Order)

ANOVA

When speaking of statistics, an analysis of variance (ANOVA) within

the study’s structure aims at testing to ensure the study has resulted

in validly applicable statistics.

Another key question to ask when reviewing a study is “does the

study structure aim at validly applicable statistics?” If yes, the study

results are said to be an ANOVA.

Experimental Group or Control Group

In quantitative studies, the most common design uses an

experimental group and a control group. The experimental group is

exposed to the independent variables, while the control group is not.

Then both groups are measured on the dependent variable.

Before deciding upon the experimental group upon which to base the

study, the variables being studied need to be defined. Variables

represent a relationship that is tested by the research design. In a

quantitative study, this requires at least one independent variable and

one dependent variable, although more complex studies may include

more than one of each. Independent variables are those that are

assumed to have some impact or influence on the dependent, or

outcome variable. For example, in a study that examines the intent to

remain in one’s job, the dependent variable, the effects of one’s

relationship with the manager is one possible independent variable

that can influence the decision to stay. The dependent variable is

always the expected outcome measure and the independent

variables are the ones that are theorized to have an impact on

changes in the dependent variable. It is important to note if anything

in the literature review explains why the researchers chose the

particular independent and dependent variables that they did. In a

well-designed study, the rationale for variable selection will be

obvious from the review of past literature.

Main Lesson Overview A

This module takes a closer look at the two basic types of research

introduced in the previous module: Qualitative Research and

Quantitative Research. The module also begins the process of

determining whether a study is well designed and implemented, and

whether the findings are solid enough to merit incorporation into

practice.

Qualitative research begins with identifying a broad topic to be

explored and studied, rather than a narrowly designed research

question; it does not use a research question as such, or a research

hypothesis. Since it is focused on a general study for identifying and

studying broad concepts, it involves the organization and

interpretation of non-numeric data for the purpose of discovering

important patterns or relationships. It may or may not involve

literature review, and it doesn’t use a formal sample selection, data

analysis, or statistical interpretations. That noted, however, qualitative

research plays an important role in the activities of healthcare

leaders. The panel at left lists basic components of a qualitative

study.

Main Lesson Overview B

Basically, the way to determine if a study is bad is to measure it

against the criteria of a good study. This sounds simple, but often

times, studies are labeled as good, valid, when they really do not

measure up. It is the role of reviewers to make sure all of the I’s are

dotted and the T’s are crossed. Decisions based on unsound studies

can have potentially large negative financial implications and can

result in injuries and death.

Random Assignment

Random assignment of the study participants into either an

experimental group or a control group: i.e. does the study use

random sample selection of subjects, or are they matched on key

variables?

Statistical Analysis

The analysis of the data collected, the statistics, is an essential

component of the research design. The choice of the statistics to use

is affected by the types of data collected, the expected relationships

between the independent and dependent variables, and the format of

the research question.

The discussion section of the research is where the researches pull

together the entire study, discuss their findings, and tie the results

back to the research question. Discussion focuses on the statistics

resulted from the study. Do they reveal a significant relationship

between the variables? That is, did the independent variables affect

the dependent variable in a way the researchers had anticipated?

When reviewing a study, it is important to compare the results, the

statistics of the study, in the discussion section to the initial review of

the literature and the research question.

Valid Research

Study Validity determines whether the independent variable is really

having an effect on the dependent variable, as opposed to the study

being affected by variables extraneous to the study.

Are the instruments used to measure variables valid and reliable?

This research aspect can be determined by looking at the way the

instruments were originally built and tested. The researches should

use instruments that have been used for the type of study under

question and put through a series of analyses that confirm the

concepts of the instrument validity and reliability. Validity

demonstrates that the instrument measures the abstract concepts it is

supposed to measure. For example, the back depression inventory

has been shown to measure accurately the intensity of depression in

multiple studies involving thousands of patients. Consequently, it is

said to have validity.

Valid Set Criteria

Four of the most important factors included in a valid research study

are:

1. Does the study address a valid clinical question?

2. Do the study group participants replicate the overall population

under study?

3. Does the study randomly assign participants into the experimental

or control groups?

4. Does the study’s structure aim at validly applicable statistics?

In the final analysis, a study will fail to be valid if it does not contain

basic elements within its structure.