research concepts for HIM (health information management)

AB120916_Ch04.pptx

Home >Information Systems homework help >research concepts for HIM (health information management)

Health Informatics Research Methods: Principles and Practice, Second Edition

Chapter 4: Experimental and Quasi-Experimental Research

Learning Objectives

Differentiate between experimental and quasi-experimental research types and methodologies and decide when each should be used in health informatics.

Distinguish between the pretest-posttest control group method and the Solomon four-group method and how they are used in health informatics

Explain when the posttest-only control group method should be used.

Provide examples of when to use the one-shot case study.

Demonstrate when to use the one-group pretest-posttest method and when to use the static group comparison method.

Experiment

Begin with a hypothesis

Test it

Refine hypothesis

Test again

Reach conclusions

Try to establish cause and effect

Experimental Research

Most powerful when trying to establish cause and effect

Expose participants to different interventions

In order to compare the result of these interventions with the outcome

Independent Variable

The intervention or factor you wish to measure in order to determine if it will have an effect on the outcome or disease under study.

Examples: Medications, diet, exercise, education, health information system

Dependent Variable

The outcome, end point, or disease under study

Examples include: Survival time for patients with cancer, reduction of pressure sores in patients using specific type of wheelchair, decrease in the number of adverse events in health care facilities using a CPOE

Dose-Response Relationship

Experimental research also tries to determine a dose-response relationship.

If a new medication has slowed the progression of cancer, will a higher dose slow the progression even faster?

Or if a specific factor is removed from the environment it may also decrease the progression of a certain disease.

Use Experimental Research

Consider the following

Eligibility of appropriate participants

Randomization

Ethical Issues

Quasi-Experimental Research

Similar to experimental research but does not include randomization of participants

Independent variable may not be manipulated by the researcher, and there may be no control group

It may be used over time with something other than individual participants

Example of Quasi-Experimental Research

Study the effects of automated coding system to determine if there is an increase in hospital reimbursement before and after the system is implemented

Study the cost/benefits of using an EHR before and after its implementation. Cost/benefits of a paper-based system is compared to the cost of an EHR in all HIM functions.

Study Design	Characteristics	Diagram
Pretest-posttest control group method	Randomly assigned to intervention or non-intervention (control) group Pretests given to both groups Posttests given to both groups after intervention	Treatment: R---O---X---O Control: R---O--- ---O
Solomon four group method	Two intervention groups Two control groups Randomization used to assign to all four groups Pretest for one pair of intervention and control groups Same intervention used in both groups Posttest used in all four groups	Treatment: R---O---X---O Control: R---O--- ---O Treatment: R--- ---X---O Control: R--- --- ---O
Posttest only control group method	Randomization used for assignment into intervention and control groups No pretest given Intervention given to one group only Posttest given to both groups	Treatment: R--- ---X---O Control: R--- --- ---O

R = randomization

O = observation

X = intervention

Overview of Experimental Research Designs

Study Design	Characteristics	Diagram
One-shot case study	Simple design One group Intervention Posttest	Treatment: ---X---O
One group pretest-posttest method	One group Pretest Intervention Posttest	Treatment: O---X---O
Static group comparison method	Two groups Intervention No intervention Posttest for both groups	Treatment: ---X---O Control: --- ---O

Overview of Quasi-Experimental Study Designs

Elements

Randomization

When study participants are randomly chosen to be in the experimental, control, or comparison group using a random method, such as probability sampling, so that each participant has an equal chance of being selected for one of the groups.

Intervention = experimental group

No intervention = control group

Different intervention = comparison group

Example of Randomization

Researcher may be interested in determining whether individuals retain more if they do higher levels of exercise before they learn how to use the PHR

Three different groups of participants will be established.

First group will run for 30 minutes before sitting down in front of the computer to learn to use their PHR

Second group will walk for 30 minutes before learning to use the PHR

Third group will not do any type of exercise before learning to use the PHR

Therefore, the first group is called the experimental group, the second is called the comparison group and the third is called the control group and randomization will be used

Example of Randomization (cont.)

Develop a list of all the study participants and number them

Pick out each number and allocate to a particular group. For example, the first number drawn will go into the experimental group, the second number to the control group and the third number to the comparison group and so forth until all the numbers are drawn and participants allocated

The goal is to have the experimental, control, or comparison groups as similar as possible except for the intervention under study

Randomization techniques can be performed using statistical software programs

Comparison Group

May be unethical to withhold a certain intervention from one group of participants

Some experimental research studies do not contain a control group but instead use two comparison groups.

The intervention under study is still used but all members of the study are receiving some type of intervention

For example, if researchers are assessing the effect of using HIEs to decrease the incidence of hospital-acquired infections in four nursing facilities in a particular region, then two of the nursing facilities will use the HIE based data and two of the other nursing facilities will need to utilize some other type of database in order to minimize the unethical consequences of not providing any type of data

Crossover Design

A crossover design can be used to minimize the unethical effects of not providing certain types of interventions

Includes using one group of participants as both the experimental group and the control group. A group of participants start out by being assigned to the experimental group and receive this intervention for a certain period of time, such as 6 months or a year. After they receive the intervention, they cross over to receiving no intervention or another comparison intervention for another 6 months to a year

Example of Crossover Design

May be used when studying whether certain types of telerehabilitation will improve the outcomes of patients with multiple sclerosis

Patients may start out using sensors and body monitoring and then cross-over to using a PDA or the traditional in-house therapy monitoring in order to monitor their functional levels after treatment

The same group is used as the control (or comparison group) and the experimental group

Observation

Pretest—Observing the experimental and control or comparison groups before the intervention

Posttest—Observing the experimental and control or comparison groups after the intervention

Examples of Observation

Blood pressure taken before and after the administration of medication, diet, or exercise

Questionnaire given to determine levels of depression before and after a medication intervention.

Observing a group of individuals before the administration of a policy or procedure change and then observing them again after the change has been in place for one month.

Examples of Observation (cont.)

Midtests—Observations administered neither before or after the intervention but during the middle of the particular study

Other observations may be conducted several months or years after the intervention ends to determine its long term impact.

Time-series tests—Conducted throughout the study period as a new policy or law is implemented

Might be used to examine the number of breaches of confidentiality after the implementation of HIPAA. These rates could be compared to rates before HIPAA was implemented

Control Group

Use of the control group allows the researcher to determine if the effect seen is really due to the intervention and not other extraneous factors or confounding variables

In clinical trials when medication is being tested as the intervention, the control group is given a placebo so that they are as similar as possible to the intervention group but not receiving the medication under study

Treatment

Treatments or interventions are also the independent variable

Use of experimental medications, changes in an individuals’ behavior such as smoking or alcohol cessation, or changes in a particular assistive device, technology, software or system.

Should be administered in the same way for all participants in the experimental group.

Treatment (cont.)

Example: If physical therapists are educated and trained online in using a new rehabilitation EHR system, the level (hours of training), quality (content of the online education and training), and hands-on application (amount of time using the EHR system) should be the same for all physical therapists in the experimental group

Control group may consist of those physical therapists that will receive the traditional in-class education and training.

Hypothesis: Those physical therapists trained on-line or with distance education will be the same or better than those trained using the in-class method.

Experimental Studies Pretest/Posttest Control Group Method

Similar to the randomized controlled trial (RCT) or clinical trial

Pretest-posttest control group method provides an intervention that may include a specific program or system change than a medication or treatment

Participants are randomly assigned to either the intervention (experimental) or a non-intervention (control) group

Control/Comparison group may receive a different intervention other than the one under study

Pretests are given to both groups at the same time to assess their similarities and differences

Posttests are given to both groups to determine the effect of the intervention

Example in Health Informatics

An experimental pretest-posttest control group design was used to assess a smartphone mobile application that provided personalized, real-time sun protection advice to adults aged 18 and older who owned an Android smartphone (Buller et al. 2015)

The mobile app, called Solar Cell, provided sun protection advice

The authors hypothesized that providing personalized information to adults through a mobile app when they are in the sun may help reduce sun exposure

Participants were randomly assigned to either the intervention or control group. Those in the intervention group completed a pre-test survey, used the mobile app and received information when in the sun and then completed a post-test survey. Those in the control group received no intervention

Results of the study indicate that the Solar Cell app provided some assistance in sun protection but that it was not as strong as the researchers anticipated

However, it was found that the Solar Cell app may be beneficial to those with high risk skin types or those who spend a great deal of time outdoors to make appropriate prevention decisions that reduce their exposure to the sun

Solomon Four-Group Method

Two experimental groups which both receive the intervention

One group receives a pretest and posttest while the other experimental group receives a posttest only

Two control groups are also used in this design

One control group receives a pre and posttest while the other group receives the posttest only

All participants are randomly assigned to the groups

This method controls for pretest exposure but also requires more time, effort, and cost due to the additional groups

Example of Solomon Four Group

Adaptation of the Solomon four-group design was used by researchers evaluating the effectiveness of a multi-media tutorial in the preparation of dental students to recognize and respond to domestic violence (Danley et al. 2004)

First experimental group of dental students was randomly assigned to take the pretest, the tutorial (intervention), and then a posttest

The second experimental group first took the tutorial and then the posttest

The third group (control group) took the pretest and then the posttest

Posttest-Only Control Group Method

Participants are randomly assigned to an experimental group or a control group and posttests are the only means of observation

No pretests are used

This is done to reduce the effect of familiarity with exposure to a pretest

Not using a pretest eliminates the ability to assess an improvement in scores from before the intervention to after the intervention

Example of Posttest-Only Control Group Method

The experimental posttest only control group method was used by researchers assessing the effect of community nursing support on clients with schizophrenia (Beebe 2001)

24 participants randomly assigned to control group (routine follow-up care and informational telephone contact at 6 and 12 weeks)

Experimental group(weekly telephone intervention plus routine follow-up care for 3 months)

All were followed for 3 months after hospital discharge to determine the length of survival as well as frequency and length of stay for re-hospitalizations

Quasi-Experimental Studies One-Shot Case Study

This study is a simple design in which an intervention is provided to one group which is followed forward in time after intervention to assess the outcome (posttest)

No randomization, no control group, and no pretest is included

No baseline measurement to provide a comparison to the intervention outcome

Example of One-Shot Case Study

Researchers conducted a quasi-experimental one shot case study to determine if an automated two-way messaging system will help HIV-positive patients comply with complex medication treatments (Dunbar et al. 2003)

19 HIV-positive patients enrolled and received two-way pagers that included reminders to take all medication doses and follow any dietary requirements

No control group

Outcome measures consisted of the number of times participants reported missing one or more medication doses, medication side effects, and participant’s satisfaction level in using the messaging system

One-Group Pretest-Posttest Method

Similar to one-shot case study except that the pretest is used before the intervention

No control group and no randomization

Used when it is unethical or inappropriate to withhold the intervention from a group of participants

Example of One-Group Pretest-Posttest Method

Researchers assessed the timeliness and access to healthcare services using telemedicine in individuals aged 18 and younger in state correctional facilities (Fox et al. 2007)

Data were collected one year before implementation of the telemedicine program and two years after implementation

The telemedicine intervention consisted primarily of remote delivery of behavioral health care services

Timeliness of care and use of healthcare services before and after telemedicine implementation was examined

The data was collected primarily from medical records and other claims and information assessment logs

Static Group Comparison

Two groups are examined

One with the intervention

One without the intervention

Posttest is given to assess the result of the intervention

There are no pretests and no randomization, but a control group is used

Example Static Group Comparison

Researchers assessed the use of alcohol in patients after a traumatic brain injury (TBI) based on patients’ and relatives’ reports (Sander et al. 1997)

This study examined the validity of patients’ reports by comparing them to relatives’ descriptions of post-injury alcohol use

In this design, researchers use the brain injury as the intervention and then assess via a post-injury questionnaire whether drinking habits as perceived by the patient with the TBI and the close relative are similar or different

Internal and External Validity

Internal validity demonstrates that the dependent variable (outcome measure) is only caused by the independent variable (intervention) rather than other confounding variables

External validity is concerned with being able to generalize the results to other populations

(Campbell and Stanley 1963).

Factors Affecting Internal Validity History

History or the events happening in the course of the experiment that could impact the results.

Researcher collects level of functioning data on hip replacement patients before and after the use of a new physical therapy device. During the time that this device is being used, the developer becomes ill and unable to fully train all physical therapists in its proper use. Therefore, the study may be affected by inadequate time in training rather than the device itself.

Factors Affecting Internal Validity Maturation

Maturation and refers to the natural changes of research subjects over time due to the length of time that they are in the study.

For example, older individuals may become very fatigued after completing a training session on using a computer to manage their finances. Their fatigue could then affect their responses on the posttest.

Factors Affecting Internal Validity Testing

Testing is the effect created once exposed to questions that may be on the posttest

Example: Participants of a study that is assessing whether a course module on the use of privacy and security within the electronic health record (EHR) improves their knowledge content of this subject, use a pretest and posttest to assess whether there is improvement due to the course module. Since the students are already exposed to the pretest and are able to think of some of the test questions, they may change their answers on the posttest and do better by learning from the pretest. Therefore, the use of the pretest is what may be causing the improvement in test scores more so than the course module on privacy and security of the EHR

Factors Affecting Internal Validity Instrumentation

Instrumentation—Changes in instruments, interviewers, or observers may all cause changes in the results.

Example: Interviewers may probe for answers more from one individual they are interviewing more so than others, if training is not performed consistently across all interviewers.

Factors Affecting Internal Validity Statistical Regression

Statistical regression (regression toward the mean)—When extreme scores of measurement tend to move toward the mean because they have extreme scores, not because of the intervention under study.

Example: Coders who performed poorly on the ICD-10-CM coding exam are selected to receive training. The mean of their posttest scores will be higher than their pretest scores because of statistical regression not necessarily because of the ICD-10-CM training session.

Factors Affecting Internal Validity Selection

Selection—When there are systematic differences in the selection and composition of subjects in the experimental and control groups based on knowledge or ability.

Example: One group of subjects who have viewed an instructional video on how to give themselves insulin injections is compared to another group which has not watched this video. No randomization is used.

Factors Affecting Internal Validity Attrition

Attrition—The withdrawal of subjects from the study. Those individuals who leave a study can be very different than those who remain in the study and the characteristics of these individuals can affect the results.

Example: A study which focuses on trying to reduce the number of incomplete medical records due to incomplete nursing documentation have 15 nurses leave the experimental group and 2 nurses leave the control group. The 15 nurses who leave the group may be very different than those who remain in the experimental group.

Also, the difference in the numbers of nurses who leave each group may be a problem.

Factors Affecting Internal Validity Interaction

An interaction of factors or a combination of the factors discussed previously may lead to bias in the final results

The researcher needs to be aware of the effect of a combination of some of the factors discussed above and their impact on internal validity

(Shadish and Cook, 1998, Key, 1997, Shi, 1997).

Factors that Affect External Validity

Testing

Selection bias

Participants are chosen who are frequently under medical care

Volunteers

Participants who receive compensation

All may be different than the general population

Control for Internal and External Validity

Randomization—Most powerful to control for selection, regression to the mean, interaction of factors, improves external validity because subjects are not pre-selected but uses random assignment

Use of control or comparison groups—help control for effects of history, maturation, instrumentation, interaction of factors

(Key, 1997, Shi, 1997)

Poor Experimental Procedures

Control group exposed to part of the intervention

Multiple treatment interference

Length of time of treatment intervention

Loss of participants

Summary

Experimental study designs are one of the most powerful designs to use when trying to prove cause and effect.

Quasi-experimental study designs are also very effective but tend to have many more problems with external validity since most do not include randomization of subjects

Researchers in health informatics choose to use the quasi-experimental design for many reasons such as ethical considerations, the difficulty in randomization of subjects and small sample size (Harris et al. 2006).

Several examples of experimental and quasi-experimental studies and the methodology used in the health informatics and healthcare setting demonstrate that this study design is a viable option for health informatics research.