Descriptive survey

Igotyou

ab120908_part_2_ch03-ch07_slides_1.ppt

Home >Information Systems homework help >Descriptive survey

Health Informatics Research Methods:

Principles and Practice

Part 2: Chapters 3-8

Survey Research

Chapter 3

Overview of Survey Research

Choose a topic of study
Formulate criteria to develop questions about that topic
Can explore a disease, community, organization, culture, health information system or software etc.
Random sample of subjects chosen to answer the questions in a standardized format

Overview of Survey Research

Develop the questionnaire
Develop the cover letter that explains the study and directions for completing the questionnaire
Pretest the questionnaire for validity and reliability
Disseminate the survey via email, mail, Web, fax or use the survey during interviews

Overview of Survey Research

Researcher must choose the best medium to disseminate survey
Overall goal of survey research is to collect the most appropriate and accurate data that will answer the questions pertaining to the research topic.

Survey Creation

Use/Adapt Existing Surveys
Health Information National Trends Survey (HINTS)
Created a population-based survey that tracked trends in the use of communication technologies as a source of cancer information

Survey Creation

Use/adapt existing surveys cont’d
National Center for Health Statistics
National Health Interview Survey (NHIS)
National Ambulatory Medical Care Survey (NAMCS)
Health Assessment Questionnaire
Adapt several different questionnaires into one

New Survey Development

Need to consider the following items:
Content
Audience
Medium
Sample or survey entire population
Statistics to be generated

Advisory Committee

Focus group of experts in survey design and the topic under study
Assist in phrasing the questions
May also seek assistance from organizations/departments which provide assistance and guidance in survey design, development, and analysis of results
Inclusion criteria

Types of Questions

Open-ended (unstructured or qualitative)
Close-ended (structured or quantitative)
Scales:
Nominal
Ordinal
Interval
Ratio

Pilot Test Survey

Pilot test survey on a small group of respondents
The sample should reflect the true sample of respondents
Provide accurate simulation of administration of survey questionnaire
Review all comments and discuss with advisory board
Incorporate into final survey

Test Survey for Validity

Face Validity

Criterion-related validity

Construct validity

Content validity

Examines how survey looks

Accuracy of intended

survey

Agreement between theoretical concept and survey

Survey captures the information intended to measure

Test Survey for Reliability

Reliability or consistency of survey

Reliability Coefficient :
Cronbach’s Alpha
Measures whether survey has internal consistency—Do all variables measure the same concept?
Should be measured on an interval or ratio scale
Normally distributed
Reliability Coefficients close to 1.00 have very high internal consistency or reliability
Test-Retest for Reliability
Measures whether survey is consistent over time or when given multiple times
Correlation coefficient between the relationship between two total scores given two different times
Those coefficients close to 1.00 show strong reliability.

Factor Analysis in Refining Survey

Sometimes researchers need to refine the number of questions used in a survey.
Factor analysis is a statistical technique in which a large number of variables are summarized and reduced down to a smaller number based on similar relationships among those variables.

Audience

Need to know your audience so that questions on the survey can be built so that they are able to answer them
Use clear, unambiguous terms
Do not use terms that are unclear, such as “not sure”
Pilot test the survey to capture terms that should not be included in the original development of the survey

Framing of Questions

Order of questions is important
Demographic data is usually first, followed by more broad or general questions, followed by more specific questions with ranked or ordinal type responses or open-ended type questions
Provide a checklist of possible responses
HIPAA Privacy Rule Implementation Study by Firouzan

Incentives

Consider whether incentives should be provided to individuals who complete the survey
Some incentives may influence responses so be careful when choosing the type of incentive
Researcher should decide if it is necessary and what type of incentive should be used if any

Confidential Reponses

Discuss how the individual’s responses will be kept confidential in the cover letter or instructions
Institutional Review Board (IRB) will also need to see how information collected will be kept confidential
Examples:
individual non-identifying number and password can be part of the URL in a Web-based survey,
no identifying information linked with survey,
separate database used to collect demographic information but not linked to responses
All information reported in presentations and published material should be reported in aggregate form

Limitations

Every study design has limitations
Survey research design includes the following limitations:
Inaccurate responses due to:
Not understanding the question or instructions
Not having appropriate time to complete the survey
Not able to recall past experiences to answer the question
Too long and respondents may tire
May exaggerate responses to questions—i.e. salary

Reduce Limitations By

Ask questions that:
are unambiguous
Do not require extensive recollection
Jog the respondent’s memory with pictures, graph, table etc.
Provide ranges to choose from especially if the question is more sensitive in nature

Type/Medium

Web-based
Email
Mail
Fax
Group

Web-based Surveys

Advantages:
Reduced cost when compared to paper
Little or no data entry
Ease of data analysis
Use of pop-up instructions and drop-down boxes
Ability to present questions in random order
(Gunn, 2002)

Web-based Surveys

Disadvantages
Missing respondents who do not have a computer or access to Internet
Increased up front time in the development of the questionnaire
Hire a person with skills in Web-based survey design and development
Difficult making changes to survey once on the Web
Respondents more reluctant to provide responses over Internet due to lack of confidence in privacy and security

Other Types of Surveys

Mail—Still used effectively in health informatics especially when surveying physicians, nurses and other health care providers
Email—very similar to Web-based surveys but time to develop may not be as extensive, however, automatic response to database is limited
Fax---Similar to mail surveys and can be used so that one captures all possible respondents
Group---Paper survey made available to participants at a conference, retirement community, physician practice etc.

Examples of Survey Research in Health Informatics

EHR/ASTM study by Watzlaf et al, 2004 used all types of surveys (Web-based, mail, fax, group) with the major focus being Web-based. Intent was to increase response rate using all types of medium.
Criswell et al, 2002 used mail surveys to determine if physicians in family practice residency programs used PDAs

Examples of Survey Research in Health Informatics

Murff et al, 2001 used mail surveys for physicians to determine their satisfaction with using two different CPOE systems. Used the Questionnaire for User Interface Satisfaction (QUIS).
Patel et al, 2005 used both survey and face-to-face interviews to collect pre-and post-course data on students enrolled in the Woods Hole Course in Medical Informatics. Interviews were used to supplement the survey questions.

Examples of Survey Research in Health Informatics

Couper et al, 2007 used both mail surveys and phone interviews to collect data on weight management and also were interested in which method produced a better response rate.
Results: phone interview (59% response rate) and mail (55% response rate).

Distribution of Survey

Researcher should determine HOW the survey will be distributed
Email or phone each of the respondents selected to participate and explain the purpose of the study and their role
Ask if they are willing to participate
Determine the best way to send the survey and obtain contact information for them
Assure respondent that their answers will be kept confidential and that only aggregate data will be used when reporting results

Distribution of Survey

Discuss the cover letter content
Discuss incentives if they are used
Once the respondent agrees to participate, contact them via email, mail, fax, etc. and provide a copy of the cover letter and survey.
Make sure that the deadline date to return the survey is bolded in the cover letter (See Appendix 3C for sample of Cover Letter)
After 2 weeks with no response, send Follow-up letter and reiterate the importance of study and the need for their participation. (See Appendix 3E for sample of Follow-Up Letter)

Sample and Sample Size

Most survey research is performed on a sample
Census survey includes the entire population
If a sample is chosen, make sure that it includes an accurate representation of the population under study. In this way, the characteristics of the sample participants are similar to the population characteristics.

Sampling Methods

Stratified Random Sampling—separate the population by certain characteristics such as physician specialty, nursing units, DRGs, and then choose the sample
Systematic Random Sampling---draw the sample from a list of items such as diagnoses, I9 codes, or discharges and select every nth case

Sampling Methods

Cluster Sampling—separate first into a city block, randomly choose residences and then sample everyone within that residence
Convenience Sampling—not random, can generate quick results but the results should not be generalized to the population. Example: When one surveys everyone in a specific HIT department to determine their knowledge of HL7

Sample Size Calculation

Refer to Table 3.5 and pages 68 and 69 for an Example of the Sample Size Calculation for the EHR study by Watzlaf et al, 2004.

Response Rate

Very important in survey research
Even a low response rate may prove beneficial if it is in an area or topic that has not been researched in depth before

Response Rates

Methods used to increase response rates include:
Follow up letters, emails, phone calls, fax
Do not be overly annoying
Include the title of the research study, when the survey questionnaire was sent, the importance of the study, and how important the respondent’s reply is to the research study
Reiterate how data will be kept confidential
Explain any incentives
Attach survey again –do not make the respondent look for previous survey sent

Statistical Analysis of Survey Study Data

Usually quite simple
Frequencies and percentages
Correlation coefficieints
Tests of significance
Confidence Intervals
Open-ended questions should be analyzed using content analysis

Examples of Statistical Analysis of Survey Study Data

See Tables 3.6 – 3.12 for a simple data display for the EHR study by Watzlaf et al, 2004.

Summary

Development of survey instrument
New survey or adapt existing one
Address the following areas:
Content
Audience
How Administered (Type/Medium)
Sample or Population
Type of Statistics

Summary

Choose advisory committee of experts to review the survey
Determine the different types of questions and response types or scales
Pilot Test the survey
Feedback reviewed with Advisory Committee
Make changes based on the pilot survey

Summary

Validity and Reliability Testing
Cronbach’s alpha
Test-Retest
Factor Analysis
Other issues to consider
Incentives
How to maintain confidentiality
Minimize bias or error
Increase response rate through follow-up
Appropriate statistical analysis

Health Informatics Research Methods:

Principles and Practice

Chapter 4

Outline of Discussion

Observing
Why do Observational Research
Non-Participant
Naturalistic
Simulation—Usability
Case Study
Focused Interview

Informal Conversational
Standardized Open-Ended
General Interview Guide
Focus Group
Participant Observation
Ethnography
Content Analysis

Observing

Take a minute to observe something near you. It can be anything on your desk, in your office or home.
Look at its color, shape, function, age, etc.
Jot down some information about it.
Let’s discuss

Introduction

Qualitative research
Perceptions
Interactions
Feelings
Attitudes
Depth

Why choose Observational Research?

New topic
Transitional program
Background for larger study
Robust, rich data
Observation, field interviews, medical record reviews, ethnographic methods, combination of all

Attention To

Observation site
Time period
What will be observed
How it will be recorded
Who will conduct the observation
How data will be analyzed
How results will be disseminated

Non-Participant Observation

Non-Participant Observation

Observes actions of study participants with limited interference.
Example: Observing how employees react to a new documentation software system

Three types: Naturalistic observation, simulation observation, case study

Naturalistic Observation

Observing behaviors or actions that occur naturally in the environment
Useful to see if study participants are following a procedure, rule, law, policy
Participants should not know what the researcher is observing or when in order to simulate the “normal” environment.

Naturalistic Observation Example

We examined the reasons for underutilization of the cancer registry

Informal discussions with physicians
Naturalistic observations of cancer registry
Semi-structured interviews with physicians and nurses
Compile data to enhance the new cancer registry software system

Can anyone think of another example of naturalistic observation in HIM?

Simulation Observation

Naturalistic observation conducted in environment created for participants
Hackett, Parmanto et al, 2007 dissertation research observed individuals with visual disabilities using different websites chosen for the participants. Goal is to see if the transcoder is effective in improving websites for individuals with visual disabilities.
Usability study---different tasks developed

Usability Studies

How fast can the learner use the PHR system (Ease of learning)?
How fast can the learner accomplish the tasks? (Efficiency of use)?
How effectively can the user learn the PHR system upon several different uses? (Memorability)
How often do errors occur, how serious and what do users do to manage the error? (Error frequency and severity)
How much does the user like using the system? (Subjective Satisfaction)

(Hackett et al, 2007, Dissertation Research, An Exploration into Two Solutions to Propagating Web Accessibility for Blind Computer Users)

Case Study

Non-participant observation
Individual, group, institutional case study
Collects demographic, disease, religious, social, cultural, technological system and software, community etc.
Field notes, tape recordings, video, images, etc.

Examples Case Study

Who can provide an example of an individual, group or institutional case study?

How does the case study differ from the ethnographic analysis?

Focused Interview

Informal conversational

Standardized open-ended

General Interview guide

Informal Conversational

No set questions developed, moves forward based on what the study participant would like to discuss

Example: What is the meaning of living with dementia while in a nursing facility?
Other examples in HIM?

Standardized Open-Ended

Specific questions are used to interview the study participants

Example: Used to study automated coding software and the potential to decrease fraud and abuse---See examples

Examples of Standardized Open-Ended

Automated Coding Study:
Government:
What problems do you foresee in relation to fraud and abuse when the Electronic Health Record (EHR) is used?
Are you aware of incorrect coding or abuse detected with Natural Language Processing (NLP)? If you are familiar with the approach of the NLP, was it a rules-based approach or data-driven approach? Please describe.

Examples of Standardized Open-Ended

Vendor
What type of automated coding system do you provide?
When was your first installation of the automated coding system? How many installations (users/clients) do you have and in what settings?
What is the average installation and training time?

Examples of Standardized Open-Ended

User
What is the level of accuracy on coding and billing?
How is the automated coding system used with the EHR?
How is the automated coding system used within the coding and billing process?
What are the anti-fraud features available and how do they link to the automated coding system?

Other Examples—Standardized Open-Ended

Other examples of how the standardized open-ended interview could be used in HIM?

General Interview Guide

Outline of issues is used to conduct the informal interview

Examples: Medical schools used the general interview guide to reassess team based learning.
Other examples in HIM?

Focus Group

A focus group is a group of subjects, usually experts in the particular area of study, who are brought together to discuss a specific topic.

Example: Focus group of experts used in evaluating ICD-10-CM and its effectiveness in capturing public health related diseases—See example

Example of Website Tables – Alzheimer’s progression

Step 1

Step 2

Step 3

Questions

1. After review of the public health diagnoses reportable list, are there any diagnoses that should be added, deleted or changed? If so, please explain.

2. Do the hypotheses and explanations that relate to the coding of the reportable diagnoses provide enough information so that changes to the coding system can be made? If not, please specify which sections need further detail.

3. Do the ranked scale data and explanations related to differences in the ICD-10-CM and ICD-9-CM coding systems make sense? Do you need additional information to clarify any cases? If so, which ones.

4. Based on the information provided to you, what recommendations do you have to improve the ICD-10-CM for public health reporting?

Focus Group

Other ways focus group can be used in HIM

Participant Observation

Observer part of environment being observed
Will true feelings etc. be observed when participant is also observer?

Example of Participant Observation

Hofler et al, 2005 HIPAA compliance

Ethnography

Delving into a particular culture or organization in great detail in order to learn everything there is to know about them and to develop new hypotheses.

Ethnography

Not objective
Includes opinions of researcher
No two ethnographers will examine a specific culture or organization the same way
Focuses on people, culture, life

Field Notes

Brief notes written at field site
Description (everything researcher can remember about the event such as a meeting, encounter etc.)
Analysis (linking step 1 & 2 to research questions)
Personal viewpoints

(Hall, 2007)

Ethnography

Can include both qualitative and quantitative approaches
Participative and non-participative methods
With participative: extensive field notes
Open-ended and unstructured interviews
Documents pertinent to setting

Ethnography--Cyclical

Develop

Tentative

Hypotheses

Review

Literature

Collect Data

Analyze Data

Revise tentative hypotheses

Create new hypothesis

Day in the Life

Let’s discuss our ethnographic examples

Ethnography Questions

What parts of the ethnography did you find the most difficult to collect and why?
Do you think the ethnographic method is a good way to collect research data? Why or why not?
Do you think the ethnographic method is the best method to use when collecting observational research data related to health care and health informatics? Why or why not?

Analysis of Data

Content Analysis
Examine textual data to detect recurrent terms and emerging themes reflective of culture or facility
Examples
Automated coding study

Constant Comparative Method

Grounded Theory
Comparing incidents applicable to each category
Integrating categories
Define the theory
Writing the theory

Example--CCM

Tang, 2007—Patients with Colorectal Cancer
Open analysis of interviews
Transcripts read and words reflected respondents’ ideas and thoughts labeled
Patterns derived from actual wording
Categorizations developed from interpretation and grouping of codes
Words with similar meaning grouped into categories for further analysis
Grouping categories developed
All information reflected back to theory or objectives of study

Software
Content Analysis

Count word frequencies
Category frequencies
Cluster analysis—groups together words used in similar contexts
Co-word citation—examines occurrence of pairs of words
May be expensive and require training

Summary

Non-Participant Observation: Observes actions of study participants with limited interference. Example: Observing how employees react to a new documentation software system
Participant Observation: Researcher is a part of the environment he or she is observing Example: Used to assess how well employees in health care facilities abide by HIPAA

Summary

Naturalistic: One example of non-participant observation in which behaviors and events are recorded as they occur naturally in the normal environment
Example: naturalistic observations conducted to determine whether the cancer registry is underutilized
Simulation: Observing participants in an environment that has been created for them rather than their normal environment
Example: observing individuals with visual disabilities use different websites to determine if the websites are accessible.

Summary

Case Study: Non-participant observation when the researcher wants to thoroughly assess an individual, group, or institution.

Individual: Record and collect as much information as possible about a particular individual as they progress through a certain disease, procedure, treatment, cultural or system change.

Example: individual patient is followed and evaluated to determine how they use an assistive technology device.

Group: Very similar to individual case study except that interviews or observations are conducted on a group of individuals

Example: Assessing the moral reasoning skills of medical students using ethical issues.

Institutional: Observing a particular health care institution or facility to determine how it conducts a particular process, system or procedure.

Example: Department of Veterans Affairs health care system used the institutional case study method to describe how their electronic health record system is used in home-based primary care programs

Summary

Focused Interview: Interview used in observational or qualitative research in to collect in-depth, rich, robust information.
Focus group: Group of subjects, usually experts in the particular area of study, who are brought together to discuss a specific topic.
Example: Focus group of experts used in evaluating ICD-10-CM and its effectiveness in capturing public health related diseases.
Informal conversational: No set questions developed, moves forward based on what the study participant would like to discuss
Example: What is the meaning of living with dementia while in a nursing facility?
Standardized open-ended: Specific questions used to interview the study participants.
Example: Used to study automated coding software and the potential to decrease fraud and abuse
General interview guide: Outline of issues used to conduct the informal interview
Example: Medical schools used the general interview guide to reassess team based learning.

Summary

Ethnography: Delving into a particular culture or organization in great detail in order to learn everything there is to know about them and to develop new hypotheses.
Example: Used to assess interactions between physicians and patients when using the EHR.

Questions

Experimental and
Quasi-Experimental Research

Chapter 5

Experiment

Begin with a hypothesis
Test it
Refine hypothesis
Test again
Reach conclusions
Try to establish cause and effect

Experimental Research

Most powerful when trying to establish cause and effect
Expose participants to different interventions
In order to compare the result of these interventions with the outcome

Independent Variable

The intervention or factor you wish to measure in order to determine if it will have an effect on the outcome or disease under study.
Examples: medications, diet, exercise, education, health information system

Dependent Variable

The outcome, end point or disease under study
Examples include: survival time for patients with cancer, reduction of pressure sores in patients using specific type of wheelchair, decrease in the number of adverse events in health care facilities using a CPOE

Dose-Response Relationship

Experimental research also tries to determine a dose-response relationship.

If a new medication has slowed the progression of cancer, will a higher dose slow the progression even faster?

Or if a specific factor is removed from the environment it may also decrease the progression of a certain disease.

Use Experimental Research

Consider the following:

Eligibility of appropriate participants

Randomization

Ethical Issues

Quasi-Experimental Research

Similar to experimental research but does not include randomization of participants.
Independent variable may not be manipulated by the researcher, and there may be no control group
It may be used over time with something other than individual participants

Example of Quasi-Experimental Research

Study the effects of automated coding system to determine if there is an increase in hospital reimbursement before and after the system is implemented
Study the cost/benefits of using an EHR before and after its implementation. Cost/benefits of a paper-based system is compared to the cost of an EHR in all HIM functions.

R = randomization

O = observation

X = intervention

Overview of Experimental Research Designs

Study Design	Characteristics	Diagram
Pretest-posttest control group method	Randomly assigned to intervention or non-intervention (control) group Pretests given to both groups Posttests given to both groups after intervention	R---O---X---O R---O--- ---O
Solomon four group method	Two intervention groups Two control groups Randomization used to assign to all four groups Pretest for one pair of intervention and control groups Same intervention used in both groups Posttest used in all four groups	R---O---X---O R---O--- ---O R--- ---X---O R--- --- ---O
Posttest only control group method	Randomization used for assignment into intervention and control groups No pretest given Intervention given to one group only Posttest given to both groups	R--- ---X---O R--- --- ---O

Overview of Quasi-Experimental Study Designs

Study Design	Characteristics	Diagram
One-shot case study	Simple design One group Intervention Posttest	---X---O
One group pretest-posttest method	One group Pretest Intervention Posttest	O---X---O
Static group comparison method	Two groups Intervention No intervention Posttest for both groups	---X---O --- ---O

Elements

Randomization
When study participants are randomly chosen to be in the experimental, control, or comparison group using a random method, such as probability sampling, so that each participant has an equal chance of being selected for one of the groups.
Intervention = experimental group
No intervention = control group
Different intervention = comparison group

Example Randomization

Researcher may be interested in determining whether individuals retain more if they do higher levels of exercise before they learn how to use the PHR.
Three different groups of participants will be established.
First group will run for thirty minutes before sitting down in front of the computer to learn to use their PHR,
Second group will walk for thirty minutes before learning to use the PHR,
Third group will not do any type of exercise before learning to use the PHR.
Therefore, the first group is called the experimental group, the second is called the comparison group and the third is called the control group and randomization will be used.

Example Randomization cont’d

Develop a list of all the study participants and number them
Pick out each number and allocate to a particular group. For example, the first number drawn will go into the experimental group, the second number to the control group and the third number to the comparison group and so forth until all the numbers are drawn and participants allocated.
The goal is to have the experimental, control, or comparison groups as similar as possible except for the intervention under study.
Randomization techniques can be performed using statistical software programs

Comparison Group

May be unethical to withhold a certain intervention from one group of participants,
Some experimental research studies do not contain a control group but instead use two comparison groups.
The intervention under study is still used but all members of the study are receiving some type of intervention.
For example, if researchers are assessing the effect of using RHIOs to decrease the incidence of hospital-acquired infections in four nursing facilities in a particular region, then two of the nursing facilities will use the RHIO based data and two of the other nursing facilities will need to utilize some other type of database in order to minimize the unethical consequences of not providing any type of data.

Cross-over Design

A cross-over design can also be used to minimize the unethical effects of not providing certain types of interventions.
It includes using one group of participants as both the experimental group and the control group. A group of participants start out by being assigned to the experimental group and receive this intervention for a certain period of time, such as 6 months or a year. After they receive the intervention, they cross-over to receiving no intervention or another comparison intervention for another 6 months to a year.

Example Cross-over Design

May be used when studying whether certain types of telerehabilitation will improve the outcomes of patients with multiple sclerosis.
Patients may start out using sensors and body monitoring and then cross-over to using a PDA or the traditional in-house therapy monitoring in order to monitor their functional levels after treatment.
The same group is used as the control (or comparison group) and the experimental group.

Observation

Pretest—observing the experimental and control or comparison groups before the intervention
Posttest—observing the experimental and control or comparison groups after the intervention

Examples Observation

Blood pressure taken before and after the administration of medication, diet, or exercise
Questionnaire given to determine levels of depression before and after a medication intervention.
Observing a group of individuals before the administration of a policy or procedure change and then observing them again after the change has been in place for one month.
Other types of observations are not administered before or after the intervention but during the middle of the particular study. These observations are called midtests.

Other observations may be conducted several months or years after the intervention ends to determine its long term impact.

Other observations may be conducted throughout the study period as a new policy or law is implemented. These are called time-series tests. For example, a researcher might use the time-series test to examine the number of breaches of confidentiality after the implementation of HIPAA. These rates could be compared to rates before HIPAA was implemented.

Control Group

Use of the control group allows the researcher to determine if the effect seen is really due to the intervention and not other extraneous factors or confounding variables.
In clinical trials when medication is being tested as the intervention, the control group is given a placebo so that they are as similar as possible to the intervention group but not receiving the medication under study.

Treatment

Treatments or interventions are also the independent variable.
Use of experimental medications, changes in an individuals’ behavior such as smoking or alcohol cessation, or changes in a particular assistive device, technology, software or system.
Should be administered in the same way for all participants in the experimental group.
For example, if physical therapists are going to be educated and trained on-line in using a new rehabilitation EHR system, the level (hours of training), quality (content of the on-line education and training), and hands-on application (amount of time using the EHR system) should be the same for all physical therapists in the experimental group.
The control group may consist of those physical therapists that will receive the traditional in-class education and training.
The hypothesis is that those physical therapists trained on-line or with distance education will be the same or better than those trained using the in-class method.

Experimental Studies
Pretest/Posttest Control Group Method

Similar to the randomized controlled trial (RCT) or clinical trial
Pretest-posttest control group method provides an intervention that may include a specific program or system change than a medication or treatment.
Participants are randomly assigned to either the intervention (experimental) or a non-intervention (control) group.
Control/Comparison group may receive a different intervention other than the one under study.
Pretests are given to both groups at the same time to assess their similarities and differences.
Posttests are given to both groups to determine the effect of the intervention.

Example in Health Informatics

Shegog et al, 2001---Assessing the impact of a computer-assisted instruction (CAI) program on factors related to asthma self-management behavior.
Baseline data collected on asthma self-management skills
Children in experimental group used CAI program
One week later post-test data (assessment of asthma self-management skills and attitudes toward CAI collected

Solomon Four-Group Method

Two experimental groups which both receive the intervention
One group receives a pretest and posttest while the other experimental group receives a posttest only.
Two control groups are also used in this design
One control group receives a pre and posttest while the other group receives the posttest only.
All participants are randomly assigned to the groups.
This method controls for pretest exposure but also requires more time, effort and cost due to the additional groups.

Example of Solomon Four Group

Adaptation of the Solomon four-group design was used by researchers evaluating the effectiveness of a multi-media tutorial in the preparation of dental students to recognize and respond to domestic violence (Danley et al, 2004).
First experimental group of dental students was randomly assigned to take the pretest, the tutorial (intervention), and then a posttest.
The second experimental group first took the tutorial and then the posttest.
The third group (control group) took the pretest and then the posttest.

Posttest-Only Control Group Method

Participants are randomly assigned to an experimental group or a control group and posttests are the only means of observation.
No pretests are used.
This is done to reduce the effect of familiarity with exposure to a pretest.
Not using a pretest eliminates the ability to assess an improvement in scores from before the intervention to after the intervention.

Example of Posttest-Only Control Group Method

The experimental posttest only control group method was used by researchers assessing the effect of community nursing support on clients with schizophrenia (Beebe, 2001).
24 participants randomly assigned to control group (routine follow-up care and informational telephone contact at 6 and 12 weeks) and
Experimental group(weekly telephone intervention plus routine follow-up care for 3 months).
All were followed for 3 months after hospital discharge to determine the length of survival as well as frequency and length of stay for re-hospitalizations.

Quasi-Experimental Studies
One-Shot Case Study

This study is a simple design in which an intervention is provided to one group which is followed forward in time after intervention to assess the outcome (posttest).
No randomization, no control group, and no pretest is included
No baseline measurement to provide a comparison to the intervention outcome.

Example of One-Shot Case Study

Researchers conducted a quasi-experimental one shot case study to determine if an automated two-way messaging system will help HIV-positive patients comply with complex medication treatments (Dunbar et al, 2003).
19 HIV-positive patients enrolled and received two-way pagers that included reminders to take all medication doses and follow any dietary requirements.
No control group
Outcome measures consisted of the number of times participants reported missing one or more medication doses, medication side effects, and participant’s satisfaction level in using the messaging system.

One-Group Pretest-Posttest Method

Similar to one-shot case study except that the pretest is used before the intervention.
No control group and no randomization.
Used when it is unethical or inappropriate to withhold the intervention from a group of participants.

Example of One-Group Pretest-Posttest Method

Researchers assessed the timeliness and access to healthcare services using telemedicine in individuals aged 18 and younger in state correctional facilities (Fox, et al, 2007).
Data were collected one year before implementation of the telemedicine program and two years after implementation.
The telemedicine intervention consisted primarily of remote delivery of behavioral health care services.
Timeliness of care and use of healthcare services before and after telemedicine implementation was examined.
The data was collected primarily from medical records and other claims and information assessment logs.

Static Group Comparison

Two groups are examined;
One with the intervention
One without the intervention
Posttest is given to assess the result of the intervention.
There are no pretests and no randomization but a control group is used.

Example Static Group Comparison

Researchers assessed the use of alcohol in patients after a traumatic brain injury (TBI) based on patients’ and relatives’ reports (Sander et al, 1997).
This study examined the validity of patients’ reports by comparing them to relatives’ descriptions of post-injury alcohol use.
In this design, researchers use the brain injury as the intervention and then assess via a post-injury questionnaire whether drinking habits as perceived by the patient with the TBI and the close relative are similar or different.

Internal and External Validity

Internally validity demonstrates that the dependent variable (outcome measure) is only caused by the independent variable (intervention) rather than other confounding variables.
External validity is concerned with being able to generalize the results to other populations

(Campbell and Stanley, 1963).

Factors Affecting Internal Validity
History

History or the events happening in the course of the experiment that could impact the results.
Researcher collects level of functioning data on hip replacement patients before and after the use of a new physical therapy device. During the time that this device is being used, the developer becomes ill and unable to fully train all physical therapists in its proper use. Therefore, the study may be affected by inadequate time in training rather than the device itself.

Factors Affecting Internal Validity
Maturation

Maturation and refers to the natural changes of research subjects over time due to the length of time that they are in the study.
For example, older individuals may become very fatigued after completing a training session on using a computer to manage their finances. Their fatigue could then affect their responses on the posttest.

Factors Affecting Internal Validity
Testing

Testing is the effect created once exposed to questions that may be on the posttest.
For example, participants of a study that is assessing whether a course module on the use of privacy and security within the electronic health record (EHR) improves their knowledge content of this subject, use a pretest and posttest to assess whether there is improvement due to the course module.
However, since the students are already exposed to the pretest and are able to think of some of the test questions, they may change their answers on the posttest and do better by learning from the pretest.
Therefore, the use of the pretest is what may be causing the improvement in test scores more so than the course module on privacy and security of the EHR.

Factors Affecting Internal Validity
Instrumentation

Instrumentation--Changes in instruments, interviewers, or observers may all cause changes in the results.
For example, interviewers may probe for answers more from one individual they are interviewing more so than others, if training is not performed consistently across all interviewers.

Factors Affecting Internal Validity
Statistical Regression

Statistical regression or regression toward the mean is when extreme scores of measurement tend to move toward the mean because they have extreme scores, not because of the intervention under study.
For example, coders who performed poorly on the ICD-10-CM coding exam are selected to receive training. The mean of their posttest scores will be higher than their pretest scores because of statistical regression not necessarily because of the ICD-10-CM training session.

Factors Affecting Internal Validity
Selection

Selection is when there are systematic differences in the selection and composition of subjects in the experimental and control groups based on knowledge or ability.
For example, one group of subjects who have viewed an instructional video on how to give themselves insulin injections is compared to another group which has not watched this video. No randomization is used.

Factors Affecting Internal Validity
Attrition

Attrition is the withdrawal of subjects from the study. Those individuals who leave a study can be very different than those who remain in the study and the characteristics of these individuals can affect the results.
For example, a study which focuses on trying to reduce the number of incomplete medical records due to incomplete nursing documentation have 15 nurses leave the experimental group and 2 nurses leave the control group. The 15 nurses who leave the group may be very different than those who remain in the experimental group.
Also, the difference in the numbers of nurses who leave each group may be a problem.

Factors Affecting Internal Validity
Interaction

An interaction of factors or a combination of the factors discussed above may also lead to bias in the final results.
Therefore, the researcher needs to be aware of the effect of a combination of some of the factors discussed above and their impact on internal validity

(Shadish and Cook, 1998, Key, 1997, Shi, 1997).

Factors that Affect External Validity

Testing
Selection bias
Participants are chosen who are frequently under medical care
Volunteers
Participants who receive compensation
All may be different than the general population

Control for Internal and External Validity

Randomization—most powerful to control for selection, regression to the mean, interaction of factors, improves external validity because subjects are not pre-selected but uses random assignment
Use of control or comparison groups---help control for effects of history, maturation, instrumentation, interaction of factors

(Key, 1997, Shi, 1997)

Poor Experimental Procedures

Control group exposed to part of the intervention
Multiple treatment interference
Length of time of treatment intervention
Loss of participants

Summary

Experimental study designs are one of the most powerful designs to use when trying to prove cause and effect.
Quasi-experimental study designs are also very effective but tend to have many more problems with external validity since most do not include randomization of subjects
Researchers in health informatics choose to use the quasi-experimental design for many reasons such as ethical considerations, the difficulty in randomization of subjects and small sample size (Harris et al, 2006).
Several examples of experimental and quasi-experimental studies and the methodology used in the health informatics and healthcare setting demonstrate that this study design is a viable option for health informatics research.

Epidemiological Research

Chapter 6

Introduction

Epidemiology examines patterns of disease occurrence in human populations, and the factors that influence these patterns in relation to time, place, and persons.
Essential tool when developing specific research methodologies in health informatics.
This chapter provides examples of epidemiological principles to study disease and health informatics.

Types of Epidemiology

Epidemics -- what caused them and how they could be controlled and prevented.
Expanded rapidly beyond the study of infectious diseases into the study of all types of illnesses.
Cancer epidemiology; pharmaco-epidemiology; environmental epidemiology, nutritional epidemiology; chronic disease epidemiology, health services epidemiology,

Epidemiology and Health Informatics

Epidemiological principles can be used to study any type of behavior, outcome, occurrence, community, or healthcare system. The key is to know which epidemiological study design to use to inspect a particular problem.
Epidemiological principles and study designs are used to examine many of the health informatics systems and structures that sustain the healthcare system today.

Example

Researchers (Bell et al. 2003) used a cross-sectional study to determine whether physician offices located in high-minority and low-income neighborhoods in southern California have different levels of access to information technology than offices located in lower-minority and higher-income areas.

Example cont’d

Use epidemiological principles similar to those that Snow developed.

Researched physician offices in targeted geographic areas and neighborhoods to determine the use of different types of health information technology.

Even though they did not establish the cause of any particular disease, they determined whether or not socioeconomic demographics play a part in the use of information technology.

Infectious Disease Model

Gordis 2004, 16; Lilienfeld 1994, 37-38).

Host

Agent

Environment

Age, gender, race religion,

Marital status, ethnicity, genomics,

Social behaviors, anatomy & physiology

Prior illness or disease

Nutritional, chemical,

Physical, infectious

Physical Environment

Tornado, flood, hurricane, war

Occupational Environment

Infectious Disease

Chronic Disease Model

Component I

Component II

Component IV

Component III

Chronic disease model:

Example of lung disease

Smoking or

Air Pollution

Family History or Genetics

Infant respiratory infection

Poor Nutrition

Using the Epidemiological Models of Causation in Health Informatics

Epidemiological model: Health informatics example—

Computer-assisted coding (CAC)

Host

Agent

Environment

Experience

Training

Understanding of CAC system

Computer problems

Coding errors in system

User friendly

Encoder issues

Documentation in EHR incomplete

Structured text

Free text

Artificial Intelligence

Chronic disease model:

Example of reluctance to use PHR

Privacy and Security Issues

Inaccessible systems

Unfamiliar with computers and PHR systems

Additional Time to Develop and Use

Epidemiological Study Designs

Descriptive Study
Cross-sectional or prevalence study
Analytic Studies
Retrospective (Case-Control) Study
Prospective Study
Experimental Study
Clinical and community trial

Progression of Epidemiological Study Designs

Analytic Study Design:

RetrospectiveCase-Control
Prospective Study
Historical-Prospective Study

Experimental Study:

Clinical Trial
Community Trial

Descriptive Study Design:

Cross-Sectional
Prevalence

Components of the cross-sectional or prevalence study

Describes health characteristic at one point or period in time
Generates hypotheses
Determines whether the disease or health characteristic exists now
Generates new ideas
Performed when very little is known about a topic
Excellent design when studying new concepts in health informatics
Leads to analytic studies

Prevalence Rate

Below is an example of how a prevalence rate is determined:

Number of U.S. ambulatory healthcare facilities that use digital radiology systems

Number of ambulatory care facilities in the US

where N = 1000 if expressing the rate per 1,000 facilities, 10,000 if expressing the rate per 10,000 facilities, and so forth.

X N

Sensitivity and Specificity

Sensitivity and specificity rates can be used in prevalence studies when assessing correct measurement or correct labeling.
True Positives (TP): Correctly categorize true cases as cases (cases are individuals with the disease or outcome) = VALID labeling
False Negatives (FN): Incorrectly label true cases as non-cases (non-cases are those individuals without the disease or outcome = INVALID labeling

Sensitivity and Specificity

True Negatives (TN): Correctly label non-cases as non-cases = VALID labeling
False Positives (FP): Incorrectly label non-cases as cases = INVALID labeling
Sensitivity = Percentage of all true cases correctly identified where TP/(TP+FN)
Specificity = Percentage of all true non-cases correctly identified where TN/(TN+FP)

(Lilienfeld and Stolley 1994)

Example of Prevalence Study

The American Hospital Association (AHA) (2007) conducted a prevalence study by surveying AHA member hospitals to determine their use of health information technology. Survey instruments were sent to hospital chief executive officers (CEOs) from all types of hospitals and from different geographic areas across the country.

Analytic Study Designs: Case-Control (Retrospective)

Steps to follow when conducting a case-control (retrospective) study

Step 7	Design the instrument used to collect the exposure or risk factor data. Collect it through phone or in-person interviews, self-report questionnaires, abstracts from existing sources such as the EHR, cancer registry, birth certificates, death certificates, financial records and so forth.
Step 8	Analyze the data to include the appropriate statistics.
Step 9	Summarize the results and determine if they support or refute the hypothesis
Step 10	Publish the results

Step 1	Determine the hypothesis and decide whether to use prevalence (existing cases of disease) or incidence cases (new cases of disease)
Step 2	If prevalence cases, seek out cases from the state or hospital-based cancer registry. If incidence cases, have health care facilities provide new cases as they are treated.
Step 3	Decide who will be part of the study by using inclusion criteria such as ICD-9-CM codes, laboratory reports, radiology reports, medical records, and so forth, which all validate the disease under study.
Step 4	Randomly select the cases by obtaining a list of possible cases (either from the state or hospital-based cancer registry or from a list of ICD-9-CM codes and so forth) and use a systematic sample by choosing every 5th case.

Analytic Study Designs: Case-Control (Retrospective) cont’d

Step 5

Choose controls from siblings or friends, who are of similar age, gender, socioeconomic status, or from the same hospital. Controls should be similar to the cases for all characteristics except the disease under study. For example, if studying melanoma, choose controls from the same hospital-affiliated cancer registry as the case but who has another type of cancer such as colon cancer or lung cancer. Select these controls from a list of cancer cases identified by their ICD-9-CM code and validate the diagnosis through pathology reports and medical records. Also, choose controls from this list that are similar in age by at least five years.

Step 6

Decide whether matching of the cases and controls will be used on certain variables. Matching on variables such as age, gender, race and so forth should only be used when the researcher is certain that there is a relationship between that variable and the dependent variable. For example, age is always related to cancer because as we age, our chance of developing cancer increases. Therefore, age becomes what is called a confounding variable because it may be the underlying factor that is leading to the development of the cancer instead of the specific risk factor that one is trying to prove is related. Therefore, when studying cancers, matching should be done for age.

Analytic Study Designs: Case-Control (Retrospective) cont’d

Step 7	Design the instrument used to collect the exposure or risk factor data. Collect it through phone or in-person interviews, self-report questionnaires, abstracts from existing sources such as the EHR, cancer registry, birth certificates, death certificates, financial records and so forth.
Step 8	Analyze the data to include the appropriate statistics.
Step 9	Summarize the results and determine if they support or refute the hypothesis
Step 10	Publish the results

Example—Odds Ratio

The odds ratio for this example is:

AD = (200) x (100) = 20,000 = 4

BC (500) x (10) 5,000

The value of 4 means that those individuals that use tanning lamps are 4 times more likely to develop melanoma than those individuals who do not use tanning lamps.

If the odds ratio for this particular example equaled 1, then it means that the risk for melanoma is actually equal for the cases and controls and that use of tanning lamps is not a risk factor for melanoma. If an odds ratio is less than 1.0, this means that the factor (the use of tanning lamps) actually decreases the risk of disease and provides a protective effect.

Example: Case-Control Study in Health Informatics

Hippisley-Cox et al. (2005) used the case-control design to examine the relationship between myocardial infarction (MI), and use of non-steroidal anti-inflammatory drugs (NSAID). The authors of this study used a research database called QRESEARCH to examine this relationship.

Cases were patients aged 25 to 100 identified as having an acute myocardial infarction (MI) for the first time recorded from Read Codes (similar to SNOMED codes) during a four-year study period.

Controls were those individuals with a diagnosis of coronary heart disease, but without an MI, matched to each case by age, year, gender, and physician practice.

Odds ratios were computed.

Confounding variables also were collected and controlled for, and include smoking and comorbitities such as diabetes, hypertension, coronary heart disease, osteoarthritis, rheumatoid arthritis, and obesity.

Cohort (Prospective) Study Design

This study design has two groups of study participants:
One with the exposure (independent variable)
One without the exposure (dependent variable).
Both groups are then followed forward in time to determine if and when they develop the disease or outcome variable under study.

Calculating the Relative Risk

The calculation for the relative risk is:

Incidence rate of the exposed group

Incidence rate of the unexposed group

The incidence rate is:

Number of new cases of a disease over a period of time

Population at risk

where N = 1000 if expressing the rate per 1,000 people, or N = 100,000 if expressing the rate per 100,000 people and so forth.

Population at risk refers to those free of the disease at the start of the study.

X N

Calculating the Relative Risk cont’d

The calculation for the relative risk is:
Incidence Rate of Exposed = A
(A+B)
Incidence Rate of Unexposed = C
(C+D)
Relative Risk: [A / (A + B )]
[C / (C + D)]

Calculating the Relative Risk cont’d

The relative risk for this example is:
Incidence Rate of Exposed = 200
275
Incidence Rate of Unexposed = 25
325
Relative Risk: [0.727]
[0.077] = 9.4
In this hypothetical example, the relative risk of 9.4 is very high for the association between use of video games and migraine headaches, and those children who play video games are almost nine times more likely to develop migraine headaches than those who do not play video games.

Prospective Study Example in Health Informatics

Baxt et al. (1996) conducted a prospective study that compared the accuracy of physicians diagnosing patients with acute myocardial infarction (MI) to an artificial neural network.

Compared data collected by physicians when evaluating 1,070 patients who entered the emergency department of a teaching hospital in California with anterior chest pain to the neural network diagnosis of the same patients.

Patients were o followed over time by review of their medical records in the outpatient department or via telephone to determine their final diagnoses, which were validated by serum creatine kinase levels and EKG evidence.

Experimental Study Designs in Epidemiology

Experimental research studies expose participants to different interventions (independent variables) to compare the result of these interventions with the outcome (dependent variables).

Two examples of experimental research studies in epidemiology include the clinical and community trial.

Clinical Trials

Clinical trials are designed to help healthcare professionals test new approaches to the diagnosis, treatment, or prevention of different diseases.

Patients who are at high risk for developing these diseases are often the ones who participate in the clinical trial.

The clinical trial is designed to test new medications (most common) and surgical procedures, as well as new treatments or combinations of treatments to prevent disease.

Community trials

Very similar to clinical trials but take place in a particular community and have less control over the intervention than one would have with the clinical trial.

The community trial’s goal is to produce changes in a specific population within a community, organization, or association.

Participation includes all members of the community and the intervention tends to be provided throughout the population

(Friis et al. 2004, 322-323; UPMC 2008).

Clinical and Community Trial Protocol

Rationale and background

Specific aims

Randomization

Blinding or masking

Types and duration of treatment

Number of subjects

Criteria for including and excluding participants

Outline of treatment procedures

Clinical and Community Trial Protocol

Procedures for observing and recording side effects

Informed consent

Analysis of data

Dissemination of results

Types of Clinical Trials

Treatment trials test experimental treatments, new combinations of medicines, different types of surgery, radiation or chemotherapy

Prevention trials aim to prevent disease in a person who has never had the disease or to prevent it from advancing or reoccurring.

Diagnostic trials are conducted to find better tests, procedures, or screenings to detect a disease or condition.

Types of Clinical Trials cont’d

Screening trials examine the best method to detect diseases or health conditions.

Quality of life trials explore methods used to improve comfort and the quality of life for individuals with a chronic disease

Clinicaltrials.gov 2007

Phases of Clinical Trials

Phase I, II, III, or IV based on the size of the population and the intervention being tested. The FDA provides guidelines for the different types of clinical trials.

Phase I clinical trials usually test a new drug or treatment in a small group of people (20-80)

Phase II clinical trials study the intervention in a larger group of people (100-300)

Phase III the study drug or treatment is given to even larger groups of people (1,000-3,000)

Phase IV clinical trials include studies that collect additional information after the drug has been marketed, such as the drug’s risks, benefits, and optimal use.

Clinical Trials in Health Informatics

Shea (1996) examined several different experimental studies comparing computer-based clinical reminder systems to manual reminder card-type systems in ambulatory preventive care settings.

They found 16 different randomized controlled trials from 1975-1994 in which computer-based clinical reminders were used for several different preventive services such as hypertension follow-up, influenza vaccine, pneumococcal vaccine, mammography, fecal occult blood test, pap smear, tetanus vaccine, dental screening, smoking assessment, dietary assessment, and so forth.

They found that computer-generated reminders when compared to manual reminders increased preventive practices by 77 percent.

The use of the experimental study to examine the effectiveness of the computer-based clinical reminder system provided thorough evaluation of this system.