Plan program Evaluation

Welcome to HSC 375

Program Planning and Evaluation

Fall 2020 Dr. Sharon Lipperman-Kreda, Week 8

Today’s planToday’s planToday’s planToday’s plan

• Quantitative evaluation

• Operational definitions

• Variables and types of variables

• Elements of evaluation design

• Types of research designs

• Next week

Quantitative Evaluation Quantitative Evaluation Quantitative Evaluation Quantitative Evaluation

• Uses numerical data to evaluate program outcomes

• Analyzed using statistics

• The goals and objectives guide the evaluation design: • Assess the outcomes the program intends to change

• Identify the times to measure potential effects

• Evaluators consider a range of plausible and alternative explanations after getting results

• You will not actually carry out a formal evaluation, but you will describe plans and methods to evaluate your program

Operational DefinitionsOperational DefinitionsOperational DefinitionsOperational Definitions

• A description/definition of something in terms of the operations (procedures, actions, or processes) by which it could be observed and measured

• Researcher must provide clear and concise definitions for each relevant construct of their program’s goals and objectives

• Example 1 (operational age definition): young adults • Individuals ages 18 – 25 years

• Example 2 (operational concept definition): stress • A score greater than 10 in a test developed to assess stress

What Are Variables?What Are Variables?What Are Variables?What Are Variables? • A variable is an attribute that describes a person, place, thing, or idea

• The value of the variable can "vary" from one entity to another

• It can be changed or manipulated during an experiment or intervention

Participant ID Age Number of

Discrimination

Experiences in

Past Year

Highest Grade

of School

General

Physical Health

Education Level 1=less than high school; 2=High school graduate; 3=Some college; 4=Bachelor’s degree;

General physical health 1= excellent; 2=very good; 3= good; 4= fair; 5=poor

Substance Use Among Sexual And Gender Minorities: Substance Use Among Sexual And Gender Minorities: Substance Use Among Sexual And Gender Minorities: Substance Use Among Sexual And Gender Minorities:

Association With Police Discrimination And Police MistrustAssociation With Police Discrimination And Police MistrustAssociation With Police Discrimination And Police MistrustAssociation With Police Discrimination And Police Mistrust

Abstract: We investigated associations between experiences with

police discrimination, police mistrust, and substance use in a

convenience sample of 237 sexual and gender minority (SGM) adults in

California. In a cross-sectional survey, collected between January 2016

and July 2017, participants reported substance use, lifetime

experiences with SGM-related police discrimination, police mistrust,

demographics and SGM visibility. In adjusted logistic regression

models, we found a positive association between lifetime police

discrimination and past-two-week heavy episodic drinking. Police

mistrust also was positively associated with past-month marijuana use.

Citation: Lipperman-Kreda S, Wilson I, Hunt G, Annechino R, Antin TMJ. Substance use among sexual and gender minorities: Association with police discrimination

and police mistrust. Sexuality, Gender & Policy. First published 29 June 2020. https://doi.org/10.1002/sgp2.12019.

Substance Use Among Sexual And Gender Minorities: Substance Use Among Sexual And Gender Minorities: Substance Use Among Sexual And Gender Minorities: Substance Use Among Sexual And Gender Minorities:

Association With Police Discrimination And Police MistrustAssociation With Police Discrimination And Police MistrustAssociation With Police Discrimination And Police MistrustAssociation With Police Discrimination And Police Mistrust

Abstract (continued): Several significant interactions between lifetime

police discrimination or police mistrust with other socially stigmatized

identities including being African American, insecure housing, and

being a gender minority on a few substance use outcomes suggest that

effects of police discrimination and mistrust on substance use are

stronger among participants with multiple stigmatized identities.

Results suggest the importance of policies and interventions that focus

on eliminating police discrimination and increasing police legitimacy to

reduce risk of substance use among SGM individuals.

Citation: Lipperman-Kreda S, Wilson I, Hunt G, Annechino R, Antin TMJ. Substance use among sexual and gender minorities: Association with police discrimination

and police mistrust. Sexuality, Gender & Policy. First published 29 June 2020. https://doi.org/10.1002/sgp2.12019.

Types of VariablesTypes of VariablesTypes of VariablesTypes of Variables

• Independent Variable (IV) • The variable the researcher changes, controls, or is assumed to have a direct

effect on the dependent variable

• Believed to cause the change

• Example: Experiences with police discrimination; police mistrust

• Dependent Variable (DV) • The variable that is affected by or reacts to the independent variable

• Example: daily cigarette smoking; past-month alcohol use; past-two-week heavy episodic drinking (i.e., have five or more drinks in a row); past-month marijuana use

Types of Variables Types of Variables Types of Variables Types of Variables –––– Continued Continued Continued Continued

• Mediating Variable • A variable that links the independent and the dependent variables

• Whose existence explains the relationship between the other two variables

• IV  Mediating Variable  DV

• Example: Experiences with police discrimination  stress  daily cigarette smoking

• Moderating Variable • A variable that affects the relationship between the independent variable and

the dependent variable

• Example: Other socially stigmatized identities (1) being African American, (2) insecure housing, and (3) being a gender minority

• Association between any lifetime police discrimination and past month alcohol use was significantly stronger among participants who reported housing insecurity compared to those who did not report housing insecurity

Types of Variables Types of Variables Types of Variables Types of Variables –––– Continued Continued Continued Continued

• Confounding Variables

• Another variable which may explain changes DV

• Example: Age may also explain changes in substance use

• It can be difficult to separate the true effect of the IV from the effect of the confounding variable.

• It is important to identify potential confounding variables and plan how you will reduce their impact

• Controlled Variables

• A variable which the researcher holds constant (controls for)

• So relationships may be analyzed without interference

• Example: gender identity, race/ethnicity

Important notes about types of variables:Important notes about types of variables:Important notes about types of variables:Important notes about types of variables:

• Independent versus dependent variables refer to the relation between two variables (i.e., variable A depends on changes in variable B)

• The role of specific variable (e.g., dependent variable) changes from one research to another

• Study 1 examined effects of close friends' approval of alcohol use (IV) on adolescents’ nighttime weekend drinking (DV)

• Study 2 assessed effects of nighttime weekend drinking (IV) on school performance among high school students (DV)

• Study 3 investigated how effects of close friends’ approval of alcohol use (IV) on adolescents’ nighttime weekend drinking (DV) vary by levels of school performance (Moderating Variable)

Categorical versus Continues VariablesCategorical versus Continues VariablesCategorical versus Continues VariablesCategorical versus Continues Variables

• Categorical variables contain a specific number of categories or distinct groups

• Education levels, race, ethnicity, gender, age groups

• Categorical data may not have a logical order (e.g., gender)

• Continuous variables are numeric variables that have an infinite number of potential values between any two values

• A continuous variable can be numeric or date/time

• Annual salary, weight, number of cigarettes smoked

Elements of Evaluation Design: Observations Elements of Evaluation Design: Observations Elements of Evaluation Design: Observations Elements of Evaluation Design: Observations

or Measurementsor Measurementsor Measurementsor Measurements • Observation is the process of watching/assessing an individual or

group of individuals (population) for the purpose of data collection

• A Single Observation

• When the variables are observed and recorded only once

• A school climate survey collected from students in the Spring of 2021

• Multiple Observations

• When measuring the variables several times over a define period

• Collecting pretest survey data before an intervention and then again after the intervention

• A school climate survey collected from students before and after a program to assess whether an intervention created a positive school climate

Elements of Evaluation Design: Treatments or Elements of Evaluation Design: Treatments or Elements of Evaluation Design: Treatments or Elements of Evaluation Design: Treatments or

ProgramsProgramsProgramsPrograms • In evaluation, it is common to have one group of individuals

participating in the treatment/intervention and one control group that does not get the treatment/intervention

• A research project designed to evaluate the effects of environmental prevention strategies to reduce alcohol- related problems (e.g., enforcement of DUI, party patrols) in 24 midsized US California communities – 12 communities were intervention sites (i.e., received the intervention) and 12 were controlled sites (i.e., did not receive the intervention)

Elements of Evaluation Design: Group Elements of Evaluation Design: Group Elements of Evaluation Design: Group Elements of Evaluation Design: Group

Assignment Assignment Assignment Assignment

• Prior to initiating the program, individuals, schools, communities etc. may be assigned to the treatment or control groups to assess program effects

• Multiple approaches for group assignment

• Random group assignment

• Conducted using a variety of methods

• Example: Identify two communities (community A & Community B) that match on multiple demographic characteristics (e.g., SES level, % of racial/ethnic groups, urban) and randomly assign one as intervention and one as control

Elements of Evaluation Design: Group Elements of Evaluation Design: Group Elements of Evaluation Design: Group Elements of Evaluation Design: Group

Assignment Assignment Assignment Assignment

• Quota group assignment

• Researchers use census data to determine what percent of individuals are in each group (e.g., evaluators recruit specific groups of people based on their ethnicity to reflect the US population)

• Every nth group assignment

• Researchers select individuals or data in a preset and specific pattern (e.g., in a name list of college students every 10th

person is assigned to the intervention group)

• Consider limitations such as if a list by alphabetic order those with common names may have more chances to be selected

Elements of Evaluation Design: ConstructsElements of Evaluation Design: ConstructsElements of Evaluation Design: ConstructsElements of Evaluation Design: Constructs

• A concept, thought or notion that is more challenging to measure

• Examples: self-esteem, stress, motivation, expectancies

• Formed by grouping several specific measures together

• Example: positive youth development (leadership, public speaking, community service etc.)

• Construct Validity

• The degree to which the construct measures the concept being studied

• If the construct has low validity, then the study results cannot be generalized to other study populations

• If the construct has high validity, when the construct is used with a different study population, similar results yield high generalizability

Types of Research Design: True Experimental Types of Research Design: True Experimental Types of Research Design: True Experimental Types of Research Design: True Experimental

DesignDesignDesignDesign • When study participants are assigned to groups (e.g., intervention

versus control) through a random assignment

• True Experimental Design includes:

1. Manipulating the independent variable (e.g., intervention, medication to be tested)

2. Random assignment of participants to equivalent groups

• Participants do not know their group assignment

• Double-blind study – when both participants and evaluators do not know the assignment

3. Allows to explore cause and effect relationships as random assignment ensures groups are similar and only IV is different

• Example: A clinical trial to test a treatment to COVID-19

Types of Research Design: QuasiTypes of Research Design: QuasiTypes of Research Design: QuasiTypes of Research Design: Quasi----Experimental Experimental Experimental Experimental

Design Design Design Design

• Does not use random assignment

• Researchers do not manipulate the independent variable

• May involve using multiple observations (pre- and post-intervention)

• May test existing groups (participants in three own pre-selected types of therapies: group therapy, individual therapy, online therapy)

• Cannot determine cause and effect relationships

• Cannot exclude all the reasons why a relationship does exist

• Possible confounding bias cause by the dissimilarity among groups (e.g., those who originally chose one type of therapy)

Types of Research Design: NonTypes of Research Design: NonTypes of Research Design: NonTypes of Research Design: Non----Experimental Experimental Experimental Experimental

DesignDesignDesignDesign

• The most common design

• Non-Experimental Research involves: • No manipulation of the independent variable • No random assignment of participants • No cause and effect conclusions due to alternative

explanations • Cannot determine cause and effect relationships (i.e.,

there may be other factors that can explain the relationships)

How to Plan your Evaluation?How to Plan your Evaluation?How to Plan your Evaluation?How to Plan your Evaluation?

1. What is your study population? People, schools, cities, communities etc. number, demographics (e.g., age, gender, race/ethnicity). Consider operational definitions when needed.

2. What is/are the variable/s you need to assess to evaluate EACH of your objectives?

3. Per each identified variable, please indicate its type (e.g., dependent variable)

4. What are the operational definitions of your study variables? E.g., a score test for leadership skills

5. What variables would you consider as important control variables?

How to Plan your Evaluation?How to Plan your Evaluation?How to Plan your Evaluation?How to Plan your Evaluation?

6. What is your evaluation design? (e.g., True experimental design)

7. Given your evaluation design and objectives, at what point in time would you need to assess each variable? (e.g., month/year)

8. How would you measure each of your variables as part of your evaluation? Please consider what method you will use (e.g., qualitative, quantitative, surveys, observations, interviews) and what specific measures, scales, or questions you would use

Next WeekNext WeekNext WeekNext Week

• Milestone 3 (logic model) is due by next class • view syllabus for specific due date • Week 2 class slides for definitions and examples of logic models

• Read chapter 7 and class slides • Quiz is after next class (view syllabus for specific due date)

• Next class - team meetings to work on your evaluation plan • Make sure to get the newest Zoom version • Have the questions presented for planning your evaluation ready for your teams

• Notice a small change to the syllabus – switched Discussion Activity 3 with submission of milestone 4 to allow more time to develop a draft evaluation plan

• Stay Healthy and Safe!

Logic Model TemplateLogic Model TemplateLogic Model TemplateLogic Model Template

Program: Program name

Goal: Program goal

Inputs Activities Outputs Outcomes

What we invest What we do Who we reach What we

produce

Why we do it:

Short-term

results

Why we do it:

Intermediate

results

Why we do it:

Long-term results

For example:

• Staff

• Time

• Materials

• Money

For example:

• Conduct

workshops

and meetings

• Train staff

• Work with

media

For example:

• Community

members

• Health

providers

For example:

• Services

• A Media

campaign

• Trained staff in

health clinics

For example:

• Awareness

• Skills

• Intentions

For example:

• Behavior

change

• Policies

For example:

• Improve health

• Civic engagement