Critique Treatment Outcome study NEED in 8hours

SAMPLING

For what population do you want to test the new therapy?

Students should be able to give a concrete population to which they would like to generalize their results. For example, perhaps we are interested in seeing if our treatment is effective for all veterans with PTSD. Again, further specification can make implementation easier, but of course less specification might enable broader generalization, as long as we sample well.

Describe your sample. Name 5 specific characteristics.

Students should be able to state who is in their sample, from where they are obtaining their sample, and list at least 5 characteristics that are relevant to the example. Examples may be age, gender, ethnic diversity, SES, or things more specific for your psychological disorder. They will need to have these characteristics when they consider the representativeness of the sample to the population. These answers should be clear and specific.

What does it mean to generalize? Why is generalization important?

Use the definition from the list of concepts: “Generalization means the results of the study can be applied to people other than the actual participants of the study and to circumstances outside the actual study.”

Mention external validity as it relates to generalization & representativeness. Use the vocabulary, or use the “language of experimental method.” This is why defining our population and sample clearly is important.

What specific methods will you use to recruit your participants?

Again, this will depend on your example so make sure it is appropriate for your population/sample. For example, if you are using veterans, it may make sense to recruit from the VA. Be careful not to introduce sampling bias, which leads to an unrepresentative sample, e.g. only people who use social media.

How could your methods of recruitment affect the characteristics of your sample? Why is that important?

Consider vocabulary terms like selection, how representativeness is affected, sampling bias, etc. within the context of the example. The idea is to minimize a systematic sampling bias and increase representativeness.

What incentives could you provide to increase participation? How could these incentives affect or confound the results?

Reference information about the potential effects of incentives, including sampling bias and consequent unrepresentativeness, demand characteristics, subject bias, ethics risks, and other potential confounds. Consider the possibility of misinformation/faulty data in order to continue participation and receive the incentive.

PROCEDURE (THERAPISTS)

Describe the basic procedure of your experiment.

Students can take some liberty here. They may need some assistance designing how the specific treatments can be implemented.

What specific part of the procedure will be different for the different groups?

Identify the independent variable (the therapy) only.

How will you determine who the therapists are for the different groups?

Consider competence, standardization, how therapists operationalize the independent variable, and control for experimenter effects with multiple therapists and randomization of the therapists.

How will you make sure the therapy is provided correctly?

Students should come up with strategies to operationalize treatment. Strategies could include training for competency, monitoring of therapists, and using manualized treatments.

Will the therapist know to which therapy condition specific subjects are assigned or can they be blind to the treatment condition? If they can be blind, how will you achieve this?

A therapist can not be blind to the therapy they are providing.

What characteristics of the therapist could affect the results? If so, how could you control for these potential experimenter effects?

Examples of characteristics could include: demographics, appearance, competence, personality (e.g. empathy, interpersonal style, etc.).

These variables should be controlled with randomization across these variables (except for competence), balanced conditions, and if not we could evaluate the data (e.g. by gender of therapist) after collection of the data for some of these effects.

Give an example of how the therapist’s expectations or bias could affect or confound the results?

These confounds may result in a loss of experimental control if they introduce any other variables, communicate their expectations, or even inadvertently bias the subjects (see examples from Ppt). Give special consideration to procedure and the possibility of the therapist communicating expectations.

How could you control for these possibilities?

Similar to the response above: operationalization of therapy/instruction & training for therapists, so no confounds are introduced from experimenter bias. Also, randomization of therapist assignment can help.

PROCEDURE (OUTSIDE THERAPY)

Describe the basic procedure of your experiment.

This will be specific to your example. Students may need assistance with this but encourage them to brainstorm how they would conduct the experiment.

What specific part of the procedure will be different for the different groups?

Anything outside of the different therapies is NOT part of the independent variable, so nothing should be different outside the different types if therapy. Otherwise, this will introduce confounds.

What instructions will you give to the participants?

Students should brain storm ideas but they need to make sure to standardize the instructions to have experimental control.

Who will measure or assess the dependent variable?

Again, this is dependent on the example. Students may advocate for an independent third party, who can be blinded to the treatment conditions.

Will the instructor or the assessor know to which therapy condition different subjects are assigned or can they be blind to the treatment condition? If they can be blind, how will you achieve this? Answer for each one:

The person giving instructions (instructor)

The assessor measuring variables

Students can blind the instructors/assessors by ensuring that the instructions/assessment procedure are the same and the instructor is never informed as to which treatment subjects receive, even inadvertently by the subjects themselves. For this reason, the instructions should be clearly-structured with no idle conversation.

What characteristics of the instructor or the assessor could affect the results? If so, how could you control for these potential experimenter effects?

Very similar to the previous section. Examples of characteristics could include: demographics, appearance, competence, personality (e.g. empathy, interpersonal style, etc.).

These variables should be controlled with randomization, balanced conditions, and if not they should evaluate the data after collection of the data for some of these effects.

Give an example of how the instructor/assessor’s expectations or bias could affect or confound the results?

Example may be loss of experimental control if they introduce any other variables, communicate their expectations, or even inadvertently bias the subjects (see examples from Ppt).

How could you control for these possibilities?

Examples of strategies could include: standardized (controlled) instruction & assessment procedure, so no confounds introduced from experimenter bias and study is well-controlled. Also randomization of instructor & assessor assignment (if using multiple) can help.

SUBJECTS/PARTICIPANTS

How will you determine who is allocated to which group?

Randomization : Participants should be randomly assigned so that each participant has an equal chance of being in either treatment group. This is done to reduce the likelihood of confounding variables between groups.

What would be an example of an uncontrolled group difference that could be a confounding variable? Why is that important?

Give any number of examples (gender, age, diagnosis, etc.). Mention how uncontrolled group differences are a threat to internal validity.

How can you control for group differences?

Randomization .

What expectations might subjects themselves have about the experiment’s results and how could these expectations influence the results? How could you control for this?

Address subject bias within the context of the example. Subject bias: Participants’ behavior is influenced by what they think the experimenter’s expectations are

Give an example of the “good subject” and the “negativistic (or bad) subject.”

Give an example of each that works with the experiment example.

“Good participants” behave according to what they think the experimenter’s expectations are, to help prove the hypothesis.

“Negativistic participants” behave to disprove what they think the experimenter’s hypothesis is.

If subjects are concerned about how they are being evaluated, how could this evaluation apprehension affect or confound the results? How could you control for this?

Define evaluation apprehension and come up with an example of how this can effect results. Examples could be enhanced performance, increased anxiety, falsifying data. A potential control could be reassuring participants that there is “no correct answer,” and no judgment of desirability.

What other subject motives and goals could affect the results? How could you control for these?

Students could discuss potential effect of incentives, also possible experimenter effects on subjects. Subjects’ bias might be specific to certain experimenters.

ETHICS

How would you protect the subjects’ privacy?

Anonymity should be ensured when possible, and there needs to be established confidentiality with therapist.

What possible psychological harm could occur to subjects as a result of this research? How could this risk be minimized?

This will depend on your example. A few general risks could be creating anxiety, if subjects don’t improve then therapy could be “wasting time”, treatment could make things worse.

How will you achieve informed consent?

Provide a clear general description about how they will inform participants of the study, the ability to stop anytime, what the benefits and risks are, and that they will sign statement that they understand.

Would you deceive the subjects in any way?

We should not need to deceive subjects, but if we do, we need to provide a good rationale as to why. Possible reasons could be to limit inaccuracies of reporting and to possibly omit specifics to keep participants blinded to conditions (relevant below).

What information about the study would you provide to the subjects before they participate?

As mentioned above, this should include the procedure, general purpose (treatment outcome), risk and protection from harm, ability to stop, instructions of general expectations. This should be the same for all subjects.

What information would you withhold from subjects as you describe the study before they participate? What is the purpose of withholding this information?

Be specific as to what information you omit and why. This could mean keeping participants blinded to the treatment specifics to minimize subject bias.

What would you address in your debriefing of subjects after the study?

Participants should be told about the assigned conditions, the specific purpose of the study (to test therapy differences), and possibly the results of study at a later time.

How would you address any negative effects that might occur? Be specific.

Students should come up with a strategy of how they would attempt to correct any potential problems. One strategy could be providing a therapy that is known to be effective.

What other ethical concerns can you identify for your study? How will you address these?

Possible concerns could be using incentives, withholding effective treatment, using deception, breach of confidentiality, and possible emotional distress depending on the treatment. For each ethical concern, students should attempt to find a solution.