Pre-registration assignment
Final Year Project Preregistration Assignment
Dr Andrew Russ
1
Lecture Overview
The Importance of Replication in Science
The ‘Replication Crisis’ in Psychology
Open Science and Preregistration
Your Assignment
2
The Importance of Replication in Science
The process of repeating research to determine the extent to which findings generalize across time and across situations
We work under the understanding that some of our findings may be incorrect (we reduce type 1 and type 2 error but can’t eliminate them entirely under normal circumstances)
Reproducibility is a key requirement of good science, but to what extent it characterises current research is unknown (Open Science Collaboration, 2015)
3
Direct (Exact) Replication
To determine whether the results come out the same
Exactly recreate the scientific methods used in conditions of an earlier study
Conceptual Replication
Attempt to confirm the previous findings using a different set of specific methods that test the same idea
Same hypothesis is tested, but using a different set of methods and measures.
The Importance of Replication in Science
4
The Replication Crisis
Replicability crisis; reproducibility crisis
Methodological crisis in which scientists have found that results of many scientific studies are difficult or impossible to replicate/reproduce on subsequent investigation
Caused by Journals’ strong preference to publish significant findings (file-drawer effect) and acceptance of a range of analyses
5
6
Openness and System Programs
The Principles of Open Science
Open data
Open source
Open access
Open methodology
Open peer review
Open educational resources
7
The assignment
A simplified version of the preregistration challenge offered by Open Science
Overall upper word limit is 2,000 words (not 10% over)
Includes (section word limits are guides only):
Title
Research Description 200-300 words
Hypotheses/Research Question
Methodology Plan 750-1,000 words
Analysis Design 200-500 words
Ethical Design 150-500 words
8
title
Provide the working title of your study. It may be the same title that you will eventually submit as your final report, but it is not a requirement.
More info: The title should be a specific and informative description of a project. Following APA style (7th edition, Journal Article Reporting Standards) it should name the two or three main variables involved, the relationship between them being tested (making clear whether this is based on a causal or a correlational design), and the participant population sampled. Unless the research is highly exploratory in nature, vague titles such as 'Fruit fly preregistration plan' are not appropriate.
Example: Effect of sugar on brownie tastiness among University students. (This is a food lab experiment varying sugar between conditions. Because it is an experiment with random assignment to a sugar condition, you can use the causal phrase “effect of”.)
Example 2: Relationship between sugar and dessert preference in an online sample. (This is a correlational study asking for opinions of different brands of desserts that have more or less sugar in their recipe. Because it is only correlational, “relationship between” is more effective).
9
Research description
Please give a brief description of your study, including some background, the gap in knowledge which the study is designed to fill, a brief description of the kind of methods used, and the purpose of these methods. This section is like an abbreviated introduction to your study up until the hypothesis, which is entered in the next box.
Example: Though there is strong evidence to suggest that sugar affects taste preferences, the effect has never been demonstrated in brownies. Therefore, among university students in a lab session, we will measure taste preference for four different levels of sugar concentration in a standard brownie recipe to determine if the effect exists in this pastry.
More info: The description should be no longer than the length of an abstract. It can give some context for the proposed study, but great detail is not needed here for your preregistration. References are optional.
10
Hypotheses/ research question
List specific, concise, and testable hypotheses. State them as a relationship between a hypothetical situation of the real world (“If taste affects preference”) and the kind of observation that would help confirm that (“then mean preference will be higher for higher concentrations of sugar.”) Please state if the hypotheses are directional or non-directional. If directional, state the direction. A predicted effect is also appropriate here. If a specific interaction is important to your research, you can list that as a separate hypothesis.
Example: If taste affects preference, then mean preference indices will be higher with higher concentrations of sugar. This is a directional hypothesis (as the sugar variable gets higher, the preference variable does.)
Example 2: If there are age differences in preference for milkshake ingredients, then children in different years in school should show different preferences for strawberry, vanilla, chocolate, and orange. This is a non-directional hypothesis (the exact match-up between ages and flavours is not predicted) OR
11
Hypotheses/ research question
If you do not have a testable hypothesis due to the exploratory nature of your study, outline your research question. How will your study further your knowledge of this topic?
Example 3. This exploratory study will code open-ended responses to a question about a moral dilemma involving animal research, in order to produce a model of how people justify different moral concerns in this situation.
12
Methodology plan
Describe your methodology. This will look a lot like a method section written in the future tense. Make sure you are clear enough that a reader unfamiliar with this area of research or your study will understand what you will be doing. In a typical study, this should include separate sections for:
Participants. Say what population you will sample, and how (students using the RPS, online participants recruited via friends, male adults in a yoga class). Consult with your supervisor to describe how many participants you will ideally need to recruit.
Include your study design (i.e. what are the variables? Is it a within-subjects design etc?).
Describe the setting and context given to participants in general terms (“e.g. Participants will be brought into a lab and told they will be taking part in a taste test.”)
If it is an experiment, state whether any variables are manipulated between-participants, and how.
Also describe variables that will be compared within-participants, or used as dependent measures. Say how you are measuring them, for example by using questionnaires or other instruments. Sub-headings might be helpful here.
The description should flow in chronological order, and describe how manipulations can vary between subjects, so that we get an idea of what participants will be going through.
Example: We have a between-subjects design with 1 factor (amount of sugar in mg) with 4 levels. Taste perception will be measured using a rating scale with 5 levels: ….<insert some suitable rating options here>
More info: This section can be written in a variety of ways. The key is for a researcher to be as detailed as is necessary given the specifics of their study.
13
Analysis plan
First, describe as well as you are able the preparation of the data for analysis and any exclusion criteria. For example, you may have a ten-item scale of extraversion that you will average into a measure. You may be acquiring physiological data that your supervisor will help you turn into four indexes of autonomic bodily arousal. As an exclusion criterion, you might exclude participants who answer to a check item that they have failed to understand the vignette they were supposed to read.
What statistical model will you use to test each hypothesis? Please include the type of model (e.g. t-test, ANOVA, correlation, multiple regression, chi-square test, etc) and explain what will be included as predictors, outcomes, or covariates. Please specify any interactions or follow-up analyses you anticipate running. If you plan on using any controls, or manipulation checks you may mention that here. If your study will use qualitative methods how will you analyze the data?
Example: We will use a one-way between-subjects ANOVA to analyze our results. The manipulated, categorical independent variable is 'sugar' whereas the dependent variable is our taste index.
More information: This is perhaps the most important and most complicated question within the preregistration. As with all of the other questions, the key is to provide a specific recipe for analyzing the collected data. Ask yourself: is enough detail provided that someone else could run the same analysis again the same way?
14
ethical plan
Which parts of your design require a consideration of ethical issues? Consider the four major areas: Consent, right to withdraw, anonymity/confidentiality, and effects on participants. Identify any issues your study may have and put in place a plan to deal with these issues. Some projects will have very few ethical issues but all will have something that can be anticipated and dealt with.
Example: We are priming participant responses to a task by telling them they scored high/low in a preceding personality test. We will tell participants this was a deception, and will explain why this was necessary both verbally and in writing in the debrief. We will include <relevant contact information> for anyone who has been left feeling uncomfortable as a result of this study.
More information: This is not as easy as it may seem. Make sure you consider all the specific ethical concerns raised by your study (even the analysis) and carefully think through solutions at each point. Does your design require deception? Will participants be exposed to stimuli they may find distressing?
15
Secondary analyses
Complete this assignment as though you were collecting the data
It is very important you understand the rationale and methodology of the data collection for you final report.
16