Read Instructions

INVITED EDITORIAL: Let's Do It Again: A Call for Replications in Psi Chi J o u rn a l o f Psychological Research J o h n E. E d lu n d R o c h e s te r In s titu te o fT e c h n o lo g y

S c ie n c e is said to b e su ffe rin g fro m a crisis o f r e p li c a b i li t y ( I o a n n i d is , 2 0 0 5 ). T h is crisis o c c u rs w h e n s c ie n tific s tu d ie s fail

to b e s u p p o r te d by s u b s e q u e n t re s e a r c h . T h e challenges posed by th e replication crisis address the fundam ental n atu re o f science an d the p ublic’s u n d e r s t a n d i n g o f it. N u m e r o u s c o n t r i b u t i n g reasons for th e rep licatio n crisis have b e e n n o ted in c lu d in g d a ta fa lsific a tio n (S te e n , 2 0 1 1 ), th e pressures o f ten u re an d p ro m o tio n (Varian, 1998), questionable research practices (Simmons, Nelson, & Sim onsohn, 2011), th e tendency o f jo u rn a ls to w ant to publish particularly novel papers (Steen, 2011), and the p reference for publishing significant results (de W inter & H appee, 2013). T hese factors all in c re a se th e o d d s o f in a c c u ra te in fo rm a tio n b e in g p u b lish e d , w hich in tu rn is in c o rp o r a te d in to tex ts, as h a p p e n e d w ith th e d e ta ils o f th e original investigation in th e Kitty G enovese case, which led to th e fam ous bystander apathy studies (Griggs, 2015). T he p rim ary goal o f this editorial is to briefly discuss th e factors th at have co n trib u ted to th e re p lic atio n crisis, tech n iq u e s em ployed by various jo u rn a ls in th e field to deal with the crisis, and how Psi Chi Journal o f Psychological Research (PCf) is responding.

Build U p to th e R eplication Crisis Perhaps th e biggest indication o f the psychological re p lic atio n crisis was a series o f p ap e rs th a t were com pletely fabricated by several differen t authors (L e v elt C o m m itte e , 2 0 1 2 ). In th e s e cases, th e au th o rs in question were discovered to have com ­ pletely fabricated th e ir d ata based o n a n u m b e r o f factors ra n g in g from th e inability o f coauthors to get access to data to a statistical analysis o f raw data from the papers suggesting th at the d ata was faked (S im onsohn, 2013).

Some have looked at the data fabrication crisis as a serie s o f u n r e la te d a n d iso la te d in c id e n ts,

perhaps driven by personal flaws o r am bition. O th ­ ers, however, have looked at systemic features in aca­ dem ia as a potential influence on this p h e n o m e n a (Nosek, Spies, & Motyl, 2012). For instance, it has long been noted that ten u re an d prom otion in aca­ dem ia is driven largely by the n u m b er an d quality of publications (Varian, 1998). Early career research­ ers in the field (graduate students, post-docs, an d assistant faculty) are pressured to publish early and often, an d this can lead to academ ics taking steps to drive u p th eir publication counts. It is im p o rtan t to n o te th at these steps rarely include data fabrica­ tion. Rather, these steps often include publishing single-study short reports, which increase the likeli­ hood o f a type one e rro r (Ledgerwood & Sherm an, 2012), using selective analyses and m anipulations of degrees o f freedom to reach statistical significance (know n as “p -h a c k in g ”; S im o n so h n , N elso n , & Simons, 2014), an d engaging in unw arran ted self­ citations to increase th e n u m b e r o f citations o n e has (Purvis, 2006).

O th e r issues are m ore nuanced. T h e problem s w ithjournals seeking to publish novel findings and journals having a publication bias toward publishing significant results are arguably quite intertw ined. It has long been known by scientists that studies with­ o u t significant effects often en d u p in a file drawer, never to be seen again. Occasionally, meta-analyses reveal these file-drawer effects, b u t often th e re is a significant lag tim e if these meta-analyses are ever run. Additionally, in my experience an d that o f my colleagues, jo u rn a l editors have been n o ted to say th at significant replications an d extensions b elong in specialty jo u rn als, ra th e r than m ore widely dis­ trib u ted jo u rn als. O f course, th e re are exceptions The Psi Chi Journal editorial team invited Dr. Edlund to write an editorial as a m em ber o f the Psi Chi Research Advisory Commitee (RAC). As p art o f discussions between the RAC and the PCJ editorial team, we have em barked on a Replication Initiative to stimulate engagem ent with replication of psychological research studies.

SPRING 2 0 1 6

PSI CHI JO URNAL OF PSYCHOLOGIC AL RESEARCH

COPYRIGHT 2016 BY PSI CHI, THE INTERNATIONAL HONOR SOCIETY IN PSYCHOLOGY (VOL. 21, NO. 1/ISSN 2164-82041 5 9

A Call fo r Replications | Edlund

SPRING 2 0 1 6

PSI CHI JO URNAL OF

PSYCHOLOGIC AL RESEARCH

to these g en e ral trends. F or exam ple, studies on th e sex d ifference in jealousy have many nonsig­ nificant (and arguably underpow ered: B erm an & Frazier, 2005) an d significant findings published in p ro m in en tjo u rn als (Sagarin etal., 2012). However, th e com parative ease o f publishing th e nonsignifi­ cant findings in this field are likely attributable to n u a n c e d reasons su rro u n d in g the effect an d the theoretical a p p ro ach e d taken.

C h a n g in g P rac tic e s t o A d d re s s t h e Crisis As a field, num ero u s suggestions have em erged on how to deal with th e u n d erly in g reasons fo r th e crisis o f confidence. O n e ap p ro ach em ployed by journals is en c o u rag in g p re reg istere d studies. In this ap p ro ach , a study’s background, hypotheses, a n d m ethods are in d ep e n d e n tly reviewed before any data are collected. T h e goal o f this approach is to am eliorate the focus on publishing significant results because these studies are g u aran teed publi­ cation based on the quality o f th e research ra th e r th a n o b ta in in g results th a t re a c h a significance level o f p < .05.

A n o th e r te c h n iq u e th a t has b e e n em ployed is th e use o f massive collaborative p ro jec ts th a t c o n d u c t m u ltip le re p lic a tio n s a t th e sam e tim e such as the ManyLabs studies (Alogna et al., 2014) o r the Collaborative Replications an d E ducational P ro je c t (G ra h e , B ra n d t, IJz e rm a n , & C o h o o n , 2014). This tech n iq u e has many advantages such as the inclusion o f m o re diverse participants, and a g re a te r c o n fid e n c e in th e veracity o f th e d ata collected, d u e to the large n um bers o f collabora­ tors an d sh arin g o f th e raw data. This a p p ro ach also provides advantages to researchers who have m ore lim ited resources at th eir hom e institutions to be involved in th e larg e r scientific e n te rp rise an d discourse.

T h e s e a p p r o a c h e s c e r t a i n l y h a v e t h e i r s tre n g th s a n d ad v an tag es. H ow ever, they w ork best w hen in co rp o rated into a long-term research ag e n d a looking at an already established research question. New research ers are less likely to have su cc essfu l p r e r e g is tr a tio n s o r to b e le a d in g a m ultiple-lab based project. As such, th ere rem ains a n e e d for outlets th at are willing to publish high- quality replications.

M o v in g F o rw a rd PCJ has always published high-quality psychologi­ cal papers, ra n g in g from p ap e rs fe a tu rin g novel hypotheses (P eters, H o lg re e n , & Oswald, 2015) to studies focused o n re p lic a tio n (Casad & Lee,

2014). In collaboration with th e Psi Chi Research Advisory C o m m itte e , th e PCJ e d ito ria l team is la u n c h in g a re p lic atio n initiative with this issue. As p a r t o f this initiative, PCJ will e n c o u ra g e th e submission o f replication studies, provide a special notation in PCJ for replication studies, an d add the keyword “rep licatio n ” to reviewers’ expertise. PCJ enthusiastically welcomes reviewers with expertise in reviewing replication studies. T h e PCJ editorial team hopes th a t these structural changes will com ­ m u n ic a te th e im p o rta n c e o f re p lic a tio n studies a n d e n c o u ra g e r e s e a rc h e rs to e n g a g e in th e ir execution an d dissem ination o f findings, w hether significant o r not.

T h ere are two ends o f the spectrum on replica­ tion: direct replication an d conceptual replication (Schmidt, 2009). A d irec t replication attem pts to m ost direcdy recreate th e study th a t is b ein g repli­ cated (in materials, procedures, a n d particip an ts). In p ra ctice, th e re a re very few p u b lish e d d ire c t replications. M ore com m only a study will feature a d ire c t re p lic atio n a n d an ad d itio n al m an ip u la­ tion o r ex ten sio n . Alternatively, in a co n c ep tu al re p lic atio n , th e basic re searc h q u estio n is re p li­ cated , b u t with d iffe re n t m aterials, p ro c e d u re s, o r p articip an ts th a t are still conceptually linked. PCJ has always b e e n o p en to b o th conceptual an d direct replications.

Ultimately, the biggest factor th at will be evalu­ ated in the submission o f any PCJ m anuscript will be th e quality o f th e m an u scrip t. T h e adequacy o f th e sam ple size fo r th e c o n c lu sio n s re a c h e d (Anderson & Maxwell, 2015) will rem ain an im por­ tan t factor in decisions. Additionally, th e quality of th e m aterials a n d m eth o d s will be evaluated (see Uncles & Kwok, 2013, for a discussion o f m aterials in the co n tex t o f replication science). Finally, the adequacy o f th e analyses a n d the re p o rtin g o f the en tire m anuscript will be considered as well.

As an illustration o f w hat good replication sci­ ence can look like to Psi Chi, we p o in t to K eeran an d Burmeister (2015) featured in the current issue o f PCJ. H ere, K eeran an d B urm eister em ployed a prim arily d ire c t re p lic atio n a p p ro a c h , alth o u g h several changes a n d th e ir p o ten tial im pact were n o ted . Like all m anuscripts em ploying a replica­ tion ap proach, th e p a p e r was evaluated on its own m erits with the replicative approach b eing n eith er a b e n e fit n o r a h in d ra n c e to a c c e p ta n c e o f th e m anuscript.

Ultimately, it is o u r h o p e th a t o u r field co n ­ tinues to em brace th e benefits o f replications in psychology. T he field is strengthened by increasing

60 COPYRIGHT 2016 BY PSI CHI, THE INTERNATIONAL HONOR SOCIETY IN PSYCHOLOGY (VOL. 21, NO. 1/ISSN 2164-8204)

Edlund | A Call for Replications

the precision a n d co n fid en ce o f o u r conclusions th ro u g h u sin g b o th c o n c e p tu a l a n d d ire c t re p ­ lications. It is with th ese goals in m in d th a t PCJ explicitly invites you to su b m it your replications fo r consideration in PCJ.

References Alogna, V. K , Attaya, M. K., Aucoin, P, Bahnik, §., Birch, S„ Birt, A. R____

Zwaan, R. A. (2014). Registered replication report: Schooler and Engstler- Schooler (1990). Perspectives on Psychological Science, 9, 556 -5 7 8 doi:10.1177/1745691614545653

Anderson, S. F., & Maxwell, S. E. (2015, July 27). There’s more than one way to conduct a replication study: Beyond statistical significance. Psychological Methods. Advance online publication. doi:10.1037/ met0000051

Berman, M. I., & Frazier, P. A. (2005). Relationship power and betrayal experience as predictors of reactions to infidelity. Personality and Social Psychology Bulletin, 3 1 ,1617-1627. doi:10.1177/0146167205277209

Casad, B. J., & Lee, J. K. (2014). Benevolent sexism and perceptions of gender conforming and nonconforming women abuse victims. Psi Chi Journal o f Psychological Research, 1 9 ,1 36-143. Retrieved from http:// www.psichi.org/?page=193JNFALL2014

de Winter, J., & Happee, R. (2013). Why selective publication of statistically significant results can be effective. PLoSONE, 6(6), e66463. doi:10.1371/ journal.pone.0066463

Grahe, J. E , Brandt, M., IJzerman, H , & Cohoon, J. (2014). Replication education. APS Observer, 27. Retrieved from http://w w w .

p s y c h o lo g ic a ls c ie n c e .o rg /in d e x .p h p /p u b lic a tio n s /o b s e rv e r/2 0 1 4 / march-14/replication-education.html

Griggs, R. A. (2015). The Kitty Genovese story in introductory psychology textbooks: Fifty years later. Teaching o f Psychology, 4 2 149-152 doi:10.1177/0098628315573138

loannidis, J. P. A. (2005). Why most published research findings are false. PLoS Medicine, 2(8), e124. doi:10.1371 /joumal.pmed.0020124

Keeran, D. C., & Burmeister, J. (2015). Side-effect effect take 2. Psi Chi Journal o f Psychological Research, 2 1 ,2 -5 . Retrieved from http://w w w . psichi.org/?page=211 JNSpring2016

Ledgerwood, A., & Sherman, J. W. (2012). Short, sweet, and problematic? The rise of the short report in psychological science. Perspectives on Psychological Science, 7 ,6 0 -6 6 . doi:10.1177/1745691611427304

Levelt Committee. (2012, November 28). Ealende wetenschap: De frauduleuze onderzoekspraktijken van sociaal-psycholoog Diederik Stapel [Flawed science: The fraudulent research practices o f social psychologist Diederik Stapel], Retrieved from http://pubman.mpdl.

mpg.de/pubman/item/escidoc:1569964/component/escidoc:1569967/ Eindrapport_Commissie_Levelt.pdf

Nosek, B. A., Spies, J. R., & Motyl, M. (2012). Scientific utopia: II. Restructuring incentives and practices to promote truth over publishability. Perspectives on Psychological Science, 7,615-631 ,doi:10.1177/1745691612459058

Peters, N., Holmgreen, L., & Oswald, D. (2015). It’s just a joke: Reactions to and justifications for sex stereotypes in advertisements. Psi Chi Journal o f Psychological Research, 2 0 , 1 60-168. Retrieved from https://w ww . psichi.org/?203JNFall2015

Purvis, A. (2006). The h index: Playing the numbers game. Trends in Ecology and Evolution, 2 1 ,422. doi:10.1016/j.tree.2006.05.014

Sagarin, B. J., Martin, A. L., Coutinho, S.A., Edlund, J. E., Patel, L.,Skowronski, J. J., & Zengel, B. (2012). Sex differences in jealousy: A meta-analytic examination. Evolution and Human Behavior, 3 3,5 95-614. doi:10.1016/j. evolhumbehav.2012.02.006

Schmidt, S. (2009). Shall we really do it again? The powerful concept of replication is neglected in the social sciences. Review o f General Psychology, 13,90-1 0 0 . doi:10.1037/a0015108

Simmons, J. P„ Nelson, L. N „ & Simonsohn, U. (2011). False-positive psychology: Undisclosed flexibility in data collection and analysis allows presenting anything as significant. Psychological Science, 22, 1 3 5 9 - 1366. doi:10.1177/0956797611417632

Simonsohn, U. (2013). Just post it: The lesson from two cases of fabricated data detected by statistics alone. Psychological Science, 2 4 ,1875-1888 doi:10.1177/0956797613480366

Simonsohn, U., Nelson, L. D„ & Simmons, J. P. (2014). P-curve: A key to the file-drawer. Journal o f Experimental Psychology: General, 143,5 3 4 -5 4 7 doi:10.1037/a0033242

Steen, R. G. (2011). Retractions in the scientific literature: Do authors deliberately com m it research fraud? Journal o f Medical Ethics: Journal o f the Institute o f Medical Ethics, 3 7 , 1 13-117. doi1 0 1136/ jm e.2010.038125

Uncles, M. D., & Kwok, S. (2013). Designing research with in-built differentiated replication. Journal o f Business Research, 6 6 ,1398-1405. doi:10.1016/j.jbusres.2012.05.005

Varian, H. R. (1998). Future of electronic journals: Some speculations about the evolution of academic electronic publishing. The Journal o f Electronic Publishing, 4(1). doi:10.3998/3336451.0004.105

Author Note. J o h n E. E d lu n d , R ochester In stitu te o f Technology. T h e a u th o r wishes to th an k th e m em b e rs o f th e Psi Chi

R esearch Advisory C om m ittee fo r th e ir h e lp with this initiative a n d especially C harlie Ebersole, J o n G rah e, a n d Rick M iller fo r th e ir insightful co m m en ts on an early draft.

C o rre sp o n d e n c e c o n c e rn in g this article sh o u ld be a d d ressed to J o h n E. E d lu n d a tjo h n .e d lu n d @ rit.e d u .

SPRING 2016

PSI CHI JOURNAL OF PSYCHOLOGICAL RESEARCH

COPYRIGHT 2 0 1 6 BY PSI CHI, THE INTERNATIO NAL HONOR SOCIETY IN PSYCHOLOGY (VOL. 21, NO. 1/ISSN 2 16 4-82041 61

Copyright of Psi Chi Journal of Psychological Research is the property of Psi Chi Journal of Undergraduate Research and its content may not be copied or emailed to multiple sites or posted to a listserv without the copyright holder's express written permission. However, users may print, download, or email articles for individual use.