Richard Lucas and I are editing a special issue of the Journal of Research in Personality dedicated to replication (Click here for complete details). This blog post describes the general process and a few of my random thoughts on the special issue. These are my thoughts and Rich may or may not share my views. I also want to acknowledge that there are multiple ways of doing replication special issues and we have no illusions that our approach is ideal or uncontroversial. These kinds of efforts are part of an evolving “conversation” in the field about replication efforts and experimentation should be tolerated. I also want to make it clear that JRP has been open to replication studies for several years. The point of the special issue is to actively encourage replication studies and try something new with a variant of pre-registration.
What is the General Process?
We modeled the call for papers on procedures others have used with replication special issues and registered reports (e.g., the special issue of Social Psychology, the Registered Replication Reports at PoPS). Here is the gist:
- Authors will submit proposals for replication studies by 1 July 2015. These extended abstracts will be screened for methodological rigor and the importance of the topic.
- Authors of selected proposals will then be notified by 15 August 2015.
- There is a deadline of 15 March 2016 to submit the finished manuscript.
We are looking to identify a set of well-designed replication studies that provide valuable information about findings in personality psychology (broadly construed). We hope to include a healthy mix of pre-registered direct replications involving new data collections (either by independent groups or adversarial collaborations) and replications using existing datasets for projects that are not amenable to new data collection (e.g., long-term longitudinal studies). The specific outcome of the replication attempt will not be a factor in selection. Indeed, we do not want proposals to describe the actual results!
Complete manuscript will be subjected to peer-review but the relevant issues will be adherence to the proposed research plan, the quality of the data analysis, and the reasonableness of the interpretations. For example, proposing to use a sample size of 800 but submitting a final manuscript with 80 participants will be solid grounds for outright rejection. Finding a null result after a good faith attempt that was clearly outlined before data collection will not be grounds for rejection. Likewise, learning that a previously used measure had subpar psychometric properties in a new and larger sample is valuable information even if it might explain a failure to find predicted effects. At the very least, such information about how measures perform in new samples provides important technical insights.
Why Do This?
Umm, replication is an important part of science?!?! But beyond that truism, I am excited to learn what happens when we try to organize a modest effort to replicate specific findings in personality psychology. Personality psychologists use a diverse set of methods beyond experiments such as diary and panel studies. This creates special challenges and opportunities when it comes to replication efforts. Thus, I see this special issue as a potential chance to learn how replication efforts can be adapted to the diverse kinds of studies conducted by personality researchers.
For example, multiple research groups might have broadly similar datasets that target similar constructs but with specific differences when it comes to the measures, timing of assessments, underlying populations, sample sizes, etc. This requires careful attention to methodological similarities and differences when it comes to interpreting whether particular findings converge across the different datasets. It would be ideal if researchers paid some attention to these issues before the results of the investigations were known. Otherwise, there might be a tendency to accentuate differences when results fail to converge. This is one of the reasons why we will entertain proposals that describe replication attempts using existing datasets.
I also think it is important to address a perception that Michael Inzlicht described in a recent blog post. He suggested that some social psychologists believe that some personality psychologists are using current controversies in the field as a way to get payback for the person-situation debate. In light of this perception, I think it is important for more personality researchers to engage in formal replication efforts of the sort that have been prominent in social psychology. This can help counter perceptions that personality researchers are primarily interested in schadenfreude and criticizing our sibling discipline. Hopefully, the cold war is over.
[As an aside, I think it the current handwringing about replication and scientific integrity transcends social and personality psychology. Moreover, the fates of personality and social psychology are intertwined given the way many departments and journals are structured. Social and personality psychology (to the extent that there is a difference) each benefit when the other field is vibrant, replicable, and methodologically rigorous. Few outside of our world make big distinctions between social and personality researchers so we all stand to lose if decision makers like funders and university administrators decide to discount the field over concerns about scientific rigor.]
What Kinds of Replication Studies Are Ideal?
In a nut-shell: High quality replications of interesting and important studies in personality psychology. To offer a potentially self-serving example, the recent replication of the association between I-words and narcissism is a good example. The original study was relatively well-cited but it was not particularly strong in terms of sample size. There were few convincing replications in the literature and it was often accepted as an article of faith that the finding was robust. Thus, there was value in gaining more knowledge about the underlying effect size(s) and testing to see whether the basic finding was actually robust. Studies like that one as well as more modest contributions are welcome. Personally, I would like more information about how well interactions between personality attributes and experimental manipulations tend to replicate especially when the original studies are seemingly underpowered.
What Don’t You Want to See?
I don’t want to single out too many specific topics or limit submissions but I can think of a few topics that are probably not going to be well received. For instance, I am not sure we need to publish tons of replications showing there are 3 to 6 basic trait domains using data from college students. Likewise, I am not sure we need more evidence that skilled factor analysts can find indications of a GFP (or general component) in a personality inventory. Replications of well-worn and intensely studied topics are not good candidates for this special issue. The point is to get more data on interesting and understudied topics in personality psychology.
I hope we get a number of good submissions and the field learns something new in terms of specific findings. I also hope we also gain insights about the advantages and disadvantages of different approaches to replication in personality psychology.