The list enables a charting of what Gelman and Loken (2014) dubbed the garden of forking paths in the analysis of data i.e., the many different analytic decisions that could be or could have been made with a given data set. By pointing out many different researcher degrees of freedom, we hope to raise awareness of the risk of bias implicit in a lot of research designs in psychology and beyond. The goal of this paper is to present a list of researcher degrees of freedom that can be used in research methods education, as a checklist to assess the quality of preregistrations, and to determine the potential for bias due to (arbitrary) choices in unregistered studies. Therefore, for confirmatory aspects of the study, the word “only” is key (e.g., “we will test only Hypothesis A in the following unique manner”). A preregistration should also be exhaustive because the stipulation that one will test Hypothesis A in a certain way does not preclude the possibility that one can also test Hypothesis B in the study. For instance, just indicating one’s use of a certain scale as the main outcome measure in an experiment typically does not preclude the researcher to attempt many different ways in how to score the items of the scale in his or her pursuit for statistical significance. Our own experiences with preregistration taught us that this specification is no easy task and that maneuverability remains if preregistrations are not sufficiently specific, precise, or exhaustive. For instance, the syntax for the statistical analyses should preferably be created in advance to be run (once) on the collected data to yield the final statistical results. Hence, a preregistration specifies the project in such a way that all potential contingencies in formulating hypotheses, and designing, running, analyzing, and reporting are covered. Finally, a preregistration should exclude the possibility that other steps may also be taken (it should be exhaustive). Moreover, each described step should allow only one interpretation or implementation (it should be precise). That is, the ideal preregistration should provide a detailed description of all steps that will be taken from hypothesis to the final report (it should be specific). To disallow researchers to still use researcher degrees of freedom, it is crucial that preregistrations provide a specific, precise, and exhaustive plan of the study. For instance, this format is now used in the journals Cortex, Comprehensive Results in Social Psychology, and Perspectives on Psychological Science (for Registered Replication Reports). In addition, over two dozen journals now use a format of registered reports ( Chambers, 2013) in which the registrations themselves are subject to peer review and revisions before the data collection starts, and the report is accepted for publication regardless of the direction, strength, or statistical significance of the final results. An increasing number of journals now support preregistration for confirmatory research (e.g., Eich, 2014). Although “planned research” more accurately describes this preregistered research, we will employ the commonly used term “confirmatory research” to describe it. Preregistration requires the researcher to stipulate in advance the research hypothesis, data collection plan, specific analyses, and what will be reported in the paper. Hence, researcher degrees of freedom play a central role in the creation of (published) research findings that are both hard to reproduce in a reanalysis of the same data and difficult to replicate in independent samples ( Asendorpf et al., 2013).Īmong many potential solutions to counter inflated effects and elevated chances of finding false positive results caused by researcher degrees of freedom, one solution has received most attention: preregistration ( de Groot, 1956/2014 Wagenmakers et al., 2012 Chambers, 2013). Second, their strategic use in research may inflate effect sizes ( Ioannidis, 2008 Bakker et al., 2012 Simonsohn et al., 2014 van Aert et al., 2016). First, researchers’ opportunistic use of them greatly increases the chances of finding a false positive result ( Ioannidis, 2005 Simmons et al., 2011 DeCoster et al., 2015), or a Type I error in the language of Neyman–Pearson’s variant of null hypothesis testing (NHST). These choices are also called researcher degrees freedom ( Simmons et al., 2011) in formulating hypotheses, and designing, running, analyzing, and reporting of psychological studies, and they have received considerable recent interest for two main reasons. These choices could affect the outcome of significance tests applied to the data, and hence the conclusions drawn from the research. From the inception of the first study idea to the final publication, psychological studies involve numerous choices that are often arbitrary from a substantive or methodological point of view.
0 Comments
Leave a Reply. |
AuthorWrite something about yourself. No need to be fancy, just an overview. ArchivesCategories |