The discussion of the reliability of measures in the previous section also applies to studies as a whole. A study is better if its results can be replicated in a follow-up study. If a paper reports surprising findings, look for replications of the research. There are various types of replications, as indicated in Table 2 (based on Helmig, Spraul & Tremp, 2012 and Clemens, 2017).
In case the same hypothesis is tested in different ways in different data sets, the aim is in the bottom row of Table 2 on the right, called generalization: do new tests reveal similar results across different samples? Finding similar results using identical measures and procedures but different datasets, e.g. with a different sample, as in the bottom left cell, demonstrates reproducibility (Open Science Collaboration, 2015). Within a certain dataset, using different ways to test a hypothesis, for instance by adding control variables, excluding some observations or using different statistical analyses, a result may prove to be more or less robust (top right). Finally, results of research should be verifiable (top left): repeating the analysis of the same data using the exact same procedures should produce the exact same results.
Research should be reported such that replication is possible. The sampling of participants, measures used, construction of variables, treatment of outliers, the analyses conducted, and robustness checks should be reported in such a way that any researcher with access to the raw data or with resources to collect new data can replicate the study. Did the authors take this quote by Einstein to heart, engraved at his memorial at the National Academy of Sciences in Washington, DC?
If it is not clear to you which decisions the authors made in the design of the research (and why) the study is more difficult to replicate. Note that sometimes details on the research design are described in supplementary materials posted online.
If you’re reading articles reporting experimental results from multiple studies, it is a good idea to use the p-curve web app (http://www.p-curve.com/app/) to see how likely the results are. You might find that the results reported in the paper are ‘too good to be true’, or ‘p-hacked’ (Simonsohn, Nelson, & Simmons, 2014). If the results are too good to be true, this does not imply that the authors deliberately engaged in research misconduct (RM) or questionable research practices (QRP); it may also the be the case that the findings are a ‘lucky shot’, a statistical fluke that cannot be replicated. Such results are more likely to occur when the statistical power of the research design is too low. Increasing the number of observations is a general strategy that increases statistical power.
Obviously, you should apply the principles discussed here to your own research and live by Einstein’s commandment. Make sure that your research can at least be verified. Compare science with cooking. You should think of the documentation of the steps you took in your research as if you are writing a recipe. If you omit important details, like the temperature that the oven should have, or the duration of the bake, it is not of much use to tell your readers to know which ingredients went into the cake. Your job is to make the next baking attempt another success, just like you had it.
Figure 17. Practice and duty in reproducibility
Imagine yourself trying to bake that fantastic cake again, five years from now. Especially if you have a complicated recipe, you better write down the measures of the ingredients, the details of your oven, pots and pans and the utensils you used. Unlike many famous cooks who keep secrets or the Swedish chef from Sesame Street who has a habit of erratic working with random objects, a good scientist is completely transparent about every step in the research. Transparency ensures the best conditions for successful replication. If you compare conducting tests in the social sciences with cooking, the science is rather dull. It is a cold-blooded and rational application of techniques. There is no creative artistic involved. You do not want to be like the cook saying ‘I do not remember what I did’ when asked how she got to the award winning dish.