4.8. Meta-criteria for quality of research

Additional aspects that you can take into account when evaluating the quality of previous research are meta-criteria such as the funding source, the level of transparency, and the publication outlet in which the research appeared. Funding sources, transparency and publication outlets are associated with the level of scrutiny that a study has received. Studies funded by grants from government agencies are more likely to be of high quality than studies funded by parties that have a financial stake in the results. Studies that do not provide access to the data used, do not include the materials used, and do not provide the code for the analyses reported are not transparent. They are more likely to include mistakes than studies that rely on open data, materials and code. Studies published in international, peer-reviewed academic journals are likely to be better than published in national journals without peer review or unpublished studies.

Figure 18. Bullshit Bingo

Some academics use the rule that research that is not published should not be cited. This rule should not be used, because it contributes to selective citation of positive findings. Published research is much more likely to report positive results than unpublished research. For this reason, meta-analyses of published research are usually selective as well, and should not be taken as the most credible form of evidence. Instead, look at the design and methodology of relevant research, regardless of whether it is published or not.

When you report on previous research, always try to find attempts to replicate the research. There are interesting new tools that help you do this, such as scite.ai. Did more recent studies with similar research questions, but with different data and methods arrive at similar conclusions? Then you can be more confident that the initial findings are reliable and can be generalized. Studies that have not been replicated at all are more likely to be false positives.

A rule of thumb in the evaluation of research quality that many people use is that the higher the impact factor of a journal in which a paper has appeared, the higher the level of scrutiny a study has received. However, publication in a highly ranked, peer-reviewed, international journal does not guarantee that the research is of high quality. Even so-called ‘top journals’ struggle to keep up the quality of the research they publish (Brembs, 2018). Editors of journals with a higher impact factor reject a larger proportion of manuscripts, often without even sending them out for peer review (‘desk rejects’). Also, editors often base their decision to send out an article for peer review on criteria that have little to do with the quality of the study. Finally, peer review is not waterproof. Recently, cases of fraud have been detected even in publications in very prestigious journals such as Science. An increasing number of publications have been retracted in the past years. The number of retractions is even higher in the more prestigious journals (Fang, Steen, Casadevall, 2012). This trend illustrates the importance of an independent, critical assessment of the quality of previous research on your research problem.

A better criterion than the impact factor of a journal may therefore be the number of citations to the paper itself rather than to the journal in which it appeared. However, this metric is also difficult to interpret. Obviously, citations increase with age: older research has had more time to get cited. Also, systematic literature reviews and meta-analyses typically receive more citations than individual research reports, even though they typically do not pay attention to the quality of the research presented. Research published in larger fields, such as economics or psychology, receives more citations than research published in smaller fields, such as sociology or anthropology. Finally, research that is too good to be true may receive citations precisely because it cannot be replicated.