## 3.6. Quality of Research

When you are writing a theory section, it is tempting to review the literature by summing up the findings of previous research in an uncritical manner. This is in fact a good start of your literature review. A crucial next step, however, is to review the results of previous research in a critical way.

It may seem presumptuous to criticize established scholars, especially if you are an undergraduate student writing a thesis. But it is very important to be aware of the possibility that published research may be sloppy, contains mistakes, faulty reasoning, weak evidence, or in the worst case: it may be entirely false. You may have heard people use the expression ‘top journal’ to express their confidence in research published in certain ‘high impact’ journals with high prestige. However, it is very important to always remain critical of previous research, regardless of the prestige of the outlet in which it was published, or the prestige of the author. We will return to this in the next chapter.

How do you know whether you can have confidence in published research? Instead of relying on the prestige of the outlet, think about the claims in a research paper, and ask yourself the question: how do the authors know this? A rule that sets the bar quite high, but is very useful for a critical perspective is that if something seems too good to be true, it usually is.

The results and conclusions in published research are based on the nature and the quality of the design of the research. Have a close look at the research design. Is the design suitable to answer the research question? Pay specific attention to causal inference: to what extent does the research design allow for causal inference? Is it the best possible research design? If so, you can have more confidence in the results. We will discuss research designs in section 4.2.

Second, have a look at the data and methods. Can you actually check the data? Are they openly accessible? Or is the description in the published research paper the only source you can study? Are the codes and procedures used to analyze the data available for re-analysis? Or is the description you can read in the paper the only way to check the results?

You can evaluate the quality of data and methods in at least three respects:

1. Sampling: which units were selected for the study and why?
2. Reliability and validity of measures: to what extent are the measures reliable, accurate and representative?
3. Replicability and replication: to what extent have others checked the research?

This list is not only good for a critical discussion of previous research, but also a list of potential limitations of your own research. In your discussion section, discuss the limitations to your findings by revisiting these aspects of the quality of research. There are many more issues to the quality of research than I can discuss here. You can use the checklist in the appendix and read more in Bekkers (2016). In criticizing previous research, be polite. You are not at war. Criticizing a previous study only after you have read it closely.

The use of less stringent data collection methods and statistical tests leads to a higher likelihood of false positive results. To avoid false positives, more stringent research designs are to be preferred. More about this in the next chapter.