## 3.4. Writing a literature review

A literature review is an important element in a research proposal, a master’s thesis, a PhD dissertation, and papers reporting empirical research. A literature review can take many forms, depending on its purpose, the audience it addresses, and the breadth and depth of the research it describes. In the case of an empirical paper, the purpose of the literature review is to provide the basis of the theoretical model that you are going to test. This is also the purpose of the literature review for a bachelor thesis, master thesis, and dissertation. Your review should address a relevant research question and should generate the hypotheses to be tested in your own research.

When you are writing the review, keep in mind that the audience is wider than the person who happens to be your supervisor. Your work should contribute to the academic debate. With your thesis you become one of the scholars who are working in the field you are interested in. When you are developing a research proposal or a dissertation plan, the literature review is a more comprehensive endeavor, and should form the basis for at least the empirical work you plan to do. In the best case your review also inspires the work of others.

In all cases, your review should present an integrated narrative of the research on your research problem, and an overview of relevant questions that are not yet answered in existing research. Do not present the studies you read one by one, in a laundry list ‘he said, she said’ manner (Thomson, 2017).

Instead, present the hypotheses investigated and tell the reader to what extent these hypotheses have been supported or rejected. Order your hypotheses logically rather than in the chronological order in which the research you have read has been published. It works well to present the key variables in the literature in the form of causal or cluster model. When you discuss results of previous research on a specific hypothesis, identify the relations between publications.

Your review is not merely a summary of prior research, but also provides an assessment of the quality of that research. Offer criticism on past research. Do so only after you have read the research. Which problems with the methodology of previous research can you identify? Remember that your thesis should contribute: it should address a new question, solve a theoretical puzzle, or repair a methodological weakness. Therefore your literature review should not be a laundry list of summaries and abstracts of potentially relevant publications.

3.4.1. Gathering materials

Start your literature review by gathering relevant publications on your research problem. First select a database. Personally I use Google Scholar, https://scholar.google.nl/, because it is the most comprehensive database of academic literature available, covering 93-96% of all published articles (Martín-Martín, Orduna-Malea, Thelwall & López-Cózar, 2018). Other databases you can use are PubMed, Scopus, and the catalogue of your local university library.

The first step in your review is finding the right keywords. I recommend that you enter the dependent and independent variables from your model as keywords.

In all likelihood you will get thousands of results, among which many are irrelevant. Your first task is to decide which of these results are relevant and which are not. You can think of the results you get for your query as opening up a storage locker bought through an auction (Figure 12). Once the door is opened you will see a lot of junk, some of which you can use.

My advice would be to start with a pile of papers to read that is as small as possible, making sure that the papers you have on that pile are absolutely necessary to read because they are of key importance. Start with the three most relevant publications, and put previous literature reviews on top. Create a second pile of less relevant papers. Quickly screen publications based on their titles and the first lines of the abstracts.

The second step in your search is refining or changing the keywords. If you get very few relevant results you will need to change the keywords you use. If you get too many results, but not many relevant ones, you will need to refine the keywords. Think of synonyms for the variables in your model, e.g. ‘religious affiliation’ rather than ‘religion’, or ‘level of education’ rather than ‘achievement in education’. Findings that are relevant for your research may be published in different disciplines, each with their own jargon. What is called ‘sex discrimination’ in one field may be called ‘gender discrimination’ in another.

If you are lucky, you will find an inventory list of all the items in storage: a literature review of previous research. Someone who already did your job. I recommend you start your literature review by locating the most recent literature review on the combination of concepts in your model. Search for this guide first by adding “literature review” to your search terms. In all likelihood, however, you will not find a literature review that precisely matches your research question. In this case, you will have to do the review on your own.

The third step is screening the abstracts. As you go through them, set a high bar. Keep the pile of relevant publications as small as possible. Weed out publications that do not fit your selection criteria from the gross list of potentially relevant results. After exclusion of irrelevant publications you can start reading.

Gathering materials is an iterative process. Keep in mind that you may have used the wrong keywords, and change or refine them when they do not produce relevant research.

The fourth step is prioritizing the list of relevant publications. To decide which papers are relevant for your research question, you need selection criteria for inclusion in your literature review. Start with a fairly narrow list of selection criteria. You can think of criteria such as:

1. Research question: exclude publications that do not answer your research question and do not provide results bearing on your hypotheses;
2. Publication status: only include publications in international academic journals, exclude working papers, book chapters;
3. Language: exclude publications in languages that you and your audience cannot read;
4. Publication date: exclude publications before (or in more rare cases: after) a certain date;
5. Country of origin: exclude publications from specific geographic areas;
6. Participants: exclude publications using specific categories of actors as participants such as children, elderly persons, organizations;
7. Methodology: exclude research with specific research designs, such as case studies, experiments or surveys, or research with specific statistical models;
8. Citations: you can assume that publications that many others refer to are important in the field.

The fifth step is a forward search, starting from the small selection of publications that are most relevant. What happened after the last comprehensive literature review was published? You can find those by searching for publications that cite the literature review. In Google Scholar you can easily find those by clicking on ‘Cited by …’ below the first three lines of the abstract. Once again, set a high bar. Select only those publications that fit your criteria.

The sixth step is to find related research. In the first small selection of publications you will find references to previous research. What are the key publications? In Google Scholar you can easily find related articles by clicking on ‘Related articles’ below the first three lines of the abstract. You can also use CoCites to find similar articles.

3.4.2. Reading articles

When you are collecting previous research for your literature review, never start with reading the paper in full. If you fully examine each and every object that you find in the storage locker, assessing all of its properties, you will never finish. You’re much better off making a quick judgment of whether you can use it, and if so, setting it aside for further inspection later. So when you get a large number of search results, use a similar approach. First look at the title of a paper and the key words, and then examine the abstract. If you get the impression from the abstract that the publication is not (very) relevant, put it at the bottom of your pile. Find the most recent literature review on your research problem and start with reading this article. This gives you an idea of the state of the art in the literature at the time of writing. Pay special attention to the final sections of papers, which often contain a set of recommendations for future research. Especially if the paper is relatively recent, it is unlikely that anyone has executed these plans.

When you do get to read an article in full because it is in your small pile of most relevant publications, it is useful to structure the reading. I often do so by scribbling acronyms in the margin. First identify the research question (RQ) that the paper answers. Next, identify the theories and hypotheses (H1, H2…) that the paper discusses to answer the research question. Next, identify the variables in the analysis: the dependent variable (Y), independent variable (X), Mediating (Med) and Moderating (Mod) variables. What data do the authors use: when and where are they collected, and how many observations (n) are included? What kind of analysis do the authors conduct (e.g., OLS, Fixed Effects models), and what are the results? Finally, which issues limit the validity and reliability of the results?

Keep a log of these elements in a table, with a column for each of these elements.

Authors | Year | DOI | Theory | Y | X | Med | Mod | Data | n | Analysis | Result | Limit

3.4.3. Include references

Write a full reference for each of the publications that you have read. There are many different styles of references. The APA style is one of the most commonly used styles. You can save a great deal of work by organizing your references in a reference manager software package, such as Zotero.

A reference consists of at least four elements:

1. Author; 2. Year of publication; 3. Title of publication; 4. Where the publication can be found.

These elements should be included in references of all types: journal articles, books, reports, presentations, blogs, op-eds, presidential addresses, prize lectures, websites, and so on.

Always provide a list of references at the end of your thesis, also when your text is still ‘work in progress’, and when you have not yet cited all references in the text. Make a separate list of references to be included. This allows your supervisor to see whether you have found relevant publications that you have not yet worked into the text. You may also want to keep a separate list of publications that you have not yet read or included in your thesis text. This allows your supervisor to think along and suggest additional publications.

Your reference list is a single list of sources, in alphabetical order. Do not create separate categories for ‘books’, ‘articles’ and ‘websites’. Make sure that in the final version of your thesis the reference list contains no references that you have not cited in the text. Also make sure that the reverse is true, and that all references in the text also occur in the reference list.

Make sure you have read all the publications you refer to. Avoid including references to publications you have not read. If you think that a study you have read about in another study is so important that it deserves a mention in your text, make an effort to find it. Leave it out if you cannot find it. In the worst case, if you know that a study is important but you cannot get it, refer to a study as “Cited in: [author (year) mentioning the gem]”.

It is a good principle to refer to the source of statements. When the same reference is underlying an entire section, it is enough to include the reference only once.