# Research Quality Checklist for Experiments

Preregistration

1. Was the study preregistered prior to the data analysis?

If yes:

2. The preregistration is posted at: [URL]

3. When was the preregistration posted? Before data were collected, before data were explored, before data were analyzed

The preregistration describes:

4. Inclusion and exclusion criteria for participation

5. All procedures for assigning participants to conditions

6. All procedures for randomizing stimulus materials

7. Any procedures for ensuring that participants, experimenters, and data-analysts were kept naive (blinded) to potentially biasing information

8. A rationale for the sample size used (e.g., an a priori power analysis)

9. All measures completed by the participants (including those not used in the study)

10. The data preprocessing scripts (cleaning, transformations, normalization, smoothing)

11. A plan for handling missing data (e.g., unit non-response, attrition between rounds, dropout from survey)

12. The intended statistical analysis for each research question (including information about the sidedness of the tests, inference criteria, corrections for multiple testing, model selection criteria, prior distributions and other plans)

Sampling

13. Does the paper provide a rationale for the sample size used, preferably an a priori power analysis?

14. Does the paper report how participants were recruited for the experiment?

15. Does the paper report eligibility criteria for participation in the experiment? E.g., with respect to nationality, gender, age, experience, prior knowledge on the subject matter?

Procedure

16. Does the paper report where and when the experiment was conducted?

17. For a lab experiment: does the paper report how participants were welcomed, by whom, how and where they were seated, how they completed the experiment, and how they were debriefed? Were experimenters blind to the hypotheses?

18. Were participants fully informed? If not, was deception necessary? Were participants asked which hypotheses they suspected were being tested?

19. Does the paper report how participants were paid, and which conditions determined the size of the payment?

Materials

20. Does the paper provide details about IRB approval (e.g., location, registration number)?

21. Does the paper describe or provide a link to all instructions and stimuli for participants?

22. Does the paper describe or provide a link to all measures completed by the participants (including those not used in the study)?

23. Does the paper provide a link to the processed data?

24. Does the paper provide a link to the code/script used to preprocess the data?

25. Does the paper provide a link to the code/script used to obtain the results reported?

27. Does the paper mention which software (including version) was used?

Randomization

28. Does the paper report how participants were randomized into conditions and how stimulus materials were randomized?

29. Are numbers of observations per condition approximately equal to the design for randomization?

30. Does the paper report checks on deviations from randomization (e.g., comparison of covariates per condition)?

Manipulation and attention check

31. Does the manipulation fit the theoretical mechanism?

32. Is the manipulation clean? What other consequences did the manipulation have, in addition to the theoretically relevant effect?

33. Does the paper report a manipulation check? Is it clear whether participants who failed the manipulation check included or excluded? Are results reported for both choices?

34. Particularly for online experiments: were attention checks included?

35. Are the materials and measures validated (e.g. in previous research or a pilot study)?

Dependent variable

36. To what extent is the dependent variable a valid measure of the construct in the theory and hypotheses?

37. For experiments on charitable giving: did participants make an actual donation decision, multiple decisions of which only one was executed, or did they state donation intentions? Were donation decisions by participants actually executed? If they were not executed, did participants believe they would be?

38. Does the paper report analyses of multiple dependent variables? If so, how are the dependent variables interrelated? If the paper reports analyses of multiple dependent variables in different studies, why? Were these changes planned, and do they follow from the arguments in the hypotheses?

Analyses

39. Does the paper present a table of descriptive statistics?

40. Does the paper distinguish explicitly between "hypothesis testing" (i.e., ‘confirmatory’, prespecified) and "exploratory" (i.e., not prespecified) analyses?

41. Does the paper report how missing data and participant dropout was handled (e.g., replaced, omitted).

42. If covariates are included: does the paper report results without covariates? If only an analysis including covariates is reported, ask for an additional analysis without covariates.

43. If outliers were removed: does the paper report results without outliers? If only an analysis excluding outliers is reported, ask for an additional analysis including outliers.

44. If the analysis includes scales: does the paper report the reliability or a factor analysis of each scale? Is the reliability sufficiently high?

45. In an analysis with moderator variables: is the main effect included? Are ordinal/linear variables in the interaction term centered?

46. Does the paper report adequate statistical tests, given the measurement scales of the variables (nominal, ordinal, linear) and their distribution (normal, non-normal)?

47. Does the paper use appropriate statistical models, given the structure of the data (nesting, repeated measures) and are the statistical analyses reported correctly? Are the test-statistics accurate? Are sample sizes reported for each cell of the design?

48. Does the paper report robustness checks?

Interpretation

49. Does the paper interpret the effect sizes in a substantive manner?

50. Does the paper discuss alternative explanations of the results?

51. Does the paper discuss limitations to the research?