Chapter 6 Concluding the Reproduction

Once you have completed each of the reproduction stages for all of your claims of interest, you will be ready to submit your work. For a reproduction to be considered complete and ready for submission, you must have assessed at least one display item.

Before submitting a reproduction, you will be able to modify your answers to any entry in the ACRE platform. After you hit “Submit,” however, you will not be able to modify your reproduction attempt any further. If you wish to modify your reproduction after submitting it, you will have to record a new reproduction attempt on the platform and link to the previously completed reproduction.

6.1 Outputs

A completed reproduction will consist of three different types of outputs:

1. Revised reproduction package – Deposit your revised version of the original reproduction package in a trusted repository, such as Dataverse, openICPSR, Figshare, Dryad, Zenodo, or the Open Science Framework. You should submit a revised reproduction package any time that you perform any type of improvement to the original reproduction package. This revised reproduction package is expected to be self-contained as it might be used by future reproducers for assessment and improvement. If your new reproduction package is larger than 2Gb (?)[A: how is this determined, it sounds arbitrary?], or it contains data that you don’t have permission to share, remove the specific files from your reproduction package and add a reference to the original reproduction package [A: what does making a reference mean and look like in this context? I suggest providing an example of how to do that.].

Before submitting the reproduction, make sure that your revised reproduction package fulfills the following requirements:

  • Has a digital object identifier (DOI). All of the trusted repositories referenced above generate DOIs. This is a stable, citable link that will allow others to easily find your work.
  • Is titled following this convention: revised reproduction package for - title of the paper - last name of reproducer - year when the reproduction was completed
  • If it is not self-contained, indicate the DOI of the last reproduction package used for the reproduction (the original or a previously revised reproduction package), and the relative file location of the missing files (e.g. /data/raw/large_file.csv).

2. Reproduction report(s) – At the end of each stage, the Social Science Reproduction Platform will give you the option to generate a report summarizing your work for that stage. Once you’ve submitted your final reproduction, the platform will also automatically generate a final report with its own DOI (different from the DOI used for the revised reproduction package). If you are conducting this reproduction as part of a supervised course or project, your instructor or advisor should have defined the structure of those reports (e.g., Stage 1 and 2 reports may be part of problem set 1, and Stage 3 and 4 reports may be a part of problem set 2). Each report will contain the following information:
- Scoping Report: basic information about the paper, descriptive statistics about the number of claims identified and the subset that was assessed, and a summary of the claims and associated specifications.
- Assessment Reoprt: a summary of the display items assessed and their connections with the claims.
- Improvement Report: descriptions of improvements implemented and of improvements suggested for future reproductions, and an updated reproducibility score, if any.
- Robustness Report: summary of all of the analytical choices identified, and a description of the results of robustness tests to reasonable new specifications, if any. - Final reproduction report: includes everything, plus final comments, and it cannot be edited after submission.

3. Your data – You will be able to export a .csv file containing all of your responses recorded as part of your reproduction attempt.

6.2 Anonymity and data sharing

  • Anonymity: You may choose to post your reproduction anonymously for a period of time (TBD) after submitting it. During this embargo period, unidentifiable data from your reproduction (i.e., numerical and categorical form responses) will be visible to other platform users, but your identity and identifiable data (i.e., text form responses) will not be revealed.

  • Using the data from your reproduction: The Social Science Reproduction Platform will aggregate descriptive statistics from all submitted reproductions recorded to produce reproducibility metrics across disciplines, sub-disciplines, journals, and topical bodies of literature. If you choose to remain anonymous and are not a student who is using the SSRP as part of a course, results from your work will still be integrated into these descriptive statistics. However, we will separately display statistics for all reproductions and those not conducted anonymously or as part of a course.