Chapter 5 Concluding the Reproduction

Once you have completed each of the reproduction stages for all your claims of interest, you will be ready to submit your work. The minimal requirement to have a completed reproduction attempt is that at least one display item has been assessed.

Before submitting a reproduction attempt, you will be able to modify your answers to any entry in the ACRE platform. After you hit “Submit”, however, you will not be able to modify your reproduction attempt any further. If you wish to modify your reproduction attempt after submission, you will have to record a new reproduction attempt on the ACRE platform and link to the previously completed reproduction attempt.

5.1 Outputs

By the end of a reproduction attempt you will have created 3 different types of outputs:

  1. Revised reproduction package – Submit your revised version of the original reproduction package into a trusted repository. Examples of trusted repositories include: Dataverse, openICPSR, Figshare, Dryad, Zenodo, Open Science Framework and other. You should submit a revised reproduction package any time that you perform any type of improvement to the original reproduction package. This revised reproduction package is expected to be self-contained, as it might be used by future reproducers for assessment and improvement. If your new reproduction package is larger than 2Gb (?)[A: how is this determined, it sounds arbitrary?], or it contains data that you don’t have permissions to share, remove the specific files from your reproduction package and add a reference to the original reproduction package [A: what does making a reference mean and look like in this context? I suggest providing an example of how to do that.].

Before submitting the reproduction, make sure that your revised reproduction package fulfills the following requirements:
- Has a digital object identifier (DOI). All of the trusted repositories referenced above generate a DOIs. This is a stable, citable link that will allow others to find your work.
- It is titled following this convention: revised reproduction package for - title of the paper - last name of reproducer - year when the reproduction was completed - If it is not self-contained, please indicate the DOI of the last reproduction package used for the reproduction (the original or a previously revised reproduction package), and the relative file location of the missing files (e.g. /data/raw/large_file.csv).

  1. Reproduction report(s) At the end of each stage, the ACRE platform will give you the option to generate a report which summarizes your work completed as part of that stage. Once you’ve finalized and submitted your reproduction attempt, ACRE will automatically also generate a final report with its own DOI (different from the DOI used for the revised reproduction package). If you are performing this reproduction as part of a class or thesis your instructor/advisor should have defined the structure of those reports (e.g. reports of stages 1 and 2 could be part of problems set 1, and reports 3 and 4 could be a part of problem set 2). Each report will contain the following information:
    • Scoping: basic information about the paper, descriptive statistics on the number of claims identified and on the subset that was assessed, and a summary of the claims and associated specifications.
    • Assessment: a summary of the display items assessed and their connections with the claims.
    • Improvement: descriptions of improvements implemented and of improvements suggested for future reproduction attempts, and an updated reproducibility score, if any.
    • Robustness: summary of all of the analytical choices identified, and a description of the results of robustness tests to reasonable new specifications, if any.
    • Final reproduction report: includes everything, plus final comments, and it cannot be edited after submission.
  2. Your data. You will be able to export a .csv file containing all of your responses recorded as part of your reproduction attempt.

5.2 Anonymity and data sharing

  • Anonymity: It is possible to choose that your reproduction attempt is posted anonymously for up to one year after it was submitted. During this. During this period your reproduction attempt will be visible to other users of the platform, but your identity will not be revealed.

  • Using the data from your reproduction attempt: The ACRE platform will aggregate descriptive statistics from all reproductions attempts recorded on the platform to produce metrics of reproducibility across economic sub-disciplines, journals, and bodies of literature.