Chapter 6 Concluding the Reproduction

Once you have completed each of the reproduction stages for all of your claims of interest, you will be ready to submit your work. For a reproduction to be considered complete and ready for submission, you must have assessed at least one display item.

Before submitting a reproduction, you will be able to modify your answers to any entry in the ACRE platform. After you hit “Submit,” however, you will not be able to modify your reproduction attempt any further. If you wish to modify your reproduction after submitting it, you will have to record a new reproduction attempt on the platform and link to the previously completed reproduction. Tip: the platform allows to clone entire reproductions up to the scoping stage. If you also want to bring in content from some of the labor-intensive parts in later stages (describing all the inputs, documenting analytical choices) you can download the csv file of previous forms and upload them to a new reproduction.

6.1 Outputs

A completed reproduction will consist of three different types of outputs:

1. Revised reproduction package – Deposit your revised version of the original reproduction package in a trusted repository, such as Dataverse, openICPSR, Figshare, Dryad, Zenodo, or the Open Science Framework. You should submit a revised reproduction package any time that you perform any type of improvement to the original reproduction package. This revised reproduction package is expected to be self-contained as it might be used by future reproducers for assessment and improvement. If your new reproduction package is larger than the capacity of your trusted repository (around 2GB), or it contains data that you do not have permission to share, remove the specific files from your reproduction package and add a reference to the original reproduction package. For example if your reproduction package contains a file that cannot be shared, locate leave a note in the readme file with its relative location (e.g. /data/large_confidential_file.csv) and specific instructions on how to obtain it.

Before submitting the reproduction, make sure that your revised reproduction package fulfills the following requirements:

  • Has a digital object identifier (DOI). All of the trusted repositories referenced above generate DOIs. This is a stable, citable link that will allow others to easily find your work.
  • Is titled following this convention: revised reproduction package for - title of the paper - last name of reproducer - year when the reproduction was completed
  • If it is not self-contained, indicate the DOI of the last reproduction package used for the reproduction (the original or a previously revised reproduction package), and the relative file location of the missing files (e.g. /data/raw/large_file.csv).

2. Reproduction report(s) – At the end of each stage, the Social Science Reproduction Platform will give you the option to generate a report summarizing your work for that stage. Once you’ve submitted your final reproduction, the platform will also automatically generate a final report with its own DOI (different from the DOI used for the revised reproduction package). If you are conducting this reproduction as part of a supervised course or project, your instructor or advisor should have defined the structure of those reports (e.g., Stage 1 and 2 reports may be part of problem set 1, and Stage 3 and 4 reports may be a part of problem set 2). Each report will contain the following information:

  • Scoping Report: basic information about the paper, descriptive statistics about the number of claims identified and the subset that was assessed, and a summary of the claims and associated specifications.
  • Assessment Report: a summary of the display items assessed and their connections with the claims.
  • Improvement Report: descriptions of improvements implemented and of improvements suggested for future reproductions, and an updated reproducibility score, if any.
  • Robustness Report: summary of all of the analytical choices identified, and a description of the results of robustness tests to reasonable new specifications, if any.
  • Final reproduction report:* includes everything, plus final comments, and it cannot be edited after submission.

3. Your data – You will be able to export several .csv files containing all of your table-form responses recorded as part of your reproduction attempt.

6.2 Visibility and data use

Before submitting your reproduction, we ask that you select one of three privacy models that determine what parts of your reproduction will be visible to other SSRP users upon publishing. We recommend using the public, identified model, which allows other users to see the entirety of your reproduction and credits you directly as its author. However, you may choose to publish your reproduction anonymously under either the class anonymous or temporary anonymous models, where other SSRP users can access unidentifiable data from your reproduction (e.g., numerical and categorical responses), but not your identity and identifiable data (free-form narrative answers such as notes, descriptions, summaries, and explanations). Please refer to the Terms of Use to learn more about each privacy model; however, here they are in a nutshell:

  1. Public, identified: this is the default privacy setting. You are listed as the reproduction’s author, and other users can see the entire reproduction.

  2. Class anonymous: allows you to conceal your identity and free-form narrative answers (notes, descriptions, summaries, explanations) indefinitely if you conducted the reproduction as part of a class, project, or some other assignment for credit at the university/post-secondary level. Other users can see basic class identifiers (course name, year, institution university, instructor’s name, and email address) and categorical responses.

  3. Temporary anonymous: allows you to embargo your identity and free-form narrative answers for up to 4 years after submitting. Requires an annually recurring opt-out to maintain the embargo. At the expiration of the embargo period or if you fail to opt-out, privacy settings are updated to “public, identified.”

The Social Science Reproduction Platform will aggregate descriptive statistics from all submitted reproductions recorded to produce reproducibility metrics across disciplines, sub-disciplines, journals, and topical bodies of literature. If you choose to remain anonymous and are not a student using the SSRP as part of a course, your work may still be integrated into these descriptive statistics.