Chapter 2 Scoping

The goal of this stage is to help you define the scope of your reproduction by identifying key scientific claims and the specific display items on which you will focus.

Once you have identified your declared paper, get familiarized with it and choose the specific display items on which you will focus for the remainder of the exercise.

2.1 Read and summarize the paper

Depending on how much time you have, we recommend that you write a short (1-2 page) summary of the paper. This will help remind you of the key elements to focus on for the reproduction, and demonstrate your understanding of the paper (for yourself and others like your instructor or advisor).

When reading or summarizing the paper, try to answer the following questions:

  • Would you classify the paper’s scientific claims as mainly focused on estimating a causal relationship, estimating/predicting a descriptive statistic of a population, or something else?
  • How many scientific claims (descriptive or causal) are investigated in the paper?
  • What is the population for which the estimates apply?
  • What is the population that is the focus of the paper as a whole?
  • What are the main data sources used in the paper?
  • How many display items are there in the paper (tables, figures, and inline results)?
  • What is the main statistical or econometric method used to examine each claim?
  • What is the author’s preferred specification (or yours, if the authors are not clear)?
  • What are some robustness checks for the preferred specification?

2.2 Record scope of the exercise

By now you should have a fairly good understanding of the paper’s content. You do not, however, need to have spent any time reviewing the reproduction package in detail.

At this point, you should clearly specify which part of the paper will be the main focus of your reproduction. Focus on specific estimates, represented by a unique combination of claim-display item-specification as represented in figure 0.1. If you plan to scope more than one claim, we strongly recommend starting with just one and recording your results. You can then initiate another record in ACRE later for the second (or third, fourth, etc.) claim to reproduce using the materials and knowledge you developed in the first exercise. You can, however, reproduce more than one claim if you are already familiar with the paper.

In the Assessment stage, the reproduction will be centered around the display item(s) that contain the specification you indicate at this point.

Declare specific main estimates to reproduce.

Identify a scientific claim and its corresponding preferred specification, and record its magnitude, standard error, and location in the paper (page, table #, and table row and column). If the authors did not explicitly chose a particular estimate, you will be asked to select one. In addition to the preferred estimate, reproduce up to five estimates that correspond to alternative specifications of the preferred estimate.

Declare possible robustness checks for main estimates (optional).

After reading the paper, you might wonder why the authors did not conduct a specific robustness test. If you think that such analysis could have been done within the same methodology and using the same data (e.g., by including or excluding a subset of the data like “high-school dropouts” or “women”), please specify a robustness test that you would like to conduct before starting the Assessment stage.

These are the elements you will need for the Scoping stage. You now have all the elements necessary to complete Survey 1.

2.3 Setup your own revised reproduction package.

As part of the scoping stage, you have now identified the reproduction package of the original authors. In addition to these materials, we recommend that you create your own copy and called it a revised reproduction package. As you work through the next stages (assessment, improvement, and robustness) it is likely that you will modify some of the contents of the reproduction package. Keeping a record of those changes will help you as reproducers (eg. to document your assessments, or to communicate with original authors) and will likely help future reproducers allowing them to build on top of any contribution that they have created.

Please deposit the starting copy of your revised reproduction package (i.e. just the copy of the original reproduction package) in a trusted repository. Examples of trusted repositories include: Dataverse, openICPSR, Figshare, Dryad, Zenodo, Open Science Framework and others. We encourage reproducers to use version control software (Git) during their reproductions, and suggest they submit the entire repository (or link it to their GitHub repository).