The Social Science Reproduction Platform

class: center, middle, inverse, title-slide

.title[
# The Social Science Reproduction Platform
]
.subtitle[
## Fall RT2 - Online
]
.author[
### Fernando Hoces de la Guardia, BITSS
]
.date[
### December 2023 | <a href="https://bitss.github.io/ssrp-rt2-fall2023/">slides</a>
]

---

background-image: url("Images/iceberg.jpg")
background-size: 50%
background-position: 100% 50%
# Motivation 1/3: Share The Entire Scholarly Output

<style>

.center2 {
  margin: 0;
  position: absolute;
  top: 50%;
  left: 50%;
  -ms-transform: translate(-50%, -50%);
  transform: translate(-50%, -50%);
}

pre.sourceCode {
    max-height: 200px;
    overflow-y: auto;
}

/*
.remark-slide-number {
  position: inherit;
}

.remark-slide-number .progress-bar-container {
  position: absolute;
  bottom: 0;
  height: 4px;
  display: block;
  left: 0;
  right: 0;
}

.remark-slide-number .progress-bar {
 height: 100%;
 background-color: blue;
}
*/
</style>

</style>

.pull-left[
.font120[
 <blockquote>
 Clarebout Principle:
 
 “An article about computational science in a scientific publication is not the scholarship itself, it’s merely scholarship advertisement. The actual scholarship is the complete software development environment and the complete set of instructions which generated the figures.”
 
.right[ <cite> Buckheit and D.L. Donoho (1995, [2009](https://academiccommons.columbia.edu/doi/10.7916/D8QZ2MDS/download))</cite> ]
 </blockquote>
]
]

---
background-image: url("Images/iceberg.jpg")
background-size: 50%
background-position: 100% 50%

# Motivation 1/3: Share The Entire Scholarly Output

.pull-left[
.font120[
 <blockquote>
 Clarebout Principle:
 
 “An article about computational science in a scientific publication is not the scholarship itself, it’s merely scholarship advertisement. The actual scholarship is the complete software development environment and the complete set of instructions which generated the figures.”
 
.right[ <cite> Buckheit and D.L. Donoho (1995, [2009](https://academiccommons.columbia.edu/doi/10.7916/D8QZ2MDS/download))</cite> ]
 </blockquote>
]
]
---
# Motivation 1/3: Share The Entire Scholarly Output

.font130[
Potential benefits of following the Clarebout Principle

Well discussed potential positive effects on:
- Pedagogy  
- Incremental generation of knowledge 
]
--

.font130[
Under discussed:

- Possible positive effect on diversity, equity and inclusion: no connections or language skills ("appropriate politeness") required to obtain materials 
]

???
An under discussed benefit of this principle is that increases access to knowledge for students who are outside of elite schools (no connections or "appropriate politeness" required to obtain materials).

---

# Motivation 2/3: "Reproducibility Crisis"

.font170[
| Replication in Social Sciences (same method, different sample) 	| Reproduction in Economics (same data and methods) 	|
|-------------------------------------------------------------------	|------------------------------------------------------	|
| OSC ([2015](https://docs.google.com/document/d/1mm_4HZnEz_2Bh8XuiS2tpqCH08MFPyqUhi1baKPqR8Y/edit#heading=h.7vqf2cziid7z)): 30%-60% 	 | Chang & Li ([2015](https://www.nowpublishers.com/article/Details/CFR-0053)): 43% 	|
| Camerer et. al. ([2016](https://science.sciencemag.org/content/351/6280/1433)): ~60% 	| Gertler et. al. ([2017](https://www.nature.com/articles/d41586-018-02108-9)): 14% 	|
| Nosek & Camerer et. al. ([2018](https://www.nature.com/articles/s41562-018-0399-z)): ~60% 	| Kingi et. al. ([2018](https://hautahi.com/static/docs/Replication_aejae.pdf)): 43% 	|
| Klein et. al. ([2018](https://journals.sagepub.com/doi/10.1177/2515245918810225)): 50% 	| Wood et. al. ([2018](https://journals.plos.org/plosone/article?id=10.1371/journal.pone.0209416#abstract0)): 25% 	|
]

---
# Motivation 3/3: Prevent Loss of Knowledge

Every semester, graduate students around **the world** take an Empirical/Applied [`...`] course (e.g., Labor Economics, Social Psychology). A typical assignment consists of reproducing the results of a paper and, possibly, testing the robustness of its results.

| Stage 	| New Knowledge 	|
|:-----------------:	|:------------------------------------------------------------:	|
| 	| 	|
| 	| 	|
| 	| 	|
| 	| 	|

---
count:false

# Motivation 3/3: Prevent Loss of Knowledge

---
count:false

# Motivation 3/3: Prevent Loss of Knowledge

---
count:false

# Motivation 3/3: Prevent Loss of Knowledge

---
count:false

# Motivation 3/3: Prevent Loss of Knowledge

---

# Motivation 3/3: Prevent Loss of Knowledge 

.font130[
- In each of the previous steps there was valuable new knowledge 
- This knowledge is not usually disseminated 
 - End of semester report, or presentation that is not published. 
 - Language across courses is not standard (reproduction? replication?)
 - Current attempts to record this are closer to a full paper format (see [Replication Wiki](https://replication.uni-goettingen.de/))
- As a result there is a large amount of missed knowledge. 
 - Each new semester, new students repeat similar excesses and have to reinvent the wheel. 
 - Lost opportunity for building on top of previous exercises. 
 - Lost opportunity for aggregating the new knowledge.
]

---
name: guidelines

# Context for the SSRP

.font130[
- American Economics Association (AEA) creates first data policy in 2006.

- Must publish some data (waivers available)
]
--
.font130[
- AEA updates [policy in 2019](https://www.aeaweb.org/journals/policies/data-code/). 
    - Must post all data and code. Publication is conditional on verifying reproducibility (if confidential: must document extensively) 
    - A new requirement is to post all cleaning code, even for data that is not public
]
--

.font130[
- We should expect high levels of computational reproducibility after 2019 (AEA). 
]

.font130[

- We should not demand 100% reproducibility before, but we could identify the gaps and try to improve some.

]
---
# .font90[**A**ccelerating **C**omputational **Re**producibility: Framework]

## Beyond Binary Judgments

Reproductions can easily gravitate towards adversarial exchanges.
  - Early career researchers (ECR) have incentives to emphasize unsuccessful reproductions 
  - Original authors have a more senior position and can use it to deter in-depth reproductions from ECRs. 
  - The media also focuses on eye-catching headlines

### Our approach:

We do not want to say 
 > "Paper X is (ir)reproducible"    
 
  
--
We do want to say 
 > "Result Y in paper X has a high/low level of reproducibility according to several reproduction attempts. Moreover, improvements have been made to the original reproduction package, increasing its reproducibility to a higher level"

---
count:false

# .font90[**A**ccelerating **C**omputational **Re**producibility: Framework]

## Beyond Binary Judgments

### Our approach:

We do not want to say 
 > "Paper X is (ir)reproducible"

We do want to say 
 > "Result Y in paper X has a high/low **level** of reproducibility according to **several** reproduction attempts. Moreover, **improvements** have been made to the original reproduction package, **increasing** its reproducibility to a higher level"

---
count: true  
background-image: url(Images/paper-claims.svg)
background-size: 610px
background-position: 75% 0%
  
# ACRe Framework

.pull-left[
.font100[
Each **reproduction attempt** is centered 
around scientific **claims** (following [SCORE](https://replicats.research.unimelb.edu.au)).

One paper can contain several claims.

Each claim may be supported by various 
**display items**: tables, figures & inline results.

A reproduction attempt is at the claim level, 
and reproducers must record their 
**specifications** of interest.

Key challenge: **standardization** of concepts 
and formats.

]
                                
                                  
                                               DI: Display Item 
                                                S: Specificaiton
]
.pull-right[ 
]

---
count: true

# .font70[Large part of this exercise is about standardization]

.pull-left[
- Computational Reproduction (or Reproduction)

- Replication (will not mention this term again!)

- Reproduction attempt

- Reproduction package

- Claim

- Display item

- Specification

- Preferred specification

- Raw data

- Analysis data
]

.pull-right[

- Candidate paper

- Declared paper

- Reproduction tree

- Complete Workflow

- Computationally Reproducible from Analytic data (CRA)

- Computationally Reproducible from Raw data (CRR)

- Reasonable test

- Feasible test

- Minimal effort

]

---
count: false

# .font70[Large part of this exercise is about standardization]

.pull-left[
- **Computational Reproduction (or Reproduction)**

- **Replication (will not mention this term again!) **

- Reproduction attempt

- **Reproduction package**

- Claim

- Display item

- Specification

- Preferred specification

- **Raw data**

- **Analysis data**
]

.pull-right[

- **Candidate paper**

- **Declared paper**

- Reproduction tree

- Complete Workflow

- Computationally Reproducible from Analytic data (CRA)

- Computationally Reproducible from Raw data (CRR)

- Reasonable test

- Feasible test

- Minimal effort

]

---
background-image:  url(Images/home_page.png)
background-size: contain
count:true

.center[
# Demo: [socialsciencereproduction.org]()
]
---
background-image:  url(Images/stages.svg)
background-size: contain

# Stages

---
background-image: url(Images/select.svg),  url(Images/select_paper.png) 
background-size: 400px,  700px
background-position: 95% 100%,   0% 10%
count:false

# Select a paper

---
background-image: url(Images/select.svg), url(Images/select_example.png), url(Images/select_paper.png) 
background-size: 400px,  600px, 700px 
background-position: 95% 100%,  100% 0%, 0% 10%
count:true

# Select a paper

---
background-image: url(Images/scope.svg), url(Images/scope_paper.png)
background-size: 400px,  700px
background-position: 95% 100%, 0% 10%
count:false

# Scope

---
background-image: url(Images/scope.svg), url(Images/scope_paper2.png)  , url(Images/scope_paper.png)
background-size: 400px,  500px, 700px
background-position: 95% 100%,   90% 0%, 0% 10%
count:false

# Scope

---
background-image: url(Images/scope.svg), url(Images/scope_example.png), url(Images/scope_paper2.png)  , url(Images/scope_paper.png)
background-size: 400px,  600px, 500px, 700px
background-position: 95% 100%,  100% 75%,  90% 0%, 0% 10%
count:true

# Scope

---
background-image: url(Images/assess.svg), url(Images/assess_paper1.png)
background-size: 400px, 120px
background-position: 95% 100%, 0% 10%
count:false

# Assess 
---
background-image: url(Images/assess.svg), url(Images/assess_paper2.png)
background-size: 400px,  120px
background-position: 95% 100%, 0% 10%
count:false

# Assess

---
background-image: url(Images/assess.svg), url(Images/assess_example1.png), url(Images/assess_paper2.png)
background-size: 400px,  500px, 120px
background-position: 95% 100%,  15% 50%, 0% 10%
count:false

# Assess

---
background-image: url(Images/assess_example2.png), url(Images/assess.svg), url(Images/assess_example1.png)  , url(Images/assess_paper2.png)
background-size:  600px, 400px,  500px, 120px 
background-position: 100% 80%,95% 100%,    15% 50%, 0% 10%
count:true

# Assess

---
background-image: url(Images/improve.svg)
background-size: 700px
background-position: 50% 20%

# Improvements

## Three types of improvements:

.font130[

1. Improvements at the paper level 
2. Improvements at the display-item level  
3. Specific future improvements   
 ]
---

background-image: url(Images/improve.svg)
background-size: 300px
background-position: 0% 100%

# Improvements: Paper-level

.font140[
- Use version control software (Git/Github).
- Improve documentation: comments, indentations, object names, etc.   
- Re-organize the reproduction package into a set of folders and sub-folders that follow [standardized best practices](https://www.projecttier.org/tier-protocol/specifications/#overview-of-the-documentation), and add a master script that executes all the code in order, with no further modifications. [See AEA's reproduction template](https://github.com/AEADataEditor/replication-template).  
- Literate programming environment (e.g., Jupyter notebooks, RMarkdown)
- Re-write code using a differenet statistical software (ideally open source, like R, Python, or Julia).
- Set up a computing capsule (e.g., [Binder](https://mybinder.org/) and [Code Ocean](https://codeocean.com/)).
]
---
background-image: url(Images/improve.svg)
background-size: 300px
background-position: 0% 100%

# Improvements: Display item-level

.font120[
- Adding missing raw data: files or meta-data  
  - Example: "Add raw temperature and relative humidity data"
- Adding missing analytic data files
  - Example: "Copy the row files from Data folder into new `Analysis\trade cost\Input`"
- Adding missing analysis or cleaning code
  - Example: "Replaced broken Wald bootstrap code with updated code/command"
- Debugging code  
  - Example:  "was counting each group 4 times in round 1, so fixed that"

]
---
background-image: url(Images/improve.svg)
background-size: 300px
background-position: 0% 100%

# Improvements: future possible

We ask reproducer to leave concise and actionable tasks for other reproducers in the future.

Example 1: 
>"Revise the .aml and .bat code scripts to reflect reorganized structure"

Example 2: 
> "Provide data and codes generating the other two figures in the paper, which are not given in the replication file."

Example 3: 
> "Table 3 can be reproduced identically from the [...] analytic data files. I was not able to reproduce the analytic data files due to lack of access to ArcGIS software, but the code scripts and raw data files [...] are included in the reproduction package."

---
background-image: url(Images/robust.svg)
background-size: 700px
background-position: 50% 20%

# Robustness Checks

## Two main parts for robustness:

.font130[
1. Increase the number of robustness checks
2. Justify the appropriateness of a specific test
 ]
 
---
background-image: url(Images/robust.svg), url("Images/robustness checks.svg")
background-size: 300px, 500px
background-position: 0% 100%, 100% 60%

# Robustness

**Robustness checks:** any possible change in a computational choice, both in data analysis and data cleaning
--

.pull-left[
**Reasonable specifications** ([Simonsohn et. al., 2018](https://urisohn.com/sohn_files/wp/wordpress/wp-content/uploads/Paper-Specification-curve-2018-11-02.pdf)):

1. Sensible tests of the research question
 2. Expected to be statistically valid, and 
 3. Not redundant with other specifications in the set.

Reproducers will be able to record two types of contributions:

- Mapping the universe of robustness checks
- Proposing a specific robustness check
]
---
background-image: url(Images/robust.svg)
background-size: 400px
background-position: 95% 95%

# Robustness & Reproducibility

.pull-left[
Robustness with level 1

Robustness with levels 5-9

]

.pull-left[
Robustness with levels 2-4

Robustness with level 10

<img src="Images/robustness_lvl10.svg" width="60%" style="display: block; margin: auto;" />
]
---
background-image: url(Images/completed_repro1.png)
background-size: 450px
background-position: 0% 0% 
count:false
# Completed Reproduction: [Example](https://www.socialsciencereproduction.org/reproductions/40/published/index)

---
background-image: url(Images/completed_repro2.png), url(Images/completed_repro1.png)
background-size:  300px, 450px
background-position:  35% 0%, 0% 0%   
count:false
# Completed Reproduction: [Example](https://www.socialsciencereproduction.org/reproductions/40/published/index)

---
background-image: url(Images/completed_repro3.png) ,url(Images/completed_repro2.png), url(Images/completed_repro1.png)
background-size:  600px ,  300px, 450px
background-position:  100% 10%, 35% 0%, 0% 0%   
count:true
# Completed Reproduction: [Example](https://www.socialsciencereproduction.org/reproductions/40/published/index)

.pull-right[

.font130[
 
 
 - **Share it:** Original authors, instructors, other researchers, add it to your CV
 - **Discuss it:** Discourse forum for reproductions
 - **Cite it:** Reproduction DOI
 ]

]

---

# Promoting a Constructive Exchange

#### 1 - Contacting the original author(s) when there is no reproduction package
#### 2 - Contacting the original author(s) to request specific missing items of a reproduction package
#### 3 - Asking for additional guidance when some materials have been shared  
#### 4 - Response when the original author has refused to share due to *undisclosed reasons*   
#### 5 - Response when the original author has refused to share due to legal or ethical restrictions of the data
#### 6 - Contacting the original author to share the results of your reproduction exercise
#### 7 - Responding to hostile responses from original authors

#### Under development: sample responses form authors to reproducers
---

# .font80[Example 1: There is no reproduction package]

.font70[
>**Subject:** Reproduction package for `[“Title of the paper”]`

>Dear Dr. `[Lastname of Corresponding Author]`,
>
> I am contacting you to request a reproduction package for your paper titled `[Title]` which was published in `[Reference]`. A reproduction package may contain (raw and/or analytic) data, code, and other documentation that makes it possible to reproduce paper. Would you be able to share any of these items?
>
>I am a `[position]` at `[Institution]`, and I would like to reproduce the results, tables, and other figures using the reproduction materials mentioned above. I have chosen this paper because `[add context ...]`. Unfortunately, I was not able to locate any of these materials on the journal website, Dataverse `[or other data and code repositories]`, or in your website.
>
>I will record the result of my reproduction attempt on ACRE [...]. With your permission, I will also record the materials you share with me, which would allow access for other reproducers and avoid repeated requests directed to you. Please let me know if there are any legal or ethical restrictions that apply to all or parts of the reproduction materials so that I can take that into consideration during this exercise.
>
>In addition to your response above, would you be available to respond to future (non-repetitive) inquiries from me or other reproducers conducting an ACRE excercise? Though your cooperation with my and/or any future request would be extremely helpful, please note that you are *not required to comply*.

>Since I am required to complete this project by `[date]`, I would appreciate your response by `[deadline]`.
>
>Let me know if you have any questions. Please also feel free to contact my supervisor/instructor `[Name (email)]` for further details on this exercise. Thank you in advance for your help!
>
>Best regards,  
>`[Reproducer]`
]

---
count:false

# .font80[Example 1: There is no reproduction package]

.font70[
>**Subject:** Reproduction package for `[“Title of the paper”]`

>Dear Dr. `[Lastname of Corresponding Author]`,
>
> I am contacting you to request a reproduction package for your paper titled `[Title]` which was published in `[Reference]`. A reproduction package may contain (raw and/or analytic) data, code, and other documentation that makes it possible to reproduce paper. Would you be able to share any of these items?
>
>I am a `[position]` at `[Institution]`, and I would like to reproduce the results, tables, and other figures using the reproduction materials mentioned above. I have chosen this paper because `[add context ...]`. **Unfortunately, I was not able to locate any of these materials on the journal website, Dataverse `[or other data and code repositories]`, or in your website**.
>
>**I will record the result of my reproduction attempt on ACRE [...]. With your permission, I will also record the materials you share with me, which would allow access for other reproducers and avoid repeated requests directed to you. Please let me know if there are any legal or ethical restrictions that apply to all or parts of the reproduction materials so that I can take that into consideration during this exercise.**
>
>In addition to your response above, would you be available to respond to future (non-repetitive) inquiries from me or other reproducers conducting an ACRE excercise? **Though your cooperation with my and/or any future request would be extremely helpful, please note that you are *not required to comply*.**

---

# .font80[Example 1: Following up on additional materials]

**Template email:**

>**Subject:** Clarification for reproduction materials for `[“Title of the paper”]`

>Dear Dr. `[Lastname of Corresponding Author]`,
>
>Thank you for sharing the materials. They have been immensely helpful for my work.
>
>Unfortunately, I ran into a few issues as I delved into the reproduction exercise, and I think your guidance would be helpful in resolving them. `[Describe the issues and how you have tried to resolve them. Describe whatever files or parts of the data or code are missing. Refer to examples 1 and 2 below for more details]`.  
>  
>Thank you in advance for your help.
>  
>Best regards,  
>`[Reproducer]`

---
count:false
# .font80[Example 1: Following up on additional materials]

**Template email:**

>**Subject:** Clarification for reproduction materials for `[“Title of the paper”]`

>Dear Dr. `[Lastname of Corresponding Author]`,
>
>Thank you for sharing the materials. They have been immensely helpful for my work.
>
>Unfortunately, I ran into a few issues as I delved into the reproduction exercise, and I think your guidance would be helpful in resolving them. **`[Describe the issues and how you have tried to resolve them. Describe whatever files or parts of the data or code are missing. Refer to examples 1 and 2 below for more details]`**.  
>  
>Thank you in advance for your help.
>  
>Best regards,  
>`[Reproducer]`

---

# An example of well described issues:

.font80[
>Specifically, I am attempting to reproduce OUTPUT X (e.g., table 1, figure 3). I found that the following components are required to reproduce to reproduce OUTPUT X:

```md
         OUTPUT X
            └───[code] formatting_table1.R
                ├───output1_part1.txt  
                |   └───[code] output_table1.do           
                |       └───[data] analysis_data01.csv
                |          └───[code] data_cleaning01.R*
                |             └───[data] UNKNOWN
                └───output1_part2.txt  
                    └───[code] output_table2.do           
                        └───[data] analysis_data02.csv
                           └───[code] data_cleaning02.R
                              └───[data] admin_01raw.csv* 
```

>I have marked with an asterisk (\*) the items that I could not find in the reproduction materials: data_cleaning01.R and admin_01raw.csv. After accessing these files, I will also be able to identify the name of the raw data set required to obtain output1_part1.txt. This is to let you know that I may need to contact you again if I cannot find this file (labeled as UNKNOWN above) in the reproduction materials.
>
>I understand that this request will require some work for you or somebody in your research group, but I want to assure you that I will add these missing files to the reproduction package for your paper on the ACRE platform. Doing this will ensure that you will not be asked twice for the same missing file.
]

---
count:false

# An example of well described issues:

.font80[
>Specifically, I am attempting to reproduce OUTPUT X (e.g., table 1, figure 3). I found that the following components are required to reproduce to reproduce OUTPUT X:

```md
         OUTPUT X
            └───[code] formatting_table1.R
                ├───output1_part1.txt  
                |   └───[code] output_table1.do           
                |       └───[data] analysis_data01.csv
*               |          └───[code] data_cleaning01.R*
*               |             └───[data] UNKNOWN
                └───output1_part2.txt  
                    └───[code] output_table2.do           
                        └───[data] analysis_data02.csv
                           └───[code] data_cleaning02.R
*                             └───[data] admin_01raw.csv* 
```

>I have marked with an asterisk (\*) the items that I could not find in the reproduction materials: **data_cleaning01.R** and **admin_01raw.csv**. After accessing these files, I will also be able to identify the name of the raw data set required to obtain output1_part1.txt. This is to let you know that I may need to contact you again if I cannot find this file (labeled as **UNKNOWN** above) in the reproduction materials.
>
>I understand that this request will require some work for you or somebody in your research group, but I want to assure you that I will add these missing files to the reproduction package for your paper on the ACRE platform. **Doing this will ensure that you will not be asked twice for the same missing file.**
]

---
background-image:  url(Images/use1.png)
background-size:  500px
background-position:   0% 0%   
count:false
# How Do I Get Started?

.center[
.font120[
 &nbsp; &nbsp;&nbsp;&nbsp;&nbsp; [**Use it**](https://www.socialsciencereproduction.org) for your class 
&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;or independent project
]
]

---
background-image: url(Images/use2.png), url(Images/use1.png)
background-size:  600px, 500px
background-position:  35% 100%, 0% 0%   
count:false
# How Do I Get Started?

.font120[
Consult the supporting 
[**guide**](https://bitss.github.io/ACRE/)
]

---
background-image: url(Images/use3.png) ,url(Images/use2.png), url(Images/use1.png)
background-size:  400px,  600px, 500px
background-position:  100% 0%, 35% 100%, 0% 0%   
count:true
# How Do I Get Started?

.font120[
Consult the supporting 
[**guide**](https://bitss.github.io/ACRE/)
]
.right[
.font120[
[**Ask**](https://forum.socialsciencereproduction.org) questions
]
]

---
background-image: url(Images/metrics3.png)
background-size:  contain
count:true
# Current Users of the SSRP

---
class: inverse, center, middle

# .font140[ Thank You]

.font190[
<acre@berkeley.edu>
]