Accelerating Computational Reproducibility with the Social Science Reproduction Platform

class: center, middle, inverse, title-slide

# Accelerating Computational Reproducibility with the Social Science Reproduction Platform
## Metascience Conference
### Fernando Hoces de la Guardia, BITSS
### September 2021 | <a href="https://bitss.github.io/acre-slides-metascience2021/">slides</a>

---

background-image: url("Images/BITSSlogo.png"), url(Images/cega.png)
background-size: contain, 200px
background-position: 50% 100% , 100% 10% 
count:true

#[BITSS](https://bitss.org)

<style>

.center2 {
  margin: 0;
  position: absolute;
  top: 50%;
  left: 50%;
  -ms-transform: translate(-50%, -50%);
  transform: translate(-50%, -50%);
}

pre.sourceCode {
    max-height: 200px;
    overflow-y: auto;
}

/*
.remark-slide-number {
  position: inherit;
}

.remark-slide-number .progress-bar-container {
  position: absolute;
  bottom: 0;
  height: 4px;
  display: block;
  left: 0;
  right: 0;
}

.remark-slide-number .progress-bar {
  height: 100%;
  background-color: blue;
}
*/
</style>

<style type="text/css">
# CSS for including pauses in printed PDF output (see bottom of lecture)
@media print {
  .has-continuation {
    display: block !important;
  }
}

</style>

.pull-left[
The Berkeley Initiative for Transparency in the Social Sciences works to improve </br>the credibility of science by </br>
advancing transparency, </br> reproducibility,  rigor, and </br> ethics in research and policy analysis.

.font130[**Core ACRe**]

Aleks Bogdanoski  
 
 Albert Chae

Fernando Hoces  de la Guardia
 
 Katie Hoeberling

Edward Miguel
 
Lars Vilhuber
 
]
.pull-right[ .right[
We are part of the Center for Effective Global Action ([CEGA](https://cega.berkeley.edu/)).   
 </br></br></br></br></br></br></br>

.font130[[**Many Others**](https://bitss.github.io/ACRE/contributions.html#list-of-contributors-guidelines-content-and-source-code)]

CEGA staff

Undergrad and Graduate RAs

Catalysts

Outside Collaborators   
(Researchers, Programmers)
  ]]

---
count:false
# Motivation: Computational Reproducibility

.pull-right[
.font130[
    Clarebout Principle:
    </br></br></br></br></br></br></br></br></br></br></br></br>

]

]

---

background-image: url("Images/iceberg.jpg")
background-size: 50%
background-position: 100% 50%
count:false
# Motivation: Computational Reproducibility

.pull-right[
.font130[
    "Clarebout Principle":
    </br></br></br></br></br></br></br></br></br></br></br></br>
.right[
<cite> Buckheit and D.L. Donoho (1995, [2009](https://academiccommons.columbia.edu/doi/10.7916/D8QZ2MDS/download))</cite>
]
]

]

---
background-image: url("Images/iceberg.jpg")
background-size: 50%
background-position: 100% 50%
count:false
# Motivation: Computational Reproducibility

.pull-left[
Every semester, graduate students around **the world** take an Empirical/Applied [`...`] course. A typical assignment consists of reproducing the results of a paper and, possibly, testing the robustness of its results.

|       Stage       	|                         New Knowledge                        	|
|:-----------------:	|:------------------------------------------------------------:	|
|  </br></br> 	|                                          	|
| </br></br>            	|  	|
| </br></br>           	|  	|
| </br></br>   	|  </br>          	|

]

.pull-right[
.font130[
    "Clarebout Principle":
    </br></br></br></br></br></br></br></br></br></br></br></br>
.right[
<cite> Buckheit and D.L. Donoho (1995, [2009](https://academiccommons.columbia.edu/doi/10.7916/D8QZ2MDS/download))</cite>
]
]

]

---

background-image: url("Images/iceberg.jpg")
background-size: 50%
background-position: 100% 50%
count:false
# Motivation: Computational Reproducibility

.pull-left[
Every semester, graduate students around **the world** take an Empirical/Applied [`...`] course. A typical assignment consists of reproducing the results of a paper and, possibly, testing the robustness of its results.

|       Stage       	|                         New Knowledge                        	|
|:-----------------:	|:------------------------------------------------------------:	|
| Scope </br> (select and verify) 	| Data and code exist?                                         	|
| </br></br>            	|  	|
| </br></br>           	|  	|
| </br></br>   	|  </br>          	|

]

.pull-right[
.font130[
    "Clarebout Principle":
    </br></br></br></br></br></br></br></br></br></br></br></br>
.right[
<cite> Buckheit and D.L. Donoho (1995, [2009](https://academiccommons.columbia.edu/doi/10.7916/D8QZ2MDS/download))</cite>
]
]

]

---

background-image: url("Images/iceberg.jpg")
background-size: 50%
background-position: 100% 50%
count:false
# Motivation: Computational Reproducibility

.pull-left[
Every semester, graduate students around **the world** take an Empirical/Applied [`...`] course. A typical assignment consists of reproducing the results of a paper and, possibly, testing the robustness of its results.

|       Stage       	|                         New Knowledge                        	|
|:-----------------:	|:------------------------------------------------------------:	|
| Scope </br> (select and verify) 	| Data and code exist?                                         	|
| Assess            	| Degree of reproducibility for <br>specific part of the paper 	|
| </br></br>           	|  	|
| </br></br>   	|  </br>          	|

]

.pull-right[
.font130[
    "Clarebout Principle":
    </br></br></br></br></br></br></br></br></br></br></br></br>
.right[
<cite> Buckheit and D.L. Donoho (1995, [2009](https://academiccommons.columbia.edu/doi/10.7916/D8QZ2MDS/download))</cite>
]
]

]

---
background-image: url("Images/iceberg.jpg")
background-size: 50%
background-position: 100% 50%
count:false
# Motivation: Computational Reproducibility

.pull-left[
Every semester, graduate students around **the world** take an Empirical/Applied [`...`] course. A typical assignment consists of reproducing the results of a paper and, possibly, testing the robustness of its results.

|       Stage       	|                         New Knowledge                        	|
|:-----------------:	|:------------------------------------------------------------:	|
| Scope </br> (select and verify) 	| Data and code exist?                                         	|
| Assess            	| Degree of reproducibility for <br>specific part of the paper 	|
| Improve           	| E.g. fixed paths, libraries,<br>added missing files, etc.    	|
| </br></br>   	|  </br>          	|

]

.pull-right[
.font130[
    "Clarebout Principle":
    </br></br></br></br></br></br></br></br></br></br></br></br>
.right[
<cite> Buckheit and D.L. Donoho (1995, [2009](https://academiccommons.columbia.edu/doi/10.7916/D8QZ2MDS/download))</cite>
]
]

]
---
background-image: url("Images/iceberg.jpg")
background-size: 50%
background-position: 100% 50%
count:true

# Motivation: Computational Reproducibility

.pull-left[
Every semester, graduate students around **the world** take an Empirical/Applied [`...`] course. A typical assignment consists of reproducing the results of a paper and, possibly, testing the robustness of its results.

|       Stage       	|                         New Knowledge                        	|
|:-----------------:	|:------------------------------------------------------------:	|
| Scope </br> (select and verify) 	| Data and code exist?                                         	|
| Assess            	| Degree of reproducibility for <br>specific part of the paper 	|
| Improve           	| E.g. fixed paths, libraries,<br>added missing files, etc.    	|
| Test robustness   	| Results are robust to additional <br>specifications          	|

]

.pull-right[
.font130[
    "Clarebout Principle":
    </br></br></br></br></br></br></br></br></br></br></br></br>
.right[
<cite> Buckheit and D.L. Donoho (1995, [2009](https://academiccommons.columbia.edu/doi/10.7916/D8QZ2MDS/download))</cite>
]
]
]

---
# .font90[**A**ccelerating **C**omputational **Re**producibility: Framework]

## Beyond Binary Judgments

Reproductions can easily gravitate towards adversarial exchanges.
  - Early career researchers (ECR) have incentives to emphasize unsuccessful reproductions 
  - Original authors have a more senior position and can use it to deter in-depth reproductions from ECRs. 
  - The media also focuses on eye-catching headlines

--

### Our approach:

We do not want to say 
 > "Paper X is (ir)reproducible"    
 
  
--
We do want to say 
 > "Result Y in paper X has a high/low level of reproducibility according to several reproduction attempts. Moreover, improvements have been made to the original reproduction package, increasing its reproducibility to a higher level"

---
count:false

# .font90[**A**ccelerating **C**omputational **Re**producibility: Framework]

## Beyond Binary Judgments

Reproductions can easily gravitate towards adversarial exchanges.
  - Early career researchers (ECR) have incentives to emphasize unsuccessful reproductions 
  - Original authors have a more senior position and can use it to deter in-depth reproductions from ECRs. 
  - The media also focuses on eye-catching headlines

### Our approach:

We do not want to say 
 > "Paper X is (ir)reproducible"

We do want to say 
 > "Result Y in paper X has a high/low **level** of reproducibility according to **several** reproduction attempts. Moreover, **improvements** have been made to the original reproduction package, **increasing** its reproducibility to a higher level"

---
count: true  
background-image: url(Images/paper-claims.svg)
background-size: 610px
background-position: 75% 0%
  
# ACRe Framework

.pull-left[
.font100[
Each **reproduction attempt** is centered <br>
around scientific **claims** (following [SCORE](https://replicats.research.unimelb.edu.au)).

One paper can contain several claims.

Each claim may be supported by various </br>
**display items**: tables, figures & inline results.

A reproduction attempt is at the claim level, <br>
and reproducers must record their <br> 
**specifications** of interest.

Key challenge: **standardization** of concepts <br> 
and formats.

]
                                
                                  
                                               DI: Display Item 
                                                S: Specificaiton
]
.pull-right[ 
]

---
background-image:  url(Images/home_page.png)
background-size: contain
count:true

.center[
# Demo: [socialsciencereproduction.org]()
]
---
background-image:  url(Images/stages.svg)
background-size: contain

# Stages

---
background-image: url(Images/select.svg),  url(Images/select_paper.png) 
background-size: 400px,  700px
background-position: 95% 100%,   0% 10%
count:false

# Select a paper

---
background-image: url(Images/select.svg), url(Images/select_example.png), url(Images/select_paper.png) 
background-size: 400px,  600px, 700px 
background-position: 95% 100%,  100% 0%, 0% 10%
count:true

# Select a paper

---
background-image: url(Images/scope.svg), url(Images/scope_paper.png)
background-size: 400px,  700px
background-position: 95% 100%, 0% 10%
count:false

# Scope

---
background-image: url(Images/scope.svg), url(Images/scope_paper2.png)  , url(Images/scope_paper.png)
background-size: 400px,  500px, 700px
background-position: 95% 100%,   90% 0%, 0% 10%
count:false

# Scope

---
background-image: url(Images/scope.svg), url(Images/scope_example.png), url(Images/scope_paper2.png)  , url(Images/scope_paper.png)
background-size: 400px,  600px, 500px, 700px
background-position: 95% 100%,  100% 75%,  90% 0%, 0% 10%
count:true

# Scope

---
background-image: url(Images/assess.svg), url(Images/assess_paper1.png)
background-size: 400px, 120px
background-position: 95% 100%, 0% 10%
count:false

# Assess 
---
background-image: url(Images/assess.svg), url(Images/assess_paper2.png)
background-size: 400px,  120px
background-position: 95% 100%, 0% 10%
count:false

# Assess

---
background-image: url(Images/assess.svg), url(Images/assess_example1.png), url(Images/assess_paper2.png)
background-size: 400px,  500px, 120px
background-position: 95% 100%,  15% 50%, 0% 10%
count:false

# Assess

---
background-image: url(Images/assess_example2.png), url(Images/assess.svg), url(Images/assess_example1.png)  , url(Images/assess_paper2.png)
background-size:  600px, 400px,  500px, 120px 
background-position: 100% 80%,95% 100%,    15% 50%, 0% 10%
count:true

# Assess

---
background-image: url(Images/impr_rob.svg)
background-size: contain

# Improvements & Robustness

### Three types of improvements:

.font130[
1. Improvements at the paper level 
2. Improvements at the display-item level  
3. Specific future improvements   
 ]

<br><br><br><br><br><br>

## Two main parts for robustness:

.font130[
1. Increase the number of robustness checks
2. Justify the reasonableness of a specific test
 ]

---
background-image: url(Images/robust.svg)
background-size: 400px
background-position: 95% 95%

# Robustness & Reproducibility

.pull-left[
Robustness with level 1

<img src="Images/robustness_lvl0.svg" width="60%" style="display: block; margin: auto;" />

Robustness with levels 5-9

<img src="Images/robustness checks.svg" width="60%" style="display: block; margin: auto;" />

]

.pull-left[
Robustness with levels 2-4

<img src="Images/robustness_lvl2_4.svg" width="60%" style="display: block; margin: auto;" />

Robustness with level 10

<img src="Images/robustness_lvl10.svg" width="60%" style="display: block; margin: auto;" />
]
---
background-image:  url(Images/completed_repro1.png)
background-size:   450px
background-position:  0% 0%   
count:false
# Completed Reproduction: [Example](https://www.socialsciencereproduction.org/reproductions/40/published/index)

---
background-image: url(Images/completed_repro2.png), url(Images/completed_repro1.png)
background-size:  300px, 450px
background-position:  35% 0%, 0% 0%   
count:false
# Completed Reproduction: [Example](https://www.socialsciencereproduction.org/reproductions/40/published/index)

---
background-image: url(Images/completed_repro3.png) ,url(Images/completed_repro2.png), url(Images/completed_repro1.png)
background-size:  600px ,  300px, 450px
background-position:  100% 10%, 35% 0%, 0% 0%   
count:true
# Completed Reproduction: [Example](https://www.socialsciencereproduction.org/reproductions/40/published/index)

.pull-right[

.font130[
 <br><br><br><br><br><br><br>
 
 - **Share it:** Original authors, instructors, other researchers, add it to your CV
 - **Discuss it:** Discourse forum for reproductions
 - **Cite it:**    Reproduction DOI coming soon
 ]

]

---
background-image:  url(Images/use1.png)
background-size:  500px
background-position:   0% 0%   
count:false
# How Do I Get Started?

.center[
.font120[
    &nbsp; &nbsp;&nbsp;&nbsp;&nbsp;  [**Use it**](https://www.socialsciencereproduction.org) for your class <br>
&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;or independent project
]
]

---
background-image: url(Images/use2.png), url(Images/use1.png)
background-size:  600px, 500px
background-position:  35% 100%, 0% 0%   
count:false
# How Do I Get Started?

.center[
.font120[
    &nbsp; &nbsp;&nbsp;&nbsp;&nbsp;  [**Use it**](https://www.socialsciencereproduction.org) for your class <br>
&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;or independent project
]
]

<br><br><br><br><br><br><br><br>
.font120[
Consult the <br> supporting <br>
[**guide**](https://bitss.github.io/ACRE/)
]

---
background-image: url(Images/use3.png) ,url(Images/use2.png), url(Images/use1.png)
background-size:  400px,  600px, 500px
background-position:  100% 0%, 35% 100%, 0% 0%   
count:true
# How Do I Get Started?

.center[
.font120[
    &nbsp; &nbsp;&nbsp;&nbsp;&nbsp;  [**Use it**](https://www.socialsciencereproduction.org) for your class <br>
&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;or independent project
]
]

<br><br><br><br><br><br><br><br>
.font120[
Consult the <br> supporting <br>
[**guide**](https://bitss.github.io/ACRE/)
]
.right[
.font120[
[**Ask**](https://forum.socialsciencereproduction.org) questions
]
]

---
class: inverse, center, middle

# .font140[ Thank You]

<html><div style='float:left'></div><hr color='#EB811B' size=1px width=100%></html>

.font190[
<acre@berkeley.edu>
]