The SSRP and the Life Cycle of Reproducibility

class: center, middle, inverse, title-slide

# The SSRP and the Life Cycle of Reproducibility
## BITSS Open Research Seminar
### Fernando Hoces de la Guardia, BITSS
### October 2022

---

background-image:  url(Images/tier.png), url(Images/ssrp-logo.png), url(Images/i4r.png)
background-size: 25%, 35%, 25%
background-position: 0% 50%, 50% 50%, 100% 50% 
count:true

<style>

.center2 {
  margin: 0;
  position: absolute;
  top: 50%;
  left: 50%;
  -ms-transform: translate(-50%, -50%);
  transform: translate(-50%, -50%);
}

pre.sourceCode {
    max-height: 200px;
    overflow-y: auto;
}

/*
.remark-slide-number {
  position: inherit;
}

.remark-slide-number .progress-bar-container {
  position: absolute;
  bottom: 0;
  height: 4px;
  display: block;
  left: 0;
  right: 0;
}

.remark-slide-number .progress-bar {
  height: 100%;
  background-color: blue;
}
*/
</style>

<style type="text/css">
# CSS for including pauses in printed PDF output (see bottom of lecture)
@media print {
  .has-continuation {
    display: block !important;
  }
}

</style>

# Social Science Reproduction Platform (SSRP)

---
background-image: url("Images/iceberg.jpg")
background-size: 50%
background-position: 100% 50%

# .font90[Take-away From Project TIER's Talk]

.pull-left[
.font130[
 - We should document and test well our work as researchers (using the TIER protocol) so others can easily inspect and learn from our entire scholarly contribution

- Let's share the entire "iceberg" when producing new research
 
 - But what about research that has already been published?
 
 ]
]

---

# Motivation: Prevent Loss of Knowledge

.font120[Every semester, graduate students around **the world** take an Empirical/Applied [`...`] course (e.g., Labor Economics, Social Psychology). A typical assignment consists of reproducing the results of a paper and, possibly, testing the robustness of its results. ]

--

|       Stage       	|                         New Knowledge                        	|
|:-----------------:	|:------------------------------------------------------------:	|
|  </br></br> 	|                                          	|
| </br></br>            	|  	|
| </br></br>           	|  	|
| </br></br>   	|  </br>          	|

---
count:false

# Motivation: Prevent Loss of Knowledge

.font120[Every semester, graduate students around **the world** take an Empirical/Applied [`...`] course (e.g., Labor Economics, Social Psychology). A typical assignment consists of reproducing the results of a paper and, possibly, testing the robustness of its results. ]

|       Stage       	|                         New Knowledge                        	|
|:-----------------:	|:------------------------------------------------------------:	|
| Scope </br> (select and verify) 	| Data and code exist?                                         	|
| </br></br>            	|  	|
| </br></br>           	|  	|
| </br></br>   	|  </br>          	|

---
count:false

# Motivation: Prevent Loss of Knowledge

.font120[Every semester, graduate students around **the world** take an Empirical/Applied [`...`] course (e.g., Labor Economics, Social Psychology). A typical assignment consists of reproducing the results of a paper and, possibly, testing the robustness of its results. ]

|       Stage       	|                         New Knowledge                        	|
|:-----------------:	|:------------------------------------------------------------:	|
| Scope </br> (select and verify) 	| Data and code exist?                                         	|
| Assess            	| Degree of reproducibility for <br>specific part of the paper 	|
| </br></br>           	|  	|
| </br></br>   	|  </br>          	|

---
count:false

# Motivation: Prevent Loss of Knowledge

.font120[Every semester, graduate students around **the world** take an Empirical/Applied [`...`] course (e.g., Labor Economics, Social Psychology). A typical assignment consists of reproducing the results of a paper and, possibly, testing the robustness of its results. ]

|       Stage       	|                         New Knowledge                        	|
|:-----------------:	|:------------------------------------------------------------:	|
| Scope </br> (select and verify) 	| Data and code exist?                                         	|
| Assess            	| Degree of reproducibility for <br>specific part of the paper 	|
| Improve           	| E.g. fixed paths, libraries,<br>added missing files, etc.    	|
| </br></br>   	|  </br>          	|

---
count:false

# Motivation: Prevent Loss of Knowledge

.font120[Every semester, graduate students around **the world** take an Empirical/Applied [`...`] course (e.g., Labor Economics, Social Psychology). A typical assignment consists of reproducing the results of a paper and, possibly, testing the robustness of its results. ]

|       Stage       	|                         New Knowledge                        	|
|:-----------------:	|:------------------------------------------------------------:	|
| Scope </br> (select and verify) 	| Data and code exist?                                         	|
| Assess            	| Degree of reproducibility for <br>specific part of the paper 	|
| Improve           	| E.g. fixed paths, libraries,<br>added missing files, etc.    	|
| Test robustness   	| Results are robust to additional <br>specifications          	|

.font120[
.center[**Key challenge: Standardization**]
]

---
count: true  
background-image: url(Images/paper-claims.svg)
background-size: 610px
background-position: 75% 0%
  
# Standarized Scope of a Reproduction

.pull-left[
.font100[
Each **reproduction attempt** is centered <br>
around scientific **claims** (following [SCORE](https://replicats.research.unimelb.edu.au)).

One paper can contain several claims.

Each claim may be supported by various </br>
**display items**: tables, figures & inline results.

A reproduction attempt is at the claim level, <br>
and reproducers must record their <br> 
**specifications** of interest.

<br><br>

]
                                
                                  
                                               DI: Display Item 
                                                S: Specificaiton
]

---
background-image:  url(Images/home_page.png)
background-size: contain
count:true

.center[
# Demo: [socialsciencereproduction.org]()
]
---
background-image:  url(Images/stages.svg)
background-size: contain

# Stages

---
background-image: url(Images/scope.svg), url(Images/scope_paper.png)
background-size: 400px,  700px
background-position: 95% 100%, 0% 10%
count:false

# Scope

---
background-image: url(Images/scope.svg), url(Images/scope_paper2.png), url(Images/scope_paper.png)
background-size: 400px, 500px, 700px
background-position: 95% 100%, 90% 0%, 0% 10%
count:false

# Scope

---
background-image: url(Images/scope.svg), url(Images/scope_example.png), url(Images/scope_paper2.png)  , url(Images/scope_paper.png)
background-size: 400px,  600px, 500px, 700px
background-position: 95% 100%,  100% 75%,  90% 0%, 0% 10%
count:true

# Scope

---
background-image: url(Images/assess.svg), url(Images/assess_paper1.png)
background-size: 400px, 120px
background-position: 95% 100%, 0% 10%
count:false

# Standzarized Assessment 
---
background-image: url(Images/assess.svg), url(Images/assess_paper2.png)
background-size: 400px,  120px
background-position: 95% 100%, 0% 10%
count:false

# Standzarized Assessment

---
background-image: url(Images/assess.svg), url(Images/assess_example1.png), url(Images/assess_paper2.png)
background-size: 400px,  500px, 120px
background-position: 95% 100%,  15% 50%, 0% 10%
count:false

# Standzarized Assessment

---
background-image: url(Images/assess_example2.png), url(Images/assess.svg), url(Images/assess_example1.png)  , url(Images/assess_paper2.png)
background-size:  600px, 400px,  500px, 120px 
background-position: 100% 80%,95% 100%,    15% 50%, 0% 10%
count:true

# Standzarized Assessment

---
background-image: url(Images/improve.svg)
background-size: 700px
background-position: 50% 20%

# Standarized Improvements

</br></br></br></br></br></br></br></br>

## Three types of improvements:

.font130[

1. Improvements at the paper level (e.g., improve documentation following [TIER's protocols](https://www.projecttier.org/tier-protocol/specifications/#overview-of-the-documentation))
2. Improvements at the display-item level  
3. Specific future improvements   
 ]

---
background-image: url(Images/robust.svg)
background-size: 700px
background-position: 50% 20%

# Robustness Checks

</br></br></br></br></br></br></br></br>

## Two main parts for robustness:

.font130[
1. Increase the number of robustness checks
2. Justify the appropriateness of a specific test
 ]
 
---
background-image: url(Images/robust.svg), url("Images/robustness checks.svg")
background-size: 300px, 500px
background-position: 0% 100%, 100% 60%

# Robustness
.font120[
**Robustness checks:** any possible change in a computational choice, both in data analysis and data cleaning
]
--

.font110[
.pull-left[
**Reasonable specifications** ([Simonsohn et. al., 2018](https://urisohn.com/sohn_files/wp/wordpress/wp-content/uploads/Paper-Specification-curve-2018-11-02.pdf)):

1. Sensible tests of the research question
 2. Expected to be statistically valid, and 
 3. Not redundant with other specifications in the set.

Reproducers will be able to record two types of contributions: 
- Mapping the universe of robustness checks
- Proposing a specific robustness check
]
]
---
background-image:  url(Images/completed_repro1.png)
background-size:   450px
background-position:  0% 0%   
count:false
# .font90[Completed Reproduction: Generate Knowledge and Get Credit]

---
background-image: url(Images/completed_repro2.png), url(Images/completed_repro1.png)
background-size:  300px, 450px
background-position:  35% 0%, 0% 0%   
count:false
# .font90[Completed Reproduction: Generate Knowledge and Get Credit]

---
background-image: url(Images/completed_repro3.png) ,url(Images/completed_repro2.png), url(Images/completed_repro1.png)
background-size:  600px ,  300px, 450px
background-position:  100% 10%, 35% 0%, 0% 0%   
count:true
# .font90[Completed Reproduction: Generate Knowledge and Get Credit]

.pull-right[

.font130[
 <br><br><br><br><br><br><br>
 
 - **Share it:** Original authors, instructors, other researchers, add it to your CV
 - **Discuss it:** Discourse forum for reproductions
 - **Cite it:**    Each reproduction has a citable DOI
 - [Example here](https://www.socialsciencereproduction.org/reproductions/40/published/index)
 ]

]

---
background-image:  url(Images/use1.png)
background-size:  500px
background-position:   0% 0%   
count:false
# How Do I Get Started?

.center[
.font120[
    &nbsp; &nbsp;&nbsp;&nbsp;&nbsp;  [**Use it**](https://www.socialsciencereproduction.org) for your class <br>
&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;or independent project
]
]

---
background-image: url(Images/use2.png), url(Images/use1.png)
background-size:  600px, 500px
background-position:  35% 100%, 0% 0%   
count:false
# How Do I Get Started?

.center[
.font120[
    &nbsp; &nbsp;&nbsp;&nbsp;&nbsp;  [**Use it**](https://www.socialsciencereproduction.org) for your class <br>
&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;or independent project
]
]

<br><br><br><br><br><br><br><br>
.font120[
Consult the <br> supporting <br>
[**guide**](https://bitss.github.io/ACRE/)
]

---
background-image: url(Images/use3.png) ,url(Images/use2.png), url(Images/use1.png)
background-size:  400px,  600px, 500px
background-position:  100% 0%, 35% 100%, 0% 0%   
count:true
# How Do I Get Started?

.center[
.font120[
    &nbsp; &nbsp;&nbsp;&nbsp;&nbsp;  [**Use it**](https://www.socialsciencereproduction.org) for your class <br>
&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;or independent project
]
]

<br><br><br><br><br><br><br><br>
.font120[
Consult the <br> supporting <br>
[**guide**](https://bitss.github.io/ACRE/)
]
.right[
.font120[
[**Ask**](https://forum.socialsciencereproduction.org) questions
]
]

---
background-image: url(Images/metrics3.png)
background-size:  contain
count:true
# Who Is Using The SSRP?

---
class: inverse, center, middle

# .font140[ Thank You]

<html><div style='float:left'></div><hr color='#EB811B' size=1px width=100%></html>

.font190[
<acre@berkeley.edu>  
<fhoces@berkeley.edu>
]