The Social Science Reproduction Platform

class: center, middle, inverse, title-slide

.title[
# The Social Science Reproduction Platform
]
.subtitle[
## RT2
]
.author[
### Fernando Hoces de la Guardia, BITSS
]
.date[
### 06 June 2024  <a href="https://fhoces.github.io/RT2-ssrp-2024/">Slides</a>
]

---

background-image: url("Images/iceberg.jpg")
background-size: 50%
background-position: 100% 50%

<style>

.center2 {
  margin: 0;
  position: absolute;
  top: 50%;
  left: 50%;
  -ms-transform: translate(-50%, -50%);
  transform: translate(-50%, -50%);
}

pre.sourceCode {
    max-height: 200px;
    overflow-y: auto;
}

/*
.remark-slide-number {
  position: inherit;
}

.remark-slide-number .progress-bar-container {
  position: absolute;
  bottom: 0;
  height: 4px;
  display: block;
  left: 0;
  right: 0;
}

.remark-slide-number .progress-bar {
  height: 100%;
  background-color: blue;
}
*/
</style>

<style type="text/css">
# CSS for including pauses in printed PDF output (see bottom of lecture)
@media print {
  .has-continuation {
    display: block !important;
  }
}

</style>

# Motivation I: Computational Reproducibility

.pull-left[
<br>
.font120[
    Clarebout Principle:
    </br></br>
    “An article about computational science in a scientific publication is not the scholarship itself, it’s merely scholarship advertisement. The actual scholarship is the complete software development environment and the complete set of instructions which generated the figures.”
.right[  <cite> Buckheit and D.L. Donoho (1995, [2009](https://academiccommons.columbia.edu/doi/10.7916/D8QZ2MDS/download))</cite>]
]

]

---
background-image: url("Images/iceberg.jpg")
background-size: 50%
background-position: 100% 50%
count:false
# Motivation I: Computational Reproducibility

.pull-left[
<br>
.font120[
    Clarebout Principle:
    </br></br>
    “An **article** about computational science in a **scientific publication** is not the scholarship itself, it’s merely scholarship **advertisement**. The **actual scholarship** is the **complete software development environment and the complete set of instructions** which generated the figures.”
.right[  <cite> Buckheit and D.L. Donoho (1995, [2009](https://academiccommons.columbia.edu/doi/10.7916/D8QZ2MDS/download))</cite>]
]

]

---

# Motivation II: Prevent Loss of Knowledge

.font120[Every semester, graduate students around **the world** take an Empirical/Applied [`...`] course (e.g., Labor Economics, Social Psychology). A typical assignment consists of reproducing the results of a paper and, possibly, testing the robustness of its results. ]

--

|       Stage       	|                         New Knowledge                        	|
|:-----------------:	|:------------------------------------------------------------:	|
|  </br></br> 	|                                          	|
| </br></br>            	|  	|
| </br></br>           	|  	|
| </br></br>   	|  </br>          	|

---
count:false

# Motivation II: Prevent Loss of Knowledge

.font120[Every semester, graduate students around **the world** take an Empirical/Applied [`...`] course (e.g., Labor Economics, Social Psychology). A typical assignment consists of reproducing the results of a paper and, possibly, testing the robustness of its results. ]

|       Stage       	|                         New Knowledge                        	|
|:-----------------:	|:------------------------------------------------------------:	|
| Scope </br> (select and verify) 	| Data and code exist?                                         	|
| </br></br>            	|  	|
| </br></br>           	|  	|
| </br></br>   	|  </br>          	|

---
count:false

# Motivation II: Prevent Loss of Knowledge

.font120[Every semester, graduate students around **the world** take an Empirical/Applied [`...`] course (e.g., Labor Economics, Social Psychology). A typical assignment consists of reproducing the results of a paper and, possibly, testing the robustness of its results. ]

|       Stage       	|                         New Knowledge                        	|
|:-----------------:	|:------------------------------------------------------------:	|
| Scope </br> (select and verify) 	| Data and code exist?                                         	|
| Assess            	| Degree of reproducibility for <br>specific part of the paper 	|
| </br></br>           	|  	|
| </br></br>   	|  </br>          	|

---
count:false

# Motivation II: Prevent Loss of Knowledge

.font120[Every semester, graduate students around **the world** take an Empirical/Applied [`...`] course (e.g., Labor Economics, Social Psychology). A typical assignment consists of reproducing the results of a paper and, possibly, testing the robustness of its results. ]

|       Stage       	|                         New Knowledge                        	|
|:-----------------:	|:------------------------------------------------------------:	|
| Scope </br> (select and verify) 	| Data and code exist?                                         	|
| Assess            	| Degree of reproducibility for <br>specific part of the paper 	|
| Improve           	| E.g. fixed paths, libraries,<br>added missing files, etc.    	|
| </br></br>   	|  </br>          	|

---
count:false

# Motivation II: Prevent Loss of Knowledge

.font120[Every semester, graduate students around **the world** take an Empirical/Applied [`...`] course (e.g., Labor Economics, Social Psychology). A typical assignment consists of reproducing the results of a paper and, possibly, testing the robustness of its results. ]

|       Stage       	|                         New Knowledge                        	|
|:-----------------:	|:------------------------------------------------------------:	|
| Scope </br> (select and verify) 	| Data and code exist?                                         	|
| Assess            	| Degree of reproducibility for <br>specific part of the paper 	|
| Improve           	| E.g. fixed paths, libraries,<br>added missing files, etc.    	|
| Test robustness   	| Results are robust to additional <br>specifications          	|

.font120[
.center[**Key challenge: Standardization**]
]

---
count: true  
background-image: url(https://media.tenor.com/4D-9tETtpFIAAAAd/football-goal.gif)
background-size: contain
# Motivation II: Prevent Loss of Knowledge

[](https://media.tenor.com/4D-9tETtpFIAAAAd/football-goal.gif)
---
count: true  
background-image: url(Images/paper-claims.svg)
background-size: 610px
background-position: 75% 0%
  
# Standarized Scope of a Reproduction

.pull-left[
.font100[
Each **reproduction attempt** is centered <br>
around scientific **claims** (following [SCORE](https://replicats.research.unimelb.edu.au)).

One paper can contain several claims.

Each claim may be supported by various </br>
**display items**: tables, figures & inline results.

A reproduction attempt is at the claim level, <br>
and reproducers must record their <br> 
**specifications** of interest.

<br><br>

]
                                
                                  
                                               DI: Display Item 
                                                S: Specificaiton
]

---
background-image:  url(Images/stages.svg)
background-size: contain

# Stages

---
background-image: url(Images/assess_example2.png)
background-size:  70%
background-position: 50% 50%
count:true

# Standzarized Assessment

---
background-image:  url(Images/home_page.png)
background-size: contain
count:true

.center[
# Demo: [socialsciencereproduction.org]()
]
---

# Try It Yourself!
.font130[

- Create an account

- Think of a paper (go to a Journal and choose a random paper if you can't think of a particular paper)

- Check if it's already in the SSRP

- If not, start a new entry

- Quickly verify if there is any reproduction package available and declare as abandoned if there is none

- If there is a reproduction package continue the reproduction later on!
]
---
background-image: url(Images/scope.svg), url(Images/scope_paper.png)
background-size: 400px,  700px
background-position: 95% 100%, 0% 10%
count:false

# Scope

---
background-image: url(Images/scope.svg), url(Images/scope_paper2.png), url(Images/scope_paper.png)
background-size: 400px, 500px, 700px
background-position: 95% 100%, 90% 0%, 0% 10%
count:false

# Scope

---
background-image: url(Images/scope.svg), url(Images/scope_example.png), url(Images/scope_paper2.png)  , url(Images/scope_paper.png)
background-size: 400px,  600px, 500px, 700px
background-position: 95% 100%,  100% 75%,  90% 0%, 0% 10%
count:true

# Scope

---
background-image: url(Images/assess.svg), url(Images/assess_paper1.png)
background-size: 400px, 120px
background-position: 95% 100%, 0% 10%
count:false

# Standzarized Assessment 
---
background-image: url(Images/assess.svg), url(Images/assess_paper2.png)
background-size: 400px,  120px
background-position: 95% 100%, 0% 10%
count:false

# Standzarized Assessment

---
background-image: url(Images/assess.svg), url(Images/assess_example1.png), url(Images/assess_paper2.png)
background-size: 400px,  500px, 120px
background-position: 95% 100%,  15% 50%, 0% 10%
count:false

# Standzarized Assessment

---
background-image: url(Images/assess_example2.png), url(Images/assess.svg), url(Images/assess_example1.png)  , url(Images/assess_paper2.png)
background-size:  600px, 400px,  500px, 120px 
background-position: 100% 80%,95% 100%,    15% 50%, 0% 10%
count:true

# Standzarized Assessment

---
background-image: url(Images/improve.svg)
background-size: 700px
background-position: 50% 20%

# Standarized Improvements

</br></br></br></br></br></br></br></br>

## Three types of improvements:

.font130[

1. Improvements at the paper level (e.g., improve documentation following [TIER's protocols](https://www.projecttier.org/tier-protocol/specifications/#overview-of-the-documentation))
2. Improvements at the display-item level  
3. Specific future improvements   
 ]

---
background-image: url(Images/robust.svg)
background-size: 700px
background-position: 50% 20%

# Robustness Checks

</br></br></br></br></br></br></br></br>

## Two main parts for robustness:

.font130[
1. Increase the number of robustness checks
2. Justify the appropriateness of a specific test
 ]
 
---
background-image: url(Images/robust.svg), url("Images/robustness checks.svg")
background-size: 300px, 500px
background-position: 0% 100%, 100% 60%

# Robustness
.font120[
**Robustness checks:** any possible change in a computational choice, both in data analysis and data cleaning
]
--

.font110[
.pull-left[
**Reasonable specifications** ([Simonsohn et. al., 2018](https://urisohn.com/sohn_files/wp/wordpress/wp-content/uploads/Paper-Specification-curve-2018-11-02.pdf)):

1. Sensible tests of the research question
 2. Expected to be statistically valid, and 
 3. Not redundant with other specifications in the set.

Reproducers will be able to record two types of contributions: 
- Mapping the universe of robustness checks
- Proposing a specific robustness check
]
]
---
background-image:  url(Images/completed_repro1.png)
background-size:   450px
background-position:  0% 0%   
count:false
# .font90[Completed Reproduction: Generate Knowledge and Get Credit]

---
background-image: url(Images/completed_repro2.png), url(Images/completed_repro1.png)
background-size:  300px, 450px
background-position:  35% 0%, 0% 0%   
count:false
# .font90[Completed Reproduction: Generate Knowledge and Get Credit]

---
background-image: url(Images/completed_repro3.png) ,url(Images/completed_repro2.png), url(Images/completed_repro1.png)
background-size:  600px ,  300px, 450px
background-position:  100% 10%, 35% 0%, 0% 0%   
count:true
# .font90[Completed Reproduction: Generate Knowledge and Get Credit]

.pull-right[

.font130[
 <br><br><br><br><br><br><br>
 
 - **Share it:** Original authors, instructors, other researchers, add it to your CV
 - **Discuss it:** Discourse forum for reproductions
 - **Cite it:**    Each reproduction has a citable DOI
 - [Example here](https://www.socialsciencereproduction.org/reproductions/40/published/index)
 ]

]

---
background-image:  url(Images/use1.png)
background-size:  500px
background-position:   0% 0%   
count:false
# How Do I Get Started?

.center[
.font120[
    &nbsp; &nbsp;&nbsp;&nbsp;&nbsp;  [**Use it**](https://www.socialsciencereproduction.org) for your class <br>
&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;or independent project
]
]

---
background-image: url(Images/use2.png), url(Images/use1.png)
background-size:  600px, 500px
background-position:  35% 100%, 0% 0%   
count:false
# How Do I Get Started?

.center[
.font120[
    &nbsp; &nbsp;&nbsp;&nbsp;&nbsp;  [**Use it**](https://www.socialsciencereproduction.org) for your class <br>
&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;or independent project
]
]

<br><br><br><br><br><br><br><br>
.font120[
Consult the <br> supporting <br>
[**guide**](https://bitss.github.io/ACRE/)
]

---
background-image: url(Images/use3.png) ,url(Images/use2.png), url(Images/use1.png)
background-size:  400px,  600px, 500px
background-position:  100% 0%, 35% 100%, 0% 0%   
count:true
# How Do I Get Started?

.center[
.font120[
    &nbsp; &nbsp;&nbsp;&nbsp;&nbsp;  [**Use it**](https://www.socialsciencereproduction.org) for your class <br>
&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;or independent project
]
]

<br><br><br><br><br><br><br><br>
.font120[
Consult the <br> supporting <br>
[**guide**](https://bitss.github.io/ACRE/)
]
.right[
.font120[
[**Ask**](https://forum.socialsciencereproduction.org) questions
]
]

---
background-image: url(Images/paper-title.png)
background-size: contain
count:true
# First Insights Coming Out of the SSRP

---
background-image: url(Images/paper-claims.svg), url(Images/table-2.png)
background-size: 200px, contain
background-position: 90% 100%, 50% 50%
count:true
# First Insights: Some Utilization

---
background-image: url(Images/figure-2.png)
background-size: contain
count:true
# First Insights: Distribution of Scores for Display Items

---
background-image: url(Images/figure-3.png)
background-size: contain
count:true
# Can Aggregate at Paper Level In Different Ways

---
class: inverse, center, middle

# .font140[ Thank You]

<html><div style='float:left'></div><hr color='#EB811B' size=1px width=100%></html>

.font190[
<acre@berkeley.edu>  
<fhoces@berkeley.edu>
]